Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: The paper presents a deep Markov factor analysis (DMFA), which is a generative model that employs Markov property in a chain of low dimensional temporal embeddings as well as spatial inductive assumptions in order to capture temporal dynamics in functional magnetic resonance imaging (fMRI). The paper shows that DMFA has capability to cluster fMRI data in its low temporal embedding in regards to subject and cognitive state variability. The advantages of DMFA are demonstrated both through synthetic and application data.
main_review: The paper meets and exceeds the criteria of originality, quality, clarity and significance. The paper clearly indicates previous methodologies in the direction of their method (i.e., factor analysis and dynamic factorization), while providing ample citations for each. The development of DMFA is clearly presented with explicit and easy-to-understand diagrams. The estimation method (variational inference) is also clearly presented and makes it easy to reproduce for any interested practitioner. The simulation experiments show the clear utility of the  method for the generative model to learn from complex spatiotemporal dynamics presented in fMRI data. And finally, several application datasets are presented (e.g., in Autism, Depression, Ablation, etc.). The reviewer sees no major issues with the current work, and sees the paper as well-polish and an excellent example of a high-quality submission.

But due to some comments from other reviewers, I do see some significant issues with methodology/innovation, as well as a lack of proper sensitivity analysis. Therefore I will reduce my score.
limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The authors present deep Markov factor analysis (DMFA) to capture temporal dynamics in functional magnetic resonance imaging data. The method chosen by the authors is relevant since existing methods overlook the highly nonlinear and complex temporal dynamics of neural processes when factorizing their imaging data. The authors claim that their method (DMFA) is able to cluster fMRI data in its low dimensional temporal embedding, which enables validation of fMRI related hypotheses.
main_review: Reasons to accept:

- If the authors claims are correct, the study is ground breaking in fMRI data analysis since the current models do not encode temporal dynamics. The voxel activations have non-linearity and inherent temporal dependencies, which are explored using the methods mentioned by authors.

- The paper is well structured and explained. The theory behind Deep Markov Factor Analysis (DMFA) is explained perfectly.

- The authors thoroughly explored the current state-of-the art related to the paper. The comparisons are clear in the text, table as well as the figures.

- Limitations an ablation study are explained clearly.

- Ablation study is clearly explained by the authors.

 

Reasons to decline:

- Are the claims mentioned in following lines correct? The authors mentioned in line number 95-98 that RNN-based methods do not estimate any spatial factors, and are harder to

train on long sequences. Is this verified on Convolutional RNN models (Wang et al., 2019) and LSTM/GRU models (Yan et al., 2019)? Are there no recent papers using transformer based models?

- Better visualizations. Hard to interpret Figure 2 (b) and Figure 3 (Middle) images.
limitations_and_societal_impact: Summary:

The authors explored a relevant problem and can be widely used in neuroimaging to transfer high dimensional imaging data into low dimensional and interpretable representations.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper introduces Deep Markov Factor Analysis (DMFA). DMFA can model spatiotemporal BOLD fMRI time series by using time-varying latent factors that govern time-varying spatial maps consisting of spatial mixtures of RBFs. Latent time series are associated to a categorical clustering variable in order to enable grouping. Certain links in the Bayesian diagram are modeled using deep neural networks. Optimization is done by a variational approach.
Experiments are performed on a simulated data set with known ground truth, the ABIDE autism data set, and a Depression study data set. Some clustering of relevant clinical outcomes is observed.



main_review: DMFA lies at the intersection of the Topographical Factor Analysis approach (ref 14), unsupervised TV-regularized resting-state analysis as performed in (https://www.nature.com/articles/ncomms8751) and variational autoencoders for time series (https://arxiv.org/abs/1506.02216). The latter two papers are not mentioned.

Several concerns arise around this work:

What is the point in using Gaussian blobs to model spatial maps? The argument that the idea is inherited from TFA is insufficient. Why not just use voxel maps at a certain resolution? Not all brain regions are shaped like spherical Gaussians or sparse sums thereof. The reconstructed maps show this very clearly -- pretty much all detail is lost.

What is the point of *temporally varying* spatial Gaussian blobs? Surely brain regions remain in place over time. One could make the argument that “activity moves around”, but it remains true that certain spatially fixed regions receive more or less blood over time. It seems that this feature adds unnecessary complexity and hinders interpretability

The synthetic model is not very helpful since it basically generates data according to the model used. Why not generate some fake brain data based on a different model, like ICA components?

The experimental evaluation does not seem to contain any clearly stated numbers enabling comparisons to other methods. It is great that the method finds some factors in an unsupervised manner, but please state the accuracy over the full datasets for the task. It may be helpful to look at Abraham et al 2017 (ref from the paper) for the data curation procedure as this is an important thing to get right.

Why is there no comparison to the method presented in (https://www.nature.com/articles/ncomms8751), adapted to this task? This provides a simpler baseline.

limitations_and_societal_impact: There is no reason to believe there would be a negative societal impact to this work. That there would be a strongly positive one is also unlikely.

Limitations were not really discussed.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: Extending topographical factor analysis (TFA), this paper proposes a model, deep Markov factor analysis (DMFA), that models temporal dynamics in embedding space. The model is applied to both simulated and real fMRI data.
main_review: Originality

The proposed model is a novel extension of TFA with implementation details carefully worked out.


Quality

The method seems solid, but the results are not as impressive. A few comments below.
1. Is TFA always better than parcellating and averaging? Parcels are often generated from external datasets whereas latent representation are often built from datasets that are being analyzed. Would the former provide cleaner prediction evaluation?
2. How is using Gaussian blob more interpretable than spatial ICA, which tends to generate biologically interpretable networks?
3. The current simulation seems to be based on assumptions of the proposed model, and model hyperparameters are set to the ground truth. Also, quantitative comparison against TFA, HTFA, NTFA, and RKN should be provided.
4. How were hyperparameters selected for real data?
5. DMFA only beats ROI/ICA + SVM for some ABIDE datasets. Using p<0.02 as threshold would not survive multiple comparison correction for the number of datasets, i.e. 0.05/10 = 0.005. 
6. What model is used for estimating likelihood in Table 2? Is likelihood estimated per subject? What is the standard deviation? If consider the errorbar, is DFMA’s Nats significantly lower than other methods?


Clarity

The paper is clear and concise. A few minor comments below.
1. Would ST-GCN fall under the scope of this paper? If so, please compare.
2. More description on TFA would be useful for readers to get a better sense of the mentioned limitations.


Significance

The model would be useful for modeling any spatiotemporal data, but the results do not convincingly show better performance over existing methods.


Post Author Responses

I have read through the other reviews and responses, and will keep my original score mainly since classification improvements are only seen for half of the tested datasets.
limitations_and_societal_impact: An elaborate discussion societal impact is provided in the paper. Limitation on implementation is also provided.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper proposes deep Markov factor analysis (DMFA) that uses the Markov process to capture the temporal dynamics in the fMRI dataset and maps the high spatial dimensions to a low-dimensional feature space. DMFA can cluster fMRI responses into “low dimensional temporal embedding” based on subject or cognitive state. The empirical studies on synthetic and real fMRI datasets illustrate that DMFA generates better performance in comparison with state-of-the-art techniques --- such as NTFA and HTFA.
main_review: The following are the major concerns and minor comments:

1) My first question is about the prior and trail distribution(s). You can replace the combination of Normal distribution and RBF kernel with the Gaussian Process (Wu, 2019, J. Machine Learning Research) or even a Deep Gaussian Process (Roger Frigola-Alcalde, 2015, Thesis at University of Cambridge). The author(s) should explain why they chose these assumptions and how they can affect the performance of the analysis.

2) The proposed method is compared mostly with the same type of factor analysis. I would like to see comparisons with other paradigms of analyzing using similar datasets. For instance, comparing the performance of the proposed method with the graph-based approach (e.g., Mingliang Wang et al., 2021, Medical Image Analysis) or functional connectivity-based ones --- e.g., Lei Sun et al. 2021 Artificial Intelligence in Medicine.

3) The proposed method can be summarized in the form of an algorithm --- perhaps at the end of Section 3.

4) In this paper, the notations are confusing. In the regular papers, scalers are denoted by small letters, vectors are defined with small letters (highlighted by bold), matrices are denoted by capital letters using bold. In this paper, they are a lot of conflicts. It is so hard to trace what is a set, a matrix, or even a distribution.

5) The equations should have numbers. It is hard to trace them in the current format.

6) In this paper, "embedding" is used to describe the temporal low dimensional space. Mathematically, embedding spaces have greater dimensions than the original spaces (not lower). For clarity, I suggest the phrase "low-dimension feature space" be used here instead.

7) The brain images in Figures 2--4 and S1 should have value bars to make it clear what each color represents. Additionally, some subfigures do not have legends. The font size in these figures should also be increased.

limitations_and_societal_impact: Please see the main comment section.
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: The paper Deep Markov Factor Analysis (DMFA): Towards Concurrent Temporal and Spatial Analysis of fMRI Data proposes a scalable approach to factorize the data of each subject into a product of a small number of spatial and temporal components.
Following on the work of (Manning et al., Topographic Factor Analysis, Plos ONE 2014) on topographical factor analysis, each spatial component is given by a radial basis function. The temporal components are modeled from a set of latent satisfying the Markov property. The first latent is conditioned on the data class giving to the model clustering capabilities. Variational inference is used to derive a lower bound of the log-likelihood which is then optimized via stochastic gradient descent.
In a first experiment on resting state data (Autism dataset), DMFA is used to obtain subject specific atlases that allow to seggregate ASD and control subjects with an accutacy of 60 % (it is definitely hard to read).
On a second experiment (Depression dataset), MDD and controls are exposed to different musical stimuli and DMFA is able to seggregate well the data of different subjects without supervision. In some extend it gives some hint about the kind of stimuli and the conditions of the subjects but results are not so convincing (the authors are honest about this). 
In a third experiment they show that DMFA gives better results than other methods in terms of held-out log-likelihood and prediction of the next time point given the history.
The code for DMFA is available. Pre-trained model are given and the code to generate the figures from pre-trained model works.

main_review: # 1. Originality
The model proposed by the authors uses the same spatial prior as in (Manning et al., Topographic Factor Analysis, PloS one 2014) and a temporal prior similar to (Krishnan et al. Deep kalman filters, 2015.).
The combination of these two techniques is new.
I find that the related work section is well done overall and I find that the difference with previous work is clearly stated. In the section "Factor Analysis in fMRI" I would have added techniques related to IVA (Lee JH et al. Independent vector analysis (IVA): multivariate approach for fMRI group study., Neuroimage 2008), MultiViewICA (Richard Hugo et al., Modeling Shared responses in Neuroimaging Studies through MultiView ICA, NeurIPS 2020) or TensorICA (Beckmann et al., Neuroimage, 2005).

# 2. Quality
## 2.1 Sensitivity analysis needed
The proposed method has many hyper-parameters to tune (number and size of hidden layers, size of latent factors,  number of spatial maps to cite just a few). Some of these parameters vary from one experiment to the next such as  the size of hidden layers between synthetic and real experiments or the value of $\sigma_0$ between the autism and depression dataset.
Ideally, it would be nice to have sensitivity curves for all hyper-parameters but the fact that there are no sensitivity curves for the hyper-parameters that vary across experiments is problematic I think.

## 2.2 Autism experiment
1. Unless I missed something, the model is fit on the full dataset (train and test set) so that the time-courses of interest can be recovered in all subjects. I find this worrying because there is leakage of information from the test set. Ideally, the parameters of the model would be learnt on the train data and fixed when the test data are used. However with DMFA this is not possible because there is one set of parameters per subjects. I suggest here another experiment using naturalistic stimuli that I believe yields a cleaner pipeline: the authors could use the forrest dataset where subjects listen to an audio version of forrest gump (Hanke et al., Scientific data, 2014) to learn the spatial representations. Then they could use the second part of the dataset where the same subjects are listening to music of different genre and attempt to recover the music genre from the projected data.
2. I find it quite surprising to use temporal ICA sources to build a functional connectivity matrix. These sources are typically independent and in most ICA algorithms, they are normalized so the functional connectivity matrix should always be identity unless I am wrong. Maybe it makes more sense to use spatial ICA sources to get the atlases and then project the data onto these atlases ?
3. A standard baseline would be to use the 444 regions of the BASC Atlas (Bellec, Multi-level bootstrap analysis of stable clusters in resting-state fmri., NeuroImage 2010) to get connectivity matrices before SVM is applied on them (with python the atlas is easy to get using nilearn https://nilearn.github.io/modules/generated/nilearn.datasets.fetch_atlas_basc_multiscale_2015.html ). I fill it is a more natural baseline than the ones tried by the authors.
4. The result for DMFA (without SVM) are missing when the full dataset is used. Why did the authors make this choice ?
5. I would have expected to see the results of competitive unsupervised methods (NTFA, HTFA) as well as the ablation study done here so that we can compare the methods based on the decoding accuracy they yield.
6. The authors state that competitive methods for the same task reach 69 % accuracy (l256). I believe it would be fair to display this information in Figure 3 so that we can actually see where DMFA stands (I believe DMFA is below 69 % isn't it ?)

## 2.3 Depression experiment
There is clear evidence that the different subjects are in different clusters. However, what is interesting here in my opinion is whether the model can retrieve the condition of the different subjects (or some characteristics of the displayed stimuli) better than alternatives. About the condition of subjects authors say: (l284) "MDD group data points are fairly populated in the center of temporal latent while control group are dispersed across latent space". I find this statement a bit fuzzy. As it is, this experiment does not allow me to formulate a clear conclusion about whether the model works better than alternatives or not. 

## 2.4 Comparison with state of the art
I am not sure I fully understood this part. 
1. This comment asks for more pedagogy. The likelihood is intractable. How can the authors evaluate it ?
2. Leaving aside the problem of information leakage from the test set, I think the previous experiments on task fMRI or rest fMRI data provide more classical benchmark in neuroscience and more importantly yield results that are relevant in clinical applications (if a method can classify better than alternatives whether a patient is ASD or not this would really yield practical benefits).  Although the held-out likelihood / prediction of next sample suggests that DMFA is a better generative model, I find it hard to know whether these metrics are really reliable and if it is of any use in neuroscience.

# 3. Clarity

I find this submission extremely difficult to follow.

## 3.1 Notation and organisation issues
1. The notations become clearer in Table 1 but when reading section 3, I don't know the dimensions of the latent variables $z_n^F$, the domain and co-domain of function $\mu_{\theta}^W$ etc. The space in which any parameter leaves should be defined straight away at least in the text if it is not done directly in the formulas. This is really an important point as it makes the submission difficult to understand.
2. The notations $\mathcal{N}$ and $Norm$ are redundant unless I missed something. All of these notations should be properly defined in a notation section.
3.  Are $\sigma_{\theta}^F$ matrices ? I believe so but I am not sure. Matrices, vectors and reals should have a different notation (using uppercase, bold and lowercase for example). Another example  l132 $g$ lives in $[0, 1]$ but in Table 1 it lives in $[0, 1]^2$. I understand these are typos but with clearer notations, it becomes obvious to know what is correct.
4. I believe $\sigma_0$ is never defined.

## 3.2 Motives for modelization choices
1. Having a common name for groups of parameters such as $\theta$ or $\phi$ may be practical but I find it confusing in the sense that it hides the fact that some parameters are shared and some are not. In general it is not always clear which parameters are shared or not. The notations give some hints but it would be better if this was explicitely stated in the text. In addition, I believe it is important to understand why these choices are made. For example, you assume that the spatial components are subject specific as well as the temporal components. These assumptions are not made in all models (for example in (Calhoun, A method for making group inferences from functional MRI data using independent component analysis.,  Human Brain Mapping 2001) the temporal part is assumed to be shared across views). Other parameters are shared (such as the neural network functions) and we might want to know why.
2. Table 1 describes a particular architecture. It would be nice to understand how the authors end up chosing this particular one and what is the performance of other choices.

## 3.3 Readability
1. The figures are too small especially Figure 3 (right) or the legend in Figure 4 (a). In Figure 3 (right) this is particularly problematic as it is difficult to read the accuracy values. A grid might also help.
2. Table 1 is difficult to understand. There is this layer column which goes down to 10 layers whereas if I understand correctly, none of the networks have more than 1 hidden layer.
3. In order to facilitate the reading of the derivation of the ELBO it would have been nice to see a more detailed version in appendix.
4. (l309) "trained each with their default hyperparameter": what are the defaults hyper-parameters for DMFA ? For other models ? According to which paper or which package ?

## 3.4 Reproducibility
1. l226 "These three groups of sources are periodically activated according to some random weights". The type of randomness should be precised.
2. In the depression dataset there is no value for $S$. 
3. I find it difficult to reproduce the figures from scratch using the code as it involves downloading the full dataset and pre-processing it myself (in the README there is no script to get the dataset although there seems to be one to do the pre-processing).  What I would have hoped for is a small synthetic experiment that I can run in a short amount of time and is just there to show me that the code works.

# 4 Significance
1. Ignoring the fact that some information may leak from the test set, results displayed in Figure 3 are only marginally better than SVM after applying an atlas. For a method that takes 12 000 minutes to run, I think we need a marked improvement if we want other researchers to use it. 
2.  However, I would like to highlight that with fMRI data it is quite difficult to study the temporal structure when the TR is as high as 2s. I would suggest the authors to take datasets where the TR is much lower such as (Feinberg, et al. "Multiplexed echo planar imaging for sub-second whole brain FMRI and fast diffusion imaging." PloS one 2010) so that they have a higher chance of seeing strong effects.


# 5 Minor comments
l157 The comment that amortized variational inference could lead to over-fitting is wrong in my opinion. Amortized variational inference is strictly less powerful than variational inference as in amortized variational inference the parameters are restricted to leave in the image of the parametrized function while in variational inference they are completely free.   
limitations_and_societal_impact: Authors have a broader impact section in appendix.
review_title: null

Meta-review: Dear authors,

the overall consensus is that your work has merit in terms of method and experiments
although there are a number of concerns about some limited comparison with alternative
baseline approaches. Despite this concern I am inclined to flip the decision on the
positive side as code is available and you can still insert some numbers in the camera
ready as proposed.

Best regards,
The AC

Decision: Accept (Poster)