Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The manuscript proposed a Generative Communication Model by individually modeling the signals (by an S-VAE) and the communication sent at each graph node (by an RNN+MLP). Experiment on synthetic data shows that the model can successfully recover the designed communication between regions. Experiment on real (HCP) fMRI data shows good classification accuracy of the task type, and meaningful inter-region communication patterns can be discovered. 
strength_and_weaknesses: Strength: Explicit modeling of the inter-region communication is both interesting and important for the fMRI data analysis and, more generally, the majority of multi-variate time series data analysis. The design of the model by S-VAE and RNN is reasonable and well-implemented. 

Weakness: First of all, it will be important to compare the proposed model with the “traditional deep learning” approaches. Specifically, it will be interesting to see ablation experiments of 1) training an S-VAE using the signals from all regions as input and output (with certain regularizations) and 2) removing the communication model to train the individual S-VAEs only. The reviewer is also curious about how the proposed model would scale to the region (node) size, as the current experiment on real data includes a relatively small (<20) number of regions. This concern is especially prominent as 1) the manuscript did not provide computational cost for training the model, and 2) each node will have its own S-VAE and RNN+MLP to be trained. The final concern is more of a naming convention issue, as generally, the communication model developed in this work is not considered “Generative.” Rather, it performs a binary classification on whether a specific inter-region communication will happen at a specific time point. The methodology section of this manuscript also suffers from clarity issues; see comments below. 
clarity,_quality,_novelty_and_reproducibility: Clarity: While overall, the manuscript is easy to read and the figures (especially Fig. 1) help a lot in understanding the design of the model, the methodology section can be improved for its organization. Specifically, the section “Generative communication models” is more like an introduction. Further, the paragraph led by the sentence “One recent model that has been used to model this dynamical system is LFADs…” mixed the S-VAE and RNN+MLP models together, making it difficult to understand each component clearly. Finally, equations 8-9 can use more explanations for the symbols used and their relationships to the corresponding models. 

Quality: The quality of the manuscript is good, and the results are well-supporting the authors’ claims. 

Reproducibility: In addition to the public source code promised by the authors, the proposed model shall be reproducible based on the description in the methodology and the Implementation sections. 
summary_of_the_review: An interesting and important attempt to model the connectivity of fMRI/time series data via explicit modeling of the inter-region communication. More investigations into how and why the model can work will be needed.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper models the fMRI activity of multiple brain regions simultaneously, with sparse communication between them. 
strength_and_weaknesses: This paper seems like a reasonable extension of the popular neural modeling approaches (ex. LFADS) to fMRI and multiple brain regions. 

I have some reservations:
1. I am not confident on the identifiablity of the model. If the generator for each region is expressive enough, it might not be necessary to have any communication from other brain regions. 

2. I am not confident of the instantaneous sparsity as biologically realistic. If region A is not connected with region B, then the communication will be 0 for all time points. This points to some sort of group sparsity over time/trials. 

3. Do you recover the ground-truth communication on individual trials? In simulations (Fig 3), all results are averaged across multiple trials. In my experience, it is hard to get a truly sparse signal as posterior for individual trials in VAE setting. The authors should clarify whether the sampled posterior is sparse or not.

4. The fMRI analyses seem weak. The biological insights seem mixed and not too clear. 
clarity,_quality,_novelty_and_reproducibility: Writing is clear. 
Novel neural modeling architecture, but the analyses are not too convincing. 
summary_of_the_review: More analyses on identifiability, or more empirical analysis would be helpful. At the current state, the paper seems more suited for a neuroscience journal after using it for scientific insights.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This work proposes new generative model of communication in graph networks, applied here to fMRI data. The network attempts to overcome weaknesses of purely correlational approaches such as pearson correlation by explicitly modeling the directional communication between different nodes in the network, and giving control of the sparsity of communication between nodes. In the model, each node possess two networks, the first a S-VAE that reconstructs the signals based on the observed data and the estimated communication and task events, and the second a network that determines whether to communicate with another node using a binary signal. This allows for the unique directional information from each node to be precisely delineated, and to regularize the sparsity of communication using a Laplacian constraint. The manuscript offers a simple example of how the model could work in practice and then applies it to an fmri dataset consisting of 3 emotional, language, and motor tasks. Because there is no ground truth for the fmri data, they look at the ability to classify tasks based on the estimated communication dynamics, and compare this to the classification ability using dFNC, a pearson correlation based approach. They find superior performance of the generative model when using shorter time windows for the dFNC approach. Lastly, they demonstrate reconstruction of a communication graph for each task for the S-VAE approach, and find some concordance with biological intuition. 
strength_and_weaknesses: 1) The model is well described and the experiments have clear descriptions for reproducibility wrt the data used and network training parameters. 
2) The model explicitly incorporates the inductive bias of sparsity, and models directional binary communication between regions. 
3) There are experiments with real data and synthetic data to demonstrate the model quality and types of insights it can deliver. 


Weaknesses
1) Paucity of benchmarks and comparisons. I found the approach interesting, but comparing a generative model to a windowed pearson correlation felt lacking. It would be nice to see other metrics considered, especially granger causality which is often used or transfer entropy. This would be especially nice on a benchmark dataset or synthetic dataset for causal prediction, but also for the directional communication graphs in Figure 5. 
In the LFADS paper that is mentioned for instance, LFADS is compared for latent factor estimation against comparable techniques in a fairly convincing synthetic data experiments. 
One of the challenges in evaluating the manuscript is that while the model will certainly produce a set of communication values when applied to a dataset, it isn’t clear in what data and parameter regimes it is effective. The model will give some answer when applied to a dataset, but it isn’t clear 
2) I found the description of the synthetic data could be better. How were the synthetic experiments prepared and what is the alignment metric that is used? I found the description, especially the post hoc alignment of data confusing. What are the individual data points in Figure 3? 
3) In figure 4, unless I am incorrect the VAE model has been trained on the tasks before hand, so there is a greater amount of information stored in the network weights. This may contribute to accuracy classifying tasks, but also likely introduces some amount of bias. 


Nits: 
Reference formatting is weird – names are not separated from the text.
Figure 2 – in the cartoon there is no noise in the trace for R3, and the noise appears hand drawn (the rise in signal at T1 for R1 is not a function/does not pass the ‘vertical line test’). 
Pg 4 after eq 7 typo ‘intense’
Pg 4 bottom missing equal signs
Figure 3A – what is the definition of this metric
“We train one model on all three tasks such that the prediction of the task is not influenced by the fact that the model was specifically trained to model communication for that task” – technically it is still influenced by the task, just less so. 
In the introduction the authors mention an advantage over correlational approaches to function over shorter timescales, but this approach requires training data and time to initialize the state of RNNs and account for time lags, eg in the Figure 2 cartoon there is a time gap in communication that is claimed to be learned by the model. 


clarity,_quality,_novelty_and_reproducibility: I found the manuscript hard to follow in sections, for instance the intuition section and the model description, where the notation can be hard to follow and may benefit from a table of parameters in the appendix. The descriptions are adequate to reproduce the results and while the code is not yet available it is slated to be posted.  I do believe the model is novel. 
summary_of_the_review: I am a bit torn with this manuscript. I think that the model holds some value and has a structure (directionality, explicit communication) and inductive biases (sparsity) that could be useful , but I have reservations about the quality of presentation and comparisons given. Without benchmarks for synthetic data or better sense of how the model compares to other approaches it is unclear when it should be used. All models are wrong, some are useful, and it would be nice to know the exact circumstances and advantages this model has – in practice – compared to other approaches. I am inclined to reject unless there is significant revisions wrt benchmarks and clarity of exact situations where the model should and shouldn’t be deployed.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: In this paper, the authors propose a method for non-linear generative communication from human brain functional data, addressing three problems with connectivity approaches by explicitly modeling the directionality of communication, finding communication at each timestep, and encouraging sparsity in communication. They simulate temporal data to evaluate the proposed method that has sparse communication between embedded nodes. The proposed model can predict the expected communication dynamics. Overall, I found this paper to be difficult to follow, and I am not sure how the proposed method is novel with regard to machine learning.
strength_and_weaknesses: Please refer to the review section.

clarity,_quality,_novelty_and_reproducibility: Please refer to the review section.
summary_of_the_review: a) There may be some practical applications of this paper, but it is mainly ambiguous whether its contribution(s) will be accepted as part of the machine learning area. There seems to be just a combination of some machine learning algorithms in this paper, which are then applied to a short set of data in the paper.

b) Besides the syntactic data, the only real fMRI dataset used to evaluate the proposed method is HCP-1200. For you to be able to evaluate a new approach, you will have to use data from a variety of sources.

c) I found this paper hard to follow as well. Some parts of the problem formulation are hard to understand. 

d) The authors mentioned modeling the cognitive pathways from white matter — i.e.,  these regions communicate through signal propagation along the cortex and along white matter tracts over longer distances. Several theories explain fMRI may not be a good proxy to shed light on the white matter functional activities. Instead, many studies used DTI to model the regions’ communications. As an example, please see the following paper: “Learning Macroscopic Brain Connectomes via Group-Sparse Factorization”. Authors should clarify the scientific fundamental behind their proposed theory/approach.

e) The performance of the proposed method should be compared with similar state-of-the-art approaches.

f) There are some minor linguistic and typo problems in this paper.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper proposes modeling fMRI measurements with latent dynamical system expressed in terms of a macroscale communication dynamics using coupled sequential VAEs. The latent structure of the communication is then interpreted. The reviewers found the premise that causal communication links could be inferred from extremely coarse (poor spatial and temporal resolution), and purely observational measurements such as fMRI to be implausible. Further, no evidence is given to validate the predictions of the model. The reviewers found the model to be of questionable utility to both neuroscientists and machine learners.
justification_for_why_not_higher_score: Lack of utility and validation.
Decision: Reject