CommsVAE: Learning the brain's macroscale communication dynamics using coupled sequential VAEs
Authors: Eloy Geenjaar
Noah Lewis
Amrit Kashyap
Robyn Miller
Vince Calhoun
Venue: Submitted to ICLR 2023 (2023)
Keywords: variational autoencoder
computational neuroscience
graphs
fMRI
sequential variational autoencoder
graph learning
communications

Abstract: Communication within or between complex systems is commonplace in the natural sciences and fields such as graph neural networks. The brain is a perfect example of such a complex system, where communication between brain regions is constantly being orchestrated. To analyze communication, the brain is often split up into anatomical regions that each perform certain computations. These regions must interact and communicate with each other to perform tasks and support higher-level cognition. On a macroscale, these regions communicate through signal propagation along the cortex and along white matter tracts over longer distances. When and what types of signals are communicated over time is an unsolved problem and is often studied using either functional or structural data. In this paper, we propose a non-linear generative approach to communication from functional data. We address three issues with common connectivity approaches by explicitly modeling the directionality of communication, finding communication at each timestep, and encouraging sparsity. To evaluate our model, we simulate temporal data that has sparse communication between nodes embedded in it and show that our model can uncover the expected communication dynamics. Subsequently, we apply our model to temporal neural data from multiple tasks and show that our approach models communication that is more specific to each task. The specificity of our method means it can have an impact on the understanding of psychiatric disorders, which are believed to be related to highly specific communication between brain regions compared to controls. In sum, we propose a general model for dynamic communication learning on graphs, and show its applicability to a subfield of the natural sciences, with potential widespread scientific impact.

Under review as a conference paper at ICLR 2023
COMMSVAE: LEARNING THE BRAIN'S MACROSCALE COMMUNICATION DYNAMICS USING COUPLED SE- QUENTIAL VAES
Anonymous authors Paper under double-blind review
ABSTRACT
Communication within between complex systems is commonplace in the nat- ural sciences and fields such as graph neural networks. The brain is a perfect example of such complex system, where communication between brain regions is constantly being orchestrated. To analyze communication, the brain is often split up into anatomical regions that each perform certain computations. These regions must interact and communicate with each other to perform tasks and sup- port higher-level cognition. On macroscale, these regions communicate through signal propagation along the cortex and along white matter tracts over longer dis- tances. When and what types of signals are communicated over time is an un- solved problem and is often studied using either functional or structural data. In this paper, we propose a non-linear generative approach to communication from functional data. We address three issues with common connectivity approaches by explicitly modeling the directionality at each timestep, and encouraging sparsity. To evaluate model, we simulate temporal data that has sparse communication between nodes embedded in and uncover the expected communication dynamics. Sub-
communication, finding communication
show that our model sequently, we apply our model to temporal neural data from multiple tasks and show that our approach models communication that is more specific to each task. The specificity of our method means it can have an impact on the understanding of psychiatric disorders, which are believed to be related highly specific communi- cation between brain regions compared to controls. In sum, we propose general model for dynamic communication learning on graphs, and show its applicability to subfield of the natural sciences, with potential widespread scientific impact.
INTRODUCTION
Characterizing macroscale communication between brain regions is a complex and difficult prob- lem, but is necessary to understand the connection between brain activity and behavior. The effect of neural systems each other, connectivity, has also been linked to psychiatric disorders, such as schizophrenia FFriston 2002). Hence, gaining a deeper understanding of the dynamics underly- ing the communication between brain regions is both from the perspective of understanding how brain facilitates higher-order cognition and also to provide insight into and consequently how psychi- atric disorders arise. Static functional network connectivity (sFNC) and dynamic functional network connectivity (dFNC), computed respectively as the Pearson correlation between regional activation timeseries over the full scan duration (sFNC) or on shorter sliding windows (dFNC), are among the most widely reported measures of connectivity between brain regions Hutchison et al. (2013). These approaches calculate the correlation between the timeseries of each brain region find their coherence, either across the full timeseries (sFNC) or by windowing the timeseries and calculating the correlation within each window (dFNC). Although extensions have been proposed Hutchison et al. (201)), along with more complex connectivity measures, such as wavelet coherence Yaesoubi et al. 2015). multiplication of temporal derivates Shine et al. (2015). and Granger causality Roe- broeck al. 2005); Seth et al. (2015), Pearson correlation remains the most prevalent measure of brain network connectivity. And even the less commonly employed metrics have issues stemming from some combination of sensitivity to noise, linearity, symmetry, coarse timescales. Most im-

---

Under review as a conference paper at ICLR 2023
portantly, these approaches do not directly model the communication, but rather analyze it post-hoc. The pursuit of instantaneous communication between brain regions [Sporns et al. (2021) and gen- erative approaches to model communication can potentially lead to models that closely resemble effective connectivity AAvena Avena-Koenigsheegee et al. (2018). An important advantage of generative models is that they allow to move away from post-hoe inference from context-naive metrics to- ward simulating macroscale brain communication. We propose the use of recurrent neural networks as generative models of communication on both simulated data and functional magnetic resonance imaging (fMRI) data to validate and demonstrate the specificity of our model. Our method com- plements connectivity metrics, since does more than quantify the aggregation of communication Avena-Koenigsberg et al. (2018), but directly simulates it, and can thus be analyzed using those same connectivity metries.
Creating an accurate generative model of the macroscale communication dynamics in the brain is hard, due to its complexity. However, there are some general design principles the brain follows Sterling & Laughlin| (2015). Generally, the brain tries to minimize its energy use, which is likely also true for communication in the brain. Macroscale communication is an energy-intensive process and involves white matter tracts, which are essentially highways that connect spatially separate parts of the brain. Hence, the amount of information and the number of times information is sent should generally be limited. The bits needed convey limited by the brain using sparse coding, which at a lower scale is how neurons encode and communicate information Olshausen & Field (2004). Although mechanistically different, due metabolic and volume constraints, macroscale commu- nication presumably exhibits strategies similar to sparse coding to efficiently transfer information between neural populations [ullmore & Sporns 2012); Sterling & Laughlin (2015). To incorpo- rate this inductive bias into our model, we arize the communication from one region in our model another using KL-divergence term to Laplace distribution. The communication itself is modeled as Laplace distribution and by minimizing its divergence from Laplace distribution we encourage sparser temporal communication. Encouraging sparse temporal communication implies that information is only sent when necessary, and lower rates of communication lead to reductions in the brain's 's energy requirements.
As far as we are aware, this is the first model that explicitly models communication dynamics be- tween vertices on a graph, specifically between brain regions. The connectivity metrics that are used currently assume that the connectivity between brain regions is stationary, e.g. Granger causality, and lack a generative model. Although non-linear generative models have been proposed for con- nectivity (Stephan al.] (2008), however, determining the correct model is intractable Lohmann eet al. (2012). Furthermore, most methods quantify the coupling between brain regions and do not consider potentially rapid changes in fMRI signals that can be traced from region to region. In this paper, we specifically try to model the communication between brain regions, which likely relates to these abrupt and parsimonious changes in the signal. This means that our model is finding a type of interaction between brain regions, or more generally nodes on graph, that has never been studied in this way before, to the best of our knowledge. The assumptions we make thus also differ from commonly used connectivity metrics. Firstly, we assume directionality and communication with the same temporal resolution as the original signal. Hence, we completely move away from windowed approaches, that quantify the coupling between brain regions within windows. Secondly, assume that brain regions are largely independent and can be modeled by a dynamical system with sparse inputs. Lastly, we aim to learn fully generative model of fMRI data, where the communications and initial state of each brain region's 's dynamical system can be sampled from a simple distribution. This also implies that our method is hard to compare with more common connectivity metrics, and we expect it to exhibit different behavior since it is modeled under different assumptions. We pro- pose this model because we believe has tremendous value and complements many connectivity metrics in a meaningful way, not only in the neuroimaging community, but also in other scientific fields, such as sociology Barnes ( (1969). computational biology or chemistry Balaban (1985). and traffic prediction models Zhao et al. (2019).
The goal of our model is to get a better idea of communication dynamics in the brain. To evaluate whether our model is equipped to find the underlying dynamics of a known generative model, we first train on simulated data. After we show that our model finds the correct generative model from the simulated data, we apply our model to neuroimaging data with fairly well-established neural pathways. Although the pathways are well-established, we do not know the exact ground truth communication underlying fMRI data. We do know that communication dynamics depend on
2

---

Under review as a conference paper at ICLR 2023
the task it is trying to perform, even outside of explicit windows in which these tasks performed. Hence, to evaluate our model on fMRI data, we train it on three different tasks and use logistic regression to predict what task small window of timesteps is from using the communication model predicts. We compare our model to dFNC and show that our method is more sensitive to the task, especially for small windows where explicit task is performed. This has implications for both resting-state fMRI and task IMRI, since our model may be able to provide deeper look into how tasks shape macroscale communications in the brain. Understanding task responses allows us to more accurately understand their correlates to behavior and how communication in the brain shapes certain behavior. Since our results indicate that our model is more sensitive to communications even at resting-state that are still related to task. Being sensitive even outside of external driving of fMRI activity is impactful for resting-state fMRI and psychiatric disorder research, where we expect communication dynamics to be different between controls and patients. Furthermore, since we establish our method on simulated data and then extend it to real data, we provide an initial framework for scientists that may want to extend our method into their area of expertise.
2
METHOD
Formal definition of our model The data structure we assume in this work is simple graph (V; E). with N vertices (V), and M edges (E), as shown in Figure D Generally, we can assume fully-connected graph, since our proposed model can learn to not send any communication over an edge, meaning it essentially 'trims' that edge. Each vertex si has a corresponding timeseries x xi...r with 7' timesteps, this timeseries is modeled as the following dynamical system.
x
(x' (t), u (t))
(1)
A)
8)
Sequentill VAE (5- VAB)
Encoder
Decoder
Router
9
Communication RNN [Comma)
SAGE
+
WW
RNN
MLP
Figure 1: This figure shows the composition of a single node from a graph on the left. The com- munication and reconstruction RNNs are completely independent of each other, the communication RNN sends signals to other nodes, whereas the reconstruction RNN reconstructs the original time- series from outside communication and an initial condition 20 (right).
One recent model that has been used model this dynamical system LFADs Pandarinath et al. (2018), where they model the external inputs u(t) samples from normal distribution parameter- ized by a controller that has access to the hidden state of the encoder and decoder. The controller can be fairly opaque in what it is concretely modeling, however, if we have access to timeseries for multiple nodes, we can replace the input from the controller with input coming from the neighbors of a node. This incorporates more explicit inductive bias in our model. Formally, the external inputs for vertex of are communications c and external events e(t). We use the notation + i to mean all incoming communication from neighbors of vertex v². - to mean all outgoing com- munication from vertex v' to its neighbors, and j + i to mean communication from vertex v² specifically to vertex v². The external events are known external perturbations system, such as the block design of task. To model the dynamical system of each vertex sepa- rately, we train a recurrent neural network (RNN), which we call the decoder, to find the non-linear function (F)) that describes its dynamics. First, we consider the case where the model receives no inputs, this is a transient dynamical system, that only depends on its initial state z/. We find the initial inputs to the dynamical systems using an encoder, with the following probability distribution p(z' xx....) A (z': pág ot). This distribution is parameterized using bi-directional RNN (the encoder), and we sample the initial state from that distribution zo p(z' xx....)) The encoder and
the dynamical
3

---

Under review as a conference paper at ICLR 2023
decoder together form sequential variational autoencoder (S- VAE) Zhao et al. (2017): Kingma & Welling (013). The S-VAE visualized in Figure [B. The decoder can be expressed as parame- terizing the following distribution p(2) (2))). which allows to train it using the evidence lower
bound (HLBO) Kingma & Welling (2013).
We extend this model by adding two types of inputs, namely the communication from other ver- tices, and external events. The communication from one vertex to another is also modeled as a recurrent neural network, with ef sample from Laplace distribution. p(c) [xi...-1)
(c ( oi r) Communication from vertex vertex j at each timestep t thus sam-
pled from the following Laplace distribution ci p(c]""ixx (x)...)- Modeling the communication timeseries between vertices distributions for each timestep allows us regularize the communica- tion. To encourage sparse communication between brain regions, we minimize the KL-divergence between the communication timeseries and a zero-mean unit-variance multi-dimensional Laplace distribution during training. The communication network is visualized in Fi. Note that com- munication from vertex to vertex j computed at time arrives at the decoder at time t + 1 to ensure the model is temporally causal. Thus, the decoder now models the following distribution p( z c...r-1,e1... ternal events only affect one brain region, after which the signal propagates through communication. The external events are represented as one when the external event is ac- tive and when it is inactive. Our model trains vector for each event type. Each entry in the vector corresponds to the probability that an event should be inserted as an external event in that brain region. The vector parameterizes Gumbel-Softman distribution Daan et al.) (2016), which we sample from to determine the index of the brain region that gets the external event as input.
where ei...' are the external events. In our model, we assume that ex- other brain regions
So far, each communication network predicts communication to another brain region independently. To allow the model to suppress information that has already been sent by other brain regions, we add router network that similar the forget gate in long short-term memory (LSTM) network Hochreiter & Schmidhuber (197)). The router network receives all the incoming communication c timestep and this vector is fed through a multi-layer perceptron (MLP), which has the same number of outputs as the size of the communication vector c." The output is transformed through a sigmoid to be between [0, 1] and multiplied with the incoming communication vector. This allows the router to block information sent from certain vertex that is already captured by information sent from other vertices. The router is visually represented in Figure [
With simplified notation, thus have the following equations for the S-VAE.
h = RNN Enc (xi...)
(2)
M Wr, o0 = = Wi
2b ~ ~ (Hb. o%)
(3)
L...T = RNNI (z), C.ir e...r)
Then, for the communication network we obtain the following equations.
h = RNN Corm
(5)
, - MLP M
Comm. Ju (h) 1),
MLP
Comm, (hi_ c L o
(6)
ci = c,t O a (MLP) (c,
Router
Where o()) is sigmoid function, and MIP MI and MI.P Comm, share every layer, except the last one.
Comm,
We can then adapt the evidence lower-bound (ELBO) Kingma Welling 2013) as follows, where d' (zh, c.. x refers to the approximate posterior, and Pb (x.... Z C...r, eI...) is the generative network. in our case the recurrent decoder.
N
((0,0,x...T)
=
DKL (as(z) xx....) p(z))
DDLL (clT.r (xf...) p(c))
(8)
+ (gi .xii (logró (xi... a eill eI..r)
4

---

Under review as a conference paper at ICLR 2023
Where p(z) and p(c) are the priors for the initial inputs and the communications and is the sparsity multiplier, respectively. Specifically, the first row of the equation S refers to the KL-divergence between the initial hidden states zg. learned by the encoder (4') and the prior p(z), which is zero- mean one standard deviation Gaussian. The second row of the equation refers to the KL-divergence,, weighted by a sparsity parameter A, between the communication timeseries (ci and a prior p(c), which is a zero-mean 0.1 standard deviation Laplace distribution. This encourages the model to learn sparse communication timeseries between the vertices. The final row of the equation refers to the expected log-likelihood, or the reconstruction, of the signal that is modeled by the decoder (p))) We assume that the fMRI signal for each region is normally distributed with a constant standard deviation, which means that optimizing the expected bogllikelihood is equivalent to minimizing the mean-squared error. In our work, we decide minimize the mean-squared error because we found it to be more stable.
Intuition and synthetic data To gain some intuition about our model, we explain how our model works on the simulated data we use in this work. We assume that if part of the signal is explained by a pattern in another region at previous timestep, communication could have occurred. A simple ex- ample is communication between two regions, where the timeseries for each region is uniform noise. Let us assume there is a linearly decaying pulse in the first region at a time 71 Uniform(O, 70), that decays until T2 ~ T1 + 20. Once the pulse has decayed in the first region at time T2, a pulse starts in the second region that linearly decays time T3 T2 + 20. These are exactly the pa- rameters we use to create our synthetic data. Now, if we only observe one of the two timeseries, the pulses will seem sampled from a uniform distribution the decoder of that region. However, if we have access to both regions at the same time and enough training data, our model should be able to figure out that the pulse in the first region structurally precedes the pulse in the second region. In our synthetic data, we add a third region that models a sinusoid and has a pulse after the second region's 's pulse that linearly decays at time T4 T3 + 20. This region also has a different underlying signal, namely a randomly shifted sinusoid (shifted between Uniform (-20, 20)) to show that our model can separate global signal from communication. visual example of our synthetic data is shown in Figure 2 We expect that the initial inputs 2p models the global signal so that the communication our model predicts is not affected by the sinusoidal signal. As mentioned previously, fMRI data also exhibits global signals that may affect connectivity metrics, such as physiological noise. Hence, we can infer whether our model is affected by the global signal in our simulation data to make it more realistic when transferred to biological data. The communication
thus expect is shown in Figure 2B. Namely, the communication model in region I should learn to send a signal to region 2 when the pulse in its region has almost decayed, and region 2's communication model should learn to send pulse to region 3 once its pulse has almost decayed. The expected communications in our synthetic data are shown in Figure 2B
Simulated Imput dignals
Expected communication
02
Th- Unlformjo 1-90]
Comm from R3-91 Comm from RI -02
12: 20
16- 13
This figure shows the simulated input signals on the left and the communication we would expect our model to find. Both signals are drawn using graphical design software. The communica- tion would happen from R1 to R2 (middle right) right before T2, and from R2 to R3 (bottom right) right before T3.
Figure 2:
Biological data The fMRI data we use in this work is obtained from the ppen-aceess, under data usage terms, HCP-1200 dataset Van Essen al. (203)) for all subjects with cortical surface time-
5

---

Under review as a conference paper at ICLR 2023
series data. We use resting-state fMRI (rs-fMRI) and three tasks in our work, the motor, emotion processing, and language processing tasks. The data is registered using multimodal surface regis- tration (MSM) Robinson et al. 2014 2018), and surfaces are constructed using Freesurfer Glasser et al.] 2013); Fischl (2012). We use the Y atlas Yeo et al. (201T) to separate the cortical sur- face data into 17 separate regions. The spatial information within each region is averaged, as is com- monly done in communication analyses of the brain. Each subject's 's timeseries is then independently band-pass filtered independently (0.01 0.15Hz) and linearly detrended using the Nileam package Abraham et al. (2014). Each task has a different length, so we cut off tasks at 176 timesteps, which the smallest common size of the timeseries. For the rs-IMRI data we use the first 200 timesteps to make the length similar to the task data. The motor task consists of visual cue, after which a subject has to start performing a motor task. The motor tasks a subject performs are related to the left hand, right hand, left foot, right foot, and tongue. For the emotion task, the subject sees visual cue and then has to match two shapes shown at the bottom of the screen to a shape at the top. This sub-task is referred to as the neutral task, and the other task has the subject match one of two faces at the bottom of the screen with a face shown at the top. The faces either look fearful or angry. The language task consists of a sub-task where the participant has perform arithmetic and push a button for the right answer. The story sub-task presents the subjects with a short story of 5-9 sentences. The timing of each task is shown, after convolving it with SPM's hemodynamic response function Penny et al. (201I). in Appendix B We train one model on all three tasks such that the prediction of the task is not influenced by the fact that the model was specifically trained to model communication for that task.
Implementation Our model is implemented in Pytorch Paszke et al. (2017) and trained on an internal cluster using single NVIDIA V100 and NVIDIA 2080 GPUs, with a batch size of 8, the Adam optimizer| Kingma Ba (2014), EE- weight decay, a learning rate of 0.01, 0.1 epsilon, and 0.9, 0.999 as betas. We reduce the learning rate when it plateaus using a scheduler. At each plateau, we reduce the learning rate by multiplying with 0.95, with a patience of 6 epochs, and a minimum learning rate of 1E 5. We train each model for 200 epochs and perform early stopping when the loss has not improved for 50 epochs, for the simulated data across 8 seeds: (42, 1337, 9999, 50, 100, 500, 1000, 5000). The model is trained on the first three seeds for the biological data, this is due to computational limitations. Since we use multiple sparsities, the number of runs quickly explodes, and each model on biological data takes about 7 8 : 30 hours train. All of the code will be made publicly available after the double-blind review has concluded on GitHub. The hidden size we use for the encoder, decoder, and communication model are all 16. For the synthetic data we use sparsities: (0.001, 0.005, 0.01, 0.05, 0.1, 0.5. 1.0, 2.0). For the real data we use smaller number of sparsities due to computation limitations, but also because we have narrowed the sparsity down at this point to: (0.001, 0.005, 0.01, 0.05, 0.1, 1.0). To not mix subjects across the training, validation, and test set, we split them based on the subjects. Hence, a subject occurs with every task in one of the sets it was assigned to and in no other sets. Data is z-scored across the temporal dimension of the data for each subject independently.
Experiments To test whether our model is interesting as a complement to connectivity metrics, we evaluate our model in three different ways. To get a better idea of how much the controller improves our model, we perform each experiment with both models. We refer to the model without the controller as Comms VAE and the model with the controller as Meta-Comms VAE in the rest of the text. Firstly, evaluate model on the aforementioned synthetic data. We calculate the correspondence between the expected communications and the communication that our model predicts on test set and look how the sparsity parameter affects the average correspondence to the test set. The correspondence is calculated by convolving normal distribution over an impulse response centered on the start of the event in that brain region. Then we correlate the communication between the resulting signal and the communication to that brain region for different shifts of the communication signal. Finally, we sum the maximum absolute correlations for R1- > R2 and R2- > R3, and subtract the sum of two times the average absolute communication for all other brain regions. This corresponds to expecting zero communication in all other brain regions, and a spike right before the start of the actual event. By using shifts to calculate the correlation we do not make assumptions about the lag that is learned between the communication RNN and the decoder. Note that when we use communication in these experiments, we take the mean of the Gaussian or Laplace distribution, since it is the most probable sample under that distribution. Then, train both our models an fMRI dataset with three tasks, each model is trained on all three tasks

---

Under review as a conference paper at ICLR 2023
conjointly not to induce any task-specific model noise that can affect the downstream prediction. Since we do not have a ground truth for the communication of the fMRI signal, we determine the best model by looking the reconstruction eπror the validation set with respect to the sparsity, see Appendix [ We use the model that performs the best across the seeds and use for task prediction. We use 4 different window sizes 15, 10, 20, and 50] compare our model dFNC, Granger causality Granger (1969), transfer entropy Lobier et al. (2014), multiplication temporal derivatives Shine et al. (2015). and the full IMRI signal in the window for each region. This allows us to determine whether our model has better temporal resolution since the quality of the correlation will likely get lower with a smaller dNFC window. For each window, we take the absolute value of our model's 's communication (since it can be both negative and positive) and sum it across each window. Then, we train a logistic regression classifier on the training and validation set using the temporally summed communication to predict what task each subject in the test set's window is from. This results in a classification accuracy across time for the four windows, and we perform the same routine for the connectivity metric comparisons. Finally, we look at the absolute correlation between the communication timeseries for each directed edge and the sub-tasks, we average the correlation across seeds and the subjects in the test set. We visualize the averaged correlations that are higher than 80% of the maximum value or higher than 0.2, whichever value is higher. This experiment allows us see whether our method sends biologically realistic compared to a ground-truth.
RESULTS
Synthetic data The results for the synthetic data are shown in Figure 3 Generally, we see trend where higher sparsity parameters increase the correspondence to our expected communication on the test set, up to point where increased sparsity suppresses any communication. There is peak around 0.01 in Figure BA for both models, after which the correspondence with the expected com- munication decreases. This exactly supports our hypothesis that encouraging sparse communication helps. To show what the communication looks like in the two models we test, we take the test set synthetic data, predict the communication for each, and use the synthetic generation parameters to align the communication timeseries. After aligning the communication timeseries, we take the aver- age over the test set for the best seeds for each model at 0.01 sparsity from Figure 3A and visualize what the communication looks like in Figure 33. Right before the pulse ends in the region 1, it communicates with region 2 before its pulse begins. The communication from region 2 to region 3 starts exactly when the pulse is almost linearly decayed, right before the pulse starts in region 3. The main difference between our Meta-Comms CommsVAB and CommsVAB model is that in Figure 33. the Comms VAE's communications between region and region 2 are closer to the actual pulse. Generally, we do not see trend in terms of a difference between the two CommsVAB models, at least for the synthetic data. This is probably because the task itself is sparse and hence being able to increase sparsity using the controller unnecessary. An example of the reconstruction of validation data by our model is shown in Appendix
IMRI data The results to test both the specificity of our generative model and how instantancous the communication is are shown in Figure 4 We produce these results for the best model we find based on the reconstruction error on the validation set, see Appendix A The best model is the Meta- CommsVAr model with sparsity of 0.05. However, the standard deviation over the reconstruction higher than for 0.01, and since the difference in the average is negligible between the two spar- sities, we select 0.01 for visualizations and the next two results because it seems more stable. Our results with this model clearly show that our method outperforms all of the connectivity metrics we compare it to, and even the fMRI data itself. This indicates that our model finds representations of the tasks in the rs-fMRI data that are intuitive and not linearly separable initially. This true for every window, and every window size. Note that both Granger causality and transfer entropy per- form so poorly in terms of classification accuracy, is likely because they are inaccurate when fit in small windows. However, our method only has small decline in classification accuracy for smaller window sizes, which supports our hypothesis that our model can find instantaneous communication. Our method already starts being task task-specific early in time, meaning that even before any of the tasks start, the communication our model finds in the brain is task-specific, which is likely to be true if we knew the ground truth of communication. One reason that task accuracy increases over time
7

---

Under review as a conference paper at ICLR 2023
Aigimmntt with expecced comme
Meta Commicaa
Meto Commovas
Dortm Frame Comm From 922-122
Figure 3: This figure shows the performance of models for different sparsity parameters (x-axis) on the left and comparison of our model's communications to the ground truth event times (black) on the right. The smaller. slightly transparent, dots in the left plot are the performances of different seeds for each model, and the larger bold dot is the average performance of the seeds. The values at which events happen are uniformly sampled,
is hard to visualize when the model communicates since the communication is different for every test trial due to differences in event times. Hence, to average the communication over the test set, we use the event times to shift the timeseries so they align. The right figure shows the average over these aligned timeseries for the test set.
is that the communication model only has access to points previously, so early on in the timeseries, does not have enough information to start sending high-quality communication. After few win- dows, the accuracy of the task increases and is then relatively stable. This is different for the dFNC, which fluctuates, likely based on when the tasks start, but our model is specific to the task and es- sentially the same across all of the windows in the timeseries. Thus, our model is task-specific when there relatively little task signal from either of the tasks in a certain window. Another result gather from Figure is that the lower- triangar part of the matrix is worse at predicting the task than the upper-triangular part of the communication matrix for each window. The reason include these results is that the dFNC is symmetric and the lower and upper triangular parts of the matrix are thus the same. To make the comparison fair, we compare separately compare the triangular parts of communication matrix with dFNC. Since our method is directed, can also take the whole matrix, which outperforms both the lower and upper triangular parts of that matrix, as expected.
(line]
dFNC
Upper triangular Granger causality
Lower triangular
Full matrix
Rave data
Transfer entropy
Multiplicative temporal derivatives
Figure 4: This figure shows four panels, each with a different window size to compute both our model's communication density and dFNC. The y-axis shows the classification accuracy of the task given only that window on the test set.
Our last experiment aims to evaluate whether our model communication between regions that aligns with the sub-tasks in the data. We use naming conventions for these regions from Kahali et al. (021). The hemodynamic response model over time for each sub-task is shown in Appendix B The average correlation across subject and seeds is shown in each the sub-tasks, and color of the title indicates whether it is sub-task of the motor task (blue), emotion task (red), language task (orange). Two of the most easily verifiable correlations to sub-tasks are the visual cues in the bottom left of Figure The edges from the early visual area (V-A) somatomotor B (SM-B) and limbic A (L-A) are correlated to the hemodynamic response of the visual cue. This is a convincing result because of the nature of the task, namely motor execution after the visual cue. Motor execution has
8

---

Under review as a conference paper at ICLR 2023
been shown to be related to both the somatomotor region well as the limbic region Mogenson et al. (1980). Although the latter is relatively understudied, our results indicate a potentially fruitiul direction when related to communication. Furthermore, the visual cue for the emotion task is related to communication from the early visual area (V-A) to control A (C-A), which is a cognitive control area. Given that the emotion task requires the participant to match shape (neutral) to a target in the top of a screen, the communication from the visual area to the cognitive control area also aligns with biological intuition. Other notable communication occurs towards two default mode networks (D-A, D-B). This aligns with previous research, which indicates sub-networks of the default mode network are coupled with language networks Gordon al. (022)).
face (fear)
Left hand
Left foot
Right hand
Right foot
Tongue
Luk
5.0 Viaa wnd
04.0 Dinil MONTONN
CC Coros C LO Dimbec
DC DeGL
Figure 5: This figure shows the correlation between communication from our model and sub-tasks in the data. Notably, the visual cue in both the motor and emotion task induce sending from the early visual areas (V-A) to networks implicated in motor execution.
4 DISCUSSION
This work proposes a generative model for macroscale communication in the brain that we show pre- dicts directed, instantaneous, and task-specific communication dynamics. We compare our model on a synthetic task that tests the requirements we set out for our model and find that our model can accu- rately uncover the communication dynamics we incorporate into the synthetic data. Then, we train our two models on three fMRI tasks and find that reconstruction error decreases with higher sparsity parameters, indicating that the sparse prior on the brain communication dynamics fits our data. The sparsity of macroscale brain communication needs to be studied further but aligns with findings in computational neuroscience Olshausen & Field (2004) and the design principles of the brain [teer] ling & Laughlim (2015). We use the best model based on reconstruction error over seeds to predict what task small time windows are from in a test set and compare the classification accuracy with a multiple commonly used connectivity metrics, both directed, and undirected. The performance of our model does not decrease as drastically with smaller window sizes, supporting our claim that our model predicts instantancous communication. Then, understand how communication related to sub-tasks, we calculate the correlation between each sub-task and the communication timeseries of each edge. After averaging over test subjects, and seeds, we visualize the highest sub-task cor- relations. Almost all of the strong connections we find are directed, with correlations up to 0.64. The directions and the strength of the communication also align with neuroscientific priors we have about how regions affect each other. Thus, our results confirm what we set out to incorporate into our model from the beginning.
We want to expand on this model by applying it to stst fMRI data, and explore whether our model can be used to more deeply understand psychiatric disorders. As an initial step, we have looked at how well our model can reconstruct 200 timesteps of resting-state fMRI data in Appendix G Furthermore, since this model is generative, we can start to probe the model and interpret the response functions when we synthetically increase communication from one region to another at certain time points. We can design communication patterns to test how communication in different regions affects the signal in others.

---

Under review as a conference paper at ICLR 2023
5 REPRODUCIBILITY
By making our code available after the double-blind process has concluded, we hope that our work can be reproduced and inspected by other scientists. Most the parameters necessary for reproduc- tion are mentioned in the experiments section, and a longer description of the formulas and math behind the model provided the methods section. We have explicitly added section on the intu- ition behind the model so people can get a mental grasp of how the model works and that hopefully also helps people reproduce our work.
REFERENCES
Alexandre Abraham, Fabian Pedregosa, Michael Fickenberg, Philippe Gervais, Andreas Mueller, Jean Kossaifi, Alexandre Gramfort, Bertrand Thirion, and Gaël Varoquaux. Machine learning for neuroimaging with scikit-learn. Frontiers in neuroinformatics, 8:14, 2014.
Andrea Avena-Koena Koeniggenngssereer. Bratislav Misic, and Olaf Sporns. Communication dynamics in com- plex brain networks. Nature reviews neuroscience, 19(1):17-3, 2018.
Alexandru T Balaban. Applications of graph theory in chemistry. Journal of chemical information and computer sciences, 25(3):334-343, 1985.
John A Barnes. Graph theory and social networks: technical comment on connectedness and connectivity. Sociology, 3(2):215-233) 1969.
Ed Bullmore and Olaf Sporns. The economy of brain network organization. Nature reviews neuro- science, 13(5)333--34, 2012.
Bruce Fischl. Freesurfer. Neuroimage, 62(2):774-781, 2012.
Karl Friston. Dysfunctional connectivity in schizophrenia. World Psychiatry, 1(2):66, 2002.
Matthew Glasser, Stamatios Sotiropoulos, J Anthony Wilson, Timothy Coalson, Bruce Fischl, Jesper Andersson, Junqian Xu, Saad Jbabdi, Matthew Webster, Jonathan R Polimeni, al. The minimal preprocessing pipelines for the human connectome project. Neuroimage, 80:105-124, 2013.
Evan M Gordon, Timothy O Laumann, Scott Marek, Ryan V Raut, Caterina Gratton, Dillan J New- bold, Deanna Greene, Rebecca S Coalson, Abraham Z Snyder, Bradley Schlaggar, al. Default-mode network streams for coupling to language and control systems. Proceedings of the National Academy of Sciences, 117(29):1338--111 17319, 2020.
Clive WJ Granger. Investigating causal relations by econometric models and cross-spectral methods. Econometrica: journal of the Econometric Society, pp. 424 438, 1969.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8):
1735-1780, 1997.
R Matthew Hutchison, Thilo Womelsdorf, Elena A Allen, Peter A Bandettini, Vince D Calhoun, Maurizio Corbetta, Stefania Della Penna, Jeff Duyn, Gary Glover, Javier Cootzlle-Caastillo, et al. Dynamic functional connectivity: promise, issues, and interpretations. Neuroimage, 80: 360-378, 2013.
Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterizztion with gumbel-softmax. arXiv preprint arXiv:1611.01 144, 2016.
Sayan Kahali, Mareus E Raichle, and Dmitriy A Yablonskiy. The role of the human brain neuron- glia-synapse composition in forming resting-state functional connectivity networks. Brain Sci- ences, 11(12):1565, 2021.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv: 14/2.14/2.880 2014.
10

---

Under review as a conference paper at ICLR 2023
Diederik P Kingma and Max Welling. arXiv:13/2.611, 2013.
Auto-encoding variational
bayes.
arXiv preprint
Muriel Lobier, Felix Siebenhühner, Satu Palva, and J Matias Palva. Phase transfer entropy: a novel
phase-based measure for directed connectivity in networks coupled by oscillatory interactions. Neuroimage, 85:853-872, 2014.
Gabriele Lohmann, Kerstin Erfurth, Karsten Müller, and Robert Turne. Critical comments on dy- namic causal modelling. Neuroimage, 59(3):2322-25 2329, 2012.
Gordon J Mogenson, Douglas I. Jones, and Chi Yiu Yim. From motivation to action: functional interface between the limbic system and the motor system. Progress in neurobiology. 14(2-3): 69-97, 1980.
Bruno Olshausen and David Field. Sparse coding of sensory inputs. Current opinion in neuro- biology, 14(4))481-487, 2004.
Chethan Pandarinath, Daniel J O'Shea, Jasmine Collins, Rafal Jozefowiez, Sergey D Stavisky, Jonathan C Kao, Eric M Trautmann, Matthew T Kaufman, Stephen Ryu, Leigh R Hochberg, et Inferring single-trial neural population dynamics using sequential auto-encoders. Nature methods, 15(10)8805-115 2018.
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang. Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. 2017.
William D Penny, Karl J Friston, John T Ashburner, Stefan J Kiebel, and Thomas E Nichols. Statis- tical parametric mapping: the analysis of functional brain images. Elsevier, 2011.
Emma C Robinson, Saad Jbabdi, Matthew F Glasser, Jesper Andersson, Gregory C Burgess, Michael P Harms, Stephen M Smith, David C Van Essen, and Mark Jenkinson. Msm: new flexible framework for multimodal surface matching. Neuroinage, 100:414 4266444444646 2014.
Emma C Robinson, Kara Garcia, Matthew F Glasser, Zhengdao Chen, Timothy S Coalson, Antonios Makropoulos, Jelena Bozek, Robert Wright, Andreas Schuh, Matthew Webster, et al. Multimodal surface matching with higher-order smoothness constraints. Neuroimage, 167:453-465, 2018.
Alard Roebroeck, Elia Formisano, and Rainer Goebel. Mapping directed influence over the brain using granger causality and fmri. Neuroimage, 25(1):230-22) 2005.
Anil K Seth, Adam B Barrett, and Lionel Barnett. Granger causality analysis in neuroscience and neuroimaging. Journal of Neuroscience, 35(8):3293-3297 2015.
James M Shine, Oluwasanmi Koyejo, Peter Bell, Krzysztof Gorgolewski, Moran Gilat, and Rus- sell A Poldrack. Estimation of dynamic functional connectivity using multiplication of temporal derivatives. Neurolmage, 122:399-400 2015.
Olaf Sporns, Joshua Faskowitz, Andreia Sofia Teixeira, Sarah A Cutts, and Richard F Betzel. Dy- namic expression of brain functional systems disclosed by fine-scale analysis of edge time series. Network Neuroscience, 5(2):405 2021.
Klaas Enno Stephan, Lars Kasper, Lee M Harrison, Jean Daunizeau, Hanneke EM den Ouden, Michael Breakspear, and Karl J Friston. Nonlinear dynamic causal models for fmri. Neuroimage, 42(2):649-662, 2008
Peter Sterling and Simon Laughlin. Principles of neural design. MIT press, 2015.
David C Van Essen, Stephen M Smith, Deanna M Barch, Timothy EJ Behrens, Essa Yacoub, Kamil Ugurbil, Wu-Minn HCP Consortium, et al. The wu-minn human connectome project: an overview. Neuroimage, 80:62-79,
Maziar Yaesoubi, Elena A Allen, Robyn L Miller, and Vince D Calhoun. Dynamic coherence analysis of resting fmri data to jointly capture state-based phase, frequency, and time-domain information. Neuroimage, 10:1333, 2015.

---

Under review as a conference paper at ICLR 2023
BT Thomas Yeo, Fenna M Krienen, Jorge Sepulcre, Mert R Sabuneu, Danial Lashkari, Marisa Hollinshead, Joshua Roffman, Jordan W Smoller, Lilla Zöllei, Jonathan R Polimeni, et al. The organization of the human cerebral cortex estimated by intrinsic functional connectivity. Journal of neurophysiology, 2011.
Ling Zhao, Yujiao Song, Chao Zhang, Yu Liu, Pu Wang, Tao Lin, Min Deng, and Haifeng Li. T-gen: A temporal graph convolutional network for traffic prediction. IEEE Transactions on Intelligent Transportation Systems, 21(9):3848-388 3858, 2019.
Shengjia Zhao, Jiaming Song, and Stefano Ermon. Towards deeper understanding of variational autoencoding models. arXiv preprint arXiv:1702.0865.. 2017.
12

---

Under review as a conference paper at ICLR 2023
A TASK FMRI DATA RECONSTRUCTION ERROR ACROSS SPARSITY
Figure 6 shows that the Ceta-CommmVVAB outperforms the Comms VAE on all sparsities, but also has clear dip around certain sparsity. Since we do know the ground truth of the commu- nication in the fMRI dataset as we did with the synthetic data, the improvement in reconstruction error for higher sparsities shows promise for hyperparameter searches on other datasets. Note the reconstruction error is squared error, summed over time and averaged over the number of regions.
CommsVAE
64
Meta-CommsVAE Baseline
reconstruction loss
62
60
58
56
Validation
54
52
50
10 -3
10-2
10-1
100
Sparsities
Figure 6: This figure shows the average reconstruction of both models on the fMRI dataset, with standard deviation computed across the seeds. The Meta-CommsVAE outperforms the Comms VAE for all sparsities and performs the best at sparsity of 0.025.
13

---

Under review as a conference paper at ICLR 2023
B TASKS HEMODYNAMIC RESPONSE
Figure 7 shows the hemodynamic response for each of the tasks for the 176 timesteps we use in our model. The legend indicates what sub-task the hemodynamic response corresponds to.
Rightloct Uoot.
00
Ematioo
Viaol
-82
LanguRge
0.5
Figure This figure shows the average hemodynamic response of the test set.
14

---

Under review as a conference paper at ICLR 2023
C EXAMPLE RECONSTRUCTION OF THE SYNTHETIC DATA
Figure shows an example of the reconstruction of the synthetic data. The red line the input signal, and the blue line is the reconstruction. The first row is the first region, the second row the second region, and the last row is the third region. The columns are four random timeseries from the validation set. This shows that our model can still model the global signal, even for a shifted sinusoid, with almost perfect communication dynamics, see Figure
04
0.4
63
23-
0.0
2.-
-02
-02
0.4
26
100
50
150
30
120
-1.
10.4
-04
-0.
-1:0
50
120
153
50
100
150
50
350
30c
150
0.75
0.4
W
0.73
W
0.50
22-
0229
0.2
0.00
2:25
2.00
-.00
150
352
355
150
Figure 8: This figure shows an example of the reconstruction on the synthetic data for our CommsVAE model with sparsity 0.01. The red line is the input signal, the blue line the recon- structed signal, each row corresponds to a different region, and each column to a different timeseries from the validation set.
15

---

Under review as a conference paper at ICLR 2023
D SCALING
Our model in its current form applied to data with less than twenty regions. The scaling of the model more regions is shown in Figure and indicates fairly linear scaling for the inference time, and exponential scaling in the parameters.
1000
10
800
Number of parameters (M)
Inference time [ms]
600
400
200
0
0
25 50
75
100
0
25 50
75
100
Number of regions
Number of regions
Figure 9: The scaling of the inference time in milliseconds, and the scaling of the parameters with more regions/nodes added in our network. For the inference time, the scaling seems be largely linear, with slight exponential component, and for the number of parameters, the scaling is expo- nential. The hidden sizes of the network are 16 for these calculations, similar to the ones used for the task experimentation in this work. Note that we believe that our implementation could be more efficient and potentially lead to linear scaling the inference time.
16

---

Under review as a conference paper at ICLR 2023
E ABLATION: NO COMMUNICATION
As way to further our understanding of the types of signal that are modeled using the communica- tion, we compare model trained without communication model trained with communication in Figure 10
No communication
With communication
W MWA
Visua
Mann
M
Somatomotor
Deral attention
Mmmm
Dontal attention
w
wmma
Ventral
Mmmt
Limbic 8
Limbic A
Control
Wwww
Control
Defowit
Man
Defoult
Mor Vly"
Subject
Subject
Subject
Subject
Subject
Subject
Subject
Subject
Figure 10: To indicate the difference between using communication and not using communication, we visualize the reconstruction on 4 subjects in the validation set. The left side of the figure shows a model trained without any communication, and the right side shows a model trained with com- munication. The red line indicates the true signal, the blue line indicates the reconstruction in both sub-figum fub-figures.
17

---

Under review as a conference paper at ICLR 2023
F INDIVIDUAL TRIALS FOR SIMULATED DATA
The individual trials of our model (sparsity=0.01) on the test set. Figure ITA shows a zoomed-in version for the first 50 test trials, and Figure LIB shows the overview of all trials. Each trial exhibits the expected behavior, namely communication right at the end or atleast within the window between TI and T2, and T2 and T3, meaning between the event in the region that is sending and the region that is receiving.
A
Time
R1
R2->R3
blue/grey overlay time between the spikes, and in green/red the communication. Green means is communication.
Zoomed in version of sub-figure B
Trials
B
R1
R2
R3
RI
R2
R3
R1
R3
R3
Figure 11: This figure shows the communication of the first 50 test trials in sub-figure A, and an overview all trials in sub-figure B.
18

---

Under review as a conference paper at ICLR 2023
G RESTING-STAIE FMRI
We train our model with a larger hidden size (128) on resting-state fMRI (rs-fMRI) data to ensure that sparsity also increases the reconstruction performance with higher sparsities on data that is externally structured. We find a similar pattern as with the task data, where higher sparsities improve the reconstruction error of the model in Figure and 13
CommsVAE
54
Meta-CommsVAA
52
50
Volidation
46
10
2
169
Sparsities
Figure 12: The validation reconstruction loss compared to sparsities for rs-fMRI data
rr-fMRI reconstruction
Visall
Visual
Somotomator
Somotemator
1,1
Anthm
Desai attertion A
Am
Desal attontion
Ventral atsersion
Nin
cntrrllatieriion
Limble B
Norw win
Limble
Control A
Control
Control
DefaultA
Mon war
Defeutt
Detestt
Subject
Subject 2
Subject
Subject
Figure 13: This figure shows the reconstruction of the rs-fMRI using our model with the best sparsity (0.1).
To make sure the communication between brain regions is sparse, especially for rs-fMRI data, we also visualize the first four brain regions for two subjects the validation set, see Figure [4
19

---

Under review as a conference paper at ICLR 2023
Subject
Subject
Visual A
Incoming communication
Each line of the subpiots neighboring brain region's communication to region
Visual B
Somatomotor
Somatomotor
Figure 14: This figure shows the communication timeseries for the first four brain regions and two different subjects. Each line in the sub-plots refers to a brain region that is sending to the respective brain region, and the x-axis the communication over time.
20


Reviews:
[{"review_id": "mOeWzmK2gL", "reviewer": "Reviewer_qncm", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The manuscript proposed a Generative Communication Model by individually modeling the signals (by an S-VAE) and the communication sent at each graph node (by an RNN+MLP). Experiment on synthetic data shows that the model can successfully recover the designed communication between regions. Experiment on real (HCP) fMRI data shows good classification accuracy of the task type, and meaningful inter-region communication patterns can be discovered. ", "summary_of_the_review": "An interesting and important attempt to model the connectivity of fMRI/time series data via explicit modeling of the inter-region communication. More investigations into how and why the model can work will be needed. ", "strength_and_weaknesses": "Strength: Explicit modeling of the inter-region communication is both interesting and important for the fMRI data analysis and, more generally, the majority of multi-variate time series data analysis. The design of the model by S-VAE and RNN is reasonable and well-implemented. \n\nWeakness: First of all, it will be important to compare the proposed model with the “traditional deep learning” approaches. Specifically, it will be interesting to see ablation experiments of 1) training an S-VAE using the signals from all regions as input and output (with certain regularizations) and 2) removing the communication model to train the individual S-VAEs only. The reviewer is also curious about how the proposed model would scale to the region (node) size, as the current experiment on real data includes a relatively small (<20) number of regions. This concern is especially prominent as 1) the manuscript did not provide computational cost for training the model, and 2) each node will have its own S-VAE and RNN+MLP to be trained. The final concern is more of a naming convention issue, as generally, the communication model developed in this work is not considered “Generative.” Rather, it performs a binary classification on whether a specific inter-region communication will happen at a specific time point. The methodology section of this manuscript also suffers from clarity issues; see comments below. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "FBfmDPT-6P5", "reviewer": "Reviewer_kyae", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work proposes new generative model of communication in graph networks, applied here to fMRI data. The network attempts to overcome weaknesses of purely correlational approaches such as pearson correlation by explicitly modeling the directional communication between different nodes in the network, and giving control of the sparsity of communication between nodes. In the model, each node possess two networks, the first a S-VAE that reconstructs the signals based on the observed data and the estimated communication and task events, and the second a network that determines whether to communicate with another node using a binary signal. This allows for the unique directional information from each node to be precisely delineated, and to regularize the sparsity of communication using a Laplacian constraint. The manuscript offers a simple example of how the model could work in practice and then applies it to an fmri dataset consisting of 3 emotional, language, and motor tasks. Because there is no ground truth for the fmri data, they look at the ability to classify tasks based on the estimated communication dynamics, and compare this to the classification ability using dFNC, a pearson correlation based approach. They find superior performance of the generative model when using shorter time windows for the dFNC approach. Lastly, they demonstrate reconstruction of a communication graph for each task for the S-VAE approach, and find some concordance with biological intuition. ", "summary_of_the_review": "I am a bit torn with this manuscript. I think that the model holds some value and has a structure (directionality, explicit communication) and inductive biases (sparsity) that could be useful , but I have reservations about the quality of presentation and comparisons given. Without benchmarks for synthetic data or better sense of how the model compares to other approaches it is unclear when it should be used. All models are wrong, some are useful, and it would be nice to know the exact circumstances and advantages this model has – in practice – compared to other approaches. I am inclined to reject unless there is significant revisions wrt benchmarks and clarity of exact situations where the model should and shouldn’t be deployed. ", "strength_and_weaknesses": "1) The model is well described and the experiments have clear descriptions for reproducibility wrt the data used and network training parameters. \n2) The model explicitly incorporates the inductive bias of sparsity, and models directional binary communication between regions. \n3) There are experiments with real data and synthetic data to demonstrate the model quality and types of insights it can deliver. \n\n\nWeaknesses\n1) Paucity of benchmarks and comparisons. I found the approach interesting, but comparing a generative model to a windowed pearson correlation felt lacking. It would be nice to see other metrics considered, especially granger causality which is often used or transfer entropy. This would be especially nice on a benchmark dataset or synthetic dataset for causal prediction, but also for the directional communication graphs in Figure 5. \nIn the LFADS paper that is mentioned for instance, LFADS is compared for latent factor estimation against comparable techniques in a fairly convincing synthetic data experiments. \nOne of the challenges in evaluating the manuscript is that while the model will certainly produce a set of communication values when applied to a dataset, it isn’t clear in what data and parameter regimes it is effective. The model will give some answer when applied to a dataset, but it isn’t clear \n2) I found the description of the synthetic data could be better. How were the synthetic experiments prepared and what is the alignment metric that is used? I found the description, especially the post hoc alignment of data confusing. What are the individual data points in Figure 3? \n3) In figure 4, unless I am incorrect the VAE model has been trained on the tasks before hand, so there is a greater amount of information stored in the network weights. This may contribute to accuracy classifying tasks, but also likely introduces some amount of bias. \n\n\nNits: \nReference formatting is weird – names are not separated from the text.\nFigure 2 – in the cartoon there is no noise in the trace for R3, and the noise appears hand drawn (the rise in signal at T1 for R1 is not a function/does not pass the ‘vertical line test’). \nPg 4 after eq 7 typo ‘intense’\nPg 4 bottom missing equal signs\nFigure 3A – what is the definition of this metric\n“We train one model on all three tasks such that the prediction of the task is not influenced by the fact that the model was specifically trained to model communication for that task” – technically it is still influenced by the task, just less so. \nIn the introduction the authors mention an advantage over correlational approaches to function over shorter timescales, but this approach requires training data and time to initialize the state of RNNs and account for time lags, eg in the Figure 2 cartoon there is a time gap in communication that is claimed to be learned by the model. \n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "8cqRIKdoo-V", "reviewer": "Reviewer_zo6y", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, the authors propose a method for non-linear generative communication from human brain functional data, addressing three problems with connectivity approaches by explicitly modeling the directionality of communication, finding communication at each timestep, and encouraging sparsity in communication. They simulate temporal data to evaluate the proposed method that has sparse communication between embedded nodes. The proposed model can predict the expected communication dynamics. Overall, I found this paper to be difficult to follow, and I am not sure how the proposed method is novel with regard to machine learning.", "summary_of_the_review": "a) There may be some practical applications of this paper, but it is mainly ambiguous whether its contribution(s) will be accepted as part of the machine learning area. There seems to be just a combination of some machine learning algorithms in this paper, which are then applied to a short set of data in the paper.\n\nb) Besides the syntactic data, the only real fMRI dataset used to evaluate the proposed method is HCP-1200. For you to be able to evaluate a new approach, you will have to use data from a variety of sources.\n\nc) I found this paper hard to follow as well. Some parts of the problem formulation are hard to understand. \n\nd) The authors mentioned modeling the cognitive pathways from white matter — i.e.,  these regions communicate through signal propagation along the cortex and along white matter tracts over longer distances. Several theories explain fMRI may not be a good proxy to shed light on the white matter functional activities. Instead, many studies used DTI to model the regions’ communications. As an example, please see the following paper: “Learning Macroscopic Brain Connectomes via Group-Sparse Factorization”. Authors should clarify the scientific fundamental behind their proposed theory/approach.\n\ne) The performance of the proposed method should be compared with similar state-of-the-art approaches.\n\nf) There are some minor linguistic and typo problems in this paper.\n", "strength_and_weaknesses": "Please refer to the review section.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "1: The main claims of the paper are incorrect or not at all supported by theory or empirical results.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "1CumX3ymAd", "reviewer": "Reviewer_sVVG", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper models the fMRI activity of multiple brain regions simultaneously, with sparse communication between them. ", "summary_of_the_review": "More analyses on identifiability, or more empirical analysis would be helpful. At the current state, the paper seems more suited for a neuroscience journal after using it for scientific insights. ", "strength_and_weaknesses": "This paper seems like a reasonable extension of the popular neural modeling approaches (ex. LFADS) to fMRI and multiple brain regions. \n\nI have some reservations:\n1. I am not confident on the identifiablity of the model. If the generator for each region is expressive enough, it might not be necessary to have any communication from other brain regions. \n\n2. I am not confident of the instantaneous sparsity as biologically realistic. If region A is not connected with region B, then the communication will be 0 for all time points. This points to some sort of group sparsity over time/trials. \n\n3. Do you recover the ground-truth communication on individual trials? In simulations (Fig 3), all results are averaged across multiple trials. In my experience, it is hard to get a truly sparse signal as posterior for individual trials in VAE setting. The authors should clarify whether the sampled posterior is sparse or not.\n\n4. The fMRI analyses seem weak. The biological insights seem mixed and not too clear. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject