Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The current manuscript presented a connectome constrained latent variable model for whole-brain calcium imaging of C.elegans nervous system. The whole-brain calcium imaging of C.elegans was collected while C.elegans was undergoing chemosensory testing, and the dataset has been published recently. In the current study, the authors aimed to present a model that could predict the single-neuron and the single-trial activity of this dataset. Specifically, the activity of each neuron of the whole brain imaging was modeled by latent variable analogous to the voltage of one unit in the model network. The connection between model network units was constrained by the connectome. The authors showed that the connectome constraints significantly improved the prediction power of the model, in predicting the activity of missing units, as well as predicting the single-trial activity of hold-out warms. Overall, the authors have provided a clear description of the model and carried out a good amount of experiments to evaluate different model variants.
main_review: •	I believe that connectome constraints help improve the model performance in the current setup. However, the results are not surprising. The question is whether the benefit of connectome constraints was due to insufficient training data. When you have enough data, would an unconstrained model or a loosely constrained model perform as well as a connectome constrained model, meaning that a model could learn reasonable connection structure through sufficient training? 

•	How does the latent space look like? The current model predicted the latent variable of neuron voltage and used a differential equation to further predict the observed calcium signal. One major question I had was whether such a simplified calcium-voltage model was sufficient for C.elegans neurons. The units of model networks were passive non-spiking units, while in real brain calcium signal was mainly generated by suprathreshold neuronal activities. Although the problem might be partially taken care of by the nonlinearities of the model, it is not clear whether that was sufficient. On the other hand, I’m wondering whether the oversimplified calcium-voltage model could be one reason causing the relatively worse single neuron prediction performance of sensory neurons. Considering the spiking properties of sensory neurons, they might be more sensitive to inaccurate calcium-voltage models compared to interneurons and motor neurons.

•	How does warm prediction look like? Could authors show some population trajectories? Could authors discuss what had led to the inaccuracy in whole warm predictions, whether due to inaccurate prediction of a particular subset of neurons or whether due to inaccurate prediction of overall brain state? 

summary_of_the_review: In general, I vote for accepting the current paper if the authors could address all of my concerns above. I think the authors have presented a model for a novel whole-brain calcium imaging which was only recently published. Overall, the modeling strategy makes sense. I like that the authors tested on different variants of the model, which are all relevant, and provided insights on model selection. One limitation of the current work is the model generalization. I think the application of the model is limited to the current training data. I’m not convinced that the current model could generalize to chemosensory tasks, other than the ones in the training sets.  My major concern is about the clarity of some parts of the paper and some additional improvement in the work. I hope the authors could address my concerns in the rebuttal.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper develops a latent variable model with biologically meaningful parameters for the worm *C. elegans* whole brain neural traces. Given the connectivity between the neurons, the neural traces are modeled as stochastic leaky integrators with either conductance or current based synaptic input model. Given the voltage traces, the observed calcium signals are modeled as first order leaky integrators added by noise to account for the observation noise. External input is nonlinearly transformed and fed into the neurons to account for the stimulus-dependent modulation of neural activity. Multiple connectivity models (fully trainable, fully trainable with $L_2$ regularization, trainable weight magnitudes, trainable global scale) are used to investigate whether biological connectome constraints help with neural activity predictions or not. Variational inference is used to infer a posterior distribution over neural activity trajectories with factorized Gaussian distribution as the variational family. Results on neuron-hold-out and worm-hold-out experiments suggest that connectome constraints and conductance-based modeling improves model prediction over unseen neurons and worms.
main_review: The question of whether anatomical constraints help with improving model predictions and constraining the space of generative models of neural activity is an important and significant question in computational neuroscience. This paper takes a step towards addressing this question by developing biological models that have interpretable parameters (connectivity weights, neuron time constants, resting membrane potentials) yet are flexible to capture the variability across the population or individual neurons (by incorporating noise in the latent and observed variables). If a model resembles the biological generative process of the data up to appropriate amount of details, we would expect that incorporating more biological constraints would help with model's predictive performance. Therefore this approach not only could be used for model selection, but also could provide insight about why biological systems prefer certain architectures or mechanisms over others.

While the paper is clear and interesting, there are several issue that need more clarification as listed below.

* [Hierarchical Bayesian models](https://www.biorxiv.org/content/10.1101/621540v1.article-info) are already developed to study the dynamics in *C. elegans*. How would the results change if the anatomical constraints are imposed on the already established models such as the one mentioned above. Some comparison with this paper would make the claims stronger.
* In the results, why neuron-held-out correlation is worse than worm-held-out? Am I misunderstanding this? At least intuitively shouldn't we expect to have better performance when a single neuron is missing compared to all the neurons from a worm missing?
* How did the authors perform neuron-hold-out experiments? If a neuron is missing in the training dataset, then how do we have a latent variable corresponding to it? Is the missing neuron treated as missing observations? At the test time, do we use the "reconstruction" for evaluating the correlations? Is the reconstruction the mean of the posterior or is it based on samples?
* For the neuron-hold-out experiments, pairs of neurons (107) are considered. Why is there 107 pairs and why are we considering pairs of neurons instead of removing a single neuron? Again I feel like I'm misunderstanding the neuron-hold-out experiments.
* Is it possible to visualize the reconstructed activity of all the neurons (for a hold-out-worm and training worm) in the appendix? This would be important for biology experts and can provide further insights into the proposed model.
* How should we ensure that the model is learning time dynamics? The factorized Gaussian family does not incorporate any dynamical model or smoothness constraints in my understanding, is that right?
* When using anatomical constraints for model selection we need to control and account for as many confounding factors as possible that can potentially lead to misinterpretations. These factors include (but not limited to) controlling for the number of parameters across different models (more parameters can lead to overfitting, can make the optimization harder, can make the inference harder since it's in a higher dimensional space), and successful training (what is the convergence criterion?). 
* In Fig. 3 the error bars seem too large, are the differences significant in the violin plots? In addition, some neurons have really bad predictive performance, can we focus on those neurons to understand why this is happening? Is it due to lack of measurements form their neighboring neurons? Is it because they are less predictable and more impacted by unmeasured sources of variability?


I have some minor comments to, and would like the authors to clarify.

* What is $\theta$? Please write down $\theta$ explicitly using the notations introduced in the paper.
* According to Wikipedia, *C. elegans* has 302 neurons, not 300. Can you clarify this?
* What is $\phi$? Are you using a neural network to parametrize $\phi$? If so what kind of architecture is used?
* How does the regularization work? What is the probabilistic interpretation of adding the $|| |M| -|C| ||$ regularization? Is this equivalent to defining a normal prior? Please clarify this. Why did the authors use $L_2$ regularization? Isn't $L_1$ regularization associated with sparsity? If so, what is the probabilistic interpretation of this in the variational inference framework?





summary_of_the_review: While the approach is very interesting and the problem considered is important, the paper lacks comparisons and clearer description and investigation of the results. Furthermore, some justification on specific choices need to be done. See above for more detailed comments.

Post discussion: I'm increasing my score to "accept" since most of my comments are incorporated, I thank the authors for their hard work in revising the paper and incorporating additional experiments/visualizations.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The authors propose a biologically constrained latent linear dynamical model of the C. elegans nervous system. They use connectomic information (including chemical vs electrical synapses) to constrain connections between units during inference. They fit the model to calcium imaging data from whole C. elegans using a variational approach. They find that biological constraints improve model performance and validity using both withheld neuron and across worm metrics.
main_review: This paper makes some additional developments on the C. elegans latent variable modeling literature: including both chemical and electrical synapse information, adding conductance-based synapses, and nonlinear calcium observations with a time constant. However, I don’t believe it’s the first linear latent variable model to consider connectomic constraints (or fitting linear models to C elegans data with variational inference). See equation 1 in Mena et al (2017). Also see Linderman et al (2019) for cross-worm comparisons and predictions of held-out neurons.

The clarity of the methods and conclusions could be improved.
Methods;
-Does 'g' in equation 5 denote the softplus? It's not clear whether the 'same' in the sentence before refers to g being the same across neurons, same as the softplus definition earlier in the section or both.
	-Is there a reference supporting this observation model? Is the threshold of 'g' fixed?

-Page 4 section 2.1 redundantly refers to Bargmann et al about nonspiking neurons in C. elegans twice in the first two paragraphs. I'd suggest moving this reference earlier to the introduction where you first mention that you're using a nonspiking model as many in the ICLR audience may not know this fact and it'd more quickly support your modeling assumptions.

-Last paragraph of page 3 "A latent variable model....". I found this paragraph a little confusing. I wasn't sure what many sentences were referring to, especially with the introduction of the "black-box" networks (2nd sentence vs the black-box CCN brought up in the third sentence).

-The explanation of variational inference over SMC (top text on page 3, starting “inference instead of SMC”) was a little unclear what the authors wanted to convey with “generates a different type of distribution and is more computationally efficient because it allows direct sampling from the approximate posterior.” A more explicit definition of “different” would help clarity (or removing this altogether) and it’s also unclear whether sampling from the posterior is used to help validate/apply the model here.

Conclusions:
-The conductance-based synapses showed a performance improvement in the neuron holdout, but not really in the worm holdout condition. Why is this and does it impact the conclusions?

-How does the calcium observation model impact the model fit? This is an important contribution of the proposed model, but it's not clear if the Ca dynamics are necessary compared to, for instance, a simple linear-Gaussian model (given the observation window size).
summary_of_the_review: The paper highlights that known biological constraints can be included tractably within latent variable models of biological neural networks (in this case, the C. elegans). Moreover, they provide some evidence that these constraints improve model fits.
The model goes beyond existing linear latent variable models of C. elegans. However, the conclusions do not include strong measures comparing against existing methodology and it's difficult to gauge how much advancement this model provides. While the presentation clarity was adequate, organization and model presentation could be improved.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper uses VAE (variational autoencoder) to model neural activities of C.elegans observed in calcium imaging. The main contribution is making the encoder-learned posterior distribution close to a biophysically constrained prior, as well as designing the decoder based on biophysical rules.
main_review: Strength
- Adding biophysical constraints into neuron activity modeling, and conducting both neuron and worm hold-out experiments to validate the model.
- Alleviate the issue of trail-to-trail variability.

Weakness — not clearly written at all:
- About the prior p(v|h) generation: it only mentions “Pθ(v|h) is a biophysically realistic connectome-constrained network with passive point-neuron dynamics” without pointing out what the network is: are the only trainable parameters in the network W^c_ji and W^e_ji, and a MLP  layer for encoding h_i(t), or are there any other layers? And since it has subscript θ, the same as the decoder network, should I assume they are the same network? If so, how can the decoder be trained since it needs the prior to calculate KL divergence for the loss?
- There’s no detail about the exact network architecture, layer, input/output dimensions, data processing/ training procedure, etc. This makes the whole logic even harder to comprehend.
- Apart from clarity, the technical novelty in computer science is a bit insignificant. Empirically, the results are reasonable but not interesting enough, and no comparison is made with other baseline modeling methods. I would suggest a re-writing (rephrasing, better connecting between sections, putting many appendix contents like A2 and A5 into the main text, etc.) and re-submitting to a neuroscience journal.
summary_of_the_review: Overall, the paper provides a good aspect in modeling neuron activities by using connectome constrains as VAE prior, but it’s not clearly written and lacks novelty or significance in both technical and empirical aspects.
review_title: null

Meta-review: The authors build an encoding model of whole-brain brain activity by integrating incomplete functional data with anatomical/connectomics data. This work is significant from a computational neuroscience perspective because it constitutes a proof of concept regarding how  whole brain calcium imaging data can be used to constrain the missing parameters of a connectome-constrained, biophysically detailed model of the C. elegans nervous system. There were issues related to clarity in the initial submission which all appeared to have been addressed in the final revision. This paper received 3 accepts (including one marginal accept) and 1 reject. The paper was discussed and the reviewers (including the negative reviewer) were unanimous that the current submission should be accepted.
Decision: Accept (Poster)