Connectome-constrained Latent Variable Model of Whole-Brain Neural Activity
Authors: Lu Mi
Richard Xu
Sridhama Prakhya
Albert Lin
Nir Shavit
Aravinthan Samuel
Srinivas C Turaga
Venue: ICLR 2022 Poster (2022)
Keywords: connectome
latent-variable model
variational autoencoder
biophysics
whole-brain
neural activity
calcium imaging
caenorhabditis elegans
voltage
generative model
inference network

Abstract: The availability of both anatomical connectivity and brain-wide neural activity measurements in C. elegans make the worm a promising system for learning detailed, mechanistic models of an entire nervous system in a data-driven way. However, one faces several challenges when constructing such a model. We often do not have direct experimental access to important modeling details such as single-neuron dynamics and the signs and strengths of the synaptic connectivity. Further, neural activity can only be measured in a subset of neurons, often indirectly via calcium imaging, and significant trial-to-trial variability has been observed. To address these challenges, we introduce a connectome-constrained latent variable model (CC-LVM) of the unobserved voltage dynamics of the entire C. elegans nervous system and the observed calcium signals. We used the framework of variational autoencoders to fit parameters of the mechanistic simulation constituting the generative model of the LVM to calcium imaging observations. A variational approximate posterior distribution over latent voltage traces for all neurons is efficiently inferred using an inference network, and constrained by a prior distribution given by the biophysical simulation of neural dynamics. We applied this model to an experimental whole-brain dataset, and found that connectomic constraints enable our LVM to predict the activity of neurons whose activity were withheld significantly better than models unconstrained by a connectome. We explored models with different degrees of biophysical detail, and found that models with realistic conductance-based synapses provide markedly better predictions than current-based synapses for this system.

Published as a conference paper at ICLR 2022
CONNECTOME-CONSTRRANEDD LATENT VARIABLE MODELS OF
NEURAL ACTIVITY
Lu Mi Richard Xu', Sridhama Prakhya' Allbert Lin", Nir Shavit", Aravinthan D.T. Samuel Srinivas C. Turaga
1 HHMI Janelia Research Campus
2 Harvard University
3 MIT
(xur, prakhyas, turagas jjanel haw...hi.... .org (a bertlin, samuell 1 @g. ha rrvaadddee eddu {lumi, shanir) @mit.edu
ABSTRACT
The availability of both anatomical connectivity and brain-wide neural activity mea- surements in C. elegans make the worm promising system for learning detailed, mechanistic models of an entire nervous system in a data-driven way. However, faces several challenges when constructing such model. We often do not have direct experimental access to important modeling details such as single-neuron dynamics and the signs and strengths of the synaptic connectivity. Further, neural activity can only be measured in subset of neurons, often indirectly via calcium imaging, and significant trial-to-triaa variability has been observed. To address these challenges, we introduce camnectome-consrrcc
latent variable model (CC-LVM) of the unobserved voltage dynamics of the entire elegans nervous system and the observed calcium signals. We used the framework of variational autoencoders to fit parameters of the mechanistic simulation constituting the genera- tive model of the LVM calcium imaging observations. A variational approximate posterior distribution over latent voltage traces for all neurons efficiently inferred using inference network, and constrained by prior distribution given by the biophysical simulation of neural dynamics. We applied this model to an experimen- tal whole-brain dataset, and found that connectomic constraints enable our LVM to predict the activity of neurons whose activity were withheld significantly better than models unconstrained by comnectome. We explored models with different degrees of biophysical detail, and found that models with realistic conductance- based synapses provide markedly better predictions than current-based synapses for this system.
INTRODUCTION
The anatomical connectivity of the entire C. elegans nervous system, including both chemical and However, well-calibrated and predictive connectome-constriindd mechanistic models of this nervous system have yet to be demonstrated [8] 7). This is partially because whole-brain recordings are insufficient com- pletely constrain computational models. First, the single-neuron and synapse dynamics are generally unknown. Second, the connectome does not directly inform the signs and strengths of individual synapses. Third, the response properties of sensory neurons are incompletely known. Further, it is unclear what level biophysical detail necessary to reproduce the essential computational function of the C. elegans nervous system.
electrical synapses, been known for several decades (3) 27 31
Here, we used whole-brain calcium imaging data to constrain the missing parameters a connectome- constrained, biophysically detailed model of the C. elegans nervous system. We started with simplified non-spiking passive point-neuron model of the voltage dynamics of individual neurons in the circuit. We modeled inputs the neurons from both electrical and chemical synapses.
I cqual contribution

---

Published as a conference paper at ICLR 2022
The chemical synapses are modeled nonlinearly, with either current-based or conductance- based biophysics. The challenge of fitting such a model to data is two-fold. First, the voltage dynamics of the neurons are not directly observed, but rather indirectly measured via their slow calcium dynamics. Second, there is significant trial-to-trial variability in neural activity. suggesting strong initial state dependence in the neural responses to the same sensory stimulus (9. These issues can both be addressed by treating the collective voltage signals of all neurons in the nervous system as an unobserved latent variable whose dynamics are determined by the simplified connectome-constraindd biophysical model with unknown neuronal and synaptic parameters.
Our womnectome-cnsttranne latent variable model (CC-LVM) of voltage dynamics of the C. elegans nervous system is a large-scale latent variable model with a very bigh-dimensional latent space consisting of voltage dynamics of 300 neurons over minutes of time at the simulation frequency of 160 Hz. The generative model for these latent variables is described by stochastic differential equations modeling the nonlinear dynamies of the network activity. We developed a variational autoencoder- based framework for inferring the unobserved voltage dynamics from the observed calcium dynamics of only a subset of the neurons in the nervous system.
The C. elegans nervous system consists of 300 neurons, divided into 118 distinct classes (often bilaterally symmetric pairs) 126] 14, 25). These neurons can be categorized into sensory neurons, internuurnns and motor neurons. The majority of these neurons, about 200, are concentrated in the head of the animal. The synaptic connectivity of C. elegans has been mapped with electron microscopy, providing researchers with complete connectome 30 31.
We applied the CC-LVM to whole-brain calcium imaging data which captured 170 of the 300 neurons in multiple worms as they responded to chemosensory stimuli (2) In principle, an accurate model of the nervous system constrained by incomplete activity measurements but a complete description of the animal's anatomical connectivity can enable accurate predictions of activity in neurons which were not recorded. We tested this hypothesis by using the CC-LVM to predict the activity of neurons which were measured but whose activity was withheld during model training. We also used the model to predict the activity of entire worms, by holding out single trials during training. The CC-LVM predicted activity of withheld neurons and withheld worms significantly better than models unconstrained by the connectome, demonstrating the utility of the connectome even when little is known about the signs and strengths of individual connections. Further, we found that models with conductance- based synapses provide superior predictions to models with current-based based synapses. This is surprising because while conductance- based synapses are more mechanistically accurate, they do not necessarily support efficient inference [8] CC-LVMs thus provide new tool for connectome and activity constrained modeling of neural circuits and for discovering the appropriate level of detail of biophysical model for given system. This work provides framework not only for modeling the C. elegans nervous system in particular, but also the neuronal networks of other biological systems.
1.1 PRIOR WORK
Previous work in modeling the C. elegans nervous system has focused on creating network simulations based on anatomical connectome data with unknown single neuron synaptic biophysics [18] 28 [9
13. These network models provide a holistic view of the entire nervous system and were validated by comparing simulated locomotion with movements of live animals [Л]] However, they were not able fit against prerecorded calcium fluorescence data, so their simulated neuronal activities were not confirmed. Work has also been done on modeling unlabeled neuron populations but these models cannot predict activity at a single neuron resolution because the activity data is not mapped to specific cells (4. Recent advances in machine learning have enabled increasingly sophisticated latent variable models that uncover structure from data generated by advanced neural interfacing technologies (21. This framework has been used in other contexts to infer neuronal voltage dynamics from high- higg-diimnsiooal calcium fluorescence recordings 20 I 24 Despite their utility, these models do not account for connectomic data. Our work employs both LVMs and connectomic constraints to create model that is informed both by neural population dynamics and anatomical network structure.
Recently. Bayesian models have been widely applied in neuronal datasets to infer variables such as spiking activity. neural dynamics, and connectivity [8] (2). In particular, Linderman et al. [[7] created a hierarchical state space model of C. elegans neural activity. However, because such a model is not mechanistic, it cannot predict the activity of neurons that were never measured. Warrington
2

---

Published as a conference paper at ICLR 2022
Generative mode!
Patl.voo)
(o) C. elegans connectome A Whis, of 7985
(0) Recorded calcium dynamics
sensary neuron
res mounon
rpconder fonded rourse
VCC
WEARS
perponded reurzn
electical syrapse
unscanded
1,40
T2200
N7:50
1968
ol (u) Inferred lasent dynamics
(d) Labeled whole brain caleium imaging
(b) Inference network Gulvh.ol
4p
120 neurone recarted
een recosiin oingig lanss
uap moont
grscaeded
comcitter lyyee e sparrplnn byyes
(+) Trini-o-trial varisbillty in mensured calcium fluoreecance trace
Tyea
NU
10
Figure 1: Anatomical and functional whole-brain datasets available in C. elegans. a. The connectome- constrained generative model produces prior distribution of voltage dynamics and fluorescence traces given chemosensory stimulus. Observed data highlighted in color and r the upsampling factor from the acquisition rate to the simulation rate. b. The inference network generates a posterior distribution voltage dynamics given observed fluorescence traces and chemosensory stimuli. c. The connectome of C.elegans nervous system, with chemical connections in blue and electrical connections in orange [30] Neurons recorded in the functional imaging dataset [32] are colored green. d. Labeled whole-brain calcium imaging of 170 neurons in the head of the worm while a panel chemosensory stimuli were presented [32). Deterministic multicolor labels allow for the identification of all neurons. e. Sample fluorescence activity traces of three measured neurons, with stimulus delivery marked by the colored bars. Note the significant rtal-to--trial variability in neural activity.
et al. 28 used sequential Monte Carlo (SMC) to impute the intracellular voltage potentials of C. elegans neurons from 49 recorded calcium traces (12). Their work combined neuronal, body and calcium observation simulators in order model locomotion 3] Their simulator produced series of exemplar neuronal voltage traces, but they did not test their model's ability to predict the activity of unrecorded neurons, so the accuracy of their inferences remains unknown.
Our model uses a variational auto-encoder (VAE) to perform inference instead of SMC which is more computationally efficient because it allows direct sampling from the approximate posterior. The CC-LVM was trained on activity recordings of 170 neurons (32), a large fraction of the 300 total neurons in the C. elegans nervous system (29 31. We validated our model by evaluating its ability to predict the activity of neurons held out from the training data. Finally, we searched a space of
generative models and optimized voltage predictions by comparing them to calcium activity data.
CONNECTOME-CONSTRRANNDD LATENT VARIABLE MODEL
We constructed connectome-constann system, where each node in our network represents a specific neuron in the animal. The activity of each neuron was modeled with a latent variable analogous to voltage. Since the neurons in C. elegans do not spike 2], the dynamics of cach neuron in the network is represented by a stochastic non-spiking passive leaky integrator equation with learned time constants and resting membrane potentials [5) The neurons were coupled by both chemical and electrical synapses with learned weights [5). We allowed these weights to be non-zero only where the connectome indicates the existence of synapses. Given a set of learned parameters (weights, time constants, and resting membrane potentials), the dynamics of the network define prior distribution over neural activity trajectories. Importantly, the stochastic nature of the neuron voltages causes deviations from perfectly deterministic dynamics, allowing us to model the observed variability in single-neum neuron dynamics. This variability has several potential sources, including the unmeasured initial states of the neurons and our incomplete knowledge of the sensory inputs driving the nervous system.
latent variable model (CC-LVM) of the C. elegans nervous
A latent variable model of this scale with a nonlinear generative model defined by the stochastic dynamics is difficult to fit because of the high number of parameters. To address this challenge, we used the probabilistic inference framework of variational autoencoders (VAE) (2 to train black-box voltage inference network to predict posterior distribution over neural activity trajectories. We then used this inference network to train the parameters of the CC-LVM. The resulting LVM has a biologically realistic generative model of the nonlinear neural dynamics of the C. elegans nervous system, and black-box temporal convolutional inference network which, given sensory stimulus
3

---

Published as a conference paper at ICLR 2022
and calcium imaging measurements, predicts factorized Gaussian distribution over the voltages of all the neurons in the network. Several variants of the LVM were developed: we tested models in which the synaptic connections were modeled either current-based or conductance-baat -bssed [5], and evaluated different types of connectome constraint. The LVM was optimized with the ELBO (evidence lower bound) objective.
2.1 NETWORK MODEL WITH PASSIVE POINT-NEURON VOLTAGE DYNAMICS
Neurons in the C. elegans nervous system are largely non-spiking 22, so we model the voltage dynamies for these neurons as a passive point neurons with a single electrical compartment. Let E RV denote the voltages of the N neurons. The voltage (t( for each post-synaptic neuron at the time was calculated using a first-order leaky integrator equation given by
(t), + (t) st( + (t) + + of + (t).
(1)
where is the voltage time constant, o; is the chemosensory input provided to only the sensory neurons, sf is the chemical synaptic input, of is the electrical synaptic input, wpeat is the resting neuron voltage.
We studied two variations of the model, the current-based model and the conductance- -based model, which differ in their formulations of chemical synaptic input 8f. Since neurons in the C. elegans nervous system are largely non-spiking, we model the chemical synapses as having graded release of neurotransmitter, rather than the all-or-none quantal release seen in spiking neurons. In both models, we model the amount of neurotransmitter released, (1 (t)). in proportion to the pre-synaptic voltage 1 followed by softplus activation g(-) which sets a minimum voltage below which there no synaptic release. We use g(-), denote softplus function for the rest of the paper. In our current- model, synaptic input se to post-synaptic neuron i is directly proportional pre-synaptic neurotransmitter concentration:
N
(t) = Σ (f))
(2)
where W represents the chemical synaptic weight between pre-synaptic neuron and post-synaptic neuron i. W,, can be positive or negative depending on if the synaptic connection is excitatory or inhibitory, If neurons j and i are not connected, Wf. is set to zero. In the conductance-based model, we model the synaptic current entering the post-synaptic neuron with more biophysical detail as
N s5()) = (E,, (()W,,((0)
(3)
Here, the pre-synaptic neurotransmitter concentration Wf (f)) more accurately modeled as proportional to the conductance at the post-synaptic terminal. The post-synaptic current is then given by the product of the synaptic conductance and the difference between the post-synaptic voltage (t) and the synaptic reversal potential Ej..
In contrast to the current-based synapse, whose input independent of the post- nost-synaptii voltage i (t). the more biophysically accurate conductance- synapse model also has a dependence on the post-synaptic voltage. Additionally, this model decouples the sign of the synapse from the strength. The reversal potential of a synapse F.j: dictates whether a synapse is excitatory or inhibitory. A large and positive Ej corresponds to an excitatory synapse causing depolarization of the postsynaptic neuron. Conversely, an inhibitory synapse will have negative Ej; causing hyperpolarization. In this model, we can now independently train the sign of a synapse and its non-negative strength W which is not easily possible with current- based synapses. In both the current-basee and conductance-baas models, the following equation was used to represent electrical synaptic inputs:
N Σ
(t)
(4)
where WI. is restricted be non-negative and vy is the potential difference between presynaptic and postsynaptic neurons. We also restrict W = We because the potential differences between electrical synapses are symmetric. To directly compare the outputs of the LVM neural activity
4

---

Published as a conference paper at ICLR 2022
measurements, model must generate calcium signals from the voltage traces. We model the calcium concentration [Ca], of each neuron i as first-order leaky integrator, driven by voltage-gate calcium channels with the same nonlinear current-voltage (I-V) function g(ve):
[ca[[Ca):(() + [Cal,(t) g(v:(1),
(5)
where g()) represents softplus activation, 7[Ca] is time constant shared across all neurons. We then map calcium concentration [Ca] into the measured calcium fluorescence signals f via an affine transform with scalar a- and bias B*,
fi(t) = a [Ca),() + B5 + (t)
(6)
with measurement noise represented by noise amplitude o, and noise term (t) NN(0, These equations are simulated in discrete time using Euler integration.
2.2 VARIATIONAL. INFERENCE
Note that voltage dynamics in equation iis a deterministic equation. We provide further flexibility to the model in order to model the significant trial-to-trial variability we observed in Figure Such variability could be caused by different initial state of neurons, or by unknown, unmeasured inputs from environment. In order to address this, we allow for the voltage dynamics to deviate from the deterministic dynamics, and introduce a Gaussian prior over these deviations. This results in the following stochastic version of the voltage dynamics,
(7) Where of is the noise amplitude, and the standard normal noise term is of (O With this approach, we re-formulated the neuronal voltage as a latent variable a with the generative model given by 7 and must perform Bayesian inference to estimate their value. Exact Bayesian inference is intractable for this sigh-dimensional nonlinear stochastic dynamical system. For this reason, we developed computationally efficient strategy for inferring an approximate posterior distribution P(v)f) over the latent variable given the measured fluorescence data f using variational Bayes in the framework of the variational autoencoder [15] see derivation in Appendix A.
(t),(t) + vt( = (t) + st) + u + o,(t) + (.]e
2.3 INFERENCE NETWORK
The inference network @o(uff o) generates the approximate posterior distribution of the latent variable v (voltage for all neurons at all time points) given the measured fluorescence data f and the sensory inputs It consists of multiple ID convolutional layers that transform the input data into a Gaussian distributions (t(() ~ (N)(()), oz (()) for each neuron = 1,.., N at every time point t 0 .... D. The approximate posterior distribution Qo [ o) is factorized as
Qo(uff.o) = II
Qo C=O....D
(8)
D is the length of the time window of data fed into the inference network, and @ is its parameters.
2.4 GENERATIVE MODEL
The generative model outputs the calcium fluorescence trace f and neuronal voltage e given sensory inputs It can be formulated as
Po(f = Pe(flu.o)P&(eio)
where contains parameters of the generative model. Po(v)o) a biophysically realistic connectome- constrained network with passive point-neuron dynamics. It outputs prior distribution over voltage v given sensory input o. We use a nonlinear sensory mapping I/ to transform our chemosensory input to neuronal stimulus to the sensory neurons. The stimuli is represented as set of binary vectors hx(t), which are given the presence of stimulus at time and 0 otherwise. (offle, o) is another realistic model that maps the voltage into reconstructed fluorescence f. given sensory inputs The voltages u are sampled from the approximate posterior distributions generated by the inference network. Details of both distributions are described in Appendix A.2
5

---

Published as a conference paper at ICLR 2022
model
current-based
conductance-basee
weightaaammeter
learned dense
learned conne. sparsity
learned
learned Iearned
conne.
conne.
conne.
sparse
dense
sparse total count sparsity
count
count
range T
00, xc] 0o, 20)( co, 00)
[0, 0o)
(0)
[0,00
0,
co)
(0,00)
[
0
1
1
1
[
0
W
M
v
a
0,01
0.01
0.01
0.01
0.01
0.01
0.002
range
[0, 00)
(0)
[0, 00)
00)
(0)
00)
00)
00,00
00
1
1
[
0
1
1
[
>
0]
W
Mi
V
V
V
0.01
0.01
0.01
0.01
0.01
0.01
0.065
Table Definitions and constraints for the LVM variants. We evaluated a total model variants. Across these variants, we explored two methods of modeling chemical synapses: current- based and conductance-basee (Sec. 2.1). We also lested several different levels of connectome constraint. (Sec. (2.6). For both the chemical synapse ji and electrical synapses W the weight matrix W M,,, where T3, is the sparsity matrix, M₁t is the matrix connection magnitudes, and a global scalar. Trainable parameters are indicated by v. We compared the comeecoomecconstraieed versions to two unconstrained learned models, a Iearned dense model in which all neurons are connected, a learned spanse model using a L1 regularization |M [|. and a learned total
count model with L1 regularization with connectome synapse count C' using |I ||| M [|1 ||C||
2.5 OBJECTIVE FUNCTION FOR GENERATIVE MODEL AND INFERENCE NETWORK
The variational objective of our latent variable model uses the ELBO The reconstructed calcium fluorescence f is then fit to the measured fluorescence data by maximizing the ELBO for each measured time point. The objective function is combination of reconstruction loss between the reconstructed and measured fluorescence traces and the KL divergence between voltage prior and posterior. (details in Appendix A.5)
L --EEy-qirll -(eu-Q( [log (P& (fle, 0))] + DKL (Qo(elf.o)
(10)
Optimization of the KL divergence with the reconstruction loss allows our inference network to learn the time dynamics of our neuronal voltages because the difference between the posterior and prior distributions is made small.
2.6 CONNECTOME CONSTRAINT FOR MODEL WEIGUTS
To build the model using chemical and electrical synapse count, C and C as connectome constraint, we factorized the chemical synaptic weight W as Wf. =f where a represents a global scaling factor which scales synapse counts to per-synapse currents or conductances depending on the model class. Tý binary value which indicates the connectivity, and MG, reflects the magnitude of connection strength. The electrical synaptic weights matrix We. is similarly defined We compared the performance the connectome constrained models to
ja
as W5: = a T unconstrained ones (Table
mathematically as two N x df Appendix (A.6). The anatomical connectome can be represented matrices: one containing the chemical synapse counts (C"), and one containing the sizes of electrical synapses (C"). We designed the connectome sparsity constraint by fixing the sparsity of our model to that of the connectome. For chemical connections, the model sparsity T5 ICP, O). and likewise for electrical connections, Tí. 1 0j. For both chemical and electrical connections, the magnitudes Mj, and Mj, are trainable.
We applied the connectome sparsity constraint to both current and conductance- based versions the LVM. In the conductance-basse model, we were able to apply the even stronger connectome count constraint: directly assigning the synapse counts the chemical connection magnitudes (Mf, = C and the synapse sizes to the electrical connection magnitudes (Mf. Cj.). Under this constraint, both the sparsity and magnitude are fixed, and only the global scaling factors a and a are trainable. Additionally, we evaluated another variant of this constraint, connectome count". This constraint uses parameters of connectome count- to initialize M;.- MG. is then allowed to deviate from C. during re-training. We compared our connectome- consorrinnd models to two unconstrained models. The learned dense model used fully connected sparsity matrices T and T", with the magnitudes M and M as trainable parameters. The learned sparse model uses Ln regularization both magnitude matrices: | [| and M The learned total count model preserves the total synapse count for
Sinne in the cuseent-based model, the range of W is (c),, and because the signs of the synaptic connections are unknown, we could not directly apply the connectome count constraint it.

---

Published as a conference paper at ICLR 2022
both magnitude matrices: ||| 1a and | 12 a and a are 0.002 and 0.065 respectively, which is the same in the conne. model. Thus, it regularizes the sum of elements in M and M equal to that of C and C
EXPERIMENTS
3.1 DATASET
We applied the CC-LVM to a calcium imaging dataset in which immobilized, pan-neuronally labeled C.elegans were presented with a panel of chemosensory stimuli (2-butanone, 2,3- ..3-pentanedione, and NaCI) (32) The activity of 170 neurons in the head of the animal was measured across 21 individuals. Each recording lasts for 5 min, with an acquisition rate of Hz. In this dataset, every neuron in the worm brain was identified in vivo using deterministic multicolor landmark, allowing whole brain dynamics to be mapped onto the anatomical connectome with single-cell resolution for the first time. The connectome constraints we applied utilized the anatomical connectivity data from (30). The complete hermaphrodíte C.eegans was reconstructed using electron microscopy, and the connections between its 300 neurons were mapped. The connectome contains 3464 chemical synapses and 1031 electrical synapses. More details can be found in appendix A.6
3.2 NEURON HOLDOUT EVALUATION
We hypothesized that the CC-LVM which incorporates the complete connectome but has access only to the measured activity of a subset the neurons may nevertheless be able to accurately predict the activity of unmeasured neurons. This would constitute experimentally testable predictions from our mechanistic model. We tested this hypothesis by performing neuron holdout evaluations, withholding single bilateral pair of measured neurons from the model during both training and testing. Note that our latent variable model always infers the voltages of all neurons regardless of how many neurons observed in any experiment, since the latent states the voltages of the entire nervous system. The goal of this evaluation is to test the model's ability to predict the neural activity of neurons which were never observed at any time. We fit a version of the model with pair of neurons withheld, and then compare the model predictions for the same pair of neurons with the ground truth calcium fluorescence measurements. We repeated this evaluation for all 107 measured neuron pairs. Because many of the neurons in C. elegans are bilaterally symmetric, the symmetric pairs tend to have high correlations with each other. In order to prevent the model from predicting one neuron solely based on its sibling, we removed the symmetric neurons in groups of two. For every neuron pair, we performed the neuron holdout evaluation for both current and conductance models with different levels of connectome constraints. For these experiments, (details in Appendix (A.4) between the predicted and measured calcium fluorescence magnitudes to quantify the predictive performance (Figure 2 and Table 2, more predicted traces in Appendix A.9). For each evaluation, we trained 4 models with random initialization, and reported both the mean value and standard error (SE) for each model. We found connectome sparsity and connectome count constrained models were better at predicting traces of held-out neurons than unconstrained models.
calculated the correlation coefficient
3.3 WORM HOLDOUT EVALUATION
Another evaluation method we performed was to hold out the data from handful of individual worms, train the model on the remaining worms, and predict the activity of the neurons of the held-out individuals. For each of the 9 model variants, we trained on 15 worms, and tested the model on 6 withheld worms. In each case, we trained 4 models with different random initialization. For the worm holdout traces, we calculated the correlation coefficient between the predicted and measured calcium fluorescence. We also reported ELBO. reconstruction loss and KL divergence (Appendix (A.4) on tested worms to represent the fitting performance. We reported the mean and standard error (SE) from models trained with 4 different random initializations. We found that models using the connectome count constraints performed better at predicting calcium traces from held out worms in Table
7

---

Published as a conference paper at ICLR 2022
model
current-based
conductance-based
learned
learned conne. learned learned
learned
conne.
conne. conne.
holdout
metric
dense
sparse sparsity
dense
sparse
total count sparsity count'
count
mean
-0.104
0.001
0.087
0.081
0.008
-0.033
0.262
0.318
0.319
neuron
corr f
SE
10.013
#0.010
#0.021
+0.012
=0.011
#0.011
#0.016
#0.016
10001
mean
0.465
0.439
0.440
0.457
0.445
0.470
0,463
0.474
0.468
cor
SE
+.0006
+0.007
+0.006
+0.006
-0006
+00006
+0.006
10.006
1.0006
ELBO
mean
-0.772
-0.826
-0.803
-0.849
-08662
-0.931
-0.898
-0.854
-0.697
worm
SE
10.037
#0.036
#0.036
10.033
=0.035
10.033
10.032
#0.036
1.066
recon +
mean
0,745
0.765
0.754
0.771
0.780
0.805
0.082
0.744
0.636
SE
+0.037
+0.037
+0.035
+0.033
= -0.035
+0.032
+0.031
+0.035
+0.065
KLD L
mean
0.028
0.061
0.049
0.078
0.082
0.126
0.077
0.110
0.061
SE
0.001
+0.003
+0.002
+0.003
-0.002
+0.005
+0.003
+0.006
+0.002
Table 2: Holdout evaluation results. We performed neuron holdout evaluations on each of the 107 measured neuron pairs in the dataset, and report the average correlation coefficient between the predicted and measured fluorescence traces (Sec. [.2]. We also performed worm holdout evaluations, witholding of the individuals from the training (Sec. (33). For the worm holdouts, we report several metries: correlation coefficient, ELBO, reconstruction loss, and KL. divergence. Overall, the conductance-basel model under connectome count constraint produced the best predictions among the 9 models.
3.4 COMPARING THE DIFFERENT MODEL VARIANTS
The gold standard for comparing generative models in machine learning to compare the model evidence P(f) on a test dataset. However, this is intractable for many generative models, and so the evidence lower bound (ELBO) is used for model fitting and sometimes also for model comparison. However, the ELBO is not a tight lower bound, and so is not guaranteed to provide the same ordering over models as the model evidence P(f). In this paper, exploiting the interpretable nature of our mechanistic generative model, we developed a stricter criterion for comparing models based on the neuron holdout evaluation. Neuron holdout predictive performance is biologically meaningful, as it evaluates the ability of a model to make testable predictions about neurons which were not directly measured. We found that overall, the conductance-basee CC-LVMs under connectome count constraint make the best neuron holdout predictions quantified by the correlation coefficient. We found that model accuracies at neuron holdout prediction are not very tightly correlated with their test ELBO. While connectome count² still achieves the best performance, the learned dense model achieves the second best performance, despite performing worst in the neuron holdout evaluation. This suggests that the learned dense model has potentially discovered an equally accurate but mechanistically incorrect way to model the calcium imaging data.
Stronger connectome constraint improves the neuron holdout predictions. We found that our CC-LVM with the connectome count constraint produced significantly better predictions than CC- LVM with the connectome sparsity constraint, and both connectome constrained models outperformed unconstrained LVMs with the same dynamics (Table 2 This suggests that the neuron activity pre- dictions are improved by a stronger connectome constraint. Note that connectome count² achieves slightly better performance than connectome count', its parameters get initialized from connec- tome count', then are allowed to deviate during re- re-training. We also explored whether CC-LVMs achieve better performance due solely to the smaller number of trainable parameters. Comparing models to the learned sparse LVM with sparsity regularization, we found that the learned sparse model performed worse. This indicates that the connectome constraint improves predictability due to its topology, not due to the number of trainable parameters.
the maneectome-cossrrainee
Perhaps relatedly, we found superior predictions for interneurons and motor neurons, compared to sensory neurons. While the inputs to motor neurons and especially interneurons are largely contained in the connectome, the inputs sensory neurons are not well constrained by the connectome since they also respond to external stimuli. Instead, we trained a black-box model to predict the tunings of sensory neurons to experimentally presented odors, but this likely does not fully characterize all the sensory inputs to the worm. This result additionally suggests the utility of the connectome, for neurons where the connectome provides information about the dominant sources of input.
Conductance-based models outperform current-based models. We found that conductance- based models achieve better performance in neuron holdout evaluations, despite needing to estimate the reversal potential E in addition to the non-negative strength of a chemical synapse c This could be for several reasons. The biophysical model of conductance-ban based synapse might be a more accurate description of reality. There are also two technical advantages to this model. First, conductance based synapses allow us to make the fullest use of the connectomic data by using the non- negative synapse counts. Second, separating the optimization of the sign of synapse from its magnitude might allow
8

---

Published as a conference paper at ICLR 2022
Neuron holdout predictive performance
(+)
sensory neurons
interneurons
motor neurons
1.0
1.0
correlation
-05
-0.5
-0.5
(b)
2-butanone
2,3 pentanedione
NaC1 measured fluorescence trace
sensory neuron: BAG
interneuron: AVA
motor neuron: DB02
6R/F
(min)
current
current
current
conductance
conductance learned total count
conductance
conductance
learned
learned
connectome
learned
learned
connectome
connectome
connectome
dense
sparse
sparsity
dense
sparse
sparsity
count
count
Figure 2: LVM predictive performance across neuron types. Violin plots comparing the distribution of activity for the 9 LVM variants. Neurons divided into sensory, inter, motor neurons. Measured traces and predictions made by the models for three selected neurons: sensory neuron BAG, interneuron AVA, and motor neuron DB02.
correlation coefficients between predicted and measured
for easier optimization of both. contrast, the sign and magnitude of a current- based synapse are coupled in the same parameter, and have observed strong local minima preventing the efficient optimization of the sign and the magnitude. Accordingly, we found that conductance- based models converge connectome closer the counts given by the original data with less variance due the removal of the neuron sign parameter (Appendix AA7 We found that a synapse initialized as excitatory unlikely to change its sign after optimization. For the same reasons, we are unable to effectively use the non non-neettvve synapse counts contained in the connectomic data to inform the magnitude of the current-base based synapse while leaving the sign unconstrained.
DISCUSSION
In this work, we proposed a connectome-constrrainne latent variable model (CC-LVM) of the C.legann nervous system. Our latent variable framework enables the estimation of the initial state of the network dynamies and compensates for modeling imperfections. This flexibility allows us to fit models to calcium imaging data with significant trial-to-trial variability. In contrast to black-box generative connectome-
models used LVMs neural data modeling [21, we developed a biophysics- constrained, mechanistically detailed generative model for the neural dynamics. We evaluated current- current-based and conductance-bassee models for synapses. We used connectomic data to constrain the model to different degrees. Since CC-LVMs enable the prediction of neurons which were not used to fit the model, we were able to simulate and validate the process of making biologically relevant and experimentally testable predictions of the activity of unmeasured neurons. We found that the conductance-bart based model produces much more accurate predictions in neuron holdout evaluation and worm holdout evaluation. Knowledge of the connectome strongly constrains in silico predictions of individual neurons whose activity were not measured. We hope that this mechanistic model can be used to predict the effects of experimental perturbations, leading causal understanding of neural computation in neural circuits. The research in this paper was enabled by recent advances in whole-brain calcium imaging identified neurons (32
While we have only analyzed the first datasets to be collected in this manner, we hope that rich database of such recordings across diversity of conditions will lead to achieving the dream of standard models of the C. elegans nervous system [1] which generalize across different sensory modalities and behaviors. Rapidly developing whole-brain imaging technologies and advances in high-th thghuhhroughput connectomics connectivity data available other organisms eventually lead to accurate mechanistic models of these larger, more complex biological networks.
making nigh-dimensional neural activity and anatomical
[3 33) [ [6]. We hope that this approach will

---

Published as a conference paper at ICLR 2022
5
REPRODUCIBILITY
We trained each of our LVMs on 1 Quadro RTX 8000. Training procedures are described in We released our software and datasets (https://github com/ TuragaLab wornvae for reproducibility.
Appendix A.3 A.4 A.5
ACKNOWLEDGEMENT
This work was funded by the Howard Hughes Medical Institute. Lu Mi is supported by Intel and Math Works. Albert Lin and Aravinthan Samuel were funded by NSF Physics of Living Systems (NSF 18068 18); NSF Ideas (NSF IOS-1555914) and the NIH (1 UOI N6111697--1), We thank William Bishop, Roman Vaxenburg, Core Francisco Park, Helena Casademunt for comments on the manuscript.
REFERENCES
[1] Aitchison, L., Russell, L., Packer, A. M., Yan, J., Castonguay, P., Hausser, M. and Turaga, S. C. [2017], Model-based bayesian inference of neural activity and connectivity from all-optical interrogation of neural circuit, in 'Advances in Neural Information Processing Systems', Vol. 30, Curran Associates, Inc.
[2] Bargmann, C. I. [1998], 'Neurobiology of the Caenorhabditis elegans genome", Science
[3] Boyle, J. H., Berri, S. and Cohen, N. [2012], 'Gait modulation in C. elegans: integrated neuromechanical model', Front. Comput. Neurosci.
[4] Chen, X., Randi, F., Leifer, A. M. and Bialek, W. [2019]. 'Searching for collective behavior in small brain', Phys. Rev. E.
[5] Dayan, P., Abbott, L. F. et al. [2003], 'hheoretical neuroscience: computational and mathemati- cal modeling of neural systems', Journal of Cognitive Neuroscience 15(1), 154-155.
[6] Dorkenwald, S., Mckellar, C., Macrina, T., Kemnitz, N., Lee, K., Wu, J., Popovych, S., Bae, A.,
Mitchell, E., Nehoran, B., Jia, Z., Zung, J., Brittain, D., Collman, F,, Jordan, C., Silversmith, W., Baker, C., Deutsch, D., Kumar, S., Burke, A., Gager, J., Hebditch, J., Moore, M., Morejohn, S.. Silverman, B., Willie, K., Willie, R., Murthy, M. and Seung, S. [2020], 'FlyWire: Online community for whole-brain connectomics', bioRxiv pp. 1-34.
[7] Gleeson, P., Lung, D., Grosu, R., Hasani, R. and Larson, [2018), 'c302: a multiscale framework for modelling the nervous system of Caenorhabditis elegans', Philos Trans R Soc Lond B Biol Sci.
[8] Gonçalves, P. J., Lueckmann, J.-M., Deistler, M., Nonnenmacher, M., Öcal, K., Bassetto, G., Chintaluri, C., Podlaski, W. F., Haddad, S. A.. Vogels, T. P., Greenberg, D. S. and Macke, H. [2020], 'Training deep neural density estimators to identify mechanistic models of neural dynamics', eLife.
[9] Gordus, A., Pokala, N., Levy, S., Flavell, S. W. and Bargmann, C. I. [2015], 'Feedback from network states generates variability in probabilistic olfactory circuit", Cell.
[10] Izquierdo, E. J. and Beer, R. D. [2013], *Connecting a connectome to behavior: An ensemble of neuroanatomical models of elegans klinotaxis" PLOS Computational Biology 92).. 1-20.
[11] Izquierdo, E. J. and Beer, R. D. [2016]. 'The whole worm: brain-body-ennirooneee models of C. elegans", Curr Opin Neurobiol
[12] Kato, S., Kaplan, H. S., Schrödel, T., Skora, S., Lindsay, T. H., Yemini, E.,, Lockery, S. and Manuel, Z. [2015]. 'Global brain dynamics embed the motor command sequence of Caenorhabditis elegans', Cell.
10

---

Published as a conference paper at ICLR 2022
[13] Kim, J., Santos, J. A., Alkema, M. J. and Shlizerman, E. [2019], 'Whole integration of neu- ral connectomics, dynamics and bio- tio-mechanics for identification of behavioral sensorimotor pathways in Caenorhabditis elegans', biorXiv preprint biorXi::1711.0188
[14] Kimble, J. and Hirsh, D. [1979], "The postembryonic cell lineages of the hermaphrodite and male gonads Caenorhabditis elegans', Dev. Biol. 70(2). 396-31664 417-
[15] Kingma, D. and Welling, M. [2014], 'Auto-cncoding variational bayes', arXiv preprint 1rXiv:1312.614 14.
[16] Lin, A., Witvliet, D., Hernandez-uunez L., Linderman, S. W., Samuel, A. D. T. and Venkat- achalam, V. [2022], 'Imaging whole-brain activity to understand behaviour', Nat Rev Phys
[17] Linderman, S., Nichols, A., Blei, D., Zimmer, M. and Paninski, L. [2019]. 'Hierarchical recurrent state space models reveal discrete and continuous dynamics of neural activity in C. elegans', biorXiv.
[18] Liu, H., Kim, J. and Shlizerman, E. [2018]. 'Functional connectomics from neural dynamics: probabilistic graphical models for neuronal network of Caemorhabditis elegans', Philos Trans R Soc Lond B Biol Sci.
Philos
[19] Olivares, E., Izquierdo, E. and Beer, [2021], 'A neuromechanical model of multiple network rhythmic pattern generators for forward locomotion in C. elegans', Front. Comput. Neurosci
[20] Pandarinath, C.,, O' O'Shea, D. J., Collins, Jozefowicz. R., Kao, C., Trautmann, E. M.. Kaufman, M. T., Ryu, S. I., Hochberg, L. R., Henderson, J. M., Shenoy, K. V., Abbott, L. and Sussillo, D. [2018), 'Inferring single-trial neural population dynamies using sequential auto-encoders', Nature Methods.
[21] Pei, F., Ye, J., Zoltowski, D., Wu, A., Chowdhury, R. H., Sohn, H., O'Doherty, J. E., Shenoy, K. V., Kaufman, M. T., Churchland, M., Jazayeri, M., Miller, L. E., Pillow, J., Park, I. M., Dyer, L. E. and Pandrinath, C. [2021], 'Neural latents benchmark '21: Evaluating latent variable models of neural population activity', arXiv preprint arXiv:2109.044
[22] Rezende, J., Mohamed, S. and Wierstra, [2014], 'Stochastic backpropagation and approx- imate inference in deep generative models', arXiv preprint 101.44440..408.
[23] Scheffer, L. K., Xu, C. S., Januszewski, M., Lu, Z., Takemura, S. Y., Hayworth, K. J., Huang, G. B., Shinomiya, K., Maitin-Shepprd,, J., Berg, S., Clements, J., Hubbard, P. M., Katz, W. T., Umayam, L., Zhao, T., Ackerman, D., Blakely, T., Bogovic, J.. Dolafi. T., Kainmueller, D., Kawase, T., Khairy, K. A., Leavitt. L., Li, P. H., Lindsey, L., Neubarth, N., Olbris, J., Otsuna, H., Trautman, F. T., Ito, M., Bates, A. S., Goldammer, J., Wolff, T., Svirskas, R., Schlegel, P., Neace, F. R., Knecht, C. J., Alvarado, C. X., Bailey, D. A., Ballinger, S., Borycz, J. A., Canino, B. S.. Cheatham, N., Cook, M., Dreher, M., Duclos, O., Eubanks, B., Fairbanks, K., Finley, S.. Forknall, N., Francis, A., Hopkins, G. P., Joyce, E. M., Kim, S., Kirk, N. A., Kovalyak, J., Lauchie, A., Lohff, A., Maldonado, C., Manley, E. A., McLin, S., Mooney, C., Ndama, M., Ogundeyi, O., Okeoma, N., Ordish, C., Padilla, N., Patrick, C., Paterson, T.. Phillips, E. E., Phillips, E. M., Rampally, N., Ribeiro, C., Robertson, M. K., Rymer, J. T., Ryan, S. M.. Sammons, M. Scotr, A. K., Scott, A. I.., Shinomiya, A., Smith, C., Smith, K., Smith, N. I.,, Sobeski, M. A., Suleiman, A., Swift, J., Takemura, S., Talebi, I., Tarnogorska, D., Tenshaw, E.. Tokhi, T.. Walsh, J. J., Yang, T., Horne, J. A., Li, F., Parekh, R., Rivlin, P. K., Jayaraman, V., Costa, M., Jefferis, G. S., Ito, K., Saalfeld, S., George, R., Meinertzhagen, I. A., Rubin, G. M., Hess, H. F., Jain, V. and Plaza, S. M. [2020], 'A connectome and analysis of the adult drosophila central brain', eLife 9, 1-74.
[24] Speiser, A., Yan, J., Archer, E., Buesing, L., Turaga, S. C. and Macke, J. H. [2017], FFatt amortized inference of neural activity from caleium imaging data with variational autoencoders", arXiv preprint arXiv: 1 1.011666..
[25] Sulston. J. E. [1983], 'Neuronal cell lineages in the nematode Caenorhabditis elegans.', Cold Spring Harbor Symposia on Quantitative Biology 48 Pt 2, 443 452.
11

---

Published as a conference paper at ICLR 2022
[26] Sulston, J. E. and Horvitz, H. R. [1977], 'Post-embryonic cell lineages of the nematode, Caenorhabditis elegans', Dev. Biol. 56(1). 110-156.
[27] Varshney, L. R., Chen, B. L., Paniagua, E., Hall, D. H. and Chklovskii, D. B. [201 'Structural properties of the Caenorhabditis elegans neuronal network', PLoS Comput. Biol. 7(2).
[28] Warrington, A., Spencer, A. and Wood, F. [2019]. 'The virtual patch clamp: Imputing e. elegans membrane potentials from calcium imaging'. arXiv preprint 1rXiv:
[29] White. J. G., Southgate, E., Thomson, J. N., Brenner, S. et al. [1986], 'The structure of the nervous system of the nematode cacnorhabditis elegans', Philos Trans R Soc Lond Biol Sci 314(1 165), 1-340.
[30] White, J., Southgate, E., Thomson, and Brenner, S. [1986), "The structure of the nervous sys- tem of the nematode Caenorhabditis elegans', Phils Trans R Soc of Lond B, Biol Sci 314(1165),
LP 340.
[31] Witvliet, D., Mulcahy, B., Mitchell, K., Meirovitch, Y., Berger, D. K., Wu, Y., Liu, Y., Koh, W. X., Parvathala, R., Holmyard, D. et al. [2020], 'Connectomes across development reveal principles of brain maturation in C. elegans', bioRxiv.
[32] Yemini, E., Lin, A., Nejatbakhsh, A., Varol, E., Sun, R., Mena. G. E., Samuel, A. D., Paninski, L., Venkatachalam, V. and Hobert, O. [2019], NNurrPAA: A Neuronal Polychromatic Atlas of Landmarks for Whole-Bras Imaging in C. elegans', bioRxiv 676312.
[33] Zheng, Z., Lauritzen, J. S., Perlman, E., Robinson, C. G., Nichols, M., Milkie, D., Torrens, O., Price, J., Fisher, C. B., Sharifi, N., Calle-Schuler, S. A., Kmecova, L., Ali, I. J., Karsh, B., Trautman, E. T., A, B. J., Hanslovsky, P., Jefferis, G. S. X. E., Kazhdan, M., Khairy, K., Saalfeld, S., Fetter, R. D. and Bock, D. [2018), 'A Complete Electron Microscopy Volume of the Brain of Adult Drosophila melanogaster', Cell 174(3), 730-743.e22.
12

---

Published as a conference paper at ICLR 2022
A APPENDIX
A.1 VARIATIONAL INFERENCE
We used Bayesian inference (VAE) to compute the posterior probability over the latent variables (neuron voltages) given the data f (measured calcium fluorescence),
P(f)))
P(ulf)
(11)
P(f)
To solve this problem, needed compute the model evidence,
P(f) = - f P P(o)dd
(12)
However, computing P(f) mathematically intractable problem, so we instead used variational inference to approximate the value of P(wlf) [15]. Denoting the approximate posterior Q(vlf). we
can formulate evidence lower bound (ELBO) as
log P(f) 2 C [o P(f. v) log Q(flv)],
(13)
which tightens the lower bound on log P(f) as L is maximized. This allowed us to improve the inference network and find the posterior such that Q(vlf) = P(u)f) f), since the Bullback-Leible (KL) divergence, which measures the distance between the distributions Q and P. minimized by maximizing ELBO.
A.2 PRIOR AND POSTERIOR DISTRIBUTION AT DISCRETE TIME STEP
Using Euler discretization with a time step of At, the first-order leaky integrator equation in Equation] I in the network model is
(u At) (t))/AA v1) st(t) s5()) + ufest os(t At),
(14)
Then the voltage (t + At) of neuron at time step t + At is formulated as
(t At) = (t(T:(s((t) + st)) + ureat + o, At) (t)) + (t))),
(15)
This equation indicates that the voltage state (t( at the current time step determined by function over multiple components from the last time step - At, including the past voltage state, the chemical and electric synaptic inputs, and the sensory input:
(t + At) = (1), (t)),,
(16)
For the posterior distribution (v((()) ((t), the inference network outputs (t()) and o,(t) for every neuron i at each time step E. The posterior distribution for the voltage generated by the inference network at each time step is (t) ~ ((,,(t)) (t)), given sequence of measured fluorescence data fO),..., f (DAt) and sensory input o0)).... .... o (Dtt) with total number of steps D + 1. We then used the reparameterization trick to sample the voltage vi v1) from this posterior distribution to generate a sequence of voltage sample from e; (0) ..., va (DAI).
For the prior distribution, we used the voltage samples uy (1) from the posterior distribution at the last step the past voltage state in F to calculate the prior distribution Po(o At) (t At)) at time step f + At. The prior distribution is defined as N (F(((,, (t), (t), (f), o, + At)), of). of is the constant standard deviation of the prior distribution that is trainable during optimization. At the initial time step 0, we defined another trainable parameter μ4 (0) to represent the mean of the voltage prior distribution.
Similarly, we used Euler integration to discretize the first-order leaky integrator equation (Equation 5). The calcium concentration [Ca),(t At) of neuron at time step f. + At is given by
13

---

Published as a conference paper at ICLR 2022
Chemosensory Stimulus o
Fluorescence Trace f
Sensory Encoder
Inference Network
Infer Voltage
Reconstruction Loss
Connectome
Approximate Voltage Posterior Qg(vlf,0)
Constrained
Network
Reparameterized Sampling
Voltage Samples v
Objective Function
Generative Model Po
Divergence
Generative Model Ps Generate
Evidence Lower Bound (ELBO)
Generate Volttage Voltage Prior (a(v)o)
Dn.(Os(vlf, o))l
Trace
Reconstruction Loss KL Divergence
Reconstructed Fluorescence Py(fiv,0)
Figure 3: Schematic of the connectome-constraieee latent variable model (CC-LVM), Sensory inputs are fed through biophyiscally realistic connectome-constriineed network to generate the voltage prior Po (oolo)) To train the model, we first used an inference network to infer from the data the approximate voltage posterior Qe(v))) f). Fluorescence traces are then reconstructed
generative model P& (f) o). We then compared the reconstructed fluorescence traces to the measured traces, and optimized the model with evidence lower bound (ELBO), with combination of reconstruction loss and KL divergence.
[Ca]i(t + Ar) At/TCC - (Ca).(t) + [Ca].(f),
(17)
We used a sequence of voltage samples (t),,, =0, O.... D&t as the inputs to this formula to get sequence calcium concentration from the measured fluorescence trace at time step 0 using [Ca], = (F) - 25)/a² Equation in the generative model Po generates the output distributions Po (t) of fluorescence trace at time step as (Ca((C))((t) + 31, ..). The values a' , are trainable fluorescence parameters.
[Ca,( The initial concentration is estimated
A.3 MODEL ARCHITECTURE
The calcium fluorescence data is given in matrix of dimensions (N,T) where N is the total number of neurons in the connectome. The chemosensory stimuli input is modeled as a matrix of dimensions (3, T) which consists of 3 one-hot encoding vectors that 1 when the stimuli is present and 0 otherwise. Each of the 3 vectors represents one of the three chemical stimuli presented: 2-butanone and p.n-pentanedione, NaCI.
The caleium fluorescence matrix is passed to the input layer of our inference network Q(vlf o) which is a ID convolutional layer. The output of the input layer then gets passed through an upsampling layer, ReLU nonlinearity, another 1D convolutional layer, and then another upsampling layer. The first convolutional layer has 2N input channels, 2NV output channels and kernel size of 11; the second has 2N input channels, 2N output channels and kernel size of 21. The first upsampling layer upsamples by a factor of 4 and the second upsamples by a factor of 10. These factors account for the fact that we simulate our network times faster with upsampling factor r 40 than the recording rate. The sensory input passed through a 1D convolutional layer with input channels, 2.N output channels and kernel size of 21. This is then passed through ReLU and then concatenated to the output of the 2nd convolution on our calcium data. The merged matrix is then 1D convolved again with 2N inputs, 2N outputs and kernel size of 41. This is one of the final outputs of our inference network and gives a (N, 7) matrix that represents the means of our posterior distribution. To generate the standard deviation of our factorized posterior distribution, we pass our merged data through another convolutional layer with the same dimensions but different parameters. We pass that output through nonlinear affine transform (using a softplus nonlinearity) to rescale the output to an appropriate level of noise. The parameters of this inference network are all the weights and biases of the convolutions and upsampling layers described above, which we denote by á the paper.
The input to the generative model the same sensory input that we pass to our inference network. The first layer of our generative model is single layer nonlinear transform that converts our 3- dimensional input to an N dimensional input that is to any non-sensory neurons. The next layer is recurrent neural dynamics, given by Equation [ which passes information to itself in the next time step based on the connectivity given by the connectome. Although some sources say that C. neurons
elegans has 302 neurons, only simulate 300 because two cells originally classified are not connected to the rest of the connectome Witvliet et al. (3I]. The simulated voltage for each neuron in the connectome is passed to calcium simulation layer (Equation 5). The final layer of
14

---

Published as a conference paper at ICLR 2022
model is calcium to fluorescence affine transformation (Equation 6 The output of this final layer then compared to the ground truth fluorescence data to calculate the reconstruction loss. Below, we have included some pseudocode describing the architecture of our network.
Inference "etrencee_Networke ( cwwciumm fluor, missing _ata_mask, sensory _nnput)) calcium_inpul concatenate (calcium, (calcium_fluor, ffuorr missing_data_mant convi_out relu (convl (calcium_input)
upl_out = (psample(conyl__utt conv2 out relu (conv2 (up_ooo_ (upl_out))
up2_out upsample (conv22
sensory_conv_oue = relu (conv3 (snno((sssyyyyinnuut)
merge__calcium_sensor = concatenate (up2_out, sensory_conv_out)
mean mean_latent_nneuon_nwoo
conv4 (merged__calc_sennon)
std [___latnnt_ncuron_vlllll softp lus (conv5 (merged_caleim_sennse ))
sample vvolage wawellatetttttt latent_neuron, neuron_ voltage rand norm std latent seuroo_voolaae
return sample_latem_.nouroorrollll
Generative Generative_Model (samlee latenttattttt_teeeaeeee___eee__e neuron_ sensory_input)>
#Equation neuron pooltage_dynantin = inntggattorccomeettoeeee
sumple_latent_nnurooo.oooo sensory_input)
#Equation
calcium__concentration
= lcaty_interrttorccaliinnnn
#Equation
fluorescence trace = nonlinear affine
affine_ transform
calcium_ con.cetrtnntnniionnnn.mmmm dynamcis)
return fluorescence_trace
A.4 FORMULATTION FOR OBJECTIVE FUNCTIONS AND METRICS
Here we provide the mathematical formulations for the objective functions and metrics we used in this work.
The evidence lower bound (ELBO) a combination of reconstruction log likelihood and the negative KL divergence. The reconstruction log-likelihood, given the predicted fluorescence trace f(()) of neuron i at time step as output distribution (f) distributtin.NN(((())...... And f; is formulated
i((1)= = a² + B5.
(18)
Given the measured fluorescence trace ft(t), number neurons N, and the total number of time steps D + calculated the reconstruction negative log likelihood the reconstruction loss
DA Σ
Cicon
(2 exp
2(01)2 Σ (1.0)) -(00)-((()))
(19)
The KL divergence describes the distance between voltage prior and posterior distribution. We represent the posterior distribution generated from the inference network ((()()), for
15

---

samples for Published as a conference paper at ICLR 2022
neuron i at each time step t. We then sample (t()) from the posterior distribution. According to Equation 16 we represent the mean (t) of prior distribution (t)((t), as
(1) - =(0)==(((())))
(20)
where of is a trainable parameter.
We then use closed-form of KL divergence defined between two univariate Gaussian distributions
DAE N Σ
o.¹ o: (z((t)
) (a:(1))² + (4((1) - &((1)²
DEL
log
(21)
2(
==
1=1
We minimize the objective function C -ELBO during optimization given by
L Lreson + DKL.
(22)
In the worm holdout evaluation, for each neuron pair, we calculated the ELBO, reconstruction loss and KL divergence metrics on each holdout worm, for a total worms. We also trained each model setting 4 times under the same constraints and different initializations. We calculated the mean and
standard error of ELBO, reconstruction loss and KL divergence based on 4 x each model.
For both neuron and worm holdout evaluations, we calculated the correlation coefficients between the measured ft) and reconstructed fluorescence traces ft).. The correlation coefficient for each neuron is simply
DAE
((tt))((t))))))))))))))) (f(()) Fe(t))
T1
t=0
(23)
2 DAr
(R() -(())
Σ (A(11- ((())-)((1))
In the neuron holdout evaluation, for each neuron pair, we trained model in which the ground truth of said neuron pair was removed from the training data. There were 107 pairs and 170 measured neurons in total. We trained each model 4 times with different initializations under each of 8 connectome constraints, for a total of 8 x 107 x 4 = 3424 different models. We calculated the mean and standard error of r; based on 4 x 170 680 samples for each model.
In the worm holdout evaluation, we calculated these metrics on each holdout worm, for a total of worms. Each worm has about 170 measured neurons. We trained each model 4 times under the same connectomic constraints and with different initializations. We calculated the mean and standard error of r, based on x 170
4080 samples for each model in evaluation.
A.5 GRADIENT- BASED OPTIMIZATION
We used PyTorch minimize the ELBO using gradient descent with respect the parameters of both the generative and inference models,
For LVMs which used connectome count constraint with a and af trainable. The objective function is
f = L(T, L(T, a , oo, E,MLP,
(24)
For LVMs which used other constraints including connectome count and learned dense, learned sparse, where M and R trainable. The objective function is
f M M a , a0 o E. E.MLP,
(25)
16

---

Published as a conference paper at ICLR 2022
where á is all the parameters of the inference network. We used a clamp in PyTorch to constrain the weight magnitude M and M to be non-negative in several models. We restrict the M to be symmetric by defining it as the sum of upper triangular matrix and its transpose. Our simulation time step is 6.25ms. Each window size D lasts for 7.5s with 30 imaging steps. We used weights of 1 for both the reconstruction loss and KI. divergence. We used Adam with learning rate scheduler perform the optimization. We used an initial learning rate of 3e with a learning rate schedular with step size of 50, and a gamma of 0.5. We also set gradient clip value of 1. Each model was trained for 300 epochs in which each epoch is one full pass through all the training data.
In the paper, we use the variable 0 to denote the trainable parameters of our generative model. The parameters in 0 are given by
0 = [T)T] M aM,M,,,F,, , 0" of E.H.
(26)
where 7 and [Ca] are the time scales of the neuron and calcium dynamics. M and M are the magnitudes of neuronal connections for chemical and electrical synapses respectively. of and Bf are the scaling and bias parameters of the fluorescence affine transform. a" and of are the noise magnitudes in the neural dynamics equation (Equation D and the fluorescence equation (Equation () The variable H represents the parameters of the sensory mapping that transforms chemosensory data into neural stimulus.
The variable denotes all the trainable parameters of the inference network. These are just all the weights and biases of the 5 different convolutional layers described in section A.3
A.6 CONNECTOME CONSTRAINTS FOR MODEL WEIGHTS
We constructed CC-LVMs on both current-based and conductance- based synapses with different levels of connectomic constraints. Given anatomical connectome data with two N x N matrices: chemical synapse counts (C"), and sizes of electrical synapses (C"). The chemical synaptic weight W5. as W - T Mi, where a represents a global scaling factor. 75 a binary value which indicates the connectivity, and Mý, reflects the magnitude of connection strength. The electrical synaptic weights matrix Wi; is similarly defined as Wi; Mf. We provide more implementation details of each constraint below.
conductance-based connectome count We directly assign the synapse counts the chemical connection magnitudes (M2, Cj,) and the synapse sizes to the eletrical connection magnitudes (My, = Ci,) Under this constraint, both the sparsity Тý and Tý are fixed to the connectome sparsity, magnitude Mú, and Mf, are fixed to the connectome synapse count, and only the global scaling factors a and a are trainable. Mf, and Mj, are restricted to be non-negative.
conductance-based + connectome count We initialize all model parameters with the parameters from the trained conductance-baasd connectome count model. And then the sparsity T5 and TV are fixed to the connectome sparsity, global scaling factors af and af are fixed to 0.01. And now magnitude Mf. and Mí, are trainable during the re-training. Mí, and Mf. are restricted to be non-negative.
conductance-based connectome sparsity For chemical and electrical connections, the model sparsities are T5 = 1[C > 0) and T: = 1[C), > 0) respectively. For both chemical and electrical connections, the magnitudes Mc, and M9, are trainable. The sparsity Tý; and TV are fixed to the connectome sparsity and global scaling factors, a and a", are fixed to 0.01. MM. and Mf. are restricted to be non-negative.
Inductance-based learned total count For chemical and electrical connections, the model is fully connected with the sparsity TỤ, and TV = respectively. For both chemical and electrical connections, the magnitudes Mf. and Mý, are trainable. The model is constrained to preserve the same total synapse count as connectome count ||C*]| for both magnitude matrices adding ||| |[ 12 and ||I | - |C in the objective function. a and a are defined as 0.002 and 0.065, which
the same the trained conne.coun model, that regularizes the sum of elements in M equal to that of C and likewise for M and C The sparsity TV; and yj are fixed to 1 and Mf, and M3, are restricted to be non-negative.
17

---

Published as a conference paper at ICLR 2022
model
current-hased
conductance-based
consistency
weight
learned
learned conne.
learned
learned
learned
conne.
conne. conne.
dense
sparse
sparsity
dense
sparse
total count sparsity
count
count
W
0.142
0.084
0.497
0.082
0.084
0.092
1.000
1.000
1.000
inter
W
0,024
0.000
0.748
0.097
0,000
0.010
0.997
1.000
0.997
W
00002
-0.001
0.469
-0.001
0.001
0.003
1,000
1.000
1.000
connectome
W
-0.002
0.000
0.654
0.006
0.000
0.007
0.998
1.000
0.996
Table 3: Inter-consistency and connectome-consssennyyy Correlation between multiple model
weights with different initializations, and correlation coefficient between model weights and connec- tome counts. Evaluations are performed for models fitted on a single worm and across current-based and conductance-basee models with different levels of connectome constraints. The CC-LVM is more consistent among different random initializations and more highly correlated the connectome counts.
conductance-based learned sparse For chemical and electrical connections, the model is fully connected with the sparsity Тý; = 1 and Tū = respectively. For both chemical and electrical connections, the magnitudes M,, and My are trainable. Mcanwhile, it uses an additional L regularization on both magnitude matrices: || and ||M[]] that the fraction of nonzero elements in M is small, and likewise for M The sparsity Tý and Tý; are fixed to 1 and global scaling factors of and af are fixed to 0.01. Mj. and Mý, are restricted to be non-negative.
conductance-basee + learned dense For chemical connections and electrical connections, the model is fully connected with sparsity Tj = and Tj 1 respectively. For both chemical and electrical connections, the magnitudes Mú; and M;, are trainable. The sparsity matrices, T5 and T); are fixed. Global scaling factors a and a are also fixed to 0.01. Mj, and Mf, are restricted to be nón-negative.
current-based + connectome sparsity For chemical and electrical connections, the model sparsity is Tý = 1[0% > 0] and TV = 1[C > >0] respectively. For both chemical and electrical connections. the magnitudes Mf; and Mf; are trainable. The sparsity Tū; and TV are fixed to the connectome sparsity and global scaling factors, a and a", are fixed to 0.01. Mj. are restricted to non-negative, while the sign of Mf, is not restricted.
current-based + learned sparse For chemical and electrical connections, the model fully connected with sparsity Tú 1 and 7V 1. For both chemical and electrical connections, the magnitudes Ma, and Mj, are trainable.
both chemical and electrical connections, the magnitudes Mf, and Mc; are trainable. Meanwhile, it uses an additional Lı regularization on both magnitude matrices: || and || that the fraction of nonzero elements in M is small, and likewise for M The sparsity Tý. and Tý, are fixed to 1, global scaling factors, a and a are fixed to 0.01. Mí, are restricted be noganega,,,, while the sign of Mf; is not restricted.
current-basee based + learned dense For chemical and electrical connections, the model is fully connected with Tý 1, and 7í 1. For both chemical and electrical connections, the magnitudes Mf. and Mf, are trainable. The sparsity Tý, and Tý are fixed to 1 and global scaling factors a and af are fixed to 0.01. Mf, restricted to be non-negative, while the sign of Mf, is not restricted.
A.7 MODEL WEIGHT CONSISTENCY ANALYSIS
We used the correlation coefficient to quantify the inter-consistency of models trained with different random initializations, well as the commectome-consistennyy which is the correlation coefficient between model weights and connectome counts.
In particular, for the connectome consistency, we compare the consistency of our model weights and connectome synapse count. We calculate the correlation coefficient r, between the absolute value of model weights [Wayl and connectome synapse count Cıy along each row i. Then it is averaged across N rows and models with identical constraints but different random initializations to get F as our metric for comnectome-cnnsistneey.
18

---

Published as a conference paper at ICLR 2022
Worm holdout predictive performance
sensory neurons
Interneurons
motor neurons
1.0
3.0
correlation
0.5
-05
-0.5
-0.5
-3.6
-1.0
current +
current
current connectome
conductance
conductance +
conductance +
conductance connectome sparsity
conductance connectome
conductance connectome count
learned
learned sparse
learned
learned
learned
dense
sparsity
sparse
total
Figure 4: Worm holdout evaluation: Violin plots comparing the distribution of correlation coeffi- cients between predicted and measured neuron activity for the LVM variants. Neurons are divided into sensory, inter, and motor neurons.
N
- -ct)
Ti
(27)
N ( (
Cis Cis) )²
For inter-consistency, we evaluate how model weights are affected by random initialization after training. We calculate the correlation coefficient between the weights W of model and weights W2 of model along each row i. Model and model 2 have the constraints but different random initializations. The correlation coefficient between them is
N
Σ 3=1
W! Wh (7)) W
Ti
(28)
N
(W] W Σ
W3 w3)
This coefficient, r;, is then averaged across N rows and each combination of 2 models within the models with random initialization to get r; as our metric for inter-consistency.
As shown in T the conductance- based model parameters more consistent under different random initializations. Optimization of the conductance-bassee model is easier because the chemical weights We of the conductance-hrate based synapses sign variable. Mcanwhile, the model parameters are better constrained with the connectome sparsity or counts, which enables it to consistent across different initialization.
always positive and do not need trainable
A.8 PREDICTIVE PERFORMANCE FOR WORM HOLDOUT V.S. NEURON HOLDOUT
We demonstrate the violin plot for the model performance in worm holdout evaluation in Figure We found that the results from worm holdout are different from those the neuron holdout experiment in Figure Here, all 9 LVM variants perform similarly across different categories. Our connectome- constrained models do not outperform other unconstrained baselines.
We also studied whether the predictive performance of worm holdouts and neuron holdouts were correlated. In Figure 5 we show the scatter plot of predictive performance for worm holdouts neuron holdouts. Each point represents neuron from the same worm. We also report the correlation coefficient between the predictive performance for worm holdout and neuron holdout for each LVM variant. We found that connectome-constraineed model achieves the highest correlation,
19

---

Published as a conference paper at ICLR 2022
holdout holdout predictive performance
con .0.37
con D
com
current + learned dense
current learned sparse
current + connectome sparsity
worm holdout predictive performance
com: 0.245
con: 0.041
con: 0.020
conductance learned
conductance learned sparse
conductance learned count
com: 0.389
cor: 0.423
com 0.434
05-
conductance connectome sparsity
conductance conductance connectome count
connectome count
-30
neuron holdout predictive performance
Figure 5: Scatter plot for predictive performance of worm holdout v.s. neuron holdout. Each point represents one neuron. Our best model connectome count achieves highest correlation, which indicates highest consistency for neurons which gets better performance in both evaluations.
and also indicates the highest consistency for neuron sets which gets better performance in both evaluations.
A.9 PREDICTED HOLDOUT TRACES WITH VARIOUS CONSTRAINTS
For each neuron, we performed both neuron holdout evaluation and worm holdout evaluation for current-based and conductance-basee models with different levels of connectomic constraint.
For neuron holdout evaluation, plot additional predicted traces from 15 representative neurons, including 12 with good performance (top), and 3 with worse performance (bottom) in Figure models with connectome count constraint outperforms other baselines.
For worm holdout evaluation, we show the results on the same 15 neurons from LVMs in Figure 7 all models achieve comparable results.
A.10 PREDICTED TRACES AND VOLTAGE LATENT SPACE FOR NEURON HOLDOUT EVALUATION
For neuron holdout evaluation, we also show the predicted traces and voltage latent space for all measured neurons. We show 189 neurons in Figure 8 Figure and Figure TO In each subplot, reported the measured fluorescence trace ground truth, predicted fluorescence trace ground truth, and the mean and standard deviation (std) of voltage latent space (posterior distribution) from the inference network.
A. PREDICTED TRACES AND VOLTAGE LATENT SPACE FOR WORM HOLDOUU EVALUATION
For worm holdout evaluation, similarly, we also show the predicted traces and voltage latent space for all neurons. We show 300 neurons from train worm in Figure II Ii-Furure| [5] Meanwhile, also show 300 neurons from a test worm in Figure [6--Figure 20 In each subplot, we reported the measured fluorescence trace ground truth (for recorded neurons), predicted fluorescence trace ground
20

---

Published as a conference paper at ICLR 2022
Neuron holdout predictive performance
fluorescence
Z-butanone
2.3-pentanedione
sensory nearon: AFD
interneuron: AVB
Interneuron: AVE
sensory neuron: AWB
motor neuron: ILID
sensory neuron: OLQV
interneuron:
interneuron: RIB
AF/1
sensory
Intemeuron: RID
interneuron:
sensory AWC
VB01
TImin)
current
current
conductance learned
conductance
conductance learned sparse
conductance connectome sparsity
conductance
conductance connectome count
learned
learned dense
connectoree
total count constraint
sparse
sparsity
dense
count
Figure LVM predictive performance across neuron types for neuron holdout evaluation. Measured traces and predictions made by the 9 LVMs for 15 neurons including sensory neurons, interneurons and motor neurons.
truth, and the mean and standard deviation (std) of voltage latent space (posterior distribution) from the inference network.
21

---

Published as a conference paper at ICLR 2022
Worm holdout predictive performance
measured fluorescence trace
2- 2-butanone sensory neuron: AFD
2.3-pentanedione
NaC1
interneuron:
Interneuron: AVE
sensory neuron: AWB
motor neuron: IL1D
sensory neuron: OLQV
intereuron:
AP/6
sensory neuron: OLL
interneuron: RID
interneuron: RIM
sensory neuron: AWC
Interneuron: AVK
motor neuron: VBD1
7/min]
current
current
current + conductance
conductance total count
conductance
conductance
conductance conductance connectome
leamed
learned
connectome
learned dense
learned sparse
connectome sparsity
connectome count
sparse
Figure 7: LVM predictive performance across neuron types for worm holdout evaluation. Mea- sured traces and predictions made by the LVMs for 15 neurons including sensory neurons, interneu- rons and motor neurons.
For the train worm, find most of the neurons have good predictive performance compared to the measured ground truth. That indicates optimization is saturated on the train set. However, the test worm has relatively worse predictive performance, which indicates the generalization gaps between
22

---

Published as a conference paper at ICLR 2022
each individual trial (animal). While some neurons still achieve good results, that indicates model still generalized to unseen individuals to some degree.
Interestingly, for both train and test worms in the worm holdout evaluation, find those unmeasured neurons still have reasonable patterns. That indicates our model successfully avoids the degeneracy of solutions for those unmeasured neurons.
23

---

Published as a conference paper at ICLR 2022
Neuron holdout prediction and voltage latent space connectome count
2-butanone
P-pentanndione
NaCI
predicted fluorescence trace
voltage latent mean
voltage latent std
measured fluorescence trace
intemeuron: ADAL
Intereuron: ADAR
Intermuron: ADEL
Internurron AGER
sensory neuron ADFL
sensory neuron ADFR
sencory
sensory neuron
sensory neuron: AVOL
sensory neuron: AFOR
Interneroo: AVAL
Internerron: AIR
intereuronn AUL
Interneuron: AIBR
Internuroo:
interneronn: AIMR
Interneuron, ANL
Interneuronn AIRR
Internuron:
Interneuronn
intemeuron: AZZL
intereuron. AIZR
Interneuoon ALA
sensory neuron AGR
sencory neuron
sensory neuron:
sensory neuron: ASER
sensory neuron ASGL
sensery neuron: ASGR
sensory neuron: ASHL
sensory
sensory
sensory neuron:
sensery neuron: ASIL
sensory neuron: ASR
sensory neuron: ASKL
sensory neuron:
sensory neuron:
sensory neuron: AJAR
Interneuron AVAL
Interneuron: AARR
intermuronn AVBL
intemuuronn AVBR
Intemeuron: AVOL
Internuroon AVDR
Interneronn AVEL
Intemeuron: AVER
Interneuron, AVEL
WA Wad
Intemeuronn AVER
Intemeuron
intermuronn
intemeuron: AWR
intemsuron: AVL
Intemeuron: AVER
internerror AVKL
Interneuron: AVER
Internuron:
sensory neuron ANAL
sensory neuron: AWAR
sensory neuron: AWEL
sensory neuron:
sensery nouTon: ANCL
sensery nEWION HWIR
T[min]
Figure& 8: Neuron holdout evaluated on connectome count CC-LVM predicted traces and the corresponding voltage latent space across all recorded neurons (part 1).
24

---

Published as a conference paper at ICLR 2022
Neuron holdout prediction and voltage latent space for connectome count? 2
2-butanone
2-3ppentanedione
NaCI
predicted fluorescence trace
voltage latent mean
voltage latent std
measured flucrescence trace
sensory neurore
sonsory neuron:
intemeuroo: CEPOL
interneuron: CEPOR
Intemeuron: CEPVL
intemeuron: CEPVR
motor neuron: DA01
moter neuren: D801
motor neuron: DB02
motor neuron: DOO1
sensory neuron: FUPL
sensory neuron: FLPR
Hart Judteat We
phoryngeal neuron: IIL
pharyngeal neuron, IR
pharyngeal neuron: 12L
pharyngeal
pharyngeal neuron
pharyngeal neuron
pharyngeal nouron:
pharyngeal nouron:
motor neuron: ILLL
ILIOL
ILIOR
motor neuron: ILIVL
motor neuron: ILIVA
sensory neuron: 1L21.
sensory
sensory
samsory
sersory neuron: 1.2VL.
sennory neuran: L2VR
pharyngeal neuron: ML
pharyngeal neuron:
pharyngeal neuron:
pharyngeal neurons
pharyngeal neuron: MER
pharyngeal neuron: M4
pharyngeal neuron: MS
pharyngeal neuren:
pharynggal neurore MCR
pharyngeal neuron:
pharyngeal NSML
pharyngeal NSMR
OLLL
sensory neuron: OLLR
sensory neuron: OLOOL
sensory neuron: OLOOR
OLOVL
neuron: OLOVR
Internuron: RIAL
interneuron: BAR
interneuron RIBL
interneuron: RBR
intemeuron: RIFL
intereuron: RIFR
Interneuro:: RIGL
Intereuron:
internuron:
Intemeuron:
T(min)
Figure 9: Neuron holdout evaluated on connectome count?: CC-LVM predicted traces and the corresponding voltage latent space across all recorded neurons (part 2).
25

---

Published as a conference paper at ICLR 2022
Neuron holdout prediction and voltage latent space for connectome count?
2-butanone
2.-pentanedione
NaCI voltage latent mean
predicted fluorescence trace
-...
voltage latent
measured fluorescence
interneroon RIR
Intereuron: RIPL
Intereuronn RIPR
Intemeuron: AR
Intereuron: RIS
motor neuron: RML
motor neuron: ANDOL
motor neuron: RMOOR ae
motor neuron: RMDVL
motor neuror: RMDVR
motar neuror: RMEL
motor neuron:
RNED
motor RMEV
RMTL
motor nouron: RVER
Internouron: RMGLL Mauar
interneroon AMGR
sensory
sensory neuron: SAAOR
sensory neuron: SAML
sensory neuron: SAAVR
intereuror::
interneuror: SABVL
interneron: SABVR
/FF
A Maromol
motor SIADI
motor neuron: SADR
motor SAA
motor nHAUOO: SLAVT!
motor neuUon: SIBOL
motor TwurOn: SIBOA
motor neuron: SIMVL
motor neuron:
motor neurOR:
motor neuron: SHBOR
MOSST neurOn: SMBVL
motor nouron: SMBVR
mator neuron: SMOOL
motor neuron: SMOOR
motor neUTOn: SNOVI
motor SMOVR
URADL
URAOR
motor neuron: URAVE
moter nouron: URAVR
sensory neuron: URBL
sensory neuron:
sensory neuron: URXIL
sensory neuron:
sensory neuron: URYDL
sensory neuron: URYOR
sensory neuron: URYVL
sersory neuron URIVR
motor neuron: VADI
motor neuron VBOI
VB02
VOO1
T[min]
Figure 10: Neuron holdout evaluated connectome count?: CC-LVM predicted traces and the corresponding voltage latent space across all recorded neurons (part 3).
26

---

Published as a conference paper at ICLR 2022
Train worm: worm holdout prediction and voltage latent space for connectome count?
2- 2-butanone
2,3 pentanedione
NaCI
predicted fluorescence trace
voltage latent mean
voltage latent std
measured fluorescence trace pharyngeal neuron:
phoryngeal neuron:
pharyngeal neuron:
pharyngeal neuron:
pharyngeal neuron:
pharyngeal neuron:
pharyngeal
pharyngeal neuren:
pharyngeal neuron
pharyngeal el neuron MZL
pharyngeal neuron M2R
pharyngeal neuron: M3L
pharyngeal neuron: M3R
pharyngeal neuron M4
pharyngeal neuron: MS
pharyngeal neuron: MCL
pharyngeal neuron: MCR
pharyngeal neuran: MI
pharyngeal neuron: NSML
pharyngeal neuron: NSMR
sensory neuron: ASIL
sansary neuron: ASI
semsory neuron: ASE
semsrry neuron: ASJR
sensory neuron: AWAL
sensory neuron: AWAR
sensory neuron: ASOL
sensery neuron: ASGR
sensory neuron: AVBL
sensory neuron. AWBR
sensory nouron: ASEL
sensory nouron: ASER
sensory nouron: ADFL
sensory neurOn: ADFR
sensory neuron: AFDL
semsory neuron: AFOR
sensory neuron: ANCL
ANCR
sensory ASKL
ASKR
ASHL
sensory ASHR
ADLL
ADLR
sensory BAGL
semsory
Semsory
MEnsary
sensory neuron:
sensory
sensory neurO::
sensory neuron:
sensory
sencory neuron: SOOR
sensory nouron: AQR
sensory neuron:
sensory neuron:
sensory neuron: ALMR
sensory neuron:
sensory neuror: PVN
T(min)
Figure 11: A train worm evaluated on worm holdout using connectome count2 2 constraint: CC-LVM predicted traces and the corresponding voltage latent space across all neurons including unmeasured neurons (part 1).
27

---

Published as a conference paper at ICLR 2022
Train worm: worm holdout prediction voltage latent space for connectome count
2 2-butanone
2.3-pentanedione
NaCI
predicted fluorescence trace
voltage latent
voltage latent std
measured fluorescence trace
sensory neuren:
sensory
sensery neuron: FLPL.
sensory neuron:
sensory neuron:
sensory neuron: PVDL
sensory neuron: PVDR
Interneroon ADEL
Interneuron: ADER
Intereuron: POEL
Interneuron: POER
Interneuoon: PHALL
Intemeuron: PHAR
Intemeuron: PHBL
Interneuron: PHBR
intereuror: PHCL
nnterneuor:: PHOR
sensory neuren: IL2OL
sensory neuron: IL2OR
sensory neuron: 1L21.
sensory neuror: ELR
sensory neuror: 12VL
sensory neuren: 1.2
interneuronn CEPOL
Internuronn: CEPOR
Interneuron: CEPAL
interneuron: CEPAR
AF/F
sensory neuron: URIDL
sensory neuron: URYOR
sensory neuron: URIVL
sensory neuron: URIVR
sensory neuron: OLLL
sensory neuron: OLLA
sencory neuron CLOOL
sensery neuron: OLOOR
sensory neuron OLOVE
sensory neuron: OLOVR
moter nEUTON: ILIDL
moter newron: ILIDA
motor ILIL.
mOGOT TREUOIC ILIR
moter nwuron: ILVL
motor LIVR
interneuror: ANN
interneuron: AIN
Intereuronn AML
Intereuronn AIMR
Interneuron:
URBL
URBR
Interneuron
Interneroonn AYL
Interneuron: AIY
Interneron:
AIAR
sensory
sensory
T[min]
Figure 12: A train worm evaluated on worm holdout using connectome count constraint: CC-LVM predicted traces and the corresponding voltage latent space across all neurons including unmeasured neurons (part 2).
28

---

Published as a conference paper at ICLR 2022
Train worm: worm holdout prediction and voltage latent space for connectome count?
2 butanone
2,3 pentanedione
NaCI
predicted fluorescence trace
voltage latent mean
voltage latent std
measured fluorescence trace Interneuron
interneroon: ALA
interneronn: PVOL
intemsuron: ADAL
internuron: ADAR
Interneuron: RIPL
We
intertaurrn: RMR
intereuron BOUL
internuroo: BOUR
intemeuron: PVR
intemeuron AVR
internuron: AVFR
AsALA M Mima wwaess
Astoo SyAAAbo ARAANAA
Internurron AVHL
Internuroon AVHR
Internerro: PVPL
Interneronn PVPR
Intereuron: LUAL
internuron: LUAR
Interneuron: PVNL
Intereuron: PAN
Interneuronn AVG
internuroon DVB
Internuronn: RIBL
internuroo: RIBA
interouron: RIGL
internuroo: RIG
intemeuron: RMGL
AF/FF
interneroon: AMGR
Internuron: AIBL
internuroo: AIBR
Interneuron: RCL
Interneroon RICR
sensory neuron: SAAOL
sensory neuron: SAAOR
sensory neuron: SAHVL
sensory neuron: SAARR
Interneuron: AVLL
Interneuron AKR
Interneuron: DVC
intereuron: AVLL
internurom: AVR
interneuron:
interneuron AVDL
interneronn
intemeuron:
intemeuron: PAWL
interneuron:
interneuron:
intemeuron:
intemearor:
internuroo:
Intereeronn
Intemeuron:
neuron:
Intemeuror:
Interneuron:
Interesron:
T(min)
Figure 13: A train worm evaluated worm holdout using connectome ?ount constraint: CC-LVM predicted traces and the corresponding voltage latent space across all neurons including unmeasured neurons (part 3).
29

---

Published as a conference paper at ICLR 2022
Train worm: worm holdout prediction and voltage latent space for connectome count
2-butanone
2.3-pentanedione
NaCI
predicted fluorescence trace
voltage latent mean
voltage latent
measured fluorescence trace
Internouron; PVCL
Intereuron: PVCR
Interneuron: RIPL
intemsuron: RPP
motor neuron: URADL
motor nauron URADR
motor neuron: URAVL
motor neuron:
motor nouron: RNEL
motor RMOOL
RMOOR
RMOL
motor RMDR
motor neuron: RMOVL
RNDVR
motor neuron: RNL
motor neuron: RMR
motor neuron: RVHL
mater neuron: RMB
internuro:: SABD
internaurton: SABVL
Intereuron: SABVR
motor neuron: SHOOL
motor neuron: SHOOA
motor nOUTOn: SHOVL
motor DEUTON: SHOVR
motor neuUon: SHOOL
Aubreel
SMBOR
motor ReuTon:
SHBVR
SIDOL
SIBR
SIADI
motor SMOR
motor neuron: SIAVL
motor SLAVR
motor FAUTON: DAG1
motor neuron: DA02
motor neuron: DA03
motor neuron: DA04
motor neuron: DAGS
motor neuron: DADS
motor neuron: DAG7
motor neuron: DADE
motor neuron: DAOS
motor neuron: PQA
motor neuUon: D801
motor neurDn:
motor neuron: D803
motor neuron: 0804
motor neuron:
motor nouren:
T(min)
Figure 14: A train worm evaluated on worm holdout using connectome count constraint: CC-LVM predicted traces and the corresponding voltage latent space across all neurons including unmeasured neurons (part 4).
30

---

Published as a conference paper at ICLR 2022
Train worm: worm holdout prediction and voltage latent space for connectome count
2-butanone
2,3 pepettnnedioee
NaCl
predicted fluorescence trace
voltage latent mean
voltage latent std
measured fluorescence trace
A506
motor
neuron:
motor neuron:
motor
motor neuron:
motor neuron:
neurom:
neuron:
motor neuron: DOOS
motor neuron:
mobor neuron:
neuron: VA03
motor neuron:
motor
neuron: VA12
motor neuron: VB01
motor neuron:
motor neuron: V803
neUIGR:
motor
motor VB06
nEURGn:
motor
motor neuron: V809
motor neuron: VB10
motor neuron:
motor neuron: VD01
motor neuron: VD02
motor neuron:
motor neuron: VD04
MotOr neuron:
motor neuron:
motor neuron: VD07
motor neuron: VIDB
motor neuron:
motor neuron: VD10
motor neuron: VOLI
motor neuron: VD12
motor neuron: VD13
motor neurnn:
motor neuron:
motor neuron: VC01
motor newron: VC02
motor neuron:
motor nouron: VC04
motor neurOn VCD5
motor neuron:
T(min]
Figure 15: A train worm evaluated on worm holdout using connectome count2 constraint: CC-LVM predicted traces and the corresponding voltage latent space across all neurons including unmeasured neurons (part 5).
31

---

Published as a conference paper at ICLR 2022
Test worm: worm holdout prediction and voltage latent space for connectome count?
2 butanonne
2,3 --peetanedoonee
NaCI
predicted fluorescence trace
voltage latent mean
voltage latent std
measured fluorescence trace pharyngeal reurre 12L
pharyngeal neusen:
phuryngeal neuron 12R
phanyngoal neuron: I2R
pharyngeal neuron: B
pharyngeal neuron 14 wyy
pharyngeal neuron: iS
pharyngeal neuror: iS
pharyngeal neuron. MI
pharyngeal neuron: MZL
pharyngeal neuron: MER
pharyngeal neuron NIL
pharyngeal neuron: NaR
pharyngeal nouron: M4
pharyngeal neuron: MS
pharyngeal neuron: MCL
pharyngeal neuron NCR
pharyngeal neuron: MI
pharyrgeal neuron: NSML
pharyngeel neuron: NSMR
sensory neuron: ASIL
sensory neuson: ASI
sensory neuror: ASIL
Sensory nouron: ASJR
sensory neuron: ANAL
sensory neuron: AWAR
sensory neuror:
sensory neurOR:
sensory neuron ANDL
sensory neuron
AF
ASEL
sensory
ADFR
AFOL
AFOR
neuron, ANCL
AWER
sensory ASKL
sensory neurON ASKR
sensory neuron: ASHL
sensory nauton: ASHR
sensory neuron: ADLL
sensory neuron: ADUR
sensory neuron: BAGL
sencory neuron, BAGR
sensery neuron URXI
sensory neuron URXR
sensory neuron: AUNL
sensory neuron: ALXR
semsory neuron: PUNL
sensory neuron: PLNR
sensory nouron: SDQL
sensory neuron: SOOR
sensory nouron: AOR
sensery neuron: POR
Sensorr nouron: ALML
Sensory NEURON: ALMR
sensory neuren: AVM
sensory neuron: PVM
T(min)
Figure 16: A hold-out (test) worm evaluated on worm holdout using connectome count? con- straint: CC-LVM predicted traces and the corresponding voltage latent space across all neurons including unmeasured neurons (part 1).
32

---

Published as a conference paper at ICLR 2022
Test worm: worm holdout prediction and voltage latent space for connectome count
2- z-butanone
2,3- 3--pentanedione
NaCI
predicted fluorescence trace
voltage latent mean
voltage latent std
measured fluorescence trace
sencory neuron: PLML
sensory neuron: PLMR
sensory neuron: FLPL
sensory neuron: ELPR
sensory neuron: DVA
sensrry neuron: PVDL
semsory neuron: PVDR
Internuron: ADEL
internuroon ADER
intemeuron: POEL
intereuronn POER
Interneuron PHAL
Internuron: PHAR
Internuron: PHBL
intemeuron: PHBR
intemuronn PHC
internuron: PHCR
sensory neuron: IL2OL
sensory neuron: Aa Ma lmades
sensory neuron
sensery neuror:
sensory
sensory neuron:
intornuroor CEPOR
sensory URYDL
sensory UEYDR
semsory URYVL
sensory URYVR
OLLL
OLLA
semsory neurOn: CLOOL
OLOOR
sersory neuron: OLOVE
sensory nauron: OLOVR
motor neuron: L1DL
motor ReuTon: ILIOR
motor neuron: ILIL
motor neuron: ILIR
motor nouror: ILIVL
TOOOr neuron: ILIVR
Internerron: ANL.
Interneuron: AIR
interneuron: ANL
Interneroon: AIRR
interneuro:: RM
sensory neuron: URBL
sensory neuron: URBR JAMMAA
interneror: RIR
interneuron: APL
Interneroon AIR
Interneuron: NAL
interneuron: AIA
sensory neuron: AUAL
semsory neuron: AUAR
interneronn ALL
intemeuron: AIZ
interneuron:
T(min)
Figure 17: A hold-out (test) worm evaluated on worm holdout using connectome count? con- straint: CC-LVM predicted traces and the corresponding voltage latent space across all neurons including unmeasured neurons (part 2).
33

---

Published as a conference paper at ICLR 2022
Test worm: worm holdout prediction and voltage latent space for connectome count?
2-butanone
2,3- 23-pentanedione
NaCI
predicted fluorescence trace
voltage
voltage latent std
measured fluorescence trace
Interneuron; ALA
Intereuronn PVOL
Interneronn PVOR
Interneroon ADAL
Intereuron:
Intemeuronn RFL
Internernon PVR
Interneroon AVTL
Internuronn AVTR
intemeror: ANL
Internuronn AVR
Interneroon PVPL
intemeuror: PVPR
internuronn LUAL
intemeuror: WUR
interuuro:: PVNL
Internuronn PVNR
interneronn AVG
Intemeuronn
Intereuron:
Intemeuron: RIOL
Internerron
Interneuron RMGL
interneuron; RNGR
Intereuron:
Interneuton: AIBR
Internuront RILL
interneroon RIC
sensory neuron: SAADL
sensory neuron: SAAOR
sensory neuron: SAAV
sensory neuron: SAARR
internuronn AVL
internuron:
interneroo: DVC
Intemeuron AML
intemeuron:
Internuron:
intemuuoo: AVOL
Interneron:
Interneroo:
interneron: PMWL
Interneuron: PVWR
internuroo: RIL
Internuronn RIAR
interneronn RIML
Internerron RMR
Intereuronn AVEL
Internuronn AVER
motor neuron RMFL
motor neuren: RMPR
internuron:
intemeuron: AVEL
T(min)
Figure 18: A hold-out (test) worm evaluated on worm holdout using connectome count con- straint: CC-LVM predicted traces and the corresponding voltage latent space across all neurons including unmeasured neurons (part 3).
34

---

Published as a conference paper at ICLR 2022
Test worm: worm holdout prediction and voltage latent space for connectome count
2 butanone
2,3 pentanedione
NaC1
predicted fluorescance trace
voltage latent mean
voltage latent std
measured fluorescence trace
PVER
Interneroon
motor URADL
URADR
motor neuron: URAVL
motor neuron: URAVR
mator neuron: RMEL
motor neurons RNER
motor neuror: RMED
motor neuron: ANEV
motor neuron: AMOOL
motor neuron: AMDOR
motor neuron: AMDL
motor neuron: RMDR
moOtor nouron: RMDVL
motor neuren: RMOVR
motor RMHR
Internuron: SABO
interneroon SABM
Interneuron: SABVR
motor neuron: SMOOL
motor neuron: SMOOO
mator neuron: SNOVL
motor neLION: SNOVI
motor FRRUFOR SMBOL
motor neuron: SHIOR
motor NAAUOS: SMIVL
motor NAUPOR: SNIVA
moTor REUFOR SIBVL
SIADL
SIADR
motor SHVVL
moter SLAR
moter neuron: DADI
motor neuron: DA02
motor neuron: DA03
motor neuron DAD4
motor neuron: DADS
moter neuron: DA06
motor neuron DA07
motor neuron: DADS
motor neuron: DA09
motor neuron: PDA
D604
D855
D606
motor neuron: D607
motor neuror: ASD1
motor neuron: AS22
1(min]
Figure 19: A hold-out (test) worm evaluated on worm holdout using connectome count con- straint: CC-LVM predicted traces and the corresponding voltage latent space across all neurons including unmeasured neurons (part 4).
35

---

Published as a conference paper at ICLR 2022
Test worm: worm holdout prediction and voltage latent space for connectome count
2 b-uuaanone
2.3-pentanedione
NaCI
predicted fluorescence trace
voltage latent mean motor neuron: A504
voltage latent std
measured fluorescence trace motor neuran; A50S
metor neuron:
ASD7
motor A508
A509
motar AS10
motor neuron: AS11
motor neuron: POB
motor neuron DO01
motor neuron: DO02
motor neuron: DO33
mator neuron DO04
motor neuron: DOOS
motor neuron: DOO6
motor neuron: VADI
motor neuron: VA02
motor neuron: VAD3
motor neuror: VADA
motor neuron: VADS
motor nouron: VADS
MOOOY nowron: VA0?
motor neuron: VADE
motor neuurr: WADS
motor neurOr: VA30
motor neuron: WALI
LF/FF
motor neuron: VA12
motor neuren: VBOL
motor neuron: VB02
motor neuron: VB03
motor neuron: VB04
motor neuron: VB0S
V807
VB10
VB11
mator VD01
motor newron: VD02
motor VD03
motor neuron: VD04
motor neuron: VDD5
motor neuron: VD06
motor neuron: VD07
moter meuron: VDDS
motor neuron: VO09
motor neuron VD10
motor neuron VDL1
motor neuron: VD12
motor neuron VD13
motor neuron: HELL
motor neuron: HSWR
motor nouron: VCO1
motor neuron: VC02
motor nouron: VC03
motor neuTon VCD4
moter neuron VCD5
motor neuron VCD6
T(min]
Figure 20: A hold-out (test) worm evaluated on worm holdout using connectome count con- straint: CC-LVM predicted traces and the corresponding voltage latent space across all neurons including unmeasured neurons (part 5).
36


Reviews:
[{"review_id": "wiqtXanoPMH", "reviewer": "Reviewer_5GJS", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The question of whether anatomical constraints help with improving model predictions and constraining the space of generative models of neural activity is an important and significant question in computational neuroscience. This paper takes a step towards addressing this question by developing biological models that have interpretable parameters (connectivity weights, neuron time constants, resting membrane potentials) yet are flexible to capture the variability across the population or individual neurons (by incorporating noise in the latent and observed variables). If a model resembles the biological generative process of the data up to appropriate amount of details, we would expect that incorporating more biological constraints would help with model's predictive performance. Therefore this approach not only could be used for model selection, but also could provide insight about why biological systems prefer certain architectures or mechanisms over others.\n\nWhile the paper is clear and interesting, there are several issue that need more clarification as listed below.\n\n* [Hierarchical Bayesian models](https://www.biorxiv.org/content/10.1101/621540v1.article-info) are already developed to study the dynamics in *C. elegans*. How would the results change if the anatomical constraints are imposed on the already established models such as the one mentioned above. Some comparison with this paper would make the claims stronger.\n* In the results, why neuron-held-out correlation is worse than worm-held-out? Am I misunderstanding this? At least intuitively shouldn't we expect to have better performance when a single neuron is missing compared to all the neurons from a worm missing?\n* How did the authors perform neuron-hold-out experiments? If a neuron is missing in the training dataset, then how do we have a latent variable corresponding to it? Is the missing neuron treated as missing observations? At the test time, do we use the \"reconstruction\" for evaluating the correlations? Is the reconstruction the mean of the posterior or is it based on samples?\n* For the neuron-hold-out experiments, pairs of neurons (107) are considered. Why is there 107 pairs and why are we considering pairs of neurons instead of removing a single neuron? Again I feel like I'm misunderstanding the neuron-hold-out experiments.\n* Is it possible to visualize the reconstructed activity of all the neurons (for a hold-out-worm and training worm) in the appendix? This would be important for biology experts and can provide further insights into the proposed model.\n* How should we ensure that the model is learning time dynamics? The factorized Gaussian family does not incorporate any dynamical model or smoothness constraints in my understanding, is that right?\n* When using anatomical constraints for model selection we need to control and account for as many confounding factors as possible that can potentially lead to misinterpretations. These factors include (but not limited to) controlling for the number of parameters across different models (more parameters can lead to overfitting, can make the optimization harder, can make the inference harder since it's in a higher dimensional space), and successful training (what is the convergence criterion?). \n* In Fig. 3 the error bars seem too large, are the differences significant in the violin plots? In addition, some neurons have really bad predictive performance, can we focus on those neurons to understand why this is happening? Is it due to lack of measurements form their neighboring neurons? Is it because they are less predictable and more impacted by unmeasured sources of variability?\n\n\nI have some minor comments to, and would like the authors to clarify.\n\n* What is $\\theta$? Please write down $\\theta$ explicitly using the notations introduced in the paper.\n* According to Wikipedia, *C. elegans* has 302 neurons, not 300. Can you clarify this?\n* What is $\\phi$? Are you using a neural network to parametrize $\\phi$? If so what kind of architecture is used?\n* How does the regularization work? What is the probabilistic interpretation of adding the $|| |M| -|C| ||$ regularization? Is this equivalent to defining a normal prior? Please clarify this. Why did the authors use $L_2$ regularization? Isn't $L_1$ regularization associated with sparsity? If so, what is the probabilistic interpretation of this in the variational inference framework?\n\n\n\n\n", "summary_of_the_paper": "The paper develops a latent variable model with biologically meaningful parameters for the worm *C. elegans* whole brain neural traces. Given the connectivity between the neurons, the neural traces are modeled as stochastic leaky integrators with either conductance or current based synaptic input model. Given the voltage traces, the observed calcium signals are modeled as first order leaky integrators added by noise to account for the observation noise. External input is nonlinearly transformed and fed into the neurons to account for the stimulus-dependent modulation of neural activity. Multiple connectivity models (fully trainable, fully trainable with $L_2$ regularization, trainable weight magnitudes, trainable global scale) are used to investigate whether biological connectome constraints help with neural activity predictions or not. Variational inference is used to infer a posterior distribution over neural activity trajectories with factorized Gaussian distribution as the variational family. Results on neuron-hold-out and worm-hold-out experiments suggest that connectome constraints and conductance-based modeling improves model prediction over unseen neurons and worms.", "summary_of_the_review": "While the approach is very interesting and the problem considered is important, the paper lacks comparisons and clearer description and investigation of the results. Furthermore, some justification on specific choices need to be done. See above for more detailed comments.\n\nPost discussion: I'm increasing my score to \"accept\" since most of my comments are incorporated, I thank the authors for their hard work in revising the paper and incorporating additional experiments/visualizations.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "vKJ4YADd02", "reviewer": "Reviewer_UhCh", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper makes some additional developments on the C. elegans latent variable modeling literature: including both chemical and electrical synapse information, adding conductance-based synapses, and nonlinear calcium observations with a time constant. However, I don’t believe it’s the first linear latent variable model to consider connectomic constraints (or fitting linear models to C elegans data with variational inference). See equation 1 in Mena et al (2017). Also see Linderman et al (2019) for cross-worm comparisons and predictions of held-out neurons.\n\nThe clarity of the methods and conclusions could be improved.\nMethods;\n-Does 'g' in equation 5 denote the softplus? It's not clear whether the 'same' in the sentence before refers to g being the same across neurons, same as the softplus definition earlier in the section or both.\n\t-Is there a reference supporting this observation model? Is the threshold of 'g' fixed?\n\n-Page 4 section 2.1 redundantly refers to Bargmann et al about nonspiking neurons in C. elegans twice in the first two paragraphs. I'd suggest moving this reference earlier to the introduction where you first mention that you're using a nonspiking model as many in the ICLR audience may not know this fact and it'd more quickly support your modeling assumptions.\n\n-Last paragraph of page 3 \"A latent variable model....\". I found this paragraph a little confusing. I wasn't sure what many sentences were referring to, especially with the introduction of the \"black-box\" networks (2nd sentence vs the black-box CCN brought up in the third sentence).\n\n-The explanation of variational inference over SMC (top text on page 3, starting “inference instead of SMC”) was a little unclear what the authors wanted to convey with “generates a different type of distribution and is more computationally efficient because it allows direct sampling from the approximate posterior.” A more explicit definition of “different” would help clarity (or removing this altogether) and it’s also unclear whether sampling from the posterior is used to help validate/apply the model here.\n\nConclusions:\n-The conductance-based synapses showed a performance improvement in the neuron holdout, but not really in the worm holdout condition. Why is this and does it impact the conclusions?\n\n-How does the calcium observation model impact the model fit? This is an important contribution of the proposed model, but it's not clear if the Ca dynamics are necessary compared to, for instance, a simple linear-Gaussian model (given the observation window size).", "summary_of_the_paper": "The authors propose a biologically constrained latent linear dynamical model of the C. elegans nervous system. They use connectomic information (including chemical vs electrical synapses) to constrain connections between units during inference. They fit the model to calcium imaging data from whole C. elegans using a variational approach. They find that biological constraints improve model performance and validity using both withheld neuron and across worm metrics.", "summary_of_the_review": "The paper highlights that known biological constraints can be included tractably within latent variable models of biological neural networks (in this case, the C. elegans). Moreover, they provide some evidence that these constraints improve model fits.\nThe model goes beyond existing linear latent variable models of C. elegans. However, the conclusions do not include strong measures comparing against existing methodology and it's difficult to gauge how much advancement this model provides. While the presentation clarity was adequate, organization and model presentation could be improved.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "fAz1z0eOaW", "reviewer": "Reviewer_aiwW", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength\n- Adding biophysical constraints into neuron activity modeling, and conducting both neuron and worm hold-out experiments to validate the model.\n- Alleviate the issue of trail-to-trail variability.\n\nWeakness — not clearly written at all:\n- About the prior p(v|h) generation: it only mentions “Pθ(v|h) is a biophysically realistic connectome-constrained network with passive point-neuron dynamics” without pointing out what the network is: are the only trainable parameters in the network W^c_ji and W^e_ji, and a MLP  layer for encoding h_i(t), or are there any other layers? And since it has subscript θ, the same as the decoder network, should I assume they are the same network? If so, how can the decoder be trained since it needs the prior to calculate KL divergence for the loss?\n- There’s no detail about the exact network architecture, layer, input/output dimensions, data processing/ training procedure, etc. This makes the whole logic even harder to comprehend.\n- Apart from clarity, the technical novelty in computer science is a bit insignificant. Empirically, the results are reasonable but not interesting enough, and no comparison is made with other baseline modeling methods. I would suggest a re-writing (rephrasing, better connecting between sections, putting many appendix contents like A2 and A5 into the main text, etc.) and re-submitting to a neuroscience journal.", "summary_of_the_paper": "This paper uses VAE (variational autoencoder) to model neural activities of C.elegans observed in calcium imaging. The main contribution is making the encoder-learned posterior distribution close to a biophysically constrained prior, as well as designing the decoder based on biophysical rules.", "summary_of_the_review": "Overall, the paper provides a good aspect in modeling neuron activities by using connectome constrains as VAE prior, but it’s not clearly written and lacks novelty or significance in both technical and empirical aspects. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "NAXXjQuvAQl", "reviewer": "Reviewer_iieb", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "•\tI believe that connectome constraints help improve the model performance in the current setup. However, the results are not surprising. The question is whether the benefit of connectome constraints was due to insufficient training data. When you have enough data, would an unconstrained model or a loosely constrained model perform as well as a connectome constrained model, meaning that a model could learn reasonable connection structure through sufficient training? \n\n•\tHow does the latent space look like? The current model predicted the latent variable of neuron voltage and used a differential equation to further predict the observed calcium signal. One major question I had was whether such a simplified calcium-voltage model was sufficient for C.elegans neurons. The units of model networks were passive non-spiking units, while in real brain calcium signal was mainly generated by suprathreshold neuronal activities. Although the problem might be partially taken care of by the nonlinearities of the model, it is not clear whether that was sufficient. On the other hand, I’m wondering whether the oversimplified calcium-voltage model could be one reason causing the relatively worse single neuron prediction performance of sensory neurons. Considering the spiking properties of sensory neurons, they might be more sensitive to inaccurate calcium-voltage models compared to interneurons and motor neurons.\n\n•\tHow does warm prediction look like? Could authors show some population trajectories? Could authors discuss what had led to the inaccuracy in whole warm predictions, whether due to inaccurate prediction of a particular subset of neurons or whether due to inaccurate prediction of overall brain state? \n", "summary_of_the_paper": "The current manuscript presented a connectome constrained latent variable model for whole-brain calcium imaging of C.elegans nervous system. The whole-brain calcium imaging of C.elegans was collected while C.elegans was undergoing chemosensory testing, and the dataset has been published recently. In the current study, the authors aimed to present a model that could predict the single-neuron and the single-trial activity of this dataset. Specifically, the activity of each neuron of the whole brain imaging was modeled by latent variable analogous to the voltage of one unit in the model network. The connection between model network units was constrained by the connectome. The authors showed that the connectome constraints significantly improved the prediction power of the model, in predicting the activity of missing units, as well as predicting the single-trial activity of hold-out warms. Overall, the authors have provided a clear description of the model and carried out a good amount of experiments to evaluate different model variants.", "summary_of_the_review": "In general, I vote for accepting the current paper if the authors could address all of my concerns above. I think the authors have presented a model for a novel whole-brain calcium imaging which was only recently published. Overall, the modeling strategy makes sense. I like that the authors tested on different variants of the model, which are all relevant, and provided insights on model selection. One limitation of the current work is the model generalization. I think the application of the model is limited to the current training data. I’m not convinced that the current model could generalize to chemosensory tasks, other than the ones in the training sets.  My major concern is about the clarity of some parts of the paper and some additional improvement in the work. I hope the authors could address my concerns in the rebuttal.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors build an encoding model of whole-brain brain activity by integrating incomplete functional data with anatomical/connectomics data. This work is significant from a computational neuroscience perspective because it constitutes a proof of concept regarding how  whole brain calcium imaging data can be used to constrain the missing parameters of a connectome-constrained, biophysically detailed model of the C. elegans nervous system. There were issues related to clarity in the initial submission which all appeared to have been addressed in the final revision. This paper received 3 accepts (including one marginal accept) and 1 reject. The paper was discussed and the reviewers (including the negative reviewer) were unanimous that the current submission should be accepted.
Decision: Accept (Poster)