Modeling Dynamic Neural Activity by combining Naturalistic Video Stimuli and Stimulus-independent Latent Factors
Authors: Finn Schmidt
Polina Turishcheva
Suhas Shrinivasan
Fabian H. Sinz
Venue: NeurIPS 2025 poster ()
Keywords: Dynamics of neural representations
probabilistic neural predictive models
primary visual cortex of mice
variational encoding models
stimulus-independent variability
Primary area: neuroscience_and_cognitive_science
TL;DR: This paper presents a probabilistic model combining video inputs with stimulus-independent latent factors to model dynamic neural activity in mouse V1.

Abstract: The neural activity in the visual processing is influenced by both external stimuli and internal brain states. 
Ideally, a neural predictive  model should account for both of them.
Currently, there are no dynamic encoding models that explicitly model a latent state and the entire neuronal response distribution.
We address this gap by proposing a probabilistic model that predicts the joint distribution of the neuronal responses from video stimuli and stimulus-independent latent factors. 
After training and testing our model on mouse V1 neuronal responses, we find that it outperforms video-only models in terms of log-likelihood and achieves improvements in likelihood and correlation when conditioned on responses from other neurons. 
Furthermore, we find that the learned latent factors strongly correlate with mouse behavior and that they exhibit patterns related to the neurons’ position on the visual cortex, although the model was trained without behavior and cortical coordinates.  
Our findings demonstrate that unsupervised learning of latent factors from population responses can reveal biologically meaningful structure that bridges sensory processing and behavior, without requiring explicit behavioral annotations during training.
The code is attached to the submission.

Modeling Dynamic Neural Activity by Combining Naturalistic Video Stimuli and Stimulus-Independent
Latent Factors
Finn Schmidt finn. Setmmiitt@ui-ggettingeee
Polina Turishcheva'
uni-goettingen..e serrsacheerdeeuun--petttigggggde
Suhas Shrinivasan shrinivasan&cs. uni uni-uni--iiee -o--gggttnnnnnneee de
Fabian H. Sinz
s sinz@uni-goe -loz@unn-goetttigen..d
Institute of Computer Science and Campus Institute Data Science, University of Göttingen, Germany
Lower Saxony Center for Al & Causal Methods in Medicine, Germany
Abstract
The neural activity in the visual processing is influenced by both external stimuli and internal brain states. Ideally, neural predictive model should account for both of them. Currently, there are no dynamic encoding models that explicitly model latent state and the entire neuronal response distribution. We address this gap by proposing probabilistic model that predicts the joint distribution of the neu- ronal responses from video stimuli and stimulus-independent latent factors. After training and testing our model mouse V1 neuronal responses, we find that it out- performs video-only models in terms of log-likelihood and achieves improvements in likelihood and correlation when conditioned responses from other neurons. Furthermore, we find that the learned latent factors strongly correlate with mouse behavior and that they exhibit patterns related to the neurons' position the visual cortex, although the model was trained without behavior and cortical coordinates. Our findings demonstrate that unsupervised learning of latent factors from popula- tion responses can reveal biologically meaningful structure that bridges sensory processing and behavior, without requiring explicit behavioral annotations during training. Code: gifnuk.com/iinaahhcddtttt...
Dynamic Latent State
Introduction
Predicting the activity of sensory neurons a major goal of computational neuroscience towards understanding the mechanisms of information encoding in the brain. In particular, accurately predicting neural responses in the primary visual cortex (VI) response to a given stimulus could provide deeper insights into how the brain processes visual information. However, this task remains challenging because neural activity in the visual cortex exhibits variability not only across different visual stimuli but also across repeated presentations of the same stimulus [38 42). This variability arises from the influence of numerous unobservable factors that the visual cortex integrates into its processing, For example, the activity in the visual cortex is influenced by factors such behavioral tasks 13, 22), attention 12 and general brain states correlated with behavior [15] 32 35 40). Recording all of these additional variables is not feasible, especially because most of them unknown (4). Thus, understanding the sensory processing of the visual cortex requires models that capture both stimulus-dependent and shared stimulus-conditionel variability [5]. One way to do
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

A. Problem statement: animal watches the same video, but different brain states lead to different neuronal responses. B. Pro- posed solution: infer the brain state with a probabilistic latent-variable model that predicts marginal Inflated- (ZIG) distribu- tions for each neuron.
Figure 1:
Predictive model
Nouronal
recordings
Encode bniin stare
ZIG
Iosa
it is to model internal fluctuations using latent variables that capture a shared state across neurons, explaining the variability in responses the same stimulus. To this end, such models require probabilistic framework to effectively incorporate and infer these latent variables.
The majority of video encoding models for neuronal activity in visual cortex focuses on models that predict neural activity conditioned solely on the stimulus (39 43 47 or on mapping neural activity to meaningful low-dimensional latent spaces, independent or only conditioning on the underlying task or stimulus (20 24 36 41]. To the best of our knowledge, only a few models exist that capture the correlated variability of large neuronal populations and the visual stimuli input [5]. None of them are designed for dynamic video-based stimuli.
In this paper, we address this gap with the following contributions:
We propose probabilistic neuronal predictive model that accounts for both dynamic visual stimuli and latent brain state. The brain state is extracted from a subset of neuronal activity (Fig. I
Our model surpasses comparable models without latent in predicting neuron response distributions in terms of log-likelihood.
We show that the model's latent variables are meaningful, as they are strongly correlated with
mouse behavior (as expected from experimental work (32 35 40) and exhibit topographic patterns
along the cortical surface despite no exposure to behavioral data or cortical position data.
2 Related work
Existing data-driven neural response prediction methods fall into three categories:
deterministic models that predict neural activity from visual stimuli but do not account for response uncertainty and variability [e.g. 3
23
39
43
46 47),
probabilistic models focused on deriving a latent state from neural responses, sometimes condi-
tioning on the external sensory inputs 20 36 41 55).
3
models, using both sensory input and neuronal activity to predict other neurons' responses 5 25
App. A summarizes related works in terms of inputs, outputs, and presence of latents.
Deterministic models for visual cortex Recent advances in deep learning, particularly convolu- tional neural networks (CNNs) trained on image recognition tasks [10 I 33 52], have significantly improved predictive models. However, while task-driven models work well for responses from macaque visual cortex, they do not necessarily work as well for mouse visual cortex (9] Since our focus is on recordings from the mouse visual cortex, we focus on data-driven models, trained end-to-end on neuronal activity in the following. Early work focused on developing specialized stimulus feature representations [called cores, (2. 16 or readout architectures [27 [9 that map the
output of the core to neuronal representations. Sinz al. [39] extended the core video stimuli, adding a gated recurrent unit, and incorporating behavioral data such as pupil dilation and running speed. The inclusion of these behavioral variables allows the model to capture aspects of latent brain state and their interaction with neuron selectivity. For instance, Franke et al. [17 showed that behavioral activity modulates stimulus selectivity in mouse visual cortex when processing colored natural scenes. Höfling et al. (23), Vystrčilová et al. [46] further improved core architectures with factorized 3D convolutions for dynamic stimuli, achieving faster training and better performance than GRU-based models. In parallel, Wang et al. [47 introduced a foundational model for mouse visual cortex with a preprocessing module that adjusts for pupil position and integrates recurrent components into the core. Recently, transformer- based models 3 28 have shown promising results.
While these models reproduce biological phenomena and achieve state-of-the-ar performance, they all train point estimators, i.e., deterministic networks that predict a single value for each trial. The lack of a proper probabilistic model prevents them from computing lielliiels and limits their ability to model the stochastic distribution of neuronal activity. Wu et al. [51] proposes variational

---

model for the network weights to model neurons' distributions, but this model only takes visual stimuli as input and does not model an independent latent state.
Probabilistic models focused on a latent state Previous work has made significant progress in reducing high-dimensional neuronal recordings to smooth low-dimensional manifolds, often interpreted as latent states. However, these models do take the animal's sensory input into account, which is one strong driving force of neuronal activity, in particular in sensory areas. Yu et al. [54) introduced Gaussian Process Factor Analysis to map neural activity over time into a low-dimensional space, assuming Gaussian distribution for neural activity given the latent variable at each time point. Jensen et al. [24] scaled this approach using variational inference. Gokcen et al. 20 extended further by incorporating time delays to model interactions between neural populations, revealing distinct latent dimensions for shared and region-specingg neural dynamics. Building on this, Gokcen et al. [2 also introduced a shared latent space across sessions. However, these models assume a simple relationship between latent variables and neural activity.
To extend these approaches to nonlinear modeling, other works [36 autoencoders. The encoder learns the approximate posterior {<]ly, a) for latent variables z based on neural responses y and auxiliary data a, while the decoder reconstructs neural data from z. Sussillo et al. uses a generative recurrent network to model temporal dependencies in 2 for neuronal spikes. Zhu et al. [57] extended this work for calcium recordings using pero-inflated gamma distribution as a loss. In contrast, Schneider et al. [36] trains with contrastive loss, integrating behavior or task information to produce similar embeddings for comparable neural activity. Similarly, Zhou and Wei [56] models dependencies between task labels u, latents 2, and observations with two-stage non-contrastive approach. However, those methods are focused on encoding neuron responses into meaningful latent embeddings and do not predict neuron responses based on external stimuli inputs like images or videos.
56
57
use variational
Models using sensory and neuronal input to predict other neurons' activity Recent work on recurrent state space models (RSSMs) used latent states for neuronal response modeling. For instance, Glaser et al. 19 uses observed neural activity and exogenous factors to derive latent state. However, their model limited discrete states. Zoltowski et al. [58] are able to predict both discrete and continuous random states. However, their model is limited to a decision-making task. Apart from RSSMs, Kim al. 25 uses flow model, taking both task-relevant inputs and some neurons to predict responses of the other neurons, but focuses on the auditory cortex, where both task and neuronal inputs are time series. To our knowledge, Bashiri et al. 5 is the only work combining visual input with a latent state. It extends the core-readout framework by adding flow-modeled, ciimulus-condttion
variability, including noise correlations, to the mean activity predicted by the core-readout model. However, the model of Bashiri et al. (5] is limited to static stimuli. Since their approach requires covariance matrix between the responses of all neurons to an image, it scales quadratically with the number of neurons. This makes it computationally too expensive for temporal dependencies, since it would scale quadratically in the number of neurons and timepoints.
3 Proposed latent model
In this work, we modify a well-established deterministic core-readout framework - consisting of factorized 3D convolutional core [46] and a Gaussian readout [9] to probabilistic latent state architecture (details in Ap..C). Specifically, we add an encoder that takes a subset of neurons as input, reduces their dimensionality, and derives latent variable. This latent state is then combined with the transformed visual input to predict the activity of other neurons from the same session. This model is equivalent to a latent variable model that predicts neuronal activity from two independent factors i) latent state and ii) the input video. In contrast to Bashiri et al. [5), our model is video-based and predicts time series of neuronal response and the corresponding marginal distribution for each neuron and time point. We designed the model to predict neuron's 's activity conditioned on any subset of neurons from the same experiment, In addition, it can explicitly infer the latent state, allowing us to analyze its relation to other external factors such as the behavior of the animal.
Model architecture Our model predicts time-varying neuronal responses y E NxT to a video stimulus x E W WXXT,, where N is the number of neurons, T the number of time points, and W and H are the frames' width and height. The prediction also depends on a dynamic, stimulas-independent latent factor e IR RxT with latent dimension & < N (Fig To obtain probabilisticall grounded
3

---

A
Nauron
Figure 2: A. A 3D convolutional neural network- based core extracts features of the video input x. The read-out learns the spatial position (purple dot). The deterministic Poisson baseline predicts the mean response via per-neuron affine func- tion from the learned feature vector w For the ZIG baseline, we double the dimension of w wl..0) to predict response-distribuie parame- ters g, of ZIG distribution.
Responses
Core Rai
ZIO Busaine
Video x
Distriuution
Encoder Modeling a/:ly)
Pesportisss
agmodutt
GAU
BLUGIOO
2- MatLoll
Distiiuttona
B. For the latent model we keep the core-read-our architecture w(..0) and additionally add the latent variable predict g, 0. The encoder (approximate posterior) first reduces the dimensionality of masked neuronal responses y. Afterwards, a GRU computes the posterior q(z yy ~ (u((y), oI), from which we sample the latent state when conditioning on neurons. For the marginal distribution p(y x) we sample the latents from the prior /V(0, I) and integrate out z via Monte-Carlo sampling. C. Directed graphical model of video, latent, and response variables; the dashed line indicates the variational approximation.
latent state, we combine ideas from Bashiri et al. [5] and Zhu et al. [7 by modeling neuronal responses conditioned on stimulus and latent factors using Zero-Inflated Gamma (ZIG) distribution, which provides more realistic assumption for calcium imaging data compared to widespread Poisson assumption. In calcium imaging of neurons, the deconvolved fluorescence signals often contain many zero or near-zero values when neurons are inactive, together with positively skewed values when activity occurs. The ZIG distribution naturally captures this behavior by combining a point mass at zero (to model silent periods) with gamma distribution that represents the continuous, rasitive-valued calcium activity during neural firing. ZIG is mixture of a uniform distribution for zero responses and shifted Gamma distribution for nonzero neuronal responses [49]
N,T II Sp
)) II
PzIG 2y
(x((x, fr (aai Kg. Oze z; v)) Mappp
where P is simultaneously the width of the uniform part and the shift of the Gamma distribution, K; the shape parameter per neuron ger the nonzero response probability, and Ose the scale parameter for each neuron at time-step t. fr(: Ks, () is the probability density function of a Gamma distribution shifted by p. Our latent model predicts the distribution parameters 0 and q based on a given video-stimulus, the encoded latent factors, and the model's 's parameters v/.
die (x, z, v = sigmoid f( (x: 2)) + w/!) 22.) Oze Ox, z, v)) ELU (x: + W. (0)
+ 1.
R; is fitted independent of the stimulus once per neuron on the train set via moment matching avoid un-identifiability conditions during model training (i.e. mean of the Gamma distribution k 0 can be realized by infinitely many combinations of and 0). For each neuron and time point, we use the mean of the predicted ZIG distribution as the response prediction.
We model (( and f((0) using a typical core-readout architecture (Fig A where the core extracts features of the video-input, and the readout maps the relevant features from the core-output to the individual neurons. While the core has the same architecture we doubled the feature vectors of the classical Gaussian readout to predict both q and We additionally tested to decode the latent variables replacing z, with smoothed latent variables g(z,) in Eq. (T However, this did not improve the models' performance (App. D). The chimulus-indepenes latent factors are modeled with an isotropic Gaussian prior across time and factors p(z) = -O, I). We model an approximate posterior (Fig. 2 as a Gaussian with the mean as a function of the responses y, the encoder parameters d, and independent variance: g(zly; Φ) N(uyy; Φ), IT))
model without latent state,
Training We fit the model by maximizing P / via evidence lower bound (ELBO) via
variational inference 7 26]:
PzIG(y (x)= 2 (log((yl,,3) ()) + DuL (z)()) p(z)],
(2)
where (.) represents expected value and DKL. the Kullback-Leibles divergence. The likelihood term
(log p(y i2, x)) (xyy) of the ELBO is computed via Monte-Carle sampling by drawing samples

---

from the approximate posterior y). We found that 150 samples are optimal (App. B Since (z(z)y) p(z) are Gaussians, we computed the Kullback-Leibler divergence analytically. In all of the experiments, the latent model was initialized with a pretrained video-only ZIG model, unless stated otherwise.
Marginalization To compare the latent model with the baselines, which do not take neuronal activity as input, we sample the latent from the prior and marginalize z via Monte-Carlo sampling to compute the response likelihoods given only the video p(y[x):
1 Th Σ Ellyxx,2
p(ylx) /
dz 1
(3)
App. B shows that the log- oogllikelhhodd stabilizes and plateaus after approximately 1000 samples.
Experiments
Dataset
We trained and evaluated our models on the data from the five mice of the SENSORIUM
competition 43 The SENSORIUM IDs of the mice are in App. Responses are 2-photon calcium traces from- 40, 000 excitatory neurons of five mice in layers 2-5 of the primary visual cortex. They were recorded at 8 Hz while the head-fixed mice viewed naturalistic gray-scale videos at 30 Hz. The signals were synchronized and upsampled to 30 Hz. During training, we used video input with shape (W, H, T) (64, 36, 80), while for evaluation, used the whole length T for each video. Behavioral -ocimoott-toooootoo speed, pupil dilation, and center position also recorded and resampled to 30 Hz.
Baseline models To compare with the standard non- con-pobbabilistic model, we use the SENSORIUM
43 baseline model with a Poisson loss. To disentangle the impact of the ZIG loss and the latent modeling, we use a video-only ZIG-model (without latent) as second baseline. All models share the same hyperparameters wherever possible (Tab. () Since we aim to use correlations between latent variables and behavioral variables [5] 40 as external validation for the model (see below), we
excluded behavioral data during training for all models in contrast to previous work [39 47).
50,, we use aaagle--rial-oorelattoo averaged across neurons to measure model performance. For each neuron i, we compute the correlation between the neuron response predictions Ujts and the actual responses J/jss over all time points of all videos j in the evaluation data. We evaluate the model in two scenarios. First, we want to compare the model with the baselines, which do not have any additional neuronal input. Hence, in the "Non-Conditioned" scenario, we sample latents (!)) from the prior and use the mean of the response distribution predictor
Evaluation Building upon prior work 39
8イ - D...-pxxe p(y-p(y/,,x, x))(z) = (yy (xxxy:/pxxx..xxxxpee))
Here, Yty the predicted time series of the response of neuron to video j.
Then we want to check if adding responses as input helps. So, in the second "Conditioond scenario, we derive the latent state from subset of the population of neurons ýr (Xr) yosyyyn x, )p((yz)) )xl where I is an index set. Algorithmically, the only difference between the scenarios is how we sample z.
As we model full response distributions, we also compute the log-likelihood evaluate how well those distributions are predicted. For our latent model, the latent variables z are marginalized out via Monte-Carlo sampling (Eq. 33; for experiment details see App.
Con-Conditioned evaluation: Latent model improves response distribution modeling The latent model outperforms the video-only ZIG model in terms of log-likelihood (Tab. I. demonstrating the improved capability of the latent model for capturing the full response distributions. Since the neuronal responses continuous and the Poisson distribution is for discrete values only, evaluating the ligelioel of the Poisson model is not meaningful.
Adding a latent to our ZIG model does not hurt predictive performance and yields comparable correlation. The ZIG and latent models have about one percentage point lower correlation compared to the Poisson baseline model (Tab. I. This happens because the ZIG distribution is not an
5

---

exponential family, which leads to trade-off between modeling full distributions (likelihood) and conditional means (correlation) [30.. We test this claim by training the means of the predicted ZIG distribution with Poisson loss ("Poisson loss ZIG'), which leads to a comparable performance with the Poisson baseline model. Details are in the App. G
To test there is better distributional fit than the Gamma distribution for the positive responses, we replaced it with more flexible normalizing- based distribution while keep- ing everything else fixed (latent-state and stimulus dependence). However, adding flow layers did not improve the performance of our ZIG model, suggesting that the ZIG distribution already provides an adequate fit to the neuronal response distributions (App. Finally, we tested whether our latent model also improves performance when combined with different core sises-spcciiclll...... a Vision Transformer (ViT) core as [28] The quantitative trends are the same: the ZIG model with the ViT core performs slightly worse than the Poisson baseline due
Table Predictive performance of models. Log-likelihood is computed in Bits per Neuron and Time. Standard error of the mean (SEM) is reported across three models with different initializations.
Poisson Baseline
Poisson Loss Video-only
Latent ZIG
ZIG
ZIG
Pearson
Correlation Log-
0.195 20.003
0.194
=0.003
0.183 #0.003
0.182 +0.004
Likelihood f
-0.98 #0.04
-0.30 +0005
to the optimization trade-off, whereas the latent model improves performance beyond the Poisson baseline (see App. for details).
Distributional modeling helps out-of-distribution stimuli Point-estimate models, like Poisson baseline model, empirically struggle on OOD stimulus distributions such as gratings or when response distributions shift with stimuli. Since ZIG and latent
moving dots (Tab. 2 PPossson D models learn per-neuron distributions, we hypothesized that they might generalize to OOD stimuli. To assess generalization, we eval- uated our models on the SENSO- RIUM bonus track, which provides neural recordings collected in re- sponses -iittiidiitiittioon (OOD) video stimuli-such as pink-noise clips, whose spatiotemporal statistics signif- icantly diverge from the statistics of naturalistic videos used during train- ing. Learning per-neuron distributions indeed improves OOD performance (Tab. 'ZIG'). The latent model per- forms similarly when sampling from
Table 2: Correlations on half of the neurons for out-of- distribution (OOD) stimuli, averaged across three OODs per mouse: (1) Poisson baseline, (2) ZIG model, (3) Latent ZIG without conditioning, half of the neurons. SEM is across mice.
(4) Latent ZIG conditioned on
Non Conditioned
Poisson
ZIG
Conditioned
Mouse 1
0.058
0.091
0.091
0.156
Mouse 2
0.097
0.128
0.128
0.174
Mouse 3
0.110
0.134
0.133
0.181
Mouse 4
0.084
0.118
0.115
0.185
Mouse 5
0.064
0.091
0.087
0.153
the prior (Tab, [ Won-Conditionedd). Conditioning on half the neurons fur- ther boosts correlations (Tab. 'Con-
Averuge
0.08
0.11
0.11::0.007
0.17::0.009
.Con-
ditioned'), showing that encoding neuron responses also generalizes to OOD.
Increasing the latent dimension improves performance in the 'Conditioned' scenario but could lead to overfitting In the 'Conditioned" scenario, the dimensionality the information bottleneck: bigger latent dimension helps to pass more information from the neuronal input. However, it requires approximating a higher-diie digheer-immesiional distribution, which might be vulnerable to overfitting. To explore the optimal latent dimension,
the latent variable is
assessed the predictive correlation of models with (I) different latent dimensions & and (2) varying portions of responses yr report the correlation
as input. Throughout, mouse, which was never used as input, i.e., where not part of the index set I. Given half of the neurons responses as input in the 'Conditioned" scenario (Fig. A), our latent model achieves a performance of up to 0.27 (=0.003), a boost of 0.09 compared to the video only prediction performance of 0.18. However, in the Noon-Codditione" scenario, increasing the latent dimension actually decreases the correlation from 0.18 of the latent model to 0.16 (Fig. (Fig.BA). Beyond k = 80 we observe no significant changes. A reason for the decrease in the -ondiondii scenario could be that increasing
quarter of the neuron population (n x 2000) per
A).
'The Poisson baseline numbers are lower than in
43
we do not behavior data

---

the latent dimension enhances the latent model's encoding capabilities, while the video-pree processing component remains unchanged. As a result, the model may rely too much on the encoded responses in the latent variables, which might hurt the correlation when sampling from the prior.
We further explore the effect of conditioning on different portions of the neuron population the predictive performance of latent model with low latent dimension (k = 5) and high latent dimension (k 100).
For the low-dimensional latent model, the performance levels off when conditioned on 30% of the neuron population, reaching a correlation of about 0.23 (+0.004). In contrast, the diseesinnns latent model keeps improving with more neurons given to the encoder, finally reaching a correlation of 0.28 (+00004) when 75% of neuron responses given (Fig. 3 B).
In App. M we further tested whether our baseline models achieve comparable performance if we include behavioral variables as replacement for the latent state. However, the respective models do not exceed correlation of 0.20. This indicates that our latent state encodes high-dimensional brain state that is not captured by few behavioral dimensions, in line with previous work [40].
Increasing latent dimension improves log-likelihood in "Non-Conditioned" scenario Because the correlation decreases with the latent dimensionality in the Non-Conditiones scenario, one might expect decline in likeliiiddood performance as well, while it is actually growing with higher latent dimension (Fig. C). To analyze this apparent discrepancy, we checked how the variance of the latent factors affects the log-likelihodd. Since the average norm of the latent feature vectors (9.0) increases with the latent dimension, the variance of distribution w; Zz ~ NO, [[::1?) increases as well (Fig. C).
Eq. show that if w. Z₁ has an increased variance, we sample the distribution parameters q. 0 from broader range. Hence, high- diteensinnii model is more likely to put probability mass more extreme response values and does not concentrate all the probability mass around the mean. Examples can be seen in App. This improves the likeliiel log p(y x) z log( Pyy/xx z())) - -lo(()) L), since its more likely to sample a probability density function p(- xx ( with a positive value even for extreme response values y. Thus, high -ighdimensional model predicts more realistic response distributions by better accounting for potential outliers.
0.30
0.2
3.0
Correltton
0.25
Logakee hoed
2.5
homm
0.20
-0.2
2.0
-0.4
0.15
0.10
zamplig from or
conditioning 50% in 40000
-0.8
Lg kchreed
age Ly
2.0
Aue agt Lo
0.5
100
150
200
0.1
10 20
30 50
75
80
125
150
200
Latent Dimeession
Percenlage neurons conditioned
Latent Dimension
Figure 3: A Average prediction-response correlation across models with varying latent dimensions. B Average conditioned correlation for low- vs. high-dimensional latent models, if different portions of the neuron population are given. C Log-likklihood vs. average norm of latent feature vectors w (g) (8) across latent dimensions. Error bars: SEM over 3 seeds; some points omitted to save training cost.
Latents primarily reflect a stimulus-independens brain-state If we sample from the prior, the latent independent from the vidco input a priori. However, we condition the latents on neuron responses, i.c., sampling the latents from the approximate posterior y), g(zly) can still carry stimulus information due to the collider structure x x- z in the graphical model (Figure 2C).
To test whether latents sampled from q(ll)) reflect Himmlls-iindpendeee residual stimulus content, we ran two control experiments. In the first experiment, we want to ensure that we cannot predict the latent state from the video statistics, therefore, we compute the local contrast Ga for each time point and each neuron measured via the pixel variance in the receptive field of the neuron for the given video frame
brain states rather than
7

---

The receptive field is determined by the maximal receptive field of the convolutional core- as determined by the kernel sizes-tt the readout location of the neuron.
We performed a linear regression argminw;s Σ, I|Wc, + b - %e and computed the R² values of the actual latents z, and the predicted values W c; + b on the test set. We performed a 5-fold cross- validation on the validation data to compute the average R² values. For 4 of the 5 mice, the video contrast does not explain the variance of the latents. However, for one mouse we get values up to 0.27 (+10.01) indicating that the latents sometimes do encode residual stimulus information (Tab. Col. 1).
Table 3: (1) Contrast R²: (2) Conditioned cor- relation; (3) Correlation for mixed repeats; (4) Correlation int the Non-Conditioned scenario. SEM averaged over neurons and videos.
In the second experiment, we use repeated stimuli to validate that the model encodes brain state-related and not jisual-input-relate information. We split the neuronal popula- tion into two halves y = (Yone half, Yother shuffled the matching of the halves across trials that presented the same stimulus, and used the shuffled pairs to predict Yome,half via
Contrast (R*)
Corr.
Corr.
Corr. Non- Conditioned
sume rep.
mixed rep.
Mouse 1
0.05=am
0.17
0.12+0.01
0. 0.12=a00
Mouse 2
-0.19=020
0.19
0.16
0,15=aol
Plyone half x, .ether.hatt)) where i and T(i) de- note the index of the original and shuffled trial. This approach ensures that Yother corresponds to the same video but a different brain state. If primarily captures video responses, the performance should not decline significantly. Conversely, if z mostly reflects brain state, performance should drop to the level without conditioning or worse. Tab. Lab.BCon Col. 2,3 shows that correlation drops for all mice when conditioning on mixed repeats versus same-repeat responses, indicating that the latents encode a stimulus-independentt brain state.
(+)
Mouse 0.27.1001
0.2010.02
0.1710.02
0.161001
Mouse 4 - -0,03=0044
0.24
0.21
0. 0,182001
Mouse 0.01=
0.21 +0.02
0.15
0. 0.14ga01
Averuge 0.02:20.00
0.20
0.16:10.11
0.15=0009
Sanity check: Latent variables strongly corre- late with behavior
1.0-
It is widely known that latent brain state is strongly correlated with be-
Pupil dilation
Treadmill speed
haviour 5 32 40, therefore, we use behaviour to test whether the latent variables sampled from our approximate posterior capture relevant inter-
Average Correcation
0.6
nal states of the brain. Specifically, we computed a canonical correlation analysis (CCA), which finds the linear combination of the latent variables (()),..., (*) with maximal correlation to a cho- sen behavioral variable like pupil dilation or tread- mill speed.
0.4
a.0
mouse
mouse
mouse
mouse
mouss
Although our model has not seen any behavioral data during training, the learned latents show strong correlations with behavioral data (Fig. 4. suggesting that our latents indeed encode mean- ingful brain-state related information. Analysis
Figure 4: Canonical correlation of behavior and latent for each mouse. The CCA analysis was done with 5-fold cross-validation in an 80/20 split. The error bars indicate the standard error of the mean of cross-validation.
details are in App. L
To assess how meaningful these correlations are,
Table 4: Average canonical correlation across all mice for our latent model (ZIG latent) and a CEBRA model trained on the same data. SEM is computed over the different validation correla- tions in the cross-validation.
we compare against a CEBRA baseline (36)
trained on the same data, with hyperparameters tuned to maximize behavioral correlation and with- out visual-stine -ttmuuulss input or held-out neurons (de-
tails in App. D. While the CEBRA model attains slightly higher correlation with pupil dilation, our latent model achieves much higher and more sta- ble correlation with treadmill speed, yielding over- all comparable performance (Table
CEBRA
ZIG latent
Pupil dilation t Treadmill speed f
0.66 (0.004)
0.60 (0.01)
0.27 (0.006)
0.42 (0.02)
Feature vectors of latent model exhibit topographic organization Bashiri al. (5] analyzed
the relation between latent state and position in the visual space or cortex. This inspired us to explore whether our latent space relates to the cortical positions of neurons as well. The latent space model
has learned two feature matrices, (!) and w(w), where the columns w w are the feature vectors
8

---

for each neuron i. These vectors serve as neuron cerron-specffic weights that project the shared latent state onto that neuron's 's response- response-distriuutee distributton parameters (see Eq. D
First, we visualized these vectors and noticed a spatial organization on the cortical surface. Specif- ically, we computed the most relevant direction in the weight spaces by singular value de- composition (SVD) on the feature matrices, extracting the singular vectors U1, "2, E R that explain most of the variance in the weight vectors. Subsequently, we project each weight vector onto the first three SVs via Mj
and " w. for = 1,2.3 and each neuron i.
um depth
depth
Freely leamned
Position based
Then we plotted the neurons using their cortical xy positions and using the SV val-
First SD
ues as colors (Fig. 5 Columns 1,2). The visualization for ( in Fig. [ simi-
lar visualizations for w())) are provided App.MM
Second SD
Although our model was not explicitly pro- vided with cortical position data during training. it still learned patterns that suggest a spatial organization by cortical position (Fig. 5 Col. 1-3). These patterns appear independent of the neurons' depth the cortex (Fig. Col. and 2). Thus, ron's position seems to contain relevant in- formation for predicting its responses. Be- yond the third singular dimension, we did find more spatially organized patterns (App.M) M
6000
Third SD
400
200
200 400
800
Figure 5: Color gradient maps of the latent feature vec- tors w along the first three Singular Dimensions for mouse 9. All recorded neurons are located within 600pm x 600μmm square in the cortex. Their depth dif- fers at most 200 jum. The first three columns display the color maps for model with freely learned feature vectors, trained without any knowledge of cortical po- sitions. Columns 1-2 use only neurons from a specific depth for both SVD and visualization; Column in- cludes all neurons for both. Column 4 shows model predicting latent features from cortical positions. Rows show the top three singular dimensions in descending order.
Sharing parameters via cortical coordi- nates reduces model size with minimal accuracy loss. The previous experiment suggests that cortical positions are impor- tant for latents, hence, we could use them to learn the latent feature vectors and de- crease the model size.
Instead of storing a separate pair of feature vectors w(9.9) (q.0) for every neuron, we let two small 2-layer MLPs map the neuron's cor- tical position (Px, Py Ps) to w! This substitution removes thousands of free pa- rameters, as we remove the big feature ma- trices, each of size & N (latent dimension x number of neurons), model parameters for each of the five mice. The decrease log-likelihood and conditioned correlation only marginal (Tab. 5) This confirms that neighboring neurons naturally have similar latent feature vectors. The topo- graphic maps of coordinate- based W(..e) (Fig.5) column 4) closely match those from the response-based (freely learned) latent model, indicating that these structures are Correlation t
Table 5: Comparison of log-likelihood (in bits per Neu- ron and Time) and conditioned correlation of models with differently learned latent feature vectors w()), w(4), The feature vectors are learned as independent model parameters per neuron (freely learned), predicted from cortical positions via an MLP same vector is shared across neurons, but can differ in scale (same). All models use a latent dimension k = 12. Standard error of the mean (SEM) is reported across three models with different initializations.
resitiion-asedd or the
Freely learned
Position based
Same
Log-LLikelhhn 1
-0.34 10.05
-0.69 10.44
-0.88 40.04
Conditioned
0.24 #0.003
0.22 +0.005
0.18 +0.003
strongly present in the data. As a control, we trained a model where all neurons shared the same feature vector, differing only by per-neuron scale. Its worse performance shows that neurons differ not just in response strength, but also in which latent dimensions they emphasize across the cortex.

---

Cortical-based latent mapping can extrapolate to unseen neurons To test if the cortical-basee mapping from the previous experiment generalizes unseen neurons, we re- moved N = 500 neurons per mouse during the first phase of training the latent model. In the second phase, we only finetune the
Table Correlation for 500 held-out neurons per mouse, conditioned on half of the training neurons ("Neurons'). ) We compare a model trained on all neurons (" ('Original") with model that must extrap- olate the mapping for the latents to those neurons ( ("Extrapolation') SEM is across mice.
weights and spatial position of pl9) and fit
of the video-encoding part (Eq. (.. and the la-
tent feature vectors w; and w (9) are computed by the previously trained MLP. We compare the performance of the fine-tuned model on those N 500 neurons against model which was trained end-to-end on all neurons (Tab, 6 'Orig- inal'). We find that the position-base based latent model can extrapolate the mapping of latents to unseen neurons using only their cortical position with almost no drop in predictive performance, demonstrating that the learned cortical-bact based mapping can extrapolate to new neurons using cortical coordinates alone.
Neurons
Original
Extrapolation
Mouse
3932
0.185
0.185
Mouse
3954
0.255
0.245
Mouse
3970
0.205
0.205
Mouse
4101
0.215
0.210
Mouse 10
4061
0,205
0,205
Average
4003
0.213 :0.005
0.210 =0.004
Discussion
Cortical neuron activity variability arises primarily from two sources: stimulus-driven variability and internally driven variability from unobserved processes, such as behavioral tasks or brain states, which induce correlated activity across neurons. In this work, we showed that adding latent factors to a video encoding model enables the prediction of a joint dynamic response distribution and captures biological variables by implicitly learning correlations between latent factors and behavior. We also showed that the influence of the latent variables on the neuronal response is topographically organized in robust way and that the latent variables are not primarily driven by the visual input.
Limitations and possible extensions Although it remains difficult to isolate how much variance is explained by visual input alone [34,, our control analyses suggest that the latent variables largely encode indeuuendnndeeeeentt factors; a residual stimulus-dri -unules-driven component, however, cannot be fully excluded. The model also uses more parameters per neuron than the video-only baseline. For simplicity, we implemented our latent method on a competition baseline [43]: future work should assess its performance when paired with state-of-the-an core representations.
Future extensions could relax distributional assumptions toward non-isotropic or non-Gaussian priors, replace the ZIG loss with a flow-based objective, or embed the method in new state-of-the-al transformer models [45]. Our method should integrate with most of the modern models that predict neuronal responses. It has to be further investigated whether the robust topographic organization of the latents reflects a biologically meaningful length scale, spatially organized brain waves [33, or simply residual retinotopic influences not captured by the video model.
Acknowledgements We thank Dominik Becker, Michaela Vystrčilová, Lucas Mohrhagen, Flo- rain Seifert, Alexander Ecker for the technical help and insightful discussions. Computing time was made available on the high-performance computers HLRN-IV at GWDG at the NHR Center NHR @ Göttingen. The center is jointly supported by the Federal Ministry of Education and Re- search and the state governments participating in the NHR (www.nhr- verein. dwwpeiindd//seeeeeeeeeeee paruuserrr--rttrr...... FHS acknowledges the support of the German Research Foundation (DFG): SFB 1233, Robust Vision: Inference Principles and Neural Mechanisms Project-ID 276693517 and SFB 1456, Mathe- maties of Experiment Project-ID 432680300, and the European Research Council (ERC) under the European Union's 's Horizon Europe research and innovation programme (Grant agreement No 101171526). FHS acknowledges the support of the Lower Saxony Ministry of Science and Culture (MWK) with funds from the Volkswagen Foundation's 's wakkunff.iederrachsseeee program (project name: CAIMed Lower Saxony Center for Artificial Intelligence and Causal Methods in Medicine; grant number: ZN4257. This project has received funding from the European Research Council (ERC) under the European Union's 's Horizon Europe research and innovation programme (Grant agreement No. 101041669).
www.nhhr-verein.cr
10

---

References
[1] Takuya Akiba, Shotaro Sano, Toshihiko Yanase, Takeru Ohta, and Masanori Koyama. Optuna: A next-generation hyperparameter optimization framework. In Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining, pages 2623-2631, 2019.
[2] J Antolik, S B Hofer, J A Bednar, and T D Mrsic-flogel. Model constrained by visual hierarchy improves prediction of neural responses to natural scenes. PLoS Comput. Biol., pages 1-22, 2016.
[3] Antonis Antoniades, Yiyi Yu, Joe S Canzano, William Yang Wang, and Spencer Smith. Neu- roformer: Multimodal and multitask generative pretraining for brain data. In The Twelfth International Conference on Learning Representations, 2024. URL UtLhtttss://oo httpss// net/forrum2dd-W8888:22
[4] Mehdi Azabou, Vinam Arora, Venkataramana Ganesh, Ximeng Mao, Santosh Nachimuthu, Michael Mendelson, Blake Richards, Matthew Perich, Guillaume Lajoie, and Eva Dyer. A unified, scalable framework for neural population decoding. In A. Oh, T. Nau- mann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neu- ral Information Processing Systems, volume 36, pages 44937 Curran Associates, htwp:///nnnnneeei.c...eceepeeeee 2cc.aa/226641117777772aa8a2aapap
Inc., 2023. URL
pttps//appee//22////////22/ pdf
[5] Mohammad Bashiri, Edgar Walker, Konstantin-Klemean Lurz, Akshay Jagadish, Taliah Muham- mad, Zhiwei Ding, Zhuokun Ding, Andreas Tolias, and Fabian Sinz. A flow-based latent state generative model of neural population responses to natural images. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, volume 34, pages 15801-15815. Curran Associates, Inc., 2021. URL https htt.tpps://prrceeeinngeeooooeeoe http://rroeeddingg.eeeeeess.eppp www.wp..rn...mm wwcppppr_eaaeeeeeeeeeeeeeeeeeeee filel/l//lffe. waweeeffflleppppee///eee /2pp///2//////lle
.dd
[6] Martin Bjerke, Lukas Schott, Kristopher T Jensen, Claudia Battistin, David A. Klindt, and Benjamin Adric Dunn. Understanding neural coding on latent manifolds by sharing features and dividing ensembles. In The Eleventh International Conference on Learning Representations, 2023. URL httthtttt//ppererrtii/ Toa
net/ wettfoormmidddaaeeeeeeettr
[7] David M. Blei, Alp Kucukelbir, and Jon D. McAuliffe. Variational inference: A review for statisticians. Journal of the American Statistical Association, 1 112(518) (8:8888859-877 April 2017. ISSN 1537-274X. doi: 10. 1..111/00111199.0000 201.:4999..11...87777 URL http: doi org/10 1080/0 0t////111214992000 2017 1285773
[8] Farran Briggs, George R. Mangun, and W. Martin Usrey. Attention enhances synaptic efficacy and the signal-to-noise ratio in neural circuits. Nature, 499(7459):476-480, 2013.
[9] SA Cadena, F Sinz, T Muhammad, E Froudarakis, E Cobos, E Y Walker, Reimer, M Bethge, and A S Ecker. How well do deep neural networks trained on object recognition characterize the mouse visual system? In NeurlPS 20/9 Workshop Neuro AI, 2019.
[10] Santiago A Cadena, George Denfield, Edgar Y Walker, Leon A Gatys, Andreas S Tolias, Matthias Bethge, and Alexander S Ecker. Deep convolutional models improve predictions of macaque v1 responses to natural images. PLOS computational biology,
2019.
[11] Charles F Cadieu, Ha Hong, Daniel LK Yamins, Nicolas Pinto, Diego Ardila, Ethan A Solomon, Najib J Majaj, and James DiCarlo. Deep neural networks rival the representation of primate cortex for core visual object recognition. PLoS computational biology, 10(12):e1003363, 2014.
[12] Marlene R. Cohen and John H. R. Maunsell. Attention improves performance primarily by reducing interneuronal correlations. Nature Neuroscience, 12(12):1594, 2009.
[13] Marlene R. Cohen and William T. Newsome, Context-dependent changes in functional circuitry in visual area MT. Neuron, 60(1):162-173, 2008

---

[14] Laurent Dinh, David Krueger, and Yoshua Bengio. Nice: Non-linear independent components estimation. In arXiv preprint arXiv:1410.8516, 2014.
[15] Alexander S. Ecker, Philipp Berens, R. James Cotton, Manivannan Subramaniyan, George H. Denfield, Cathryn R. Cadwell, Stelios M. Smirnakis, Matthias Bethge, and Andreas S. Tolias. State dependence of noise correlations in macaque primary visual cortex. Neuron, 82(1): 235 248, 2014.
[16] Alexander S. Ecker, Fabian H. Sinz, Emmanouil Froudarakis, Paul G. Fahey, Santiago A. Cadena, Edgar Y. Walker, Erick Cobos, Jacob Reimer, Andreas S. Tolias, and Matthias Bethge. A rotation-equivariant convolutional neural network model of primary visual cortex. In Inter- national Conference on Learning Representations, 2019. URL btttp:://openreview: net/ Forum'id=HHfUUiAqqk)
[17] K. Franke, F. Willeke, K. Ponder, M. Galdamez, N. Zhou, T. Muhammad, S. Pa- tel, E. Froudarakis, J. Reimer, F. H. Sinz. and A. S. Tolias. State-dependent pupil di- lation rapidly shifts visual feature selectivity. Nature, 610(7930):128 134, 2022. doi: 10.1 101100//118866 186-022-0270----3 /ttps:/sss///
[18] Eloy Geenjaar, Donghyun Kim, Riyasat Ohib, Marlena Duda, Amrit Kashyap, Sergey Plis, and Vince Calhoun. Learning low low-dimensional dynamics from whole-brain data improves task capture, 2023. URL httph::tt///ttt //rrxiv//xxvvv httg/bbs/2330 14369
[19] Joshua Glaser, Matthew Whiteway, John P Cunningham, Liam Paninski, and Scott Linderman. Recurrent switching dynamical systems models for multiple interacting neural populations. Advances in neural information processing systems, 33: 33:14867-148788 2020.
[20] Evren Gokeen, Anna I Jasper, João Semedo, Amin Zandvakili, Adam Kohn, Christian K Machens, and Byron M Yu. Disentangling the flow of signals between populations of neurons. Nature Computational Science, 2(8):512 525, August 2022.
[21] Evren Gokcen, Anna Jasper, Alison Xu, Adam Kohn, Christian Machens, and Byron M Yu. Uncovering motif's of concurrent signaling across multiple neuronal populations. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems, volume 36, pages 3471 4-31222 Curran Associates, /ppp////22/////lee///
Inc., 2023. URL htttttt:://pocceeeteitto neurips. wwwpppppppp..epeeceeeeeeeeeeee.p 6cf 71777777766.5bbb222c000000bbbbbb Ponp://eeeeeeeaaaae-oooooooooooe pdf
Conf
[22] Ralf M. Haefner, Pietro Berkes, and József Fiser. Perceptual makisi as probabilistic inference by neural sampling. Neuron, 90(3):649 -60, 2016.
[23] Larissa Höfling, Klaudia P Szatko, Christian Behrens, Yuyao Deng, Yongrong Qiu, David Alexander Klindt, Zachary Jessen, Gregory W Schwartz, Matthias Bethge, Philipp Berens, et al. A chromatic feature detector in the retina signals visual context changes. Elife, 13:e86860, 2024.
[24] Kristopher Jensen, Ta-Chu Kao, Jasmine Stone, and Guillaume Hennequin. Scalable bayesian gpfa with automatic relevance determination and discrete noise models. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, volume 34, pages 10613-10626. Curran Associates, Inc., 2021. URL /ttp neurips. cc/paper.ppp../ f wwwes//leieeee pitee///apee//2///fiiee// .dff
[25) Timothy Doyeon Kim, Thomas Zhihao Luo, Tankut Can, Kamesh Krishnamurthy, Jonathan W Pillow, and Carlos D Brody. Flow-field inference from neural data using deep recurrent networks. bioRxiv, 2023.
[26] Diederik Kingma and Max Welling. Auto-encoding variational bayes, 2022. URL https //arxiv. org/ /t////b////111 wtt./o//////a///////////// 6114
[27] David A. Klindt, Alexander S. Ecker, Thomas Euler, and Matthias Bethge. Neural system identification for large populations separating "what" and "where". In Advances in Neural Information Processing Systems (NeurIPS). Curran Associates, Inc., November 2017.
12

---

[28] Bryan M. Li, Isabel Maria Cornacchia, Nathalie Rochefort, and Arno Onken. VIt large- scale mouse response prediction using vision transformer. Transactions on Machine Learning Research, 2023. ISSN 1835-8856. /
qHZs2p4ZD4
[29] Konstantin- Klemens Lurz, Mohammad Bashiri, Konstantin Willeke, Akshay Jagadish, Eric Konstantin-Klemens
Wang, Edgar Y. Walker, Santiago Cadena, Taliah Muhammad, Erick Cobos, Andreas Tolias, Alexander S. Ecker, and Fabian H. Sinz. Generalization in data-driven models of primary visual cortex. International Conference on Learning Representations (ICLR), 2021. URL
https:://ppnzeevee...es
[30] Konstantin-Klene Lurz, Mohammad Bashiri, Edgar Y. Walker, and Fabian H. Sinz. Bayesian oracle for bounding information gain in neural encoding models. In The Eleventh International Conference on Learning Representations, 2023. URL https::///ppnteeeee
id=iYC5hOKqug
[31] Jude F. Mitchell, Kristy A. Sundberg, and John H. Reynolds. Spatial attention decorrelates intrinsic activity fluctuations in macaque area V4. Neuron, 63(6))879-888, 2009.
[32] Cristopher M Niell and Michael P Stryker. Modulation of visual responses by behavioral state in mouse visual cortex. Neuron, 65(4):472-479, 2010.
[33] Galen Pogoncheff, Jacob Granley, and Michael Beyeler. Explaining properties with biologically constrained deep learning architecture. Advances in Neural Information Processing Systems,
2023.
[34] Dean A Pospisil and Wyeth Bair. The unbiased estimation of the fraction of variance explained by a model. PLoS computational biology, 17082220 2021.
[35] Jacob Reimer, Emmanouil Froudarakis, Cathryn R. Cadwell, Dmitry Yatsenko, George H. Denfield, and Andreas S. Tolias. Pupil fluctuations track fast switching of cortical states during quiet wakefulness. Neuron, 84(2):355-32,, 2014.
[36] Steffen Schneider, Jin Hwa Lee, and Mackenzie Weygandt Mathis. Learnable latent embeddings for joint behavioural and neural analysis. Nature, 617(7960)3360-360 03800,368.0000 May 2023.
[37] K Seeliger, L Ambrogioni. Y Güçlütürk, LM van den Bulk, U Güçlü, and MAJ van Gerven. Endd-toend neural system identification with neural information flow. PLoS Comput Biol, 17(2): e1008558, Feb 2021. doi: 10. 1/.371/jouall..bbi... 1008555
1008558.
[38] Michael N. Shadlen and William T. Newsome. The variable discharge of cortical neurons: implications for connectivity, computation, and information coding. Journal of Neuroscience, 18(10):880-3386 1998.
[39] Fabian Sinz, Alexander S Ecker, Paul Fahey, Edgar Walker, Erick Cobos, Emmanouil Froudarakis, Dimitri Yatsenko, Zachary Pitkow, Jacob Reimer, and Andreas Tolias. Stim- ulus domain transfer in recurrent models for large scale cortical population prediction on video. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, ed- itors, Advances in Neural Information Processing Systems, volume 31, Curran Associates, Inc., 2018. URL http::////o/o neurips. hts/pape__fileee wawesrffilllpppppee hppppper.fleeeeppprr////////111a /atppp/ppee///eleee/ 9d634c589d6
Paper .ddf
[40] Carsen Stringer, Marius Pachitariu, Nicholas Steinmetz, Charu Reddy, Matteo Carandini. and Kenneth D Harris. Spontaneous behaviors drive multidimensional, brainwide activity. Science, 364(6437):255, 2019. doi: 10. 1tt////ccncc.aaa/55500
[41] David Sussillo, Rafal Jozefowicz, L. F. Abbott, and Chethan Pandarinath. Lfads latent factor analysis via dynamical systems, 2016. URL ltttss///txxiii. httpp:://aaxiv.og//bb/////////// 06315
[42] George J. Tomko and Donald R. Crapper. Neuronal variability: non-stationary responses to identical visual stimuli. Brain Research, 79(3):405-418, 1974.
13

---

[43] Polina Turishcheva, Paul G. Fahey, Laura Hansel, Rachel Froebe, Kayla Ponder, Michaela Vystrčilová, Konstantin F. Willeke. Mohammad Bashiri, Eric Wang. Zhiwei Ding, Andreas Tolias, Fabian H. Sinz, and Alexander S. Ecker. The dynamic sensorium competition for predicting large-scale mouse visual cortex activity from videos. arXiv, preprint, 2023. doi: 10.485500arXxi.235...00 19654. https://// org/10. 10. 200..48550/raa arXiv 2305 19654
[44] Polina Turishcheva, Max F Burg, Fabian H Sinz, and Alexander S Ecker. Reproducibility of predictive networks for mouse visual cortex. Advances in Neural Information Processing Systems, 37:7930-7956, 2024.
[45] Polina Turishcheva, Paul Fahey, Michaela Vystrčilová, Laura Hansel, Rachel Froebe, Kayla Ponder, Yongrong Qiu, Konstantin Willeke, Mohammad Bashiri, Ruslan Baikulov, et al. Ret- rospective for the dynamic sensorium competition for predicting large-scale mouse primary visual cortex activity from videos. Advances in Neural Information Processing Systems, 37: 1 118907-118929, 2024.
[46] Michaela Vystrčilová, Shashwat Sridhar, Max F. Burg. Tim Gollisch, and Alexander S. Ecker. Convolutional neural network models of the primate retina reveal adaptation to natural stimulus statisties. bioRxiv, 2024. doi: 10. 101/2024.03 03..0.0366600000.... URL https://www http:///wwwbbbwxivoo biorxiv
axg./onntent/eally/222222
03. 583740
[47] Eric Y Wang. Paul G Fahey, Zhuokun Ding, Stelios Papadopoulos, Kayla Ponder, Marissa A Weis, Andersen Chang, Taliah Muhammad, Saumil Patel, Zhiwei Ding, al. Foundation model of neural activity predicts response new stimulus types. Nature, 64008058):440-477,, 2025.
[48] Yule Wang. Zijing Wu, Chengrui Li, and Anqi Wu. Extraction and recovery of spatio- temporal structure in latent dynamics alignment with diffusion models. In A. Oh, T. Nau- mann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neu- ral Information Processing Systems, volume 36, pages wttp://e////prr/22////10000 wwwrinurriicooo cc/paper httpppperflleeppppppeeeeeeee
Curran Associates,
Inc., 2023. btL "abboo0aad6111444466666666222220 Conference pdf
[49] Xue-Xin Wei, Ding Zhou, Andres Grosmark, Zaki Ajabi, Fraser Sparks, Pengcheng Zhou, Mark Brandon, Attila Losonczy, and Liam Paninski. A zero-inflated gamma model for post- deconvolved caleium imaging traces. May 2019. doi: 10.1101/637652. URL httt:://dx.... doi
org/10. 1000/176667
[50] Konstantin F Willeke, Paul G Fahey, Mohammad Bashiri, Laura Pede, Max F Burg, Christoph Blessing, Santiago A Cadena, Zhiwei Ding, Konstantin-Klena Lurz, Kayla Ponder, et al. The sensorium competition on predicting large-scale mouse primary visual cortex activity. arXiv preprint 20Xiv:2206.08666, 2022.
[51] Nan Wu, Isabel Valera, Fabian Sinz, Alexander Ecker, Thomas Euler, and Yongrong Qiu. Probabilistic neural transfer function estimation with bayesian system identification. Plos Computational Biology, 20(7):e1012354, 2024.
[52] Daniel LK Yamins, Ha Hong, Charles F Cadieu, Ethan Solomon, Darren Seibert, and James DiCarlo. Performance-opimiin hierarchical models predict neural responses in higher visual cortex. Proceedings of the national academy of sciences, 11 111123)8619-8624, 2014.
[53] Zhiwen Ye, Matthe S Bull, Anna Li, Daniel Birman, Tanya L Daigle, Bosiljka Tasic, Hongkui Zeng, and Nicholas A Steinmetz. Brain-wide topographic coordination of traveling spiral waves. bioRxiv, page 2023. 12..1.....505555 December 2023.
[54] Byron M Yu. John P Cunningham, Gopal Santhanam, Stephen Ryu, Krishna V Shenoy, and Maneesh Sahani. Gaussian-procest factor analysis for low-dimensional single-trial analysis of neural population activity. Advances in neural information processing systems, 21, 2008.
[55] Byron M. Yu, John P. Cunningham, Gopal Santhanam, Stephen I. Ryu, Krishna V. Shenoy, and Maneesh Sahani. Gaussian-process factor analysis for low-dimensional single-trial analysis of neural population activity. Journal of Neurophysiology. 102()):614-635, 2009. doi: 10.1152/jn. 90941.2008. Epub 2009 Apr 8; Erratum in: Journal of Neurophysiology. 2009 Sep:102(3):2008.
14

---

[56] Ding Zhou and Xue-Xin Xin Wei. Learning identifiable and interpretable latent models of high- dimensional neural activity using pi-vae. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 724777277. Curran Associates, Inc., 2020. URL http:ttt///roeeenngssstttt neuriis ec/ paper_ files/ 2ttpar/filleeappe///2///////////
[57] Feng Zhu, Harrison A Grier, Raghay Tandon, Changjia Cai, Anjali Agarwal, Andrea Gio- vannucci, Matthew Kaufman, and Chethan Pandarinath. A deep learning framework for inference of single-trial neural population dynamics from calcium imaging with subframe temporal resolution. Nature neuroscience, 25(12)1124--1734 2022.
[58] David Zoltowski, Jonathan Pillow, and Scott Linderman. A general recurrent state space framework for modeling neural dynamics during decision-making. In International Conference on Machine Learning, pages 16800-1691, PMLR, 2020.
15

---

A Related Works Overview
An overview of related work, which either predicts neuron responses for given natural stimuli, uses latent representations for encoding neuron responses or takes behavior of the animal into account, is summarized in App. A
Table 7: Summary of Reviewed Literature: Neural activity categorizes if the recorded neural activities are dynamic or static, and their role as input/output of the model. Stimulus details the type of stimulus components. Learned Latent indicates if the used model has a latent space and if distribution is computable for the latent (probabilistic). Behavior describes the involvement and role of behavioral data. Datasets lists the datasets, including subject types, data collection methods, tasks performed during recordings, and neuron sample sizes.
References
Neural activity
Stimulus
Learned latent
Bchavior
Datasets
Schneider et al. 36
Dynamic Input
No
Yes, probabilistic
Input
Rats, Mice, Monkey
2p and electro- physiology
10- 1000 Neurons
Wang al.
47
Dynamic Output
Video
No
Input
Mice, 2p. passive 140,000 Neurons 40,000 Neurons
Turishcheva et al.
43
Sinz al.
39
Kim et al.
25
Dynamic In/Output
Audio
Yes, probabilistic
No
Synthetic and Rats, audio decision- making 67 Neurons
Antoniades ct al.
3
Dynamic Output
Video
Yes, not probabilistic
Output
Mice, 2p, passive 386 Neurons Monkey 27,373 Neurons
Azabou et al.
No
Gokcen et al.
20
Dynamic Input
No
Yes, probabilistic
No
Macaque V1-3
120 Neurons
Sussillo ct al.
Synthetic
30 Neurons
Zhou and Wei |56
Dynamic In/Output
No
Yes, probabilistic
Task
Monkey, reaching-task Rat, running
Input
192, 120 Neurons Monkey, Rat task
Wang 48
No
200 200 Neurons
Bjerke al. 6
No
Mousc,Ran Rats 26 and 149 Neurons Macaque, reaching-task 200 Neurons
Jensen et al.
Geenjaar et al.
Dynamic In/Output
No
Yes not, probabilistic
No
MMRI
Seeliger et al.
Bashiri et Dynamic Output
37
Vidco
No
No
MMRI
Statie
Image
Yes. probabilistic
Output
Mouse VI/LM, 2p.passive
Output
4,000 Neurons
16

---

B Monte-Carlo Approximations
We have approximate expectation values via Monte-Cart sampling two times in model setup. First, during training we have to calculate the likelihood (log p(y|z, z, x)) in the ELBO via Monte-Carlo sampling (Eq. E Second, during evaluation, we draw samples from the prior distribution (0, Ig), where k is the latent dimension, for calculating the log-likelihood log ply (x) as described in Eq. 2
The ELBO was approximated with up to 150 samples. More samples yield memory issues during training on a single A100 GPU. We observed the log- ligelliedoood performance during inference to increase with the sample size during training as the ELBO is approximated more accurately (Fig. 6). For the Monte-Carhe approximation of the marginalized log-likelihood during inference. we tested different sample sizes ranging up to 5000. It was not possible to test more than 5000 samples, because at that point we ran into memory issue on single A100 GPU, However, Fig. 7 indicates that the log-like likelihood stabilizes and approaches plateau after approximately 1000 samples. We thus resorted to drawing 1000 samples to approximate the marginalized log- log-likelihon likeliihdoodd. The correlation performance of the latent model was not strongly influenced by the sample sizes.
-0.7
-0.0
-0.0
Log-lkkelhhos
Log Lioelhoood
0.9
-0.9
-1.0
-11
-1.1
10 Number of Samples to Approximate RO
100 150
10
100
1000
5000
Number prior samples for evaluation
(measured in bits per neuron and time) of models trained with different sample sizes to approximate the ELBO during training. The error bar indi- cates the standard error of the mean of the log- likelihood for models initialized and trained on 3 different seeds. Due to the computa- tional cost training model, the standard error was calculated only at single point.
Figure 6:
Figure 7: Log-likelihood (measured in bits per neuron and time) for different sample sizes L of the prior during evaluation. Error bars indicate the standard error of the mean for sampling from different seeds.
Table 8: Hyperparameter Configuration
General
C Model Architecture and Hyperparmeters Setting
Learning rate
0.005
Core
All experiments and training of models were done a single A100 GPU with 80GB of memory.
Number Layers
Temporal Kernel Size first Layer
The video processing part consists of factorized 3D-
Spatial Kernel Size
(11,11)
CNN block followed by a Gaussian readout ((23)). Each convolutional layer consists of factorized 3D
first Layer
Spatial Kernel Size
(5.5)
convolution across spatial and temporal dimensions followed by a batch normalization layer and an ELU activation function. We use a variational autoen- coding approach for the latent representations. dropout layer is applied to the neuron responses be- fore they are fed into the encoder. This prevents the model from learning correlations between spe- cific neurons, thereby encouraging the learning of global latent representations. The encoder applies linear layer reducing the dimensionality of neuron responses N 8, 000 (NV ranging from 7800 8200
other Layer
Spatial Kernel Size
(5,5)
other Layer
Channels per Layer
(32,64,128)
Encoder
GRU
Number Layers
Dropout probability
50%
Output Dim Linear Layer
42
Output Dim GRU
17

---

Table 10: An overview of the details of each experiment conducted in the main text. In the row for Table 1 we only refer to the latent model.
Positional
Pre-trained Video-Part
Latent Dimension I
Evaluation Neurons
Used Mice
Data
(per mouse)
ID'S in 12
Table
True
True
12
Half
6-10
Table 2
False
True
12
Half
1-5
Table
True
True
12
Half
1-5
Table 4
True
True
12
Half
6-10
Table
False
False
12
500
6-10
Figure 3
True
True
1-200
Quarter
6-10
Figure
True
True
1-200
Quarter
6-10
Figure 5
True
True
12
All
9
depending on the mouse as part the SENSORIUM dataset 44 to n < N. For each time step, the same linear layer is applied independently. The linear layer is followed by layer normalization and an ELU activation function. An individual linear layer was trained for each mouse. The output is processed by a one-layer recurrent neural network with gated recurrent units producing the means ((y;;) ()) of the approximate posterior gz) Φ) N (y((;; v) II)). a is learnable model parameter of the encoder. is combined with the outputs from the video encoding part computing the parameters
for the probability log p(y|z, x) as in Eq. (T. For a graphical illustration, see Fig. 2 We searched
over various hyperparameters (Table 8 using Optuna [I]
D Optimizing Encoder Architecture
To ensure optimal encoding of the neuron responses, we tested a 3D factorized CNN and a GRU architecture. We further tested different layer sizes ranging from 1-5. We found that one-layer GRU encoder performs best in terms of correlation and log-likelihood.
Further, we tried to add a "smoother" g to further smooth the latents in time. Compared to the model reported above, z; replaced by g(z,) in Eq. (I). We tested how model with a CNN- smoother or GRU-smoother model performed compared to model without smoother. As Tab. g indicates, replacing the GRU with a CNN does not make a significant difference. However, the model without smoother had the best performance in log- ligeeikll and conditioned correlation (Tab.
Table 9: The columns present the performance of models with 12-dimensional latent space, each trained using either a GRU smoother, CNN smoother, or no smoother.
E Experiment Details
Depending on the analysis setup, we adjusted the model architecture and training configuration as needed. For experiments examining cortical patterns revealed by the latent model, we ensured that no positional data was provided. Although the Gaussian readout allows neuron positions be used for spatial predictions the video feature map, disabled this option for these experiments (Tab. 10 'Positional Data" column). Additionally,
GRU
No Smoother Smoother Smoother
CNN
Log-Likelihood Conditioned
-0.74
-0.77
-0.30
0.23
0.22
0.24
Correlation
we found that initializing the vidco-processing weights of our latent model with those from pre-trained video-only ZIG model improved predictive performance (Tab. [T]. Therefore, we applied pretrained weights for the video component whenever the experiment setup allowed (Tab, TT 'Pretrained Video-Part' column). Since we tested different latent dimensions, we specify the latent dimensions used for each experiment (Tab. 10 'Latent Dimension' column). Lastly, we report the number of neurons used during evaluation (Tab. 10 'Evaluation Neurons" column). If the same number of neurons was used for evaluation across experiments, the selected neurons remained consistent. For instance, if one-quarter of the neurons were evaluated, the same subset was used in all experiments. For exact number of neurons per mouse, see Tab. 12
18

---

Table 11: Comparison of latent model with latent dimension & = 12, initialized using weights from pre-trained video-only ZIG model, against a latent model trained from scratch. We evaluate performance based on log-likelihood and correlation, considering both Conditioned Correlation (conditioning half of the neurons) and Prior Correlation (sampling from the prior). Evaluation is conducted on half of the neuron population.
Pre-trained
Scratch
Log-L LigeLiielihhod f
-0.30
-0.36
Conditioned
0.24
0.23
Correlation
Non-Conditioned Correlation 1
0.18
0.10
F
Mice IDs
For reproducibility, list the Sensorium ID's of the mice, which were used for training and evaluation in our experiments, in Tab. [2] the main part, they were referred as mice 1-10. Further, Tab. 12 shows the exact number of recorded neurons per mouse.
Table 12: Mice 1-10 of the main text with corresponding SENSORIUM IDs ('IDs'). The total numbers of recorded neurons per mouse are tracked (" N'eurooss).
Neurons
ID
Mouse 1
7440
29156-11-10
Mouse 2
7928
29228-2-10
Mouse 3
8285
29234-6-9
Mouse 4
7671
29513-3-5
Mouse 5
7495
29514-2-9
Mouse 6
7863
29515-10-12
Mouse 7
7908
29623-4-9
Mouse 8
7939
29712-5-9
Mouse 9
8202
29647-19-8
Mouse 10
8122
29755-2-8
G Trade-off between Modeling distribution and Modeling conditional Mean
As the ZIG distribution is not an exponential family with the data mean as a sufficient statistic, better modeling of the full response distribution (i.e., higher log-likelihood) does not necessarily translate into more accurate conditional-mean predictions (i.e., higher correlation).
To check this, we trained ZIG model using a Poisson loss during training. For this, we insert the means of the predicted ZIG distributions into the Poisson loss. For each neuron i and timepoint our ZIG model predicts the ZIG-parameter qar (x; v) and 0 (x; vb) (as in Fig. A) based on the video input x and its parameters vb. With these we analytically compute the mean of the ZIG distribution Ase Pse(gie(x; vb), Oze (xx(x; w)), Ki, p). During training, we then use Poisson Loss between those predicted responses ra and the observed neural responses ra as objective:
min Σ Pse (ux(x 1b). (x),)))
ki,)) k.,, - Vit llog ia Fa(aa(x;;)) 1b),
(x(;; (ic
Consistent with our hypothesis above, shown in Tab. 13 training video-only ZIG model with Poisson loss raises its correlation from 0.183 to 0.195, comparable to the Poisson baseline model.
We observe similar pattern with our latent model: swapping its training objective from ZIG to Poisson loss yields notable boost correlation (especially when conditioning on half of the neuronal population) (Tab. B
19

---

Table 13: Predictive performance the Poisson Baseline model, video-only ZIG model trained with Poisson loss, and a video-only ZIG model trained with ZIG loss. Further, the predictive performance Latent ZIG model trained on Poisson-loss and ZIG-loss is reported with and without conditioning half of the neurons' responses. Log-likelihood is measured in bits per neuron and time. Performance is evaluated only on half of the neurons that are not used for conditioning. Within and "Conditioned" best result is set in boldface.
each "ooioonNNo-oootiiii
Poisson ZIG Poisson-loss
ZIG ZIG-loss
Non-Conditioned
Conditioned
Baseline
Latent Poisson-loss
Latent ZIG-Ioss
Latent Poisson-loss
Latent ZIG-loss
Pearson Correlation
0.195
0.195
0.183
0.184
0.183
0.266
0.230
H
Flow Model
5
successfully introduced flow to enhance predictive capabilities of their model for neural data Hence, tested, how adding flow T applied our neural responses y impacts our model.
We used Gaussian distribution base distribution which is independent across time and neurons. A Gaussian distribution has the advantage, that we do not have to restrict the transformed responses T(y) be positive. To capture the peak zero, the responses below the threshold p are modeled by a uniform distribution, while the flow used to capture responses above the threshold. We model the base distribution independently over time. Thus, for given latent space 2, and video x, the likelihood of the transformed neuron responses T'y:) at time point is given by:
(1= 11-90e(x, qia(x, Zt)
1 -
p(T (v.))x))...))
II (Gaa<))
II (Bn2Z))
[ir((x 2:) (N
P
The flow model consists of two parts. (I) A flow model T, with learnable parameters, that transforms the neural responses y, for each time point the same way. The flow acts independently across the N neurons: T = [Ti (B11), (22))..... Tv (UNe)) (2) A latent space model, as described in Sec. 3 which predicts the response distribution of the transformed responses r, == Tyy.). Thus, the expected value (x, 2,) and response likelihood ga (ze, x) are predicted by our latent space model. To determine the actual response likelihood p(y: Ze. x), we use the change of variable formula:
ply x) p(T()) 2 x det VyT())
As we choose to apply the flow on each neuron and time dimension separately, this results in diagonal Jacobian:
det V VyT(r) II
Oyat
To predict response for neuron at time point I, we compute the mean of the random variable ya 1; (rw) via Monte-Cart sampling:
BTT (rw)) f T, (ree) p(ralx, dra
1 T.
Σr, (m((m)
For the experiments, used simple flow consisting of two functions:
affine softplus
The affine layer has learnable scale parameter d; and bias parameter b; for each neuron. computes )ffine(y:) = azı + b;. The inverse softplus layer is used to make sure that response values 7 7 are positive for sample ý from the base distribution.
We tested different flows consisting of more functions, where affine function and non-linear
20

---

function alternated. We tested log, tanh, exp. ELU as non-linear functions. For example. we experimented with the flow used in [5]:
T = affine exp oaffine ELU o affine o ELU affine o log oaffine
However, all those models either collapsed or could not improve the performance of the first tested simple flow.
Tab. 14 shows, that a model with simple flow indeed improved the average log-likelihood compared base model, which was trained without flow (7 identity) using Gaussian base distribution p(yl x. = Ny; p12, 2), I) for non-zero responses. However, the default latent space, which models ZIG distributions, has still the highest log-likelihood of -0.74 bits per time and neuron. When the encoder sees half of the neurons, the ZIG model slightly outperformed the Gaussian model with and without flow terms of correlation (Tab. 14
Table 14: Comparison between default latent ZIG model, Gaussian latent model and gaussian
latent model with flow
ZIG Model
Gaussian Model
Flow Gaussian
Model
Log- log-likklihood in Bits per Neuron and Time
-0.88
-2.1
-1.2
Conditioning on half of neurons
0.229
0.219
0.211
I Comparison with a Vision Transformer Core
To test the transferability of our latent method, we repeated the main comparison using Vision
Table 15: Performance comparison using Vision
Transformer (ViT) core.
Transformer (ViT) core following Li et al. (28). The trends are qualitatively the same as with the 3D CNN core: the ZIG model with the ViT core performs slightly worse than the Poisson base- line, consistent with the optimization trade-off between modeling full distributions and condi- tional means discussed in the main text. In con- trast, the latent model improves the performance beyond the Poisson baseline, demonstrating that our approach generalizes across different core architectures.
Poisson
Video-only
Latent
Baseline
ZIG
ZIG
Correlation 0.17 (0.007) 0.155 (0.099) 0.153 (0.007)
Log-
Likelihood 1 -1.2 (0.09) -0.45 (0.06)
While we did not obtain state-of- tf-the-rrttt perfor- mance with ViT-likel due limited hyper- parameter tuning-the the results indicate that the latent method we propose is readily transferable to other cores.
Table 16: Predictive performance of Poisson
Baseline model, a video-only ZIG-model with be- havior and latent model without behavior.
Additionally, conditioning on half of the neurons further boosts the correlation up to 191 (0.006), showing that encoding neuron responses also works effectively with the ViT core.
Poisson Baseline
Video-only ZIG
Latent
Conditioned Correlation Log- Likelihood 1
0.20
0.19
0.27
-0.94
0.18
Models with Behavior
To assess whether measured behavior could substitute for our inferred latent state, we augmented both the Poisson baseline and the video-only ZIG model with the two measured behavioral variables: treadmill speed and pupil dilation. These behavioral variables were concatenated as additional input channels into each model's core as 43 47 As reported in Tab. [6] the latent state model (with latent dimension &- = 150, and conditioned on half of the neurons held out from evaluation) surpasses both eehavior-augmente not included for the latent model.
models in terms of correlation and log-likelihood. Behavior was
21

---

K Heat Maps of Response Distributions
As reported in the main paper, a high-dimensional latent state yields higher log-likelihood. We explained this effect by better "coverage" of the response space with probability mass. Here, visualize this effect. One can see that a high-dimensional latent model is more likely to put some probability mass broader range of response values and does not concentrate all the probability mass around the mean (Fig. [ Thus, extreme response values, as in column of Fig. & are covered with non-zero probability. A low-dimensional model predicts them with almost probability (parts of the response traces are in the gray areas of the heatmap).
Neuron
Neuron 2
Neuron 3
8 6 2 Log-likelihood
100
-10
G 4 I o Log-likellhood
10
Figure 8: Neuron response traces and model predictions for one video recording. The black line represents the actual response of each neuron over time, while the cyan line represents the model's predicted response, which is the mean of the predicted ZIG distribution. For each time point, the predicted response ZIG-distribution is visualized as a heat map along the y-axis. The top row displays predictions from model with latent dimension of k 100, while the bottom shows predictions from a model with a latent dimension of & = 10.
L Behavior Analysis
For the pupil dilation and the treadmill speed of each mouse, we performed one CCA analysis each. The CCA analysis was done with 5-fold cross-validation. We split the recording time with an 80/20 ratio and fit the CCA weights on the train split. The correlations were computed between the CCA combination Σ, w
(cca) (1) and the behavioural variable (pupil dilation or treadmill speed) on the test
split. w, (coa) are the CCA weights and (1)) ...., (a) E RTT are the latent variables. For each mouse, we computed the average correlation across the cross-validation. For two selected videos and mice, we plotted their normalized pupil dilation and treadmill speed against the corresponding normalized CCA combination of the latent over the whole video time, which corresponds to ~ 300 time points (Fig. 9
Table 17: CEBRA hyperparameter search (Optuna) and se- lected setting used for all mice.
CEBRA baseline explained. To com- pare how good our models is we used CEBRA (36 as a baseline. CEBRA performs dimensionality reduction on neural activity using InfoNCE con- trastive learning, where positive and negative pairs are defined by auxil- iary variables such as time behavior.
Hyperparameter
Search range
Selected value
Time offset Batch size Learning rate
5 5-20 (integer) {32, 64, 128, 256. 512. 1024)
6
64
[10-4, 10-2] (log-uniform)
0.003
Latent dimension
2-32 (integer)
24
22

---

When the auxiliary variable is discrete, for example left or right wheel turn, it selects positives uniformly from all samples with the same label. When the variable is continuous, such as running speed or pupil direction, it chooses random point within time window around the sample and then find the closest match in the dataset using either Euclidean or cosine distance; this sample becomes the positive pair, which adds diversity and prevents repeatedly selecting the same example. Negative pairs are sampled randomly. For decoding, CEBRA encode neural responses, find the nearest latent vectors for responses in the training set, and returns their associated behavioral variables as predictions.
CEBRA hy perparameetess tuning, For each mouse, we fitted one CEBRA model independently and computed -ehavi-b correlations using the same CCA protocol and the same train/validation split as for our latents. We tuned CEBRA's hyperparameters with Bayesian optimization (Optuna), exploring batch size, latent dimensionality, learning rate, and time offset, and selected the setting that maximized the average validation behavior correlation. The search ranges and the selected values (used for all mice) are summarized in Tab. 17
Nomalbed Bar
Aupi Belwwer COO Lateers
Fasi Beeavaa CCA ef Laneets
Time Tramnss
Time Thammnn
Norollass BchoHortutte
Spooid bohowke
Normall Bchatlor-ater
Spoed Serwwtor
CCA Latarts
210
Time Frames
Time Frames
Figure 9: Normalized behavior and CCA combination latent factors during two selected videos
M Additional Brain Maps
In Fig. 5 we showed only the first three singular dimensions of the latent feature matrix w(1), In rows and of Fig. 10 we compare those same three dimensions for both w() and w(o), They exhibit largely the same spatial patterns, aside from sign flips in columns I and 3. Beyond the first three dimensions, we did not observe any notable structure: Row 2-4 of Fig. displays singular dimensions one through nine for ("),, Beyond the fourth singular dimension, they do not appear to align with cortical neuron positions and seem randomly distributed across the xy-plane.
N Latent model performance on out-of-distributioo data
For the training of the latent model, we usually initialize the weights of the core, which encodes the video input, with the weights of pretrained video-only ZIG model, which doesn't 't have latent state. If we train the latent model from scratch, the model seems to over-rely on the information of the latent state, which encodes responses of half of the neurons' population during training. When this information is marginalized out "oNon-Conditioned)) ), the model performs poorly in terms of correlation Tab. 18 However, this can be prevented by loading a pretrained core ensuring that the latent state mostly encodes additional information, which is not contained in the output of the core Tab. 19 When conditioning on half of the neurons" responses, both training procedures yield Condiiioondd".
comparable results (Tab. Tab. 18 Tab.
23

---

1st SD
2nd SD
Figure 10: Row 2-4: Color gradient maps of the latent feature vectors w; (4) the first nine Singular Dimensions (SD) for mouse
Values for (
along
4. Row 1: The color gradient maps of the first three singular dimensions of the latent feature vectors w; are plotted. All recorded neurons are located within 600μmm x 600pmm square in the cortex. Their depth differs most 200 jam. The model was trained without any knowledge of cortical positions and freely learnable feature vectors.
1st SD
2nd
3rd SD
Values for W
4th SD
5th SD
6th SD
Values for W
7th SD
8th SD
600
9th SD
2
Values for W
1
0
-1
200
-
-3
200
400
600
Table 18: Latent model without ZIG pretraining, k: = 200
Table 19: Latent model with ZIG pretraining. k = 150
Non-Conditioned
Conditioned
Non Non-Conditioned
Conditioned
Mouse 6
0.016
0.170
Mouse
0.086
0.172
Mouse 7
0.029
0.196
Mouse
0.121
0.192
Mouse 8
0.022
0.211
Mouse
0.125
0.206
Mouse 9
0.026
0,209
Mouse
0.107
0.207
Mouse 10
0.015
0.184
Mouse 10
0.082
0.179
Broader impact
Accurate models of neural variability deepen our insight into how brains transform sensory informa- tion and may shed light on the disruptions underlying neurological disorders. Specifically, a more precise model that integrates internal brain states, stimulus-driven activity, and anatomical structures such as retinotopy or memberships to certain brain areas could uncover deeper insights into the cortex's computational principles. Although training on existing. broadly applicable datasets to maximize the scientific yield each experiment. Moreover, approaches like the model presented here help to reduce the number of animal experiments by enabling rescarchers to explore functional principles of brain processes with faithful models in silico.
models still depends animal data, we rely
24

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract mentions all core contributions, and in the introduction, our core contributions are listed in more detail.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: In the Discussion section, we provided a paragraph to discuss the limitations of our work.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model cell-speciiiaatinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g. if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
25

---

Answer:
[NA]
Justification:
We do not make any theoretical claims that require a formal proof.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: Our model's complete architecture is specified in Section 3 and all training
and optimization hyperparameters are exhaustively listed in Appendix C dataset evaluate is public 43
The SENSORIUM
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
26

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: The Sensorium data is publicly available. Our code is added in the supplemen- tary material and will be made public upon publication.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nnis.cc.
mublic//uudees/Codeebbbiisg
www.:sswbbssssoooooooooooo for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (httpa
/ hwtp/u/lic/gguudee/eeeeeee/c/eee
ass asionPlicyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: All relevant hyperparameter choices are listed in Further, for each experi- ment, we listed all necessary details for reproducibility to the best of our knowledge.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Wherever possible, we reported error bars to make sure that our results are statistically significant.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emror bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
27

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: All experiments be run single A100 GPU with 80GB memory as stated in
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't fidn't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics httpp/////eerrpps.ccpcpppbb////a
Answer: [Yes]
Justification: We carefully checked that our research in the paper is conform with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We addressed the potential social impacts in our Broader impact section (App- O).
Guidelines:
28

---

The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negatíve societal impacts include potential malicious or unintended uses (e.g.. disinformation, generating fake profiles, surveillance), fairness considerations (c.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: To the best of our knowledge, our model and the SENSORIUM dataset do not present significant misuse risks, no additional release safeguards required.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: All creators and owners of original assets are properly cited and mentioned in the paper. Further, made that don't 't hurt any licenses in our paper.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
29

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, sxperswithcode com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re- re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
There are no new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
Neither Crowdsourcing nor research with human subjects were done in our
paper.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
30

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: LLM's were not included in the core method development of our research.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy httepp///eeerrpeeccc//ooeeerrrre for what should or should not be described.
31


Reviews:
[{"review_id": "GHQ96Xv7Nb", "reviewer": "Reviewer_FxVu", "summary": "This paper presents a probabilistic model for predicting neural activity in the mouse primary visual cortex (V1) in response to video stimuli. The model aims to account for both the stimulus and unobserved internal brain states by incorporating a latent variable inferred from a portion of the recorded neural population. The authors evaluate this model on a public dataset, reporting improvements in log-likelihood over a video-only model. The study also analyzes the learned latent variables, showing they correlate with the mouse's behavior and have a spatial organization related to the cortical layout of the neurons.", "questions": "1. The justification for using a Zero-Inflated Gamma (ZIG) distribution over a Poisson one is clear, given that ZIG is better suited for the continuous and sparse nature of calcium imaging data. However, the choice of the Gamma component itself is worth probing. Have you considered other heavy-tailed distributions to model the positive responses? Do you have any intuition on whether this might help close the approximation gap?\n\n2. Here is a suggestion for a baseline that could further contextualize your model's performance. Instead of a direct ZIG observation model, an alternative would be a more biophysically-inspired model. One could first predict underlying discrete “spikes” with a Poisson model and then apply a deterministic temporal convolution to model the slow dynamics of the calcium transients. A comparison against such a benchmark would be very interesting to see and would significantly strengthen the paper's claims about the effectiveness of your end-to-end framework.", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "## Strengths:\n\n **A clear contribution:** \n\nThe paper addresses the important problem of modeling stimulus-independent variability in neuroscience. The proposed model, which combines a dynamic video encoder with a latent variable framework for predicting full response distributions, is a logical step forward in the field. \n\n**Sufficient experimental validation:** \n\nThe authors perform a number of experiments to validate their approach. They compare against reasonable baselines, test on out-of-distribution stimuli, and perform control analyses to investigate the properties of the learned latent variables.\n\n\n**Biologically relevant findings:** \n\nA key result is that the latent variables, learned without supervision from behavior or location data, correlate with external variables like mouse behavior and cortical topography. This suggests the model is capturing meaningful structure in the neural data. \n\n\n## Weaknesses:\n\n**1. Performance on key metrics:** \n\nA notable weakness is that the proposed model performs worse on the single-trial correlation metric compared to a simpler Poisson baseline. While the authors report an improvement in log-likelihood, the trade-off with the correlation metric, which is a standard measure of predictive accuracy in this domain, is a significant concern. \n\n**2. Model and training complexity:** \n\nThe latent variable model introduces considerable complexity over the baselines. The appendices show that the best performance is achieved only when initializing from a pre-trained model, which suggests that training from scratch is challenging and may lead to poor solutions. This could be a practical barrier to adoption. \n\n**3. Purity of latent states:** \n\nThe central claim is that the latents capture stimulus-independent variability. However, the authors' own analysis shows that for one of the five mice, the latents contain residual stimulus information. This finding, while reported transparently, somewhat undermines the model's core premise.", "was_revised": true, "final_justification": "The paper introduces a relatively novel probabilistic latent variable model for predicting neural activity. It is technically sound, and the experiments support the claims.", "rebuttal": "Thank you for your review, and for viewing our work to have “clear contribution” and  “sufficient experimental validation” and providing valuable feedback. \n\nHere we address the weaknesses and questions raised during the review.\n\nAddressing raised weaknesses:\n\n**Performance on key metrics**: We acknowledge that the ZIG model slightly underperforms the Poisson baseline in lines 240-245 of our main text (L240--245) due to the optimization trade-off, since ZIG is not an exponential family distribution (see [1]), and maximizing its likelihood does not automatically maximize its correlation since mean is not a sufficient statistic of the ZIG distribution [1].  \nHowever, our primary goal was not to outperform baselines on point-estimate predictions (correlation) but to more accurately model the entire neuronal response distribution (log-likelihood) along with latent factors, which are known to be a strong signal in neuronal populations [2]. Models that are solely based on sensory input cannot capture these noise correlations.\nWe chose ZIG as a more biologically fitting distribution for deconvolved calcium trace signals, which often include many zero or near-zero values when neurons are inactive, along with positively non-integer skewed values when activity occurs, and our choice also agrees with the literature [3, 4]. Compared to Poisson, which is a discrete distribution, for ZIG, we can also compute the exact likelihood. We chose ZIG for these reasons.\nWhile we report the correlation to facilitate comparison with other models, we want to highlight that likelihood is a complete metric as it captures the full stimulus-conditioned distribution rather than correlation, which just captures the conditional mean. Thus, we believe likelihood aligns better with our aim to model both variability and higher moments of neural activity. \n\n\n\n**Model complexity**: We agree that our approach involves a two step procedure—training a standard video-to-neuron model before training the latent module--instead of training everything from scratch, as we show that we get better results this way (reported in Appendix L). However, we do not think that this is necessarily a weakness since using a pretrained backbone is now standard practice in many modern machine learning pipelines, models consisting of multiple separate components are seldom trained fully from scratch. Additionally, we view our two step approach to also provide us with modularity: one can simply use an existing pre-trained video-neuron encoder and fine-tune with the latent module on top, further promoting practical adoption.\n\n**Purity of latent states**: Our claim about the latent separation stems from the independence structure of the latent variables that follows a factor analysis model (Fig. 2C), where latents are a priori independent of the stimulus – giving a bias toward stimulus independence. Our objective function (ELBO), $E_z [\\log p(y \\mid x, z)] - \\mathrm{KL}\\bigl[q(z \\mid y) | \\mathcal{N}(0,1)\\bigr]$, encourages this via the KL term that tells $z$ to remain isotropic unless it helps predict the responses. In the analysis you are referring to, we sample from the approximate posterior $q(z \\mid y)$. Due to the collider structure $x \\to y \\leftarrow z$, the posterior $q(z \\mid y)$ can still carry stimulus information. Our control experiment checks whether these samples encode stimulus signals, and we will make this subtlety clearer.\n\nAddressing raised questions:\n\n**Q1**: Thank you for this suggestion—we had very similar thoughts and in fact ran a series of follow‑up experiments in which we replaced the Gamma component of the ZIG distribution by a more flexible normalizing‑flow-based distribution, while keeping everything else fixed (latent-state and stimulus dependence). The normalizing flow consists of a Gaussian base distribution, followed by a series of invertible transforms, $T$. To capture the peak at zero, the responses below a threshold $\\rho$ are still modeled by a uniform distribution as in our original ZIG setup. For each neuron, the flow has learnable parameters. Thus, for responses $y_i>\\rho$ of neuron $i$,  we apply the flow $T_i$:  $ p(T_i(y_i)\\mid z,x)= \\mathcal{N} \\left( \\mu(x, z), 1 \\right) $, where $z$ denotes the latent variable, $x$ the video and $\\mu(x, z)$ is the predicted mean of the Gaussian.\nThe actual response likelihood follows from the change--of--variables rule\n$ p(y_i \\mid z,x)= p\\bigl(T_i(y_i)\\mid z,x\\bigr) \\cdot \\bigl|\\det\\nabla_{y_i}T_i(y_i)\\bigr|. $\n\nWe also tested replacing the Gaussian base distribution with a Gamma positive‑tail, and a different number of flow layers, ranging from 2-9. The flow layers alternated affine with non‑linear functions, including {log⁡,exp⁡,tanh⁡,softplus,ELU}. Our best combination, a two‑layer $\\text{affine}\\rightarrow \\text{softplus}^{-1}$ flow with a Gaussian base distribution, could not improve upon our ZIG model (see below). \n\nThus, we concluded that the ZIG distribution is already a pretty good fit to the neuron responses. Further, we concluded that it is tricky to close the approximation gap- the trade-off between correlation and likelihood- that arises for non-exponential distributions [1].\n\n|   | ZIG Model    | Flow Model      |\n|-|-|-\n| Conditioned Correlation | 0.23     | 0.22   |\n| Log-likelihood | -0.98    | -1.2   |\n\nWe will integrate these experimental results as part of the appendix.\n\n\n**Q2**:  Thank you for the interesting suggestion. We agree that a model producing discrete spikes and followed by convolutional calcium dynamics would mimic the biological process and data collection better and is scientifically interesting. We tested this briefly in the past. However, because we could not make it work well, we abandoned this idea. \n\n\nWe hope these comments make our goals and choices clearer and help address some of the concerns raised.\n\n#### References:\n[1] Konstantin-Klemens Lurz, Mohammad Bashiri, Edgar Y. Walker, and Fabian H. Sinz. (2023) Bayesian oracle for bounding information gain in neural encoding models. In The Eleventh International Conference on Learning Representations.\n\n[2] Stringer, C., Pachitariu, M., Steinmetz, N., Reddy, C. B., Carandini, M., & Harris, K. D. (2019). Spontaneous behaviors drive multidimensional, brain‑wide population activity. Science, 364(6437), 255. https://doi.org/10.1126/science.aav7893\n\n[3] Zhou, Ding, and Xue-Xin Wei. \"Learning identifiable and interpretable latent models of high-dimensional neural activity using pi-VAE.\" Advances in neural information processing systems 33 (2020): 7234-7247.\n\n[4] Zhu, F., Grier, H. A., Tandon, R., Cai, C., Agarwal, A., Giovannucci, A., ... & Pandarinath, C. (2022). A deep learning framework for inference of single-trial neural population dynamics from calcium imaging with subframe temporal resolution. Nature neuroscience, 25(12), 1724-1734."}, {"review_id": "GyN77gfQlf", "reviewer": "Reviewer_qQZJ", "summary": "The paper presents a probabilistic latent-variable framework that models neural data to videos by combining stimulus-driven features with a low-dimensional, stimulus-independent brain state, addressing variability that cannot be explained by the visual input alone. Building on a 3D CNN core and a Gaussian readout, the model predicts a Zero-Inflated-Gamma (ZIG) response distributions whose parameters are jointly modulated by the video and the inferred latent variables. Compared with Poisson or video-only ZIG baselines, it improves log-likelihood and single-trial correlation. These results demonstrate that unsupervised inference of latent variables can enhance predictive accuracy and discover biologically meaningful structure without requiring behavioural supervision.", "questions": "1. This paper compared a video-only ZIG-model without latent to disentangle the impact of the ZIG. Could you provide a latent-Poisson baseline using the same video core? One suggestion is to keep the CNN core and readout but predict Poisson rates, then compare likelihood and correlation with the latent-ZIG model. This will help us understand whether the observed performance gains come from the ZIG likelihood itself or from the latent design.", "limitations": "The authors have discussed the limitations in Section 5.", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "**Strengths**:\n\n1. The authors proposed a 3D video encoder with a Zero-Inflated-Gamma (ZIG) likelihood whose parameters are jointly shaped by stimulus features and a latent state. This combination goes beyond the common Poisson-mean setting, making the model capture both firing-rate modulations and trial-to-trial variability in a mathematical way. \n\n**Weaknesses**:\n\n1. A minor concern is that this paper spends several dense paragraphs summarizing prior methods, where many only loosely connected to the proposed latent-ZIG model, which distracts from the main ideas and makes the paper feel unfocused.  A brief overview in the main text with further details moved to the supplementary material would improve clarity and readability.", "was_revised": true, "final_justification": "The authors' rebuttal adequately addresses my concerns.", "rebuttal": "We appreciate your review and constructive feedback. We believe we can address the raised weakness and question effectively. Specifically:\n\nAddressing raised weakness:\n\n**Clarity of related works section**: In the final manuscript, we will refine the related works section to more clearly position our contributions within the broader literature. Briefly, our method builds on core-readout models to predict dynamic neural activity to video stimuli, but introduces a probabilistic formulation that also includes learning of latent factors underlying neural activity. Existing probabilistic or latent-state models typically ignore the stimulus altogether, while stimulus-driven models are either deterministic or overlook latent states. We aim to clarify this landscape and clearly situate our work at the intersection of these approaches.\n\n\nAddressing raised questions: \n\n**Q1**: Thanks for raising this point. In fact, we had the same question. Therefore, this experiment is already in Appendix G. \nWe keep the 3D CNN and Gaussian read‑out unchanged and train the latent model with a Poisson loss. As we cannot directly compute likelihood on a discrete Poisson distribution for continuous signals, we insert the means of the predicted ZIG distributions into the Poisson loss. For each neuron $i$ and timepoint $t$, our ZIG model predicts the ZIG-parameters $q_{it}(\\mathbf{x};\\psi)$ and $\\theta_{it}(\\mathbf{x};\\psi)$ (as in Figure 2A) based on the video input $\\mathbf{x}$ and its parameters $\\psi$. With these, we analytically compute the mean of the ZIG distribution $\\hat{r_{it}}\\bigl((q_{it}(\\mathbf{x};\\psi),\\theta_{it}(\\mathbf{x};\\psi)\\bigr)$. During training, we then use Poisson Loss between those predicted responses $\\hat{r_{it}}$ and the observed neural responses $r_{it}$ as an objective. Basically, we parametrize a mean firing rate (needed for Poisson) via a ZIG distribution.\nIn addition to Appendix G below, we report both correlation and likelihood, showing that a latent model trained with Poisson loss has a worse likelihood performance and a similar correlation performance compared to the ZIG latent model. Moreover, adding a latent improves the likelihood compared to the video-only baselines, regardless of the used loss function.\n|    | Video-only ZIG    | Video-only Poisson      | Latent ZIG    | Latent Poisson     |\n|-|-|-|-|-\n| Correlation  | 0.183     | 0.195 | 0.183     | 0.184   |\n| Log-likelihood  | -0.98     | -1.61 | -0.30    | -1.39   |\n\n\nWe hope that these comments helped to understand our paper a bit better, and we will improve the writing based on your feedback."}, {"review_id": "xjaXH29jjF", "reviewer": "Reviewer_WMJZ", "summary": "This submission introduces a probabilistic latent-variable model that predicts the full distribution of mouse V1 calcium responses using a video-encoding core and an inference model capturing a low-dimensional, stimulus-independent brain state. Compared with video-only baselines, the model (i) raises test log-likelihood, (ii) generalises better to OOD inputs when conditioned on partial population activity; (iii) the learned latents correlate with behaviour and reveal a cortical topography without supervision.", "questions": "1. Have you tried replacing the 3D CNN core with a recent ViT-based core such as V1T? Even a small-scale experiment would show whether the latent state provides performance gains, which would be quite important, in my opinion. \n\n2. Could you please add standard errors or statistical significance results where they are missing?\n\n3. On the significance of the paper: could the authors please clarify whether the inference of latent variables is performed conditioned on the neuronal responses given the same stimuli and from the same timesteps? If so, I struggle to justify the significance, or rather the claimed improvement, in the model's ability to better predict the responses of other neurons. If one assumes that a subset of neurons at timestep $t$ can inform the prediction of (or can contain information about) other neurons, then it is only natural that a model that conditions its prediction on a subset of neurons can theoretically perform better than a model that does not. As such: Is a comparison to models that do not condition on the subsets of neurons justified / fair / meaningful? I would be happy to hear your clarifications on this matter.", "limitations": "Yes", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "1. Quality: \n- (+) The paper is technically sound with a well-formulated probabilistic model. \n- (-) It would be important to include errors in Table 4. The claims cannot be fully substantiated without them.\n\n2. Clarity:\n- (+) Clearly written and well-organised\n- (-) Minor typos (e.g. “non-condititoned”, “percise”). \n- (-) Readers unfamiliar with ZIG would definitely benefit from a short paragraph before Eq. 1.\n\n3. Significance:\n- (+) First probabilistic model that jointly handles natural-movie input and models latent brain state for neuronal response prediction, filling a gap in dynamic encoding models;\n- (+) The latent factors capture behaviour and cortical topography without supervision;\n- (-) Comparisons stop at the Sensorium CNN/GRU baseline; the authors acknowledge more recent transformer cores [3, 27] but do not test on them, so the incremental benefit on modern architectures is unknown.\n- (-) Impact seems questionable due to the modest gain in correlation against baselines and due to the fact that conditioning requires simultaneous recording of many neurons. This would limit immediate applicability to smaller-scale datasets.\n\t    \n4. Originality:\n- (+) Novel combination of (a) ZIG, (b) latent-state inference from a subset of neurons, and (c) dynamic video core. Prior latent models either ignore stimulus or are image-based; the present work unifies both for video. The discovery of topographic maps in the weights that connect latents to neurons is interesting.\n- Methodologically the pieces (variational auto-encoding, ZIG, 3D CNN core) are known, but the novelty lies in their integration.", "was_revised": true, "final_justification": "The authors have addressed my questions with new experiments and further clarifications. \n\nI am keeping my score at borderline accept, as I do not believe I have the necessary knowledge of the related work to be confident in my assessment. \n\nNevertheless, having spent some time on this paper, I do not have any reason to consider rejection.", "rebuttal": "Thank you for your positive review, referring to our paper as “technically sound with a well-formulated probabilistic model” and “clearly written and well-organised”. We also appreciate your constructive feedback.\nWe believe we can address all your concerns and questions:\n\nAddressing raised weaknesses:\n\n**W1**: Error bars on Table 4:\nThanks for pointing this out. We ran the experiments again with models initialized on 2 more seeds and reported the standard error of mean along 3 seeds with the slightly changed results of Table 4 below:\n|    | Freely learned    | Position based     | Same    | \n|-|-|-|-\n| Log-Likelihood  | -0.32 (0.05)     | -0.70 (0.04)| -0.88  (0.04)  \n| Conditioned Correlation | 0.24 (0.003)    | 0.22 (0.005) | 0.18 (0.003)\n\n**W2** Introducing ZIG: Thanks for your suggestion. We would incorporate an intuitive introduction like this in the text: “In calcium imaging of neurons, the measured fluorescence signals often include many zero or near-zero values when neurons are inactive, along with positively skewed values when activity occurs. A zero-inflated gamma distribution models this by combining a point mass at zero (to capture silent periods) with a gamma distribution (to represent the continuous, positive-valued calcium activity during neural firing). This allows for more accurate modeling of the sparse and bursty nature of neural activity.”\n\n**W3** Comparisons with transformer cores: See **Q1**\n\n**W4** Limited impact:  We would like to address your point regarding limited impact. First, advances in recording techniques have made large-scale datasets increasingly common, with several open-access resources now available, including MiCRONS [1], Sensorium 2023 [2], and the Allen Brain Observatory [3]. Deep learning based predictive models, shown to be superior in predictive performance, typically rely on such datasets [4], and hence we do not see data scale as a limiting factor for our approach—especially since the data we use is publicly available. Regarding the applicability to smaller-scale datasets, we believe our work can serve as a strong pretrained initialization and could adapt to smaller-scale datasets via fine-tuning.\nSecond, our primary goal was not to outperform baselines on point-estimate metrics like correlation, but to more accurately model the full neuronal response distribution (log-likelihood), while additionally capturing the influence of unobserved latent states alongside stimulus-driven activity—a well-known strong signal in large-scale populations  [5].\nLastly, there is evidence that the underlying latent state is effectively high-dimensional [5].  Therefore, it cannot easily be estimated using behavioural variables as proxy. Other existing methods ([3, 35, 40, 55, 56] from the original manuscript) either ignored the stimuli or were not probabilistic and thus ignored the noise correlations, both limiting the extraction and interpretability of the latent state vectors.\n\nSince we show that our learned latents states correlate with behavior, this allows us to design realistic in-silico experiments, supporting in-vivo causal testing. We hence believe our model trained on a large-scale dataset has a direct experimental impact. \n\n**W4** Typos:  Thanks for catching the typos! We will thoroughly go through the manuscript again.\n\nAddressing raised questions:\n\n**Q1**:\nBelow, we show that the trends are qualitatively the same - e.g., the ZIG model with the ViT core is a bit worse than Poisson due to the optimization trade-off discussed in lines 199-204,  and then the latent model actually improves the performance above the Poisson baseline. \nWhile we did not obtain the state-of-the-art performance using ViT during rebuttal---very likely attributable to limited hyperparameter tuning---we hope that this experiment is sufficient to show that the latent method we introduced is transferable towards other cores.\nBelow, we rerun the Table 1 comparison, this time replacing the 3‑D CNN with a ViT backbone for every model.\n|    | Poisson baseline   | Video-only ZIG      | Latent ZIG    |\n|-|-|-|-\n| Correlation  | 0.17 (0.007)    | 0.155 (0.009) | 0.153  (0.007)   | \n| Log-likelihood  | -     | -1.2 (0.09) | -0.45 (0.06)   | \n\nAdditionally, Conditioning on half the neurons further boosts correlation up to 0.194 (0.006), showing that encoding neuron responses works with a different core, too.\n\n**Q2**: See W1\n\n**Q3**: Yes, you are right, in the ‘Conditioned’ scenario, we give a subset of responses from the same time and movie frame as input for the model. We fully agree that it would be unfair to compare such a model to a baseline that lacks this extra information. We reported this experiment to show that our model additionally **allows** us to condition on other neurons and baselines that predict neural activity to stimuli currently cannot. \nThus, our main experimental results in Table 1 report the performance of all models without conditioning on any response. See lines 179-180 for evaluation details of this “Non-Conditioned” scenario. In the ‘Non‑Conditioned’ scenario, we still report Pearson correlation, the field’s standard metric, and our video‑only ZIG model’s correlation remains close to the Poisson baseline, while also allowing exact log probability calculation (scoring the full-response distribution) and how including the learned latents improves the log probability. \n\n\nRegarding existing works that allow conditioning on other neurons, to the best of our knowledge, there are currently no public baselines that support this. The closest is Neuroformer [6], but its original implementation does not support multisession training. Scaling it to a dataset like Sensorium 2023 would require substantial modifications—such as adjusting context window sizes and architectural changes—making it unsuitable as an out-of-the-box baseline.\n\nHowever, as discussed in W4, our primary goal and the significance of the paper is not to outperform baselines on correlations but to create an accurate and biologically plausible model for the entire neuronal response distribution (log-likelihood) and the latent state. Such models could enable more previously unavailable analyses, such as the cortical topography analysis from Figure 5. \n\nWe hope these comments shed some light on our motivation, technical, and design choices. We also hope that it will help the reviewer reconsider some criticism.\n\nReferences:\n[1] Zhou, P., Reimer, J., Zhou, D., Pasarkar, A., Kinsella, I., Froudarakis, E., Yatsenko, D. V., Fahey, P. G., Bodor, A., Buchanan, J., Bumbarger, D., Mahalingam, G., Torres, R., Dorkenwald, S., Ih, D., Lee, K., Lu, R., Macrina, T., Wu, J., da Costa, N., Reid, R. C., Tolias, A. S., & Paninski, L. (2020, March 25). EASE: EM‑assisted source extraction from calcium imaging data. bioRxiv. https://doi.org/10.1101/2020.03.25.007468 \n\n[2]  Turishcheva, P., Fahey, P. G., Hansel, L., Froebe, R., Ponder, K., Vystrčilová, M., Willeke, K. F., Bashiri, M., Willeke, K. F., Bashiri, M. P., Willeke, K. F., Ecker, A. S., Sinz, F. H., Ding, Z., Tolias, A. S., & Ecker, A. S. (2024, July 12). The Dynamic Sensorium competition for predicting large‑scale mouse visual cortex activity from videos. arXiv. https://doi.org/10.48550/arXiv.2305.19654\n \n[3] de Vries, S. E. J., Lecoq, J. A., Buice, M. A., Groblewski, P. A., Ocker, G. K., Oliver, M., … Koch, C. (2020). A large‑scale standardized physiological survey reveals functional organization of the mouse visual cortex. Nature Neuroscience, 23(1), 138–151. https://doi.org/10.1038/s41593-019-0550-5\n\n[4] Lurz, K.‑K., Bashiri, M., Willeke, K. F., Jagadish, A., Wang, E., Walker, E. Y., Cadena, S. A., Muhammad, T., Cobos, E., Tolias, A. S., Ecker, A. S., & Sinz, F. H. (2021, January 12). Generalization in data‑driven models of primary visual cortex, ICLR 2021\n\n[5] Stringer, C., Pachitariu, M., Steinmetz, N., Reddy, C. B., Carandini, M., & Harris, K. D. (2019). Spontaneous behaviors drive multidimensional, brain‑wide population activity. Science, 364(6437), 255. https://doi.org/10.1126/science.aav7893\n\n[6] Antoniades, A., Yu, Y., Canzano, J., Wang, W., & Smith, S. L. (2023). Neuroformer: Multimodal and multitask generative pretraining for brain data. arXiv preprint arXiv:2311.00136."}, {"review_id": "9VFKdhmA3Q", "reviewer": "Reviewer_cFTH", "summary": "This paper proposes a novel algorithm for predicting neural responses to dynamic naturalistic stimuli while modeling stimulus-independent variability in the neuronal population (internal brain states). \n\nMost previous approaches tend to either predict neural activity from stimuli without modeling a latent state or focus on modeling a latent state from neural responses. Some previous work predicts neural activity based on sensory and neuronal input. However, the only previous work combining visual input with a latent state is restricted to static stimuli. \n\nTo bridge this gap, the authors propose a model to predict neural activity combining video visual input with a latent state. Technically, it consists of a 3D convolutional neural network “core” that processes the video frames and a latent variable that could be conditioned upon neural activity and captures stimulus-independent fluctuations. The response of each neuron is modeled with a Zero-Inflated Gamma (ZIG) distribution whose parameters depend on both the video-driven features and the latent state. ZIG is chosen because it aligns better with calcium imaging data.\n\nThe model is evaluated on neural recordings from mouse primary visual cortex (V1) in response to grayscale natural video stimuli (from the SENSORIUM dataset). The proposed video+latent model improves the log-likelihood of the neural responses over baselines and also achieves comparable Pearson correlation. In addition, the proposed method seems to handle out-of-distribution stimuli better. \n\nMore importantly, the latent states highly correlate with behavioral variables (pupil dilation and locomotion speed), although they are not used in training. This suggests the latent state successfully captured intrinsic neural dynamics. The latent states also seem to exhibit topographic organization, although the cortical positions of the neurons are not used in the training. Those conclusions show that the latent features uncovered by the model have a meaningful biological basis.", "questions": "1. Regarding weakness #2: Would you be able to evaluate the quality of the latent factors against a baseline? I am impressed that they have a solid biological basis, but I am wondering if they are somehow better than a baseline because they are learned together in a neural activity prediction task.\n2. Mouse v1 neurons should encode other features such as orientation. Your analysis in table 3 shows that the latents encode contrast minimally. Do you think it'd be meaningful to extend your analysis to other features?\n3. For the neurons that are conditioned upon - how are they chosen? I would assume they are chosen randomly from reading the paper. Does using a different subset of neurons matter? Does restricting the choice to purely layer 2/3/4/5 matter? I am wondering if the different choice here can lead to \"better\" latent states or higher accuracy.", "limitations": "yes", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. The paper proposes a novel integration of dynamic visual stimuli and latent brain states in order to predict neural activity, addressing a gap in the literature.\n2. The latent states (learned in an unsupervised fashion during training) have a meaningful biological basis. They highly correlate with behavioral variables, and they reflect an anatomical organization.\n3. The model handles OOD samples better\n3. The paper conducted a thorough experiment and analysis\n\nWeaknesses:\n1. The proposed model (more complex than the baseline) does not substantially improve (and in fact slightly underperforms) the baseline in terms of predicting the mean neural response as measured by correlation. The authors have provided an explanation for this, but I am not sure if the method's added complexity and parameters.\n2. To remedy the above weakness, maybe the authors can show that the latent states (the other important part learned in the model) are somehow \"better\" than a baseline. But such a baseline is missing from the paper.", "was_revised": true, "final_justification": "I appreciate that the authors have conducted additional experiments to address my concerns and curiosity. Overall, this is a technically solid paper, and the results are interesting; I believe this is a meaningful contribution to the research community.", "rebuttal": "Thank you for your positive review, highlighting the novelty of our work and our “thorough experiments and analysis”.\n \nBelow we address your questions, and we will incorporate these replies into our manuscript:\n\n\n**Q1**:  You raise a valid point about baselines for the latent factors. The most relevant and commonly used \"baseline\" for internal brain states are observable behavioral variables, and we performed this exact comparison. However, to address your question, we compared it with CEBRA  (Schneider et al., 2023). Please note, that CEBRA does not take visual stimuli as input or have held-out neurons. We trained CEBRA, which has the same latent dimension ($k=12$) as our model, on our data by fitting for each mouse one CEBRA model. Then, we performed the same canonical correlation analysis (CCA) with the CEBRA latents as we did for our latents (lines 322-327). “  Specifically, CCA “finds the linear combination of the latent variables $\\mathbf{z_{(1,\\dots,T)}}^{(1)},\\dots,\\mathbf{z_{(1,\\dots,T)}}^{(k)}$ with maximal correlation to a chosen behavioral variable like pupil dilation or treadmill speed.”  The CCA analysis was done with 5-fold cross-validation in an 80/20 split.\nBelow, we can see the canonical correlation with pupil dilation and treadmill speed on the test set. The error bars indicate the standard error of the mean of cross-validation. Especially for treadmill speed, the latents of our model do correlate stronger with behavior compared to the CEBRA latents, probably because explicitly separating stimulus information actually helps.     \t\n\t\n|  Canonical Correlation with pupil dilation  | CEBRA    | ZIG latent     \n|-|-|-\n|Mouse 1  |   0.65 (0.003)  | 0.66 (0.007) |\n|Mouse 2  |   0.48 (0.008)  | **0.59** (0.009) |\n|Mouse 3  |   0.52 (0.007)  |**0.64** (0.03) |\n|Mouse 4  |   0.59 (0.004)  | 0.59 (0.003) |\n|Mouse 5  |   **0.85** (0.001)  | 0.60 (0.01) |\n|**Average**  |   0.62 (0.005)  | 0.62 (0.01) |\n\n\n|  Canonical Correlation with treadmill speed  | CEBRA    | ZIG latent     \n|-|-|-\n|Mouse 1  |   0.19 (0.003)  |**0.26** (0.02) |\n|Mouse 2  |   0.48 (0.008)  |**0.61** (0.02) |\n|Mouse 3  |   0.35 (0.003)  | **0.57** (0.005) |\n|Mouse 4  |   0.26 (0.01)  | **0.33** (0.03) |\n|Mouse 5  |   0.26 (0.007)  | **0.44** (0.007) |\n|**Average**  |   0.24 (0.005)  | **0.49** (0.02) |\n\n**Q2**: Yes, you are right that mouse V1 has orientation-selective neurons (as well as phase-selective neurons). However, this information should be encoded in the model's core and readout weights and not in the latents [1] as it is a function of the neuron that does not change from trial to trial, changing the response based on the stimuli.\nHowever, the key idea of analysis in Table 3 is to show that latents are mainly encoding stimuli independent information. This is expected from the model and we wanted to test against the case that the model uses the latent state to explain variability that it cannot explain with the video-driven part. In theory, it could also do that for orientation, but we’d expect this to be more complex. In addition, orientation is not as easily extracted from natural video as local contrast. We thus focused on contrast because it's a fundamental, low-level stimulus property that can be directly computed from the video frames. This made it a straightforward control that latents do not encode contrast (e.g., stimulus) information - the low R² values in Table 3 confirm this.\nTo further probe stimulus independence beyond contrast, we ran an additional test: we predicted the held‑out neurons for trial A while conditioning on the responses of trial B (same movie, different brain state). If latents mainly carry stimulus information, performance should be unaffected, as stimulus does not change; but if they carry brain-state‑specific information, accuracy should fall. We observed the decrease in performance (Fig. 3, lines 300‑315), confirming that the latents represent a video‑independent brain state.\n\n**Q3**: Yes, your assumption is correct, and they are chosen randomly. We’ll clarify this in our final version. To answer your question, we ran additional experiments where we masked neurons based on their cortical coordinates. To test how sensitive the model is to that choice, we ran three experiments: we sorted the neurons by their cortical x, y, or z coordinate and then masked the first 50 % of each ordered list. Masking by x or y (cortical surface) caused a small drop in performance, whereas masking by z (cortical depth) left accuracy unchanged relative to random masking. This mirrors Fig. 5: latent patterns show clear topographic organisation across the cortical surface but remain largely constant with depth.\nBelow are the results of the different masking strategies, we reported the standard error of the mean across the five mice.\n|    | Random    | x-coordinate      | y-coordinate     | z-coordinate      |\n|-|-|-|-|-\n|Conditioned Correlation  |   0.24 (0.002)  | 0.225 (0.003) | 0.22 (0.003)    |  0.24 (0.002)   |\n\nWe will incorporate this analysis into the appendix.\n\nWe hope that these additional comments and experiments will help you to understand our method better and revisit the evaluation.\n\nReferences:\n\n[1] Ustyuzhaninov, I., Burg, M. F., Cadena, S. A., Fu, J., Muhammad, T., Ponder, K., Froudarakis, E., Ding, Z., Bethge, M., Tolias, A. S., & Ecker, A. S. (2022, February 10). Digital twin reveals combinatorial code of non‑linear computations in the mouse primary visual cortex [Preprint]. bioRxiv. https://doi.org/10.1101/2022.02.10.479884"}]

Author rebuttal:

Meta-review: This paper proposes a probabilistic encoding model for mouse V1 activity that combines naturalistic video input with stimulus-independent latent factors, modeling full response distributions through a Zero-Inflated Gamma (ZIG) likelihood. The key scientific finding is that these learned latents, though inferred without supervision, align with behavioral variables such as pupil dilation and locomotion and exhibit spatial organization across cortical surface. Empirically, the model improves test log-likelihood over video-only baselines, generalizes better on out-of-distribution stimuli, and achieves comparable correlation to Poisson models while offering a richer account of trial-to-trial variability. This unification of stimulus-driven and latent state modeling fills a notable gap in dynamic encoding approaches.

The strengths of the paper are multiple: (1) a well-formulated probabilistic framework that addresses stimulus-independent variability directly, (2) biologically meaningful latent states that emerge without behavioral supervision, and (3) solid experimental validation across likelihood, correlation, OOD generalization, and interpretability analyses. Reviewers appreciated that the model captures meaningful latent structure not just statistically but also in terms of behavior and cortical topography. While some weaknesses were noted—slightly lower correlation compared to Poisson baselines, training complexity, and questions about latent purity—the authors provided thoughtful clarifications and additional experiments, including normalizing-flow likelihoods, latent-Poisson baselines, neuron-selection sensitivity, and transfer to ViT cores. These helped establish robustness and clarify trade-offs.

After rebuttal and discussion, reviewers generally agreed that the benefits of the proposed approach outweigh its limitations. The correlation–likelihood trade-off is well justified, the latent states show stronger behavioral correlations than CEBRA baselines, and added experiments confirmed the method’s flexibility across architectures and datasets. Although not all concerns are fully resolved, the consensus is that this is a technically sound and biologically meaningful contribution that advances dynamic neural modeling. I therefore recommend acceptance.
Decision: Accept (poster)