Time-Evolving Dynamical System for Learning Latent Representations of Mouse Visual Neural Activity
Authors: Liwei Huang
Zhengyu Ma
Liutao Yu
Huihui Zhou
Yonghong Tian
Venue: NeurIPS 2025 poster ()
Keywords: Visual Cortex
Latent Variable Models
Dynamical System
Primary area: neuroscience_and_cognitive_science

Abstract: Seeking high-quality representations with latent variable models (LVMs) to reveal the intrinsic correlation between neural activity and behavior or sensory stimuli has attracted much interest. In the study of the biological visual system, naturalistic visual stimuli are inherently high-dimensional and time-dependent, leading to intricate dynamics within visual neural activity. However, most work on LVMs has not explicitly considered neural temporal relationships. To cope with such conditions, we propose Time-Evolving Visual Dynamical System (TE-ViDS), a sequential LVM that decomposes neural activity into low-dimensional latent representations that evolve over time. To better align the model with the characteristics of visual neural activity, we split latent representations into two parts and apply contrastive learning to shape them. Extensive experiments on synthetic datasets and real neural datasets from the mouse visual cortex demonstrate that TE-ViDS achieves the best decoding performance on naturalistic scenes/movies, extracts interpretable latent trajectories that uncover clear underlying neural dynamics, and provides new insights into differences in visual information processing between subjects and between cortical regions. In summary, TE-ViDS is markedly competent in extracting stimulus-relevant embeddings from visual neural activity and contributes to the understanding of visual processing mechanisms. Our codes are available at https://github.com/Grasshlw/Time-Evolving-Visual-Dynamical-System.

Time-Evolving Dynamical System for Learning Latent Representations of Mouse Visual Neural Activity
Liwei Huang 1.2, Zhengyu Ma²; Liutao Yu?, Huihui Zhou', Yonghong Tian 1,2,3
School of Computer Science, Peking University, China
2eng Cheng Laboratory, China
School of Electronic and Computer Engineering. Shenzhen Graduate School, Peking University, China huanglv20@stu, pku. edu.cn, {mazhy, yult, zhouhh)@pche ac. cn, yhtianēpku. edu. cn
Abstract
Seeking high-qualin representations with latent variable models (LVMs) to reveal the intrinsic correlation between neural activity and behavior or sensory stimuli has attracted much interest. In the study of the biological visual system, naturalistic visual stimuli are inherently high-dimensional and time-dependent, leading to intri- cate dynamics within visual neural activity. However, most work LVMs has not explicitly considered neural temporal relationships. propose Time-Evolving Visual Dynamical System (TE-ViDS), sequential LVM that decomposes neural activity into low- www-dimensional latent representations that evolve over time. To better align the model with the characteristies of visual neural activity, we split latent representations into two parts and apply contrastive learning shape them. Extensive experiments synthetic datasets and real neural datasets from the mouse visual cortex demonstrate that TE-ViDS achieves the best decoding performance naturalistic scenes/movies, extracts interpretable latent trajectories that uncover clear underlying neural dynamics, and provides new insights into differences in visual information processing between subjects and between cortical regions. In summary, TE- ViDS is markedly competent in extracting stimulus-relevant embeddings from visual neural activity and contributes to the understanding of visual processing mechanisms. Our codes are available at https::/giituu..cm// www.Crasssww//mmiieteteeoee Evolving Visual- Visual-Dynante DanaaDaaaaaayyassnee
cope with such conditions,
Introduction
With the rapid development of neural recording technologies, researchers are able to simultancously record the spiking activity of large populations of neurons, opening up avenues for exploring the brain [55]. For nigh-dimensional neural data analysis, an important scientific problem is how to account for the intrinsic correlation between neural activity and behavioral patterns or sensory stimuli.
One influential approach is latent variable models (LVMs). which construct low-dimensional latent representations that bridge behavior stimuli and explain neural activity well (47
5
56 26 34
To deal with the increasing scale of neural population activity. LVMs have evolved from simple mathematical models early on, such as principal component analysis and factor analysis 9 44]
to complex artificial neural networks. More recently, advanced deep learning algorithms have enabled LVMs extract high-quality representations from neural activity without knowledge of experimental labels [59 42 22 36) or to incorporate behavioral information into models to constrain the shaping 23). These approaches have contributed the analysis neural
of latent variables [37 25 45 52
"Correspondngg author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

activity in various ways, including predicting neural responses 42 27
decoding related motion
patterns or simple visual scenes 36 48 and constructing interpretable latent structures 63 3].
Although LVMs have sparked strong interest in uncovering the underlying dynamical structure of neural activity, most studies have focused on neural data recorded from motor brain regions under
specific controlled behavioral settings [13 42 63 36 43 such as pre-planned reaching movements
[8] There is paucity of studies on LVMs exploring neural data from the visual cortex [21 62 48
Given the significant challenge of elucidating the correlation between neural activity and visual stimuli
the development of robust models for extracting vision-related latent representations is critically imperative. Furthermore, due to the complex visual neural dynamics, incorporating temporal structure into models to obtain time-dependent latent representations may be a key point for the analysis of visual neural activity. However, there is also lack of LVMs that explicitly model neural temporal relationships under visual stimuli.
15
28 58 17.
In this work, we propose the Time-nvolving Visual Dynamical System (TE-ViDS), which aims to generate high-quality latent representations from visual neural activity by disentangling neural components related to visual stimuli from those influenced by internal states. First, we introduce temporal structures to explicitly establish temporal relationships in latent variables, allowing them to evolve over time and capture the temporal dependency inherent in neural activity. Second, to sufficiently utilize the characteristics of visual neural activity, we adopt the split structure approach [36] and design distinct loss functions to construct two specialized parts of latent representations. The external latent representations aim to capture stimulus-relevant components within visual neural activity, while the internal latent representations reflect the dynamical internal states that influence the animal's 's sensory capability [39 4. In doing so, TE-ViDS can generate latent representations that are more relevant to visual stimuli while also explaining the effects of internal states. We evaluate our model on synthetic and mouse visual datasets, comparing it with leading alternatives. Our model demonstrates its ability to construct meaningful latent representations, yield a greater degree of correlation between neural activity and visual stimuli, and explain the visual information processing mechanisms of the mouse visual cortex. Specifically, our main contributions are as follows.
We introduce state factors to filter temporal information, enabling TE-ViDS to progressively compress neural activity and evolve latent variables. Regarding the distinct objectives of the two parts of latent representations, we apply self-supervised contrastive learning to shape the external and utilize a time-dependent prior distribution to guide the internal.
Through evaluation on synthetic datasets, we show that our model more effectively recovers latent structure and handles time-sequential data well.
Through evaluation mouse visual datasets, we demonstrate that our model outperforms alternative models in decoding natural scenes and natural movies, as well as in extracting clear temporal trajectories of neural dynamics at different time scales.
Further analysis reveals that our model can explain the potential variability in visual infor- mation processing between subjects and provide new evidence for the functional hierarchy of the mouse visual cortex.
2 Related Work
With the advancement of deep learning, the application of cutting-edge learning algorithms and the innovative design of model structures have greatly promoted the development of LVMs in neuroscience. Some prominent works are summarized below.
The approaches based on neural networks have become major avenues for discovering latent repre- sentations underlying neural activity, which better elucidate the mechanisms of neural representations. well-known model is latent factor analysis via dynamical systems (LFADS), which used RNNs in a sequential VAE framework to extract precise firing rate estimates and predict observed behavior Recurrent switching linear dynamical 41 also introduced recurrent structures, facilitating the understanding of complex nonlinear neural dynamics. Through specific latent variable designs, Pi-VAE 663 and Swap-VAr (36) built interpretable latent structures linked to motor behavior pat- terns. Furthermore, numerous studies have made significant contributions to achieving the goal of dissociating behaviorally relevant and irrelevant components in neural activity. Targeted neural
for single-trial data on motor cortical datasets [4
29
30
systems (rSLDS) [35 53 and low-rank RNNs

---

A
B
Recorded spikes
Roconstructed
RNN
rates
Encoder
Decoder
Eatered Insenal
RNN
Figure I: The method overview. A. The illustration of ViDS for analyzing visual neural activity in the mouse visual cortex. The encoder extracts spatial features from sequential spike data. The latent variables are evolved conditionally on features of the encoder and RNNs' state factors over time. The decoder maps latent variables inferred firing rates. B. The illustration of different learning objectives for the two parts of latent representations of ViDS. For external latent representations, we apply contrastive loss to encourage them to distinguish the stimulus-relevant components. Given reference sample (white dot), the red dot is positive sample and the orange dots negative samples. For internal latent representations, we use the KL divergence to constrain their distribution to time-dependent prior distribution.
dynamical modeling (TNDM), based on LFADS, constructed a two-pathway structure for separation [25] PSID [55. DFINE [1, and DPAD 46) were dedicated to introducing supervised information into dynamical systems to capture behavior-relevant dynamics.
In the visual domain, several latent variable models have made an effort to advance the understanding of visual neural dynamics. One work used Gaussian process models [19] and found that anesthesia- induced internal state fluctuations lead to correlated variability, while another proposed network- based linear dynamical system (ILDS) that captures neural variability under drifting grating stimuli 21 In addition, rectified latent variable model built latent variables that show a spectrum of functional groups as neurons the mouse visual cortex under drifting gratings 40, and a flow-based generative model recovered the distribution of visual neural activity [7]. Recently, CEBRA, self- supervised learning model, obtained consistent latent representations and made progress in decoding movies [48]. Although these studies cover various types of models, most of them are limited to simple visual stimuli and the task of reconstructing neural responses, leaving a gap for constructing high-quality latent representations from visual neural activity under naturalistic stimuli.
Methods
To generate latent representations that can effectively explain the characteristics of visual neural activity, introduce the Time-Exolving Visual Dynamical System (TE-ViDS), which evolves two parts of representations over time. In this section, we elaborate the concrete implementations of the model architecture and model learning (visualized in Figure D.
Basic notations. Neural responses recorded from large numbers of neurons are mostly in the form of spikes, which regarded discrete events. In practice, it is common to discretize period of time into small time windows and calculate the number of spikes within each window. Consequently, for the neural activity of population of neurons over period of time, define sequence input as x = (xi, X2,..., Xr) E RT'xN XN which represents spike counts of N neurons within time windows. The corresponding low-dimensional latent variable is denoted z (Z1,22,- 22,..... ... ZT) E TR
3.1 Model Architecture
As aforementioned, TE- ViDS compresses visual neural activity into external latent representations and internal latent representations (z, = 1z z (*) I). To evolve the two latent variables, we incorporate two dynamical systems into our model, using the state factor h, to sift and accumulate temporal
information & 20 [2]. In this way, the latent variables of the current time step are only conditioned on the input and state factors of the previous time steps, facilitating the capture of dynamics causal order.
3

---

External latent variables. This part of latent representations aims to capture the stimulus components of neural activity. To maintain our focus on variability arising from internal brain states, we construct external latent variables as deterministic values:
stimulus-relevant relevant
Ze
= (fx()))
(I)
Internal latent variables. This part of latent representations aims to reflect dynamical internal states that contain high variability and noise. To model such dynamics, internal latent variables are constructed as stochastic values that evolve over time. The tractable parameterized distribution (approximate posterior) is conditioned on both x, and h while the prior distribution is conditioned solely on h endowing with a certain degree of spontaneity:
(1) X1:ts la-1
(u o 1), 1, 02 == f
(xx(x))
(2)
(8) d N p.. 1), lū T2.cl prior (
(8)
(8)
(3)
Neural activity reconstruction. High-quality latent representations, compressed forms of the original neural activity, are required to effectively reconstruct their input. In practice, to enrich the dynamic information for a more accurate reconstruction, the spike count observation is related not only to the latent variables but also the internal state factors:
&
(e) Zi: 21 11--1 P(r.), = fdee (2,0) :aahhh -
(
where the parameterized distribution P is chosen to be Poisson 21, i.e., the actual inferred neural activity is spike firing rates rg.
The state factor is updated by GRU By selectively integrating and exploiting input and latent variables, the state factor crucial for capturing complex sequential dynamics. Besides, since the animal's i dynamical internal states are inevitably affected by visual stimuli, h, and h.d) corresponding to the latent representations are updated differently:
Recurrent neural networks.
h = .(e) (fx(x)), h(
(5)
h = JiR (xx(x,,, 2/(0 2
All the functions/ f above are parameter-learnable neural networks (see Appendix A for details).
3.2 Model Learning
Given that the two parts of latent representations hold different objectives for disentanglement, introduce the following loss function to optimize all learnable parameters in an end-to-end manner:
L = Lrecons + +B &Leantrustivee + Erogular.
The first term which deals with the objective of inferring neural activity is formulated as T E [(),,, where Lp is Poisson negative log likelihood.
The second term encourages external latent representations to distinguish stimulus-relevant compo- nents through self-supervised contrastive learning. Specifically, for a given sample (XI...., Xr)) we randomly select another sequence offset by several time steps as a positive sample, denoted Xpos = (X)+A)...XT+A)) where A can be positive negative. The offset is smaller than the length of the sequence to ensure that the positive sample pairs overlap, i.c., the visual stimuli cor- responding to the positive sample pairs are similar, which establishes association between the external latent variables and the visual stimuli. Then, mini-bated of negative samples is randomly selected from the entire training set. Finally, we compute the external latent variables of all selected samples and apply the NT-Xent loss [10] as Lcoutrastive to them. In doing so, external latent repre- sentations the positive sample pairs are driven to distinguished from the negative samples. For this term, we do not apply the time-wise operation, but flatten the temporal and spatial dimensions of external latent variables for the loss computation. addition, to enhance the effect of the positive sample, we adopt the practice of the swap operation (36). We exchange the external latent variables

---

of the given sample with those of the positive sample while keeping the internal latent variables unchanged. The new latent representations are then used to compute new inferred firing rates and an additional reconstruction loss.
The third term measures the difference between the prior and the approximate posterior of the internal latent variables by the KI. divergence, formulated as / Σ DxI, (0) | Besides, we compute the L2 norm of the expectation and log- variance of the prior distribution as a regularization to avoid excessive fluctuations over time and to stabilize model training.
and y are hyperparameters used to control the severity of the penalty for each loss term. The entire loss function actually obeys the strategy of maximizing the evidence lower bound (ELBO) of the marginal log likelihood in the VAE framework [31], while also accounting for the impact of introducing the contrastive loss [57] A detailed derivation is given in Appendix
Experiments
To evaluate the utility of the latent representations constructed by TE-ViDS for the analysis of visual neural activity, we perform a series of experiments to compare our model with alternative LVMs from two aspects, similar to studies focusing on motor brain regions (36] 48]. First, we quantify the performance decoding visual stimuli using latent representations. which has long served as research hotspot for unraveling the mechanisms of visual processing [28] [58] Second, we assess the clarity of latent temporal trajectories extracted from neural dynamics.
4.1 Datasets
We carry out the experiments on two commonly used synthetic datasets and a well-known publicly available mouse visual neural dataset.
Synthetic datasets. The two synthetic datasets are crucial for evaluating the fundamental ability of LVMs to construct accurate latent variables. One is non-temporal dataset generated from several sets of labels 663] [3]]3]], facilitating the assessment of class discrimination capabilities. The other is
temporal dataset constructed by the Lorenz system test for extracting temporal relationships
21 54]. A detailed description of the generating procedure is presented in Appendix C
Mouse visual neural dataset. We use subset of the Allen Brain Observatory Visual Coding dataset
5T. which has been used in a variety of studies, including the construction of brain-like networks
[50, the modeling of functional mechanisms 66 [4], and the decoding of visual stimuli 48 This
dataset was collected by Neuropixel probes from six mouse visual cortical regions simultaneously. including VISp, VISI, VISrI, VISal, VISpm, and VISam. Notably, neural activity was recorded while mice passively viewed visual stimuli without any task-driven behavior. The dataset comprises 32 mouse. In this work, choose analyze five mice (subjects) that
sessions, each involving have the maximum number of recorded neurons (see Appendix D). and these neurons are evenly distributed across all regions (the coefficient of variation for the number of neurons across six regions is below 0.5). We focus on neural activity in response to natural scenes and a natural movie. For natural scenes, 118 images are presented in random order, each for 250ms and 50 trials. The neural activity in the form of spike counts is binned into 10-ms windows so that each trial contains 25 time points. The natural movie 30 seconds long with frame rate of 30Hz, presented for 10 trials. We bin the spike counts with sampling frequency of 120Hz and align them with the movie timestamps. resulting in four time points for each frame.
Since spike responses are quite variable across trials even under identical experimental conditions, we conduct the basic evaluation on "held-out" trials. Specifically, for each dataset, we randomly split all trials into 80% for training, 10% for validation, and 10% for test.
4.2 Evaluation Metric
Following previous studies 54
48
apply quantitative metrics for evaluation.
Reconstruction score. For the two synthetic datasets, we use linear regression to the latent variables of models to the true latent variables, and report the R² as the reconstruction score.
5

---

E
Grazn Trus
Stap VAE
CEIMA
New-
Bosi
Lave2
Love?
LPATR
VAV
Figure 2: Results on synthetic datasets. A. The true latent variables of the non-temporal dataset. B-D. The inferred latent variables of our model and some alternative models. E. The reconstruction scores of all models on the non-tem con-telporal and temporal datasets. The standard error is computed on 10 runs with different random initializations.
Decoding score. For the mouse visual neural dataset, we quantify the performance of models in decoding natural scenes/natural movie frames. In terms of natural scene decoding, we obtain the latent variables of the last 20 time points (50ms-250ms) in each trial, since there a response latency in the mouse visual cortex when presented with static stimuli [51]. These latent variables are then concatenated into a vector to form the latent representations of each trial, thereby retaining maximal temporal information. We use the KNN algorithm to classify the latent representations of each trial, i.e, to decode the corresponding natural scenes. In terms of natural movie frame decoding, we compute the latent variables of four time points within each frame, averaging them create the latent representations for that frame. We also use the KNN to predict movie frames based on latent representations (900 frames in total, i.e., 900 classes). The specific KNN procedure is as follows: First, we fit the KNN on the training set and search for the optimal hyperparameter (the number of neighbors) on the validation set, using classification accuracy as the metric. The search range is set to odd numbers between 1 and 20, with odd numbers chosen to reduce decision uncertainty. Then, the accuracy on the test set is reported as the decoding score. Notably, for movie frame decoding. we follow the established approach of CEBRA [48). considering an error of less than Is (constraint window) between the predicted frame and the true frame as correct prediction.
4.3 Alternative Models
For comprehensive analysis, we compare TE-ViDS with several leading LVMs, including three generative models (a sequential: LFADS 42 a supervised: VA- 63 and a self-supervised: Swap- VAE (36)) and nonlinear encoding method with contrastive learning (CEBRA) [48) Specifically, VAE incorporates additional supervised information for latent variable construction, but does not model temporal dynamics. Swap- VAE utilizes contrastive learning construct separated latent variables, aligning with our disentanglement goal, but it also does not explicitly model temporal dynamics. For the two models that are able model the temporal relationships, LFADS processes sequential neural activity using bidirectional RNNs, while CEBRA encodes temporal features of sequential data with fixed convolutional kernels. Furthermore, for fair comparisons, we build small version of our model (TE-ViDS-smas small) with fewer trainable parameters than Swap-VAF (see Appendix
F for the number of trainable parameters).
For each dataset, all models are set to latent variables of the same dimension and trained for more than 20,000 iterations until convergence. Additionally, we apply the hyperparameter grid search for all models. More details of the training setup are presented in Appendix
4.4 Results on Synthetic Datasets
First, we visualize the reconstructed latent variables on the non-temporal dataset (Figure -D)) TE-VIDS reliably separates the different clusters as well as recovers the structure of the true latent variables to form clear arcs. In contrast, some of the alternative models fail to construct similar structures despite separating clusters (Swap- VAE and CEBRA), and others even struggle to separate four clusters (LFADS and VAE, Appendix G). Quantitatively, the reconstruction scores also suggest that our model outperforms all alternative models 2
For the original temporal dataset, ViDS performs significantly better than those models that process sequential spikes at each time point independently, and moderately better than LFADS
'i VAE incorporates the label prior during training, but the inferred latent variables are built without the label prior at the evaluation stage. This way is used in all experiments.

---

Table 1: The decoding scores (%) for 118 natural scenes on the mouse visual neural dataset. The standard error computed on runs with different random initializations.
Models
Mouse 1
Mouse 2
Mouse
Mouse
Mouse 5
PCA
0.59
1.53
1.53
0.80
0.85
LFADS
30.76:1:0.55
16.46:1:0.9
22.20:10.26
19.69-10.38
4.69:1:.22
Pi-VAE
7.49:10.77
19.42:10.62
22.92:L0.37
2.22:10.64 13.71
Swap-VAE
32.81
24.34:10.57
14.36-t1.31
14.85:11.29
3.32+00.7
CEBRA
1.53::115
3.42::0.07
4.86:1011
2.81=10.33
1.08=0.100
TE-V -S-ViDs-small
27.08=22.6
23.95=1.399
1..0
34.95110900
9.93
50.8610.81
27.2410.477
29.90
38.0510.53
9.44
1.0
50
TEANLSS
Clas
Clas
Clas
(latonal)
Clas
0.6
Decoding Score (%)
40
Claa
Clas
Claa
Clas
0.2
Cbac
-.2
20
Clas
Clas
Clas
4.6
10
Clas
Clo
Clas
Clos
Mouse
Mouse 2
Mouse 3
Mouse 4
Mouse
Clas
Clas
Class
Clos
Clas
Clas Dle
Figure 3: Results the mouse neural dataset under natural scene stimuli. A. The decoding scores (%) of the full, external and internal latent representations of TE- VEDV for 118 natural scenes. B. RSMs computed on the original neural representations and (Mouse and Mouse 2). Each element in matrix is the similarity between two trials' representations. Each small square involves comparisons between two natural scenes, containing 50 trials.
latent representations, respectively
which also uses RNNs process sequential data (Figure 2 :). Moreover, when shuffle the time dimension for each trial data in the original dataset to obtain a dataset without temporal dependency, the performance of our model and LFADS shows drastic degradation. However, for models utilizing time-jittered positive samples, the performance reduction observed in Swap- VAE and CEBRA was comparatively less severe. These results demonstrate the superiority of our model in dealing with temporal data and its sensitivity to temporal relationships.
4.5 Results on the Mouse Neural Dataset under Natural Scene Stimuli
As shown in Table T TE- ViDS achieves the highest decoding scores for all mice with noticeable improvement over other models. In particular, CEBRA, which can encode temporal relationships, instead performs poorly this downstream task, suggesting that using time filters to extract temporal neural features is less suitable in this case. In contrast, the time-evolving latent representations constructed by our model reliably capture temporal relationships encoded in the mouse visual cortex under different static scene stimuli, which may be helpful for natural scene discrimination. We further evaluate the decoding performance the external and internal latent variables separately (Figure 3A). We find that the external latent representations significantly outperform the internal latent representations, supporting our assumption that the former are tuned to coamulus-relevan components of neural activity, while the latter point stimulus-irelevant internal states.
Although our model achieves the highest performance, we find that the decoding scores for different mice show large differences. We further apply Representational Similarity Analysis (RSA) [33 32 to the original neural activity and TE-Vil TE-ViDS's latent representations. Specifically, we calculate the representational similarity between the representations to each pair of trials using the Pearson correlation coefficient, yielding representational similarity matrices (RSMs). We select ten scenes that elicit the strongest average responses for visualization. By comparing the RSMs of the original

---

Table 2: The decoding scores (%, in is window) for natural movie frames on the mouse visual neural dataset. The standard error is computed on 10 runs with different random initializations.
Models
Mouse 1
Mouse
Mouse
Mouse 4
Mouse 5
PCA (baseline)
8.44
28.77
25.42
21.56
11.69
LFADS
0.94
26.57-1:.44
26.77:12.23
24.76+t1.80
12.69 1.38
Pi-VAE
10.24:10.31
42.51
36.96+10.00
38.31
18.09
Swap-VAE
12.19440.20
51.31
45.96:10.34
41.53-10.63
22.70++0.42
CEBRA
10.62::0.18
52.76=0.899
61.01110.76
42.11=0073
22.33
TE- ViDD---maall
13.23+0.25
64.09=10.41
59.36110.30
53,46:10.58
29.60#:.26
TE-ViDS
13.88:10.19
65.38
59,88+0.72
54.33:10.54
30.1810.400
A 7h
TOAN0
siredy
Cosamiai Window 63
TE-VIS
VAE
E
CEBRA
10
-1
Figure 4: Results on the mouse neural dataset under natural movie stimuli. A. The decoding scores (%) of the full, external and internal latent representations of TE-ViDSS B. The decoding scores (%) for movie frames under different constraint windows of predicted frames and the true frames. C-E. Visualization results latent trajectories (Mouse 2). Each color corresponds to all frames within 1s. Small dots denote one frame. Large dots denote the average among group of frames. The red dashed line connects averages. F. RSMs computed on the original neural representations and TE-ViDS's latent representations (Mouse 2). Each element is the similarity between two frames' representations. Each small square involves comparisons between two trials, containing 900 frames.
neural activity and TE-Vin ViDS (Figure B), we show that model can extract clearer intrinsic patterns of neural activity and reduce the impact of useless noise, facilitating the unraveling of information observe that the RSMs of TE-ViDS
processing mechanisms in the visual cortex. Moreover, between two mice exhibit significantly different patterns. For Mouse 1, there are two redder blocks, top left and bottom right, in each of the small squares, suggesting that the neural representations of closer trials in time are more similar and divided into two periods, even under different visual stimuli. This phenomenon may be due differences in the internal states of this mouse during the two periods, and the effect of such states on neural activity is even stronger than that of visual stimuli previous
some extent. For Mouse 2, there is such clear change in state. These results echo studies, showing that sensory performance in mice is strongly influenced by their internal state 4), which may also be an important reason for the variability in neural activity between subjects.
In addition to tasks on "held-out" trials, we perform more challenging task on "held-out" stimuli. All models are trained with neural responses to 95 natural scenes (approximately 80% of all scenes), and their decoding performance is evaluated on the remaining 23 scenes. The results (see Appendix H show that TE- ViDS outperforms other models,
for these unseen stimuli.
In summary, these results demonstrate the advantages of our model decoding natural scene stimuli and extracting fine neural dynamics (see latent trajectories over time in Appendix []] The latent representations constructed by our model help to explain potential differences visual information processing between individuals, which deserves further exploration.
8

---

A
Natural Scenes
B
Natural Movic
Decoding (%)
Decoding Score
6
VBpp
VISI
Visai
Vipee.
ViSam
V15p
ViSi
Visai
ViSye
ViSee
Figure The decoding scores (%) of TE- ViDS for natural scenes/natural movie frames on six mouse visual cortical region datasets. The box plots are based on 10 runs with different random initializations.
4.6 Results on the Mouse Neural Dataset under Natural Movie Stimuli
TE-ViDS performs best for four mice (Table 2) over LFADS than over the others. We also show that the external latent representations are more stimulus releevatteee (Figure 4A). The results of Figure show that TE-Vil VEDV consistently outperforms alternatives across a broad range constraint windows, especially in finer- finer-grained frame decoding.
Specifically, our model gains more advantage
For visualization, we reduce the dimensions of the latent representations of each movie frame using ISNE. We set all frames within 1s group and show the trajectories of latent representations for the middle 10s of the movie (Figures -E We focus on Mouse 2, for which most models achieve the highest scores. See Appendix for the results of the other parts of the movie and the other models). Compared to alternative models, the representations of TE-ViDS show clear temporal structures along movie frames with less overlap and entanglement between different time groups.
For further analysis, we also compute RSMs of the original neural activity and TE-ViDS on Mouse 2 (Figure 4 Different from the RSMs in Figure 33 each element in these matrices the rep- resentational similarity between the representations to each pair of movie frames and each small square involves comparisons between two trials. The comparison between neural RSM and TE-ViDS RSM provides the same evidence that our model extracts clearer patterns and reduces noise. In each small square VEDSSD RSM, brighter regions are near the diagonal, showing that the neural representations of neighboring frames are more similar, even for different trials. This pattern suggests that under continuous and long-duration visual stimuli, this mouse has a stronger neural response to stimuli and is less affected by its internal state. This may reveal differences in the mouse's 's visual encoding ability under short-duration static stimuli and long-duration continuous stimuli.
To conclude, our model achieves the highest decoding scores for natural movie frames. Furthermore, we also perform experiments on "held-out" movie frame stimuli, in which our model performs best on decoding movie frames (Appendix H). Importantly, our model constructs meaningful latent representations related to the content and temporal structure of movie stimuli at large time scales, providing insights into visual information processing mechanisms for long- duratiin stimuli.
4.7 Experiments for Mouse Cortical Regions
To explore the ability of six mouse visual cortical regions process visual stimuli, we randomly sampled 150 neurons from each of the six regions of five selected mice to construct visual neural activity datasets, respectively. We train TE-VIDS on these six datasets and evaluate its decoding performance for natural scenes/natural movie frames. The experiment is repeated ten times.
As shown in Figure the decoding performance varies considerably across regions and the perfor- mance trends across regions between decoding the two types of stimuli are essentially the same. The anatomical hierarchy of the mouse visual cortex [5I] shows that VISp is the primary region, while VISI, VISrI and VISal are the mid-level regions, and VISpm and VISam are the high-level regions. Additionally, VISrl is multi-sensory region [14] that receives multi-modal sensory input. Our results show that the decoding performance is higher in the primary and mid-level regions than the high-level regions, with the lowest performance observed in VISri. First, the poor performance of VISrl may be due to the fact that, as multi-sensong region, it is not well driven by visual stimuli alone. Second, while some previous studies have suggested that the mouse visual cortex is organized in parallel structure 49 24 the differences in decoding performance across regions may provide

---

evidence that there is heterogeneous and specialized visual encoding information for each brain region. These results echo the anatomical work [5]] some extent and provide new insights into the functional hierarchy of the mouse visual cortex.
5
Discussion
This work presents novel latent variable model, TE-ViDS, by introducing temporal structures to explicitly establish temporal relationships and constructing two parts of latent representations to disentangle the stimulus-relevant components from visual neural activity. The results of synthetic and mouse neural datasets demonstrate that our model outperforms alternative models and builds latent representations that are strongly correlated with visual input, revealing intrinsic correlations between visual neural activity and visual stimuli. A series of ablation studies demonstrates the effectiveness of our model's 's components (Appendix []] Furthermore, TE-ViD ViDS aids in explaining the variability in visual information processing between subjects and provides computational evidence for the functional hierarchy of the mouse visual cortex, which may contribute to the computational neuroscience community.
There is paucity of research analyzing visual neural activity with LVMs. CEBRA has built consistent latent representations from multimodal visual neural activity [48). Our work takes a further step the development of powerful LVMs to yield stimulus-correlated latent representations and capture neural dynamics. However, there are still some limitations. First, in the absence of recorded internal states or behavioral information, variables and find it difficult to interpret the functional role of inferred internal latent representations. Second, our model exhibits substantial variability in decoding performance across individual mice. Subsequent analyses suggest that the variability in neural activity between subjects and trials [60] (38 may be caused by the animal's internal states. This could represent potential breakthrough for LVMs to consistently construct high-quality stimulus-correlated latent representations across conditions, but further exploration required.
unable to quantitatively assess the interpretability of internal latent
Last but not least, our approach is not limited to studying visual neural spikes from mice and can be extended neural data from other species and modalities investigate broader principles of biological coding mechanisms, facilitating the development of computational neuroscience.
Acknowledgments and Disclosure of Funding
This work is supported by grants from the Shenzhen KQTD (No. 20240729102051063) and the National Natural Science Foundation of China (No. 62027804, No. 62425101, No. 62332002, No. 62088102, and No. 62206141).
References
[1] Hamidreza Abbaspourazad, Eray Erturk, Bijan Pesaran, and Maryam M Shanechi. Dynamical flexible inference of nonlinear latent factors and structures neural population activity. Nature Biomedical Engineering, 8(1):85-108, 2024.
[2] Parima Ahmadipour, Omid G Sani, Bijan Pesaran, and Maryam M Shanechi. Multimodal spiking and field potential population
subspace identification for modeling activity. Journal of Neural Engineering, 21(2):026001, 2024.
[3] Mikio C Aoi, Valerio Mante, and Jonathan W Pillow. Prefrontal cortex exhibits multidimensional dynamic encoding during decision-making. Nature neuroscience, 23(11):1410-1420, 2020.
[4] Zoe Ashwood, Nicholas A Roy, Iris R Stone, International Brain Laboratory, Anne Urai, Anne K Churchland, Alexandre Pouget, and Jonathan W Pillow. Mice alternate between discrete strategies during perceptual decision decision-making, Nature Neuroscience, 25(2):201-215 2022.
[5] Giwon Bahg. Daniel G Evans, Matthew Galdo, and Brandon M Turner. Gaussian process linking functions for mind, brain, and behavior. Proceedings of the National Academy of Sciences, 117(47)22338-290 294096 2020.
10

---

[6] Shahab Bakhtiari, Patrick Mineault, Timothy Lillicrap, Christopher Pack, and Blake Richards. The functional specialization of visual cortex emerges from training parallel pathways with self-supervised predictive learning. In Advances in Neural Information Processing Systems 34, pages 181644251177 2021.
[7] Mohammad Bashiri, Edgar Walker, Konstantin- Klemens Lurz, Akshay Jagadish, Taliah Muham- mad, Zhiwei Ding, Zhuokun Ding, Andreas Tolias, and Fabian Sinz. A flow-based latent state generative model of neural population responses to natural images. In Advances in Neural Information Processing Systems 34, pages 15801-15815, 2021.
KonstnntinKKlemens
[8] Justin Bayer and Christian Osendorfer. Learning stochastic recurrent networks, 2015.
[9] Kevin L. Briggman, Henry DI Abarbanel, and WB Kristan Jr. Optical imaging of neuronal 2005.
populations during decision-making. Science,
[10] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In Proceedings of the 37th International Conference on Machine Learning, volume 119, pages 1597-1607. PMLR, 2020.
[11] Kyunghyun Cho, Bart van Merrienboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using rmn encoder- decoder for statistical machine translation, 2014.
[12] Junyoung Chung, Kyle Kastner, Laurent Dinh, Kratarth Goel, Aaron C Courville, and Yoshua Bengio. A recurrent latent variable model for sequential data. In Advances in Neural Information Processing Systems 28, 2015.
[13] Mark M Churchland, John P Cunningham, Matthew T Kaufman, Justin D Foster, Paul Nuyu- jukian, Stephen Ryu, and Krishna Shenoy. Neural population dynamics during reaching. Nature, 487(7405):51-555 2012.
[14] Saskia EJ de Vries, Jerome A Lecoq, Michael A Buice, Peter A Groblewski, Gabriel K Ocker, Michael Oliver, David Feng, Nicholas Cain, Peter Ledochowitsch, Daniel Millman, et al. A large-scale standardized physiological survey reveals functional organization of the mouse visual cortex. Nature neuroscience, 23(1):138-151, 2020.
[15] James DiCarlo, Davide Zoccolan, and Nicole C Rust. How does the brain solve visual object recognition? Neuron, 73(3):415-434, 2012.
[16] Laurent Dinh, Jascha Sohi-Dickstein, and Samy Bengio. Density estimation using real NVP. In International Conference on Learning Representations, 2017.
[17] Changde Du, Kaicheng Fu, Jinpeng Li, and Huiguang He. Decoding visual neural representa- tions by multimodal learning of maaain-vssulllinng Analysis and Machine Intelligence, (5(9):1070011077) 2023.
features. IEEE Transactions on Patlern
[18] Eva L Dyer, Mohammad Gheshlaghi Azar, Matthew G Perich, Hugo L Fernandes, Stephanie Naufel, Lee E Miller, and Konrad P Körding. A cryptography-baan based approach for movement decoding. Nature biomedical engineering, 1(12):96--976 2017.
[19] Alexander S Ecker, Philipp Berens, R James Cotton, Manivannan Subramaniyan, George H Denfield, Cathryn Cadwell, Stelios M Smirnakis, Matthias Bethge, and Andreas S Tolias. State dependence of noise correlations in macaque primary visual cortex. Neuron, 82(1):235-244, 2014.
[20] Otto Fabius and Joost R. van Amersfoort. Variational recurrent auto-encoders, 2015.
[21] Yuanjun Gao, Evan W Archer, Liam Paninski, and John P Cunningham. Linear dynamical neural population models through nonlinear embeddings. In Advances in Neural Information Processing Systems 29, 2016.
[22] Joshua Glaser, Matthew Whiteway, John P Cunningham, Liam Paninski, and Scott Linderman. Recurrent switching dynamical systems models for multiple interacting neural populations. In Advances in Neural Information Processing Systems 33, pages 14867-14878, 2020.
11

---

[23] Rabia Gondur, Usama Bin Sikandar, Evan Schaffer, Mikio Christian Aoi, and Stephen L Keeley. Multi-modal gaussian process variational autoencoders for neural and behavioral data. In International Conference on Learning Representations, 2024.
[24] Liwei Huang, Zhengyu Ma, Liutao Yu, Huihui Zhou, and Yonghong Tian. Long-range feedback spiking network captures dynamic and static representations of the visual cortex under movie stimuli. Advances in Neural Information Processing Systems 37, pages 11432-1 1455,, 2024.
[25] Cole Hurwitz, Akash Srivastava, Kai Xu, Justin Jude, Matthew Perich, Lee Miller, and Matthias Hennig. Targeted neural dynamical modeling. Advances in Neural Information Processing Systems 34, pages 29379-2939222 293922, 2021.
[26] Mehrdad Jazayeri and Srdjan Ostojic. Interpreting neural computations by examining intrinsic and embedding dimensionality of neural activity. Current opinion in neurobiology, 70: 12--10., 2021.
[27] Jaivardhan Kapoor, Auguste Schulz, Julius Vetter, Felix Pei, Richard Gao, and Jakob H. Macke. Latent diffusion for neural spiking data, 2024.
[28] Kendrick N Kay, Thomas Naselaris, Ryan J Prenger, and Jack L Gallant. Identifying natural images from human brain activity. Nature, 15277185) 2008,
[29] Mohammad Reza Keshtkaran and Chethan Pandarinath. Enabling hyperparameter optimiza- tion in sequential autoencoders for spiking neural data. In Advances in Neural Information Processing Systems 32, 2019.
[30] Mohammad Reza Keshtkaran, Andrew R Sedler, Raeed Chowdhury, Raghav Tandon, Diya Basrai, Sarah L Nguyen, Hansem Sohn, Mehrdad Jazayeri, Lee E Miller, and Chethan Pandari- nath. A large-scale neural network training framework for generalized estimation of single-trial population dynamics. Nature Methods, 19(12):1522-1577 2022.
[31] Diederik Kingma and Max Welling. Auto-encoding variational bayes, 2013.
[32] Nikolaus Kriegeskorte, Marieke Mur, and Peter A Bandettini. Representational similarity analysis-connecting the branches of systems neuroscience. Frontiers in systems neuroscience, 2:4, 2008
[33] Nikolaus Kriegeskorte, Marieke Mur, Douglas A Ruff, Roozbeh Kiani, Jerzy Bodurka, Hossein Esteky, Keiji Tanaka, and Peter A Bandettini. Matching categorical object representations in inferior temporal cortex of man and monkey. Neuron, 60(6): 2222111144 141, 2008.
[34] Christopher Langdon, Mikhail Genkin, and Tatiana A Engel. unifying perspective on neural manifolds and circuits for cognition. Nature Reviews Neuroscience, 24(6):363-31 377, 2023.
[35] Scott Linderman, Matthew Johnson, Andrew Miller, Ryan Adams, David Blei, and Liam Paninski. Bayesian Learning and Inference in Recurrent Switching Linear Dynamical Systems. In Proceedings of the 20th International Conference on Artificial Intelligence and Statisties, volume 54 of Proceedings of Machine Learning Research, pages
2017.
[36] Ran Liu, Mehdi Azabou, Max Dabagia. Chi- Chi-Heng Lin, Mohammad Gheshlaghi Azar, Keith Hengen, Michal Valko, and Eva Dyer. Drop, swap, and generate: A self-supervised approach for generating neural activity. In Advances Neural Information Processing Systems 34, pages 10587-10599, 2021.
[37] Valerio Mante, David Sussillo, Krishna Shenoy, and William Newsome. Context-dependent computation by recurrent dynamics in prefrontal cortex. nature, 503(7774))778-88) 84, 2013.
[38] Tyler Marks and Michael J Goard. Stimulus-dependent representational drift in primary visual cortex. Nature communications, 12(1):5169, 2021.
[39] João Marques, Meng Li, Diane Schaak, Drew N Robson, and Jennifer M Li. Internal state dynamics shape brainwide activity and foraging behaviour. Nature, 577 -23, 2020.
12

---

[40] Jeremiah B Palmerston and Rosa HM Chan. Latent variable models reconstruct diversity of neuronal response to drifting gratings in murine visual cortex. IEEE Access, 9:75741-75750, 2021.
[41] Matthijs Pals, A Erdem Sagtekin, Felix Pei, Manuel Gloeckler, and Jakob H Macke. Infer- ring stochastic low-rank recurrent neural networks from neural data. In Advances in Neural Information Processing Systems 37, pages 18225-18264, 2024.
[42] Chethan Pandarinath, Daniel J O' Shea, Jasmine Collins, Rafal Jozefowicz, Sergey D Stavisky, Jonathan C Kao, Eric M Trautmann, Matthew Kaufman, Stephen Ryu, Leigh Hochberg. et al. Inferring single-trial neural population dynamics using sequential auto-encoders, Nature 2018.
methods, 15(10):805-3
[43] Felix Pei, Joel Ye, David Zoltowski, David Zoltowski, Anqi Wu, Raced Chowdhury, Hansem Sohn, Joseph O' Doherty, Krishna V Shenoy, Matthew Kaufman, Mark Churchland, Mehrdad Jazayeri, Lee Miller, Jonathan Pillow, Il Memming Park, Eva Dyer, and Chethan Pandarinath. Neural latents benchmark '21: Evaluating latent variable models of neural population activity. In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks, volume 1, 2021.
[44] Patrick T Sadtler, Kristin M Quick, Matthew D Golub, Steven M Chase, Stephen I Ryu, Elizabeth C Tlyle-Kabbar Byron M Yu, and Aaron P Batista. Neural constraints on learning. Nature, 512(7515):423 426, 2014.
[45] Omid G Sani, Hamidreza Abbaspourazad, Yan T Wong, Bijan Pesaran, and Maryam M Shanechi. Modeling behaviorally relevant neural dynamics enabled by preferential subspace identification. Nature Neuroscience, 24(1):140-144 2021.
[46] Omid G Sani, Bijan Pesaran, and Maryam M Shanechi. Dissociative and prioritized modeling of behaviorally relevant neural dynamics using recurrent neural networks. Nature neuroscience, 27(10):203320455 2024.
[47] Shreya Saxena and John P Cunningham. Towards the neural population doctrine. Current opinion in neurobiology, 55:103-111, 2019.
[48] Steffen Schneider, Jin Hwa Lee, and Mackenzie Weygandt Mathis. Learnable latent embeddings for joint behavioural and neural analysis. Nature, 617(7960):360-368 2023.
[49] Jianghong Shi, Eric Shea-Brown, and Michael Buice. Comparison against task driven artificial neural networks reveals functional properties in mouse visual cortex. In Advances in Neural Information Processing Systems 32, 2019.
[50] Jianghong Shi, Bryan Tripp. Eric BheaaBrrown, Stefan Mihalas, and Michael A. Buice. Mousenet: A biologically constrained convolutional neural network model for the mouse visual cortex. PLOS Computational Biology. 18(9):e1010427, 2022
[51] Joshua Siegle. Xiaoxuan Jia, Séverine Durand, Sam Gale, Corbett Bennett, Nile Graddis, Greggory Heller, Tamina K Ramirez, Hannah Choi, Jemnifer A Luviano, et al. Survey of spiking in the mouse visual system reveals functional hierarchy. Nature, 592((852):86-99) 2021.
[52] Jonnathan Singh Alvarado, Jack Goffinet, Valerie Michael, William Liberti III, Jordan Hatfield, Timothy Gardner, John Pearson, and Richard Mooney. Neural dynamics underlying birdsong practice and performance. Nature, 599((786))635--6)) 6699, 2021.
[53] Jimmy Smith, Scott Linderman, and David Sussillo. Reverse engineering recurrent neural networks with jacobian switching linear dynamical systems. In Advances in Neural Information Processing Systems 34, pages 16700-16713, 2021.
[54] David Sussillo, Rafal Jozefowicz, I.. F. Abbott, and Chethan Pandarinath. Lfads analysis via dynamical systems, 2016.
latent factor
[55] Anne E Urai, Brent Doiron, Andrew M Leifer, and Anne K Churchland. Large-scale neural recordings call for new insights to link brain and behavior. Nature neuroscience, 25(1):11-19, 2022.
13

---

[56] Saurabh Vyas, Matthew D Golub, David Sussillo, and Krishna V Shenoy. Computation through neural population dynamics. Anmual review of neuroscience, 43(1):2449-2) 275, 2020.
[57] Yu Wang, Hengrui Zhang, Zhiwei Liu, Liangwei Yang, and Philip S Yu. Contrastvae: Con- trastive variational autoencoder for sequential recommendation. In Proceedings of the 31st ACM international conference on information & knowledge management, pages 2056-2066 2022.
[58] Haiguang Wen, Junxing Shi, Yizhen Zhang, Kun-Han Lu, Jiayue Cao, and Zhongming Liu. Neural encoding and decoding with deep learning for dynamic natural vision. Cerebral cortex, 2664444413-4416,, 2018.
[59] Anqi Wu, Nicholas A. Roy, Stephen Keeley, and Jonathan W Pillow, Gaussian process based nonlinear latent structure discovery in multivariate spike train data. In Advances in Neural Information Processing Systems 30, 2017.
[60] Xia, Tyler D Marks, Michael Goard, and Ralf Wessel. Stable representation of a naturalistic movie emerges from episodic activity with gain variability. Nature communications, 12(1):5170, 2021.
[61] Byron M Yu. John P Cunningham, Gopal Santhanam, Stephen Ryu, Krishna V Shenoy, and Maneesh Sahani. Gaussian-process factor analysis for low-dimensional single-trial analysis of neural population activity. Advances in Neural Information Processing Systems 21, 2008.
[62] Yuan Zhao and II Memming Park. Variational latent gaussian process for recovering single-trial dynamics from population spike trains. Neural computation, 1966)11233 2017.
[63] Ding Zhou and Xue-Xin Wei. Learning identifiable and interpretable latent models of high- dimensional neural activity using pi-vae. In Advances in Neural Information Processing Systems 33, pages 7234-7247, 2020.
14

---

A Detailed Structure of TE-ViDS
The encoder and decoder of TE- TE-ViDS are derived from Swap- VAE. The encoder consists of three blocks, the first two which are sequentially stacked with a linear layer, batch normalization, and a ReLU activation. The last block differs from Swap-VAL VAE that it additionally introduces the hidden states of the GRU as input. The output dimensions of the three blocks are N. M, and M, where N is the number of neurons and M is the number of latent variables. The decoder is symmetric structure of the encoder, where the first two blocks are also sequentially stacked with a linear layer, batch normalization, and a ReLU activation, and the last block linear layer followed by SoftPlus activation. We set the dimensions of the two latent variables to be equal and use a one-layer GRU for each of them, where the dimensions of the hidden states are equal to the dimensions of the latent variables. The operations of each module are shown in Figure 6
Overall
Prior
Encode
Decode
Recurrent
Figure 6: The operations of each module in TE-ViDS.
B Derivation of the Loss Function of TE-ViDS
The loss function of TE-ViDS obeys the strategy maximize the likelihood of the joint sequential distribution p(x1.)). Involving the latent variables ZI.T, we have the variational lower bound:
Togp(x117)
q q(11r|xx:7) log (():))))
/ q(2)))) ((1)r)x1e)) log
p(x 21.T)
dz
p(21.1 X1:1)
/ ((xI))xxiT) log
(Z X1.T)
dzi.T / ((Z)IT(xIT) log (( (1::)
p(XI:T, ZI.T)
dz dzi.r
p
21.T X1:T)
KL(g(Zz:T X (x.r)) + ((Z17)(X)T)))) log p(XIT, ZI:T) dz
[ X1:T)
/ ((z1T|xi:T) log
p(X1.T, Z1.T) dZ
(Z X
2 / ((11 (x:) log
p(XITT, Z1:T) (Z X1.T)
d21:r Lcoutrast:
where p(X1:T, B:r)) is the joint distribution as well as p(21:T(x1:T) and ((T) is the true posterior and the variational approximate posterior, respectively. The true posterior is intractable.
Considering Equations an 5
know that zi"" is deterministic values and h (6) is a function of x
and 2 (8)
Therefore, we have the factorization:
T
(i:T)
II
p(x,
Iz
1n 2/ x1-1 )p(z, X1 202-1)
(8)
T I
{( Xlit, x (1)
(9)
15

---

where 0(22 X1: 1
p 2,
(8)
(8) (c) |11-011 2 and p(xp|2 Zi:e x1:0-1) are the distributions
defined by Equations 2 and respectively. Based on the above factorization, we decompose the variational lower bound
f ((21:1/x1r)) log
P(IITT, Z1:T)
dz1:r
( (x1:))
- =ʃ (((2:r/Xx:))
T
pxx, [2] 21.0 x1# (p(2
(e)
X1:t- 1,
(8) dz
log
(10)
Xl:tx 111-11
Σ (f
(x log
p(x, 11:21 2 x x1
1p(2)
Xlit-11 2
2
dz1.
( X1:t, (i:t-1)
When we simplify the above log- likelihood a function g(Xi:t, Z1:1), we have:
-/ (1 ( XI.T, (1) (4ft-19((x1s)) 2 d1drr dZi.T-1
/ (1 B1d)) / a(2 XI.T, - )dzr
dZ11T-1
(11)
/
9 ((:1)))x: (x1. 21.)d21
Therefore, we further decompose Equation 10 as:
/ ((21T|xx:T)) log
p(XI.T, Z1:T) d21.T
q(21:T) x1:r)
( ((ana(x1))) log p(xe win 2 X1
)p(z) (1)
X1:0- (2)
2(001) da
q(z)" x1:e. 21:0-1
(2)
(111) 2 Zize X1 - )dzz
(e)
p(z,") (2) x1:t- 2 (((=1) dz
/ (((ene))ie))) log
g(z)")
X1:2; (
T
(1 (0g1 (z
Σ
1:23 2i0 X1:1 ddzz
(c)
(12)
/
X1:1, z 2 1lpp(t (1)
( 2
logp(x11210) (6)
KL(g(Z))) ( XXiee 2 1)pp))lp((2 (
X1:t-1; 1, 11) dai.
(2)
=Bgen.r Σ
log (x (0) x1xt-1)
KL(g(2)) Xx:e (0)
1)lpz) X1x-11 (1
16

---

Finally, for a given sequential data x, we have the loss function:
-logpxxlk 2 x1a-1) (L(((z))) (0) X1st, (id lp(a X
212--1)
+
+contrast
tm
reconstraction
regularization loes
(13)
where the first and second terms correspond to Lp and DKL in the main text, respectively.
Since we assume Poisson distribution for the inferred neural activity, Lp is the Poisson negative log-likelihood:
Lp(x),r)) log
r -t X₂!
-xx log ri + r + logxx! x₂!
(14)
log r, + r, x, logx, X, 2
log (2xx²).
As for DKL, under the assumption that both the prior and the approximate posterior are Gaussian, we have:
DKL(z) (2()) = -fote" ((2z)" X1.0, (12 log
q(2,
X1:t,
111-1)
(4)
dzi
(2)
p(zz
(6)
X1:6-
2 2
x
(1)
2 exp
2 2,c
= =f ((2) (x 2 2 log
d2i
(1)
21) Pha.e)
V 2.1
exp
20
(1(00
27
E )
11 %
E (1)
Z
log o,t + log o.t
20
25
1
+ + 11
+ R2 z,t
+
2,8
log az + log o.t
2
20%
(Matt P2,) log 2 + log à2
-1
2
(15)
Finally, we apply NT-Xeet loss as the contrastive loss:
sim (z(e) Zpos /T)
exp
Lcontrast llog exp
(16)
sim
(((e)
Zposs
(c)
/T]
+
Σ exp
sim (zle), zires /T)
where sim(+, is the cosine similarity and 5 is the temperature coefficient.
C Generating Procedure of Synthetic Datasets
First, we generate labels u; from four uniform distributions on 2ix (2f+1)xX) J, (0. 2,3,,, in preparation for building four clusters. Second, for each cluster, we sample 2-dimensional latent variables from independent Gaussian distribution with sin Us 5 cos u,) mean and (0.6 0.5] cos ui), 0.5| cos u,l) variance. Third, we feed sampled
Non-temporal dataset.
-
17

---

latent variables into RealNVP network [16] to form firing rates of 100-dimensional observations and generate the synthetic neural activity from the Poisson distribution. Each cluster contains 4,000 samples. This dataset, being label-dependent, is used to evaluate the models' ability to construct discriminative latent variables.
Temporal dataset. We generate three dynamic latent variables from the Lorenz system, consisting of 30 simulated neurons then computed by
of a set of nonlinear equations. The firing randomly weighted linear readouts from the Lorenz latent variables. Synthetic neural activity is also generated from the Poisson distribution. The hyperparameters of the Lorenz system follow a previous work [54]. We run the Lorenz system for Is (Ims for a time point) from five randomly initialized conditions. Each condition contains 20 trials. This dataset is for assessing the models' ability to represent temporal information.
D Characters of Mouse Visual Neural Dataset
The full names and abbreviations neurons for all chosen mice is also presented.
cortical regions are listed in Table and the number of
Table 3: Characters of the neural dataset.
Cortical Region
Abbreviation
Mouse
Mouse 2
Mouse 3
Mouse 4
Mouse 5
primary visual cortex
VISp
75
51
93
63
52
lateromedial area
VISI
39
30
56
38
20
rostrolateral
VISrl
49
24
58
44
41
anterolateral area posteromedial area
VISal
42
51
43
71
46
VISpm
62
90
17
19
64
anteromedial area
VISam
94
72
49
60
64
Number of Trainable Parameters of All Models
The number of model parameters is roughly proportional the number of input neurons. We present the number of parameters for the Mouse dataset in Table 4
Table The number of model parameters for the Mouse dataset.
LFADS
pi-VAE
Swap- VAE
CEBRA
TE-ViDS-small
TE-VIDS
Number of parameters
0.45M
0.49M
0.38M
0.71M
0.29M
0.68M
Training Setup
We list the information of training datasets and the hyperparameters of the model training in Table For each model, we perform the grid search for the learning rate (1.0 x 10-5 to 1.0 x 10 3) and the weights of each loss term (0.01 10) to achieve optimal performance on the validation set.
Table 5: The information of each dataset and the hyperparameters of the training.
Datnset
Training Size
Validation Size
Test Size
Latent Dimension
Leurning Rate
Optimizer
non- tereeeemporal synthetic dataset temporal synthetic dataset neural datasct under natural socnes
12800
3200
32
0.000S
Adam
80000
20000
8
0.001
Adam
14750
0.0001
Adam
visual neural dataset under natural movie
28800
3600
3600
128
0.0001
Adam
Furthermore, we present the preprocessing implementation of training samples for all models.
Visual neural dataset. For the neural activity under natural scenes, each sample input TE-ViDS sequential data from 5 time points and the offset of positive samples from target samples is within #3
18

---

time points. For the neural activity under the natural movie, each sample consists of 4 time points and the maximum absolute offset As for alternative models. pi- VAE and Swap- VAE take the neural activity of an independent time point an input sample. LFADS processes samples with the same length as our model. For CEBRA, following the original approach [48, we take the surrounding points centered on the target point to form a sequence (5 time points for natural scenes and 4 for the natural movie).
Non-temporal dataset. Since this dataset does not involve temporal relationships, each sample can be conceptualized sequence comprising a single time step. the context of contrastive learning for such data, the approach of defining positive pairs through temporal shifting is inapplicable. Instead, we adopt a strategy provided in CEBRA, selecting positive samples based on the distance of their respective labels, thereby implementing a form of supervised contrastive learning.
Temporal dataset. TE-ViDS and LFADS 50ms of data input, while the other models take data at one time point. The offset of positive samples is set to 5ms for our model.
All models for all datasets are trained on GPU (NVIDIA A100).
G Additional Results of Alternatives on Synthetic Non-Temporal Dataset
TH-ViDS-Small
pi-VAE
Latent
Latent
Latent 2
Figure 7: The inferred latent variables of alternatives on the non-temporal dataset.
H
Additional Experiment on the Visual Neural Datatsets
We train models with neural activity under approximately 80% natural scene/movie stimuli and evaluate them using "held-out" stimuli, i.e., completely unseen scenes/movies. As Tables and show, VE-V consistently achieves the best performance such challenging tasks.
Table 6: The decoding scores (%) for "held-out" natural scenes.
Models
Mouse
Mouse 2
Mouse 3
Mouse 4
Mouse 5
LFADS
48.96#1.13
34.78+1.88
39.22+42.47
38.17=1.60
15.74
VAE
23.57+3.17
37.04+2.42
41.91+1..0
20.70:42.05
10.35+1.10
Swap- VAE
Wwap-VAE
55.48::2.34
17.65
36.61+3.21
44.09:2.39
14.17
CEBRA
5.83++0.58
8.87++0.00
13.04++1.40
9.39-+0.77
4.96++0.69
TE-ViDS-smal TE-ViDS
62.61+2.50
41.30
47.57+3..06
50.52++2.77
23.30
65.91+1131
38.70+
46.52+2.09
53.57++1.88
24.52+
Table 7: The decoding scores (%, in is window) for "held-out" natural movie frames.
Models
Mouse
Mouse
Mouse
Mouse 4
Mouse 5
LFADS
38.8940.98
51.78
49.0612.04
46.28 .89
39.22+
Pi-VAE
38.8941.34
65.39-10.95
66.56+1.31
1..1 1..1
Swap-VAr VAE
3..22+1183
59.72+
58.33+1.62
57.72+11.77
44.50+
CEBRA
3..44+11004
54.39+1.41
52.89+0.84
48.8941088
3..89+1.660
TE- ViDD-Sllmlll
39.28+22.15
68.06+11.09
72.50+0.81
65.50+10.7
47.83+1447
TE-ViDS
42.00-10.52
66.28+1107
69.0640.84
66.28+11.43
1..00
19

---

I Additional Results of Latent Trajectories on the Visual Neural Datatsets
For the mouse neural dataset under natural scene stimuli. we visualize the latent representations by embedding them in two dimensions using tSNE (Figure(8). We focus on Mouse 1, for which most of the models achieve the highest scores, We select scenes that elicit the strongest average responses for visualization. Specifically, we reduce the dimensions of latent representations at single time point and take the average across all trials, to show the latent trajectories over time for each scene. The result of TE- ViDS exhibits clear temporal structure for different natural scenes. For LFADS, its ability encode temporal features of sequential neural activity results clear temporal structures, but the latent representations of different classes are largely intermingled. For pi- pi-VAF, Swap- VAE, and CEBRA, their latent trajectories show varying degrees of entanglement over time. These results suggest that our model effectively distinguishes between category information and captures temporal information from neural dynamics well.
For the mouse neural dataset under natural movie stimuli, in addition to Figures 4C-E, we visualize the results of all models and all three parts of the movie for Mouse (Figure) 9.
TEAIDS
TE-106
FFADS
yo
Staup
-10
Figure 8: Visualization results of latent trajectories on the mouse visual neural dataset under natural scene stimuli (Mouse 1).
TEAISS wwet
P-VAR
Soap VARl
CHRRA
(a) The first 10s
TEABS eaal
Seap
(b) The middle 10s
TEADDS-aad
(c) The last 10s
Figure 9: Visualization results of latent trajectories on the mouse visual neural dataset under natural movie stimuli (Mouse 2).
J Ablation Studies
We perform ablation studies for several aspects of TE- ViDS's components and neural activity input dimensions explore their impact performance.
The loss function and the recurrent module of VEDV..S. To show the effectiveness of the com- ponents of our model, conduct some ablation studies the loss function and the recurrent
20

---

Table 8: The decoding scores (%) of ablation studies for the loss function, the recurrent module and the split design of VE-ViDS on the mouse visual neural dataset under natural scene stimuli.
Models
Mouse I
Mouse 2
Mouse 3
Mouse 4
Mouse 5
TE-ViDS
50.86=1081
27.24
2..90=2.44
38.05:0.53
9.44
w/o negative samples
21.85+1 1.86
9.71=0.72
12.32+1.22
20.03++ 1.26
6.20+=0.43
w/o contrastive loss
30.17:11.74
8,24-1.33
13.76:0.87
23.42:10.99
7.19=0.28
w/o swap operation
28.80:10.86
14.27=0.42
18.92:40.93
15.97=0.85
4.68=0.22
contrastive loss and swap operation
6.8810.64
3.92:1022
4.17::0.22
6.58
2.56:1..12
with temporal independent prior
28.97+11.16
9.05++0.64
15.10+0.98
15.54+0.699
3,97+0032
GRU-Vanilla RNN
4..64+1.78
25.61=0.68
20.58
34.15+0.43
7.17+00.0
GRU-+LSTM
49.97=0.50
26.92=0.59
29.31+0.23
36.73=0,41
9.24=0.27
Non-recurrent
33.32+0.99
21.78=0037
22.63+0047
27.27+0.64
7.69+0.40
Extemal--nll
45.41+1.116
21.49+00.6
23.83++0.73
27.39+0.67
6,80++.23
Internal-Only
2.10+0.21
1.78:10.19
1.44
2.49-0222
1.46:10.12
Table 9: The decoding scores (%, in 1s window) of ablation studies for the loss function, the recurrent module and the split design of TE- ViDS on the mouse visual neural dataset under natural movie stimuli.
Models
Mouse
Mouse 2
Mouse 3
Mouse 4
Mouse 5
TE-ViDS
13.88-0.15
65.38+0.36
59.88+0.72
54.33++0.54
10.18+0.40
w/o negative samples
11327=0.36
4..59+
45.67+0060
44.17++0.42
19.93-+0.35
w/o contrastive loss
11.24=0.23
47.98=0.90
44.09=0.67
44.02=0.47
18.24=0.41
w/o swap operation
10.22.10.31
49.3910.57
45.30100.4
43.12.10.57
21.83
w/o contrastive loss and swap operation
9.16++0.37
24.84+1.10
22.33+0.881
26.49+00.6
12.02 22.02-+0.49
with temporal independent prior
12.09=0.116
57.7410.62
53.87:10449
46.90.10.45 48
22.92.10.43
GRU->Vanill RNN GRU-+ISTM
13.08-0.31
63.1910.55
59.1311..22
53.83110.41
29. 02.22-0.44 4040
12.77-0.22
64.69+0.53
6.600+0060
54.37 0.41
28.50+00.61
Non-rocurrent
11.14=0.30
03.26=0.48
48.31+0047
43.81=00.40
22.98=10444
External-Oni Internal-Oni,
12.16-0.24
63.3310.29
58.64110.77
52.11
30.24.10.54
7.57+00.24
11.73+0.43
12.86+0.35
10+0.33
9.89++.22
module (Tables 8 and 19. In terms of contrastive learning, we first exclude negative samples from the computation the contrastive loss and use only the cosine distance between the external latent variables of positive sample pairs as the loss function. In other words, we only bring the positive pairs closer. This results in decrease in performance, suggesting that negative samples are useful. Next, remove either the contrastive loss the swap operation and observe similar impact for both. Finally, we remove both, meaning that there are no more objectives related to contrastive learning. The significant decrease in performance suggests that contrastive learning plays a crucial role in models. In terms of the regular loss of the internal latent variables, we originally assumed that the prior distribution is deeeddeen..nt. When assume that it is independent standard normal distribution at each time step, the model's 's performance degrades, demonstrating that time-dependen assumptions about the prior also important. In terms of the recurrent module, the results suggest that GRU is a better choice when considering the trade-off between performance and computational efficiency. Besides, we evaluate non-recurrent version of our model by setting the time steps of GRU to 1, which demonstrates that the recurrent module plays a critical role.
The split design of latent variables. We build two models based on TE-ViDS, one with external variables only (External-Onie and the other with internal variables only (Internal-Only) Both models perform worse than TE-VIDS, confirming the split design's 's importance and effectiveness.
The dimension latent variables and the number of input neurons. We perform ablation studies on the number of latent variables and input neurons. As shown in Figures [0] and [1] first, the performance saturates gradually as the dimension the latent variables increases, which suggests that it is sufficient to choose a dimension in reasonable range (not much fewer than input neurons). Second, performance decreases as the number of sampled neurons decreases, suggesting that for each mouse, all recorded neurons contribute to the representation of visual stimuli.
21

---

Mouse
Mouse
Mouse
Mouse
Mouse
45000
19.55
Deoring Score
15
61 12E
95 Lotent Dimenssoo
160
56
1032
56 125
Loten Dumension
Lotent Dimension
Latent Uiiensionn
Latene Dimensiio
(a) Natural Scenes
Mouse
Mouse
Monse
Mouse
Mouse
Decoding (%)
64
21.5
45
25
150
M 195 Losent Dunzision
05 155
10115" "2""
"010"""""""""" "IS""T
Lusent Dineesioo
Latent Dimeniion
Lateet Dincesionn
Latene Dimension
(b) Natural Movie
Figure 10: The results of ablation studies on the dimension of latent variables.
Mouse
Mouse
Mouse
Mouse
Mouse
Deoddnn (o
12
22e Number of Input Neurona
Namber of Leput Neurons
Number of Tiput Neusona
Number Input Input Neunas
Number of Input Neurans
Natural Scenes
Monoo
Dooalnng Scoee
307197 Numberr Teput Nemoome
20 Number Neurons
Namber of Neunoro.
Number Iupui Naarns
Number of Impul Ntamua
(b) Natural Movie
Figure 11: The results of ablation studies the number of input neurons.
K
Additional Experiment on the Mouse Visual Neural Dataset from CEBRA
The neural dataset used CEBRA also preprocessed from the Allen Brain Observatory Visual Coding dataset. The dataset is generalized by sampling different numbers of neurons from the same visual cortical region of all mice, without taking into account the variability between subjects. We evaluate model this dataset. The results (Figure 12 show that model performs better the primary and mid-level regions, while CEBRA performs better on the high-level regions. Moreover, in the case of sample with 40 time steps (CEBRA reported their highest performance in this setting), our model can achieve a performance of more than 95% with fewer trainable parameters (TE-ViC ViDS: 0.44M: CEBRA: 1. 1.09)).
VISA
ViSai
ViSues
javer
Algiring Fila
nover lil
paap trll
PR
Wny 151 Neason
20
Namnber Neurons
Narbe: Neroon
Number Neurzas
Namther oftega Noures
Figure 12: The decoding scores (%, in Is window) on the mouse neural dataset used in CEBRA.
22

---

NeurIPS Paper Checklist
The checklist is designed encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No], or [NA].
[NA] means either that the question Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justification right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will published with the paper.
The reviewers of your paper will be asked use the checklist as one of the factors their evaluation. While "[Yes] is generally preferable to "[No]" ", it is perfectly acceptable to answer "[No] providedd proper justification is given (e.g.. "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering binary way, we
"[No] or "[NA]" is not grounds for rejection. While the questions are phrased acknowledge that the true answer is often more nuanced, please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS Paper Checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Please see abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]
Justification: Please Section
23

---

Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: The work does not involve proof of theory.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We detail all the techniques for reproducing the results of our work in Section and Appendix F
Guidelines:
The answer NA means that the paper does not include experiments.

---

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g.. with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome describe the particular way they provide for reproducibilityy In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: The codes and data are available atapllbiaa/n att/::///ghaaaaccc/////s//////// Visual- Biaall-Dymmiial--SSy -ystem
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.c cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https //tpps.cc/publicc///cccc
www.ded/eeooeeuummemccoo ssi ssionPolicy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
25

---

Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer:
[Yes]
Justification: Please Section and Appendix
F
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: Please see the results in all Tables and Figures.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by enror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Please see Appendix
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
26

---

The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics httpp::/neurrpst http:/rppw...pppuooooccccaccaaae
Answer: [Yes]
Justification: Our work conforms with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g.. if there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: Our work does not involve social impact.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Answer: [NA]
Justification: Our work does not release data or models with a high risk of misuse.
Guidelines:
The answer NA means that the paper poses such risks.
27

---

Released models that have high risk for misuse or dual-use use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: The datasets used in paper publicly available.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g.. website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com /dtaaset has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: Our work does not introduce new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moddl part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: Our work does not involve crowdsourcing experiments nor research with human subjects.
28

---

Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Our work does not involve crowdsourcing experiments nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMS if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification:
The LLM is used only for writing.
Guidelines:
The answer NA means that the core method development in this research does involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy /tap:::///eurraii cc/ //t.ooeeenncee/225/// for what should or should be described.
29


Reviews:
[{"review_id": "O7HKysB4NO", "reviewer": "Reviewer_NU2a", "summary": "The authors:\n- Introduce a new latent variable model (LVM) which employs two RNNs to encode spike trains from mouse visual cortex into two latent variable vectors\n- The first, ‘external’ latent variable is a deterministic representation of stimulus-dependent neural activity, while the second, ‘internal’ latent variable stochastically encodes the noisy component of the neural activity\n- The networks and their link functions are jointly trained to maximise the ELBO of the neural data under a time-dependent Possion generative model and a contrastive loss on the external latent variables\n- The positive examples for contrastive loss in the external latent space are selected by encoding a temporally offset input sequence; negative samples are randomly selected from the dataset", "questions": "- Please could the authors clarify, e.g. in the appendix, how this method was applied to non-temporal synthetic data? Namely, how was sequential data generated from the stationary latent variables, thereby allowing application of the contrastive loss?\n- An ablation is performed where the contrastive loss is removed. Did the authors consider another ablation where the negative examples are instead drawn from the same sequence but with longer time delays, akin to how negative samples are drawn for CEBRA? This may help isolate the importance of the chosen form of contrastive loss.\n- In section 4.5, the authors claim that qualitative differences in the RSA for mice 1 and 2 are because “sensory performance in [mouse 1] is strongly influenced by their internal state.” Do the authors expect that this internal state is represented in some way by their internal latent variable? For example, could we expect similarity in the internal latents in the two blocks of trials for mouse 1, but not for mouse 2? This would help illuminate the architectural choices of the model too.\n- Why does the decoding score concatenate vectors in the scenes case, but average them in the movies case?", "limitations": "There are no relevant negative societal impacts associated with this work.\n\nLimitations of the model are not discussed to great extent, nor are limitations of the analysis provided. \n\nThe authors focus on one neural dataset, so a small acknowledgement of the feasibility of application to other datasets, e.g. other mouse brain areas, other species, and other tasks, would help contextualise the study.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths\n- This is a very interesting, novel LVM, and the authors provide a convincing set of results comparing their model to existing LVMs, both with and without time dependence.\n- Furthermore, they provide an extensive range of ablation studies to their method, indicating the importance of each component of their full model. They also show that their model’s success is not due to its complexity, given that a smaller number of parameters also achieves state of the art performance on real and synthetic datasets.\n- Finally, they go beyond LVM performance metrics, and uncover representational similarities of neural representations which are not evident in the raw neural data.\n\n\nWeaknesses\n- Architectural choice wasn’t always clear to me. While I appreciate the division of representation between stimulus and internal noise, specific dependencies in section 3.1 were not fully motivated, and authors less familiar with existing cited models could benefit from an explanation of each component. Other explanations lack some clarity, for example, details of the swapping operation and the intuition for its benefit to training.\n- Without extra details, potentially provided in cited works, it is not obvious to me how the model was applied to non-temporal data in the first place. Both the generative model and the form of the contrastive loss function require a sequence of activity, \nIn section 4.4 it is stated that “Swap-VAE and CEBRA [suffer] a slight degradation due to the use of time-jittered positive samples.” Did the authors mean SWAP-VAE and pi-VAE - as it appears from Fig. 2E that CEBRA suffers a drastic decrease in performance due to time shuffling.\n- In section 4.6, the authors claim that “our model achieves significantly higher 273 decoding scores than LFADS and is slightly better than the others” - it would be helpful to provide details on statistical tests they performed to qualify this statement.\nIn its current form, I find the claim that the model “provides new evidence for the functional hierarchy of the mouse visual cortex” too strong. The corresponding section 4.7 simply provides a variety of performance metrics for various parts of mouse visual cortex, but does not provide sufficient evidence to support a functional hierarchy. Such a claim would require, e.g., equivalent analysis for multisensory stimulus exposure experiments.\n- Alternatively, finding a way to cross-validate the model trained one brain region in encoding activity from another region may help reveal functional dependencies between them. One method could be to train the model on one region, then retrain only the encoder and decoder weights on a second brain region, to ascertain whether the regions share similar time-dependencies.", "was_revised": true, "final_justification": "As per my response to rebuttal 1, the authors intend to extend their results to fill the gaps I identified in my review. As such, I am strengthening my recommendation for acceptance.", "rebuttal": "We thank the reviewer for the constructive and detailed comments. We will do our best to address the comments and answer the questions. Below are our detailed responses.\n\n**R1. Architectural choice and swapping operation.**\n\nOur model leverages the architecture of sequential VAEs, with many component dependencies derived from the classic model VRNN. Specifically, in the encoder, the external latent variables $z^{(e)}$ and internal latent variables $z^{(i)}$ are conditioned on the input neural activity, as well as the respective state variables ($h^{(e)}$ and $h^{(i)}$) of two distinct RNNs. This conditioning facilitates the capture of temporal information in a causal order. The prior distribution of $z^{(i)}$ is conditioned solely on the state variables. In the decoder, the reconstructed neural activity is conditioned on the $z$ and $h^{(i)}$. The deliberate inclusion of $h^{(i)}$ here is intended to enrich the information of neural dynamics, providing a more precise reconstruction. Finally, regarding the update for the RNN state variables, $h^{(e)}$ depend solely on the input neural activity and their values from the previous time step. Differently, $h^{(i)}$ depend not only on the input neural activity and their previous time-step values, but also on the latent variables. This design choice reflects a consideration that animals' internal states are susceptible to the influence of current external visual stimuli.\n\nThe swapping operation for details is as follows: For a given sample and its corresponding positive sample, we exchange their external latent variables while keeping their internal latent variables unchanged. This process yields two new latent variables. We then decode neural firing rates using these new latent variables and compute the additional reconstruction loss. The intuition is this: our model is trained to reduce the distance between the external latent variables of positive sample pairs. Consequently, if the model has learned an effective external latent space, it should still accurately reconstruct neural activity even after this swap. This robust reconstruction, even with interchanged external latent variables from positive pairs, significantly enhances the training process by imposing strong regularization and encouraging more meaningful latent representations.\n\nWe will include the necessary clarifications in our revised version.\n\n**R2. How the model was applied to the non-temporal dataset.**\n\nFor the model input of non-temporal datasets, each sample can be conceptualized as a sequence comprising a single time step (as described in Appendix F). In the context of contrastive learning for such data, the approach of defining positive pairs through temporal shifting is inapplicable. Instead, we adopt a strategy provided in CEBRA, selecting positive samples based on the distance of their respective labels (Appendix C), thereby implementing a form of supervised contrastive learning.\n\n**R3. The sentence on line 232.**\n\nWe apologize for the imprecision in the original statement on line 232. Our intended meaning was that, while both our model and LFADS exhibited substantial performance degradation, the performance reduction observed in Swap-VAE and CEBRA was comparatively less severe (although CEBRA's decline was still notable). We will revise the sentence to clarify this distinction.\n\n**R4. Statistical tests for the sentence on line 272.**\n\nFollowing the reviewer's suggestions, we conducted a one-sided t-test comparing the performance of our model against baseline models, with the alternative hypothesis stating that our model's mean performance is superior. The results (t-statistic > 10; p-value < 1e-5) demonstrate that our model consistently exhibits significantly higher performance than all other models across nearly all mice, with the sole exception being the comparison with CEBRA in Mouse 3. The intent of the description on line 272 is to emphasize the differential magnitude of performance gains, specifically, that our model's advantage over LFADS is greater than that over the other models.\n\n**R5. The experiment for mouse cortical regions.**\n\nIn the current results (Figure 5), performance variations across brain regions, under differing visual stimuli, exhibit similar patterns. This at least indicates heterogeneous and specialized visual encoding information across each brain region. However, we acknowledge that the initial conclusion regarding a sequential functional hierarchy was overly strong, and we will temper this statement in the revised version.\n\nFollowing the reviewer's suggestion, we fine-tuned a model pre-trained on one region to another. This involves training only the first encoder layer and the last decoder layer, while freezing all other parameters. Tables r1 and r2 present these results, with rows indicating the pre-trained region and columns the fine-tuned region. We observe that even when pre-trained in different brain regions, the performance of the fine-tuned model shows no significant difference. The performance is lower than that of a model trained directly on the target region. This phenomenon suggests that neural activity across different brain regions may share similar latent spaces but encode distinct visual information.\n\nTable r1: The fine-tuned results (\\%) for decoding natural scenes.\n\n||VISp|VISl|VISrl|VISal|VISpm|VISam|\n|-|:-:|:-:|:-:|:-:|:-:|:-:|\n|VISp|39.68±1.83|18.88±1.83|4.00±0.18|24.00±1.30|12.95±1.26|9.08±0.80|\n|VISl|25.76±0.60|20.07±1.15|3.22±0.44|23.93±1.49|9.46±0.79|7.46±0.77|\n|VISrl|23.73±0.90|15.29±1.13|5.64±0.31|20.00±1.53|9.69±0.84|7.69±0.56|\n|VISal|25.76±0.67|17.53±1.79|4.64±0.48|27.97±1.23|12.27±1.87|8.24±0.54|\n|VISpm|25.05±1.74|14.00±0.73|3.59±0.45|19.53±1.54|13.42±1.00|7.66±0.83|\n|VISam|24.37±1.84|16.31±1.47|4.75±0.27|22.68±1.02|11.29±0.48|10.22±0.53|\n\nTable r2: The fine-tuned results (\\%) for decoding natural movie frames.\n\n||VISp|VISl|VISrl|VISal|VISpm|VISam|\n|-|:-:|:-:|:-:|:-:|:-:|:-:|\n|VISp|35.33±1.79|24.44±0.76|15.33±0.84|29.84±1.87|22.60±0.88|23.56±1.07|\n|VISl|28.71±1.63|28.71±0.88|13.96±0.74|31.53±1.21|20.82±0.74|22.80±0.97|\n|VISrl|28.62±1.56|24.44±0.49|15.78±0.61|30.44±1.27|21.73±0.75|23.33±0.70|\n|VISal|28.64±1.52|24.44±0.91|14.60±0.72|38.80±1.97|20.07±0.64|25.09±1.56|\n|VISpm|28.73±1.54|24.09±0.62|14.00±0.34|31.09±1.13|25.07±1.51|23.04±0.96|\n|VISam|31.18±1.72|23.51±0.96|14.24±0.25|32.80±1.17|21.02±1.08|27.20±0.91|\n\n**R6. Ablation study about the negative samples.**\n\nIn response to the reviewer's suggestions, we investigate an alternative negative sampling strategy: drawing negative samples from the same sequence but with longer time offsets. Specifically, for the scene dataset, we use a 100 ms offset for negative samples (compared to 30 ms for positive samples). For the movie dataset, negative samples are generated with offsets of approximately 330 ms and 165 ms (versus an approximate 16 ms offset for positive samples). As detailed in Tables r3 and r4, this negative sampling approach results in a slight performance decline, with a larger decrease observed when the negative sample offset is smaller. These results indicate the feasibility of this alternative strategy, yet they also highlight a trade-off: selecting appropriate offsets for different datasets increases the computational cost associated with hyperparameter search. Our current methodology, which involves randomly selecting negative samples from the entire training dataset, emerges as a more generalized and efficient approach. Besides, we carefully review the code of CEBRA, confirming that it adopts the same random sampling strategy.\n\nTable r3: The ablation results (\\%) for decoding natural scenes.\n\n||Mouse 1|Mouse 2|Mouse 3|Mouse 4|Mouse 5|\n|-|:-:|:-:|:-:|:-:|:-:|\n|Random|50.86±0.81|27.24±0.47|29.90±0.43|38.05±0.53|9.44±0.20|\n|Longer offset (100 ms)|48.20±0.48|26.46±0.52|27.49±0.32|36.47±0.39|8.31±0.23|\n\nTable r4: The ablation results (\\%) for decoding natural movie frames.\n\n||Mouse 1|Mouse 2|Mouse 3|Mouse 4|Mouse 5|\n|-|:-:|:-:|:-:|:-:|:-:|\n|Random|13.88±0.19|65.38±0.36|59.88±0.72|54.33±0.54|30.18±0.40|\n|Longer offset (300 ms)|13.00±0.31|63.96±0.47|59.37±0.84|53.78±0.45|29.37±0.54|\n|Longer offset (165 ms)|12.83±0.29|63.24±0.70|58.58±0.41|52.40±0.53|28.62±0.49|\n\n**R7. Do the authors expect that this internal state is represented by their internal latent variable?**\n\nIn fact, the RSM of mouse 1 in Figure 3B has already shown regular changes in the internal state, i.e., the two redder blocks in the top-left and bottom-right corners, as discussed in lines 255–258 of the text. Upon checking the RSM of the internal latent variables, we observe that this pattern is even more pronounced. Subsequently, a quantitative assessment of the similarity between two blocks of trials, performed using the Pearson correlation coefficient, reveals that the similarity for Mouse 1 (0.248) is much higher than that for Mouse 2 (0.022). This finding provides further corroboration for our statement that visual sensory performance is influenced by internal states.\n\n**R8. Why does the decoding score concatenate vectors in the scenes, but average them in the movies?**\n\nThe presentation duration of static scenes (250 ms) necessitates the preservation of dynamic changes within their latent representations for effective stimulus discrimination. Consequently, a concatenation operation is employed to retain maximal temporal information. In contrast, the frame rate of the movie is 30 Hz, meaning that each movie frame lasts only about 33 ms. Using an averaging operation for the latent representation of a single movie frame does not result in much information loss but can reduce the computational complexity of KNN algorithms.\n\n**R9. Limitations.**\n\nThe main limitation is the inability to quantitatively analyze internal latent variables reflecting internal states. Moreover, as noted in lines 332-334, extending our method to other brain regions and species represents a promising avenue for future exploration."}, {"review_id": "E5TqDnFFb2", "reviewer": "Reviewer_RtPx", "summary": "The paper presents a dynamical latent variable model that separates stimulus-related and internal neural dynamics using a hybrid VAE and contrastive learning framework. The model consists of two distinct latent pathways: a deterministic external latent space to capture stimulus-driven variability, and a stochastic internal latent space to capture trial-specific dynamics. A key feature is the use of contrastive learning and a swap-based reconstruction loss to enforce a functional separation between these two latent spaces. The model is evaluated on neural recordings from mouse visual cortex and the authors claim it uncovers both representational structure and hierarchical processing.", "questions": "Below is a list of questions, concerns, and suggestions that I had while reading the paper. Some relate to specific modeling choices and assumptions, while others concern interpretation and framing of the results:\n\n**Questions for which I am happy to increase the rating if properly addressed:**\n\n- What are the exact “new insights” that this paper is providing “into the intrinsic correlation between neural activity and visual stimuli” (line 54), as the authors mention in the intro? same question for “visual information processing mechanisms of the mouse visual cortex”? I agree the method has potential for being used to generate insights, but if you are claiming that it is providing the insight then the insight should be provided, rather thoroughly and convincingly. I am specifically curious about the “new” insight, as opposed to confirming previous findings (maybe even with better quality) or improved performance.\n- The authors motivate the choice of making the external latent deterministic in biological terms (i.e. “to reduce the noise inherent in neural activity”). Why is that desired, from the perspective of modelling biological data? Neural responses to the same stimulus exhibit trial-to-trial variability that is not fully attributable to latent brain states or structured noise (i.e. individual neurons exhibit intrinsic variability), so a biologically faithful model would allow $\\mathbf{z}^{(e)}$ to reflect some uncertainty.\n- The decoder takes $\\mathbf{h}_{t-1}^{(i)}$ as input in addition to the latents $\\mathbf{z}_t^{(e)}$ and $\\mathbf{z}_t^{(i)}$. If the latents are meant to capture the full structure of the neural dynamics, what is the justification for passing the RNN hidden state upstream of $\\mathbf{z}_t^{(i)}$ into the decoder? To me, this blurs the role of $\\mathbf{z}_t^{(i)}$ as a sufficient representation of the internal factors that effect high-dimensional neural data. Moreover, if the hidden state of the internal RNN is necessary, why not include the hidden state of the external RNN as well?\n- How using denoised responses (i.e. using the predictions of any powerful predictive model) and computing RSM is “facilitating the unraveling of information processing mechanisms in the visual cortex”? The motivation for comparing the RSMs across two mice (Fig. 3B) was the observed difference in decoding score (i.e. mainly concerned with external factors), but the explanation is more concerned with internal states. The way I understood the explanation is that it looks like the two mice have different internal states and therefore the decoding score is different across subjects. First of all, I do not find this argument convincing, and even if I was convinced, I am not sure if this qualifies as “new insights into differences in visual processing between subjects” (mentioned in the abstract).\n- Regarding the explanation of the observed diagonal structure across all video trials (Fig. 4F), I agree that this indicates the mouse neural representation is driven by stimulus-related responses—especially since internal states are expected to vary across trials. The persistence of the diagonal structure thus likely reflects the dominance of the stimulus-driven component. To support this claim more directly, it would be helpful to see RSMs computed separately on the external and internal latents. The expectation is that the RSM of the external latent should preserve the diagonal structure across trials, while this pattern should be absent in the RSM of the internal latent.\n- I find the interpretation of the decoding scores across visual areas in Fig. 5 somewhat unclear. First, while the authors suggest that lower decoding in VISrl supports a hierarchical organization, they simultaneously attribute it to multi-sensory integration, which weakens the hierarchy argument. Second, if the cortex is indeed hierarchically organized, one might expect higher decoding scores in high-level areas due to more specialized and invariant representations. Instead, the highest decoding appears in VISp, the primary area, which seems counterintuitive if semantic discriminability is the goal. Could the authors elaborate how these results support a hierarchical structure, and how one might disentangle the effects of hierarchy from those of multi-modal integration?\n- The claim that the results amount to a “potential breakthrough” seems overstated. While I agree that the model may offer useful insights into sources of trial-to-trial variability, their interactions, and their temporal structure, the evidence does not clearly demonstrate a substantial advancement over prior work in uncovering computational mechanisms in visual areas. I suggest rephrasing this statement to more accurately reflect the scope of the contribution.\n\n**Other (minor) questions/suggestion:**\n\n- Could the authors explain how using the contrastive learning objective along with the swap reconstruction loss enforces a functional separation between the two latent factor sets (i.e. $\\mathbf{z}^{(i)}$ and $\\mathbf{z}^{(e)}$)? What is the contribution of each of these losses?\n- It would help the reader if the authors highlight the main differences between their approach and those mentioned in the related work (at least the most relevant ones that they use for model comparison). Some differences are mentioned (e.g Swap-VAE assumes independence over time) but for LFADS and CEBRA there is only a one sentence description and it is not immediately clear what the difference is (e.g. no internal-external factorization in LFADS). A more clear contrast of important aspects can help understanding the methodological contribution of the presented method.", "limitations": "Although the authors indicate in the checklist that limitations are discussed in Section 5 (Discussion), I could not find any explicit mention of limitations in the text. However, below are some limitations that I think are worth discussing (as mentioned also in my questions):\n\n- The deterministic nature of the external latent may artificially suppress trial-to-trial variability that is genuinely stimulus-related.\n- The evaluation focuses primarily on decoding and RSM structure, with limited connection to actual mechanistic hypotheses about neural computation. For instance, do the learned internal latent trajectories reflect known internal states (e.g., arousal, locomotion)? Or do the external latent encodes interpretable stimulus properties?\n- The interpretation of subject differences and hierarchical processing are speculative", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths:**\n\n- The text is clearly written with a logical flow and was easy to follow\n- Proposes a novel combination of contrastive learning, latent swapping, and variational sequence modeling.\n- Explicit modeling of internal vs. external factors in neural data is conceptually appealing and clearly structured.\n- The swap-based reconstruction loss is a clever mechanism for enforcing disentanglement.\n\n**Weaknesses:**\n\n- Several modeling choices (e.g., deterministic external latents, use of internal GRU state in the decoder) are not clearly justified given the biological modeling goals.\n- Interpretations in the results sections often overreach the evidence presented (e.g., \"potential breakthrough\") and claims about “new” insights into neural processing mechanisms and hierarchical structure are not strongly supported by the presented analyses.", "was_revised": true, "final_justification": "Considering the authors’ responses to my concerns and their discussion with the other reviewers, I believe the merits of this work outweigh its limitations, and I have decided to maintain my positive evaluation.", "rebuttal": "We thank the reviewer for the thoughtful and detailed comments. We will do our best to address the comments and provide detailed responses point by point, below.\n\n**R1. About new insights.**\n\nOur work offers two key insights into visual information processing mechanisms in the mouse visual cortex. First, the encoding capabilities for visual information are influenced by the animals' internal state, potentially accounting for variability in neural activity both across individuals and within the same individual over time. Second, under long-duration continuous visual stimuli, the neural activity of mice becomes more focused on encoding visual stimuli and is less influenced by their internal state. Further details on these points are elaborated in R4 and R5.\n\nWe acknowledge, however, that certain phrasing, such as claims of providing \"new insights\" into the intrinsic correlation between neural activity and visual stimuli, was imprecise. This should be revised to state that our model captures a greater degree of correlation between neural activity and visual stimuli. We will implement these necessary modifications in the revised manuscript.\n\n**R2. About the deterministic external latent variables.**\n\nWe agree that factors beyond neuronal noise and internal brain states contribute to trial-to-trial variability. However, spontaneous neural activity, which is highly correlated with internal states, is recognized as a predominant source of cortical variability [1].\n\nTo comprehensively model variability involving external inputs, it would be necessary to incorporate auxiliary information, such as neural activity from other cortical regions (e.g., encoding other sensory inputs, behaviors, or memories). Given that this work analyzes neural activity within visual regions, an attempt to rashly model such externally driven variability (e.g., by pre-specifying the variance of external latent variables) risks both an ineffective representation of this variability and a detrimental impact on the modeling of internal state-related latent variables.\n\nTherefore, to maintain our focus on variability arising from internal brain states, we have opted to simplify the modeling of external latent variables as deterministic values.\n\n**R3. About the inputs of decoder.**\n\nWe admit that the choice of state variables as input to the decoder is primarily motivated by engineering considerations. First, following the architectural dependencies established in the VRNN, we introduce internal state variables in the decoder to further enrich the dynamic information, extending beyond what is captured by internal latent variables, thereby facilitating a more accurate reconstruction of neural activity. Second, since the external latent variables are deterministic functions of external state variables, introducing external state variables into the decoder yields less information gain. Therefore, we omitted this inclusion to mitigate computational complexity.\n\n**R4. How computing RSM is \"facilitating the unraveling of information processing mechanisms in the visual cortex\"?**\n\nExternal latent variables reflect neural activity components related to visual stimuli; consequently, their performance in decoding these stimuli cannot surpass the inherent validity of the visual information encoded by the neural activity itself. Therefore, observed differences in decoding scores across individual mice are more plausibly attributed to differences in the visual information encoded by their respective visual neural activity, rather than being a direct consequence of the external latent variables. More abstractly, these variations may be linked to the visual sensory capabilities of different mice at various times.\n\nBuilding upon this premise, our analysis of the RSM reveals distinct patterns of internal state changes (as reflected by the internal latent variables) across mice, as detailed in lines 255-260 and further elucidated in R7 to Reviewer NU2a. We thus hypothesize that such inter-individual differences in sensory performance may be influenced by the internal state of each animal, potentially serving as a significant factor contributing to inter-individual variability. This investigation into the influence of internal states on sensory performance supports our broader aim of exploring and explaining the mechanisms of visual information processing across different individuals.\n\n**R5. About the Figure 4F.**\n\nAs suggested by the reviewer, we compute the RSM for external latent variables and internal latent variables separately. As expected, the RSM for external latent variables consistently exhibits a clear diagonal structure, even for different trials. In contrast, the RSM for internal latent variables lacks such a structure, displaying no consistent organization across trials. These findings further support the hypothesized distinct roles of the external and internal latent variables in encoding different aspects of neural activity. We will incorporate these additional results into the revised manuscript.\n\n**R6. The interpretation of the decoding scores across areas.**\n\nFirst, our conclusions regarding hierarchical organization are primarily drawn from observed differences in decoding performance between primary and high-level cortical regions (lines 310-313). We wish to clarify that the discussion concerning the lowest decoding performance in VISrl (a multisensory brain region) serves merely as a potential explanation for this specific observation, rather than as an argument for hierarchical organization. Second, we offer the following explanation for the superior decoding performance observed in VISp. The neural datasets used in our experiments include categories with similar semantic content (e.g., various natural scenes featuring the same animals, or movie frames that are temporally distant but depict similar characters). Consequently, high-level cortical regions (VISpm and VISam), which primarily encode high-level visual features such as semantic information, may struggle to distinguish these subtle categorical differences. Conversely, the primary visual cortex (VISp), which encodes low-level visual features like texture and orientation, is better equipped to discriminate these fine-grained distinctions.\n\nNevertheless, we acknowledge that our initial conclusions regarding functional hierarchy were overly strong. The current results primarily suggest that the visually encoded information is heterogeneous across different mouse cortical regions. We will temper our conclusions accordingly in the revised manuscript. Furthermore, the interactive effects of hierarchical structure and multimodal integration represent a compelling avenue for future exploration, likely necessitating a synergistic analysis of neural activity across multiple sensory cortices.\n\n**R7. About the potential breakthrough.**\n\nIt appears there may be a misunderstanding regarding the \"potential breakthrough\" referenced in lines 330-331. We wish to clarify that this phrase pertains to a promising direction for future research—specifically, the challenge of constructing stable, high-quality latent representations across diverse experimental conditions. It is not presented as a contribution of our work. In fact, the observed variability in our model's decoding performance across different mice is a recognized limitation of this work, underscoring the need for further investigation into robust modeling.\n\n**R8. What is the contribution of each of these losses?**\n\nOur approach achieves disentanglement between external and internal latent variables through multiple training objectives.\n\nFirst, as described in lines 132–141, contrastive learning is applied to the external latent variables. This mechanism encourages external latent variables corresponding to sample pairs with small temporal offsets (i.e., under similar visual stimuli) to cluster together, thereby establishing an association between these variables and the visual input. Concurrently, a time-dependent prior distribution with spontaneity is used to guide the internal latent variables via the KL divergence. The application of these distinct loss functions to the two types of latent variables is instrumental in promoting their disentanglement.\n\nSecond, the swapping operation (details in R1 to Reviewer NU2a) further reinforces this disentanglement. This operation involves exchanging the external latent variables between positive sample pairs while preserving their internal latent variables. This manipulation specifically aims to further reduce the distance between the external latent variables of positive pairs. By doing so, it compels the model to isolate and strengthen the distinct roles of the external and internal latent variables, thereby enhancing their disentangled representation.\n\n**R9. The main differences between our model and alternative models.**\n\nWe have provided a brief description of the differences in Section 4.3. Here, we elaborate on the main differences between our proposed model and alternatives:\n\n* pi-VAE: Unlike our self-supervised approach, it incorporates additional supervised information (specifically, visual stimulus categories) for latent variable construction. Additionally, pi-VAE does not model the temporal dynamics inherent in neural activity.\n\n* Swap-VAE: Similar to pi-VAE, Swap-VAE does not explicitly model the temporal information of neural activity, operating under an assumption of temporal independence.\n\n* LFADS and CEBRA: A key distinguishing feature is that neither LFADS nor CEBRA performs a decomposition into external and internal latent variables. Moreover, the temporal relationships modeled by these approaches are not strictly causal. The former employs bidirectional RNNs to capture temporal relationships, while the latter uses temporal convolutional kernels to extract time information.\n\n[1] Spontaneous behaviors drive multidimensional, brainwide activity. Carsen Stringer, et al. Science 2019."}, {"review_id": "dZxtLJUxro", "reviewer": "Reviewer_iXM1", "summary": "This work presents a latent variable model for time-varying neural activity in mice, using an encoder-decoder architecture with a temporally evolving latent space split into two components: “external” and “internal.” The external component is a deterministic function of neural activity, aimed at capturing stimulus-dependent information, while the internal component is stochastic and intended to capture neural variability. The model is trained using a combination of a variational lower bound and a contrastive loss. On synthetic datasets, it shows improved latent and stimulus reconstruction. Applied to the large-scale Allen Brain Observatory Visual Coding dataset, the model outperforms baselines on natural scene and movie decoding, where decoding is framed as a scene/frame classification task.", "questions": "1. Why use the contrastive loss and only on z^e? It’s not clear to me what the motivation there is precisely.\n\n2. I understand that the combination of the reconstruction term and KL divergence forms the standard ELBO. However, it’s unclear how the full objective, which includes the additional contrastive loss, is still considered an ELBO. In Appendix B, you derive the ELBO and then add a negative contrastive loss term, but it’s not clear whether the ELBO inequality still holds under this modified objective. As I understand it, the contrastive term is not necessarily independent of the ELBO (z^{e} appears in the reconstruction loss)—so minimizing the contrastive loss could potentially conflict with maximizing the ELBO. It would be helpful if you could clarify how these objectives interact and whether the resulting objective still maintains a valid ELBO interpretation.\n\n3. There seems to be substantial variation in decoding performance across the different mice (esp mouse 5 vs rest in Figure 3 and mouse 1 vs rest in Figure 4). Do you have any hypotheses as to why this might be the case? \n\n4. What are the main limitations of your model and approach? What future directions are you considering? Are there specific neuroscientific questions you aim to address with your model?\n\nPlease also see questions under \"weaknesses\".", "limitations": "Limitations of the model and approach are not discussed. Please see my question 4.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "### Overall assessment\n\nEdit: note that this assessment was written prior to the rebuttal phase. See below for final justification.\n\nThis paper presents a promising direction for latent variable modeling of neural activity but is currently significantly limited by unclear motivation, under-explained methods, and insufficiently rigorous evaluation.\n\n**To help me raise my scores**, I encourage the authors (where feasible) to: **(1)** clarify the model architecture and its novelty; **(2)** justify the use of the contrastive loss and its interaction with the ELBO; **(3)** substantiate the choice of simulated data and corresponding evaluation; **(4)** strengthen the evaluation with more rigorous metrics (e.g., log-likelihood/ELBO on spikes), justify the decoding setup; **(5)** clearly articulate the contribution relative to prior work, and importantly **(6)** reflect and discuss the paper's limitations.\n\nDetails below.\n\n## Strengths\n\n- This work tackles an important and timely problem in computational neuroscience and machine learning—analyzing time-varying neural activity in response to natural and complex visual stimuli—by developing a latent variable generative model of the neural activity. \n- The model explicitly separates stimulus-related and internally driven components in a temporally evolving latent space leading to more structured and interpretable representations of neural activity. \n- The paper presents results on the publicly available, large-scale Allen Brain Observatory Visual Coding dataset consisting of neural activity across six different cortical regions and five different mice, recorded using modern Neuropixel probes. It also includes comparisons with several existing latent variable models, making it overall a suitable benchmark for evaluating approaches to modeling mouse visual neural activity. \n\n## Weaknesses\n\n**Summary**: The core contributions, motivation, and methodological choices are often unclear or underexplained. Key concerns include vague problem framing, limited detail on synthetic datasets, coarse decoding evaluation with scalability issues, and missing assessments of spike reconstruction and internal latent utility. The writing would also benefit from revision for clarity, grammar and style.\n\n1. **Lack of clarity in problem motivation, methodological contribution, and model positioning:** In the initial sections—abstract, introduction, and related work—I found it difficult to identify the core methodological contribution, the specific scientific question being addressed, and the gap in prior work that the model aims to fill. While these sections hint at high-level goals, such as using latent variable models (LVMs) to capture correlations between neural activity and visual stimuli or behavior, they do not clearly present the motivation for applying LVMs in this context, provide a concise description of the proposed approach, or explain how it meaningfully differs from existing models. The rationale behind key design choices, particularly in relation to modeling neural activity to visual stimuli, is also unclear. A more focused and structured presentation of these elements before diving into the technical details would greatly enhance the clarity and accessibility of the paper.\n2. **Lack of clarity and motivation in model architecture and proposed methodology:** In Section 3, the method is not clearly motivated or presented in terms of the model architecture and the specific relationships between variables. While Figure 1 offers a helpful high-level overview, it omits key variables such as $h_t$, $x_t$, $z_t^e$, $z_t^i$, and $r_t$, which are central to the model. Although the text outlines how these variables interact, the rationale behind their roles and the choices made is not clearly explained, making it difficult to develop an intuitive understanding. The appendix provides a graphical representation of the model, but it lacks a descriptive summary of the relationships among these variables and the motivation for these specific design choices, especially given that the model being proposed is quite complex. The same goes for the loss; why exactly include the contrastive loss? For clarity and completeness, I would recommend the authors include a concise and self-contained explanation of the model and its components directly in the main text, even if some fundamental choices are based on previous work. Important, why are these choices central to the context of neural activity in response to visual stimuli (if they are intended as such) and how are they exactly different from the baselines considered? This would also substantially help understand these specific baselines were chosen, which at the moment feels not well-justified.\n3. **Synthetic data lacks sufficient detail**: The descriptions of the synthetic datasets lack essential detail. The current explanation—“[Line 166] Synthetic datasets roughly point to two properties of visual neural activity…”—is vague and uninformative, and does not clarify what the synthetic data actually looks like, how it’s constructed, or what aspects of visual neural activity it is intended to model. \n4. **Unclear whether synthetic results evaluate the model’s core contributions:** When it comes to the results, while they show that their model is superior in reconstructing the true latents and the observations, it is not clear whether it tests their model’s novelty — be it the split internal-external latent variables or their loss functions — or its relevance to neural activity to visual stimuli. It’s difficult to judge how good of an evaluation this represents for their proposed model.\n5. **Decoding task treated as coarse grained classification with limited scalability:** In their main evaluation on real data, the authors frame the task as decoding natural scenes and movie frames, but in practice, they treat it as a classification problem: predicting which labeled image (out of 118 natural scenes) or movie frame (out of 900) was shown to the animal. For movie frames, predictions are considered correct if they fall within +/-1 second—about 30 frames if you consider 30 fps—making this a coarse-grained temporal task rather than frame-level decoding. This approach also doesn’t scale well to longer videos or larger scene sets that are typical, where the label space becomes unwieldy. Additionally, choices like searching for K (in their KNN) in odd numbers from 1 to 20 are ad hoc and not well justified.\n6. **Missing evaluation of spike reconstruction and internal latent utility:** While it is encouraging to see the model outperform all baselines on scene and frame decoding, with the external latent clearly showing higher stimulus decodability, the paper does not evaluate the model’s ability to reconstruct spike counts. Given that the core task is to model the distribution of spike counts—albeit through latent variables—it would be more rigorous to report the log-likelihood (or ELBO) on held-out spike data and compare it to other models that support such evaluation. Additionally, the paper lacks a benchmarking task to assess whether the “internal” latent component captures meaningful structure. Showing that it does not aid stimulus decoding seems to me insufficient to demonstrate its utility or interpretability.\n7. **Scope for improvement in written clarity and style:** The paper would benefit from careful editing to better align with the tone and conventions of scientific writing. Many sentences feel unwieldy and imprecise:\n\n(i) “[L38–39]: *Given that revealing the correlation between neural activity and visual stimuli is a challenging research highlight in the neuroscience community, it is desperately necessary to develop a strong model for extracting vision-related latent representations.*”\n\n(ii) “[L48]: *The external latent representations aim to magnify stimulus-relevant components of visual neural activity, while the internal latent representations point to dynamical internal states that have an impact on the animal’s perceptive capability.*”\n\n(iii) “[L117] *The high-quality latent representations are compressed information on the original neural activity, so they are needed to reconstruct the input neural activity well.*”\n\nIn above examples, the phrasing could be made more concise, neutral, and aligned with typical scientific expression.", "was_revised": true, "final_justification": "The authors have addressed concerns about the work’s positioning in the literature, provided explanations for model and loss design, and agreed to rewrite parts of the model description for clarity. They have also extended their evaluation by including log-likelihood scores (ELBO) showing superior performance over comparable models in the literature.\n\nHowever, three concerns remain. First, the quality of the learned internal latent variable is unclear—while it shows a weak correlation with stimulus, the work could be strengthened by also measuring correlations with observed behaviors and demonstrating a stronger trend. Second, the choice of synthetic datasets is relatively simple; more expressive datasets would better test the model’s ability to uncover latent factors alongside stimulus information. Third, although the authors argue that adding the contrastive term to the loss does not violate the ELBO, this still needs to be formally proven or reproduced from prior work.\n\nOverall, I find the work valuable and would increase my score commensurate with how effectively these points are addressed.", "rebuttal": "We thank the reviewer for the notable and perceptive comments. We will do our best to address the reviewer's concerns and answer the questions in the following.\n\n**R1. Problem motivation, the core methodological contribution and the gap in prior work.**\n\n**Problem motivation and scientific objective**: LVMs have proven highly influential in studies for the motor cortex, primarily by extracting low-dimensional neural embeddings that encode motor features and capture motor-related neural dynamics. However, LVMs have limited application in the analysis of visual neural activity. In our work, the scientific objective is to capture low-dimensional, stimulus-relevant embeddings from visual neural activity, particularly in the context of naturalistic visual stimuli. More specifically, we aim to leverage LVMs to disentangle neural components related to visual stimuli from those influenced by internal states. Through this disentanglement, we seek to further explain the observed variability in visual neural activity across subjects and across cortical regions.\n\n**The core methodological contribution**: We have provided a concise approach description in lines 45-50. Here, we make a more detailed clarification. First, we introduce temporal structures to explicitly establish temporal relationships for latent variables in a causal order. This design allows the latent variables to evolve over time, capturing the temporal dependencies inherent in neural activity. Second, we implement a split structure and design distinct loss functions to build two specialized sets of latent variables: external latent representations are trained to capture stimulus-relevant components of visual neural activity and internal latent representations are developed to reflect the animal's dynamic internal states.\n\n**The gap in prior work**: Existing studies via LVMs on visual neural activity primarily focus on simple visual stimuli and do not investigate extracting meaningful neural embeddings under naturalistic visual stimuli. More importantly, there was a lack of LVMs capable of splitting latent representations to explicitly differentiate between external stimulus-relevant and internal state-related components in visual neural activity in a self-supervised manner. For detailed differences in structure and design from previous models, please refer to R9 to Reviewer RtPx.\n\n**R2. Model architecture and methodology of model training.**\n\n**Model architecture**: Our model's core comprises external and internal latent variables, representing distinct visual neural activity components, alongside state variables for temporal processing. These variables interdepend through modules, fully described in the manuscript; here, we highlight design motivations.\n- Many component dependencies are adapted from the classic VRNN.\n- In the encoder, both external and internal latent variables are conditioned on input neural activity and their respective RNN state variables, crucial for causal temporal capture.\n- The internal latent variable's prior is conditioned solely on state variables, promoting internal state spontaneity.\n- The decoder reconstructs neural activity conditioned on both latent and internal state variables, with internal state variables enriching neural dynamics for more precise reconstruction.\n- Finally, external RNN state variables depend only on current input and previous state, while internal state variables also depend on latent variables, reflecting how external visual stimuli influence an animal's internal states.\n\n**Loss functions**: The objective of multiple training terms within our designed loss functions is to achieve disentanglement between external and internal latent variables, compelling them to reflect distinct components of visual neural activity.\n- As in lines 132–141, contrastive learning is applied to the external latent variables. It encourages clustering external latent variables corresponding to sample pairs with small temporal offsets (i.e., under similar visual stimuli), associating these variables with the visual input. Concurrently, a time-dependent prior distribution exhibiting spontaneity guides the internal latent variables via KL divergence. The differential application of these distinct loss functions to the two sets of latent variables is instrumental in promoting their disentanglement.\n- The swapping operation (detailed in R1 to Reviewer NU2a) further reinforces this disentanglement. It compels the model to strengthen the distinct roles of the external and internal latent variables, thereby enhancing their disentangled representation.\n\nFor the reason for the baseline choices, please refer to R1 to Reviewer RfuT.\n\n**R3. The experiments of synthetic datasets.**\n\n**We have provided the detailed generation procedure for the two synthetic datasets in Appendix C**. Specifically, the non-temporal dataset is constructed using several label clusters to represent distinct visual stimulus categories, while the temporal dataset is designed to characterize the temporal relationships inherent in neural activity.\n\nThese synthetic experiments are crucial for evaluating the fundamental ability of LVMs to construct accurate latent variables—a standard preliminary step in this field. The verification of our model's core contribution and its efficacy in extracting latent variables related to actual visual stimuli is subsequently accomplished through experiments with real visual neural activity datasets.\n\n**R4. About the decoding task.**\n\nFor decoding tasks and coarse-grained (within +/-1 second) movie frame classification, **we largely follow CEBRA's established approach**, as it has conducted extensive analysis of mouse visual cortex activity. This alignment facilitates fair baseline comparisons. In addition, our model consistently outperforms others in finer-grained (the minimum is +/-0.2 second) frame classification, as shown in Figure 4B.\n\nNevertheless, this approach has limitations with longer movies or larger scene datasets. Future work will explore more generalized metrics for evaluating latent variable-visual stimuli relationships, such as comparing the similarity between latent variables and low- or high-level visual features.\n\nFor k-value selection, empirical values were derived considering dataset size, latent variable dimension, and category count. An odd $k$ value was chosen to reduce decision uncertainty.\n\n**R5. Evaluation of spike reconstruction and internal latent variables.**\n\nWe evaluate spike reconstruction on the test set for models other than CEBRA. Our model consistently demonstrated superior performance, as indicated by lower Poisson negative log-likelihood values (Tables r1, r2).\n\nWe acknowledge the current lack of quantitative metrics for internal latent variables. While preliminary results suggest low correlation with visual stimuli, future work will require more comprehensive datasets to rigorously validate whether these variables accurately reflect an animal's internal state.\n\nTable r1: The reconstruction results for the scene dataset.\n\n||Mouse 1|Mouse 2|Mouse 3|Mouse 4|Mouse 5|\n|-|:-:|:-:|:-:|:-:|:-:|\n|LFADS|0.234±3.6e-5|0.222±4.5e-5|0.232±3.5e-5|0.208±5.8e-5|0.226±5.7e-5|\n|pi-VAE|0.252±8.7e-5|0.241±2.8e-5|0.248±2.4e-5|0.227±4.0e-5|0.245±7.1e-5|\n|Swap-VAE|0.202±2.1e-4|0.189±2.6e-4|0.196±2.5e-4|0.181±2.9e-3|0.196±7.3e-4|\n|TE-ViDS|0.195±7.6e-4|0.145±1.6e-4|0.184±4.9e-3|0.147±4.8e-3|0.174±1.3e-3|\n\nTable r2: The reconstruction results for the movie dataset.\n\n||Mouse 1|Mouse 2|Mouse 3|Mouse 4|Mouse 5|\n|-|:-:|:-:|:-:|:-:|:-:|\n|LFADS|0.134±3.9e-4|0.196±5.1e-4|0.186±5.5e-4|0.192±5.3e-4|0.163±5.4e-4|\n|pi-VAE|0.137±1.7e-4|0.195±6.6e-5|0.185±7.7e-5|0.192±9.2e-5|0.163±7.8e-5|\n|Swap-VAE|0.100±2.7e-4|0.153±2.9e-4|0.144±2.4e-4|0.149±2.0e-4|0.119±2.3e-4|\n|TE-ViDS|0.086±4.0e-5|0.128±8.4e-5|0.118±1.0e-4|0.122±1.5e-4|0.096±4.7e-5|\n\n**R6. The improvement in written.**\n\nWe will revise those sentences mentioned by the reviewers in the revised manuscript.\n\n- L38-39: Given the significant challenge of elucidating the correlation between neural activity and visual stimuli, the development of robust models for extracting vision-related latent representations is critically imperative.\n\n- L48: External latent representations aim to capture stimulus-relevant components within visual neural activity, while internal latent representations reflect dynamic internal states influencing an animal's sensory capabilities.\n\n- L117: High-quality latent representations, as compressed forms of original neural activity, must effectively reconstruct their input.\n\n**R7. Why use the contrastive loss?**\n\nPlease refer to R2.\n\n**R8. Whether the resulting objective still maintains a valid ELBO?**\n\nThe ELBO derivation hinges on introducing an approximate posterior for latent variables and utilizing variational inference. Contrastive learning, applied to external latent variables under the approximate posterior's input-conditioned constraint, acts as a regularizer without conflict. Its non-negative loss term also preserves the ELBO inequality.\n\n**R9. The variation in decoding performance across mice.**\n\nWe have further investigated this phenomenon in Sections 4.5 and 4.6. Our findings suggest that it may stem from variations in the internal states of different mice at distinct stages, which, in turn, influence their capacity for encoding visual information.\n\n**R10. Limitations.**\n\nWithout simultaneous internal state recordings, interpreting the functional roles of inferred internal latent variables is limited. Future work will analyze datasets incorporating both visual neural activity and state monitoring. Subsequently, we will extend this approach to other brain regions and experimental paradigms to investigate broader principles of biological coding mechanisms."}, {"review_id": "hWrWYgUus2", "reviewer": "Reviewer_RfuT", "summary": "Time-Evolving Dynamical System for Learning Latent Representations of Mouse Visual Neural Activity\n\nThis paper presents a method for time-varying representations in a neural network for visual data decoding in a latent space. The method essentially relies on two RNNs - one of which is regularized towards external behavioral variables and another that is regularized with a contrastive loss to be \"smooth\". The method essentially is a mild variation of existing work, some of which is cited but many others that are omitted. I think really understanding what this method has to offer would require much more rigorous comparison to the many editing dynamical system LVM models.\n\n\nMajor comments:\n\n\"most work on LVMs has not explicitly considered neural temporal relationships\" \nOff the top of my head here are the models I can think of:\n - LDS: https://pubmed.ncbi.nlm.nih.gov/22663075/\n - POGLM: https://arxiv.org/pdf/2402.01263\n - PLINE: https://proceedings.mlr.press/v139/kim21h.html\n - DPAD: https://www.nature.com/articles/s41593-024-01731-2\n - rsLDS: https://arxiv.org/abs/1610.08466\n - CREIMBO: https://openreview.net/forum?id=28abpUEICJ\n - dLDS: https://jmlr.org/papers/v25/23-0777.html\n - PSID: https://pubmed.ncbi.nlm.nih.gov/33169030/\n - DFINE: https://pubmed.ncbi.nlm.nih.gov/38082181/\n - GP-VAE: https://arxiv.org/pdf/2310.03111\n - RP-GSSM: https://arxiv.org/abs/2505.23569\n - POYO: https://openreview.net/forum?id=sw2Y0sirtM\n\n\nMany of the above are much more reasonable to compare to - especially the both regularized AND sequential (like PSID or DFINE) than the comparisons provided in the paper.\n\nThe disentangling is a major part of the paper and is similar to PSID (sequential - cited above) and the now many nonlinear CCA style methods (supervised - examples listed below). These comparisons may be more appropriate than CEBRA.\n - https://proceedings.mlr.press/v28/andrew13.html\n - https://openreview.net/forum?id=5FUq05QRc5b\n - https://arxiv.org/abs/2408.12091\n - https://arxiv.org/abs/2312.13455", "questions": "I'm not really a fan of CEBRA but even I find the decoding scores in Figure 3 suspiciously low (also for pi-VAE). Do the authors have an explanation for that? It seems like something significant went wrong, potentially a bug? \n\nThe separation of internal and external stimuli can be challenging and similar methods (DCCA, Lyu et al, SPLICE, etc listed above) rely on information theoretic approaches to quantify disentangling. Here the primary metric is the decoding of external variables. Is this sufficient? Would estimates of the information show that these numbers are low enough to consider the internal variables truly internal?", "limitations": "Limitations are not discussed, and it seems at the very least the limitation of data requirements to fit this model should be discussed (especially when there are linear system alternatives that would take much less data to fit).", "rating": 2, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n - The model derivation seems correct\n - The results for the model in absolute terms seem interesting\nWeaknesses\n - The comparisons are lacking in appropriate baselines\n - There are many missing references\n - The novelty of the model is fairly limited", "was_revised": true, "final_justification": "While I think that the authors did try hard to address the comments, there are a number of new baselines and I feel that the paper needs significant editing to incorporate these changes. Unfortunately, it's very hard to assess these new baselines given the NeurIPS format and I think that a new submission to the next conference would be much stronger.", "rebuttal": "We thank the reviewer for the pointed and constructive comments. We will do our best to address the comments and answer the questions.\n\n**R1. References and baselines.**\n\nWe sincerely appreciate the reviewer's valuable suggestions regarding additional Latent Variable Model (LVM) references, and we will gladly incorporate a discussion of these works into our revised manuscript. However, we respectfully contend that these suggested models may not be more appropriate for direct comparison with our method than the baselines we have already employed.\n\nFirst, our work proposes a novel visual dynamical system that leverages self-supervised contrastive learning to disentangle neural activity components related to visual stimuli from those influenced by internal brain states. A core characteristic of our approach is the absence of any supervised signals (e.g., external behavioral variables) in the model's training. Given these specific model characteristics, we selected our current baselines. LFADS is a classic and widely recognized dynamical system, providing a robust benchmark for temporal neural data. pi-VAE incorporates external supervised information (and supporting discrete variables) to construct interpretable latent variables, contrasting with our self-supervised objective. Swap-VAE utilizes contrastive learning to construct separated latent variables, aligning with our disentanglement goal, although it does not explicitly model temporal dynamics. CEBRA also employs contrastive learning for latent variable construction. Critically, the work on CEBRA has conducted extensive analyses of neural activity in the mouse visual cortex and our experimental setup for visual neural activity largely follows its established methodology. This direct alignment in an experimental context makes CEBRA a particularly pertinent baseline.\n\nSecond, most of the models mentioned by the reviewer are either oriented towards other research topics or possess characteristics that render them less applicable to our specific context. For instance, models such as LDS and PLINE primarily focus on modeling the dynamics of synthetic data or motor behaviors in the motor cortex. Similarly, PSID, DFINE and DPAD are designed to separate behavior-relevant and behavior-irrelevant neural dynamics using supervised information, which differs from our self-supervised approach. MM-GPVAE is tailored for multimodal inputs, while CREIMBO and POYO focus on modeling population dynamics across brain regions or individuals. Finally, models like RP-GSSM and DCCA are primarily concerned with learning low-dimensional representations of images or videos, which fall outside the scope of our neural activity analysis.\n\nNevertheless, as suggested by the reviewer, we incorporate PSID, DFINE, and DCCA into our experiments on visual neural activity for comparison. Notably, the supervised versions of PSID and DFINE leverage continuous, high-temporal-resolution behavioral variables as supervised information to separate behavior-relevant and behavior-irrelevant neural dynamics. This approach presents significant challenges for application to the disentanglement of visual-related and visual-unrelated neural activity in our context. Specifically, biological experiments often utilize static visual stimuli, or dynamic stimuli with temporal resolutions considerably lower than that of neural activity. Furthermore, natural visual stimuli are high-dimensional and complex signals. The various methods of constructing visual information for supervision introduce strong human-induced priors that could influence the nature of the constructed latent variables. For example, using pixel values, low-order visual features, or high-order semantic information for supervision corresponds to visual information encoded by different regions of the visual pathway. Therefore, we adopt the unsupervised versions of PSID and DFINE. As presented in Table r1, our model consistently demonstrates superior performance compared to these models.\n\nTable r1: The results (\\%) for decoding natural movie frames.\n\n||Mouse 1|Mouse 2|Mouse 3|Mouse 4|Mouse 5|\n|-|:-:|:-:|:-:|:-:|:-:|\n|PSID|8.11|38.33|39.56|29.67|14.11|\n|DFINE|10.36±0.56|55.53±0.71|46.56±0.38|45.49±0.68|22.96±0.63|\n|DCCA|12.76±0.54|60.13±0.64|58.47±0.87|43.31±1.42|26.82±1.11|\n|TE-ViDS|13.88±0.19|65.38±0.36|59.88±0.72|54.33±0.54|30.18±0.40|\n\n**R2. The low performance of CEBRA in Table 1.**\n\nWe think that CEBRA demonstrates stronger performance in distinguishing between coarse-grained categories but less capability for finer-grained distinctions. First, this is suggested by Figure 8 in the Appendix, where, despite CEBRA's capacity to differentiate some categories through contrastive learning, considerable overlap is observed in the neural latent representations corresponding to other categories. This limitation is further evinced in the movie decoding experiment (Figure 4B), where CEBRA's performance declines more significantly as the constraint window narrows (i.e., an increased number of movie frame categories).\n\nTo corroborate this view, we further perform an experiment on static scene decoding. We train the models on neural data of the five natural scenes that elicit the strongest neural responses and evaluate their performance on the five-classification task. As shown in Table r2, although CEBRA's performance remains inferior to our model, it achieves a relatively good level, supporting our hypothesis regarding its better performance on coarser distinctions.\n\nFurthermore, we note that the lower performance of pi-VAE is confined to Mouse 1, with its performance on other mice being at an intermediate level, thus ruling out the possibility of a bug.\n\nTable r2: The results (\\%) for decoding five natural scenes.\n\n||Mouse 1|Mouse 2|Mouse 3|Mouse 4|Mouse 5|\n|-|:-:|:-:|:-:|:-:|:-:|\n|LFADS|79.6±2.8|78.4±4.3|68.0±4.3|71.2±1.7|61.6±3.6|\n|pi-VAE|97.2±1.0|81.2±1.8|80.8±1.4|85.6±2.1|69.2±2.0|\n|Swap-VAE|98.0±0.8|69.6±2.3|77.2±1.6|76.4±2.3|84.8±2.2|\n|CEBRA|96.4±1.3|76.8±1.9|80.4±1.2|72.8±1.9|56.8±2.7|\n|TE-ViDS|100.0±0.0|95.6±1.2|94.8±1.1|94.8±0.6|90.4±1.0|\n\n**R3. The evaluation for internal latent variables.**\n\nWhile our current findings indicate a low correlation between internal latent variables and visual stimuli, we acknowledge the absence of quantitative metrics in this work to definitively verify if these variables reflect an animal's internal state. To address this limitation, we will utilize more comprehensive datasets (e.g., incorporating behavioral and physiological recordings) for robust evaluation in the future.\n\n**R4. The data requirements.**\n\nThis represents a trade-off: our model, being more complex than linear alternatives, incurs higher computational costs when applied to smaller datasets. However, for larger datasets, where linear models are prone to underfitting, our model demonstrates superior scalability and the capacity to capture more intricate relationships."}]

Author rebuttal:

Meta-review: This paper introduces a time-dependent latent variable model that incorporates separate componentns for stimulus and noise, and is trained using a contrastive learning framework.  Its strengths include a novel modeling approach to an important problem in neuroscience (i.e., resolving stimulus and noise underlying neural responses) and is well written.  Weaknesses involved lack of justification for some modeling choices and inadequate comparisons to other baseline methods.  Ultimately, I found the authors' rebuttal responses compelling and feel that the reservations of reviewer RfuT are not fatal to the paper's contributions.  (I agree, for example, that some of the alternate methods -- eg that used supervised approaches to identifying signal latents, or methods focused on low-D motor variables -- are not entirely relevant benchmarks to the unsupervised method proposed here).  I thus feel the paper makes a valuable contribution to the NeurIPS community and am happy to recommend it for acceptance.  Please be sure to address all reviewer comments and criticisms in the final manuscript.
Decision: Accept (poster)