Title: Time-Evolving Dynamical System for Learning Latent Representations of Mouse Visual Neural Activity
Venue: NeurIPS 2025
Accepted: yes
Keywords: Visual Cortex

Latent Variable Models

Dynamical System
Abstract: Seeking high-quality representations with latent variable models (LVMs) to reveal the intrinsic correlation between neural activity and behavior or sensory stimuli has attracted much interest. In the study of the biological visual system, naturalistic visual stimuli are inherently high-dimensional and time-dependent, leading to intricate dynamics within visual neural activity. However, most work on LVMs has not explicitly considered neural temporal relationships. To cope with such conditions, we propose Time-Evolving Visual Dynamical System (TE-ViDS), a sequential LVM that decomposes neural activity into low-dimensional latent representations that evolve over time. To better align the model with the characteristics of visual neural activity, we split latent representations into two parts and apply contrastive learning to shape them. Extensive experiments on synthetic datasets and real neural datasets from the mouse visual cortex demonstrate that TE-ViDS achieves the best decoding performance on naturalistic scenes/movies, extracts interpretable latent trajectories that uncover clear underlying neural dynamics, and provides new insights into differences in visual information processing between subjects and between cortical regions. In summary, TE-ViDS is markedly competent in extracting stimulus-relevant embeddings from visual neural activity and contributes to the understanding of visual processing mechanisms. Our codes are available at https://github.com/Grasshlw/Time-Evolving-Visual-Dynamical-System.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nTime-Evolving Dynamical System for Learning Latent Representations of Mouse Visual Neural Activity\n\nThis paper presents a method for time-varying representations in a neural network for visual data decoding in a latent space. The method essentially relies on two RNNs - one of which is regularized towards external behavioral variables and another that is regularized with a contrastive loss to be \"smooth\". The method essentially is a mild variation of existing work, some of which is cited but many others that are omitted. I think really understanding what this method has to offer would require much more rigorous comparison to the many editing dynamical system LVM models.\n\n\nMajor comments:\n\n\"most work on LVMs has not explicitly considered neural temporal relationships\" \nOff the top of my head here are the models I can think of:\n - LDS: https://pubmed.ncbi.nlm.nih.gov/22663075/\n - POGLM: https://arxiv.org/pdf/2402.01263\n - PLINE: https://proceedings.mlr.press/v139/kim21h.html\n - DPAD: https://www.nature.com/articles/s41593-024-01731-2\n - rsLDS: https://arxiv.org/abs/1610.08466\n - CREIMBO: https://openreview.net/forum?id=28abpUEICJ\n - dLDS: https://jmlr.org/papers/v25/23-0777.html\n - PSID: https://pubmed.ncbi.nlm.nih.gov/33169030/\n - DFINE: https://pubmed.ncbi.nlm.nih.gov/38082181/\n - GP-VAE: https://arxiv.org/pdf/2310.03111\n - RP-GSSM: https://arxiv.org/abs/2505.23569\n - POYO: https://openreview.net/forum?id=sw2Y0sirtM\n\n\nMany of the above are much more reasonable to compare to - especially the both regularized AND sequential (like PSID or DFINE) than the comparisons provided in the paper.\n\nThe disentangling is a major part of the paper and is similar to PSID (sequential - cited above) and the now many nonlinear CCA style methods (supervised - examples listed below). These comparisons may be more appropriate than CEBRA.\n - https://proceedings.mlr.press/v28/andrew13.html\n - https://openreview.net/forum?id=5FUq05QRc5b\n - https://arxiv.org/abs/2408.12091\n - https://arxiv.org/abs/2312.13455\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n - The model derivation seems correct\n - The results for the model in absolute terms seem interesting\nWeaknesses\n - The comparisons are lacking in appropriate baselines\n - There are many missing references\n - The novelty of the model is fairly limited\n\nQUESTIONS:\nI'm not really a fan of CEBRA but even I find the decoding scores in Figure 3 suspiciously low (also for pi-VAE). Do the authors have an explanation for that? It seems like something significant went wrong, potentially a bug? \n\nThe separation of internal and external stimuli can be challenging and similar methods (DCCA, Lyu et al, SPLICE, etc listed above) rely on information theoretic approaches to quantify disentangling. Here the primary metric is the decoding of external variables. Is this sufficient? Would estimates of the information show that these numbers are low enough to consider the internal variables truly internal?\n\nLIMITATIONS:\nLimitations are not discussed, and it seems at the very least the limitation of data requirements to fit this model should be discussed (especially when there are linear system alternatives that would take much less data to fit).", "TITLE": "Official Review", "RECOMMENDATION": 2, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work presents a latent variable model for time-varying neural activity in mice, using an encoder-decoder architecture with a temporally evolving latent space split into two components: “external” and “internal.” The external component is a deterministic function of neural activity, aimed at capturing stimulus-dependent information, while the internal component is stochastic and intended to capture neural variability. The model is trained using a combination of a variational lower bound and a contrastive loss. On synthetic datasets, it shows improved latent and stimulus reconstruction. Applied to the large-scale Allen Brain Observatory Visual Coding dataset, the model outperforms baselines on natural scene and movie decoding, where decoding is framed as a scene/frame classification task.\n\nSTRENGTHS AND WEAKNESSES:\n### Overall assessment\n\nEdit: note that this assessment was written prior to the rebuttal phase. See below for final justification.\n\nThis paper presents a promising direction for latent variable modeling of neural activity but is currently significantly limited by unclear motivation, under-explained methods, and insufficiently rigorous evaluation.\n\n**To help me raise my scores**, I encourage the authors (where feasible) to: **(1)** clarify the model architecture and its novelty; **(2)** justify the use of the contrastive loss and its interaction with the ELBO; **(3)** substantiate the choice of simulated data and corresponding evaluation; **(4)** strengthen the evaluation with more rigorous metrics (e.g., log-likelihood/ELBO on spikes), justify the decoding setup; **(5)** clearly articulate the contribution relative to prior work, and importantly **(6)** reflect and discuss the paper's limitations.\n\nDetails below.\n\n## Strengths\n\n- This work tackles an important and timely problem in computational neuroscience and machine learning—analyzing time-varying neural activity in response to natural and complex visual stimuli—by developing a latent variable generative model of the neural activity. \n- The model explicitly separates stimulus-related and internally driven components in a temporally evolving latent space leading to more structured and interpretable representations of neural activity. \n- The paper presents results on the publicly available, large-scale Allen Brain Observatory Visual Coding dataset consisting of neural activity across six different cortical regions and five different mice, recorded using modern Neuropixel probes. It also includes comparisons with several existing latent variable models, making it overall a suitable benchmark for evaluating approaches to modeling mouse visual neural activity. \n\n## Weaknesses\n\n**Summary**: The core contributions, motivation, and methodological choices are often unclear or underexplained. Key concerns include vague problem framing, limited detail on synthetic datasets, coarse decoding evaluation with scalability issues, and missing assessments of spike reconstruction and internal latent utility. The writing would also benefit from revision for clarity, grammar and style.\n\n1. **Lack of clarity in problem motivation, methodological contribution, and model positioning:** In the initial sections—abstract, introduction, and related work—I found it difficult to identify the core methodological contribution, the specific scientific question being addressed, and the gap in prior work that the model aims to fill. While these sections hint at high-level goals, such as using latent variable models (LVMs) to capture correlations between neural activity and visual stimuli or behavior, they do not clearly present the motivation for applying LVMs in this context, provide a concise description of the proposed approach, or explain how it meaningfully differs from existing models. The rationale behind key design choices, particularly in relation to modeling neural activity to visual stimuli, is also unclear. A more focused and structured presentation of these elements before diving into the technical details would greatly enhance the clarity and accessibility of the paper.\n2. **Lack of clarity and motivation in model architecture and proposed methodology:** In Section 3, the method is not clearly motivated or presented in terms of the model architecture and the specific relationships between variables. While Figure 1 offers a helpful high-level overview, it omits key variables such as $h_t$, $x_t$, $z_t^e$, $z_t^i$, and $r_t$, which are central to the model. Although the text outlines how these variables interact, the rationale behind their roles and the choices made is not clearly explained, making it difficult to develop an intuitive understanding. The appendix provides a graphical representation of the model, but it lacks a descriptive summary of the relationships among these variables and the motivation for these specific design choices, especially given that the model being proposed is quite complex. The same goes for the loss; why exactly include the contrastive loss? For clarity and completeness, I would recommend the authors include a concise and self-contained explanation of the model and its components directly in the main text, even if some fundamental choices are based on previous work. Important, why are these choices central to the context of neural activity in response to visual stimuli (if they are intended as such) and how are they exactly different from the baselines considered? This would also substantially help understand these specific baselines were chosen, which at the moment feels not well-justified.\n3. **Synthetic data lacks sufficient detail**: The descriptions of the synthetic datasets lack essential detail. The current explanation—“[Line 166] Synthetic datasets roughly point to two properties of visual neural activity…”—is vague and uninformative, and does not clarify what the synthetic data actually looks like, how it’s constructed, or what aspects of visual neural activity it is intended to model. \n4. **Unclear whether synthetic results evaluate the model’s core contributions:** When it comes to the results, while they show that their model is superior in reconstructing the true latents and the observations, it is not clear whether it tests their model’s novelty — be it the split internal-external latent variables or their loss functions — or its relevance to neural activity to visual stimuli. It’s difficult to judge how good of an evaluation this represents for their proposed model.\n5. **Decoding task treated as coarse grained classification with limited scalability:** In their main evaluation on real data, the authors frame the task as decoding natural scenes and movie frames, but in practice, they treat it as a classification problem: predicting which labeled image (out of 118 natural scenes) or movie frame (out of 900) was shown to the animal. For movie frames, predictions are considered correct if they fall within +/-1 second—about 30 frames if you consider 30 fps—making this a coarse-grained temporal task rather than frame-level decoding. This approach also doesn’t scale well to longer videos or larger scene sets that are typical, where the label space becomes unwieldy. Additionally, choices like searching for K (in their KNN) in odd numbers from 1 to 20 are ad hoc and not well justified.\n6. **Missing evaluation of spike reconstruction and internal latent utility:** While it is encouraging to see the model outperform all baselines on scene and frame decoding, with the external latent clearly showing higher stimulus decodability, the paper does not evaluate the model’s ability to reconstruct spike counts. Given that the core task is to model the distribution of spike counts—albeit through latent variables—it would be more rigorous to report the log-likelihood (or ELBO) on held-out spike data and compare it to other models that support such evaluation. Additionally, the paper lacks a benchmarking task to assess whether the “internal” latent component captures meaningful structure. Showing that it does not aid stimulus decoding seems to me insufficient to demonstrate its utility or interpretability.\n7. **Scope for improvement in written clarity and style:** The paper would benefit from careful editing to better align with the tone and conventions of scientific writing. Many sentences feel unwieldy and imprecise:\n\n(i) “[L38–39]: *Given that revealing the correlation between neural activity and visual stimuli is a challenging research highlight in the neuroscience community, it is desperately necessary to develop a strong model for extracting vision-related latent representations.*”\n\n(ii) “[L48]: *The external latent representations aim to magnify stimulus-relevant components of visual neural activity, while the internal latent representations point to dynamical internal states that have an impact on the animal’s perceptive capability.*”\n\n(iii) “[L117] *The high-quality latent representations are compressed information on the original neural activity, so they are needed to reconstruct the input neural activity well.*”\n\nIn above examples, the phrasing could be made more concise, neutral, and aligned with typical scientific expression.\n\nQUESTIONS:\n1. Why use the contrastive loss and only on z^e? It’s not clear to me what the motivation there is precisely.\n\n2. I understand that the combination of the reconstruction term and KL divergence forms the standard ELBO. However, it’s unclear how the full objective, which includes the additional contrastive loss, is still considered an ELBO. In Appendix B, you derive the ELBO and then add a negative contrastive loss term, but it’s not clear whether the ELBO inequality still holds under this modified objective. As I understand it, the contrastive term is not necessarily independent of the ELBO (z^{e} appears in the reconstruction loss)—so minimizing the contrastive loss could potentially conflict with maximizing the ELBO. It would be helpful if you could clarify how these objectives interact and whether the resulting objective still maintains a valid ELBO interpretation.\n\n3. There seems to be substantial variation in decoding performance across the different mice (esp mouse 5 vs rest in Figure 3 and mouse 1 vs rest in Figure 4). Do you have any hypotheses as to why this might be the case? \n\n4. What are the main limitations of your model and approach? What future directions are you considering? Are there specific neuroscientific questions you aim to address with your model?\n\nPlease also see questions under \"weaknesses\".\n\nLIMITATIONS:\nLimitations of the model and approach are not discussed. Please see my question 4.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents a dynamical latent variable model that separates stimulus-related and internal neural dynamics using a hybrid VAE and contrastive learning framework. The model consists of two distinct latent pathways: a deterministic external latent space to capture stimulus-driven variability, and a stochastic internal latent space to capture trial-specific dynamics. A key feature is the use of contrastive learning and a swap-based reconstruction loss to enforce a functional separation between these two latent spaces. The model is evaluated on neural recordings from mouse visual cortex and the authors claim it uncovers both representational structure and hierarchical processing.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n\n- The text is clearly written with a logical flow and was easy to follow\n- Proposes a novel combination of contrastive learning, latent swapping, and variational sequence modeling.\n- Explicit modeling of internal vs. external factors in neural data is conceptually appealing and clearly structured.\n- The swap-based reconstruction loss is a clever mechanism for enforcing disentanglement.\n\n**Weaknesses:**\n\n- Several modeling choices (e.g., deterministic external latents, use of internal GRU state in the decoder) are not clearly justified given the biological modeling goals.\n- Interpretations in the results sections often overreach the evidence presented (e.g., \"potential breakthrough\") and claims about “new” insights into neural processing mechanisms and hierarchical structure are not strongly supported by the presented analyses.\n\nQUESTIONS:\nBelow is a list of questions, concerns, and suggestions that I had while reading the paper. Some relate to specific modeling choices and assumptions, while others concern interpretation and framing of the results:\n\n**Questions for which I am happy to increase the rating if properly addressed:**\n\n- What are the exact “new insights” that this paper is providing “into the intrinsic correlation between neural activity and visual stimuli” (line 54), as the authors mention in the intro? same question for “visual information processing mechanisms of the mouse visual cortex”? I agree the method has potential for being used to generate insights, but if you are claiming that it is providing the insight then the insight should be provided, rather thoroughly and convincingly. I am specifically curious about the “new” insight, as opposed to confirming previous findings (maybe even with better quality) or improved performance.\n- The authors motivate the choice of making the external latent deterministic in biological terms (i.e. “to reduce the noise inherent in neural activity”). Why is that desired, from the perspective of modelling biological data? Neural responses to the same stimulus exhibit trial-to-trial variability that is not fully attributable to latent brain states or structured noise (i.e. individual neurons exhibit intrinsic variability), so a biologically faithful model would allow $\\mathbf{z}^{(e)}$ to reflect some uncertainty.\n- The decoder takes $\\mathbf{h}_{t-1}^{(i)}$ as input in addition to the latents $\\mathbf{z}_t^{(e)}$ and $\\mathbf{z}_t^{(i)}$. If the latents are meant to capture the full structure of the neural dynamics, what is the justification for passing the RNN hidden state upstream of $\\mathbf{z}_t^{(i)}$ into the decoder? To me, this blurs the role of $\\mathbf{z}_t^{(i)}$ as a sufficient representation of the internal factors that effect high-dimensional neural data. Moreover, if the hidden state of the internal RNN is necessary, why not include the hidden state of the external RNN as well?\n- How using denoised responses (i.e. using the predictions of any powerful predictive model) and computing RSM is “facilitating the unraveling of information processing mechanisms in the visual cortex”? The motivation for comparing the RSMs across two mice (Fig. 3B) was the observed difference in decoding score (i.e. mainly concerned with external factors), but the explanation is more concerned with internal states. The way I understood the explanation is that it looks like the two mice have different internal states and therefore the decoding score is different across subjects. First of all, I do not find this argument convincing, and even if I was convinced, I am not sure if this qualifies as “new insights into differences in visual processing between subjects” (mentioned in the abstract).\n- Regarding the explanation of the observed diagonal structure across all video trials (Fig. 4F), I agree that this indicates the mouse neural representation is driven by stimulus-related responses—especially since internal states are expected to vary across trials. The persistence of the diagonal structure thus likely reflects the dominance of the stimulus-driven component. To support this claim more directly, it would be helpful to see RSMs computed separately on the external and internal latents. The expectation is that the RSM of the external latent should preserve the diagonal structure across trials, while this pattern should be absent in the RSM of the internal latent.\n- I find the interpretation of the decoding scores across visual areas in Fig. 5 somewhat unclear. First, while the authors suggest that lower decoding in VISrl supports a hierarchical organization, they simultaneously attribute it to multi-sensory integration, which weakens the hierarchy argument. Second, if the cortex is indeed hierarchically organized, one might expect higher decoding scores in high-level areas due to more specialized and invariant representations. Instead, the highest decoding appears in VISp, the primary area, which seems counterintuitive if semantic discriminability is the goal. Could the authors elaborate how these results support a hierarchical structure, and how one might disentangle the effects of hierarchy from those of multi-modal integration?\n- The claim that the results amount to a “potential breakthrough” seems overstated. While I agree that the model may offer useful insights into sources of trial-to-trial variability, their interactions, and their temporal structure, the evidence does not clearly demonstrate a substantial advancement over prior work in uncovering computational mechanisms in visual areas. I suggest rephrasing this statement to more accurately reflect the scope of the contribution.\n\n**Other (minor) questions/suggestion:**\n\n- Could the authors explain how using the contrastive learning objective along with the swap reconstruction loss enforces a functional separation between the two latent factor sets (i.e. $\\mathbf{z}^{(i)}$ and $\\mathbf{z}^{(e)}$)? What is the contribution of each of these losses?\n- It would help the reader if the authors highlight the main differences between their approach and those mentioned in the related work (at least the most relevant ones that they use for model comparison). Some differences are mentioned (e.g Swap-VAE assumes independence over time) but for LFADS and CEBRA there is only a one sentence description and it is not immediately clear what the difference is (e.g. no internal-external factorization in LFADS). A more clear contrast of important aspects can help understanding the methodological contribution of the presented method.\n\nLIMITATIONS:\nAlthough the authors indicate in the checklist that limitations are discussed in Section 5 (Discussion), I could not find any explicit mention of limitations in the text. However, below are some limitations that I think are worth discussing (as mentioned also in my questions):\n\n- The deterministic nature of the external latent may artificially suppress trial-to-trial variability that is genuinely stimulus-related.\n- The evaluation focuses primarily on decoding and RSM structure, with limited connection to actual mechanistic hypotheses about neural computation. For instance, do the learned internal latent trajectories reflect known internal states (e.g., arousal, locomotion)? Or do the external latent encodes interpretable stimulus properties?\n- The interpretation of subject differences and hierarchical processing are speculative", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors:\n- Introduce a new latent variable model (LVM) which employs two RNNs to encode spike trains from mouse visual cortex into two latent variable vectors\n- The first, ‘external’ latent variable is a deterministic representation of stimulus-dependent neural activity, while the second, ‘internal’ latent variable stochastically encodes the noisy component of the neural activity\n- The networks and their link functions are jointly trained to maximise the ELBO of the neural data under a time-dependent Possion generative model and a contrastive loss on the external latent variables\n- The positive examples for contrastive loss in the external latent space are selected by encoding a temporally offset input sequence; negative samples are randomly selected from the dataset\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n- This is a very interesting, novel LVM, and the authors provide a convincing set of results comparing their model to existing LVMs, both with and without time dependence.\n- Furthermore, they provide an extensive range of ablation studies to their method, indicating the importance of each component of their full model. They also show that their model’s success is not due to its complexity, given that a smaller number of parameters also achieves state of the art performance on real and synthetic datasets.\n- Finally, they go beyond LVM performance metrics, and uncover representational similarities of neural representations which are not evident in the raw neural data.\n\n\nWeaknesses\n- Architectural choice wasn’t always clear to me. While I appreciate the division of representation between stimulus and internal noise, specific dependencies in section 3.1 were not fully motivated, and authors less familiar with existing cited models could benefit from an explanation of each component. Other explanations lack some clarity, for example, details of the swapping operation and the intuition for its benefit to training.\n- Without extra details, potentially provided in cited works, it is not obvious to me how the model was applied to non-temporal data in the first place. Both the generative model and the form of the contrastive loss function require a sequence of activity, \nIn section 4.4 it is stated that “Swap-VAE and CEBRA [suffer] a slight degradation due to the use of time-jittered positive samples.” Did the authors mean SWAP-VAE and pi-VAE - as it appears from Fig. 2E that CEBRA suffers a drastic decrease in performance due to time shuffling.\n- In section 4.6, the authors claim that “our model achieves significantly higher 273 decoding scores than LFADS and is slightly better than the others” - it would be helpful to provide details on statistical tests they performed to qualify this statement.\nIn its current form, I find the claim that the model “provides new evidence for the functional hierarchy of the mouse visual cortex” too strong. The corresponding section 4.7 simply provides a variety of performance metrics for various parts of mouse visual cortex, but does not provide sufficient evidence to support a functional hierarchy. Such a claim would require, e.g., equivalent analysis for multisensory stimulus exposure experiments.\n- Alternatively, finding a way to cross-validate the model trained one brain region in encoding activity from another region may help reveal functional dependencies between them. One method could be to train the model on one region, then retrain only the encoder and decoder weights on a second brain region, to ascertain whether the regions share similar time-dependencies.\n\nQUESTIONS:\n- Please could the authors clarify, e.g. in the appendix, how this method was applied to non-temporal synthetic data? Namely, how was sequential data generated from the stationary latent variables, thereby allowing application of the contrastive loss?\n- An ablation is performed where the contrastive loss is removed. Did the authors consider another ablation where the negative examples are instead drawn from the same sequence but with longer time delays, akin to how negative samples are drawn for CEBRA? This may help isolate the importance of the chosen form of contrastive loss.\n- In section 4.5, the authors claim that qualitative differences in the RSA for mice 1 and 2 are because “sensory performance in [mouse 1] is strongly influenced by their internal state.” Do the authors expect that this internal state is represented in some way by their internal latent variable? For example, could we expect similarity in the internal latents in the two blocks of trials for mouse 1, but not for mouse 2? This would help illuminate the architectural choices of the model too.\n- Why does the decoding score concatenate vectors in the scenes case, but average them in the movies case?\n\nLIMITATIONS:\nThere are no relevant negative societal impacts associated with this work.\n\nLimitations of the model are not discussed to great extent, nor are limitations of the analysis provided. \n\nThe authors focus on one neural dataset, so a small acknowledgement of the feasibility of application to other datasets, e.g. other mouse brain areas, other species, and other tasks, would help contextualise the study.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-03"}