Title: Extracting task-relevant preserved dynamics from contrastive aligned neural recordings
Venue: NeurIPS 2025
Accepted: yes
Keywords: neural latent embedding

neural latent dynamics

contrastive learning
Abstract: Recent work indicates that low-dimensional dynamics of neural and behavioral data are often preserved across days and subjects. However, extracting these preserved dynamics remains challenging: high-dimensional neural population activity and the recorded neuron populations vary across recording sessions. While existing modeling tools can improve alignment between neural and behavioral data, they often operate on a per-subject basis or discretize behavior into categories, disrupting its natural continuity and failing to capture the underlying dynamics. We introduce $\underline{\text{C}}$ontrastive $\underline{\text{A}}$ligned $\underline{\text{N}}$eural $\underline{\text{D}}$$\underline{\text{Y}}$namics (CANDY), an end‑to‑end framework that aligns neural and behavioral data using rank-based contrastive learning, adapted for continuous behavioral variables, to project neural activity from different sessions onto a shared low-dimensional embedding space. CANDY fits a shared linear dynamical system to the aligned embeddings, enabling an interpretable model of the conserved temporal structure in the latent space. We validate CANDY on synthetic and real-world datasets spanning multiple species, behaviors, and recording modalities. Our results show that CANDY is able to learn aligned latent embeddings and preserved dynamics across neural recording sessions and subjects, and it achieves improved cross-session behavior decoding performance. We further show that the latent linear dynamical system generalizes to new sessions and subjects, achieving comparable or even superior behavior decoding performance to models trained from scratch. These advances enable robust cross‑session behavioral decoding and offer a path towards identifying shared neural dynamics that underlie behavior across individuals and recording conditions. The code and two-photon imaging data of striatal neural activity that we acquired here are available at https://github.com/schnitzer-lab/CANDY-public.git.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nAuthors develop a method to generate a shared embedding space of neural recordings from many sessions to capture presumably low-dimensional preserved dynamics. They also fit a linear dynamical system to model the temporal evolution in the latent space derived from the created embedding. Their method uses contrastive learning methods to enforce that neural states with similar behavioral observations are mapped close by and vice versa. They apply their method to a synthetic, a mouse, and a monkey dataset and show that the learned embedding maintains similarity observed in the behavioural observations across sessions and subjects and allows the decoding of the observed behaviours.\n\nSTRENGTHS AND WEAKNESSES:\nIt’s great that the method was tested across multiple datasets, which supports its robustness. Similarly, the comparison to other state-of-the-art methods is much appreciated, especially with the use of grid search to tune those methods for a fair evaluation. The analysis on the importance of different loss terms is interesting and adds depth to the understanding of the method.\n\nThe paper provides clear definitions of the loss components and the overall setup. However, it should be made clearer which specific loss terms were used to generate the results shown. This would help readers better interpret the findings and reproduce the experiments.\n\nIncorporating an LDS (linear dynamical system) component in this method is a nice addition to improve interpretability. However, the paper doesn’t clearly demonstrate how this addition can be practically leveraged.\n\nQUESTIONS:\n(1) The method encourages behavioral similarity by design, but it would be valuable to understand whether other meaningful properties are preserved or not. For example, does similarity between neural recordings persist in the embedding space? Is proximity in time or trials reflected?\n\n(2) Regarding generalization, are you using the behavioral supervision loss here? If not, it’s unclear why the method would generalize. There doesn't seem to be anything explicitly enforcing that the embedding space is shared across sessions. Could you please clarify?\n\n(3) The approach assumes that the decoded behavioral observations are task-relevant and can be reliably read from the neural data. An important control would be to shuffle the neural recordings relative to the behavioral data and show that under these conditions, the method cannot find a “good” embedding. Including such a control would help confirm that the embedding genuinely reflects neural-behavioral relationships.\n\n(4) To better assess generalization, it would be helpful to test whether the model can map previously unseen neural states to the correct behavioral outputs. One suggestion would be to hold out specific trial types (e.g., certain movement angles or task periods) from training and include them only in the test set. Are these held-out examples mapped to the correct behavioral location in the embedding space? Can their associated behavior still be accurately decoded? This would help evaluate whether the embedding captures meaningful behavioral structure, even for novel data, or where its limitations are.\n\nLIMITATIONS:\nThe approach is interesting and the results on preserving behavioral similarity are promising. However, it would be important to discuss whether preserving behavioral similarity is always desirable. In neuroscience, it's common for different cognitive processes to produce identical behavioral outputs - such as during evidence accumulation in integration tasks, where animals may show the same behavior while engaging in opposite internal processes. It would strengthen the paper to include a discussion of when and where this method is appropriate, and what the limitations might be in such cases. Alternatively (but probably outside the scope of this paper): How does the method perform on error trials? It would be interesting to see whether the model distinguishes between correct and incorrect trials in the embedding, and whether behaviorally similar but cognitively distinct trials are treated differently.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work proposed CANDY, and end-to-end framework that aligns neural and behavior data using rank-based contrastive learning, and it also integrate with a linear dynamical system to capture the temporal associations. The optimization include a contrastive loss to align the neural latent embeddings with the continuous behavior samples, a behavior decoding loss, as well a linear dynamical system with parameters to be jointly optimized. It contributes to increase the generalizations across data, which allows learning the embeddings across sessions and subjects (also include unseen sessions), and demonstrates superior performance compared to existing baselines.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n1. This method contribute to an important question on representation learning from neural activities for behavior decoding, and aim to improve the generalization of the method across subjects and sessions. \n2. Compared to existing approaches, this model focused on using contrastive loss aligning with continuous behaviors, and incorporated with a dynamical constraint to enable temporal consistency, which is superior than existing baselines such as CEBRA.\n3. This methods are evaluated on multiple datasets from mouse to macaque, demonstrate its generalizations across animal species. \n4. It demonstrates superior performance on decoding performance and its performance are less sensitive to choice of hyperparameters such as latent dimension.\n\n**Weakness**\n1. Explain the choice of linear dynamical system, discuss whether it would be sufficient to capture the nonlinearity and nonstationarity of the dynamical system. It would be valuable to introduce ablation study on the complexity of dynamical system.\n2.  Lack of comparisons with other baseline for behavior decoding such as stich-LFADS, POYOs, NDT-2, etc. [34-39] as discussed in the related works.\n\nQUESTIONS:\n1. In Figure 3b, it is unclear how CANDY is superior compared to CEBRA?\n2. What is the computational cost and latency of the proposed method compared to existing approaches?\n3. Are the parameters estimated in the linear dynamical system eqn (4) and eqn (5) identifiable and interpretable?\n\nLIMITATIONS:\n1. Lack of evaluations and comparisons with other SOTA baselines discussed in the related works [34-39]. \n2. The assumption of linear dynamical system (LDS) as a prior in the model might not be sufficient to capture the complexity of the neural dynamical systems,  and generalizable to more complex situations.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose CANDY, a dimensionality reduction method for neural data that fits a shared LDS over multiple sessions with aligned embeddings. A key step is using a contrastive loss to align embeddings  based on the continuous behavioural label associated with each time point. This encourages the learned embeddings to group together neural data points with similar behaviours, even across sessions. They apply the method to one ground truth simulation, and a couple datasets. One interesting finding is that the contrastive loss is necessary to find behaviourally relevant embeddings, and simply using a supervised behavioural loss is not sufficient.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\nThis is a really nice paper with a cool idea to align latents using their behavioral labels by leveraging contrastive learning.The claims are generally well supported, and the analyses of the contribution of the contrastive loss are interesting.  \n\nAs far as I know the model is a novel application of an existing rank-based contrastive loss to a new problem regarding alignment of neural dynamics (although I am not an expert on contrastive learning so I am not 100% sure on the novelty). It is certainly a step forward compared to CEBRA or DFINE. I believe the model will be broadly interesting to NeurIPS and the analyses of the contrastive loss offers some insights regarding the important considerations for aligning latents. I can certainly see future researchers building upon this idea for new alignment algorithms. \n\nIt's well written and clear for the most part, which I appreciated as a reviewer.\n\nWeaknesses:\n\nAlthough I really like this submission and find it to be high quality overall, there are two fundamental points that really should be improved.\n- The method is demonstrated in two neuroscience data sets, although one dataset receives much more attention in the main text. But the paper is missing ground truth experiments on out-of-sample data.\n- The discussion is very short and should be expanded to include more limitations and relationship to other work on latent dynamics inference and contrastive learning in neuroscience.\n\nQUESTIONS:\n- Line 112, definition of S_h,j, I think that should be \\leq, not \\geq. Is that a typo?\n- The ground-truth analyses (section 4.1) should include examples of where the model breaks down. For example, what happens if the behavioural readout is highly nonlinear, or worse yet if the mapping varies from session to session?\n- Lines 260-266: Can the authors give more details here? Is the model trained from scratch only trained on a single session? \n- In neuroscience experiments, it is sometimes unclear if different animals are using the same dynamical strategies to perform a task. Is the model robust to different latent dynamics across sessions? Can it be used to discard sessions that are deemed too dynamically different from the other sessions?\n\nLIMITATIONS:\nSomewhat. The authors could be clearer about the limitations of their model and when they expect it to fail, for example in out-of-sample data where there may be session-specific nonlinearities.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose CANDY, a contrastive learning approach to align latent encodings of neural activity across multiple sessions and subjects by using continuous behavioral variables as a shared anchor and per-session neural encoders. In addition to the contrastive objective, there is an optional behavior prediction objective and a k-step ahead neural prediction loss obtained by additionally fitting a linear dynamical system to the latent space and a nonlinear latent-to-neural decoder. The method is validated using a mouse wheel-turning task with calcium imaging and a monkey center-out reaching task using electrophysiological data. CANDY produces latent spaces that successfully predict behavior and generalize across recording sessions and individuals. Additionally, the latent distance matrices (across timebins) better reflect behavioral distances than other methods (CEBRA and DFINE). Lastly, in ablation experiments, the authors show the contrastive loss is necessary for the good performance they observe.\n\nSTRENGTHS AND WEAKNESSES:\nContrastive learning for neural/behavioral modeling is a promising research direction with several encouraging recent results (CEBRA, MARBLE, etc.). \n\nThe various components of the method are well-justified: the session-specific encoders, the  behaviorally-anchored neural embeddings, the neural decoders, and the linear dynamical system. However, these components are all fairly standard, so I consider CANDY to have modest novelty.\n\nThe behavioral prediction results (Figures 3d-e, S3b-c) seem to be very good compared to baseline models. Additionally, the neural/behavioral alignment as visualized in Figures 3c and S4b is very impressive. Generally, the experiments seem to be high quality.\n\nMARBLE (Gosztolai et al. 2025, Nature Methods), as a contrastive learning method for neuroscience applications, is very relevant and should be cited. BLEND (Guo et al. 2025, ICLR) is also shares similar aims to CANDY. Additionally, there should be a discussion of the specific ways in which MARBLE and CEBRA differ from CANDY. Contrastive learning in neuroscience could easily fill a whole paragraph in the related works section.\n\nIs the LDS useful at all? It does not seem to help predictive performance to use the filtered latents (e.g. Figure 3 d vs. e). But it may still help the model learn a good latent space during training. I view the lack of evidence for the usefulness of the LDS as a major weakness of the paper, and would consider raising my score if can be addressed.\n\nQUESTIONS:\nDo you have evidence that the LDS improves the model? (See last paragraph of \"Strengths and Weaknesses\")\n\nWill the wheel turning data be made public upon publication? It seems like it would be a useful resource.\n\nMinor Comments\n* Figure 4a and 4b should plot the contrastive-but-not-behavioral embeddings and matrices from Figure 3 for easy comparison.\n\nLIMITATIONS:\nTraining a new encoder for each session could be cumbersome and may not work as well for sessions with neural but not behavioral data.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-03"}