Extracting task-relevant preserved dynamics from contrastive aligned neural recordings
Authors: Yiqi Jiang
Kaiwen Sheng
Yujia Gao
E. Kelly Buchanan
Yu Shikano
Seung Je Woo
Yixiu Zhao
Tony Hyun Kim
Fatih Dinc
Scott Linderman
Mark Schnitzer
Venue: NeurIPS 2025 spotlight ()
Keywords: neural latent embedding
neural latent dynamics
contrastive learning
Primary area: neuroscience_and_cognitive_science

Abstract: Recent work indicates that low-dimensional dynamics of neural and behavioral data are often preserved across days and subjects. However, extracting these preserved dynamics remains challenging: high-dimensional neural population activity and the recorded neuron populations vary across recording sessions. While existing modeling tools can improve alignment between neural and behavioral data, they often operate on a per-subject basis or discretize behavior into categories, disrupting its natural continuity and failing to capture the underlying dynamics. We introduce $\underline{\text{C}}$ontrastive $\underline{\text{A}}$ligned $\underline{\text{N}}$eural $\underline{\text{D}}$$\underline{\text{Y}}$namics (CANDY), an end‑to‑end framework that aligns neural and behavioral data using rank-based contrastive learning, adapted for continuous behavioral variables, to project neural activity from different sessions onto a shared low-dimensional embedding space. CANDY fits a shared linear dynamical system to the aligned embeddings, enabling an interpretable model of the conserved temporal structure in the latent space. We validate CANDY on synthetic and real-world datasets spanning multiple species, behaviors, and recording modalities. Our results show that CANDY is able to learn aligned latent embeddings and preserved dynamics across neural recording sessions and subjects, and it achieves improved cross-session behavior decoding performance. We further show that the latent linear dynamical system generalizes to new sessions and subjects, achieving comparable or even superior behavior decoding performance to models trained from scratch. These advances enable robust cross‑session behavioral decoding and offer a path towards identifying shared neural dynamics that underlie behavior across individuals and recording conditions. The code and two-photon imaging data of striatal neural activity that we acquired here are available at https://github.com/schnitzer-lab/CANDY-public.git.

Extracting task-relevant preserved dynamics from contrastive aligned neural recordings
Yiqi Jiang' Kaiwen Sheng" Yujia Gao E. Kelly Buchanan', Yu Shikano', Tony Hyun Kim 1. Yixiu Zhao', Seung Je Woo', Fatih Dinc SSott Linderman Mark Schnitzer"
Stannoor University CCrnegie Mellon University
University of California, Santa Barbara
"Howard Hughes Medical Institute
iygjiang, kwsheng, sw11, machnitz)@@staffoddd.ddt
Abstract
Recent work indicates that low-dimensional dynamics of neural and behavioral data often preserved across days and subjects. However, extracting these preserved dynamics remains challenging: high-dimensional neural population activity and the recorded neuron populations vary across recording sessions. While existing modeling tools can improve alignment between neural and behavioral data, they often operate per-subject basis or discretize behavior into categories, disrupting its natural continuity and failing to capture the underlying dynamics. We introduce Contrastive Aligned Neural DY (CANDY), end-to-end framework that aligns neural and behavioral data using rank-based contrastive learning, adapted for continuous behavioral variables, to project neural activity from different sessions onto a shared low-dimensional embedding space. CANDY fits a shared linear dynamical system the aligned embeddings, enabling an interpretable model of the conserved temporal structure in the latent space. We validate CANDY on synthetic and real-world datasets spanning multiple species, behaviors, and recording modalities. Our results show that CANDY is able to learn aligned latent embeddings and preserved dynamics across neural recording sessions and subjects, and it achieves improved cross-session behavior decoding performance. We further show that the latent linear dynamical system generalizes to new sessions and subjects, achieving comparable models trained from scratch. These advances enable robust cross-session behavioral decoding and offer path towards identifying shared neural dynamics that underlie behavior across individuals and recording conditions. The code and two-photon imaging data of striatal neural activity that acquired here are available at https: //tbtbb.... com/schnitzes waaeerrlaa/aaaaeea Laaa//CADYYPDD cocooy-pulcc.ge.o git.
superior behavior decoding performance to
Introduction
Understanding how complex behavior arises from high-ding sigh-dimensional neural activity is a central goal in systems neuroscience. Several studies have found that high digh-dimensional neural activity can often be reflect lower-dimensional set of latent states that evolve over time as an animal perceives, plans, and executes actions [1-7]. Notably, growing evidence indicates that behaviorally relevant latent states and dynamics are preserved across recording sessions and subjects [8-10]. However, reliably uncovering these preserved latent variables is challenging due to substantial variability in neural recordings and behavior among animals. Specifically, different sessions often measure different subsets of neurons, and trial durations and timing vary widely animals initiate or complete movements at different speeds. These discrepancies make it difficult to consistently identify and align preserved latent dynamics across time and subjects.
'These authors contributed equally to this work.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Traditionally, researchers have modeled each session separately and then aligned the latent spaces post hoc using techniques like canonical correlation analysis (CCA) [10-13]. This approach often assumes that trials have identical durations, which can limit its applicability to datasets with variable trial lengths. On the other hand, unsupervised approaches to modeling neural dynamies addresses the variability across sessions by employing session-specifer encoders to project neural activity into a common latent space, but it does not explicitly ensure that the resulting latent dynamics are behaviorally aligned [14, 15]. More recently, CEBRA aligns embeddings across sessions based on behavioral similarity; however, it discretizes behavior into categories' which overlooks the natural continuity of behaviors, and it does not model the temporal dynamics of the latent space [16]. While state-space models, such as linear dynamical systems (LDS) [17-19], switching linear dynamical systems (SLDS) [20, 21]. and nonlinear state-space systems [22, 23], model temporal structure, they lack built-in mechanisms to align dynamics across sessions.
In this study, we introduce CANDY, Contrastive Aligned Neural DYnamics, novel framework that integrates neural recordings from multiple sessions and extracts preserved, behaviorally-reeevan latent dynamics for a task. CANDY simultancously aligns multi-session embeddings and estimates latent dynamics, eliminating the need for post-hoc alignment and enabling direct recovery of interpretable temporal structures across sessions. CANDY projects neural activity from different subjects onto common space using a subject-specific encoder (Fig. IA). It uses contrastive learning to align the latent embeddings, guided by the continuous behavioral data across sessions, such that the latent embeddings capture shared, behaviorally relevant factors (Fig. IB). At the same time, CANDY learns latent dynamics with LDS to predict how latent states evolve over time (Fig. IC).
Our key contributions include: 1) Methodologically, we develop a new framework to extract preserved dynamics underlying continuous behavior across sessions and animals using rank- contrastive objective and a linear dynamical system. 2) Conceptually, we show that contrastive alignment is essential for uncovering preserved dynamical systems, not merely decoding aid. 3) Biologically, we show that a dynamical model trained on one set of animals transfers to held-out individuals, indicating the existence of preserved dynamical structures in specific brain region under given task. 4) Experimentally, we collected a new two-photon calcium imaging dataset in mice striatum across multiple days and animals.
2 Related work
Neural latent dynamics Several methods have been developed for extracting low-dimensional
latent factors from high-dimensional neural measurements [24-43]. Many methods also model the temporal dynamics of the latent factors using. for example, state-space models [24-26], sequential
variational autoencoders [14, 40], low-rank recurrent neural networks [41, 42], diffusion models [43, 44], or teacher-student distillation [45]. However, these models typically focus on uncovering the dynamics of a single recording session, instead of the dynamics that preserved across multiple
sessions.
Multi-session stitching in neuroscience Early efforts to learn shared neural latent spaces across sessions and subjects often required post-hoc alignment [9-11]. More recent approaches have projected neural data from different sessions to common space via session-specific MLPs without explicit alignment or incorporated behaviors as supervisory signals to constrain the latent space [14, 16, 46]. In parallel, there has been a trend in building large-scale transformer-ba models by integrating heterogeneous neural recordings from multiple experiments and labs [47-52). However, both approaches leave unclear which parts of the common space are actually aligned across sessions and what their behavioral relevance is. Additionally, the common latent space is less interpretable without explicit characterization of its dynamics.
foundation
Contrastive learning Contrastive learning has emerged as a powerful technique for representation learning across modalities [53-59]. Previously, it has achieved great success in aligning different discrete modalities across multiple domains [55-58]. Recently, machine learning scientists have developed regression-awarr representation learning techniques by contrasting samples against each other based on their ranking in the target space [53, 59]. Such contrastive learning has been shown to be effective for extracting shared temporal structure from time-series data across modalities in other
'Though CEBRA allows continuous behavioral variables, treats continuous labels many discrete classes.

---

A
B
Contrastive leaminng
C
embedcing
Mouse
M6.
Anchor
latent dynamics
inferred embedding
Naural activly
Noural decodar
Neural adivlly
Di:
Shared embedding space
Bahavior data
Contrastive learning
(optional:
suparvision loss)
Shared latent dynamics system
My -%."
Contraelive leaming
10 Me
simi
2D embedding space
Bahavior data
Birrbarit
Shared latent dynamics system
Kalman prediction and updata
AM
Neural activtty
Neural activity
Noural deopder
Figure 1: Illustration of the CANDY architecture that aligns latent embeddings and extracts the preserved dynamics across sessions. A. The model architecture includes aligning the neural embeddings across sessions and extracting preserved task-relevant LDS. B. Illustration of the rank-based contrastive learning [53]. For an anchor embedding (colored in purple), other embeddings are ranked by their Li distance with respect to the anchor's behavior in the ID behavior space. The loss then enforces that distances in the 2D embedding space reflect this behavioral ranking. C. A zoom-in view of the shared behavior constrained latent embedding space and shared latent dynamics system depicted in (A). The latent dynamics is modeled as an LDS with a dynamics matrix A and an emission matrix C the latent embeddings space shared across sessions.
fields, such as videos, sensing physics, and robotics planning [60-62]. In neuroscience, recent studies have explored contrastive learning for extracting meaningful neural latent representations. CEBRA [16] embeds neural activity with noncausal convolutional neural network by using a contrastive objective temporal similarity and discretized behavioral similarity. MARBLE [63] introduces geometric deep learning framework that embeds local flow fields computed from neural activities into a latent space using contrastive objectives. However, it requires post-hoc alignment for down- stream behavior decoding across sessions. While both methods produce useful low-dimensional representations, they do not explicitly model shared latent dynamics across sessions or individuals.
3 Methods
Extracting task-relevant neural dynamics that are preserved across recording sessions, spanning days and subjects, essential to understand the common strategy that underlies behavior across days and subjects. In this section, we present CANDY, an end-to-end framework that combines anchored contrastive embedding alignment with a shared LDS. In Section 3.1 we describe our nonlinear encoder, which uses a behavior-anchoree contrastive loss to align embeddings across all sessions; Section 3.2 details how we fit a shared LDS to these embeddings; and Section 3.3 presents the joint training strategy that integrates both objectives into unified optimization.
Suppose have M sessions (e.g., different days or animals), indexed by s l,..., M. Session comprises (() trials, where trial has ((.,)) time bins. At each time bin observe the high- dimensional neural activity x (s.5) E g and the associated continuous behavior y! (s.i) E IR"", where (+)) is the number of recorded neurons session s and ny is the dimensionality of the behavioral measurement. For notional simplicity, drop the trial index i in the following sections.
3.1 Extracting and aligning the nonlinear latent embeddings across sessions
First we focus learning the low-dimensional latent embeddings a!!)) E TR of the high-dimensional neural activity xi
(s) that are aligned across all M sessions and can be approximated with a shared
LDS prior. As depicted in Fig. IA, we employ an autoencoder architecture, where each session
has its own neural activity encoder foc: V + IR (a deep neural network parameterized by
(5) weights ((*) for each session) that maps xt to a(
3

---

To capture the preserved dynamics in the embedding space and to enable flexible casual inference. To ensure that ((x)) encodes the shared, behaviorally relevant signals in common space across sessions, we employ a behavior- behavior-anchored contrastive loss that draws together latent embeddings from time points with similar behaviors and separates those with dissimilar behaviors (Fig. IB). Specifically, we employ the Rank-N-Contran loss [53], which ranks every pair of embeddings a!") in each minibatch (total of N pairs) by their behavioral distance and enforces that distances in the latent space follow the same ordering. Imposing this constraint across all pairs yields an embedding space in which inter-embedding distances faithfully mirror the corresponding continuous behavior. For notational simplicity, we drop the session superscript on û, for the remainder of Section 3.1, since all sessions' embeddings are jointly aligned.
Given an anchor ün with the associated behavior Uh. for any other embedding új, we define a set of embeddings that have higher ranks than ûj in terms of the behavioral label distance with respect to the anchor as Sh, := {uz k + h, (yh. ykk 2 (yn, y/)] where d(-:-) is the L1 distance between the two behaviors. In other words, Sn.jj is defined anchor ün is larger than the distance between úp, and ūj. and hence are less similar to the anchor ún than ūj. Define the "likelihood" associate with úi given the anchor and the higher rank set as:
set of embeddings whose distance to the
exp (sim (i), ú,)) /T)
P P(ū Un Sn.s) Liyes exp (sim (#n, 0n.) /T)
where sim(..-) the similarity measure between two latent embeddings (e.g., negative Le norm) and r is the temperature parameter. Here, the likelihood will be maximized to push uy closer to the anchor un the embedding space than any other embeddings in the rank set Sh.j.
We define a contrastive objective based on the ranking of the continuous behavior samples:
N
Ccontrastvve
log P (āz un SR.j)
N(N
A= jth
This contrastive loss not only promotes behavioral relevance in the latent embeddings but also serves as an alignment mechanism across sessions by using behavior as a shared anchor.
In some experiments, we additionally incorporate behavior supervision loss that explicitly encour- ages learned embeddings to capture the behavioral outputs:
M
(,(0)
1
MSE (gT (i((0)
Liehavior
M
2=1
where (-) is a session-invariant linear mapping from latent embeddings â, to behavioral variables yt- We evaluate the effect and discuss the limitations of this behavior supervision loss in Section 4.2.3.
3.2 Learning the preserved dynamical systems across sessions on the latent embedding space
we treat the aligned latent embedding a)*) as noisy observations of single LDS that is shared across M sessions (Fig. 1C). Concretely, we assume latent state 2, E RR= that evolves and generates emheddings via
(+) Az(a) + Wes W, ~ N((,0)) Q)
2
(4)
a/6) Ozl (5) + U2.
is a ((,
(5)
with A E Ra (dynamics matrix), C E Dxn, (emission matrix), and the noise covariance Q and R shared across all sessions. We set n. = D so that the latent dynamics have the same dimensionality as the embedding space. This two-layer design enables real-time causal filtering, non-causal smoothing, and principled handling of missing data via Kalman filter, while the neural embedding captures complex structure without sacrificing tractable inference.
To causally and flexibly infer the latent state, run a standard Kalman filter forward the embeddings of each session. At each time t, the filter update gives us the causally estimated posterior
state (s) (e)
[1:]] from which we compute the filtered embedding v.()) = Cz(()

---

We learn the parameters of the shared LDS, C, RJ. by minimizing k-step-ahead neural
prediction loss, one the standard methods for training LDS [22, 64-66]. The high-dimensional
neural activity is reconstructed from neural decoder face IR + R a deep neural network with parameters (1*)) for each session:
= focr (u())
(6)
where ut is the filtered embedding. a result, the overall loss for the LDS is:
1 M -(
Σ MSE(A (o) 2taje =
Si-step
M
3.3 Combining the latent embeddings and the preserved latent dynamics
Finally, we train all components of CANDY, the encoder parameters |== the neural activity decoder parameters {Ø(*) J==1 ( the shared LDS parameters {A, C, Q, RR,, and the optional behavior
decoder parameters 7, in end-to-end manner by minimizing the combined loss:
LCANDY Li-sttp + Avcntrastive Abehavior Cochavior
(8)
optional
where Acontrastive and Abchavior are the loss weights that control the relative contributions of the align- ment and behavior- susaviiossuprrisiion terms, respectively. Notably, the behavior supervision loss is optional for alignment, whose effect and limitation will be discussed in detail in Section 4.2.3.
4 Results
We first validated CANDY using a synthetic dataset. For this initial test, a single latent dynamical system projected onto three distinet neural observation spaces using different nonlinear mappings. This experiment demonstrated that CANDY can reliably recover the shared underlying dynamics, when the observed data come from heterogeneous sources. Next, evaluated CANDY's performance on two real-world neural datasets spanning species and recording modalities: two-photon calcium imaging recordings from mouse dorsolateral striatum during a wheel-turning task that collected and made available with this paper, and publicly available clectrophysiological datasets from macaque motor cortex during a center-out reaching task [1, 3]..
Inferred lalent dynamics (causal)
Spiral ground truth dynemice
Neural observations
Acvetrvtioe
Ascernattive
Aontraative 0.0001
LOUTNN
0.0
Neuron
0.0
Neuron
1.0
-10
-1
Animal
Animal
Animal 3
Figure 2: Recovering shared latent dynamics from multiple heterogeneous observations. A. Ground-truth spiral dynamics trajectory. B. Simulated neural observations for three animals, each generated by applying a distinct nonlinear mapping from the same underlying low-dimensional trajectory in (A). C. Latent dynamics inferred by CANDY for each animal using three different contrastive loss scales Acontrastive-
4.1 Simulation experiment
To validate CANDY's ability to recover a common dynamical system from heterogeneous obser- vations, generated three synthetic "animals" follows. Given the plausibility of spiral-like

---

dynamics in neural activity [67], we first simulated two-dimensional LDS whose trajectories trace out a spiral (Fig. 2A) following equations:
cos(O) (i)
sin(0) cos(O)
Zt = A2-1,, r
with radial decay r = 0.99 and rotation angle 0 = π 17/5 15. The continuous behavioral signals ya E R were then obtained via shared linear mapping y/+ zz (see Appendix A. for details). Finally, for each animal we applied subject-specific nonlinear transformation fgc. : -R² - IR3 with Gaussian noise to produce neural observations (Fig. 2B; see Appendix A.1 for details).
CANDY was trained jointly on all three datasets with latent dimensionality D set to 2. Each subject-specific MIP encoder 110) TR3 IR2 learns to project the noisy 3D observations back onto the shared 2D manifold, leveraging a contrastive objective between latent embeddings and behaviors. As shown in the left panel of Fig. 2C, the inferred trajectories of each subject, as well as the shared LDS, were correctly identified with contrastive learning despite distinct nonlinear distortions and noise. If we set the contrastive learning loss scaler smaller, the inferred trajectories exhibit weaker alignment, and the learned LDS becomes less accurate (Fig. 2C, middle). If we set the contrastive scaler to a much smaller value, the alignment fails and the shared LDS is not accurately identified (Fig. 2C, right). This suggests that behavior- anchored contrastive learning is essential to align latent embeddings across sessions and to uncover the preserved LDS.
To evaluate the generalizability of the learned LDS to a held-out animal, we generated a new neural observation from the same spiral dynamical system with a distinct nonlinear mapping (Fig. A-A-B; see Appendix A. for details). The LDS was then frozen, and only the encoder for the new session was trained. The resulting inferred latent dynamics for this new animal were found be consistent with the dynamic flow field of the pretrained LDS (Fig. SIC), demonstrating that the LDS can generalize to new subjects.
4.2 Mouse wheel-turning task
We evaluated CANDY on mouse wheel-turning dataset collected by two-photon calcium imaging (Fig. 3A). We recorded neural activity from the dorsolateral striatum of three adult mice (IDs: 24, 25, 26): with 4 sessions each from mice 24 and 26 and 5 sessions from mouse 25, for a total of 13 sessions and approximately 3,000 neurons (~220 neurons per session). Each session comprised roughly 200 trials, during which animals rotated manipulandum to center randomly initialized visual cue on an iPad screen. Calcium signals were sampled at 30Hz, resulting in 20-40 time bins per trial aligned to cue onset and one of three trial-end events: centered, cursor exiting the screen edge (left or right), or timeout. To assess both within- and cross-subject generalization, we split data from two mice (24 and 26) to training (60%), validation (10%), and testing (30%) sets, and held out the third mouse (25) entirely for generalization evaluation. Detailed imaging parameters and preprocessing procedures are provided in Appendix A.4.
4.2.1 The alignment of latent embeddings preserved behavioral structures across sessions
To assess CANDY's ability to align latent embeddings across sessions while preserving behaviorally relevant information, we compared to two baselines: multi sestii-sssionn CEBRA [16], which also trains a shared embedding all sessions jointly, and DFINE-supervised [22], which trains separate embeddings per session. We used grid search to optimize hyperparameters for both the DFINE- supervised model and multi-session CEBRA model (see Appendix B for details). We first visualized each model's latent embeddings in one 2D space via principal component analysis (PCA) (Fig. 3B). CANDY successfully aligns latent embeddings from multiple sessions into single, coherent manifold, where there is no clear distinction in the latent embeddings of different sessions (Fig. 3B, left). Multi-session CEBRA is also able to bring latent embeddings into common subspace with no session- clusters (Fig. 3B, middle). In contrast, DFINE-supervised produces latent embeddings that occupy distinct and disjoint regions of the latent space, indicating an absence of correspondence across sessions (Fig. 3B, right).
We then quantified the embedding geometry by comparing pairwise Euclidean distances in the latent embedding space to behavioral dissimilarities computed from wheel-turn velocities (Fig. 3C). Sorting time bins by ascending velocity magnitude, plotted both the behavioral distance

---

matrix and the latent embedding distance matrix of each model to facilitate direct comparison between latent-space geometry and the true behavioral structure. The latent embeddings" distance matrix of CANDY (Fig. 3C, top right) exhibits a clear block-diagonal pattern that closely mirrors the behavioral matrix (Fig. 3C, top left) - with blocks corresponding to rightward, leftward, or stationary movements demonstrating that CANDY's latent embeddings faithfully encode behavioral structures. By comparison, multi-session CEBRA's distance matrix (Fig. 3C, bottom left) shows only week correspondence to the behavioral structure, and DUINE-supervised (Fig. 3C, bottom right) fails to recover any block structure. To quantitatively assess how well the latent embedding as the
geometry can reflect the task-relevant behavioral structure. we defined alignment Pearson correlation between the Euclidean distance matrices of behavior and the corresponding latent embeddings. We showed that CANDY achieves significantly higher alignment score than multi-session CEBRA among all latent embedding dimensionalities (Fig. S2).
iPod screen
Ground truth
(embucdieg))
CANDY
CEBRA
DFINE supervised (cmboddinng
PLS
SLD6
LDS
Whool balacdy (amv))
Animal
Hoad-Tbod mouse
Animal 2
WWW
W
Mouse: wheel-tuming task
Controller: wheal
Cumudatedd tial time
150
B
CANDY
CEBRA (muli session)
DFINE (supervised)
C
Behaviors
(sorted)
Whod speed difference
CANDY
bins sess.)
Time bins (all seco.)
Whee velocity (cmls)
aleady
slaudy
Eucldean
Time bins (all sess.
Time bins
CEBRA
DFINE
PC2
(mult session)
(supervised)
PC1
Trmee bine (all soss.)
Euclddaa dellance
Time bine (all poss.)
Euclidea dettace
Time bins (all se69.)
Time bins (ell sess.) )
E
Mult Mult-sesion oull-sesion behavor
decoding perforance
CANDY (embbedding)
Mutt-easiion behevior
decccing becodinnpparbbaaaa parfomanca (R2
[dynamics)
CANDY
0.6
0.8
(mudil-ssss)) )
DFINE supervised (dynamics)
0.4
DFINE-wupee IFPNE-opperieed
(emmbeiiiing)
Silch-LFFAD
0.0
0.0
SLDS
PCA
Latent embedding dimension
32
2
Latent dynamics dimension
64
LDS
Figure 3: CANDY aligns latent embeddings and dynamics across sessions resulting in higher behavior decoding performances. A. Left: illustration of the mouse wheel-turning task, where positive velocity indicates rightward turning and negative velocity indicates leftward turning. Right: sample behavior traces (including a zoomed-in view) from two randomly selected testing sessions. The traces of DuuvEE-upervvsed, LDS, SLDS, and PLS are from behavior decoders trained on individual sessions separately. B. Top: Scatter plot of the first two principal components (PC1 vs. PC2) of the aggregated latent embeddings across all sessions, with points colored by instantaneous wheel velocity. Bottom: Same PC1-PC2 projection as the top row but colored by session IDs. Columns: (i) CANDY, (ii) multi-session CEBRA, and (ii) DFINE-supervised.. C. Pairwise heatmaps over time bins sorted by behavioral magnitude, comparing behavioral differences (top left) and latent embedding distances of the three methods illustrated in (B). The latent embedding distance is defined as the Euclidean distance between the latent embeddings from two time bins with rows and columns ordered identically to the top left panel. D-E. Multi-session behavior decoding performance (R²) evaluated with a aession-agnostic behavior decoder (Appendix C.2.1) on (D) neural latent embeddings and (E) latent dynamics. Solid line: average over sessions. Performance for each session averaged over 5 seeds. Error bar: s.e.m over the 8 sessions.
7

---

These findings indicate that CANDY successfully aligns representations across sessions while capturing the inherent behavioral structure within its learned embeddings, all without the usage of behavioral supervision loss.
CANDY
Covernatto
Algmmeet score
Conntratton
Cumtrrocivve
Comiination
hehhorio
C behavvinr
bbchavio
Whre velocelly (emil))
04
0.0
Latent embedding dimension
deccding parforinner (RP)
Muti-session behavior
0.8
Camastar
PC2
PC1
Trial
Trial end
2
16
32
64
B
I Labont embedding dimension
Comtractive
Contenstiooe
Conntrattio
deccding performance (R3)
Lilbuvice
Lustavion
C Leluviie
Multi-seesion behavior
0.8
Time bins (all se80.)
Eudiidaan distanns
Time bine (all sees,)
10
Eudidenn distonoe
Time sees.)
10
Eudliden dietande
0.7
0.-
16
32
Time bins
Time (mil
Time (all sess.)
Latent dynemics dimension
Figure 4: Ablation study of behavior supervision and contrastive loss. A. PC1-PC2 projections of latent embeddings aggregated across all sessions. Top row is colored by instantaneous wheel velocity; bottom by session ID. Columns show (i) CANDY supervision loss, (ii) behavior supervision loss only (no contrastive loss), and (iii) unsupervised (neither behavior supervision loss nor contrastive loss). B. The pairwise distance matrix of the latent embeddings (similar Fig. 3C) extracted by the three methods mentioned in panel (A). C. Alignment score for different latent embedding dimensionalities. Shaded area: s.e.m 5 random seeds. D-E. Multi-session behavior decoding R² value evaluated with a session-agnostic behavior decoder (Appendix C.2.1) on (D) latent embeddings and (E) causally inferred latent dynamics. Solid lines: average over 8 sessions. Performance for each session is averaged over 5 seeds. Error bar: s.c.m. over the 8 sessions. Red background: disable the loss listed above; green background: enable the loss listed above.
4.2.2 Behavior decoding from latent embeddings and dynamical factors
To assess how well the aligned representations support behavior decoding, we trained a session- agnostic decoder to predict the behavioral variable each time step from either the latent embeddings or the inferred latent dynamics across all sessions (Appendix C.2.1).
CANDY demonstrates high behavior decoding performance on latent embeddings across all sessions with much lower dimensions needed than other methods (Fig. 3D). These results show that CANDY can uncover a behaviorally relevant low-dimensional structure, supporting the hypothesis that shared latent embedding space exists across sessions. Across all latent dimensions, CANDY yields higher decoding accuracy than multi-session CEBRA [16], DFINE-supervised [22], partial least squares regression (PLS), and PCA, with markedly larger gains at moderate dimensions and a diminishing margin very large dimensions where DFINE-supervised is essentially par.
Next, we evaluated the decoding performance on the causally inferred latent dynamics extracted by CANDY, { }.M 1-
1 to test the effectiveness of the preserved LDS. Similarly, the behavior decoding performance plateaus at only latent dimensions (Fig. 3E). In contrast, latent dynamics of models built on unaligned embeddings, including superieuee [22], Stitch-LFADS [14], LDS, and SLDS [26], all underperform CANDY across all dimensionalities and require substantially more dimensions to converge. We verified that all models correctly learned neural dynamics within their respective sessions by evaluating the decoding performance with session-specific decoders (Fig. S3; Appendix C.2.2 and D.3 for details). Furthermore, we demonstrated that the extracted LDS from
8

---

CANDY can forecast future behaviors with the pretrained behavior decoder, substantially better than DFINE-superviee (Fig. S4; see Appendix D.4 for details).
4.2.3 Behavior supervision loss cannot replace contrastive loss for alignment
To assess the impact of behavior supervision embedding and dynamics alignment, we compared four training regimes: (i) CANDY (Anchavior = 0), (ii CANDY with behavior-superrision loss
(Apehavior 1), (iii) CANDY without contrastive learning (Ncontrastive
0) and (iv) a fully ablated
model (Acomirastive
Abehuviur = O).
We showed that only with contrastive learning can the latent embeddings be aligned and capture the continuous behavioral structure (Fig. 3B-C and Fig. 4A-B). Contrastive learning enables the models to achieve the highest alignment score and augmenting CANDY with behavior supervision can further increase the alignment score (Fig. 4C), and thus better reflect the behavioral structure in the latent embedding space.
However, behavior supervision loss alone cannot replace contrastive learning for alignment. The model with only behavior supervision loss fails to align latent embeddings across sessions (Fig. 4A, middle), breaks the block- diak-diagonal behavioral structure (Fig. 4B, middle), and achieves a substantially lower alignment score (Fig. 4C). Interestingly, all three variants, the original contrastive- only model, the combined contrastive + behavior model, and the behavior-only model, achieve similar behavior decoding performance from the latent embeddings (Fig. 4D). Nevertheless, when we decode behavior from the inferred latent dynamics, the behavior-only model performs poorly (Fig. 4E), indicating that correct latent dynamics cannot be learned without embedding alignment via contrastive learning.
Finally, in an ablation experiment where we removed both losses entirely, the latent embeddings collapse into misaligned and behaviorally uninformative spaces (Fig. 4C-D). Together, these results underscore that while behavior supervision sharpens alignment, it cannot replace the contrastive loss: aligning embeddings via contrastive learning is critical not only for embedding-level decoding but also for uncovering the preserved LDS across sessions.
In conclusion, the contrastive loss is indispensable for initial, cross-session embedding alignment, while behavior supervision loss further enhances that alignment once established.
4.2.4 Generalization of the preserved LDS to held-out sessions.
To assess CANDY's ability to generalize to a novel subject, we first pretrained the model (with latent dimension D 8) on 8 sessions from two animals. We froze the shared LDS and the behavior decoder from the pretrained model, and trained only the subject-specife encoder and neural decoder on a held-out third animal (5 sessions).
A
Decode from the neural latent embeddings
Decode from the neural latent dynamics
Muli-eeesion behevior decoding performance (RP)
0.9
Mulli-seeston behewfor decading performance (R*)
0.9
0.0
0.8
Fixed pre trained LDS
Fixed pre reeetrannd LDS
0.6
Trein from screich
0.6
Train from acraich
10%
20%
50%
80%
100%
10%
20%
50%
80%
100%
Percenlage of training data
Percenlage training data
Figure 5: Generalization of CANDY to held-out sessions. A. Decoding performance (measured by R²) using the latent embeddings ûn as a function of the percentage of held-out session training data. Blue curve: results with the fixed pretrained LDS and session session-agnostic behavior decoder (Appendix 1.2.1) from CANDY; brown curve: training the model from scratch on the held-out data. B. Same comparison for decoding from the inferred latent dynamics z. Solid line: average over 5 sessions and each session's 's performance is averaged over 5 model seeds. Error bar: s.e.m. over the 5 sessions.
With the pretrained LDS and behavior decoder, training with just 10% of the held-out data already yields R² x 0.7. Increasing the fraction to 50% pushes performance to R² 0.8. Crucially, at every iraction-including 100%-thr the model with pretrained LDS and behavior decoder outperforms

---

model trained from scratch on the same sessions. Similarly, behavior decoding from latent dynamics (Fig. 5B) achieves relatively high R 0.8 with 50% of the data. The fact that our frozen LDS generalizes seamlessly, producing high decoding accuracy without re-estimating the LDS parameters, confirms that the temporal structure learned by CANDY truly preserved across subjects.
4.3 Monkey center-out reaching task
We also validated CANDY on a dataset of a different species, a different recording modality and a different behavioral task (see Appendix A.5 for dataset and preprocessing details). Spikes trains in the motor cortex of two monkeys are recorded during a center-out reaching task (Fig. S5A), CANDY achieves higher decoding performance of the hand velocity in x and y directions on latent embeddings over 40 sessions than multi-session CEBRA, -upNe-ssper,ised, PLS and PCA (Fig. S5A-B) and higher decoding performance on latent dynamics than -FUNE-supervised, LDS and SLDS (Fig. S5C) using a session-agnosti behavior decoder. Moreover, we demonstrated that the extracted LDS from CANDY can forecast future behaviors with the pretrained behavior decoder, substantially better than DUINE--uperviee (Fig. S6; Appendix D.4). Additionally, CANDY captures the underlying behavioral structure in the latent embedding space suggested by a substantially higher alignment score than multi-session CEBRA (Fig. S7). Furthermore, the pretrained LDS generalizes to two held-out sessions, achieving similar decoding performance on both latent embeddings and latent dynamics as the model trained from scratch on all held-out sessions (Fig. S8). These confirm that CANDY is able to align the latent embeddings and extract the preserved dynamics from various species and recording modalities.
5 Discussion
In this work, we introduced CANDY, an end-to-end framework that integrates rank-based contrastive alignment with dynamical modeling uncover a preserved LDS shared across sessions and subjects. The alignment of the latent embeddings via rank-based contrastive learning preserves the intrinsic continuity in the task-relevan behavioral data. This enables the temporal structure within the embedding space to be faithfully modeled by an interpretable dynamical system. Across synthetic dataset with distinct nonlinear observations and two real-world datasets, two-photon calcium imaging from mouse striatum, collected by ourselves, and electrophysiological spike trains from macaque motor cortex, CANDY consistently: (i) aligned latent embeddings across sessions, (ii) preserved the intrinsic structure of the continuous behavior in the latent embeddings, (iii) achieved accurate behavior decoding on both aligned latent embeddings and shared latent dynamics, and (iv) learned a preserved LDS that is generalizable to unseen sessions.
Looking ahead, several directions could further enhance CANDY's versatility and impact. First, incorporating nonlinear dynamics could capture richer temporal patterns beyond the linear Gaussian assumption. While CANDY's expressive encoders can project complex dynamics onto a linearly tractable latent space (Fig. S9; see Appendix A.2 for details), this risks obscuring the true underlying nonlinear mechanisms. Second, while extracting preserved neural computations is a primary goal, identifying the complementary subject-distineti ing the underlying strategy that govern motor behaviors. Third, while CANDY demonstrates strong generalization of the pretrained LDS and behavior decoder to unseen sessions, generalization to completely novel behavioral states poses a fundamental challenge for both CANDY and existing approaches. Specifically, we held out one moving direction of the monkey center-out reaching task entirely for testing, and demonstrated that both CANDY (lest It2: mean 1209, s.e.m. 0.0319) and CEBRA (test R2: mean -0.2138, s.e.m. 0.0248) failed to generalize to the unseen behavior category, which requires future exploration. Fourth, though CANDY primarily focused on linear behavior mapping, extending this nonlinear behavior mapping or even subject-specific behavior readout is an interesting future direction. Finally, beyond extracting shared dynamics, CANDY shows promise as diagnostic tool for discovering dynamical heterogeneity. Our synthetic experiments (Fig. S10; see Appendix A.3 for details) indicate that when presented with sessions from disparate underlying dynamics, the model provides clear signals of alignment failure, such as unaligned latent trajectories and a marked performance drop in the session-agnostic decoder. The crucial next step is to validate this capability on experimental data where heterogeneity is suspected.
dynamical strategies [68] is useful for understand-
10

---

Acknowledgments and Disclosure of Funding
Dr. Mark Schnitzer gratefully acknowledges funding from the Simons Foundation Collaboration on the Global Brain and the Vannevar Bush Faculty Fellowship Program of the U.S. Department of Defense. Yiqi Jiang and Fatih Dinc are supported by Stanford University's Mind, Brain, Computation and Technology program under the Stanford Wu Tsai Neuroscience Institute. Kaiwen Sheng is supported by Stanford Bioengineering department fellowship. Fatih Dinc is supported in part by the National Science Foundation under Grants PHY-2309 135 and 2313150, and the Gordon and Betty Moore Foundation Grant No. 2919.01 9.002 to the Kavli Institute for Thcoretical Physics (KITP). We thank Lili Tai and Yu Jiang for computing resources support and Dr. Itamar Landau for fruitful discussions.
References
[1] Mark M Churchland, John Cunningham, Matthew T Kaufman, Justin Foster, Paul Nuyu- jukian, Stephen I Ryu, and Krishna V Shenoy. Neural population dynamics during reaching. Nature, 457574 2012.
[2] Patrick T Sadtler, Kristin M Quick, Matthew D Golub, Steven M Chase, Stephen I Ryu, Elizabeth C Tyler-Kabara, Byron M Yu, and Aaron P Batista. Neural constraints on learning. Nature, 512(75155:423-426, 2014.
[3] Juan A Gallego, Matthew G Perich, Stephanie N Naufel, Christian Ethier, Sara A Solla, and Lee E Miller. Cortical population activity within a preserved neural manifold underlies multiple motor behaviors. Nature communications, 9(1):4233, 2018.
[4] Matthew Perich, Juan A Gallego, and Lee E Miller. A neural population mechanism for rapid learning. Neuron, 100(4):964 976, 2018.
[5] Oleg Rumyantsev, Jérôme A Lecoq, Osear Hernandez, Yanping Zhang, Joan Savall, Radoslaw Chrapkiewicz, Jane Li, Hongkui Zeng, Surya Ganguli, and Mark J Schnitzer. Fundamental bounds on the fidelity sensory cortical coding. Nature, 580(7801):100-105
2020.
[6] Manuel Beiran, Nicolas Meirhaeghe, Hansem Sohn, Mehrdad Jazayeri, and Srdjan Ostojic. Parametric control flexible timing through low-dimensional neural manifolds. Neuron, 111 (5):739-755 753, 2023.
[7] George Mountoufaris, Aditya Nair, Bin Yang, Dong- Wook Kim, Amit Vinograd, Samuel Kim, Scott W Linderman, and David Anderson. A line attractor encoding persistent internal state requires neuropeptide signaling. Cell, 187((1):5998--6)) 6015, 2024.
[8] James V Haxby, J Swaroop Guntupalli, Andrew C Connolly, Yaroslav O Halchenko, Bryan R Conroy, M Ida Gobbini, Michael Hanke, and Peter J Ramadge. A common, high-dimensional model of the representational space in human ventral temporal cortex. Neuron, 72(2):404 416, 2011.
[9] Alan D Degenhart, William E Bishop, Emily R Oby, Elizabeth C "Tyer-Kabaan Steven M Chase, Aaron P Batista, and Byron M Yu. Stabilization of brain-computer interface via the alignment of dimennsnni spaces of neural activity. Nature biomedical engineering, 4(7):672-665. 2020.
[10] Mostafa Safaie, Joanna C. Chang, Junchol Park, Lee E. Miller, Joshua T. Dudman, Matthew G. Perich, and Juan A. Gallego. Preserved neural dynamics across animals performing similar behaviour. Nature, 623(7988):755-771 2023.
[11] Juan Gallego, Matthew G Perich, Raeed H Chowdhury, Sara Solla, and Lee E Miller. Long-term stability of cortical population dynamics underlying consistent behavior. Nature neuroscience, 23(2):260-2)) 270, 2020.
[12] Sadegh Ebrahimi, Jérôme Lecoq, Oleg Rumyantsev, Tugce Tasci, Yanping Zhang, Cristina Irimia, Jane Li, Surya Ganguli, and Mark Schnitzer. Emergent reliability in sensory cortical coding and inter-area communication. Nature, 605(79111;713-721, 2022.

---

[13] Alex H Williams. Equivalence between representational similarity analysis. centered kernel alignment, and canonical correlations analysis. bioRxiv, pages 2024-10, 2024.
[14] Chethan Pandarinath, Daniel J O' Shea, Jasmine Collins, Rafal Jozefowicz, Sergey D Stavisky, Jonathan C Kao, Eric M Trautmann, Matthew T Kaufman, Stephen Ryu, Leigh R Hochberg, et al. Inferring single-trial neural population dynamics using sequential auto-encoders. Nature methods, 15(10):805-815 2018.
[15] Brianna M Karpowicz, Yahia H Ali, Lahiru N Wimalasena, Andrew R Sedler, Mohammad Reza Keshtkaran, Kevin Bodkin, Xuan Ma, Daniel B Rubin, Ziv M Williams, Sydney S Cash, al. Stabilizing brain-computer interfaces through alignment of latent dynamics. Nuture Communications, 16(1):4662, 2025.
[16] Steffen Schneider, Jin Hwa Lee, and Mackenzie Weygandt Mathis. Learnable latent embeddings for joint behavioural and neural analysis. Nature, 61767960):30--366,, 2023.
[17] Jakob H Macke, Lars Buesing, John P Cunningham, Byron M Yu, Krishna V Shenoy, and Maneesh Sahani. Empirical models of spiking in neural populations. Advances in neural information processing systems, 24, 2011.
[18] Yuanjun Gao, Evan W Archer, Liam Paninski, and John P Cunningham. Linear dynamical neural population models through nonlinear embeddings. Advances in neural information processing systems, 29, 2016.
[19] João D Semedo, Amin Zandvakili, Christian K Machens, Byron M Yu, and Adam Kohn. Cortical areas interact through a communication subspace. Neuron, 102(1):249-259, 2019.
[20] Joshua Glaser, Matthew Whiteway, John P Cunningham, Liam Paninski, and Scott Linderman. Recurrent switching dynamical systems models for multiple interacting neural populations. Advances in neural information processing systems, 33: 33:14867-14878, 2020.
[21] Amber Hu, David Zoltowski, Aditya Nair, David Anderson, Lea Duncker, and Scott Linderman. Modeling latent neural dynamics with gaussian process switching linear dynamical systems. Advances in Neural Information Processing Systems, 37:33805-33835, 2024.
[22] Hamidreza Abbaspourazad, Eray Erturk, Bijan Pesaran, and Maryam M. Shanechi. Dynamical flexible inference of nonlinear latent factors and structures in neural population activity. Nature Biomedical Engineering. 8(1):85-100 2024.
[23] Orren Marniol-Tambout David M Zoltowski, E Mika Diamanti, Lucas Pinto, Carlos D Brody, David W Tank, and Jonathan Pillow. Modeling state-dependent communication between brain regions with switching nonlinear dynamical systems. In The Twelfth International Conference on Learning Representations, 2024.
[24] Omid G Sani, Hamidreza Abbaspourazad, Yan T Wong, Bijan Pesaran, and Maryam M Shanechi. Modeling behaviorally relevant neural dynamics enabled by preferential subspace identification. Nature Neuroscience, 24(1):140-11 149, 2021.
[25] Parsa Vahidi, Omid G Sani, and Maryam M Shanechi. Modeling and dissociation of intrinsic and input-driven neural population dynamics underlying behavior. Proceedings of the National Academy of Sciences, 121(7):e2222881122 2024.
[26) Scott Linderman, Matthew Johnson, Andrew Miller, Ryan Adams, David Blei, and Liam Paninski. Bayesian learning and inference in recurrent switching linear dynamical systems. In Artificial intelligence and statistics, pages 914-922. PMLR, 2017.
[27] Byron M Yu, John P Cunningham, Gopal Santhanam, Stephen Ryu, Krishna V Shenoy, and Maneesh Sahani. Caussian-proces: factor analysis for low-dimensional single-trial analysis of neural population activity. Advances in neural information processing systems, 21, 2008
[28] Liam Paninski, Yashar Ahmadian, Daniel Gil Ferreira, Shinsuke Koyama, Kamiar Rahnama Rad, Michael Vidne, Joshua Vogelstein, and Wei Wu. new look at state-space models for neural data. Journal of computational neuroscience, 29: 19:1071126, 2010.
12

---

[29] David Sussillo and Omri Barak. Opening the black box: low-dimensional dynamics in high- dimensional recurrent neural networks. Neural computation, 25(3):626-649, 2013.
[30] John P. Cunningham and Byron M. Yu. Dimensionality reduction for large-scale neural recordings. Nature Neuroscience, 17011):100-1509 2014.
[31] Carsen Stringer, Marius Pachitariu, Nicholas Steinmetz, Matteo Carandini, and Kenneth D Harris. High-dimensional geometry population responses in visual cortex. Nature, (7766)))): 361-365, 2019.
[32] Saurabh Vyas, Matthew D Golub, David Sussillo, and Krishna V Shenoy. Computation through neural population dynamics. Annual review of neuroscience, 43(1):249 2020.
[33] Lea Duncker and Maneesh Sahani. Dynamics on the manifold: Identifying computational dynamical activity from neural population recordings. Current opinion in neurobiology, 70: 163-170, 2021.
[34] Timothy D Kim, Thomas Z Luo. Jonathan W Pillow, and Carlos D Brody. Inferring latent dynamics underlying neural population activity via neural differential equations. In International Conference Machine Learning, pages 5551-5561. PMLR, 2021.
[35] Carsen Stringer and Marius Pachitariu. Analysis methods for large-scale neuronal recordings. Science, x8666722)ecdd7722, 2024.
[36] Joana Soldado -Mggranne, Valerio Mante, and Mancesh Sahani. Inferring context-dependent computations through linear approximations of prefrontal cortex dynamics. Science Advances, 10($1)xead14743, 2024.
[37] Amit Vinograd, Aditya Nair, Joseph H Kim, Scott W Linderman, and David Anderson. Causal evidence of a line attractor encoding an affective state. Nature, 634(8035):910-118 2024.
(38] Omid G Sani, Bijan Pesaran, and Maryam M Shanechi. Dissociative and prioritized modeling of behaviorally relevant neural dynamics using recurrent neural networks. Nature neuroscience, 27(10):20332000 2045, 2024.
[39] Parsa Vahidi, Omid G Sani, and Maryam Shanechi. Braid: Input-driven nonlinear dynamical modeling of behavirrl data. In The Thirteenth International Conference on Learning Representations, 2025.
[40] Cole Hurwitz, Akash Srivastava, Kai Xu, Justin Jude, Matthew Perich, Lee Miller, and Matthias Hennig. Targeted neural dynamical modeling. Advances in Neural Information Processing Systems, 34:29379-29392. 2021.
[41] Adrian Valente, Jonathan W Pillow, and Srdjan Ostojic. Extracting computational mechanisms from neural data using low-rank rnns. Advances in Neural Information Processing Systems, 35: 2022.
[42] Fatih Dine, Adam Shai, Mark Schnitzer, and Hidenori Tanaka. Cornn: Convex optimization of recurrent neural networks for rapid inference of neural dynamics. Advances in Neural Information Processing Systems, 36:5 11111177311000 2023.
[43] Yule Wang, Zijing Wu, Chengrui Li, and Anqi Wu. Extraction and recovery of spatio-temporal structure latent dynamics alignment with diffusion model. Advances in Neural Information Processing Systems, 36, 2024.
[44] Jaivardhan Kapoor, Auguste Schulz, Julius Vetter, Felix Pei, Richard Gao, and Jakob H Macke. Latent diffusion for neural spiking data. Advances in Neurul Information Processing Systems, 37:1 181 18119-118155 2024.
[45] Zhengrui Guo, Fangxu Zhou, Wei Wu, Qichen Sun, Lishuang Feng, Jinzhuo Wang, and Hao Chen. Blend: Behavior-guides neural population dynamics modeling via privileged knowledge distillation. arXiv preprint 2327 2024.
13

---

[46] Daiyao Yi, Simon Musall, Anne Churchland, Nancy Disentangled multi-subjectt and social behavioral representations through constrained subspace variational autoencoder (cs-vae). bioRxiv, pages 2022-09, 2022.
and Shreya Saxena.
[47] Mehdi Azabou, Vinam Arora, Venkataramana Ganesh, Ximeng Mao, Santosh Nachimuthu, Michael Mendelson, Blake Richards, Matthew Perich, Guillaume Lajoie, and Eva Dyer. A unified, scalable framework for neural population decoding. Advances in Neural Information Processing Systems, 36, 2024.
[48] Joel Ye, Jennifer Collinger, Leila Wehbe, and Robert Gaunt. Neural data transformer 2: multi- context pretraining for neural spiking activity. Advances in Neural Information Processing Systems, 36, 2024.
[49] Antonis Antoniades, Yiyi Yu, Joseph Canzano, William Wang, and Spencer LaVere Smith. Neuroformer: Multimodal and multitask generative pretraining for brain data. International Conference on Learning Representations, 2024.
[50] Yizi Zhang, Yanchen Wang, Donato Jiménez-Benetó, Zixuan Wang, Mehdi Azabou, Blake Richards, Renee Tung, Olivier Winter, Eva Dyer, Liam Paninski, et al. Towards a" universal translator" for neural dynamics at single-cell, single-spike resolution. Advances in Neural Information Processing Systems, 37:80495-8055 2024.
[51] Mehdi Azabou, Krystal Xuejing Pan, Vinam Arora, Ian Jarratt Knight, Eva I. Dyer, and Blake Aaron Richards. Multi-session, multi-task neural decoding from distinct cell-types and brain regions. The Thirteenth International Conference on Learning Representations, 2025.
[52] Eric Y Wang, Paul G Fahey, Zhuokun Ding, Stelios Papadopoulos, Kayla Ponder, Marissa A Weis, Andersen Chang, Taliah Muhammad, Saumil Patel, Zhiwei Ding, et al. Foundation model of neural activity predicts response new stimulus types. Nature, 64008058):440-477,, 2025.
[53] Kaiwen Zha, Peng Cao, Jeany Son, Yuzhe Yang, and Dina Katabi. Rank-n-contrast: learning continuous representations for regression. Advances in Neural Information Processing Systems, 36, 2024.
[54] Phuc H. Le-Khac, Graham Healy, and Alan F. Smeaton. Contrastive representation learning: framework and review. IEEE Access, 8: 199997,-193934, 2020.
[55] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference machine learning. pages 8748-80 8763. PmL.R, 2021.
[56] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pages 15597-607. PmLR, 2020.
[57] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages
2020.
[58] Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. Advances in neural information processing systems, 33:18661-18673, 2020.
[59] Chenggang Chen, Zhiyu Yang, and Xiaoqin Wang. Neural embeddings rank: Aligning 3d latent dynamics with movements. Advances in Neural Information Processing Systems, 37: 141461-141489, 2024.
[60] Yuchong Sun, Hongwei Xue, Ruihua Song, Bei Liu, Huan Yang, and Jianlong Fu. Long-form video-language pre-training with multimodal temporal contrastive learning. Advances in neural information processing systems, 35:38032-38045, 2022.
14

---

[61] Shengzhong Liu, Tomoyoshi Kimura, Dongxin Liu, Ruijie Wang. Jinyang Li, Suhas Diggavi, Mani Srivastava, and Tarek Abdelzaher. Focal: Contrastive learning for multimodal time- series sensing signals in factorized orthogonal latent space. Advances in Neural Information Processing Systems, 01377399473334 2023.
[62] Benjamin Eysenbach, Vivek Myers, Ruslan Salakhutdinov, and Sergey Levine. Inference via interpolation: Contrastive representations provably enable planning and inference. Advances in Neural Information Processing Systems, 37:58901-58925 2024.
[63] Adam Gosztolai, Robert Peach, Alexis Amaudon, Mauricio Barahona, and Pierre Van- dergheynst. Marble: interpretable representations of neural population dynamies using geomet- ric deep learning. Nature Methods, pages 1-9, 2025.
[64] Yingying Li and Na Li. Leveraging predictions in smoothed online convex optimization via gradient-basee algorithms. Advances in Neural Information Processing Systems, 33:14520 14531. 2020.
[65] Ayesha Vermani, I1 Memming Park, and Josue Nassar. Leveraging generative models for unsupervised alignment of neural time series data. In The Twelfth International Conference on Learning Representations, 2023.
[66] Danijar Hafner, Timothy Lillicrap, Ian Fischer, Ruben Villegas, David Ha, Honglak Lee, and James Davidson. Learning latent dynamics for planning from pixels. In International conference machine learning, pages 2555-2565. PMLR, 2019.
[67] Abigail A Russo, Ramin Khajeh, Sean R Bittner, Sean M Perkins, John Cunningham, Lau- rence F Abbott, and Mark M Churchland. Neural trajectories in the supplementary motor area and motor cortex exhibit distinct geometries, compatible with different classes of computation. Neuron, 107 758, 2020.
[68] Marino Pagan, Vincent D) Tang, Mikio C Aoi, Jonathan W Pillow, Valerio Mante, David Sussillo, and Carlos D Brody. Individual variability of neural computations underlying flexible decisions. Nature, 639(8054):421-422) 2025.
[69] Fatih Ding, Hakan Inan, Oscar Hernandez, Claudia Schmuekermair, Omer Hazon, Tugce Tasci, Biafra Ahanonu, Yanping Zhang, Jérôme Lecoq, Simon Haziza, et al. Fast, scalable, and statistically robust cell extraction from large-scale neural calcium imaging datasets. bioRxiv, 2021.
[70] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv, 2014.
15

---

Appendix
The appendix of this paper is structured follows:
Section A introduces the evaluated datasets.
Section B details the compared baselines in our experiments.
Section C summarizes the training configuration we applied in this paper.
Section D presents more comprehensive evaluation results.
Section discloses compute resources used for this paper.
A Dataset
A.1 Simulated spiral dynamics
We generated latent trajectories by simulating the spiral dynamics from Eq. 9 for 50 time steps starting from an initial state 20 (0.5, 0.5). For each trial, we added independent Gaussian noise (0, df with = 0.01 to the resulting latent states.
The behavior variable y generated from the latent state via linear mapping:
W + Ey
(SI)
where W is defined as identity matrix of size Rx and cy is gaussian noise sampled from (00, 12) with og = 0.001.
The neural observation xz for each session were generated via:
= Later (Zx) - e
(S2)
with the nonlinear mapping for each session defined as
24 0
Lacus
2, [
(S3)
sin
20 cos 2, [1])
2,[0]
famz)
0.872, (0)
0.5
(2(()) 0.5 co 2, [1)
(S4)
(0.5z, + 0.87 sin
+2((0) 0.87 cos 2 [1]]
0.71 2/ [0] 0.71 sin 2, + 0.71 cos(2 (1)
fam
2,
(S5)
-0.71 00] + 0.71 sin (0) + 0.71 cos(2 =;[1]]
The observation noise Ex
was sampled independently for each session from N/00, 1213) with
T₂ 0.1.
For the held-out animal in Fig. S1, the nonlinear mapping is defined as
24 00
f000
2, (0) sin 2 (1])
(S6)
sin
A.2 Simulated limit cycle
We simulated the limit cycle dynamics using the Van der Pol oscillator as
22
dt
(S7)
dz2
μ(1
22) 21
dt
16

---

where 1.0. We numerically integrated these equations from an initial condition of 20 (0.05, 0.05) for 200 time steps generate the latent limit cycle trajectory. Independent Gaussian noise ~ ~O, with Ø= = 0.01 was added to the resulting latent trajectories for each trial.
The behavioral variable y/+ was generated according Eq. S1 with oy 0.001. The neural observation for each session was generated according to Eq. S2 with the session- nonlinear mappings focn), fares, and fac corresponding to those defined in Eqs. S3-S5, respectively. The observation noise was sampled independently for each session from (00, TxT3) with 0₂ 0.1.
A.3 Simulated multiple dynamics
We simulated three sessions from two distinct dynamical systems: spiral dynamics described in Appendix A. and limit cycle dynamics as described in Appendix A.2. The neural observations of two sessions were generated from the spiral dynamics 2-piral using the nonlinear mappings of Eq. S3 and S4. The neural observation of the third session was generated from the limit cycle dynamics eycle using a distinct nonlinear neural observation mapping: 21
2x 0
21 [1
(S8)
coos(2 2, 20 + sin (2- 2, [1]] (5)
For all three sessions, the observation noise ex sampled independently from (0, aala with
Tz = 0.1 and the behavioral variable y/ was generated according to Eq. S1 with By = 0.001.
A.4 Calcium imaging data acquisition and preprocessing
Calcium imaging was performed using custom- bbilt two-photon microscope, with excitation at 920 nm and fluorescence emission peaking at 512 nm (jGCaMPSm). The imaging frame size was 512 x 512 pixels with spatial resolution of 0.8 μm per pixel, and data were sampled at 30 Hz. A GRIN lens (4 mm length. mm diameter; Inscopix) was chronically implanted over the right hemisphere to target the right dorsolateral striatum. To express jGCaMP88 in medium spiny neurons, an AAV2/PHPeBBCCaM MBB-CaaalljGGCMPPS virus (5.2 x 10 gc/ml.) was injected at stereotaxic coordinates of AP +1.0 mm, MI 2.25 mm, and DV -2.4 mm from the brain surface. Following data acquisition, the imaging movies underwent preprocessing. including fluorescence decay correction, motion correction, and z-scoring. Neuronal regions of interest (ROIs) were automatically extracted using the EXTRACT algorithm [69] implemented in MATLAB 2022a, followed by manual curation to ensure accurate cell identification.
A.5 Monkey reaching task data preprocessing
We used clectrophysiological recordings from the motor cortex collected during center-out reaching task [1, 3, 4], in which monkeys were trained to move a manipulandum toward one of eight targets displayed screen. The neural and behavioral data 48324/dande http//andd........... 000688/0.......... 250122. 1735. For training, we used 40 sessions from this task recorded from subjects C and M. sessions from these subjects were held out entirely for evaluating the generalization of the pretrained LDS. A fixed 100 ms delay between neural activity and movement was assumed across all sessions. Spiking activity was binned in 20 ms intervals, smoothed with 50 ms causal Gaussian filter, and then downsampled by a factor of two to obtain firing rates. Neurons with firing rates below 0.1 Hz were excluded, and all neural data were aligned to the go cue onset. The ratio of training, validation and testing datasets is 6:1:3.
obtained from: https:/ / /doi. org/ 10.
B Baselines
For each model, we selected the hyperparameters using grid search. We trained each model five times with five random seeds. The performance was averaged over all sessions and all seeds.
B.1 LDS and SLDS
We leveraged publicly available code lhttps://githu/ dynamics matrix to be Gaussian and the emission matrix to be orthogonal Gaussian. The number of
and set the
17

---

states in the SLDS model is selected between 2, and based on the behavior decoding performance in the validation set. We used Laplace EM algorithm with 50 iterations for training and structured mean field as the variational posterior for both LDS and SLDDS.
B.2 multi-session CEBRA
CEBRA [16] learns behaviorally relevant embeddings by applying contrastive loss to finite windows of neural traces, thus non-causal. Because it relies on a feedforward convolutional network with fixed receptive field, CEBRA does not capture explicit temporal dynamics beyond the input window. At each gradient step, positive and negative pairs are drawn using a mixed time-behavior conditional sampler, which ensures that positive samples share the same discrete behavioral label and are also constrained within defined time offset. We selected the model parameters using grid search on the hyperparameters. Eventually, the model is trained via the publicly available package (https: : //ebra. a htt//doc/isstaaa iaaitaaataallnnnnnnnooooo html). We trained 10mdeel for 5000 iterations, with batch size for contrastive learning of 2048, cosine distance as the contrasive distance metric, using the Adam optimizer of learning rate 3 x 10-1 on both mouse and monkey datasets.
B.3 DFINE-supervised
DFINE-supervised [22] learns behaviorally-reelvan nonlinear embeddings and infers an LDS for single session. The nonlinear embeddings are learned via an MLP. We used the publicly available code (https: //tps:://t/ httt::///ghhttt thub. com/ ShanechiLab/torchDD supervised on all datasets via grid search. For the mouse wheel-turning tasks, the model is trained with aheedd prediction loss, Adam optimizer [70] with learning rate of 2 x 10-3 10-3, le regularization scale of 5 x 10-3, behavior loss scale Apehavior = 1, encoder and decoder MLP with hidden layers size [64, 16), leaky ReLU activation function and batch size of for 500 epochs. For the monkey center-out reaching task, the model is trained with 2-steps-ahead prediction loss, Adam optimizer with learning rate of 2 x 10-3, 3, lg regularization scale of x 10-2, behavior loss scale Abchavior 1, encoder and decoder MLP with hidden layers size [32, 32], mish activation function and batch size of 32 for 200 epochs.
We tuned the hyperparameters of DFINE-
B.4 Stitch-LFADS
Stitch-LFADS [14] inferred common RNN-based dynamics across all sessions by leveraging session- specific linear read-in and linear read-out for the neural recordings. We used the publicly available code (https: //t /i thub. ttn/aasedleer /1 -frchh We tuned the hyperparameters via grid search. For the mouse wheel-turning task, the model is trained with Adam optimizer [70] with learning rate 4 x 10-3, learning rate decay as 0.998, number of epochs as 3000, gradient clip as 5, dropout rate 0.01, dimensionality of the encoding data as 64, time steps to be used for only the initial condition encoder as O, dimensionality of the initial condition encoder 4, dimensionality of the controller input encoder as 64, lag of the controller input as 1, dimensionality of the controller as 16, dimensionality of the controller output 16, dimensionality of the initial condition 32, dimensionality of the generator as 32, prior distribution for the controller output and for the initial condition multivariate normal distribution, scaling factor for the KL divergence regularization of the initial condition as 10-3, 3 scaling factor for the KL divergence regularization the controller output as 10 3, &g regularization scale as 10
C Training configurations
C.1 Model hyperparameters
We tuned our model with grid search for all datasets, including the simulation dataset, the mouse dataset, and the monkey dataset.
For the simulation spiral dynamics dataset, we used Adam optimizer [70] with learning rate of 2 x 10-3, f2 regularization scale of 5 x 10-3, encoder and decoder MLP with hidden sizes of [3, 2). 500 training epochs, leaky ReLU activation function, batch size of 32, 2048 time points for contrastive learning, contrastive temperature 0.1, contrastive scale Acontrastive E (0.1, 0.01, 0.0001), )), & the
18

---

g(+) arg min contrastive distance measure, & as the contrastive similarity measure, 4-steps-ahead prediction loss, and Ahehavior 0, i.e., without behavior supervision loss.
For the mouse wheel-turning dataset, we used Adam optimizer [70] with learning rate of 2 x 10-3, & regularization scale of 5 10-3 encoder and decoder MLP with hidden sizes of [64, 16], 500 training epochs, leaky ReLU activation function, batch size of 32, 2048 time points for contrastive learning, contrastive temperature 7 0.2, contrastive scale Acontrastive 0.1, i as the contrastive distance measure, &g as the contrastive similarity measure, 4-steps-ahead prediction loss, and Abchavior = 0 for Fig. and Abehavior for Fig. when the behavior supervision loss is enabled.
For the monkey center-out reaching dataset, we used Adam optimizer [70] with learning rate of 2 x 10 lz regularization scale of 2 x 10 encoder and decoder MLP with hidden sizes of [32, 32), 200 training epochs, mish activation function, batch size of 32, 2048 time points for contrastive learning, contrastive temperature
0.5, contrastive scale Acontrativee 0.1, & the contrastive distance measure, &g as the contrastive similarity measure, Abeluviue = 1, and 2-steps-ahead prediction loss.
C.2 Evaluating behavior decoding performance of latent embeddings and latent dynamies
We defined two types of behavior decoding performance evaluations: session-agnostic decoding (universal, trained across all sessions) and session-specific decoding (trained independently per session).
C.2.1 Session-agnostic behavior decoding
We concatenated latent embeddings or latent dynamics across all sessions into design matrix U E RNDD, where N = Σ (.n) T(+.), The corresponding behavioral targets y E R RNxxg concatenated in the same order. Each column of U was standardized to zero mean and unit variance, and we fit an f--regularzzed linear model (Lasso):
B, arg min
y - U3 + A|[8||,,
where À was selected via 10-fold cross vrlss-validation on the training data. Decoding performance was then reported on the held-out test set.
C.2.2 Session-specific behavior decoding
For each session s E {l.... M}. we concatenate latent embeddings or latent dynamics across all sessions into a design matrix U(+) N where (( Σn ( (+..)) The corresponding
behavioral targets yl() e [g were concatenated in the same order. Each column of U was standardized to zero mean and unit variance, and we fit an f-rrgularizzd linear model (Lasso);
3 N(s) - + A|[l||2,
D Additional experimental results
D.1 Simulation experiment nonlinear dynamics
To show that CANDY can align the latent dynamical factors trajectories generated from a nonlinear dynamical system, we simulated three distinct noisy neuronal observations generated from the same underlying limit cycle dynamics (see Appendix A.2 for details). The latent dynamical factors are aligned and correctly inferred even though CANDY uses LDS model the dynamics (Fig. S9), showing the ability of the model to characterize potential nonlinearities in the dynamics with the nonlinear encoder design choice.
D.2 Simulation experiment differentiation of different latent dynamics
To assess whether CANDY can detect when sessions employ different underlying dynamical strategies, conducted controlled synthetic experiment. We generated three sessions of neural activity with
19

---

identical linear behavioral readouts but distinct latent dynamics. Two sessions were simulated using a spiral attractor, while the third followed a limit-cycle attractor (see Appendix A.3 for details).
We trained CANDY jointly on all three sessions. The inferred latent trajectories revealed that the Two spiral sessions were consistently aligned, whereas the limit-cycle session was not aligned with them (Fig. S10A Left). On the other hand, when the three sessions are all generated from the spiral dynamics, the latent trajectories are aligned (Fig. S10A Right). This indicates that CANDY can aligned sessions with shared dynamics while isolating sessions that deviate from the common structure.
We next compared behavioral decoding using both session session-agnostic decoder and session-specific decoders (Appendix C.2). When latent dynamics were not shared across sessions (spiral vs. limit- cycle), session-specifee decoders achieved significantly higher accuracy than the session-agnostic decoder (p < 10-1, Wilcoxon signed-rank test; Fig. S10B). In contrast, when all three sessions generated from spiral dynamics, the two decoding strategies achieved nearly identical performance (no significant difference, Wilcoxon signed-rank test; Fig. S1OB).
These results demonstrate that CANDY not only recovers shared latent dynamics but may also serve as diagnostic tool to flag sessions (or subjects) that adopt different dynamical strategies. We view this as a promising extension of our framework for large-scale, cross-subject neural analyses.
D.3 Behavior decoding performance of single sessions
For the mouse wheel-turning task, we additionally trained session-specific behavior decoders on individual datasets (see Appendix C.2.2). This analysis serves as control to verify that all models were trained and evaluated correctly (Fig. S3). The results further support our main evaluation strategy, where session-agnostic agnostic behavior decoding (Appendix C.2.1) provides principled measure of alignment across sessions.
D.4 Estimated dynamics flow field and forecasting results
To assess whether CANDY correctly learned the preserved dynamics across sessions, we performed neural-to-behavior forecasting evaluation both the mouse wheel-turning dataset and the monkey center-out-reaching dataset. By selecting the best model parameters and initialization seed using the validation set, showed that the trajectory of latent dynamical factors from each trial (21:)) follows the dynamical flow field closely by projecting the original latent dimensions onto the top two principal components for visualization (Fig. S4A & Fig. S6A). We then performed causal k-step forecasting using the trained LDS and the behavior decoder, applied to filtered latent factors to ensure strict causality. These analyses demonstrated that the learned LDS in CANDY enables effective neural-to -o-bavvio forecasting, consistently outperforming DFINE-supervised (Fig. S4B and Fig. S6B; Appendix C.2.1; Appendix C.2.2).
D.5 Random shuffling behavior decoding performance
To confirm that CANDY's embeddings indeed capture genuine neural-behavioral relationships rather than model artifacts, we conducted a shuffling control experiment. Specifically, we randomly permuted the behavioral data relative to the neural recordings 20 times disrupt their correspondence. Under these conditions, the model failed to converge: the training loss remained high, the learned embeddings lacked coherent structure, and no interpretable session alignment emerged (Fig. S1 S11A- B). Furthermore, the behavior decoding R² value dropped to near zero when evaluated across sessions, both from neural latent embeddings and from latent dynamics (Fig. S1IC-D). These findings demonstrate that CANDY's ability extract structured, behaviorally meaningful embeddings critically depends on the true correspondence between neural activity and behavior.
D.6 Analysis of temporal structure in the latent space
To examine whether CANDY's 's latent space captures meaningful temporal structure beyond behavioral alignment, we analyzed the similarity between embeddings across time within individual trials. For each trial, we computed the pairwise Fuclidean distance between embeddings at all time bins, resulting in temporal similarity heatmap (Fig. S12A). These heatmaps show that embeddings from
20

---

temporally adjacent bins remain close in latent space, while embeddings from bins further apart in time become progressively more distant.
To quantify this relationship, we averaged Euclidean distances as a function of temporal separation across all trials and sessions (Fig. S12B). The results reveal clear monotonic increase in distance with temporal lag, indicating that CANDY produces embeddings that evolve smoothly over time. This temporal continuity is consistent with the latent dynamical system (LDS) component of the model and demonstrates that the embeddings preserve not only behavioral similarity but also intrinsic trial structure.
E
Resources for reproducibility
Most experiments performed in this work were performed on Stanford University Sherlock CPU clusters. Running the experiment of contrastive batch size equal to 2048 requires 120GB memory. Some experiments were performed on desktop with Intel(R) Core(TM) i9-10900X CPU with 10 physical cores, 256G RAM, and an NVIDIA GeForce RTX 4070 GPU, or a desktop with an AMD Ryzen 9 9950X 16-core 32-thred CPU, 192G RAM, and NVIDIA GeForee RTX 4070 Ti GPU.
21

---

Supplementary Figures
A
B
C
dynamics (cousa)
Spiral ground fruh dynamics
Noural observations
Pretrained animals
animal
Neuron 3
Nauurnn 2
Newron
Animal 1
Antmal
Animal
Antmal 4 (hald-out animal)
Figure S1: The pretrained LDS generalizes unseen synthetic animals. A. Ground-truth spiral dynamics trajectory. B. Simulated neural observations for three pretrained animals and one held- out animal, each generated by applying a distinet nonlinear mapping from the same underlying low-dimensional trajectory in (A). C. The dynamic flow field of the LDS pretrained on the three animals and their corresponding latent dynamics inferred by the pretrained CANDY (left) and the latent dynamics of the held-out animal inferred by the fine-tuned CANDY (right).
0.8
Figure S2: Quantifying the alignment of latent embeddings to behavioral structure. We compared the align- ment score for CANDY (blue) and multi-session CEBRA (purple) using the mouse wheel sioeel-trnnin dataset. Solid lines: the average alignment score over 5 random seeds. Shaded areas: s.e.m. over 5 random seeds.
CEBRA
(multi sosii-ssssoon
Alignment
0.4
CANDY
0.0
Latent embedding dimension
Single session behavior decoding performance (R²)
CANDY (ombadsing))
decoding performnnc (R*)
CANDY (dynamics)
Single session behavior
0.8
0.8
CEBRA (muli-snss)
DRINE (dynamics)
DRINE (ombedding)
Btich-LFADS
PLS
SLDS
16
64
16
32
C4
LD8
embedding
Latent dynamics dimension
Figure S3: Behavior decoding performance on single session. A. The behavior decoding R² on single sessions by training behavior decoder for each session separately. B. The behavior decoding R² on single sessions by training behavior decoder for each session separately. Solid line: average over 8 sessions; for each session, model performance averaged over random seeds. Error bar: s.c.m over 8 sessions.
22

---

B
CANDY
Behtavirr decoding performance (R1
DFINE-uupenised (sessionssgnosic decodar)
PC2 PC1
DFINE-aupervsed (s0 wow.ion--peeffccco dacader)
02-
Triil start
Trial and
0.0
Rightward trial
Leflworrd
k-stop ahoad foracasting
Figure S4: CANDY successfully learned the preserved dynamics enabling high neural-to- behavior forecasting in the mouse wheel-turning dataset. A. Visualization of the dynamics flow field in the latent subspace, projected onto the first PCs of both the latent trajectories (21:1) and the learned flow field. Trial types are color-coded (green: rightward trial; pink: leftward trial), with the trial start (red dot) and the trial end (gray dot) marked. The original latent dimension is four. B. Solid line: neural- to-bbehvviorrr forecasting using session session-agnostic decoder (Appendix C.2.1) across all sessions (solid line). Dash line: neural-to-behan (Appendix C.2.2). Performance of CANDY (blue) is compared with DFINE- (orange). Line: average over 8 sessions. Shaded area: s.e.m over sessions.
forecasting using session-specins decoders
Ground Iruth
CANDY (ambodding)
CEBRA (mulli-eess)
DFINE superrppddd (embedding)
PLS
SLDS
LDS
velocty
10
20
a
Monkey center-out reach task
Controler: Maripuiantum
10
Time
(8)
decoding performanse (82)
CANDY (ambeddingg)
Mutt-ssesion behaviar
0.0
CANDY (dynamics)
Mulli-setsion behavior
CEBRA (mulii-ess
DFINE DFINE-supervised (dynamics)
DFINE-aupervied
(ombedding)
necoding
SLDS
PLS
0.0
PCA
16
Letent embedding dimension
Latent dynamics dimension
Figure S5: CANDY achieves higher behavior decoding performances in the monkey center- out reaching task. A. Left: illustration of the monkey center-out reaching task. Right: sample behavior traces (including a zoom-in view) on the testing datasets from two randomly selected sessions. The model and the behavior decoder of DFINE-supervised. LDS, SLDS, and PLS are trained on individual sessions separately. Top and bottom rows in each subplot are the velocity in x and y directions respectively. B-C. Multi-session behavior decoding performance (R²) evaluated with universal linear behavior decoder (Appendix C.2.1) on (B) latent embeddings and (C.) latent dynamics. Solid line: average over 40 sessions; for each session, model performance is averaged over random seeds. Error bar: s.e.m. over the 40 sessions.
23

---

Behavior decoding perforaanee (R2)
0.5
CANDY
0.0
(session-agnosti decoder)
2
DFINE- //Neesupd (session-specifie decoders)
-0.5
L PC
Trial start
-1.5
Trial end
K-step ahead forecasting
Figure S6: CANDY successfully learned the preserved dynamics enabling high neural-to- behavior forecasting in the monkey center-out reaching dataset. A. Visualization of the dynamics flow field the latent subspace, projected onto first two PCs of both the latent trajectories (21.T) and the learned flow field. Eight target directions of the task are color-coded, with the trial start (red dot) and the trial end (gray dot) marked. The original latent dimension is two. B. Solid line: neural-to-bel to-bavvve forecasting using session-agnostic decoder (Appendix C.2.1) across all sessions (solid line). Dash line: coural-oo-bchavio forecasting using session- cession-specific decoders (Appendix C.2.2). Performance of CANDY (blue) is compared with DFINE--upervised (orange). Line: average over 40 sessions. Shaded area: s.e.m over 40 sessions.
A
CANDY
CEBRA (muli--eessoo)
CANDY
Trme bins (all socss.)
Eudkdian dieteraaa mama
Behaviors (sorted)
Eucleean destance
Aligmment score
0.4
(embedding)
Time bins (eil sess.)
CEBRA
CEBRA (multi-session)
(muli
0.0
Time bins (al scess.)
bins soss.)
Eudidiar disarmaa
-02
16
embedding dimension
PC2
Time bins (sil
PC1
Figure S7: CANDY aligns latent embeddings across sessions in the monkey center-out reaching
task. A. Top: Scatter plot of the first two principal components (PCI vs. PC2) of the latent embeddings across 40 sessions, with points colored by the target direction. Bottom: Same PC1-PC2 projection the top row but colored by session IDs. Columns: (i) CANDY, (ii) multi-session CEBRA. B. Pairwise heatmaps over time bins sorted by behavioral angle, comparing behavioral differences (top left) and latent embedding distances of the two methods in A. The latent embedding distance is defined as the Euclidean distance between the latent embeddings with rows and columns ordered identically the left panel. C. The alignment (purple). Solid lines: the average alignment score over 5 random seeds. Error bar: s.e.m. over random seeds.
for CANDY (blue) and multi-session CEBRA
24

---

Decode from the neural latent embeddingg
Decode from the neural latent dynamics
Average test R2 score
Average test R2 score
0.
0.4
Fixed pre-trsined LDS
Fixed pre-trained LDS
Train from scratch
Train from scratch
10%
20%
50%
80%
100%
10%
20%
50%
80%
100%
Percentage of training data
Percentage of training data
Figure S8: Generalization of CANDY to held-out sessions in the monkey center-out reaching task. A. Decoding performance latent embeddings ú across sessions function of the fraction of held-out session data used. The blue curve shows the results with the fixed pretrained LDS and behavior decoder from CANDY; the brown curve shows training from scratch on the held-out data. B. Same comparison for decoding from the inferred latent dynamics. Solid line: average performance over held-out sessions, and each session's performance averaged over 5 random seeds. Error bar: s.e.m. over 2 held-out sessions.
A Neural observations
Limit cycle ground truth dynamics
C Inferred menifold trajectory (fitering)
Anmal
Animal 2
Animal
Naaron uounoo
Animll
Animal
Neuran
Figure $9: Recovering shared latent factors trajectory from nonlinear dynamics. A. Simulated neural observations of three animals, each generated by applying distinct nonlinear mapping from the same underlying low-dimensional trajectory (B). B. Ground-truth limit cycle dynamics trajectory. C. Latent trajectory inferred by CANDY for each animal with latent dimension set to four.
A
dynamics
Identical dynamics
B
Dx10
Bahavior decoding
performance
Session-specfic behavior docader
Session-agnostic behavior decoder
0.2
Diffrent
Itttiil
Bynamica
Synamice
Dim
L Dim
Trial start
Trial and
Figure S10: Detecting divergent latent dynamics across sessions. A. Latent factor trajectories inferred by CANDY in a synthetic dataset with three sessions. Left: two sessions were generated from the identical spiral dynamics, while the third followed limit-cycle dynamics (Appendix A.3). Right: all three sessions were generated from identical spiral dynamics (Appendix A. B. Behavior decoding accuracy of session-are aggnsti (Appendix C.2.1) and session-specifi decoders (Appendix C.2.2). All tests are two-sided Wilcoxon signed-rank tests. Error bar: s.c.m over 20 seeds.
25

---

Figure S11: Shuffled neural-behavioral control confirms that CANDY relies on genuine neural-behavioral relationships. A. Same as Fig.3B but using the shuffled neural berralb data. Top: colored by in- stantaneous wheel velocity. Bottom: col- ored by session IDs. B. Latent embedding distance of the shuffled data. The latent embedding distance defined the same as Fig.3C. C-D. Multi-session behavior de- coding performance (R²) evaluated with a session-agnostie decoder (Appendix C.2.1) on (C) neural latent embeddings and (D) latent dynamics. Solid line: average over sessions; for each session, shuffling perfor- mance is averaged over 20 random shuffles.
Aprea vabochty (cmmss)
Multi-sessian behaviar
decading performance
CANDY jembodiingl
0.4
Shuffed jenttedling)
PC2
L
0.0
32
Latent embedding dimension
Triel atort Trial end
decoding perforannc (F²)
Multi session behavior
0.8
B
CANDY
(shutfled)
0.6
CANDY (dynniccs)
Time bins (all sess.)
Eudldeer dietence
0.4
0.2
(dymamcca)
Shuffed
0.0
Time bins 6ess.)
Latent dynamics dimension
Shaded area: S.e.m over 8 sessions.
Eudiddean distance
Awerage Eudideen
distannce bistanosbttween
latnel ombeddings
Time bins (haa)
Ime bins (tai))
Time bins (tral]
(tai))
Time bins (tria)
Time tine (viel)
Time bina (tria))
Time bins (trial)
Temporal distance (bins)
Figure S12: Latent embeddings reflect temporal proximity within trials. A. Example temporal similarity maps from four representative trials. Each heatmap the pairwise Euclidean distance between embeddings at all time bins within the trial; embeddings from temporally adjacent bins remain close in latent space. B. Average Euclidean distance between embeddings as a function of temporal distance. Solid line: average over all trials and sessions. Shaded area: s.e.m. across all trials and sessions.
26

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We validated that the extracted neural latent embeddings are aligned and the preserved dynamics is generalizable by doing extensive experiments, including synthetic dataset, mouse dataset, and publicly available monkey dataset to support our claim. We run our models for each variant using five different model seeds ensure that the results are not random. We also did in-depth analysis and compare with multiple SOTA baselines to justify our claim the model.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We discuss limitations in our "Discussion" section, acknowledging our linear dynamical systems could be enhanced with nonlinear formulations to capture richer temporal patterns, and noting potential for future integration with external covariates and brain- machine interfaces.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best
27

---

judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: The paper does not present theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results,
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We illustrated the model architecture, disclosed the model parameters, and also add details on the training in the appendix for the reader to reproduce the main experimental results. We clearly address each piece of our models in the Method section. For each figure, we clearly say the meaning and calculation for the x-axis and y-axis, either in the main text in the appendix. We provide explanation for all evaluation methods and metrics, either in text or in equations.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
28

---

(c) If the contribution is new model (e.g.. a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We disclosed the synthetic dataset, all the code, and the public dataset is available online.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cm cc/ http://deeuumssiionnneeeeeeeeeee
public/gue Ceallic/giies/Coooe
for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: / //nips.com httppsscccppbbcccccccccccccccl hww//uideeee des//dee//e /CodeSubbiiiiccc wswonnolliiieeeeeeeee oniolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification: All details of the hyperparameters for all models (our model and other bench- marking models) are provided in the appendix. We also emphasize that hyperparameters were chosen via grid search. The data splits are also disclosed in the main paper for the mouse dataset and appendix for the monkey dataset.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, appendix, or as supplemental material.
7. Experiment statistical significance
29

---

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: For all line charts, added error bars shaded areas to represent the standard error of mean. In the caption, we clearly address the solid line and error bars. As we don't t have any bar plots, we did report any statistical significance.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2- 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative eтror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We described our computing resources in section E "Resources for repro- ducibility".
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https:
Answer: [Yes]
Justification: We confirm that there are no harms in our code and our code follows the NeurlPS Code of Ethics. We confirm that the paper and code preserve anonymity.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
30

---

If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: Our work fundamental research focused on understanding preserved neural dynamics across sessions and subjects in system neuroscience, without direct societal impacts beyond advancing basic neuroscience knowledge.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g.. disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our work focuses on fundamental neuroscience research methods for analyzing neural recordings from mice and monkeys. There are no foreseeable dual-use applications or potential for misuse that would require specific safeguards.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
31

---

12. Licenses for existing assets
Question:
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We uesd publicly available code for baseline comparison from appropriate repositories with proper citations. Publicly available monkey dataset are appropriately specified in the paper and referenced.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g.. website). the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We made our code and synthetic datasets publicly available. The appendix provides detailed model hyperparameters, preprocessing procedures, and implementation details sufficient for reproducing our results.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots. if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: No crowdsourcing or human subjects were involved in this work. Our study uses only synthetic dataset, public monkey dataset, and laboratory mice experiments.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
32

---

Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: No crowdsourcing or human subjects were involved this work. Our study uses only public monkey dataset and laboratory mice experiments.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality the research, declaration not required.
Answer: [NA]
Justification: We did not use LLMs part of the methodology experimental approach. Our work relies entirely on traditional methods without any LLM components influencing the research design, implementation, outcomes.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our L.LM policy (https:/ http://eeuuppppeeee for what should or should not be described.
33


Reviews:
[{"review_id": "ktdj0eDRck", "reviewer": "Reviewer_6wRv", "summary": "The authors propose CANDY, a contrastive learning approach to align latent encodings of neural activity across multiple sessions and subjects by using continuous behavioral variables as a shared anchor and per-session neural encoders. In addition to the contrastive objective, there is an optional behavior prediction objective and a k-step ahead neural prediction loss obtained by additionally fitting a linear dynamical system to the latent space and a nonlinear latent-to-neural decoder. The method is validated using a mouse wheel-turning task with calcium imaging and a monkey center-out reaching task using electrophysiological data. CANDY produces latent spaces that successfully predict behavior and generalize across recording sessions and individuals. Additionally, the latent distance matrices (across timebins) better reflect behavioral distances than other methods (CEBRA and DFINE). Lastly, in ablation experiments, the authors show the contrastive loss is necessary for the good performance they observe.", "questions": "Do you have evidence that the LDS improves the model? (See last paragraph of \"Strengths and Weaknesses\")\n\nWill the wheel turning data be made public upon publication? It seems like it would be a useful resource.\n\nMinor Comments\n* Figure 4a and 4b should plot the contrastive-but-not-behavioral embeddings and matrices from Figure 3 for easy comparison.", "limitations": "Training a new encoder for each session could be cumbersome and may not work as well for sessions with neural but not behavioral data.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Contrastive learning for neural/behavioral modeling is a promising research direction with several encouraging recent results (CEBRA, MARBLE, etc.). \n\nThe various components of the method are well-justified: the session-specific encoders, the  behaviorally-anchored neural embeddings, the neural decoders, and the linear dynamical system. However, these components are all fairly standard, so I consider CANDY to have modest novelty.\n\nThe behavioral prediction results (Figures 3d-e, S3b-c) seem to be very good compared to baseline models. Additionally, the neural/behavioral alignment as visualized in Figures 3c and S4b is very impressive. Generally, the experiments seem to be high quality.\n\nMARBLE (Gosztolai et al. 2025, Nature Methods), as a contrastive learning method for neuroscience applications, is very relevant and should be cited. BLEND (Guo et al. 2025, ICLR) is also shares similar aims to CANDY. Additionally, there should be a discussion of the specific ways in which MARBLE and CEBRA differ from CANDY. Contrastive learning in neuroscience could easily fill a whole paragraph in the related works section.\n\nIs the LDS useful at all? It does not seem to help predictive performance to use the filtered latents (e.g. Figure 3 d vs. e). But it may still help the model learn a good latent space during training. I view the lack of evidence for the usefulness of the LDS as a major weakness of the paper, and would consider raising my score if can be addressed.", "was_revised": true, "final_justification": "* The paper explores behavioral-aligned contrastive learning for neural data with latent dynamics, a very promising and interesting research direction.\n* The model performs well in terms of behavioral predictions, and is able to align data from multiple individuals and recording sessions on two real-world datasets. \n* Resolved in discussion period: the learned dynamics are informative\n* Could be strengthened: evidence of behaviorally relevant shared neural dynamics across subjects. The evidence presented in Figure 5 is, in my view, fairly weak.", "rebuttal": "Thank you for the summary and your constructive feedback. Please find our point-by-point responses to your comments and questions below. In all cases, we either performed new experiments or added clarification to the text to address your concerns. We look forward to hearing your assessment.\n\n> Weakness 1: The various components of the method are well-justified: … so I consider CANDY to have modest novelty.\n\nWhile these are indeed important building blocks of our work, we respectfully note that CANDY’s contributions go well beyond simply combining standard modules:\n\nFirst, our **methodological** advance lies in *developing a new framework to extract preserved dynamics underlying continuous behavior across sessions and animals using a rank-based contrastive objective and a linear dynamical system*. This shows two specific novelties compared to previous methods: (1) this goes beyond the limitation of other behaviorally-anchored neural embedding methods, such as CEBRA, which have to discretize behaviors before applying the contrastive objective; (2) the behaviorally-anchored neural embeddings are also constrained by a linear dynamical system, which assists the learning of the preserved dynamics by maintaining the temporal structure of the embeddings within a trial. \n\nSecond, as a **conceptual** novelty, we show that *contrastive alignment is essential for uncovering preserved dynamical systems*, not merely a decoding aid. This is especially important for neuroscience as neural recordings show different degrees of heterogeneity, including number of neurons and the exact location of the recording. In Figures 2 and 4, removing the contrastive term causes the latent trajectories and the LDS to collapse. This highlights a new role of contrastive learning: as an alignment tool for uncovering preserved dynamical-structure across sessions, rather than just representation learning as in CEBRA. \n\nThird, our work addresses an important **biological** question: while earlier work has shown that neural activity motifs are preserved across animals performing the same tasks [1], no work to date has investigated whether these motifs can be supported by the same dynamical system equations. Here, we show that an LDS trained on one set of animals transfers directly to held-out individuals, *indicating the existence of preserved dynamical structures in a specific brain region during a specific task*. In Figures 5 and S5, we froze both the learned LDS and the behavior decoder and still achieved high decoding performance on unseen animals/sessions. This demonstrates that the latent dynamics inferred by CANDY are not only interpretable, but also consistently preserved across subjects in the mouse striatum during the wheel-turning task and in the monkey motor cortex during the reaching-out task. This is important because this shows a unique cross-subject generalization of dynamical models of the animal brains.\n\nFinally, on the **experimental** side, we ***designed and collected a new 2p calcium cross-animals dataset ourselves***, which includes 3 animals with 5 sessions each (15 sessions total), and each session contains ~200 trials with recordings from ~200 neurons in the striatum. We will make the dataset available to the community if accepted. \n\nWe hope these clarifications would address your concerns of the originality of this work.\n\n> Weakness 2: MARBLE … should be cited. … discussion of the specific ways in which MARBLE and CEBRA differ from CANDY… the related works section.\n\nThank you for referencing these papers. We agree that contrastive learning is a rapidly evolving area in neuroscience and appreciate the opportunity to better position our method in this context. We will add the following paragraph to the Related Work section:\n\n“**Contrastive learning in neuroscience**. Recent studies have explored contrastive learning for extracting meaningful neural latent representations. CEBRA [2] embeds neural activity with noncausal convolutional neural network by using contrastive objective on temporal similarity and discretized behavioral similarity.  MARBLE [3] introduces a geometric deep learning framework that embeds local flow fields computed from neural activities into a latent space using contrastive objectives. However, it requires *post hoc* alignment for downstream behavior decoding across sessions. While both methods produce useful low-dimensional representations, they do not explicitly model shared latent dynamics across sessions or individuals.”\n\n> Weakness 2: BLEND also shares similar aims to CANDY.\n\nBLEND develops a model-agnostic paradigm to distill behaviorally relevant neural latent embeddings from an existing model for use on new datasets lacking behavior signals. In contrast, CANDY develops a framework to extract preserved dynamics across animals by using behavior as an anchor to align heterogeneous neural populations (e.g., number of neurons and location of the recordings). We will add the following sentences to the Related work section:\n“The BLEND framework [4] uses a teacher-student setup to distill behaviorally relevant information to student networks when there is a lack of behavior data in a new dataset, but does not focus on extracting shared task-relevant dynamics across animals.”\n\n> Weakness 3 & Q1: Is the LDS useful at all? …\n\nThis is a critical aspect of our work and you are correct that the LDS is not designed to improve behavior decoding performance. Below, we highlight two distinct reasons to address why LDS is useful:\n\nFirst, the LDS module is essential for addressing an important scientific question: **Can we extract a preserved and latent dynamical system (not just representations!) underlying neural activity across animals performing similar tasks?** While contrastive learning methods like CEBRA can produce behaviorally meaningful embeddings, they do not model the underlying dynamics and thus cannot determine whether shared dynamical motifs exist across subjects. This is precisely what CANDY was designed to address. More importantly, in Section 4.2.4, Figures 5 and S5, we demonstrate that a pretrained LDS learned from several animals generalizes to new animals with high decoding performance, without retraining the LDS. This provides **biological evidence** that the **neural activities in the same brain region can exhibit shared dynamics across individuals in similar well-trained behaviors**.  \n\nSecond, LDS is a key component because our methodological novelty lies in **extracting a preserved, interpretable dynamical system via behaviorally aligned neural embeddings, not in boosting behavior decoding performance using LDS**. As shown Figures 2 and 4, omitting the alignment step prevents recovery of a coherent LDS. Fig. 4 further illustrates that **contrastive objective is critical for uncovering preserved dynamics**: replacing contrastive loss with behavior supervision achieves similar decoding from embeddings but *fails* to recover meaningful dynamics, as seen in much worse decoding from the latent dynamical factors. To our knowledge, this is the first demonstration of behavior guided alignment is essential for isolating latent dynamical systems that generalizes across both sessions and animals. \n\nIn summary, the LDS module is essential for our scientific goal of testing shared dynamical motifs across individuals. In the revised manuscript, we will add these clarifications to directly address this concern. We welcome further questions and look forward to your reassessment.\n\n> Q2: Will the wheel turning data be made public upon publication?\n\nYes! We are committed to open science and will release the striatal 2p calcium imaging data upon acceptance. We hope you will consider this as an additional crucial contribution of the paper to the community.\n\n> Q3: Fig. 4a and 4b … for easy comparison.\n\nGreat suggestions! For easy comparison, we will put the first column of Fig. 3B to Fig. 4A and the top right similarity matrix in Fig. 3C to Fig. 4B. Then we will move the C-E to the third row in Fig. 4. \n\n> Limitations\n\nTraining a session-specific encoder is not a limitation but a necessary design choice due to the substantial heterogeneity in neural recordings across sessions. Behavior signals serve as a key for aligning latent neural dynamics across sessions. Without them, embeddings drift arbitrarily in latent space. As shown in Fig. 4, when both contrastive and behavioral losses are removed, thus removing behavioral supervision, the embeddings fail to align, resulting in failure to recover consistent underlying dynamics. Existing *post hoc* alignment methods similarly depend on behavioral information to align latent embeddings across sessions [1,3], further underscoring the necessity of behavioral signals. Moreover, adapting to new sessions with a new encoder, as demonstrated in Figures 5 and S5, only requires a few trials of data.\n\n\nWe hope our clarifications on the methodological, conceptual, biological, and experimental novelties, along with the added content in the Related work and Discussion sections, and the explanation of the usefulness of LDS address your concerns. If there are any other points we can clarify or further experiments we can conduct to strengthen your assessment, please let us know.\n\n[1] M. Safaie, et al. \"Preserved neural dynamics across animals performing similar behaviour.\" Nature 623.7988 (2023): 765-771.\n[2] S. Schneider, Steffen, J. H. Lee, and M. W. Mackenzie. \"Learnable latent embeddings for joint behavioural and neural analysis.\" Nature 617.7960 (2023): 360-368.\n[3] A. Gosztolai, et al. \"MARBLE: interpretable representations of neural population dynamics using geometric deep learning.\" Nature Methods (2025): 1-9.\n[4] Z. Guo, et al. \"BLEND: Behavior-guided Neural Population Dynamics Modeling via Privileged Knowledge Distillation.\" The International Conference on Learning Representations (2025)."}, {"review_id": "3Moap78qGa", "reviewer": "Reviewer_oySm", "summary": "The authors propose CANDY, a dimensionality reduction method for neural data that fits a shared LDS over multiple sessions with aligned embeddings. A key step is using a contrastive loss to align embeddings  based on the continuous behavioural label associated with each time point. This encourages the learned embeddings to group together neural data points with similar behaviours, even across sessions. They apply the method to one ground truth simulation, and a couple datasets. One interesting finding is that the contrastive loss is necessary to find behaviourally relevant embeddings, and simply using a supervised behavioural loss is not sufficient.", "questions": "- Line 112, definition of S_h,j, I think that should be \\leq, not \\geq. Is that a typo?\n- The ground-truth analyses (section 4.1) should include examples of where the model breaks down. For example, what happens if the behavioural readout is highly nonlinear, or worse yet if the mapping varies from session to session?\n- Lines 260-266: Can the authors give more details here? Is the model trained from scratch only trained on a single session? \n- In neuroscience experiments, it is sometimes unclear if different animals are using the same dynamical strategies to perform a task. Is the model robust to different latent dynamics across sessions? Can it be used to discard sessions that are deemed too dynamically different from the other sessions?", "limitations": "Somewhat. The authors could be clearer about the limitations of their model and when they expect it to fail, for example in out-of-sample data where there may be session-specific nonlinearities.", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\nThis is a really nice paper with a cool idea to align latents using their behavioral labels by leveraging contrastive learning.The claims are generally well supported, and the analyses of the contribution of the contrastive loss are interesting.  \n\nAs far as I know the model is a novel application of an existing rank-based contrastive loss to a new problem regarding alignment of neural dynamics (although I am not an expert on contrastive learning so I am not 100% sure on the novelty). It is certainly a step forward compared to CEBRA or DFINE. I believe the model will be broadly interesting to NeurIPS and the analyses of the contrastive loss offers some insights regarding the important considerations for aligning latents. I can certainly see future researchers building upon this idea for new alignment algorithms. \n\nIt's well written and clear for the most part, which I appreciated as a reviewer.\n\nWeaknesses:\n\nAlthough I really like this submission and find it to be high quality overall, there are two fundamental points that really should be improved.\n- The method is demonstrated in two neuroscience data sets, although one dataset receives much more attention in the main text. But the paper is missing ground truth experiments on out-of-sample data.\n- The discussion is very short and should be expanded to include more limitations and relationship to other work on latent dynamics inference and contrastive learning in neuroscience.", "was_revised": true, "final_justification": "The authors have made a strong case for CANDY, including promising results in comparison to stitch-LFADS, and helpful clarifications about the difference between their model and competitors. I think the model hits the perfect balance between simplicity and expressivity to make an impact in neuroscience. Inferring latent dynamics across sessions is a very important question in neuroscience, and this is one of the few methods I've seen that tries to do this while maintaining an interpretable architecture. The authors therefore have my full support for this paper.", "rebuttal": "Thank you for this summary, which hits the mark perfectly. We appreciate your support of our manuscript and the constructive feedback throughout the review. Please find below our point-by-point responses to your questions and comments.\n\n> Weakness 1: … one dataset receives much more attention in the main text. \n\nAn important reason for this is because we collected this dataset ourselves specifically to evaluate generalization in a setting where we had full control over preprocessing and task design. Nonetheless, we show applications to the public monkey dataset, as is the convention in the field currently. We believe that the use and introduction of **a new, cross-subject calcium imaging dataset**, which will be made publicly available upon acceptance, is a strength of our study and part of the novelty. We hope it provides value to the community by expanding the types of datasets used to evaluate neural dynamics models. \n\n> … missing ground truth experiments on out-of-sample data.\n\nWe acknowledge this limitation, but emphasize that we showed in Figures 5 and S5 that the LDS and the decoder can be generalized to a held-out animal. To prove this on a dataset with ground truth, we added an additional experiment on the synthetic dataset by helding-out one session that shared the same spiral latent dynamics but different neural observations readout and noise perturbation. By fixing the LDS and tuning the encoder parameters only for the held-out session, we observe that the model can correctly infer the spiral latent trajectory on the held-out dataset, following the pretrained LDS flow field. We will add this result to the revised manuscript. \n\n> Weakness 2: The discussion … expanded to include more limitations and relationship to other work on latent dynamics inference and contrastive learning ...\nLimitations: … clearer about the limitations of their model and when they expect it to fail, … session-specific nonlinearities.\n\nWe agree with this assessment. In the initial submission, due to page limit, we had to shorten our discussion section. In the revised manuscript, we will include more work from the broader neuroscience literature that employs contrastive learning:\n\n“**Contrastive learning in neuroscience.** Recent studies have explored contrastive learning for extracting meaningful neural latent representations. CEBRA [1] embeds neural activity with noncausal convolutional neural network by using contrastive objective on temporal similarity and discretized behavioral similarity.  MARBLE [2] introduces a geometric deep learning framework that embeds local flow fields computed from neural activities into a latent space using contrastive objectives. However, it requires *post hoc* alignment for downstream behavior decoding across sessions. While both methods produce useful low-dimensional representations, they do not explicitly model shared latent dynamics across sessions or individuals.”\n\nand add the following to an explicit **Limitations and outlook** section which will be included in the manuscript:\n\n“CANDY offers a principled approach for aligning population activity and extracting shared latent dynamics across sessions and subjects, but it has limitations. First, it relies on a linear dynamical system (LDS), which may be too restrictive for highly nonlinear neural dynamics. While Koopman theory suggests that higher-dimensional LDSs can approximate complex systems, we lost the ability to truly reveal the dynamical structure and the latent dimension. Second, CANDY assumes a shared linear behavioral readout across subjects, which may overlook preserved dynamics with nonlinear behavioral readout or subject-specific readouts. Although we found this assumption sufficient in our experiments, incorporating distinct behavioral readouts is a promising direction for future work.”\n\n> Q1: Line 112, definition of S_h,j, I think that should be \\leq, not \\geq. Is that a typo?\n\nNo, it should be \\geq. Please allow us to elaborate: $S\\_{h,j}$ is defined as a set of variables whose distance to anchor $\\hat{u}_h$ is larger than the distance between anchor $\\hat{u}_h$ and $\\hat{u}_j$, and hence are less similar to $\\hat{u}_h$ than $\\hat{u}_j$. Therefore in the contrastive loss, we pull $\\hat{u}_j$ closer to $\\hat{u}_h$ compared to other variables in $S\\_{h,j}$.\n\n> Q2: The ground-truth analyses … where the model breaks down. … if the behavioral readout is highly nonlinear, or … the mapping varies from session to session?\n\nGreat question! In the model we presented, we always set the behavior decoder to be linear. To address your question, we added a few more synthetic experiments:\n- Same spiral dynamics and same nonlinear behavior readout \n- Same spiral dynamics but different linear readouts for each session\n\nWe observe that CANDY breaks in both situations. We believe this is an interesting future direction. One potential approach is that contrastive learning cannot just take in the behavior label naively and a behavior encoder might be needed for such a situation for recovering the preserved dynamics. We will add this to the Limitations and outlook section as written in the above response. \n\n> Q3: Lines 260-266: ... Is the model trained from scratch only trained on a single session?\n\nWe apologize for the confusion. No, the model trained from scratch was trained on all held-out sessions. We will change L263 from “model trained from scratch on the same sessions” to “model trained from scratch on all held-out sessions”.\n\n> Q4: ... Is the model robust to different latent dynamics across sessions? Can it be used to discard sessions that are deemed too dynamically different from the other sessions?\n\nGreat question! To explore if the model can distinguish sessions with different dynamics, we added a synthetic experiment where two sessions were generated from the same spiral dynamics, while the third used limit-cycle dynamics. All three shared the same linear behavioral readout. \n\nWhen trained on all three sessions, CANDY successfully aligned the two spiral sessions, while the third remained misaligned in the latent space. The decoding $R^2$ value for each dataset is only $\\approx 0.9, 0.6, 0.1$ respectively, revealing a mismatch in the underlying dynamics. Based on the misalignment observed in the latent space, we identified the divergent session and excluded it. Retraining on the two aligned sessions improved decoding back to $R^2\\approx 0.99$. \n\nThis result suggests that CANDY can not only recover shared latent dynamics but potentially also serve as a diagnostic tool to flag sessions or subjects that deviate in their dynamical strategies. We see this as an exciting direction for extending this work. We will include the new synthetic experiment in the Appendix and add the following to the above Limitation and outlook section: “Third, while CANDY could facilitate the identification of distinct underlying dynamics as shown in the synthetic experiment, distinguish different neural processes that yield similar behaviors is an interesting extension to this work.” \n\n\nWe appreciate your thoughtful and constructive questions, which have helped us strengthen both the conceptual and empirical aspects of our work. We hope that our explanation and the newly conducted experiments would address your concerns. We look forward to your reassessment!\n\n[1] Schneider, Steffen, Jin Hwa Lee, and Mackenzie Weygandt Mathis. \"Learnable latent embeddings for joint behavioural and neural analysis.\" Nature 617.7960 (2023): 360-368.\n[2] Gosztolai, Adam, et al. \"MARBLE: interpretable representations of neural population dynamics using geometric deep learning.\" Nature Methods (2025): 1-9."}, {"review_id": "i7xlfSec8P", "reviewer": "Reviewer_P5Sz", "summary": "This work proposed CANDY, and end-to-end framework that aligns neural and behavior data using rank-based contrastive learning, and it also integrate with a linear dynamical system to capture the temporal associations. The optimization include a contrastive loss to align the neural latent embeddings with the continuous behavior samples, a behavior decoding loss, as well a linear dynamical system with parameters to be jointly optimized. It contributes to increase the generalizations across data, which allows learning the embeddings across sessions and subjects (also include unseen sessions), and demonstrates superior performance compared to existing baselines.", "questions": "1. In Figure 3b, it is unclear how CANDY is superior compared to CEBRA?\n2. What is the computational cost and latency of the proposed method compared to existing approaches?\n3. Are the parameters estimated in the linear dynamical system eqn (4) and eqn (5) identifiable and interpretable?", "limitations": "1. Lack of evaluations and comparisons with other SOTA baselines discussed in the related works [34-39]. \n2. The assumption of linear dynamical system (LDS) as a prior in the model might not be sufficient to capture the complexity of the neural dynamical systems,  and generalizable to more complex situations.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "**Strengths**\n1. This method contribute to an important question on representation learning from neural activities for behavior decoding, and aim to improve the generalization of the method across subjects and sessions. \n2. Compared to existing approaches, this model focused on using contrastive loss aligning with continuous behaviors, and incorporated with a dynamical constraint to enable temporal consistency, which is superior than existing baselines such as CEBRA.\n3. This methods are evaluated on multiple datasets from mouse to macaque, demonstrate its generalizations across animal species. \n4. It demonstrates superior performance on decoding performance and its performance are less sensitive to choice of hyperparameters such as latent dimension.\n\n**Weakness**\n1. Explain the choice of linear dynamical system, discuss whether it would be sufficient to capture the nonlinearity and nonstationarity of the dynamical system. It would be valuable to introduce ablation study on the complexity of dynamical system.\n2.  Lack of comparisons with other baseline for behavior decoding such as stich-LFADS, POYOs, NDT-2, etc. [34-39] as discussed in the related works.", "was_revised": true, "final_justification": "Thanks for the detailed responses from the authors, especially on clarification on using a LDS model, while still could introduce nonlinearity during the modeling. And adding comparisons with existing baseline Stitch-LFADS LFADS, and demonstrates superior performance. Those efforts have addressed most of my concerns, I would like to improve my score to borderline accept as positive score. Thanks for your efforts!", "rebuttal": "We thank the reviewer for their constructive feedback throughout their review. Below, we provide point-by-point responses to your questions and comments. In all cases, we either added a new experiment or clarification to the manuscript.\n\nTo begin with, we’d like to address a few misunderstood contributions in the summary, which we believe is the main reason behind confusions later in the review.\n\n*increase the generalizations across data*: Our goal is not to improve behavior decoding generalization across datasets. Instead, we started with a biological question: **how to extract the task-relevant preserved latent dynamics**, motivated by evidence of shared strategy in certain brain regions shared across sessions while doing similar tasks [1]. We used CANDY as a framework for investigation and demonstrated its effectiveness on two datasets: a new 2p calcium imaging dataset on mouse wheel-turning task collected by ourselves (will be released upon acceptance) and a public monkey electrophysiology dataset.\n\n*demonstrates superior performance compared to existing baselines*: We are not aiming to improve cross-session behavior decoding performance. Instead, we treat behavior decoding from neural embeddings and dynamic factors as a **diagnostic** way to **evaluate alignment quality and whether a preserved dynamics is correctly captured**. If our model matches or exceeds benchmark performance despite biologically motivated constraints, it suggests the preserved dynamics are well learned. Surprisingly, our model shows superior performance, further supporting both the existence of preserved dynamics and our model’s effectiveness.\n\nTo emphasize these distinctions, we will revise the contributions at the end of the introduction as:\n\n“Our key contributions include:\n1) Methodologically, we develop a new framework to extract preserved dynamics underlying continuous behavior across sessions and animals using a rank-based contrastive objective and a linear dynamical system. \n2) Conceptually, we show that contrastive alignment is essential for uncovering preserved dynamical systems, not merely a decoding aid.\n3) Biologically, while prior work shows preserved neural motifs across animals [1], none have tested if these can arise from the same dynamical system. We show that a dynamical model trained on one set of animals transfers to held-out individuals, indicating the existence of preserved dynamical structures in a specific brain region under a given task.\n4) Experimentally, we collected a new cross-day multi-animal 2p calcium dataset and will release it upon acceptance.”\n\n> Weakness1 & Limitation2\n\nOur primary goal is to build a model that **extracts preserved, interpretable latent dynamics shared across sessions**. Beyond its clear interpretability, CANDY leverages learnable nonlinear encoders to grant the latent space far greater flexibility than a standard LDS would allow [2]. Intuitively, if we map neural activities to latent variables via $z = f(x)$ and impose an LDS prior $\\dot z = A z$, the induced dynamics on the observations become $\\dot x = (\\nabla f(x))^{-1} A\\,f(x)$, where $\\nabla f(x)$ is the Jacobian of $f$. Since $f(x)$ is learnable and nonlinear, the resulting latent variables induce nonlinear dynamics on the observed neural activity.\n\n“While we focus here with an LDS prior to test our core hypothesis of the existence of preserved dynamics, whether replacing the LDS with different types of nonlinear dynamics in the CANDY framework can be beneficial is an interesting future direction.” We will add this to the Discussion section.\n\nTo address this concern empirically, we performed a new synthetic limit cycle experiment, involving *nonlinear*, *nonstationary* dynamics. Despite this complexity, CANDY successfully aligns the sessions and recovers the shared 2D limit cycle in the latent space when emphasizing on the contrastive loss (similar as Fig. 2C). This shows that the combination of a flexible nonlinear encoder with a linear dynamics prior is sufficient to recover complex underlying dynamics. We will add this to Fig. 2.\n\n> Weakness2 & Limitation1\n\nWe believe there is a misunderstanding about the primary motivation of our work and we apologize if the information in the Related work is misleading.\n\nWhile benchmarking large-scale models on behavior decoding is important, CANDY is not designed to focus on maximizing behavior decoding performance, as we said above. We will add “These transformer-based architectures focus primarily on behavior decoding, whereas CANDY focuses on extracting interpretable preserved dynamics across sessions.” to the Related work.\n\nIt is a great suggestion to compare with Stitch-LFADS since it also models a shared dynamical structure. We grid-searched 800 hyperparameters for both datasets independently. The mean±s.e.m of testing $R^2$ are (table entry: using a universal behavior decoder \\ session-specific behavior decoder):\n\nMouse dataset\n|Latent dim|Stitch-LFADS|CANDY|\n|-|-|-|\n|2|-0.0093±0.0051 \\ 0.5380±0.0340|0.6787±0.0130 \\ 0.6798±0.0133|\n|4|0.0260±0.0839 \\ 0.6267±0.0195|0.7727±0.0157 \\ 0.7859±0.0146|\n|8|0.0315± 0.0291 \\ 0.6910±0.0151|0.7976±0.0111 \\ 0.8062±0.0136|\n|16|0.0734±0.0556 \\ 0.7223±0.0131|0.8046±0.0119 \\ 0.8108±0.0125|\n|32|0.1504±0.0836 \\ 0.7548±0.0116|0.7917±0.0159 \\ 0.8005±0.0135|\n|64|0.2704±0.0758 \\ 0.7743±0.0136|0.7937±0.0134 \\ 0.8023±0.0124|\n\nMonkey dataset\n|Latent dim|Stitch-LFADS|CANDY|\n|-|-|-|\n|2|-0.0136±0.0044 \\ 0.0920±0.0075|0.6453±0.0107 \\ 0.6178±0.0118|\n|4|-0.0078±0.0056 \\ 0.2185±0.0106|0.6089±0.0130 \\ 0.6158±0.0119|\n|8|-0.0001±0.0080 \\ 0.3822±0.0108|0.6015±0.0126 \\ 0.6106±0.0122|\n|16|-0.0020±0.0089 \\ 0.5041±0.0096|0.5714±0.0134 \\ 0.6095±0.0122|\n\nThese results indicate that stitch-LFADS failed to align the dynamics across sessions.\n\n> Q1\n\nThis figure is not meant to show CANDY’s superiority over CEBRA, but to highlight the advantage of those trained with contrastive learning over those without. Specifically, Fig. 3B shows that contrastive methods (CANDY and CEBRA) can align neural latent embeddings across sessions using behavior as an anchor, a property not observed in DFINE, which serves as a baseline without contrastive alignment.\n\nDirect comparisons to CEBRA are shown in Figures 3C-D, S2, S3B and S4. CANDY produces latent embeddings with significantly stronger correlations to ground-truth behavior and higher behavior decoding performance, indicating **more interpretable** and **behaviorally meaningful** representations.\n\n> Q2\n\nDue to limited access to high-performance GPUs, all experiments were run on CPU. On an AMD Ryzen 9 9950X, training CANDY on 8 sessions (~250 neurons, ~30 time steps each, batch size 2048) takes about 10 hours. For comparison, Stitch-LFADS takes ~8 hours and CEBRA ~6 hours under similar settings. With a high-VRAM GPU, we expect CANDY’s training could be accelerated by an order of magnitude or more.\n\n> Q3\n\nTheoretically, both are important questions of current research. Empirically, the answer is yes.\n\nIdentifiability: Building on recent work on identifiable VAEs, CEBRA proves that latent variables learned with categorical contrastive loss are linearly identifiable. We apply a contrastive loss suited for regression, while preserving the same structure. We additionally fit a linear dynamical system to model these embeddings and LDS parameters are identifiable up to orthogonal transformations. Therefore, though there is still a gap in the broader literature regarding the identifiability in the case of joint fitting of contrastive objectives and their dynamics, our work builds on existing identifiability proofs in terms of the distinct parts of the model. We will include a discussion of this in the manuscript.\n\nInterpretability: While LDS identifiability means that the exact matrices (e.g., dynamics matrix A) are not uniquely determined, the topology and temporal evolution of the latent trajectories are still meaningful in a task-aligned context, which is precisely our contribution. We aligned the latent variables using the contrastive loss and continuous behavioral signals, and illustrate their interpretability with examples:\n\n- In Fig. 3B, neural latent embeddings colored by wheel velocity reveal a smooth 1D manifold aligned with task structure. To further highlight this, we will update the figure with an additional row colored by trial type (leftward vs. rightward cursor onset), where the two wings of the trajectory correspond cleanly to the two trial conditions, reinforcing the interpretability of the learned latent geometry.\n- In Fig. 3C, the distance matrix of CANDY’s latent embeddings (top right) shows a block-diagonal structure that matches the behavioral similarity matrix (top left), indicating that the latent space faithfully encodes behavioral structure.\n- The behavioral interpretability is also evident in the center-out reaching task (Fig. S4A), where the latent representations clearly organize the eight reach directions into distinct clusters.\n\nIn summary, while LDS and encoder parameters are expected to be linearly identifiable, empirically, the learned latent trajectories are behaviorally aligned, geometrically structured, and interpretable, supporting CANDY’s goal of uncovering shared task-relevant dynamics across sessions. In that regard, our work provides an important empirical contribution to the literature. We hope these clarify your question.\n\nWe hope our clarifications on the motivations and contributions of the paper and the choice of LDS address your concerns. If there are any other points we can clarify or further experiments we can conduct to strengthen your assessment, please let us know.\n\n[1] Safaie, Mostafa, et al. \"Preserved neural dynamics across animals performing similar behaviour.\" Nature 623.7988 (2023): 765-771.\n[2] Abbaspourazad, Hamidreza, et al. \"Dynamical flexible inference of nonlinear latent factors and structures in neural population activity.\" Nature Biomedical Engineering 8.1 (2024): 85-108."}, {"review_id": "sq7GqWy3Yr", "reviewer": "Reviewer_JGvB", "summary": "Authors develop a method to generate a shared embedding space of neural recordings from many sessions to capture presumably low-dimensional preserved dynamics. They also fit a linear dynamical system to model the temporal evolution in the latent space derived from the created embedding. Their method uses contrastive learning methods to enforce that neural states with similar behavioral observations are mapped close by and vice versa. They apply their method to a synthetic, a mouse, and a monkey dataset and show that the learned embedding maintains similarity observed in the behavioural observations across sessions and subjects and allows the decoding of the observed behaviours.", "questions": "(1) The method encourages behavioral similarity by design, but it would be valuable to understand whether other meaningful properties are preserved or not. For example, does similarity between neural recordings persist in the embedding space? Is proximity in time or trials reflected?\n\n(2) Regarding generalization, are you using the behavioral supervision loss here? If not, it’s unclear why the method would generalize. There doesn't seem to be anything explicitly enforcing that the embedding space is shared across sessions. Could you please clarify?\n\n(3) The approach assumes that the decoded behavioral observations are task-relevant and can be reliably read from the neural data. An important control would be to shuffle the neural recordings relative to the behavioral data and show that under these conditions, the method cannot find a “good” embedding. Including such a control would help confirm that the embedding genuinely reflects neural-behavioral relationships.\n\n(4) To better assess generalization, it would be helpful to test whether the model can map previously unseen neural states to the correct behavioral outputs. One suggestion would be to hold out specific trial types (e.g., certain movement angles or task periods) from training and include them only in the test set. Are these held-out examples mapped to the correct behavioral location in the embedding space? Can their associated behavior still be accurately decoded? This would help evaluate whether the embedding captures meaningful behavioral structure, even for novel data, or where its limitations are.", "limitations": "The approach is interesting and the results on preserving behavioral similarity are promising. However, it would be important to discuss whether preserving behavioral similarity is always desirable. In neuroscience, it's common for different cognitive processes to produce identical behavioral outputs - such as during evidence accumulation in integration tasks, where animals may show the same behavior while engaging in opposite internal processes. It would strengthen the paper to include a discussion of when and where this method is appropriate, and what the limitations might be in such cases. Alternatively (but probably outside the scope of this paper): How does the method perform on error trials? It would be interesting to see whether the model distinguishes between correct and incorrect trials in the embedding, and whether behaviorally similar but cognitively distinct trials are treated differently.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "It’s great that the method was tested across multiple datasets, which supports its robustness. Similarly, the comparison to other state-of-the-art methods is much appreciated, especially with the use of grid search to tune those methods for a fair evaluation. The analysis on the importance of different loss terms is interesting and adds depth to the understanding of the method.\n\nThe paper provides clear definitions of the loss components and the overall setup. However, it should be made clearer which specific loss terms were used to generate the results shown. This would help readers better interpret the findings and reproduce the experiments.\n\nIncorporating an LDS (linear dynamical system) component in this method is a nice addition to improve interpretability. However, the paper doesn’t clearly demonstrate how this addition can be practically leveraged.", "was_revised": true, "final_justification": "The authors present an interesting method for constructing shared latent embedding spaces across neural recording sessions using behavioural observations. The approach is tested on synthetic, mouse, and monkey datasets, demonstrating robustness and generalizability. They further compare their method to existing methods in a fair and helpful manner. I appreciate the authors’ thoughtful rebuttal, which directly addressed key concerns through additional experiments and clarifications. While some limitations to their method remain, these limitations are not critical, and the authors acknowledge these limitations clearly and outline directions for future work. Overall, I believe this is a strong and timely contribution to the field.", "rebuttal": "Thank you for your clear summary of our work, and the constructive feedback. Please find below our point-by-point responses to your comments and questions. In all cases, we either added a new experiment or specific clarifications to the text to address your concerns. We look forward to your assessment!\n\n> Weakness 1: … it should be made clearer which specific loss terms were used to generate the results shown...\n\nThank you for this suggestion. We used only the model loss and contrastive loss for Figures 2-3 and S1-S2. We added behavior supervision loss for Figures 5 and S3-S5. So, in total, we use two variations of the algorithm. We will make it clear in the paper by using a separate name for when the behavior supervision loss is used or not: CANDY and CANDY-sup.\n\n> Weakness 2: … the paper doesn’t clearly demonstrate how LDS addition can be practically leveraged.\n\nThank you for making this point. Before discussing interpretability, we want to emphasize that the LDS component in CANDY is **not just an add-on**, but is central to the **scientific question** motivating this work: **can we extract a preserved latent dynamical system (not just representations!) underlying neural activity across different animals performing similar tasks?** While contrastive learning methods like CEBRA can yield behaviorally aligned embeddings, they do not model underlying temporal structure, and therefore cannot test whether shared latent dynamics exist across individuals. CANDY is designed to fill this gap. We chose LDS as a simple, interpretable, and computationally efficient way to model latent dynamics, and because LDSs are effective for local approximations of neural dynamics [1]. Figures 2 and 4 show that contrastive alignment is critical for recovering consistent dynamics, whereas behavioral supervision alone fails to do so, as indicated by weaker decoding from dynamics factors. Moreover, Section 4.2.4 and Figures 5 and S5 demonstrate that an LDS pretrained on multiple animals can be transferred to new animals without retraining. This provides compelling empirical evidence for the existence of shared latent dynamics across individuals.\n\nIn terms of interpretability, Fig. 3 shows meaningful topology in the embedding space. In Fig. 3B, latent embeddings colored by wheel velocity reveal a smooth, 1D manifold consistent with task structure. We added the same analysis (will be placed in the Appendix) using LDS dynamical factors and observed similarly structured, low-dimensional geometry aligned with both wheel velocity and trial type. Moreover, we added an analysis on the relationship between the dynamical factors and behavioral variables across sessions. Specifically, we computed the correlation between each dimension of the LDS latent state and the observed wheel velocity across sessions when latent dimension = 8. We found dynamical factors of CANDY exhibited correlations $\\approx$ 0.68±0.04 with the behavior, whereas that of DFINE and SLDS exhibited correlations $\\approx$ 0.23±0.034 and 0.12±0.001.\n\n> Q1\n\nGood point! Beyond behavioral alignment (Figures 3B-C, S2, and S4), we find that CANDY’s latent embeddings also reflect meaningful temporal structure. We added an analysis of pairwise distances within trials and found that temporally adjacent time points remain close in latent space. Plotting Euclidean distance versus temporal separation shows clear temporal continuity in the embeddings, exactly what the LDS was designed to capture. We will add it to the Appendix.\n\n> Q2\n\nYes! We used a behavior supervision loss in the generalization experiments, since  the pretrained sessions were unseen, so contrastive comparisons weren’t possible. In L257, we stated that both the LDS and behavior decoder from the pretrained model were frozen. To clarify, we will revise the sentence to “We froze the shared LDS and behavior decoder from the pretrained model and trained only the subject-specific encoder and neural decoder on a held-out third animal using both contrastive and behavior supervision losses (5 sessions).”\n\n> Q3\n\nThanks for the suggestion! We performed this control and the results are as expected.\nWe randomly shuffled behavioral data relative to neural activity 20 times to break their correspondence. Under these conditions, the model fails to converge—the training loss remains high, embeddings lack coherent structure, and behavior decoding drops to chance. Visualizations show no session alignment or interpretable organization, in contrast to Fig. 3B-C. These results confirm that CANDY’s ability to extract structured, behaviorally meaningful embeddings depends on genuine neural–behavioral relationships, not just model architecture. We will include these findings in the Appendix.\n\n> Q4\n\nThanks for sharing this important suggestion! In the current manuscript, we tested whether the learned linear dynamical system and the behavior decoder can be generalized to unseen neural states from new subjects performing similar behaviors on both our mouse wheel-turning dataset (to be released if accepted) and a standard monkey center-out reaching dataset. Results show superior performance of the pretrained LDS and behavior decoder compared to training from scratch on the same sessions. \n\nTo address your question, we tested whether the model can generalize to new behavioral states by holding out a movement direction in the monkey dataset. Unfortunately, both CANDY (test $R^2$: mean -0.1209, s.e.m. 0.0319) and CEBRA (test $R^2$: mean -0.2138, s.e.m. 0.0248) failed to generalize to the unseen behavior category, which will be added to the Appendix. This reflects the broader challenge of out-of-distribution generalization in machine learning. We will add the following to the Limitations and Outlook section: \"While CANDY shows strong generalization of the pretrained LDS and behavior decoder to unseen sessions, both CANDY and CEBRA failed to generalize to unseen trial types (Appendix), which can be explored in future work.\"\n\n> Limitation 1: … discuss whether preserving behavioral similarity is always desirable… when and where this method is appropriate, and what the limitations ... \n\nGreat point! Several recent studies have shown that different subjects can rely on similar neural dynamics within the same brain area to drive both well‑trained behaviors [2] and more naturalistic actions [3]. In our work, we likewise find that both mice in the wheel‑turning task and monkeys in the center‑out-reaching task exploit preserved dynamics. That said, it remains unclear whether such preserved dynamics generalize across all behavioral paradigms or brain regions, especially given that distinct internal processes can sometimes produce identical observable actions. \n\nTo address this, we added a synthetic experiment where two sessions were generated from the same spiral dynamics, while the third used limit-cycle dynamics. All three shared the same linear behavioral readout. When trained on all three sessions, CANDY successfully aligned the two spiral sessions, while the third remained misaligned in the latent space. With a shared behavior decoder, the decoding performance is only ($R^2 \\approx 0.9, 0.6, 0.1$ respectively), revealing a mismatch in underlying dynamics. Based on the misalignment observed in the latent space, we identified the divergent session and excluded it. Retraining on the two aligned sessions improved decoding back to $R^2\\approx 0.99$. This result suggests that a drop in cross‑session decoding performance can serve as an initial indicator of when latent dynamics are not shared. \n\nWe will make explicit in the Limitation and outlook section by adding: “While CANDY can facilitate the identification of distinct underlying dynamics as shown in the synthetic experiment, developing methods to distinguish truly distinct neural processes that yield similar behaviors remains an important direction for future work.”\n\n> Limitation 2: … How does the method perform on error trials? … whether the model distinguishes correct and incorrect trials … behaviorally similar but cognitively distinct trials are treated differently. \n\nThank you for this insightful suggestion. As our primary goal in this manuscript is to demonstrate CANDY’s ability to recover preserved, behaviorally relevant dynamics, we have not yet evaluated its performance on error trials. Nonetheless, we agree that probing whether CANDY embeddings can distinguish correct from incorrect trials, especially when the overt behavior remains similar, would be highly informative. Due to time constraints, we will reserve the full analysis of correct versus error trial dynamics for the period leading up to the camera‑ready deadline, if accepted, and include it in the appendix.\n \n\nWe appreciate your thoughtful and challenging questions, which have helped us strengthen both the conceptual and empirical aspects of our work.  We hope that our explanation and the newly conducted experiments would address your concerns. We look forward to your reassessment!\n\n[1] Abbaspourazad, Hamidreza, et al. \"Dynamical flexible inference of nonlinear latent factors and structures in neural population activity.\" Nature Biomedical Engineering 8.1 (2024): 85-108.\n[2] Safaie, Mostafa, et al. \"Preserved neural dynamics across animals performing similar behaviour.\" Nature 623.7988 (2023): 765-771.\n[3] Nair, Aditya, et al. \"An approximate line attractor in the hypothalamus encodes an aggressive state.\" Cell 186.1 (2023): 178-193."}]

Author rebuttal:

Meta-review: The authors propose a poweful new framework for learning cross-animal shared dynamics from neural recordings called CANDY using a novel contrastive objective. This is an exciting step away from simple representation learning to latent dynamics alignment using contrastive learning with a potential to be useful for "understanding". The experiments and comparisons were satisfactory, especially the neuroscience datasets. The authors rebuttal was great. Nice work.

Although completely different in methodology and approach, the AC believes that the following paper attempts to solve a generalized version of the problem. This is just for information. No action needed.
Vermani, A., Nassar, J., Jeon, H., Dowling, M., & Park, I. M. (2024, October 7). Meta-dynamical state space models for integrative neural data analysis. The Thirteenth International Conference on Learning Representations (ICLR). https://openreview.net/forum?id=SRpq5OBpED
Decision: Accept (spotlight)