Neural Embeddings Rank: Aligning 3D latent dynamics with movements
Authors: Chenggang Chen
Zhiyu Yang
Xiaoqin Wang
Venue: NeurIPS 2024 poster (2024)
Keywords: Dimensionality reduction
Latent dynamics
Brain-machine interfaces
Neural decoding
Contrastive learning
Primary area: neuroscience_and_cognitive_science
TL;DR: We reduce high-dimensional neural dynamics to only three dimensions and decode movements using just linear and logistic regression

Abstract: Aligning neural dynamics with movements is a fundamental goal in neuroscience and brain-machine interfaces. However, there is still a lack of dimensionality reduction methods that can effectively align low-dimensional latent dynamics with movements. To address this gap, we propose Neural Embeddings Rank (NER), a technique that embeds neural dynamics into a 3D latent space and contrasts the embeddings based on movement ranks. NER learns to regress continuous representations of neural dynamics (i.e., embeddings) on continuous movements. We apply NER and six other dimensionality reduction techniques to neurons in the primary motor cortex (M1), dorsal premotor cortex (PMd), and primary somatosensory cortex (S1) as monkeys perform reaching tasks. Only NER aligns latent dynamics with both hand position and direction, visualizable in 3D. NER reveals consistent latent dynamics in M1 and PMd across sixteen sessions over a year. Using a linear regression decoder, NER explains 86\% and 97\% of the variance in velocity and position, respectively. Linear models trained on data from one session successfully decode velocity, position, and direction in held-out test data from different dates and cortical areas (64\%, 88\%, and 90\%). NER also reveals distinct latent dynamics in S1 during consistent movements and in M1 during curved reaching tasks. The code is available at https://github.com/NeuroscienceAI/NER.

Neural Embeddings Rank: Aligning 3D latent dynamics with movements
Chenggang Chen, Zhiyu Yang, Xiaoqin Wang Department of Biomedical Engineering, Johns Hopkins University aueng-gang, Bronn.ccee@ppp. chen@jhu. edu
Abstract
Aligning neural dynamics with movements a fundamental goal in neuroscience and brain-machine interfaces. However, there is still a lack of dimensionality reduction methods that can effectively align low-dimensional latent dynamics with movements. To address this gap, we propose Neural Embeddings Rank (NER), technique that embeds neural dynamics into a 3D latent space and contrasts the embeddings based movement ranks. NER learns regress continuous represen- tations of neural dynamics (i.e., embeddings) on continuous movements. We apply NER and six other dimensionality reduction techniques neurons in the primary motor cortex (M1), dorsal premotor cortex (PMd), and primary somatosensory cortex (SI) as monkeys perform reaching tasks. Only NER aligns latent dynamics with both hand position and direction, visualizable in 3D. NER reveals consistent latent dynamics in MI and PMd across sixteen sessions over year. Using a linear regression decoder, NER explains 86% and 97% of the variance in velocity and position, respectively. Linear models trained on data from one session successfully decode velocity, position, and direction in held-out test data from different dates and cortical areas (64%, 88%, and 90%). NER also reveals distinct latent dynamics in S1 during consistent movements and in MI during curved reaching tasks. The code is available at http:///gtthhhcoommeennnneeeee/
Introduction
has long been thought that individual neurons in the motor and premotor cortex, similar those in the sensory cortex, are tuned to specific movement parameters such as direction. However, this static and receptive field-based neural representation fails explain movement trajectories during simple tasks like reaching. Recent studies have found that the activities of multiple simultaneously recorded neurons, which fire spikes time-dependent manner, encode reaching movements [10]. Unlike the one-dimensional dynamics from single neuron, understanding how movements are represented by these high-dimensional neural dynamics is challenging. In systems neuroscience and brain-machine interfaces, there is significant motivation reduce these high-dimensional neural dynamics to low-dimensional latent dynamics for at least three reasons:
First, visualizing neural dynamics. This involves a trade-off between dimensionality and explained variance. To explain a complex reaching task, least six dimensions are typically required. For example, in eight-direction center-out reaching task, [II [ [12) selects fifteen dimensions for the PMd, ten for the MI, and eight for the S1. Therefore, we need to further reduce dimensionality of these flowddimensional" latent dynamics. Currently, we still lack method that can directly explain enough variance within three dimensions. Second, compuring movements-relatt After dimensionality reduction,
latent dynamics. visualize the trajectories of latent dynamics over time. For instance, [5] reveals rotational latent dynamics during reaching tasks using principal component analysis (PCA). 2 found that animals performing similar tasks exhibit similar latent dynamics.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

However, these trajectories do not align precisely with the reaching movements: when the hand reaches in eight directions, the trajectories of latent dynamies are neither in eight distinct directions nor well-separated, often appearing entangled. Third, decoding movements using latent dynamics. Decoders trained on individual neural activities are commonly used to predict movements [3]. However, a drawback of using indívidual neural activities is that when the identities of neurons change during long-term recordings, the decoding performance deteriorates [2] Additionally, is infeasible to decode movements from different brain areas animals. Decoders trained on latent dynamics facilitate long-term and cross-animal decoding 9 25]. Since latent dynamics do not fully capture neural dynamics, decoding performance is often suboptimal with linear decoders, necessitating the use of nonlinear decoders or deep neural networks [25]. Thus, decoding movements using a linear decoder without hyperparameters remains significant challenge.
As our goal to extract latent dynamics that are most informative about movements, we chose to train the latent dynamics using movements as the target. Several recent studies have trained latent dynamics to classify different movement directions or positions using variational autoencoders (VAE) 18 or contrastive learning [27 []. In this paper, are inspired by the fact that many features, including movements, are continuous, and that the function of many biological neurons is not classification but regression. For example, neurons exhibit monotonic tuning to light intensity and sound levels [2] [24]. Even for discrete features like faces, face cells exhibit ramp- samp-shaped tuning to different features 81. Additionally, 75% of faces could be correctly decoded using a linear regression decoder 4 Thus, we trained latent dynamics to regress movement by minimizing the ranking loss.
32
We are motivated by the fact that CEBRA treats continuous labels as many discrete classes, which cannot be well separated in low-dimensional space. These classes are also highly imbalanced, with many more near-zero classes (Fig. D
Our two main contributions are as follows:
We introduce Neural Embeddings Rank (NER), a dimensionality reduction method that contrasts paired samples in the embedding space and ranks neural embeddings to align them with continuous movement labels (Fig. [T)). Without introducing additional hyperparameters and using the same inputs, NER addresses the high dimensionality and class imbalance issues present in CEBRA.
We demonstrate that NER reveals behavior-aligned latent dynamics in all twenty sessions from three different brain areas in three monkeys. The trajectories of latent dynamics vary across different behaviors. We show that all movement parameters (directions, velocities, and positions) can be decoded using linear decoders trained on data from one session, tested on data from different dates over one year, and across different brain areas and hemispheres.
vel. (cm/s) e
200
X
b
anchor Vi
continuous label
0
same label [+At
positive V)
negative
NER
-2000
200
At = "augmentation"
2nd
1st
3rd
label dist. w.r.t. anchor
0
CEBRA
O
rank emb. w.r.t. label
pred. (cm/s)
CEBRA
positive Vj A
OA
1 Pair
N. Pair
-200
2200
negative
O
1 P. Pair
3 N. Pair
1 Pair
N. Pair
0
NER
1 P. Pair
1 N. Pair
-200
-200 vel. (cm/s) 200
1 P. Pair 3 N. Pair
1 P. Pair
N. Pair
Figure I: a Top: Hand velocities of X-Y coordinates across three trials. Note that the distribution of velocities is highly imbalanced, with many near-zero values. Bottom: Real (X-axis) versus predicted (Y-axis) Y-coordinate hand velocities using linear regressor. The predicted velocities in CEBRA are much smaller than those in NER, as CEBRA mispredicts infrequent large velocities as frequent small velocities. b Batch size is three (in real experiments, is 512), with two batches (one for e; and one for Uy and vy) combined in NER. Both models use embeddings within specific time offset (e.g., 10 ms) of the anchor as the augmented embedding, which shares the same label/color as the anchor.

---

a
Hand positions! velocities & directions 80
Noural latent dynamics
Aligned latent dynamics
Decoder
Decoded velocities!
posiiions & directions
Train
velooises
directions
M1
M1
dynamics
270
piVAE
neuron
/mmmmm
day
high
1 year later
198
200
contra M1
NER
low
latent
Jasi PMd
same
day
Time
Figure 2: NER aligns 3D latent dynamics with movements and enabling cross sessions movement decoding. Top: The monkey performs center-out reaching task in eight directions using a planar manipulandum. Hand velocity is computed from hand position. Bottom: The monkey moves the cursor outer targets to receive rewards (image from (23)) Top:
Neural dynamics are recorded using 96-channel Utah array in two monkeys. Monkey C has implants in the MI of the right hemisphere first and in MI and PMd of the left hemisphere second. Monkey has an implant in area 2 of the S1 of the left hemisphere (image from brainmuseum. org). During the task, all monkeys use the hand contralateral to the implanted hemisphere. Bottom: Spiking activities from multiple neurons (44 to 211 I) are recorded during multiple trials (168 to 1038) of the behavioral task. Dimensionality reduction reduces 200-dimensional neural dynamics 3D latent dynamics. Top: Neural dynamics from 190 dimensions (neurons) in the PMd are reduced to 3D latent dynamics. Bottom: Trial-averaged latent dynamics. Data is from Monkey C (date: 2016 10-14). Top: Linear and logistic regression decoders are trained the same independent variables (latent dynamies) but different dependent variables (hand velocities and directions) using 80% of train data. Bottom: The trained decoder from 80% of train data is used to predict movements on the 20% held-out test data. The model trained on one day predicts movements across one year, the contralateral MI, and ipsilateral PMd. Top: Two linear decoders trained from decode hand velocities (positions) and directions with R-squared accuracy of 86% (96%) and peak accuracy of 97%, respectively, on the 20% held-out test data. Bottom: Decoders trained on the same day (diagonal) have much better decoding performance than models trained on different conditions (off-diagonal) using previous dimensionality reduction methods, whereas method has higher and consistent decoding performance.
2 Related work
There are at least five categories of dimensionality reduction methods: Linear methods, such as PCA, jPCA [5) demixed PCA (dPCA) (16), and preferential subspace identification (PSID) 26). PCA captures the majority of variance in the data, jPCA reveals rotational dynamics in monkey reaching, dPCA highlights task-related components, and PSID extracts latent dynamics that predict motion during reach versus return epochs. Nonlinear methods like uniform manifold approximation and projection (UMAP) [19] and t-distributed stochastic neighbor embedding (tSNE) (28 extensively used in biological data, such as for identifying neuron cell types [7]. These methods reveal identities but often collapse temporal dynamics that resemble neural trajectories. UMAP with labels has been applied for dimensionality reduction in recent studies (27 32). Generative methods using recurrent neural networks or Transformers, such as latent factor analysis via dynamical systems (LFADS)
21 AutoLFADS (15), RADICaL [3]] and Neural Data Transformer (NDT) 30 These methods model tngle-triall variability in neural spiking activity better than PCA, though they often require explicit assumptions about underlying data statistics. Label-guided generative methods using VAEs, including Poisson identifiable VAE (piVAE) [32), Swap 18, and targeted neural dynamical modeling (TNDM) [14]. For example, PiVAE uses both discrete and continuous labels to shape embeddings, revealing well-separated but less movement-aligned embeddings. Contrastive learning methods, introduced to learn robust, generalizable representations of neural population dynamics,

---

such as CEBRA 27 and Mine Your Own view (MYOW) M. Compared to piVAE, AutoLFADS, and UMAP, CEBRA provides more identifiable latent dynamies corresponding different hand directions in $1, although these latent dynamics trajectories do not correlate well with movements.
3 Model
NER is inspired by previous studies [1 and uses the same data sampling and neural feature encoder as CEBRA [27] to extract neural embeddings. Fig. Th illustrates the difference between NER and CEBRA. CEBRA treats each embedding in batch as discrete class. For an anchor, it contrasts with its augmented embedding as a positive pair and three randomly sampled embeddings as negative pairs. NER, on the other hand, ranks six embeddings according to their continuous labels. contrasts an anchor with its augmented or first embedding as a positive pair and the remaining four embeddings as negative pairs. NER continues by contrasting the second embedding as a positive pair and the remaining three embeddings negative pairs. This process repeats until all embeddings have been positively contrasted with the anchor. NER learns a regression-awan representation that orders all embeddings in a batch.
Mathematically, we define as the high-dimensional neural dynamics, ʃ as the feature encoder, and v = f(x) as the low- low-dimessional neural embeddings. The batch size N is set to 512, and the temperature 7 fixed at Data augmentation in both CEBRA and NER is achieved by selecting embeddings whose labels fall within a specific offset (e.g., 10 ms) of the anchor's 's label. Note that we did not fine-tune the temperature and offset, as the only difference between CEBRA and NER the loss function. The selection of these hyperparameters will have similar effect on both models.
In each iteration, CEBRA receives positively contrast with the anchors, and a third batch of randomly sampled embeddings on that will negatively contrast with the anchors. The anchor loss I in CEBRA is:
batch of anchors U, one batch of embeddings vy that will
exp(sim((s, (;))m CEBRA -log Σ NN=rerp(simml vn)/T)
where sim(.,) represents the similarity between two neural embeddings (e.g., negative L2).
In each iteration, NER receives one batch of anchors vú one batch of augmented embeddings Uy Uk that will either positively negatively contrast with anchor, and a third batch of labels y. The anchor loss in NER is:
Σ 2N 2NN=1J4i log exp(sim((s, (1),s
NER
2N
Empest exp(simm Ux)/T)
There are two key differences from CEBRA. First, batches of U; and Vj. 1 are merged, and labels y duplicated, resulting in a batch of 2N for both emheddings and labels. This ensures that each anchor and its augmented embedding exist within the same batch. Second, we introduce St, : {vk k A i, d,d(y, yu) d(y;, } to denote the set of embeddings Us that are of lower ranks than Uj in terms of label distance relative to Us where d(-,-) is the distance measure between two labels (e.g., L1). Intuitively, for an anchor U; any other embedding vy in the batch is positively contrasted with it, enforcing the similarity between and 12 to be greater than that between and any other embedding 2y in the batch if the label distance between y/4 and yı is larger than that of
y: and yj- Minimizing 'EE aligns the order of embeddings with their corresponding label orders (6)
relative to the anchor.
By ranking all the embeddings within a batch, NER also addresses the class imbalance issue and effectively represents infrequent classes (Fig. [T].. In each batch, small percentage (e.g.. 5%) of embeddings may come from infrequent classes, such as large velocities. In CEBRA, only a single augmented embedding is positively contrasted with its anchor, only 5% of anchors have access to these infrequent embeddings. In NER, all 2N - embeddings can contrast with an anchor, allowing 100% of anchors to access the 5% of infrequent embeddings.
4 Results
To fairly evaluate our dimensionality reduction method against related approaches, we selected six representative methods from various categories: PCA, dPCA, UMAP (with and without labels).

---

PIVAE, and CEBRA. To avoid bias from single session in specific brain area, where pi piV and CEBRA were previously tested, we conducted experiments across MI, PMd, and SI, covering total of twenty sessions (Table Statistical results for each comparison are provided (Table 2 Figure illustrates the pipeline of this study. We primarily used linear decoders, and additionally included nonlinear k-nearest neighbors (kNN) decoder. To maintain consistency with movement decoding in the original CEBRA paper. we used a 16-dimensional (16D) CEBRA model, which has a stronger representation capacity and demonstrates better performance than the 3D model in kNN decoder.
4.1 Molement-aligned latent dynamics were consistent over years in M1
Across all ten sessions in the left and right hemispheres of M1, NER consistently revealed neural embeddings that aligned with movement (Fig Fig(Da). Notably, during the initial movement stage, the latent dynamics converged on the same starting points, forming a pinwheel structure resembling the ground truth movements. Furthermore, observed nearly identical neural embeddings in both hemispheres, even when data collection was separated by over a year.
M1 in Left Hemisphere
M1 In Right Hemisphere
a 2016-09-29
2016-10-06
2016-10-14
2016-10-21
2015-03-13
2015-03-19
NER
b
CEBRA
C
piVAE
Figure 3: NER reveals consistent, movement-aligned latent dynamics in M1. Single-trial (top) and averaged (bottom) latent dynamics from six sessions across one year in two hemispheres. Latent dynamics are rotated with reference to the 2016-10- 10-10-014 14 session, using one of the eight reaching directions. b-c Similar to a but using CEBRA and pi VAF. Fig. D shows the latent dynamics from the remaining four sessions for the same monkey. Fig. T0 displays single-trial and/or averaged latent dynamics revealed by five other methods. Fig. shows trial-averaged latent dynamics before rotation. Fig. 12 presents the entangled neural embeddings using PCA and the time-stimulus components revealed by dPCA.
CEBRA was the second-best method, uncovering comparable latent dynamics with both directions and positions roughly aligned with movements for both single and averaged embeddings (Fig(3b, Fig Øb). However, CEBRA had two limitations: the movement starting points were widely separated,
5

---

differing from ground truth movements, and its latent dynamics were less consistent across sessions. For example, it only revealed connected latent dynamics at movement starting points in two sessions.
piVAE ranked third, displaying direction-aligned latent dynamics in different directions with relatively separated single-trial neural emheddings (Fig/3), Fig(D)) However, while correlated with movements, the latent dynamics were not aligned with them and were less consistent across sessions. UMAP with labels showed clearly clustered neural embeddings corresponding to different angles, while UMAP without labels produced extended, less clustered latent dynamics (Fig. TT). Both methods failed to generate aligned and consistent latent dynamics. PCA and dPCA also generated identifiable latent dynamics (Fig TOp, with dPCA revealing both time and stimulus components. However, a major limitation of both methods was the mixing of single-trial neural embeddings (Fi.
In summary, NER proved to be the best method for revealing movement-aligned latent dynamics. We further evaluated its performance in PMd and S1, leveraging these aligned latent dynamics decode movements within and across sessions and to explore movement-specific latent dynamics.
4.2 Explained variance of movements using linear decoders in M1, PMd, and S1
Five dimensionality reduction methods were used to reveal single-trial latent dynamics that depended on movements. We then used these latent dynamics independent variables to explain the variance of dependent variables, namely, hand velocities, directions, and positions.
Fig. 4a shows the ground truth and predicted hand movement trajectories using latent dynamics revealed by NER in PMd. A linear regression decoder explained 90% and 98% of the variance in hand velocities and positions, respectively.
a
Hand movements
Explained movements
Directions and X-Y velocities
100
Diecction slood cotion accurcy
80-
CEBRA
60-
DIVAE
UMAP w/ label
40-
UMAP wia label
X-vel
20-
Y-vel
0-
200
4:00
600
800
1000
1200
Time from go cue Ims)
Velocities
d
Positions
M1
PMd
S1
M1
PMd
S1
0.9
0.9
0.8
0.6-
0.6
0.5-
0.5
0.2
02
01
10200
10,000
1992
19823
190839
180020
160015
300007
1000
371207
0,3
100995
102000
130009
110000
50003
200000
1500
19
10000
Figure 4: Explained variance of movements in MI, PMd, and S1. a Left: Hand movement trajectories.
Right: Predicted trajectories by two decoders. Data collected from PMd on 2016 10-14. The
explained variance for velocities and positions is 90% and 98%, respectively. b Hand direction classification accuracy using a logistic regression decoder. Shaded areas represent the standard deviation across six sessions from MI. Explained variance of hand velocities using a linear regression decoder on latent dynamics revealed by five dimensionality reduction methods (indicated by different colors and shapes). The X-axis shows session dates. Left: Ten sessions from the M1 of Monkey C. Right: Six sessions from PMd of Monkey C and four sessions from S1 of Monkey H. Similar to but for hand positions. Fig. 13 provides the direction tuning curve in PMd, the correlation between tuning curves and velocities, and explained variance for directions.
In both MI and PMd (Fig D. accuracy from the start of the go to the end of the animal's reach. This tuning curve was highly
13p a logistic regression decoder revealed the tuning of directional

---

correlated with hand velocities in the latent dynamics extracted by NER but not by CEBRA (MI: 0.93 vs. 0.28, PMd: 0.93 vs. 0.13: Fig. 13p).
NER outperformed the four other methods in all sessions for explaining the variance of both hand velocities and positions (Fig. Pp. d). For example, across ten sessions in M1, NER explained 86% of the variance in velocities, while the next best model, piVAE, explained only 35%. Similar findings were observed in PMd (89% vs. 32%) and SI (86% 47%).
In summary, combined with linear decoders, NER demonstrated the clearest velocity-dependent direction tuning and explained the largest variance in velocities and positions.
4.3 Long-term and cross-hemisphere decoding in M1
NER explains the largest variance in both hand velocities and positions in the 80% training data. Next, we tested the trained model on the remaining 20% of test data. In addition to decoding test data from the same session on single day, we also evaluated its performance on data from different sessions. We first conducted comparisons in M1 5 followed by PMd (Fig and S1 (Fig. 7
Linear logistic regression
k-nearest neighbors
NER dia: 0.71 off: 0.64
CEBRA dia:-3.1 off -3.2
DIVAE dia:-11. off:
NER dia: 0.76 off:
CEBRA 160 dia: 0.75 off:
PIVAE daa 0.87 off:
150313
15031.
0 150630
160929
161005
161007
161014
disc 0.34 off: 0.68
diac 0.74 0.71
die: 0.67 off: 8.23
dia: 0.98 off: 0.89
dia: 0.58 aff: 0:18
dia: 0.07 off: 0.13
150313
150319
160929
3 1160055
161007
181014
dia: 98% off: 90%
dia: 90% off: 79%
dia: 61% off: 44%
92% 99% 44%
150319
I 3350029
16092-9
165005
161007
161014
161021
Figure 5: Decoding within and across time and brain hemispheres over years in M1. a-d Three methods are applied to neural dynamics from ten sessions. Linear and logistic regression decoders (a, e) nonlinear k-nearest neighbors (kNN) decoder (b, d. f) trained 80% of the data and used to decode velocities, positions, and directions on the remaining 20% within-session (diagonal) and cross-session (off-diagonal). Brighter colors indicate higher decoding performance. a Same-session, cross-session, and cross-hemisphere velocity decoding using a linear regression decoder. b Similar to but using the nonlinear kNN decoder. Note that CEBRA's latent dynamics are represented in sixteen dimensions instead of three. c-d Similar to a-b but with hand positions as the decoding variable. e-f Similar to a-b but with hand directions the decoding variable.
Fig5p shows velocity decoding performance using linear regression decoder across three dimen- sionality reduction methods. Interestingly, the linear decoder could not decode hand velocities from latent dynamics generated by CEBRA (all variances were negative) and piVAE (only four positive). In contrast, all variances with NER were positive (minimum: 0.24), with performance across different sessions comparable to within-session performance (0.64 vs. 0.71). A kNN decoder (Fig (5)) achieved high performance for CEBRA and piVAE only with within-sessiee latent dynamics.
NER outperformed CEBRA and PiVAE by a substantial margin in position decoding across all conditions (Fig(5)). While the kNN decoder did not improve NER's 's performance relative to the linear decoder, enhanced within-session performance for CEBRA and pi VAE over NER (Fig 1). However, this came at the cost of cross-session decoding performance (0.89 vs. 0.18 and 0.13). Fig 5 shows direction decoding accuracy, where NER still outperformed CEBRA and pi ViVE. Similar results were observed using kNN decoder (Fig 5h.

---

In summary, compared to CEBRA and piVAE, NER consistently achieved higher performance across all sessions with a linear decoder and outperformed in cross-session decoding with the nonlinear decoder.
4.4 Latent dynamics in PMd and decoding between M1 and PMd
Next, we turned our attention to PMd. Surprisingly, NER revealed similar latent dynamics in this higher-order motor (Fig Ga). Two other dimensionality reduction methods also identified comparable latent dynamics, but these were less consistent and did not align well with movements (4). Fig. 66 shows within- and cross-session velocity decoding using linear regression decoder (left) and a kNN decoder (right). Consistent with M1, in PMd, we observed that (1) both CEBRA and piVAE underperformed with linear regression, while NER achieved stable performance across all conditions, regardless of within- or cross-session contexts; (2) NER demonstrated robustness across decoders, whereas the performance of CEBRA and piVAE varied substantially, ranging from very low to occasionally outperforming NER in within-session decoding with the kNN decoder; and (3) even with the kNN decoder, CEBRA and piVAE failed in cross-session decoding, whereas NER maintained similar performance.
We also evaluated position and direction decoding and found that NER continued to outperform the other two methods Ob, [5). Overall, NER reveals consistent latent dynamics in PMd and can effectively decode movements across PMd and M1.
a
2016 2016-09-29
10-05
10-06
10-07
10-14
10-21
Linear regression CEBRA
k-nearest neighbors CEBRA 16D
b
NER dia: 0.75 off: 0.74
PIVAE dia: off: -2.8
NER dia: 0.76
PIVAE off:
dia: -3.1 off: -3.1
L60020N
TE0 DOPRCOPM
19
PPM
dia:
16100GN1
Figure 6: Latent dynamics in PMd and decoding across brain areas. Trial-averaged latent dynamics in PMd revealed by NER, with the rotation reference set to the same session as in Fig. (i.e., 2016-10-14). b Previously used latent dynamics from MI are added. Same-date, cross-date, and cross-brain area velocity decoding using linear regression decoder (left) and a kNN decoder (right) on latent dynamics revealed by the three methods, Similar to b. Fig, shows the single-trial and averaged latent dynamics revealed by CEBRA and VAE. Fig. [5 shows hand direction decoding performance using a logistic regression decoder.
4.5 Same movements but different latent dynamics in S1
Lastly, we examined the latent dynamics and movement decoding in S1. NER revealed consistent latent dynamics in S1 (Fig 2 b). After rotating the latent dynamics with reference to the target shown in Fig they exhibit consistent yet distinct shape compared the latent dynamics observed in MI and PMd. Velocity decoding using linear regression decoder was only successful when the latent dynamics were extracted by NER (Fig Pp. d). While all three methods performed adequately for position decoding. NER outperformed both CEBRA and PiVAE across all nine conditions (Fig (F) Although SI displays different latent dynamics than and PMd, NER remains the most effective method for decoding movement both within and across sessions in S1.
8

---

a
17-12-01
12404
12-07
d
NER
CEBRA
PIVAE
dia: 0.67 off: 0.56
dia: -4.6 off:
dia: -0.38 off: -0.75
Veloctties
371120
171207
dia: off:
dia: 0.6 off: 0.45
off: 0.51
171201
Pocttons
171204
171207
Explaned
Decoded
Figure 7: Distinct latent dynamics with the same movement in S1. Trial- latent dynamies revealed by NER, using the same reference target session as in Fig. 3 Latent dynamics with the reference target set the first session in S1. Leftt On 80% held-in trials, the explained variance by linear and logistic regression decoders is 91 % and 97%, respectively. Right: On 20% held-out test trials, the trained linear decoder predicts velocities and positions with performances of 71% and 90%, respectively. Data collected on 2017-12-01. Same- and cross-date velocity decoding performance using the linear regression decoder with three dimensionality reduction methods. Similar to but for positions. Fig. 16 shows latent dynamics revealed by CEBRA and piVAE.
4.6 Straight and curved hand movements have different latent dynamics in M1
Lastly. we applied both NER and CEBRA new experiment where a monkey performed both straight and curved hand movements in different directions while neural recordings were simultane- ously collected in M1 (Fig. (p).
We first examined the latent dynamics when the monkey performed straight hand movements in six different directions (Fig Bb). Surprisingly, both single and averaged latent dynamics aligned well with the movements 8b) and displayed shape similar to the previously observed latent dynamics in MI. Next, selected three hand directions, each involving both straight and curved hand movements (p). When we trained both NER and CEBRA on individual directions, only NER revealed clearly separated latent dynamics corresponding to straight and curved movements ()) The difference between the two methods became even more pronounced when they were trained on all three directions combined: NER displayed latent dynamics for straight movements that were surrounded by those formed by curved movements. The explained variance achieved by NER was also higher than that of CEBRA across all three angles, especially on the combined angles tested more challenging condition where all six reaching movements were curved (Fig (g). In the latent space, two curved movements in same direction produced close but distinet latent dynamics Bh). While CEBRA performed comparably single directions, it struggled with combined directions 17e). NER consistently achieved higher explained variance than CEBRA 17D.
(Fig. 17 2. Finally,
Overall, NER only aligns latent dynamics with straight movements but also effectively differ- entiates curved hand movements from straight ones, demonstrating its robustness across various movement types.
Discussion
benchmark comparison of NER and six other dimensionality reduction methods across multiple brain areas and two movement tasks highlights NER', superior performance in uncovering latent dynamics. We believe the primary advantage of our method is its ability extract nearly identical latent dynamics across brain areas and over extended time periods. This capability opens new avenues for both fundamental neuroscience research and brain-machine interfaces (BMI). Previous studies 9 25 used PCA to discover preserved latent dynamics across time and in animals performing similar behaviors. In contrast, the latent dynamics revealed by NER are significantly more informative than those uncovered by PCA. We believe NER will aid neuroscientists in probing the stability of latent

---

a
b
d
6 straight move
3 pairs of straight-curve move
h
3 pairs of curve-curve move
Figure 8: Distinct latent dynamics align with different movements in M1. a A monkey performs a curved reaching task through virtual maze while neural activity from MI is recorded simultancously. The task includes multiple reaching directions, and the monkey makes curved movements when encountering barrier the trajectory (reproduced from (2)). b Hand positions for six target directions without barriers (i.e., straight movements). e Latent dynamics from a single trial (left) and averaged across trials (right). d Explained variance of hand movements (velocity: 0.79, position: 0.92). Hand directions (represented by colors) are assigned manually. e Hand movements for three target directions, both without barriers (straight movements) and with barriers (curved movements). f Latent dynamics of curved (stars) and straight (dots with black line) hand movements shown separately and combined across directions. Hand movements for three target directions, all with barriers (curved movements). h Latent dynamics trained on one pair (Ist) or three pairs (3rd) of curved movements, with decoder-explained hand movements (2nd and 4th). Fig. 17 provides comparison of results with CEBRA.
dynamics under various conditions. For BMI applications, we demonstrate that NER, combined with simple linear decoder, predict hand movements across years, brain areas, and hemispheres. This capability enables training on latent dynamics within and between subjects, allowing for movement prediction in different subjects. The linear decoder's 's lack of hyperparameters is also an advantage.
The application of NER is not limited to hand movements using neurophysiological recordings. This includes latent dynamics in the hippocampus representing the body position of running rats and latent dynamics in the visual cortex representing embedded video features [27]. Similarly, recording modalities are not limited to single-neuron electrophysiology: other methods, such as
calcium imaging, local-field potentials, and EEG, can also be used.
In our final experiments with curved movements, we manually selected three pairs of reaching tasks that involved curved trajectories. Both NER and CEBRA failed when all 108 movement conditions were trained simultancously. Furthermore, beyond the straight and curved movements examined here macaque monkeys, more complex movements, such handwriting 29 and speech (2 (0), exist primarily in humans. These movements are also continuous but may require more than a 3D) latent space for effective representation, and ranking the distances of complex movements adds further challenges. Nevertheless. uncovering the latent dynamics underlying these complex movements, which has yet to be 7c could greatly advance BMI applications 3 3)
Acknowledgments and Disclosure of Funding
We greatly appreciate the Miller and Shenoy labs for publicly releasing their experimental data macaque monkeys. We also thank the Mathis lab for the CEBRA code, which served as the basis for NER. This work was supported by National Institute of Deafness and Other Communications Disorders grant DC003180 (X. W.). No conflicts of interest are declared by the authors.
10

---

References
[1] Mehdi Azabou, Mohammad Gheshlaghi Azar, Ran Liu, Chi-Heng Lin, Erik C Johnson, Kiran Bhaskaran-Nair, Max Dabagia, Bernardo Avila-Pires Lindsey Kitchell, Keith B Hengen, et al. Mine your own view: Self-supervised learning through across-sample prediction. arXiv preprint arXiv:2/02.10166 2021.
[2] Matteo Carandini and David Heeger. Normalization canonical neural computation. Nature reviews neuroscience, 13(1)551-62, 2012.
[3] Nicholas S Card, Maitreyee Wairagkar, Carrina lacobacci, Xianda Hou, Tyler Singer-Clark, Francis R Willett, Erin M Kunz, Chaofei Fan, Maryam Vahdati Nia, Darrel R Deo, et An accurate and rapidly calibrating speech neuroprosthesis. New England Journal of Medicine, 391(7):609-66) 618, 2024.
Clark,
[4] Le Chang and Doris Tsao. The code for facial identity in the primate brain. Cell, 169(6)10013- 1028, 2017.
[5] Mark M Churchland, John P Cunningham, Matthew T Kaufman, Justin D Foster, Paul Nuyu- jukian, Stephen Ryu, and Krishna Shenoy. Neural population dynamics during reaching. Nature, 487(7405):51 2012.
[6] Chaofei Fan, Nick Hahn, Foram Kamdar, Donald Avansino, Guy Wilson, Leigh Hochberg. Kr- ishna V Shenoy, Jaimie Henderson, and Francis Willett. Plug-and-play stability for intracortical brain-computer interfaces: a one-year demonstration of seamless brain-to-text communication. Advances in neural information processing systems, 36, 2024.
[7] Cátia Fortunato, Jorge Bennasar- Vázquez, Junchol Park, Joanna C Chang, Lee E Miller, Joshua Dudman, Matthew G Perich, and Juan A Gallego. Nonlinear manifolds underlie neural population activity during behaviour. bioRxiv, 2023.
Bennasar-Vázquez,
[8] Winrich A Freiwald, Doris Y Tsao, and Margaret S Livingstone. A face feature space in the macaque temporal lobe. Nature neuroscience, 12(9): 2009.
[9] Juan A Gallego, Matthew G Perich, Raeed H Chowdhury, Sara Solla, and Lee E Miller. Long-term stability of cortical population dynamics underlying consistent behavior. Nature neuroscience, 23(2):200-27) 2020.
[10] Juan A Gallego, Matthew G Perich, Lee E Miller, and Sara A Solla. Neural manifolds for the control of movement. Neuron, 94(5):978-984 2017.
[11] Juan Gallego, Matthew G Perich, Stephanie N Naufel, Christian Ethier, Sara A Solla, and Lee E Miller. Cortical population activity within a preserved neural manifold underlies multiple motor behaviors. Nature communications, 91)::4233, 2018
[12] Cecilia Gallego-Carraeedo, Matthew G Perich, Raeed H Chowdhury, Lee E Miller, and Juan ÁI- varo Gallego. Local field potentials reflect cortical population dynamics in region-specific and frequency-dependent manner. Elife. 11:111555, 2022.
[13] Joshua Glaser, Ari S Benjamin, Raeed H Chowdhury, Matthew Perich, Lee E Miller, and Konrad P Kording. Machine learning for neural decoding. Eneuro, 7(4), 2020.
[14] Cole Hurwitz, Akash Srivastava, Kai Xu, Justin Jude, Matthew Perich, Lee Miller, and Mathhis Hennig. Targeted neural dynamical modeling. Advances in Neural Information Processing Systems, 294-29379-299997 2021.
[15] Mohammad Reza Keshtkaran, Andrew R Sedler, Raeed Chowdhury, Raghav Tandon, Diya Basrai, Sarah L Nguyen, Hansem Sohn, Mehrdad Jazayeri, Lee E Miller, and Chethan Pandari- nath. A large-scale neural network training framework for generalized estimation of single-trial population dynamics. Nature Methods, 19(12): 2022.
[16] Dmitry Kobak, Wieland Brendel, Christos Constantinidis, Claudia E Feierstein, Adam Kepecs, Zachary F Mainen, Xue-Lian Qi, Ranulfo Romo, Naoshige Uchida, and Christian Machens. Demixed principal component analysis of neural population data. elife, 5:e10989, 2016.
11

---

[17] Eric Kenji Lee, Hymavathy Balasubramanian, Alexandra Tsolias, Stephanie Udochukwu Anakwe, Maria Medalla, Krishna V Shenoy, and Chandramouli Chandrasekaran. Non-linear dimensionality reduction on extracellular waveforms reveals cell type diversity in premotor cortex. Elife, 10:e67490, 2021.
[18) Ran Liu, Mehdi Azabou, Max Dabagia, Chi-Heng Lin, Mohammad Gheshlaghi Azar, Keith Hengen, Michal Valko, and Eva Dyer. Drop, swap, and generate: A self-supervised approach for generating neural activity. Advances in neural information processing systems, 34: 34:10597-10599, 2021.
[19] Leland Mclnnes, John Healy, and James Melville. Umap: Uniform manifold approximation and projection for dimension reduction. arXiv preprint 0rXiv:/802.03422) 2018.
[20] Sean I. Metzger, Kaylo T' Littlejohn, Alexander B Silva, David A Moses, Margaret P Seaton, Ran Wang. Maximilian E Dougherty, Jessie Liu, Peter Wu, Michael Berger, et al. A high- performance neuroprosthesis for speech decoding and avatar control. Nature, 620(7976):1037- 1046, 2023.
[21] Chethan Pandarinath, Daniel J O' Shea, Jasmine Collins, Rafal Jozefowicz, Sergey D Stavisky, Jonathan C Kao, Eric M Trautmann, Matthew Kaufman, Stephen Ryu, Leigh Hochberg. et al. Inferring single-trial neural population dynamics using sequential auto-encoders. Nature methods, 15(10):805-81) 2018.
[22] Felix Pei, Joel Ye, David Zoltowski, Anqi Wu, Raeed H Chowdhury, Hansem Sohn, Joseph E O'Doherty, Krishna Shenoy, Matthew T Kaufman, Mark Churchland, et al. Neural latents benchmark 21: evaluating latent variable models of neural population activity, arXiv preprint arXiv:2109.04463 2021.
[23] Matthew G Perich, Juan A Gallego, and Lee E Miller. A neural population mechanism for rapid learning. Neuron, 10044):964-77,, 2018.
[24] Neil C Rabinowitz, Ben DB Willmore, Jan WH Schnupp, and Andrew J King. Contrast gain control auditory cortex. Neuron, 70(6): 179-1191, 2011.
[25] Mostafa Safaie, Joanna C Chang, Junchol Park, Lee E Miller, Joshua T Dudman, Matthew G Perich, and Juan Gallego. Preserved neural dynamics across animals performing similar behaviour. Nature, 623(7988))765-771, 2023.
[26] Omid G Sani, Hamidreza Abbaspourazad, Yan Wong, Bijan Pesaran, and Maryam Shanechi. Modeling behaviorally relevant neural dynamics enabled by preferential subspace identification. Nature Neuroscience, 24(1)9140-149, 2021.
[27] Steffen Schneider, Jin Hwa Lee, and Mackenzie Weygandt Mathis. Learnable latent embeddings for joint behavioural and neural analysis. Nature, 617(7960)36003366 2023.
[28] Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 911)), 2008
[29] Francis R Willett, Donald T Avansino, Leigh R Hochberg, Jaimie M Henderson, and Kr- ishna Shenoy. Higg--perfommacc brain-to-text communication via handwriting. Nature, (93 254, 2021.
[30] Joel Ye and Chethan Pandarinath. Representation learning for neural population activity with neural data transformers. arXiv preprini arXiv:2108.0022: 2021.
[31] Kaiwen Zha, Peng Cao, Jeany Son, Yuzhe Yang, and Dina Katabi. Rank-n-contrast: Learning continuous representations for regression. Advances in Neural Information Processing Systems, 36, 2024.
[32] Ding Zhou and Xue-Xin Wei. Learning identifiable and interpretable latent models of high- dimensional neural activity using pi-vae, Advances in Neural Information Processing Systems, 33:7234-7247, 2020.
12

---

[33] Feng Zhu, Harrison A Grier, Raghav Tandon, Changjia Cai, Anjali Agarwal, Andrea Gio- vannucci, Matthew Kaufman, and Chethan Pandarinath. A deep learning framework for inference of single-trial neural population dynamics from calcium imaging with subframe temporal resolution. Nature neuroscience, 25(12)1724-1174, 2022.
13

---

A Appendix supplemental material
A.1
Computer
Operating System is Ubuntu 22.04.3 LTS, computer memory is 42 GB, CPU is Intel Xcon W-2225, and GPU is NVIDIA RTX A5000.
A.2
Center-out reaching experiments in MI, PMd, and S1
A.2.1
Behavior
The dataset comprises behavioral task data from two male Macaca mulatta monkeys (Monkeys H and C). These monkeys were trained to sit in a primate chair and perform a center-out reaching task using a planar manipulandum with the hand contralateral to the implanted hemisphere. During each trial, the monkey started by moving cursor to central target. After variable waiting period, one of eight outer targets (equally spaced along a circle of 6-8 em radius) was presented. Monkeys C and differed in the task protocols:
Monkey C: Trained to wait for an auditory go cue during a delay period of 0.5-1.5 seconds while the target remained visible. Upon receiving the cue, the monkey had to move the cursor to the outer target within second and hold there for 0.5 seconds to receive a liquid reward.
Monkey H: No delay period; the monkey had to move the cursor to the outer target within second and hold there for 0.1 seconds.
For both monkeys, the trial restarted by returning the cursor to the central target. Endpoint positions of the manipulandum were recorded at 1 kHz, and task event timings were digitally logged. Hand velocity was computed as the derivative of hand position. The dataset includes 6 sessions for Monkey C and 5 sessions for Monkey H, considering only successful trials (an average of 307+221 trials per session, mean + s.d.)
Neural recordings
The dataset consists of neural recordings from two male Macaque monkeys. These recordings were obtained using 96-channel Utah electrode arrays implanted specific cortical regions.
Monkey C: Initially implanted in the right primary motor cortex (M1) and later received implants in the left MI and dorsal premotor cortex (PMd) (denoted CR and CL., respectively).
Monkey H: Implanted in area 2 of the primary somatosensory cortex of the left hemisphere.
Neural activity was recorded using Cerebus system (Blackrock Microsystems, Salt Lake City, UT)
sampling frequency of 30 kHz. The recorded signals underwent band-pass filtering (250-5000 Hz) and were converted to spike times based on threshold crossings. Spike sorting was performed using specialized software (Offline Sorter v3, Plexon, Inc, Dallas, TX) identify putative neurons.
Date
Monkey
Hemisphere
Trial
M1
PMd
SI
150313
Chewie
Right
1038
86
n/a
n/a
150309
Chewie
Right
1026
72
n/a
n/a
150629
Chewie
Right
179
49
n/a
n/a
150630
Chewie
Right
178
44
n/a
n/a
160929
Chewie
Left
208
74
114
n/a
161005
Chewie
Left
202
82
167
n/a
161006
Chewie
Left
209
63
192
n/a
161007
Chewie
Left
168
70
137
n/a
161014
Chewie
Left
740
88
190
n/a
161021
Chewie
Left
286
84
211
n/a
171201
Han
Left
292
n/a
n/a
70
171204
Han
Left
255
n/a
n/a
83
171207
Han
Left
245
n/a
72
Table 1: Datasets for the center-out reaching experiments.
14

---

Datasets
All the center-out reaching experiments using the open source data from: https:
org/ trggstaah/dattaee httpaaa/ddaaaaaatttbt1000.0.011. xdd xd2547dkt This data is released accompanying this (tte:icleee7311555 73155#datal We used all sessions from Monkey aaperlattss://efffeciciimtt http://dffeeccceeeeeerorrrraaaal wwwp/rrtggeeea../.ooo/oor..rreee
Chewie and Monkey Han. We chose these Monkeys because one session in Chewie is used by piVAE paper[32] and one session in Monkey Han is used by CEBRA paper [27]. Although the datasets come from same session in same Monkey, the temporal resolution is much higher for the datasets used by piVAE and CEBRA papers. The data Matlab format and we extract following information: igtDir (Target direction, radians for Monkey Chewie and degrees for Monkey Han), idx-goCue (The time go Cue one), vel(XY velocities), MI MI-spikes for both Chewie 2015 and Chewie 2016, and PMd-spikes only for Chewie 2016. The time bin is 30ms and we extract all the spikes after each go Cue. We extracted 40 bins for Monkey Chewie and 35 bins for Monkey Han, because most trial in Monkey Han has short acquisition window than 40 bins (afte go Cue). We smoothed the discrete spike count in the Matlab using a Gaussian kernel. The standard deviation 1.5 and kernel size is six standard deviations. We keep all the trials and neurons. The number of trials and neurons are shown in Table I Our NER is just a modification of the loss function used by CEBRA: https://gg g thub. com wwwpttiieooooooooooooooooooooooe
CEBRA The RNC loss could be downloaded from: https:: / gi thub. com/ wom/kaawweee wwwawwienhhaaaaaaa Rank N- Contrast The iterations is 20000, learning rate is 1c-4, and batch size 512. The temperature fixed to I for both NER and CEBRA. The output dimension of NER is fixed to 3. For CEBRA, we used output dimension of 3 for visualizing the latent dynamics and linear models decoding. We only used 16 dimensional embeddings for k-NN decoders. For the piVAE, we did not use the original version which is based on older version of Tensorflow. https://gi thub. cww wwwpzhddw/pp..o pi vae Instead, used the modified conv-pi- -i-VEE VAE that is already included into the CEBRA package. We fixed the random seed to 42, and using batch size of 200 and iterations of 300.
A.2.2 Curved hand movements experiments in M1 Fig 17
The MC_Maze dataset includes recordings from the primary motor and dorsal premotor cortices of a monkey performing reaches visual targets in virtual maze with an instructed delay. This dataset comprises 108 different task configurations, each varying in target positions, barrier numbers, and barrier positions. The monkey repeated each task configuration multiple times in random order, result- ing in 2,869 trials recorded in single session with 1 182 neurons and simultancous hand kinematics monitoring. This datasets could be downloaded from: /attps://dandd.. arcaivirrhiiv... org/dandi dandisst// 000128. https: //gttbb com /aadddatt//daddddoeooccoaaaaa 000128 In our works, we used the NLB2I package 22 to download the data from DANDI: / (Jupyter Notebook) is modified from: https: github. /o/ htt///lob/mmstrreee wwwpssttee/oootoooooooo http:/oteeooo/oeeeeeeeeoeoeoee /c_ aaaoss_maeeeae ipynb We used a time bin of 5ms (raw resolu- tion is mms) and Gaussian window of 50ms. For six straight movements, we only use version 0 and trial type of 13, 29, 17, 38, 6. 18. For the straight-curved and curved-curved movements, we keep all three versions of task (one straight and two curved). We removed the target angle during training. We used trial type of 13, 38, 18 for straight-curved and 37,1, 31,38, 34,18 for curved-curved movements. The iterations is 5000, learning rate is le-4, and batch size 512.
/ggttttttuubbu www. www.////walllaeettte.e.e Our code http://euraallaattteeeeeeeeeeeee
A.3 Decoders
We rotate the latent dynamics with reference to same target before decoding. We used orthogo- nal Procrustes from scipy for this purpose: https:///ocs wwwpsscccc//sopoeeooooooo ort//oc//cipy//////ooooo//././c/ httboo//ciip///ffeennceee reweeeneeeeeeeeeeeeeeeeeeeeeeeee generated/scipy.. www.ll...gga.. orthogonal
www.pporrs....ooo arthogonal__roccustees html We picked on target angle and ro- tate the whole 3D) latent dynamics using the computed orthogonal matrix. This rotation will not modify local detail or the relative positions of each reaching direction. For the linear decoders using linear and logistic regression, both are imported from "lincarmmodel" of sklearn. There is no parameter or hyperparameter for the linear regression model. For the logistic regression model, we used the following parameters: max_ max_iter=500, multi class=' multinomial k-nearest neighbors Regressor and Classifier, they are both imported from "neighbors" in sklearn. We used "GridSearchCV" in sklearn to searach the best "n neighbors" range from 2 to 1024.
Sooler='Ibffss For the
15

---

M1 Left Hemisphere
M1 in Right Hemisphere
a
2016-10-05
2016-10-07
2015-06-29
2015-06-30
NER
b
Cebra
C
PiVAE
Figure 9: NER reveals consistent and movements aligned latent dynamics MI for the remaining four sessions. Extra four sessions" latent dynamics at left and right hemisphere of Monkey C after rotating relative to target session in Fig 2016-10-14)
16

---

2015-03-13
PiVAE
UMAP
w/
UMAP
label
dPCA
PCA
Figure 10: Neural embeddings revealed by five other dimensionality reduction methods. Single trial (top) and trial averaged (bottom) latent dynamics revealed by five other dimensionality reduction methods.
160929
163005
151006
161007
161014
161021
130313
L50630
NER
Cebra
PiVAE
UMAP
w/ label
UMAP w/o
label
Figure 11: Latent dynamics without rotation. Trial-averaged latent dynamics revealed by five dimensionality reduction methods without rotation.
17

---

Monkey 2010-0z 1216-03-13
Monkey C 201 5-005-0--1 13-13
Monkey
Monkey
dit comporert
Iatestimakg compooeent
Ist midig COmP00191:
Let tine corpposa1s
200 utiuuaa componvent
Jec moaing corppceert
c50
100
6.75
23
26
Figure 12: Mixed single trial latent dynamics in PCA and time-stimulus components revealed by dPCA. Unlike other five dimensionality reduction methods, single trial latent dynamics revealed by principal component analysis (PCA) and demixed PCA (dPCA) is mixed and latency dynamics are only identifiable after averaging. In the trial- avergge latent dynamics, dPCA reveals three components (left, middle, right) at eight directions (different colors): time component aligned with go cue regardless of directions, separated stimulus component varied across time, and mixed component aligned with go cue and different for each direction.
Table 2: Statistical analysis and quantitative comparisons of the presented results. Figs. 3 and 6a
show the correlation coefficients between each pair of averaged latent dynamics. Here, "d." refers values the diagonal, "o." refers to values off the diagonal, "L" denotes the left panel, and "R" denotes the right panel. The 2nd to 4th columns display the mean and standard deviation, while the 5th 7th columns provide the t-statisties and p-values.
Fig.
NER
CEBRA
piVAE
NER vs CE
NER vs pi
CE vs pi
0.95, 0.03
0.92, 0.05
0.43, 0.25
7.9, 5.5e-10
2.99-19
14.8, 1.0e-18
5a-d
0.71, 0.1
-3.08, 0.6
-1.21, 1.4
16.8, 4.3e-08
2.5,2.5
-5.8, 2.8e-04
5a-o
0.64, 0.1
-3.22, 1.0
-2.05, 1.6
40.9, 1.7e-59
16.3, 1.7e-28
3.9e-
5b-d
0.76, 0.1
0.7
0.87.0.1
0.4.
7.2--30
-3.2, 1.1e-02
5b-o
0.70, 0.1
1.5
-1.05,
3.3e-19
1.44.6.4
8.9e-01
5c-d
0.0
0.74.000
0.67,
1.66,99c007
4.4, 1.8e-3
1.4, 1.9e-11
5c-o
0.1
0.71,0.1
0.23,0.4
19.2, 235-333
17.9,3.0e-31
12.6, 1.4e-21
5d-d
0.93,000
0.9
0.96, 0.0
0.5,
5.00--0
-3.1, 1.2e-02
5d-o
0.8
0.18.
0.13.0.4
2.0
16.5,8.0e-29
0.7,
5e-d
93,4.4 4.4
9.,
11.9
3.6--22
2.4, 4.0-020-
1.9.9.5e-02
5e-o
90, 5.9
7..9.0
43, 15.2
13.6, 1.7e-23
2.6
17.5, 1.2e-30
5f-d
92, 4.1
9.,
1.5
7.5e-.
-4.9, 8.3e-04
-..4,7.
5f-o
89, 5.8
16.3
15.9
3..1,3.26-33
5,4
2,5e-77
6a
0.87, 0.08
0.81,
0.56,
1.1e-18
29.3, 1.6e-30
2.8e22.
6bL-d
0.75, 0.0
-3.06, 0.4
-1.56, 1.1
35.2, 1.2c-12
7.2, 158
-4.8, 5.8e-04
6bL-o
0.0
-3.10. 0.3
2.788, 2.2
124.6, 5.7c-138
9.72,9.
1.le-O
6cL-d
0.78, 0.1
0.78. 0.1
0.84.
-0.0, 9.8e-01
-2.4, 3.3e-22
119,
6cL-o
0.1
-0.72,
-0.22, 0.8
1.8--6
19.8,
0.99
6bR-d
0.94, 0.0
0.75.0.0
0.58, 0.1
36.3, 834e-13
9.5, 1.3e-06
9.5.9.7e-04
6bR-o
0.93, 0.0
0.74, 0.0
0.21,
4.3e-133
2.63,2.
18.2,
6cR-d
0.93,
0.0
0.95,
-0.1, 9.3--1
2.4--
3.4e-3
6cR-o
0.92, 0.0
0.25, 0.5
0.18,
15.3,
17.2,
2.1,2.8e-01
7d-d
0.67,0.0
-4.59, 1.3
-0.38, 0.3
5.6. 3,0e-02
5,9, 2.8c-02
-5.5,
7d-o
0.56.0.1
4..79, 1.1
-0.75,
11.4, 9.0c-05
19.4, 6.7e-06
-8.3, -8.3,4.1e-00
7e-d
0.90.000
0.60.
0.73.0.1
18.5, 2.9c-03
3.1, 8.9e-02
-1.8, 2.2e-01
7e-o
0.77.
0.45.0.1
0.51,
5.9, 1.9e-03
3.3, 2.1e-02
-0.9, 4.1e-01
18

---

Explained directions in PMd and X-Y velocities
a
Direction classification accuracy (SA
100
* NER
Cebra
80
piVAE
60
UMAP w/ label
UMAP w/o label
X-vel
Y-vel
200
400
600
800
1000
1200
Time from (ms)
b
Correlation matrix among four direction tuning curves and X-Y velocities
M1
PMd
1.0
1.0
0.8
0.8
Cebra
0.52
Celora
0.38
0.6
0.5
UMAP wio label
0.96
0.66
0.90
0.4
UMAP we abel
0.82
0.62
0.84
X-vel
0.93
0.28
0.85
0.85
Xvve-
0.93
0.14
0.76
0.66
Yvel-
0.93
0.28
0.84
0.86
Yvell
0.93
0.12
0.76
0.65
NER
piVAE
UNAP label.
Xvel
Yual-
Cebra
DIVAE
UMAP wio label
Xvel
Koval
C
Explained peak classification accuracy by logistic regression
M1
PMd & S1
100
Explained variance (9)
90
Explained variance (96)
90
80
80
70
70
50
60
50
1
160929
161005
161000
161000
161014
161021
150313
150319
150629
150630
160929
161005
161000
161001
161014
161021
171201
171204
171207
1604
1611
Figure 13: Hand directions tunings and explained peak classification accuracy of hand directions in
MI, PMd, and S1. a Hand directions explained accuracy using logistic regression models trained on the latent dynamics revealed by five dimensionality reduction methods. Shaded areas are standard deviation over six sessions from PMd in the Monkey Notice the only NER reveal hand velocities dependent direction tuning curves that peak around 500 ms. b Correlation coefficients matrix between direction tuning curves and velocities M1 (left) and PMd (right). Explained variance on hand directions using logistic regression decoder.
19

---

a
2016-09-29
10-05
10-06
10-07 10-14 10-21
Cebra
b
PiVAE
Figure 14: Latent dynamics in PMd revealed by CEBRA and piVAE Single trial and trial averaged latent dynamics revealed by CEBRA and piVAE. All the figures are rotated with reference with the one session shown in Fig (2016-10- 10).
Logistic regression
k-nearest neighbors
NER dia: 94% off: 99%
CEBRA
piVAE die: 73% off: 40%
NER dis: 94% oft 92%
CEBRA 160 dia 95% offt 50%
PIVAE
dia: 92% off: 85%
des 97% off: 44%
2 TR300
1010210
PN
Figure 15: Direction decoding accuracy. Same date, cross date, and cross brain areas decoding of hand directions using linear regression (left) and k-nearest neighbors decoder trained on the latent dynamics revealed by three methods. Notice the range of color bars is 0- 100 for all six figures.
20

---

2017-12-01
12-04
12-07
Rotate with reference
to M1
Rotate with reference
to S1
b
Cebra
Rotate with reference
to S1
C
piVAE
Rotate with
reference to S1
Figure 16: Latent dynamics in S1 revealed by CEBRA and PiV. Top, single trial latent dynamics revealed by CEBRA are rotated with reference to the one session MI. Bottom, latent dynamis are rotated with reference to the first session in S1. b Similar to the bottom figure of a but using CEBRA dimensionality reduction method. Similar to c but using VAE dimensionality reduction method.
21

---

b
3 pairs of straight-curve move
a
8
3 pairs of curve-curve move
d
Figure 17: Distinct latent dynamics revealed by CEBRA. Similar to Fig but using the latent dynamics revealed by CEBRA. Ground truth motion trajectories at three directions for straight- curve movements. b Latency dynamics trained on three directions separately and combined. Notice the latent dynamics of straight hand movements (dots and black line) are either mixed (1st) with curved movements or squeezed (3rd). Latent dynamics trained by three directions combined are overlapped. Linear models predicted hand trajectories. The directions explained variance for hand velocities (directions) are 87% (94%), 83% (93%), 82% (94%), and 66% (86%). Notice all the values lower than NER which 92% (95%), 89% (96%), 90% (96%), and 81% (91 19),, especially for the combined directions. Ground truth motion trajectories at four directions for paired curve-curve movements. e Latency dynamics trained three directions separately and combined. Notice latent dynamics trained by three directions combined are overlapped. f Linear models predicted hand trajectories. The directions are manually assigned. The explained variance for hand velocities (directions) are 85% (93%), ), 81 % (92%), 85% (94%). and 61 % (80%). Notice all the values are lower than NER which are 95%) 89% (95%), 91 1% (96%), and 81 % (91% ) especially for the combined directions.
manually assigned. The
22

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The claims we made in the abstract and introduction are based on our extensive experiments over multiple dataset and benchmark comparisons of our method with six other methods.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: the last part Discussion, we mentioned two limitations of this work clearly
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full of assumptions and complete (and correct) proof?
23

---

Answer: [NA]
Justification:
There is no theoretical analysis. This work an application of contrastive
learning and deep regression Neuroscience. We do not make claim any theoretical contribution.
Guidelines:
The answer NA means that the paper does not include theoretical results, All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: Our works use the public dataset, and our code adds the loss function from one paper into the code of the other paper. Both codes are available online. There is not hyperparameter turnings. We mentioned the GPU, iterations, and batch size we used in the Appendix. We also upload our codes that generate all the Figures.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurlPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (c.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the partieular way they provide for reproducibilitty In the case of closed-source models, it may be that access to the model is limited in

---

some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification: Also the data and code already available online. We just combine two codes.
Guidelines:
The answer NA means that paper does not include experiments requiring code. Please see the NeurlPS code and data submission guidelines (https://nips.co
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
geweddei////////e/eeeeeeesssssee CodeSubmiee isi onPolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification:
We gave those details in the Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [NA]
Justification: We just show the raw, mean and median values without using any statistical metrics.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
25

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide the details of our computer that used to run those experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https: /t/prppupp.../. cetpuriss.ccppbbic//eiiiie scc//ubbic/tthiiccciddedl...i
Answer: [Yes]
Justification:
We used open-source neural spiking data from animals.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- cration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: The data we used came from neuron activities in the brains. Nothing is related to humans.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
26

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The codes have been released by others already.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [NA]
Justification:
All the data and codes are open source
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
27

---

If assets are released, the license, copyright information, and terms of use in the thcode has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
latasets.paperssit
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: Everything is open source already.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moode as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
Only authors do the experiments.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification:
We used open source data.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
28

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
29


Reviews:
[{"review_id": "Vx2haevG7F", "reviewer": "Reviewer_VFtX", "summary": "The authors propose a dimensionality reduction method, specifically to learn latent neural dynamics in a 3-dimensional space. The authors perform experiments that test the transfer of their model from one hemisphere to another, from one year to another, from one brain region to another, and how well dimensionality reduction methods separate straight and curved hand movements. Each experiment is atleast compared with CEBRA and Pi-VAE, which are both methods that also (potentially) have access to the target variables during training.", "questions": "1. Why is the generalization from one brain region to another in the same monkey with the exact same trained model important? And why should it work?\n2. I would also be interested as to why the authors believe the model performs relatively well in generalizing from one brain region to another. It is not entirely intuitive to me directly why a model trained on one region should produce latent embeddings for another region that, when rotated with the Procrustes method, obtains latent embeddings that are still linearly related to the task. My question arises because different brain regions (especially S1 compared to PMd) are entirely new neurons, with a new level of abstraction, which would intuitively require atleast some level of fine-tuning the model.\n3. Do you specifically use the CEBRA model that has access to targets during the self-supervised learning (CEBRA-behavior)? This is unclear in the text, and I could not find what type of CEBRA models the authors use given that in the original CEBRA paper [1] they propose both CEBRA time and CEBRA behavior. The same question arrises for the Pi-VAE, which is a method that can be used for both discrete and continuous labels. The proposed method in the paper uses continuous labels, but from L86 it almost sounds like the authors trained the Pi-VAE with discrete labels. If so, this comparison is unfair.\n4. Why is it important that X-Y velocity prediction accuracy is aligned with the direction tuning curve in PMd?\n5. Why are the CEBRA models 16 dimensions instead of 3 for the k-NN decoders?\n\n[1] Schneider, S., Lee, J. H., & Mathis, M. W. (2023). Learnable latent embeddings for joint behavioural and neural analysis. Nature, 617(7960), 360-368.", "limitations": "I appreciate the last sentences in the authors’ discussion that indicates where their model fails. I think it would be great if the authors include these results as well in the Appendix.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The authors perform a variety of interesting generalization tasks and generally find their model to work very well. Some of the experiments the authors perform are fairly novel in the specific type of generalization they are testing, e.g. wide vs straight reaching in Section 4.6, which is interesting. The paper is relatively clear, but does require some rewriting, especially grammatically and Section 4.6 requires a rewrite to clarify exactly how the authors are training the models.", "weaknesses": "Weaknesses:\n1. The authors do not compare their model with AutoLFADs [1] or a target-based model like CTRL-TNDM [2] that extends LFADs with labels and currently has the highest performance on the Neural Latents Benchmark [3]. Moreover, especially given that many of the tasks tested in the paper are related to generalization across a number of different variables, I believe it is important to compare the results with POYO [4], which specifically focuses on generalization.\n2. I think the authors should submit the exact code they used for these experiments, not just the code that is required to reproduce the figures once the code has been run. I am surprised by the low performance of both Pi-VAE and CEBRA and am curious how the authors specifically performed their experiments, e.g. did both methods also have access to the continuous labels like the authors’ model. The results do not instill confidence in me, even though the authors provide many specifics in their Appendix. Specifically, in the Neural Latent Benchmark [3], CEBRA obtains the second best R^2 performance on a completely held out test set of velocities. However, and I understand this is a different task, on inter-hemispheric generalization it obtains a -3 R^2 (Figure 4), for example. This would indicate that to some extent the model is essentially predicting the opposite of the signal. Given that the authors use the exact same neural feature encoder as CEBRA, I find it hard to believe that such a dramatic change comes from changing the loss function. Especially given that CEBRA-behavior would also contrastively make latent embeddings of the same velocity/direction more similar. I do believe there is potentially a benefit to the authors’ loss function, but I find the baseline results very surprising and believe a full submission of the code would help clear up this concern. Note: I am absolutely willing to be proven wrong.\n3. The authors only use one dataset to verify their results, I believe it is imperative for the authors to verify their results in another dataset as well.\n4. The authors fail to look at animal generalization, which I would say is probably more interesting even than generalization from one brain region to another.\n5. The figures in the paper are not aligned in scale (per row), and text sometimes overlaps (Figure 5). Generally the colorbars are unclear because they have different scales and are not colored. Please add one colorbar per row of figures. Moreover, Figures 9 and 10 in the Appendix have very small 3D plots that are hard to view.\n6. The authors do not provide a lot of motivation for their method and do not explain the intuitions or implications of their results. Why is their loss function better? Why does their model generalize better? Why does generalization between brain regions make sense? Etc.\n\n[1] Keshtkaran, M. R., Sedler, A. R., Chowdhury, R. H., Tandon, R., Basrai, D., Nguyen, S. L., ... & Pandarinath, C. (2022). A large-scale neural network training framework for generalized estimation of single-trial population dynamics. Nature Methods, 19(12), 1572-1577. \\\n[2] Kudryashova, N., Perich, M. G., Miller, L. E., & Hennig, M. H. (2023, March). Ctrl-TNDM: Decoding feedback-driven movement corrections from motor cortex neurons. In Computational and Systems Neuroscience (Cosyne) 2023. \\\n[3] https://eval.ai/web/challenges/challenge-page/1256/leaderboard/3183 \\\n[4] https://poyo-brain.github.io \n\nGrammatical and language weaknesses (some of these are personal preference, so feel free to ignore ones you deem unimportant):\n- L83: “Fourth, label-guided generative methods using VAE…” -> using VAEs\n- L86: “It reveals … but cannot align with movement trajectories, …” -> The revealed embeddings … but do not align well with movement trajectories.\n- L87: “… for learning robust… “ -> to learn robust\n- L98: “… Fig 1 gives the pipeline of this studies.” -> Fig 1 visualizes the pipeline used in this study.\n- L128: “… and the its less … “ -> and is less\n- L131: “Both models fails…” -> fail\n- L133: “A major limitation of both method…” -> methods\n- L134: “… for revealing movements aligned  …” -> movement-aligned\n- Figure 3 caption: “b Hand directions” -> hand direction\n- Figure 3 caption: “X-axis indicate…” -> indicates\n- Figure 4 caption: “… b, d, f are trained on the 80% … “ -> on 80%\n- Figure 4 caption: “Brighter color indicate” -> colors indicate/color indicates\n- L142: “A linear regression decoder could explain…” -> can\n- L145: “This tuning curve is highly correlate with…” -> correlated\n- L150: “Together, combined with with …” -> with\n- L154: “… and position in the 80% of training datasets …” -> in the 80% training data\n- L156: “We will performance the … “ -> perform\n- L159: “… we also tested the nonlinear decoder …” -> a nonlinear decoder\n- L165: “NER outperforms the CEBRA …” -> outperforms CEBRA\n- L165: “… over large margin …” -> by a large margin\n- L166: “A kNN decoder has no improvement over the NER, but …” -> The kNN decoder does not improve NER performance compared to the linear decoder, but for CEBRA and Pi-VAE, the kNN decoder improves within session data performance over NER.\n- L168: “However, this sacrificed …” -> this sacrifices\n- L174: “… NER revealed same …” -> the same\n- L177: “… using linear regression decoder … “ -> using a linear regression decoder\n- L180: “… NER is robust to different decoder …” -> robust to different decoders\n- L180: “… CEBRA and Pi-VAE change from failed state to outperform NER …” -> the performance of CEBRA and Pi-VAE changes from very low performance to outperforming NER on within-session decoding with the kNN decoder.\n- L198: “… when monkey perform …” -> the monkey performs/the monkeys perform\n- L205: “… NER revealed narrower latent dynamics for straight movements …” -> I am not sure what the authors mean by this sentence.\n- L210: “… CEBRA achieves compatible performance …” -> comparable performance\n- Figure 6 caption: “The reference target is same sessions used in …” -> is the same session as used in\n- Figure 6 caption: “… is first session in S1.” -> is the first session\n- Figure 7 caption: “The cartoon from [18].” -> Subfigure a is reproduced from [18].", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the detailed and constructive feedback from R#1. We thank R#1 for recognizing the contribution of our work. R#1's concern about the technical flaws in evaluating our models against CEBRA and pi-VAE is valid and deserves detailed explanations.\n\n**Strengths** Generalization across two different tasks \\\nThanks! We need to emphasize that NER is a **dimensionality reduction method** that reveals behavior-aligned latent dynamics in 3D. We think it is unfair to compare NER with POYO or other methods instead of dimensionality reduction methods on generalization because POYO has a 128D latent space.\n\n**Weaknesses1** Compare with more models\\\nWe compared NER with AutoLFADS using the same datasets (Rebuttal Figs. 2, 5). The performance of AutoLFADS is much worse than NER. We could not test CTRL-TNDM as there are no public codes. POYO is not a dimensionality reduction method (128D) and requires several days of training using eight A50 GPUs.\n\n**W2** Codes, low performance and -3 R2 of CEBRA\\\nSee the general rebuttal. We thank R#1 for raising this question about the -3 R², which is indeed confusing.\n\n**W3** More datasets\\\nWe validated NER on the rat's hippocampus data that has been used by the CEBRA and pi-VAE papers (Rebuttal Fig. 5). The performance of NER is similar to CEBRA but only requires 2D, and both are better than pi-VAE and AutoLFADS.\n\n**W4** Animal generalization\\\nWe analyzed and uploaded the datasets from Monkey Mihili during the initial submission <Fig2_NER/Cebra/piVAE_embeddings_align>. We did not report this data due to the nine-page limit. We are glad R#1 raised this question so that we could talk about animal generalization (Rebuttal Fig. 6), which we think is also very important.\n\n**W5** Figure alignment and colorbar\\\nWe believe R#1 is talking about the different scales we used in Fig. 4a-b, 5b, and 6d. We agree with R#1 and will use the same colored scale for all panels (Rebuttal Fig. 6) and show only one colorbar on the right side of each row. We will enlarge the 2D/3D latent dynamics shown in Figs. 9 and 10.\n\n**W6** Why our loss is better\\\nSee the general rebuttal and rebuttal to R#2.\n\n**W6** Why NER generalize better\\\nA well-trained monkey performs almost the same stereotyped hand movements, and only NER could align (almost perfectly) the latent dynamics with the movement trajectories. The explained variance of velocity and position in NER is >80% for all 19 sessions (Fig. 3c, d). It outperformed all other four models in any session.\n\n**W6** Why generalization between brain regions make sense\\\nSee the general rebuttal.\n\n**W7** Grammar and language\\\nWe sincerely appreciate this reviewer for carefully examining our writing in the main text and figure captions. We will include those suggestions in our revised paper.\n\n**Questions 1** Why the generalization is important and works\\\nSee the general rebuttal.\n\n**Q2** Why generalization works\\\nSee the general rebuttal. Note that NER/CEBRA models are trained on each session or dataset separately.\n\n**Q3** Whether the comparison is fair\\\nFor CEBRA, we never used CEBRA-time and always used velocities and directions as targets or labels. In our uploaded codes like <Batch_M1_0510>, it is used for both CEBRA and NER (identical data preparation and inputs). The only difference is the \"cebra\" folder, which could be the original CEBRA downloaded on 2024-01-10 or our NER_0517.\nFor pi-VAE, in our uploaded <Batch_piVAE_0518>, \"dim_u = 3\" indicates the labels. They are either location+0+1 for rat's hippocampus data (01 for left and 10 for right) or X+Y+direction for monkey's reaching data. \"continuous_index\" includes both continuous velocities and directions. Notice that directions also belong to continuous labels because their values (i.e., angles) can be ranked and pairwise distances can be calculated. We apologize for this confusion and will include this critical information in the revised paper.\n\n**Q4** Velocity prediction aligns with direction tuning in PMd\\\nA model should not classify the movement direction when there is no hand movement. However, CEBRA achieves over 80% classification accuracy when there are no movements (Fig. 3b and 12a). Neurons in the PMd are tuned to reach direction (Chandrasekaran et al., 2017, *Nature Comm.*; Glaser et al., 2018, *Nature Comm.*). After the \"go\" cue, they increased their responses, reached a peak at 500 ms, and decreased (Jerjian et al., 2020, *Elife*). The X-Y velocity prediction accuracy should be aligned with the X-Y velocity and neural responses. This is exactly what we observed in NER and pi-VAE. Note that the pi-VAE paper observed the same X-Y velocity correlated prediction accuracy using the same datasets (Fig. 3d, e in pi-VAE).\n\n**Q5** CEBRA 16D for k-NN\\\nA comparison of NER 3D with CEBRA 16D is unfair to NER. We chose 16D for k-NN because this is the exact setting used in the CEBRA paper (Fig. 3h). CEBRA has >0.9 R² for simple rat left and right running tasks under 3D (Fig. 1e in CEBRA). However, it has a much lower R² in eight-direction hand movement tasks within 8D (Supplementary Fig. 8f in CEBRA). With the nonlinear k-NN decoder, the decoding accuracy of CEBRA 16D is no better than NER 3D for the same session data but is much worse than NER 3D for cross-session data (diagonal vs. non-diagonal; Fig. 4b, d, f, and 5b, c). We will explain this in the revised paper.\n\n**Limitations** our model fails\\\nOur model fails to reveal identifiable and interpretable 3D latent dynamics when there are more than 20 conditions of movement.\nHowever, it is far better than CEBRA. Under three pairs of straight-curve movements (six conditions), the 3D latent dynamics are well separated in NER but are collapsed in CEBRA (Fig. 7f vs. Fig. 16b). The difference is even stronger under three pairs of curve-curve movements (Fig. 7h vs. Fig. 16e). Again, both models have full access to movement labels (velocity and direction).\\\n**Limitations** include results\\\nAgree and will do that.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0KRdaCGhGg", "reviewer": "Reviewer_f3GE", "summary": "The paper introduces a novel dimensionality reduction method called Neural Embedding Ranks (NER) for aligning neural dynamics with movement in brain-computer interfaces. NER uses a ranking loss to embed neural dynamics into a 3D latent space that aligns with continuous movement parameters. The authors apply NER to neural recordings from primary motor cortex (M1), dorsal premotor cortex (PMd), and primary somatosensory cortex (S1) in monkeys performing reaching tasks, comparing it with six other dimensionality reduction techniques.The paper finds that NER out-performs all other methods to align latent dynamics with both hand position and direction.", "questions": "1. Why is the Rank and Contrast loss effective in this task? Specifically, how does it differ from other contrastive-loss-based methods like CEBRA?\n2. I believe the paper would benefit if a more detailed description were given to the method section, especially a description of RNC and how your method adapted it and could be different from it.", "limitations": "The limitations are adequately addressed.", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 2, "contribution": 3, "strengths": "1. The experimental design is comprehensive, covering multiple brain areas (M1, PMd, S1), multiple monkeys, and a variety of task conditions including curved reaching movements, and the statistical rigor is evident in the consistent performance across multiple sessions and conditions.\n\n2. The benchmarking against six other dimensionality reduction techniques provides a thorough evaluation of NER's performance.\n\n3. Extensive figures and graphs were used to illustrate the statistical results of the paper, most of which were informative and well-designed.\n\n4. The effectiveness of NER is significant and proven to exceed the performance of multiple other methods under different tasks.", "weaknesses": "1. The explanation of the method is too limited. Only a very small portion of the paper is dedicated to the Model/Method section, which makes it difficult to understand the fundamental algorithm of NER. \n\n2. It is also reasonable to suspect the main method of the paper is the application of the existing method RNC, which limits the contribution of the paper.\n\n3. It is empirically evident that the method works well in the setting of the paper, but there is limited discussion on what could be the cause of such an improvement over other methods (such as CEBRA), which by the formulation of the loss function, is very similar in nature.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate R#2 for recognizing the contribution and strength of our work.\n\n**Weaknesses1** Limited explanation of method\\\n Agree. We will add a detailed explanation (see below) later.\n\n**W2** Application of existing RNC method\\\nWe came up with the idea of modifying the loss in the CEBRA paper (2023 May) using regression before the publication of the Rank and Contrast (RNC) paper (Zha et al., NeurIPS, 2023 Nov). *In the neuroscience field, ramped neural responses to continuously increasing sensory features are widely observed (as mentioned in the Introduction). It is unnatural to classify sound levels, light intensities, inter-eye distances, etc*. Before modifying it, we searched \"regression AND contrastive learning\" and found \"Supervised Contrastive Regression\" (SupCR, ICLR Rejection), which is the same paper as RNC. So we borrowed their loss. The CEBRA paper is conceptually similar to supervised contrastive learning for classification tasks (SuperCon, Khosla et al., NeurIPS, 2020), and SuperCon has been cited over 4400 times. In contrast, RNC+SupCR has only 30 citations now (9 in 2023), so it is almost impossible for neuroscientists to hear the RNC paper unless they purposely search for it. Even if we applied RNC loss in NER, there are significant differences between NER and the RNC paper: RNC encodes features into a 512D space for prediction, whereas NER reduces neural dynamics to 3D latent dynamics. We believe **NER is a neural-inspired method for neural-behavioral analysis**.\n\n**W3** Loss in CEBRA and NER\\\nTwo losses are indeed very similar! We briefly described two models in the global rebuttal with an intuitive diagram of NER vs. CEBRA in Rebuttal Fig. 7.\\\nWe will use $x$ as high-dimensional neural dynamics, $f$ as feature extractor, and $v=f(x)$ as low-dimensional neural embeddings. The batch size $N$ is 512 and temperature $\\tau$ is fixed to 1. Unlike common contrastive learning methods such as SuperCon and RNC, data augmentation in CEBRA and NER is achieved by selecting the distribution of embeddings whose labels fall within the $\\Delta t$ (e.g., 10 ms) offset of the anchor's label.\\\nIn each iteration, CEBRA gets one batch of anchor $v_i$, one batch of embedding $v_j$ that will positively contrast with anchor, and a third batch of randomly sampled embeddings $v_n$ that will negatively contrast with anchor. Anchor loss $l$ in CEBRA is:\\\n$$l_{CEBRA}^{(i)} = \\-\\log \\frac{\\exp(\\text{sim}(v_i, v_j)/\\tau)}{\\sum_{n=1}^{N} \\exp(\\text{sim}(v_i, v_n)/\\tau)}$$\nwhere ${sim}(., .)$ is the similarity between two neural embeddings.\\\nIn each iteration, NER gets one batch of anchor $v_i$, one batch of augmented embedding $v_j, v_k$ that will either positively or negatively contrast with an anchor, and a third batch of labels $y$ for $v_i$ and $v_j, v_k$ . Anchor loss $l$ in NER is:\\\n$$l_{NER}^{(i)} = \\frac{1}{2N-1} \\sum_{j=1, j \\neq i}^{2N} -\\log \\frac{\\exp(\\text{sim}(v_i, v_j)/\\tau)}{\\sum_{v_k \\in S_{i,j}} \\exp(\\text{sim}(v_i, v_k)/\\tau)}$$\nThere are two differences from CEBRA. First, batches of $v_i$ and $v_j, v_k$ will merge and labels $y$ will be duplicated, resulting in a batch of $2N$ for both embeddings and labels. The purpose here is to ensure that each anchor and its augmented embedding exist in the same batch. Second, we introduce $S_{i,j}:=\\\\{v_k \\mid k \\neq i, d(\\{y_i}, \\{y_k}) \\geq d(\\{y_i}, \\{y_j})\\\\}$ to denote the set of embeddings $v_k$ that are of higher ranks than $v_j$ in terms of label distance with respect to $v_i$, where $d(·, ·)$ is the distance measure between two labels. Intuitively, for an anchor $v_i$, any other embedding $v_j$ in the batch is positively contrasted with it, enforcing the similarity between $v_i$ and $v_j$ to be larger than that of $v_i$ and any other embedding $v_k$ in the batch if the label distance between $y_i$ and $y_k$ is larger than that of $y_i$ and $y_j$. Minimizing $l_{NER}^{(i)}$ will align the order of embeddings with their corresponding orders in the labels with respect to anchor $v_i$.\n\n**Questions1** Why RNC loss is effective\\\nNER with RNC loss aims to solve the **high dimensionality and class imbalance** issues in CEBRA.\n 1.  CEBRA is an extension and generalization of the standard InfoNCE loss for classification tasks. By treating continuous labels as many discrete classes, it faces difficulty in representing them with low-dimensional latent spaces. To compensate for this drawback, one solution is to increase its dimensionality. For example, in simple tasks like rats running on a rail, CEBRA can achieve behavior-aligned latent dynamics in 3D, but these simple tasks can be represented in 2D using NER. For relatively complex tasks like reaching in 8-directions, CEBRA requires 4D for representation and 16D for decoding, whereas NER can handle these tasks in 3D. For complex behaviors like curved reaching, CEBRA struggles in 3D, but NER works well.\n 2. Representation learning of CEBRA in 16D is still worse than NER in 3D. In each batch, a very low percentage (say 5%) of embeddings come from infrequent classes, such as large velocities. In CEBRA, only a single augmented embedding is positively contrasted with its anchor, so only 5% of $l_{CEBRA}$ have access to 5% infrequent embeddings. In NER, all the *2N-1* embeddings can contrast with an anchor, so 100% of $l_{NER}$ have access to 5% infrequent embeddings. Through the RNC loss, NER resolves the class imbalance issue and can effectively represent infrequent classes.\n\n**Q2** RNC description\\\nWe will add our answers to the revised paper.\n\n**Q2** How we adapted RNC\\\nOne just needs to replace the embeddings that will be negatively contrasted with the anchors using the labels of the anchors, and replace the InfoNCE loss with the RNC loss.\n\n**Q2** How the NER paper differs from the RNC paper\n1. NER is for dimensional reduction, while RNC is for prediction.\n2. NER operates in 3D, whereas RNC operates in 512D.\n3. For augmentation, NER uses existing nearby data, while RNC creates new data.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0yE4MvbI1d", "reviewer": "Reviewer_rR7L", "summary": "The authors propose a novel method for reducing the dimensionality of neural dynamics and aligning them with movements. This method is compared with six existing methods across different brain regions. Experiments demonstrate that all movement parameters can be accurately decoded from neural dynamics using this approach.", "questions": "- Could you please provide a statistical analysis of your results? \n- Could you also provide some quantitative comparisons?", "limitations": "The authors have mentioned limitations of their work and I do not perceive any specific negative societal impacts that would need to be addressed.", "rating": 6, "confidence": 1, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "The paper offers an extensive comparison to prior works on the dimensionality reduction task. It is important to note that baselines represent diverse techniques in the field. Additionally, all code used in the study is provided, ensuring transparency and reproducibility. The paper also includes numerous visualizations and is well-structured.", "weaknesses": "The methodology section is unclear, with equations lacking necessary explanations and annotations. Additionally, there appear to be typographical errors, such as the inconsistent notation where temperature $\\tau$ is sometimes mistakenly written as $\\textit{r}$. Moreover, all the experiments lack an assessment of statistical significance, which is crucial for validating the results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the constructive feedback from R#3. We thank R#3 for recognizing the contribution of our work.\n\n**Strengths** Extensive comparison to prior works\\\nThanks!\n\n**Weaknesses1** Unclear methodology\\\nWe briefly described our method in the global rebuttal with an intuitive diagram of our model NER vs. CEBRA in Rebuttal Fig. 7. A detailed explanation is provided in the Rebuttal to R#2.\n\n**W2** Typographical errors in formulas\\\nWe apologize for this mistake and have fixed it now (Rebuttal to R#2).\n\n**W3** Lack of statistical significance\\\nThis is a great suggestion! From the Rebuttal Table, we could validate that:\n 1. NER reveals significantly more consistent 3D latent dynamics than CEBRA and pi-VAE.\n 2. NER always significantly outperforms CERBA and pi-VAE when using a linear decoder for both within- and cross-session decoding.\n 3. NER always significantly outperforms CERBA and pi-VAE when using a k-NN decoder for cross-session decoding.\n\n**Rebuttal Table**: statistical analysis and quantitative comparisons of presented results. Figs. 2 and 5a measure the correlation coefficient between each pair of averaged latent dynamics. $\\mu$, mean; $\\sigma$, standard deviation; *t*, t-statistic. A larger value indicates a larger difference between two groups; **N**, NER; **C**, CEBRA; **pi**, pi-VAE; *F*, F-statistic. A higher F indicates a greater disparity between groups; *p*, p-value; Friedman test is a non-parametric statistical test that is similar to the ANOVA but does not assume the data is normally distributed. *s*, overall differences among the three models. d., values on the diagonal; o., values off the diagonal; L, left panel; R, right panel.\n\n| Fig. | NER $\\mu$ $\\sigma$ | CEBRA $\\mu$ $\\sigma$ | piVAE $\\mu$ $\\sigma$ | N vs C $t$ $p$ | N vs pi $t$ $p$ | C vs pi $t$ $p$ | ANOVA $F$ $p$ | Friedman $s$ $p$ |\n|-----------|---------------------|----------------------|----------------------|----------------|-----------------|-----------------|---------------|------------------|\n| 2     | 0.95, 0.03 | 0.92, 0.05 | 0.43, 0.25 | 7.9, 5.5e-10 | 15.3, 2.9e-19 | 14.8, 1.0e-18 | 226.0, 2.2e-35 | 86.2, 1.9e-19 |\n| 4a-d. | 0.71, 0.1 | -3.08, 0.6 | -1.21, 1.4 | 16.8, 4.3e-08 | 4.1, 2.5e-03 | -5.8, 2.8e-04 | 57.9, 1.4e-08 | 18.2, 1.1e-04 |\n| 4a-o. | 0.64, 0.1 | -3.22, 1.0 | -2.05, 1.6 | 40.9, 1.7e-59 | 16.3, 1.7e-28 | -9.5, 3.9e-15 | 458.7, 5.8e-71 | 156.4, 1.1e-34 |\n| 4b-d. | 0.76, 0.1 | 0.75, 0.1 | 0.87, 0.1 | 0.4, 7.0e-01 | -3.5, 7.2e-03 | -3.2, 1.1e-02 | 9.5, 1.5e-03 | 9.8, 7.4e-03 |\n| 4b-o. | 0.70, 0.1 | -1.07, 1.5 | -1.05, 1.1 | 11.5, 3.3e-19 | 15.4, 6.4e-27 | -0.1, 8.9e-01 | 95.2, 7.6e-29 | 135.2, 4.4e-30 |\n| 4c-d. | 0.94, 0.0 | 0.74, 0.0 | 0.67, 0.2 | 11.6, 9.9e-07 | 4.4, 1.8e-03 | 1.4, 1.9e-01 | 17.4, 6.3e-05 | 12.6, 1.8e-03 |\n| 4c-o. | 0.88, 0.1 | 0.71, 0.1 | 0.23, 0.4 | 19.2, 2.5e-33 | 17.9, 3.0e-31 | 12.6, 1.4e-21 | 240.0, 2.9e-51 | 166.4, 7.3e-37 |\n| 4d-d. | 0.93, 0.0 | 0.93, 0.0 | 0.96, 0.0 | 0.5, 6.6e-01 | -3.7, 5.0e-03 | -3.1, 1.2e-02 | 9.5, 1.6e-03 | 8.6, 1.4e-02 |\n| 4d-o. | 0.89, 0.0 | 0.18, 0.5 | 0.13, 0.4 | 13.1, 2.0e-22 | 16.5, 8.0e-29 | 0.7, 4.8e-01 | 115.7, 6.3e-33 | 135.8, 3.2e-30 |\n| 4e-d. | 93, 4.4 | 90, 4.6 | 81, 11.9 | 2.5, 3.6e-02 | 2.4, 4.0e-02 | 1.9, 9.5e-02 | 4.7, 2.3e-02 | 3.2, 2.0e-01 |\n| 4e-o. | 90, 5.9 | 79, 9.0 | 43, 15.2 | 13.6, 1.7e-23 | 26.0, 2.6e-43 | 17.5, 1.2e-30 | 445.6, 5.0e-70 | 160.2, 1.7e-35 |\n| 4f-d. | 92, 4.1 | 93, 4.2 | 98, 1.5 | -0.3, 7.5e-01 | -4.9, 8.3e-04 | -3.4, 7.5e-03 | 10.1, 1.1e-03 | 11.5, 3.1e-03 |\n| 4f-o. | 89, 5.8 | 57, 16.3 | 44, 15.9 | 19.1, 3.2e-33 | 25.7, 5.4e-43 | 5.6, 2.5e-07 | 297.7, 1.7e-57 | 138.8, 7.2e-31 |\n| 5a     | 0.87, 0.08 | 0.81, 0.10 | 0.56, 0.09 | 14.8, 1.1e-18 | 29.3, 1.6e-30 | 20.7, 2.8e-24 | 590.5, 1.0e-51 | 90.0, 2.9e-20 |\n| 5bL-d. | 0.75, 0.0 | -3.06, 0.4 | -1.56, 1.1 | 35.2, 1.2e-12 | 7.2, 1.8e-05 | -4.8, 5.8e-04 | 103.2, 6.6e-12 | 22.2, 1.5e-05 |\n| 5bL-o. | 0.74, 0.0 | -3.10, 0.3 | -2.78, 2.2 | 124.6, 5.7e-138 | 18.2, 9.7e-38 | -1.6, 1.1e-01 | 355.4, 2.3e-75 | 205.3, 2.6e-45 |\n| 5bR-d. | 0.78, 0.1 | 0.78, 0.1 | 0.84, 0.1 | -0.0, 9.8e-01 | -2.4, 3.3e-02 | -1.9, 7.7e-02 | 3.5, 4.8e-02 | 6.2, 4.6e-02 |\n| 5bR-o. | 0.76, 0.1 | -0.72, 1.0 | -0.62, 0.8 | 17.7, 1.8e-36 | 19.8, 3.5e-41 | -0.9, 3.7e-01 | 177.6, 1.8e-49 | 192.2, 1.9e-42 |\n| 5cL-d. | 0.94, 0.0 | 0.75, 0.0 | 0.58, 0.1 | 36.3, 8.4e-13 | 9.5, 1.3e-06 | 4.5, 9.7e-04 | 67.2, 4.3e-10 | 22.2, 1.5e-05 |\n| 5cL-o. | 0.93, 0.0 | 0.74, 0.0 | 0.21, 0.3 | 77.2, 4.3e-111 | 24.3, 2.6e-50 | 18.2, 1.1e-37 | 482.5, 1.5e-88 | 262.0, 1.3e-57 |\n| 5cR-d. | 0.93, 0.0 | 0.94, 0.0 | 0.95, 0.0 | -0.1, 9.3e-01 | -1.2, 2.4e-01 | -1.0, 3.4e-01 | 1.0, 3.9e-01 | 4.7, 9.7e-02 |\n| 5cR-o. | 0.92, 0.0 | 0.25, 0.5 | 0.18, 0.5 | 15.3, 5.6e-31 | 17.2, 2.6e-35 | 1.1, 2.8e-01 | 140.5, 3.4e-42 | 195.1, 4.3e-43 |\n| 6d-d. | 0.67, 0.0 | -4.59, 1.3 | -0.38, 0.3 | 5.6, 3.0e-02 | 5.9, 2.8e-02 | -5.5, 3.2e-02 | 31.1, 3.6e-03 | 6.0, 5.0e-02 |\n| 6d-o. | 0.56, 0.1 | -4.79, 1.1 | -0.75, 0.1 | 11.4, 9.0e-05 | 19.4, 6.7e-06 | -8.3, 4.1e-04 | 101.3, 2.3e-07 | 12.0, 2.5e-03 |\n| 6e-d. | 0.90, 0.0 | 0.60, 0.0 | 0.73, 0.1 | 18.5, 2.9e-03 | 3.1, 8.9e-02 | -1.8, 2.2e-01 | 16.1, 1.2e-02 | 4.7, 9.7e-02 |\n| 6e-o. | 0.77, 0.1 | 0.45, 0.1 | 0.51, 0.1 | 5.9, 1.9e-03 | 3.3, 2.1e-02 | -0.9, 4.1e-01 | 12.5, 1.9e-03 | 9.0, 1.1e-02 |\n\n**Q1** Statistical analysis\\\nSee answers to W3.\n\n**Q2** Quantitative comparisons\\\nIn the above Rebuttal Table, we provide the mean and standard deviation of all comparisons we have made for Figs. 2, 4, 5, and 6. The explained variance of Fig. 3a is in the caption, and the correlation coefficient of the six curves shown in Fig. 3b is in Fig. 12b. Fig. 3c, d shows the explained R² among 19 sessions for 6 models. The explained R² between NER and CEBRA in the second motor task is shown in the caption of Fig. 16.\n\nPlease let us know if you find we missed quantification for any figures or claims, or if there are other useful quantitative methods you suggest. We would be more than happy to add them.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We appreciate the Area Chair for handling our manuscript and the constructive feedback from Reviewer VFtX, f3GE, and rR7L (R#1, R#2, and R#3). We were encouraged that all three reviewers recognized our paper's contribution (3/3/3). We hope our general rebuttal letter, along with the attached 7 Rebuttal Figures and three individual rebuttal letters, addresses their concerns about the soundness (2/4/2) and presentation (2/2/3) of our paper.

1. R#1 questioned whether we fairly benchmarked our model, NER, against CEBRA (Schneider, Lee, and Mathis, *Nature*, 2023) and pi-VAE (Zhou and Wei, *NeurIPS*, 2020), specifically regarding the potential limited access to target labels when training CEBRA and pi-VAE. **We addressed this concern in four independent ways:**
   1. We evaluated 20 sessions in our submitted paper, whereas the pi-VAE paper tested only one session (PMd 20161006), which we included in our paper. This allowed for a direct comparison of the latent dynamics shown in the pi-VAE paper with NER. Our implementation of pi-VAE (Fig. 13) showed similar results to the pi-VAE paper (Fig. 3f-i). However, the latent dynamics revealed by NER are much clearer than those of pi-VAE in both single trial (Rebuttal Fig. 1) and trial-averaged (Fig. 5a) analyses.
   2. We conducted an additional experiment comparing the latent dynamics generated by NER with those in the CEBRA paper, using the same datasets (Rebuttal Fig. 2). NER outperforms CEBRA in both 2D and 3D latent spaces.
   3. We performed ablation studies by training CEBRA without continuous velocity labels (Rebuttal Fig. 3). The latent dynamics failed to represent movement trajectories using only direction labels.
   4. We uploaded our training codes and data for verification.

2. R#1 asked why CEBRA generated an R² of -3. Negative R² values occurred only when using **linear** decoders to predict **velocities** (Fig. 4a). This is because velocity changes much faster than position (Rebuttal Fig. 4a), and CEBRA fails to capture the large amplitudes of velocities that occur infrequently (Rebuttal Fig. 4b). As a result, large negative velocities are decoded as small positive velocities and vice versa (Rebuttal Fig. 4c). Consequently, the decoded velocities are smaller than the actual velocities, leading to decoded locations being smaller than the actual positions (Rebuttal Fig. 4d).

3. R#1 asked why generalization works. Animal and brain area generalization is a trending topic in the neuroscience field, as evidenced by recent studies in monkeys (Safaie et al, *Nature*, 2023) and mice (Schneider 2023; Ehret et al, *Nature Neuro.*, 2024). Generalization requires:
    1. For animal generalization, the same species of animals need to perform similar behaviors. It will not work between mice vs monkeys (Safaie 2023), and straight vs curve-reaching (Fig. 7). 
    2. The examined brain areas have similar roles during behaviors. It will not work well between cortex vs striatum (Safaie 2023), and M1/PMd vs S1 (Fig. 6). 
    3. Latent dynamics instead of population neural dynamics. It fails when directly using the neural dynamics, as demonstrated by this brain area generalization study (Gallego et al, *Nature Neuro.*, 2020).
    4. Latent dynamics are informative about behaviors. A low-performance dimensionality reduction method requires a high-dimensional latent space with a larger representational capacity. For example, 10D for Safaie 2023 and Ehret 2024, and 16D for Schneider 2023.
    5. Latent dynamics are linearly aligned with each other.
    6. Decoders trained on 80% of the latent dynamics from one session are used to decode behavior from the remaining 20% holdout latent dynamics in another session. Nonlinear decoders like kNN (Schneider 2023), SVM (Ehret 2024), and LSTM (Safaie 2023) are usually applied.\
**Together, 3D latent dynamics with a linear decoder is the most challenging setup for generalization and is the most straightforward way to benchmark dimensionality reduction methods. Only NER works (off-diagonal in Fig 4, 5).**

4. R#1 asked us to examine animal generalization (Rebuttal Fig. 6). **NER reveals almost identical 3D latent dynamics in a newly added Monkey M compared to the previous Monkey C**. Using the same dataset, NER performs much better than Safaie et al., 2023 regarding the aligned latent dynamics. Animal generalization only fails in one session (140218). Interestingly, brain hemisphere generalization (Fig. 4a) is stronger than animal generalization (0.64 vs 0.43) but weaker than brain area generalization within the same hemisphere (0.64 vs 0.74, Fig. 5b).

5. R#2, R#3, and R#1 asked us to explain the difference between CEBRA and NER, and our motivation (Rebuttal Fig. 7). In brief, CEBRA considers each embedding in a batch (say 3) as a discrete class. For an anchor, it contrasts with its augmented embedding as a positive pair and 3 randomly sampled embeddings as negative pairs. NER ranks 6 embeddings according to their continuous labels. Then it contrasts an anchor with its augmented or 1st embedding as a positive pair and the remaining 4 embeddings as negative pairs. NER does not stop here; it continues by contrasting the 2nd embedding as a positive pair and the remaining 3 embeddings as negative pairs. This process continues until all the embeddings have been positively contrasted with the anchor. *NER learns a regression-aware representation that orders all embeddings in a batch.*\
**We are motivated by the fact that CEBRA treats continuous labels as many discrete classes, which cannot be well separated in low-dimensional space. These classes are also highly imbalanced, with many more near-zero classes (Rebuttal Fig. 4a, b). NER aims to solve the high dimensionality and class imbalance issues present in CEBRA.**

6. R#3 asked us to provide statistical analysis. In brief, the latent dynamics revealed by NER are **significantly more consistent** than those by CEBRA for M1 (t=7.9, p=5.5e-10) and PMd (t=14.8, p=1.1e-18).

Meta-review: The paper address the problem of embedding neural dynamics into low-dimensional representation that is continuous in the embedded space in a way that is aligned with the movement that ensued from measured neural dynamics. The approach is a way to train CEBRA (Schneider, Lee, and Mathis, Nature, 2023) by minimizing the Rank and Contrast loss (Zha et al, NeurIPS 2023). Since this is a direct combination of two previously published techniques the paper does not dwell at the technical part of any of the combined methods and instead focuses on an extensive validation of performance of the method.

All reviewers have agreed, some after the rebuttal, that the experiments are convincing and support the statements made in the paper. This is the strongest point of the paper and its main contribution.

However, all reviewers have also pointed out that the paper a) lacks description of the method and b) the writing is not up to the standards.

b) The writing can and should be improved before the paper is ready for a publication. At least the corrections suggested by reviewer VFtX, who has done extensive copy-editing work, should be taken into account. However, there are many grammar issues that this reviewer has missed.


a) I concur that description of the method is virtually missing: the authors indicate in the main body of the paper that details are given in the Appendix, but the method is not described there either. Apparently, combination of two previously published methods may not require more than the 8 lines and 3 equations devoted in the paper to explaining the approach. However, it must be clearly stated that this is a mere combination and the reader can get all details from the CEBRA and RnC papers. Currently, the paper in terms of method description is not self-contained and thus is not reproducible. It is more of an advertisement of what a combination of CEBRA and RnC could produce without clear explanation of how to combine them.

If the paper is properly re-edited and writing is corrected, if the method section should clearly state that this is an application of unmodified CEBRA and RnC, with any deviations carefully spelled out in the appendix, then it would provide a reproducible contribution to the conference.

Conditioned on the above changes I recommend acceptance as a poster.
Decision: Accept (poster)