Title: Neural Embeddings Rank: Aligning 3D latent dynamics with movements
Venue: NeurIPS 2024
Accepted: yes
Keywords: Dimensionality reduction

Latent dynamics

Brain-machine interfaces

Neural decoding

Contrastive learning
Abstract: Aligning neural dynamics with movements is a fundamental goal in neuroscience and brain-machine interfaces. However, there is still a lack of dimensionality reduction methods that can effectively align low-dimensional latent dynamics with movements. To address this gap, we propose Neural Embeddings Rank (NER), a technique that embeds neural dynamics into a 3D latent space and contrasts the embeddings based on movement ranks. NER learns to regress continuous representations of neural dynamics (i.e., embeddings) on continuous movements. We apply NER and six other dimensionality reduction techniques to neurons in the primary motor cortex (M1), dorsal premotor cortex (PMd), and primary somatosensory cortex (S1) as monkeys perform reaching tasks. Only NER aligns latent dynamics with both hand position and direction, visualizable in 3D. NER reveals consistent latent dynamics in M1 and PMd across sixteen sessions over a year. Using a linear regression decoder, NER explains 86\% and 97\% of the variance in velocity and position, respectively. Linear models trained on data from one session successfully decode velocity, position, and direction in held-out test data from different dates and cortical areas (64\%, 88\%, and 90\%). NER also reveals distinct latent dynamics in S1 during consistent movements and in M1 during curved reaching tasks. The code is available at https://github.com/NeuroscienceAI/NER.

Neural Embeddings Rank: Aligning 3D latent dynamics with movements

Abstract

Aligning neural dynamics with movements is a fundamental goal in neuroscience and brain-machine interfaces. However, there is still a lack of dimensionality reduction methods that can effectively align low-dimensional latent dynamics with movements. To address this gap, we propose Neural Embeddings Rank (NER), a technique that embeds neural dynamics into a 3D latent space and contrasts the embeddings based on movement ranks. NER learns to regress continuous representations of neural dynamics (i.e., embeddings) on continuous movements. We apply NER and six other dimensionality reduction techniques to neurons in the primary motor cortex (M1), dorsal premotor cortex (PMd), and primary somatosensory cortex (S1) as monkeys perform reaching tasks. Only NER aligns latent dynamics with both hand position and direction, visualizable in 3D. NER reveals consistent latent dynamics in M1 and PMd across sixteen sessions over a year. Using a linear regression decoder, NER explains 86% and 97% of the variance in velocity and position, respectively. Linear models trained on data from one session successfully decode velocity, position, and direction in held-out test data from different dates and cortical areas (64%, 88%, and 90%). NER also reveals distinct latent dynamics in S1 during consistent movements and in M1 during curved reaching tasks. The code is available at https://github.com/NeuroscienceAI/NER.

1 Introduction

It has long been thought that individual neurons in the motor and premotor cortex, similar to those in the sensory cortex, are tuned to specific movement parameters such as direction. However, this static and receptive field-based neural representation fails to explain movement trajectories during simple tasks like reaching. Recent studies have found that the activities of multiple simultaneously recorded neurons, which fire spikes in a time-dependent manner, encode reaching movements [10]. Unlike the one-dimensional dynamics from a single neuron, understanding how movements are represented by these high-dimensional neural dynamics is challenging. In systems neuroscience and brain-machine interfaces, there is significant motivation to reduce these high-dimensional neural dynamics to low-dimensional latent dynamics for at least three reasons:

First, visualizing neural dynamics . This involves a trade-off between dimensionality and explained variance. To explain a complex reaching task, at least six dimensions are typically required. For example, in an eight-direction center-out reaching task, [11][9][12] selects fifteen dimensions for the PMd, ten for the M1, and eight for the S1. Therefore, we need to further reduce the dimensionality of these 'low-dimensional' latent dynamics. Currently, we still lack a method that can directly explain enough variance within three dimensions. Second, comparing movements-related latent dynamics . After dimensionality reduction, we can visualize the trajectories of latent dynamics over time. For instance, [5] reveals rotational latent dynamics during reaching tasks using principal component analysis (PCA). [25] found that animals performing similar tasks exhibit similar latent dynamics.

However, these trajectories do not align precisely with the reaching movements: when the hand reaches in eight directions, the trajectories of latent dynamics are neither in eight distinct directions nor well-separated, often appearing entangled. Third, decoding movements using latent dynamics . Decoders trained on individual neural activities are commonly used to predict movements [13]. However, a drawback of using individual neural activities is that when the identities of neurons change during long-term recordings, the decoding performance deteriorates [9]. Additionally, it is infeasible to decode movements from different brain areas or animals. Decoders trained on latent dynamics facilitate long-term and cross-animal decoding [9, 25]. Since latent dynamics do not fully capture neural dynamics, decoding performance is often suboptimal with linear decoders, necessitating the use of nonlinear decoders or deep neural networks [25]. Thus, decoding movements using a linear decoder without hyperparameters remains a significant challenge.

As our goal is to extract latent dynamics that are most informative about movements, we chose to train the latent dynamics using movements as the target. Several recent studies have trained latent dynamics to classify different movement directions or positions using variational autoencoders (V AE) [32][14][18] or contrastive learning [27][1]. In this paper, we are inspired by the fact that many features, including movements, are continuous, and that the function of many biological neurons is not classification but regression. For example, neurons exhibit monotonic tuning to light intensity and sound levels [2][24]. Even for discrete features like faces, face cells exhibit ramp-shaped tuning to different features[8]. Additionally, 75% of faces could be correctly decoded using a linear regression decoder[4]. Thus, we trained latent dynamics to regress movement by minimizing the ranking loss.

We are motivated by the fact that CEBRA treats continuous labels as many discrete classes, which cannot be well separated in low-dimensional space. These classes are also highly imbalanced, with many more near-zero classes (Fig. 1a).

Our two main contributions are as follows:

We introduce Neural Embeddings Rank (NER), a dimensionality reduction method that contrasts paired samples in the embedding space and ranks neural embeddings to align them with continuous movement labels (Fig. 1b). Without introducing additional hyperparameters and using the same inputs, NER addresses the high dimensionality and class imbalance issues present in CEBRA.

We demonstrate that NER reveals behavior-aligned latent dynamics in all twenty sessions from three different brain areas in three monkeys. The trajectories of latent dynamics vary across different behaviors. We show that all movement parameters (directions, velocities, and positions) can be decoded using linear decoders trained on data from one session, tested on data from different dates over one year, and across different brain areas and hemispheres.

Δt ≈ 'augmentation'

Y

X

a

b

vel. (cm/s)

pred. (cm/s)

0

200

-200

CEBRA

NER

1st

2nd

3rd

label dist. w.r.t. anchor

continuous label

rank emb. w.r.t. label

1 P. Pair    4 N. Pair

1 P. Pair    3 N. Pair

1 P. Pair    3 N. Pair

1 P. Pair    2 N. Pair

1 P. Pair    1 N. Pair

vel. (cm/s)

-200

200

anchor vi

negative vk

positive vj

t

same label

t+Δt

positive vj

negative vn

CEBRA

NER

0

200

-200

0

200

-200

Ours: NER

Hand positions/

velocities & directions

Aligned latent

dynamics

Decoder

Decoded velocities/

positions & directions

Neural to latent

dynamics

Train 80%

M1 same day

1 year later

contra. M1

Test 20%

1

2

199

200

S1

M1

M1

PMd

neuron

latent

1

2

3

same

day

high

low

0

90

180

270

a

b

c

d

e

Latent dynamics

velocities

directions

1

2

3

CEBRA & piVAE

DO

口

M

Time

2 Related work

There are at least five categories of dimensionality reduction methods: Linear methods, such as PCA, jPCA [5], demixed PCA (dPCA) [16], and preferential subspace identification (PSID) [26]. PCA captures the majority of variance in the data, jPCA reveals rotational dynamics in monkey reaching, dPCA highlights task-related components, and PSID extracts latent dynamics that predict motion during reach versus return epochs. Nonlinear methods like uniform manifold approximation and projection (UMAP) [19] and t-distributed stochastic neighbor embedding (tSNE) [28] are extensively used in biological data, such as for identifying neuron cell types [17]. These methods reveal identities but often collapse temporal dynamics that resemble neural trajectories. UMAP with labels has been applied for dimensionality reduction in recent studies [27, 32]. Generative methods using recurrent neural networks or Transformers, such as latent factor analysis via dynamical systems (LFADS) [21], AutoLFADS [15], RADICaL [33], and Neural Data Transformer (NDT) [30]. These methods model single-trial variability in neural spiking activity better than PCA, though they often require explicit assumptions about underlying data statistics. Label-guided generative methods using VAEs, including Poisson identifiable VAE (piVAE) [32], SwapVAE [18], and targeted neural dynamical modeling (TNDM) [14]. For example, piVAE uses both discrete and continuous labels to shape embeddings, revealing well-separated but less movement-aligned embeddings. Contrastive learning methods, introduced to learn robust, generalizable representations of neural population dynamics,

such as CEBRA [27] and Mine Your Own view (MYOW) [1]. Compared to piVAE, AutoLFADS, and UMAP, CEBRA provides more identifiable latent dynamics corresponding to different hand directions in S1, although these latent dynamics trajectories do not correlate well with movements.

3 Model

NER is inspired by previous studies [31] and uses the same data sampling and neural feature encoder as CEBRA [27] to extract neural embeddings. Fig. 1b illustrates the difference between NER and CEBRA. CEBRA treats each embedding in a batch as a discrete class. For an anchor, it contrasts with its augmented embedding as a positive pair and three randomly sampled embeddings as negative pairs. NER, on the other hand, ranks six embeddings according to their continuous labels. It contrasts an anchor with its augmented or first embedding as a positive pair and the remaining four embeddings as negative pairs. NER continues by contrasting the second embedding as a positive pair and the remaining three embeddings as negative pairs. This process repeats until all embeddings have been positively contrasted with the anchor. NER learns a regression-aware representation that orders all embeddings in a batch.

Mathematically, we define x as the high-dimensional neural dynamics, f as the feature encoder, and v = f ( x ) as the low-dimensional neural embeddings. The batch size N is set to 512, and the temperature τ is fixed at 1. Data augmentation in both CEBRA and NER is achieved by selecting embeddings whose labels fall within a specific offset (e.g., 10 ms) of the anchor's label. Note that we did not fine-tune the temperature and offset, as the only difference between CEBRA and NER is the loss function. The selection of these hyperparameters will have a similar effect on both models.

In each iteration, CEBRA receives one batch of anchors v i , one batch of embeddings v j that will positively contrast with the anchors, and a third batch of randomly sampled embeddings v n that will negatively contrast with the anchors. The anchor loss l in CEBRA is:

where sim ( · , · ) represents the similarity between two neural embeddings (e.g., negative L 2 ).

In each iteration, NER receives one batch of anchors v i , one batch of augmented embeddings v j , v k that will either positively or negatively contrast with an anchor, and a third batch of labels y . The anchor loss l in NER is:

̸

̸

There are two key differences from CEBRA. First, batches of v i and v j , v k are merged, and labels y are duplicated, resulting in a batch of 2 N for both embeddings and labels. This ensures that each anchor and its augmented embedding exist within the same batch. Second, we introduce S i,j := { v k | k = i, d ( y i , y k ) ≥ d ( y i , y j ) } to denote the set of embeddings v k that are of lower ranks than v j in terms of label distance relative to v i , where d ( · , · ) is the distance measure between two labels (e.g., L 1 ). Intuitively, for an anchor v i , any other embedding v j in the batch is positively contrasted with it, enforcing the similarity between v i and v j to be greater than that between v i and any other embedding v k in the batch if the label distance between y i and y k is larger than that of y i and y j . Minimizing l ( i ) NER aligns the order of embeddings with their corresponding label orders relative to the anchor.

By ranking all the embeddings within a batch, NER also addresses the class imbalance issue and effectively represents infrequent classes (Fig. 1a). In each batch, a small percentage (e.g., 5%) of embeddings may come from infrequent classes, such as large velocities. In CEBRA, only a single augmented embedding is positively contrasted with its anchor, so only 5% of anchors have access to these infrequent embeddings. In NER, all 2 N -1 embeddings can contrast with an anchor, allowing 100% of anchors to access the 5% of infrequent embeddings.

4 Results

To fairly evaluate our dimensionality reduction method against related approaches, we selected six representative methods from various categories: PCA, dPCA, UMAP (with and without labels),

piVAE, and CEBRA. To avoid bias from a single session in a specific brain area, where piVAE and CEBRA were previously tested, we conducted experiments across M1, PMd, and S1, covering a total of twenty sessions (Table 1). Statistical results for each comparison are provided (Table 2). Figure 2 illustrates the pipeline of this study. We primarily used linear decoders, and additionally included nonlinear k-nearest neighbors (kNN) decoder. To maintain consistency with movement decoding in the original CEBRA paper, we used a 16-dimensional (16D) CEBRA model, which has a stronger representation capacity and demonstrates better performance than the 3D model in kNN decoder.

4.1 Movement-aligned latent dynamics were consistent over years in M1

Across all ten sessions in the left and right hemispheres of M1, NER consistently revealed neural embeddings that aligned with movement (Fig.3a, Fig.9a). Notably, during the initial movement stage, the latent dynamics converged on the same starting points, forming a pinwheel structure resembling the ground truth movements. Furthermore, we observed nearly identical neural embeddings in both hemispheres, even when data collection was separated by over a year.

a

2016-09-29

2016-10-06

2016-10-14

2016-10-21

NER

CEBRA

piVAE

c

b

2015-03-13

2015-03-19

M1 in Left Hemisphere

M1 in Right Hemisphere

CEBRA was the second-best method, uncovering comparable latent dynamics with both directions and positions roughly aligned with movements for both single and averaged embeddings (Fig.3b, Fig.9b). However, CEBRA had two limitations: the movement starting points were widely separated,

differing from ground truth movements, and its latent dynamics were less consistent across sessions. For example, it only revealed connected latent dynamics at movement starting points in two sessions.

piVAE ranked third, displaying direction-aligned latent dynamics in different directions with relatively separated single-trial neural embeddings (Fig.3c, Fig.9c). However, while correlated with movements, the latent dynamics were not aligned with them and were less consistent across sessions. UMAP with labels showed clearly clustered neural embeddings corresponding to different angles, while UMAP without labels produced extended, less clustered latent dynamics (Fig. 10). Both methods failed to generate aligned and consistent latent dynamics. PCA and dPCA also generated identifiable latent dynamics (Fig.10), with dPCA revealing both time and stimulus components. However, a major limitation of both methods was the mixing of single-trial neural embeddings (Fig.10).

In summary, NER proved to be the best method for revealing movement-aligned latent dynamics. We further evaluated its performance in PMd and S1, leveraging these aligned latent dynamics to decode movements within and across sessions and to explore movement-specific latent dynamics.

4.2 Explained variance of movements using linear decoders in M1, PMd, and S1

Five dimensionality reduction methods were used to reveal single-trial latent dynamics that depended on movements. We then used these latent dynamics as independent variables to explain the variance of dependent variables, namely, hand velocities, directions, and positions.

Fig. 4a shows the ground truth and predicted hand movement trajectories using latent dynamics revealed by NER in PMd. A linear regression decoder explained 90% and 98% of the variance in hand velocities and positions, respectively.

a

b

c

d

NER

Directions and X-Y velocities

Velocities

Positions

Hand movements

Explained movements

piVAE

UMAP w/ label

X-vel

UMAP w/o label

CEBRA

Y-vel

M1

M1

PMd

PMd

S1

S1

Direction classification accuracy(

80

一

60

40

20

0

0

200

400

600

800

1000

1200

Time from go cue (ms)

0.8

★

★

★

★

★

★

★

0.8

★

★

★

★

★

★

0.9

★

★

★

★

★

★

★

★

★

★

0.9-

1.0

★

★

★

★

o.6

0.6

0.8

0.8

0.7

0.7

0.4

0.6

0.6

02/

0.1L

62609

161005

90019

161007

161014

：

161021

150313

150319

150629

150630○

0.2

160929

161005

161006

口

161007

161014

1

161021

171201

■

171204

171207°

■

0.3

0.5

0.4

160929

■

161005

900191

口

161007

161014

161021

150313-

150319

150629

150630

0.5

0.4

161005

+

■900191

161007

161014

161021

171201

■

171204

171207·

In both M1 and PMd (Fig.4b, Fig.13a), a logistic regression decoder revealed the tuning of directional accuracy from the start of the go cue to the end of the animal's reach. This tuning curve was highly

correlated with hand velocities in the latent dynamics extracted by NER but not by CEBRA (M1: 0.93 vs. 0.28, PMd: 0.93 vs. 0.13; Fig. 13b).

NER outperformed the four other methods in all sessions for explaining the variance of both hand velocities and positions (Fig. 4c, d). For example, across ten sessions in M1, NER explained 86% of the variance in velocities, while the next best model, piV AE, explained only 35%. Similar findings were observed in PMd (89% vs. 32%) and S1 (86% vs. 47%).

In summary, combined with linear decoders, NER demonstrated the clearest velocity-dependent direction tuning and explained the largest variance in velocities and positions.

4.3 Long-term and cross-hemisphere decoding in M1

NER explains the largest variance in both hand velocities and positions in the 80% training data. Next, we tested the trained model on the remaining 20% of test data. In addition to decoding test data from the same session on a single day, we also evaluated its performance on data from different sessions. We first conducted comparisons in M1 (Fig.5), followed by PMd (Fig.6) and S1 (Fig. 7).

Velocities

Positions

Linear & logistic regression

k

-

nearest neighbors

Directions

piVAE

CEBRA

NER

piVAE

CEBRA 16D

NER

a

b

c

d

e

f

dia: 0.71 off: 0.64

dia: -3.1 off: -3.2

dia: -1.2 off: -2.1

dia: 0.94 off: 0.88

dia: 0.74 off: 0.71

dia: 0.67 off: 0.23

dia: 93% off: 90%

dia: 90% off: 79%

dia: 81% off: 44%

dia: 0.76 off: 0.7

dia: 0.75 off: -1.1

dia: 0.87 off: -1.1

dia: 0.93 off: 0.89

dia: 0.93 off: 0.18

dia: 0.97 off: 0.13

dia: 92% off: 89%

dia: 93% off: 57%

dia: 99% off: 44%

0

1 or 100%

150319

161014

161021

150319

161014

161021-

150319

161014

161021

Fig.5a shows velocity decoding performance using a linear regression decoder across three dimensionality reduction methods. Interestingly, the linear decoder could not decode hand velocities from latent dynamics generated by CEBRA (all variances were negative) and piVAE (only four positive). In contrast, all variances with NER were positive (minimum: 0.24), with performance across different sessions comparable to within-session performance (0.64 vs. 0.71). A kNN decoder (Fig.5b) achieved high performance for CEBRA and piVAE only with within-session latent dynamics.

NER outperformed CEBRA and piVAE by a substantial margin in position decoding across all conditions (Fig.5c). While the kNN decoder did not improve NER's performance relative to the linear decoder, it enhanced within-session performance for CEBRA and piVAE over NER (Fig.5d). However, this came at the cost of cross-session decoding performance (0.89 vs. 0.18 and 0.13). Fig.5e shows direction decoding accuracy, where NER still outperformed CEBRA and piVAE. Similar results were observed using a kNN decoder (Fig.5f).

In summary, compared to CEBRA and piVAE, NER consistently achieved higher performance across all sessions with a linear decoder and outperformed in cross-session decoding with the nonlinear decoder.

4.4 Latent dynamics in PMd and decoding between M1 and PMd

Next, we turned our attention to PMd. Surprisingly, NER revealed similar latent dynamics in this higher-order motor area (Fig.6a). Two other dimensionality reduction methods also identified comparable latent dynamics, but these were less consistent and did not align well with movements (Fig.14). Fig. 6b shows within- and cross-session velocity decoding using a linear regression decoder (left) and a kNN decoder (right). Consistent with M1, in PMd, we observed that (1) both CEBRA and piVAE underperformed with linear regression, while NER achieved stable performance across all conditions, regardless of within- or cross-session contexts; (2) NER demonstrated robustness across decoders, whereas the performance of CEBRA and piVAE varied substantially, ranging from very low to occasionally outperforming NER in within-session decoding with the kNN decoder; and (3) even with the kNN decoder, CEBRA and piVAE failed in cross-session decoding, whereas NER maintained similar performance.

We also evaluated position and direction decoding and found that NER continued to outperform the other two methods (Fig.6c, Fig.15). Overall, NER reveals consistent latent dynamics in PMd and can effectively decode movements across PMd and M1.

Velocities

Positions

Linear regression

k

-

nearest neighbors

piVAE

CEBRA

NER

piVAE

CEBRA 16D

NER

b

c

dia: 0.75 off: 0.74

dia: -3.1 off: -3.1

dia: -1.6 off: -2.8

dia: 0.94 off: 0.93

dia: 0.75 off: 0.74

dia: 0.58 off: 0.21

dia: 0.78 off: 0.76

dia: 0.78 off: -0.72

dia: 0.84 off: -0.62

dia: 0.94 off: 0.92

dia: 0.94 off: 0.25

dia: 0.95 off: 0.19

a

2016-09-29

10-05

10-06

10-07

10-14

10-21

0

1

160929PM

161021M1

161021PM

160929PM

161021M1

161021PM

4.5 Same movements but different latent dynamics in S1

Lastly, we examined the latent dynamics and movement decoding in S1. NER revealed consistent latent dynamics in S1 (Fig.7a, b). After rotating the latent dynamics with reference to the target shown in Fig.3, they exhibit a consistent yet distinct shape compared to the latent dynamics observed in M1 and PMd. Velocity decoding using a linear regression decoder was only successful when the latent dynamics were extracted by NER (Fig.7c, d). While all three methods performed adequately for position decoding, NER outperformed both CEBRA and piVAE across all nine conditions (Fig.7e). Although S1 displays different latent dynamics than M1 and PMd, NER remains the most effective method for decoding movement both within and across sessions in S1.

piVAE

CEBRA

NER

dia: 0.9 off: 0.77

dia: 0.6 off: 0.45

dia: 0.73 off: 0.51

Velocities

Positions

d

e

a

b

c

dia: 0.67 off: 0.56

dia: -4.6 off: -4.8

dia: -0.38 off: -0.75

17-12-01

12-04

12-07

0

1

Explained

Decoded

171201

171207

171201

171207

4.6 Straight and curved hand movements have different latent dynamics in M1

Lastly, we applied both NER and CEBRA in a new experiment where a monkey performed both straight and curved hand movements in different directions while neural recordings were simultaneously collected in M1 (Fig. 8a).

We first examined the latent dynamics when the monkey performed straight hand movements in six different directions (Fig.8b). Surprisingly, both single and averaged latent dynamics aligned well with the movements (Fig.8c) and displayed a shape similar to the previously observed latent dynamics in M1. Next, we selected three hand directions, each involving both straight and curved hand movements (Fig.8e). When we trained both NER and CEBRA on individual directions, only NER revealed clearly separated latent dynamics corresponding to straight and curved movements (Fig.8f). The difference between the two methods became even more pronounced when they were trained on all three directions combined: NER displayed latent dynamics for straight movements that were surrounded by those formed by curved movements. The explained variance achieved by NER was also higher than that of CEBRA across all three angles, especially on the combined angles (Fig. 17c). Finally, we tested a more challenging condition where all six reaching movements were curved (Fig.8g). In the latent space, two curved movements in the same direction produced close but distinct latent dynamics (Fig.8h). While CEBRA performed comparably on single directions, it struggled with combined directions (Fig.17e). NER consistently achieved higher explained variance than CEBRA (Fig.17f).

Overall, NER not only aligns latent dynamics with straight movements but also effectively differentiates curved hand movements from straight ones, demonstrating its robustness across various movement types.

5 Discussion

A benchmark comparison of NER and six other dimensionality reduction methods across multiple brain areas and two movement tasks highlights NER's superior performance in uncovering latent dynamics. We believe the primary advantage of our method is its ability to extract nearly identical latent dynamics across brain areas and over extended time periods. This capability opens new avenues for both fundamental neuroscience research and brain-machine interfaces (BMI). Previous studies [9, 25] used PCA to discover preserved latent dynamics across time and in animals performing similar behaviors. In contrast, the latent dynamics revealed by NER are significantly more informative than those uncovered by PCA. We believe NER will aid neuroscientists in probing the stability of latent

d

e

a

b

c

g

f

h

6 straight

move

3 pairs of

straight-curve

move

3 pairs of

curve-curve

move

dynamics under various conditions. For BMI applications, we demonstrate that NER, combined with a simple linear decoder, can predict hand movements across years, brain areas, and hemispheres. This capability enables training on latent dynamics within and between subjects, allowing for movement prediction in different subjects. The linear decoder's lack of hyperparameters is also an advantage.

The application of NER is not limited to hand movements using neurophysiological recordings. This includes latent dynamics in the hippocampus representing the body position of running rats and latent dynamics in the visual cortex representing embedded video features [27]. Similarly, recording modalities are not limited to single-neuron electrophysiology; other methods, such as calcium imaging, local-field potentials, and EEG, can also be used.

In our final experiments with curved movements, we manually selected three pairs of reaching tasks that involved curved trajectories. Both NER and CEBRA failed when all 108 movement conditions were trained simultaneously. Furthermore, beyond the straight and curved movements examined here in macaque monkeys, more complex movements, such as handwriting [29] and speech [29, 20], exist primarily in humans. These movements are also continuous but may require more than a 3D latent space for effective representation, and ranking the distances of complex movements adds further challenges. Nevertheless, uncovering the latent dynamics underlying these complex movements, which has yet to be achieved[7], could greatly advance BMI applications [3, 6].

Acknowledgments and Disclosure of Funding

We greatly appreciate the Miller and Shenoy labs for publicly releasing their experimental data on macaque monkeys. We also thank the Mathis lab for the CEBRA code, which served as the basis for our NER. This work was supported by National Institute of Deafness and Other Communications Disorders grant DC003180 (X.W.). No conflicts of interest are declared by the authors.

References

Mehdi Azabou, Mohammad Gheshlaghi Azar, Ran Liu, Chi-Heng Lin, Erik C Johnson, Kiran Bhaskaran-Nair, Max Dabagia, Bernardo Avila-Pires, Lindsey Kitchell, Keith B Hengen, et al. Mine your own view: Self-supervised learning through across-sample prediction. arXiv preprint arXiv:2102.10106 , 2021.

Matteo Carandini and David J Heeger. Normalization as a canonical neural computation. Nature reviews neuroscience , 13(1):51-62, 2012.

Nicholas S Card, Maitreyee Wairagkar, Carrina Iacobacci, Xianda Hou, Tyler Singer-Clark, Francis R Willett, Erin M Kunz, Chaofei Fan, Maryam Vahdati Nia, Darrel R Deo, et al. An accurate and rapidly calibrating speech neuroprosthesis. New England Journal of Medicine , 391(7):609-618, 2024.

Le Chang and Doris Y Tsao. The code for facial identity in the primate brain. Cell , 169(6):10131028, 2017.

Mark M Churchland, John P Cunningham, Matthew T Kaufman, Justin D Foster, Paul Nuyujukian, Stephen I Ryu, and Krishna V Shenoy. Neural population dynamics during reaching. Nature , 487(7405):51-56, 2012.

Chaofei Fan, Nick Hahn, Foram Kamdar, Donald Avansino, Guy Wilson, Leigh Hochberg, Krishna V Shenoy, Jaimie Henderson, and Francis Willett. Plug-and-play stability for intracortical brain-computer interfaces: a one-year demonstration of seamless brain-to-text communication. Advances in neural information processing systems , 36, 2024.

Cátia Fortunato, Jorge Bennasar-Vázquez, Junchol Park, Joanna C Chang, Lee E Miller, Joshua T Dudman, Matthew G Perich, and Juan A Gallego. Nonlinear manifolds underlie neural population activity during behaviour. bioRxiv , 2023.

Winrich A Freiwald, Doris Y Tsao, and Margaret S Livingstone. A face feature space in the macaque temporal lobe. Nature neuroscience , 12(9):1187, 2009.

Juan A Gallego, Matthew G Perich, Raeed H Chowdhury, Sara A Solla, and Lee E Miller. Long-term stability of cortical population dynamics underlying consistent behavior. Nature neuroscience , 23(2):260-270, 2020.

Juan A Gallego, Matthew G Perich, Lee E Miller, and Sara A Solla. Neural manifolds for the control of movement. Neuron , 94(5):978-984, 2017.

Juan A Gallego, Matthew G Perich, Stephanie N Naufel, Christian Ethier, Sara A Solla, and Lee E Miller. Cortical population activity within a preserved neural manifold underlies multiple motor behaviors. Nature communications , 9(1):4233, 2018.

Cecilia Gallego-Carracedo, Matthew G Perich, Raeed H Chowdhury, Lee E Miller, and Juan Álvaro Gallego. Local field potentials reflect cortical population dynamics in a region-specific and frequency-dependent manner. Elife , 11:e73155, 2022.

Joshua I Glaser, Ari S Benjamin, Raeed H Chowdhury, Matthew G Perich, Lee E Miller, and Konrad P Kording. Machine learning for neural decoding. Eneuro , 7(4), 2020.

Cole Hurwitz, Akash Srivastava, Kai Xu, Justin Jude, Matthew Perich, Lee Miller, and Matthias Hennig. Targeted neural dynamical modeling. Advances in Neural Information Processing Systems , 34:29379-29392, 2021.

Mohammad Reza Keshtkaran, Andrew R Sedler, Raeed H Chowdhury, Raghav Tandon, Diya Basrai, Sarah L Nguyen, Hansem Sohn, Mehrdad Jazayeri, Lee E Miller, and Chethan Pandarinath. A large-scale neural network training framework for generalized estimation of single-trial population dynamics. Nature Methods , 19(12):1572-1577, 2022.

Dmitry Kobak, Wieland Brendel, Christos Constantinidis, Claudia E Feierstein, Adam Kepecs, Zachary F Mainen, Xue-Lian Qi, Ranulfo Romo, Naoshige Uchida, and Christian K Machens. Demixed principal component analysis of neural population data. elife , 5:e10989, 2016.

Eric Kenji Lee, Hymavathy Balasubramanian, Alexandra Tsolias, Stephanie Udochukwu Anakwe, Maria Medalla, Krishna V Shenoy, and Chandramouli Chandrasekaran. Non-linear dimensionality reduction on extracellular waveforms reveals cell type diversity in premotor cortex. Elife , 10:e67490, 2021.

Ran Liu, Mehdi Azabou, Max Dabagia, Chi-Heng Lin, Mohammad Gheshlaghi Azar, Keith Hengen, Michal Valko, and Eva Dyer. Drop, swap, and generate: A self-supervised approach for generating neural activity. Advances in neural information processing systems , 34:10587-10599, 2021.

Leland McInnes, John Healy, and James Melville. Umap: Uniform manifold approximation and projection for dimension reduction. arXiv preprint arXiv:1802.03426 , 2018.

Sean L Metzger, Kaylo T Littlejohn, Alexander B Silva, David A Moses, Margaret P Seaton, Ran Wang, Maximilian E Dougherty, Jessie R Liu, Peter Wu, Michael A Berger, et al. A highperformance neuroprosthesis for speech decoding and avatar control. Nature , 620(7976):10371046, 2023.

Chethan Pandarinath, Daniel J O'Shea, Jasmine Collins, Rafal Jozefowicz, Sergey D Stavisky, Jonathan C Kao, Eric M Trautmann, Matthew T Kaufman, Stephen I Ryu, Leigh R Hochberg, et al. Inferring single-trial neural population dynamics using sequential auto-encoders. Nature methods , 15(10):805-815, 2018.

Felix Pei, Joel Ye, David Zoltowski, Anqi Wu, Raeed H Chowdhury, Hansem Sohn, Joseph E O'Doherty, Krishna V Shenoy, Matthew T Kaufman, Mark Churchland, et al. Neural latents benchmark'21: evaluating latent variable models of neural population activity. arXiv preprint arXiv:2109.04463 , 2021.

Matthew G Perich, Juan A Gallego, and Lee E Miller. A neural population mechanism for rapid learning. Neuron , 100(4):964-976, 2018.

Neil C Rabinowitz, Ben DB Willmore, Jan WH Schnupp, and Andrew J King. Contrast gain control in auditory cortex. Neuron , 70(6):1178-1191, 2011.

Mostafa Safaie, Joanna C Chang, Junchol Park, Lee E Miller, Joshua T Dudman, Matthew G Perich, and Juan A Gallego. Preserved neural dynamics across animals performing similar behaviour. Nature , 623(7988):765-771, 2023.

Omid G Sani, Hamidreza Abbaspourazad, Yan T Wong, Bijan Pesaran, and Maryam M Shanechi. Modeling behaviorally relevant neural dynamics enabled by preferential subspace identification. Nature Neuroscience , 24(1):140-149, 2021.

Steffen Schneider, Jin Hwa Lee, and Mackenzie Weygandt Mathis. Learnable latent embeddings for joint behavioural and neural analysis. Nature , 617(7960):360-368, 2023.

Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research , 9(11), 2008.

Francis R Willett, Donald T Avansino, Leigh R Hochberg, Jaimie M Henderson, and Krishna V Shenoy. High-performance brain-to-text communication via handwriting. Nature , 593(7858):249-254, 2021.

Joel Ye and Chethan Pandarinath. Representation learning for neural population activity with neural data transformers. arXiv preprint arXiv:2108.01210 , 2021.

Kaiwen Zha, Peng Cao, Jeany Son, Yuzhe Yang, and Dina Katabi. Rank-n-contrast: Learning continuous representations for regression. Advances in Neural Information Processing Systems , 36, 2024.

Ding Zhou and Xue-Xin Wei. Learning identifiable and interpretable latent models of highdimensional neural activity using pi-vae. Advances in Neural Information Processing Systems , 33:7234-7247, 2020.

Feng Zhu, Harrison A Grier, Raghav Tandon, Changjia Cai, Anjali Agarwal, Andrea Giovannucci, Matthew T Kaufman, and Chethan Pandarinath. A deep learning framework for inference of single-trial neural population dynamics from calcium imaging with subframe temporal resolution. Nature neuroscience , 25(12):1724-1734, 2022.

A Appendix / supplemental material

A.1 Computer

Operating System is Ubuntu 22.04.3 LTS, computer memory is 42 GB, CPU is Intel Xeon W-2225, and GPU is NVIDIA RTX A5000.

A.2 Center-out reaching experiments in M1, PMd, and S1

A.2.1 Behavior

The dataset comprises behavioral task data from two male Macaca mulatta monkeys (Monkeys H and C). These monkeys were trained to sit in a primate chair and perform a center-out reaching task using a planar manipulandum with the hand contralateral to the implanted hemisphere. During each trial, the monkey started by moving a cursor to a central target. After a variable waiting period, one of eight outer targets (equally spaced along a circle of 6-8 cm radius) was presented. Monkeys C and H differed in the task protocols:

Monkey C : Trained to wait for an auditory go cue during a delay period of 0.5-1.5 seconds while the target remained visible. Upon receiving the cue, the monkey had to move the cursor to the outer target within 1 second and hold it there for 0.5 seconds to receive a liquid reward.

Monkey H : No delay period; the monkey had to move the cursor to the outer target within 1 second and hold it there for 0.1 seconds.

For both monkeys, the trial restarted by returning the cursor to the central target. Endpoint positions of the manipulandum were recorded at 1 kHz, and task event timings were digitally logged. Hand velocity was computed as the derivative of hand position. The dataset includes 6 sessions for Monkey C and 5 sessions for Monkey H, considering only successful trials (an average of 307±221 trials per session, mean ± s.d.)

Neural recordings

The dataset consists of neural recordings from two male Macaque monkeys. These recordings were obtained using 96-channel Utah electrode arrays implanted in specific cortical regions.

Monkey C : Initially implanted in the right primary motor cortex (M1) and later received implants in the left M1 and dorsal premotor cortex (PMd) (denoted as CR and CL, respectively).

Monkey H : Implanted in area 2 of the primary somatosensory cortex of the left hemisphere.

Neural activity was recorded using a Cerebus system (Blackrock Microsystems, Salt Lake City, UT) at a sampling frequency of 30 kHz. The recorded signals underwent band-pass filtering (250-5000 Hz) and were converted to spike times based on threshold crossings. Spike sorting was performed using specialized software (Offline Sorter v3, Plexon, Inc, Dallas, TX) to identify putative neurons.

Table 1: Datasets for the center-out reaching experiments.

Datasets

All the center-out reaching experiments using the open source data from: https://datadryad. org/stash/dataset/doi:10.5061/dryad.xd2547dkt This data is released accompanying this paper: https://elifesciences.org/articles/73155#data We used all sessions from Monkey Chewie and Monkey Han. We chose these Monkeys because one session in Chewie is used by piVAE paper[32] and one session in Monkey Han is used by CEBRA paper[27]. Although the datasets come from same session in same Monkey, the temporal resolution is much higher for the datasets used by piVAE and CEBRA papers. The data is Matlab format and we extract following information: tgtDir (Target direction, radians for Monkey Chewie and degrees for Monkey Han), idx-goCueTime (The time go Cue is one), vel(XY velocities), M1-spikes for both Chewie 2015 and Chewie 2016, and PMd-spikes only for Chewie 2016. The time bin is 30ms and we extract all the spikes after each go Cue. We extracted 40 bins for Monkey Chewie and 35 bins for Monkey Han, because most trial in Monkey Han has short acquisition window than 40 bins (afte go Cue). We smoothed the discrete spike count in the Matlab using a Gaussian kernel. The standard deviation is 1.5 and kernel size is six standard deviations. We keep all the trials and neurons. The number of trials and neurons are shown in Table 1. Our NER is just a modification of the loss function used by CEBRA: https://github.com/AdaptiveMotorControlLab/CEBRA The RNC loss could be downloaded from: https://github.com/kaiwenzha/Rank-N-Contrast The iterations is 20000, learning rate is 1e-4, and batch size 512. The temperature is fixed to 1 for both NER and CEBRA. The output dimension of NER is fixed to 3. For CEBRA, we used output dimension of 3 for visualizing the latent dynamics and linear models decoding. We only used 16 dimensional embeddings for k-NN decoders. For the piVAE, we did not use the original version which is based on older version of Tensorflow. https://github.com/zhd96/pi-vae Instead, we used the modified conv-pi-VAE that is already included into the CEBRA package. We fixed the random seed to 42, and using batch size of 200 and iterations of 300.

A.2.2 Curved hand movements experiments in M1 Fig 8 17

The MC_Maze dataset includes recordings from the primary motor and dorsal premotor cortices of a monkey performing reaches to visual targets in a virtual maze with an instructed delay. This dataset comprises 108 different task configurations, each varying in target positions, barrier numbers, and barrier positions. The monkey repeated each task configuration multiple times in random order, resulting in 2,869 trials recorded in a single session with 182 neurons and simultaneous hand kinematics monitoring. This datasets could be downloaded from: https://dandiarchive.org/dandiset/ 000128 , https://github.com/dandisets/000128 . In our works, we used the NLB21 package[22] to download the data from DANDI: https://github.com/neurallatents Our code (Jupyter Notebook) is modified from: https://github.com/neurallatents/neurallatents. github.io/blob/master/notebooks/mc_maze.ipynb We used a time bin of 5ms (raw resolution is 1ms) and Gaussian window of 50ms. For six straight movements, we only use version 0 and trial type of 13, 29, 17, 38, 6, 18. For the straight-curved and curved-curved movements, we keep all three versions of task (one straight and two curved). We removed the target angle during training. We used trial type of 13, 38, 18 for straight-curved and 37,1, 31,38, 34,18 for curved-curved movements. The iterations is 5000, learning rate is 1e-4, and batch size 512.

A.3 Decoders

We rotate the latent dynamics with reference to same target before decoding. We used orthogonal Procrustes from scipy for this purpose: https://docs.scipy.org/doc/scipy/reference/ generated/scipy.linalg.orthogonal_procrustes.html . We picked on target angle and rotate the whole 3D latent dynamics using the computed orthogonal matrix. This rotation will not modify local detail or the relative positions of each reaching direction. For the linear decoders using linear and logistic regression, both are imported from "linear_model" of sklearn. There is no parameter or hyperparameter for the linear regression model. For the logistic regression model, we used the following parameters: max_iter=500, multi_class='multinomial', solver='lbfgs' For the k-nearest neighbors Regressor and Classifier, they are both imported from "neighbors" in sklearn. We used "GridSearchCV" in sklearn to searach the best "n_neighbors" range from 2 to 1024.

a

2015-06-29

2015-06-30

NER

Cebra

piVAE

c

b

2016-10-05

2016-10-07

M1 in Left Hemisphere

M1 in Right Hemisphere

piVAE

UMAP

w/o

label

UMAP

w/

label

dPCA

PCA

2016-09-29

10-05

10-06

10-07

10-14

10-21

2015-03-13

2015

06-29

06-30

NER

Cebra

piVAE

UMAP

w/o

label

UMAP

w/

label

160929

161005

161006

161007

161014

161021

150313

150319

150629

150630

(Y1)

G

U

Monkey C 2015-03-13

a

b

Monkey C 2015-03-13

Monkey C 2016-10-05

Monkey C 2016-09-29

1st time component

1st stimulus component

1st mixing component

2.0

1.5

0.75

1.5

1.0

0.50

1.0

0.5

0.25

0.5

0.00

0°0

+0.25

0.0 -

0.5

+0.50

0.5 -

1.0

1.0

0.75

1.00

1.5 -

1.5

0

10

20

30

40

10

20

30

40

0

10

20

30

40

1st time component

1st stimulus component

1st mixing component

2.0

0.75

1.0

1.5

0.50

1.0

0.5

0.25

0.5

0.0

+0.25

0.00

00

-0.5

0.50

0.5

-1.0

+0.75

1.0 -

1.5

0

10

20

30

40

0

10

20

30

40

10

20

30

40

a

b Correlation matrix among four direction tuning curves and X-Y velocities

M1

PMd

NER

piVAE

UMAP w/ label

X-vel

UMAP w/o label

Cebra

Y-vel

M1

PMd & S1

c

Explained directions in PMd and X-Y velocities

Explained peak classification accuracy by logistic regression

100

accuracy

80

ion

60

ica

ssifie

clas

S

40

ection

20

Direc

0

0

200

400

600

800

1000

1200

Timefromgocue(ms)

-1.0

-1.0

NER-1

NER-1

0.8

0.8

Cebra-0.52

1

Cebra-0.38

1

piVAE-0.96

0.68

1

0.6

piVAE-

0.9

0.55

1

0.6

UMAPw/olabel-0.96

0.66

0.98

1

0.4

UMAPw/olabel-0.82

0.62

0.84

1

0.4

X-vel-0.93

0.28

0.85

0.85

1

X-vel-0.93

0.14

0.76

0.66

1

0.2

0.2

Y-vel-0.93

0.28

0.84

0.86

1

1

Y-vel-0.93

0.12

0.76

0.65

1

1

NER-

Cebra-

piVAE-

UMAP w/o label-

X-vel-

Y-vel-

0.0

NER-

Cebra-

piVAE-

UMAP w/o label-

X-vel-

Y-vel-

0.0

100

100

90

06

Explainedvariance

Explainedvariance

80

80

70

70

60

60

50

50

160929

161005

161006

161007

161014

161021

150313

150319

150629

150630

160929

161005

161006

161007

161014

:161021

171201

171204

171207

2016-09-29

10-05

10-06

10-07

10-14

10-21

a

Cebra

piVAE

b

(U

#3

(U)

Directions

Logistic regression

k

-

nearest neighbors

piVAE

CEBRA

NER

piVAE

CEBRA 16D

NER

dia: 94% off: 93%

dia: 92% off: 85%

dia: 73% off: 40%

dia: 94% off: 92%

dia: 95% off: 59%

dia: 97% off: 44%

0

100%

160929M1

161014M1

161014PM

161021M1

161021PM

a

Cebra

piVAE

b

Rotate

with

reference

to M1

Rotate

with

reference

to S1

Rotate

with

reference

to S1

Rotate

with

reference

to S1

2017-12-01

12-04

12-07

c

a

d

3 pairs of

straight-curve

move

3 pairs of

curve-curve

move

b

c

f

e

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The claims we made in the abstract and introduction are based on our extensive experiments over multiple dataset and a benchmark comparisons of our method with six other methods.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]

Justification: In the last part of Discussion, we mentioned two limitations of this work clearly

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: There is no theoretical analysis. This work an application of contrastive learning and deep regression on Neuroscience. We do not make or claim any theoretical contribution.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: Our works use the public dataset, and our code adds the loss function from one paper into the code of the other paper. Both codes are available online. There is not hyperparameter turnings. We mentioned the GPU, iterations, and batch size we used in the Appendix. We also upload our codes that generate all the Figures.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in

some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: Also the data and code are already available online. We just combine two codes.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We gave those details in the Appendix.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [NA]

Justification: We just show the raw, mean and median values without using any statistical metrics.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We provide the details of our computer that used to run those experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We used open-source neural spiking data from animals.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: The data we used came from neuron activities in the brains. Nothing is related to humans.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The codes have been released by others already.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [NA]

Justification: All the data and codes are open source

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: Everything is open source already.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: Only authors do the experiments.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: We used open source data.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
comments: SUMMARY:
The authors propose a novel method for reducing the dimensionality of neural dynamics and aligning them with movements. This method is compared with six existing methods across different brain regions. Experiments demonstrate that all movement parameters can be accurately decoded from neural dynamics using this approach.

STRENGTHS:
The paper offers an extensive comparison to prior works on the dimensionality reduction task. It is important to note that baselines represent diverse techniques in the field. Additionally, all code used in the study is provided, ensuring transparency and reproducibility. The paper also includes numerous visualizations and is well-structured.

WEAKNESSES:
The methodology section is unclear, with equations lacking necessary explanations and annotations. Additionally, there appear to be typographical errors, such as the inconsistent notation where temperature $\tau$ is sometimes mistakenly written as $\textit{r}$. Moreover, all the experiments lack an assessment of statistical significance, which is crucial for validating the results.

QUESTIONS:
- Could you please provide a statistical analysis of your results? 
- Could you also provide some quantitative comparisons?

LIMITATIONS:
The authors have mentioned limitations of their work and I do not perceive any specific negative societal impacts that would need to be addressed.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2024-07-09

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a novel dimensionality reduction method called Neural Embedding Ranks (NER) for aligning neural dynamics with movement in brain-computer interfaces. NER uses a ranking loss to embed neural dynamics into a 3D latent space that aligns with continuous movement parameters. The authors apply NER to neural recordings from primary motor cortex (M1), dorsal premotor cortex (PMd), and primary somatosensory cortex (S1) in monkeys performing reaching tasks, comparing it with six other dimensionality reduction techniques.The paper finds that NER out-performs all other methods to align latent dynamics with both hand position and direction.\n\nSTRENGTHS:\n1. The experimental design is comprehensive, covering multiple brain areas (M1, PMd, S1), multiple monkeys, and a variety of task conditions including curved reaching movements, and the statistical rigor is evident in the consistent performance across multiple sessions and conditions.\n\n2. The benchmarking against six other dimensionality reduction techniques provides a thorough evaluation of NER's performance.\n\n3. Extensive figures and graphs were used to illustrate the statistical results of the paper, most of which were informative and well-designed.\n\n4. The effectiveness of NER is significant and proven to exceed the performance of multiple other methods under different tasks.\n\nWEAKNESSES:\n1. The explanation of the method is too limited. Only a very small portion of the paper is dedicated to the Model/Method section, which makes it difficult to understand the fundamental algorithm of NER. \n\n2. It is also reasonable to suspect the main method of the paper is the application of the existing method RNC, which limits the contribution of the paper.\n\n3. It is empirically evident that the method works well in the setting of the paper, but there is limited discussion on what could be the cause of such an improvement over other methods (such as CEBRA), which by the formulation of the loss function, is very similar in nature.\n\nQUESTIONS:\n1. Why is the Rank and Contrast loss effective in this task? Specifically, how does it differ from other contrastive-loss-based methods like CEBRA?\n2. I believe the paper would benefit if a more detailed description were given to the method section, especially a description of RNC and how your method adapted it and could be different from it.\n\nLIMITATIONS:\nThe limitations are adequately addressed.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a dimensionality reduction method, specifically to learn latent neural dynamics in a 3-dimensional space. The authors perform experiments that test the transfer of their model from one hemisphere to another, from one year to another, from one brain region to another, and how well dimensionality reduction methods separate straight and curved hand movements. Each experiment is atleast compared with CEBRA and Pi-VAE, which are both methods that also (potentially) have access to the target variables during training.\n\nSTRENGTHS:\nThe authors perform a variety of interesting generalization tasks and generally find their model to work very well. Some of the experiments the authors perform are fairly novel in the specific type of generalization they are testing, e.g. wide vs straight reaching in Section 4.6, which is interesting. The paper is relatively clear, but does require some rewriting, especially grammatically and Section 4.6 requires a rewrite to clarify exactly how the authors are training the models.\n\nWEAKNESSES:\nWeaknesses:\n1. The authors do not compare their model with AutoLFADs [1] or a target-based model like CTRL-TNDM [2] that extends LFADs with labels and currently has the highest performance on the Neural Latents Benchmark [3]. Moreover, especially given that many of the tasks tested in the paper are related to generalization across a number of different variables, I believe it is important to compare the results with POYO [4], which specifically focuses on generalization.\n2. I think the authors should submit the exact code they used for these experiments, not just the code that is required to reproduce the figures once the code has been run. I am surprised by the low performance of both Pi-VAE and CEBRA and am curious how the authors specifically performed their experiments, e.g. did both methods also have access to the continuous labels like the authors’ model. The results do not instill confidence in me, even though the authors provide many specifics in their Appendix. Specifically, in the Neural Latent Benchmark [3], CEBRA obtains the second best R^2 performance on a completely held out test set of velocities. However, and I understand this is a different task, on inter-hemispheric generalization it obtains a -3 R^2 (Figure 4), for example. This would indicate that to some extent the model is essentially predicting the opposite of the signal. Given that the authors use the exact same neural feature encoder as CEBRA, I find it hard to believe that such a dramatic change comes from changing the loss function. Especially given that CEBRA-behavior would also contrastively make latent embeddings of the same velocity/direction more similar. I do believe there is potentially a benefit to the authors’ loss function, but I find the baseline results very surprising and believe a full submission of the code would help clear up this concern. Note: I am absolutely willing to be proven wrong.\n3. The authors only use one dataset to verify their results, I believe it is imperative for the authors to verify their results in another dataset as well.\n4. The authors fail to look at animal generalization, which I would say is probably more interesting even than generalization from one brain region to another.\n5. The figures in the paper are not aligned in scale (per row), and text sometimes overlaps (Figure 5). Generally the colorbars are unclear because they have different scales and are not colored. Please add one colorbar per row of figures. Moreover, Figures 9 and 10 in the Appendix have very small 3D plots that are hard to view.\n6. The authors do not provide a lot of motivation for their method and do not explain the intuitions or implications of their results. Why is their loss function better? Why does their model generalize better? Why does generalization between brain regions make sense? Etc.\n\n[1] Keshtkaran, M. R., Sedler, A. R., Chowdhury, R. H., Tandon, R., Basrai, D., Nguyen, S. L., ... & Pandarinath, C. (2022). A large-scale neural network training framework for generalized estimation of single-trial population dynamics. Nature Methods, 19(12), 1572-1577. \\\n[2] Kudryashova, N., Perich, M. G., Miller, L. E., & Hennig, M. H. (2023, March). Ctrl-TNDM: Decoding feedback-driven movement corrections from motor cortex neurons. In Computational and Systems Neuroscience (Cosyne) 2023. \\\n[3] https://eval.ai/web/challenges/challenge-page/1256/leaderboard/3183 \\\n[4] https://poyo-brain.github.io \n\nGrammatical and language weaknesses (some of these are personal preference, so feel free to ignore ones you deem unimportant):\n- L83: “Fourth, label-guided generative methods using VAE…” -> using VAEs\n- L86: “It reveals … but cannot align with movement trajectories, …” -> The revealed embeddings … but do not align well with movement trajectories.\n- L87: “… for learning robust… “ -> to learn robust\n- L98: “… Fig 1 gives the pipeline of this studies.” -> Fig 1 visualizes the pipeline used in this study.\n- L128: “… and the its less … “ -> and is less\n- L131: “Both models fails…” -> fail\n- L133: “A major limitation of both method…” -> methods\n- L134: “… for revealing movements aligned  …” -> movement-aligned\n- Figure 3 caption: “b Hand directions” -> hand direction\n- Figure 3 caption: “X-axis indicate…” -> indicates\n- Figure 4 caption: “… b, d, f are trained on the 80% … “ -> on 80%\n- Figure 4 caption: “Brighter color indicate” -> colors indicate/color indicates\n- L142: “A linear regression decoder could explain…” -> can\n- L145: “This tuning curve is highly correlate with…” -> correlated\n- L150: “Together, combined with with …” -> with\n- L154: “… and position in the 80% of training datasets …” -> in the 80% training data\n- L156: “We will performance the … “ -> perform\n- L159: “… we also tested the nonlinear decoder …” -> a nonlinear decoder\n- L165: “NER outperforms the CEBRA …” -> outperforms CEBRA\n- L165: “… over large margin …” -> by a large margin\n- L166: “A kNN decoder has no improvement over the NER, but …” -> The kNN decoder does not improve NER performance compared to the linear decoder, but for CEBRA and Pi-VAE, the kNN decoder improves within session data performance over NER.\n- L168: “However, this sacrificed …” -> this sacrifices\n- L174: “… NER revealed same …” -> the same\n- L177: “… using linear regression decoder … “ -> using a linear regression decoder\n- L180: “… NER is robust to different decoder …” -> robust to different decoders\n- L180: “… CEBRA and Pi-VAE change from failed state to outperform NER …” -> the performance of CEBRA and Pi-VAE changes from very low performance to outperforming NER on within-session decoding with the kNN decoder.\n- L198: “… when monkey perform …” -> the monkey performs/the monkeys perform\n- L205: “… NER revealed narrower latent dynamics for straight movements …” -> I am not sure what the authors mean by this sentence.\n- L210: “… CEBRA achieves compatible performance …” -> comparable performance\n- Figure 6 caption: “The reference target is same sessions used in …” -> is the same session as used in\n- Figure 6 caption: “… is first session in S1.” -> is the first session\n- Figure 7 caption: “The cartoon from [18].” -> Subfigure a is reproduced from [18].\n\nQUESTIONS:\n1. Why is the generalization from one brain region to another in the same monkey with the exact same trained model important? And why should it work?\n2. I would also be interested as to why the authors believe the model performs relatively well in generalizing from one brain region to another. It is not entirely intuitive to me directly why a model trained on one region should produce latent embeddings for another region that, when rotated with the Procrustes method, obtains latent embeddings that are still linearly related to the task. My question arises because different brain regions (especially S1 compared to PMd) are entirely new neurons, with a new level of abstraction, which would intuitively require atleast some level of fine-tuning the model.\n3. Do you specifically use the CEBRA model that has access to targets during the self-supervised learning (CEBRA-behavior)? This is unclear in the text, and I could not find what type of CEBRA models the authors use given that in the original CEBRA paper [1] they propose both CEBRA time and CEBRA behavior. The same question arrises for the Pi-VAE, which is a method that can be used for both discrete and continuous labels. The proposed method in the paper uses continuous labels, but from L86 it almost sounds like the authors trained the Pi-VAE with discrete labels. If so, this comparison is unfair.\n4. Why is it important that X-Y velocity prediction accuracy is aligned with the direction tuning curve in PMd?\n5. Why are the CEBRA models 16 dimensions instead of 3 for the k-NN decoders?\n\n[1] Schneider, S., Lee, J. H., & Mathis, M. W. (2023). Learnable latent embeddings for joint behavioural and neural analysis. Nature, 617(7960), 360-368.\n\nLIMITATIONS:\nI appreciate the last sentences in the authors’ discussion that indicates where their model fails. I think it would be great if the authors include these results as well in the Appendix.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-19"}