Cross-Modal Representational Knowledge Distillation for Enhanced Spike-informed LFP Modeling
Authors: Eray Erturk
Saba Hashemi
Maryam M. Shanechi
Venue: NeurIPS 2025 poster ()
Keywords: Knowledge distillation
Multi-session models
Neural decoding
Neuroscience
Primary area: neuroscience_and_cognitive_science

Abstract: Local field potentials (LFPs) can be routinely recorded alongside spiking activity in intracortical neural experiments, measure a larger complementary spatiotemporal scale of brain activity for scientific inquiry, and can offer practical advantages over spikes, including greater long-term stability, robustness to electrode degradation, and lower power requirements. Despite these advantages, recent neural modeling frameworks have largely focused on spiking activity since LFP signals pose inherent modeling challenges due to their aggregate, population-level nature, often leading to lower predictive power for downstream task variables such as motor behavior. To address this challenge, we introduce a cross-modal knowledge distillation framework that transfers high-fidelity representational knowledge from pretrained multi-session spike transformer models to LFP transformer models. Specifically, we first train a teacher spike model across multiple recording sessions using a masked autoencoding objective with a session-specific neural tokenization strategy. We then align the latent representations of the student LFP model to those of the teacher spike model. Our results show that the distilled LFP models consistently outperform single- and multi-session LFP baselines in both fully unsupervised and supervised settings, and can generalize to other sessions without additional distillation while maintaining superior performance. These findings demonstrate that cross-modal knowledge distillation is a powerful and scalable approach for leveraging high-performing spike models to develop more accurate LFP models.

Cross-Modal Representational Knowledge Distillation for Enhanced Spike-Informed LFP Modeling
Eray Erturk Saba Hashemi² Maryam M. Shanechi 1233,4
Ming Hsieh Department of Electrical and Computer Engineering
Thomas Lord Department of Computer Science
Alfred E. Mann Department of Biomedical Engineering *Neuroscience Graduate Program University Southern California, Los Angeles, CA
feerturk, saba. hashemi, shanechi (arrechhh@uuuuc.. @usc edu
Abstract
Local field potentials (LFPs) can be routinely recorded alongside spiking activity in intracortical neural experiments, measure a larger complementary spatiotemporal scale of brain activity for scientific inquiry, and can offer practical advantages over spikes, including greater long-term stability, robustness to electrode degradation, and lower power requirements. Despite these advantages, recent neural modeling frameworks have largely focused on spiking activity since LFP signals pose inherent modeling challenges due to their aggregate, population-lever level nature, often leading to lower predictive power for downstream task variables such as motor behavior. To address this challenge, framework that transfers high-fidelity representational knowledge from pretrained multi-session spike transformer models LFP transformer models. Specifically, we first train a teacher spike model across multiple recording sessions using a masked autoencoding objective with session-specific neural tokenization strategy. We then align the latent representations of the student LFP model to those of the teacher spike model. Our results show that the Distilled LFP models consistently outperform single- and multi- LFP baselines in both fully unsupervised and supervised settings, and can generalize to other sessions without additional distillation while maintaining superior performance. These findings demonstrate that cross-modal knowledge distillation is a powerful and scalable approach for leveraging high-performing spike models to develop more accurate I.FP models.
introduce cross- moosl knowledge distillation
Introduction
Recent advances in neural recording technologies have enabled the collection of large-scale neural datasets across multiple subjects and recording sessions and led to many advanced models of neural activity that trained for single recording sessions [1-12. Moreover, access such large-scale datasets has motivated training multi-subject and multi melti-session models of neural activity that can generalize across experimental conditions and tasks (3322 To date, the development of multi- session models has focused on spiking activity given the widespread availability of spike datasets compared other neural modalities such field potentials. Indeed, local field potentials (LFPs) remain underutilized in recent modeling efforts, despite being routinely recorded alongside spikes. Yet, LFP signals can offer complementary modality for neuroscience investigations by measuring the brain at larger spatiotemporal scales compared with neuronal spikes (23 24]. Furthermore, LFPs
are potentially advantageous for brain-computer interfaces (BCIs) 25 27 several reasons. First,
CCorresponiing author: shanechi Qusc.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

they often exhibit greater long-term term stability and robustness as they are less sensitive to small shifts in electrode positions or neuronal loss due to reflecting larger-scale population activity across many Second, they are more consistently available than spikes, particularly in chronic long-term settings where spike recordings often degrade or become unavailable over time Third, they have lower power requirements than spike signals, making them more suitable for real-time applications such as BCIs (20 35
neurons 23 24
28 3Q
[28
31
Despite these advantages, models trained on LFP signals tend to underperform compared to spike- based models in decoding tasks, especially under unsupervised and self-superrvsed training regimes [9|24 34 36 37], which are critical for modeling with unlabeled neural data and for building neural representations that are generalizable across tasks. This gap arises primarily due to inherent challenges in modeling LFP signals. Specifically, LFP signals reflect population-
aggregate activity from complex neural circuits, resulting in difficulties in isolating individual sources of task-relevant neural variability within aggregate signals B8 (39) and leading to redundant and highly correlated spatial patterns [40. In addition, high noise correlations in low-frequency bands of LFP 41
can further
obscure task-relevan information in LFP signals.
To overcome these limitations and enable training more accurate LFP models, we propose cross- modal representational knowledge distillation framework that leverages spike-based transformer models pretrained on an abundant amount of public spike datasets as teacher models to improve the quality of LFP transformer models. We enable our framework to operate in a fully unsupervised setup, which is of primary interest, while also extending to the supervised scenario. We demonstrate that representational knowledge distillation from pretrained models of high-fidelity spikes guides the LFP models toward capturing hehavior- -rreiiiiireeiiivveeeeee features, thus remarkably improving the decoding performance of LFP models while mitigating the effects of redundant and noisy components typically observed in unsupervised LFP training and preserving the generalization properties of LFP signals.
Contributions In summary, we make the following contributions: 1) We develop novel unsuper- vised cross-modaa knowledge distillation framework to transfer representational knowledge from pretrained multi-session spike teacher models to LFP-based student models. Through extensive empirical evaluation across motor cortical data from 6 monkeys that span 3 distinct datasets with LFP and LFP power signals, we demonstrate that distillation significantly improves the downstream de- coding accuracy of LFP models while also maintaining their generalizability to other sessions that are not used for distillation. The distillation gains persist even in an extended supervised setting. 2) We develop an extension that performs multi-session cross-modal knowledge distillation and show that it can further improve downstream decoding performance. 3) We also develop multi-session LFP-only baseline model (MS-LFP) trained on these motor cortical datasets, enabling rigorous evaluation and comparison of LFP models. 4) As an additional contribution and to build our teacher models, we develop an improved multi-session spike (MS-Spike) model by designing new neural tokenization strategies and training optimizations, and demonstrate that it outperforms recent state-of the--arrt baseline (NDT2 [17) on downstream behavior decoding tasks. Taken together, our results highlight cross- modal representational knowledge distillation as a powerful strategy to leverage complementary neural modalities both for investigating cross-modal neural representations and for developing robust and scalable neural decoding models for applications such as BCIs.
2 Related Work
Neural data modeling Many approaches in neural data modeling primarily focus on training models within individual recording sessions, particularly on spiking signals and using deep learning-based
approaches [4 6 12 42 or state-space models (5 43 More recently, multi-session
modeling strategies have been introduced improve generalization across sessions and subjects, employing techniques such as session-stitching (2), contrastive-learning [13), and transformer- based approaches with different neural tokenization strategies using either unsupervised [T6] [7] or supervised [14 15] 44 objectives. To develop our spike teacher models, inspired by these works, we adopted modified neural tokenization approach that led an improved multi-session spike model, which we then used as the teacher network for our novel cross-modal knowledge distillation framework. Beyond spike modeling, recent efforts have developed multi-session modeling frameworks for intracranial electreencephalography (iEEG) signals by applying temporal patch-based tokenization, in some cases combined with spatial embedding of electrode locations, and trained these models using masked autoencoding (MAE) objective [18 19), self-supervised methods 220)

---

or supervised objectives (2T]. However, intracortical LFP signals are distinct from iEEG signals, which cover larger spatial scale of neural activity than LFP due distinct electrode types. For our LFP modalities, we design a session-specific spatial patch tokenization. Given the continuous nature of L.PP recordings and intracranial recordings, we also train our multi-session LFP models under the MAE objective.
Multimodal neural modeling The goal of our cross- modall knowledge distillation is distinct from prior works that model multimodal neural signals to enable information fusion during inference, for example, using autoencoder- based architectures [45] 46 and state-space models [24 30 47- 511 Unlike our framework, the goal of these methods is to collectively model both modalities for information fusion and, as such, they require access to both modalities at inference time. In contrast, our goal is to develop improved unimodal LFP model that operates solely LFP signals after training and during inference. Developing accurate unimodal LFP models is critical both to study larger-scale circuit-level neural representations and because LFPs are the only available modality in many recording scenarios (see Introduction). While our goal is distinct from multimodal decoding, to more comprehensively evaluate our framework, also developed and compared it with multimodal baseline model of spike and LFP signals, and performed inference/decoding either by processing both modalities only unimodal LFPs.
Knowledge distillation Outside neuroscience, knowledge distillation has been widely adopted in deep learning applications to transfer the learned structure from often larger teacher model to a smaller student model [52] with successful application across various domains such as natural
language processing [53] 54 and computer vision [55] through classification-leve logit matching
or intermediary feature- alignment objectives. While most prior works focused on distilling the
knowledge of larger model to smaller model of the same modality, in domains outside neuroscience, knowledge distillation across different modalities has also been studied through alignment- strategies [57 58, including contrastive distillation objectives [59]. In neuroscience, knowledge distillation has recently been used for the distinct goal of learning under privileged information to improve behavior decoding from spiking activity by distilling knowledge from teacher model that observes behavior as privileged knowledge in addition to spikes [60]. However, cross-modal knowledge distillation between neural modalities remains unexplored. Furthermore, is unclear how such knowledge distillation can affect downstream behavior decoding. Here, we develop novel cross-modal representational knowledge distillation framework for neural modalities and show its substantial benefit for learning LFP representations by transferring knowledge from spike models.
3 Methodology
In our setup, we have access to M sessions of neuronal recordings, all containing discrete spiking activity. Among these M sessions of available recordings, only a subset L < MM sessions contain continuous LFP signals (see Table Furthermore, each recording session can differ substantially, including variations in the number of recorded spiking neurons, LFP channels, behavioral variables. and recording durations. Also, in many scenarios, neural data may be unlabeled in some sessions due to lack/difficulty of behavioral measurement or annotation, thus requiring an unsupervised approach. We develop an unsupervised modsl knowledge distillation framework that transfers knowledge about latent neural representations from multi-session spike models into LFP models. We also develop a supervised extension of our framework to show the robustness of our findings.
3.1 Neural signal tokenization
The first step toward our cross-modal knowledge distillation framework involves utilizing the avail- able large-scale spike recordings to build robust and generalizable multi-session multi-subject unsupervised model for spiking activity as the teacher model. This model should capture key neu- ronal dynamics across various experimental tasks and recording setups, thereby facilitating effective knowledge transfer to LFP signals. To achieve that, we develop multi-session spike transformer encoder model. We do so by adopting and modifying the approach in [77, namely NDT2, which uses shared space embeddings across sessions, while accounting for cross-session/suujeet spatial variability via learned session/subject tokens. Different from [17]. we encode spatial variability across sessions via session-specific space embeddings, offering a direct spatial parameterization as detailed below.
3

---

Neural tokenizer module
Mutt-sassion/subjjct neural activity (Spikes or LFP)
Embedding module
sulli-sesssonnsubjeett
Patching module
Multi- sestssiio///seeec
value emoeddinge
Shared
Session speciic space embediings
neural activity tokens
neural activily patches
Einbed
Exarpse
Spate tee
oo (1.0
50000 enbooding byar
Extepplo pain
Sessions
Option
HSpss (D)
12
Tire
spece
Flastening
For OIRPOO
JPP paldi kyyer
Seasln egass ertwnding
LTP twats
Exarale
Option
Spraton
Enarple
Cem:
UPP papcres
(Space D
Figure 1: Tokenization of multi-session spiking activity or LFP signals. First, neural signals are patched across space (i.e., neurons channels) with padding if necessary. Then, the embedding module learns value embeddings for neural signals using the same block that is shared across sessions, as well session-specific space embeddings for each spatial patch. The final token representations are formed by adding the value embeddings (shared across sessions) and session-specific space embeddings. This addition is illustrated by mixing the colors of value and space embeddings.
As shown in Fig. D. the first step is to form token sequences out of multi-session neural recordings.
T. session- } and n' is the number of recorded neurons for session i. As done in NDT2, inspired by image-like like patching of time-series signals [6]] across time, for each recording session i, we first group (patch) S neurons across space (with padding if n² is not divisible by S). Each neural patch later processed as separate token, and S hyperparameter denoting the spatial patch size. Thus, for each f, the tokenizer module generates S neural patches, which increases the total
Assume that we have spike count recordings si E M6' where time-index E
index i E {1, ........ M)
number of tokens (i.e., the sequence length processed by the transformer encoder) from T, Ti T This strictly spatial patching approach allows us to 1) maintain the temporal resolution of the original neural activity and 2) enable the encoder backbone explicitly leverage spatial relationships (e.g.. via attention mechanism) in addition to temporal patterns.
Once patches are formed, for each neural patch, first learn value embeddings using the same block shared across all recording sessions. For discrete spike counts, we learn embedding vectors for each unique spike count value, V (Vo, V ......, V&} where i denotes the maximum spike count allowed (e-g.. 5 spikes
ms bins), each V; E &R and d denotes the final token dimensionality (or latent dimensionality). Thus, we form the value embedding of each patch, denoted by F, E R&d by concatenating the value embedding vectors of each neuron's 's spike count (i.e., V,'s) within that patch.
To account for spatial variability both within and across sessions, we learn session-specific space embedding vectors for each neural patch in every session, Et,p ( Et where each
Ej E Rdd This differs from NDT2, which accounts for spatial differences across sessions by using session- and subject-specific embedding vectors, while space embeddings are shared across sessions (see Appendix appendix[A.6ffom performance comparisons of our multi-session spike model with NDT2 and details of the NDT2 baseline). Finally, we form the token embedding sequence by adding the value and space embedding vectors E, and Ej for each neural patch.
In addition to the multi-session spike model (MS-Spike), we also developed a multi-session model for LFP signals (MS-LFP). This allows us assess the benefit of distilling knowledge from multi-session spike models into LFP models more rigorously. To do so, we extended our patch-based tokenization strategy for LFP signals. However, since LFP signals are continuous-valuee we formed the value embeddings either 1) by passing each neural patch through a shared linear layer or 2) by passing the sequence of neural patches through dilated causal convolutional layer.
3.2 Pretraining and fine-tuning of multi-session models
After the tokenization, we pretrain the multi-session models in an unsupervised manner, whether for spikes or LFP, using an unsupervised masked autoencoding (MAF) objective as shown in Fig. 7 We drop 60% of tokens across space and time, and train the transformer encoder and predictor to

---

reconstruct the neural activity from the unmasked tokens. Then, we use the pretrained tokenizer and transformer encoder extract latent representations.
Once pretraining of multi-session models is complete, we fine-tune on individual recording sessions in unsupervised manner with the same MAH objective (unsupervised setting). For an unseen session, we initialize new spatial embeddings for its neural patches and optimize them during fine- tuning. While pretraining of multi-session models is always unsupervised, to show the robustness of our findings, we also explore performing the fine-tuning by further incorporating a supervised behavior regression objective. To do so, new behavior regression head is coupled to the encoder backbone. For the supervised variant, we jointly optimize MAE and behavior regression losses, whereas only behavior regression loss is optimized for the fully-supervised variant. The above gives us the MS MS-Spike, Spike, which we can now utilize to develop our cross cross-modal knowledge distillation framework. Please also see Appendix A. for additional details on pretraining and fine-tuning.
3.3 Cross-modal knowledge distillation
After training robust and generalizable models of multi-session spiking activity, we transfer knowledge from these high-performance spike models model operating solely LFP signals. We perform the knowledge distillation in the latent representational space to leverage the representational power of pretrained spike models. Specifically, we propose a cross-modal knowledge distillation framework that aligns representations across spiking activity and LFP signals as depicted in Fig. 2
The first step in this approach is fine-tuning the pretrained multi- session spike model by using one of the fine-tuning approaches de-
Finctuned sp rotary trar tarafomer enooder backbonn
Single-cess
Sp Iatent repressrtations eerlly "W"
fined in Section 3.2 either fully unsupervised or with some level of behavior supervision if desired.
Fineluned tokenizer module
LFP rotary transformer encoder bacibone
Then, we initialize a new model for the single-session LFP signal with
Single session
LFP Iatent representationn
the patch tokenization strategy de-
UFP takanizer module
scribed in Section 3.1 and shown in Fig. [] Unlike spike signals, we use a dilated convolutional layer for the value embeddings of LFP signals instead of learnable embed- dings.
Figure 2: Cross-modal knowledge distillation across spiking activity and LFP signals via representation-leve alignment.
Next, to train the LFP model using distillation, we align the latent representations of paired spike and LFP signals- after averaging the representations of spatial patches corresponding to the same timestep-by by maximizing their average cosine similarity. To prevent overfitting to the alignment objective and also allow for inclusion of LFP- spesiiciiiic dynamics, we include an additional autoen- coding loss to reconstruct the observed LFP signals. The final distillation objective combines both components:
T Σ - fo +\ 1=1
1 T 1=1 (f
<
Edistill
1
1
(1)
n
Autvencoding objective
Represeentttiion-leee higgameen//simmiaatti objective
where À the scaling hyperparameter, y, E By and s; E RT denote paired LFP and spike signals, respectively, and <.. denotes inner product. The function fa(-) is predictor linear layer used to reconstruct yı. while fe()) and (.). represent the tokenizer module and transformer encoder backbone stacks for LFP and spike modalities, respectively, Note that the spike model (-()) is kept frozen when optimizing the distillation objective, serving as the teacher in the distillation process (see Appendix A.8.2 for the ablation study on the systematic unfreezing of the teacher MS-Spike model).
Overall, the required steps for the distillation procedure can be summarized as 1) pretraining an MS-Spike model on pretraining sessions, 2) fine-tuning the MS-Spike model on a new session using one of the fine-tuning tasks, depending on the level of supervision, and 3) initializing single-session
5

---

LFP model, then training it on the new session in step 2 via the cross-modal knowledge distillation objective in Eq. [..eeee the teacher model is the fine-tuned MS- MS-Spike model from step 2. During this step, the teacher MS-Spike model is fully frozen.
4 Results
We first pretrained a multi-session spike model (MS-Spike) on 226 sessions collected from 16 subjects across 6 different datasets 62 72
(Table
2) To test generalizability to unseen sessions,
during pretraining, we held out 5 sessions from 62
63 and 3 sessions from [64 for fine-tuning,
which were amongst sessions with paired spike-LFP recordings (standard low-pass filtering was used to extract raw LFP signals in these datasets, see Appendix [A.3.] for details). Additionally, to test generalizability to unseen subjects and datasets, during pretraining, we held out all sessions from [73), which contained paired spike and LFP-power signals (standard power band features were provided in the dataset). After pretraining, we fine-tuned the MS -pike model on the spikes of the held-out sessions and then trained Distilled LFP models using the fine-tuned spike models as teachers.
As our primary comparisons, we compared the performance of knowledge Distilled LFP models with the following transformer models, which had the same architecture as described in Section 3.1
Multi-session LFP models (MS-LFP): These models were pretrained on LFP signals of recording sessions from 62 63 and (64) (25 and 9 sessions, respectively), and then fine-tuned on the LFP signals of sessions that were held out of pretraining (same as the and held-out sessions from MS- MS-Spike pretraining). Because [73] provides LFP-power signals rather than raw LFP signals, we pretrained separate multi-session models on these LFP power signals from 14 sessions across subjects and fine-tuned on 9 held-out sessions from 2 subjects, one of whom was not included in the pretraining data.
Single-session LFP models (SS-LFP): These models were trained independently on the LFP (or LFP power for 73) signals from each individual recording session.
Single-session multimodal models (SS-MM): These models were trained on the concate- nation of spike and LFP (or LFP power) signals from individual sessions, and used both modalities during training and inference.
Single-session multimodal models with zero spikes during inference (SS-MM-ZS): To directly compare with LFP-only models at inference time, we also evaluated the SS-MM models by zeroing out the spike inputs to the tokenizer inmeutiiptttttion after z-scoring), thus having the model process only LFPs while keeping the architecture unchanged.
All models were trained following the setup described in Section 3.J and Appendix A.I
4.1 Cross-modal knowledge distillation significantly improves behavior decoding
As our primary comparisons, compared the behavior decoding performance of the Distilled LFP models in the fully unsupervised setting, where MS-Spike was fine-tuned only with the unsupervised MAE objective and was used teacher for the Distilled LFP models with the objective in Eq. [ All baseline LFP models are also trained (and fine-tuned for MS-LFP) to optimize the MAE objective with linear layer for value embeddings described above.
We found that distillation leads to a remarkable improvement in decoding performance of the LFP signals as shown in Fig. 3 Distilled LFP models (orange line) consistently outperformed the models trained solely on the LFP signals, whether using multi-session (MS-LFP) or single-session (SS-LFP) 51, one-sided Wilcoxon signed-rank
training (p < 2.6 x 10 10 for both MS-LFP and SS-LFP, test). For example, for Monkeys I, C. Ch, and M, respectively, the average behavior decoding performance (1R²) of the Distilled LFP model was 0.66, 0.72, 0.77, and 0.72 (0.71 on average), whereas the MS-LFP model achieved 0.24, 0.34, 0.33, and 0.22 (0.27 on average), and the SS-LFP models underperformed the MS-LFP model (0.24 on average).
Also interestingly, we observed that Distilled LFP models performed similarly to or sometimes better than their teacher MS-Spike models, with the Distilled LFP average performance being higher than that of MS-Spike models (0.71 1 vs. 0.69 on average; p < 1.7 x 10-3, n = 51, one-sided Wilcoxon signed-rank test). Since shared information between the spike and LFP modalities has been shown to

---

Monkey
Monkey Ch
Monkey 20170124 01
00.7
0.8
vel. Vol.
MS-Spike 55-UFP
MS-UFiP
55 MM
4.4
0.5
Distilied LFP + 55 35-MM--5
Time (ms!
Monkey C el_
20109422.00 05
201000n1.07 01
20180015 ot
20101013.03
2017014..0 g1
20180929
20101006
20101007
201010221
Monkay
Monkey Ch
Monkey
0.8
vol val ve. vol.
0.77
0.5
Time imss
0.4
Monkey 20140303
0.2
6.
20140337
20140155
20140305
20140304
20140007
(mm))
Sessions
Sessiors
Figure 3: Behavior decoding performances (IR2) of unsupervised models. (a-d) Decoding results across sessions for individual monkeys (Monkeys I, C with LFP signals, and Monkeys Ch, M with LFP power signals). Error bars around each datapoint indicate standard deviations obtained across 3 different seeds. (e) The true (grey) vs. decoded behavior trajectories (velocities) from the latent representations of Distilled LFP models (orange) and MS-LFP models (blue).
be task-predictive in prior works 23 24 74), this result suggests that Distilled LFP models potentially learn to better extract shared behavior- prreiii-preeiiivveeee information compared with MS-Spiks models due to aligning the LFP representations with their associated spike representations.
To further explore the power of knowledge distillation in extracting informative embeddings from LFPs, trained multimodal SS-MM models and performed inference either with multimodal signals (SS-MM, solid green line), or only with LFP signals by zeroing out the spike signals, i.c., imputing with their global-mean after z-scoring (SS-MM-ZS, dashed green line). We found that SS-MM and SS-MM-ZS achieved 0.33 and 0.27 average decoding performance. respectively. which were again substantially lower than the Distilled LFP models (p < 2.6 10-10, for both SS-MM and SS-MM-ZS = 51, one-sided Wilcoxon signed-rank test). This result indicates that in the fully unsupervised setting, distillation is more successful in learning informative embeddings for LFP signals compared with the concatenation of spike and LFP modalities in the input to the transformer, which presents distinct difficulty of learning all collective dynamics both signals. Indeed, the distillation objective provides principled and scalable approach to unsupervised representation learning by explicitly aligning the representations across modalities.
In addition to the held-out sessions reported above, we also performed the same analysis for the sessions used in the pretraining of MS-Spike and MS-LFP models (25, 9, and 14 sessions for 62 [3]
[64] and 773,, respectively) and found that the Distilled LFP models again improve behavior decoding performance over baseline models for the pretraining sessions (see Fig.
Appendix (..
To assess the scalability of distillation, we trained Distilled LFP models with different model sizes. Fig. 10 shows that even when the Distilled LFP model capacity decreased 10 times compared to the other LFP--ased baselines, the distilled models outperformed all baselines regardless of scale.
Further, in an ablation analysis on the impact of MS-Spike pretraining dataset size (Appendix A.8.3 we found that the performance difference between the MS-Spike and MS-LFP models is not simply due to the dataset size used during pretraining but also to the differences in signal characteristics. Importantly, we found that this gap can be overcome through our cross-modal knowledge distillation.
4.2 Cross-modal knowledge distillation successfully aligns LFP and spike representations
Next, we investigated the alignment of latent representations extracted by MS- MS-LFP, and Distilled LFP models by computing 2-dimensional i-SNE representations from the sequence-averaged latent representations. As shown in Fig. 4 the latent representations of MS-Spike and Distilled L.FP models are clustered together, indicating their successful alignment through the distillation objective, whereas the latent representations of MS-LFP models clustered far away from both. Also, for all
7

---

models, the latent representations of the sessions from the same subject are clustered closer (shown by gray ovals). despite not having explicit learnable subject tokens.
We also quantified the degree of alignment across
Seedion
models at the sequence-level by computing the top-1 I
dorkka 201 20002000.2 E1 andaekey Cut #eoonooy (h 20100320 Morkey 41217
#rankey cour Marssey On J0L0UREA Parlkay 20346007
and top-5 representation retrieval accuracy and the mean rank of the paired spike and LFP sequence representations among all (-3000) sequences, We also computed the alignment at the global represen- tation level by computing the centered kernel align- ment (CKA) [75] between the MS- USS-pi MS MS-LFP, and Distilled LFP models (see Appendix A.5 for de- tails). We observed that MS-Spike and Distilled L.FP representations exhibit a high degree of alignment, achieving 0.67 top-1 aceuracy, 0.90 top-5 accuracy, low mean rank of 4.99, and CKA of 0.89. In contrast, MS-Spike vs. MS-LFP and Distilled LFP vs. MS-LFP pairs yield much worse retrieval perfor- mance in all metrics, as seen in Table I Even though the sequence-level retrieval metries for MS- MS-Spike vs. MS-LFP and Distilled LFP vs. MS-LFP are close to the random baseline, their global-level CKA values (0.70 and 0.76) are relatively high-though much lower than that of MS-Spike vs. Distilled LFP (0.89) suggesting global-level similarities between the spike and LFP representation spaces (see Appendix[AA55 for details). These results indicate that, in addition to improving the quality of LFP models, the cross-modal knowledge distillation framework can also serve as a new tool for aligning and investigating latent structure across different neural modalities at multiple spatiotemporal scales.
eider kea 28:5201.535
201
Montay C
Honkey
@ IES Sp be
XS-UFF
Isanle ab Lif power)
Montey M
Monkey!
YSUP
NS. Sple
Figure 4: t-SNE representations of time- averaged latent representations across MS- Spike, MS-LFP, and Distilled LFP models across recording sessions from subjects held-out for fine-tuning.
55-MMM-25
Monkay I
S5-LIP Ditiiled LFP 20160622.00 01
b
Morkeey
Distilled LFP on el_
Bchavtor decoding (82)
0.6-
85-
0.4
8.
022
201sete c3
2010022 of
musa-7 206c007. m010033) c2
ox
201eer as
ot
andin or moale pe paint 52 mmnor 19 Sanonn pe
az
Soruurt aaaaaa ox
42.
20100221
2014s00s 20dieii
Secsions
Sessiors
Figure 5: Behavior decoding performances (122) of Distilled LFP models on LFP signals of the sessions denoted on the x-axis that were never used in pretraining, fine-tuning, distillation for Monkeys I (a) and C (b).
4.3 Distilled LFP models can generalize to other sessions without additional distillation
To evaluate whether Distilled LFP models can generalize beyond the session used for distillation, we tested their decoding performance on LFP signals from sessions that were never used in the MS-Spike pretraining, fine-tuning, or distillation. To this end, we trained another MS-Spike model where all sessions shown in Fig. 5 were held-out during pretraining. We then fine-tuned this Mpike model separately two of the held-out sessions 20160622 01 for Monkey I and el_ for Monkey C-and trained Distilled LFP models on the LFP signals of these sessions using their corresponding fine-tuned MS- Spike models as teachers. Finally, we froze these Distilled LFP models and evaluated them on the LFP signals of the remaining held-out sessions to assess their generalization performance.
As shown in Fig. 5 Distilled LFP models trained only on single session's 's spike-LFP alignment (e.g., session 20160622_00 01 for Monkey I in panel a and session el_ for Monkey C in panel b) substantially outperform all other LFP baselines, including MS-LFP, SS-LFP, and SS-MM-ZS, which were trained (or fine-tuned, in the case of MS-LFP) on the tested sessions' L.PP signals. This result suggests that the distillation objective enables the transfer of the teacher MS-Spike model's prior
8

---

knowledge from its pretraining distribution into the Distilled LFP models, allowing them to leverage spike-aligned representations even on the held-out tested sessions, despite no spike or LFP data from those sessions being used during distillation, which was performed on a different held-out session.
4.4 Cross-modal knowledge distillation improves performance even in the supervised fine-tuning setting
Our primary goal is develop a fully unsupervised distillation framework, motivated by the broader generalization capabilities of unsupervised models and/or the sparsity of behavior measurements or annotations in many datasets. Nevertheless, we also examined whether cross cross-modal distillation improves performance even in the supervised case. In this case, multi- MS- MS-Spike and MS-LFP models are fine-tuned with a supervised variant, where we jointly optimize the MAE and behavior regression objectives. Also, all single-session models are trained with the behavior regression objective. We used a convolutional layer for value embeddings for all LFP models. The Distilled LFP model was trained with the objective in Eq. [ Jusing the above supervised teacher spike models.
As shown in Fig. 6 Distilled LFP models again consistently outperformed all LFP-only baselines, including MS-LFP, SS-LFP, and SS- MM-ZS, in behavior decoding performance (R²). As expected, unlike in the unsupervised setting, the Distilled LFP models did not sur- pass the spike-based models (MS-Spike and SS-MM), which served as an upper bound on the decoding performance. This is likely due to the direct supervision of spike models with behavior, which helps the spike model uncover more behavior-predictiv embeddings than in the unsupervised case. However, as noted above, the Distilled LFP models significantly outperformed all LFP- baselines (p 2.6 10 for MS-LFP, p < 2.9 x 10 10 for SS-LFP, and p < 2.6 x 10 10 for SS-MM-ZS, n = 51, one-sided Wilcoxon signed-rank test). These results show that the distillation framework benefits LFP modeling even when supervision with behavior is performed for fine-tuning, leading to powerful LFP models.
Model Pairs
Top-1
Top-5
Mean rank
CKA
MS-Spike Distilled LFP MS-Spike MS-LFP
0.6711
0.9041
4,99
0.89
0.0007
0.0023
1289.15
0.70
Distilled LFP MS-LFP
0.0003
0.0023
1265.57
0.76
Random
0,0003
0.0013
1513.42
0.07
Table I: Representation retrieval metrics across pairs of MS-Spike, MS-LFP, and Distilled LFP models. The last row demonstrates the metrics computed across random representations.
We also tested the distillation performance in this supervised setting for the recording sessions used in pretraining of MS- Apike and MS-LFP models. We found that the Distilled LFP models again improved downstream decoding performance over all LFP-only baselines (see Fig. 12 in Appendix A. As an additional analysis, we also trained Distilled LFP models fully-supervised setting by replacing the autoencoding objective with the behavior regression objective in Eq. ann by fine- tuning the teacher MS-spike models fully-supervised. As shown in Figs. 13 and 14 fully-supervised distillation can further improve downstream decoding performance compared to baseline methods.
4.5 Multi-session cross-modal knowledge distillation further improves performance
Next, we trained the distilled LFP models in multi multi-session setup (MS-Distille LFP) to investigate whether multi-session distillation further improves downstream decoding performance over single- session distillation. In this setup, the fully unsupervised distillation objective in Eq. | was optimized on pooled spike-LFP data across multiple sessions rather than on just the target session, and the distilled model was then fine-tuned on the behavior-labeled target session (see Appendix A212). This setting is particularly important in practical scenarios where large amounts of unlabelled paired sFike-LFP data available across multiple sessions subjects, but behavior labels are limited to small subset of sessions. Through this approach, we aim to leverage shared structure across sessions to obtain more accurate and generalizable initialization even beyond single-session distillation.
Extending to multi session distillation provided robust gains over aigle-session Distilled LFP models. MS- LFP models significantly outperformed Distilled LFP models for supervised fine-tuning (0.82 vs. 0.80, p < 1.3 x 10-4, n = 51, one-sided Wilcoxon signed-rank test), and fully-supervised fine-tuning (0.83 vs. 0.81, p < 1.3 x 10-6,
51, one-sided Wilcoxon signed-rank test) strategies as shown in Figs. T6. 17 while achieving comparable performance for the unsupervised fine-tuning
setting (0.72 vs. 0.7 (Fig. 15). Also, as shown in Figs. 18
19
20 similar results held for the

---

Monkey
Hurkey Ch
Monbey 201170224 01
0.80
0.75-
0.05-
0.80
Vel Vol.
0.70
8.65
0.81
KS-UFP
MS Spike
S5-LFP
HM
0.82
No Norkcy C el
Time (mo)
0.50-
Disilled
+ 85-MM--5
0.45
20166622.9 St
20160915 oi
20161013 ot
2017014..00 ot
20168009
20181186
20101007
20161021
20160000,
Vel.
Time Emayy
Horkey
Honkey N
29 0.55
0.90
0.00
0.58
Vel. Vel
0.80
0.66
0.70
082
Monteyy 20180103
Vel Vel
8.50
05
20140217
20140216
20140107
20140304
20140007
Time (IT))
Sessions
Figure 6: Behavior decoding performances (R²) of supervised models. Figure conventions the same as in Fig.
recording sessions used in pretraining of MS-Spike, MS-LFP, and MS-Distilled LFP models. Overall, these results highlight that multi-session distillation can leverage large-scale unlabeled neural data to build more accurate and generalizable LFP models.
5 Discussion
Here, we developed cross-modal representational knowledge distillation framework to build more accurate LFP models by leveraging high-performing spike-based models as teachers. To achieve that, we first developed an improved multi-session spike model with new neural tokenization strategy and pretrained it on publicly available motor cortical datasets. Then, we transferred the representational knowledge of spike models to LFP models through a representation-leye alignment also developed multi-session LFP-only
objective. To enable rigorous evaluations of our approach, baseline trained on motor cortical LFP signals. Our results demonstrate that cross-modal distillation significantly improves the representation quality of the LFP models, both in fully unsupervised and supervised settings, even when using distilled models with ten times fewer parameters. In addition, we showed that cross-man modal distillation be important scientific tool for aligning and investigating latent structure across multiple spatiotemporal scales of brain activity. Moreover, we showed that the distilled models can generalize LFP signals from other sessions without additional distillation, still achieving superior decoding performance compared to LFP baselines. Finally, we showed that extending the distillation approach multi-session setting can further improve decoding performance.
There several areas for future investigation. First, while incorporated substantial pretraining data, extending pretraining to incorporate larger and more diverse multi-session datasets may reveal richer latent representations across sessions, ultimately enabling more diverse evaluations and improving generalization and robustness. Second, investigating more diverse pretraining tasks beyond masked autoencoding is important future direction potentially improve the quality of the distilled models. Third, this distillation approach can be tested for knowledge transfer beyond mross-modal transfer, e.g., for transfer between different brain regions. Moreover, multimodal models represent promising complementary direction when both modalities are available at inference time, as they can leverage information from both spike and LFP signals potentially surpass unimodal performance. While we compared to a multimodal approach as a baseline, our simple proof-of- concept multimodal baseline could be extended into more sophisticated architectures the future for better multimodal performance. Nevertheless, spike signals degrade faster than L.P signals in BCIs (e.g., due to electrode tip encapsulation) 28 781, the Distilled LFP models provides powerful approach to maintain high BCI performance over long time-periode given the higher stability LFPs, thus improving the robustness and longevity of BCIs. Finally, while our results highlight the potential of L.P--baedd models as robust and accessible alternatives to spike-based models, confirming their reliability as full replacements requires further experimental validation under controlled signal degradation conditions.
10

---

Acknowledgements
This work was partly supported by the National Institutes of Health (NIH) grants RFIDA056402. R01MH123770 and R61MH135407, the NIH Director's New Innovator Award DP2-MH1 and the National Science Foundation (NSF) CRCNS program award IIS- 13271. We sincerely thank Danil Tyulmankov and Enes Burak Bilgin for their feedback, and all members of the Shanechi lab for helpful discussions.
References
[1] Yuanjum Gao, Evan W Archer, Liam Paninski, and John P Cunningham, Linear dynamical neural population models through nonlinear embeddings. In Advances in Neural Information Processing Systems, volume 29, Curran Associates, Inc., 2016.
[2] Chethan Pandarinath, Daniel J. O' Shea, Jasmine Collins, Rafal Jozefowicz, Sergey D. Stavisky, Jonathan C. Kao, Eric M. Trautmann, Matthew T. Kaufman, Stephen Ryu, Leigh R. Hochberg, Jaimie M. Henderson, Krishna V. Shenoy, L. F. Abbott, and David Sussillo. Inferring single-trial neural population dynamies using sequential auto-encoders. Nature Methods, 15(10):805 815, October 2018. Number: 10.
[3] Qi She and Anqi Wu. Neural Dynamics Discovery via Gaussian Process Recurrent Neural Networks. In Proceedings of The 35th Uncertainty in Artificial Intelligence Conference, pages 464. PMLR, August 2020. ISSN: 2640-3498.
[4] Joel Ye and Chethan Pandarinath. Representation learning for neural population activity with Neural Data Transformers. Neurons, Behavior, Data analysis, and Theory, 5(3), August 2021. arXiv:2108.cm 0102100 [es, q-bio].
[5] Omid G. Sani, Hamidreza Abbaspourazad, Yan T. Wong, Bijan Pesaran, and Maryam M. Shanechi. Modeling behaviorally relevant neural dynamics enabled by preferential subspace identification. Nature Neuroscience, 24(1): 110-14004999
January 2021.
[6] Cole Hurwitz, Akash Srivastava, Kai Xu, Justin Jude, Matthew Perich, Lee Miller, and Matthias Hennig. Targeted Neural Dynamical Modeling. In Advances in Neural Information Processing Systems, volume 34, pages 29379-29392. Curran Associates, Inc., 2021.
[7] Ran Liu, Mehdi Azabou, Max Dabagia, Chi-Heng Lin, Mohammad Gheshlaghi Azar, Keith Hengen, Michal Valko, and Eva Dyer. Drop, Swap, and Generate: A Self-Supervised Approach for Generating Neural Activity. In Advances in Neural Information Processing Systems, volume 34, pages 10587-10599. Curran Associates, Inc., 2021.
[8] Trung Le and Eli Shlizerman. STNDT: Modeling Neural Population Activity with Spatiotem- poral Transformers. Advances Neural Information Processing Systems, 15:17926-17939, December 2022.
[9] Hamidreza Abbaspourazad, Eray Erturk, Bijan Pesaran, and Maryam M. Shanechi. Dynamical flexible inference of nonlinear latent factors and structures in neural population activity. Nature Biomedical Engineering, 8(1) 8(1):85-108, January 2024. Publisher: Nature Publishing Group.
[10] Parsa Vahidi, Omid G. Sani, and Maryam M. Shanechi. Modeling and dissociation of intrinsic and input-driven neural population dynamics underlying behavior. Proceedings of the National Academy of Sciences, 121(7)=e212288712) 2024.
[11] Omid G. Sani, Bijan Pesaran, and Maryam M. Shanechi. Dissociative and prioritized modeling of behaviorally relevant neural dynamics using recurrent neural networks. Nature Neuroscience, 27(10):00332045, October 2024. Publisher: Nature Publishing Group.
[12] Parsa Vahidi, Omid G. Sani, and Maryam Shanechi. BRAID: Input-driven nonlinear dynamical modeling of behavoraa data. The Thirteenth International Conference on Learning Representations, 2025.
11

---

[13] Steffen Schneider, Jin Hwa Lee, and Mackenzie Weygandt Mathis. Learnable latent embeddings for joint behavioural and neural analysis. Nature, 617(7960):360-3665 May 2023. Publisher: Nature Publishing Group.
[14] Mehdi Azabou, Vinam Arora, Venkataramana Ganesh, Ximeng Mao, Santosh B. Nachimuthu, Michael Jacob Mendelson, Blake Aaron Richards, Matthew G. Perich, Guillaume Lajoie, and Eva L., Dyer. A Unified, Scalable Framework for Neural Population Decoding. November 2023.
[15] Mehdi Azabou, Krystal Xuejing Pan, Vinam Arora, Ian Jarratt Knight, Eva Dyer, and Blake Aaron Richards. Multi-session, multi-task neural decoding from distinct cell-types and brain regions. October 2024.
[16] Yizi Zhang, Yanchen Wang, Donato M. Jiménez-Benetó. Zixuan Wang, Mehdi Azabou, Blake Richards, Renee Tung, Olivier Winter, The International B. Laboratory, Eva Dyer, Liam Paninski, and Cole Hurwitz. Towards a "Universal Translator" for Neural Dynamics at Single- Cell, Single-Spike Resolution. Advances in Neural Information Processing Systems, 37:80495- 80521, December 2024.
[17] Joel Ye, Jennifer Collinger, Leila Wehbe, and Robert Gaunt. Neural Data Transformer 2: Multi- context Pretraining for Neural Spiking Activity. Advances in Neural Information Processing Systems, 36:80352-8033 80374,, December 2023.
[18] Daoze Zhang, Zhizhang Yuan, Yang Yang, Junru Chen, Jingjing Wang, and Yafeng Li. Brant: Foundation Model for Intracranial Neural Signal. November 2023.
[19] Christopher Wang, Vighnesh Subramaniam, Adam Uri Yaari, Gabriel Kreiman, Boris Katz, Ignacio Cases, and Andrei Barbu. BrainBERT: Self-supervised representation learning for intracranial recordings. September 2022.
[20] Geeling Chau, Christopher Wang. Sabera Talukder, Vighnesh Subramaniam, Saraswati Soedarmadji, Yisong Yue, Boris Katz, and Andrei Barbu. Population Transformer: Learning Population-level Representations of Neural Activity. October 2024.
[21] Georgios Mentzelopoulos, Evangelos Chatzipantazis, Ashwin G. Ramayya, Michelle Hedlund, Vivek Buch, Kostas Daniilidis, Konrad Kording, and Flavia Vitale. Neural decoding from stereotactic EEG: accounting for electrode variability across subjects. November 2024.
[22] Functional connectomics spanning multiple areas of mouse visual cortex. Nature, 640(8058):435 447, April 2025. Number: 8058 Publisher: Nature Publishing Group.
[23] Bijan Pesaran, Martin Vinck, Gaute T. Einevoll, Anton Sirota, Pascal Fries, Markus Siegel, Wilson Truccolo, Charles E. Schroeder, and Ramesh Srinivasan. Investigating large-scale brain dynamics using field potential recordings: analysis and interpretation. Nature Neuroscience, Publisher: Nature Publishing Group.
21(7):903 919, July 2018.
[24] Hamidreza Abbaspourazad, Mahdi Choudhury, Yan T. Wong, Bijan Pesaran, and Maryam M. Shanechi. Multiscale low-dimensional motor cortical state dynamics predict naturalistic reach- and-grasp behavior. Nature Communications, 12(1))607, January 2021. Number: 1 Publisher: Nature Publishing Group.
[25] Krishna V Shenoy and Jose M Carmena. Combining decoder design and neural adaptation in brain-machine interfaces. Neuron, 84(4):665-68) 680, 2014.
[26] Maryam M Shanechi. Brain-machine interfaces from motor to mood. Nature neuroscience, 22110):154-11544 2019.
[27] Lucine I. Oganesian and Maryam M Shanechi. Brain-computer interfaces for neuropsychiatric disorders. Nature Reviews Bioengineering, 1:653-670, 2024.
[28] Dong Wang, Qiaosheng Zhang, Yue Li, Yiwen Wang, Junming Zhu, Shaomin Zhang, and Xiaoxiang Zheng. Long-term decoding stability of local field potentials from silicon arrays in primate motor cortex during 2D center out task, Journal of Neural Engineering, 113):036009,, May 2014. Publisher: IOP Publishing.
12

---

[29] Sergey D. Stavisky, Jonathan C. Kao, Paul Nuyujukian, Stephen I. Ryu, and Krishna V. Shenoy. high performing maammme interface driven by low-frequency local field potentials alone and together with spikes. Journal of Neural Engineering, 12(3):036009, May 2015. Publisher: IOP Publishing.
[30] Han-Lin Hsich, Yan T. Wong, Bijan Pesaran, and Maryam M. Shanechi. Multiscale modeling and decoding algorithms for spike-field activity. Journal of Neural Engineering, 16(1):016018, December 2018
[31] Andrew Jackson and Thomas M. Hall. Decoding Local Field Potentials for Neural Interfaces. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 25(10):1705-177 October 2017.
[32] Eric Drebitz, Bastian Schledde, Andreas K. Kreiter, and Detlef Wegener. Optimizing the Yield of Multi-Unit Activity by Including the Entire Spiking Activity. Frontiers in Neuroscience, 13, February 2019. Publisher: Frontiers.
[33] Nur Ahmadi, Timothy G. Constandinou, and Christos-Savee Bouganis. Inferring entire spiking activity from local field potentials. Scientific Reports, 11(1):19045, September 2021. Publisher: Nature Publishing Group.
[34] Parima Ahmadipour, Omid G Sani, Bijan Pesaran, and Maryam M Shanechi. Multimodal subspace identification for modeling discrete-contimuous spiking and field potential population activity. Journal of Neural Engineering, 21(2):026001, March 2024. Publisher: IOP Publishing.
[35] Mare W. Slutzky and Robert D. Flint. Physiological properties of brain-machine interface input signals. Journal of Neurophysiology, 118(2): (202))):329-144) 1343, August 2017. Publisher: American Physiological Society.
[36] Arjun K. Bansal, Carlos E. Vargas-Irwin, Wilson Truccolo, and John P. Donoghue. Relationships among low- frequencycc local field potentials, spiking activity, and three-dimensional reach and grasp kinematics in primary motor and ventral premotor cortices. Journal of Neurophysiology, April 2011. Publisher: American Physiological Society.
[37] Nur Ahmadi, Timothy G. Constandinou, and Christos-Savvar Bouganis. Decoding Hand Kinematics from Local Field Potentials Using Long Short-Term Memory (LSTM) Network. In 2019 9th International JEEE/EMBS Conference on Neural Engineering (NER), pages 415-419, March 2019. ISSN: 1948-3554.
[38] György Buzsáki. Costas A. Anastassiou, and Christof Koch. The origin of extracellular fields and currents - EEG, ECoG, LFP and spikes. Nature Reviews Neuroscience, 13(6):407-220, June 2012. Number: 6 Publisher: Nature Publishing Group.
[39] Pablo Saatinez-Cañadaa Shahryar Noei, and Stefano Panzeri. Methods for inferring neural circuit interactions and neuromodulation from local field potential and electroenceehalloramm measures. Brain Informatics, 8(1):27, December 2021.
[40] Nur Ahmadi, Timothy G Constandinou, and Christos-Savee Bouganis. Robust and accurate decoding of hand kinematics from entire spiking activity using deep learning. Journal of Neural Engineering, 18(2):22001,, February 2021. Publisher: IOP Publishing.
[41] Andrei Belitski, Arthur Gretton, Cesare Magri, Yusuke Murayama, Marcelo A. Montemurro, Nikos K. Logothetis, and Stefano Panzeri. Low-Frequency Local Field Potentials and Spikes in Primary Visual Cortex Convey Independent Visual Information. Journal of Neuroscience, 28622):696-5709, May 2008 Publisher: Society for Neuroscience Section: Articles.
[42) Sayed Mohammad Hosseini and Maryam Shanechi. Dynamical modeling of behaviorally rele- vant spatiotemporal patterns in neural imaging data. In Forty-second International Conference on Machine Learning. 2025.
[43] Scott Linderman, Matthew Johnson, Andrew Miller, Ryan Adams, David Blei, and Liam Paninski. Bayesian Learning and Inference in Recurrent Switching Linear Dynamical Systems. In Proceedings of the 20th International Conference on Artificial Intelligence and Statistics, pages 914-922, PMLR, April 2017. ISSN; 2640-3498.
13

---

[44] Ran Liu, Mehdi Azabou, Max Dabagia, Jingyun Xiao, and Eva L. Dyer. Seeing the forest and the tree: Building representations of both individual and collective dynamics with transformers. October 2022.
[45] Daniel Kramer, Philine L. Bommer, Carlo Tombolini, Georgia Koppe, and Daniel Durstewitz. Reconstructing Nonlinear Dynamical Systems from Multi-Modal Time Series. In Proceedings of the 39th International Conference on Machine Learning, pages 11613-11633 1163.. PMLR, June 2022. ISSN: 2640-3498.
[46] Rabia Gondur, Usama Bin Sikandar, Evan Schaffer, Mikio Christian Aoi, and Stephen L. Keeley. Multi-modal Gaussian Process Variational Autoencoders for Neural and Behavioral Data. October 2023.
[47] Liam Paninski, Yashar Ahmadian, Daniel Gil Ferreira, Shinsuke Koyama, Kamiar Rahnama Rad, Michael Vidne, Joshua Vogelstein, and Wei Wu. A new look at state-space models for neural data. Journal of Computational Neuroscience, 29(1):107-126, August 2010.
[48] Mehdi Aghagolzadeh and Wilson Truccolo. Latent state-space models for neural decoding. In 2014 36th Annual International Conference of the IEEE Engineering in Medicine and Biology Society, pages 3033-3036, August 2014. ISSN: 1558-4615.
[49] Hamidreza Abbaspourazad, Han-Lin Hsieh, and Maryam M. Shanechi. A multiscale dynamical modeling and identification framework for spike- field activity. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 27(6): 1138, 2019.
[50] Anirudh Wodeyar, Mark Schatza, Alik Widge, Uri T Eden, and Mark Kramer. A state space modeling approach to real-tine time phase estimation. eLife, 10:c68803, September 2021. Publisher: eLife Sciences Publications, Ltd.
[51] Christian Song. Hin-Lin Hsich, Bijan Pesaran, and Maryam M Shanechi. Modeling and inference methods for switching regime-dependent dynamical systems with multiscale neural observations. Journal of Neural Engineering, 19(6):066119, November 2022. Publisher: IOP Publishing.
[52] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the Knowledge in a Neural Network, March 2015. mrXiv:1503.02531 [stat].
[53] Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter, March 2020. arXiv:1910.01 108 [cs].
[54] Yuxian Gu, Li Dong, Furu Wei, and Minlie Huang. MiniLLM: Knowledge Distillation of Large Language Models. October 2023.
[55] Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. FitNets: Hints for Thin Deep Nets, March 2015. arXiv:1412.6550 [es].
[56] Sergey Zagoruyko and Nikos Komodakis. Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfer, February 2017. arXiv:1612.03922 [cs].
[57] Saurabh Gupta, Judy Hoffmman, and Jitendra Malik. Cross Modal Distillation for Supervision Transfer, November 2015, arXiv: MoXiv:1507.004 [cs].
[58] Yusuf Aytar, Carl Vondrick, and Antonio Torralba. SoundNet: Learning Sound Representations from Unlabeled Video. In Advances in Neural Information Processing Systems, volume 29, Curran Associates, Inc., 2016.
[59] Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive Representation Distillation, January 2022. 1xXiv:1910.10699 [cs].
[60] Zhengrui Guo, Fangxu Zhou, Wei Wu, Qichen Sun, Lishuang Feng, Jinzhuo Wang, and Hao Chen. BLEND: Behavior-guided Neural Population Dynamies Modeling via Privileged Knowledge Distillation. October 2024.
14

---

[61] Yuqi Nie, Nam H. Nguyen, Phanwadee Sinthong, and Jayant Kalagnanam. A Time Series is Worth 64 Words: Long-term Forecasting with Transformers. September 2022.
[62] Joseph E. O'Doherty, Mariana M. B. Cardoso, Joseph G. Makin, and Philip N. Sabes. Nonhuman Primate Reaching with Multichannel Sensorimotor Cortex Electrophysiology May 2017.
[63] Joseph G Makin, Joseph E O' Doherty, Mariana M Cardoso, and Philip N Sabes. Superior uum-movement decoding from cortex with new, unsupervised-leaare Neural Engineering, (5())026010,, January 2018. Publisher: IOP Publishing.
algorithm. Journal of
[64] Robert D. Flint, Eric W. Lindberg, Luke R. Jordan, Lee E. Miller, and Mare W. Slutzky, Accurate decoding of reaching movements from field potentials in the absence of spikes. Journal of Neural Engineering, 94):046600 June 2012. Publisher: IOP Publishing.
[65] Matthew G. Perich and Lee E. Miller. Long-term recordings of motor and premotor cortical spiking activity during reaching in monkeys.
[66] Matthew G. Perich and Lee E. Miller. Altered tuning in primary motor cortex does not account for behavioral adaptation during force field learning. Experimental Brain Research, 235(9):2689-2700) September 2017.
[67] Matthew G. Perich, Juan A. Gallego, and Lee E. Miller. A Neural Population Mechanism for Rapid Learning. Neuron, 100(4))644-97..7, November 2018.
[68] Juan A. Gallego, Matthew G. Perich, Raeed H. Chowdhury, Sara A. Solla, and Lee E. Miller Long-term stability of cortical population dynamics underlying consistent behavior. Nature Neuroscience, 23(2):260-27) 270, February 2020.
[69] Joshua I. Glaser, Matthew G. Perich, Pavan Ramkumar, Lee E. Miller, and Konrad P. Kording. Population coding of conditional probability distributions in dorsal premotor cortex. Nature Communications, 1788, May 2018.
[70] Mark M. Churchland, John P. Cunningham, Matthew T. Kaufman, Justin D. Foster, Paul Nuyujukian, Stephen I. Ryu, and Krishna V. Shenoy. Neural population dynamics during reaching. Nature, (87(7405):51-555 July 2012. Publisher: Nature Publishing Group.
[71] Raeed H Chowdhury, Joshua Glaser, and Lee Miller, Area of primary somatosensory cortex encodes kinematics of the whole arm. eLife, 9:e481 (848819,,0 January 2020. Publisher: eLife Sciences Publications, Ltd.
[72] Xuan Ma, Fabio Rizzoglio, Kevin I. Bodkin, Eric Perreault, Lee E Miller, and Ann Kennedy. Using adversarial networks to extend brain computer interface decoding accuracy over time. eLife, 12:e84296, August 2023, Publisher: eLife Sciences Publications, Ltd.
[73] Cecilia Gallego-Carracedo, Matthew G Perich, Raeed H Chowdhury, Lee E Miller, and Juan ÁI- varo Gallego. Local field potentials reflect cortical population dynamics in region-specific and frequency-dependent manner. eLife, 11:773155, August 2022. Publisher: eLife Sciences Publications, Ltd.
[74] Steffen Katzner, Ian Nauhaus, Andrea Benucci, Vincent Bonin, Dario L. Ringach, and Matteo Carandini. Local Origin of Field Potentials Visual Cortex. Neuron, 61(1):35 41, January 2009. Publisher: Elsevier.
[75] Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of Neural Network Representations Revisited, July 2019. arXiv:1905.004 104144 [cs].
[76] Gaurav Sharma, Nicholas Annetta, David Friedenberg, Tony Blanco, Daphne Vasconcelos, Ammar Shaikhouni, Ali R. Rezai, and Chad Bouton. Time Stability and Coherence Analysis of Multiunit, Single-Unit and Local Field Potential Neuronal Signals in Chronically Implanted Brain Electrodes. Bioelectronic Medicine, 2(1):63-71, June 2015. Publisher: BioMed Central.
[77] Robert D. Flint, Michael R. Scheid, Zachary A. Wright, Sara A. Solla, and Marc W. Slutzky. Long-Term Stability of Motor Cortical Activity: Implications for Brain Machine Interfaces and Optimal Feedback Control. Journal of Neuroscience, 36(12):3623-36) March 2016. Publisher: Society for Neuroscience Section: Articles.
15

---

[78] Dustin A. Heldman and Daniel W. Moran. Chapter 20 Local field potentials for BCI control. In Nick F. Ramsey and José del R. Millán. editors, Handbook of Clinical Neurology. volume 168 of Brain- Brain-Comppen Computer Interfaces, pages 279-288. Elsevier, January 2020.
[79] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked Autoencoders Are Scalable Vision Learners, December 2021. arXiv:2111.00
[cs].
[80] Mingyue Cheng, Qi Liu, Zhiding Liu, Hao Zhang, Rujiao Zhang, and Enhong Chen. TimeMAE: Self-Supervised Representations of Time Series with Decoupled Masked Autoencoders, March 2023. arXiv:2303.03320 [cs].
[81] Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu. RoFormer: Enhanced Transformer with Rotary Position Embedding, November 2023. arXiv:2104.09864 [cs].
[82] Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. FlashAttention Fast and Memory-Efficient Exact Attention with IO-Awareness, June 2022. arXiv:2205.141: 14135 [cs].
[83] Ilya Loshchilov and Frank Hutter. Decoupled Weight Decay Regularization, January 2019. arXiv:1711.05 [cs].
[84] Carolina
Filipe, Mahmoud Elmakki,
and
Guilherme
Costa-Ferreira.
gilluut..co/cctiiiube/enttttttt 4 2025.
16

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: All claims made in the abstract and introduction accurately reflect the paper's contributions and scope, and are backed by our results presented throughout the manuscript.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Several limitations of the work are discussed in the Discussion section. We report all dataset details, computational requirements, and used resources in the Appendix.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sellsspeciiiationn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
17

---

Answer: [NA]
Justification:
No theoretical results are included in this work.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: We used publicly available datasets in this study, and clearly explained the preprocessing details in the Appendix. We also provide link to our GitHub repository in the Appendix that includes models and an inference notebook for reproducibility. Further, we also provide the hyperparameters and training details in the Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
18

---

5. Open access to data and code
Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: We used publicly available datasets in this study, and clearly explained the preprocessing details the Appendix. We also provide link to our GitHub repository in the Appendix that includes models and an inference notebook for reproducibility. Further, we provide the hyperparameters and training details in the Appendix.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://miis.ce))
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ //nips. httpppscccpbbllcggguuuccp / CroodeSSbmsssiooni. ssiliii for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e-g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The dataset details, hyperparameters, and other training-related details of our model and benchmark methods are specified throughout the text and in the Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We provide error bars and standard deviations for figures and tables. We provide statistical significance levels for comparisons throughout the text.
Guidelines:
The answer NA means that the paper does not include experiments.
19

---

The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (c.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We provided the computational requirements and used resources in the Ap- pendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics htttt/////arrrps..c/////////ttte
Answer: [Yes]
Justification:
Our work conforms with the NeurlPS Code of Ethics in every respect.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
20

---

Justification: We discuss the impact of our work that can improve the accuracy and robustness of LFP-based neurotechmologies, e.g., brain-computer interfaces, that can enable more stable and robust neural decoding frameworks. We also discuss the utility of our framework as tool for neuroscience investigations to align and study neural representations across different modalities. We do not expect any negative societal impacts of our work.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our work has no risk of misuse. Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: The creators of the datasets we used in our study are properly credited. We also provided appropriate references to the creators of the baseline NDT2 method, on whose methodology improved our teacher MS-Spike model.
21

---

Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re- ee-ackagedd both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: All assets such single-session and multi-session LFP baseline meth- ods, teacher multi-session spike models, single- and multi-session cross rross-modal knowledge distillation approach across spike and LFP modalities well documented throughout the Introduction, Methodology, and Appendix.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: Our work does not include crowdsourcing experiments or research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
Institutional review board (IRB) approvals or equivalent for research with human subjects
22

---

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Our work does not include crowdsourcing experiments research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Answer: [NA]
Justification: The core methodology, scientific rigor, and originality of our work do not include any LLM usage.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non standtandard components.
Please refer to our LLM policy (https : /ttpuri///euips... scc/Confereenee/222222 /LLM) for what should or should not be described.
23

---

A Appendix
A.1 Details of pretraining and fine-tuning of the multi-session models.
Pretraining To allow for the training of generalizable and robust foundational model of multi- session neural activity, we employed masked autoencoding (MAE) as the unsupervised training objective due to its success in various applications using transformer architectures [17 79 80].
Specifically, we randomly drop 3% (e.g., 60%) of the neural patch tokens (after the tokenizer module) before passing them through the transformer encoder backbone. Note that the token dropping is not only performed across time, but also across space, allowing the model to reconstruct the missing information from both temporal and spatial contexts. Then, we append learnable mask tokens to the token sequence at the positions of the dropped patches and add their corresponding space embeddings to the mask tokens to inform the model of their spatial identity. The resulting token sequence is then passed through a predictor transformer, and the output representations at the masked positions are treated reconstructions of the dropped patches. Both the encoder and predictor use transformer architectures with rotary positional embeddings [8]] (see the ablation study in Appendix [A.8.] for details about choice of using rotary transformers as the encoder backbone architecture). As the training objective, we employed a Poisson negative oog-likelhhood (NLL) for discrete spiking activity and mean-squared error (MSE) for continuous LFPs. During loss computation, any padded dimensions introduced during spatial patching are excluded. Further, to train multi-session (MS) distilled LFP models, we optimized the training objective in Eq. across multiple recording sessions rather than on individual recording sessions (see Appendix A.12 for details).
EnstrainingFFnetunnnng
Objective:
Unsupervised masked autoencoding
Poisson NLL or MSE
Mult session/sutject neural activity tokens
Multi session/subjjct neural activity
Rolary baansoormer encoder backbone
latent representations
Multi sessionnsubjecct
Rotary trans/omee predictor backbone
rost Iohnn
(Spikes LFP)
Sp plwcaaa
Predicted masked patches
Sessions BOAA
Neural tokenizer module | Add new space emteddding for new aideiios
12 2
-Ben ore
Finetuning (optional):
Ob MSF
Supervised behavior regression
Linear
12.23 Tu-T
layer
Predicted
belaavior
12
Figure 7: Overall architecture of the proposed multi-session model. As the self-supervised pretraining objective, we employ masked autoencoding (MAE) to allow for learning generalizable representations across datasets recorded during diverse experimental tasks. During fine-tuning on a single session, optionally incorporate supervised behavior regression. MSE: mean-squared error. NLL: negative log-likelihood. Rep.: Representation.
Fine-tuning After pretraining the multi-session model with the unsupervised MAE objective, we primarily fine-tune the model in unsupervised setting. This approach reflects our core goal: to build general-purpose neural representations that do not rely on task-specific supervision and remain more adaptable across recording conditions and downstream tasks. In addition to this primary unsupervised setting, we also explore two supervised fine-tuning strategies to assess the utility of learned representations. In the supervised variant, we jointly optimize MAE and behavior regression losses to align latent representations with behavior while maintaining reconstruction ability. In the fully-supervised variant, we fine-tune the model using only the behavior regression objective. All fine-tunings are performed separately on individual single-session recordings. For a new session that not encountered during pretraining, we initialize new space embeddings for its neural patches and optimize them during fine-tuning. Fine-tuning of the MS-Distilled LFP models was performed the same way as training single-session Distilled LFP models, where the model weights were initialized with the weights of pretrained ad-Distilled LFP model instead of random initialization (see Appendix
24

---

A.12 for details). Note that all model parameters are updated during fine-tuning regardless of the level of behavior supervision.
A.2 Training details, hyperparameters, and codebase
All models were trained in cluster with 8 NVIDIA RTX A6000 GPUs. We used automatic mixed precision (AMP) training and flash-attention mechanism 82 in transformer architectures for speed- up and memory efficiency purposes. Also, we implemented our framework with variable-length sequence support to remove any computation and memory overhead that could be caused by sequence padding.
To train multi multi-session spike models (MS-Spike), we performed distributed training over 4 GPUs with batch size of 64 with early-stopping patience of 50 epochs, which approximately took 50 hours (-550 epochs), but convergence started around 10 hours of training. We used a patch size (S) of 64 for spatial patching of neurons across space. Overall, the MS-Spike model was trained on spike signals from 226 recording sessions, which approximately corresponds to 120. IM tokens. After pretraining, it was fine-tuned on 17 individual recording sessions, comprising roughly 4.8M tokens.
Similarly, for multi-session LFP models (MS-LFP), we performed distributed training over 2 GPUs with batch size of 64, with early-stopping patience of 50 epochs, which approximately took 3 hours (-560 and -480 epochs for models trained on LFP and LFP power signals). We used a patch size (5) of 32 and 288 for spatial patching for LFP and LFP power signals, respectively, since LFP power signals of 73 had 9 power bands (including local motor potential) from each recording channel.
Single-session models (SS-LFF and SS-MM) and Distilled LFP models were trained on single GPUs with batch size of 32 for 400 epochs with early stopping patience of 50 epochs, which again took approximately an hour. For SS-LFP and Distilled LFP models, we used patch size (S) of 32 and 288, as done for MS-LFP models, for LFP and LFP power signals, respectively. We used the same patch size (S) as in the Distilled LFP models for training the MS-Distilled LFP models, and trained them on 2 GPUs with a batch size of 64 with early stopping patience of 50 epochs, which resulted in convergence around 500 epochs. For SS-MM models, used a patch size (S) of 96 and 352 (summation of patch sizes for MS-Spike and MS-LFP models) for concatenation of spike and LFP or extracted
LFP power signals, respectively. For all LFP models whose latent representations after training (or fine-tuning) with the MAE objective, we used a linear layer for value embeddings to avoid potential information leakage from temporal and spatial convolutions. In the supervised setting, we trained MS-LFP models with a convolutional layer for the value embeddings under the MAE objective, and then fine-tuned with the behavior regression objective ensure compatibility with single-session models. We found that using convolutional layer for value embeddings significantly improved the downstream decoding performance for distilled models and supervised single-session models. Therefore, to enable fair comparisons, we selected the best best-performing architecture for each model class.
After the pre-training of multi-session models (MS-Spike, MS-LFP, and MS-Distilled LFP), we fine-tuned them for 400 (or 150 for MS-Distille LFP) epochs, with early stopping patience of 50 epochs, which usually took around an hour on a single GPU.
For all models, we employed 10-layer transformer encoder backbones (except the scaling analysis in Fig. [0] and a hidden state dimension of 256. We used a sequential learning rate composed of 1) a linear warmup learning rate with a start factor of 0.3 that reached its maximum value of 0.000625 over 30 epochs, and 2) an exponential learning rate with a decay factor 0.995. We used Adam optimizer [83] with weight decay factor starting from 0.1 and reaching to its maximum value of 0.4 over 1000 epochs (which was never achieved for any model). For models trained/fine-tum MAE objective, we used masking probability of 0.6 that is randomly applied on neural patches across space and time, and used 4-layer transformer predictors with a hidden state dimension of 192, followed by 64-dimensional wwojeproecction. For supervised fine-tuning (combination of MAE and behavior regression objectives), we did not apply any scaling on the two loss terms, and for the distillation objective, we scaled the representation alignment objective by X = 5.
on the
We release models and an inference notebook for reproducibility at https : //ggttuu...... com/ Shanchhiaab/Crressocllliisilllss
25

---

A.3 Dataset details
All datasets used in this study are publicly available datasets as shown in Table and preprocessing details are as follows.
whose experimental
Experimental Tasks
Sessions (Pre-FU
Total duration (h)
Total Sequences
Total
Dataset
Regions
# Suhjects
Neurons
Makinn et al., 62/63 63
MI
RT
42-5
13.49
9664
12060
"Flint
MI
CO
9-3
2.03
1448
2344
M1, PMd.
*Gallego- Carracedo
CO
0-23
8.21
9884
1201
S1 (Area 2)
Peric et aL. 65
MI. PMd
RI, CO
I09-0
34.69
34613
10301
Churchland et al.,
70
MI, PMd
Muze
2
9:20
19.33
23117
1728
Chowthury et al..
71
S1 (Area 2)
TRT, CO
12-0
7.45
8334
1188
al.,
CO. ISO, Key
45.0
8.24
9042
4007
Table 2: Datasets used during pretraining and fine-tuning of multi-session spike model. Pre-Ft denotes
the number of sessions used during pretraining and held out for fine-tuming * represents datasets
that have paired LFP (or LFP power for (73) signals available. RT, CO, TRT, ISO, and Key are
abbreviations for random-target, center-out, two-workspace random-target, isometric wrist torque random-targe and key grasping, respectively.
A.3.1 Multimodal neural datasets
The following three datasets have simultaneously recorded spike and LFP (or LFP power for (33 signals, coupled with behavior signals. Therefore, we focused on the following datasets for the distillation analyses. For each dataset, applied a random 80%-20% train-test split at the segment (sequence) level.
Makin et al., 62 In this dataset, two monkeys (Monkeys I and L) performed 2D target- reaching task by controlling a cursor in a 2D virtual environment. Both monkeys were trained to perform continuous reaches to circular targets arranged in grid (e.g.. 8x8). Even though there was no inter-trial interval between sequential reaches, there existed 200 ms lockout interval after a target acquisition during which no target could be acquired. After the lockout interval, new target was randomly drawn from the set of possible targets with replacement. All sessions include neural recordings from the primary motor cortex (M1, 96 channels), and some sessions additionally include recordings from the somatosensory cortex (SI), yielding 192 channels in total.
Monkeys I and L. had 37 and 10 recording sessions, respectively, where broadband signals were publicly available for 30 sessions Monkey I. For these sessions, we extracted local field potential (LFP) signals through standard processing pipeline: 1) notch filtering at harmonics of 60 Hz (line noise the U.S), 2) high-pass filtering with 0.05 Hz cut-off frequency to remove DC drift, 3) low- pass filtering below 50 Hz for anti-aliasing. 4) common average referencing to remove interchannel correlations and 4) downsampling to milliseconds (ms) resolution (100 Hz). For spike signals used sorted units, binned them in 10 ms nonoverlapping windows, and
across all 47 sessions, discarded the units with average firing rate below Hz. As downstream behavior signals, we used cursor velocity in r and y directions. For continuous behavior signals and LFP signals, we performed z-scoring for each dimension independently.
Apart from these preprocessing steps, we did not perform further processing such as trial alignment to movement onset or temporal shifting between neural and behavioral signals. All time-series data (spikes, LFPs, and behavior) were segmented into 5-second sequences before being fed into the models. As shown in Table 2 we used all recording sessions from Monkey L and 32 sessions
from Monkey I to pretrain the MS-Spike models. Of the 32 sessions from Monkey I, 25 included broadband signals, while the remaining did not. For the MS-LFT models, we used the 25 broadband sessions from Monkey for pretraining. To evaluate generalization, we held out 5 of these broadband sessions from Monkey I as unseen sessions for fine-tuning.
Flint et al., [64] In this dataset, monkey (Monkey C) performed 2D center-out reaching task while grasping a two-link manipulandum. Monkey was trained to perform reaches from a center position to 2-cm square outer targets in an 8-target environment. Each trial of the task started with the illumination of the center target, where the monkey had hold the manipulandum for a random hold time of 0.5-0.6 seconds. After, the center target disappeared and outer target was randomly
26

---

selected from the pool of possible 8 targets, which signaled the monkey to start the reach. To obtain the reward, the monkey had to reach the outer target within 1.5 seconds and hold the manipulandum at the outer target for random time of 0.2-0.4 seconds. Then, the monkey returned back to the center target position, and the next trial started. All 12 recording sessions include neural recordings from M1 (96 channels), with L.P signals available at 2 kHz. We used 2D manipulandum velocity in x and y directions as downstream behavior signals.
To obtain paired spike and LFP signals, we followed the same preprocessing pipeline described above for Makin al., dataset 62 63 resulting in synchronized time-series sequences spikes, LFPs, for and behavior. We used 9 sessions for pretraining the MS- MS-Spike and MS- MS-LFP models and held out the remaining 3 sessions as unseen data for fine-tuning and evaluation.
Gallego-Carracede et al., [73] Four monkeys (Monkeys Ch, I., H. and M) performed an instructed delay 2D center-out reaching task using a manipulandum in this dataset. Each trial began with the monkey bringing and holding the cursor at the central target. After a variable delay period, one of eight outer targets equally spaced along a circle of 6-8 em radius was presented, followed by an auditory "go" cue that signaled the monkey to imitiate a movement toward the selected target. Monkeys Ch and M were trained to wait for the auditory go cue. whereas Monkeys H and were not. To obtain a reward, Monkeys CH and M were required to hold the cursor at the target for 0.5 seconds, whereas Monkeys H and I. were required to hold for 0. seconds. Monkeys had to return back to the central target to start a new trial. We used 2D hand velocity in x and y directions as downstream behavior signals.
There were 10, 5, 3, and recording sessions available for Monkeys Ch, H, L. and M, respectively, all of which were not used during the pretraining of the MS-Spike model. Unlike [63] and [64), this dataset did not include broadband signals or raw LFP signals, but instead, contained LFP power signals across 0.5-4 Hz, 4-8 Hz, 8-12 Hz, 12-25 Hz, 25-50 Hz, 50- 100 Hz. 100-200 Hz, and 200-400 Hz. Therefore, we trained separate MS-LFF models on 6, 5, and 3 recording sessions of Monkeys Ch, H. and L. respectively. where we held out 4 recording sessions of Monkey Ch and all sessions of Monkey M as unseen sessions for fine-tuning. Also, this dataset included spike, LFP power, and behavior signals at 30 ms resolution, unlike the other datasets used in this study. The neural signals were recorded from M1 for Monkeys Ch and M, and area 2 of S1 for Monkeys H and L. As done for other datasets, we z-scored continuous LFP power and behavior signals for each dimension independently. All time-series data (spikes, LFPs, and behavior) were segmented into 3-second sequences before being fed into the models.
A.3.2 Spike-only neural datasets
For the following spike-only neural datasets, we utilized the NeuroTask project, which provides a unified interface for accessing and processing the following datasets [84]. In this framework, both spike and behavior signals are provided in trial structure with variable lengths. Although trial event annotations (e.g.. go cue, target onset) were available for some, we did not perform trial alignment based on these markers and instead used the data as provided for consistency and to mimic more naturalistic settings. Trials longer than 5 seconds were segmented into sequences with a maximum duration of 5 seconds. For all datasets, we binned the spike counts within 10 ms nonoverlapping windows and used behavior signals in the same 10 ms resolution. None of the following datasets were included in the distillation analyses they only had spike signals available, however, performed behavior decoding analyses from the latent representations extracted by our unsupervised MS-Spike model and NDT2 (see Fig. g in Appendix (A.)).
This dataset included recordings from four monkeys (Monkeys Cp. T, Mp, and J) they performed either 2D center-out reaching task 2D random target reaching task. Specifically, Monkey Cp had 51 sessions for the center-out task and 15 for the random target task; Monkey T had 6 sessions for each task; Monkey Mp had 22 center-out and 6 random target sessions; and Monkey J had 3 sessions for the center-out task and none for the random target task (109 spike-only recording sessions). For both tasks, we used 2D cursor velocity in and directions as downstream behavior signals. The neural signals were recorded from MI for Monkeys Cp, Mp, and J, and dorsal premotor cortex (PMd) for Monkeys Cp. T, and Mp.
Perich et al., 665-69 69
In the 2D center-out reaching task, the monkeys began each trial by moving their hands to the center of the workspace and was randomly presented one of eight outer targets equally spaced in circle
27

---

after a variable waiting period. To receive a liquid reward, the monkeys were required to reach the outer target within 1 second. For the 2D random target reaching task, the monkeys reached sequentially to four targets, followed by receiving a reward. About 200 ms after reaching the target, new target appeared, and the monkeys started the reaches immediately.
Churchland et al., 70 In this dataset, two monkeys (Monkeys C1 and C2) performed maze task on a vonto-pprallle screen where they made both straight reaches and reaches that curved around one or more intervening barriers. The dataset typically included 27 different conditions, where each condition provides a particular arrangement of targets and barriers. Monkey C1 and C2 had 4 and 5 recording sessions, respectively. For both monkeys, the recordings were made from M1 and PMd. We used the 2D cursor velocity in and y directions as downstream behavior signals.
Chowdhury et al., [71] This dataset included recordings from three monkeys (Monkeys R1, R2, and R3) who used a manipulandum to reach to targets on a screen within a 20 em x 20 cm workspace across two different experimental tasks. For the first experimental task, the two- workspace experiment, the screen was divided into four quadrants, and two specific regions were selected: the far ipsilateral and the near contralateral quadrants. In each trial, one of these two workspaces was randomly selected, and the monkey performed sequence of reaches to randomly positioned targets within that workspace.
For the second experimental task, the active vs. passive center-out reaching task, each trial began with the monkey holding the cursor on a central target for randomized duration. In half of the trials, referred to as passive trials, mechanical perturbation displaced the monkey's hand toward one of four outer target directions during the center-hold period. After the perturbation (bump), the monkey actively reached to complete the trial (active trials). Both monkeys R1 and R2 had and recording sessions of center-out and two-workspace tasks, respectively, whereas Monkey R3 only had recording sessions for the workware task. For both tasks, the neural recordings were made from area 2 of S1, and we used the 2D hand velocity in and y directions as downstream behavior signals.
Ma al., 722 In this dataset, four monkeys (Monkeys XI, X2, X3, and X4), were trained to perform three different experimental tasks where the neural signals were recorded from M1. In the first experimental task, the isometric wrist task, Monkey X4 controlled the cursor on the screen by exerting forces on a small box placed around one of the hands. Flexion and extension forces moved the cursor left and right, respectively, while radial and ulnar forces moved it up and down. Each trial began with the cursor held on the center target for random duration, after which auditory "go" cue prompted the monkey to reach toward one of eight outer targets. A liquid reward was given if the monkey moved the cursor to the target within 2 seconds and held it there for 0.8 seconds. For this task, we used the 2D cursor velocity in the x and y directions as downstream behavior signals.
In the second experimental task, the key grasping task, Monkey XI eached toward and grasped a custom device placed beneath the screen using one hand. A pair of force-sensitive resistors measured the grasping forces applied during the task. At the start of each trial, the monkey placed its hand on touchpad and waited for random interval before initiating grasp. then moved the cursor into one of three rectangular targets displayed on the screen while simultaneously increasing and maintaining grasp force. For this task, we used the applied force in the x and y directions as downstream behavior signals.
In the third experimental task, Monkeys X2 and X3 performed center-out reaching task while grasping the upright handle of a manipulandum. Each trial began with the monkey moving the cursor to the center target, followed by a variable hold period, after which it reached toward one of eight outer targets spaced uniformly around circle. A trial was successful if the monkey reached the target within I second and held for 0.5 seconds to receive a liquid reward. For this task, we again used the 2D cursor velocity in the x and y directions as downstream behavior signals.
A.4 Behavior decoding
To ensure fair comparison across all models and effectively quantify the quality of inferred latent representations, we evaluated behavior decoding by training a linear regression decoder from the inferred latent representations to the behavior signals using the training trials, and assessed
28

---

performance on the held-out test trials. To extract single latent representation per time step, we applied mean-pooling over the latent representations of all spatial patches corresponding to the same time step. For supervised models, we used linear layer appended to the encoder backbone as the behavior decoder. We quantified the downstream decoding performance by computing the coefficient of determination (R2) between true and predicted behavior signals, averaged across behavior dimensions.
A.5 Representation retrieval and alignment
To quantify the distillation performance, we also quantified the representation retrieval performance between paired spike and LFP representations across MS-Spike, MS-LFP, and Distilled LFP models. To achieve that, we first computed sequence-level representations by mean-pooling the latent rep- resentations of neural patches within each sequence. All metrics were computed on test sequences from recording sessions with simultancous spike and LFP (or LFP power) recordings (see Table 2 3,013 sequences from 65 sessions).
As representation retrieval performance metrics, we then computed top-1 accuracy, top-5 accuracy, and mean rank between paired spike and LFP sequence representations. Specifically, for given LFP sequence representation, we computed the cosine similarity with all spike sequence representations and ranked them (among 3013 sequences) in descending order of similarity. Top-1 and top-5 accuracies measure the proportion of LFP representations for which their corresponding paired spike representation was the most similar or was ranked among the top five most similar spike representations, respectively. Similarly, mean rank computes the average rank of the ground- truth paired spike representations across all LFP representations. As shown in Table [ despite our distillation objective not being contrastive objective-i.e. despite it not aligning paired spike and LFP representations while repelling the other unpaired spike representations- the MS-Spike and Distilled LFP models achieved high top- 1/toptt accuracy and low mean rank, indicating strong alignment. In contrast, MS MS-Spike vs. MS-LFP and Distilled LFP vs. MS-LFP pairs yielded much lower retrieval metrics, underscoring the effectiveness of the distillation objective. Metries for the random baseline were computed across two randomly generated matrices with same shape of latent representations.
Surprisingly, we observed that the aforementioned retrieval metrics for the MS-Spike vs. MS-LFP and the Distilled LFP vs. MS-LFP pairs were nearly chance level, indicating that without distillation, there is poor alignment between the spike and LFP representations at the sequence-level, i.e. when looking alignment
basis. This can be counterintuitive, given that we later show that information from spike models can be effectively distilled into LFP models. To better understand this finding, in addition to the sequence- level alignment measured by the retrieval metrics above, we quantified the global alignment between the representations by computing the linear centered kernel alignment (CKA) scores between the representations extracted by MS -S-ie, MS-LFP, and Distilled LFP models 75). Unlike -based metrics such as top-1 accuracy, top-5 accuracy, or mean rank, which operate at the sequence-level, CKA evaluates global alignment of representational geometry as follows. Given two sets of representations matrices X E Ranx and Y E x where n is the number of sequences and dj is the representation dimensionality, CKA measures the pairwise similarity between X and Y by computing the Hilbert-Schmin Independence Criterion between the centered Gram matrices K = XX" and L = YY"" and normalizes this similarity to ensure invariance isotropic scaling and orthogonal transformations:
sequence-by-sequeee
1 P
CKA(X,Y)
IX* I
Intuitively, CKA measures whether two sets of representations encode similar relational structure, where a CKA score of indicates perfect alignment, while a score close to O indicates dissimilar representational structure (as seen in the last row of Table for the random baseline). Unlike retrieval- based metrics above, we observed that MS- vs. MS-LFP and Distilled LFP vs. MS-Spike pairs achieved substantially higher CKA scores compared to the random baseline, indicating a similar global representational geometry between them, even though they lack alignment at the individual sequence-level. However, in line with the retrieval- based metrics, the CKA score between MS-Spike and the Distilled I.PP model was consistently the highest, highlighting the success of the distillation objective in aligning both local and global representational structures.
29

---

A.6 MS-Spike vs. NDT2
As discussed in Section 3.T we adopted an unsupervised multi-subject, multi-session modeling approach similar to NDT2 [7 which tokenizes spiking activity by grouping neurons across space into neural patches and encodes spatial variabilityy via learned space embeddings. In NDT2, these space embeddings are shared across sessions and subjects, while session- and subject-specific variability handled through session-specifie and subject-specific tokens.
In contrast, our MS-Spike model uses session-specific space embeddings, where each recording session has its own learned spatial embedding for neural patches. We did not include subject- or session-level tokens, relying instead on the per-session space embeddings to account for cross-session variability. This change allows our model to flexibly represent neural dynamics across diverse sessions without enforcing a shared spatial representation across subjects.
In addition to this methodological difference, implementation of MS-Spike differs from NDT2 in several key technical aspects- sooe which also contributed to improved training efficiency:
The transformer encoder and predictor backbones of MS-Spike utilize rotary transform- ers [8]] whereas NDT2 utilized default transformer architectures with absolute position encodings.
All models in our framework incrukiingnnn MS-Spike support variable-length sequences during both training and inference. This removes the need for padding all sequences in batch to the same length, thereby reducing both computational and memory overhead. We believe that this is a critical factor in the improved scalability of our MS-Spike model.
For example, this enabled us to train on 5-second segments using batch sizes of 64. In contrast, we encountered severe memory issues when using the original NDT2 implemen- tation with 5-second segments, even at very small batch sizes such as These issues made training infeasible: at that rate, a single model would have required nearly month to complete training. Attempts to use shorter sequences (e.g.. 2 seconds) still failed due to memory constraints, and we ultimately trained the baseline NDT2 model using only 1-second sequences.
As a final debugging step to resolve this issue, we observed the training curves of NDT2 with purely supervised loss (which is computationally lighter than the unsupervised MAE objective) using 5-second segments. Even in this scenario, training still failed to converge-unlite the I-second sequence training, which converged reliably. These findings underscore both the computational limitations of the original NDT2 framework and the practical advantages of our memory-efficient, variable-length MS-Spike implementation.
We trained our NDT2 bascline with 8 subject-speciffc and session-specift tokens, used a patch size (S) of 64 for spatial patching (same as MS-Spike), used a 10-layer transformer encoder backbone as the feature extractor, and set the masking probability to 0.6, matching the configuration used in MS-Spike. As noted above, due to memory and speed limitations, we trained and evaluated NDT2 using 1-second sequences with batch size of 128. To ensure fair comparison, used the exact same train-test splits as MS-Spike. All other hyperparameters -including learning rate schedules-vye were kept their default values specified in the original NDT2 implementation. Training was performed in distributed setting across 4 GPUs for approximately 400 epochs, with total training time amounting to around 125 hours.
A detailed per-subject and per-session comparison of downstream behavior decoding performance between MS-Spike and NDT2 is presented in Fig. g We performed the behavior decoding as described in Appendix(A.4 where we inferred the latent representations from unsupervised MS-Spike and NDT2 models for the pretraining recording sessions.
Among the 226 recording sessions used in the pretraining, MS- MS-Spike outperformed NDT2 in 208 sessions, while NDT2 performed better in 18 sessions. For sessions where MS-Spike outperformed NDT2, the average improvement in decoding performance was 0.090 decoding Rt2 In contrast, for the sessions where NDT2 outperformed MS-Spike, the performance gain averaged only 0.014 R². Overall, MS-Spike significantly outperformed NDT2 in downstream behavior decoding (p < 3.97 x 10
226, one-sided Wilcoxon signed-rank test).
30

---

Hartey (Ma
1012
048
042
Sale
Vorkey JRi
Morkey
0.0
008
633
Honony (Churchland
033-
Morkes (1a ot all
0.74
046-
0.72
NO12
0551
1777
(Dwoddtury
2.40
H972
Monkey (Choadtrryy
0.30
0.50
1077
0.40
1072
45.5
Morooy (Petch
0.80
028-
025
Sessions
Sessions
Figure Per-subject and per-session downstream behavior decoding performance comparison between unsupervised MS- MS-Spike and NDT2 models. Each subplot corresponds to different monkey, with session-wise (denoted on the x-axis) decoding performances (R²) shown. MS-Spike and NDT2 performances are directly compared for each recording session. Subject identifiers and associated dataset references are noted in the subplot titles. See Appendix A.3 for detailed dataset descriptions.
31

---

A.7 Additional baseline comparisons with MSID and BRANT
To provide further evidence for the strong performance of our cross-modal knowledge distillation approach, we conducted two additional baseline comparisons. First, we trained multimodal models using a method termed multiscale SID (MSID) [4] a recent multimodal framework for modeling neural activity -using spike and LFP signals from 30 recording sessions of Monkey I, Since MSID) single-session model, a separate model was trained for each session. We used the official implementation provided by the authors and adopted the horizon hyperparameters reported in their manuscript [34] (hy = h 10). Under this configuration, MSID achieved an average behavior decoding performance (R²) ) of 0.48, which is substantially lower than that of the Distilled LFP model (R² = 0.66).
Second, we attempted to fine-tune the BRANT [8] model, i.e., a multi-session model for intracranial ERG (iFEG) signals with patch-based tokenization combined with spatial electrode location embed- dings, trained the masked autoencoding (MAH) objective. Despite our extensive attempts- freezing and updating the BRANT backbone during supervised behavior regression fine-tuning- model did not converge using LFP signals of 30 recording sessions of Monkey I. We hypothesize that this lack of convergence may stem from 1) substantial domain shift between the human iEEG data used for BRANT'S pretraining and the nonhuman primate LFP data used in our experiments, and 2) the large scale of BRANT's ' architecture (-505M parameters), which likely requires substantially larger datasets for effective fine-tuning.
A.8
Ablation studies
A.8.1
The encoder backbone architecture
In our experiments, we used rotary transformers 81 as encoder backbones due to their flexible attention mechanism, which naturally supports pretraining objectives such as masked autoencoding involving token dropping and masked token reconstruction. In addition to this architectural advantage, we further validated our choice of using rotary transformers from a performance perspective through an ablation study comparing several encoder backbone architectures, namely long short-term memory (LSTM) networks, one-dimensional convolutional neural networks (ID CNNs), and rotary transform- ers. Specifically, we trained fully supervised single-session LFP models separately on 30 recording sessions from Monkey using each of the aforementioned encoder architectures. As shown in Table
3. models with rotary transformer backbone achieved performance comparable to those with an LSTM backbone, whereas models with ID CNN backbone slightly underperformed. Therefore, we adopted rotary transformers as the default encoder backbone for all subsequent experiments.
Backbone architecture
Behavior decoding (1²2
LSTM
0.66 + 0.06
ID CNN
0.61 + 0.06
Rotary transformer
0.66 + 0.08
Table 3:
Average behavior decoding performances (R²) across different encoder backbone architec- tures for fully-super single-session LFP models trained on 30 recording sessions of Monkey I. + represents standard deviation.
A.8.2
The systematic unfreezing of the teacher MS-Spike model during cross-modal knowledge distillation
the teacher MS-Spike model was kept frozen throughout the cross-modal knowledge distillation procedure to 1) simplify training and 2) prevent representation drift in the high-performing teacher model. To validate this design choice, we conducted an ablation study on the 5 and 3 held-out recording sessions of Monkeys I and C, respectively, in which the teacher MS- model was systematically unfrozen at different stages of training the Distilled LFP models: 1) from the beginning, 2) at the 10th epoch, and 3) at the 75th epoch. As shown in Table 4 the Distilled LFP models exhibited representational collapse when the teacher MS-Spike model was unfrozen and updated from the beginning (row 1), achieving an average R² of 0818. Even though unfreczing the teacher MS- MS-Spike model at the 10th (row 2) 75th (row 3) epoch slightly improved
As detailed in Section 3.3
32

---

the performance, they still substantially underperformed (0.20 and 0.31 R², respectively) compared the default configuration, where the teacher MS-Spike model was completely frozen (row 4, 0.68 R²).. Overall, these results confirm that keeping the teacher model frozen throughout distillation provides the most stable and effective training in our experiments, though exploring more gradual or structured unfreezing strategies may offer potential future improvements.
Epoch unfrozen
Behavior decoding (122)
0
0.18 EE 0.08
10
0.20 0.07
75
0.31 + 0.06
0.68 + 0.07
Table 4: Average behavior decoding performances (R²) of Distilled LFP models when the teacher MS-Spike model's parameters were unfrozen and updated the epochs indicated the Epoch unfrozen column. "." denotes that the teacher model was kept completely frozen throughout training, as the default setup. + represents standard deviation.
A.8.3 The impact of pretraining dataset size in cross-modal knowledge distillation
A natural question arises as to whether the performance gap between the Distilled LFP and MS-LFP models mainly driven by differences the pretraining dataset sizes of their respective MS- MS-Spike teacher (i.e., 226 sessions) and MS-LFP models (i.e., 34 sessions for LFP signals), rather than by inherent signal characteristics. To investigate this, we trained new MS-Spike model (denoted as subset) using the same 34 recording sessions employed for MS-LFP training and used it as the teacher in the distillation process. On these 34 sessions, the MS- -SS-e Subset model achieved an average decoding performance of 0.54 R², outperforming the MS-LFP model (0.32 R²) but underperforming the original MS-Spike model trained on all 226 sessions (0.65 R²), likely due to the reduced pretraining set size. Importantly, shown in Fig. 9 the Distilled LFP models distilled from the MS-Spike Subset teacher (denoted by Distilled LFP Subset) again outperformed the MS-LFP models the 5 and 3 held-out sessions of Monkeys and C, respectively, achieving R² of 0.63 for the Distilled LFP Subset model vs. 0.28 for the LFP model the 8 held-out sessions, where the teacher MS-Spike Subset achieved 0.62 R² (in the 8 held-out sessions). For reference, the original MS-Spike model trained on 226 sessions and the corresponding Distilled LFP models achieved 0.69 and 0.68 R2 average decoding performance these results suggest that the performance difference between the MS- MS-Spike and MS-LFP models not simply due to the dataset size used during pretraining but also to the differences in signal characteristics. Critically, this gap can be overcome through cross-modal knowledge distillation.
these 8 held-out sessions, respectively. Overall,
Monkey
Monkey C
0.7
0.8
3066
0.7-
0.5-
0.6
0.4
0.5
MS-Spike
MS LFP
Distilled LFP Distilled LFP Subset
Behawior
0.3
0.4
0.2
0.3
20160822_01 01
20160630.00 01
20160915.01 01
20161013 03
20170124.00 01
Sessions
Sessions
Figure 9: Behavior decoding performances (R²) of the subset MS-Spike model and Distilled LFP models by using the fine-tuned subser MS-Spike models as the teachers. Performance averages are computed across all held-out fine-tuning sessions of the denoted monkeys (same sessions as in Fig.
3)
A.9 Model-size scaling
To investigate the scalability the cross-modal knowledge distillation approach, we trained Distilled LFP models with varying model sizes, i.c., models with 1, 2, 4, and 8 transformer encoder layers, in
33

---

addition to the default model size (transformer encoders with 10 layers). We found that the Distilled LFP models achieved superior performance LFP-only baselines even when the model capacity was decreased 10 times.
a
LFP signals of Monkeys and C
0.7
decoding (R2)
0.6
Average
behavior
0.5
0.4
0.3
0.2
2
8
10
b
0.8
LFP power signals of Monkeys Ch and M
eccodigg (R2)
0.7
Average
behavior
de 0oiin00
00.5
0.4
MS-LFP
0.3
SS-LFP
S5-MS-ZS
0.2
Distilled LFP
0.1
4 Number of parameters (x10*)
8
10
Figure 10: Average decoding performance (R²) of Distilled LFP models trained raw LFP signals from Monkeys and C in (a) and Monkeys Ch and M in (b) at increasing parameter scales. Perfor- mance averages computed across all held-out fine-tuning sessions of the denoted monkeys (same sessions as in Fig. 3
In addition to evaluating the model size scalability the Distilled LFP models, we also examined the scalability of the teacher MS-Spike model. Specifically, we trained teacher MS-Spike model with 4 transformer encoder layers (-4M parameters) instead of 10 layers (-10M parameters), and then trained Distilled LFP models with 10 transformer encoder layers (~10M parameters) using and 3 held-out recording sessions from Monkeys and C, respectively. In this setting, the Distilled LFP models exhibited a larger performance drop compared student model scaling, expected, achieving an average decoding performance of 0.649 R instead of 0.681 R
A.10 Cross-modal knowledge distillation performance on pretraining sessions
In addition to the fine-tuning sessions, the results of which are shown in Figs. 3 and we also tested the distillation performance on the multimodal (i.c., paired spike and LFP signals) recording sessions used in the pretraining of MS-Spike and MS-LFP models for the sake of completeness. Overall, we included 25 and 9 sessions of Monkeys I and C, respectively, in the pretraining of MS-Spike and MS-LFP (trained on LFP signals) models.For Monkeys Ch, H. and La, we included 6, 5, and 3 recording sessions, respectively. in the pretraining of the MS-LFF model trained LFP power signals. We performed both unsupervised and supervised fine-tuning of the MS- MS-Spike model on these sessions, as these subjects were completely held out during MS MS-Spike pretraining (see Table
Unsupervised setting As done for Fig. 3 we first tested the cross-modal knowledge distillation performance in the unsupervised setting. As shown in Fig, [1] Distilled LFP models consistently outperformed all LFP-only baselines and even their teacher MS-Spike models, again suggesting that distilled models effectively leverage shared, rehaavor-predictee structure across spike and LFP modalities.
For example, the average behavior decoding performance (R²) of the Distilled LFP model was 0.66, 0.69, 0.62, 0.77, and 0.68 Monkeys C, Ch, H. and La, respectively (0.67 on average), compared to 0.63, 0.71, 0.66, 0.63, and 0.51 for the MS-Spike teacher models (0.64 average). The MS-LFPP baseline achieved considerably lower performance: 0.33, 0.34, 0.21, 0.58, and 0.37 (0.34 on average), and single-session LFP (SS-LFP) models underperformed even further, with an average of 0.28.
34

---

means of Consistent with results on the fine-tuning sessions, Distilled LFP models again significantly outper- formed all LFP model variants (p 3.6 x 10 15 n 48, one-sided Wilcoxon signed-ram rank test). and also achieved significantly better performance than their MS-Spike teachers (p < 5.3 x 10-1, 1 48 one-sided Wilcoxon signed-rank test). Interestingly, the performance gains over the teacher MS-Spike models were particularly obvious for Monkeys H and La. This suggests that cross-modal knowledge distillation can still be beneficial even when the teacher model or teacher modality underperforms, highlighting its ability to fuse complementary information across modalities during training, despite relying solely on the LFP modality at inference time. Further, both Distilled LFP models and teacher MS-Spike models significantly outperformed the multimodal models (SS-MM and SS-MM-ZS, see Section
4 for details), highlighting the need for more complex unsupervised multimodal fusion approaches rather than input-level concatenation of multimodal signals.
Supervised setting Next, we compared the decoding performances in the supervised setting, following the same setup as in Fig. 6 Consistent with results obtained fine-tuning sessions, the performance gains from cross modsll distillation remained robust, albeit smaller than those observed the unsupervised setting as shown Fig. 12
For instance, the average behavior decoding performance of the Distilled LFP models was 0.75, 0.73, 0.73, 0.91, and 0.87 for Monkeys I, C, Ch, H, and Laa, respectively (0.77 on average), compared to 0.66, 0.71, 0.70, 0.88, and 0.82 for the MS-LFP model (0.71 on average) and 0.65, 0.69, 0.70, 0.90 and 0.87 (0.70 on average) for SS-LFP model. As expected, the distilled models again did not outperform the spike-based models (MS-Spike and SS-MM, with average performances of 0.77 and 0.79, respectively) the supervised setting. However, the Distilled LFP models significantly outperformed all LFP-only baselines (p < 3.8 x 10-12 for MS-LFP, < 9.0 x 10-13 10-13 for SS-LFP, and p 6.0 x 10 for SS-MM-ZA ZS, n = 48, one-sided Wilcoxon signed-rant bank test).
Interestingly, and consistent with the unsupervised setting, MS Apike models fine-tuned on Monkeys H and La underperformed even the LFP-only baselines, whereas the Distilled LFP models achieved superior performance. We hypothesize that this discrepancy stems from limited generalization of the spike modality in these subjects: while MS-Spike models achieve near- pearect decoding performance on the training set, this performance does not translate to the test set, likely due to overfitting or session-specifie nature in the spike signals. contrast, the LFP-only models demonstrate stable and generalizable decoding performance across train and test splits.
Through the distillation process, the LFP models learn to better extract shared, cross-modal structure between spike and LFP representations during training. without relying on spike signals at inference time. Crucially, this process does not merely copy the MS-Spike model's 's outputs, but instead enables the distilled model to aggregate complementary information across modalities in a way that is more robust train-test shifts. These results highlight the value of cross- modal distillation combining the shared, behaviorally relevant information present in both modalities with the inherent generalization stability of the LFP modality, ultimately improving downstream decoding performance.
A.11 Cross-modal knowledge distillation performance with fully-sueervised distillation
We also tested the distillation performance in fully-supervised setting where we replaced the autoencoding objective in Eq. with the behavior regression objective, resulting in the following distillation objective:
T
T Ioly.),
Coup distill
Σ 21 - fo (f
1-
1
Σ T
1
(3)
nyy
(lfo(
Behavior regression objective
Represecnatiinn-lwt aligumendsimilarit objective
where f-(-) is a behavior regression layer used to reconstruct behavior signal =, I² and (-)-)
was fine-tuned in a fully-supervised manner (see Section A.1 to solely optimize behavior regression objective.
To provide fairer comparison, we also fine-tuned MS-LFP models in a fully-supervised manner. As shown in Figs. 13 and [4 for recording sessions used in fine-tuning and pretraining, respectively, fully- supervised cross-modal knowledge distillation led to slight but consistent performance improvements in downstream behavior decoding performance.
35

---

For the fine-tuning sessions, supervisee Distilled LFP models achieved 0.81 R average de- coding performance across all subjects and sessions, significantly outperforming the Distilled LFP models in Fig. 6. which had an average performance of 0.80 R (p < 1.3 x 10-6, n = 51, one-sided Wilcoxon signed-rank test). Fully-supervised fine-tuned MS-L.FP models also exhibited a small gain, achieving an average of 0.76 R compared to 0.74 R for their supervised counterparts. However, fully-supervised Distilled LFP models still significantly outperformed fully-supervised MS-LFP models (p 2.6 x 10 10 r 51, one-sided Wilcoxon signed-rank test).
Similar trends were observed on the pretraining sessions, as shown in Fig. Fully-supervised Distilled LFP models achieved an average performance of 0.774 R², significantly outperforming the Distilled LFP models from Fig. 12 which achieved 0.768 R² (p 4.3 10 n = 48, one-sided Wilcoxon signed-rane -aak test). Fully- Salll-supee MS-LFP models also showed a slight improvement, reaching an average performance of 0.73 R2 compared to 0.71 R2 fo their supervised counterpart. Nevertheless, fully-supervised Distilled LFP models again significantly outperformed fully-supervised MS-LFP models (p- 8.8 x 10-11, n 48, one-sided Wilcoxon signed-rank test).
Overall, these results indicate that explicit inclusion of supervised behavior regression in the dis- tillation objective, in addition to using a fully-supervised teacher model, can further enhance the decoding performance, although the performance gains are modest compared to the improvement introduced by the distillation itself.
A.12 Multi-session (MS) extension of cross-modal knowledge distillation
So far, we performed unsupervised pretraining of MS- MS-Spike models and then fine-tuned them on individual recording sessions using different objectives varying in behavior supervision levels, which were later used as teacher models for the single-session cross- modal distillation setup. Next, investigated whether this distillation process could be extended to a multi-session setting (MS- Distilled LFP) by applying the unsupervised cross-modal distillation objective across multiple sessions, thereby leveraging large-scale unlabelled paired spike-LFP data to pretrain a single, generalizable distilled LFP model. To achieve that, we pooled paired spike-LFP signals across multiple sessions (i.c.. same sessions used for pretraining of MS-LFP model), and pretrained MS- Distilled LFP models by optimizing the unsupervised distillation objective Eq. [I] where we used the unsupervised MS-Spike model as teachers (note that the pretraining of all multi-sesion models was also unsupervised). As done for MS-LFP models, we trained different MS-Distilled LFP models for raw LFP signals (for Makin et al., 62 33 and Flint et al., (64] datasets), and LFP power signals (for Gallego et al., 733 dataset). For the MS- distiistlleed LFP model trained on LFP power signals, we first fine-tuned the teacher MS MS-Spike models in an unsupervised manner on spike signals of the Gallego et al., 733 dataset (note that this dataset was completely held-out during pretraining of the MS-Spike model).
After pretraining, we fine-tuned the MS-Distilled LFP models on held-out recording sessions (i.e.. same held-out sessions for MS-LFP models) in three different scenarios as done for MS-Spike and MS-LFP models. First, we fine-tuned MS-Distilled LFP models in an unsupervised manner by optimizing the unsupervised distillation objective in Eq. J where we used unsupervised fine-tuned MS-Spike models as teachers. Second, we performed a supervised fine-tuning where we again optimized the distillation objective in Eq. but used teacher MS- MS-Spike models that are fine-tuned in supervised manner (see Appendix I. Third, we performed fully-supervised fine-ture fine-tuning by optimizing the supervised distillation objective in Eq. 3 with fully-supervised fine-tuned MS-Spike models as teachers. Compared to single-session distillation, we investigated whether unsupervised dailllatio-bbaeee pretraining could provide a better initialization that can further improve downstream decoding performance when fine-tuned on recording sessions with behavior labels, especially through supervised fully-speervieed fine-tuning.
As shown in Figs. 16 for the recording sessions held-out for fine-tuning, MS-Distilled LFP improved performance over its single-sessio counterparts (Distilled LFP). For instance, in the supervised case, MS-Distilled LFP achieved 0.76, 0.78, 0.87, and 0.86 (0.82 on average) for Monkeys I, C, Ch, and M. respectively, and significantly outperformed Distilled LFP (p < 1.3 x 10 10-4, n 51, one-sided Wilcoxon signed-rank test) that achieved 0.74, 0.75, 0.86, and 0.86 (0.80 on average). The performance improvements were even more significant in the fully-supervised scenario (p- < 1.5 x 10-8, n = 51, one-sided Wilcoxon signed-rank test), where the MS-Distilled LFP model achieved 0.77, 0.80. 0.88, and 0.87 (0.83 on average) for Monkeys I, C, Ch, and M.
36

---

respectively, compared to 0.75, 0.78, 0.87, and 0.86 (0.81 on average), Beyond the supervised scenarios, we also compared these models in the unsupervised case and found that both models achieved comparable performances (0.71 and 0.72 on average for Distilled LFP and MS-Distilled LFPP models, respectively).
It should also be noted that the recording sessions of Monkey M were completely held-out during the pretraining of the MS-Distilled LFP model, done for the MS-LFP model. For Monkey M. we performed single-session fine-tuning of the pretrained MS-Distilled LFP model in two steps: 1) fine-tuning the MS-Spike model-the teacher model used during pretraining of the MS-Distilled LFP.- on Monkey M's s spike data, and 2) subsequently fine-tuning the pretrained MS- MS-Distilled LFP model on Monkey M's LFPs using the fine-tuned MS-Spike model from step 1 as the teacher. Through this pipeline, we aimed to transfer the spike-LFP LFP alignment learned across animals to new subject. Therefore, the results for Monkey shown Figs. 16 and 17 (bottom right panels) also demonstrate the generalization capability of the MS-Distilled L.P model to an unseen subject through fine-tuning.
We also performed the same comparisons for the recording sessions used in pretraining of MS- Distilled LFP models, they had behavior labels available. However, it is important to note that this may not always be the case in practical scenarios, where large-scale multimodal neural recordings are often unaccompanied by corresponding behavior signals. Similar held-out fine-tuning sessions, we found that iistiistleed LFP models again outperformed Distilled LFP models. In the supervised setting, MS-Distilled LFP models achieved significantly superior performance compared Distilled 48, one-sided Wilcoxon signed-rank test) where they achieved
LFP models (p < 9.6 x 10 0.79 vs. 0.75, 0.76 vs. 0.73, 0.72 vs. 0.73, 0.91 vs. 0.91, and 0.88 vs. 0.87 for Monkeys C, Ch, H. and La, respectively (0.79 vs. 0.77 on average). Similarly, the performance improvements were again more significant the fully-supervised setting (p < 1.3 x 10 10-12 n = 48, one-sided Wilcoxon signed-rank test) where MS-Distilled LFP and Distilled LFP achieved 0.79 vs. 0.76, 0.77 vs. 0.74, 0.76 vs. 0.71, 0.92 vs. 0.92, and 0,90 vs. 0.90 for Monkeys I, C. Ch, H. and La, respectively (0.80 vs. 0.77 on average). Unlike fine-tuning sessions, MS- LFP also in the unsupervised setting (p < 6.1 x 10 n = 48, one-sided Wilcoxon signed-rank test), where they achieved 0.69 vs. 0.66, 0.72 0.69, 0.63 vs. 0.62, 0.78 0.77, and 0.67 vs. 0.68 for Monkeys I, C, Ch, H. and La, respectively (0.70 vs. 0.67 on average).
LFP significantly outperformed Distilled
Overall, these results demonstrate that multi-session cross-modal distillation offers powerful ex- tension to the single-session distillation by leveraging shared structure across recordings to produce stronger and more generalizable LFP models. While single-session distillation already yields substan- tial improvements over LFP-only baselines, incorporating broader variability through multi-session distillation-basse pretraining further enhances the distilled model's capacity to capture modality- shared, behaviorally predictive features. This advantage is particularly pronounced supervised and fullyssuer fine-tuning regimes, where multi-session distilled models consistently outperform their single-session counterparts.
37

---

MS-Saike
SSUFP
Konkey
55 MMA-2
Monkey' 20161006
Behover (82)
0.70-
3.00-
0.0--
0.30-
0.20-
Time Imu
201
2010000
2010099
Mankey
0.80
Bolgrior doouding (92)
0.60
0.40
0.00
62.2
62.1
6..
Sess onG
sataee deceling (82)
0.80
Honkay Ch
Monkey
0.60-
0.40
0.20-
20150335
20150315
20101188
201c1014
Timm (ma))
Sessions
Monkey H
Monkeyy
Salaee socoling 182
0./5-
0.70
0.00-
0.53
6.45.
20171127
20171128
2017220
2017126
20171207
Senaicats
Monkey to
Monkay LO 2017073L
salave 0002912 (92)
0.68-
0.50
0.40-
0.35
0.30
2010711
TORES
20179917
Time (mi)
Sessions
Figure 11: Behavior decoding performances (R?2) of unsupervised models on the recording sessions that were used in pretraining of MS-Spike and MS-LFP models. Each panel (a-e) shows decoding results across sessions for individual monkeys (Monkeys C. with LFP signals, and Monkeys Ch, H, and La with LFP power signals) and example decoded behavior trajectories from latent representations of Distilled LFP models and MS-LFP models these sessions. The MS- MS-Spike model fine-tuned for Monkeys Ch, H. and La, as these subjects were completely held out during MS-Spike pretraining. In contrast, fine-tuning
applied for the plotted sessions here from Monkeys I and C, as these monkeys' other sessions were used during MS-Spike pretraining. For the MS-LFP models, no fine-tuning was performed for any session, since all sessions with LFP signals were included during the MS-LFP models' pretraining.
38

---

Spiie
53-MN
Monkey
55-099-5
Schooo decodin
0.60-
0.50
Time
20160004
Sessins
Monkey C e2_1
0.90
Bongeer doodinng (82)
0.70
0.60
0.50
0.40
0.20
6.2
4.1
65.
Sessions
Monkey Ch
Honkey Ch 20150630
Beloner doooiing (82)
0.80
0.70
0.60
0.5
0.60
20130313
20130118
20150629
201seba
201611005
20161014
Sessirs
Munbay
Honbay
Bohavdr (ecodig (92)
0.90
8.85
6.0
0.75
3.70-
0.6
20171127
2017130
20272201
20171204
30171207
Time (mc)
Sessions
Monkey La
Honkey Lo 2017073L
0.90
Sengelor docodng 1921
0.00
0.75
0.70
vel
20170731
20170917
Time
Sessions
Figure 12:
Behavior decoding performances (R²) of supervised models on the recording sessions that were used in pretraining MS Spike and MS-LFT models. Figure conventions are the same as in Fig. [] Unlike Fig. [I however, MS-Spike and MS-LFP models shown here were fine-tuned in supervised manner (note these underwent unsupervised pretraining).
39

---

MS-Spike Fully Sup. Finetuning
MS-LFP Fully Sup. Finetuning
-5--MM
Distilled LFP
4-S5-MM-25
Distilled
Fully Sup.
Monkey
b
Monkey
0.85
0.90
Behavior decodnng
0.75
0.70
0.86
0.65
0.60
0.55
0.82
0.50
0.80
20160622 01
20160630 01
20160915
Monkey 20161013_03 03
20170124_00
01
20160929
20161006
20161007
20161021
Monkey
Behavior decodingg (R2)
0.90
0.85
0.80
0.88
0.75
0.86
0.70
0.84
0.65.
0.82
0.60
0.80
0.55.
0.78
e5.2
20140217
20140218
20140303
20140304
20140307
61.1
Sessions
Figure 13: Behavior decoding performances (R2) in fully-supervine (Fully Sup.) setting. Each panel (a-d) shows the decoding results across sessions for individual monkeys (Monkeys I, C with LFP signals, and Monkeys Ch, H, and La with LFP power signals). Unlike Fig. 6 we demonstrate MS-Spike and MS-LFP models that fine-tuned in fully-supervised manner just with the behavior regression objective, and Distilled LFP models that are trained via the supervised distillation objective in Eq. 3 with fully-supervised MS-Spike models being teachers. Single-session models trained only through the behavior regression objective, as done for Figs. 6
40

---

MS Spice Fully Sup. finetuning
SP
Distilee LPP
MM4
Ostiled LFP Fully Sup.
Monteyl
Behawer decoding 1822
0.80
0.70-
0.60
0.50-
201800144 Of
20 2010001a
201
201
Monkay
Scoding 000
0.60
0.50
62.7
23
6.
02.4
04.5
Sessins
Morkey
0.90
Behevor deoding (ez)
0.80-
0.50
0.40
20150111
2015011
2015002
2015000
20151005
20161014
Serobing
Monkey
Benatoor scooing (22)
0.90
20171127
201711288
20171201
20171204
20171207
Sassiors
0.90
occoding
0.80
0.75
20176731
201 TORNSS
20170917
Figure 14: Behavior decoding performances (RP) in fully-supervise (Fully Sup.) setting on the recording sessions that were used in pretraining of MS MS-Spike Spike and MS-LFP models. Each panel (a-e) shows decoding results across sessions for individual monkeys (Monkeys Monkeys Ch, H. and La with LFP power signals). Unlike Fig. 12] we demonstrate MS-Spike and MS-LFP models that are fine-tuned in fully-supervised manner just with the behavior regression objective, and Distilled LFP models that are trained via the supervised distillation objective in Eq. 3 with fully- MS- MS-Spike models being teachers. Single-session models are trained only through the behavior regression objective, done for Figs.
with LFP signals, and
12 13
41

---

Monkey
Monkey Ch
Behavron decoding (R2)
0.73
0.80
0.70
0.79.
0.68
0.65
0.78
0.60
+
MS-Distilled LFP
0.76.
0.55-
Distilled
0.75.
0.74.
20160622_00 01
20160630 01
20161013 .3
20170124 .11
20160929
20161006
20161007
20161021
20160915
Monkey M
Behavio decoding (R2)
Monkey C
0.78
0.78
0.76
0.74
0.74
0.72
0.72
0.68
0.70
0.66
0.68
0.64
0.66
el.1
e5
20140217
20140218
20140303
20140304
20140307
Sessions
Sessions
Figure 15: Behavior decoding performances (R²) of unsupervised MS- MS-Distilled LFP and Distilled IFP models. Both models were trained (and then fine-tuned for the MS-Distilled LFP model) through the unsupervised distillation objective Eq. where teacher MS- Spike models were fine-tuned also in an unsupervised manner.
a
Monkey
b
Monkey
0.89
0.78
Behavier decodino
0.75
0.88-
0.73
0.70
0.87
0.65
MS-Distilled LFP Distilled LFP
0.86.
0.85
0.60
70169622_08 01
20160630.00 01
20160915 01
20161013_03
20170124_00 01
20160929
20161006
20161007
20161021
Behavier decoding (R2)
Monkey
d
0.91
Monkey M
0.82
0.90
0.80
0.89
0.88
0.76
0.87
0.74
0.86
0.72
0.70
0.84
el_1
e5.2
20140217
20140218
20140303
20140304
20140307
Sessions
Sessions
Figure 16: Behavior decoding performances (R²) of supervised MS- MS-Distilled LFP and Distilled LFP models. Both models were trained (and then fine-tur tuned for the MS-Distilled LFP model) through the unsupervised distillation objective in Eq. T while teacher MS-Spike models were fine-tuned in supervised manner.
42

---

Distilled LFP
+
MS-Distilled LFP
Distilled LFP Fully Sup.
MS-Distilled LFP Fully Finetuning
Behavior decoding (R2) 10
Monkey!
Monkey Ch
0.80
0.75
0.89
0.70
0.87
0.65
0.86
0.85
0.60
20160622.00 01
20160630 01
20160915 01
Monkey C 20161013.00 03
20170124_00 01
20160929
20161006
20161007
20161021
Monkey
Behavlor decoding (82)
0.84.
0.82
0,90
0.80
0.78
0.88
0.76
0.74
0.72
0.84
0.70
e5.2
20140217
20140218
20140305
20140304
20140307
Sessions
Sessions
Figure Behavior decoding performances (R2) of suply-suee MS-Distilled L.P and Distilled I.PP models. Both models were trained (and then fine-tuned for the MS-Distilled LFP model) through the supervised distillation objective in Eq. 3 and the teacher MS-Spike models were fine-tuned also in fully-supervised manner.
43

---

Disiiled UP
Morkey
Behwvirr decating (12))
0.75-
0.68-
0.3
0.60
0.50
20160824
20 (0118188
20l
201
0.80
decddng (83)
0.70-
0.65-
0.60
0.55
64
e
Sessions
Monkey Ch
0.80
Defueeor decodna (121
0.70-
0.60
10159313
20103177
20130829
2010uaar
20101805
20161014
Sessione
Monkey 11
Delveru uscading (821
0.75
0.70
0.68
20111127
20111228
30171201
20171226
201711007
Monkey
Selveter decoddig (121
0.64
0.62
20110111
20170917
Sessions
Figure 18: Behavior decoding performances (R²) of unsupervised MS- MS-Distilled LFP and Distilled LFP models on the recording sessions that were used in pretraining of MS-Spike, MS-LFP, and MS-Distilled L.P models. Figure conventions are the same as Fig.
44

---

Nonkey
Distiled
Ditilled UPP
0.04
0.82
0.80
0.78-
0.74
0.22
0,70
0.68
2010000 or
10160916
201100 20116092
Sessions
Monkey
0.00-
0.75-
0.60
0.55
61
62.
4.
64.3
0.004
Hunkey
0.80-
0.70
0.60-
0.50-
20133113
20190116
20156699
20158610
2016100
20161014
Sensions
0.92
0.91
0.80-
0.89
10173127
20173128
20122301
38171384
Sessions
Monkey
0.88
0.86
20110151
2017ORm5
20170817
Sessiors
Figure 19: Behavior decoding performances (R²) of supervised MS-Distilled LFP and Distilled LFP models on the recording sessions that were used in pretraining of MS-Spike, MS-LFP, and MS-Distilled LFP models. Figure conventions are the same as in Fig. 16
45

---

Diselled LFP
HS Distilee LFP
- DDsstilled LFP-
M5-Distifed LFP-
Monkay
Beeaiter decodin
0.5-
0.3
0.70
0.68
20180152
201
20
0.85
Monkey
0.80
0.70
0.69
Behaweer
0.6
0.5-
050
0.45
24.
65
Sessions
Morkey
0.80
Whheet decddin
0.70
0.80
0.30
1010033
2015031
30150099
20150000
20151005
20151014
Senaines
Monkey
0.904
Metei oaggsing (82)
0.93
0.92
0.91
0.00
0.89
688
20171127
20171128
20171201
20171204
20171207
Scatier
0.93
70.92
0.91
087
0.86
201701111
2017Ra03
2017097
Figure 20: Behavior decoding performances (R²) of fully-supervised MS-Distilles LFP and Distilled LFP models on the recording sessions that were used in pretraining MS-Spike, MS-LFP, and MS-Distilled LFP models. Figure conventions are the same as in Fig. 17
46


Reviews:
[{"review_id": "AohS5hsY8U", "reviewer": "Reviewer_LV5f", "summary": "The paper introduces an unsupervised cross-modality distillation framework for electrophysiology, combining both spikes and local field potential (LFP). They first train a teacher spike model across multiple recording sessions using a masked autoencoding objective with a session-specific neural tokenization strategy, and then align the latent representations of the student LFP model to those of the teacher spike model. When tested on motor cortical data from monkeys, their distilled LFP model consistently outperforms the other LFP baselines. In some of the sessions tested, they find some small improvements in performance of the spike only model, suggesting that including both modalities can enhance the performance of BCI. They provide evidence that the model can align modalities while also capturing individual differences in the latent embeddings.", "questions": "1. How do other existing LFP baselines compare? Please provide comparisons with other traditional deep learning methods (e.g., MLP, CNN, LSTM/state space model), as well as iEEG or other LFP-based transformer baselines to assess how performant a rotary-transformer is for LFP modalities. \n2. How does your distillation approach compare to a multi-modal training approach? Please compare with multimodal approaches like those described in Schanechi and Pesaran’s work on the topic. \n3. Can you please clarify the size of patches for the LFP vs. the spikes? How is the data split into train and test? Please describe the model evaluation procedure in detail for both modalities.\n4. How general is your distillation framework? Could it be applied to another decoding model (e.g. MtM, POYO)?\n5. What is the performance degradation while downsizing the teacher model instead of the student model?", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths:**\n- The work addresses an important challenge and aims to utilize LFP which is often discarded from analysis, and can be used with minimal preprocessing steps\n- Novel approach to use cross-modality distillation for improving decoding from LFPs\n- The results suggest that pretraining with spikes helps LFP decoding, and the incorporation of LFP can also help unimodal BCI decoding in some cases.\n\n**Weaknesses:**\n- While the results in Figure 3 demonstrate a clear advantage of the distilled multimodal model over the LFP only decoder, it is not clear whether this is a strong enough LFP baseline for this dataset. Additional baselines are needed to demonstrate the advantage of the multimodal approach. There are existing cross-modal decoders from the Schanechi and Pesaran groups, and a number of LFP-based transformer models in the literature (Brant, NeurIPS 2023; Seegnificant, NeurIPS 2024) that could be used to build a more compelling set of baselines. \n- Clarity of methods.  The entire pretraining method and finetuning is described in 3.1. This should be broken up into different subsections that delineate the pretraining objective from your downstream evaluations. Section 3.2 is also not very easy to follow. Instead of connecting back to the MAE pretraining in 3.1, the reader is pointed to pretraining in Appendix. It would be useful to give a more clear step by step guide to your method and what remains frozen vs. learned, and the dynamics of updates across the two models.\n- It is unclear whether the model has learned generalization information about the relationship between modalities, or uses LFP as additional animal-specific features. Demonstration of some across-modality transfer to new animals would be helpful to understand whether the relationship between modalities can generalize to new settings. \n- The advantage of distillation over multi-modal pretraining has not been discussed. How critical is it to pretrain with spikes and then start learning with LFP in smaller model vs. pretraining with the objective in end-to-end manner? Why have two separate models\nThere should be some alignment across modalities but also LFP carries different information, do you assume this is not useful for the downstream task and is discarded? Do you have evidence that this alignment objective is useful for decoding? Does it help to denoise the LFP? \n- Often, student teacher models require specific learning schedules. How do you unfreeze or add in supervision over training?\n- Patching over neurons doesn’t really make sense as the ordering of neurons has no specific meaning. Please justify this choice of tokenization.", "was_revised": true, "final_justification": "During the response period, the authors provided new experiments that demonstrate the utility of their method in across day and animal transfer, and attempted to provide new unimodal baselines to contextualize their results. These new experiments gave further confidence in the approach and that the results are meaningful.", "rebuttal": "> Additional baselines and expanding the distillation approach to other models\n\nWe thank the reviewer for thoughtful suggestions. As requested, we incorporated new baselines for further validations.\n\nFirst, we trained MSID (Ahmadipour et al., 2024)—a multimodal model of neural activity from suggested groups—on 30 sessions from Monkey I. MSID achieved a mean $R^2$ of $0.48\\pm0.09$, **substantially lower than our unsupervised distilled LFP models**, which achieved $0.66\\pm0.05$. The underperformance of MSID is consistent with its reliance on linear latent dynamics.\n\nSecond, we attempted to fine-tune the BRANT model, using the publicly available checkpoint. Despite extensive attempts—both freezing and updating the BRANT backbone—the model failed to converge on the Monkey I dataset. We hypothesize this is due to:\n- **Domain shift** between the human SEEG data used for pretraining and the NHP LFP data used in our setting, and\n- **Model scale**—BRANT’s ~505M parameters likely require larger datasets to fine-tune effectively.\n\nAdditionally, for LFP data without distillation, we compared our rotary transformer backbone, 10-layer LSTMs, and 10-layer 1D CNNs (all producing 256-dimensional embeddings), trained in the supervised setting. Across 30 sessions of Monkey I:\n|Model|Decoding $R^2$|\n|-|-|\n|LSTM|$0.66\\pm0.06$|\n|1D CNN|$0.61\\pm0.06$|\n|Rotary Transformer|$0.66\\pm0.08$|\n\nWhile these results are comparable, our supervised distilled LFP models still achieved significantly higher performance ($0.76\\pm 0.04~R^2$) than all of them. We selected the transformer backbone primarily for its compatibility with masked autoencoding objectives and its strong performance across diverse deep learning tasks.\n\nWhile performance is important, we emphasize that the **core contribution of our work** lies in its **conceptual advance**—enabling **high-performance unimodal LFP decoding through cross-modal distillation**, rather than simply aiming to outperform existing LFP models by changing the model architecture. \n\nImportantly, our framework is **architecture-agnostic**: the distillation approach could be applied to architectures such as BRANT, MtM, NDT2, or POYO in the future.\n\n> Clarity of methods, adding supervision, freezing\n\nWe thank the reviewer for helpful suggestions on the write-up, and we apologize for the confusion caused. In Section 3.2, we originally referred to Appendix A.1 for MS-Spike fine-tuning details. However, we will now also include a clear summary of these details directly in Section 3.1, where they are most relevant. We will also add a step-by-step guideline for training/finetuning details, summarized as follows:\n\n1. Pretraining an MS-Spike model on pretraining sessions\n2. Finetuning the MS-Spike model on one of the finetuning tasks, depending on the level of supervision. **For supervised fine-tunings, we add a regression head to the pretrained MS-Spike model, as the pretraining was unsupervised. During all fine-tunings, all model parameters are updated.**\n3. Initializing an LFP model, then training it via the cross-modal knowledge distillation objective where the teacher model is the fine-tuned MS-Spike model from step 2. **During this step, the teacher MS-Spike model is fully frozen (as described in L208-209).**\n\n> Learning schedules\n\nWe chose to keep the teacher MS-Spike model fully frozen during distillation. The decision was based on simplicity and prior evidence (Abbaspourazad et al., 2025; Li et al., 2025).\n\nAs described in response to reviewer ZtJZ item 3, we tested the impact of unfreezing the teacher model. When the teacher was **updated from the start of distillation**, the model suffered from representational collapse, achieving only $0.09~R^2$ on one session of Monkey I. We then tested a **delayed unfreezing schedule**, where the teacher network remained frozen for the 10 or 75 epochs and was updated thereafter. While these prevented collapse, they achieved slightly worse performance. \n\n> Multi-modal modeling\n\nAs highlighted in L100–101, our goal is distinct from multi-modal modeling: rather than leveraging both LFP and spike signals at inference time, we aim to build **high-performing LFP-only models** (the important applications for LFP-only models are provided in the introduction L30-38). \n\nAs also discussed in L24–28, our motivation to use a pretrained spike-based teacher is twofold:\n\n- Spike models (e.g., NDT2, MS-Spike) are well-established as SOTA for motor decoding (L39–41),\n- Public spike datasets are significantly more abundant than multimodal datasets, enabling more generalizable pretraining. (L27-28).\n\nAnother advantage of cross-modal knowledge distillation over multi-modal learning is its simplicity, i.e., not requiring complex alignment approaches as in many multi-modal works (Peng et al., 2025; Wang et al., 2019). This is particularly important when one modality is of lower quality than another (i.e., LFP vs. spikes) (Huang et al., 2021). \n\nWe agree that comparing against multi-modal baselines is important, and thus, we provided comparisons against 2 multimodal baselines: SS-MM and SS-MM-ZS (see L231-237). As shown in our results, distilled LFP models consistently outperform the SS-MM-ZS baseline (Figs. 3,4,6,10,11,12,13). Further, we now show that our distillation approach outperforms a multi-modal baseline, MSID. \n\n> Denoising & discarding information of LFP\n\nThe reviewer is correct that the distillation objective contributes to denoising the LFP signals. However, we **do not believe this process completely discards LFP-specific information** that may be complementary. We highlight 3 reasons supporting this:\n\n- **We include an autoencoding objective with the distillation loss**, encouraging the model to retain intrinsic LFP information.\n- **The distillation loss does not converge to zero** (e.g., 0.8 cosine similarity), suggesting that the model retains some degree of LFP-specific encoding.\n- As shown in Fig. 3, distilled LFP models can even **outperform their teacher MS-Spike models** on certain sessions. This would be unlikely if important LFP-specific information were discarded.\n\n> Generalization\n\nWe do demonstrate generalization to a new subject in Section 4.5, where we evaluate **multi-session (MS) distillation approach** on a **held-out animal (Monkey M)**. The procedure is as follows:\n\n1. Pretrain an MS-Spike model by including some sessions of Monkeys Ch, H, and La (in addition to the 226 sessions), **excluding any data from Monkey M**. \n2. Using the MS-Spike model from step 1 as the teacher, pretrain an MS distilled LFP-power model through the distillation objective by excluding Monkey M\n3. Fine-tune the MS-Spike model from step 1 on Monkey M’s spikes\n4. Fine-tune the MS distilled LFP model from step 2 on Monkey M’s LFPs, where the fine-tuned MS-Spike model from step 3 is used as the teacher.\n\nThis pipeline **transfers the spike-LFP alignment learned across animals** to a new subject, and **the resulting distilled LFP models generalize well to Monkey M**. We will make this clearer in the manuscript. \n\nWe would also like to ask the reviewer to kindly see item 3 in response to reviewer KUPu, showing generalizability to fully unseen recording sessions with no training/finetuning, and distillation.\n\n> Patching, data details, evaluation\n\nWe agree that the ordering of spiking neurons does not inherently carry semantic meaning. While the neuron index is arbitrary, the patched groups still represent neurons recorded from the same cortical region (M1) with shared and complementary dynamics. The learnable patch embeddings and the self‑attention mechanism allow the model to flexibly capture these correlations without assuming any fixed spatial structure. A similar patching mechanism can also be found at NDT-2. \n\nWe provided patch sizes for spike and LFP signals across different settings in Appendix A.2 (L645-646, L649-651), where we used a patch size of 64 for spikes, and a size of 32 and 288 for LFP and LFP power signals, respectively. These choices are made for computation efficiency, as small patch sizes increase the input sequence lengths. \n\nFor data splits, we perform a random 80/10/10% train/validation/test split. Models are evaluated based on downstream behavior decoding: we train a linear regression head on the embeddings produced by the encoder, detailed in Appendix A.4. During evaluation, we combine validation & test sets to improve the robustness of performance statistics, particularly for sessions with limited data.\n\n> Teacher model size\n\nPer reviewer’s request, we now performed distillation from a 4M parameter MS-Spike model, where the student distilled LFP models had 10M parameters. Mean performance in Fig. 9a, where both teacher MS-Spike and student distilled LFP models had 10M parameters, was $0.681$ $R^2$. With a 4M parameter MS-Spike model, we obtained a slightly worse mean performance of $0.649~R^2$, and a larger drop compared to student model scaling as expected, but distilled LFP models again outperformed other LFP-only baselines. \n\nOverall, we hope the further evidence above on our model’s performance and generalization answers the reviewer’s concerns and encourages the reviewer to reconsider our work.\n\nReferences:\n- Ahmadipour et al. (2024). Multimodal subspace identification for modeling discrete-continuous spiking and field potential population activity. Journal of Neural Engineering.\n- Abbaspourazad et al. (2025). Wearable Accelerometer Foundation Models for Health via Knowledge Distillation. arXiv. \n- Li et al. (2025). Distilling Knowledge from Large-Scale Image Models for Object Detection. ECCV.\n- Peng et al. (2025). Modalities Contribute Unequally: Enhancing Medical Multi-modal Learning through Adaptive Modality Token Re-balancing. ICML. \n- Wang et al. (2020). What Makes Training Multi-Modal Classification Networks Hard?. arXiv. \n- Huang et al. (2021). What Makes Multi-modal Learning Better than Single (Provably). arXiv."}, {"review_id": "ZU73A8CcJW", "reviewer": "Reviewer_KUPu", "summary": "The authors propose a method for learning behavioral-predictive representations of LFPs by first pretraining a spiking model and then distilling the learned representations into an LFP model. Additionally, session-specific tokens are used to align latent representations across recording sessions. In experiments predicting behavior, the distilled LFP models perform roughly as well as spike models, and outperform single- and multi-session LFP-only baseline models. A TSNE embedding and an analysis with centered kernel alignment confirm that the distilled LFP model achieves similar representations to the multi-session spike model. The main result also holds up in a supervised finetuning setting and a multi-session distillation setting.", "questions": "Are my assessments in \"Strengths and Weaknesses\" about the smaller LFP dataset and the necessity of paired LFP-spike data accurate?\n\nCan you say anything specific about the features of LFPs that allow the distilled LFP models to predict behavior about as well as the spike models? \n\nMinor comments:\n* The experiments section should state the number of tokens for pretraining and finetuning,", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "The idea of learning informative LFP features by distilling knowledge from a spike model is very attractive given all the benefits of LFPs listed in the abstract (stability, lower power requirements, robustness) and the generally higher predictive performance of spikes in various behavior prediction scenarios.\n\nThe transformer architecture trained using session-specific learnable tokens and a masked autoregressive objective is a reasonable and flexible approach.\n\nThe finding that an LFP-only model can predict behavior as well as a spiking model by aligning its representations is very interesting to a computational neuroscience audience.\n\nThe MS-LFP data has many fewer sessions than the MS-Spike data (34 vs. 226). It seems plausible that this could explain the poor performance of the LFP-only models. Additionally, I would expect the LFP data that is only available as LFP power to be less informative than the LFPs themselves. I find it hard to interpret the results of the LFP-only baselines given these considerations.\n\nIn the first paragraph of the Results section, there is a discussion about testing generalization to unseen subjects. But \"after pretraining, we fine-tuned the MS-spike model on the spikes of the held-out sessions,\" so it seems that there are no truly unseen subjects at the time of testing the distilled LFP model. Additionally, the results in section 4.3 are not on true hold-out sessions or animals, since they are used in pretraining. If I am following correctly, it seems the strategy taken to test the proposed method is not applicable to the case where only LFPs are available, with no shared spikes. But this is the exact scenario where the proposed knowledge distillation would be expected to be most beneficial and practical.\n\nGiven these concerns in the previous two paragraphs, I find it difficult to come away with a clean interpretation of the results of the paper.", "was_revised": true, "final_justification": "My two major reservations with original paper have been addressed. First, I find the LFP-only results (MS-LFP) much easier to interpret with the addition of the MS-Spike subset results, and the distilled LFP subset results should also help with the interpretation. Second, the new generalizability results to a completely unseen session demonstrate a stronger form of generalizability than the original paper. For these reasons, I have raised my score.", "rebuttal": "We thank the reviewer for their acknowledgement of our work’s strengths and its importance for the computational neuroscience community. Below, we answer the reviewer’s questions:\n\n> Number of different sessions across MS-LFP and MS-Spike\n\nWe appreciate this insightful observation and agree that the smaller number of sessions in the MS-LFP dataset (34 vs. 226 for MS-Spike) can indeed be a contributing factor to the performance gap between the LFP-only and spike-only models.\nHowever, this imbalance stems not from our experimental design but from **the limitations of publicly available datasets—a key motivation for our cross-modal knowledge distillation from spikes to LFPs**. As noted in L26–28, most motor neural datasets have focused on spike recordings, likely due to theirr higher signal-to-noise ratio (SNR) as well as more memory-efficient data sharing. In contrast, LFP signals are typically derived from broadband recordings sampled at high frequencies (e.g., 30 kHz), which require much larger storage (often 1–8 GB per session in datasets like Makin et al.), whereas spike timing data for the same session can be shared in a much smaller storage (100 MB–1 GB). Given this data availability difference in the neuroscience domain, recent modeling approaches largely utilized spike signals to train models on larger datasets and provide more rigorous evaluations.\n\nGiven these constraints, **we did our best to include as many publicly available datasets containing both spike and LFP signals as possible**. While the current availability of LFP data is limited, we believe that with larger-scale LFP datasets and more sophisticated pretraining strategies, it would be possible to develop more accurate MS-LFP models in the future. However, even in a future large-scale setting, we believe that the proposed cross-modal distillation would offer advantages due to the SNRs of spikes and LFP signals. As supporting evidence, we trained an MS-Spike model using the same 34 sessions as MS-LFP, which achieved $0.544\\pm0.083$ $R^2$, outperforming MS-LFP ($0.325\\pm0.042$ $R^2$) but underperforming the MS-Spike model trained on 226 sessions ($0.650\\pm0.065$ $R^2$), likely due to the reduced training set. This aligns with prior work showing spikes often carry more behaviorally relevant information, as reflected in higher decoding performance (Bansal et al., 2011). Therefore, using spike models to guide LFP models can help denoise the LFP signal and improve its behavioral relevance.\n\n> LFP vs. LFP power features\n\nWe understand and agree with the intuition that raw LFP signals should, in principle, contain at least as much information as the LFP power features, since the latter are computed from the former. As expanded below, the reason for using LFP power in some datasets was that in those datasets, raw LFP signals were not available.\n\nWe would like to highlight that the **LFP baselines trained on raw LFP signals and those trained on LFP power features are not directly comparable**, as they originate from different datasets collected during different experiments. Specifically, recordings from monkeys Ch, H, La, and M were obtained from the dataset by Gallego-Carracedo et al., where only preprocessed LFP power features are publicly available. Consequently, separate models had to be trained on these sessions. Thus, these results also reflect performance differences across distinct datasets and preprocessing pipelines rather than a strict comparison of LFP signals versus LFP power features.\n\n> Generalizability\n\nThe distillation is performed on a single-session basis by using the LFP signals of that session, after fine-tuning the pretrained MS-Spike model on the spike signals of that session, which is then used as the teacher network during distillation. Thus, it is correct that during the test time of the distilled LFP models, there are no ‘truly’ unseen subjects or sessions; when we say ‘unseen’, **we refer to the being ‘unseen’ in the pretraining of MS-Spike models**. We would like to highlight that **this is the case for other multi-session and multi-task models of neural activity, such as NDT2 or POYO**, i.e., before testing the pretrained model on a new subject/session, some degree of fine-tuning is required for all of these models as they contain subject/session-specific parameters that need to be learned. **In this manner, the reviewer’s understanding is correct, and our work does not differentiate from the prior work in this regard, but follows a similar pretraining/fine-tuning strategy.**\n\nReviewer’s understanding of results in Section 4.3 is also correct. The spike signals of sessions on the x-axis of Fig. 5 are indeed used during pretraining of the teacher MS-Spike model. However, **during the distillation objective, we use the LFP signals of a single session (20160622_01 of Monkey I)**, after fine-tuning the pretrained MS-Spike model on that same session’s spike signals to use it as a teacher network. Thus, the LFP signals of the sessions on the x-axis of Fig. 5 are never seen during parameter updates/training; **they are just used for inference**. In Fig. 5, we show that the distillation objective learns a spike-LFP alignment that generalizes beyond the recording session on which the distillation is performed. \n\nNevertheless, we understand the reviewer’s concern, and we sincerely thank them for bringing it up. **Now, we pretrained a new MS-Spike model where we first held out the recording sessions shown in the tables below (in addition to the sessions on which we performed distillation)**. Then, we performed the following:\n\n1. Pretrain a new MS-Spike model where all sessions shown in tables below are held out, in addition to session 20160622_01 of Monkey I, and all sessions of Monkey C.\n2. Fine-tune the MS-Spike model on spike signals of session 20160622_01 of Monkey I and session e1_1 of Monkey C separately \n3. Use fine-tuned MS-Spike models from step 2 as teacher and perform distillation with LFP signals of session 20160622_01 of Monkey I and session e1_1 of Monkey C separately (so distillation is only performed on single sessions)\n4. Perform inference on the LFP signals of sessions below (held-out).\n\nIn this setting, we obtained the following results. Note that **‘Distilled LFP’ and ‘MS-LFP’ refer to results in Fig. 5**, whereas ‘New Distilled LFP’ refers to the new model trained as described above. \n\nMonkey I:\n|Session|MS-LFP|Distilled LFP on 20160622_01|New Distilled LFP on 20160622_01| \n|-|-|-|-|\n|20160624_03|0.396|0.607|0.598| \n|20160627_01|0.361|0.579|0.539| \n|20160916_01|0.298|0.576|0.529| \n|20160921_01|0.336|0.566|0.423| \n|20160927_04|0.350|0.562|0.482|\n|20160927_06|0.353|0.599|0.558|\n|20160930_02|0.355|0.570|0.582|\n|20160930_05|0.338|0.617|0.562|\n|20161005_06|0.301|0.450|0.452| \n|20161006_02|0.389|0.571|0.541|\n|20161007_02|0.306|0.563|0.547|\n|20161011_03|0.310|0.546|0.504|\n|20161014_04|0.348|0.600|0.567|\n|20161017_02|0.360|0.537|0.546|\n|20161024_03|0.310|0.611|0.582| \n|20161025_04|0.406|0.661|0.640| \n|20161026_03|0.352|0.655|0.587| \n|20161027_03|0.267|0.510|0.445| \n|**Mean**|**0.341**|**0.577**|**0.538**|\n\nMonkey C:\n|Session|MS-LFP|Distilled LFP on e1_1|New Distilled LFP on e1_1| \n|-|-|-|-|\n|e2_1|0.287|0.499|0.495|\n|e2_2|0.379|0.628|0.639|\n|e3_1|0.375|0.696|0.691|\n|e3_2|0.357|0.629|0.664|\n|e3_3|0.375|0.648|0.653|\n|e3_4|0.242|0.356|0.388|\n|e4_1|0.340|0.528|0.556|\n|e4_2|0.342|0.598|0.599|\n|e4_3|0.273|0.494|0.497|\n|e5_1|0.384|0.625|0.673|\n|e5_2|0.384|0.585|0.560|\n|**Mean**|**0.340**|**0.572**|**0.583**|\n\nIn this setting, the generalization performance of the new distilled LFP of Monkey I model slightly decreased compared to the one in Fig. 5; but, it still significantly outperformed the MS-LFP model. Unlike Monkey I, the distilled LFP model of Monkey C outperformed its counterpart in Fig. 5 and MS-LFP. These results indicate that the our distillation approach can generalize to recording sessions **even when they are completely unseen, i.e., neither of the spike signals nor the LFP signals are used during training or even fine-tuning**. This result also shows that even for new sessions that do not contain paired spikes, the distilled LFP models can still offer an advantage. \n\nWe believe that this result is a great addition to our manuscript, and we sincerely thank the reviewer for their great insight on recommending that.\n\n> About LFP features that allow superior performance for LFP signals\n\nPrior work in neuroscience shows that LFP and spike signal dynamics have common dynamical modes that are dominantly predictive of downstream behavior (Abbaspourazad et al., 2021). Given this evidence, aligning the LFP representation to spike representation through the distillation objective likely allows the learning of the behaviorally relevant dynamical features in LFP time-series data. \n\n> Number of tokens\n\nThe number of tokens processed during pretraining and finetuning of MS-Spike is 120.1M and 4.8M, respectively. We will include these details in our manuscript.\n\nOverall, we thank the reviewer for their careful evaluation and for highlighting important points, which we addressed in detail with new analyses and clarifications. We hope that these responses and the new evidence provided will help the reviewer to reassess their overall evaluation of our work.\n\nReferences:\n- Stavisky et al. (2015). A high performing brain–machine interface driven by low-frequency local field potentials alone and together with spikes. JNE.\n- Abbaspourazad, et al. (2021). Multiscale low-dimensional motor cortical state dynamics predict naturalistic reach-and-grasp behavior. Nature Comm.\n- Ahmadipour et al. (2024). Multimodal subspace identification for modeling discrete-continuous spiking and field potential population activity. JNE.\n- Bansal et al. (2011). Relationships among low-frequency local ﬁeld potentials, spiking activity,\nand three-dimensional reach and grasp kinematics in primary motor\nand ventral premotor cortices. Journal of Neurophysiology."}, {"review_id": "V7qo3ebJ1G", "reviewer": "Reviewer_ZtJZ", "summary": "The authors propose a cross-modality distillation framework that leverages existing neural-spike based transformer decoders to boost the performance of LFP-based transformer decoder. The authors argue that having LFP-based transformer decoders is also important, as LFP recordings can be more easily obtained in some cases. They essentially train a LFP-based transformer by combining autoencoding loss (to ensure the latent representations can reconstruct the LFP signal) and a cosine loss (to align the latent representations of the LFP transformer decoder to the neural spike transformer decoder). The authors compared their distillation-based LFP transformer with reasonable baselines (e.g., vanilla LFP-based transformer) across multiple datasets. They show significant improvement in decoding performance of the distilled LFP-transformer in comparison to the baselines. Finally, they extend their distilled LFP-based transformer to a multi-session approach demonstrating modest improvements in decoding (increase of 0.01 and 0.02 in R2 coefficient, although statistically significant).", "questions": "Out of my own curiosity, have the authors tried not keeping the weights of the teacher model not fixed in the distillation objective? It might be possible that allowing the LFP signal to influence the representations of the spike-based transformer might lead to a better representation.", "limitations": "Yes", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n1. The paper is well-written and does a good job of explaining its problem. It is important to also develop techniques that can work with LFP data which is more readily available than its neural spike counterpart.\n\n2. The paper proposes an extension of NDT2 that performs significantly better than NDT2 (increase in behavior decoding (R2 coefficient) by roughly 0.9). The proposed extension changes the tokenization of the spiking data to allow the transformer to leverage spatial relationship in addition to the temporal ones.\n\n3. The paper uses a cross-modal knowledge distillation framework that uses neural spike-based transformer (trained with larger datasets) to train single session LFP transformer. The corresponding distilled LFP transformer significantly outperforms transformers trained using only LFP data robustly showing the value of using distillation.\n\n4. The authors perform extensive validation of their technique against reasonable baselines, and show consistent improvement.\n\n**Weakness**\n\n1. The improvements of distilled LFP models over the multi-session spike-based transformer are modest (improvement of 0.01 to 0.02 in the R2 coefficient). Since, the latent representation of the distilled LFP model is trained to be similar to the spike-neural transformer, it is not surprising that the distilled LFP model and spike-based transformers have similar decoding performance as the decoding is performed using latent representations. The fact that adding LFP data does not yield a substantial improvement seems to suggest that most of the decoding capability is resulting from the spike-based representations with the LFP data having a minor effect on improving the representation. \n\n2. The multi-modal baseline used for comparison is a bit naive. Concatenating the LFP signal and the neural spike is one of the simplest way to perform multi-modal learning. There are a lot more principled ways to perform multimodal learning, e.g., take a look at the review by Xu et al. IEEE Trans. Pattern Recognition'23. I am not expecting authors to compare their method with new multi-modal approaches, as it is not the main point of the work. I would just encourage authors to acknowledge the limitation of their multi-modal approach and discuss that the another approach to improving LFP models would be through multi-modal learning (there are multi-model learning approaches that also allow unimodal predictions). Indeed, with enough availability of both LFP data and spike-based data, it could be possible that multi-modal approaches could outperform the distillation approaches.", "was_revised": true, "final_justification": "Authors response has not impacted my score. My opinion of the work remains the same.", "rebuttal": "We thank the reviewer for their thoughtful and constructive evaluation of our work. We appreciate their recognition of our contributions and address their comments below.\n\n> Substantial improvement over MS-Spike models\n\nAs the reviewer noted, distilled LFP models do not yield substantial gains in behavior decoding compared to their teacher MS-Spike counterparts. This observation is expected and aligns with prior findings that spike signals alone outperform LFP signals in decoding, which is precisely why we chose the spike model as the teacher for distillation. However, we emphasize that our goal is to enable a high-performance unimodal LFP model, which does not need to use any spiking activity for decoding. As such, the value of our approach lies not in surpassing spike-based decoding but in enabling improved decoding just from LFP signals—**a more practical, more robust, and less invasive modality—while narrowing the gap to spike-based performance**. Further, please see item 3 in our response to reviewer KUPu, where we show that the distilled LFP models can still achieve a competitive decoding performance on a new session without any further finetuning or distillation.\n\n> Multimodal modeling\n\nWe thank the reviewer for this thoughtful comment. We agree that our multi-modal baseline—simple concatenation of spike and LFP signals at the input level—is indeed a naive approach compared to more principled multi-modal learning strategies. We acknowledge this limitation and note that more sophisticated multi-modal approaches could be explored as a complementary direction, particularly when both spike and LFP data are available at inference time (L967–968).\n\nHowever, we would also like to highlight an important challenge in this setting. Multi-modal models can degrade significantly when the higher-quality modality (spikes) is missing at inference, sometimes performing worse than unimodal baselines (Ma et al., 2022). This issue becomes even more pronounced in unsupervised multi-modal modeling, which is crucial for training generalizable models, where the lower-quality modality can dominate training, making it harder to learn robust and generalizable representations. In contrast, our cross-modal distillation framework is explicitly designed to enable high-quality inference from LFP signals alone, without requiring spikes at test time. This is critical given the practical advantages of LFPs and that spike recordings degrade over time. Please see the important applications of an LFP-alone model in the introduction (L30-38).\n\nTaking these considerations together, we believe our cross-modal distillation offers **a simpler, more scalable, and more robust approach** for the specific problem of improving LFP decoding when spike data is unavailable at inference, which is a critical scenario in neuroscience and neurotechnology applications. This argument is further supported by our new analyses.\n\n> Unfreezing the teacher network\n\nWe thank the reviewer for their question! Per the reviewer’s request, we performed an experiment on unfreezing the teacher network at different stages of the distillation training. When the teacher was **updated from the start of distillation**, the model suffered from **representational collapse**, achieving only $0.09~R^2$ on session 20160622_01 of Monkey I. We then tested a **delayed unfreezing schedule** by unfreezing the teacher network at 10th or 75th epochs of the distillation. While these prevented collapse, they resulted in slightly inferior performance: unfreezing the teacher network at 10th and 75th epochs achieved $0.668 \\pm 0.072$ and $0.674 \\pm 0.066$ average $R^2$ over 8 sessions (5 finetuning sessions of Monkey I and 3 finetuning sessions of Monkey C as in Fig. 3), where freezing the teacher network completely (**default setting**) achieved $0.681\\pm0.067$ $R^2$. \n\nWe would also like to point the reviewer to prior work showing that freezing the teacher network during distillation leads to improved student performance when the teacher is fine-tuned (Abbaspourazad et al., 2025; Li et al., 2025). Despite this prior evidence and our experience, we acknowledge that an alternative scheduling algorithm outperforming the completely frozen scenario can be developed, which we will leave as an interesting future direction.\n\nReferences:\n- Ma et al. (2022). Are Multimodal Transformers Robust to Missing Modality? IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)\n- Abbaspourazad et al. (2025). Wearable Accelerometer Foundation Models for Health via Knowledge Distillation. arXiv.\n- Li et al. (2025). Distilling Knowledge from Large-Scale Image Models for Object Detection. ECCV."}]

Author rebuttal:

Meta-review: This paper introduces an unsupervised distillation framework that transfers representations from spike-based model to a local field potential (LFP) only model, and shows that the resulting LFP models align well with spike representations and outperform LFP-only baselines in predicting behavior. The reviewers were unanimous in their assessment that the paper makes a worthwhile contribution to the literature and should be accepted.  Congratulations! Please be sure to address all reviewer comments and criticisms in the final manuscript.
Decision: Accept (poster)