Title: Cross-Modal Representational Knowledge Distillation for Enhanced Spike-informed LFP Modeling
Venue: NeurIPS 2025
Accepted: yes
Keywords: Knowledge distillation

Multi-session models

Neural decoding

Neuroscience
Abstract: Local field potentials (LFPs) can be routinely recorded alongside spiking activity in intracortical neural experiments, measure a larger complementary spatiotemporal scale of brain activity for scientific inquiry, and can offer practical advantages over spikes, including greater long-term stability, robustness to electrode degradation, and lower power requirements. Despite these advantages, recent neural modeling frameworks have largely focused on spiking activity since LFP signals pose inherent modeling challenges due to their aggregate, population-level nature, often leading to lower predictive power for downstream task variables such as motor behavior. To address this challenge, we introduce a cross-modal knowledge distillation framework that transfers high-fidelity representational knowledge from pretrained multi-session spike transformer models to LFP transformer models. Specifically, we first train a teacher spike model across multiple recording sessions using a masked autoencoding objective with a session-specific neural tokenization strategy. We then align the latent representations of the student LFP model to those of the teacher spike model. Our results show that the distilled LFP models consistently outperform single- and multi-session LFP baselines in both fully unsupervised and supervised settings, and can generalize to other sessions without additional distillation while maintaining superior performance. These findings demonstrate that cross-modal knowledge distillation is a powerful and scalable approach for leveraging high-performing spike models to develop more accurate LFP models.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a cross-modality distillation framework that leverages existing neural-spike based transformer decoders to boost the performance of LFP-based transformer decoder. The authors argue that having LFP-based transformer decoders is also important, as LFP recordings can be more easily obtained in some cases. They essentially train a LFP-based transformer by combining autoencoding loss (to ensure the latent representations can reconstruct the LFP signal) and a cosine loss (to align the latent representations of the LFP transformer decoder to the neural spike transformer decoder). The authors compared their distillation-based LFP transformer with reasonable baselines (e.g., vanilla LFP-based transformer) across multiple datasets. They show significant improvement in decoding performance of the distilled LFP-transformer in comparison to the baselines. Finally, they extend their distilled LFP-based transformer to a multi-session approach demonstrating modest improvements in decoding (increase of 0.01 and 0.02 in R2 coefficient, although statistically significant).\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n1. The paper is well-written and does a good job of explaining its problem. It is important to also develop techniques that can work with LFP data which is more readily available than its neural spike counterpart.\n\n2. The paper proposes an extension of NDT2 that performs significantly better than NDT2 (increase in behavior decoding (R2 coefficient) by roughly 0.9). The proposed extension changes the tokenization of the spiking data to allow the transformer to leverage spatial relationship in addition to the temporal ones.\n\n3. The paper uses a cross-modal knowledge distillation framework that uses neural spike-based transformer (trained with larger datasets) to train single session LFP transformer. The corresponding distilled LFP transformer significantly outperforms transformers trained using only LFP data robustly showing the value of using distillation.\n\n4. The authors perform extensive validation of their technique against reasonable baselines, and show consistent improvement.\n\n**Weakness**\n\n1. The improvements of distilled LFP models over the multi-session spike-based transformer are modest (improvement of 0.01 to 0.02 in the R2 coefficient). Since, the latent representation of the distilled LFP model is trained to be similar to the spike-neural transformer, it is not surprising that the distilled LFP model and spike-based transformers have similar decoding performance as the decoding is performed using latent representations. The fact that adding LFP data does not yield a substantial improvement seems to suggest that most of the decoding capability is resulting from the spike-based representations with the LFP data having a minor effect on improving the representation. \n\n2. The multi-modal baseline used for comparison is a bit naive. Concatenating the LFP signal and the neural spike is one of the simplest way to perform multi-modal learning. There are a lot more principled ways to perform multimodal learning, e.g., take a look at the review by Xu et al. IEEE Trans. Pattern Recognition'23. I am not expecting authors to compare their method with new multi-modal approaches, as it is not the main point of the work. I would just encourage authors to acknowledge the limitation of their multi-modal approach and discuss that the another approach to improving LFP models would be through multi-modal learning (there are multi-model learning approaches that also allow unimodal predictions). Indeed, with enough availability of both LFP data and spike-based data, it could be possible that multi-modal approaches could outperform the distillation approaches.\n\nQUESTIONS:\nOut of my own curiosity, have the authors tried not keeping the weights of the teacher model not fixed in the distillation objective? It might be possible that allowing the LFP signal to influence the representations of the spike-based transformer might lead to a better representation.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a method for learning behavioral-predictive representations of LFPs by first pretraining a spiking model and then distilling the learned representations into an LFP model. Additionally, session-specific tokens are used to align latent representations across recording sessions. In experiments predicting behavior, the distilled LFP models perform roughly as well as spike models, and outperform single- and multi-session LFP-only baseline models. A TSNE embedding and an analysis with centered kernel alignment confirm that the distilled LFP model achieves similar representations to the multi-session spike model. The main result also holds up in a supervised finetuning setting and a multi-session distillation setting.\n\nSTRENGTHS AND WEAKNESSES:\nThe idea of learning informative LFP features by distilling knowledge from a spike model is very attractive given all the benefits of LFPs listed in the abstract (stability, lower power requirements, robustness) and the generally higher predictive performance of spikes in various behavior prediction scenarios.\n\nThe transformer architecture trained using session-specific learnable tokens and a masked autoregressive objective is a reasonable and flexible approach.\n\nThe finding that an LFP-only model can predict behavior as well as a spiking model by aligning its representations is very interesting to a computational neuroscience audience.\n\nThe MS-LFP data has many fewer sessions than the MS-Spike data (34 vs. 226). It seems plausible that this could explain the poor performance of the LFP-only models. Additionally, I would expect the LFP data that is only available as LFP power to be less informative than the LFPs themselves. I find it hard to interpret the results of the LFP-only baselines given these considerations.\n\nIn the first paragraph of the Results section, there is a discussion about testing generalization to unseen subjects. But \"after pretraining, we fine-tuned the MS-spike model on the spikes of the held-out sessions,\" so it seems that there are no truly unseen subjects at the time of testing the distilled LFP model. Additionally, the results in section 4.3 are not on true hold-out sessions or animals, since they are used in pretraining. If I am following correctly, it seems the strategy taken to test the proposed method is not applicable to the case where only LFPs are available, with no shared spikes. But this is the exact scenario where the proposed knowledge distillation would be expected to be most beneficial and practical.\n\nGiven these concerns in the previous two paragraphs, I find it difficult to come away with a clean interpretation of the results of the paper.\n\nQUESTIONS:\nAre my assessments in \"Strengths and Weaknesses\" about the smaller LFP dataset and the necessity of paired LFP-spike data accurate?\n\nCan you say anything specific about the features of LFPs that allow the distilled LFP models to predict behavior about as well as the spike models? \n\nMinor comments:\n* The experiments section should state the number of tokens for pretraining and finetuning,\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces an unsupervised cross-modality distillation framework for electrophysiology, combining both spikes and local field potential (LFP). They first train a teacher spike model across multiple recording sessions using a masked autoencoding objective with a session-specific neural tokenization strategy, and then align the latent representations of the student LFP model to those of the teacher spike model. When tested on motor cortical data from monkeys, their distilled LFP model consistently outperforms the other LFP baselines. In some of the sessions tested, they find some small improvements in performance of the spike only model, suggesting that including both modalities can enhance the performance of BCI. They provide evidence that the model can align modalities while also capturing individual differences in the latent embeddings.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n- The work addresses an important challenge and aims to utilize LFP which is often discarded from analysis, and can be used with minimal preprocessing steps\n- Novel approach to use cross-modality distillation for improving decoding from LFPs\n- The results suggest that pretraining with spikes helps LFP decoding, and the incorporation of LFP can also help unimodal BCI decoding in some cases.\n\n**Weaknesses:**\n- While the results in Figure 3 demonstrate a clear advantage of the distilled multimodal model over the LFP only decoder, it is not clear whether this is a strong enough LFP baseline for this dataset. Additional baselines are needed to demonstrate the advantage of the multimodal approach. There are existing cross-modal decoders from the Schanechi and Pesaran groups, and a number of LFP-based transformer models in the literature (Brant, NeurIPS 2023; Seegnificant, NeurIPS 2024) that could be used to build a more compelling set of baselines. \n- Clarity of methods.  The entire pretraining method and finetuning is described in 3.1. This should be broken up into different subsections that delineate the pretraining objective from your downstream evaluations. Section 3.2 is also not very easy to follow. Instead of connecting back to the MAE pretraining in 3.1, the reader is pointed to pretraining in Appendix. It would be useful to give a more clear step by step guide to your method and what remains frozen vs. learned, and the dynamics of updates across the two models.\n- It is unclear whether the model has learned generalization information about the relationship between modalities, or uses LFP as additional animal-specific features. Demonstration of some across-modality transfer to new animals would be helpful to understand whether the relationship between modalities can generalize to new settings. \n- The advantage of distillation over multi-modal pretraining has not been discussed. How critical is it to pretrain with spikes and then start learning with LFP in smaller model vs. pretraining with the objective in end-to-end manner? Why have two separate models\nThere should be some alignment across modalities but also LFP carries different information, do you assume this is not useful for the downstream task and is discarded? Do you have evidence that this alignment objective is useful for decoding? Does it help to denoise the LFP? \n- Often, student teacher models require specific learning schedules. How do you unfreeze or add in supervision over training?\n- Patching over neurons doesn’t really make sense as the ordering of neurons has no specific meaning. Please justify this choice of tokenization.\n\nQUESTIONS:\n1. How do other existing LFP baselines compare? Please provide comparisons with other traditional deep learning methods (e.g., MLP, CNN, LSTM/state space model), as well as iEEG or other LFP-based transformer baselines to assess how performant a rotary-transformer is for LFP modalities. \n2. How does your distillation approach compare to a multi-modal training approach? Please compare with multimodal approaches like those described in Schanechi and Pesaran’s work on the topic. \n3. Can you please clarify the size of patches for the LFP vs. the spikes? How is the data split into train and test? Please describe the model evaluation procedure in detail for both modalities.\n4. How general is your distillation framework? Could it be applied to another decoding model (e.g. MtM, POYO)?\n5. What is the performance degradation while downsizing the teacher model instead of the student model?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-04"}