Title: A Generalist Intracortical Motor Decoder
Venue: NeurIPS 2025
Accepted: yes
Keywords: Brain-Computer Interfaces

Neuroscience

Motor Cortex
Abstract: Mapping the relationship between neural activity and motor behavior is a central aim of sensorimotor neuroscience and neurotechnology. While most progress to this end has relied on restricting complexity, the advent of foundation models instead proposes integrating a breadth of data as an alternate avenue for broadly advancing downstream modeling. We quantify this premise for motor decoding from intracortical microelectrode data, pretraining an autoregressive Transformer on 2000 hours of neural population spiking activity paired with diverse motor covariates from over 30 monkeys and humans. The resulting model is broadly useful, benefiting decoding on 8 downstream decoding tasks and generalizing to a variety of neural distribution shifts. However, we also highlight that scaling autoregressive Transformers seems unlikely to resolve limitations stemming from sensor variability and output stereotypy in neural datasets.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper builds an improved foundation model for decoding neural activity into estimates of manual behavior.  The authors use a combination of human and monkey data, about 2000 hours of data across multiple behavioral tasks.  They show improved results on several downstream decoding tasks and also perform a number of ablation and scaling experiments to explore the benefits of this large foundation model approach.  It’s a nice paper overall and well-written.\n\nMain Finding:\nThe key takeaway is that pretraining helps, primarily by improving sample efficiency. However, beyond 1.5 hours of data, there's no additional performance gain, which they attribute to data heterogeneity (in recording devices, pattern of dead channels, subjects, etc).  The generalization to out-of-distribution data is also weak in several specific cases that the authors highlight.\n\nModel Choices:\n•\tThey use a causal transformer and acknowledge that the negative results might partly due to architectural choices. \n•\tRegarding tokenization: they group spikes from 32 channels into patches per time step. This may be suboptimal, especially when transferring across devices—channel groupings lose their meaning. The model also ignores information about brain region, which is probably important for Neuropixels recordings that can span multiple regions in a single experiment.  (I’m not sure if this paper uses Neuropixels recordings but these are becoming more common in both monkey and human recordings.)\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n-\tWell-written and clear\n-\tImprovement on the SOTA\n-\tAddresses an impactful and interesting problem\n-\tRefreshingly honest about the limitations\n\nWeaknesses\n-\tSeveral of the results fig panels were pretty obscure at first read; I’d recommend looking the figures over again and trying to clarify these for non-expert readers\n\nQUESTIONS:\n-\tObvious question - is it possible to fix the out-of-distribution problems highlighted in fig 4?  (Maybe this will be the topic of the next paper.)\n-\tSome of the r2 values of the predictions are fairly low.  It might be useful to see a decomposition of the errors across different frequency bands – I’d guess that the highest frequencies are impossible to predict (due to sensor noise or unconstrained motor variability) and the r2 values for the lower frequencies may be better\n-\tIn general it would be nice to see some more predicted traces, so we get a clearer qualitative sense of what these models are predicting\n\nLIMITATIONS:\nadequately addressed", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces Neural Data Transformer (NDT) 3, a standard transformer-based model trained on large amount of neural recordings. The authors explore the extent to which the large-pretraining/foundation model paradigm apply to the field of neural decoding and BCI. To that aim, they train their model on ~2000hrs of neural data coming from either human or monkey subjects (in motor cortical areas) and test the downstream behavioural decoding performances as a function of model and dataset size. They identify that large pre-training benefits decoding when available subject data is <1.5hrs. Furthermore they explore different ID vs. OOD generalisation scenarios: (1)  different configurations when distribution shifts are due to different time or other presence/absence of spring load feedback (2) continuous/trialized training (3) unseen behaviour (novel angles in 2D center-reach).\n\nSTRENGTHS AND WEAKNESSES:\nThe paper present a comprehensive study of the benefits (and limitations) of scaling model pretraining in the field of neural decoding, a important and very active area of modern research with several potential downstream applications.\n\nWhile the proposed model employs standard tools in ML for decoder-only transformer training (e.g. rotary embeddings, custom tokens etc..), its scaling analysis reveals important and neural-specific limitations that are worthy of consideration and will inform the future development of neural decoding efforts.\n\n# Strengths\n- The paper provides extensive evaluations and supplementary experiments.\n- The identified limitations (saturation of scaling performance) are highly relevant for the field and can inform therapeutic applications.\n- The authors provide a model that can be applied to a variety of downstream task with good performance.\n\n# Weaknesses\n- The authors in the main text mention that full-comparisons with POYO were omitted as it underperformed NDT3 on the FALCON dataset. However Figure 19 in the appendix suggests a more nuanced situation. For example in the 16D Reach/Grasp EMG (FALCON M1) - mid panel in Figure 19 the POYO-like model either matched or surpassed NDT3 performance in the 100% data regime. Similar trends emerge in the other panels (e.g. in M2). Most importantly for the claims of the paper, the POYO-like model seems to enjoy more favourable scaling laws (with performance increasingly more rapidly with data size), thus hinting that perhaps scaling it to the ~2000hrs regime could have yielded better results.\n- In the paper the authors compare NDT3 vs. NDT2 (Figure 11). However NDT2 was not trained with behavioural labels and Figure 9 suggests that training with behaviour is beneficial. Hence a fairer/interesting comparison would have been NDT2 vs. NDT3-neural-only to disentangle the contribution of the design choices of NDT3.\n\nMinor points\n- In Figure 11 there is a mismatch between the legend of NDT2-100hrs (cross marker) and the plots (circle markers) - assuming color is correct.\n- There is no explicit discussion of the current limitations & the discuss itself is very limited\n\nQUESTIONS:\n1. How would NDT3-neural only compare against NDT2?\n2. Why do authors believe that NDT3 suffers high training instabilities in the H2 task (Figure 10)? Also, why does the pretrained-version have such higher initial loss? It is also interesting that the runs that exhibit training instabilities actually results in the best performance.\n3. With data quality being a crucial factor when training foundational models, what would the effect of (automated) spike-sorting be on the quality of the final model? Current pipeline only employs filtering+thresholding, yielding noisy data. While perhaps not directly applicable to the field of BCI (efficient online spike sorting is an area of active development), when applied during pre-training it could potentially improve model performance (albeit also widening the ID vs. OOD gap).\n\nLIMITATIONS:\nThere is currently no explicit discussion of the limitations of this work nor a statement about potential societal impact. The author could for example discuss the still unexplored axes that might influence model quality, e.g. spike-sorting input data, potential importance of data diversity (other brain regions) etc... or the identified pathological sensitivity to input order and stereotyped outputs.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-22"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nAuthors introduce Neural Data Transformer 3 (NDT3), a pretrained autoregressive transformer for decoding motor behavior from intracortical neural activity measured from monkeys and humans.  Pre-training the model on up to 2000 hours of neural and behavioral data, the authors evaluate the decoding performance of NDT3 on 8 motor datasets.  In this analysis, they study model performance when scaling the number of parameters in the model and the amount of pre-training data used.  The authors suggest benefits of scaling both pre-training dataset size and model parameter count, but only when the amount of downstream, task-specific data is very limited (up to ~1.5 hours of data).\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- The authors provide a wide breadth of evaluations, and highlight where NDT3 succeeds and fails.  These analyses may offer useful insights for practitioners seeking to scale their neural data decoders.\n- NDT3 achieves strong performance as compared to conventional baseline Wiener Filters in most evaluated tasks.\n- Although saturation limits were observed with relatively low quantities of motor behavior pre-training data, such pre-training strategies may be applicable in regimes where data collection is prohibitively expensive or only small amounts of new-subject calibration data can be acquired (e.g., end-user BCI applications).\n\nWeaknesses:\n- A primary motivation of this paper and neural foundation models in general is to support generalization to new tasks and subjects.  Minimal improvements are made in this regard (most notable improvements from pre-training occur when subject data is available in pre-training data or when downstream data is very limited) and little guidance is provided into how this may be achieved.\n- The benefits of pre-training are limited when there is no data from the test subject in the pre-training data.  Outside of laboratory settings, it is rarely the case that we would have training data available for a BCI user (even if from a different task), limiting practicality of this type of strategy.\n- At many points in the paper it is unclear what type of generalization is being accounted for and whether it is actually cross-subject generalization.  The authors highlight that their goal is to study \"scaling of cross-subject generalization\" (Line 125), but in many cases data from the test subject (but not task-specific data) is included in training (unless I am misunderstanding this).  I disagree that this is cross-subject generalization, since subject-specific neural data distributions can be learned from the cross task-data.\n\nQUESTIONS:\n- Clarification: Do all 200 hour and 2000 hour pre-training datasets have data from the test subject (even if not specific to the test task)?\n- If feasible, can the authors provide any numerical results for how generalization scales as a factor of number of subjects (or alternatively number of sessions, number of hours, etc.) in the training dataset while strictly excluding any data from the test subject?\n- How does performance scale with pre-training data when you can only train a linear readout (instead of fine-tuning all parameters of the model)?  Would this not be a more stringent test of the generalization of pre-trained features with scaling?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces the Neural Data Transformer 3 (NDT-3) model architecture for motor decoding from intracortical spiking activity, which is a generalist, causal transformer inspired by models such as GATO. NDT-3 builds upon NDT-2 but with some key changes: moving from a masked auto encoding (MAE) architecture to a causal transformer, and accepting behaviour tokens as inputs in addition to neural data. Other components mostly remain the same except some implementation details and evaluation schemes. The model is pretrained in self-supervised fashion on 1.75k hours of neural data, post which it is finetuned using SSL on a target calibration dataset before a decoder layer is learnt through supervised learning for behavioural decoding. NDT-3 performs comprehensive evaluations of the scaling properties of a \"vanilla\" (ViT-like) causal Transformer architecture and evaluates the cross-subject transferability of such pretrained BCI decoders. While performance results are overall positive, it highlights some concerns in cross subject transfer and handling the complexity associated with output dimensionality and generalisation to new behaviours. Overall, it is a performant model for neural decoding and highlights key avenues for future work in this subfield.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n* This is among the largest pretrained model for neural data/neural decoding, especially in the context of intracortical spiking activity, and represents a comprehensive effort at scaling model size and pretraining data.\n* While results are mostly positive for scaling, the paper highlights concerns related to output dimensionality, generalisation to heldout/unseen behaviour, and cross subject transfer of pretrained models.\n* The experimental evaluation of NDT-3 (and to some extent NDT-2 in comparison) is comprehensive, although there is a concern that only NDT-style models are considered (for the most part; see Weaknesses).\n\n**Weaknesses**\n\n* It remains an open question how many of these results, especially negative transfer results, are a consequence of design choices in NDT-style models. Some possible culprits:\n\n  * Patch tokenisation scheme adopted from ViT models\n  * Using binned spike counts instead of actual spike timings (as in POYO)\n  * Autoregressive transformer architecture which is not entirely \"causal\" – patches of neural tokens and multiple behaviours are predicted autoregressively even at the same real-world time point\n  * Pretraining + \"continual\" finetuning scheme – SSL finetuning + supervised learning but the lack of post training (maybe with RL) causing models to underperform?\n\n  and so on. While this is not meant to be a request to figure all of these questions out in one paper (which is already dense in terms of evaluations), insights would be appreciated.\n* The paper doesn't show results on FALCON H2, which is the speech task based on Willett et al. (2023) (or similar) and can have long trials of 10s+ length. This highlights some problems with Transformer-based approaches and their ability to deal with long context tasks (issues with length generalisation and/or prohibitive memory requirements) especially when dense labels are not available (the task only has trial-level sentence labels if I understand correctly). Concurrent work from Wairagkar et al. (2025) and Ryoo et al. (2025) may be worth looking at for future extensions.\n* Details on the POYO \"reproduction\" associated with Fig. 19 and Appendix D3 seem sparse/underspecified. Based on what I pieced together, there are still differences with POYO: you consider binned spikes instead of spike times, it is unclear how exactly tokenisation and training were done. I would think POYO's positive, consistent cross-subject transfer comes from unit embeddings and supervised training as opposed to cross-attention. Also, lines 1363-1364 appear to be inaccurate to me (at least as of 4 months ago): there's a checkpoint at https://github.com/neuro-galaxy/poyo and a codebase at https://github.com/neuro-galaxy/torch_brain. These might be worth looking at to iron out differences between POYO and the NDT-3 cross-attention implementation.\n* Another comment about baselines is that Jiang et al.'s recent work on NDT-MTM scaling may not entirely take into account the right hyperparameter settings at different scales, given that they only tuned them on single session and 10 session models (c.f. Yang et al. (2022)).\n* This might be me, but I found the paper dense at times, though I appreciated the insight and attention to low-level details. Re-reading it at times, I felt the message or summary of results kind of got lost with each read. It might be worth including a concise summary in list form with the key positive and negative results in one place, given the breadth of experiments.\n\n**References**\n\n* Willett, Francis R., et al. \"A high-performance speech neuroprosthesis.\" Nature 620.7976 (2023): 1031-1036.\n* Wairagkar, Maitreyee, et al. \"An instantaneous voice-synthesis neuroprosthesis.\" Nature (2025): 1-8.\n* Hee-Woon Ryoo, Avery, et al. \"Generalizable, real-time neural decoding with hybrid state-space models.\" arXiv e-prints (2025): arXiv-2506.\n* Yang, Greg, et al. \"Tensor programs v: Tuning large neural networks via zero-shot hyperparameter transfer.\" arXiv preprint arXiv:2203.03466 (2022).\n\nQUESTIONS:\n* On lines 294 and 306, did you mean Fig. 4D instead of 4C? 4C seems to be a schematic as opposed to a result on overfitting.\n* Given the prevalence of post-training in LLMs, how important do you think it is for NDT-3-style models given architectural and objective parallels?\n* Again, given the rising adoption of synthetic data pipelines in LLM pretraining, could these be used to alleviate issues of generalisation to unseen targets or other scaling issues? Are there preliminary explorations or results in this direction by authors?\n* NDT-2 had results on cross-species transfer, more specifically that monkey to human transfer did not work. Did NDT-3 alleviate these issues? Maybe I missed this but I couldn't find an explicit study of cross-species transfer, but maybe because cross-subject transfer was ambiguous/negative.\n* I see in the Appendix that you mention latency is < 20ms, are there actual values in ms for specific hardware along with memory requirements? 350M params is quite a lot so I'm curious what prediction times look like on desktop/workstation-equivalent hardware you'd find in a typical lab.\n\nLIMITATIONS:\nYes, limitations are discussed overall (the paper analyses the architecture's scaling and specific failure modes explicitly) and discusses broader impact in Appendix A.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}