A Generalist Intracortical Motor Decoder
Authors: Joel Ye
Fabio Rizzoglio
Xuan Ma
Adam Smoulder
Hongwei Mao
Gary H Blumenthal
William Hockeimer
Nicolas Guazzelli Kunigk
Dalton D. Moore
Patrick J. Marino
Raeed H. Chowdhury
J. Patrick Mayo
Aaron Batista
Steven Chase
Michael L Boninger
Charles M. Greenspon
Andrew B. Schwartz
Nicholas G. Hatsopoulos
Lee E. Miller
Kristofer Bouchard
Jennifer L Collinger
Leila Wehbe
Robert Gaunt
Venue: NeurIPS 2025 poster ()
Keywords: Brain-Computer Interfaces
Neuroscience
Motor Cortex
Primary area: neuroscience_and_cognitive_science
TL;DR: Scaling Transformers for brain-computer interfaces show gains restricted to low data regimes.

Abstract: Mapping the relationship between neural activity and motor behavior is a central aim of sensorimotor neuroscience and neurotechnology. While most progress to this end has relied on restricting complexity, the advent of foundation models instead proposes integrating a breadth of data as an alternate avenue for broadly advancing downstream modeling. We quantify this premise for motor decoding from intracortical microelectrode data, pretraining an autoregressive Transformer on 2000 hours of neural population spiking activity paired with diverse motor covariates from over 30 monkeys and humans. The resulting model is broadly useful, benefiting decoding on 8 downstream decoding tasks and generalizing to a variety of neural distribution shifts. However, we also highlight that scaling autoregressive Transformers seems unlikely to resolve limitations stemming from sensor variability and output stereotypy in neural datasets.

A Generalist Intracortical Motor Decoder
Joel Ye Fabio Rizzoglio Adam Smoulder'
Hongwei Mao?
Xuan Ma
Patrick Marino? Raeed Chowdhury? Dalton Moore
Gary Blumenthal
William Hockeimer* Nicolas G. Kunigk? J. Patrick Mayo² Aaron P. Batista" 2
Steven Chase' Adam Rouse Michael L. Boninger Charles Greenspon" R
Andrew B. Schwartz? Nicholas G. Hatsopoulos Lee E. Miller
Kristofer E. Bouchard
Jennifer L. Collinger Leila Wehbe' Robert Gaunt2t
Careegie Mellon University University Pittsburgh Northwestem University
Univeersity of Chicago University of Kansas Medical Center
Lawrence Berkeley National Laboratory
Abstract
Mapping the relationship between neural activity and motor behavior is central aim of sensorimotor neuroscience and neurotechnology. While most progress this end has relied on restricting complexity, the advent foundation models instead proposes integrating breadth of data advancing downstream modeling. We quantify this premise for motor decoding from intracortical microelectrode data, pretraining an autoregressive Transformer on 2000 hours of neural population spiking activity paired with diverse motor covariates from over 30 monkeys and humans. The resulting model is broadly useful, benefiting decoding on 8 downstream decoding tasks and generalizing to variety of neural distribution shifts. However, we also highlight that scaling autoregressive Transformers seems unlikely to resolve limitations stemming from sensor variability and output stereotypy in neural datasets. 7/gi thub. com/ jo1199/nd33
alternate avenue for broadly
Code: https
Introduction
Intracortical neural data collection is growing rapidly. This growth comprises only larger individual datasets with more neurons and higher behavioral complexity [12], but also an increase in the collective number of datasets. This wealth of data presents opportunity develop insights and applications that span datasets, provided we can reconcile their inherent diversity. Large deep networks appear very suitable for this task, so much so that the generic creation of deep networks operating on broad domain data has been termed foundation modeling (3. Efforts to create foundation models are now proliferating beyond their origins natural language processing (NLP) and computer vision (CV) into many domains of engineering and science [4]. Here, we evaluate the efficacy of foundation modeling for motor decoding from intracortical spiking activity.
Motor decoding is a valuable domain for characterizing neural data foundation models. Academic and industrial efforts create iBCIs for neuroprosthetics provide path for scaling data collection from hundreds to thousands of subject-hours, and also fuel a need for pretrained models that generalize quickly and perform robustly for new users and settings. Behavior prediction metrics for BCI performance are also more intuitive for benchmarking progress than neural data prediction or the abstract goal of providing scientific insight (e.g. with latent variable models in silico models) [ 6 Finally, recent work has shown that deep networks are able to transfer learn across motor cortical
"Correspondecc:: joelye98gmail.com
'Comespondence: rag53epitt.ede
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

A.
B.
R
0.8
Pretrained NDT3 (350M, 2000 hr)
NDT3
NDT3 from Scratch
Linear
0.6
0.4
0.2
0.0 100
101
102
Minutes of Task Data
Figure 1. A. NDT3 decodes intracortical spiking activity into low-dimensional time series for various motor effectors [] B. We aggregate decoding performance tasks with variable amounts of data (from Fig. I1 Pretrained NDT3 models reliably oulperform from-scratch models and linear baselines, up 1.5 hrs of data.
datasets collected at different timepoints, subjects, or tasks 77. These ingredients provide the motivation and means for scaling neural data modeling.
However, scaling may constrained by the unique variability present in each neural dataset. This unique variability induces irreducible error, not from the neural data signal noise, but from what the same neural data means across contexts. To be precise, we distinguish two sources of unique variability introduced by comparing across subjects. The first difference stems from the physical difference in electrodes and recorded neurons, which we term sensor variability. For example, suppose have toy 2-neuron subjects, where neuron fires leftward motion and the other fires rightward motion. No amount of pretraining will reduce the data needed from new subjects to infer each neuron's 's preferred direction. The second difference is the residual variability in subject behavior and neural dynamics. This is also significant as shared structure between neural datasets, as probed by linear methods, fade quickly beyond the data's top principal components [T0 Since pretraining cannot address the unique variability of each neural dataset, pretraining gains overall may be limited. Neural scaling efforts must therefore be alert whether performance limits are due fundamental data barriers or methodological shortcomings.
We combined simple tokenization and standard autoregressive Transformer to enable pretraining over diverse data and fine-tuning to new tasks without any task-specific parameters (Fig. TT). We pretrained this model, which we call Neural Data Transformer 3 (NDT3), using up 2000 hours of neural and behavioral data from motor neuroscience experiments with monkeys and clinical intracortical BCI (iBCI) trials with humans. We then evaluated NDT3's decoding performance on eight motor datasets (Section B.I. finding that tuning NDT3 yields models that either improve over task-specific models trained from scratch when task data under 1.5 hours, and match them beyond this limit (Fig. DB.. Further, these gains persist under real-world distribution shifts (Section (.3). These benefits may enable more complex experimental design and decrease the burden of decoder training for people using iBCIs. However, beyond the 1.5 hour limit, we observe that increasing data scale without increasing model capacity degrades performance, which provides that data heterogeneity will challenge the productivity of scaling neural data. To guide future efforts improve scaling, we analyze two measures of generalization where NDT3 currently fails, in robustness to input sensor variability and output stereotypy (Section 3.2
indicators
2 Approach
2.1 Data
NDT3 models datasets of paired neural spiking activity and behavior (Fig. 2p. Given our focus on motor decoding, most of the data comes from devices implanted in motor cortex of various monkeys
Photo courtesy of Nicho Hatsopoulos and The Chicago Tribune.

---

Hours"
10 100
Monkey
B. Neural Activity
Subjects
Fraction
0.2
0.1-
Covariates
Force
0.0
2
4
8
10 12
Kinematics
Simulotor
Covariate Dimensions
Kinematical
Noise
C.
BCI tokens:
D. Timestep
Neural Tokens:
Phase
Binned Spikes
Reward
Neur
Neur
Naur 3
Return
Bhur
Bhur
Nour
7
Causal Transformer
Trial
Start
Neur
Neur
Return
Behavior Low-D Covariates
Omitted in evaluation
Masked in evaluation
Figure 2. NDT3 Data and Model Design: A. NDT3 models paired neural spiking activity and behavioral covariate timeseries. We plot the distribution of 2000 hours of pretraining data by subjects (top) and covariate dimensionality (bottom). B. Examples of the neural and behavioral data each of the three types behavioral covariates in pretraining: kinematics, EMG (clectromyography), or forces. Not all modeled dimensions in data are meaningfully task-related (right, grey behavior). C. Neural spiking activity is tokenized in time by binning the number of spikes every 20 and in "space" using patches of channels (usually 32), as in NDT2 (8]. Behavior is low-dimensional in our data, NDT3 also pretrains on data from BCT control, which we annotate with two additional tokens. The phase token indicates whether the user is controlling or ohserving the behavior and the reward token indicates if the BCT task was completed. D. NDT3 models tokens in a single flat stream with linear readins and readouts. Every real-world timestep (shown by the blue cutout) yields several tokens, which order allow causal decoding inference-time. inference, we omit return and phase tokens and zero- mask behavior tokens.
use token per hehavior dimension, also per 20 ms timestep.
and humans (Fig. 2A). These devices are intracortical multiclectrode arrays or probes that record 30 kHz extracellular potentials. After bandpass-filtent filtering, spikes are extracted from these potentials when they exceed preset thresholds determined by the researcher. The resulting neural data in our pretraining are diverse (Fig. [ top). Data can have markedly different profiles across electrodes due to physical separation (being on different electrode arrays in the same subject (left)), and the interaction recording noise and threshold preprocessing can yield either many inactive channels (middle) or high and noisy firing (right).
The typical behaviors in the pretraining data are reaching and grasping, nearly all from experimental paradigms that consist of short, repeated trials. While neural data were always recorded from microelectrodes, motor covariate signals came from various sensors. In monkey datasets, these sensors measure actual limb activity (c.g.. BB, left: limb kinematics from optical tracking, middle: electromyography (EMG)). In human datasets, physical movements are typically not possible, so the behavioral signals are programmatically generated. These signals are "paired" with the neural data in that they are cued or otherwise instructed to the person, who will attempt or imagine the corresponding behavior, such grasping at specified force level (Fig. 2 right). This force panel also shows that pretraining, we cannot always automatically discern the primary task covariates (e.g., blue line, force, in the panel) from other recorded behavioral variables (grey). Thus, some behavior variables may be unpredictable. Finally, we include closed loop iBCI data, where some behavior is generated by an iBCI decoder (not NDT3, see modeling strategy in Section 2.2) The data used for pretraining NDT3 is detailed in Section A.3.4
We minimize preprocessing of these data to maximize the applicability of our generalist model. Kinematic signals are typically converted velocities, and all behavior (kinematics, EMG, force) normalized per dataset such that the maximum absolute value of each variable is one. Data are cut or concatenated into fixed length sequences, without additional annotation of data discontinuity. This strategy, common in language modeling while maintaining throughput of real data. We used length of two seconds as it is roughly the timescale of the behavior in our data (Fig. A). Sequences with spikes covariate variability
[12), homogenizes the data for improved GPU utilization

---

discarded. In total, this yielded about 3 million sequences, 1 billion neural tokens, or 1750 hours of recorded data. We round this to khrs in text for simplicity.
2.2 Model
NDT3 is an autoregressive Transformer with linear readin and readout layers per modality, similar to GATO or TDMs [3315). This requires data tokenization (Fig. 2C). We tokenize neural data by patching spike counts { each token vector of the binned spikes fixed temporal resolution (20 ms) and spatial dimension (32 channels). For example, spikes sampled from an electrode array with 100 channels would patch into 4 = [100/32] 32D neural tokens per 20 ms timestep. As the behavioral variables are already low-dimensional, we simply assign token per dimension at the same temporal resolution as neural data. Finally, we add tokens marking whether the behavior are generated by BCI system or by physical limb movement. While measured kinematics, EMG, or force will reflect a natural relationship with neural activity, behavioral data from BCI tasks are controlled by a program or learned decoder. We frame BCI-driven behavior suboptimal control [16] and adopt a scheme from Decision Transformers [1 timesteps where behavior is driven by neural activity (i.e. native body movement or BCI control) versus programmatic, open loop control. We also use a Return token reflecting controller quality based on future task completion. Note that these signals are only considered for pretraining, and are ablated entirely from the model at evaluation. Similarly, input behavior tokens are masked out in inference, that the model input only indicates how many behavioral dimensions must be predicted. NDT3 is trained with mean-squared error for prediction of behavioral variables, and categorical cross-entropy losses for prediction of neural spike count and return.
18 . In this scheme, we use a Phase token to denote
All modalities are flattened into single token stream, with the multiple tokens in each real-world timestep sorted to respect a canonical, causal order required for control (Fig. (2)). We use rotary embeddings [9] to track real-world timesteps, and learned, modality position embeddings to distinguish tokens within timestep. Note that NDT3's autoregressive objective will condition some tokens on other tokens from the same real- world timestep.
Pretraining and Tine-Tuning We pretrain NDT3 models over many scales of pretraining data (1.5 hrs to 2 khrs) and in sizes of 45M and 350M parameters to assess the impact of data and model scaling. Pretraining is stopped early according to validation loss or terminated at maximum of 400 epochs. The 200 hour, 45M model trains for 480 A A100-hours while the 2000 hour (2 khr) 350M model takes 20K A100-hours. Fine-tuning maintains the pretraining objectives and updates all parameters.
2.3 Evaluation Strategy
Evaluation datasets and tuning Our main evaluation (Section 3.1 uses four human and four monkey datasets sampling varied upper limb movements, which we detail in Section A.3.4 All downstream monkeys are held-out of pretraining, and all humans are held-out of < 2 khr models. That is, we evaluate scaling of cross-subject generalization, as it is the most general use case for BCI-oriented foundation models. Note the human data leakage was allowed as significant data came from humans included in me-determined evaluation datasets, and this leakage did not advantage the khr model (Section ).2.5). Each dataset contains multiple sessions of data, typically from single monkey or human. We will refer to each such setting as a "task," " distinguished from the behavioral procedure performed in each dataset. Each session has unique variability, so one fine-tuning strategy, used in
prior work 7 g 20 is to focus evaluations by tuning separate models for each evaluation session. We instead fine-tune NDT3 jointly over data combined from multiple evaluation sessions, which helps manage compute and storage demands, and better reflects that real-world deep network based BCIs typically jointly tune over multiple sessions. Fig. 12 confirms this joint tuning is comparable to per-session tuning for multi-session data.
Baselines We compare with Wiener filters (WF) and NDT2 8 WFs are conventional linear method for motor iBCI (1]. We implement them as ridge regression with multi-timestep history. NDT2 is a Transformer that uses MAE self-supervision [22 to learn across neural datasets. NDT2 currently leads on FALCON (23,, a BCI decoding benchmark that notably includes RNN baselines; we thus omit RNNs in our evaluation. We compare to NDT2 both prepared from scratch on each evaluation dataset (as in FALCON), or tuned from the public pretrained weights from pretraining on 100 hours of human data. We are also aware of two other proposals for scaled Transformers

---

spiking activity, POYO 7 and NDT-MTM (0)). We omit full comparisons with these models, as our reproduction of POYO underperforms NDT3 on FALCON when both models are trained from scratch, and NDT--MMM is qualitatively similar to NDT (see Section A.3.3)
Downstream Hyperparameters We tune all deep networks (NDT2 and NDT3) over 3 learning rates. This hyperparameter sweep is limited for computational tractability. Importantly, the same sweep is used for all tasks; we list and show its sufficiency relative to wider sweeps in Section A.3.2 The best learning rate is chosen based on the average validation score over three random seeds, and we report the average score on a separate test split.
3 Results
NDT3's pretraining advances intracortical models by an order of magnitude in data and model scale, from 200 to 2000 hours and 10M to 100M+ parameters, In Section 3.1 we show that this joint scaling is productive, but only when downstream datasets are 1.5 hours. To analyze this limit, Section 3.2 shows that despite outperforming baselines, NDT3's transfer remains pathologically sensitive to shifts in data input dimensions. Section 3.2 also shows that scaling does not improve generalization to new behaviors. Section [3.] concludes by showing that NDT3's pretraining gains, despite their limits, do generalize to novel settings provide a practical foundation for motor decoding.
3.1 Multi-scale evaluation across motor decoding tasks
To set expectations for how data scale and model size will impact performance, we first measure behavior prediction R² computed during pretraining. We report on a test split comprising multiple sessions of 2D reaching mainly from one monkey. Models can learn this as 1.5 hours of data sampled from this task are included in all pretraining datasets. The first three bars in Fig. 3 compare a model pretrained on solely these 1.5 hours, or one of two 200 hour datasets: The "Subject" model uses data from 10 other monkeys, while the "Session" model uses 200 hours from the test monkey (from separate set of experiments with similar behavior). Only the session model provides an improvement, suggesting cross- data is too distant to benefit a model that already uses 1.5 hours of task-specific data. The right three bars compare how further scaling to 2 khrs can actually degrade performance, suggesting dissimilar pretraining data can interfere with learning of the test task. This interference is mitigated by increasing model size 350M parameters, consistent with recommendations to scale model size and dataset size in tandem 24 26). However, rescued performance still remains near the 200 hour reference.
In light of previous work showing cross-subject transfer at low downstream data scales upstream saturation suggests downstream data scale might influence pretraining gain. We thus evaluate downstream at multiple data scales, illustrated in Fig. 3 These two datasets are from a human performing open loop iBCI calibration for bimanual cursor 277, and a monkey performing self-paced reach to random targets (28). Both individuals are held-out from pretraining entirely, so the task-specific data are only seen in tuning. In the bimanual task, NDT3 performance improves with increased pretraining data at all downstream data scales. The self-paced reach dataset has more ambiguous results. For example, from-scratel NDT3 is competitive at all but the smallest data scale. These examples raise the concern that pretraining efficacy will depend on not only data scale but also its quality, like the subject behavior studied. However, such variable efficacy across downstream settings is not uncommon in pretraining 22 30 and does not preclude an initial aggregate evaluation.
our
Tuning over 2000 models in 31 evaluation settings, we find that pretraining scale provides overall NDT3 outperforms the WF and NDT2 (pretrained or not). We discuss NDT2's relatively poor performance in Section A.2.7 From-scratch NDT3 performance can be improved with minimal pretraining (1.5 hrs), consistent with findings in computer vision [31]. Performance continues improving up to 2000 hours of pretraining data, but only when paired with increased model size to 350M parameters, as in pretraining. The gain of the 350M khr model over other models is significant under nonparametric Wilcoxon signed rank tests, for all comparisons except the 350M 200 hr model (Fig. BD).
benefits (Fig. 3C). To begin,
We now return to analyzing the impact of downstream dataset scale and quality on pretraining efficacy. Fig. 3E organizes downstream results by data scale, revealing that said scale is the primary factor in determining pretraining efficacy. Specifically, after normalizing by 200-hr model performance to
5

---

A.
Pretraining-tinn time evaluation
B.
Downstream evaluation
0.75
R²
Subject transfer weak
and needs larger models
R
4D Bimanual
Cursor
R²
Reach
2D Self- self-paced
Pretraining Volume (hrs)
Subject Session
45M
45M
350M
2000
0.3
0.70
0.6
200
0.65
0.5
200
200 200 khr khr
Pretrain Data Volume
LEGEND x...
NDT3 Scratch Pretrained 45M Pretrained 350M
min
0.1
0.4
10
50100%
D.
E.
200 hr Normalized R2
Downstream:
task 3-5 scales Avg.
R2
Baseline
45M param. 220M param
hD12 Seratsh
ciess Nigtiianie inp 8.000
1.5
0.3
43M 1.5
001
0.00
p-r 0.0
0.2
45N 202
0.00
0:22
1.0
0.1
0.5
0.0
Wiener Fitter
HOT2 Scratch
NOT3 Scratch
1.5 hr
2 khr
khr
350M Iehe
6:3
NO12 100
200
250
Wiener Fiter
Scretch
40H fo
438 a00
45M
May
200
0.0
10
10a
102
Minutes of Task Data
Figure 3. Evaluation diverse motor tasks: A single legend is used throughout. A. Pretraining evaluation for 5 models, the center two bars are repeated for clarity. The top evaluations across pretraining checkpoints are averaged and scattered individually. All models' pretraining data include dataset. We compare a model with just this data (1.5 hr) to using 200 hours of additional data from over 10 other monkeys (Subject) vs 200 hours from the test monkey (Session). Only the additional test monkey data improves over the calibration model. The final two models pretrain the 200 hours from the test monkey. Using 2 khrs underperforms the 200 hr model 45M parameters, with performance restored by increasing capacity to 350M. B. Examples of good and bad data-scaling in downstream multiscale evaluation on two datasets. The bottom right text shows total time used in downstream fine-tuning. The x-axis scales this full dataset down by random subsampling. Shading shows standard deviation on 3 tuning seeds. Increasing pretraining data yields performance gains at all downstream scales in the 4D task, but effects are unclear the self- saff-pace reach task. C. Downstream performance averaged for 31 settings comprised of different downstream datasets and scales, for different NDT3s and baselines. 45M NDT3s improve with data from 1.5 hrs to 200 hrs but saturate at 2 khrs. Increasing model size to 350M parameters enables further gain. D. p-values computed from FDR-corrected pairwise t-tests for each pair of models. The 350M khr NDT3 significantly outperforms other pretrained NDT3s, except the 350M 200 hr NDT3, and is the only model do so. NDT2s omitted for brevity, see Fig. 33 E. Per-task performance, normalized by the 350M 200 hr NDT3 performance, is shown against task time for different NDT3 models. Each vertical band shows models trained on the same evaluation setting, e.g. dashed lines show the evaluations from the self- reaching dataset. Model variability vanishes by 1.5 hours, implying scaling will only help below this limit.
hours of calibration data for the test
full pretraining dataset (2 khr), which includes
account for dataset variability in absolute performance, the distinction between pretrained models and from-scratel models vanishes by 1.5 hours. This convergence also explains the earlier observation of upstream saturation, as it also used 1.5 hours of calibration. The remaining variability across models at any fixed downstream dataset scale should be attributed to dataset quality. The effect significant but secondary to data scale, so we detail individual dataset results in Section [A.2.5] Moreover, a precise analysis on the impact of dataset quality is beyond the scope of this work, as our evaluation datasets have unique subjects and behaviors, while a controlled analysis sis would require multi-subject, multi-task datasets. Thus, we conclude provisionally that further scaling of NDT-style neural decoders will mainly be productive at downstream data scales <1.5 hours.
3.2
NDT3 does not generalize to novel input order and output conditions
Section 3.1 showed that pretraining gains saturate at data scales that, while larger than previously measured for neural data models, are practically low for BCI applications, as 1.5 hours can reached after few sessions of data collection. Unique variability across neural datasets does imply an

---

A.
B. R
Once pretrained, NDT3 only further transfers cross- session token-shuflles cross-session.
0.7
2khr 350M
Chonnd 7
DoloriDe
0.5
Scratch
Init Scratch/ / 2khr 350M
0.3
2. Tune
Irpuc vares
Test Culib Cnlly
Crosa-Session
Cross-Sutjecl
Channel Shu
Wulf-wwieen Shift
Token Shuffe
3. Test Co Eval
voou fired
Tune Setting
Linear (PCA-LDA)
NDT3 Scratch
Scratch 2kkhr 350M Linear (PCA-LDA)
C.
D.
Bahaviar
350M khr
E.R
drodems
0
-2
V lest dresker
-4 > KUL
X
extropooatiio??
0" 90' 180 270"
Figure 4. Pathological sensitivity to input order and stereotyped outputs. A. Setup for fine-tuning for input analysis. Colored bar marks constant physical electrode channel of neural data, boxes show how they are tokenized for NDT. Schematic uses only 8 channels and tokens, data has 96 channels and tokens. Using models initialized from scratch or from pretraining (1), we tune and evaluate performance on a self self-paced reaching dataset (3). We precede evaluation by training on intermediate setting (2) with altered neural inputs relative to (3). (3) only uses minute of data, while (2) uses several hours. B. The first three settings are no intermediate training (Test Calib Only), tuning with data Cross-Scssion data or Cross-Subject data with the same behavior. The latter settings alter cross-session data to emulate how cross-subject data might fail to transfer. Shuffle channel randomly permutes inputs, half-token shift rolls channels so that each channel uses data from i + 16, and shuffle token permutes data tokenwise, while preserving the channels grouped in each token. Gains above the Test Calib Only baseline are only preserved under token-shuffling. C. We study angular extrapolation an isometric monkey dataset where exerted forces are mapped to cursor positions 8 different angles. We use three held-in and five held-out angles. D. Predictions are made either by fitting a Wiener Filter to neural data after dimensionality reduction (Linear), or from NDT3 (Scratch, 350M 2khr). While the linear model extrapolates to held-out angles, NDT3 predictions strictly follow held-in angles. F., Pretraining improves over from-scratch in held-in angles, but far underperforms in held-out angles.
eventual ceiling on pretraining gains, but this low threshold raises the concern that our modeling decisions, like tokenization, pretraining hyperparameters, or lack of post-training, may be at fault. We next begin to dissociate these two possibilities, by highlighting NDT3's sensitivity to the specific neural inputs and behavioral outputs seen in tuning.
Inpuf order sensitivity limits cross-subject transfer. Transfer learning is greatly reduced when using cross-subject neural data relative to cross-session neural data, even when the data are collected in identical experiments and thus controlled for other variables [8]. Limited cross-subject transfer may thus be a more precise cause of limited scaling in pretraining. This hypothesis predicts that pretraining may already account for all feasible cross- subsett transfer. We show this by measuring transfer after large-scale pretraining, as schematized in Fig. 4A. Specifically, we tune NDT3 using calibration data from a single session of monkey self self-paced reach (28), optionally preceded by intermediate tuning on additional cross-session cross-subject data. Fig. 4 shows that cross-session intermediate tuning remains beneficial even after pretraining. contrast, pretrained models do not benefit from cross- subject tuning. Further, cross-subject data does not benefit Trom-scratch models beyond the pretrained baseline (Test Calib Only). This shows pretraining occludes further cross-subject transfer, but not cross-session transfer, and supports the view that scaling is limited by cross-subject variability.
Recall that cross-subject differences can in turn be decomposed into sensor variability and changes in the neural dynamics underlying subject behavior. We can isolate how NDT3 resolves sensor variability while controlling for changes in aggregate neural statistics by performing intermediate tuning on cross- sessiss data with altered input structure. We consider three alterations of decreasing strength: a complete channel shuffle, a half-token shift that preserves channel adjacencies but disrupts the channels in each (multi-channel) token, and a token shuffling (Fig. 4B Right). In all settings, intermediate tuning improves the from-scratch model, consistent with NDT's ability to transfer nontrivially across varied datasets. However, as in the cross-subjec setting, no gains outperform
7

---

the pretrained baseline (Test Calib Only). Moreover, the pretrained model only achieves positive transfer from intermediate tuning on token shuffle cross-session data. These results show NDT3's cross-session transfer, and equivalently its downstream scaling, depends on the consistency of neural data dimensions, at least at the granularity of tokens. These conclusions are also supported in extended analysis (Section A.2.6), where we vary intermediate data scales and tuning strategy.
Pretraining does not enable angular extrapolation. Increased data efficiency after pretraining suggests that NDT3 could decode a new subject's 's behavior without sampling its full dynamic range. For example, center-out tasks require the subject to make stereotyped ballistic reaches and allow us to hold out reach angles to test decoding generalization. Center-out specifically provides a useful litmus test as the neural activity underlying center-out reach can be projected to planar subspace [32), guaranteeing that some generalization is possible by construction. It is further known that non- non-pretrained deep networks fail at such held-out angle generalization generalization would be a qualitative milestone for pretraining.
33 Thus achieving this
We analyze one session of an isometric monkey center-out dataset from [33] where the monkey exerts forces in one of eight angles and its force level is mapped to cursor position (Fig. (C). We separate this data into 3 held-in and 5 held-out angles, and plot predictions from a Wiener Filter, a from-scratch NDT3, and a pretrained NDT3 (Fig. 4D). The WF, when fit to latents constructed through dimensionality reduction (PCA and LDA, see Section A.2.1 for methods), yields predictions that generalize to the held-out angle of leftward reach. Further as expected. from scratch predictions fail to generalize, and qualitatively are constrained to their nearest held-in angle. Pretrained NDT3, however, still fails to generalize, and appears to replicate from-scratch behavior of making predictions constrained to held-in angles. We quantify accuracy Fig. + showing that while pretrained NDT3 improves over Prom-scrated NDT3 overall, both models far underperform the PCA-LDA-fi fit WF on held-out generalization. This attractor strategy is undesirable for continuous BCI control (27], but is viable when the data are stereotyped, as is common in neural datasets. We replicate this result in two more settings in Section I.2. Importantly, this analysis does not confirm whether pretraining has failed to learn that simple dimensionality reduction can recover planar structure underlying ballistic reaches; plausibly, angular generalization could be achieved with a lightweight post- post-training protoco! that discourages attractors in favor of continuous decoding.
Minimal benchmarks for neural data scaling Both sensor variability and output stereotypy pose fundamental challenges in neural pretraining, rather than trivial shortcomings specific to NDT3. For example, while our analysis raises concerns with NDT's patch-wise tokenization, the alternative of neuron-level tokenization 77 still requires input-channel identification and performs poorly in our hands (Fig. 19p. Moreover, despite enthusiasm for neural foundation models, cross-subject transfer issues persist across neural modalities (4 35 Given NDT3's current state-of-the-art performance, these distillations of its current limits serve as concise benchmarks for generalization that future neural foundation models will likely need to overcome.
3.3
NDT3 does generalize to varied real world distribution shifts.
Having understood NDT's limitations under systematic synthetic shifts, we next provide examples where its pretraining usefully generalizes under real-world distribution shifts.
Neural distribution shifts Neural data are nonstationary, with shifts rising from a mix of controlled experimental variables to more speculative factors. For example, the firing rate of different channels will evolve over the course of an hour, implying distribution shift associated with change time (Fig-5h 5A top left). Shifts also occur between activity in two arm postures or whether finger motion occurs under spring load or not (Fig. 55A top middle and right). Since these shifts are common in neural data, pretraining gains should ideally be robust to their effecct We thus tune models data from one setting (in iindiistribution, ID) and measure the performance of models in that same setting and the shifted setting fout-of-distribution OOD). Positive correlation of performance in all cases implies the ID gains conferred by pretraining persist OOD. More practically, these examples suggest that pretraining benefits not dependent narrow features specific to the choice of tuning dataset.
Trial structure DNNs have been observed to overfit the temporal structure of experimental data, challenging their use in iBCI control 27 36 For example, a DNN might learn there is always no motion before the start of behavioral trial, independent of the neural activity. To date, these claims have been studied exclusively on un-pretrained deep networks. We next assess how pretraining affects
8

---

A.
Shift: Time
Shift:: Pose
Shift: Spring Lead
B.
Continuous Train
Trialized Train
R
Continuous Eval R
ID:
Center
ID: Normel
0.4
Trialized Eval
0OD: 1
doD: Edge
OOD: Spring
1.6
1.4
0.2
Firing ratio
Firing rate ratio
Firing rate ratio
hour (R?)
Edge (R
0.8
0.0
25%
50%
Session:
10 min 25%
50%
10 min
0.6
0.6
Sprina lif
0.6
R Somatosensory
R²
Oculomotor
0.7
0.4
0.6
0.3
0.4
0.6 Center (R?)
0.4
0.2
0.2
0.4
0.6
0.8
0.5
1 minute (R?)
Normal (R²)
0.4
"Session: 4 min
0.2
Session: 40 min
LEGEND x Wiener Filter x Scratch
200hr 45M 2khrr 350M
25%
50%
28 min
25%
50%
10 hr
Figure 5. Generalization of pretraining gains. A. Models fine-tuned in one distribution of data are evaluated in-distribution (ID) and out-of-distribution (OOD). Top plots show the distribution across channels of neural firing rates from OOD and trials, normalized by average ID firing rates. Lower plots scatter OOD vs ID performance, with each point being a single model with different hyperparameters. The time shift uses two human cursor datasets collected one hour apart. Models were tuned in each block and were evaluated in the second block. Pose shift uses monkey center-out reach task which was performed with the hand starting in different locations the workspace. Spring Load uses dataset of monkey ID finger motion with or without spring force feedback. B. Models are evaluated on a human open-loop cursor dataset prepared in two ways. Trialized training receives inputs according to trial boundaries, varying from 2-4 seconds in length. Continuous training receives random 1 second snippets (that can cross trial boundaries). Trialized evaluation matches trialized training, and continuous evaluation is done by streaming up to second of history. indicates points below 0.0. Continuously trained models perform well both evaluation settings, while models trained on trialized data fail in continuous evaluation. C. Multiscale fine-tuning performance of NDT3 on datasets recorded outside motor cortex, namely S1 (Somatosensorry and FEF/MT (Oculomotor).
this overfitting in open loop human cursor control data by comparing a continuous and trialized setting. In the continuous setting, we cut random one second intervals of data in training and continuously stream up to one second of data in evaluation. The trialized setting formats data to respect trial boundaries, so the model always sees data aligned to the start of behavior.
In Fig.5B, we show that models using both trialized training and evaluation outperform models with both continuous training and evaluation. This implies NDT3 will learn exploit clear trial structure in data. However, while trialized from-scratch models become subtrivial under continuous evaluation (solid blue line is off-panel), pretrained models degrade more gracefully. For example, the 350M 2 khr model evaluated continuously only performs slightly worse with trialized tuning than with continuous tuning. Pretraining NDT3 thus reduces its dependence on trial structure, which should benefit both data analysis and iBCI control. Note, however, the contrast in these results with Fig. ED. which show that DNNs clearly do overfit to tuning data in some cases. These nuances underscore the importance of rigorously evaluating model generalization in future work.
New brain areas In Fig. 5C, return to multiscale fine-tuning to test how NDT3, pretrained on motor cortex, performs in somatosensory cortex (S1) and oculomotor areas (FEF and MT). The SI data shares the same self-paced reaching behavior of Fig. 3B. while the Oculomotor data is an analogous visual pursuit task. The gain from pretraining over from-scratch models is high in S1, but also nontrivial in the Oculomotor dataset. While the former can be attributed the close interaction of sensorimotor areas, the latter implies NDT3 has learned a broader prior. Although these results are encouraging, our previous results suggest these gains could reflect common experimental artifacts like trial structure, rather than neurophysiological priors (e.g. declining subject focus over time (37). For example, this Oculomotor dataset contains 4 behavioral conditions, which may benefit from the tendency to learn classifiers shown in Fig. 4 rather than prior neural dynamics.
4 Discussion
Broader Impacts NDT3 built as generic pretrained model to accelerate BCI motor decoding workflows. To the extent it can, and with the cost and complexity of pretrained deep networks

---

over simpler decoders considered, such BCI devices are intended to restore function to individuals with disabilities impairments resulting from brain injury or disease. Human BCIs do however pose risks for data privacy and personal agency 38 which are accentuated by the adoption of pretraining paradigms. NDT3 may also be used non-human primate BCI research. We believe that the increased data efficiency of pretrained models in those settings can help reduce the need for redundant data collection.
Limitations This work provides an initial assessment of scaling up neural data decoding with a Transformer. Beyond the explicitly analyzed limitations in Section 3.2 this work can be expanded on both architectural and data design axes. Architectural choices include the use of different neural features (spike times, bandpower) or metadata features (per-day embeddings). Data design axes are harder to explore, as it is constrained by the expense of creating intracortical datasets (public or not). They are nonetheless vital to understand, as NDT3's results may poorly predict scaling trends on datasets with more diverse neural activity or behavior, with unconstrained (naturalistic) experimental paradigms, or with many more (1000s) of subjects.
Many fields are now pursuing large-scale deep learning as "a tide that lifts all boats" 39 hoping that improvements in effective pretraining will yield field-wide, downstream improvements. Such unifying abstraction would be timely for neuroscience, given the increasing volume, diversity, and complexity of modern neural data. Joining other pretraining efforts on varied modalities of neural data (Section A.I), we trained NDT3 on 2000 hours of paired neural population activity from motor cortex and behavior, and then conducted a broad downstream decoding evaluation. Consistent with other foundation models, we found the best aggregate performance comes from increasing data scale and model size jointly (Section B.Tp. We release the 350M khr NDT3 as a strong baseline foundation model for intracortical decoding from spiking activity, with the disclaimer that benefits will likely be minimal by 1.5 hours of downstream data. To move forward, we highlight input order sensitivity and output attractors as issues future foundation models should address.
Acknowledgments
JY is supported by the DOE CSGF (i.e. U.S. Department of Energy, Office Science, Office of Advanced Scientific Computing Research, Department of Energy Computational Science Graduate Fellowship under Award Number DE-SC0023112). This research was supported part by the University of Pittsburgh Center for Research Computing, RRID:SCR 022735, through the resources provided. Specifically, this work used the H2P cluster, which is supported by NSF award number DAC-2117681. This research also used resources of the NERSC, Department of Energy Office of Science User Facility, granted as part of JY's fellowship. Research reported in this publication was supported by the National Institute of Neurological Disorders and Stroke of the National Institutes of Health under Award Numbers ROINS121079 and UH3NS107714, The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health.
10

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Our abstract and introduction describe the two core results of the paper, that scaling pretraining on BCI data is productive but mainly in limited regimes.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Main results in Section 3.1 emphasize qualifications around performance gains, and Section 3.2 analyze model shortcomings.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiitiinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g. if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?

---

Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proof's provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Our codebase for pretraining and evaluation is released, and an order of 100 hours of public data can be used pretrain smaller models. Our largest scale model depends on private pretraining data, but the model weights are released. We provide fine-tuming notebook reproduce our evaluations of these models on public datasets. Code is also available to reproduce limitation analyses in Section 3.2
Justification:
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (c.g., to registered users), but it should be possible for other researchers to have some path to reproducing verifying the results.
12

---

reproducible, they 5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, described in supplemental material?
Answer: [No]
Justification: Code is available and supports the scraping and preprocessing of public data used in this work. The majority of pretraining data is private due to several factors, including the overall sensitivity of human neural data, the necessity of releasing private collaborator data being used in ongoing research, and the fact that the majority of data is unlabeled and not suitable for public release. Configurations for each experiment in the main result are documented in the codebase.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cccppbbll
Taaddes/CodeSubmisssooPPoo
for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https //npss ccc//ullicggiiis//oooeeee/ wwtosubbisssiooooiioooooooo for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer:
[Yes]
Justification: Main details are described in text, and code for all configurations are released.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Our main scaling performance is tested with paired statistical comparisons. Variability across random seeds in subsequent analysis is shown either as raw data or as standard error.
Guidelines:
The answer NA means that the paper does not include experiments.
13

---

The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results are out range (e.g. negative error rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We provide reference compute costs in GPU-hours for the 45M 200 hour and 350M 2000 hour runs, which dominated compute costs for the project.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code ethies
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics httpp::///eerr//ee http:://eeerree..cooooccccccccaa
Answer: [Yes]
Justification: have reviewed the Code of Ethics. Human and animal data in this study were collected for previous studies approved by respective home institutions and federal regulatory bodies.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
14

---

Justification: A broader impact statement is available. It is tentatively in the supplement as the negative impact fairly distant from the main focus of this work, but it can moved to main text with extra page.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation. generating fake profiles, surveillance), faimess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: We do not believe the released pretrained models have high risk for misuse as their utility limited to data from restricted clinical devices.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: Data used from public and private sources are appropriately cited. Scraping seripts with URLs are available the codebase. All other assets are original.
Guidelines:
The answer NA means that the paper does not existing assets.
15

---

The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g.. CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms of use the package should be provided. For popular datasets, ppaperswi thcode. com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-per re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: Pretrained models are made available with model cards located in the codebase. (Redacted in publication for anonymity).
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/mooel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asseot is used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots. if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: No new instructions were provided human participants for the purpose of this study.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with
human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [Yes]
16

---

Justification: No new potential risks were incurred by this study as it was conducted with data collected for other studies. These other studies were performed with the appropriate regulatory approvals. Animal datasets were collected with approval by Institutional Animal Care and Use Committees. Human datasets were collected with Institutional Review Board approval, as part of clinical trials conducted under FDA Investigational Device Exemptions.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods this research? Note that if the L.LM used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, or originality the research, declaration not required.
Answer: [NA]
Justification: LLMs not used in original components of this research.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, non-standard components.
Please refer to our LLM policy should or should not be described.
htt.pp/cccccccffeeeeeee//////// for what
For data collection, the animal datasets used in this work were collected for other studies that were approved by Institutional Animal Care and Use Committees. Human datasets were also collected for other studies. These studies were performed under an approved Investigational Device Exemption from the FDA and were approved by the Institutional Review Board the University of Pittsburgh. The clinical trial is registered at clinicaltrials.go (ID: NCT011948022 and informed consent was obtained before any experimental procedures were conducted. Details on the implants and clinical trial are described in 40 41 We discuss the potential benefit of NDT3 to reduce user burden for iBCI-based neuroprosthetics, though the dissemination of pretrained models on these data raise the privacy concern that the original human data may be recoverable from model weights. Since this seems technically challenging at this point, and since the source data are restricted binned spiking activity to begin with, we deem the risk low enough to justify the potential scientific benefit of sharing our pretrained models.
17

---

References
[1] Anne E. Urai, Brent Doiron, Andrew M. Leifer, and Anne K. Churchland. Large-seale neural recordings call for new insights to link brain and behavior. Nature Neuroxcience, 25:11-19, 2022. doi: 10.1038/ 141550021-00000--1
[2] lan H. Stevenson. Tracking advances in neural recording. Statistical Neuroscience Lab, University of Connecticut, 2023. URL hnttps://steeeensnn 1ab. uconn. edu/scaling/ Accessed September 6, 2024.
[3] Rishi Bommasani, Drew A. Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S. Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, Erik Brynjolfsson, Shyamal Buch, Dallas Card, Rodrigo Castellon, Niladri Chatterji. Annie Chen, Kathleen Creel, Jared Quiney Davis, Dora Demszky, Chris Donahue, Moussa Doumbouya, Esin Durmus, Stefano Ermon, John Etchemendy, Kawin Ethayarajh, Li Fei-Fei, Chelsea Finn, Trevor Gale, Lauren Gillespie, Karan Goel, Noah Goodman, Shelby Grossman, Neel Guha, Tatsunori Hashimoto, Peter Henderson, John Hewitt, Daniel E. Ho, Jenny Hong. Kyle Hsu, Jing Huang. Thomas Icard, Saahil Jain, Dan Jurafsky. Pratyusha Kalluri, Siddharth Karamcheti, Geoff Keeling, Fereshte Khani, Omar Khattab, Pang Wei Koh, Mark Krass, Ranjay Krishna, Rohith Kuditipudi, Ananya Kumar, Faisal Ladhak, Mina Lee, Tony Lee, Jure Leskovec, Isabelle Levent, Xiang Lisa Li, Xuechen Li, Tengyu Ma, Ali Malik, Christopher D. Manning, Suvir Mirchandani, Fric Mitchell, 7anele Munyikwa, Suraj Nair, Avanika Narayan, Deepak Narayanan, Ben Newman, Allen Nie, Juan Carlos Niebles. Hamed Nilforoshan, Julian Nyarko, Giray Ogut, Laurel Orr. Isabel Papadimitriou. Joon Sung Park, Chris Piech. Eva Portelance, Christopher Potts, Aditi Raghunathan. Rob Reich, Hongyu Ren, Frieda Rong. Yusuf Roohani, Camilo Ruiz, Jack Ryan, Christopher Ré, Dorsa Sadigh, Shiori Sagawa, Keshav Santhanam, Andy Shih, Krishnan Srinivasan, Alex Tamkin, Rohan Taori, Armin W. Thomas, Florian Tramèr, Rose E. Wang. William Wang, Bohan Wu, Jiajun Wu, Yuhuai Wu, Sang Michael Xie, Michihiro Yasunaga, Jiaxuan You, Matei Zaharia, Michael Zhang, Tianyi Zhang, Xikun Zhang, Yuhui Zhang. Lucia Zheng. Kaitlyn Zhou, and Percy Liang. On the opportunities and risks of foundation models, 2022. htL.hhttss///aree http://///aaiioooraaa///////
87258
[4] Hanchen Wang, Tianfan Fu, Yuanqi Du Wenhao Gao, Kexin Huang, Ziming Liu, Payal Chandak, Shengchao Liu, Peter Van Katwyk. Andreea Deac, et al. Scientific discovery in the age of artificial intelligence. Nature, 620(7972):47-60,, 2023.
[5] Felix Pei, Joel Ye, David M. Zoltowski, Anqi Wu, Raced Hasan Chowdhury, Hansem Sohn, Joseph E O'Doherty, Krishna V. Shenoy, Matthew Kaufman, Mark Churchland, Mehrdad Jazayeri, Lee Miller, Jonathan W. Pillow, Memming Park, Eva I. Dyer, and Chethan Pandarinath. Neural latents benchmark 221: Evaluating latent variable models of neural population activity. In Thirty-fifth Conference Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021. URL https:///peerrview..rt het.:e/wweettoooooooeoosssss
[6] Eric Y Wang, Paul G Fahey, Kayla Ponder, Zhuokun Ding, Andersen Chang, Taliah Muhammad, Saumil Patel, Zhiwei Ding, Dat Tran, Jiakun Fu, et al. Towards a foundation model the mouse visual cortex. bioRxiv, 2023.
[7] Mehdi Azabou, Vinam Arora, Venkataramana Ganesh, Ximeng Mao. Santosh Nachimuthu, Michael Mendelson, Blake Richards, Matthew Perich, Guillaume I Lajoic, and Eva Dyer. A unified, scalable framework for neural population decoding. Advances in Neural Information Processing Systems, 36, 2024.
[8] Joel Ye, Jennifer L Collinger, Leila Wehbe, and Robert Gaunt. Neural data transformer 2: Multi-context pretraining for neural spiking activity. In Thirth-seventh Conference Neural Information Processing Systems, 2023. URL http:pps://oppeeevieeeeeeeeee//e wwwp:iiwwwwweeeoonenimiem
[9] Steffen Schneider, Jin Hwa Lee, and Mackenzie Weygandt Mathis. Learnable latent embeddings for joint behavioural and neural analysis. Nature, May 2023. ISSN 1476-4687. doi: 10. 01080/441586-023-00003--66
htRLhttpss:///// wrg.ta. 1038/s41586 022118588-------0 00055-000---6---6---
[10] Juan A. Gallego, Matthew G. Perich, Stephanic N. Naufel, Christian Ethier, Sara A. Solla, and Lee E. Miller. Cortical population activity within preserved neural manifold underlies multiple motor behaviors. Nature Communications, (1):4333, Oct 2018. ISSN 2041-1723. doi: 10.1038/s41167 4670000-000.-01-0-66------- URL
https://www. http://ww..aattre.coom wwt.://atture.cooaarrrttee////ss -08
[11] Mostafa Safaie, Joanna Chang, Junchol Park, Lee Miller, Joshua Dudman, Matthew Perich, and Juan A Gallego. Preserved neural dynamics across animals performing similar behaviour. Nature, 623 (7988):765-77)) 77777,, 2023.
[12] Jonas Geiping and Tom Goldstein. Cramming: Training language model on a single gpu in one day. 2022. URL httpss//a/atx.......p/ wttp////////...aoo///// 14034
18

---

[13] Scott Reed. Konrad Zolna, Emilio Parisotto, Sergio Gomez Colmenarejo, Alexander Novikow, Gabriel Barth-Maron, Mai Gimenez, Yury Sulsky, Jackie Kay, Jost Tobias Springenberg. Tom Eccles, Jake Bruce, Razavi, Ashley Edwards, Nicolas Heess, Yutian Chen, Raia Hadsell, Oriol Vinyals, Mahyar Bordbar, and Nando de Freitas. A generalist agent, 2022. URL
[14] Ingmar Schubert, Jingwei Zhang, Jake Bruce, Sarah Bechtle, Emilio Parisotto, Martin Riedmiller, Jost To- bias Springenberg, Arunkumar Byravan, Leonard Hasenclever, and Nicolas Heess. A generalist dynamics model for control, 2023. Utt.httss://arkk httpt//ss//aaxilooo//////////// wrg./bs//./......./ 18912
[15] Team Chamelcon, Chamelcon: Mixed-modal carly-fusion foundation models. urXiv:2405.09812 2024.
arXiv preprint
[16] Josh Merel, David Carlson, Liam Paninski, and John P Cunningham. Neuroprosthetic decoder training as imitation learning. PLoS computational biology, 1004944,, 2016.
[17] Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Michael Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learnngg via sequence modeling. 2021. URL https://arxii wttp:////////...ooo///// 01345
[18] Kuang- Huci Lee, Ofir Nachum, Mengjiao Yang. Lisa Lec, Daniel Freeman, Winnic Xu, Sergio Guadarrama, lan Fischer, Eric Jang, Henryk Michalewski, and Igor Mordatch, Multi Multi-game decision transformers, 2022. URL httpps//arrxiv. http://:///aaxw.oo////absee//e/e wttpw//...wgga///....../ 15241
[19] Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding, 2023. http/shts:://rxiii www..oggg/aaaa.e. hat///bb//2e/ 225./.11.... 89864
[20] Yizi Zhang. Yanchen Wang. Donato Jimenez- Zixuan Wang. Mehdi Azabou, Blake Richards, Olivier Winter. The International Brain Laboratory. Eva Dyer, Liam Paninski. al. Towards universal translator" for neural dynamics at single-cell, single-spike resolution. arXiv preprint arXiv:2407. 2024.
[21] Chethan Pandarinath and Sliman J Bensmaia. The science and engincering behind sensitized brain- controlled bionic hands. Physiotogical Reviews, 102(2):551 604, 2022.
[22] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, Ross Girshick. Masked autoencoders are scalable vision learners, 2021.
[23] Brianna M Karpowicz, Joel Ye, Chaofei Fan, Pablo Tostado-Marcos, Fabio Rizzoglio, Clay Washingion, Thiago Scodeler, Diogo de Lucena, Samuel R Nason-Tomaszew Matthew J Mender, Xuan Ma. Eze- quiel Matias Arneodo, Leigh R Hochberg, Cynthia Chestek, Jaimie Henderson. Timothy Q Gentner, Vikash Gilja, Lee E Miller, Adam Rouse, Robert A Gaunt, Jennifer L Collinger, and Chethan Pandarinath. Few-shot algorithms for consistent neural decoding (falcon) benchmark. bioRxiv, 2024, doi: 10.1101/ 2024.09. 15.66995...11166. /ttp/http:://ww... wwwrwwv.wwiii..i.........wwwwwwo wwtp///ttttt//aa////eeo/e/////ae http://oonnenn//raa///////////// wwwpaall///2////1/////0
15. 613126
[24] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenhorn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakoh Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recogmition at scale. In International Conference Learning Representations. 2021. /RL/nttps:/ http:///ppeneeiieeee/errooooo YicbFaNTTy
[25] Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Joan Puigcerver, Jessica Yung, Sylvain Gelly, and Neil Houlsby. Big transfer (bit): General visual representation learning, 2020. URL http:/ttts::///..orrr//////
1912. 1137Ø
[26] Armen Aghajanyan, Lili Yu, Alexis Conneau, Wei-Ning Hsu, Karen Hambardzumyan, Susan Zhang, Stephen Roller, Naman Goyal, Omer Levy, and Luke Zettlemoyer. Sealing laws for generative mixed- modal language models. In International Conference on Machine Learning, pages 265-279. PMIR, 2023.
[27] Darrel R Deo. Francis R Willett, Donald T Avansino, Leigh R Hochberg, Jaimie M Henderson, and Krishna V Shenoy. Brain control of bimanual movement enabled by recurrent neural networks. Scientific Reports, 14(1):1998, 2024.
[28] Joseph E. O'Doherty, Mariana M. B. Cardoso, Joseph G. Makin, and Philip N. Sabes. Nonhuman primate reaching with multichannel sensorimotor cortex electrophysiology, May 2017. URL https //doi. ooooi.org/10... 8. 5281/zenodo. 788569
19

---

[29] Lucas Beyer, Andreas Steiner, André Susano Pinto, Alexander Kolesnikov, Xiao Wang, Daniel Salz, Maxim Neumann, Ibrahim Alabdulmohsin, Michael Tschannen, Emanuele Bugliarello, Thomas Unterthiner, Daniel Keysers, Skanda Koppula, Fangyu Liu, Adam Gryener, Alexey Gritsenko, Neil Houlsby, Manoj Kumar, Keran Rong, Julian Eisenschlos, Rishabh Kabra, Matthias Bauer, Matko Bošnjak, Xi Chen, Matthias Minderer, Paul Voigtlaender, Ioana Bica, Ivana Balazevic, Joan Puigoerver, Pinelopi Papalampidi, Olivier Henaft, Xi Xiong. Radu Soricut, Jeremiah Harmsen, and Xiaohua Zhai. Paligemma: versatile 3b vlm for transfer, 2024. WtRLhhtts:///arkk org/ ht/p//bb////2 .77226
[30] Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger. Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray. Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever. and Dario Amodei. Language models are fow-shot learners, 2020. URL http:::/arxiv..rgg////// wttporr//////...////////// 14165
[31] Rahim Entezari, Mitchell Wortsman, Olga Saukh, M Moein Shariatnia, Hanie Sedghi, Ludwig Schmidt. The role of pre-training data in transfer learning. arXiv preprint 2302. 2023.
[32] Mark M Churchland, John P Cunningham, Matthew Kaufman, Justin Foster, Paul Nuyujukian, Stephen Ryu, and Krishna V Shenoy. Neural population dynamics during reaching, Nature, 487(7405); 51-56, 2012.
[33] Fabio Rizzoglio, Ege Altan, Xuan Ma, Kevin L. Bodkin, Brian M. Dekleva, Sara A. Solla, Ann Kennedy, and Lee F. Miller. Monkey-to-human transfer of brain-computer interface decoders. bioRxiv, 2022. doi: 10. 11.01/022.11.12.51111.. 12. 515848
URL. https://ww..borrett
[34] Hubert Banville, Yohann Benchetrit, Stéphane d' Ascoli, Jérémy Rapin, and Jean-Rémi King. Scaling laws for decoding images from brain activity. arXiv preprint arXiv:2501.15322 2025.
[35] Ann-Kathrin Kiessner, Robin T Schirmeister, Joschka Boedecker, and Tonio Ball. Reaching the ceiling? empirical scaling behaviour for deep eeg pathology classification. Computers in Biology and Medicine, 178:108681. 2024.
[36] Joseph Costello, Hisham Temmar, Luis Cubillos, Matthew J Mender, Dylan M Wallace, Matthew Willsey, Parag G Patil, and Cynthia A Chestek. Balancing memorization and generalization in mns for high performance brain-mas machine interfaces. bioRxiv, pages 2023 05, 2023.
[37] Nicholas A Steinmetz, Peter Zatka-Haas. Matteo Carandini, and Kenneth D Harris. Distributed coding of choice, action and engagement across the mouse brain. Nature, 576(7786):66-2733 2019.
[38] Eran Klein, Tim Brown, Matthew Sample, Anjali R Truitt and Sara Goering. Engineering the brain: ethical issues and the introduction of neural devices. Hastings Center Report, 45(6):26 35, 2015.
[39] Samira Abnar, Mostafa Dehghami, Behnam Neyshabur, Hanie Sedghi. Exploring the limits of large scale pre-training, 2021. URL httppss://aaxvi.oor//aaaaaeea
02095
[40] Jennifer L Collinger, Brian Wodlinger, John E Downey, Wei Wang, Elizabeth C TTyler-Kabare Douglas J Weber, Angus JC McMorland, Mcel Velliste, Michael L Boninger, and Andrew B Schwartz. High- performance neuroprosthetic control by individual with tetraplegia. The Lancer, 381 (9866))557-5644 2013.
[41] B Wodlinger, E Downey, E C Kaler-Kabara, A B Schwartz, M L Boninger, and J L. Collinger. Ten- dimensional anthropomorphic arm control a human brain-machine interface: difficulties, solutions, and limitations. Journal of Neural Engineering, 12(1):01601., Feb 2015. ISSN 1741-2560, 1741-2552, doi: (0.008//1411-256012221 20110///00/////11111111 URL htttps://oopsciieeeee iop. wwt./rraca.te.roorooo 1088/1741-2566/12//11
016811
[42] Weibang Jiang, Liming Zhao, and Bao liang Lu. Large brain model for learning generic representations with tremendous EEG data BCL. In The Twelfth International Conference on Learning Representations. 2024. RtRL//tttts///peeevvviit http://///ppprreeeeeeeee//rrrrr htt///ooummiddee htw.:/nnee/forrruooooooooottrtt
[43] Zhizhang Yuan, Fanqi Shen, Meng Li, Yuguo Yu, Chenhao Tan, and Yang Yang. Brainwave: A brain signal foundation model for clinical applications, 2024, URL httpps////axxivoo/////////222 222.
10251
[44] Chaoqi Yang. M. Brandon Westover, and Jimeng Sun. Biot: Cross-data biosignal learning in the wild,
2023. URL https://arxxx wwtpaaawwwwvvrrar/raaaaaaaaa/e/e wttp.o//g.....aobb/////// 10351
20

---

[45] Rahul Thapa, Bryan He, Magnus Ruud Kjaer, Hyatt Moore, Gauri Ganjoo, Emmanuel Mignot, and James Zou. Sleepfm: Multi-modal representation learning for sleep across brain activity, eog and respiratory signals, 2024. URL https:////// http:////aarivvoogaaa/////s/
17766
[46] Armin W. Thomas, Christopher Ré, and Russell A. Poldrack. Self-supervised learning of brain dynamics from broad neuroimaging data, 2023.
[47] Josue Ortega Caro, Antonio Henrique de Oliveira Fonseca, Syed A Rizvi, Mattco Rosati, Christopher Averill, James L Cross, Prateek Mittal, Emanuele Zappala, Rahul Madhav Dhodapkar, Chadi Abdallah, and David van Dijk. BrainLM: A foundation model for brain activity recordings. In The Twelfth International Conference on Learning Representations. 2024. URL httpp::/ wetp://ne//oormmmomoooo////EE/
[48] Christopher Wang, Vighnesh Subramaniam, Adam Uni Yaari, Gabriel Kreiman, Boris Katz, Ignacio Cases, and Andrei Barbu. BrainBERT: Self superviee representation learning for intracranial recordings. In The Eleventh International Conference on Learning Representations, 2023. URL HtRL/htts::///ttg hntpps://openreview.nn Forum?id=xmet reUn6
[49] Geeling Chau, Christopher Wang, Sabera Talukder, Vighnesh Subramaniam, Saraswati Soedarmadji, Yisong Yue, Boris Katz, and Andrei Barbu. Population transformer: Learning population-level representa- tions of intracranial activity, 2024. URL httpps//aaxxxy. wwwp/r///////eeeee/////eee httpp//////...... 03844
[50] Sabera J Talukder, Jeanifer J. Sun, Matthew K Leonard, Bingni W Brunton, and Yisong Yue. Deep neural imputation: A framework for recovering incomplete brain recordings. In NeurlPS 2022 Workshop on Learning from Time Series for Health, 2022. URL
[51] Samuel M Peterson, Shiva HI Singh, Benjamin Dichter, Kelvin Tan, Craig DiBartolomeo. Devapratim Theogarajan, Peter Fisher, and Josef Prrvzzi. Ajile12 Long-term naturalistic human intracranial neural recordings and pose. Scientific Data, 9(1): 2022. ISSN 2052-4463, doi: 10. 0188844197-002-012888
URL httLhhtps:///o/ org/18. 1038/ 44 -22-8-----------
[52] Adrien Doerig, Rowan Sommers, Katja Seeliger, Blake Richards. Jenann Ismael. Grace Lindsay. Konrad Kording, Talia Konkle, Marcel A. J. Van Gerven, Nikolaus Kriegeskorte. and Tim C. Kietzmann. The neuroconnectionist research programme, 2022. URL WtRLhhtps://arxiv httpss:/arrtv.org//d/////2
83718
[53] Richard Antonello. Aditya Vaidya, and Alexander G. Huth. Scaling laws for language encoding models in fmri, 2024. URL https::/arrxx wtwpor///abb////oe 11863
[54] Tyler Benster, Guy Wilson, Reshef Elisha, Francis R Willett, and Shaul Druckmann. A cross-modal approach to silent speech with llm-enhanced recognition. arXiv preprint arXiv:2403.05588 2024.
[55] Weihao Xia, Raoul de Charette, Cengiz Öztrelli and Jing-Hao Xue. Umbrae: Unified multimodal brain decoding. In European Conference on Computer Vision (ECCV), 2024.
[56) Quilee Simeon, Leandro Venâncio, Michael A Skuhersky, Aran Nayebi, Edward Boyden, and Guangyu Robert Yang. Scaling properties for artificial neural network models a small nervous system. In SoutheastCon 2024, pages 516-522 524. IEEE, 2024.
[57] Motoshige Sato, Kenichi Tomcoka, Ilya Horiguchi, Kai Arulkumaran, Ryota Kanai, and Shuntaro Sasai, Scaling law in neural data: Non-invasive speech decoding with 175 hours of eeg data, 2024. URL https://arxiv.. http://xxxxx.orrr/bbbb//// wrg./rb///bbb/////// 07595
[58) Margaret Mitchell, Simone Wu, Andrew 7aldivar, Parker Barnes, Lucy Vasserman, Ben Hutchinson, Flena Spitzer, Inioluwa Deborah Raji, and Timnit Gebru. Model cards for model reporting. In Proceedings of the Conference on Fairness, Accountability, and Transparency, FAT '19. ACM, January 2019. doi: 10. 01555328756032287599 URL http://dx. www..o///...o..ooooooooooo.o wtt.:o///11..11/1//1111 1145/3287560 3287596
[59] Edward H Adelson, James R Bergen, et al. The plenoptic function and the elements of early vision, volume 2. Vision and Modeling Group, Media Laboratory, Massachusetts Institute of .... 1991.
[60] Gregor Bachmann and Vaishnavh Nagarajan. The pitfalls of next-token prediction, 2024. URL https
7/axxiv Varxxiv.org/ass/24e: 06963
[61] Colin Lca, Rene Vidal, Austin Reiter, and Gregory D. Hager. Temporal convolutional networks: A unified approach to action segmentation, 2016. httphhtts///axxworrrreee
88242
[62] Abhimanyu Das. Weihao Kong, Rajat Sen, and Yichen Zhou. A decoder-only foundation model for time-series forecasting, 2024. URL /ttpss/::///arxi http:///aaavvvvo//aa////// orgg/bs//31......o 19688
21

---

[63] Demetres Kostas, Stéphane Aroca-Ouellette, and Frank Rudzicz. BENDR: Using transformers and contrastive self-supervised learning task learn from massive amounts of EEG data. Frontiers in Human Neuroscience, 15, 2021. ISSN 1662-5161. doi: 10.3389//nhum.221.65333 // www..ooiissioooggrrarraaaeeeeeee wtt.e//cttrrrrre/a/cootcccccctee 10. 3389/ fww.///hhmmmmmmm
URL https
[64] Thomas Wang. Adam Roberts, Daniel Hesslow, Teven Le Scao, Hyung Won Chung, Iz Beltagy. Julien Launay. and Colin Raffel, What language model architecture and pretraining objective work best for zero-shot generalization?, 2022. URI. httttpp://aaxiiv.ooot ottgg//b//////aa/a/ .08822
[65] Nur Muhammad Mahi Shafiullah, Zichen Jeff Cui, Ariuntuya Altanzaya, and Lerrel Pinto. Behavior transformers: Cloning k modes with one stone, 2022. URL
11251
[66] Jesse Farebrother, Jordi Orbay, Quan Vuong, Adrien Ali Taiga, Yevgen Chehotar, Ted Xiao, Alex Irpan, Sergey Levine, Pablo Samuel Castro, Aleksandra Faust, Aviral Kumar, and Rishabh Agarwal. Stop regressing: Training value functions via classification for scalable deep rl, 2024. WtRLhhtss://arxiiv
abs/2403.03930
[67] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers, 2021. URL https://arxxv.orgg 2104. 14294
[68] Alex Graves, Santiago Fernández, Faustino Gomez, and Jürgen Schmidhuber. Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks. In Proceedings of the 23rd international conference Machine learning, pages 369-376, 2006.
[69] Francis R. Willett, Donald T. Avansino, Leigh R. Hochberg, Jaimic M. Henderson, and Krishna V. Shenoy. Seeehperformacc brain-to-text communication via handwriting. Nature, 593(7858):249 254, May 2021. ISSN 1476-4687.
doi: (033//441860002-0353553 541586 021 021-03566-2
URL https::/www.ww... nature. com/articles/
[70] Chen Yang, Junzhuo Li, Xinyao Niu, Xinrun Du, Songyang Gao, Haoran Zhang, Zhaoliang Chen, Xingwei Qu, Ruibin Yuan, Yizhi Li, Jiaheng Liu, Stephen W. Huang, Shawn Yuc, and Ge Zhang. The fine line: Navigating large language model pretraining with down-streaming capability analysis, 2024. URL
https://arxiv. org/ wwt.:/r////b/a/e//e// abs/2404. 81204
[71] Sachin Goyal, Ananya Kumar, Sankalp Garg, Zico Kolter, and Aditi Raghunathan. Finetune like you pretrain: Improved finetuning zero-shot vision models, 2022. http:/tpss///attoorg/aaaaas/e wrtp/b//////b//////////// 22tps/2222.006888
[72] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models, 2020.
[73] Linxing Preston Jiang, Shirui Chen, Emmanuel Tanumihardja, Xiaochuang Han, Weijia Shi, Eric Shea- Brown, and Rajesh P. N. Rao. Data heterogeneity limits the scaling effect of pretraining neural data transformers. bioRxiv, 2025. doi: 10. 11110002225.0511265555555 terrly/225//0511/2225 85. 15.5515511
Utt/https:///wet htttt:::/www...oox.....o www.wbiioiii...ooooooooo.......o www..boooxiv..rr/cotteeee org/content//
[74] Robert Flint, Erie W Lindborg, Luke R Jordan, Lee E Miller, and Mare W Slutzky, Accurate decoding of reaching movements from field potentials in the absence of spikes. Journal of neural engineering, 9(4); 046006, 2012.
[75] Chethan Pandarinath, Paul Nuyujukian, Christine H Blabe, Brittany L Sorice. Jad Saab, Francis R Willett, Leigh R Hochberg, Krishna V Shenoy, and Jaimie M Henderson. High performance communication by people with paralysis using intracortical brain-computer interface. elife, 6:e1 1854, 2017.
[76] Jonathan R Wolpaw, Niels Birbaumer, Dennis J McFarland, Gert Pfurtscheller, and Theresa M Vaughan. Brain-computer interfaces for communication and control. Clinical neurophysiology. 1136)1767779,, 2002.
[77] Beata Jarosiewicz, Anish A Sarma, Daniel Bacher, Nicolas Y Masse, John D Simeral, Brittany Sorice, Erin M Oakley, Christine Blabe, Chethan Pandarinath, Vikash Gilja. et al. Virtual typing by people with tetraplegia using a self-calibrating intracortical brain-computer interface. Science translational medicine, 7 (313):313ra119-- 31-33111193311 17a, 2015.
[78] Mariana P. Branco, Lisanne M. De Boer, Nick F. Ramsey, and Mariska J. Vansteensel. Encoding of kinetic and kinematic movement parameters in the sensorimotor cortex: A brain-computt interface perspective. European Journal of Neuroscience, 5055) 2772, September 2019. ISSN 0953- -963,816X, 1460- 9568. doi: 10. N0.1111/ejj.1433 14342. URL BRLLlnttss///oniieelbbrrr....eee wwwp://wc.ccccccc///////eeeee/ wwwpw/w/wwwooooooocce 1111/ejn. 14342
22

---

[79] Kristin M Quick. Jessica L Mischel. Patrick J Loughlin, and Aaron P Batista. The critical stability task: quantitying sensory-motor control during ongoing movement in nonhuman primates. Journal of Neurophysiology. 120K5)2114-2181, 2018.
[80] Patrick Marino, Lindsay Bahureksa. Carmen Fernández Fisac, Emily R Oby, Adam L Smoulder, Asma Motiwala, Alan Degenhart, Erinn M Grigsby, Wilssan M Joiner, Steven M Chase, et posture subspace in primary motor cortex. bioRxiv, pages 2024-08, 2024.
[81] Matthew Mender, Samuel Nason-Tomaszees Tomaszewski, Hisham Temmar, Joseph Costello, Dylan M Wallace, Matthew S Willsey, Nishant Ganesh Kumar, Theodore A Kung. Parag Patil, and Cynthia A Chestek. The impact of task context on predicting finger movements in brain machine interface. eLife, 12:e82598, jun 2023, ISSN 2050-084X.
doi: 10.7554/cLife 82598. URL https: / /ooi org/10. 7554/eLife .22598
[82] Xuan Ma, Fabio Rizzoglio, Eric J. Perreault, Lee E. Miller, and Ann Kennedy. Using adversarial networks to extend brain computer interface decoding accuracy over time. Aug 2022. doi: 10. 10.1101/2022.0..66.00777
URL https://www. xior wwt.e/oww/oo/o//oo////./
1181/2022 88. 504777v1
[83] Kendra K. Noneman and J. Patrick Mayo. Decoding continuous tracking eye movements from cortical spiking activity. International Journal Neural Systemss, page 20165726744000000, October 2024. ISSN 0129-0657, 1793-6462. doi: 10.1 14944222999990777577770 URL https://www.w wor Idscientificccc.
doi/10
[84] Chaofei Fan, Nick Hahn, Foram Kamdar, Donald Avansino, Guy Wilson, Leigh Hochberg, Krishna Shenoy, Jaimie M. Henderson, and Francis R Willett. Plug-and-play stability for intracortical brain- computer interfaces: A one-year demonstration of scamless brain-to-text communication. In Thirty-seventh Conference Neural Information Processing Systems, 2023. URL http:////gggeeeeeeeeeeffrrrrrrre STqaMghtDi
[85] Tri Dao. Flashattention-2: Faster attention with better parallelism and work partitioning, 2023. URL
https://arxtv.org/bss//// .86991
[86]
Mostafa Dehghani, Josip Djolonga, Basil Mustafa, Piotr Padlewski, Jonathan Heek, Justin Gilmer, Andreas Steiner, Mathilde Caron, Robert Geirhos, Ibrahim Alabdulmohsin, Rodolphe Jenatton, Lucas Beyer, Michael Tschannen, Anurag Arnab, Xiao Wang, Carlos Riquelme, Matthias Minderer, Joan Puigcerver, Utku Evci, Manoj Kumar, Sjoerd van Stecnkiste, Gamaleldin F. Elsayed, Aravindh Mahendran, Fisher Yu, Avital Oliver, Fantine Huot, Jasmijn Bastings, Mark Patrick Collier, Alexcy Gritsenko, Vighnesh Birodkar, Tay, Thomas Mensink, Alexander Kolesnikov, Filip Pavetić, Dustin Tran, Thomas
Cristina Vasconcelos, Kipf. Mario Lučić, Xiaohua Zhai, Daniel Keysers, Jeremiah Harmsen, and Neil Houlshy. Scaling vision transformers to 22 billion parameters, 2023. BRRLhhtps://aaxivvv org. org/ alaaaaa/..aa.eeeeea 2382224422
[87] Mitchell Wortsman, Peter J Liu, Lechao Xiao, Katic E Everett, Alexander Alemi, Ben Adlam, John D Co- Reyes, Izzeddin Gur, Abhishek Kumar, Roman Novak, Jeffrey Pennington, Jascha Sohl-Dickster Kelvin Xu, Jaehoon Lee. Justin Gilmer, and Simon Korbblith. Small-scale proxies for large-scale transformer training instabilities. In The Twelfth International Conference Learning Representations, 2024. URL
[88] Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. Transformer feed-forward layers are key-value memories, 2021. URL https://arxxw..rr//bssss 14913
23

---

A
Appendix
Contents
A.1
Related works and proposed taxonomy
24
2
Supplementary Results
25
A.2.1
Further tests of held-out angle generalization.
25
A.2.2
Ablations
26
A.2.3
Isolated scaling in neural data
27
A.2.4
Pretraining does not benefit FALCON H2 (Handwriting)
28
A.2.5
Multiscale decoding individual motor tasks
28
A.2.6
Sequential tuning similar to joint tuning
29
A.2.7
Aggregate performance on all NDT3 models with significance tests
31
A.2.8
Neural vs Behavioral Objectives
32
A.2.9
Downstream probe through pretraining
32
A.2.10
Evaluation on the Neural Latents Benchmark
33
A.2.11
Neural objectives needed in fine-tuning
34
A.3
Methods
34
A.3.I
Metrics and Evaluation
A.3.2
Training
34
A.333
Baselines and Other Transtormers
35
A.3.4
Pretraining and Evaluation datasets
38
A.3.5
Generalization analyses and Further evaluations
38
A.3.6
Architectural Details
38
A.1
Related works and a proposed taxonomy
Neural data is sufficiently diverse as support many distinct efforts train large neural data models. The scale of pretraining is somewhat larger in the non-implanted modalities, where data is more abundant. The largest EEG models have reached scale of 2.5K
to 40K hours of data 43) Current fMRI models operate in the IK [46] 7K [47 hour range. The largest models these studies are the O.IB-IB parameter range. Intracranial modalities, including sEEG 48 49, ECoG [50] 5T] and spiking activity 66, have thus far been studied at an order of magnitude smaller scales of data and model size (20-1000 hours, <0. parameters).
42
or higher volumes if also considering non-brain biosignals (EKG) [4] 45
Direct scaling neural data modeling should be distinguished from NeuroAI efforts 52 to measure how models of the human sensorimotor experience (e.g. language, vision, audio models) predict neural data ( However, multimodal efforts begin blur this distinction (54 (5), care will be
required to distinguish advances in modeling neural data, embodied data, or their interaction.
Comparing neural data models Current efforts to understand scaling in neural data Simeon et al. (56), Sato et al. (57) will have their reach limited by the specificity of every neural dataset. A meta-challenge for the field is understanding how different parameters (species, brain area, modality, task) impact scaling properties. This would be greatly aided by development of reporting practices for different neural data models. To facilitate comparison, codebase [58]. addition to the standard model card, we propose reporting an additional taxonomy aid comparisons across neural data models, using concepts.
create model card in the INT3
First: neural data models can be conceptualized as modeling slices of the plenneural function, inspired by the plenoptic function in vision [59]. The plenoptic function is model of an idcalized eye which

---

A.
Neural Data
Behavior (Held-In)
Wiener Filtor
NDT3
LEGEND
X Wiener Filter x Scratch
200hr 45M
2khrr 350M
45M
B. R²
200 hr
Monkey
0.5
0
1900 10
Held--n
00.
110
Monkey
Monkey
Monkey
-180
-90
90
180
-180 -90
90
180
Test Angle Offset
Figure 6. A. Two monkey 2D center-out datasets with 8 angular conditions. LDA projections show monkey J's data is distinetly more separable than monkey V's. We then test generalization of behavioral decoding to held-out angles after training of 8 angles (boxed in red). All NDT3s and the WF produce predictions constrained between the held-out angles. The Wiener Filter here, unlike the main text, is a direct fit on neural data without dimensionality reduction. B. We quantify performance for decoding on each angle with respect to distance from the central angle between the held-in in angles. We average performance over all 8 central angles.
parameterizes all possible images with 7 dimensions: 4D to describe the global spacetime of the view, 2D to deseribe viewing angle (spherical) or coordinate (Cartesian) of the image, and ID for wavelength. Since neural data models are primarily interested in circumscribed systems rather than the physical world, a similar global coordinate system (e.g. 4D for all possible electric potentials) would be uninformative. We thus propose reporting more qualitative coordinates:
1. Identity: The network individual being recorded.
2. Task: The behavior, stimuli, or other activity the network is reflecting.
3. Spacetime:
Coordinates specified in a network-local coordinate frame (e.g. brain area).
Second: The modeled extent of this plenneural function is conveniently discretized in three resolutions in Transformer-like sequence modeling framework: the token, the sequence, and the full training data. The token is the most granular unit of data being modeled; NDT3 models neural populations 32 neurons at a time, in 20ms bins. At the sequence input level, NDT3 models inputs from single humans monkeys, across 128-256 neurons in 2 second snippets, while performing effectively one "movement." " Finally, NDT3's pretraining spans dozens of individuals, records motor and premotor areas over 2.5K hours, over variety of arm and hand movements.
A.2
Supplementary Results
A.2.1 Further tests of held-out angle generalization.
To further support our single-dataset illustration of attractor structure in pretrained NDT3, we evaluate reach angle generalization across sessions in the isometric, force-based dataset (Monkey J) setting and a second, manipulandum-basee (Monkey J) setting with monkey movement. To begin, we visualize the separability of these neural data with LDA (Fig. 6A Neural Data). We next train decoders on every pair of angles separated by 90 degrees (one shown) and plot predictions on held-out trials from all angles. NDT3 and WFs, here directly fit to high-D data instead of after PCA-LDA, both fail to extrapolate to held-out angles, consistent with Rizzoglio et al. [33, illustrating the necessity of the dimensionality reduction for generalization in the main text. We quantify prediction performance in Fig. D. These plots again show that held-out generalization is subtrivial, while being entirely consistent with pretraining's overall narrative of improved performance in all conditions.
Methods for PCA-LDA. Our PCA-LDA preparation used in Fig. 4C-E adheres closely our standard data preparation and is directly comparable to the inputs received by the deep networks. To begin, we smooth all spike counts with an exponential kernel, as in our Wiener Filter baseline (Section A.3.3) We then z-score these smoothed spikes, and fit PCA to the high-dimensional (96D) z-scored neural activity at each timestep to extract the top 10 PCs. Finally, we fit linear discriminant analysis (LDA) to reduce this 10D PC space to neural activity to 2D. Example neural data projections are shown in Fig. 6A, showing clear radial structure that can be exploited for generalization. The training data for PCA and LDA are both restricted to only the held-in angles. Note that LDA uses categorical labels to separate the held-in reach directions but is applied without class labels after fitting. While this train-time labeling is technically not given to NDT3, this does not influence our
25

---

R2
0
-1
45M 200h
Ablate mask
Session: 60 S
25%
50%
10 min
Figure 7. Ablation of covariate masking on an open 2D Cursor + Click dataset. Covariate inputs are completely masked in inference for the default NDT3, and autoregressively generated in the ablation.
argument that NDT3 pretraining should be capable of angular generalization, as NDT3 does not confuse the reach angle of held-in predictions. The final Wiener Filter is fit to the 2D data and uses 1 second of history.
A.2.2 Ablations
We ablate the major design decisions made to enable NDT3's large scale pretraining. These ablations give us confidence that NDT3 overcomes the basic challenges encountered in development, but compute restrictions prevent more exhaustive comparisons or exploration of model design space. We encourage further work exploring the influence of different hyperparameters. In these plots, distinguish validation split performance and evaluation split performance, which is computed by batch-mode prediction (not the costly streaming evaluation used throughout main experiments).
We find the default next-step prediction objective fails for learning decoding of highly autocorrelated covariate timeseries, perhaps because simply relying teacher- forced behavioral inputs provides a severe shortcut that prevents learning of a proper neural to behavior decoding map [60]. Different time-series models have addressed this by adopting convolutional
Covariate dropout
input-output layers 61.. tokenizing along temporal dimensions (62). or learning with contrastive
We avoid introducing architectural modifications and instead adopt a simple dropout procedure that masks a portion of covariate inputs some fraction of the time. Specifically, on every training batch, two random numbers are drawn. The first, M U(0, 1), determines what fraction of covariate inputs should be masked. On 90% of batches, we also sample T ~ U'O 2] seconds, such that the mask is only applied after timestep T. That is, on 90% of batches, the model is provided prefix-prompp metrics for validation and evaluation are always computed with a prefix and full masking of non- non-prefix timesteps. In Fig. 7 we ablate covariate masking (which also removes the prefix logic), and tune on 2D Cursor + Click task. The ablated model performs subtrivially with student-forced predictions provided as input at test time. Note that the ablated model performs trivially with masked inputs (not shown).
objectives 49 63
We do not block losses on this prefix as in prefix-LMs (64]. Pretraining
BCI-phase and return conditioning NDT3's pretraining includes several hundred hours of BCT control data, where the covariates were by another decoder. We introduced phase and return conditioning tokens to differentiate the several types of BCI control data from recorded behavior. Specifically, in BCI data, NDT3 receives input tokens specifying what fraction of the behavior reflects neural input (BCI control is on) vs programmatic input (BCI control is off, as in open loop BCI calibration). Further, we provide inputs encoding reward (trial success) when trials change, and return (future reward over a 10 second horizon, which crosses data boundaries). This design is intended to evaluate the potential for a Decision-Transfore control, but we do not discuss this in this work. In Fig. 8 B, we focus on whether these inputs improves pretraining loss and R² in validation splits, which has contains BCI data, and the held-out evaluation split containing only monkey behavior. The figures show that the ablation significantly
like offline learning strategy for improved online
26

---

A.
split Validation
B.
split Validation
C.
split Validation
1.0
1.0
Covariate MSE
4x
Covariate R
45V Len numon
1kh Human mo CanstrainRosumm
Covariate R
450 200h
22th HSE1
225h Neural)
22th (hacch 1s1
3x100-
0.4
0.4
02
02
2x
split Evaluution
split Evaluation
spiit Fvaluation
4x.10
1.0
1.0
Covarlate MSE
R
Covariate R
0.6
0.6
2x
0.4
0.
4SN Litt Human
0.0100
34h Harran Ino Constra rofeeum
100
0.0
10
100
100
Epochs
Epochs
Epochs
Figure& 8. Ablations evaluated based on upstream evaluation split. A., B. Ablation of BCI control tokens C. Ablation of neural objective and covariate MSE objective in favor of classification over quantized covariates.
decreases validation split performance, and causes a slightly earlier stopping point leading worse evaluation performance. Note both models early before the full training budget of 400 epochs.
Neural reconstruction objective All main NDT3 models used a neural reconstruction objective inherited from the self-supervised learning pretraining from NDT2. We ablate this choice post-hoc and see it may actually minorly harm pretraining (validation split), though the neural objective doesn't 't harm evaluation split decoding (Fig. (C). Note the scalar weighting of neural vs covariate objectives were set to be roughly balanced in pretraining Section 2 downstream analysis on the standalone value of the neural reconstruction objective.
MSE over classification In robotics and certain generalist models 14, continuous action spaces are sometimes better decoded and controlled when quantized [65] This is because MSE is an insufficient objective when the output distribution is multimodal (e.g. one of two possible paths in robotics). While it seems unlikely that the close relationship between movement behavior and motor cortex is multimodal, multimodal behavior may be appropriate when pretrained on heterogeneous data, i.e. when similar neural activity corresponds to different behavior in two datasets. We attempted such a quantization, including HI--Gauss smoothing (66) which found to help; but this does not recover the performance of the default MSE objective (Fig. (CC) on the evaluation split. We found this performance gap persisted under fine-tuning (not shown). This suggests that NDT3 differentiates neural data inputs from different datasets.
Patch size NDT2 and NDT3 both tokenize neural data by patching them into fixed size clusters. It is unclear whether transfer learning might occur for sub-token features, which motivates the use of smaller tokens in larger datasets that might afford it [67] We change patch size to 16 and show this performs slightly worse in the 45M 200h model (Fig. C. Smaller patches (and subsequent increased neural tokens) may be more beneficial in the larger scale models, but their benefit must be weighed against their increased compute burden.
A.2.3 Isolated scaling in neural data
Due to NDT3's joint modeling of behavior and neural data, it is difficult to dissociate whether scaling gains in behavior come from improved behavioral or neural priors. To assess whether NDT3 can scale solely from neural data modeling, we pretrain a new set of 45M parameter models up to 200 hours with only causal neural data modeling objective. As before, we then tune to a downstream decoding task, this case, a Critical Stability Task dataset (CST). From a representation probing perspective, improved downstream performance implies higher quality neural representations. We use the standard single-stream autoregressive modeling objective as the rest of this work in the downstream setting, we find direct linear probing of neural representations perform worse. Fig- P compares the scaling on downstream neural and behavioral metrics after the standard fine-tuning procedure.
27

---

A.
Pretrain: Neural only
Pretrain: Neural + Bahavior
B.
Pretrain: Noural only
Pratrain: Neural Behaviaral
Neural
Neural
R²
R²
NLL
Session: 60 Task: 42 min
NLL
Session: 60
Task: 42 min
200
Pretraining Volume (hrs)
0.695
0.695
0.4
0.4
0.690
0.690
0.3
0,3
70
0.685
0.685
0.2
0.2
0.680
0.680
0.1
0.1
Session: 60 s Task: 100%
Session: 60
Task: 42 min
0.675
0.675
0.0
0.0
25
100%
10
50 100%
30
100%
Figure 9. Scaling of unsupervised pretraining After pretraining models up to 200 hours with only the neural reconstruction objective, fine-tune models in similar multiscale evaluation as in Scction with newly initialized behavioral readout. We use the CST task here. A. left shows neural Joss scales with neural-only pretraining. The right pancl plots the neural loss, also present in the standard evaluation, also scales with joint pretraining. B. left shows that with neural pretraining. downstream decoding performance saturates at a flat performance after 25 hours. This is compared against the nonsaturated scaling from joint pretraining. Colorbar common for all plots, and Xs from-scratch NDT3 models.
Fig. AA shows that downstream neural reconstruction improves with increased pretraining data either when using only the neural objective or both neural and behavioral objectives (as in the standard setting). The joint pretraining achieves advances neural metrics in all settings, illustrating that decoding behavior is a complementary objective to neural data reconstruction even for representation learning.
Fig. 9 contrasts decoding curves the two pretraining settings, in that neural pretraining has saturated decoding after just 25 hours of pretraining. This is consistent with the interpretation that the behavioral readout reflects only one aspect of the neural data. Together with the neural metric plots, this analysis shows scaling over solely neural data is possible, but also that decoding behavior is complementary pretraining objective for improving neural representation learning and decoding.
A.2.4 Pretraining does not benefit FALCON H2 (Handwriting)
We also evaluated NDT3 for decoding of letters in human-open loop handwriting task (FALCON H2). Although this is also a motor cortical decoding task, we excluded H2 from NDT3's aggregate evaluation since it is sequence-to-sequees task, we pool neural tokens at each timestep and add a linear projection and optimize with CTC loss [68] We maintain the default neural reconstruction loss and causal attention mask, and do apply data augmentation.
opposed to continuous task. To apply NDT3 to this
Note that RNNs are the current standard architecture for communication tasks like H2 23 69
Training and tuning was less stable than for our continuous decoding tasks and required more extensive hyperparameter tuning, perhaps because the overall dataset size remains small (<Ik samples), specific parameters are listed in the codebase. We observe three regimes both training and fine-tuning. First, the model can fail to achieve initial learning period. Second, the model achieve reasonable nontrivial solutions, comparable to expected performance for unaugmented RNNs (though we do quantify this). Third, some models will exhibit learning instabilities that resolve in significantly improved performance. We illustrate these regimes in example validation curves below. Overall, the third regime is rarely achieved. More relevant to the main narrative of this work, fine-tuning appears to degrade both final solution quality and reduces the range of nontrivial hyperparameters (not shown). Investigating sequence sequence objective over CTC loss would be valuable future work.
A.2.5 Multiscale decoding individual motor tasks
Fig. A plots model performance for each of the 31 evaluation settings we study in the eight primary evaluation datasets we use. Studying any individual dataset will yield variable conclusions on whether pretraining structure is helpful, underscoring the need for proposed foundation models to be evaluated across many different datasets. We see the most clear scaling with pretraining data (color gradient
28

---

100
CTC Loss
10
40
30
CER
45M 200h
10
Scratch
10
100
1000
Epochs
Figure 10. Three regimes of NDT3 training for handwriting decoding. We show validation loss and character error rates example runs from-scrated and fine-tuned NDT3s.
with red on top) the Critical Stability Task and Bimanual Task. FALCON tasks and Self-paced Reach appear minimally affected by scaling pretraining data, in that either pretrained models generally slightly above a from scratch model at all data scales with no particular best pretrained model. The 2D Click and Grasp datasets uniquely show high variability model performance and strong degradation of the 350M khr model at low data scales. Grasp instability was so high that we trained 9 seeds instead of the standard this degradation is due to the instability of full fine-tuning of large models at the extremely low data scales these datasets present (e.g. 2.5 minutes the 25% scaling). Finally, we remind that the 2D + Click, FALCON HI, and ID Grasp Force tasks are datasets from human participants that are included in the 2 khr pretraining. Surprisingly, this leakage did not provide clear benefit the 2 khr model.
better estimate model performance. We propose
These scaling plots also provide more precise context for baseline performance. NDT2 performs particularly poorly in the low data regime, while Wiener Filters perform poorly in the high data regimes.
In Fig. 11B, we illustrate qualitative predictions private datasets. These visualizations show diversity in covariate timescales and structure. They also illustrate that the summary R obscure several features of model predictions. For example, pretrained models in Cursor Y tend have false positive deflections movement. It2 also is not easily comparable tasks with continuous dynamics (CST) vs. transient dynamics (Cursor GI).
A.2.6 Sequential tuning is similar to joint tuning
In Section we showed that channel shuffling and half-token shifts were sufficient to reduce cross-session transfer to the extent of cross-subject transfer, in the context of a pretrained 350M 2khr NDT3 model and from-scratch NDT3 model. Here add 1) joint tuning results in addition the sequential tuning in the main text, 2) extensions of this analysis to multiple intermediate tuning data scales, 3) additional replication of this analysis for the 45M 200hr NDT3 model.
The comparison of sequential fine-tuning against with joint fine-tuning done for completeness. Given cross-context data and test session, either jointly tune all data (as we do in Section
29

---

Human
Human held-out of 2kh
2D+Click Cursor
7D Reach Grasp Kinematics (FALCON H1)
1D
Grasp Force
4D Bimanual
R²
Ra
R²
R²
0.4
0.6
0.4
0.3
0.3
0.4
0.3
0.2
0.2
0.2
0.2
0.1
0.1
0.1
Session: 60
0.0
Session: 80
Session: 8 min
Sessjon: min
Session: min
lask: 100%
Task:
Task: 12 min
Task: 25
0.0
0.0
0.0
25
50
100%
100%
100%
50
100%
Monkey 2D Finger Pose (FALCON M2)
16D Reach/Grash EMG (FALCON M1)
R²
R²
LEGEND
2000
Pretraining Volume (hrs)
Pretroined (NDT3]
0.4
0.6
45M
0.2
0.5
350M
0.4
NDT3 Scrotch
0.0
.... *
0.3
NDT2 Scrotch
.....6 X
-0.2
0.2
NDT2 100
Session: 64
Session: 10 min
Session: 2 min
Session: 1 hr
X Wiener
-0.4
Task:
0.1
Task:
25 30100%
50100%
100%
...
1D Critical Stability Task
2D Self self-paced reach
(au) B
Cursor
R2
R²
True
(0.23)
Scratch [o.29)
0.4
0.6
g1 (au) 2 (au)
Pretrained (0.69)
0.5
Scratch [0.
True
0.4
Pretrained (0.61)
0.1
Scretch (0.07)
Session: 60
Session: 60 s
Task:
42 min
Task: 12
0.0
4.0
8.0
12.0
16.0
0.0
10
25
50 100%
0.3
50100%
Time (s)
Figure 11. A. Fine-tuning evaluations for individual datasets. Performance on both held-out (left) and held-in (right) splits are shown side by side by FALCON datasets. We shade the standard deviation of model seeds in tuning. Different tasks show substantial variability in benefit from pretraining. We show example predictions of pretrained (45M 200h) and from-scratch NDT3 for the 2D Click Cursor task to give sense of what different prediction performances mean terms of open loop data prediction. Numbers legend are R² for that model's predictions in the shown snippet.
30

---

A.
Cross-Subjess
B. Channel Shuffle
Half-token Shift
Token Shuffle
R²
0.7
0.5
In-session baseline
0.3
Test
0.1
0.5
1
2hr 1
2 4 hrs
0.5 1 2hr
0.5 1
2 hr 0.5 1 2hr
LEGEND
Sequential
Joint
x Scratch 200hr 45M
2khrr 350M
Figure 12. A replication of Fig. 4JB but showing results for both sequential and joint fine-tuning in each setting, and on multiple fine-tuning data scales. As before, each model here tunes with some data from other settings (e.g. cross-subject data for the cross-subject panel), and a fixed amount of data on the test session. Sequential tuning first tunes on other-setting data, and then tunes on test session data. Joint tuning uses all data at once. In Fig. 4A/B, we only showed the better choice for each panel. i.e. we showed joint tuning for cross-scssion data, and sequential tuning for the other settings. In addition, we overlay how the cross-subject tuned models perform when applied half-token shifted test data, confirming that subject and session data transfer similarly shifted test data. Shading shows the standard deviation on 3 model seeds. In-session baselinc refers to the no-intermediate tuning setting in Fig. 43.
3.1 or sequentially tune first on the cross cross-context data and then on the test session. The latter is the protocol used in most other works and what we adopt in the main text for this analysis. We find for pretrained models that the results are similar, but that for from-scratch models, sequential tuning is essential for cross-subject transfer and slightly harmful for cross-session models (Fig. T2A). Sceing that sequential tuning is mainly advantageous for from-scratch models, we speculate that sequential tuning is particularly helpful for filtering learning signals in highly heterogeneous data, but pretrained models develop a capacity for doing such filtering during fine-tuning.
For the multiple intermediate data scale, only the largest scale was reported in the main text. Perfor- mance for most models decay towards the no intermediate tuning baseline. The exception here cross-session tuning, where even minor amounts of data provide substantial gain, and cross-subject transfer in the joint tuning setting for from-scratch models, where, as mentioned, the model performs very poorly.
The 45M 200hr model performs differently than the 350M 2khr at individual settings but without any changes in overall trends.
A.2.7 Aggregate performance on all NDT3 models with significance tests
We additionally report the average performance of an NDT2 model pretrained with 100 hours of human data and two NDT3 models pretrained with 25 hours and 70 hours. These models are placed in context with the models from Fig. BP in Fig. A3 Note that for NDT3 models each successively larger data scale uses strict superset of data from smaller scales. We also provide the p-values computed for the significance of the difference between each pair of models in Fig. B3 P-values are computed as FDR-corrected pairwise t-tests. The 350M khr model has p < 0.06 improvements over all but the 350M 200 hr model. Interestingly, all other pretrained models, except the 25 hr model, appear equivalent, at least statistically. We presume this is due to the fact that our evaluation of 31 task settings may be insufficiently large.
NDT2 performs relatively poorly in our evaluation. This is true even when tuning from the public checkpoint trained on 100 hours of neural data from humans, though tuning does in general improve over the from-scratch NDT2 training. We believe pretrained NDT2's performance gap with NDT3 partially due to NDT2's need to newly initialize decoding layers in each downstream task, which increases NDT2's dependence on thorough hyperparameter tuning. This makes NDT2 a poor candi- date for a foundation model. Section A.3.3 and Section AA336] describe a number of methodological innovations that likely each contribute to the remaining performance differences between NDT2 and NDT3.
31

---

NOT2 011
Sgeiicacce Lewds
R²
8 task x 3-5 scales Avg.
350M param
NDT3 Scratch 04
NOT2 100 0.06 0.04
x
Bits
9.10
RoW Column RV Difference
0.03
0.3
45M 25 hr-0.00
0.2
43M 70 hr 0.00
0.16
0.14
0.05
0.01 0.02
4SM 200 hr 0.00
6.1
9.44
0.09
0.01 0.01
-0.00
0.05
0.1
45M Khr-a 0.00
0.01
-0.10
0.0
WF
NOT2
NDT2 100 br
NDT3
1.5 hr
25
70 tv
200
2 kht
200 nr
ktr
350M khr
10
0.20
8:15
am
0.01
0.01 not om
-0.15
WF
HOT2 100 hr
HOT2
Scraten
4tM 2.5
49M 25
tr
200 fr
43M Inr
3508 200
HOTS
45M
4518
Figure 13. A. A replication of Fig. B including additional 25 hr and 70 hr model. The two additional models show the precise performance we measure may noisily related to to pretraining data scale. B. Heatmap of differences between performances of pairs of models with significance tested with CDR-rceeetted pairwise t-tests. Note that coloring is used here to indicate differences, not significance. Positive numbers with significance indicate the row model outperforms the column model.
Neural Objective (Spike count flssification)
Behavior Objective (Mean squared error)
Loss
Loss
70 hr
125 hr 1.5 hr
002389
1.5 hr
25 hr
1.71
70 hr
002299
1.69
0.0100
1.67
khr (350M)
2 khr (350M)
50
100
200
Epochs
50
100
200 Epochs
Figure 14. Pretraining curves shown for 4SM parameter NDT3s at 1.5 hour, 25 hour, and 70 hours, in addition
the curves for khr 350M parameter NDT3. We separately show metrics on neural and behavioral objectives through training. Since early stopping is used in model selection, we verify here that neither objective is overfits significantly, except for the 1.5 hour model's neural objective.
A.2.8 Neural Behavioral Objectives
In this work, the neural objective present mainly an auxiliary objective improve downstream decoding. We see that neural and behavioral objectives are complementary in Section A.2.3
In Fig. [4] provide some additional context, showing that neural objectives also improve through pretraining, i.c. that we are not overfitting the neural objective and thus degrading decoding.
A.2.9 Downstream probe through pretraining
Upstream and downstream performance are generally assumed to be correlated in pretraining studies. This assumption supports the use of a single evaluation at the end of pretraining, rather than throughout pretraining, particularly if pretraining is not overfit. It is possible, however, that the narrower tasks used in evaluation may be learned or even overfit earlier than the general pretraining task. To assess this, we conduct a small downstream probe of checkpoints every 20 epochs of pretraining for a 45M 25 hour NDT3. This probe measures the performance specifically at the 25% scaling of the bimanual task. In Fig. 15 we show this progression and compare the variability of checkpoints in pretraining with the differences across different task scales and pretraining models. The downstream probe for this task shows quick benefit from pretraining, providing the full gain over from-scratch models by the first checkpoint we evaluate epoch 20, and then plateaus. This contrasts with the
32

---

45M 25 Eval pretraining
0.26
50% scale
0.24
350m 2kh, 25% scale
Bimanual R2
0.22
25% scale
0.20
0.18
0.16
NDT3 mse, 25% scale
10% scale
0.14
300
Pretraining epochs
Figure 15. Downstream evaluation conducted through pretraining of 45M 25 hour NDT3, for the 25% scaling himanual task, provided in context final evaluation achieved by scale settings for the 45M 25 hour models, and the 25% scalings for the from-scratch NDT3 and best NDT3 (350M 2 khr). The benefits from pretraining are attained by epoch 20, and plateaus without overfitting for the rest of pretraining.
and 50% himanual task
Dataset
RTT
Maze
Maze Large
Maze Medium
Maze Small
NDT1
0.1643
0.3597
0.3739
0.3081
0.2788
SoTA
0,2010
0.3650
0.3831
0.3329
0.3458
NDT3 Scratch
0.1533
0.3093
0.2892
0.1859
0.1853
NDT3 350M 2kh
0.1695
0.2775
0.2781
0.1937
162116
Table 1. NDTI vs NDT3 co-bps (higher is better) on NLB's MC Maze and MC RTT datasets in the 20ms split. The NDT3 results were produced were the standard sweeps used in this work, e.g. three random learning rates and three random seeds. SoTA results come from LangevinFlow for RTT and Maze and MINT for the Maze scaling datasets.
pretraining plots in Fig. 14 The mismatch between upstream and downstream progress here differs from correlated ppstream-downstredd progress in single-epoch language model studies [70], which may be due to differences in domain pretraining scale.
A.2.10 Evaluation on the Neural Latents Benchmark
The Neural Latents Benchmark (5] is benchmark for evaluating latent variable modeling of neural activity. This evaluation differs from NDT3's direct decoding evaluations in that all of NLB's s metrics are derived from acausally inferred neural firing rates, akin pixel-level objective in computer vision. Note specifically that the decoding metric officially reported in the NLB is derived from submitted firing rates on the evaluation server; report decoding performance on NLB datasets without modeling neural activity is not an expected use of the benchmark.
Despite its distinct setting, the NLB provides models of motor cortical activity, providing context for NDT3's performance outside of decoding, To apply NDT3 the NLB, tuned NDT3 in a new downstream task where insert new token at each timestep, from which we linearly decode firing rates of held out neurons. We performed this tuning separately for each of the maze and random target tasks (RTT), reporting the resulting co-bps scores in Table [ NDT3 performs poorly. From-scrated training on the dingle-session benchmark datasets underperform NDT1 in all tasks. Tuning from pretrained models improved performance, sample efficiency, and robustness to hyperparameters (only performance is reported here), but did not dramatically change model competitiveness. Zhang et al. (20) reported the value of more diverse neuron-level objectives for neural activity prediction, though their submission also dramatically underperforms NLB SoTA, warranting critical examination of whether pretraining benefits low-level modeling of neural activity.
well-established datasets which evaluate
33

---

Bimanual (scale 0.25)
Self- self-paced Reaching (scale 0.1)
0.3
0.6
Eval R²
0.2
Eval R2
0.4
0.1
0.2
0.0
0.0
10
102
10
102
Epoch
Epoch
350M 2khr
45M 200hr
Scratch
Behavior + Neural
Behavior
Figure 16. We compare fine-tuning on bimanual and self- reach tasks (RTT), for variants with the neural objective and behavior objective to those with only the behavior objective. Curves from with the best validation R² a standard seed LR fine-tuning sweep are shown.
A.2.11 Neural objectives are needed in fine-tuning
Our recommended fine-tuning for NDT3 is to match pretraining in most hyperparameters (except learning rate). This includes the use of the neural reconstruction objective, even though the down- stream settings we used only evaluate behavior reconstruction. We show that including this neural objective is important in fine-tur tin-tuning in Fig. 16 Specifically, for pretrained models, ablating the neural objective cause substantial drop in performance. Interestingly, the drop is significant for the larger scale 350M 2khr NDT3. This dependence of NDT3 on the neural objective is not desirable, but consistent with findings in other pretrained models that fine-tuning performs best when consistent with pretraining settings 711] For from-scratch models, the neural objective does not cause significant impact.
A.3 Methods
A.3.1 Metrics and Evaluation
Throughout this work evaluate offline decoding of continuous covariates timeseries. The metric we specifically use is the coefficient of determination, R², as computed by scikit-leam'' 's r2 score function. R²2 is useful metric over MSE 1 represents perfect prediction and O is the by best-guess baseline, the mean of the data. In pretraining, R² is computed over the flat average of all covariate dimensions, since each datapoint has differing covariate dimensionalities. In evaluation, R² is computed as a variance- average of R2's in each covariate dimension. Another difference between training and evaluation metries that training predictions are made over batched data, while evaluation predictions are mostly computed in a streaming fashion. Streaming requires continuous neural data across different behavioral epochs, and so cannot be performed for the Oculomotor and CST datasets. We also omit it for the motor cortex self- celf-paced reach dataset, which has a very large evaluation split. Streaming allows timesteps the beginning of each sequence to leverage neural context from the preceding sequence, which raises performance slightly, as shown in the continuous vs trialized analysis (Section 3.3). We limit history in streaming evaluations to the max history seen in tuning (1 second).
achieved
A.3.2 Training
Pretraining hyperparameters were manually tuned in preliminary experiments at the 45M parameter models on small datasets. 350M models diverged at the chosen 4e peak LR, so we lowered peak
LR to 1c 4. For tuning, the explored LRs are le 4, 3e 4, 5e 4 for training from scratch and 3e 5, le - 4, 4e for fine-tuning. While this is far from an exhaustive search, we show in Fig. 17 that other regularization hyperparameters are set to reasonable defaults such that this sweep finds near optimal results for both a from scratch model and fine-tuning the 45M model. Fine-tuning, like pretraining, is early stopped with a patience of 100 epochs. Batch size is uniformly
34

---

eval set cst
eval set bimanual
evail set rtt
0.6
0.4
Eval
0.2
Exhoustive
0.0
-0.2
NDT3 Scratch
45M 200
NDT3 Scratch
45M 200
NDT3 Scratch
45M 200
Variant
Variant
Variant
Figure 17. For 3 monkeys datasets at 10% scale, we extend HP sweep to 5 LRs and dropout in [0.0, 0.3] (vs default 0.1). For fine-tuni fine-tuning. from-scratch models we also sweep Transformer width ((266, 512, 1024] vs default 512. This yields 45-model sweep on seed. We compare the range of scores achieved by this larger sweep against the standard L.R seed sweep.
also sweep weight decay in [0.0001, 0.01, 0.1] (vs default 0.1), while for
set to 16K in pretraining, and scaled to be roughly 10-20% of dataset size in fine-tuning. NDT3 from-scratch models were trained at the 1 11M parameter range. Exact model configurations for different experiments are documented in the codebase.
NDT3's simple architectural design allows us to train on batches from different tasks and dimen- sionalities. To avoid excess padding in training, we concatenate pretraining data that is otherwise discontinuous (trialized) into 2 second data. We do not add any separator tokens, this does not appear to have a performance impact for language models [2]. With mixed-precision training, the 350M parameter NDT3 can fit the 4-8K tokens in each input context in the memory of 40G NVIDIA A100 GPUs Thus we can restrict NDT3's pretraining parallelism to data- data-parallelimm.
Using Kaplan et al. 72 's equation for FLOP computation, Crowward = 2N + MMaayerTeasxaaa we compute the footprint of the 350M 2kh model. We use about 0.9B FLOPPs per token in the forward pass, and about 0.9T neural tokens processed over training, which yields pretraining footprint of about 2.4c21 FLOPs.
A.3.3 Baselines and Other Transformers
Many models have been proposed for neural data decoding. While we believe the selected Wiener Filter and NDT2 baselines are representative of standard approaches, it may seem insufficient relative to the number of available approaches. To this end, want emphasize that this work's primary aim is to evaluate the scaling of a base Transformer on neural data, rather than establish uniform superiority of model over all others. Such claims can only be established in field with many benchmarks and external evaluations, which BCI decoding currently lacks.
Wiener Filter The Wiener Filter baseline swept history of neural input up to the max length provided to NDT, and reported the H² of the best WF according to test data in primary evaluation (slightly advantaging the WF). Generalization plots in Section 3.3 report the performance of WF models at these different histories. For evaluating angular generalization, WFs were only swept up to Is history due to memory limits; performance was not varying substantially with history so we do not expect this to have impacted conclusions. The WF was for simplicity directly fit the concatenated trial data, which may have slightly negatively impacted its performance in trialized datasets (Oculomotor, CST, Generalization analyses).
cross-validated over regularization strength. We also
In the primary evaluations in Section 3.1 we considered WFs fit either independently per session in dataset or jointly on all sessions, which is helpful for sessions in very low data regimes. We report the better of the 2. In generalization analyses, for simplicity, we only report joint fits, which may cause a slight downward bias in performance.
NDT2. NDT2 is Transformer that has shown effective transfer learning across BCI datasets. It currently leads the FALCON BCI decoding benchmark and outperforms RNN decoders. NDT2 baselines were prepared with its public codebase. We trained NDT2 models both from-scratch and from the public checkpoint pretrained on 100 hours of human data. Max context length and patience were held constant across the models. This restriction to patience of 100 epochs accounts for some difference with the reported FALCON benchmark results in Karpowicz et al. 23), as we note in Table 2 The only departure we take from NDT2's default design is that we train NDT2 with both
35

---

Dataset
Patience
Held-In R²
Held-Out R
H1
100
0.567 10.034
0.453
H1 (reproduction)
250
0.62840.011
0.517=0.010
H1 (23
250
0.62
0.52
M2
100
0.563
0.352
M2 (reproduction)
250
0.582
0.391 0.391
M2 23
250
0.63
0.43
Table 2. NDT2 HI and M2 results when trained with 100 epuchs of patience (this work) in fine-tuning 250
Karpowicz et al. (23] We report mean and standard deviation of 3 model seeds on the FALCON evaluation (which is in turn a cross-scssion mean).
NDT3
NDT2
Behavior
Behavior
1 token per dimension
Fixed readout dimensionality
N3
B1
N2
B1/2
Linear readouts
Cross-attention readouts
Causa Backbone
Some neural masked Behavior omitted
N1
N2
N3
B1
Behavior masked
N1
N2
N3
token per
token per
32 channels
32 channels
1s
1s
20ms binned spikes
20ms binned spikes
Figure 18. NDT2 and NDT3 architectures differ mainly in the conversion of NDT2's representation learning backbone to NDT3's single multimodal stream. NDT3 directly intakes neural and behavioral tokens and predicts with next-step objective. NDT2 employs explicit masking of input neural tokens and extracts neural and behavioral predictions at each timestep with cross-attention layers.
neural reconstruction loss (25% masking) and supervised decoding loss. This joint objective was introduced late the NDT2 paper (8) Fig 5) and performs comparably to multi-stage training while simplifying large scale evaluation. This is true for all eight evaluation tasks except CST. In the CST task, we used only the supervised decoding loss, as the token dropout used in reconstruction can dropout all neural input.
For hyperparameter tuning, we matched NDT3's tuning budget for the pretrained NDT2 checkpoint by only exploring 3 learning rates, Given mediocre NDT2 from-scratch performance, we swept NDT2 over model sizes in addition the standard learning rates. We set the NDT2 from-scratch model sizes to 20M and 72M to be comparable with NDT3 45M, but note that the NDT2 pretrained checkpoint only 6M parameters.
NDT2 vs NDT3. NDT3 builds off of NDT2 but departs in several manners to enable more streamlined scaling and analysis over heterogeneous decoding tasks, which we overview in Fig. T8 Lower level technical changes are described in Section A.3.6] Both models relate neural data to behavior and train with both neural data and behavior prediction objectives. Both models use both these objectives in fine-tuning, but NDT2 only uses the neural objective in pretraining. NDT2's neural data objective is based on MAE 2 such that some fraction of input neural data tokens are masked and reconstructed in a decoder separate from the main backbone. However, this explicit masking was originally developed study representation learning on images, not timeseries decoding. Causal domains like BCI control can also learn nontrivial representations simply through next-step prediction. NDT2 employed both explicit MAE masking and a causal attention mask its backbone, which is redundant computationally and also reduces the context available to make predictions. NDT3 thus dispenses with the masking mechanism and uses next-step prediction alone. NDT2 also differs from NDT3 in its readout of behavior. Again, since NDT2 studied representation learning, it used additional
36

---

7D Reach/Grasp Kinematics R² (FALCON H1)
16D Reach/Grasp EMG (FALCON M1)
2D Finger Pose (FALCÓN M2)
R²
0%
0.6
0.5
0.4
0.6
0.4
0.2
0.3
0.5
0.2
0.0
Session:
Session:
*esssoon:
Session:
Session:
Session: 10 min
Task: hr
Task: 1 hr
Task: 4 hr
Task: hr
Task: 44 min
Task: 44 min
100%,
100%
100%
100%.
100%s 25
50
100%.
LEGEND X NDTt-CrossEnc (POYO) NDT2 Scratch NDT3 Scratch +DDT3 2thr 350M
Figure 19. datasets. We find that the cross-atiention does not provide benefits over patch-wise tokenization NDT3. For NDT3-CrossEne, we report the best model from 9x larger hyperparameter sweep (over model size and dropout) than the other models.
(POYO reproduction) compared with NDT3 and NDT2 on FALCON movement
cross-attention readout layers to "probe" behavior predictions at each timestep. NDT3 simplifies this two-part encoder-decoder design decoder-only design. In this flow, masked behavior tokens provided at the input and filled in through the backbone, and varying the input tokens allows us to make predictions of the corresponding behavior dimensionality.
NDT-MTM vs NDT Zhang et al. (20] proposed a Multi-task Masking method (MtM) as means of enabling improved scaling for spiking activity. MtM increases the sophistication of the masks applied in the MAE strategy used to train NDTI and NDT2, and finds improvements on varied neural data metrics. We view this as analogous to introducing more granular objectives to improve "pixel" level metrics, and believe the results are consistent with the original image MAE [22] not achieving particularly good pixel metrics. However, decoding metrics, the focus of this work, are similar to those achieved by the baseline NDT models.
Of note. Jiang et al. 73 also shows that NDT-MTM fails to scale across subjects in the mouse neural prediction setting which NDT-MTM was developed, replicating our core claim on the difficulty of cross-subject transfer. Given these similarities and that NDT-MTTM makes no claim on advancing decoding, we did not attempt comparisons.
POYO vs NDT POYO 7 is a Transformer that can pretrain on spiking activity, proposed at a similar time as NDT2. POYO uses a cross-attention encoder, which is qualitatively large departures from the more basic Transformer formula used in NDT. Specifically, rather than directly operating on patched neural data, NDT. POYO first "tokenizes" individual spikes on all channels and performs cross-attention to reduce the large set of resulting tokens to a smaller number of latent tokens. In practice, this yields pre-specified constant number of tokens per timestep, as opposed to NDT's choice of scaling the number of tokens depending on input data dimensionality.
The official POYO codebase weights available, so rather than attempt an exhaustive comparison, we sought a basic test of whether POYO's cross-attention encoder would enable improved performance and thus scaling in NDT3. A priori, we deem this unlikely as the Perceiver-based cross-attention encoder has not been widely adopted in other domains. Our re-implemented cross-altention encoder, holding all other elements constant within NDT3, performs worse than direct patching tokenization Fig. 19 For clarity, we enumerate the differences between NDT3/NDT3-CHon Ene and POYO:
supported as of writing this work, and neither were its pretrained
NDT3 uses joint neural / behavior prediction losses. POYO is trained only with behavior prediction losses.
NDT3 bins neural spike counts, while POYO encodes individual spike times. While the higher fidelity is likely useful, most BCI pipelines do not typically use individual spike times, and not all data releases include these more granular features.
Dataset-specific embeddings: Both NDT2 and POYO use learned dataset embeddings to account for dataset-specife variability. NDT3 discards these learned tokens for simplicity, and accepts the small performance hit this introduces.
37

---

Unit-specific embeddings: POYO additionally learns unit-specific embeddings for each neuron or channel, which varies across datasets. NDT's analogous spatial embedding. learned for each patch (or unit, when patch size is 1), is fixed across datasets.
While our reproduction may understate POYO's overall performance, does suggest that the main design choice that could potentially improve scaling does not outperform simple patching. POYO does outperform NDT2 M2, suggesting common choices with NDT3 such as rotary positional embeddings, might account for its reported benefits.
A.3.4 Pretraining and Evaluation datasets
Pretraining datasets were comprised of historical data from several labs, the rough composition of which is shown in Fig. 23. The evaluation behavior used during pretraining was reaching in 2 monkeys. The first monkey dataset came from public release 74].. and the second from a private dataset Chase lab). The latter had center-out reach in standard conditions and under visual feedback perturbations. The monkey in the second dataset is also present in the kkr monkey and 2kh and up model dataset sizes, though performing in a different set of experiments.
Inherent the process of large-scale scraping is a loss of detail on what precise tasks were used, so we only have qualitative description of tasks we believe are well represented. NDT3 trains on a wide variety of reaching behaviors from relatively constrained (2D center-out reaching to fixed number of targets) to relatively unconstrained (self-paced, more targets, potentially 3D) and under experimental manipulations (delayed onset, multiple targets, different error thresholds requiring more precision). These reaching behaviors are described in both endpoint kinematics and as EMG. A smaller fraction of pretraining data are isometric and force related (force exerted against manipulandums) for wrist and arm motion. Human datasets contain variety of iBCI tasks, with closed loop datasets reflecting both high and low quality control. These tasks include reach and grasp behavior from 1-10 degrees of freedom, as well as some individuated finger tasks for clicking.
We detail evaluation datasets in Table 3 Three datasets come from the FALCON benchmark [23]. two are based on public datasets (128) 27D. and three are private. Note we avoid the Neural Latents Benchmark (5) as does not directly measure decoding performance. For each evaluation dataset, we specify a tuning split and an evaluation split. Only tuning split data is changed when varying data scale. Tuning and evaluation splits are block-contiguous, i.e. trials are not interleaved, for better downstream applicability.
A.3.5 Generalization analyses and Further evaluations
Intra-session generalization Posture, spring, and angular generalization evaluate OOD performance in the standard setup of comparing in-distribution and out-of df-ittiiiiittoinnn performance directly (with changes the underlying evaluation dataset) The intra-session temporal shift analysis is evaluated in an inverted, slightly more rigorous setting. Specifically, we trained two sets of models on the two different temporal blocks, and evaluated on an evaluation split in the later block, rather than only training on the early block and evaluating on both blocks. This way, the OOD shift is measured with respect to the same evaluation dataset.
A.3.6 Architectural Details
NDT3 adopts several architectural innovations used in recent Transformer models. These were compared against baselines in preliminary experiments, but formal ablations in the final experimental setting were not conducted. We defer full description of the Transformer dimensions to the public codebase.
FlashAttention 2 [85 is used to increase training and inference speeds. On the NERSC Perlmutter cluster, with FA2, 45M NDT3 trained at about 270M neural tokens per 40G A 100 hour, 350M NDT3 trained at about 70M neural tokens per A100 hour. Note, FA2 also enables use of the 350M model for real-time (<20ms) inference latency. On an NVIDIA 4090, we see mean inference times of 4ms for the 45M parameter NDT3 and 9ms for the 350M parameter NDT3.
Positional Embeddings [19] Rotary embeddings are applied to indicate the real-world timestep of every input token. Additionally, 48 categorical learned embeddings are reserved
38

---

to distinguish token modality and position within a timestep (10 for neural, 16 for covariates, 16 for covariate constraints, for reward/retum, for dummy tokens, remainder unused).
QK Normalization 866 87 An additional layer norm applied to the query and key embeddings, before the rotary embeddings, which helped stabilize training of the 350M parameter models.
No context embeddings
Differing from NDT2, no learned embeddings for disambiguat- ing imput datasets were prepended to each input. This was removed for simplicity. Per GATO [3] and language modeling practices, we instead leave task / dataset disambiguation to the modeling process: In pretraining, the covariate maskout strategy allows for many tasks to be specified in-context (as later behavior be inferred on the basis of earlier neural-behavioral token relationships). In fine-tuning, the tuning dataset already uniquely specifies the function be learned.
Cross entropy loss for spiking data prediction: We used the standard cross entropy loss to classify spike count over the Poisson loss common many neural data architectures. Since the overall ablation of neural objective shows large impact in this work, it is likely that this decision should be evaluated with neural data related tasks rather than decoding.
We document the Transformer model shapes considered in our work in Table This shape is not systematically explored in our work, and is by historical artifact, slightly different than the shapes used in NLP/CV. Embedding parameters are negligible. One possible area of interest is that the feedforward expansion factor is in model, i.c. the MLP dimension is low. If MLPss do serve as memory stores in Transformers [88]. increasing this shape may yield more performant model size scaling, given the heterogeneity of our datasets.
39

---

Table 3. Evaluation datasets used for multiscale decoding and generalization analyses. The references provide extended description of the behavioral task. Dashed line separates datasets for Section [.1 and for analysis. Datasets use unsorted multi-umi activity and are processed in Is chops unless otherwise mentioned.
Dataset
Description
CALCON H1, M1,
separate single-subject multi-session datasets for different iBCI tasks. Data comes in high data split (held-in), and low-data split (held- out), with the intention on identifying methods that can achieve parity in the two settings. HI is an open loop human dataset for calibrating 7D reach-and-gen in a robot arm. MI is a monkey reach-and-gras to different objects with EMG recordings. M2 is monkey 2D finger movement task with cunipulandum-meassee measured kinematics. Scaling scores are reported on the test set.
M2 23
task
Self-paced reach (RTT) 28
Monkeys reach for random targets one at time in small planar workspace. We decode 2D arm velocity monkey Indy. Has neu-
ral data from M1 and S1, we use M1 in Section 3.1 and Section 3.2 and SI in Section 3.3
Bimanual Cursor Control 27
A human open loop dataset where the participant attempts movement of one or both hands to control two cursors.
2D Cursor Click (private)
Cursor control is classic iBCI endpoint 77 Two human partici- pants attempt movement according to visually cued cursor movement and audiovisual click cues. We also use this dataset for trial structure analysis in Section 3.3
Grasp force (private)
A open-loop dataset with two human participants attempting isometric power grasps. Specifically, participants were asked to match force output according to visual cues in Mujoco environment. Grasps cued were both static (instant onset, hold, and offset) or dynamic (gradually increasing force). This dataset is valuable for human iBCI study because force modulation is required in many motor behaviors, and grasp force has primarily only been characterized in monkeys until now (78). Uses second intervals due to long behavior timescale. We expect this dataset can released by end of 2024.
Critical Stability Task 79 (private, trialized, sorted)
A monkey dataset collected to study continuous control relative to ballistic movement. The monkey balances a virtual cursor on a 1D workspace for up to 6 seconds.
Posture-varied Center-Out [80 (private, trialized, sorted) Spring-load 81
A monkey center-out task, but the monkey's 's hand is adjusted to one of 6 different starting positions. We the central position center and the rest as edge.
A monkey moves fingers, clamped together in a manipulandum for effec- tive 1DoF, is neutral or under spring load.
Center-out, Monkey J 82 (trialized)
Used in Section 3.2 A monkey performs an isometric center out task. Forces are measured by the manipulandum and converted cursor veloe- ity signals.
Center-out, Monkey V (private, trialized) Oculomotor
Used in Section 3.2 A monkey reaches one of 8 radially arranged targets by moving a manipulandum (Kinarm).
A monkey visually tracks (via smooth pursuit) target that moves from center of workspace to one of four directions. A few dozen neurons are recorded on probes in each of frontal eye field (FEF) and area MT. We decode pupil velocity. The small number of neurons in this dataset required resetting NDT3 neural readin/readout layers.
pursuit (83] (private, trialized, sorted)
FALCON H2
Human open loop dataset where participant attempts movement to write letters cued on a screen [69 84]. The large number of timesteps in this dataset required resetting NDT3 neural readin/readout layers (to use fewer neural tokens).
40

---

Model
Layers
Width
MLP Size
Heads
Parameters (M)
NDT2 PT 8
4
256
256
4
6
ND13 Base
6
1024
1024
45
NDT3 Big
12
2048
2048
16
350
Table 4. Transformer Model Shapes used in this work.
41


Reviews:
[{"review_id": "VipIRx7SeM", "reviewer": "Reviewer_4GhD", "summary": "This paper introduces the Neural Data Transformer 3 (NDT-3) model architecture for motor decoding from intracortical spiking activity, which is a generalist, causal transformer inspired by models such as GATO. NDT-3 builds upon NDT-2 but with some key changes: moving from a masked auto encoding (MAE) architecture to a causal transformer, and accepting behaviour tokens as inputs in addition to neural data. Other components mostly remain the same except some implementation details and evaluation schemes. The model is pretrained in self-supervised fashion on 1.75k hours of neural data, post which it is finetuned using SSL on a target calibration dataset before a decoder layer is learnt through supervised learning for behavioural decoding. NDT-3 performs comprehensive evaluations of the scaling properties of a \"vanilla\" (ViT-like) causal Transformer architecture and evaluates the cross-subject transferability of such pretrained BCI decoders. While performance results are overall positive, it highlights some concerns in cross subject transfer and handling the complexity associated with output dimensionality and generalisation to new behaviours. Overall, it is a performant model for neural decoding and highlights key avenues for future work in this subfield.", "questions": "* On lines 294 and 306, did you mean Fig. 4D instead of 4C? 4C seems to be a schematic as opposed to a result on overfitting.\n* Given the prevalence of post-training in LLMs, how important do you think it is for NDT-3-style models given architectural and objective parallels?\n* Again, given the rising adoption of synthetic data pipelines in LLM pretraining, could these be used to alleviate issues of generalisation to unseen targets or other scaling issues? Are there preliminary explorations or results in this direction by authors?\n* NDT-2 had results on cross-species transfer, more specifically that monkey to human transfer did not work. Did NDT-3 alleviate these issues? Maybe I missed this but I couldn't find an explicit study of cross-species transfer, but maybe because cross-subject transfer was ambiguous/negative.\n* I see in the Appendix that you mention latency is < 20ms, are there actual values in ms for specific hardware along with memory requirements? 350M params is quite a lot so I'm curious what prediction times look like on desktop/workstation-equivalent hardware you'd find in a typical lab.", "limitations": "Yes, limitations are discussed overall (the paper analyses the architecture's scaling and specific failure modes explicitly) and discusses broader impact in Appendix A.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n\n* This is among the largest pretrained model for neural data/neural decoding, especially in the context of intracortical spiking activity, and represents a comprehensive effort at scaling model size and pretraining data.\n* While results are mostly positive for scaling, the paper highlights concerns related to output dimensionality, generalisation to heldout/unseen behaviour, and cross subject transfer of pretrained models.\n* The experimental evaluation of NDT-3 (and to some extent NDT-2 in comparison) is comprehensive, although there is a concern that only NDT-style models are considered (for the most part; see Weaknesses).\n\n**Weaknesses**\n\n* It remains an open question how many of these results, especially negative transfer results, are a consequence of design choices in NDT-style models. Some possible culprits:\n\n  * Patch tokenisation scheme adopted from ViT models\n  * Using binned spike counts instead of actual spike timings (as in POYO)\n  * Autoregressive transformer architecture which is not entirely \"causal\" – patches of neural tokens and multiple behaviours are predicted autoregressively even at the same real-world time point\n  * Pretraining + \"continual\" finetuning scheme – SSL finetuning + supervised learning but the lack of post training (maybe with RL) causing models to underperform?\n\n  and so on. While this is not meant to be a request to figure all of these questions out in one paper (which is already dense in terms of evaluations), insights would be appreciated.\n* The paper doesn't show results on FALCON H2, which is the speech task based on Willett et al. (2023) (or similar) and can have long trials of 10s+ length. This highlights some problems with Transformer-based approaches and their ability to deal with long context tasks (issues with length generalisation and/or prohibitive memory requirements) especially when dense labels are not available (the task only has trial-level sentence labels if I understand correctly). Concurrent work from Wairagkar et al. (2025) and Ryoo et al. (2025) may be worth looking at for future extensions.\n* Details on the POYO \"reproduction\" associated with Fig. 19 and Appendix D3 seem sparse/underspecified. Based on what I pieced together, there are still differences with POYO: you consider binned spikes instead of spike times, it is unclear how exactly tokenisation and training were done. I would think POYO's positive, consistent cross-subject transfer comes from unit embeddings and supervised training as opposed to cross-attention. Also, lines 1363-1364 appear to be inaccurate to me (at least as of 4 months ago): there's a checkpoint at https://github.com/neuro-galaxy/poyo and a codebase at https://github.com/neuro-galaxy/torch_brain. These might be worth looking at to iron out differences between POYO and the NDT-3 cross-attention implementation.\n* Another comment about baselines is that Jiang et al.'s recent work on NDT-MTM scaling may not entirely take into account the right hyperparameter settings at different scales, given that they only tuned them on single session and 10 session models (c.f. Yang et al. (2022)).\n* This might be me, but I found the paper dense at times, though I appreciated the insight and attention to low-level details. Re-reading it at times, I felt the message or summary of results kind of got lost with each read. It might be worth including a concise summary in list form with the key positive and negative results in one place, given the breadth of experiments.\n\n**References**\n\n* Willett, Francis R., et al. \"A high-performance speech neuroprosthesis.\" Nature 620.7976 (2023): 1031-1036.\n* Wairagkar, Maitreyee, et al. \"An instantaneous voice-synthesis neuroprosthesis.\" Nature (2025): 1-8.\n* Hee-Woon Ryoo, Avery, et al. \"Generalizable, real-time neural decoding with hybrid state-space models.\" arXiv e-prints (2025): arXiv-2506.\n* Yang, Greg, et al. \"Tensor programs v: Tuning large neural networks via zero-shot hyperparameter transfer.\" arXiv preprint arXiv:2203.03466 (2022).", "was_revised": true, "final_justification": "The authors answered all my questions during the rebuttal and discussion phases. The paper has a lot of interesting analyses that are valuable to the BCI + NeuroFM communities and so I recommend acceptance.", "rebuttal": "We thank the reviewer for your time and thoughtful questions. We generally agree with your summary. We would like to clarify that NDT3 is trained with both supervised behavior and self-supervised neural data prediction objectives, in pretraining and tuning. The related analyses include pretraining curves for each objective in Fig 14, an ablation of neural objectives in pretraining in Fig 9, and an ablation in fine-tuning in Fig 16 / C.11.\n\n### Weaknesses\n\n**NDT design**. We agree there is room for innovation beyond NDT3’s design. This work is meant to establish a rigorous baseline that provides clear goalposts for future work (whether by beating the 90 minute threshold, superior generalization in Fig 4’s analyses, or higher FALCON scores). We do not think these innovations will come easily, though (see global response below). Our thoughts on your suggested culprits:\n- Patch tokenisation: We were optimistic that NDT3’s patching would not be limiting because the patching scaled well in NDT2. In retrospect, this is only because those evaluation datasets were small. POYO’s tokenization is promising, but we were not successful with it.\n- Spike timing: This is a keen observation, and we have since observed that spike timing improves NDT, but we do not expect scaling to qualitatively improve from using spike timing over binned spikes, since binning effectively just lightly noises the input. Practically, binned spike times are necessary as many datasets are only packaged with binned spike data over precise spike timing.\n- Autoregressive causality: We do not see this design as a likely culprit since other work has  demonstrated scaling in domains with no sense of causality/time with random token dropout (e.g. Vision MAE).\n- Lack of post-training: We don’t believe other domains have required post-training to observe good scaling. We are hopeful post-training can improve NDT3’s utility, however, and would be interested in evaluating this in future efforts.\n\n**H2 task**: We did evaluate NDT3 on H2 (C.4, Fig 10). While length generalization and memory were not issues for us (thanks to FlashAttention), we did not find transfer from NDT3’s continuous regression pretraining to H2’s seq2seq classification task. The use of Transformers is likely not fundamentally limiting for either discrete or continuous speech tasks, though, as both Wairagkar 25 and Card 25 use Transformers.\n\n**POYO comparison**: This topic is important to detail. NDT-CrossEnc is so named because NDT’s patching is its most suspect design flaw (Section 3.2), but we tried to compare with POYO more completely than this, and did consider your three raised differences:\n1. Supervised training: NDT2 and NDT3 are both trained with supervised objectives in this work (see the aforementioned Fig 9/14/16 on its importance in NDT3). \n2. Unit embedding vs patch embedding: NDT-CrossEnc reduces patch size to 1, and each patch receives identifying embeddings, so NDT-CrossEnc also essentially receives unit embeddings. One remaining difference is that unit embeddings are distinguished across datasets. In NDT2, dataset-specific parameters were accounted for with a separate session embedding token. We dropped this in NDT3 due to its low benefit (according to NDT2’s ablations) and high cost (45M parameter NDT3 would incur about 20M new params from ~20K datasets x 1K params/dataset).\n3. Binned spikes vs spike timing: As mentioned, we cannot scale with spike timing as this data is frequently missing in BCI datasets.\n\nIn summary, we believe NDT-CrossEnc fairly translates POYO to the scaled BCI setting. We have confirmed our design with the released codebase and in correspondence with a researcher familiar with the POYO model. We did not pursue an exact reproduction as POYO has not shown cross-subject transfer that is qualitatively different from NDT2 (Both were evaluated in low-data downstream setting). We certainly agree that our documentation of NDT3-CrossEnc reproduction (Line 1354-1362) could be improved; we will restructure based on your feedback, similar to the above bullets, to improve clarity.\n\nOne remaining option would be to directly use the public POYO codebase, by adapting POYO model/dataloading/evaluations to the BCI use case (causal / binned spikes). Our confidence in doing this fairly is low, since there are open issues regarding reproducing reported results at the [original POYO code](https://github.com/neuro-galaxy/poyo/issues), and the new codebase also lacks evaluation scripts or end-to-end examples. At this point, we believe it would be most expedient to seek comparison through external benchmarks like FALCON.\n\n**NDT-MTM**: We agree NDT-MTM may not have been exhaustively optimized; NDT3 also did not use Mu-parameterization. To clarify, we omitted comparison with NDT-MTM not because of poor performance, but because its main distinction from NDT was its use of new masking strategies to improve neural data reconstruction (as opposed to behavior decoding).\n\n**Clarity**: Thank you for this note. We had compressed such summaries to fit the content limit, but will add these summaries in the camera-ready.\n\n### Questions\n**294-306**: Yes, we meant 4D. Thanks for catching this typo.\n\n**Post-training**: Thanks for asking! Post-training has not been explored in the neural FM space, but we included the analysis in 4C to specifically highlight a target for post-training, to prioritize a decoder that will generalize to more behaviors when the neural activity would support it. We allude to this in Line 257. One pertinent question will be how to preserve this post-training through the final fine-tuning that all neural models currently require.\n\n**Synthetic data**: To attempt to improve generalization to unseen channel orderings, we piloted the use of channel shuffling augmentations in pretraining. This helped at small scales but began to harm performance by 200 hours of pretraining data. We did not report this as we think this warrants a more thorough investigation that was beyond the scope and content limits of this paper. Other synthetic augmentations from the BCI literature are definitely worth testing for neural foundation models, but have mostly been used to address small signal nonstationarities over time, so we do not expect it to change cross-subject scaling.\n\n**Cross-species**: We did not explicitly mention cross-species transfer in this work, but you raise a good point, so we will clarify the following in the Discussion: The 200 hour model does show cross-species transfer, as it only pretrains on monkey data and positively benefits human evaluations. We do not think this is due to differences between NDT2/NDT3, but rather a result of training and evaluating at a larger scale. \n\n**Latency**: With KV cache enabled and at 1 second context length, we see mean inference times of about 4ms and 9ms for the 45M and 350M models on an NVIDIA 4090. On a 4070, the 350M model slows to ~15-18ms. We add these timings to Line 1412.\n\nThank you for all these suggestions, and we hope that our clarifications and comments (justification of NDT3 and our POYO reproduction, resolution of other weaknesses and questions) address your main critiques and respectfully request reconsideration of the borderline score.\n\n---\n_We attach below a global rebuttal summary if you are interested in how your feedback relates to other reviewer’s comments:_ \n\n4GhD/iNS9/cDht sought more understanding of how NDT’s design might cause the reported limitations. We share this desire but believe addressing it clearly will require scaling alternative models, and we exhausted our resources scaling NDT3 (20K A100 hours). We do believe that NDT3’s design was appropriate as the first choice to scale to 100M+ parameters as 1) NDT2 showed the main choice of patchwise tokenization worked well in the previous order of magnitude, 2) NDT3 outperformed the alternate candidate (POYO) in our hands (Fig 19), and 3) NDT3 closely adheres to the vanilla Transformer that successfully scales in other fields. We are hopeful for future improvements but note that cross-subject transfer is broadly challenging beyond our design and neural modality (Banville 25, Apicella 24). The most promising path forward appears to be scaling to thousands of subjects (Kaifosh 25, Aristimunha 25), which is not yet feasible for implanted BCIs. Instead, we personally are actively exploring richer neural data features and alternate tokenization, as some reviewers have suggested. More generally, we hope that NDT3’s 90 minute saturation mark and limitation analyses have provided clear goalposts for the field. \n\nThe reviewers requested various improvements to clarity. We agree the work is dense and unclear at times, which likely caused some reviewers to request analyses or question claims that were explored in the appendix. We will add 1) summary boxes to our main text sections (4GhD) and 2) a table of contents in the appendix. We will also 3) extend the discussion with an explicit limitations section and restore the impact statement to the main text (iNS9), and 4) tweak figures to improve readability (cDht). Finally, we 5) will add a reference to the supplementary analysis that supports the rigor of our cross-subject evaluation (HeMw).\n\n---\nApicella 24, Toward cross-subject and cross-session generalization in EEG-based emotion recognition: Systematic review, taxonomy, and methods. https://doi.org/10.1016/j.neucom.2024.128354\n\nBanville 25, Scaling laws for decoding images from brain activity. https://arxiv.org/pdf/2501.15322\n\nKaifosh et al, 2025, A generic non-invasive neuromotor interface for human-computer interaction. https://www.nature.com/articles/s41586-025-09255-w\n\nAristimunha 25, EEG Foundation Challenge. https://arxiv.org/abs/2506.19141v1\n\nCard et al, biorXiv 2025. Long-term independent use of an intracortical brain-computer interface for speech and cursor control"}, {"review_id": "t6jUKqQwJP", "reviewer": "Reviewer_HeMw", "summary": "Authors introduce Neural Data Transformer 3 (NDT3), a pretrained autoregressive transformer for decoding motor behavior from intracortical neural activity measured from monkeys and humans.  Pre-training the model on up to 2000 hours of neural and behavioral data, the authors evaluate the decoding performance of NDT3 on 8 motor datasets.  In this analysis, they study model performance when scaling the number of parameters in the model and the amount of pre-training data used.  The authors suggest benefits of scaling both pre-training dataset size and model parameter count, but only when the amount of downstream, task-specific data is very limited (up to ~1.5 hours of data).", "questions": "- Clarification: Do all 200 hour and 2000 hour pre-training datasets have data from the test subject (even if not specific to the test task)?\n- If feasible, can the authors provide any numerical results for how generalization scales as a factor of number of subjects (or alternatively number of sessions, number of hours, etc.) in the training dataset while strictly excluding any data from the test subject?\n- How does performance scale with pre-training data when you can only train a linear readout (instead of fine-tuning all parameters of the model)?  Would this not be a more stringent test of the generalization of pre-trained features with scaling?", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- The authors provide a wide breadth of evaluations, and highlight where NDT3 succeeds and fails.  These analyses may offer useful insights for practitioners seeking to scale their neural data decoders.\n- NDT3 achieves strong performance as compared to conventional baseline Wiener Filters in most evaluated tasks.\n- Although saturation limits were observed with relatively low quantities of motor behavior pre-training data, such pre-training strategies may be applicable in regimes where data collection is prohibitively expensive or only small amounts of new-subject calibration data can be acquired (e.g., end-user BCI applications).\n\nWeaknesses:\n- A primary motivation of this paper and neural foundation models in general is to support generalization to new tasks and subjects.  Minimal improvements are made in this regard (most notable improvements from pre-training occur when subject data is available in pre-training data or when downstream data is very limited) and little guidance is provided into how this may be achieved.\n- The benefits of pre-training are limited when there is no data from the test subject in the pre-training data.  Outside of laboratory settings, it is rarely the case that we would have training data available for a BCI user (even if from a different task), limiting practicality of this type of strategy.\n- At many points in the paper it is unclear what type of generalization is being accounted for and whether it is actually cross-subject generalization.  The authors highlight that their goal is to study \"scaling of cross-subject generalization\" (Line 125), but in many cases data from the test subject (but not task-specific data) is included in training (unless I am misunderstanding this).  I disagree that this is cross-subject generalization, since subject-specific neural data distributions can be learned from the cross task-data.", "was_revised": true, "final_justification": "Updating my score to a 4 based on rebuttal discussions with the authors.  This discussion helped resolved misclassifications that I had about their cross-subject generalization training and evaluation methodology.  What remains unresolved is a more detailed analysis of how cross-subject generalization scales, quantified in terms of amount of training data and number of subjects, when absolutely no data is available from the subject in pre-training and variable amounts of data (e.g., 10, 20, ... minutes of subject data) available for fine-tuning or linear decoding.\n\nRegardless of this unresolved aspect, I believe this study offers valuable insights into challenges associated with scaling models for neural data modeling.", "rebuttal": "We thank you for your time and detailed review of our paper. We agree with your summary with one small clarification: although 90 minutes of downstream data is far from trivial. As evidence, prior studies (POYO, NDT2) only showed improvements when tuning datasets of single experimental sessions lasting ~1-10 minutes, and intracortical neural data releases overall remain short; monkey studies often publish with 100-200 minutes total. In this framing, the fact that NDT3’s benefit saturates at 90 minutes is a non-issue. We stated that the benefit is of limited practical utility, because in recent years BCI research has begun adopting multi-session modeling as a norm (Line 132), and in this regime 90 minutes of calibration data could be achieved after say, 3-5 BCI sessions. In this more realistic regime, unaddressed by prior neural pretraining work, we envision NDT3 as still being useful when onboarding new BCI users (in the lab and the real world), or when testing new behavioral paradigms with monkeys. \n\nWith this context, we next address the main body of your critiques and questions, which center on the issue of cross-subject generalization. \n\nOur view is that our work identifies for the first clear time that there are significant cross-subject transfer issues when scaling models on spiking activity. Thus NDT3’s contribution is not to improve cross-subject transfer in this regime, because there is no baseline to improve over at our scale of evaluation, but to definitively establish the challenge of cross-subject generalization in this domain. We agree that the former interpretation would justify much of your critique (i.e. weakness 1), and we believe our miscommunication may have caused much of your subsequent review. Specifically, addressing weakness 3 as to whether our evaluation addresses cross-subject generalization, we state at the start of 2.3, (line 122):\n> Our main evaluation uses four human and four monkey datasets [...] All downstream monkeys are held-out of pretraining, and all humans are held-out of < 2 khr models. That is, we evaluate **scaling of cross-subject generalization.**\n\nThat is, only the 2khr model contained data from some evaluation subjects (Addressing Q1). Of course, the 2khr setting is our largest scale, and we realize now that we asserted (Line 125) that the work nonetheless evaluated cross-subject generalization without further justification. The following explanation was omitted:\n\n> We did not hold humans out of our full 2khr model since we have so few human datasets to begin with, and pre-determined evaluation datasets overlapped with subjects with significant data. To alleviate concerns on subject leakage, we verified that the 2khr model, with subject leakage, did not outperform other models on the test datasets with leaked subjects (Fig 11, top row), and did outperform other models on a held-out human evaluation (Fig 11, top right). This observation (Line 1157), supports our claim that NDT3’s demonstrated benefits are all in a setting where no evaluation subject data was seen in pretraining (contrasting with the critique in weakness 2), and that our evaluation reflects scaling cross-subject gains overall. To summarize, **our pretraining data rarely included data from evaluation subjects, and where it did, models did not benefit from this data.** \n\nIf you may now agree that we did evaluate cross-subject generalization, we can address Q2, on scaling with the number of subjects. Our main results (Fig 3C/D) show that performance increases with (cross-subject) pretraining data hours. Individual task results varied greatly, so we did not attempt more precise scaling laws. More precise quantification of the scaling benefit of cross-subject hours is measured in NDT2 Fig 3, 4 [Ye 23], for one downstream evaluation. Unfortunately, your strongest request of identifying scaling with respect to the number of training subjects is challenging because the data available in different subjects is heterogeneous in volume and quality (Fig 2A) and more simply, because the number of subjects in intracortical neural data is low overall. We hope that our work motivates future neural data collection efforts with hundreds of subjects, to analyze precisely your requested analysis. \n\nFinally, your Q3 is interesting, but unfortunately we have not run analyses on how probes might scale with pretraining. We agree probes would provide a stricter view of pre-trained features. We chose to evaluate E2E fine-tuning to best reflect downstream usage. Downstream BCI deployment most likely will value E2E tuning’s greater or equal performance relative to probes because tuning cost is not significant at NDT3’s size. It is our understanding that linear probes are most frequently used to evaluate representation learning when model pretraining differs from downstream application (e.g. ImageNet/CLIP to generic vision tasks). This is also of interest for NDT3, so probe-based evaluations are of interest for future work.\n\nWe believe your main concern with this work has been around pretraining leakage and our claim of cross-subject generalization. We will add a summary of our response here around Line 125. Thank you for advocating for more clarity on this point. We hope that this addition and our response here have alleviated your concerns, and respectfully request a reconsideration of your score if so.\n\nYe et al, 2023. Neural Data Transformer 2. https://openreview.net/pdf?id=CBBtMnlTGq\n\n---\n_We attach below a global rebuttal summary if you are interested in how your feedback relates to other reviewer’s comments:_ \n\n4GhD/iNS9/cDht sought more understanding of how NDT’s design might cause the reported limitations. We share this desire but believe addressing it clearly will require scaling alternative models, and we exhausted our resources scaling NDT3 (20K A100 hours). We do believe that NDT3’s design was appropriate as the first choice to scale to 100M+ parameters as 1) NDT2 showed the main choice of patchwise tokenization worked well in the previous order of magnitude, 2) NDT3 outperformed the alternate candidate (POYO) in our hands (Fig 19), and 3) NDT3 closely adheres to the vanilla Transformer that successfully scales in other fields. We are hopeful for future improvements but note that cross-subject transfer is broadly challenging beyond our design and neural modality (Banville 25, Apicella 24). The most promising path forward appears to be scaling to thousands of subjects (Kaifosh 25, Aristimunha 25), which is not yet feasible for implanted BCIs. Instead, we personally are actively exploring richer neural data features and alternate tokenization, as some reviewers have suggested. More generally, we hope that NDT3’s 90 minute saturation mark and limitation analyses have provided clear goalposts for the field. \n\nThe reviewers requested various improvements to clarity. We agree the work is dense and unclear at times, which likely caused some reviewers to request analyses or question claims that were explored in the appendix. We will add 1) summary boxes to our main text sections (4GhD) and 2) a table of contents in the appendix. We will also 3) extend the discussion with an explicit limitations section and restore the impact statement to the main text (iNS9), and 4) tweak figures to improve readability (cDht). Finally, we 5) will add a reference to the supplementary analysis that supports the rigor of our cross-subject evaluation (HeMw).\n\n---\nApicella 24, Toward cross-subject and cross-session generalization in EEG-based emotion recognition: Systematic review, taxonomy, and methods. https://doi.org/10.1016/j.neucom.2024.128354\n\nBanville 25, Scaling laws for decoding images from brain activity. https://arxiv.org/pdf/2501.15322\n\nKaifosh et al, 2025, A generic non-invasive neuromotor interface for human-computer interaction. https://www.nature.com/articles/s41586-025-09255-w\n\nAristimunha 25, EEG Foundation Challenge. https://arxiv.org/abs/2506.19141v1"}, {"review_id": "nHotEZTlaO", "reviewer": "Reviewer_iNS9", "summary": "The paper introduces Neural Data Transformer (NDT) 3, a standard transformer-based model trained on large amount of neural recordings. The authors explore the extent to which the large-pretraining/foundation model paradigm apply to the field of neural decoding and BCI. To that aim, they train their model on ~2000hrs of neural data coming from either human or monkey subjects (in motor cortical areas) and test the downstream behavioural decoding performances as a function of model and dataset size. They identify that large pre-training benefits decoding when available subject data is <1.5hrs. Furthermore they explore different ID vs. OOD generalisation scenarios: (1)  different configurations when distribution shifts are due to different time or other presence/absence of spring load feedback (2) continuous/trialized training (3) unseen behaviour (novel angles in 2D center-reach).", "questions": "1. How would NDT3-neural only compare against NDT2?\n2. Why do authors believe that NDT3 suffers high training instabilities in the H2 task (Figure 10)? Also, why does the pretrained-version have such higher initial loss? It is also interesting that the runs that exhibit training instabilities actually results in the best performance.\n3. With data quality being a crucial factor when training foundational models, what would the effect of (automated) spike-sorting be on the quality of the final model? Current pipeline only employs filtering+thresholding, yielding noisy data. While perhaps not directly applicable to the field of BCI (efficient online spike sorting is an area of active development), when applied during pre-training it could potentially improve model performance (albeit also widening the ID vs. OOD gap).", "limitations": "There is currently no explicit discussion of the limitations of this work nor a statement about potential societal impact. The author could for example discuss the still unexplored axes that might influence model quality, e.g. spike-sorting input data, potential importance of data diversity (other brain regions) etc... or the identified pathological sensitivity to input order and stereotyped outputs.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "The paper present a comprehensive study of the benefits (and limitations) of scaling model pretraining in the field of neural decoding, a important and very active area of modern research with several potential downstream applications.\n\nWhile the proposed model employs standard tools in ML for decoder-only transformer training (e.g. rotary embeddings, custom tokens etc..), its scaling analysis reveals important and neural-specific limitations that are worthy of consideration and will inform the future development of neural decoding efforts.\n\n# Strengths\n- The paper provides extensive evaluations and supplementary experiments.\n- The identified limitations (saturation of scaling performance) are highly relevant for the field and can inform therapeutic applications.\n- The authors provide a model that can be applied to a variety of downstream task with good performance.\n\n# Weaknesses\n- The authors in the main text mention that full-comparisons with POYO were omitted as it underperformed NDT3 on the FALCON dataset. However Figure 19 in the appendix suggests a more nuanced situation. For example in the 16D Reach/Grasp EMG (FALCON M1) - mid panel in Figure 19 the POYO-like model either matched or surpassed NDT3 performance in the 100% data regime. Similar trends emerge in the other panels (e.g. in M2). Most importantly for the claims of the paper, the POYO-like model seems to enjoy more favourable scaling laws (with performance increasingly more rapidly with data size), thus hinting that perhaps scaling it to the ~2000hrs regime could have yielded better results.\n- In the paper the authors compare NDT3 vs. NDT2 (Figure 11). However NDT2 was not trained with behavioural labels and Figure 9 suggests that training with behaviour is beneficial. Hence a fairer/interesting comparison would have been NDT2 vs. NDT3-neural-only to disentangle the contribution of the design choices of NDT3.\n\nMinor points\n- In Figure 11 there is a mismatch between the legend of NDT2-100hrs (cross marker) and the plots (circle markers) - assuming color is correct.\n- There is no explicit discussion of the current limitations & the discuss itself is very limited", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their careful reading of our manuscript (e.g. the Fig 11 typo catch), and we agree with your summary. We address your concerns individually below.\n\n**Weaknesses**\n- **POYO-scaling**: Due to evaluation noise, we do not believe that any model is significantly different in the 100% M1 point or M2 50%/100% points. Nonetheless, we do also see that POYO started worse than NDT3 but is at parity by the larger data scales. You raise an interesting interpretation that this might imply that POYO will scale faster with more data. We had interpreted this trend as meaning that our POYO reproduction trained poorly at lower scales. Overall, Fig 19 is open to interpretation. For example, NDT2 sees similar slopes as POYO in the right half of each of these plots, so by similar argument we could have aimed to scale NDT2. Since we could not afford to scale pretraining on multiple methods, we judged it best to scale NDT3 after designing it for scaled training and verifying it had highest overall performance in scratch settings. We look forward to future work that provides more rigorous comparison. We believe we have made it relatively easy to demonstrate better scaling than NDT3 with clear goalposts (90 minutes, tying with baselines at 100% FALCON scale, Sec 3.2 generalization analyses).\n- **NDT2 objectives**: You have identified a subtle implementation detail! In the NDT2 codebase, most models are indeed prepared with a neural-only SSL objective, and probed for behavior labels. However, some of NDT2’s experiments (e.g. Fig 5) simplify this 2-step training into a single model that trains with both neural and behavioral objectives. For fairness (as you say), we have used this joint training objective through our evaluation of NDT2 in this work. We mentioned this briefly in Line 1317-1318. To clarify, we now modify Line 1317 to read: \n> “The only departure we take from NDT2’s default design is that we train NDT2 with both a neural reconstruction loss (25% masking) and a supervised decoding loss. This joint objective was introduced late in the NDT2 paper (Fig 5) and performs comparably to multi-stage training while simplifying large scale evaluation.”\n\n**Questions**\n- **NDT3-Neural against NDT2**: See above.\n- **H2**: In conversation with researchers familiar with the H2 task, we speculate that our irregular training is just a result of overparameterized Transformers with insufficient regularization. H2 only contains about 600 data points in total. Most works in the field use small RNNs, except Card 25, which adopted the Transformer after collecting much larger volumes of data. The pretrained model loss is higher likely because NDT3 pretrains on continuous regression tasks while H2 is a Seq2Seq task. We do not see a reason NDT3 could not perform reasonably on H2 if its pretraining mixed in a larger dataset of Seq2Seq spiking decoding tasks.\n- **Spike-sorting**: We agree with your intuition that spike-sorted inputs could enrich the representations learned in pretraining. Practically, this choice is often already made for us, as few datasets are released with broadband data or sorted spikes, and some BCI labs do not save the broadband data due to its large storage footprint. We are broadly interested in the development of models that support richer neural data features, but the field needs convincing evidence that the additional overhead of these features are justified.\n\n**Limitations**: We have initially omitted an explicit Limitations section as our results have a section (3.2) analyzing methodological limitations. We acknowledge, though, that there are other limitations in our work unaddressed by 3.2, and will include some of your suggestions in a new explicit Limitations section in camera-ready. Similarly, we will restore our impact statement from section A in the appendix to the main text.\n\nBy contextualizing why we chose to scale NDT3 over POYO, clarifying NDT2’s joint objective, and lengthening the discussion, we hope we have addressed your concerns on the work’s weaknesses, and ask that you might consider a non-borderline score. Thanks for your time!\n\nCard et al, biorXiv 2025. Long-term independent use of an intracortical brain-computer interface for speech and cursor control\n\n---\n_We attach below a global rebuttal summary if you are interested in how your feedback relates to other reviewer’s comments:_ \n\n4GhD/iNS9/cDht sought more understanding of how NDT’s design might cause the reported limitations. We share this desire but believe addressing it clearly will require scaling alternative models, and we exhausted our resources scaling NDT3 (20K A100 hours). We do believe that NDT3’s design was appropriate as the first choice to scale to 100M+ parameters as 1) NDT2 showed the main choice of patchwise tokenization worked well in the previous order of magnitude, 2) NDT3 outperformed the alternate candidate (POYO) in our hands (Fig 19), and 3) NDT3 closely adheres to the vanilla Transformer that successfully scales in other fields. We are hopeful for future improvements but note that cross-subject transfer is broadly challenging beyond our design and neural modality (Banville 25, Apicella 24). The most promising path forward appears to be scaling to thousands of subjects (Kaifosh 25, Aristimunha 25), which is not yet feasible for implanted BCIs. Instead, we personally are actively exploring richer neural data features and alternate tokenization, as some reviewers have suggested. More generally, we hope that NDT3’s 90 minute saturation mark and limitation analyses have provided clear goalposts for the field. \n\nThe reviewers requested various improvements to clarity. We agree the work is dense and unclear at times, which likely caused some reviewers to request analyses or question claims that were explored in the appendix. We will add 1) summary boxes to our main text sections (4GhD) and 2) a table of contents in the appendix. We will also 3) extend the discussion with an explicit limitations section and restore the impact statement to the main text (iNS9), and 4) tweak figures to improve readability (cDht). Finally, we 5) will add a reference to the supplementary analysis that supports the rigor of our cross-subject evaluation (HeMw).\n\n---\nApicella 24, Toward cross-subject and cross-session generalization in EEG-based emotion recognition: Systematic review, taxonomy, and methods. https://doi.org/10.1016/j.neucom.2024.128354\n\nBanville 25, Scaling laws for decoding images from brain activity. https://arxiv.org/pdf/2501.15322\n\nKaifosh et al, 2025, A generic non-invasive neuromotor interface for human-computer interaction. https://www.nature.com/articles/s41586-025-09255-w\n\nAristimunha 25, EEG Foundation Challenge. https://arxiv.org/abs/2506.19141v1"}, {"review_id": "yuyTgOSTf5", "reviewer": "Reviewer_cDht", "summary": "This paper builds an improved foundation model for decoding neural activity into estimates of manual behavior.  The authors use a combination of human and monkey data, about 2000 hours of data across multiple behavioral tasks.  They show improved results on several downstream decoding tasks and also perform a number of ablation and scaling experiments to explore the benefits of this large foundation model approach.  It’s a nice paper overall and well-written.\n\nMain Finding:\nThe key takeaway is that pretraining helps, primarily by improving sample efficiency. However, beyond 1.5 hours of data, there's no additional performance gain, which they attribute to data heterogeneity (in recording devices, pattern of dead channels, subjects, etc).  The generalization to out-of-distribution data is also weak in several specific cases that the authors highlight.\n\nModel Choices:\n•\tThey use a causal transformer and acknowledge that the negative results might partly due to architectural choices. \n•\tRegarding tokenization: they group spikes from 32 channels into patches per time step. This may be suboptimal, especially when transferring across devices—channel groupings lose their meaning. The model also ignores information about brain region, which is probably important for Neuropixels recordings that can span multiple regions in a single experiment.  (I’m not sure if this paper uses Neuropixels recordings but these are becoming more common in both monkey and human recordings.)", "questions": "-\tObvious question - is it possible to fix the out-of-distribution problems highlighted in fig 4?  (Maybe this will be the topic of the next paper.)\n-\tSome of the r2 values of the predictions are fairly low.  It might be useful to see a decomposition of the errors across different frequency bands – I’d guess that the highest frequencies are impossible to predict (due to sensor noise or unconstrained motor variability) and the r2 values for the lower frequencies may be better\n-\tIn general it would be nice to see some more predicted traces, so we get a clearer qualitative sense of what these models are predicting", "limitations": "adequately addressed", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n-\tWell-written and clear\n-\tImprovement on the SOTA\n-\tAddresses an impactful and interesting problem\n-\tRefreshingly honest about the limitations\n\nWeaknesses\n-\tSeveral of the results fig panels were pretty obscure at first read; I’d recommend looking the figures over again and trying to clarify these for non-expert readers", "was_revised": true, "final_justification": "It's a good paper, and the authors had good responses to the reviews.  I maintain my score.", "rebuttal": "Thank you for the positive assessment. We respond to your points below.\n- Re: brain region information. Nearly all devices in our data are placed in hand/arm regions in the motor cortex. Within this scope, even the coarse anatomy varies across humans (and between species), so it is not clear how to use a general coordinate system or discriminative region label. Such labels may be non-critical, thankfully, as [Azabou 25] shows that the neural activity itself is predictive of the brain region they are sourced from.\n- Fixing out of distribution in Fig 4: We hope so, but do not have a clear path forward. By thought experiment, we infer that POYO should be more robust to the synthetic shuffles, but likely only with the a priori foresight to tune the unit embeddings first. \n- Error decomposition / traces: Frequency-band error decomposition is an interesting suggestion. In constrained monkey experiments, decoding R2s are quite high, so sensor noise is likely not dominant. We didn’t observe salient qualitative changes in predictions beyond previous decoding models, but will add in an example trace prediction for some of our evaluation tasks in the appendix.\n\nAzabou et al, 2025. Multi-session, multi-task neural decoding from distinct cell-types and brain regions\n\n\n---\n\n_We attach below a global rebuttal summary if you are interested in how your feedback relates to other reviewer’s comments:_\n\n4GhD/iNS9/cDht sought more understanding of how NDT’s design might cause the reported limitations. We share this desire but believe addressing it clearly will require scaling alternative models, and we exhausted our resources scaling NDT3 (20K A100 hours). We do believe that NDT3’s design was appropriate as the first choice to scale to 100M+ parameters as 1) NDT2 showed the main choice of patchwise tokenization worked well in the previous order of magnitude, 2) NDT3 outperformed the alternate candidate (POYO) in our hands (Fig 19), and 3) NDT3 closely adheres to the vanilla Transformer that successfully scales in other fields. We are hopeful for future improvements but note that cross-subject transfer is broadly challenging beyond our design and neural modality (Banville 25, Apicella 24). The most promising path forward appears to be scaling to thousands of subjects (Kaifosh 25, Aristimunha 25), which is not yet feasible for implanted BCIs. Instead, we personally are actively exploring richer neural data features and alternate tokenization, as some reviewers have suggested. More generally, we hope that NDT3’s 90 minute saturation mark and limitation analyses have provided clear goalposts for the field. \n\nThe reviewers requested various improvements to clarity. We agree the work is dense and unclear at times, which likely caused some reviewers to request analyses or question claims that were explored in the appendix. We will add 1) summary boxes to our main text sections (4GhD) and 2) a table of contents in the appendix. We will also 3) extend the discussion with an explicit limitations section and restore the impact statement to the main text (iNS9), and 4) tweak figures to improve readability (cDht). Finally, we 5) will add a reference to the supplementary analysis that supports the rigor of our cross-subject evaluation (HeMw).\n\n---\nApicella 24, Toward cross-subject and cross-session generalization in EEG-based emotion recognition: Systematic review, taxonomy, and methods. https://doi.org/10.1016/j.neucom.2024.128354\n\nBanville 25, Scaling laws for decoding images from brain activity. https://arxiv.org/pdf/2501.15322\n\nKaifosh et al, 2025, A generic non-invasive neuromotor interface for human-computer interaction. https://www.nature.com/articles/s41586-025-09255-w\n\nAristimunha 25, EEG Foundation Challenge. https://arxiv.org/abs/2506.19141v1"}]

Author rebuttal:

Meta-review: In this paper the authors investigate using a pre-trained model to map neural activity to motor behavior. They pre-trained an autoregressive transformer on >2000 hours of intracortical neural and motor data from more than 30 monkeys and humans. Their results show the pre-training of the model is broadly beneficial, improving performance across eight different downstream decoding tasks and generalizing well to shifts in neural data. However, the authors also conclude that simply scaling such models is unlikely to resolve fundamental limitations arising from sensor variability and stereotyped patterns in neural datasets.

The primary concerns of the reviewers were centered on the true extent and limitations of the model's generalization capabilities, as well as the context provided by the baseline comparisons. A major recurring point, raised by multiple reviewers, was the weakness in cross-subject and out-of-distribution generalization. One reviewer questioned the definition of "cross-subject generalization" used in the paper, arguing that including any data from the test subject (even from a different task) in the pre-training set limits the practicality of the approach and does not represent a true zero-shot transfer scenario. This concern was linked to specific architectural choices, such as the patch-based tokenization of neural data, which reviewers suggested might not be robust to heterogeneity across different subjects, tasks, and recording hardware.

Furthermore, reviewers expressed a need for more rigorous comparisons to alternative models to better contextualize NDT-3's performance and limitations. Specifically, the comparison to the POYO model was noted as being insufficient or revealing a more nuanced story than presented in the main text, with one reviewer suggesting POYO might have more favorable scaling properties in some regimes. A desire for a fairer ablation study against the previous NDT-2 model was also mentioned to better isolate the benefits of the new architecture from the benefits of pre-training with behavioral data. Minor points on the paper's density and the clarity of some figures were also raised.

Despite these concerns, the reviewers unanimously recommended acceptance, driven by several common positive assessments. All reviewers lauded the paper as a comprehensive, large-scale, and highly valuable study in the important and challenging field of neural decoding. They recognized it as a significant effort, representing one of the largest pre-trained models for intracortical spiking activity to date. A key strength highlighted was the paper's honesty and transparent analysis of its own limitations. Rather than only focusing on successes, the authors' detailed exploration of where scaling helps and where it fails (e.g., the performance saturation and generalization issues) was seen as a crucial and insightful contribution that will inform future work in the BCI and neural foundation model communities. This combination of achieving strong performance, improving on the state-of-the-art, and providing a nuanced look at the challenges ahead was the clear basis for the positive recommendations.

With the reviewers reaching consensus that this paper should be accepted, and a final average score of 4.5, a decision of Accept (poster) was decided on.
Decision: Accept (poster)