Neural Encoding and Decoding at Scale
Authors: Yizi Zhang
Yanchen Wang
Mehdi Azabou
Alexandre Andre
Zixuan Wang
Hanrui Lyu
International Brain Laboratory
Eva L Dyer
Liam Paninski
Cole Lincoln Hurwitz
Venue: ICML 2025 spotlightposter (2025)
Keywords: neural coding
multimodal
multi-task
transformer
masked modeling
electrophysiology
neuroscience
brain region
Neuropixels
Primary area: applications->neuroscience_cognitive_science

Abstract: Recent work has demonstrated that large-scale, multi-animal models are powerful tools for characterizing the relationship between neural activity and behavior. Current large-scale approaches, however, focus exclusively on either predicting neural activity from behavior (encoding) or predicting behavior from neural activity (decoding), limiting their ability to capture the bidirectional relationship between neural activity and behavior. To bridge this gap, we introduce a multimodal, multi-task model that enables simultaneous Neural Encoding and Decoding at Scale (NEDS). Central to our approach is a novel multi-task-masking strategy, which alternates between neural, behavioral, within-modality, and cross-modality masking. We pretrain our method on the International Brain Laboratory (IBL) repeated site dataset, which includes recordings from 83 animals performing the visual decision-making task. In comparison to other large-scale modeling approaches, we demonstrate that NEDS achieves state-of-the-art performance for both encoding and decoding when pretrained on multi-animal data and then fine-tuned on new animals. Surprisingly, NEDS's learned embeddings exhibit emergent properties: even without explicit training, they are highly predictive of the brain regions in each recording. Altogether, our approach is a step towards a foundation model of the brain that enables seamless translation between neural activity and behavior.

Neural Encoding and Decoding at Scale
Yizi Zhang
Yanchen Wang"!
Mehdi Azabou'
Alexandre Andre Zixuan Wang' Hanrui Lyu²
The International Brain Laboratory'
Eva Dyer² Liam Paninski 1.4 Cole Hurwitz!. 1.1
Columbia University
Universityy of Pennsylvania "The International Brain Laboratory
Northwesteen University
'Equal Contribution
Abstract
1. Introduction
Systems neuroscience has experienced a revolution in data acquisition, from brain-wide recordings using Neuropixels probes (Jun et al., 2017; Steinmetz et al., 2019: 2021; IBL et al., 2023; Khilkevich et al., 2024; Chen et al., 2024), to automated tracking of behavior from video data (Mathis et al., 201 8; Pereira et al., 2022; Biderman et al., 2024). While these multi-animal, multimodal datasets open new avenues for understanding the relationship between neural activity and behavior, they also necessitate the development of analysis approaches that can effectively leverage massive volumes of data (Paninski & Cunningham, 2018; Stringer & Pachitariu, 2024).
Recent work has demonstrated that large-scale, multi-animal models are powerful tools for char- acterizing the relationship between neural activ- ity and behavior. Current large-scale approaches, however, focus exclusively on either predicting neural activity from behavior (encoding) or pre- dicting behavior from neural activity (decoding), limiting their ability to capture the bidirectional relationship between neural activity and behav- ior. To bridge this gap, we introduce a multi- modal, multi-task model that enables simulta- neous Neural Encoding and Decoding Scale (NEDS). Central to our approach is novel multi- task-masking strategy, which alternates between neural, behavioral, within-man modality, and cross- modality masking. We pretrain our method on the International Brain Laboratory (IBL) repeated site dataset, which includes recordings from 83 ani- mals performing the same visual decision-makin task. In comparison to other large-scale mod- cls, we demonstrate that NEDS achieves state- of-the-art performance for both encoding and decoding when pretrained on multi-animal data and then fine-tuned on new animals. Surpris- ingly, NEDS's learned embeddings exhibit emer- without explicit training, they are highly predictive of the brain regions in each recording. Altogether, our approach is a step towards a foundation model the brain that enables seamless translation between neu- ral activity and behavior. Project page and code: https:/ //ibl -edd.nndd.... gi trithuu.i.. io/
Recent efforts to address this challenge have focused on developing large-scale models that can be trained across multiple sessions and animals (Azabou et al., 2024; Ye et al., 2024; Zhang et al., 2024b;a; Posani et al., 2024; Wang et al., 2025). These works demonstrate that with increas- ing scale of neural data comes improved performance on downstream tasks such predicting neural activity from behavior (encoding) or predicting behavior from neural ac- tivity (decoding). Despite their promise, most preexisting large-scale modeling approaches are constrained by their task-specific nature, focusing exclusively on either encoding (Posani et al., 2024; Wang et al., 2025) or decoding (Az- abou et al., 2024; Ye et al., 2024; Zhang al., 2024a). This limits their ability to model the bidirectional relationship between neural activity and behavior. Overcoming this limi- tation will require multimodal models capable of seamlessly translating between different modalities.
gent properties:
address this need, we propose a method for simulta- neous Neural Encoding and Decoding at Scale (NEDS). Our approach leverages recent advances in self- self-supervised learning and multimodal masked modeling (He et al., 2022; Mizrahi et al., 2023), training a transformer- to jointly capture the relationship between neural activity and behavior. We utilize a multi-task. tastiiiins-mms strategy (Tay et al., 2022; Zhang et al., 2024b) that alternates between neu-
Correspondence to: Yizi Zhang
model
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Neural Encoding and Decoding at Scale
ral, behavioral, within-modality, and cross-modal masking (shown in Figure 1 A)). After training. NEDS can perform decoding by masking behavior and predicting it from neural activity, or encoding by masking neural activity and predict- ing it from behavior. Our multi-rask--mskingg framework unifies encoding and decoding, enabling flexible and scal- able analysis of neural activity and behavior.
decade (Gilja et al., 2015; Pandarinath et al., 2017; Wil- lett al., 2023; Metzger et al., 2023). Encoding models classically take the form of linear or "generalized linear models" (GLMs) (Paninski, 2004; Truccolo et al., 2005). Again, recent advancements in deep learning have led to considerable improvements in this domain (Wen et al., 2018; Schrimpf et al., 2018; Ustyuzhaninov et al., 2022; Willeke et al., 2022; Li et al., 2023: Turishcheva et al., 2024; Wang et al., 2025). Encoding models can provide insights into the brain, such as how uninstructed movements dominate neural variability (Musall et al., 2019) or how neural correlates of decision-making are distributed across many brain regions (IBL et al., 2023).
We evaluate our approach on the International Brain Labora- tory (IBL) repeated site dataset (IBL et al., 2022), which con- sists of Neuropixels recordings targeting the same brain re- gions across 83 mice performing the same decision-making task. We benchmark NEDS on encoding and decoding of key task variables including whisker motion, wheel velocity, choice, and the "block" prior (Findling et al., 2023). We first demonstrate that NEDS outperforms an equivalent uni- modal encoding and decoding method. We then compare NEDS to preexisting large-scale modeling approaches in- cluding POYO+ (Azabou et al., 2025) and NDT2 (Ye et al., 2024). We demonstrate that NEDS achieves superior perfor- mance compared to these approaches when pretrained on trial-aligned data from 73 animals and fine-tuned on data from 10 held-out animals. Finally, we show that pretrained NEDS exhibits emergent properties; without explicit train- ing, the latent representations learned by NEDS are highly predictive of the brain regions in each session. Taken to- gether, NEDS represents a new paradigm for scale neurobehavioral modeling, bringing us closer to a founda- tion model of the brain at single-cell, single-spike resolution. The contributions of this work include:
Multimodal models of neural activity and behavior. Recent advancements in large-scale electrophysiology and video tracking have facilitated the collection of massive neurobe- havioral datasets. This has led to the development of novel multimodal approaches for understanding the relationship between neural activity and behavior (Sani et al., 2021; Hur- witz et al., 2021b; Gondur et al., 2023; Sani et al., 2024; Schulz et al., 2025). One common strategy is to use latent variable modeling to learn shared embedding space for neural and behavioral data (Hurwitz et al., 2021a), as demon- strated by methods like PSID (Sani et al., 2021) and TNDM (Hurwitz et al., 2021b). More recently, masked modeling ap- proaches, such as the masked VAE (M-VAE) proposed in a preprint by Schulz et al. (2025), have shown promise. In this work, structured masking is used to learn the conditional distributions between neural activity and behavior. Similar to NEDS, this allows for performing encoding and decoding after training. A limitation of M- M-VAE is its sequential VAE architecture VAE), which poses scaling challenges and limits its applicability to multi-animal datasets.
A unified modeling approach for encoding and decod- ing of behavior at scale (NEDS) that achieves state-of- the-art performance in both tasks.
A demonstration that both encoding and decoding per- formance scale meaningfully with the amount of pre- training data and model capacity (i.e., scaling).
Large-scale models for neural analysis. Traditional analy- ses of neural data and behavior have been limited to single- animal, single-session models. Recent evidence suggests that this paradigm neglects shared information between an- imals performing similar tasks (Safaie et al., 2023; Zhang et al., 2024a). To exploit this shared information, ongoing work has focused on developing large-scale models that can be trained on neural data and behavior from many animals and brain regions (Pandarinath et al., 2018; Lurz et al., 2020; Azabou et al., 2024; Ye al., 2024; Zhang al., 2024a;b; Vermani al., 2024; Wang al., 2025). These works draw inspiration from the success of large-scale foundation mod- els in natural language processing (Radford et al., 2015 8; 2019; Achiam et al., 2023) and the natural sciences (Jumper et al., 2021; Abramson et al., 2024), which demonstrate remarkable generalization abilitics driven by broad pretrain- ing. Notable examples of large-scale models for neural analysis include POYO+ (Azabou al., 2025), a multi-task. multi-animall decoding approach based on POYO (Azabou et al., 2024) and NDT2, a masked modeling approach for
A demonstration of emergent properties: pretraining NEDS across many animals produces embeddings that accurately predict the brain regions in each session.
2. Related Work
Neural encoding and decoding models. Neural decoding and encoding are complimentary analyses in systems neu- roscience. Decoding quantifies what information is present in the neural activity while encoding quantifies how neu- ral activity represents this information (Paninski & Cun- ningham, 2018). Traditional decoding models include lin- ear regression, reduced rank regression, or more recently, deep learning approaches (Glaser et al., 2020). Improve- ments in decoding methodologies have significant potential for real-world applications, such as brain- caan--comuter inter- faces, which have seen remarkable progress over the past
2

---

Neural Encoding and Decoding at Scale
neural prediction that can be fine-tuned for decoding (Ye et al., 2024). Critically, both these models are task-specific: POYO+ is restricted to decoding and NDT2 can be fine- tuned for decoding but not encoding. To improve the scien- tific utility of large-scale neural models, we argue that they must be capable of performing both encoding and decoding.
veloping large-scale generalist models (Tay et al., 2022;
Zhang al., 2024b). We extend this paradigm to model the relationship between neural activity and behavior. Con- cretely, let us denote X as neural activity and Y as set of behaviors the animal is performing. We define a set of masking schemes as follows (shown in Figure IA):
Neural masking. The neural activity X is fully masked, and the model learns to predict it based on behavior Y. This masking scheme trains the model to predict the conditional expectation E[[ YY, effec- tively teaching it to perform neural encoding. Impor- tantly, we alternate between predicting neural activity using all behavioral variables and using one behavioral variable at a time. This approach allows the model to perform encoding with respect to each behavioral variable during inference, enabling the ranking of each behavior's 's importance in explaining neural variability,
3. Methods
In this section, we present a multimodal, multi-task model for Neural Encoding and Decoding at Scale (NEDS). The key innovation of our approach lies in the use of multi- task-masking in which NEDS alternates between neural, behavioral, within-modaie ing training (Figure IA). This approach enables NEDS to learn the conditional expectations between neural activity and behavior, unifying neural encoding and decoding within a single framework. We implement NEDS as a multimodal transformer in which each modality is tokenized indepen- dently and processed through a shared transformer (Figure IB). NEDS enables scalable and flexible analysis of neural activity and behavior from massive, multi-animal datasets.
and cross- cross-modal masking dur-
Behavior masking. The behavior Y is fully masked, and the model learns to predict it based on neural ac- tivity X. This masking scheme trains the model to predict the conditional expectation E[Y | X]. effec- tively teaching it to perform neural decoding.
3.1. Dataset
Within-modalle random masking. Random to- kens are masked and then reconstructed from either the neural activity X or behavior Y. This masking scheme, which models the conditional expectations [ Xummasked] and IE[Y masked Yunnusked], im- proves the model's s ability to learn modality modality-specific specific representations by capturing intra-modal dependencies.
For all analyses in this paper, we use the IBI. repeated site dataset (IBL et al., 2022). This dataset consists of Neuropix- els recordings collected from 10 labs with standardized experimental pipelines. The recordings target the same five brain regions across 83 adult mice performing a complex decision-making task. We train on data from up to 73 ani- mals, holding out 10 animals for evaluation. One animal in the training set has two insertions, resulting a total of 74 training sessions. For the neural data, we use trial-aligned, spike-sorted data (Chapuis et al., 2022), excluding neurons with firing rates less than 2 Hz. We utilize total of 27,380 neurons from 225 brain regions We bin the neural activity using 20ms windows and fix the trial-length to 2 seconds (100 time bins). For the behavioral data, we use four task variables: whisker motion, wheel speed, choice (left/right), and the "block" prior (Findling et al., 2023). The block is the prior probability of the stimulus appearing on the left or right side: (1) 20/80% right, (2) 80/20% left, or (3) 50/50%. Whisker motion and wheel velocity are binned at 20ms res- olution. We exclude trials in which mice exhibited reaction time outliers, as defined by the IBI. Brain-Wide Map (IBL. et al., 2023). We also evaluate our approach on a monkey reaching dataset (Pei et al., 2021) detailed in Appendix H.
Cross-modal random masking. Random tokens are masked and then reconstructed from both the neural ac- tivity X and behavior Y. This masking scheme, which models the conditional expectation [[Xmaskocd Y Ymasked I X Xummasked, Yunmaskedd) improves cross- cross-modal learning by encouraging the model to learn the joint relationship between the modalities.
Each of these masking schemes encourages the model to capture a diffcrent aspect of the relationship between neural activity and behavior. By mixing between these masking schemes during training, NEDS is be able to seamlessly switch between encoding and decoding at inference, achiev- ing state-of-the-arr performance in both tasks. We ablate these masking schemes in Appendix B and find that using all masking schemes together yields the best performance.
3.2. Multi-task-masking for neural activity and behavior
3.3. Architecture
Previous work has demonstrated that multi-task-maste masking, which involves alternating between different masking schemes during pretraining, is powerful approach for de-
Modality-specifie tokenization. We map both neural and behavioral data to token sequences using modallty-speeciff tokenizers (i.e., línear embeddings). For the neural data,
3

---

Neural Encoding and Decoding at Scale
A
Condilioning
B
Tokenization
Spike
Continuous Behavior
Discrete Behavior
pl | Behavior
175
125
Tokeeizer
Tokenizer
Tokenizer
P( Behaviar Neural)
75
Taepprt broodding
Nodaly
Neural Masking
Ereadding
0.5
Titte [5]
1.5
2.0
Decson Brisdding
NEDS
Behavior Masking
Mededd
Continanua
Transformer
Belawior
Encoder
2.0
Within-Modality Random Masking
NEDS
Modal Random Masking
Betasior
Discrete
Left
Loft
Lott
Left
1
Decoder
NEDS
Figure 1. Schematic illustration of NEDS. (A) Neural encoding and decoding can be interpreted as modeling the conditional probability distributions between neural activity and behavior (Schulz et al., 2025). In NEDS, we utilize a multi-task- masking approach (Tay al., 2022; Zhang al., 2024a) to model the conditional expectations of these distributions as well as to encourage cross-modal and within-modality representation learning. This is achieved by alternating between neural, behavioral, within- modality, and cross- cross-modal masking during training. (B) We implement NEDS using multimodal cansformer-basod architecture. We utilize modality speciifcc tokenizers that convert spike counts and continuous behaviors into 20ms temporal tokens and discrete behaviors into sequences of repeated tokens, aligning with the temporal resolution of the continuous data. We then add temporal, modality, and session embeddings to the tokens. We train NEDS by masking out tokens according to the masking schemes from (A) and then predicting them with modality-speciifee decoders. Our multimodal architecture builds on work from other domains (He et al., 2022; Mizrahi et al., 2023; Fang et al., 2024).
we tokenize each 20ms time bin, generating a sequence of T' tokens per trial. For the continuous behavioral data, we again utilize temporal tokenization yielding sequences of T' tokens. For discrete behavioral data, we replicate each value T times to align with the temporal resolution of the continuous data and convert it into a sequence of T tokens (shown in Figure 1B). To differentiate between modalities, we add learnable modality-specific embeddings to each to- ken (ablated in Appendix G). We also add rotary position embeddings (RoPE) (Su et al., 2024; Azabou et al., 2024) to each token to help the model distinguish data from different timesteps. Tokens from all modalities are concatenated into a single unified sequence, which serves as input to the shared transformer encoder. Tokenizing each modality separately provides flexibility for incorporating additional modalities and cases where modalities are missing.
Modality-specfiic decoders. We use separate linear decoders to reconstruct each modality. This is done by first splitting the transformer encoder output by modality. Then, each modality subset is transformed using its corresponding de- coder to reconstruct the original data. This design enables flexible decoding across modalities and allows the model to function effectively even when some modalities are missing, making it suitable for diverse and heterogencous datasets.
Session-specifie adaptation. To enable training across multi- ple sessions, use session-specific input matrices to trans- form neural activity and behavior into fixed-dimensional embeddings (Pandarinath et al., 2018). We also add ses- sion embeddings to each token to help the model account for session-specific variations (Azabou et al., 2024; Zhang et al., 2024a). For decoding, we use session- and modality- specific decoders for each animal.
Multimodal transformer encoder. To process the multi- modal sequence of tokens, we employ an encoder-only transformer architecture composed of standard transformer blocks with pre-normalization and feed-forward layers. Cen- tral to this architecture is the self-attention mechanism (Vaswani, 2017), which computes pairwise interactions be- tween all tokens in the input sequence. This mechanism enables the model to capture dependencies within modali- ties, across modalities, and across time steps.
3.4. Generative process
To train NEDS, we randomly sample masking scheme Mj to apply each sequence in the batch. We transform neural and behavioral data into a fixed-dimensional space using Wx inpur E NxD and Wy inpur Wnnuur E BxD, where N and B are the dimensions of neural activity and behavior, respectively. We then tokenize the neural data and behav-
4

---

Neural Encoding and Decoding at Scale
ioral data using modality-ppeciffe tokenizers (see Section
metric. For decoding wheel speed and motion energy, we quantify performance using single-trial R², which measures the proportion of variance explained after accounting for the trial average.
3.3). After tokenization, apply the sampled masking scheme by replacing selected tokens with a learned mask token. We then add positional embeddings (PE) to encode three types of structure: modality (PFmodalit). which differ- entiates neural and behavioral tokens; temporal ordering (PErempora)), which captures the sequence position of each to- ken within session; and session identity (PEsession). which provides session-specific context. These embeddings are summed with the tokenized inputs before being processed by the transformer, which outputs representations for neural activity (ex) and behavior (ey). The training is as follows; M₁ [[[eural, behavior, within-modie cross-modal)
4.2. Baselines
We compare NEDS against number of linear and non- linear baselines. For our linear baselines, we compare to a standard linear regression (used in IBL. et al. (2023)) and a multi-session reduced-rant neural encoding (Posani et al., 2024) and decoding (Zhang et al., 2024a) algorithm. Both the linear and reduced-rank models use neural activity (or behavior) across all timesteps predict behavior (or neural activity) at specific timestep. For our non-linear base- lines, we compare NEDS to two recent transformer- -bassd methods: POYO+ (Azabou et al., 2025) and NDT2 (Ye et al., 2024). POYO+ is a multi-session, multi-task neu- ral decoding algorithm based on the original POYO model (Azabou et al., 2024). NDT2 a multi-session, masked modeling approach for neural self-prediction that can be fine-tuned for decoding. These algorithms represent the current state-of-the-ard in neural decoding of spiking activ- ity. Importantly, each model in our paper has different pretraining objective: POYO+ performs multi-task neural decoding of choice, block, wheel, and motion energy across all animals, NDT2 performs self-supervised prediction of neural activity across all animals, and NEDS performs mul- timodal, multi-task-maa mask-masking across all animals (described in Section 3.2). We utilize re-implemented version of NDT2 for all our analyses.
Zx = Tokenizer Wx input (X))
Zy Tokenizer(W: imput (Y))
Z = M の [Zx, Z
Zpss = Z + memoaliity + Peremporal + PEsession
ex, EY = Transformer( 4pos)
Doisson(W*ahn (ex))
MSE(W continuous MSS((Ccominuoux
(EY))
Cross-Entropy
(W (W didierett (ex))
(1) where @ denotes pointwise multiplication, Wrate E R RDN,, and Wroomimoous and W discrrett map the outputs of the trans- former to the continuous and discrete behaviors, respec- tively. We assume that X modeled by a Poisson emission model with time-varying rates. For continuous behaviors, we assume Y is modeled by Gaussian with fixed variance, corresponding minimizing the MSE loss. For discrete behaviors, we assume Y is modeled by categorical distri- bution, corresponding to minimizing a cross-entropy loss.
5. Experiments
4. Evaluation
For all experiments, we evaluate the performance of each model on 10 held-out animals. For these 10 animals, we split the trials into training (70%), validation (10%), and test (20%) sets. The metrics (detailed in Section 4. are computed on the test trials for each heldout animal.
4.1. Tasks
We evaluate the ability of NEDS to model the relationship between neural activity and behavior using two supervised prediction tasks:
5.1. Single-session
1. Neural encoding: Predicting spiking activity from the behavioral and task variables, including choice (left/right), "block" prior, wheel speed, and whisker motion energy. For this task, we use all task variables collectively and individually, allowing us rank each variable's 's contribution to driving neural responses. We utilize the co- co-bps metric (Pillow et al., 2008; Pei et al., 2021) which measures neural reconstruction perfor- mance using bits per spike.
We first evaluate NEDS on single-session neural encoding and decoding. For this experiment, we train NEDS on the training trials from one heldout animal and then evaluate its performance on the test trials for the same animal. We com- pare NEDS to the linear baselines and a unimodal version of NEDS without multimodal, multi-task training (using the same transformer architecture). We conducted extensive hy- perparameter tuning by initializing 50 random models with hyperparameters randomly selected from predefined ranges, The model with the best hyperparameters was chosen based on its validation set performance (see Appendix C for the hyperparameter ranges used in this experiment).
2. Neural decoding: Predicting behavioral and task vari- ables from spiking activity. For choice and block pre- diction, we use classification accuracy as the evaluation
5

---

Neural Encoding and Decoding at Scale
Spike Encading
Wheel Decoding
Whisker Decading
Choic Decoding
Block Decoding
2.4
1.0
0.7-
0.9-
0.6
0.6
0.5-
0.8-
R-dequazeed
Aecorary
0.5
0.2-
0.4
0.7
0.3-
0.3-
D.c-
3.0
0.
0.43
0.50
0.50
0.06
n.1
[0.20 0.39
0.58
0.50
0.5
10.00
0.84
0.82
0.03
0.01 0.5
0,54
0.04 0.03 o.po
Ainwac
Glegle-
Iinssc
0od-
Single-
Lizene
Single
Lineaz XBR
Una-
Singie-
Linsez
Aingle- Mali-
NESE
suna
MESS
Haza
HESE
HEDS
SEDS
Spike Encoding
Wheel Decoding
Whisker Decodinn
Choice Deooding
Block Decoding
0,90
0.30
0.65
0,60
0.85
0.75
24%1
0.50
7%
0.45
11 %
%%
4%
0.15
0.30
0.5C
0.65
0.45
0.60
0.70
0.85
0.75
0.90
stingee-Sesele
Single-Beesll
Stigle-Session
Single-Session
single-Sessicn
Ground Truth
Single Session NEDS Mecinaa
Multi Session NEDS
Ground Truth Seeal
Single Session NEDS
Multi Session NEDS Tetal
Tetal
Netmiiied Fixing Nate
Wheel speed
AM
2.0
2.0
Time (s)
2.0
2.0
2.0
2.0
Thmee Ost
0
Ground Truth
byye 50 Single-eessinn NEDS
Pyps Multi-Session
Whieler F
Sa
Tria1 Index
Nortion Kowsgy
Nww
15)
2.0
Time (5)
2.0
Time (c)
2.0
20 0
2.0
Time
2.0 5
25
Figure 2. Quantitarive and qualittative evaluation of single-session and multi-session NEDS. (A) We evaluate multi-session NEDS and single-session NEDS models against our linear baselines and the session, unimodal variant of NEDS. Our results show that multi-session NEDS consistently outperforms all baselines across all tasks, while single-session NEDS outperforms all baselines except block decoding. These findings demonstrate the advantages of multimodal training and cross-animal pretraining for neural encoding and decoding. Among the baseline models, RRR has the fewest parameters (1,000 20,000 on average). Linear models contain approximately 40,000 70,000 parameters on average. Both the hinlle-eessinn unimodal and multimodal NEDS share the same transformer encoder size (~> million parameters). The multi-session NEDS is the largest model with 12 million parameters in its transformer encoder. (B) A scatterplot comparison of multi-session NEDS pretrained on 74 sessions vs. single-session NEDS. Fach dot corresponds to an individual session. The green value in the bottom right of each subplot displays the relative improvement of the 74-session NEDS over single-session NEDS. (C) A comparison of the predicted avval-aeer firing rates for sinllesession and multi-session NEDS against the ground truth trial-averaged spike counts for selected neurons. Predictions from multi-session NEDS more closely matches the ground truth. (D) Each compares single-session and multi-session NEDS predictions of single-trial variability for neuron against the ground truth. Singlc-trial variability is obtained by subtracting the neuron's 's peristimulus time histogram (PSTH) from its activity in cach trial. Only selected trials are shown for visualization purposes. (E, F) The predicted wheel speed and whisker motion energy from both the single-session and multi-session NEDS are shown alongside ground truth behaviors for each trial.
5.2. Multi-session
such as session embeddings, input matrices, and linear de- coders. We perform multi-task decoding for NEDS and POYO+ and single-task decoding for NDT2 as it does not support multi-task decoding.
In our multi-session experiments, we evaluate NEDS, POYO+, and NDT2 on the 10 held-out animals after pre- training on trials from 73 animals. The goal of this ex- periment is to assess each model's ability to leverage in- formation from multiple animals to improve generalization to unseen animals. After pretraining, we fine-tune each approach on the training trials from 10 held-out animals, allowing adaptation of session- specific model components
Hyperparameter searches become prohibitively expensive in terms of cost and compute when pretraining across multiple animals. To address this, we perform hyperparameter tuning for all methods on a subset of 10 animals. Specifically, we initialize 50 models with hyperparameters randomly sam-
6

---

Neural Encoding and Decoding at Scale
pled from a predefined range and select the best model on the validation performance. Once identify the optimal architecture, we scale it to 73 animals by increasing model capacity while keeping all other hyperparameters fixed. Dur- ing fine-tuning, we fix the pretrained model architecture and initialize 30 random models to optimize session- -ppciiiic hyperparameters, including learning rate, weight decay, and mask ratio (see Appendix for more details). Our approach attempts to balance practical compute challenges with rigor- ous fine-tuning (see Section 7 for additional discussion).
All
Choice
Block
Wheel
Whisker
0.27 + 0.03
0.06 + 0.02 0.10 + 0.02 0.12 0.02
0.11 0.02
Table I. Ranking the importance of task variables for driving new- ral activity. We use multi-session NEDS to rank the importance of each variable for neural encoding. We measure encoding perfor- mance in bits per spike (bps), where higher values indicate better performance. We show the average and standard error of the en- coding performance for each task variable across the 10 animals.
importantly, these results demonstrate that our multi-task masking strategy enables NEDS to accurately perform both neural encoding and decoding after training.
5.3. Brain region classification with neuron embeddings
Although NEDS is trained to perform neural encoding and decoding, we want to understand what additional informa- tion about the neurons is captured by the latent embeddings of our model (e.g., brain region information). There are two matrices in NEDS which store neuron-specific information: the input matrix for neural activity (Wxinpu and the out- put matrix for neural activity (Wrille) from Equation For each neuron k we can extract a neuron-specific embedding by taking the corresponding row l in each matrix and con- catenating the rows together. We refer to the concatenated embeddings for a neuron k as C₁.
6.2. Multi-session
In our multi-session experiment, we find that NEDS bene- fits significantly from pretraining on 74 sessions, yielding substantial improvements in both encoding and decoding (Figure 2A) and achieving the best performance across all baselines for all tasks. Figure 2B shows a scatter plot com- paring the performance of single-session and multi-session NEDS across each task the 10 held-out sessions. We find that multi-session NEDS consistently outperforms single- session NEDS across all tasks, with decoding improving by 4-1 1 % and encoding improving by 24%. Figure 2C-F shows the qualitative improvements of NEDS over single- session NEDS for both encoding and decoding.
We now want to evaluate how well the embedding Ck pre- diets the brain region where neuron i is located. To test this, we train linear support vector classifier on the neuron embeddings obtained from the 10 held-out animals after fine-tuning Our analysis focuses five main brain re- gions: the posterior thalamus (PO), lateral posterior nucleus (LP), dentate gyrus (DG), hippocampal CA1 (CA1), and anteromedial visual area (VISa). VISa encompasses several sub-regions which we treat as a single region for this anal- ysis. We evaluate brain region classification using f-fold cross-validation across all neurons. To understand the im- portance of multimodal, multi-session training, compare the neuron embeddings from our pretrained NEDS to single- session unimodal and multimodal versions of NEDS. For the unimodal versions of NEDS, we extract neuron embed- dings by taking the corresponding row in Write for encoding and Wx input for decoding (i.e., no concatenation..
In Figure 3. we compare NEDS to two sfate-of-the-art large- scale models for neural decoding: POYO+ and NDT2. After pretraining each approach across 74 sessions and fine-tuning on 10 heldout sessions. we find that NEDS outperforms NDT2 by 11-37% and POYO+ by 1-13% across all decod- ing tasks. Importantly, NEDS is also able to perform neural encoding unlike these methods.
We also demonstrate that multi-session NEDS can be used to quantify how much each task variable drives neural ac- tivity. Table 1 shows a comparison of how well NEDS can predict neural activity from all task variables as well as from each task variable individually. We observe that the movement-related variables, such as wheel speed and whisker motion energy, explain larger proportion of neural variance than cognition-related variables like choice and block. This is consistent with the findings from IBL et al. (2022). An interesting finding is that encoding performance for all task variables combined is significantly higher than for any individual task variable, suggesting that the task variables capture distinct information about neural activity.
6. Results
6.1. Single-sessioon
In our single-session experiment, we find that NEDS outper- forms the linear baselines in neural encoding and decoding of both continuous and discrete behaviors, with the excep- tion of the block prior (shown in Figure 2A). Additionally, multmooaa NEDS surpasses its unimodal counterpart on all tasks except block decoding, improving encoding by 5% and decoding by 2-7%, highlighting the advantages of jointly modeling both modalities during training. Most
6.3. Brain region classification with neuron embeddings
Figure presents an analysis of neuron embeddings ex- tracted from NEDS, following a similar evaluation pipeline used for POYO+ embeddings in Azabou et al. (2025). De-
7

---

Neural Encoding and Decoding at Scale
Wheel
Whisker
Choice
Block
0.70
0.95
0.65
0.80
0.33
0.50
0.80
13%
12%
1%
7%
0.4
0.6
0.45
Multi-Session 20X0+
Muiti-Sessaae FO10+
Molti-Session 2OX04
Multi-9esstee 20Y0+
0.65
0.85
0.45
0.75
37%
29%1
11%1
12%
0.50
0.65
0.45
0.60
0.70
0:85
0.75
0.90
Moiti-Bessicn ND12
Hulti-Besssin NDT2
Multi-Bession NDT2
Mul Molti-10 Multi-Bession B0T2
Figure 3. Comparing NEDS POYO+ and NDT2. We compare multi-session NEDS POYO+ and NDT2 after pretraining 74 sessions, evaluating all models on neural decoding tasks across 10 held-on out sessions. We measure the performance of choice and block decoding with accuracy and the wheel speed and whisker motion energy using single-trial R Each dot corresponds to individual session. The green value in the bottom right each subplot displays the relative improvement of NEDS over POYO+ and NDT2.
A
B
C
Brain Region Classification Accuracy
80.4%
PO
0.85
0.10
0.01
0.01
LP
0.07
0.79
0.06
0.04
0.04
50.0%
45.0%
0.00
0.11
True
CA1
0.00
0.06
0.88
Bra Region
FO
LP
VISa
0.03
0.05
0.02
0.04
0.87
DG
CA2
Dnimodal
Unimodal
Single-
Mallii-
PO
CA1
ViSa
Escoding
Decoding
Bessicn NEDS Sessico NEDS
Prodicted Label
Figure 4. Brain region classification with neuron embeddings from NEDS. (A) UMAP projection of NEDS neuron emheddings (detailed Section 5.3), color-coded by distinct brain regions. (B) Classification accuracy of brain regions using neuron embeddings obtained from single-session unimodal, multimodal NEDs, and multi-session, mulit-modi modal NEDS. (C) Confusion matrix showing the brain region classification performance of neuron embeddings from multi-session NEDS.
spite receiving no explicit brain region information during training, the embeddings of multi-session NEDS exhibit striking pattern: different brain regions show clear separabil- ity (Figure 4A). We compare the brain region classification performance of neuron embeddings extracted from single- session unimodal NEDS, single-session multimodal NEDS, and multi-session NEDS in Figure 4B. We find that the classifier achieves higher accuracy with neuron embeddings from multimodal NEDS than from unimodal NEDS, Pre- training across 74 sessions further improves performance, with multi-session NEDS achieving the highest classifica- tion accuracy of all the methods (83%). In Figure 4C, we show the brain region classification confusion matrix for multi-session NEDS. Analyzing the classifier's errors re- veals potential similarities between specific brain regions, such as PO and LP. These regions, both located in the pos-
terior thalamus, are known to play key roles multimodal sensory integration (Allen al., 2016). Surprisingly, we also find similarities between LP and DG which suggests that these regions might have subset of neurons with simi- lar functionality. Future work could further investigate the nature of these similarities and their implications for multi- modal sensory processing and memory-related functions.
7. Discussion
In this work, we introduce an approach for Neural Encoding and Decoding Scale (NEDS). Our method leverages culti-task-mastee alternating between neural, behavioral, within-modality, and cross-modal masking during training. This approach enables NEDS to accurately predict behavior from neural activity (decoding) and neural activity from behavior (encoding) after training. We demonstrate that pre-
8

---

Neural Encoding and Decoding at Scale
training NEDS across 73 animals improves both encoding and decoding performance heldout animals, surpassing state-of-the-art methods such as POYO+ and NDT2 on the IBL repeated site dataset. Finally, we demonstrate that the learned neuron embeddings of NEDS display emergent properties: they are highly predictive of the brain regions in each recording without explicit training.
ethically, future extensions human data will require care- ful consideration of privacy, consent, and responsible use. Overall, our work contributes to building foundation models that could drive innovations in brain-computer interfaces and neurological therapies.
References
This work has several limitations. First, we rely exclu- sively on trial-aligned data for pretraining and evaluation of all models. While this approach is standard in systems neuroscience, it limits the amount of data available for pre- training. In contrast, Azabou et al. (2024) pretrained POYO on unaligned data, enabling the use of much larger datasets. Pretraining NEDS with unaligned data could enhance its generalizability and an exciting direction for future work. Second, due to computational constraints, were unable to conduct extensive hyperparameter tuning when pretraining models on data from all 73 animals. The development of standardized benchmarks, such as the FALCON benchmark introduced by Karpowicz et al. (2024), will facilitate more rigorous and scalable model comparisons moving forward.
Abramson, J., Adler, J., Dunger, J., Evans, R., Green, T. Pritzel, A., Ronneberger, O., Willmore, L., Ballard, A. J., Bambrick, J., et al. Accurate structure prediction of biomolecular interactions with alphafold 3. Nature, pp. 1-3, 2024.
Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L.,, Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al. Gpt-4 technical report. arXiv preprint arXiv:2303.0877 087744 2023.
Allen, A. E., Procyk, C. A., Howarth, M., Walmsley, L., and Brown, T. M. Visual input to the mouse lateral posterior and posterior thalamic nuclei: photoreceptive origins and retinotopic order. The Journal of Physiology, 594(7): 1911-1929, 2016
Looking ahead, extending the training paradigm introduced in NEDS to incorporate additional modalities, such as local field potentials and electrophysiological features, presents promising direction. Recent studies have shown that com- bining a neuron's 's activity with its electrophysiologccal prop- erties enables accurate prediction its cell-type (Beau et al., 2025; Lee al., 2024; Yu al., 2025). Incorporating these additional features into NEDS could enhance its ability to identify and predict cell-type specific functionality. Overall, NEDS introduces flexible and scalable paradigm for mod- eling the relationship between neural activity and behavior.
Azabou, M., Arora, V., Ganesh, V., Mao, X., Nachimuthu, S., Mendelson, M., Richards, B., Perich, M., Lajoie, G., and Dyer, E. A unified, scalable framework for neural population decoding. Advances in Neural Information Processing Systems, 36, 2024.
Azabou, M., Pan, K. X., Arora, V., Knight, I. J., Dyer, E. L., and Richards, B. A. Multi-session, multi-task neural decoding from distinct cell-types and brain regions. In The Thirteenth International Conference on Learning Representations, 2025. URL https:t//tpeenttee //tp:r////ppeevviee. net/ /
Acknowledgments
This project was supported by the Wellcome Trust (209558 and 216324), National Institutes of Health (1U19NS123716), the Simons Foundation, the National Science Foundation (NSF award CIF:RI:2212182 NSF CA- REER award IIS-214077) and by DoD OUSD (R&E) under Cooperative Agreement PHY-2229929 (The NSF AI Institute for Artificial and Natural Intelligence), as well as generous gifts from the Alfred Sloan Foundation, the McK- night Foundation, and the CIFAR Azrieli Global Scholars Program.
Beau, M., Herzfeld, D. J., Naveros, F., Hemelt, M. E., Agostino, F., Oostland, M., Sánchez-López,, A., Chung, Y. Y., Maibach, M., Stabb, H. N., et al. A deep- learning strategy to identify cell types across species from high-density extracellular recordings. Cell, 2025. doi: 10. 10.1016/j wwt.///.ell2222.01...
Biderman, D., Whiteway, M. R., Hurwitz, C., Greenspan, N., Lee, R. S., Vishnubhotla, A., Warren, R., Pedraja, F. Noone, D.,, Schartner, M. M., et al. Lightning pose: improved animal pose estimation via semi-supervised learning, bayesian ensembling and cloud-native open- source tools. Nature Methods, 21(7):1316 2024.
Impact Statement
This work advances scalable, multimodal models that im- prove our understanding of the bidirectional relationship between neural activity and behavior, with potential appli- cations in neuroscience and neurotechnology. While our approach relies publicly available animal data collected
Biewald, L. Experiment tracking with weights and biases, 2020. URL https: // //www/wwwww/////o/ wandb. com/.
Chapuis, G., Faulkner, M., Harris, K. D., Huntenburg, J. M., Hurwitz, C., Lee, H. D., Paninski, L.., Rossant, C., Roth,

---

Neural Encoding and Decoding at Scale
N., Steinmetz, N. A., al. Spike sorting pipeline for the international brain laboratory. 2022.
IBL, Benson, B., Benson, J., Birman, D., Bonacchi, N., Carandini, M., Catarino, J. A., Chapuis, G. A., Church- land, A. K., Dan, Y., et al. A brain-wide map of neural activity during complex behaviour. bioRxiv, pp. 2023-07, 2023.
Chen, S., Liu, Y., Wang, Z. A., Colonell, J., Liu, L. D., Hou, H., Tien, N.-W., Wang, T., Harris, T., Druckmann, S., et al. Brain-wide neural activity underlying memory- guided movement. Cell. 187(3):676-691, 2024.
Jumper. J., Evans, R., Pritzel, A., Green, T., Figurnov, M., Ronneberger, O., Tunyasuvunakool, K., Bates, R., Zidek, A., Potapenko, A., et Highly accurate protein structure prediction with alphafold. Nature, 596(7873):533-589 2021.
Fang, C., Sandino, C.,, Mahasseni, B., Minxha, J., Pouransari, H., Azemi, E., Moin, A., and Zippi, E. Pro- moting cross- cross-modee modal representations to improve multi- modal foundation models for physiological signals. arXiv preprint arXiv:2410. 164.4 2024.
Jun, J. J., Steinmetz, N. A., Siegle, J. H., Denman, D. J., Bauza, M., Barbarits, B., Lee, A. K., Anastassiou, C. A., Andrei, A., Aydın, ç., et al. Fully integrated silicon probes for high-density recording of neural activity, Na- ture, 551 (7679)):23222 2017.
Findling, C., Hubert, F., Laboratory, I. B., Acerbi, L., Ben- son, B., Benson, J., Birman, D., Bonacchi, N., Bruijns, S., Carandini, M., et al. Brain-wide representations of prior information in mouse decision-making. BioRxiv, pp. 2023-07, 2023.
Karpowicz, B. M., Ye, J., Fan, C., Tostado-Marcos, P., Riz- zoglio, F., Washington, C., Scodeler, T., de Lucena, D., masoo-Tomaszzwski, R., Mender, M. J., et al. Few- shot algorithms for consistent neural decoding (falcon) benchmark. bioRxiv, pp. 2024 09, 2024.
Gilja, V., Pandarinath, C., Blabe, C. H., Nuyujukian, P., Simeral, D., Sarma, A. A., Sorice, B. L., Perge, A., Jarosiewicz, B., Hochberg, L. R., al. Clinical trans- lation of high-performance neural prosthesis. Nature medicine, 21(10): 142--115,, 2015.
Khilkevich, A., Lohse, M., Low, R., Orsolic, I., Bozic, T., Windmill, P., and Mrsic-Flogel, T. D. Brain-wide dynamics linking sensation to action during decision- making, Nature, 634(8035):890-900, 2024.
Glaser, I., Benjamin, A. S., Chowdhury, H., Perich, M. G., Miller, L. E., and Kording, K. P. Machine learning for neural decoding. eneuro, 7(4), 2020.
Lee, E. K., Gul, A., Heller, G., Lakunina, A.. Jaramillo, S., Przytycki, P., and Chandrasekaran, C. Physmap- interpretable vivo neuronal cell type identification us- ing multi-modas analysis of electrophysiological data. BioRxiv, pp. 2024-02, 2024.
Gondur, R., Sikandar, U. B., Schaffer, E., Aoi, M. C.,, and Keeley, S. L. Multi-modal gaussian process variational au- toencoders for neural and behavioral data. arXiv preprint arXiv:2310.03111, 2023.
Li, B. M., Cornacchia, I. M., Rochefort, N. L., and Onken, A. VIt: large-scale mouse v1 response prediction using a vision transformer. arXiv preprint arXiv:2302.03023, 2023.
He, K., Chen, X., Xie, S., Li, Y., Dollár, P., and Girshick, R. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 16000-16009, 2022.
Liaw, R., Liang, E., Nishihara, R., Moritz, P., Gonzalez, J. E., and Stoica, I. Tune: A research platform for dis- tributed model selection and training. arXiv preprint arXiv: DrXiv:/807. /000 055 2018
Hurwitz, C., Kudryashova, N., Onken, A., and Hennig, M. H. Building population models for large-scale neural recordings: Opportunities and pitfalls. Current opinion in neurobiology. 70:64-73. 2021a.
Lurz, K.-K., Bashiri, M., Willeke, K., Jagadish, A. K., Wang, E., Walker, Y., Cadena, A., Muhammad, T., Cobos, E., Tolias, A. S., al. Generalization in data-driven models of primary visual cortex. BioRxiv, pp. 2020-10. 2020.
Hurwitz, C., Srivastava, A., Xu, K., Jude, J., Perich, M., Miller, L., and Hennig, M. Targeted neural dynamical modeling. Advances in Neural Information Processing Systems, 34:29379-29392. 2021b.
IBL, Banga, K. Benson, J., Bonacchi, N., Bruijns, S. A., Campbell, R., Chapuis, G. A., Churchland, A. K., Dava- tolhagh, M. F., Lee, H. D., et al. Reproducibility of in- vivo sloctrophy measurements in mice. bioRxiv, pp. 2022-05, 2022.
Mathis, A., Mamidanna, P., Cury, K. M., Abe, T., Murthy, V. N., Mathis, M. W., and Bethge, M. Deeplabcut: mark- erless pose estimation of user-defined body parts with deep learning. Nature neuroscience. 21(9):1281-1288 2018.
10

---

Neural Encoding and Decoding at Scale
Metzger, S. L., Litrlejonn, K. T., Silva, A. B., Moses, D. A., Seaton, M. P., Wang, R., Dougherty, M. E., Liu, J. R., Wu, P., Berger, M. A., al. A high-performance neuropros- thesis for speech decoding and avatar control. Nature, 620(7976): 2023.
Pillow, J. W., Shlens, J., Paninski, L.,, Sher, A., Litke, A. M., Chichilnisky, E., and Simoncelli, E. P. Spatio-temporal correlations and visual signalling in complete neuronal population. Nature, 454(7207):995- 99999999),99999999 2008.
Posani, L., Wang, S., Muscinelli, S., Paninski, L., and Fusi, S. Rarely categorical, always sigh-dimensional: how the neural code changes along the cortical hierarchy. bioRxiv, pp. 2024-11, 2024.
Mizrahi, D., Bachmann, R., Kar, O., Yeo, T., Gao, M., De- hghan, A., and Zamir, A. 4m: Massively multimodal masked modeling. Advances in Neural Information Pro- cessing Systems, 36:58363-58408, 2023.
Radford, A., Narasimhan, K., Salimans, T., Sutskever, I., et al. Improving language understanding by generative pre-training. 2018.
Musall, S., Kaufman, M. T., Juavinett, A. L.,, Gluf, S., and Churchland, A. K. Single-trial neural dynamics are domi- nated by richly varied movements. Nature neuroscience, 22(10):1677-1666, 2019.
Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I., al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
Pandarinath, C., Nuyujukian, P., Blabe, C. H., Sorice, B. L., Saab, J., Willett, F. R., Hochberg, L. R., Shenoy, K. V., and Henderson, J. M. High performance communica- tion by people with paralysis using an intracortical brain- computer interface. elife, 18548, 2017.
Safaie, M., Chang, J. C., Park, J., Miller, L. E., Dudman, J. T., Perich, M. G., and Gallego, A. Preserved neural dynamics across animals performing similar behaviour. Nature, 623(7988) 2023
Sani, O. G., Abbaspourazad, H., Wong, Y. Pesaran, B., and Shanechi, M. M. Modeling behaviorally relevant neural dynamics enabled by preferential subspace identi- fication. Nature Neuroscience, 24(1):140-149) 2021.
Pandarinath, C., O'Shea, J., Collins, J., Jozefowicz, R., Stavisky, S. D., Kao, J. C., Trautmann, E. M., Kaufman, M. T., Ryu, S. I., Hochberg, L. R., al. Inferring single- trial neural population dynamics using sequential auto- encoders. Nature methods,
2018.
Sani, O. G., Pesaran, B., and Shanechi. M. M. Dissociative and prioritized modeling of behaviorally relevant neu- ral dynamics using recurrent neural networks. Nature neuroscience, 27(10):003322453 2024.
Paninski, L. Maximum likelihood estimation of cascade point-process neural encoding models. Network: Compu- tation in Neural Systems, 15(4)2443,200 2004.
Schrimpf, M., Kubilius, J., Hong, H., Majaj, N. J., Rajaling- ham, R., Issa, E. B., Kar, K., Bashivan, P., Prescott-Roy, J., Geiger, F., et al. Brain-score: Which artificial neu- ral network for object recognition is most brain-like? BioRxiv, pp. 407007, 2018
Paninski, L. and Cunningham, J. P. Neural data science: ac- celerating the experimnn--nallysistthoon cycle in large- scale neuroscience. Current opinion in neurobiology, 50: 232-241, 2018.
Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V.,
Schulz, A., Vetter, J., Gao, R., Morales, D., Lobato-Rios, V., Ramdya, P., Gonçalves, P. J., and Macke, J. H. Modeling conditional distributions of neural and behavioral data with masked variational autoencoders. Cell Reports, 44 (3), 2025.
Thirion, B., Grisel, O., Blondel, M., Prettenhofer, P., Weiss, R., Dubourg, V., Vanderplas, J., Passos, A., Cour- napeau, D., Brucher, M., Perrot, M., and Duchesnay, E. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825-288 2830, 2011.
Steinmetz, A., Zatka-Haas, P., Carandini, M., and Harris, K. D. Distributed coding of choice, action and engage- ment across the mouse brain. Nature, Nature,,56(77))) 576(778)):666 273, 2019.
Pei, F,, Ye, J., Zoltowski, D., WU, A., Chowdhury, R. H., Sohn, H., O' O'Doherty, J. E., Shenoy, K. V., Kaufman, M. T., Churchland, M., et al. Neural latents bench- 21: evaluating latent variable models of neural population activity. arXiv preprint 3XXv:2109.04463. 2021.
Steinmetz, N. A., Aydin, C., Lebedeva, A., Okun, M., Pa- chitariu, M., Bauza, M., Beau, M., Bhagat, J., Böhm, C., Broux, M., et al. Neuropixels 2.0: A miniaturized high-density probe for stable, long-term brain recordings. Science, 372(6539) 2021.
Pereira, T. D., Tabris, N., Matsliah, A., Turner, D. M., Li, J., Ravindranath, S., Papadoyannis, E. S., Normand, E., Deutsch, D. S., Wang, Z. Y., et al. Sleap: A deep learning system for multi-animal pose tracking. Nature methods, 19(4):4864499 2022.
Stringer, C. and Pachitariu, M. Analysis methods for large- scale neuronal recordings. Science, 38666722) 2024.

---

Neural Encoding and Decoding at Scale
Su, J., Ahmed, M., Lu, Y., Pan, S., Bo, W., and Liu, Y. Roformer: Enhanced transformer with rotary position embedding, Neurocomputing, 167013, 2024.
Ye, J., Collinger, J., Wehbe, L., and Gaunt, R. Neural data transformer 2: multi-context pretraining for neural spik- ing activity. Advances in Neural Information Processing Systems, 36, 2024.
Tay, Y., Dehghani, M., Tran, V. Q., Garcia, X., Wei, J., Wang, X., Chung, H. W., Shakeri, S., Bahri, D., Schuster, T., et al. U12: Unifying language learning paradigms. arXiv preprint arXiv:2205.05133 2022.
You, Y., Li, J., Reddi, S., Hseu, J., Kumar, S., Bhojanapalli, S., Song, X., Demmel, J., Keutzer, K., and Hsieh, C.-J. Large batch optimization for deep learning: Training bert in 76 minutes. arXiv preprint
2019.
Truccolo, W., Eden, U. T., Fellows, M. R., Donoghue, J. P., and Brown, E. N. A point process framework for relating neural spiking activity to spiking history, neural ensemble, and extrinsic covariate effects. Journal of neurophysiol- ogy, 93 2005.
Yu, H. Lyu, H., Xu, Y., Windolf, C., Lee, E. K., Yang, F. Shelton, A. M., Winter, O., Laboratory, I. B., Dyer, E. L., Chandrasckaran, C.,, Steinmetz, N. A., Paninski, L., and Hurwitz, C. L. In vivo cell-type and brain region classification via multimodal contrastive learning. In The Thirteenth International Conference on Learning Representations, 2025. URL https: hattps://openreret net fwrum?orummmmmm (atprrmmmdd-11111e11111
Turishcheva, P., Fahey, G., Vystrčilová, M., Hansel, L., Froebe, R., Ponder, K., Qiu, Y., Willeke, K. F., Bashiri, M., Wang, E., et al. The dynamic sensorium competition for predicting large-scale mouse visual cortex activity from videos. ArXiv, pp. arXiv-2305, 2024.
Zhang, Y., Lyu, H., Hurwitz, C., Wang, S., Findling, C., Hubert, F., Pouget, A., Laboratory, I. B., Varol, E., and Paninski, L. Exploiting correlations across trials and behavioral sessions improve neural decoding. bioRxiv, 2024a.
Ustyuzhaninov, I. Burg, M. F., Cadena, S. A., Fu, J., Muhammad, T., Ponder, K., Froudarakis, E., Ding, 7 Bethge, M., Tolias, A. S., et al. Digital twin reveals com- binatorial code of non-linear computations in the mouse primary visual cortex. BioRviv, pp. 2022-02, 2022.
Zhang, Y., Wang, Y., Benetó, D. J., Wang, Z., Azabou, M., Richards, B., Winter, O., Laboratory, T. I. B., Dyer. E., Paninski, L.., et al. Towards a "universal translator" for neural dynamics single-cell, single-spike resolution. ArXiv, pp. arXiv-2400 2024b.
Vaswani, Attention is all you need. Advances in Neural Information Processing Systems, 2017.
Vermani, A., Nassar, J.. Jeon, H., Dowling, M., and Park, I. M. Meta-dynamical state space models for integrative neural data analysis. arXiv preprint arX1v:2410.0545, 2024.
Wang, E. Y., Fahey, P. G., Ding, Z., Papadopoulos, S., Pon- der, K., Weis, M. A., Chang, A., Muhammad, T., Patel, S., Ding, Z., et al. Foundation model of neural activity predicts response to new stimulus types. Nature, 640 (8058):470-477, 2025.
Wen, H., Shi, J., Zhang, Y., Lu, K.-H., Cao, J., and Liu, Z. Neural encoding and decoding with deep learning for dynamic natural vision. Cerebral cortex, 28(12):4136 4160, 2018.
Willeke, K. F., Fahey, P. G., Bashiri, M., Pede, L., Burg, M. F., Blessing, C., Cadena, S. A., Ding, 7.,, Lurz, K.-K., Ponder, K., et al. The sensorium competition pre- dicting large-scale mouse primary visual cortex activity. arXiv preprint urXiv:2206.08666 2022.
Willett, F. R., Kunz, E. M., Fan, C., Avansino, D. T., Wilson, G. H., Choi, E. Y., Kamdar, F., Glasser, M. F., Hochberg, L. R., Druckmann, S., et al. A high--egttie gegh-perffrmance speech neuroprosthesis. Nature, 020 2023.
12

---

Neural Encoding and Decoding at Scale
A. Neural encoding metric (BPS)
We use bits per spike as metric to evaluate neural encoding performance. The bits per spike for a neuron is defined as follows:
1 (C(r; - -(Fn
bits/spike spp log
where Xa.r represents the neural activity of the target neuron n at time t, r is the inferred firing rate from the model, F.. is the mean firing rate for neuron n, and ssp is its total number of spikes. We then average the bits per spike across all neurons test session to get the final performance.
The bits per spike metric is closely related to the deviance, standard metric generally used in Statistics, e.g., generalized linear models (GLMs) use deviance as goodness-offff measure for statistical model. The deviance compares the goodness-of-fit of the model of interest, e.g., NEDS, to a baseline null model, where the goodness-of-fit is measured by the model log likelihood. The bps is normalized version of the deviance metric, which compares the model predictions to the average firing rate of the neuron for the trial. The bps further normalizes the deviance by the spike count so that the metric can be comparable across neurons regardless whether the neurons are active or inactive.
B. Masking scheme ablation
In NEDS, introduce cross-modal and within-modalle random masking alongside neural and behavior masking for encoding and decoding (details in Section 3.2). To assess the importance of these additional masking schemes for model performance, we perform an ablation study by training NEDS with each scheme individually removed. We evaluate the models on 10 held-out test sessions for single-session training. The findings are presented in Table 2.
Masking Scheme Removed
Encoding
Decoding
Spike
Choice
Block
Wheel
Whisker
None Cross-Modal
0.20 = 0.00
0.84 0.03
0.83 + 0.02
0.57 0.03
0.52 0.03
0.20 = 0.02
0.83 + 0.05
0.83 + 0.02
0.50 = 0.02
0.46 + 0.03
Within-Modality
0.10 = 0.02
0.84 + 0.05
0.81 + 0.03
0.52 = 0.03
0.46 + 0.03
Table 2. Ablation study of modhitt and cross-modal masking. We evaluate the impact of the cross-modal and within-modalit masking schemes on the performance of NEDS in both single-session experiments. The metrics are computed on the held-out test sessions.
In Table 2, we observe that both within-modality and cross-modal masking have large impact on the model's performance. Removing the within-modalit masking scheme results in a large drop in performance across all metrics, particularly in encoding. Removing the modsl masking scheme has a more modest impact on encoding, but has a large effect on continuous behavior decoding.
C. Model and hyperparsameter details
NEDS
Preprocessing. For both neural encoding and decoding, utilize the raw spike counts without preprocessing. We normalize the continuous behavior to range between -1 to 1.
Losses. For encoding, we use the Poisson negative log- ligliikel as the loss function, while for decoding, we apply cross-entropy loss for discrete behaviors and mean squared error for continuous behaviors.
Hypeeparaameter. To select the optimal model architecture and optimizer configuration for our single-session and multi- session models, we use Ray Tune (Liaw et al., 2018) randomly sample 50 hyperparameter combinations from the ranges specified in Table 3. The hyperparameters for single-session models are chosen by Ray Tune based on the ranges specified in Table 3. For each session, we choose 50 randomly sampled models and select the best hyperparameters according to validation performance. To determine the optimal model architecture for multi-session NEDS, we conduct hyperparameter tuning by evaluating 50 randomly sampled models on 10 pretraining sessions. After selecting the architecture, we retain most of its configuration unchanged and only increased the model depth when scaling up to 74 pretraining sessions. When finetuning pretrained models, Ray Tune is used only to optimize the mask ratio, weight decay, and learning rate, as the model architecture remains fixed and inherited from the multi-session model. During finetuning, we randomly sample 30
13

---

Neural Encoding and Decoding at Scale
Hyperparameter
Value Range
Embeddligg Dimension
[128, 256, 512]
Head Dimension
[256, 512, 1024]
Depth
[5, 6] Uniform(0.1, 0.4)
Mask Ratio
Weight Decay
Log-Unifommn Cee-Uniformm(.0000 0.1)
Learning Rate
Log-Uniform((.0000 0.001)
Table 3. The range of possible NEDS model and optimizer hyperparameters from which Ray Tine randomly samples combinations.
Hyperparameter
Value
Embedding Dimension
256
Head Dimension
512
Number of Heads
8
Depth
22
Mask Ratio
0.1
Dropout Rate
0.2
Attention Dropout Rate
0.4
Weight Decay
0.01
Learning Rate
0.003
Batch Size
1024
Table 4. Hyperparameters used for training 74-session multimodal NEDS,
combinations of these hyperparameters for each session.
The hyperparameters of our 74-session multimodal model are provided in Table The transformer model comprises 12 million parameters. When including session-specifee input matrices and linear decoders, the total number of model parameters increases to 150 million. The dataset consists of roughly 13 million tokens, obtained from 14 hours of neural and behavioral data. The total neuron hours (calculated as the number of neurons multiplied by the number of hours) amounts to roughly 400,000 in total.
NDT2
For NDT2, re-implement the model to align with the workflow of NEDS and ensure consistency in evaluation methods. NDT2 follows encoder-decoder architecture, specifically masked autoencoder adapted for neural spike data. The workflow consists of three main steps. As first step, the model is pretrained with the self-supervised learning (SSL) task of reconstructing masked neural patches. As a second step, the model is fine-tuned for session of interest with the previous SSL. setting. As a final step, the encoder is frozen and a new decoder is trained to decode a task variable. For the SSL pretraining, as the original NDT2, we use the Poisson negative log- ligllikeli as the loss function. For decoding, we apply cross-entropy loss for discrete behaviors and mean squared error for continuous behaviors.
For discrete behaviors, extend NDT2 to classification the original NDT2 only allows for regression (e.g., velocity decoding). To predict discrete behaviors at the trial level, avoid generating predictions at every timestamp by using single query token in the decoder. This query token must be linked to a specific timestamp within the trial to enable temporal attention. We experimented with three options: the first, average, and last timestamp. As expected, using the last timestamp produced the best results, as it allows the query token to leverage the most contextual information due to causal temporal attention masking.
To select the optimal model architecture and optimizer configuration for NDT2 pretraining, we use Weights Biases (Biewald, 2020) to randomly sample 50 hyperparameter combinations from the ranges specified in Table 5. Due to limited compute budget, we perform our hyperparameter search 10 randomly selected sessions with encoder that has 0.5/1.7/6. million parameters (depending on the embedding size). The decoder and learning rate scheduler match those of the original NDT2 for pretraining and the other steps. The best hyperparameters for NDT2 pretraining are chosen based on the validation performance (i.e. validation SSL loss).
While scaling up pretraining sessions, we observed instabilities in the training. To mitigate this, we reduced the
14

---

Neural Encoding and Decoding at Scale
Hyperparameter
Value Range
Embedding
[128, 256, 512]
Patch Size
[32, 64]
Mask Ratio
Uniform I, 0.5)
Weight Decay
(og-Uniform(000000 0.1)
Learning Rate
Log-Uniform((.000) 0.01)
Table The range of possible NDT2 model and optimizer hyperparameters from which Weights and biases randomly samples combina- tions.
learning rate by factor of 2 (from 0.001 to 0.0005), resulting in more stable training. To give NDT2 more capacity model 74 sessions of data, we increased the encoder depth from 3 to 6, resulting in 12.9 million parameters encoder. Additionally, we experiment with scaling the encoder up to 44 million parameters by increasing the encoder depth to 12 and the encoder head dimension to 2,250. However, the pretraining curve shows a similar trend as the 12.9 million parameters encoder, converging at a comparable epoch with similar validation results.
The hyperparameters for NDT2 74-sessions pretraing model are detailed in Table 6.
Hyperparameter
Value
Embedding Dimension
512
Patch Size
64
Encoder Depth
6
Decoder Depth
2
Encoder Head Dimension
1024
Decoder Head Dimension
512
Encoder/Decoder Number of Heads
Encoder/Decoder Dropout
0.1
Mask Ratio
0.11
Weight Decay
0.001
Learning Rate
0.0005
Batch Size
1024
Table 6. Hyperparameters used for pretraining 74-session NDT2.
For the SSL finetuning individual sessions, use the best hyperparameter configurations from pretraining. We performed sweep over the learning rate, but we found it did not result in significant changes in convergence or validation performance. During SSL finetuning, the decoding loss exhibits high variance. To prevent premature stopping and poor checkpoint selection, we apply running average using a sliding window over the last 10 validation epochs to stabilize loss monitoring.
To obtain the decoding result for specific session, we freeze the encoder that was trained with SSL finetuning. For each task, we train a new transformer decoder that adapts to the specific task. To obtain the best possible performance, we tested 4 different learning rates (i.e., 0.0004, 0.0002, 0.0001, 0.00005) for each session-task pair. Regardless of the learning rate used, we always report the best session-task result obtained from the best validation result.
POYO+
Following the tuning strategies for other baselines, we first do a random hyperparameter tuning search using 10 sessions. We perform 100 runs then report the hyperparameters for the best model based on the validation accuracy. We then train a larger model on sessions. The final 74-session model has 3.2 million parameters. The hyperparameters for the model can be found in Table 7. The pretrained model is then finetuned on the 10 held-out sessions. We use early stopping during finetuning and report the test metrics.
RRR
For the RRR encoder, both the behavior inputs and spike count outputs are z-scored. As a result, the loss function used is mean squared error. During evaluation, the model predictions are converted back to the original data scale. For the RRR decoder, we retain the raw spike count inputs without preprocessing and normalize only the continuous behavior outputs to range of -1 1. We use cross-entropy loss for decoding discrete behaviors and mean squared error for decoding
15

---

Neural Encoding and Decoding at Scale
Hyperparameter
Value
Embedding Dimension
128
Head Dimension
64
Number of Latents
32
Depth
Number of Heads
FFN Dropout
0.3
Linear Dropout
0.3
Attention Dropout
0.3
Weight Decay
le-3
Learning Rate
le-3
Batch Size
128
Table Hyperparameters used for training the 73 session POYO model
continuous behaviors.
To select the optimal model hyperparameter and optimizer configuration for our single-session RRR models, we used Ray Tume to randomly sample 50 hyperparameter combinations from the ranges specified in Table The RRR model's maximum allowable rank is 100 (full rank), matching the number of time bins in the IBL data. We select the model rank from specified range. The RRR encoder and decoder require different optimizers: the encoder uses LBFGS, while the decoder uses AdamW. As result, they require different ranges of learning rate. For multi-session RRR, we use an encoder rank of 4, learning rate of 0.001, a weight decay of 0.01, and batch size of 16.
Hyperparameter
Value Range
Encoder Rank
Randint(2, 50)
Encoder Learning Rate
Log-Uniform(0.001, 1)
Decoder Rank
Randint(2, 50)
Decoder Weight Decay
Log-Uniform(0.000 0.1)
Decoder Learning Rate
Log-Uniform(0.0000, 0.001)
Table 8. The range of possible RRR model and optimizer hyperparameters from which Ray Tane randomly samples combinations.
Linear
We begin by preprocessing the input and output of the linear encoder and decoder, standardizing them to follow a normal distribution. Next, we use seikit-learn's 's (Pedregosa et al., 201 GridSearchC' function to select the regularization strength from the set [0.0001, 0.001, 0.01, O.1, 1, 100, 1000, 10000] when fitting the linear encoder and decoder. During evaluation, the model predictions are then converted back the original data scale.
D. Model size effect
A model's parameter count provides general measure of its expressivity, but its ability to generalize also depends on the scale of the training dataset. When excessively large models are trained on limited data, they are more susceptible to overfitting, capturing noise rather than learning meaningful patterns. practice, models with fewer parameters can outperform larger ones on smaller datasets by maintaining a better balance between model capacity and data constraints. To systematically investigate this trade-off for NEDS, we perform an ablation where we train NEDS with million (3M) and 12 million (12M) parameters single-session data from the 10 held-out sessions.
In our main experiments, we use 3-million-paranee parameter model for single-session NEDS and I 12-million-paran parameter model for 74-session NEDS. As shown in Table 9, model size significantly impacts single-session performance in NEDS. We find that a 12-million parameter -arlmeenn model overfits when trained on a single session, leading to degraded performance in both encoding and decoding tasks. This finding supports our choice of a smaller model for single-session NEDS and larger model for multi-session training.
16

---

Neural Encoding and Decoding at Scale
Model Size
Encoding
Decoding
Spike
Choice
Block
Wheel
Whisker
3 Million
0.20 + 0.00 0.09 + 0.06
0.84 + 0.03
0.83 + 0.02
0.57 0.03
0.52 + 0.03
12 Million
0.83 + 0.15
0.83 + 0.06
0.51 + 0.07
0.46 + 0.10
Table Effect of model size on NEDS performance. We evaluate the effect of model size on the performance of NEDS single-session experiments. All metries are computed based on the 10 held-out sessions.
E. Benchmark of NEDS vs. other multi-session models
In Table 10, we show the results of our comparison of NEDS to two ofate-oftthe-rr large-scale models for neural decoding: POYO+ and NDT2. We also compare NEDS to a linear multi-session decoder (Zhang et al., 2024a) trained on 74 sessions. We provide the mean and standard error for each method across the 10 held-out sessions.
Method
Encoding
Decoding
Spike
Choice
Block
Wheel
Whisker
RRR
0.83 0.04
0.84 + 0.03
0.45 + 0.02
0.41 t 0.03
NDT2
0.81 + 0.05
0.79 + 0.04
0.47 + 0.04
0.46 + 0.05
POYO+
0.90 + 0.02
0,81 + 0.01
0,51 + 0.03
0.52 + 0.02
NEDS
0.27 + 0.03
0.91 + 0.03
0.86 + 0.02
0.64 + 0.02
0.59 + 0.03
Table 10. Benchmark of NEDS and other multi-session models. We evaluate all decoding models on choice and block (accuracy), and wheel speed and whisker motion energy (single-trial R²). Each bascline is pretrained on 74 sessions of IBL data and fine-tuned on 10 unsoon, held-out sessions. Wc show the average and standard error for all methods across the 10 sessions.
F. Training details
NEDS
We train both the single-session and multi-session NEDS using the Adam optimizer. The 74-session models are trained on 16 Nvidia RTX8000 GPUs (each with 48GB memory) in under 2 days for a total of 2000 epochs. Single-session multimodal NEDS and unimodal encoding NEDS can be trained on a single Nvidia A40 GPU in less than 2 hours, also for 2000 epochs. Single-session unimodal decoding NEDS requires less than 30 minutes on one Nvidia A40 GPU for the same number of epochs. Finetuning can be similarly performed using a single GPU. For the extensive hyperparameter tuning experiments. use 4 Nvidia A40 or V V100 GPUs for single-session NEDS (including both training from scratch and fine-tuning), and complete the process in less than 1.5 days.
NDT2
We train NDT2 using the AdamW optimizer as original NDT2 docs. For the 50 hyper parameters search experiments, we use Nvidia H100 GPU (80GB memory) per experiment. In total, it took 90 hours for running all experiments. For the large scale pretraining experiment on 74 sessions, we use 4 Nvidia H200 GPUs for hours and 30 minutes reach 600 use 1 Nvidia H100 GPU (80GB memory) per session for
epochs. For the SSL. finetuning experiment
the 10 sessions, total of 2 hours. Finally, for the 160 (= 4 learning rate configurations x 4 tasks 10 sessions) decoding experiments, we use Nvidia H200 GPU per experiment for a total of 10 hours 30 minutes.
POYO+
We use the official implementation of POYO+ which allows us to train the model on four decoding tasks simultaneously. A major difference from the original model is the use of trial-aligned data, with the context window fixed to seconds, and no temporal jittering used during data sampling, which is an important source of data augmentation used during training. We do use unit dropout data augmentation. When training POYO+, we use the LAMB optimizer (You et al., 2019) with weight decay. The learning rate is held constant, then decayed towards the end of training (last 25% of epochs), using cosine decay schedule. The model is trained in supervised manner on four different behavior decoding tasks. To balance the scale of the different tasks, we increase the weight of the regression tasks by a factor of 20. Training is done on a single A A100 GPU for 100 epochs, lasting around 6 hours.
RRR
17

---

Neural Encoding and Decoding at Scale
We train the single-session RRR encoder using the LBFGS optimizer, which converges quickly under a minute, typically within 20 epochs. For the single-session RRR decoder, we use the AdamW optimizer, requiring less than 20 minutes on single Nvidia A40 GPU for 1000 epochs. For hyperparameter tuning experiments, we use one CPU for single-session RRR encoder, and one Nvidia A40 GPU for single-session RRR decoder, completing the process within day. We train the multi-session RRR for 2000 epochs on a single Nvidia A40 GPU, completing the process in two days.
G. Embedding ablations
We performed ablation study single-session data (corresponding Figure 2) to evaluate the importance of the temporal and modality embeddings. Session embeddings were not ablated, they are essential for distinguishing data from different sessions. The ablation results, presented in Table 11, demonstrate that the embeddings generally improve performance across all five tasks.
Model Variant
Encoding (bps)
Choice (Acc)
Block (Ace)
Wheel (R²)
Whisker (R²)
NEDS (multi-session)
0,27 = 0.08
0.91 + 0.08
0.87 = 0.05
0.64 + 0.05
0.59 = 0.08
NEDS (single-session)
0.20 = 0.06
0.84 + 0.12
0.83 + 0.07
0.57 + 0.09
0.52 = 0.10
NEDS (w/o modality embed)
0.20 + 0.07
0.85 + 0.12
0.83 + 0.05
0.52 + 0.07
0.46 + 0.09
NEDS (w/o temporal embed)
0.24 + 0.07
0.84 + 0.12
0.84 + 0.05
0.52 + 0.08
0.48 + 0.09
NEDS (w/o temporal + modality)
0.25 = 0.07
0.86 + 0.11
0.83 + 0.06
0.53 + 0.07
0.47 = 0.10
Table 1. Ablation of modality and temporal embeddings in NEDS. We ablate the embeddings for single-session NEDS and test the performance on the IBL repeated site dataset. Reported values are means + standard deviations across 10 test sessions.
H. NEDS on monkey reaching dataset
We tested NEDS on the MC-RTT primate motor task dataset (Pei et al., 2021), which differs significantly from the IBL visual Dacision-making dataset several ways: (1) MC-RTT uses Utah arrays, whereas IBI. relies on Neuropixels recordings, (2) MC-RTT involves monkeys, while IBL uses mice. (3) MC-RTT focuses on a random target reaching motor task, unlike the visual decision- task in IBL (4) MC-RTT data is unaligned, compared to the trial-aligned structure in IBL.
As the MC-RTT dataset contains only a single recording session, we compared the single-session variant of NEDS against an MLP decoder. Encoding quality was measured via bits-per-spike (bps), and decod- ing performance for finger velocity was measured using R?. We utilized the MLP and data splits from https: petpss:/githhe //tttbbb.. com/ seanmperkins http//biidddddddee decoding algorithms. The neural and behavioral data was binned at 20ms. To predict each time step, we used a time history of 13 bins which is the default in the repository. We tuned the learning rate of the MLP, keeping other parameters (e.g. model sizes and optimizers) fixed to the defaults provided. We trained 30 random models for NEDS and chose the best validation performance, similar to the procedure in our main text. The results, summarized in Table 12, show that NEDS can work well across recording modalities, species, and tasks.
which has implementations of few common neural
Method
Encoding (bps)
Decoding (Vel R²)
MIP
0.66
Unimodal NEDS
0.04
0.65
Multimodal NEDS
0.07
0.72
Table 12. Benchmark of NEDS the MC-RTT dataset. We evaluate single-session NEDS and baseline MT.P on finger velocity decoding (single-trial R²). Multimodal NEDS outperforms both the unimodal variant and MI.P.
18


Reviews:
[{"review_id": "LM26OiVnTO", "reviewer": "Reviewer_MVUE", "summary": "The paper introduces Neural Encoding and Decoding at Scale (NEDS), a multimodal, multi-task model designed to simultaneously predict neural activity from behavior (encoding) and behavior from neural activity (decoding) using large-scale, multi-animal datasets. NEDS employs a novel multitask-masking strategy that alternates between neural, behavioral, within-modality, and cross-modality masking, implemented within a transformer-based architecture. The model is pretrained on the International Brain Laboratory (IBL) repeated site dataset, comprising Neuropixels recordings from 83 mice performing a visual decision-making task, and fine-tuned on held-out animals. The main findings include: (1) NEDS achieves state-of-the-art performance in both encoding and decoding compared to baselines like POYO+ and NDT2; (2) performance scales with pretraining data and model capacity; and (3) NEDS’s latent embeddings exhibit emergent properties, predicting brain regions with 83% accuracy without explicit training", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer’s suggestion to evaluate the generalizability of our model on additional datasets, tasks, and unaligned data. Given the complexity of the neural recordings we analyze in our paper, spanning multiple brain regions and animals, we intentionally focused on a small set of well-defined, trial-aligned behaviors (e.g., choice) for our evaluation. We agree with the reviewer that it would be interesting to test NEDS across more complex datasets and tasks. To address this, we are currently training multiple baselines models and NEDS on the MC-RTT primate motor task dataset [1], which differs significantly from the IBL visual decision-making task and is also unaligned. We plan to include these results in our response once the experiments are complete.\n\n[1] Pei, Felix, et al. \"Neural Latents Benchmark'21: Evaluating latent variable models of neural population activity.\" arXiv preprint arXiv:2109.04463 (2021).", "claims_and_evidence": "The claims made in the submission are largely supported by clear and convincing evidence. The authors claim that NEDS outperforms existing large-scale models in both encoding and decoding, which is substantiated by quantitative comparisons with POYO+, NDT2, and linear baselines across tasks like choice, block prior, wheel speed, and whisker motion energy. The evidence includes performance metrics such as bits per spike (bps) for encoding and accuracy/$R^2$ for decoding, computed on 10 held-out animals. The claim of scalability with pretraining data is supported by the improved performance of multi-session NEDS over single-session NEDS. The emergent property of brain region prediction is convincingly demonstrated through a linear classifier achieving 83% accuracy on neuron embeddings. No claims appear problematic, as the results are consistently backed by experimental data and visualizations.", "methods_and_evaluation_criteria": "The proposed methods and evaluation criteria are well-suited to the problem of modeling bidirectional relationships between neural activity and behavior. The multitask-masking strategy is a logical extension of masked modeling techniques, allowing flexibility for both encoding and decoding within a single framework. The use of the IBL dataset, with its standardized recordings across 83 mice, is an appropriate benchmark for evaluating scalability and generalization in systems neuroscience. Evaluation metrics—bits per spike for encoding and accuracy/$R^2$ for decoding—are standard and meaningful for assessing neural prediction tasks. The comparison with state-of-the-art models (POYO+, NDT2) and linear baselines ensures a robust evaluation. However, the reliance on trial-aligned data and simple behavioral variables (e.g., wheel speed, choice) may limit the generalizability to more complex tasks, though this is acknowledged as a limitation in Section 7.", "theoretical_claims": "The paper does not present formal theoretical proofs requiring verification.", "experimental_designs_or_analyses": "I reviewed the experimental designs and analyses in Sections 5 and 6, focusing on single-session, multi-session, and brain region classification experiments. The designs are sound: pretraining on 74 sessions and fine-tuning on 10 held-out sessions is a standard approach, and the train-validation-test split (70%-10%-20%) is appropriate. The ablation study on masking schemes (Appendix B) validly isolates the contribution of each component, showing that within-modality and cross-modal masking enhance performance. Hyperparameter tuning using Ray Tune (Appendix C) is rigorous. The brain region classification experiment is well-executed, using 5-fold cross-validation and comparing unimodal vs. multimodal embeddings.", "relation_to_broader_scientific_literature": "NEDS builds on prior work in neural encoding and decoding, it's main algorithmic inspiration being He's 2022 work on Masked Autoencoders for computer vision tasks. Based on this idea, it extends unimodal approaches like POYO+ (decoding-focused) and NDT2 (decoding-capable) by unifying encoding and decoding, addressing a gap noted in the literature.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Strengths: The paper’s originality lies in its creative combination of masked modeling and multi-task learning, enabling a flexible, bidirectional model—a significant advance over task-specific predecessors. The clarity of explanation is strong. The potential for real-world impact, such as brain-computer interfaces, enhances its significance. The emergent brain region prediction also adds an valuable insight.\n\nWeaknesses: The reliance on mice data and simple behavioral tasks (e.g., wheel speed, whisker motion) limits the demonstration of NEDS’s capability for complex behaviors like visual decoding. While the authors suggest extensions to other modalities (Section 7), the current scope feels narrow given the “foundation model” ambition. The computational constraint on hyperparameter tuning for multi-session models (Section 7) is a minor weakness, though mitigated by practical tuning on a subset.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "Generalizability to Complex Tasks: The evaluation focuses on simple behaviors (e.g., choice, wheel speed) in mice. Have you tested or plan to test NEDS on more complex tasks, such as visual decoding or multi-step decision-making? \n\nPotential for unaligned data: The authors mentioned this in the limitations. Given POYO is capable of training on unaligned data and the claim that NEDS's potential for this extension, how would this be possible for NEDS and why isn't it explored in this work?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "iGWqtQExn1", "reviewer": "Reviewer_geJL", "summary": "This paper proposes NEDS, a multimodal, multi-task auto-encoder to learn meaningful representations of neural activity and behavior. In brief, the model is based on an encoder-only transformer that tokenized spikes in a similar scheme to NDT (linear projection of binned spikes), as well as both continuous discrete behaviors, then decodes the same information from a multi-head decoder. This configuration allows the authors to perform multiple tasks: predicting behavior from neural activity, neural activity from behavior, and solving different masking problems. The authors demonstrate their model on the International Brain Lab (IBL) dataset, showcasing that it performs better at many of tasks than a slate of baseline models: linear decoders, reduced rank regression, unimodal models, POYO+ and NDT2. They find that the latent embeddings from the model correlate with which brain regreion the data came from, bringing confidence that it has a learned a meaningful representation of the neural data.", "questions": "", "limitations": "", "rating": 5, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate that the reviewer found our paper well-written and our evaluation convincing. We agree that evaluating NEDS on additional datasets would make our paper stronger. In response, we are currently applying NEDS to a primate motor task dataset (MC-RTT [1]) to demonstrate its generalizability across different recording modalities, species, behavioral tasks, and data structures. We plan to include these results in our response once the experiments are complete.\n\n[1] Pei, Felix, et al. \"Neural Latents Benchmark'21: Evaluating latent variable models of neural population activity.\" arXiv preprint arXiv:2109.04463 (2021).", "claims_and_evidence": "Yes, the claims are well-substantiated and the evidence is clear and compelling.", "methods_and_evaluation_criteria": "To evaluate this model, the key question is whether the proposed method works better than prior methods: they show this through extensive and convincing evaluation with an appropriate slate of benchmark models. Indeed, the extent of the improvements they demonstrate is remarkable compared to prior art; I was impressed with how much better this did than their well-tuned baseline. A secondary consideration is whether the model gives new insights, and their Figure 4 is an interesting proof of concept of that; though I should add that region decoding is a fairly artificial and rather easy task, it is nevertheless reassuring that this works.", "theoretical_claims": "No theoretical claims in this paper.", "experimental_designs_or_analyses": "I did check the soundness of their experiments and analyses. Again, this is pretty well-trodden territory, the IBL dataset is a well-understood dataset, used prior in many papers, including in prior work from Zhang et al. (2024), and the dataset selection, model selection, etc. seem appropriate and in line with the literature.", "relation_to_broader_scientific_literature": "The proposed method builds on prior work, including some of the insights from POYO (a multi-head decoder), NDT (tokenization scheme), and the universal translator from Zhang et al. 2024 (multi-mask scheme). The key innovation is using one model to co-embed behavior and spikes. This deserves a publication in and of itself.", "essential_references_not_discussed": "None.", "other_strengths_and_weaknesses": "There are nice figures, the text is well-written. It would be nice to see this applied to other datasets in a follow-up paper, but there's plenty here to warrant publication.", "other_comments_or_suggestions": "The reference to Azabou et al. (the original POYO paper) has the wrong date, it was published at NeurIPS 2023, not 2024.", "questions_for_authors": "The paper is very clear, no further questions.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NQ0UEoVrti", "reviewer": "Reviewer_MJ1y", "summary": "This paper introduces Neural Encoding and Decoding at Scale (NEDS), a multimodal, multi-task model that simultaneously performs neural encoding (predicting neural activity from behavior) and neural decoding (predicting behavior from neural activity) by bridging behaviors and neural activity with a shared masked training Transformer. The framework is evaluated on 83 mice and shows good results in single-session and multi-session cases.\n\n## update after rebuttal\nThanks for the responses. I keep my original score and hope to see these supplements and further discussion in the camera-ready version.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> It could be better to extend the results to other recordings, or you may discuss the heterogeneity of the neuropixel data to show the capability of the model.\n\nWe agree with the reviewer's suggestion to extend NEDS to other datasets. To address this, we are currently training NEDS on a primate motor task dataset to show its generalizability across different recording setups, species, and behavioral tasks. In future work, we are excited to extend our approach to larger and more diverse datasets.\n\n> Just one hold-out test is a bit thin, and it would be beneficial to do at least one or two more validations to make sure the results are stable, albeit at a cost.\n\nWe agree that additional validation would strengthen the evaluation. Our results rely on a fixed hold-out set due to computational and experimental constraints, and while more rigorous cross-validation would provide a more accurate assessment of model differences, it is currently challenging given the scale of our datasets and models. The development of shared benchmarks will help enable more robust comparisons across different approaches; we plan to participate in such benchmarks in future work.\n\n> Have to test the impact of the token length? How do you prepare the continuous and discrete data to cover an overall behavior?\n\nThis is a great question and something we have not explored. We utilize the tokenization scheme from [1] for our model where the neural data is binned at a specific resolution (20ms) and then each time step is passed into the transformer as a token. We fix the context length to 2 seconds, aligned to movement onset, for all trials and animals. The model performance is more likely influenced by the time bin size rather than token length, as suggested in previous work [1]. We are currently training NEDS on two different bin sizes on a monkey dataset. We are excited to further explore this in future work. For more details on how we tokenized continuous and discrete behaviors, please refer to Section 3.3, “Modality-Specific Tokenization,” in the main paper.  At a high-level, the discrete data is transformed into tokens that are repeated multiple times to match the resolution of the continuous data.\n\n> Is there any difference between the data from different labs (total 10, mentioned in the section 3.1)?\n\nThis is an interesting question! Part of the goal of the International Brain Laboratory (IBL) was to reproduce the same experiment and data across multiple labs. To do this, the IBL utilized a standardized experimental protocol and quality metrics to ensure consistent data quality across labs [2]. As can be seen in Figure 7 of [2], the authors conducted an experiment using all neurons in the IBL repeated site datasets to predict their brain region and lab identity. The findings indicate that while brain region identity can be reliably decoded from single-neuron profiles, lab identity cannot be inferred from the data.\n\n> It would be beneficial to give some details about the four kinds of behaviors, or give some references.\n\nWe appreciate the feedback and will incorporate more descriptions of the behaviors into the final camera-ready.\n\n> Have you introduced the design of the objective functions and some model details, such as the position embeddings.\n\nYes, details about the objective function and position embeddings can be found in Section 3.3, “Architecture,” of the main paper.\n\n> Is it possible to test the performance in the animal-independent condition, which means test the performance with fine-tuning on the test 10 mice?\n\nCurrently, fine-tuning is required for evaluating these models to align the session and neuron-specific weights into a shared representation. Zero-shot performance of these models is an exciting future direction.\n\n> Statistical analysis would make the results better.\n\nWe agree that statistical analysis would improve our results. To address this in the limited time window during the rebuttal, we are currently re-running our single-session analysis with different random seeds to determine whether model performance was significantly influenced by randomness. We plan to include these results in our response once the experiments are complete.\n\n[1] Pei, Felix, et al. \"Neural Latents Benchmark'21: Evaluating latent variable models of neural population activity.\" arXiv preprint arXiv:2109.04463 (2021).\n\n[2] International Brain Laboratory, et al. \"Reproducibility of in vivo electrophysiological measurements in mice.\" bioRxiv (2024): 2024-12.", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "Yes, that's an empirical but interesting study.", "experimental_designs_or_analyses": "Yes, good.", "relation_to_broader_scientific_literature": "Yes, some recent works have been involved for comparison.", "essential_references_not_discussed": "No. The reference is well.", "other_strengths_and_weaknesses": "Strengths: That’s a valuable research focusing on handling neural encoding and decoding with a shared model. Substantial experiments show effectiveness of multi-modal training to achieve good results.\n\nWeaknesses: 1) The evaluation was performed with fixed 10 mice held out from 83 mice, which may introduce randomness. 2) In my view, the work successfully validates the effectiveness of using multi-session data for pretraining and training shared models with brain and behavior data, instead of “scaling” for a large model. Therefore, the title may not be suitable for the content, though that’s cool with a short title.", "other_comments_or_suggestions": "1. It could be better to extend the results to other recordings, or you may discuss the heterogeneity of the neuropixel data to show the capability of the model.\n2. Just one hold-out test is a bit thin, and it would be beneficial to do at least one or two more validations to make sure the results are stable, albeit at a cost.\n3. Have to test the impact of the token length? How do you prepare the continuous and discrete data to cover an overall behavior?\n4. Is there any difference between the data from different labs (total 10, mentioned in the section 3.1)?\n5. It would be beneficial to give some details about the four kinds of behaviors, or give some references.\n6. Have you introduced the design of the objective functions and some model details, such as the position embeddings.\n7. Is it possible to test the performance in the animal-independent condition, which means test the performance with finetuning on the test 10 mice?\n8. Statistical analysis would make the results better.", "questions_for_authors": "Please see the suggestions part.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hUGD8Px2NG", "reviewer": "Reviewer_Fbeh", "summary": "This article introduces a multimodal, multi-task model named \"Neural Encoding and Decoding at Scale (NEDS)\" for large-scale neural encoding and decoding. The model employs a novel multi-task masking strategy, enabling simultaneous bidirectional prediction between neural activity and behavior—predicting neural activity from behavior (encoding) and predicting behavior from neural activity (decoding). NEDS was pre-trained on a large-scale multi-animal dataset and fine-tuned on new animal data, demonstrating exceptional performance and generalization capabilities.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> The use of mask-based pre-training for Transformers is quite common. I am curious whether the three embeddings—Modality Embedding, Temporal Embedding, and Session Embedding—are truly effective...\n\nWhile masked modeling is a common objective for training transformers, it is not yet clear which masking schemes work well for modeling neural activity and behavior. We aimed to address this question in our paper and with our proposed method.\n\nWe appreciate the suggestion to ablate the different embeddings for NEDS. To do this, we performed an ablation study on single-session data (10 test sessions) to evaluate the importance of the temporal and modality embeddings. Session embeddings were not ablated, as they are essential for distinguishing data from different sessions. The ablation results, presented in the following table, demonstrate that the embeddings generally improve performance across the 5 tasks we evaluate (with a few exceptions). We want to note that these are single-session ablations and will be more sensitive to hyperparameters.\n\n|                          | Encoding        |      | Decoding                         |      |      |      |\n|--------------------------|----------------|------|----------------------------------|------|------|------|\n|                          | Encoding (bps) | Choice (Acc) | Block (Acc) | Wheel (R2) | Whisker (R2) |\n| NEDS (multi-session)     | **0.267** ±0.080 | **0.909** ±0.084 | **0.865** ±0.052 | **0.641** ±0.051 | **0.586** ±0.083 |\n| NEDS (single-session)    | 0.203 ±0.062 | 0.840 ±0.115 | 0.827 ±0.072 | 0.568 ±0.089 | 0.523 ±0.097 |\n| NEDS (w/o modality embed) | 0.198 ±0.070 | 0.846 ±0.116 | 0.831 ±0.046 | 0.516 ±0.066 | 0.459 ±0.086 |\n| NEDS (w/o temporal embed)    | 0.243 ±0.070 | 0.842 ±0.121 | 0.835 ±0.053 | 0.523 ±0.082 | 0.478 ±0.085 |\n| NEDS (w/o temporal + modality embed) | 0.246 ±0.069 | 0.859 ±0.106 | 0.830 ±0.063 | 0.530 ±0.067 | 0.469 ±0.095 |\n\n> The paper only compares two methods, POYO+ and NDT2, on a single dataset. Although some performance improvements were achieved, the overall framework does not differ fundamentally from prior work. The experimental section is also limited to evaluating performance in terms of encoding and decoding, lacking insightful scientific findings. \n\nPrior work in modeling spiking neural activity and behavior focuses on modeling a single direction of the relationship (i.e., decoding or encoding). NEDS introduces a new framework for modeling neural activity and behavior by jointly modeling the two modalities, allowing for simultaneous encoding and decoding at test time (POYO+ and NDT2 solely focus on decoding). We believe this multimodal approach is novel for this domain.\n\nWe agree that more experiments are needed to extract scientific insights. However, encoding and decoding remain key tools for understanding what and how information is represented in the brain. For example, the International Brain-wide Map [1] used these methods to characterize how visual, sensory, and motor information are distributed across the mouse brain. We are excited to extend this with NEDS, leveraging large-scale, multi-animal data to capture richer information and reveal how neuronal functional profiles relate to brain anatomy, as shown in Figure 4.\n\n> My main concern lies in the potential value of the contribution of the neuro-behavioral conversion presented in this article. Compared to previous works, this article seems more like an incremental project and does not address fundamental issues in the field, such as whole-brain alignment for neuro-behavioral conversion or conversion specific to the human brain...\n\nWe thank the author for the suggested citation which we would be happy to include. We want to emphasize that we strongly believe that our contribution is not incremental. Currently, there are very few foundation models trained on spiking data, contributing to the novelty of our approach. Also, current foundation modeling approaches for spiking activity have primarily focused on decoding, whereas, to the best of our knowledge, no prior work has unified neural encoding and decoding.\n\nWe note key differences from NeuroBOLT [1]: NeuroBOLT translates between neural modalities (fMRI and EEG) without modeling behavior, and uses separate encoders per modality, unlike NEDS, which uses a shared transformer to unify representations.\n\nWe also feel that the critique related to whole-brain alignment does not apply to spiking recordings which are spatially restricted. Whole-brain alignment is a more tractable problem in fMRI research due to its broader spatial coverage, but is not typically addressed in invasive electrophysiology.\n\n[1] International Brain Laboratory, et al. \"A Brain-Wide Map of Neural Activity during Complex Behaviour.\" bioRxiv (2024).\n\n[2] Li Y, Lou A, Xu Z, et al. NeuroBOLT: Resting-state EEG-to-fMRI Synthesis with Multi-dimensional Feature Mapping[J]. Advances in Neural Information Processing Systems, 2025, 37: 23378-23405.", "claims_and_evidence": "Convincing enough.", "methods_and_evaluation_criteria": "The methodology presented in this paper bears resemblance to previous works such as POYO+ and NDT2. Moreover, the IBL repeated site dataset serves as a benchmark within the domain, underscoring the significance of this study in advancing the alignment research between neural and behavioral modalities. The contributions of this paper are thus noteworthy for their potential to enrich our understanding of the intricate interplay between neural activity and behavior.", "theoretical_claims": "The use of mask-based pre-training for Transformers is quite common. I am curious whether the three embeddings—Modality Embedding, Temporal Embedding, and Session Embedding—are truly effective. Can the author provide evidence to support this?", "experimental_designs_or_analyses": "The paper only compares two methods, POYO+ and NDT2, on a single dataset. Although some performance improvements were achieved, the overall framework does not differ fundamentally from prior work. The experimental section is also limited to evaluating performance in terms of encoding and decoding, lacking insightful scientific findings.", "relation_to_broader_scientific_literature": "NDT2 is a Transformer-based spatiotemporal encoder-decoder model that undergoes unsupervised pre-training through Masked Autoencoding (MAE). And POYO+ is a novel multi-task, multi-session neural decoding approach capable of decoding neural activity from different cell types and brain regions.\n\nThis article combines encoding and decoding to form a neuro-behavioral conversion, an idea which I find unsurprising given its similarity to concepts already prevalent in the paper by Li et al. [1].\n\n[1] Li Y, Lou A, Xu Z, et al. NeuroBOLT: Resting-state EEG-to-fMRI Synthesis with Multi-dimensional Feature Mapping[J]. Advances in Neural Information Processing Systems, 2025, 37: 23378-23405.", "essential_references_not_discussed": "Li Y, Lou A, Xu Z, et al. NeuroBOLT: Resting-state EEG-to-fMRI Synthesis with Multi-dimensional Feature Mapping[J]. Advances in Neural Information Processing Systems, 2025, 37: 23378-23405.", "other_strengths_and_weaknesses": "My main concern lies in the potential value of the contribution of the neuro-behavioral conversion presented in this article. Compared to previous works, this article seems more like an incremental project and does not address fundamental issues in the field, such as whole-brain alignment for neuro-behavioral conversion or conversion specific to the human brain, among others.", "other_comments_or_suggestions": "No.", "questions_for_authors": "I don't have any more questions; I am looking forward to the author's response to the concerns I raised earlier.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper is along the recent trend of foundation models in neuroscience. It establishes a multi-modal multi-task encoding-decoding approach to predict neural activity from behavior and vice versa. The paper tests this approach in a single-session and multi-session setting, establishing that pre-training on multiple sessions improves performance over the single-session case and prior approaches like NDT2 and Poyo+. All reviewers applaud the paper as being well written and making a noteworthy contribution (albeit with varying degree of enthusiasm). Hence I believe this is a strong case for acceptance.

A few comments to the authors for the camera-ready version:
- The ablation table in response to reviewer Fbeh seems misleading. If I read your rebuttal correctly, these experiments were done on single-session NEDS but you include and boldface multi-session NEDS in this table, suggesting somewhat misleadingly that removing the modality and temporal embeddings decreases performance, while it does not when properly compared to the single-session version in row 2 (in fact, performance increases in one case and is unchanged in another).
- That multi-animal pretraining improves performance has been shown several years ago in the context of visual encoding models by [Lurz et al., ICLR 2021](https://openreview.net/forum?id=Tp7kI90Htd) – would probably make sense to mention it in the related work.
Decision: Accept (spotlight poster)