OmniMouse: Scaling properties of multi-modal, multi-task Brain Models on 150B Neural Tokens
Authors: Konstantin Friedrich Willeke
Polina Turishcheva
Alex Gilbert
Goirik Chakrabarty
Hasan Atakan Bedel
Paul G. Fahey
Yongrong Qiu
Marissa A. Weis
Michaela Vystrčilová
Taliah Muhammad
Lydia Ntanavara
Rachel E Froebe
Kayla Ponder
Zheng Huan Tan
Emin Orhan
Erick Cobos
Sophia Sanborn
Katrin Franke
Fabian H. Sinz
Alexander S. Ecker
Andreas S. Tolias
Venue: ICLR 2026 Poster (2026)
Keywords: Scaling laws
Multimodal Transformers
Foundation models
Visual cortex
Neural encoding models
Neural decoding
Behavioral prediction
Calcium imaging
Primary area: applications to neuroscience & cognitive science
TL;DR: Scaling a unified model of neural activity, video, and behavior across 73 mice reveals that brain modeling is data-limited, not parameter-limited, even at unprecedented recording scale.

Abstract: Scaling data and artificial neural networks has transformed AI, driving breakthroughs in language and vision. Whether similar principles apply to modeling brain activity remains unclear. Here we leveraged a dataset of 3.1 million neurons from the visual cortex of 73 mice across 323 sessions, totaling more than 150 billion neural tokens recorded during natural movies, images and parametric stimuli, and behavior. We train multi-modal, multi-task models that support three regimes flexibly at test time: neural prediction, behavioral decoding, neural forecasting, or any combination of the three. OmniMouse achieves state-of-the-art performance, outperforming specialized baselines across nearly all evaluation regimes. We find that performance scales reliably with more data, but gains from increasing model size saturate. This inverts the standard AI scaling story: in language and computer vision, massive datasets make parameter scaling the primary driver of progress, whereas in brain modeling -- even in the mouse visual cortex, a relatively simple system -- models remain data-limited despite vast recordings. The observation of systematic scaling raises the possibility of phase transitions in neural modeling, where larger and richer datasets might unlock qualitatively new capabilities, paralleling the emergent properties seen in large language models. Code available at \url{https://github.com/enigma-brain/omnimouse}.

Published as a conference paper at ICLR 2026
OMNIMOUSE: SCALING PROPERTIES OF MULTI-MODAL, MULTI-TASK BRAIN MODELS ON 150B NEURAL TOKENS
Konstantin F. Willeke--H1a4, Polina Turishcheva'4, Alex Gi2 Goirik Chakrabarty", Hasan A. Bedell23, Paul G. Fahey12a, Yongrong Qiu123, Marissa A. Weist, Michaela Vystrčilová', Taliah Mubammad Lydia Ntanavara Rachel E Froebe123, Kayla Ponder123, Zheng Huan Tan12a, Emin Orhan', Erick Cobos?*, Sophia Sanborn Katrin Franke123, Fabian H. Sinz Alexander S. Eckeri4s, Andreas S. Tolias
Toliast1233.89 1,277,8,9
DDpparmeen of Ophthalmology. Byers Eye Institute, Stanford University
"Stanford Bio-X, Stanford University
Ww Tsai Neurosciences Institute, Stanford University
Institute of Computer Science and Campus Institute Data Science, University Göttingen
Max Planck Institute for Dynamies and Self-Organte Self-Organization, Göttingen
Nattioal Center for Computational Sciences, Oak Ridge National Laboratory
"Center Neuroscience and Artificial Intelligence, Baylor College of Medicine
*Department of Neuroscience, Baylor College of Medicine Department of Electrical Engineering, Stanford University *Corresponding author: willeke@staanfrd. edu
Equul contribution
ABSTRACT
Scaling data and artificial neural networks has transformed AI, driving break- throughs in language and vision. Whether similar principles apply to modeling brain activity remains unclear. Here we leveraged a dataset of 3. million neurons from the visual cortex of 73 mice across 323 sessions, totaling more than 150 bil- lion neural tokens recorded during natural movies, images and parametric stimuli, and behavior. We train multi-modas multi-task models that support three regimes flexibly at test time: neural prediction, behavioral decoding, neural forecasting, or any combination of the three. OmniMouse achieves state-of-the-art performance, outperforming specialized baselines across nearly all evaluation regimes. We find that performance scales reliably with more data, but gains from increasing model size saturate. This inverts the standard AI scaling story: in language and computer vision, massive datasets make parameter scaling the primary driver of progress, whereas in brain modeling even in the mouse visual cortex, relatively simple system models remain data-limited despite vast recordings. The observation of systematic scaling raises the possibility of phase transitions in neural modeling, where larger and richer datasets might unlock qualitatively new capabilities, par- alleling the emergent properties in large language models. Code available at https: bittps://gitee //github. com/ eni iaaiigm--aa brwin/ catinooommmu
Scaling model full
Subsets of recording sessions
Nouronal tesporues
Visual stmuukus
>1500 tokens
Scaling data
1M
64
A$
SM
Unittec mi dtimmdal muiti task model
20M
Test loss
.15
80M
300M
16
32
64
323
Number of recordingg from mouse visual cortex
5M
20M
60M 10
201
1079
10
10
101*
107P
Flaps
Flaps
Figure I:
A. OmniMouse unifies neural prediction, behavior decoding, and forecasting tasks. B. Scaling model size on 150+ billion neural tokens shows performance saturation. unlike language models. C. In contrast, scaling data consistently improves performance across all model sizes.

---

Published as a conference paper at ICLR 2026
1 INTRODUCTION
Scaling models and data has driven recent progress machine learning, with large language, vision, and multi-modal models showing consistent performance gains and enabling foundation models that unity tasks across domains. A natural question is whether models of the brain can also benefit from scaling. The mouse visual cortex, with large-scala neural datasets (MICTONS Consortium et al. de Vries et al.] [019]] Angelaki et al. 2025) and standardized benchmarks (Willeke et al. Turishcheva al. 2004), offers natural setting to investigate this, yet compared with the internet-scale corpora used in language and vision, available neural datasets are much smaller, more fragmented, and less diverse. The neuroscience community has recently started to work toward al.
2021
2022
foundation models for EEG (Chau et al. 2024 Chen et al. 20224] Cui et al. [2024)
2024
Kostas et al. [2221] Yang et | [202]] Thapa et al. 2024 Li et al. (2024). 1MRI (Caro et al. 2023
Dong al. [024] Kan et al. 2022: T'homas al. [0022] d'Ascoli et al. 2025), MEG (Csaky al.
2024]. and intracranial signals (Zhang et al. 2023; Wang et al. 2023). Prior work in this direction
focused on isolated modalities (Ye et 2023| Azabou al.] (023), single predictive task (Wang) et all 2025), lacked scalability across datasets Ye & Pandarinath [021] Mi er al., 2023;] Antoniades
al. 2024), ) omitted stimulus (Zhang et al. 2025) and behavioral information (Jiang al. 2025
Mi et al.) (023).
In this work, we introduce OmniMouse, a multi-modal, multi-task architecture for modeling neu- ronal activity in the mouse visual cortex. Unlike prior models that are typically restricted to a single modality, OmniMouse combines single-neuron tokenization, video encoding and behavioral decod- ing into a unified architecture. Our model design enables flexible masking, allowing the model to handle arbitrary combinations of neural forecasting, stimulus-conditionee response prediction, sub- population prediction, and behavioral decoding. We train OmniMouse on one of the largest single- neuron datasets to date: 323 recordings from the visual cortex of 73 awake mice viewing naturalistic movies, images, and parametric stimuli, totaling over 150 billion neuronal activity tokens from over 3 million neurons.
Our main findings and contributions are:
Systematic scaling analysis: We find that performance improves systematically with more data, but saturates with model size beyond moderate scales. This suggests that data, not model size, is currently the bottleneck for predictive accuracy in neural modeling.
A multi-modal, multi-task model for visual cortex: OmniMouse handles both single- modality and multi-modal inputs, supporting any combination of forecasting and stimulus-conditioned pre- diction across different neurons, stimuli, time points, and animals.
OmniMouse achieves state-of-the-art performance: When compared to strong specialized base- lines, OmniMouse outperforms prior specialized methods across nearly all tasks, showing that our architecture is competitive independent of data scale advantages.
2 RELATED WORK
Large-scale deep learning models for single-neuron predictions. Deep learning has advanced
predictive modeling in neuroscience, particularly in vision (Cadieu et al.
2014 Batty et al., 2017
Klindt al. 2017
McIntosh et al. 2016 Cadena al. 2019 Kindel al.
2019 Walker al.
2019
Zhang et al. 2018 Ecker et al. 2018 Sinz et al. 2018 Burg et al. 2021 Cowley & Pillow 2020
CNN-based approaches introduced shared leature cores with per-neuron readouts (Antolik et al.
2016
Klindt et al. 2017; McIntosh et al.) 2016 Sinz et al. 2018 Lurz et al. 2021). culminating in mu mulmi- "digital twins' that capture biological phenomena beyond training data (Ustyuzhaninov
et all
2022 Wang et al. 2025). With the shift to transformers, new variants have explored ViT
cores
(Li et al. (003). hybrid convolution-attention designs (Lin et al. 2024; Pierzchlewiez et al., and spatial- spatial-transfoore transformerr readouts (Saha et al. 2024). though most still omit video input.
2023
Transformer models were also applied to response-to-response modeling, beginning with NDT YYe & Pandarinath, 2021), extended to multiple animals (Ye et al. 2203) and neuronal masking (Zhang et all 2024). STDNT (Le & Shlizerman (2022) extended NDT by explicitly modeling correlations across neurons but it did not consistently outperform NDT. While NDT projects all neurons together via linear layers, Quantformer (Calcagno et al.) 2224)) also a transformer- based forecaster, introduced neuron-specific tokens to handle any number of neurons. POYO (Azabou et al., 2023) introduced
2

---

Published as a conference paper at ICLR 2026
binning, and POYO+ (Azabou et al. 2025) extended this to discrete classification tasks such as stimulus orienta-
hingle-neuron tokenization for behavior decoding, removing the need for
tion. POCO (Duan et al. 2025) combined POYO and NDT tokenization to predict neuronal activity
from neural history and sub-populations. Representing the most significant scaling of this frame-
work, NEDS (Zhang et al.] 2025) modeled approximately 30,000 neurons across 74 sessions with multi-task loss. None of these models incorporate visual stimuli.
To study the combined effect of brain state and visual input on neuronal responses, Bashiri et al. (2021) used a CNN branch for static stimuli with an additional flow-branch for trial-to-trial correla- tions. For dynamic video stimuli. Schmidt al. (2025) modeled a latent brain state probabilistically, using baTeb response tokenization. Similarly, Neuroformer (Antoniades et al. (024) used past activity and visual input but is limited to single sessions and cannot flexibly condition on subsets of neurons or response history. CEBRA (Schneider et al., (023), contrastive encoder, also mapped activity to behavior or stimuli, accounting for inter- neuron correlations. The closest analog to Om- niMouse outside single-cell neuroscience is d'Assooli et al. (2025). who predicted fMRI responses from concatenated video, text, and audio embeddings.
General scaling laws in deep learning. Large-scale models in language and vision exhibit pre- dictable improvements with scale, described by empirical "scaling laws". Kaplan et al. (2020) first showed that performance follows power-law trends in model size, dataset size, and compute. Hoffmann et al.] 2022) refined this with "Chinchilla scaling", prescribing proportional growth of model and data size for optimal efficiency. Whether these frameworks extend scientific do- mains, where data are multi-modal, complex, noisy, and limited, is less clear. Examples such as AlphaFold3 (Abramson et al. 2024) suggest that systematic scaling of both models and datasets can drive major advances in AT for science.
Scaling neuroscience models. There is no consensus on whether classic machine learning scaling laws apply to single-neuron data. Jiang et al. (2025) questioned their applicability. analyzing the NDT-based model of Zhang et al. (2024). Jiang et al. (2025) argued that cross cross-session variability and thus implicit data heterogeneity is crucial for scaling benefits, though it remains unclear if these results generalize to different mouse tasks or model architectures. Again using an NDT-based model but on motor cortex microelectrode data from monkeys and humans, Ye al. (2025) reported that scaling is constrained by data variability, which pretraining alone cannot fully overcome. Con- sistent with this view, POCO (Duan al. 2025) used calcium imaging show that recordings improve predictive performance, aligning with earlier results Lurz et al. (202I). However, POCO included fewer than 90,000 neurons, mostly from zebrafish (~77,000). Neural saturation has also been observed: Gokce & Schrimpi (2024) found that behavioral alignment improves with model size, but neural aligament plateaus, with gains concentrated in higher-level visual areas. In contrast, Antonello et al.) (2023) reported no such saturation when predicting language and audio iMRI re- sponses, suggesting that scaling limits may depend on the modality and data regime. The largest single-cell response-tor to-bbavvoo prediction model is POYO+ Azabou et al. (2025) with ~ 100,0000 neurons, which did not perform extensive scaling analyses.
3 DATASET
Neuronal responses. We compiled a dataset of more than 3 million single-unit neuronal record- ings from 73 animals (Fig. The dataset contains excitatory neurons' responses visual cortex recorded via wide-field two-photon calcium imaging at 6-14 Hz in awake, head-fixed, behaving mice (Sofroniew et al. 2016), with spiking activity extracted by CAIMAN (Giovannucci et al. 2019).
Visual stimuli. The mice were presented with naturalistic images sampled from ImageNet (Rus- Sakovsky et al, 2015) and videos sampled from cinematic movies and the Sports- IM dataset (Karpa- Thy et al, 2014). In addition, mice were shown parametric stimuli such static and drifting Gabors PPetkov & Subramanian 2007), directional pink noise, flashing Gaussian dots, random dot kine- matograms (Morrone et al. | 2000), and model-generated stimuli (similar to Walker et al.) 2019). All stimuli were presented at 30-66 Hz, at varying resolutions with images presented for 500 ms and preceded by 300-500 ms blank screen.
3

---

Published as a conference paper at ICLR 2026
B
Stiuuus presentation, entire experiment
Train
Recording sessions
Animals
69
Natura video
Natural
316
Natural images
10s
Neurons
3.1 M
76,669
Parametric
Duration (h)
543
14,6
Other images
Nouran hours
5.73
159,759
Neur. Activily samples
155.4
4.6 B
Bahavior samples
293.0 M
7.9 M
523
Behavior
Rumning speed
frames
38.5
1.1 M
Avg
31%
17%
17%
3.%
position Pupil size
Image frames
743,130
17,391
Total Stimullus duration
Figure 2: Data. A. Data were collected via calcium imaging from head-fixed mice running on wheel while viewing visual stimuli. Behavior variables include pupil center z and y positions, pupil dilation and its derivative and running speed. B. Dataset statistics. C. Different visual stimuli were presented across sessions, with stimulus types varying by session.
Behavior variables. Our dataset contains five behavior variables: running speed, recorded at 50- 100 Hz, and four pupil variables: pupil center z and y positions, pupil dilation and its derivative, all recorded at 20 Hz.
Data utilization. We reconstruct the visual stimulus presented throughout the entire recording, en- abling continuous representation of the full experimental timeline including blank periods across all diverse visual paradigms. For model training, downsample all behaviors to 20 Hz, visual stimuli to 30 Hz, and linearly upsample all neuronal responses to 30Hz to be comparable to the SEN- SORIUM 2023 benchmark. All token counts throughout this work refer the number of samples before upsampling to avoid artificially inflating the dataset size.
4
OMNIMOUSE ARCHITECTURE
4.1 TOKENIZATION
The input OmniMouse is temporal chunk of multi-modal data: neural responses, stimulus frames, and behavioral traces, all time-aligned at their respective sampling rates. This chunk is paired with a masking configuration specifying which samples from each modality should encoded (i.e., un- masked) and which subset of the masked samples should serve as reconstruction targets. After en- coding each modality as "token" embeddings, we apply the mask by simply removing masked tokens from the input and constructing query embeddings for the targets. Below we describe this process for each modality.
Neuronal responses. To enable per-neuron, per-sample masking, we adopt the tokenization from abou al. 2025)) Duan et al. (0025), For P: neurons (possibly sampled from a larger recorded popu- lation) from a session i, construct a tensor of Sg calcium trace samples and apply strided ID con- volution (stride SR. window wx) along the temporal dimension: feewv P. xSR > R xT' Dands where X₂ n2 = Scoms(Fi) RR), Demodel is the model's latent dimension, and 7 S-wa + is the number of tokens per neuron. sR and Wn. control the size and overlap of the token
sn
groups.
We augment each token with learned identity embeddings for its neuron, session, and animal, similar to Azabou et al. (2025). To limit the number per-neuron parameters and decouple from model size Antolík et al. (2016); Klindt et al. (2017); McIntosh et al. (2016), each embedding table has fixed dimension D embeed = 128 and a corresponding linear projection W IR Danas x Danbal into model space. We flatten the neuron and temporal dimensions and obtain:
ID W,E.(NN) W,E,(1) W.E.()) P.Xxxannts
Z Flatten (XR + ID) E T PTTx Daiae
(2)
where N, E R contains neuron IDs. The masking configuration for neural activity specifies which tokens to mask and which to reconstruct. These masked and target tokens removed from Zn
4

---

Published as a conference paper at ICLR 2026
Mosked responses
Maskad umibdddings
Neuronal reeponses
umbsaldings
Concalonate
multi moliiammtall features
Identity embeddingg
Gross-
Respamss
Behavior
Aciiaal
Gross- Attenttion Decoder
Transformer
Bataniier projection
Visual Stimutus
VIT pasiian emboddinng
Figure 3: Model architecture. OmniMouse introduces unified framework that handles arbitrary combinations of neural forecasting, sub-population prediction, stimulus encoding, and behavioral decoding through flexible masking. We adopt single-neuron, single-time-chunk tokenization and a cross-attention encoder (following POYO+ (Azabou et al. (2025)), along with analogous queries the multi-modal cross-attention decoder. step-by-step summary of the training procedure is provided in Appendix A.4
to create the encoder input Z E (R.TT [R(..T-Rmaskeel Pmasksd) Dasiel For target tokens, we construct decoder queries by simply using their identity embeddings, yielding QR D IR musss X Each query reconstructs the samples uniquely captured by the corresponding token. In addition to its embedding vector, each token or query assigned timestamp indicating its relative temporal position within the chunk context. For neuronal response embeddings, we use the timestamp from the first of its 8n samples.
Visual stimuli. Given Sv frames Sx of grayscale video, we use the first ten layers of randomly-nittaalized initialized Hiera vision transformer Ryali et al. (2023) followed by a linear projection to extract spatiotemporal embeddings Zv E H Sy DDanee where H', W", and Sy result from Hiera's s convolutional patch embedding and hierarchical pooling.
The masking configuration specifies a contiguous unmasked frame range to encode. Because Hiera's early stages use local temporal attention, each output token has well-defined temporal window, yielding the final encoder input, Zv. Each video embedding is assigned the timestamp from the first frame within its temporal receptive field.
Behavioral traces. For Sg samples from each of the five behavioral variables, construct sin- gle behavior tensor (IR (B)xS$), which is either completely masked or completely unmasked. When unmasked, we use shared linear layer to project the traces along the temporal dimension and add learned channel-specific embeddings (as well as the appropriate session and animal ID embeddings), yielding encoder input sequence ZB 5x Punda, When behavior masked, we instead use as target and construct queries with the channel and session/animal ID embeddings QB E R25x Dadds, which are passed to the shared decoder to reconstruct each channel's 's full trace. Because each behav- ioral token/query captures the entire context window, we simply assign them to timestamp 0.
4.2 MASKING
We train OmniMouse with 119 structured masking configurations (Tab. (S6). including our core evaluation tasks (forecasting, popula- tion prediction, stimulus encoding, and behavioral decoding) as well as numerous systematic variations that reduce or combine context across modalities.
Neuronal population ation size
Prediction target
buffer gap
context
Population context
Neural responses. We define consistent prediction target used by all masking configurations-thee final second (30 samples) of activity from 3072 randomly selected neurons and vary unmasked context along two axes: Population context provides contemporane- ous activity from a subset of non-targ non-target neurons, while causal con- text provides temporally preceding activity from all neurons. We
0.0s
Figure 4: Masking of neu- ronal responses
5

---

Published as a conference paper at ICLR 2026
also combine masking along these axes (i.e., provide causal context from only subset of neurons), forcing the model to interpolate along both the population and temporal dimensions. To maintain consistent shapes across sessions, we first randomly sample 4096 neurons (of which 3072 become targets) per session before applying masks. Finally, we enforce a 5-sample buffer gap Fig. between causal context and prediction target to prevent any upsampling artefacts.
Visual stimuli. As noted above, we define a visible frame range, and vary the start time and duration. Frames preceding the target enable forecasting from video, while contemporaneous frames enable stimulus encoding tasks.
Behavior. We either fully provide behavioral traces as encoder input fully mask them and use them as decoder reconstruction targets.
4.3 MODEL
After tokenization, we process the multi-modal embeddings through three stages: (I) encoder that compresses unmasked neuronal and behavioral tokens into a fixed-length latent sequence, (2) a multi- modal fusion stack that integrates these latents with features of the visual stimulus, and (3) decoder that reconstructs target neuronal responses and behavioral traces from the fused representation. Full hyperparameter specifications are provided in App. A.
Encoding, First, we concatenate the unmasked neuronal response and behavioral embeddings into single input sequence. ZRB Z ZB| Note that depending on the masking configuration, the length of Zn will vary and Z may be absent (when hehavior masked). Following Azahou et al. (2023 2025), we compress this variable-length sequence into fixed set of latent embeddings via cross-attention with learned queries. We use M unique query embeddings, repeating each N times, for N' different timestamps uniformly spaced across the context window, yielding M x query embeddings total. To each of the unique embeddings, we also add the session and animal identity embeddings. We use a slightly larger M than in prior work to avoid an information bottleneck with larger neural populations.
Within the cross-attentin block, we implement local sliding-windn wiitio attention: each query and input token is assigned a local temporal window (with length determined by its modality), and attention is masked between any pair of features whose windows do not overlap. To retain global information flow and avoid attention sinks, we additionally append G global register tokens (Darcet et al.) 2023) to the query embeddings, which attend to the entire key sequence. Depending on the masking configuration, remove query repeats for portions of the context window with overlapping input features.
Multi-modal fusion. We concatenate the encoded latent sequence with any unmasked video features from the Hiera encoder and pass the result through stack of L, multi-modal transformer layers (Tab. S5 consisting of self-attention and a forward network. We interleave layers with local sliding-window attention and global layers with no attention masking at a ratio of 5:1. This design allows efficient processing of long sequences while periodically enabling full cross-modal and long- range temporal interaction. All transformer layers throughout the model (including the encoding and decoding cross attention blocks) use ID-RoPE (Su et al. 2024) computed from each token's ' timestamp, encode relative timing both within and across modalities.
Decoding, We extend the cross-attention decoder of Azabou et al. (2025) to support neuronal re- sponse prediction alongside behavioral decoding- The fused multi- mudii- features from the final transformer layer serve as keys and values in a shared decoder cross- cross-attention block. As queries, we concatenate the neuronal target queries QR and, when behavior a prediction target, the behavioral queries QB. No explicit task modality embeddings are added. This pross-attention also uses local causal
attention, where each query attends only to latent features within its tempo- ral neighborhood. After cross-attention and shared feed-forward layer, the outputs routed to modality modalltt-speccifc linear readouts that project from das back to the target dimensionality: & samples per neuronal query and S. behavioral samples per behavioral query.

---

Published as a conference paper at ICLR 2026
Forecasting
Population prediction
Stimulus encading
Forecasting
r stimulus
Population stimuus
Behasior
marked
Video
masied
masked
Nearonal responses
maskied
masked
masked
masked
masked
target
Tostless
.10
Single correaatio
24
33
31.
.40
-35
.3-
1M
-00
23
-35
3M
-50
-10
20M
.10
.25
50M
201H 2014
10.50
10
1234
10.34
3
10
10th
10TH 10
10"
30 1.10 10.00
10
10
34
10
10a
Flops
Figure 5: Task-specific performance gains with model scaling. Top row: masking schema. Middle row: Test loss. Bottom row: single-trial correlation. A. Forecasting; predicting one second neuronal activity, conditioned only on past neuronal activity B. Population prediction; conditioned on sub- population of N = 1024 neurons. C. Stimulus-encoding Neuronal encoding conditioned on the visual stimulus. D. Stimtlus-conditionee forecasting: same as forecasting, but also conditioned on the visual stimulus. E. siimulu--condttione population prediction; same as population prediction (N 1024 neurons in population context), with additionally provided context of visual stimulus.
4.4 TRAINING
We trained our model to predict both neuronal responses and behavioral traces, using Poisson loss (averaged across neurons) for neural encoding and mean squared error (MSE) loss for behavior de- coding. To balance the two objectives, the behavioral loss down- wwwn-weettted by factor of 0.1 so that its scale matches the magnitude of the Poisson loss. For our scaling experiments, we trained models end-to-end on either the complete dataset of 323 sessions or on constructed collections of 8, 16, 32 or 64 sessions to study data scaling effects. These nested collections were designed so that larger collections always contained all sessions from smaller oncs, ensuring consistent evaluation.
We followed Hu et al. 2024); Wen et al. (2024); Hägele et al. (2024) and trained our model with warmup followed by a constant learning rate for at least 250k steps, saving checkpoints every 20k steps. This warmup-stable training phase serves dual purpose: it trains the model convergence while simultaneously producing a dense series of intermediate checkpoints that span a wide range of compute budgets. To obtain clean final evaluation point from each checkpoint, training from each saved checkpoint for an additional 10k steps using an rate decay, which anneals the learning rate near-zero and allows the model to settle into local optimum. Each such decayed checkpoint then yields one evaluation point in our scaling plots. This procedure allows us to densely populate the compute axis of our scaling curves without training separate models from scratch at each compute budget. See App. A.5 for infrastructure details.
continue
learning
5 UNIFIED EVALUATION FRAMEWORK
All scaling experiments use a standardized evaluation protocol on the same mice to ensure fair com- parison across models, baselines, and conditions. We chose seven mice (evaluation mice) comprised of five publicly available datasets from SENSORIUM 2023 and two test mice from SENSORIUM 2022. For all analyses, we use the held-out set provided by these datasets. As in model training, we define a consistent prediction target used by all evaluation configurations: the final second (30 sam- ples) of neuronal activity of randomly chosen 3072 neurons in each session are the prediction target. We evaluate five response prediction tasks (Fig. 5) well as a behavior decoding task (Fig. 6
Predicting one second of neuronal activity given the previous second (i.e. one second of causal context only). We use MtM (Zhang et al. 2024). model that supports causal and popula- tion masking. as the baseline.
Forecasting.
Population prediction. Provided with a population context of unmasked N 256 neurons over the full 2-second context window, predict a target population. This task assesses how much of the trial-to-trial variability can be explained by simultaneously recorded neurons. As in the forecasting
7

---

Published as a conference paper at ICLR 2026
B
Average oss
locaton
Pupil
Running speed
Behavior Decoding
Behavior decoding
masking configuration
1M
Single thal coreecoollee comelation
.75
.0
predict behardor
Testless lest
20M1
beltastor marked
40
8561
pmporriins
355
A0
.30
2014
2014
1,000
1044
2014
10:4
10.0
1044
2014
10145
10%0
1014
10.0
1009
10th
1021
Flops
Flops
Flops
Flops
Figure 6: Behavior decoding scales with model size. A. Masking for behavior decoding. B. De-
coding loss averaged across all behaviors variables. C-E. Prediction performance for behavioral decoding.
task, we chose MIM (Zhang et al. 2024) as the baseline model. For the scaling analysis in Fig. we provide N 1024 neurons population context.
Stimulus-encoding. Given the full two seconds visual stimulus, predict neuronal activity for the final second. The neuronal encoding competitions SENSORIUM 2023 (Turishcheva et al. 2024)
and SENSORIUM 2022 (Willeke et al. 2022) establish strong baselines for this task. Stimulus-conditionee forecasting. This task additionally provides the full seconds of visual stimulus as context to the forecasting task. We use use Schmidtt et al. (2025) as a baseline model, which also conditions response prediction on neurons and video.
Stimulus-conditionee population prediction. With the full two seconds of visual stimulus con- text in addition to N 256 unmaksed neurons. Again, we use (Schmidt et al. (2025) as the baseline. Behavior decoding. Given the activity of all neurons for the full two seconds, predict all behavioral traces. CEBRA (Schneider et al. 2023) and Poyo+ (Azabou et al. 2025) are used as baselines.
For fair comparison, we train the baselines on the same amount of data OmniMouse. We use the smallest collection of eight mice from data-scaling experiment (Fig. 7 as a data-matched compari- son. Implementation details and hyperparameters for each bascline are provided in App. B For all experiments, consistent with SENSORIUM 2022/2023 competitions, we opt for single-trial correla- tion, i.e. the pearson correlation between ground truth and predicted signal, the single evaluation metric, without any correction for the noise ceiling. This measure allows for an unbiased comparison of predictivity across all tasks and conditions.
RESULTS: THE BENEFITS OF SCALING
models are compute- or parameter-limited. Because collecting neuronal data is costly, we first asked if existing models are already limited by compute parameters, or if more data would still improve performance. To answer this question, we trained models on all 323 sessions while scaling width and depth as in Tab. $5 We evaluated five neuronal response mask- ing strategies (Fig. 5 top row): two based on response dynamics (forecasting and population con-
Current
Table I: Baseline comparisons. Results displayed in bold indicate the highest score per task in either the data-matched condition (top) or when using the full dataset (bottom). Baselines were evaluated for all conditions that they support, with X denoting unsupported condition. Conditions: Forecasting (Fest), forecasting + stimulus (Fest+S),, population prediction (Pop) with n = 256 visible neurons, population prediction stimulus (Pop+S). Behavioral decoding: Average across behaviors (Avg), pupil-location (Gaze), pupil-size (Pupil), and rumning speed (Running).
Neuronal Activity Prediction Pop+S
Behavior Decoding
Model
Fest Fest+$
Pop
Avg
Gaze
Pupil
Running
MtM (Zhang et al. 2024]
0.12
X
0.07
X
X
X
X
X
8 sessions
Latent Model (Schmidt 2025
0.18
X
0.16
X
X
X
CEBRA (Schnekder et al. 2023
X
X
X
0.53
0.52
0.55
0.51
POYO+ Azabou et al. 2025
X
X
X
0.55
0.56
0.63
0.47
OmniMouse-5M
0.18
0.25
0.25
0.27
0.59
0.68
0.66
0.44
323 sessions
OmniMMuue-IM
0.18
0.31
0.27
0.35
0.68
0.75
0.73
0.55
OmniMMuueeSMM
0.22
0.32
0.28
0.35
0.69
0.76
0.74
0.57
OmniMouss-20N,
0.23
0.33
0.29
0.37
0.75
0.78
0.75
0.73
OmniMouse-SOM
0.25
0.34
0.29
0.37
0.77
0.80
0.76
0.75
OmniMouse-300M
0.25
0.34
0.30
0.37
0.76
0.80
0.76
0.73
8

---

Published as a conference paper at ICLR 2026
Table 2: Sensorium bench- mark results. Compari- son against Sensorium 2022 and 2023 competition winners across both tracks. See ta- ble SS7 for additional compar- isons.
Sensorium 2022
Sensorium 2023
Main
Bonus
Main
Bonus
Competition Winner
0.33
0.45
0.29
0.22
OmniMouse-80M
0.37
0.45
0.33
0.30
text), two analogous variants that additionally condition on video video-condittoned forecasting and video-conditioned population context), and one stimulus-encoding strategy (video & behavior). For each strategy, models ranged from 1M to 300M parameters, and we tracked both test loss and single- trial correlation as function of total compute (model FLOPs, excluding FLOPs of neuron-specific parameters). Performance improved across all neuronal prediction tasks as model size increased up to 80M parameters (Fig. 5). Beyond this point, gains were minimal, as loss curves saturated overfit, indicating that current models are data-limited rather than compute- or centmmeer-limiteee
OmniMouse achieves state-of-the-art performance. Our large-scale model outperforms all base- lines across six evaluation regimes for both response prediction and behavior decoding (Tab. D Cru- cially, these gains are not simply due to training on more data: in data- matched comparisons, where identical datasets, our model outperforms
OmniMouse and baselines are trained and evaluated strong specialized methods across all tasks except for decoding of running speed. This demonstrates that the architectural and masking design of OmniMouse provides advantages independent of data scale. OmniMouse also sets a new state the art on the Sensorium 2022 and 2023 competitions (Tab. 2. For the Sensorium 2023 competition, backhone and only neuron-specific parameters trained, and (2) with full end-to- end training on the same 10-mouse competition dataset (see Tab. [Sr]for details), and surpass the prior state of the art in both cases. We note that while our data-matched setting uses the same training data, our framework additionally enables training across video boundaries an information not available for the previous models.
evaluate model (1) with frozen pretrained
OmniMouse-8ii
Behavior prediction shows the most promising scaling dynamics on the available data. To char- acterize the scaling of behavior decoding, we used the same models and evaluated their ability to predict pupil location, pupil size. and running speed from neuronal activity only (Fig. Across all three settings, performance improved smoothly with compute budget, reminiscent of classic scaling- law behavior. Larger models consistently achieved higher single-trial correlations, albeit with an indication of saturation at the largest scale tested. The models had not yet fully converged for the behavioral decoding task, and longer training could have improved performance further, even on the largest model. These results show that behavioral prediction continues to improve with model scaling and may benefit from further increases in capacity.
Scaling dataset size improves performance. To study how dataset size affects performance, trained three model sizes- 5M, 20M, and 80M - on nested collections of 8, 16, 32, 64, and 323 sessions, such that the larger collections are supersets of the smaller ones (Fig. 7 For evaluation, we test the model on the same held-out test set of the same seven mice that were contained in all
Nested dataset structure
test
trial conrecation
Mupillocation
Pupil
Hunning speed
.25
3M
.55
Number of recording sessions
8
16
32
64
5.5
1049
20M
32
50M
64
16
64
323
Hopp
Number recording sessions
G
H
Single tral trancoobbatnnn
Forocasting
Population context
Stimulus-chiven
Forecasting stimuuus
Populat contest stimulus
24
35
27
20
.22
16
32
64
323
16
32
64
323
8
16
32
64
323
16
32
64
323
16
32
64
323
ofrecordinggggoooooooooooooooin sessions
Figure 7: Scaling data improves model performance. A. Nested datasets structure. B. Test loss for different model and data sizes, averaged across all response prediction tasks. C-K. Performance improvements for all tasks when scaling dataset from 8 to 323 sessions.

---

Published as a conference paper at ICLR 2026
collections (Fig. (C-)). In all cases, performance improved with the number of sessions, exhibiting predictable lata-scaling trends. Larger models consistently benefited more from additional data. The larger models required a minimum size of the training set to outperform the smaller models and the performance gap widened as the dataset increased in size. Behavior decoding benefited the most from data sealing (Fig. (Fig. [F--), showing no saturation and large performance differences between 5M and 80M models. For responses, the strongest gains were observed for tasks that included video input (Fig. (CE)), where the 80M models continued to improve even beyond 100 sessions, suggesting that they remained
rather than capacity- limited. forecasting and population context tasks showed larger benefits from scaling of both data and model sizes. The gaps between 20M and 80M models (Fig. Z B) increased faster compared to the tasks with video input, which could indicate lack of diversity of the visual stimuli in our dataset. Overall, these results highlight that scaling both model size and data quantity is synergistic and necessary to approach peak predictive performance.
OmniMouse enables systematic evaluation of how neuronal context shapes predictive perfor- mance. Lastly, we assessed the model's generalization by testing masking conditions not seen during training (Fig. ST varying causal and population context size. Performance scaled smoothly with additional context, demonstrating that OmniMouse learns generalizable representations that en- ables systematic analyses of contextual contributions to neural variability (see Fig. [S2] Fig. [S3] and App.C). The ability to flexibly control and combine different sources of context within single model opens an exciting avenue for future study, enabling precise quantification of how each factor shapes neural variability and their interactions across brain areas, stimuli, and behavioral states.
DISCUSSION
In this work, we introduce OmniMouse, a multi-modal, multi-task model of mouse visual cortex that jointly integrates neural activity, video stimuli, and behavioral traces across animals within a single unified architecture. Trained on one of the largest single-neuron dataset to date, OmniMouse achieves state-of-the-art performance across tasks, outperforming strong specialized baselines on neural response prediction, activity forecasting, and behavioral decoding. The breadth and scale of both model and dataset enable systematic study of scaling behavior in single-neuron brain models.
Our motivation for studying scaling laws is practical: if brain models are to become foundation models for neuroscience, it is essential to ask whether current data can sustain scaling. Despite using naturalistic movies and images, we find that performance saturates with model size, suggesting data- not compute- as the limiting factor. This is consistent with recent observations that scaling benefits in neural models are constrained by data variability rather than model capacity (Jiang et al.. 2025] Ye et al. 2025 Gokce & Schrimpi 2024). Even in the relatively simple mouse visual system, richer tasks, more varied stimuli, and larger-scale recordings are needed to support continued scaling. At the same time, relatively sparse sampling already yields strong models: with 60,000 neurons from eight mice, predictive accuracy is high, likely due to redundancy in neural codes (Stringer et al. 2019). Additional gains from larger datasets appear modest, paralleling language (Kaplan et al. 2022 Hoffmann et al.) 2022)- yet in those domains, small improvements have triggered phase transitions to qualitatively new abilities (Wei et al. 2022]. By analogy, richer neuroscience data may similarly unlock new capabilities in brain models, revealing deeper principles of neural computation.
Limitations. Our work has several limitations. First, OmniMouse parameters scale linearly with the number of neurons, as it learns per-neuron embeddings. This makes training computationally expensive and may limit scaling to larger datasets. Second, large-scale transformers remain difficult to interpret and prone to overparameterization issues, which constrain the biological insights that can be drawn. Lastly, the behavioral data present in our data is limited to spontaneous activity and is thus unclear if this approach can transfer to more complex behaviors.
Future work. Future work could extend to stimulus decoding (Benchetrit et al. 2023 Bauer et al. 2024
Zhu et al. 2025) and more precise study of training dynamics of modality interactions and tuski-task learning to improve the masking recipe. Beyond calcium imaging in mouse visual cor- tex, models could integrate other data types such electrophysiological recordings, diverse animal species, and more multi-modal stimuli such as audio. Finally, jointly modeling visual input, neuronal responses, and behavior enables analysis of spontaneous and evoked activity (Stringer et al., 2019). revealing how brain state shapes sensory processing and core principles of computation.
10

---

Published as a conference paper at ICLR 2026
REPRODUCIBILITY STATEMENT
To ensure the reproducibility of our results, we provide the complete source code for our multi-modal model, including scripts for training, evaluation, fine-tuning, and inference, available at httth:://// github. com/eni gma- brain/ omnimouse upon publication. Regarding the dataset, which consists of large-scale neuronal responses from the visual cortex and naturalistic visual stimulation, we have detailed the data acquisition and processing pipeline Appendix E. While the full dataset currently undergoing final preparation due to its unprecedented scale, we are committed to releasing publicly within six months.
ACKNOWLEDGEMENTS
This work generously supported by The James Fickel Enigma Project Fund. AST acknowl- edges the support of the National Science Foundation and of DoD OUSD (R&F) under Cooperative Agreement DBI-2229929 (The NSF AI Institute for Artificial and Natural Intelligence). This work was also supported by the Intelligence Advanced Research Projects Activity (IARPA) via Department of Interior/Interior Business Center (Dol/IBC) contract number DI6PC00003. This work was also supported by the German Research Foundation (SFB 1233, Robust Vision: Inference Principles and Neural Mechanisms, project number 276693517; SFB 1528 Cognition of Interaction, project number 454648639; project IDs 432680300 (SFB 1456, project B05) and 515774656, the European Research Council (ERC) under the European Union's Horizon Europe research and innovation programme (grant agreement numbers 101041669 and 1011715261 FHS is further supported by the German Federal Ministry of Education and Research (BMBF) via the Collaborative Research in Computa- tional Neuroscience (CRCNS) (FKZ 01GQ2107). FHS and ASE acknowledge the support of the Ministry of Science and Culture of Lower Saxony through funds from the program zukunft.niede sachsen of the Volkswagen Foundation for the "CAIMed Lower Saxony Center for Artificial Intel- ligence and Causal Methods in Medicine" project (grant no. ZN4257). This research used resources of the Oak Ridge Leadership Computing Facility at the Oak Ridge National Laboratory. which is supported by the Office of Science of the U.S. Department of Energy under Contract No. DE-AC05- 000R22725, KFW and AG sincerely thank Craig Kapfer, Kurt Stine, and the whole Marlowe team at Stanford, for the support and early access to the supercomputing cluster. We are also thankful for the support of Zoe Ryan and Bruce McGowan and the NVIDIA support team for guidance during all stages of this project. PT, MAW and MV thank the computing time granted by the Resource Allo- cation Board and provided on the supercomputer Emmy//rete at NHR-Nord @ Göttingen as part of the NHR infrastructure. Additional compute for this work was conducted with computing resources under the projects nim00010 and nim00012. We also thank Surya Ganguli, Dan Yamins, and the entire team of the Enigma Project at Stanford for helpful discussions.

---

Published as a conference paper at ICLR 2026
REFERENCES
Josh Abramson, Jonas Adler, Jack Dunger, Richard Evans, Tim Green, Alexander Pritzel, Olaf Ron- neberger, Lindsay Willmore, Andrew Ballard, Joshua Bambrick, al. Accurate structure pre- diction of biomolecular interactions with alphafold 3. Nature, 630(8016)4935550,, 2024.
Dora Angelaki, Brandon Benson, Julius Benson, Daniel Birman, Niccolò Bonacchi, Kcénia Bougrova, Sebastian A Bruijns, Matteo Carandini, Joana A Catarino. et al. A brain-wide map of neural activity during complex behaviour. Nature, 145(8079)1177-11,, 2025.
Ján Antolík, Sonja Hofer, James A Bednar, and Thomas D Atsie-Flogel. Model constrained by
visual hierarchy improves prediction of neural responses to natural scenes. PLoS computational biology, (2(6):e 1004927, 2016.
Richard Antonello, Aditya Vaidya, and Alexander Huth. Scaling laws for language encoding models in fmri. Advances in Neural Information Processing Systems, 36:21895-2199 2023.
Antonis Antoniades, Yiyi Yu, Joseph Canzano, William Wang. and Spencer La Smith. Neu- roformer: Multimodal and multitask generative pretraining for brain data, 2024. URL https:
7/arxiv. org /bss//3/111 20b.s//21100166. 00136,
Mehdi Azabou, Vinam Arora, Venkataramana Ganesh, Ximeng Mao, Santosh Nachimuthu, Michael Mendelson, Blake Richards, Matthew Perich, Guillaume Lajoie, and Eva Dyer. A unified, scalable framework for neural population decoding. Advances Neural Information Processing Systems, 36:44937-44956, 2023.
Mehdi Azabou, Krystal Xuejing Pan, Vinam Arora, Ian Jarratt Knight, Eva L Dyer, and Blake Aaron Richards. Multi-session, multi-task neural decoding from distinct cell-types and brain regions. In The Thirteenth International Conference Learning Representations, 2025.
Mohammad Bashiri, Edgar Walker, Konstantin-Klemens Lurz, Akshay Jagadish, Taliah Muhammad, Zhiwei Ding, Zhuokun Ding, Andreas Tolias, and Fabian Sinz. flow-based latent state gener- ative model of neural population responses to natural images. Advances in Neural Information Processing Systems, 34:15801-15815, 2021.
Eleanor Batty, Josh Merel, Nora Brackbill, Alexander Heitman, Alexander Sher, Alan Litke, EJ Chichilnisky, and Liam Paninski. Multilayer recurrent network models of primate retinal gan- glion cell responses. International Conference on Learning Representations, 2017.
Joel Bauer, Troy W Margrie, and Claudia Clopath. Movie reconstruction from mouse visual cortex activity. bioRxiv, pp. 2024-06, 2024.
Yohann Benchetrit, Hubert Banville, and Jean-Rémi King. Brain decoding: toward real-time recon- struction of visual perception. arXiv preprint 2111v:2310.19812 2023.
David H Brainard and Spatial Vision. The psychophysics toolbox. Spatial vision, 10(4):433-436. 1997.
Max F Burg, Santiago A Cadena, George H Denfield, Edgar Y Walker, Andreas S Tolias, Matthias Bethge, and Alexander S Ecker. Learning divisive normalization in primary visual cortex. PLOS Computational Biology. 17(6)xe0000028, 2021.
Santiago A. Cadena, George H. Denfield, Edgar Y. Walker, Leon A. Gatys, Andreas S. Tolias, Matthias Bethge, and Alexander S. Ecker. Deep convolutional models improve predictions of macaque responses to natural images. PLOS Computational Biology, 15(4)ee100687, April 2019. doi: 10. 1 1006897.
Charles F Cadieu, Ha Hong, Daniel I. K Yamins, Nicolas Pinto, Diego Ardila, Ethan A Solomon, Najib J Majaj, and James DiCarlo. Deep neural networks rival the representation of primate IT cortex for core visual object recognition. PLoS Comput. Biol., 10(1 10112):e1003963, 2014.
Salvatore Calcagno, Isaak Kavasidis, Simone Palazzo, Marco Brondi, Luca Sitå, Giacomo Turri, Daniela Giordano, Vladimir R. Kostic, Tommaso Fellin, Massimiliano Pontil, and Concetto Spampinato. Quantformer: Learning to quantize for neural activity forecasting in mouse visual cortex, 2024. URL ltttss///axxii: http:://axxvvv.oroeeee wwtp:/orr//bbb/...///222/
12

---

Published as a conference paper at ICLR 2026
Josue Ortega Caro, Antonio H de O Fonseca, Christopher Averill, Syed A Rizvi, Matteo Rosati, James L Cross, Prateek Mittal, Emanuele Zappala, Daniel Levine, Rahul M Dhodapkar, et al. Brainlm: A foundation model for brain activity recordings. bioRxiv, pp. 2023-09, 2023.
Geeling Chau, Christopher Wang, Sabera Talukder, Vighnesh Subramaniam, Saraswati Soedarmadji, Yisong Yue, Boris Katz, and Andrei Barbu. Population transformer: Learning population-level representations of neural activity. ArXiv, pp. arXiv-2406, 2024.
Yuqi Chen, Kan Ren, Kaitao Song, Yansen Wang. Yifan Wang, Dongsheng Li, and Lili Qiu. Eeg- former: Towards transferable and interpretable large-scale eeg foundation model. arXiv preprint arXiv:2401. 10278, 2024.
BR Cowley and JW Pillow. work models of visual cortex. In H. Larochelle, M. Ranzato, Hadsell, M.F. Balcan, and Lin (eds.), Advances in Neural Information Processing Systems 33, pp. 21591-21603, Curran Asso- ciates, Inc., 2020.
"gaudy" images improve the training of deep neural net-
Richard Csaky, Mats WJ van Es, Oiwi Parker Jones, and Mark Woolrich. Foundational gpt model for meg. arXiv preprint Til1:2404.092555 2024.
Wenhui Cui, Woojae Jeong, Philipp Thölke, Takfarinas Medani, Karim Jerbi, Anand A Joshi, and Richard M Leahy. Neuro-gpt: Towards foundation model for eeg. In 2024 IEEE International Symposium on Biomedical Imaging (ISBI), pp. 1-5. IEEE, 2024.
Timothée Darcet, Maxime Oquab, Julien Mairal, and Piotr Bojanowski. Vision transformers need registers. arXiv preprint arXiv:2309. 16588, 2023.
Stéphane d'Ascoli, Jérémy Rapin, Yohann Benchetrit, Hubert Banville, and Jean-Rémi King- Tribe: Trimodal brain encoder for whole-brain fmri response prediction. arXiv preprint arXiv:2507.22229 2025.
Saskia F. J. de Vries, Jerome A. Lecoq, Michael A. Buice, Peter A. Groblewski, Gabriel K. Ocker, Michael Oliver, David Feng, Nicholas Cain, Peter Ledochowitsch, Daniel Millman, Kate Roll, Marina Garrett, Tom Keenan, Leonard Kuan, Stefan Mihalas, Shawn Olsen, Carol Thompson, Wayne Wakeman, Jack Waters, Derric Williams, Chris Barber, Nathan Berbesque, Brandon Blan- chard, Nicholas Bowles, Shiella D. Caldejon, Linzy Casal, Andrew Cho, Sissy Cross, Chinh Dang, Tim Dolbeare, Melise Edwards, John Galbraith, Nathalie Gaudreault, Terri L. Gilbert, Fiona Griffin, Perry Hargrave, Robert Howard, Lawrence Huang, Sean Jewell, Nika Keller, Ulf Knoblich, Josh Larkin, Rachael Larsen, Chris Lau, Eric Lee, Felix Lee, Arielle Leon, Lu Li, Fuhui Long, Jennifer Luviano, Kyla Mace, Thuyanh Nguyen, Jed Perkins, Miranda Robertson, Sam Seid, Eric Shea-Brown, Jianghong Shi, Nathan Sjoquist, Cliff Slaughterbeck, David Sulli- van, Ryan Valenza. Casey White, Ali Williford, Daniela M. Witten, Jun Zhuang, Hongkui Zeng, Colin Farrell, Lydia Ng, Amy Bernard, John W. Phillips, R. Clay Reid, and Christof Koch. A large-scale standardized physiological survey reveals functional organization of the mouse visual cortex. Nature Neurosciemce, 23( 133 December 2019. doi: (038///4193-009-00000 URL https://doi...... org/
Zhiwei Ding, Dat T. Tran, Kayla Ponder, Zhuokun Ding, Rachel Froebe, Lydia Ntanavara, Paul G. Fahey, Erick Cobos, Luca Baroni, Maria Diamantaki, Eric Y. Wang, Andersen Chang, Stelios Pa- padopoulos, Jiakun Fu, Taliah Muhammad, Christos Papadopoulos, Santiago A. Cadena, Alexan- dros Evangelou. Konstantin Willeke, Fabio Anselmi, Sophia Sanborn, Jan Antolik, Emmanouil Froudarakis, Saumil Patel, Edgar Y. Walker, Jacob Reimer, Fabian H. Sinz, Alexander S. Ecker, Katrin Franke, Xaq Pitkow, and Andreas S. Tolias. Bipartite invariance in mouse primary visual cortex. bioRxiv, 2025a. doi: 10.1101/2023.03 15.5222236.....566... URI. htt.hhttts///wwtt www.:///w//www........o.o///.... org/ content/earl] 2025/ 0155/04/150 19/2023 13. 532836.
Zhuokun Ding, Paul G Fahey, Stelios Papadopoulos, Eric Y Wang, Brendan Celii, Christos Pa- padopoulos, Andersen Chang, Alexander B Kunin, Dat Tran, Jiakun Fu, et al. Functional con- nectomics reveals general wiring rule in mouse visual cortex. Nature, 440(8058):4594400 469, 2025b.
Zijian Dong, Ruilin Li. Yilei Wu, Thuan Tinh Nguyen, Joanna Chong, Fang Ji, Nathanael Tong, Christopher Chen, and Juan Helen Zhou. Brain-jepa: Brain dynamics foundation model with
13

---

Published as a conference paper at ICLR 2026
gradient positioning and spatiotemporal masking. Advances in Neural Information Processing Systems,
2024.
Yu Duan, Hamza Tahir Chaudhry, Misha B Ahrens, Christopher D Harvey, Matthew G Perich, Karl Deisseroth, and Kanaka Rajan. Poco: Scalable neural forecasting through population conditioning. arXiv preprint arXiv:2506.1 010222006.14557 14957, 2025.
Alexander S. Ecker, Fabian Sinz, Emmanouil Froudarakis, Paul G. Fahey, Santiago A. Cadena, Edgar Y. Walker, Erick Cobos, Jacob Reimer, Andreas S. Tolias, and Matthias Bethge. A rotation- equivariant convolutional neural network model primary visual cortex. arXiv, 2018.
Paul G Fahey, Taliah Muhammad, Cameron Smith, Emmanouil Froudarakis, Erick Cobos, Jiakun Fu, Edgar Y Walker, Dimitri Yatsenko, Fabian H Sinz, Jacob Reimer, et al. global map of orientation tuning in mouse visual cortex. BioRXiv, pp. 745323, 2019.
Emmanouil Froudarakis, Philipp Berens, Alexander S Ecker, R James Cotton, Fabian H Sinz, Dimitri Yatsenko, Peter Saggau, Matthias Bethge, and Andreas S Tolias. Population code in mouse VI facilitates readout of natural scenes through increased sparseness. Nat. Neurosci., 17(6):851-877, June 2014.
Marina E Garrett, Ian Nauhaus, James H Marshel, and Edward M Callaway. Topography and areal organization of mouse visual cortex. J. Neurosei., 34(3))
September 2014.
Andrea Giovannucci, Johannes Friedrich, Pat Gunn, Jérémie Kalfon, Brandon Brown, Sue Ann Koay, Jiannis Taxidis, Farzanch Najafi, Jeffirey L Gauthier, Pengcheng Zhou, et al. Caiman an open source tool for scalable calcium imaging data analysis. elife, 8:e38173, 2019.
Abdulkadir Gokce and Martin Schrimpf. Scaling laws for task-optimized models of the primate visual ventral stream. arXiv preprint S.GG::22111007712 2024.
Alexander Hägele, Elic Bakouch, Atli Kosson, Loubna Ben Allal, Leandro Von Werra, and Martin Jaggi. Scaling laws and compute-optimal training beyond fixed training durations. In A. Glober- son, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomezak, and C. Zhang (eds.), Advances in Neural Information Processing Systems, volume 37, pp. 76232-76264. Curran Associates, Inc., 2024. URL bttpp//tps/// neurips cc/paper files/ (01les/paper2222///11 pdf.
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Ruther- ford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training compute-optimal large language models. arXiv preprint MrXXi:2203315555 2022.
Shengding Hu, Yuge Tu, Xu Han, Chaoqun He, Ganqu Cui, Xiang Long, Zhi Zheng, Yewei Fang, Yuxiang Huang, Weilin Zhao, al. Minicpm: Unveiling the potential of small language models with scalable training strategies. arXiv preprint 2xXiv:2404.06355 2024.
Linxing Preston Jiang, Shirui Chen, Emmanuel Tanumihardja, Xiaochuang Han, Weijia Shi, Eric Shea-Brown, and Rajesh PN Rao. Data heterogeneity limits the scaling effect of pretraining neural data transformers. bioRxiv, pp. 2025-05, 2025.
Wei-Bang Jiang, Li-Ming Zhao, and Bao-Liang Lu. Large brain model for learning generic repre- sentations with tremendous eeg data in bei. arXiv preprint wxxvv:2405.1876 2024.
Keller Jordan, Jeremy Bernstein, Brendan Rappazzo, @femmbearbssky.ooccaal, Boza Vlado, You Ji- acheng, Franz Cesista, Braden Koszarsky, and @ @Gra662304977 modded-nanogott Speedrunning the nanogpt baseline, 2024. URI. wtt.aa/..oooeeeeeeerrrrrraaa///e
nanogpt
Xuan Kan, Wei Dai, Hejie Cui, Zilong Zhang, Ying Guo, and Carl Yang. Brain network transformer. Advances in Neural Information Processing Systems, 35:25586-25599, 2022.
Craig Kapfer, Kurt Stine, Balasubramanian Narasimhan, Christopher Mentzel, and Emmanuel Can- des. Marlowe: Stanford's 's gpu- based computational instrument, 2025. URL //eoooo..
14751899.
14

---

Published as a conference paper at ICLR 2026
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.0823 2020.
Andrej Karpathy, George Toderici, Sanketh Shetty, Thomas Leung, Rahul Sukthankar, and Li Fei- Fei. Large-Scale video classification with convolutional neural networks. In 2014 IEEE Conference on Computer Vision and Pattern Recognition, pp. 1725-1732, June 2014.
William Kindel, Elijah Christensen, and Joel Zylberberg. Using deep learning to probe the neural code for images in primary visual cortex. Journal of vision, 19(4):29 29, 2019.
Mario Kleiner, David Brainard, and Denis Pelli. What's new psychtoolbox-3? 2007.
David Klindt, Alexander S Ecker, Thomas Euler, and Matthias Bethge. Neural system identification for large populations separating "what" and "where". Advances in neural information processing systems, 30, 2017.
Demetres Kostas, Stephane Aroca-Ouellette Ouellette, and Frank Rudzicz. Bendr: Using transformers and contrastive self-supervised learning task learn from massive amounts eeg data. Frontiers in Human Neuroscience, 15:653659, 2021.
Trung Le and Eli Shlizerman. Stndt: Modeling neural population activity with spatiotemporal trans- formers. Advances in Neural Information Processing Systems, 111117926-7999 2022.
Bryan M Li, Isabel M Cornacchia, Nathalie L Rochefort, and Arno Onken. Vlc large-scale mouse v1 response prediction using a vision transformer. arXiv preprint arXiv:2302.03003, 2023.
Yamin Li, Ange Lou, Ziyuan Xu, Shengchao Zhang, Shiyu Wang, Dario Englot, Soheil Kolouri, Daniel Moyer, Roza Bayrak, and Catie Chang. Neurobolt: Resting- state eeg-to-fin reg-to-fimri synthesis with multi-dimensional feature mapping. Advances in Neural Information Processing Systems, 37:23378 2222378-23405. 2024.
Isaac Lin, Tianye Wang, Shang Gao, Shiming Tang, and Tai Sing Lee. Incremental learning and self-attention mechanisms improve neural system identification. arXiv e-prints, pp. arXiv-2406, 2024.
Konstantin Klemenn Lurz, Mohammad Bashiri, Konstantin Willeke, Akshay Jagadish, Eric Wang, Edgar Y. Walker, Santiago A Cadena, Taliah Muhammad, Erick Cobos, Andreas S. Tolias, Alexan- der Beker, and Fabian H. Sinz. Generalization in data-driven models of primary visual cortex. In International Conference Learning Representations, 2021. URL net/forum ? ddd=Tp/99TTttdddd
Alexander Mathis, Pranav Mamidanna, Kevin M Cury, Taiga Abe, Venkatesh N Murthy, Macken- zie Weygandt Mathis, and Matthias Bethge. DaeplaabCut: markerless pose estimation of user- defined body parts with deep learning. Nat. Neurosci., 21(9):1281-12899 September 2018.
Lane T McIntosh, Niru Maheswaranathan, Aran Nayebi, Surya Ganguli, and Stephen A Baccus. Deep learning models of the retinal response to natural scenes. Adv. Neural Inf. Process. Syst., 29 139ps):1369-1377, 1377, 2016.
Lu Mi, Trung Le, Tianxing He, Eli Shlizerman, and Uygar Sümbül. Learning time-invariant rep- resentations for individual neurons from population dynamics. Advances in Neural Information Processing Systems, 36:46007-46026, 2023.
MICrONS Consortium. Functional connectomics spanning multiple areas of mouse visual cortex. Nature, 64008058):435-447, April 2025.
MICrONS Consortium, Alexander Bae, Mahaly Baptiste, Agnes L Bodor, Derrick Brittain, Joann Buchanan, Daniel J Bumbarger, Manuel A Castro, Brendan Celii, Erick Cobos, Forrest Collman, Nuno Maçarico da Costa, Sven Dorkenwald, Leila Elabbady, Paul G Fahey, Tim Fliss, Emmanouil Froudakis, Jay Gager, Clare Gamlin, Akhilesh Halageri. James Hebditch, Zhen Jia, Chris Jor- dan, Daniel Kapner, Nico Kemnitz, Sam Kinn, Selden Koolman, Kai Kuchner, Kisuk Lee, Kai Li, Ran Lu, Thomas Macrina, Gayathri Mahalingam, Sarah McReynolds, Elanine Miranda, Eric
15

---

Published as a conference paper at ICLR 2026
Mitchell, Shanka Subhra Mondal, Merlin Moore, Shang Mu, Taliah Muhammad, Barak Nehoran, Oluwaseun Ogedengbe, Christos Papadopoulos, Stelios Papadopoulos, Saumil Patel. Xaq Pitkow, Sergiy Popovych, Anthony Ramos, R Clay Reid, Jacob Reimer, Casey M Schncider-Mizzll, H Se- bastian Seung. Ben Silverman, William Silversmith, Amy Sterling, Fabian H Sinz, Cameron I. Smith, Shelby Suckow, Zheng H Tan, Andreas S Tolias, Russel Torres, Nicholas I. Turner, Edgar Y Walker, Tianyu Wang, Grace Williams, Sarah Williams, Kyle Willie, Ryan Willie, William Wong, Jingpeng Wu, Chris Xu, Runzhe Yang, Dimitri Yatsenko, Fei Ye, Wenjing Yin, and Szi-Chich Yu. Functional connectomics spanning multiple areas of mouse visual cortex. bioRxiv, pp- 0211007.2..4440222 July 2021.
Shervin Minaee, Tomas Mikolov, Narjes Nikzad, Meysam Chenaghlu, Richard Socher, Xavier Am- atriain, and Jianfeng Gao. Large language models: A survey. arXiv preprint arXiv:4402.0619. 2024.
M C Morrone, M Tosetti, D Montanaro, A Fiorentini, G Cioni, and D C Burr. A cortical area that responds specifically to optic flow, revealed by fMRI. Nat. Neurosci., ((12):1322-133)) December 2000.
Humza Naveed, Asad Ullah Khan, Shi Qiu, Muhammad Saqib, Saeed Anwar, Muhammad Usman, Naveed Akhtar, Nick Barnes, and Ajmal Mian. A comprehensive overview of large language models. ACM Transactions on Intelligent Systems and Technology, 16(5):1-72, 2025.
Denis G Pelli. The videotoolbox software for visual psychophysics: transforming numbers into movies. Spatial vision, 10(4):437-442, 1997.
Nicolai Petkov and Easwar Subramanian. Motion detection, noise reduction, texture suppression, and contour enhancement by spatiotemporal gabor filters with surround inhibition. Biol. Cybern., 97(5-6):423444 439. December 2007.
Pawel A Pierzchlewicz, Konstantin F Willeke, Are F Nix, Pavithra Elumalai, Kelli Restivo, Tori Shinn, Cate Nealley, Gabrielle Rodriguez, Saumil Patel, Katrin Franke, Andreas S Tolias, and Fabian H Sinz. Energy guided diffusion for generating neurally exciting images. In Advances in Neural Processing Systems (NeurIPS 2023), pp. 1023.05.18.5411 176,, May 2023.
Jacob Reimer, Emmanouil Froudarakis, Cathryn R Cadwell, Dimitri Yatsenko, George H Denfield, and Andreas Tolias. Pupil fluctuations track fast switching of cortical states during quiet wake- fulness. Neuron, 84(2):355-362, 2014.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C Berg. and Li Fei-Fei. ImageNet large scale visual recognition challenge. Int. J. Comput. Vis., 1 1513)2111252,, December 2015.
Chaitanya Ryali, Yuan-Ting Hu, Daniel Bolya, Chen Wei, Haoqi Fan, Po-Yao Huang, Vaibhav Aggar- wal, Arkabandhu Chowdhury, Omid Poursaeed, Judy Hoffman, al. Hiera: A hierarchical vision transformer without the anllsslnd-wsssles.... In International conference on machine learning, pp. 29441-29454, PMLR, 2023.
Shreya Saha, Ishaan Chadha, et al. Modeling the human visual system: Comparative insights from response-optimized and task-optimized vision models, language models, and different readout mechanisms. arXiv preprint arXiv:2410. 140.31111, 2024.
Finn Schmidt, Polina Turishcheva, Suhas Shrinivasan, and Fabian H. Sinz. Modeling dynamic neural activity by combining naturalistic video stimuli and simulls-inddeendder latent factors, 2025.
URL httts::///axiii org/ abs///1o....///////// 16136
Steffen Schneider, Jin Hwa Lee, and Mackenzie Weygandt Mathis. Learnable latent embeddings for joint behavioural and neural analysis. Nature, 617777660:360-366 2023.
Fabian Sinz, Alexander Ecker, Paul Fahey, Edgar Walker, Erick Cobos, Emmanouil Froudarakis, Dimitri Yatsenko, Zachary Pitkow, Jacob Reimer, and Andreas Tolias. Stimulus domain transfer in recurrent models for large scale cortical population prediction on video. Advances in neural information processing systems, 31, 2018.
16

---

Published as a conference paper at ICLR 2026
Nicholas James Sofroniew, Daniel Flickinger. Jonathan King, and Karel Svoboda. A large field of view two-photon mesoscope with subcellular resolution resoluuton for in vivo imaging. elife, 5:e14472, 2016.
Carsen Stringer, Marius Pachitariu, Nicholas Steinmetz, Charu Bai Reddy, Matteo Carandini, and Kenneth Harris. Spontaneous behaviors drive multidimensional, brainwide activity. Science, 364(6437) 2019.
Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: En- hanced transformer with rotary position embedding. Neurocomputing, 568:127063, 2024.
Rahul Thapa, Bryan He, Magnus Ruud Kjaer, Hyatt Moore, Gauri Ganjoo, Emmanuel Mignot, and James Zou, Sleepfm: Multi-modal representation learning for sleep across brain activity, ecg and respiratory signals. arXiv preprint arXivv2405.17766, 2024.
Armin Thomas, Christopher Ré, and Russell Poldrack.
Self-supervised learning of brain dynam- ics from broad neuroimaging data. Advances in neural information processing systems, 35: 22255-21269, 2022.
Polina Turishcheva, Paul Fahey, Michaela Vystrčilová, Laura Hansel, Rachel Froebe, Kayla Ponder, Yongrong Qiu, Konstantin Willeke, Mohammad Bashiri, Ruslan Baikulov, et al. Retrospective for the dynamic sensorium competition for predicting large-scale mouse primary visual cortex activity from videos. Advances in Neural Information Processing Systems, 37:1 18907-118899,, 2024.
Ivan Ustyuzhaninov, Max F Burg, Santiago A Cadena, Jiakun Fu, Taliah Muhammad, Kayla Ponder, Emmanouil Froudarakis, Zhiwei Ding, Matthias Bethge, Andreas S Tolias, et al. Digital twin re- veals combinatorial code of non-linear computations in the mouse primary visual cortex. BioRxiv, pp. 2022-02, 2022.
Edgar Y Walker, Fabian H Sinz, Erick Cobos, Taliah Muhammad, Emmanouil Froudarakis, Paul G Fahey, Alexander Ecker, Jacob Reimer, Xaq Pitkow, and Andreas Tolias. Inception loops dis- cover what excites neurons most using deep predictive models. Nat. Neurosci., 22(12):000--2055 December 2019.
Christopher
Wang, Vighnesh Subramaniam, Adam Uri Yaari, Gabriel Kreiman, Boris Katz, Igna- cio Cases, and Andrei Barbu. Brainbert: Self-supervised representation learning for intracranial recordings. arXiv preprint arXiv:2302. 14367, 2023.
Eric Y Wang, Paul G Fahey, Zhuokun Ding, Stelios Papadopoulos, Kayla Ponder, Marissa A Weis, Andersen Chang, Taliah Muhammad, Saumil Patel, Zhiwei Ding, et al. Foundation model of neural activity predicts response new stimulus types. Nature, 640(8058):470 477, 2025.
Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yo- gatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. Emergent abilities of large language models. June 2022.
Kaiyue Wen, Zhiyuan Li, Jason Wang, David Hall, Percy Liang, and Tengyu Ma. Understanding warmup-stable-decay learning rates: river valley loss landscape perspective. October 2024.
Konstantin F Willeke, Paul G Fahey, Mohammad Bashiri, Laura Pede, Max F Burg. Christoph Bless- ing, Santiago Cadena, Zhiwei Ding, Konstantin- Klemmens- Lurz, Kayla Ponder, et al. The senso- rium competition on predicting large-scale mouse primary visual cortex activity. arXiv preprint arXiv:2206.0866 2022.
Chaoqi Yang, M Westover, and Jimeng Sun. Biot: Biosignal transformer for cross-data learning in the wild. Advances in Neural Information Processing Systems, 36:78240-78260. 2023.
Joel Ye and Chethan Pandarinath. Representation learning for neural population activity with neural data transformers. arXiv preprint arXiv:22008 0et.:2208..111111 01210, 2021.
Joel Ye, Jennifer Collinger, Leila Wehbe, and Robert Gaunt. Neural data transformer 2: multi-context pretraining for neural spiking activity. Advances in Neural Information Processing Systems, 36: 80352-80374, 2023.
17

---

Published as a conference paper at ICLR 2026
Joel Ye, Fabio Rizzoglio, Adam Smoulder, Hongwei Mao, Xuan Ma, Patrick Marino, Raeed Chowd- hury, Dalton Moore, Gary Blumenthal, William Hockeimer, et al. A generalist intracortical motor decoder. bioRxiv, pp. 2025-02. 2025.
Daoze Zhang, Zhizhang Yuan, Yang Yang, Junru Chen, Jingjing Wang, and Yafeng Li. Brant: Foun- dation model for intracranial neural signal. Advances in Neural Information Processing Systems, 36:26304 26321. 2023.
Yimeng Zhang, T-S Tai Sing Lee, Ming Li, Fang Liu, Shiming Tang, Tai Sing. Lee Ming, Li Fang, Liu Shiming, T-S Tai Sing Lee, Ming Li, Fang Liu, and Shiming Tang. Convolutional neural network models VI responses to complex patterns. Comput. Neurosci., pp. 1-22, 2018.
Yizi Zhang. Yanchen Wang, Donato Jiménez-Benan Zixuan Wang, Mehdi Azabou, Blake Richards, Renee Tung, Olivier Winter, Eva Dyer, Liam Paninski, et al. Towards a" universal translator" for neural dynamics at single-cell, single-spike resolution. Advances in Neural Information Processing Systems, 37:80495-80521, 2024.
Yizi Zhang, Yanchen Wang. Mehdi Azabou, Alexandre Andre, Zixuan Wang, Hanrui Lyu, The In- ternational Brain Laboratory, Eva Dyer, Liam Paninski, and Cole Hurwitz. Neural encoding and decoding at scale, 2025. URL http:httt///aaa
orgg h/tg//bb/250444 08201
Yu Zhu, Bo Lei, Chunfeng Song, Wanli Ouyang, Shan Yu, and Tiejun Huang. Multi-modal latent variables for cross-individual primary visual cortex modeling and analysis. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pp. 1228-1236, 2025.
18

---

Published as a conference paper at ICLR 2026
A IMPLEMENTATION DETAILS
We provide a complete specification of the hyperparameters, masking configurations, and training procedure used for OmniMouse.
A.1 HYPERPARAMETERS
Tables S4 summarize the full set of hyperparameters for the final OmniMouse model. We or- ganize them into four groups: data and tokenization [Tab.[S]] which specifies how each modality is sampled, chunked, and converted into token embeddings; encoder latent queries (Tab. [S2). which defines the learned bottleneck between the variable-length input and the fixed-length neuronal and behavioral latent sequence; transformer architecture (Tab. [S} which covers the shared configura- tion of the encoder, fusion, and decoder stages; and training (Tab. S4). which lists the loss functions, optimizer settings, and learning rate schedule.
Table S1: Data & tokenization hyperparameters.
Parameter
Description
Value
Neural responses
Sampling rate
30 Hz
SR
Samples context window
60
wR
Conv. kernel size (samples)
SR
Conv. stride (samples)
5
local-ate ocallattention window size
Attention mask local hiddng-winndow size Neurons sampled per batch
333
P.
4096
Interpolation buffer
Gap between context and targets (samples)
Num. reconstructed neurons
Size of masked sub- suh-population to he reconstructed
3072
Num. reconstructed samples
Duration of reconstruction target (samples)
30
Visual stimuli
Frame rate
Video frame rate
30
Sy
Frames per context window
local-attention window size
Attention mask local sliding sliding-window window size Spatial resolution
.2
C' x H x V
x 36 x 64
Hiera encoder
Layers per stage
Hierarchical stage depths Output dimension of patch embedding
(2, 8)
Embedding dimension
128
Num. attention heads
MLP ratio
FFN hidden dim embed dim Convolutional patch embedding
2.67
Patch kernel (t, h, w)
(6, 7,,7,7)
Patch stride (t, h, w))
Patch embedding stride
(2, (2,2,2)
Patch padding (1, It, w)
Patch embedding padding
(2, 1,3)
Query stride (t, ht Mask unit size (t, h, w)
Pooling stride per stage
(1,
Local attention window
(1, (1,8,8)
(T' x H' W" x D')
Output feature shape projepro))ttion)
(30, 8, 16, 768)
Drop path rate
0.25
Behavioral traces Sampling
Behavior sampling rate
20 Hz
Ss
Samples per context window
40
Channels
Eye tracker (4) + treadmill (1) Attention mask local Aiddingwwnndww size
local-atto local-attention window size
19

---

Published as a conference paper at ICLR 2026
Table S2: Encoder latent query hyperparameters.
Parameter
Description
Value
M
Unique latent query groups
640
N
Repeats per group (covers context window)
6
Global register tokens
256
Total latents
M N G
4096
local-attention window size
Attention mask local sliding-window size (excluding globals)
.333
Table S3: Transformer architecture hyperparameters. Values of hyperaparameters that vary with scaling are indicated with ? (see table Tab. $5
Parameter
Description
Value
de dm
Identity embedding dimension
256
Model latent dimension
Multi-modal transformer layers
h
Number of attention heads (all stages)
?
d,
Attention head dimension
dm /n/hh
Normalization type Norm
RMSNorm
10
Pre-norm
Normalization applicd before attention/FFN Normalization applicd to Q and K projections Stochastic depth FFN hidden dim / dm Gated activation function Intericaving ratio of local sliding-window attention layers Activation neural predictions
QK-normalization
Drop path rate
0.0
FFN hidden multiplier
2.67
FFN activation
SiLU (SwiGLU) 5:1
Local:global ratio
Response output activation
Softplus(3=0.1)
Behavior output activation
Activation on behavior predictions
None (identity)
Table S4: Training hyperparameters.
Parameter
Description
Value
Loss
Neural loss type
Poisson negative log- oig-likelihood
Behavior loss
Mcan squared error
Ab
Behavior loss down-weighting factor
0.1
Optimization
Optimizer
AdamW
Learning rate
7 10
Warmup steps
Linear warmup duration
20 000
Learning rate decay type
Inverse square root (10k steps)
Learning rate decay steps 81,82 32
10k steps
Adam momentum parameters regularization
0.9, 0.98
Weight decay
Gradient clipping type
Norm
Gradient clipping value
1.0
20

---

Published as a conference paper at ICLR 2026
A.2
MODEL SCALING
Tab. summarizes the five model scales used in our scaling experiments, ranging from 1M to 300M parameters. We scale along two primary axes: model width (dm.) and depth (I., the number of transformer layers in the multi-modal fusion stack). Importantly, because we decouple the identity embedding dimension de from the model width via learned linear projections (as described in Sec. number of neuron- session-, and animal-specific parameters (px) remains fixed across scales.
Table S5: Scaling variants of OmniMouse. multi-modal transformer layers; dm model dimen- sion; h: number of attention heads; de: dimensions of all embeddings; PL: multi-modal transformer layer parameters; Par: model parameters (excluding neuronal embeddings); pvv all neuronal, ses- sion, and animal parameters; pr: total parameters; S: sequence length.
Model
dm
PL
Ph
PN
Pr
OmniMouse- IM
256
4
256
1.7M
6M
779M
885M
4096
OmniMouse-SWM
256
5.1M
10.4M
779M
891M
4096
OmniMouse- 20M
512
256
19.1M
29.1M
779M
810M
4096
OmniMouse-8000
12
768
12
256
88M
115M
779M
894M
4096
OmniMouse-30000
24
1024
16
256
308M
348M
779M
1.IB
4096
A.3 MASKING CONFIGURATIONS
Tab. enumerates all 119 structured masking configurations used during training. All configura- tions share fixed prediction target: the final 30 samples (1 s) of activity from up to 3072 randomly selected neurons, Rows enumerate the systematic variations over population context, causal context, video context, and behavior encoding/decoding Entries in brackets denote multiple configurations swept over the listed values.
Table S6: Masking configurations.
Mask
Behavior
Video (last
Visible
Pop. Context
Causal Context (from to)
Predicted
frames visible)
Neurons
(from
to)
Bchavior
1-3
[64, 256, 1024]
60
4
X
4096
60
5-7
X
[64, 256, 1024]
0 - 60
8-19
[64, 256, 1024, 4096]
[0, 10, 15] + 25
20-28
[64, 256, 1024]
25 + 60
[0, 10,
25
29-37
X
[64, 256, 1024]
25 + 60
[0, 10, 15]
25
38-40
10
[64, 256, 1024]
10 + 60
41-52
[64, 256, 1024, 4096]
[0, 10, 15] + 25
53 58
10
[64, 256, 1024, 4096]
25 + 60
[10, 15] 25
59-61
20
[64, 256, 1024)
20 + 60
62-73
[64, 256, 1024, 4096]
[0, 10, 15] + 25
74-79
20
[64, 256, 1024]
25 50
[10, 15] 25
80-82
20
[64, 256, 1024]
30 - 60
83 94
[64, 256, 1024, 4096]
[0, 10, 15] + 25
95-100
[64, 256, 1024]
25
[10, 15] 25
101-103
40
[64, 256, 1024]
30 + 50
104-111
40
[64, 256, 1024, 4096]
[10, 15] - 25
112-114
[64, 256, 1024]
+ 40
115-118
50
[64, 256, 1024, 4096]
10 20
119
60
21

---

Published as a conference paper at ICLR 2026
A.4 OMNIMOUSE TRAINING ALGORITHM
Algorithm App. A.4 provides a concise summary of the full OmniMouse training loop, from tok- enization through loss computation. At each training step, single masking configuration is sampled uniformly along with a batch of multi-modal data chunks from a single session.
Algorithm Train OmniMouse
neural responses R. video frames V. behavioral traces B, masking set M, loss weight a Sample masking configuration m ~ .M uniformly
Input:
// Tokenization
Tokenize R via strided ID convolution; add neuron, session, and animal identity embeddings ZR Tokenize Hiera encoder and linear projection + Zv
Tokenize B via linear projection; add channel, session, and animal identity emheddings -+ 7 // Masking
Remove masked tokens from Z - encader input 7 build decoder queries from target identity embeddings - Qn Remove -on-ge tokens from Zy + encoder input Zy if behavior unmasked then include Z in encoder input
else build decoder queries from channel/session/anins. embeddings - Q I/ Encoding
Compress RZ Za} into M X latents via cross-attention with learned queries and G global registers // Mudti- modal Fusion
Concatenate latents with Zv
pass through T. transformer layers +
// Decoding
Decode [Qn, Qa) via local causal cross-atiention over FF
apply modality-specific linear readouts - Rumnets // Loss and Update
Compute (())= PoissonLossi Rurgnn, Rugges) + a MSE( B)
Update fe via gradient descent on L(()
A.5 COMPUTE INFRASTRUCTURE AND TRAINING PROCEDURES
All models were trained on Marlowe (Kapfer et al. 2025 Stanford's NVIDIA DGX H100 SuperPOD cluster. We scaled training from a single node (8 GPUs per node) for smaller models up to 10 nodes for OomniMous-33000
Distributed training. Because each batch is sampled from single session, we are able to signif- icantly streamline memory and distributed communication overhead by allocating groups of data sessions across ranks. This allows us to partition parameters into shared weights (the transformer backbone) and rank-specific weights (i.e. identity embeddings for the sessions on each of the ranks). Accordingly, we implement a custom DDP-style training loop in which only the shared parameters are synchronized across ranks. Gradients for shared parameters are packed into flat, pre-allocated buckets and reduced asynchronously via all reduce. While the collective is in flight, each rank independently clips and steps the rank-specinf optimizer on its local embedding parameters, ef- fectively overlapping communication with computation. Once the reduction completes, the aver- aged shared gradients are unpacked, clipped, and stepped with a second optimizer. This bucketed, caamuunicattinoovellapeeee
scheme keeps GPU utilization high even at large node counts. Our dis- tributed training strategy achieved record #22 on the nanoGPT speedrun (Jordan et al. (024)) com- petition leaderboard] demonstrating competitiveness beyond neuroscience applications.
Compilation and kernel selection. We compile the model with torch. compile using the max- autotune mode. The structured sparsity of our local sliding- window and causal attention masks is implemented via PyTorch's FlexAttention API, which compiles the block-sparse mask patterns into fused Triton kernels and avoids materializing the full attention matrix. Moreover, we specifi- cally design the tokenization strides and local-attention window sizes for each modality to align with FlexAttention's 's block-size boundary of 128 tokens, ensuring that the sparse block structure incurs no wasted computation from partially occupied blocks. Similarly, we design all masking configurations
22

---

Published as a conference paper at ICLR 2026
to produce fixed input sequence length of 4096 tokens for the multi modall fusion transformer. This fixed shape eliminates recompilation across masking configurations and enables torch. compile to cache single optimized sub-graph, for the entire training run. To further minimize overhead, we careful to cache compiled FlexAttention masks associated with each unique masking configuration.
Numerical precision. All training uses automatic mixed precision with BF16 for forward and back- ward passes, while maintaining full FP32 precision for master weights and optimizer states. We disable BF16 reduced-precision accumulation ensure numerical stability during training.
23

---

Published as a conference paper at ICLR 2026
B BASELINES
To establish baseline comparisons while managing computational costs, we train state-of-the-arr baseline models on the smallest nested dataset containing eight mice (the seven evaluation mice plus one additional training mouse). This approach ensures that all methods are compared under identical conditions while keeping baseline training tractable. We train all baselines on 8 recordings from 8 unique mice fully released mice from the sensorium 2023 competition (keeping the orig- inal train- train-validation validation test splits), mice from the sensorium 2022 competition that were used for the test split and session from the MiCRONs collection, which was used for training only. The same 8 mice were used in the smallest scaling experiment from Fig.
B.1 CEBRA
CEBRA explanation: CEBRA performs dimensionality reduction on neural activity using InfoNCE contrastive learning, where positive and negative pairs are defined by auxiliary variables such as time or behavior. When the auxiliary variable is discrete, for example left or right wheel turn, it selects positives uniformly from all samples with the same label. When the variable is continuous, such running speed pupil direction, it chooses random point within time window around the sample and then find the closest match in the dataset using either Euclidean or cosine distance; this sample becomes the positive pair, which adds diversity and prevents repeatedly selecting the same example. Negative pairs are sampled randomly. For decoding, CEBRA encode neural responses, find the nearest latent vectors for responses in the training set, and returns their associated behavioral variables as predictions.
Model hyperparameters: We trained a joint model for 8 mice, using batch size of 512 and learning rate of 3 10 The network contained 256 hidden units and produced 128-dimensional outputs (both doubled relative to the Allen example htttp:://ccct
aw//acas//a/aaaaaaaaaaaasa.aa demo notebooks/Demm Allen.htmm Training ran for up to 50,000 iterations with cosine distance as the loss metric. The model used temperature of 1, time-delta conditioning to enable behavior mode, and time offsets of 5. As CEBRA requires same frequencies between responses and behavior, both were resamples to 20 Hz, in order to compute correlation on the same predictions as for the OmniMouse. Please note that downsampling from 30 Hz responses is not reducing any information as responses were upsampled from 6-16 Hz to 30 Hz and the upsampling is done with nearest- neighbr interpolation.
B.2 UNIVERSAL SPIKE TRANSLATOR
Universal Spike Translator explanation: The Universal Spike Translator Zhang et al. (2024) per- forms sup-euupee modeling approach called multi-taskkmasking (MtM). The model alternates between masking out and reconstructing neural activity across different time steps and neurons. It uses learnable token that provides the model with context about the specific masking scheme that is being applied during training, allowing for "mode switching" at test time for different downstream tasks. During training, the masking schemes are sampled randomly which are: (1) Neuron mask- ing: Randomly masks individual neurons and reconstructs their activity using the unmasked neurons as context. (2) Causal masking: Masks future time steps and predicts them using the past steps as context.
Model hyperparameters: We started with the default hyperparameters from "ndtl stitch- "ession_ttaineer"
ing pr__propping" and "ssl_session trainer" configs from https://githuu: www..//oo/ooccreeecc tz/ IBL model and performed a Bayesian hyperparameter search using weights & biases, https: The sweep configs are available at attps:///ithii: com/ sensorium competition IBL MtM model/tree wandb sweep, Please note that compared to our forecasting settings, IBL does not take behavior as model input.
//docs wandb..nnbb........o htt///oddls/wweeer
B.3 LATENT DYNAMIC MODEL
Latent dynamic model explanation: This is a probabilistic model that predicts the joint distribu- tion of neuronal responses from naturalistic video stimuli and himulus-independers latent factors. Specifically, the model predicts time-varying neuronal response using a Prroo--nlate--aaaat distribution to model the distribution of neuronal responses conditioned on the stimulus and the latent
(ZIG)
24

---

Published as a conference paper at ICLR 2026
factor. This is modification of the deterministic factorized 3D convolutional core and a Gaussian readout, where we have an additional encoder that takes a subset of neurons as input to derive a la- tent variable. This latent variable is then combined with the transformed visual input predict the activity of other neurons in the session. The model is trained by maximizing the Evidence Lower Bound (ELBO) of (y)a( via variational inference.
Model hyperparameters: For both SENSORIUM 2023 baseline and (Schmitt et al. (2025) baseline we used the default hyperparameters from Schmidt al. (2025): 3 layer core with both spatial and temporal kernel 11 in the first layer and 5 on the layer two and three. For more details see App.C from Schmidt et al. (2025). All data modalities were upsampled to 30 Hz as both SENSORIUM 2023 baseline and Schhmidt et al. (2025) latent model require all modalities to have the same frequencies. Both SENSORIUM 2023 baseline and Schmiitt et al. (2025) latent model predict 42 samples from a 60-frame video input, we always used only last 30 frames for evaluation, to make it consistent with OmniMouse, who was trained to predict 30 samples. Please note that OmniMouse supports flexible size of predictions, while SENSORIUM 2023 baseline and [Schmidt et al. (2025) latent model has 'burn period', e.g. cannot predict responses for the first few frames of the given video.
B.4 POYO+
2025) performs multi-task neural decoding by tok- enizing neural activity. The tokeization follows POYO (Azabou et al., (023) and encodes raw data into unit-level embeddings and spike times. As POYO+ is working with calcium imaging and not spikes, it also encodes the amplitude of a neuronal response not just unit-level embeddings. Its en- coder uses cross-attention to keep the number of tokens fix despite potentially non-fixed input size. For flexible decoding, the model constructs query tokens by summing learned task embedding, a pession-speciffe embedding, and an output timestamp. It is trained end-to-ent aed-to-end on a mixture of classification, regression, and segmentation tasks, using weighted sum of enssopy and mean- squared error losses. To generate predictions, multi-modal decoder queries the latent space via cross-attention and passes the resulting tokens through task-speeific linear layers to reach the target dimensionality.
POYO+ explanation: POYO+ (Azabou et al.
Model hyperparameters: We ran an extensive hyperparameter search (>100 runs) on POYO+. We used Bayesian sweep via Weights & Biases for hyperparameter optimization; the code and sweep settings are available at /tttps://gitaeti hotpau/..ooooollytttooooorrdd hrape///toooooborrn/oeaeaaeaeaee brain traainttree/ight, mice www.hearweeeeaeeeeeeeeeeeaeeeere www.aaarrree.aamppppsoooooooo plus.
C SUPPLEMENTAL RESULTS
OmniMouse enables systematic evaluation of how neuronal context shapes predictive perfor- mance. We evaluated OmniMouse on conditions not seen during training. systematically varying neuronal history duration (10-25 samples) and population context size (16-2048 neurons) for pop- ulation context tasks. Performance scaled smoothly with context availability across all conditions Fig. S2 When video was available, performance plateaued more quickly for forecasting but contin- ued to improve for population context, suggesting that nearby neurons carry complementary infor- mation beyond visual input. These systematic evaluations demonstrate that OmniMouse has learned generalizable representations of neural variability, enabling quantitative assessment of how different sources of context-temporee temporal history contribute to explaining variability in neural responses.
Furthermore, we hypothesized that harder tasks might benefit more from scaling, as shown for large language models (Minaee et al.] 2024, Naveed et al., 2025). To test this, we varied the causal context E [10. 15, 20, 25I) for the forecasting tasks and population context size (context E [2, 1024, 2048)), with smaller contexts being more challenging prediction tasks. We also compared performance with and without 2 seconds of video input. Fig. S2 confirms our hypothesis: performance improves consistently as context grows, hence, bigger context indicates an easier task. Non- Non-video video conditioned regimes scale more steeply. However, contrary to LLMs, in our scaling model size does not preferentially benefit harder tasks: across all tasks, results for different model sizes do not meaningfully change for harder tasks.
25

---

Published as a conference paper at ICLR 2026
Example predictions for different mesking strategies
Neuronal encoding
Neuron
T=0.59
rm0.65
Neuron
r-0.63
Neuron
Giound trth
Neurona activty
rw0.55
rw0.59
1=0.41
reuron al aacthiyy
Grmund
rw0.75
rwo, P0
rw0.77
nearone population
pedicctios
Stimulus othen
belundor 1024 no pooe
2.0
4.3
2.0 Time seconds
2.0 Time
Time seconds
Pup location
Pupé size
Running speed
Schaworl bso nommlledd
0.94
Behawlor decoding
Ground buth
Indrencia louses
Behwrrer
Prediction Grcund mrah
pad
Time soconds
Time scconds
Time seconds
Figure S1: Example predictions of neuronal activity and behavioral variables. A. Here we show three example neurons and their ground truth neuronal activity for 4 seconds (black). We also show the model prediction of OmniMouse for three evaluation conditions: population context of 64 neu- rons (orange), Hiimulus-encodiin (blue), stimulus behavior neuron context (green). The predic- tive performance, shown as pearson correlation r is increasing with more information provided to the model. Our model is designed disentangle the relative contributions of sensory input, behavior, and population dynamics to individual neurons' activity. B. Ground truth and predictions for behav- ioral variables.
Forecasting
Forecasting + stimulus
Population prediction + stimulus
Population prediction
Stemutssennddgg
Singlo tral conmlation
.40
-30
-20
5tmoilu encodtnng
ng
-0M
80M
302M
0.35
0.835
1024
1021
Causal context
Causal context
N Neuurons in context
N nenuoons in context
Figure S2: Using the models capabilities to investigate context lengths for forecasting and pop- ulation context tasks. A. Forecasting with a change of causal context, i.e. how many samples of neuronal activity are unmasked. A causal context length of 10 corresponds to one third of a second of neuronal activity. B. Same change of forecasting context as A, but with video. C. Performance improvements in addition to video with population context, # neurons in context = 0 means that all neurons are masked, and the model conditions its prediction purely on the visual stimulus. In all panels, the stimulus-encoding performance is denoted by the gray box for ease of comparison. Re- markably, as seen in panel A, forecasting a whole second of neuronal activity given the past second yields comparable predictive performance as showing the entire two seconds of the visual stimulus. D. Population context only, 16 2048 neurons
D SENSORIUM COMPETITION EVALUATION
We evaluate our model on the SENSORIUM 2023 competition test set, which allows direct compari- son against the state of the art model of predicting mouse visual cortex responses from video stimuli. We use OmniMouse-8000 80M, freeze the entire model, and train only the neuron and animal embeddings using the released training data of five mice provided by the competition. For SENSORIUM 2022, report the results of OmniMouse-8000 80M trained the largest dataset collection, which included the training set of the two animals used in SENSORIUM 2022.
D.I DISTRIBUTION OF SESSIONS PER MOUSE
26

---

Published as a conference paper at ICLR 2026
Neuronal activity predict ons
Behavior decoding: Pupil location
0.8
0.40
Single correlation
Single trlal corretation
lespense liehay 30m
0.6
0.30
Raseoee Stim
Resparse Behavior
Hesporas Sémulus
only
0.3
36
64
250
1024
15
64
2556
1024
Number of neurons In contect
Number of neurons In contestt
Figure S3: Systematic evaluation across mask configurations. We evaluate the neuronal predic- tion and behavior decoding performance of OmniMouse-80M by systematically varying the model inputs via masking. Only masks using N [64, 256, 1024] have been Mouse generalizes to unseen conditions, and allows to systematically study the contribution of visual stimulus, behavioral variables, and neuronal (sub)-population activity. A. Neuronal activity predic- tions given different amounts of visible neurons in context. B. Behavioral decoding.
during training. Omni-
Table S7: Sensorium 2023 benchmark results. Models with suffix denote ensemble predictions. We use n=5 models in the OmniMouse ensemble, from different random seeds, which determines model initialization and ordering of training batches. t indicates higher is better. We either run the full multi-modal training, only train the model with single masking condition( Unimodal) predicting neuronal responses conditioned on behavior and visual stimulus comparable to all other models of the competition.
Model
Training
Main track
OOD track
DwiseNeuro-> V
Turishcheva et al. 2024
end-to-end
0.291
0.221
end-to-end
0.288 + .003
0.256 + .002
end-to-end
0.332
0.296
OmniMouse-SM
end-to-end
0.295 1: .005
0.263 1: .003
OmniMouse-5MM
end-to-end
0.327
0,293
OmniMouse-SOM
frozen
0.313 1: ,001
0.274 1: .001
OmmiMouse-800--X
frozen
0.327
0.288
12
Number Sessione
17997
20692
10857
164 18
1990
27083
21057
27424
2,560
27660
28122
21:54
99156
22279
20661
21533
76614
26776
77470
r95
10997
22364
15702
25555
18891
16342
27802 20022
26-43
28739
71844
29670
19139
2925
15415
36520
25404
27542
10457
22084
2//5
23555
26656
23964
25508
18833
26426
2014
58
201.5
2
18522
Mouse
Figure S4: Distribution of 316 sessions across 69 mice. More than 100 sessions come from first
10 mice.
27

---

Published as a conference paper at ICLR 2026
E DATASET DETAILS
E.0.1 NESTED SCALING DATASET CONSTRUCTION
The nested dataset was constructed such that for the 7 mice we conducted evaluation on mice we had repeated sessions, such that the number of repeats grew proportionally to the dataset growth, and 4 other mice had single session. As session session-per-menn per-mice distribution is highly skewed, the other sessions were samples randomly.
E.1
NEUROPHYSIOLOGICAL EXPERIMENTS
Model evaluation was performed neurophysiological data from Sensorium 2022 ( (Willlee al. 2022). Mouse 1 and 2, evaluation animals for Sensorium and Sensorium Plus tracks) and Sensorium
2023 (Turiihchevaa et al. 2024), all animals). Model training was performed historical data, including data from MICIONS Consortium (2025), Wang et al. (2025), Ding et al. (2025b), Ding
all (2025a), Fahey et al. (2019), Willeke et al. (2022) Turishcheva et (2024). but also included data not previously published.
All procedures were approved by the Institutional Animal Care and Use Committee of Baylor Col- lege of Medicine. Seventy-eight mice (Mus musculus, 32 females, 46 males, P50-155 on day of first scan) expressing GCaMP6s in excitatory neurons via
and Ail62 transgenic lines (recom- mended and generously shared by Hongkui Zeng at Allen Institute for Brain Science; Jackson Labs stock 023527 and 031 031562, respectively) were anesthetized and 4 mm craniotomy was made over the visual cortex of the right hemisphere as described previously (Reimer et al. 2014, Froudarakis ct al, 2014). In two of the seventy-three animals, GCaMP6s was additionally expressed in inhibitory neurons via DLX5-CreER (Jackson Labs stock 010705), following treatment with tamoxifen (oro- gastric gavage of tamoxifen (Sigma Aldritch T5648) dissolved corn oil (Sigma Aldritch C8267) at 15 mg/mL, 200 mg/kg body weight, two doses two days apart, second dose >= 13 days before the first included scan).
Mice head-mounted above cylindrical treadmill and calcium imaging was performed ing Chameleon Sa-Sapp laser (Coherent) tuned to 920 nm and a large field of view mesoscope [Sofroniew et al. 2016) equipped with custom objective (excitation NA 0.6, collection NA 1.0, 21 mm focal length). Laser power after the objective was increased exponentially as a function of depth from the surface according to:
P= Po x (()/L.)
(3)
Here P is the laser power used target depth z, PO is the power used at the surface (typically not exceeding 25 mW), and Lg is the depth constant (160-220 ju). The greatest laser output of ca. 112 mW was used at approximately 400-500 jmm from the surface.
The cramiotomy window was leveled with regards to the objective with six degrees of freedom. Pixel- wise responses from ROI spanning the cortical window (1.7-4 mm diameter FOV, >0.2 px/jum, superficial cortex, >2.47 Hz) to drifting bar stimuli were used to generate a sign map for delineating visual areas (Garrett al. 2014). In some but not all cases where the imaging field of view spanned multiple areas, area boundaries on the sign map were manually annotated. Imaging FOV of varying dimensions were targeted lie within the boundaries of visual cortex, and may span between primary visual cortex and surrounding higher visual areas depending on the scan design.
Scan dimensions typically fell into one of three categories. Local field of view scans contained multiple imaging planes at different depths (10-13 planes, most commonly with 5 μm Z spacing but ranging between 3 and 45 μm z spacing), with each plane spanning 600-630 x 600-630 um (240-252 x 240-252 pixels, 0.4 px/jum resolution ), acquired most commonly at 7.98 Hz (range 1.34-8.31 Hz). Large field of view scans contained single imaging planes at a single depth, with each plane scanning 1.5 3 mm diameter (0.33- 0.4 px/um resolution), acquired at between 6.5 12.4 Hz. In between are scans containing multiple imaging planes at different depths (2-5 planes, with variable interplane spacing between 5 and 150 jum), with each plane spanning approximately 0.8-1.2 mm diameter (0.4- 0.6 px/pum resolution), acquired at between 6.3 and 9.6 Hz. Scans with multiple planes, especially at high sampling densities (ex. 5 jum spacing), have high likelihood of multiple segmented traces
28

---

Published as a conference paper at ICLR 2026
emerging from multiple planes intersecting with the soma of single neuron in single scan. Multiple scans were also often collected from the same animal, and as a result single biological neurons may be recorded across multiple scans.
Movic of the animal's ' eye and face was captured throughout the experiment. A hot mirror (Thorlabs FM02) positioned between the animal's left eye and the stimulus monitor was used to reflect an IR image onto a camera (Genie Nano C1920M, Teledyne Dalsa) without obscuring the visual stimulus. The position of the mirror and camera were manually calibrated per session and focused on the pupil. Field of view was manually cropped for each session to contain the left eye in its entirety, although across different experiments the field of view may have additionally contained more or less of the face, centered or not centered on the eye, or characterized the pupil at different resolutions. Video was captured at ca. 20 Hz. Frame times were time stamped in the behavioral clock for alignment to the stimulus and scan frame times. Video was compressed using Labview's MJPEG codec with quality constant of 600 and stored in an AVI file.
Light diffusing from the laser during scanning through the pupil was used to capture pupil diameter and eye movements. A DeepLabCut model (Mathis et al. 2018) was trained as previously described TTrishhheva et al.] 2024) on 17 manually labeled samples from 11 animals to label each frame of the compressed eye video (intraframe only H.264 compression, CRF:17) with 8 eyelid points and pupil points at cardinal and intercardinal positions. Pupil points with likelihood >0.9 were fit with the smallest enclosing circle, and the radius and center of this circle was extracted. Frames with < pupil points with likelihood >0.9, or producing circle fit with outlier 5.5 standard deviations from the mean in any of the three parameters (center x, center y, radius) were discarded. Gaps in behavior were replaced by linear interpolations over the whole session, if there were more than frames with gaps, then the video is removed.
The mouse was restraistt during imaging but could walk on a treadmill. Rostro-caudal tread- mill movement was measured using a rotary optical encoder (Accu-Coder 1ST-01SF-200ONV F03-S1) with a resolution of 8000 pulses per revolution, and was recorded approx. 50-100 Hz in order to extract locomotion velocity.
Visual stimuli were presented with Psychtoolbox 3 in MATLAB (Brainard Vision| 1997 Kleiner et all 2007] Pelli (197) to the left eye with a 31.8 x 56.5 cm (height x width) monitor (ASUS PB258Q) with a resolution of 1080 1920 pixels positioned 15 cm away from the eye. When the monitor is centered
and perpendicular to the surface of the eye at the closest point, this corre- sponds to a visual angle of 3.8 %/em at the nearest point and 0.7 %/cm at the most remote corner of the monitor. As the craniotomy coverslip placement during surgery and the resulting mouse position- ing relative to the objective is optimized for imaging quality and stability, uncontrolled variance in animal skull position relative to the washer used for head-mounting was compensated with tailored monitor positioning on a six dimensional monitor arm. The pitch of the monitor was kept in the vertical position for all animals, while the roll was visually matched to the roll of the animal's head beneath the headbar by the experimenter. In order to optimize the translational monitor position for centered visual cortex stimulation with respect to the imaging field of view, we used a dot stimulus with a bright background (maximum pixel intensity) and a single dark square dot (minimum pixel intensity). Dot locations were randomly ordered from a grid tiling a portion of the screen, either a 10 x 10 grid tiling a central square (approx. 90º width and height, 10 repeats per location, 200-300 ms presentation at each location), or a 5 x 8 grid tiling the majority of the monitor (approx. 93 height and 119 width, 20 repeats per location, 200 ms presentation at each location). The final monitor position for each animal was chosen in order to center the population receptive field of the scan field ROI on the monitor, with the yaw of the monitor visually matched to be perpendicular to and 15 cm from the nearest surface of the eye at that position.
A photodiode (TAOS TSL253) was sealed to the top left corner of the monitor, and the voltage was recorded at 10 kHz and timestamped on the behavior clock (MasterClock PCIe-OSC-HSO-2 card). Simultaneous measurement with a luminance meter (LS--100 Konica Minolta) perpendicular and targeting the center of the monitor was used to generate a lookup table for linear interpolation between photodiode voltage and monitor luminance in cd/m² for 16 equidistant values from 0-255, and one baseline value with the monitor unpowered.
29

---

Published as a conference paper at ICLR 2026
At the beginning of each experimental session, collected photodiode voltage for 52 full-screen pixel values from 0 to 255 for one second trials. The mean photodiode voltage for each trial Vp was fit a function of the pixel intensity Vin:
pd
B
A V
(4)
in order to estimate the value of the monitor (=z 1.50 - 1.76). All stimuli were shown with no correction.
During the stimulus presentation, sequence information was encoded in 3 level signal according to the binary encoding of the flip number assigned in-order. This signal underwent a sine convolution, allowing for local peak detection to recover the binary signal. A linear fit was applied to the trial timestamps in the behavioral and stimulus clocks, and the offset that was applied to the data to align the two clocks, allowing linear interpolation between them. The mean photodiode voltage of the sequence encoding signal at pixel values 0 and 255 was used to estimate the luminance range of the monitor during the stimulus, with typical maximum values of approx. 10-12 cd/m?.
30


Reviews:
[{"review_id": "pqpzUajSo6", "reviewer": "Reviewer_SAAW", "summary": "The authors present a scaling analysis using a large neural model. The model is trained on multi modal data from head-fixed mice in a VR task consisting of visual stimulus, locomotion speed and pupil features recording. The neural data is calcium imaging confined to the visual cortex, for an overall tally of 3M neurons on 78 subjects. The model is based previous work POYO, with an additional hierarchical vision encoder to input the visual stimulus data. The authors present learning curves as a function of model size (measured in parameters) and show that performance saturates with model size, but does not saturate with input dataset size. The model performance is evaluated on a 2023 decoding benchmark with scores beating the baselines in all categories.", "questions": "- What reasonable steps could be taken to reproduce those results from a third-party ?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "The scale analysis is a welcome contribution to the field.\n- The discussion correctly assesses limitations of the current approach, namely the narrow brain region and behavior protocol.\n- Despite the narrow brain region and behaviour type studied, the authors manage to show that their model is still data limited, which makes the claim that dataset size is a limiting factor even more compelling.", "weaknesses": "- The performance is compared to baselines that may not be state of the art models. For example in behaviour decoding tasks, to justify the claim of SOTA we may be more interested in others large transformer based ANNs results than in the contrastive learning CEBRA method\n- No clear statement about sharing of code, models and private data for others to address the point above or even reproduce the proposed results, while the authors have made heavy use of public resources.\n\n\nMinor comments:\n- figure 1bc: label y-axis missing (loss)\n- Code / model / data availability statement", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "niBgkP6q4G", "reviewer": "Reviewer_Lnv7", "summary": "This paper presents OmniMouse, a multi-modal, multi-task transformer model trained to predict activity in the mouse visual cortex. The model is trained on a new, massive-scale dataset of 3.3 million neurons from 323 recording sessions, totaling over 150 billion neural tokens.\n\nThe primary contribution is a systematic study of scaling laws, training models from 1M to 300M parameters. The authors find that performance gains saturate with increasing model size but scale reliably and consistently with increasing dataset size. This leads to the central conclusion that current models of the visual cortex are data-limited, not parameter-limited, a finding that inverts the standard scaling narrative in mainstream AI. The largest model achieves new state-of-the-art performance on several predictive tasks, including neural forecasting, stimulus-driven prediction, and behavioral decoding.", "questions": "1. To provide a fair architectural comparison, what is the performance of a smaller OmniMouse model (e.g., OmniMouse-80M) when trained only on the 8-mouse dataset? How does that model compare to the baselines (IBL, CEBRA, etc.) trained on the same 8-mouse dataset? This would isolate the architectural contribution from the data-scaling contribution.\n2. Have you explored neuron-agnostic tokenization strategies? For example, could a neuron be represented by an embedding derived from its anatomical coordinates, its functional tuning properties (e.g., a pre-computed receptive field), or its relative position to other neurons, rather than a unique learned ID?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. **Scaling study for brain foundation model**: To my knowledge, this is the first work to systematically apply the scaling laws methodology to a single-neuron, multi-modal brain foundation model of this magnitude. The primary finding is that the model performance is data-limited rather than parameter-limited, which provides a clear directive for future progress, emphasizing the need for larger and more diverse datasets.\n2. **Dataset and model contribution:** The assembly of a 150B+ token dataset from 3.3M neurons is a major contribution in itself. Furthermore, OmniMouse-300M outperforms strong, specialized baselines (e.g., IBL, CEBRA) across the Sensorium 2023 competition even with its core weights frozen (training only neuron/animal embeddings).\n3. **Flexible and generalizable prediction:** The model's multi-task design, based on flexible masking of modalities, is effective. The paper shows that this design allows the model to learn representations that generalize to contextual variations not seen during training.", "weaknesses": "1. **Datasets scale misalignment in baseline comparison:**  The OmniMouse-300M model was trained on the full 323-session dataset, whereas all baselines were trained only on the smallest 8-mice data collection to reduce computational cost. It is unclear how much of OmniMouse's SOTA performance is due to its superior architecture versus simply having access to ~40x more data than the baselines it's compared against.\n2. **Generalization bottleneck on per-neuron identity embeddings:** The use of per-neuron (and per-session/animal) identity embeddings limits the generalization of the model. As shown in Table 1, these neuronal parameters account for the vast majority of the model's total parameters (e.g., 779M $p_N$ vs. 348M $p_M$ for the \"300M\" model). The model cannot perform zero-shot prediction on a new, unseen neuron or animal; it must be fine-tuned. A true foundation model should learn a general representation of a neuron rather than memorizing 3.3 million specific instances.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dDOmuwMuZq", "reviewer": "Reviewer_Cuf1", "summary": "This paper studies whether the scaling laws in AI also apply to modeling neural activity. Using a dataset with 3.3 million neurons from the mouse visual cortex (78 mice, 323 sessions, and over 150 billion neural tokens), the authors trained multi-modal, multi-task transformer models ranging from 1M to 300M parameters. These models can perform neural prediction, behavioral decoding, and neural forecasting. Empirically, the authors found that performance scales with data quantity but saturates with increasing model size, indicating that current modeling efforts in neuroscience are data-limited.", "questions": "1. The single-trial correlation metrics in Fig. 5 are quite low. Could the authors clarify the reason for this? For example, is it due to the use of naturalistic video stimuli rather than repeated trials? Also, why was correlation chosen as the primary evaluation metric, given that it only captures linear relationships and does not account for nonlinear dependencies or variance structure in the data?\n\n2. The paper claims that scaling in neuroscience is limited by data rather than model size, unlike in AI. In Fig. 5, it seems that as data increases, model size should also grow; otherwise, performance may saturate. This implies that model scaling is still necessary when handling larger datasets. In AI, there is already abundant data, which may make increasing model size particularly helpful. I am curious if the authors have considered this factor.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper makes a timely contribution by systematically studying scaling laws in neurofoundation models using an unprecedentedly large dataset. The work is novel in introducing the first large-scale, multi-modal, multi-task transformer that unifies neural encoding, decoding, and forecasting with naturalistic video inputs. The paper is also well written and organized.", "weaknesses": "1. The paper is impressive in scale, but several aspects could be improved. The contribution is empirical rather than methodological. The proposed transformer largely builds on existing model designs (e.g., POYO+ and prior multi-modal fusion techniques), with limited architectural innovation beyond scaling and integration. Clarifying which components are novel vs. adapted from prior work would help better explain the contribution.\n\n2. The paper would benefit from more qualitative analyses or visualizations to show what the model has learned.\n\n3. Although the inclusion of naturalistic video inputs is interesting, the current analyses do not disentangle how much information comes from visual vs. behaviors (e.g., pupil location, size, running speed).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "D9K6L7lWhS", "reviewer": "Reviewer_8RvB", "summary": "This paper presents a model architecture and training methodology and uses it to analyze data and model scaling trends for neuroscience data. It discovers positive trend in data scaling, and a limited scaling on the model side (suggesting the need for more data). The overall message is an important contribution. My main concerns are with one of the baseline used in evaluation, and some concerns about phrasing.", "questions": "**Questions**\n- It is unclear whether the dataset used for pretraining is public or private.\n- Can you please expand on \"restart training from intermediate checkpoints every 20k steps\"? (From section 4, Training paragraph) Which intermediate checkpoints do you restart from? Why restart? Why not just continue with a step change in LR?\n- In section 5, Forecasting: Why \"40 frames of behavior\" were used to condition the forecasting? Why not 30?", "limitations": "", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Analysis of both data scaling and model scaling trends is very valuable for the community. One of the first papers to do that.\n- Evaluation strategy is well designed, and a good number of tasks are included.", "weaknesses": "I like the overall message of this paper, but I think the problems mentioned below need to be addressed before I can recommend acceptance.\n\n**Main remaks**\n- I would prefer the reasoning for upsampling the data was put in the main text (I see it is in the appendix right now.)\n- (Opinionated) Just using number of neurons to compare dataset sizes doesn't seem like a good idea. This is reference to the line: \"We used a dataset of over 3 million single-unit neuronal recordings – an order of magnitude larger than...\". After all, I could have 1 second long recordings of 10 million neurons, and that dataset would not be considered big. To me, it seems a metric like \"neuron-hours\" would be better. i.e. including both the number of neurons and the recording durations to indicate data size.\n- Section 3, Data utilization: \"A key novelty is our ability to sample...\". Can you explain how this is novel in context to existing work? As far as I know, POYO and POYO+ have used this kind of arbitrary continuous sampling, and it has been a part of the open-source [`torch_brain`](https://github.com/neuro-galaxy/torch_brain) package for quite some time.\n- My biggest issue: When comparing the performance of behavior decoding, why not compare with something like POYO (individual behaviors), or POYO+ (multi-behavior)? These models (and many other recent methods) have proven to be much better than the CEBRA baseline. Comparing with something that is not a leading method for behavior decoding diminishes the results. Including stronger baselines would help convince the reader that the suggested expensive large-scale semi-supervised training does indeed lead to better performance, better than simple supervised learning using strong methods. I understand the paper is more focused on scaling, but it is important having a better reference of where purely supervised methods are in comparison.\n\n**Nits**\n- Fig 1. Missing y-axis labels\n- Some places, such as paragraph 2 of Introduction should have em-dashes `---` instead of en-dashes\n- Line 071 - comma after task - \"single modality, task, or dataset\"\n- Typo: Fig 3 caption - \"tookens\"\n- Line 249 - \"shared linear\" -> \"shared linear layer\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)