Inpainting the Neural Picture: Inferring Unrecorded Brain Area Dynamics from  Multi-Animal Datasets
Authors: Ji Xia
Yizi Zhang
Shuqi Wang
Genevera I. Allen
Liam Paninski
Cole Lincoln Hurwitz
Kenneth D. Miller
Venue: NeurIPS 2025 poster ()
Keywords: neuroscience
autoencoder
Neuropixels datasets
inpainting
brain area interactions
eletrophysiology.
Primary area: neuroscience_and_cognitive_science
TL;DR: We designed and tested an autoencoder model for stitching together multi-area multi-animal neuronal recording datasets and inpainting neural dynamics from unobserved brain areas in each experiment.

Abstract: Characterizing interactions between brain areas is a fundamental goal of systems neuroscience. While such analyses are possible when areas are recorded simultaneously, it is rare to observe all combinations of areas of interest within a single animal or recording session. How can we leverage multi-animal datasets to better understand multi-area interactions? Building on recent progress in large-scale, multi-animal models, we introduce NeuroPaint, a masked autoencoding approach for inferring the dynamics of unobserved brain areas. By training across animals with overlapping subsets of recorded areas, NeuroPaint learns to reconstruct activity in missing areas based on shared structure across individuals. We train and evaluate our approach on both synthetic data and two multi-animal, multi-area Neuropixels datasets. Our results demonstrate that models trained across animals with partial observations can successfully in-paint the dynamics of unrecorded areas, enabling multi-area analyses that transcend the limitations of any single experiment.

Inpainting the Neural Picture: Inferring Unrecorded Brain Area Dynamics from Multi-Animal Datasets
Xia
Yizi Zhang
Center for Theoretical Neuroscience, Zuckerman Mind Brain Behavior Institute,
Department of Statistics, Center for Theoretical Neuroscience,
Kavli Institute for Brain Science, Columbia University j bx24848columbia edu
Zuckerman Mind Brain Behavior Institute,
Kavli Institute for Brain Science,
Grossman Center for the Statistics of Mind, Columbia University
Shuqi Wang
Genevera Allen
Laboratory of Computational Neuroscience, École polytechnique fédérale de Lausanne (EPFL)
Department of Statistics,
Irving Center for Cancer Dynamics,
Center for Theoretical Neuroscience,
Zuckerman Mind Brain Behavior Institute,
Kavli Institute for Brain Science, Columbia University
Liam Paninski
Cole Lincoln Hurwitz
Department of Statistics, Center for Theoretical Neuroscience,
Center for Theoretical Neuroscience, Zuckerman Mind Brain Behavior Institute, Kavli Institute for Brain Science Grossman Center for the Statistics of Mind, Columbia University
Zuckerman Mind Brain Behavior Institute, Kavli Institute for Brain Science, Grossman Center for the Statistics of Mind, Columbia University
Kenneth D. Miller
Center for Theoretical Neuroscience, Zuckerman Mind Brain Behavior Institute, Kavli Institute for Brain Science, Columbia University
Abstract
Characterizing interactions between brain areas is a fundamental goal of systems neuroscience. While such analyses are possible when areas are recorded simulta- neously, it is rare observe all combinations of areas of interest within single animal or recording session. How can we leverage multi-animal datasets to better understand multi-area interactions? Building multi-animal models, we introduce NeuroPaint, a masked autoencoding approach for inferring the dynamics of unrecorded brain areas. By training across animals with overlapping subsets of recorded areas, NeuroPaint learns to reconstruct activity in missing areas based shared structure across individuals. We train and evalu- ate our approach on synthetic data and two multi-animal, multi-area Neuropixels datasets. Our results demonstrate that models trained across animals with partial observations can successfully in-pain the dynamics of unrecorded areas, enabling
recent progress in large-scale,
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

multi- analyses that transcend the limitations of any single experiment. Code is available the following github repository: NeuroPaint
Introduction
Understanding how brain areas coordinate their activity to support complex behaviors, such as memory-guided actions and perceptual decision-making. is a central challenge in systems neu- roscience [77]. Advances in electrophysiological recording techniques now enable single-cell, single-spike resolution measurements across dozens of interconnected brain areas [25 43 44) These
developments have driven the collection of brain-wide datasets from hundreds of behaving mice, cre-
ating new opportunities to study distributed neural computation at scale [28] 10 27 30]. Capitalizing
on these datasets requires computational methods capable of modeling inter-area interactions across animals and tasks.
Traditional approaches to modeling inter-area interactions rely on pairwise neuronal correlations [2]
or low-dimensional linear methods such as communication subspace analysis [39] More expressive
models, such as DLAG [15), mDLAG [16], and dCSFA [22], capture aspects temporal and spatial
dependencies but typically assume fixed communication structure and can scale poorly to large
datasets. Most recently, large-scale, muulti-animal approaches like multi-taskkmaskinn masking (MtM) have
shown that pretraining across animals with recordings from overlapping brain areas improve cross-area prediction [61]. However, these methods are limited to modeling only the areas observed within given recording session and do not take advantage of key opportunity in multi-animal datasets: using data from multiple recording sessions across animals to infer activity in unrecorded areas of any given session.
In this paper, we introduce a new framework for analyzing multi-animal, multi-area recordings, NeuroPaint, that explicitly leverages shared anatomical structure across animals to model both recorded and unrecorded brain areas. Building on the masked transformer architecture introduced in [61] approach incorporates two key innovations: (1) masking strategy tailored for unrecorded areas, in which unrecorded brain areas are treated the same as masked areas during inference; (2) an architecture that models latent dynamics separately for each brain area, enabling interpretation of area-to-area interactions. To our knowledge, this is the first method specifically designed to infer the dynamics of unrecorded brain areas using data from both other animals and other sessions of the same animal.
We evaluate our method on synthetic data and two large-scale, brain-wide mouse datasets 9
28
We
find that training across animals with overlapping subsets of recorded brain areas enables NeuroPaint to reliably infer the latent dynamics of missing areas. We compare our approach to LFADS 46 and generalized linear regression, in which we directly predict activity in missing areas from observed activity, demonstrating that inferred latents provide significantly more predictive information about unrecorded areas. Taken together, this work introduces a new paradigm for analyzing brain-wide activity, demonstrating how shared structure across animals can be exploited to in-paint dynamics from missing brain areas and opening new possibilities for multi-area analyses that transcend the limitations of single-subject recordings.
2 Related work
Advances in electrophysiology now enable simultaneous recordings across study concurrent signaling among distributed neuronal popu- 24 28 generative models such as DLAG and mDLAG have been developed to uncover shared latent dynamics across brain areas 22 T6). While these models offer interpretability. they typically assume fixed bidirectional communication structures and suffer from limited scalability. Multi-area recurrent dynamical models have also been proposed to capture inter-area interactions more flexibly (20] 35 34 [4, However, none of these approaches can infer the dynamics of brain areas that are unrecorded during given session.
Multi-area models. multiple brain areas 225
44
59 50
lations 1
Stitching or quilting multi-session recordings. Recently, several groups have developed so-called "stitching" 42 or "quilting" (5x] approaches to infer functional connectivity or latent dynamics from multi- recordings with partially overlapping sets of neurons. Some of these methods

---

directly estimate the covariance 55
6 63
noise correlations 41 55 or functional connectivity
using generalized linear models 42 and graphical models [53 while others infer latent factors or Among these, LFADS 33 is most relevant to our work, as infers latent dynamies using nonlinear state space model and can be extended to multi-session settings via linear stitchers. However, state space approaches process time points sequentially and impose strong assumptions on temporal dynamics and noise structure, limiting their scalability and flexibility in large-scale, multi-animal datasets. Moreover, these approaches cannot directly infer neural dynamics in unrecorded brain areas.
neural dynamics across sessions 51 31
Large-scale models for neural analysis. Recent work suggests that scaling models across animals and brain areas can be beneficial, as neural activity exhibits shared structure across individuals and models have emerged as powerful tools for modeling these multi-animal neural datasets, including methods such as POYO+ [3. a multi-task decoder across animals; NDT
regions [37 2 58 62]. Transformer-bart
(88 which uses masked modeling for neural prediction; and MtM 61 and NEDS [62]. which utilize multi-taskkmaskkee to learn spatiotemporal structure in neural activity and the bidirectional relationship between neural activity and behavior. However, these models overlook a key opportunity in multi-animal, multí-area datasets: inferring neural dynamics in unrecorded brain areas of one animal by leveraging shared structure across animals.
3 Methods
In this work, we present NeuroPaint, a transformer-basel masked modeling approach that predicts neural dynamics in both recorded and unrecorded brain areas using observed activity. In multi-animal extracellular recordings, each session samples neurons from only a subset of brain areas, leaving others unrecorded (Fig. E3 E). With sufficient overlap across animals, shared structure can be leveraged to infer missing activity. To model this, we randomly mask recorded areas and train the model to reconstruct them, using set of low-dimensional latents for each brain area (even when unrecorded), to enable inference of missing brain area dynamics across sessions.
3.1 Architecture
The architecture of NeuroPaint has four main components (Fig. (Fig.[C): (1) cross-attention 23 stitcher that maps neural activity from each unmasked brain area to area-specific embedding factors; (2) a tokenizer that transforms these embedding factors into tokens and adds mask tokens for masked and unrecorded areas; (3) a transformer encoder that processes all tokens to produce latent factors for each brain area; and (4) a generalized linear stitcher that reconstructs neural activity from latent factors via a linear readout followed by an exponential nonlinearity. Most parameters are shared across sessions, with few session-specifes parameters noted below.
Cross-attention read-in stitcher. We use a cross-attention stitcher (shown in Fig. I to map neural activity into a shared latent space, producing area-specific "embedding factors" that are consistent across sessions. This module builds on the Perceiver-IO architecture [23] []. using latent tokens to reduce input length. Neural activity is tokenized at the neuron level, with each neuron providing keys and values. A fixed set of learnable latent tokens act as queries, ensuring that the output embedding factors have the same size as the latent tokens. To construct each neuron token, area embedding
we concatenate the neural activity embedding with three learnable embeddings: (encoding the neuron's 's brain area). a hemisphere embedding (indicating left or right hemisphere), and unit embedding (unique to each neuron). The cross-attention stitcher then outputs distinct set of embedding factors for each brain area. For more detailed description of the cross-attention stitcher, Appendix A.2
We adopt cross-and cooss-attention stitcher transformation must be highly expressive. Neural activity can vary substantially across sessions, and aligning into consistent across-session dynamics requires non-linear transformations. At the same time, minimizing the number of parameters is critical to avoid overfitting. The cross-attention stitcher supports parameter sharing across sessions and brain areas, improving generalization. Only the unit embeddings remain session-specific, each session includes a distinct set of recorded neurons.
57 as the input-side place of a linear stitcher B3
3

---

A
Cross-Atiention Silicher Spaa-Spacffie
Tokweker
Sppiin Trals
MLP
300
X
X
Fastora
Time
Araia Humisphere, Unit Enbbeddingo
MersPaiic
NeuroPaint
Cross-Attettio
Toleen
Temporal Embedding
Area Embbeddings
Latent Foctor
GLM Stasher
!!!-!
Emtradang
000
GD0 D
Area
||00
000
Fardoos
Area (Moskod)
Transformer
(Maskedd)
Erbbading
0
000 0
(Umnoooded))
1000
Letort Factors
Figure 1: Schematic illustration of NeuroPaint. (A) Neuropixels probes only record subset brain arcas simultaneously. The brain schematic summarizes probe locations across IBL. sessions, and the table below shows variability in recorded neuron counts by area and session. In contrast to previous approaches (e.g.. mDLAG [16], NDT 5 [88)
MtM [0]] LFADS (3 NeuroPaint is the first method that can infer latent dynamics including unrecorded areas, using large-scale multi-session, multi-animal datasets. (B) use a cross-atee attennattnnioon stitcher to convert spike into tokens, and concatenate area, hemisphere, and unit embeddings to form the keys and values. We use learnable latent tokens as queries to reduce input length and produce area-specific embedding factors. Bold symbols indicate parameters shared across sessions, while non-bold symbols denote area-specific parameters. (C) NeuroPaint uses cunsformer-baset architecture with cross-attention stitchers (as shown in B) encode spike counts into arca-specific embedding factors that then tokenized into temporal tokens. We temporal and area embeddings to tokens, which are passed through the transformer and a linear layer to produce area-specific latent factors. During training, we mask tokens from sampled brain areas, and predict them using area-specific GLM stitchers.
for each brain
Tokenizer for embedding factors. The cross-attention stitcher produces area-specific embedding factors, which are then tokenized by treating each time step as a separate token. We replace tokens corresponding masked areas with learnable mask token. For each unrecorded area, also include learnable mask tokens, which allows the model to infer latent factors for unrecorded areas at test time. Inspired by masked autoencoders, which use positional embeddings preserve the spatial location of image patches [21]. we add a brain-area embedding to each token to encode its anatomical location. Although area information is already added in the aesseatttntion stitcher, adding it to mask tokens is crucial to ensure that the model can distinguish which brain area each token represents. Temporal information encoded using rotary positional embeddings (RoPE) 45 2].
Transformer encoder. To process the sequence tokens from multiple areas and animals, we utilize encoder-only transformer architecture composed of standard transformer blocks followed by a linear layer (52). By computing interactions between all pairs of tokens in the input sequence, the self-attention mechanism of the transformer encoder allows the model to capture dependencies across brain areas and time steps. The encoder outputs latent factors for all brain areas of interest, including those that were unrecorded in a given session.
Generalized linear read-out stitcher. generalized linear read-out stitcher maps the latent factors back to neural activity using a linear layer followed by exponential activation. This component kept intentionally simple to preserve the interpretability of latent factors, ensuring that those for unrecorded areas closely reflect actual neural activity. Its parameters specific each session and brain area. The number latent factors per area chosen based on the maximum linear dimensionality of that area's 's smoothed neural activity across sessions (see Appendix
3.2 Masking scheme
To enable the model to infer missing activity from unrecorded areas, we utilize inter-area masking scheme: in each training batch, randomly select masking percentage between 0% and 60%,

---

and mask out that proportion of the recorded brain areas. We then train the model to reconstruct their activity. Since our goal is to predict activity unrecorded brain areas, inter-area masking alone is insufficient for generalization. To address this, we introduce additional loss terms designed to improve the model's ss ability to infer activity in unrecorded areas.
3.3 Loss function
The loss function consists of three components: the reconstruction loss, consistency loss, and regularization loss. The reconstruction loss is a standard Poisson negative log-likelihood and is used to evaluate the accuracy of predicted firing rates against observed spike counts. The consistency loss constrains each brain area's embedding factors to preserve stable correlation structure between factors across sessions by penalizing deviations from a session-averaged target correlation (see Appendix A.3 and for details). This constraint helps the model generalize to unrecorded areas in given session by encouraging the embedding factors to encode information predictive of all observed areas across sessions. Importantly, the consistency loss is applied to embedding factors (see Fig. IC for definition), which are related to the neural activity through learned nonlinear transformations. That is, we do not assume stable correlation between areas at the level of raw spike data or latent factors. Finally, the regularization loss penalizes rapid temporal fluctuations in the latent space, promoting smoother latent dynamics. Formal definitions of the consistency and regularization losses can be found in Appendix A.3
4 Experiments
Our approach builds upon two core assumptions: (1) neural activity in each brain area lies on a low-dimensional manifold, allowing a fixed set of fow-dimensional latent factors to explain most of the variance in the high-dimensional neural data; and (2) a consistent and potentially nonlinear mapping exists between the latent dynamics of different brain areas. To test our proposed model, we first apply its performance on two large-scale Neuropixels datasets spanning multiple animals and brain areas, where these assumptions are expected to hold approximately.
synthetic dataset constructed to satisfy both assumptions. We then evaluate
In the synthetic data, ground truth firing rates from unrecorded areas are available. For the real data, we treat one recorded area per session as unrecorded by holding it out during training, allowing evaluation during test with ground ground-truth data. We measure how well the inferred latent factors capture dynamics in the held-out area by training. during the test phase, generalized linear model (GLM) to predict firing rates for that area from its inferred factors. Note that these GLMs cannot be learned during training for the held-out areas because their spike data are not made available during training.
All trials used for evaluating the latent factors come from a test set that was excluded during NeuroPaint training. We split this test into 60% training and 40% testing trials for cross-validation of the GLM performance. Performance is quantified using deviance fraction explained (DFE), a normalized goodness-of-fit metric (see Appendix A.7]. We use DFE as an intuitive, bounded metric where indicates perfect prediction, O matches a null model using the average firing rate, and negative values indicate worse- worse-than-
performance. However, when the null model closely matches the ground truth. DFE can be low even if the model performs well. Conversely, DFE values near may reflect overfitting-no not in the conventional sense of poor generalization across data splits, but that the model may capture high-frequency spike noise rather than meaningful structure in the firing rates.
4.1 Datasets
Synthetic dataset. To generate synthetic neural data, we simulate an autonomous recurrent neural network (RNN) comprising five brain areas, each with 200 RNN units (see Appendix A.6). Units within each area are densely connected (all-to-all), while inter-area connections are sparse (1%). The recurrent weights induce chaotic dynamics, resulting in trial-to-triall variability driven by different initial conditions. Counterintuitively, the chaotic dynamics generated by such randomly connected networks are often low-dimensional- -oweenniiiiooooouuiinnnnne a subspace whose dimensionality is much smaller than the number of RNN units [II. For each synthetic session, spike trains are produced using -peccfc- GL. with sparse weights (2%), mapping RNN activity to spikes. This setup yields distinct neuron populations per session, each arising from partially overlapping subsets of RNN units.
5

---

Synthelic session
Synthetic session
Devience Explained B
Prediction performances for unrecorded areas (10 sessions)
Neuree
Troi
0.1
0.0
Bascline (OLM)
NouroPain
Basslin (LMADS]
Nourofain: recon. + consist. 0977
recon, rog.
Spike data
Splke data
spika cata from recorded arozs
GLM
splke data from the
GT
spike data from unmoostted sewa
unrecorded crea
uncocorded areao
Thaa
Thal
spilm data from recorded
LFADS
whered latwrt factors
spike data from the unreconded arca
GT. DFE 0.23
GT. DFE 0.37
for recorded anass
Thal
Thal
5.0
NeuroPam
latent factors for the unrecorced area
spike data from the unrecorded area
2.5
recorded areas
NeuroPaint, DFE 0.18
NeuroPaint, DFE 0.33
fraction exclained Bynthelic session (urrocoded)
Meon fring rale
Thtal
Thal
5.0
2.5
recon: ConNi
0.5
0.5
0.5
3.4
LFADS, DFE 0.15
LFADS, DFE 0.28
3.2
Thta
Thai
5.0
2.5
GLM, DFE 0.12
GLM, DFE 0.24
2.8
Thaal
5.0
2.5
0.5
0.0
0.5
Time
Time
Basoline (GLM)
Baseline (UFADS)
Upper bound (GT)
Figure 2: Quantitative and qualitative comparison of NeuroPaint with baseline models on synthetic datasets. (A) Synthetic spike trains are generated across 10 sessions using a shared underlying RNN, with five brain arcas of 200 units cach. Spike trains are generated using session- and arca-specific GLMs with sparse, random weights, allowing partially overlapping RNN units to contribute differently across sessions. (B) Prediction performance for synthetic neurons in unrecorded and one standard error across neurons each method, while the flow diagram shows the computation of DFE. (C) Spike data and ground truth firing rates (GT) for two example synthetic neurons, compared with firing rate predictions from NeuroPaint (trained with reconstruction and consistency losses) and the two baselines (GLM and LFADS). (D) Prediction performance for synthetic neurons from area (unrecorded) in session 1. Each dot represents neuron, showing DFE achieved by NeuroPaint (trained reconstruction and consistency loss), the two baselines, and the upper bound. Dots are color-coded by each neuron's 's mean firing rate.
across 10 sessions. The bar plot displays the mean DFE
In each session, out of 5 areas are designated as "recorded," and the rest as unrecorded. Each area contains 20 60 simulated neurons per session.
IBL dataset. We use the International Brain Laboratory (IBL) brain-wide map dataset 28 B). This dataset consists of Neuropixels recordings collected from 12 labs which utilize a standardized experimental pipeline. The recordings target 279 brain areas across 139 adult mice performing the same visual decision- bocision-making task. The probe was localized after the experiments using reconstructed histology and the brain areas annotated. We utilize trial- aligedd, spike-sorted data from 20 mice (1 session per mice), and select 8 brain areas for analysis: PO, LP, DG, CA1, VISa, VPM, APN, and MRN (see Appendix A. for full area names). From these recordings, neurons for training and evaluation. We bin the spike trains using 10 ms windows and we fix the trial-length to seconds (200 time bins). Trials are categorized into two types: left choice and right choice, Each of the 20 selected sessions has 3 to 7 brain areas simultaneously recorded. We randomly hold out one recorded brain area per session for evaluation.
(Fig. A
have total of 21568
MAP dataset. The Mesoscale Activity Project (MAP) dataset records neural activity underlying memory-guided movement in mice using Neuropixels probes targeting motor cortex, thalamus, midbrain, and hindbrain regions, spanning 293 cortical and subcortical structures [TO] (Fig 3 F). includes 173 sessions from 28 mice. We use trial-aligned, spike-sorted data from 40 sessions across 16 mice, binned in 10 ms windows over fixed 4-second trials (400 time bins). Analyses are restricted to two trial types, hit left and hit right, which are also used to compute the consistency loss. For our analysis, we focus on 8 brain areas, highly relevant to the task [19 29 ALM, 1Orb, vIOrb, Pallidum, Striatum, VAL-VM, MRN, and SC (see Appendix A. for full area names). Each of the selected sessions has to brain areas simultaneously recorded. We randomly hold out one recorded brain per session for evaluation.
and 2 orbital areas:

---

4.2 Baselines
We compare NeuroPaint against linear and non-linear baselines. We use a standard generalized linear model (GLM) for our linear baseline. For our con-linear baseline, we compare to Latent Factor Analysis via Dynamical Systems (LFADSS) 33 46], a sequential variational autoencoder that has demonstrated strong performance in capturing latent neural dynamics. We utilize re-implemented version of LFADS 38 for all our analyses, as discussed below.
GLM. We implement GLMs that map from instantaneous spike activity in all the recorded areas to instantaneous spike activity in an unrecorded (in the synthetic dataset) or held-out (in the Neuropixels datasets) area. The GLM consists of linear layer followed by an exponential nonlinearity, and is trained using the Poisson negative log-likelihood loss. We fit a separate GLM for each session and each held-out area.
LFADS. We ce-implement session LFADS [46 331 to model shared neural dynamics across multiple animals. The objective of LFADS maximize the likelihood of observed neural activity by reconstructing spike trains from a oow-dimensional latent dynamical system. For each session's 's neural population, multi-session LFADS learns a linear projection layer (read-in stitcher) to embed the neural activity into shared latent space. It also learns a corresponding set of read-out stitchers to map the latent representations back to neural activity space. Unlike NeuroPaint, LFADS embeds activity from all recorded areas into a shared latent space but cannot infer latent dynamics for unrecorded areas, as is only trained to reconstruct observed data and does not use masked modeling to predict missing areas. Most importantly, LFADS lacks area-specific latent factors, which limits its interpretability in multi-area regimes. We apply principal component regression to select the latent dimensionality and to pre-condition both the read-in and read-out stitchers [33 (see Appendix (A.S). A.5).. Further details on architecture, weight initialization, and implementation can be found in Appendix 10 and AA.II
In summary, we evaluate the ability of each approach to predict neural activity in unrecorded areas, by using a supervised GI.M (trained during the test phase) to predict neural activity in held-out areas. Specifically, we compare the predictive power of three types of inputs to the GLM: (1) spike activity from recorded areas (corresponding to the GLM baseline), (2) LFADS latent factors shared across recorded areas, and (3) NeuroPaint's 's area-specific latent factors for the unrecorded area. Although we are comparing NeuroPaint's s performance with these other baselines, they are not comparable in the important sense that NeuroPaint is the only approach that explicitly learns separate latent factors for each brain area, including those unrecorded ones, whereas the two baselines cannot infer srea-specific latent dynamics.
5 Results
5.1 Synthetic dataset
We evaluate NeuroPaint synthetic dataset with known ground truth firing rates, enabling compu- tation of an upper-bound DFE using spike data in the unrecorded areas and true rates. The model is trained on recorded areas from 10 synthetic sessions and evaluated on its ability to infer single- neuron firing rates in unrecorded areas. The architecture follows Section 3.1 excluding hemisphere embeddings due to the absence of hemispheric structure in the simulation. We compare two model variants, one trained with reconstruction plus regularization loss, the other with reconstruction plus consistency loss; combining all three losses (not shown) results in over-regularization and perfor- mance degradation below baseline. This is likely due to the fast, high-firing-rate dynamics in the synthetic data; as suggested by an additional experiment on separate low-firing-rate synthetic dataset, where the full model (all three losses) outperforms the ablated variant lacking one loss term (see Appendix A.6 and Supplementary Fig. 2) We compare each model against GLM and LFADS baselines and the upper bound. The consistency-loss variant outperforms the two baselines and the regularized model, and is the closest to the upper bound (Fig. PB. D). As shown in Fig. 2C, NeuroPaint more accurately captures temporal structure than GLM, which tends to overfit to Poisson noise, and slightly outperforms LFADS in recovering fine-grained features of the ground truth firing rates. These results show that under the idealized conditions of the synthetic data, i.e. low-dimensional activity and consistent cross-area mapping, NeuroPaint can accurately infer neural dynamics in unrecorded areas.
7

---

A IBL dataset
Namber of neurons
MAP dalaso!
Number neurons
200
100
Seesin
0.06
session For CA3 (held outi
High teee J Low
Desforee Tranter Exphyeod
Fell Hone
session For
0.30
2.04
0.00
Scundart
0.09
0.02
0.04
0.02
0.00
D
Suceiie (LFAOS)
log aplie rales
0.00
Seasing IL1A31
0.0
0.9 log colo
252, session
125, secsion CAL
H
7. session ALM
NeuruPaint, DFE 0.21
NeuroPaint, DFE 0.06
NeuroPaint, DFE 0.21
NeuroFaint, DFE 0.15
Tra
0.2
6.1
0.05
0.02
0.00
6.2
LIADS, DFE 0.14
0.2
LFADS,
LFADS, OFE 32
LFADS, DFE
0.05
0.2
0.00
0.02
50 osser
50m OF15O
detay
posponse
St.m
respomse
Figure 3: Quantitative and qualitative comparison of NeuroPaint with baseline models on IBL and MAP datasets. (A) Number neurons recorded per area in 20 IBL. sessions. (B) In the IBL experimental setup, mice perform a visual decision-making task by turning a wheel to the left or right to indicate whether the stimulus presented on the left or right screen (28). (C) Left: The mean and standard error of DFE for LFADS and NeuroPaint, calculated for neurons from held-out areas pooled across 20 IBL sessions, excluding the worst 1% of neurons for visualization; see Fig. ę for results without exclusion. Right: Per-neuron DFE for the held-out area (CA1) of session 18. (D) Spike trains and predicted firing rates for two example neurons, compared between NeuroPaint and LFADS predictions. (E) Number of neurons recorded area in 40 sessions from MAP dataset. (F) The MAP dataset records neural activity from mice performing memory guudddd directional licking task, in which they are instructed lick left right according to auditory tones presented before fixed delay period.
[. (G, H) show similar content to (C, D), but with neurons from held-out areas pooled across 40 MAP sessions
(see Fig. STB for results with the worst 1% exclusion).
5.2 IBL and MAP datasets
In the Neuropixels datasets, although we have ground truth spiking data, the absence of ground truth firing rates precludes computation of an upper-bound DFE. While absolute DFE values may appear lower than those observed in the synthetic data, they reflect the intrinsie performance ceiling set by the low and sparsely modulated firing rates typical of real neuronal activity. Rather than indicating suboptimal performance, these values underscore the challenge of the task and the robustness of the model's 's performance under realistic biological constraints.
We train two separate NeuroPaint models, one for the 20 IBL sessions and one for the 40 MAP sessions. In both datasets, we compare NeuroPaint to LFADS, which is the highest performing baseline for single-neuron prediction accuracy in held-out areas. The GLM baseline performs poorly across both datasets due to its inability to model non-linear, cross-area interactions in real neural data 56 54
and is therefore omitted from the comparison (results are provided in Appendix A.9)..
Across held-out areas, NeuroPaint outperforms LFADS by a large margin in both the IBI. and MAP datasets (Fig. 3C, G; Fig. $[]. Example raster plots of individual neurons further show that NeuroPaint captures structured, single-trial dynamics that LFADS fails to recover (Fig, 3D. H). In particular, LFADS exhibits large outlier errors for some neurons (e.g., neuron 20 in Fig. BH), whereas NeuroPaint delivers more stable and accurate firing rate predictions. In the neural data, which has relatively low firing rates compared the synthetic data of Fig. 2, the combination of all three loss terms performs best: ablation experiments show that removing either the regularization or the consistency loss degrades performance relative to the full NeuroPaint model (see Appendix A. Supplementary Fig. Moreover, increasing the number of parameters in the LFADS model does not close the performance gap to NeuroPaint (see Appendix A51 suggesting that NeuroPaint's advantage is not explained by model size alone. These results demonstrate that NeuroPaint not
8

---

only infers interpretable, area-specific latent factors for unrecorded brain regions, but also achieves state-of-the-ar predictive performance.
5.3 Interpretable area-specific latent dynamics revealed by NeuroPaint
We use the MAP dataset to illustrate that NeuroPaint generates consistent and interpretable area- specific latent dynamics across trials and sessions. See similar results for IBL dataset in Ap- pendix A.14
Preprocessing. To highlight dynamic structure over static offsets, we subtract the temporal mean of each trial from each latent factor before computing correlations or visualizing dynamics in Fig. This preprocessing step helps mitigate spurious correlations driven by differences in the latent factor's s non-zero temporal means.
Consistent and context-dependent latent dynamics. We evaluate (1) the consistency of latent factors across trials for both recorded and unrecorded areas, and (2) the context-dependent variability of latent factors that reflects distinct behavioral conditions, such as hit left vs. hit right trials where mice respond by licking in different directions. We find that the inferred area-specific latent factors exhibit consistent temporal structure across trials within the same behavioral context (Fig. A even in sessions where the corresponding area unrecorded (Fig. 4B). Additionally, the latent factors exhibit clear context-dependent variability that captures task-relevant differences; for instance, the latent factors for area SC show distinct patterns between hit left and hit right trials (Fig. A To quantify the consistency and context-dependen variability,, we compute the Pearson correlation between latent factors across all trial pairs, averaged over brain areas (see Appendix T First, latent factors from the same brain areas exhibit positive correlations across trials, regardless of whether those areas were recorded or not in a given session. Second, trial pairs of the same type have higher correlations than those of different types, suggesting that the latent factors capture context-dependent variability. (see Appendix A.. for more details)
To directly evaluate whether the latent factors capture stimulus- or rehavior-dependee variability, we performed decoding analysis from the latent factors. We found that the latent factors most areas reliably support decoding for both stimulus and behavioral choice, demonstrating that they encode meaningful, resk-relevant information. (see Appendix(A.. A.17 for more details)
Inferring area-to-area interactions from inpainted neural data. We demonstrate that Neu- roPaint's 's inferred latent factors enable novel analyses of arca-to-area interactions during behavior. As shown in Fig. 4C, the latent factors from both recorded and unrecorded areas in a single trial reveal rich and distinct temporally structured activity across the 8 selected brain areas. With fully inpainted neural picture at a single-trial resolution across 40 sessions, NeuroPaint enables analyses on area-to-area interactions that were previously infeasible with Neuropixels recordings, which lack simultaneous coverage of all areas. As one such analysis, we quantify representational similarity across all pairs of brain areas. For each area and each trial period (stimulus, delay, and response), we first compute a time-by-time representational dissimilarity matrix (RDM), defined as one minus the pairwise correlation between flattened latent factors. We then measure area-to-area similarity by correlating the upper-triangular elements of these RDMs, averaged across trials over multiple sessions (Fig. (D)). This analysis reveals that inter-area relationships evolve across behavioral epochs: similarity is relatively low during stimulus and delay periods but increases during the response period, which is consistent with previous findings of increased inter-area coordination during motor output [9. While Chen et al. [9] were limited to analyzing interactions between ALM and a few simultaneously recorded areas, NeuroPaint overcomes this limitation by enabling comparisons across all area pairs, even including unrecorded areas, via latent inpainting. Furthermore, we observe that the two orbital areas (IOrb and vIOrb) exhibit high mutual similarity, reflecting their shared anatomical and functional roles [2 In contrast, these areas show weak similarity to the remaining brain areas, even during the response period, which aligns with prior knowledge about the involvement of the selected non-orbital areas in task-relevant computations 19 29
8 49 9

---

hi left stats
SC, single trai, session 5
hit right itals
SC Recoried St. Lnwarree
SC, Hit left, Single tria
Irial
thall
trial
Lasen Fartor
Sassion
Seasion
Saxsion 42
Time averaged activly subrracted
Time Session
trist
trall
rrill 3
triat
Session
Seasion
Laben: factore
24
-25
Rocorded
wroas session 172. right, single
Represantation simlarty
ALM
Körb
viorb
Pullidum
Stmukus period
Detry period
Reeponse period
Iee-
doo
46
Striatum
VAL-N
MRN
SC
65
04
to
3
-25
ditay
ieprore
Figure 4: NeuroPaint's 's area- specifce latent factors are consistent, capture context dependent variability. and enable inference of area-to araa-t interactions. (A) Inferred latent factors for the Superior Colliculus (SC), an unrecorded area, during example trials across the hit left and hit right conditions in single session. The bottom row shows the same latent factors with the trial- averaged activity subtracted. (B) Inferred single-trial latent factors for SC (both recorded and unrecorded) across six sessions. (C) Latent factors for all eight areas (both recorded and unrecorded) in a hit right trial from a single session in the MAP dataset. (D) Representation similarity analysis (RSA) of latent factors across eight brain areas (recorded and unrecorded) for the stimulus, delay, and response periods in the MAP dataset, with values averaged across trials and pooled over sessions.
6 Discussion
In this work, we introduce NeuroPaint, a masked transformer-basse model that infers the dynamics of unrecorded brain areas by leveraging shared activity structure across animals. Our experiments on synthetic and large-scale Neuropixels datasets show that NeuroPaint outperforms both linear and non-linear baselines, including LFADS and GLMs, in predicting single-neuron activity in held- out areas. Beyond predictive accuracy, NeuroPaint produces interpretable latent dynamics that are consistent across sessions and sensitive to behavioral context, enabling new forms of cross-area and cross-session analysis that were inaccessible with existing tools.
Despite these strengths, several limitations remain. First, training NeuroPaint is computationally expensive, as the self-attention mechanism scales quadratically with the number of brain areas and time steps. Future work could address this by incorporating sparse or low-rank attention mechanisms g. Second, the latent dimensionality for each brain area is not manually selected, it estimated via principled procedure (see Appendix A.5). Nonetheless, the procedure involved design choices that introduce some arbitrariness, and we find that the selected number of latent factors often exceeds the intrinsic dimensionality of the predicted firing rates in each brain area (see Appendix A.18). More automated hyperpameter selection methods, such as Population Based Training (26.. could reduce this arbitrariness. Finally, while we demonstrate proof-of-conced results on two datasets, our experiments involve a limited subset of brain areas and sessions relative to the full datasets. Scaling to hundreds of sessions is straightforward, but extending to hundreds of brain areas will require architectural innovations and improved training strategies. Nonetheless, this represents a critical step toward building truly brain-wide models.
Acknowledgments and Disclosure of F Funding
This project was supported by the Simons foundation (543017). National Institutes of Health (I551DA056397, IK99NS144599 the Gatsby Foundation. This work used NCSA Delta GPU at National Center for Suppercomputing Applications through allocation OTH250001 from the Ad-
10

---

vanced Cyberinfrastructure Coordination Ecosystem: Services & Support (ACCESS) program, which supported by U.S. National Science Foundation grants (2138259, 2138286, 2138307, 2137603, and 2138296). We are grateful to Susu Chen, Thinh Nguyen, Nuo Li for discussions about accessing the MAP dataset. We thank Mehdi Azabou for discussion about the design of cross-attention stitcher. We thank Manuel Beiran and David Clark for discussion about neural dynamics in random recurrent neural networks used in the synthetic dataset.
References
[1] Misha B Ahrens, Jennifer M Li, Michael Orger, Drew Robson, Alexander Schier, Florian Engert, and Ruben Portugues, Brain-wide neuronal dynamics during motor adaptation in zebrafish. Nature, 485 (7399))471-477, 2012.
[2] Mehdi Azabou, Vinam Arora, Venkataramana Ganesh, Ximeng Mao, Santosh Nachimuthu, Michael Mendelson, Blake Richards, Matthew Perich, Guillaume Lajoie, and Eva Dyer. unified, scalable framework neural population decoding. Advances in Neural Information Processing Systems, 36: 44937-44956, 2023.
[3] Mehdi Azabou, Krystal Xuejing Pan, Vinam Arora, Ian Jarratt Knight, Eva L Dyer, and Blake Aaron Richards. Multi-session, multi-task neural decoding from distinct cell-types and brain regions. In The Thirteenth International Conference on Learning Representations, 2025. URL http::://peeereeeeesee het/forum/id-IuuUvcCCOC
[4] Kush Banga, Julius Benson, Jai Bhagat, Dan Biderman, Daniel Birman, Niccolò Bonacchi, Sebastian Bruijns, Kelly Buchanan, Robert AA Campbell, Matteo Carandini, et al. Reproducibility of in vivo electrophysiological measurements in mice. eLife, 13:RP100840, 2025.
[5] William E Bishop and Byron M Yu. Deterministic symmetric positive semidefinite matrix completion. Advances in Neural Information Processing Systems, 27, 2014.
[6] Andersen Chang. Lili Zheng, and Genevera I Allen. Low-rank covariance completion for graph quilting with applications functional connectivity. arXiv preprint arXiv:2209.08273. 2022.
[7] Andersen Chang, Lili Zheng, Gautam Dasarathy, and Genevera I Allen. Nonparanormal graph quilting with applicatioon to calcium imaging. Stat, 12(1)xe623, 2023.
[8] Beidi Chen. Tri Dao, Eric Winsor, Zhao Song, Atri Rudra, and Christopher Ré. Scatterbrain: Unifying sparse and low-rank attention. Advunces in Neural Information Processing Systems, 34:17413-1741 2021.
[9] Susu Chen, Yi Liu, Ziyue Aiden Wang, Jennifer Colonell, Liu D. Liu, Han Hou, Nai-Wen Tien, Tim Wang, Timothy Harris, Shaul Druckmann. Nuo Li, and Karel Svoboda. Brain-witd wide neural activity underlying memory-guide movement. Cell, 187(3)667-698) 691.e16, February 2024. ISSN 00928674, doi: 10. 10. 2023.12.035 URL https:
elsevier com wttp://eetiieee////0////////5//0
[10] Susu Chen, Yi Liu, Ziyue Aiden Wang, Jennifer Colonell, Liu D Liu, Han Hou. Nai-Wen Tien, Tim Wang. Timothy Harris, Shaul Druckmann, et al. Brain-wide neural activity underlying memory guided movement. Cell, 187(3):676 077766691111 191, 2024.
[11] David Clark, LF Abbott, and Ashok Kumar. Dimension activity in random neural networks. Physical Review Letters, 131(11):11840) 2023.
[12] Marlene R Cohen and Adam Kohn. Measuring and interpreting neuronal correlations. Nature neuroxcience, 14(7):811 1-189, 2011.
[13] Peiran Gao and Surya Ganguli. On simplicity and complexity in the brave new world of large-scale neuroscience. Current opinion in neurobiology, 32:148 155, 2015.
[14] Joshua Glaser, Matthew Whiteway, John P Cunningham, Liam Paninski, and Scott Linderman. Recurrent switching dynamical systems models for multiple interacting neural populations. Advances neural information processing systems, 33:14867: 18877-1888888 2020.
[15] Evren Gokeen, Anna Jasper, João D Semedo, Amin Zandvakili, Adam Kohn, Christian K Machens, and Byron M Yu. Disentangling the flow of signals between populations of neurons. Nature Computational Science, 2(8):512-52 525, 2022.

---

[16] Evren Gokcen, Anna Jasper. Alison Xu, Adam Kohn, Christian K Machens, and Byron M Yu. Uncovering motifs of concurrent signaling across multiple neuronal populations. Advances in Neural Information Processing Systems, 36, 2024.
[17] Joshua I Gold and Michael N Shadlen. The neural hasis of decision making. Annu. Rev. Neurosci., 30(1)) 535-574, 2007.
[18] Zengeai Gao, Hidchiko K. Inagaki, Kayvon Daie, Shaul Druckmann, Charles R. Gerfen, and Karel Svoboda. Maintenance of persistent activity in frontal thalamocortical loop. Nature, 545(7653) May 2017. ISSN 0028-0836, 1476-4687. doi: 10. 1038/nature223244. URLhttpa. //wpw.//wwwww. www. nature. com/
articles/nature2225
[19] Zengcai V. Guo, Nuo Li, Daniel Huber, Eran Ophir, Diego Gummisky Jonathan T. Ting, Guoping Feng. and Karel Svoboda. Flow of Cortical Activity Underlying Tactile Decision in Mice. Neuron, :1)19 194, January 2014. ISSN 08966273. doi: 10. (0.6016/j.... (0..000/..euooo.20000 elsevier http:////eeeeeeee////////////1/0
20t..m..11.....00000 URL https::::////iiiiigga
[20] Diksha Gupta, Charles D Kopec, Adrian G Bondy, Thomas Z Luo, Verity A Elliott, and Carlos D Brody. multi-region recurrent circuit for evidence accumulation in rats. BioRxiv, 2024.
[21] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 16000-16009, 2022.
[22] Rainbo Hultman, Kyle Ulrich, Benjamin D Sachs, Cameron Blount, David E Carlson, Nkemdillm Ndubuizu, Rosemary C Bagot, Eric M Parise, Mai-Anh T Vu, Neil M Gallagher, ct al. Brain-wide electrical spatiotemporal dynamics encode depression vulnerability. Cell, 173(1):166-188)) 2018.
[23] Andrew Jaegle. Felix Gimeno, Andy Brock, Oriol Vinyals, Andrew Zisserman, and Joao Carreira. Perceiver: General perception with iterative attention. In International conference on machine learning, pages 4651- 4664. PMLR, 2021.
[24] Xiaoxuan Jia, Joshua H Siegle, Séverine Durand, Greggory Heller, Tamina K Ramirez, Christof Koch, and Shawn R Olsen. Multi-regional module-based signal transmission in mouse visual cortex. Neuron, 110(9): 1585-1598, 2022.
[25] James Jun, Nicholas Steinmetz, Joshua Siegle. Daniel Denman, Marius Bauza, Brian Barbarits. Albert K Lee. Costas A Anastassiou, Alexandru Andrei, Çagatay Aydın, et al. Fully integrated silicon probes for high-density recording of neural activity. Nature, 551(7679):232-236, 2017.
[26] Mohammad Reza Keshtkaran, Andrew R Sedler, Raeed H Chowdhury, Raghav Tandon, Diya Basrai, Sarah I. Nguyen, Hansem Sohn, Mehrdad Jazayeri, Lee E Miller, and Chethan Pandarinath. A large-scale neural network training framework for generalized estimation of single-trial population dynamics. Nature Methods, 19(12):1522-13 15777 2022.
[27] Andrei Khilkevich, Michael Lohse, Ryan Low, Ivana Orsolic, Tadej Bozic, Paige Windmill, and Thomas D Mrsic-Flogel. Brain-wide dynamics linking sensation to action during decision-making. Nature, 634 (8035):8909900, 2024.
[28] International Brain Laboratory, Brandon Benson, Julius Benson, Daniel Birman, Niccolò Bonacchi, Keénia Bougrova, Scbastian A Bruijns, Mattco Carandini, Joana A Catarino, Gaelle A Chapuis, et al. A brain-wide map of neural activity during complex behaviour. biorxiv, pages 2023-07, 2023.
[29] Nuo Li, Tsai-Wen Chen, Zengcai V. Guo, Charles R. Gerfen, and Karel Svoboda. A motor cortex circuit for motor planning and movement. Nature, 1197751) March 2015. ISSN 0028-0836, 1476-4687. doi: 10. 0088/mature14178 httpp:/tt http://wwww.aature..ooraaaaaaeea
[30] Camden J MacDowell, Alexandra Libby, Caroline I Jahn, Sina Tafazoli, Adel Ardalan, and Timothy J Buschman. Multiplexed subspaces route neural activity across brain-wide networks. Nature communica- tions, 16(1):3359 2025.
[31] Marcel Nonnenmacher, Srinivas Turaga, and Jakob H Macke. Extracting lww-dimensional dynamics from multiple large-scale neural population recordings by learning to predict correlations. Advances in neural information processing systems, 30, 2017.
[32] Seung Wook Oh, Julie A Harris, Lydia Ng. Brent Winslow, Nicholas Cain, Stefan Mihalas, Quanxin Wang. Chris Lau, Leonard Kuan, Alex M Henry, et al. A mesoscale connectome of the mouse brain. Nature, 508 (7495):007-21) 224, 2014.
12

---

[33] Chethan Pandarinath, Daniel J O' Jasmine Collins, Rafal Jozefowicz. Sergey D Stavisky, Jonathan C Kao, Eric M Trautmann, Matthew T Kaufman, Stephen I Ryu, Leigh R Hochberg. al. Inferrigg single-trial neural population dynamics using sequential auto-encoders. Nature methods, 15(10)880-813 2018.
[34] Matthew G Perich and Kanaka Rajan. Rethinking brain-wide interactions through multi-region 'network of networks' models. Current opinion in neurobiology, 65:1 65:146-151, 2020.
[35] Matthew G Perich, Charlotte Arlt, Sofia Soares, Megan E Young, Clayton P Mosher, Juri Minxha, Eugene Carter, Ueli Rutishauser, Peter H Rudebeck, Christopher D Harvey, al. Inferring brain-wide interactions using constranndd recurrent neural network models. BioRxiv, pages 2020- 12, 2020.
[36] Lorenzo Posani, Shuqi Wang, Samuel P Muscinelli, Liam Paninski, and Stefano Fusi. Rarely categorical, always nigh-dimenssonal: how the neural code changes along the cortical hicrarchy. bioRxiv, pages 2024-1 11, 2025.
[37] Mostafa Safaie, Joanna C Chang, Junchol Park, Lee E Miller, Joshua T Dudman, Matthew Perich, and Juan A Gallego. Preserved neural dynamics across animals performing similar behaviour, Nature, 623 (7988):765-7711 2023.
[38] Andrew R Sedler and Chethan Pandarinath. Ifads-torch: A modular and extensible implementation of latent factor analysis via dynamical systems, arXiv preprint arXiv:2.00.0 2023.
[39] João D Semedo, Amin Zandvakili, Christian K Machens, Byron M Yu, and Adam Kohn. Cortical areas interact through a communication subspace. Neuron, 102(1):249-259 2019.
[40] Haim Sompolinsky, Andrea Crisanti, and Hans-Jurgen Sommers. Chaos in random neural networks. Physicul review letters, 61()) 1988.
[41] Oleksandr Sorochynskyi. Stéphane Deny, Olivier Marre, and Ulisse Ferrari. Predicting syuchronous firing of large neural populations from sequential recordings. PLoS computational biology, 17())=e108850,, 2021.
[42] Daniel Soudry, Suraj Keshri, Patrick Stinson, Min-hwan Oh, Garud Iyengar, and Liam Paninski. Efficient" shotgun" inference neural connectivity from highly sub- sub-sa activity data. PLoS computational biology. 11(10):e100444, 2015.
[43] Nicholas Steinmetz, Peter Zatka-Hart Haas, Mattco Carandini, and Kenneth D. Harris. Distributed coding of choice, action and engagement across the mouse brain. Nature, 576(7786);266 273, December 2019 ISSN 0028-0836, 1476-4687.
doi: 10. 10.10088/441186--111-77-7---- URL httpp:// www. nature. com/
[44] Nicholas A Steinmetz, Cagatay Aydin, Anna Lebedeva, Michael Okun, Marius Pachitariu, Marius Bauza, Maxime Beau, Jai Bhagat, Claudia Böhm, Martijn Broux, et al. Neuropixels 2.0: A miniaturized high- density probe for stable, long-term brain recordings. Science, 372/6539) 2021.
[45] Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neuroconputing, 568: 127063, 2024.
[46] David Sussillo, Rafal Jozefowicz, LF Abbott, and Chethan Pandarinath. Lfads-latent factor analysis via dynamical systems. arXiv preprint (Et:::600006111 2016.
[47] Karel Svoboda, susu chen, and Nuo Li. Single cell reconstruction data from MouseLight 2023. doi: (0225788jancciaa2448833
database and related Matlab analysis scripts.
vl. httLLltttts://aaaaee htw.aaaeiia....aareeeee...e figshare data_ Lirta_from_hie Mrwm_mooseeiigttttmmm www.eeeeihggdddddaaeeeeeeeeeeeee aarpee__nddd_eeeeeeeeee aanddddeeeed__aaaaeaaeeeeeeeeeee Mattldddaaaabbbaaaaaa httpaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[48] Antti Tarvainen and Harri Valpola. Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results. Advances neural information processing systems, 30, 2017.
[49] Alyse Thomas, Weiguo Yang, Catherine Wang, Sri Laasya Tipparaju. Guang Chen, Brennan Sullivan, Kylie Swickatowski, Mahima Tatam, Charles Gerfen, and Nuo Li. Superior colliculus bidircction- ally modulates choice activity in frontal cortex. Nature Communications, 14(1):7358, November 2023. ISSN 2041-1723. doi: 10. 001000884114477002-45522222 URL httpp://///t /ww. nwwure. www.:auee..c///ticcceeeeeee 0146670023-435222
[50] Eric M Trautmann, Janis K Hesse, Gabriel M Stine, Ruobing Xia, Shude Zhu, Daniel J O' O'Shea, Bill Karsh, Jennifer Colonell, Frank F Lanfranchi. Saurabh Vyas, et al. Large-scale high high-densit density brain-wide neural recording in nonhuman primates. bioRvir, pages 2023 02, 2023.
13

---

[51] Srini Turaga, Lars Buesing, Adam M Packer, Henry Dalgleish, Noah Pettit, Michael Hausser, and Jakob H Macke. Inferring neural population dynamics from multiple partial recordings of the same neural circuit. Advancey Neural Information Processing Systems, 26, 2013.
[52] A Vaswani. Attention is all you need. Advances in Neural Information Processing Systems, 2017.
[53] Giuseppe Vinci, Gautam Dasarathy, and Genevera Allen, Graph quilting: graphical model selection from partially observed covariances. arXiv preprint arXiv:1912.05573, 2019.
[54] Martin Vinck, Cem Uran, Georgios Spyropoulos, Trene Onorato, Ana Clara Broggini, Marius Schneider, and Andres Canales-Johnson. Principles of large-scale neural interactions. Newron, 111(7):987-002. 2023.
[55] Adrien Wohrer, Ranulfo Romo, and Christian K Machens. Linear readout from neural population with partial correlation data. Advances in Neural Information Processing Systems, 23, 2010.
[56] Ji Xia, Anna Jasper, Adam Kohn, and Kenneth D Miller. Cireuit-motivated generalized afline models characterize stimulus-dependent visual cortical shared variability, Iscience, 27(8), 2024.
[57] Joel Ye and Chethan Pandarinath. Representation learning for neural population activity with neural data transformers. arXiv preprint arXiv:2108.01211 2021.
[58] Joel Ye, Jennifer Collinger, Leila Wehbe, and Robert Gaunt. Neural data transformer 2: multi-context pretraining for neural spiking activity. Advances in Neural Information Processing Syxtems, 36, 2024.
[59] Zhiwen Yc, Andrew M Shelton, Jordan Shaker, Julien Boussard, Jennifer Colonell, Sahar Manavi, Susu Chen, Charlie Windolf, Cole Hurwitz, Tomoyuki Namima, et al. Ultra-high density electrodes improve detection, yield, and cell type specificity of brain recordings. bioRxiv, 2023.
[60] Yizi Zhang, Hanrui yu, Cole Hurwitz, Shuqi Wang, Charles Findling, Felix Hubert, Alexandre Pouget, International Brain Laboratory, Erdem Varol, and Liam Paninski. Exploiting correlations across trials and behavioral sessions to improve neural decoding. bioRxiv, 2024.
[61] Yizi Zhang, Yanchen Wang, Donato Jiménez-Benetó, Zixuan Wang, Mehdi Azabou, Blake Richards, Renee Tung, Olivier Winter, Eva Dyer, Liam Paninski, et al. Towards a" universal translator" for neural dynamics single-cell, single-spike resolution. Advances in Neural Information Processing Systems, 37: 80495 80521, 2024.
[62] Yizi Zhang. Yanchen Wang, Mehdi Azabou, Alexandre Andre, Zixuan Wang, Hanrui Lyu, The Interna- tional Brain Laboratory, Eva Dyer, Liam Paninski, and Cole Hurwitz. Neural encoding and decoding at scale. arXiv preprint arXiv:2504.08201, 2025.
[63] Lili Zheng. Zachary Rewolinski, and Genevera Allen. A low-rank tensor completion approach for imputing functional neuronal data from multiple recordings. In 2022 IEEE Data Science and Learning Workshop (DSLW), pages 1-6. IEEE, 2022.
14

---

A Technical Appendices and Supplementary Material
A.1 Dataset details
This section provides the full anatomical names and corresponding acronyms of the brain areas analyzed throughout the paper.
IBL dataset. hippocampal CAI (CAI), anterior/anterommddal visual area (VISa), ventral posteromedial nucleus (VPM), anterior pretectal nucleus (APN), and midbrain reticular nucleus (MRN).
The posterior thalamus (PO), lateral posterior nucleus (LP), dentate gyrus (DG),
MAP dataset. The anterior lateral motor cortex (ALM), orbital area, lateral part (1Orb), orbital area, ventrolateral part (vIOrb), pallidum combined with globus pallidus, external segment (Pallidum), striatum combined with caudoputamen (Striatum), ventral anterior-lateral complex of the thalamus combined with ventral medial nucleus of the thalamus (VAL-VM), midbrain reticular nucleus (MRN), and superior colliculus, motor related (SC). ALM is identified following methods described in [47].
A.2 Generative process
For the N, neurons in area r. we construct neural data tokens Zx, E N by concatenating neural acivity X, E RRN, X. with area, hemisphere, and unit embeddings: Zarea RD_, Zhemi RDD, uuit E TR.D,, where T T + Da Dr, + Da, We then compute keys (K₂) and values (V,) for the cross-attention module using linear projections WK, Wv E RD xT" A shared set of learnable latent tokens Zo E "XxT" acts as the query Q. The resulting cross-attention output is projected via a multi-layer perceptron to obtain embedding factors Zembed, E PxT
For each trial, a masking proportion is chosen from a uniform distribution from to 0.6. p <- 0.05, then no areas are masked, otherwise, if the trial has recordings from R areas (where held-out areas are considered unrecorded), then Ceiling[p * RJ areas are randomly chosen to be masked. A variable M, is set 1 for masked areas and O for unmasked areas. The embedding factors Zembod,r are tokenized by treating each timestep as a token, resulting in T tokens Zuokon E T' where an MIP projects each token from RR. to H For masked or unrecorded areas, these neural data tokens are replaced by a learned mask token Zmask We define binary indicator U, {0, 1} to denote whether area r is unrecorded. Each token is then concatenated with new learnable area embedding Z'uca RD''' producing the input tokens Zimpii E RR''H' H' H + Da' These token sequences are passed through transformer (with RoPE applied to queries and keys), and the outputs (also E RT'²') are projected via Wilatent E R 18' xD, to obtain latent factors Zistem.r E RTT'D. Finally, firing rate predictions X, are computed via linear readout W IR PR,.N,, followed by exponentiation.
The training procedure is as follows:
Cross-attention stitcher
Remaining modules of NeuroPaint
Zx.
[X,, Zureas Zhemis Zunit]
Zioken
TTokenizer( (ereeiizer(Zebeerr)" rembed, if M, = 0 U, =0
Kr, V,
Wx(ZX,) Wv
5,
otherwise
Q
Zo
input (t) = [[] - 8,)) + 8, (lok (t), Zrreal
Z Zrross-attn.r
Cross-Attin Cross-Attention(( Q, K,, Vo)
Ziatent.r XT = Poisson (exs(onnex(((())) (Zancun. Wrate) ))
(Transformer CecefformerRRoEE.. ))
Zembod,r
MMLP(Zcross-ant)
The model assumes Poisson emission process with time-varying rates. For notational simplicity, (I) describe the generative model using data from one trial in one session. Symbols with the subscript r denote area-specific parameters, while those without are shared across areas. Most parameters shared across sessions, except for Zunit and Wrate
A.3 Loss function details
The consistency and regularization losses formally defined below.
15

---

To ensure the embedding factors maintain stable correlation structure across sessions and penalize deviations from a aassion- target, we introduce the following consistency loss.
Definition A.1 (Consistency loss). Let Lembed, (t) E R.' denote the embedding factors for area E [R] time under trial rype b E [B). Define Kmodel (r, r') RRPx the Pearson correlation matrix between 2 embed, (t) and Z embeed, (t) computed across time after aggregating time steps over trials of type b within a batch. The corresponding target correlation matrix Karger (r, 1r, E RP'P is (t) across time steps in trials of type b, aggregated over sessions. The consistency loss is then defined as follows.
obtained by averaging the correlation between Z Zembed.- (t) and [ r.
B
R
R
Leoosist. Σ r1=1ml
1 cos (vec ((ange((;)))) vvec (Kandee (1,))(rr))))
where cos(:;)) )) denotes the cosine similarity, and vecl-) is vectorization operator that flattens the correlation matrix, using only the upper triangular elements when =
The consistency loss encourages each area's embedding factors to encode information predictive of all areas observed across sessions, in addition to those recorded in the current session, thereby improving generalization to unrecorded areas. In practice, approximate the session-averaged target K( uunget (r, r') using a buffer of past batch correlations, computed with an exponential moving average (EMA) of the cross-attention stitcher to improve stability (Appendix A.4 (88)
Under the assumption that neural dynamics evolve smoothly over time, we define the following regularization loss.
Definition A.2 (Regularization loss), Let Zeatemt (t) E R D. denote the latent factors at time t for area E [R] and let Ziatenay i) denote its i-th component. The regularization loss is defined as follows.
T-1
n Dr
Cug = Σ
Ziatent, (t + 1, Z Iatent, i)
The regularization loss penalizes abrupt changes across consecutive time steps the latent space, thereby improving temporal smoothness of the learned latent factors.
The relative weighting of the loss terms was selected to ensure that the reconstruction loss remains the primary driver during training, while the consistency and regularization terms serve as auxiliary constraints. Specifically:
The reconstruction loss is scaled by a weight of and normalized by the number of timesteps and neurons.
The consistency loss is scaled by weight of and normalized by the number of pairs.
The regularization loss is scaled by weight of 0.1 and normalized by the number of timesteps and latent factors.
A.4 Exponential moving average (EMA) of the cross-attention stitcher
To stabilize the correlation target calculated from the area-specific embedding factors, we maintain an EMA version of the cross-attention stitcher during training. each iteration, the parameters in the EMA version are updated as weighted sum of their previous values and the current weight of the cross-attention stitcher in the main model, controlled by a decay rate a. Specifically, the EMA parameter 0 is updated as follows:
FMA
FMA +(1 a)
To reduce bias in early training when parameter estimates are still unstable, we adapt the decay rate based on the number of iteration steps nsep:
a min(1
nstep + O
(5)
16

---

which increase a as training progresses. Here mmxx = 0.999. This ensures faster adaptation in the early iterations and more stable averaging later on. We use the EMA cross-attention stitcher to compute the area-specific embedding factors and then compute the correlation between embedding factors for each batch, and store them running buffer. These stored correlation matrices are then averaged to generate the target correlation matrix for each area pair and trial type for the current iteration.
A.5 Determining the area-specific latent factor dimension NeuroPaint and LFADS
NeuroPaint. To determine the number of latent factors for each brain area, we estimated the dimensionality of smoothed neural activity across sessions. For each area and session, we first smoothed the spike trains with a Gaussian window (standard deviation = ms), concatenated all trials, and subtracted the mean to center the data. We then computed the participation ratio of the resulting activity matrix, which provides a measure of its linear dimensionality [13]. This yielded one dimensionality estimate per area per session. To ensure sufficient capacity, we set the number of latent factors for each brain area to the maximum participation ratio observed across sessions for that area, plus margin of 10 dimensions. We observed that for most brain areas, the estimated dimensionality saturated as the number of recorded neurons increased, suggesting that the low-di low-dimensional assumption mentioned at the end of the introduction is approximately satisfied by the neural data. This procedure ensures that NeuroPaint is equipped with enough latent factors to model the full range of dynamics expressed in each area.
Area
Dim
PO
61
LP
84
DG
72
CAI
39
VISa
35
VPM
43
APN
58
MRN
24
ALM
29
10rb
31
vlOrb
37
Pallidum
23
Striatum
31
VAL-VM
49
MRN
22
SC
21
Table 1: NeuroPaint latent factor dimensions picked for areas from IBL and MAP datasets.
uses per-session linear read-in and read-out layers to align neural activity from different sessions into a shared latent space. These session-specifie "stitching" layers map observed spike counts to input factors (read-in) and latent factors to firing rates (read-out). The shape of these matrices can vary to match the number of neurons recorded in each session. A shared encoder, generator, and factor matrix are shared across sessions and learned jointly from all sessions. The per-session read-in and read-out matrices are learned using data from only the corresponding session. To promote alignment across sessions, we follow the standard procedure to initialize the weights of the read-in and read-out matrices using principal component regression (PCR)
LFADS.
The multi-session LFADS 33
38) PCR maps the trial aver-gee firing rates from each individual session to the shared principal components across all sessions. This provides the model with an input that already in a shared subspace and is critical for ensuring a shared set of dynamics is learned. We first reshape the trial- averaged firing rates from all sessions into a matrix of size (Nrimepoints X (eoodiiioons x (Nsessions X /neuronss ) and PCA is applied to identify a set of global principal components (PCs). We find the number of PCs needed explain 90% of the variance, which defines the dimensionality of the latent factors used in LFADS, Each session's 's data is then projected onto this global PC space via Ridge regression, and the
17

---

resulting weights and biases from the trained regression model are saved to be used to initialize the read-in weights and biases in LFADS.
A.6 Details of the simulated recurrent neural network in the synthetic dataset
To generate synthetic spike data with multi-area dynamics, we simulated a continuous-time recurrent neural network (RNN). This RNN is composed of areas, each area contains 200 units. Each unit has tanh nonlinearity and with time constant 25 ms. We simulate the network using timestep of At = 10 ms. Unit activity evolves according to the following update rule:
h++1 = (1 - 3)h+ + 3 tanh(Whe))
(6)
Here B At W is the recurrent connectivity between units. h2 denotes unit activities at time step t.
The network receives no external input, its activity is driven entirely by autonomous recurrent dynamics. Each recurrent connection has a weight sampled from a normal distribution Wij ~ (O /N), here Way is the weight between unit i and unit j. N = 1000 is the total number of units in the network, we pick g = 3 so that we will have chaotic circuit dynamics [40]. Connectivity between areas is sparse: each pair of units are connected with 1% probability. Units within the same area have all-to-all connectivity.
For each trial, we randomly initialize the unit activity in the network. Due to chaotic dynamies, each trial will have qualitatively different dynamics. We simulate trials from the same RNN across synthetic sessions.
For each synthetic session, we simulate neuronal spike trains from the unit activity using GLMs. Each session has its set of synthetic neurons, with the number of neurons in each area sampled uniformly from the range [20, 60). The number of trials in each synthetic sessions are sampled uniformly from the range [200, 300].
Each area-specific and session-specific GLM first projects the unit activity onto neuronal instantaneous log firing rates using a sparse linear layer (2% sparsity). The log firing rates for each synthetic neuron are then scaled to a fixed range ([0,2]). Next, we pass the log rates to an exponential nonlinearity and Poisson emission process. For each session, we held-out spike activities generated from 1-2 RNN areas as unrecorded.
In summary, partially overlapped RNN units contribute to the simulated spike trains across synthetic sessions.
A.7 Deviance fraction explained (DFE)
Deviance fraction explained is metric that ranges from 0 to 1, where a larger value indicates better model fit. It is defined as:
Dmodel D
where the deviance terms are given by:
D model log p(x MMat log p(x |M)
D = log (xppp((/////M0 Maa)) - log p(x Maan)
Here, Man represents the saturated model, while Mu represents the null model. Both the saturated and null models assume that the instantaneous spike data is generated from an i.i.d. Poisson distribution. The saturated model (Maa) assumes that the mean of the Poisson distribution is given by the instantancous spike count, while the null model (Mout) assumes that the mean given by the time-averaged and tvial-averaged spike count.
A.8 Evaluation on Synthetic Data with Lower Firing Rates (Matching Real Data Statistics)
To better align our synthetic data with the firing rate statisties observed in the real neural recordings, we generated an additional synthetic dataset, where the log firing rates of each neuron were scaled to a fixed range of [33,3), in contrast to the range [0, 2) used in the previous synthetic dataset shown in Fig. 2 This adjustment results in lower and more realistic firing rates.
18

---

We evaluated the performance of several models for predicting neural activity in the unrecorded brain areas on this synthetic dataset: two baselines (GLM and LFADS) and multiple variants of NeuroPaint (including the linear version described in A.12 and other variants trained with different combinations of loss terms). Overall, we found that NeuroPaint trained with all three loss terms (reconstruction, consistency and regularization) achieved the best performance for this low-rate synthetic dataset (see Fig. $2). unlike for the higher-rate synthetic dataset discussed in the main text (Fig. 2 Empirically, we found that including the consistency loss was particularly important for improving NeuroPaint's s performance, enabling it to outperform the baselines.
It is important to note that for the synthetic dataset, the GLM baseline also performs very well, suggesting that linear model is sufficient to predict neural activity one area based on data from other areas, i.c. to capture inter-areal communication in this simplified setting. In future work, we plan to construct more complex and biologically realistic synthetic datasets that include strong nonlinear inter-areal dependencies, to better evaluate the advantage of nonlinear models.
A.9 GLM baseline performance on the IBL and MAP datasets
We evaluated GLM baselines on both the IBL and MAP datasets. For the IBL dataset, the mean deviance fraction explained (DFE) across neurons in the held-out areas was extremely negative (-3.8 109²), indicating severe overfitting or model mismatch. Notably, 90.7% of neurons had negative DFE values. For the MAP dataset, the mean DFE was -5.1 x 1023_ 63.3% of neurons exhibited negative DFE values.
A.10 Model and hyperparameter details
To avoid overfitting, we add dropout (40%) to all the attention layers and a dropout (20%) the very beginning. Other hyperparameters of the NeuroPaint model is listed in Table 2 Notations for the hyperparameters are introduced in [A.]] For synthetic dataset, we use 24 latent factors for each area. For the IBL. and MAP dataset, set the dimensionality of latent factors according to A.5
NeuroPaint.
Hyperparameter
Value
Da
20
Dp
3
De
50
T
400 (MAP), 200 (IBL)
T
512
P H
48
236
D'
20
H²
256
number of transformer layers
5
Table 2: Hyperparameters used for NeuroPaint.
We use a version of LFADS (Ifads-torch) re-implemented in PyTorch by 38 The hyperparameters of our multi-session LFADS are provided in Table The shared encoder, generator, and factor matrix comprise about 0.3 million parameters. When including session-specific read-in in and read-out matrices, the total number of model parameters increases to 0.4 million, 0.5 million, and 0.6 million for the synthetic, IBI. and MAP datasets. The latent factor dimension of multi-session LFADS chosen via PCR (Appendix A.5). We set the latent factor dimension to 28, 11, and 27 for the synthetic, IBI. and MAP datasets. The L.FDDS model computes KI. divergence between posteriors and priors for both initial condition and inferred input distributions, which are added to the reconstruction cost in the variational ELBO. The priors multivariate normal for the initial conditions and autoregressive multivariate normal for the inferred inputs.
LFADS.
The linear, session-specific, and area-specific stitchers take as input the flattened population activity from 3 neighhoring bins centered around the target time point. Prior to this, the binned spike counts are smoothed using ID Gaussian kernel with standard deviation of 5
19

---

Hyperparameter
Value
Initial Condition Encoder Dimension
100
Controller Input Encoder Dimension
100
Controller Input Lag
1
Controller Dimension
100
Controller Output Dimension
6
Initial Condition Dimension
100
Generator Dimension
100
Controller Output Prior Temporal Decay Constant
Controller Output Prior Innovation Variance
0.1
Initial Condition Prior Mean
0
Initial Condition Prior Variance
0.1
Dropout Rate
0.02
Coordinated Dropout Rate
0.3
Weight Decay
0.0
Learning Rate
0.0003
Batch Size
1024
Table Hyperparameters used for multi-session LFADS.
time bins. The stitchers output area-specific embedding factors, which, along with the latent factors, are set to have the same dimensionality as in the original NeuroPain model. The session-sharee reduced-rank linear transformation applied the concatenated embeddings has a fixed rank of 20. The population activity in the masked areas, which the model is trained to predict, is also smoothed using a Gaussian kernel with the same standard deviation of time bins.
A.11 Training details
NeuroPaint. We trained our model on 2-12 Nvidia A40 or A100 GPU using Adam optimizer for 1000 epochs with learning rate of (1e-3/ 3/56 x global batch size) using OneCycleLR scheduler. We put a weight decay 0.01 to avoid overfitting. We utilized batch size of 16 each GPU during the training. Global batch size is 16x the number of GPU nodes. We split our dataset based on the session to training, validation, and test set with a proportion 60%, 20%, and 20%. saved the model checkpoint based on the validation loss. The 10-session model for the synthetic dataset is trained with 2 GPU nodes in around hours. The 20-session model for the IBI. dataset trained with GPU nodes in around 35 hours. The 40-session model for the MAP dataset is trained with 12 GPU nodes in around 20 hours.
We train the multi-session LFADS using the AdamW optimizer without learning rate scheduler, which is the default setting in the Ifads-torch package. All multi-session LFADS models trained on single Nvidia A40 A100 GPU for 1000 epochs. The best model checkpoint is selected based on the reconstruction performance on the validation set. The 10-session model for the synthetic dataset is trained in under hours. The 20-session LFADS for the IBL dataset takes approximately 10 hours, while the 40-session LFADS for the MAP dataset is trained in about 16 hours.
LFADS.
ChuroPaint-Linear Training details are provided in Appendix A.12 All models are trained on single NVIDIA RTX 2080 Ti GPU and complete within one hour.
A.12 NeuroPaint--Lnear
We introduce NeuroPaint-Linn Linear, linear variant of the original NeuroPaint model. Designed for simplicity and interpretability.
Architecture NeuroPaint-Lan Linear retains the core architectural components of NeuroPaint, compris- ing three main components:
20

---

1. Linear, session-specific, and area-specific stitchers, which map population activity to set of embedding factors.
2. A low-rank, session-shared weight matrix, which transforms the concatenated embedding factors (across all areas) into area-specific latent factors, capturing a mapping between the latent dynamics of different brain areas that remains consistent across sessions and animals.
3. Linear, session-specifie, and area-specific readout layers, which map latent factors to predicted population activity (for all the recorded and unrecorded areas).
While the overall structure closely mirrors that of NeuroPaint, NeuroPaint- introduces three simplifications:
1. The cross-attention readin-stitchin is replaced by simple linear mapping-
2. The transformer encoder is replaced with reduced-rank linear transformation, restricting inter-area mappings to linear operations.
3. In contrast to the nonlinear NeuroPaint, where embeddings for masked or unrecorded areas are replaced with learned mask token after tokenization, set these embedding factors to zero directly, bypassing the tokenizer.
In addition, NeuroPaint- operates in a local and inneeiddependent fashion: latent factors and the corresponding predicted activity are computed independently at each time step, using only the recorded activity from short temporal window (3 time bins, i.e. 1 bin) around that time-without leveraging long temporal context.
Training details Training follows a similar inter-area masking strategy used in NeuroPaint. For each epoch, one recorded area per session is randomly selected and masked from the input. The model is then trained predict the activity of the masked area.
Optimization is performed using the L-BFGS algorithm in full-batch mode, minimizing the mean squared error computed across all time steps and all neurons in the masked areas over sessions. For each training epoch, L-BFGS is reinitialized from scratch and run until convergence. We train models for each of the MAP and IBL datasets for 50 epochs each (we only show results for the MAP dataset), using the default L-GGGS hyperparameters provided by PyTorch. We chose L-BFGS and mean squared error due to their stability during training and their effectiveness.
Comparison with standard reduced-rank regression methods NeuroPaint-Liner differs fun- damentally from standard reduced-rank regression approaches used in encoding models 4 36 and inter-ares areal correlational analyses (e.g., communication subspace) (39,, both in its objectives and methodology. The key innovations lie in the introduction of a session-shared mapping from the full set of area-specific embedding factors to area-specific latent factors, and the use of an inter-area masking strategy during training. Together, these components enable NeuroPaint-Linen to predict the dynamics of unrecorded brain areas based solely on the activity of recorded areas within the same session.
A.13 Impact of loss terms on NeuroPaint performance (MAP dataset)
To evaluate the contribution of each designed loss term on real data, we compared the performance of different NeuroPaint variants on the MAP dataset. In line with the synthetic data results Fig. and Fig. $2 the inclusion of the consistency loss was essential for enabling NeuroPaint outperform the LFADS bascline. In addition, NcuroPaint trained with all three loss terms performs the best. Surprisingly, SeuroPaint-Lincat a purely línear variant of the model, also performed competitively, comparably with the LFADS bascline, highlighting the strength of the overall NeuroPaint framework and its potential to reveal interpretable inter-areal communication among all areas of interest, including both recorded and unrecorded ones.
Interestingly, we found that the variant of NeuroPaint trained with reconstruction and regularization loss (but without consistency loss) exhibited unstable performance across datasets. For example, while it performed reasonably well on the original synthetic dataset (Fig. 2 its performance degraded substantially on the lower firing rate synthetic data (Fig. .2) and on the MAP dataset (Fig. $3] This suggests that, in the absence of consistency constraint, regularization can sometimes hinder model
21

---

performance possibly by wve-tonnsta the latent space or penalizing useful variability. These observations underscore the stabilizing role of the consistency loss guiding the latent representation. especially under more realistic data distributions.
A.14 Interpretable area-specific latent dynamics in the IBL dataset revealed by NeuroPaint
Similar to results shown in Fig. here examine the inferred latent factors for the IBL dataset. Unlike the latent factors for SC in the MAP dataset, the latent factors for MRN in the IBL dataset did not show strong trial-type modulations (Fig. SHAA.. However, in line with results for the MAP dataset, the inferred latent factors in the IBL dataset remain consistent across trials over sessions, regardless of whether given area recorded unrecorded (Fig. $4 B). We evaluated area-to- area representation similarity based on the inferred latent factors, across three trial periods, defined as three consecutive 0.5-second intervals after stimulus onset (Fig. S4C.D). Consistent with anatomical expectations, DG and CA1, both belonging to the hippocampus, exhibited high similarity with each other but not with other areas. In addition, two thalamic nuclei, PO and VPM, also exhibit high similarity across the three periods. Notably, the overall inter-areal similarity structure changed over time: although coherent sub-networks of areas remained present throughout the trial, they became more fragmented in the later periods, exhibiting both reduced similarity and a narrower set of similar areas. This dynamic change likely reflects a shift in task phase from visual processing to cciisin-mmaing and motor execution.
A.15 LFADS with larger architectures does not outperform NeuroPaint
To address the concerns regarding potential model capacity limitations in LFADS, we trained LFADS models on the MAP dataset with larger architectures (256 units in encoder, controller and generator). We tested the number of latent factors of 243 (matching the number of pooled NeuroPaint latent factors across areas) and 49 (matching the largest single-area NeuroPaint latent factors). To maintain training stability and prevent loss divergence, we reduced the learning rate from 3 to 2e here. Training converged successfully under this setting. The DFE for neurons held-out areas, pooled over 40 sessions, is reported in Table 4 We found that NeuroPaint continued to outperform LFADS with larger architectures in terms reconstruction accuracy. Interestingly. the higher-capacity LFADS models performed worse than the smaller model used in our main experiments, potentially due to overfitting given the increased number of parameters. These results suggest that the performance gap is not solely attributable architectural capacity, and that the paradigm introduced in NeuroPaint is the key to its improved performance.
LFADS variants
NeuroPaint
27 factors, 100 units
49 factors,
243 factors,
(used in Fig. 3G.H)
256 units
256 units
(used in Fig. 3.H
DFF
0.08 + 0.006
2.93 1020
-9.12 1020
0.10 + 0.003
DFE (Worst 5% excluded)
0.10 + 0.002
0.08 + 0.002 -9.31 + 1.31
0.12 + 0.002
Table 4: Performance comparison across LEADS models of different capacities and NeuroPaint, evaluated by DFE (meantsst.e. over neurons the held-out areas).
A.16 Calculating correlations between trial pairs of latent factors (MAP dataset)
To assess the consistency and tria--type-deeendent modulation of inferred latent factors, we computed Pearson correlations between trial pairs of area-specific latent factors. All analyses were performed separately for each brain area.
Preprocessing. For each trial, the latent factor matrix was first mean-subtracted across time for each factor remove the static offsets. The resulting matrix was then flattened into vector by concatenating all factors.
Consistency results in the left block of Table 5 between flattened latent-factor vectors for every pair of trials, and grouped trial pairs into three cate- gories: 1) Recorded Recorded: both trials from sessions where the area is recorded 2) Unrecorded
For each area, we computed Pearson correlations
22

---

Table 5: Mean Pearson correlation between latent factors computed from trial pairs pooled across all sessions and averaged over brain areas (mean + standard deviation). See Appendix A.16 for calculation details.
From all sessions
Trial-average subtracted, within a session
Trial pairs
Pearson's 's correlation
Trial pairs
Pearson's 's correlation
Recorded Recorded
0.25 0.05
Hit Left
0.59 0.28
Unrecorded Unrecorded
0.57 + 0.10
Hit Right
0.59 + 0.28
Recorded Unrecorded
0.35 0.06
Across Trial Type
0.53 0.31
Unrecorded: both trials from sessions where the area is unrecorded 3) Recorded & Unrecorded: one trial from session where the area is recorded, one from unrecorded session.
For each area, we computed the averaged correlation within each group. The reported mean was obtained by averaging these per-area values across all areas, and the standard deviation reflects variability across areas.
trial-type-dependent modulation results in the right block of Table 5
To control for the session-
to-session variability, we perform the following analysis on trial pairs within each session, treating each session separately, and again also separately for each area.
For each session, we first computed the trial-averaged flattened latent factor by averaging over all hit trials. We then subtracted this average from each trial's flattened latent factor.
We calculated Pearson correlations between all pairs of trials within session and grouped them into three categories, based on the trial types of the trial pair 1) Hit Left 2) Hit Right 3) Across Trial Type: one hit left and one hit right trial.
For each area and session, we averaged correlations within each group. The reported means and standard deviations were computed by averaging over sessions and areas.
A.17 Decoding choice and stimulus from NeuroPaint latent factors for held-out areas
To evaluate whether the latent factors capture the stimulus and behavioral variability, performed decoding analysis on the MAP dataset. Specifically, we used logistic regression with cross-validation to decode either stimulus (high/low auditory tone) or choice (lick left/lcck right/no lick) from the latent factors for each held-out area, and we compared the balanced accuracy to that decoding from held-out spike data (see Table 6 Stimulus was decoded using summed activity over the last 100 ms of the stimulus period; choice was decoded using summed activity over the last 100 ms of the delay period. Note that due to randomness, no data from VM-VAL was held out the MAP dataset. result, we are unable to report results for this area.
We found that decoding accuracy for inferred latent factors is typically slightly higher than that from held-out spike data, suggesting that the inferred latent factors successfully capture the stimu- lus/behavioral variability in the brain area. This decoding improvement also suggests that the inferred latent factors provide denoised and more complete representation than the held-out population spike data subsampled from the area of interest.
ALM
10rb
vlOrb
Pallidum
Striatum
MRN
SC
0.60
0.58
0.53
0.65
0.53
0.62
0.66
Stimulus
spike data
+0.12
+0.08
=0.08
=0.07
10.04
=0.08
#0.08
(chance level= 0.5)
0.71
0.50
0.61
0.80
0.73
0.68
0.76
latent factors
+0.08
#0.13
=0.13
=0.13
10.06
=0.12
#0.09
0.40
0.41
0.37
0.46
0.58
0.45
0.52
Choice
spike data
10.15
+0.13
+0.12
-0.17
10.13
=0.09
+0.21
(chance level= 0.33)
0.50
0.51
0.57
0.68
0.63
0.63
0.70
latent factors
10.14
10.13
10.14
20.1
10.11
20.16
10.16
Table 6: Decoding accuracy from spike data latent factors for the held-out areas with logistic regression, evaluated by balanced accuracy (seants.t.d. over sessions).
23

---

A.18
Intrinsic dimensionality of predicted firing rates for each brain area from NeuroPaint latent factors
In Table we report the intrinsic dimension of the reconstructed firing rates based on the NeuroPaint latent factors. Specifically, for each recorded area in the MAP dataset, we calculated the participation ratio of the predicted firing rates, which quantifies the effective number of linear dimensions contribut- ing to the variance. Notably, the number of latent factors we used is much larger than the intrinsic dimensionality of predicted firing rates, suggesting that similar performance may be achievable with fewer latent factors. We will explore this in future work.
ALM
1Orb
vlOrb
Pallidum
Striatum
VAL-VM
MRN
Number of latent factors Intrinsic
31
37
31
22
4.71
4.82
4.21
3.51 1133
4.19 1:.53
5.63 11.40
3.90 +:0.88
4.50
dimensionality (mean=ts..d.)
11.45
+1.40
11.43
1.05
Table 7: Intrinsic dimensionaliy of predicted firing rates reported against the number of latent factors from NeuroPaint for the MAP dataset.
IBL dataset
MAP dataset
LFADS vs NeuroPaint neurons the held-out areas
vs NeuroPaint
(excluded worst pooled
pooled 20 sessions
out
Deviarce Fracten Expliined
Expllaned
-0.25
-0.75
Derdance
1.25
0.00
LIADS
Neaiofant
LPADS
Marrhaat
0.6
Neurohain
HeuroPain:
mpli
Beseline (UFADS)
00g H2)
Daseline (LFADS)
05g HQ)
Supplemental Figure 1: Prediction performance for neurons from held-out brain areas across the two datasets. (A) Top: mean and standard error of DFE on the IBL. dataset across all neurons, here LFADS yields mean DFE on order of -el66 16, primarily due to a subset neurons with extremely poor predictions; compare Fig. BC where the 1% of neurons with the poorest predictions were excluded. (B) Top: mean and standard error of DFE on the MAP dataset, excluding the 1% of neurons with the poorest predictions; compare Fig. 3G, which included all neurons. Bottom: per-neuron DFE for all the held-out areas in 40 MAP sessions.
Bottom: per-neuron DFE for all the held-out areas 20 IBL. sessions.
24

---

Unrecorded areas
(excluude the worst 1%)
10 synthetic sessions (low firing rates)
Unrecorded areas
Deviance fracton Exilarnd
0.6
10 synthetic sessions (low firing rates)
Expla
0.6
0.4
0.4-
0.
0.-
0.0
Bacnone
Hearehonn
Ray
Heurotkant comrses
BisunnFall ResercCtE CoREECRRR.. rig
Upoar Smord
0.0
Hecrehaunt
BrisoPaicc mry
Opeat Emund
(01)
eron
(GT)
Supplemental Figure 2: Prediction performance from baselines, variants of Neuro and ground
truth firing rates for neurons from unrecorded areas in the synthetic dataset with lower firing rates. (A) Mean and standard error of DFE on the synthetic dataset with lower firing rates across all neurons. (B) Mean and standard error of DFE on the synthetic dataset with lower firing rates, excluding the
1 % of neurons with the poorest predictions.
Held-ou out MAP 40 sessions
Deviance Fraction Explained
0.100
0.075
0.050
0.025
0.000
Bacatin (URDS)
HenrsPhent
Neurspaint
feccnn
Ricewreg
Hesrohuist returres condis
recon Promiiiis red
NassoPaint
Supplemental Figure 3: Prediction performance from LFADS and variants of NeuroPain for neurons from held-out brain areas MAP dataset. Bar plot shows mean and standard error of DFE on MAP dataset across all neurons. For visuallization purpose, we truncate the y axis below -0.01. For
NeuroPaint trained with reconstruction and regularization loss, its mean of DFE is -3.83, standard error of DFE is 0.
25

---

dhoose left
MANL single
choose right
DRNN Recorded BREN (nveavted MRN tntal
Sacsion
Seedion
Sension
Tvoraovergged actitty subradted
The
Tex
6
trial
trall
Irsi
Sassion
Sesaton
Session
11
Tim
Time
Time
1iee
Time
Time
Reconded area Unaccrded
session choose left ding
Representation Sintarity
Po
DG
CA1
9-55
U5 1.
10- 15
12-
00-
042
03
01
00
Taoo
140
Supplemental Figure 4: NeuroPaint's area-specific latent factors for the IBL. dataset. (A) Inferred latent factors for the Midbrain Reticular Nucleus (MRN), an unrecorded area in session from the IBI. dataset, during example trials across the choose left and choose right conditions. The bottom row shows the same latent factors with the avial-averaged activity subtracted. (B) Inferred single-trial latent factors for MRN (both recorded and unrecorded) across six sessions. (C) Latent factors for all eight areas (both recorded unrecorded)
choose left trial from a single session in the IBL dataset. (D) Representation similarity analysis (RSA) of latent factors across eight brain areas (recorded and unrecorded) for choose-left trials for three periods after stimulus onset in the IBL dataset, with values averaged across trials and pooled over sessions.
26

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer:[
Justification: We clearly declare and introduce main contributions inferring unrecorded brain area dynamics.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Please refer to the discussion section.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model sell-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
27

---

Answer:
[NA]
Justification:
This paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: Implementation details are included in the Appendix and the github repository to be shared.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, descríbing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e-g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
5. Open access to data and code
28

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: We use public datasets for our experiments. Our code attached for repro- ducibility of our results.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips... cc/
Policyllyyyyyy for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
//nips.ccpublle wwwppppgpuuiiieeeep /otpc//uuudeeeooeuuubiiiiii SroodeSubiissi. ssiiiiolcccy for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We specify all the training and hyperparameters details in Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification:
We report error bars whenever necessary in the plots.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi-
dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for
example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
29

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma enror bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We document the computing resources used in Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics httpp////aaaarg...pppopo///ccaaa
Answer: [Yes]
Justification: We carefully read and follow the NeurIPS Code of Ethics. All of our code and main text are anonymous.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: Our work is not specifically linked to any social applications, except for its relevance fundamental neuroscience research.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
30

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper does include any high risk data and all results and models free to share.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
We clearly indicate our cited existing assets by citing the work.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
31

---

If assets are released, the license, copyright information, and terms of use in the thcode has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
latasets.paperssit
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moode as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: Our paper does not involve any human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
32

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The core method development in this research does involve LLMs as any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM for what should or should not be described.
33


Reviews:
[{"review_id": "Es1he0c2Gb", "reviewer": "Reviewer_EKaW", "summary": "Monitoring and understanding neural population activity across brain areas is critical for the understanding of neural computation. However, in practice recordings fail to monitor all brain regions simultaneously. To overcome this limitation, this work introduces a new masking autoencoder approach to learn representation across animals and brain areas to predict neural activity in no recorded areas. The masking training strategy in combination with a reconstruction and consistency loss function allows for the generalization properties of the model. The model was tested in synthetic and two neural datasets outperforming alternative existing methods.", "questions": "Understanding data need would further illustrate the applicability of the model for experimental use. For example, what are the data demands to retrain the model from scratch? And using some transfer learning strategy? Would it be possible to train a foundation model of sorts that generalizes across regions, animals and/or species?\n\nDoes the model generalize to other recording methods like calcium recordings? \n\nRegarding information content. How many latent dimensions were used for each brain region? Is that enough to capture intrinsic, stimulus and behavioral variability? Does one have to manually adjust this hyperparameter if not adequate?", "limitations": "The authors mention some limitations, but clearly stating data demands is critical to also understand the impact and applicability of the method. Moreover, understanding the information content of the latent and reconstructed signals should be fully characterized to understand the limitations.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "The model addresses a critical need for neuroscience discovery and was clearly formulated. The training paradigm also shows innovation needed to address this problem. The model was evaluated in simulation and two neural recordings showing the promise of the approach.\n\nWhile the model was evaluated in terms or reconstruction, it would be important to understand the relevance of the method to also measure stimuli or behavioral decodability to understand the information content present in the reconstructed neural activity. It would also be important to test, possibly in simulation, the model requirements and limitations, such as data demands, trial repeats, number of latent dimensions or number of recorded cells per area.", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for recognizing the innovation and importance of our approach. We respond to the questions and concerns raised as follows. \n\n>Understanding data need would further illustrate the applicability of the model for experimental use. For example, what are the data demands to retrain the model from scratch? \n\nWe agree that understanding data requirements is important for assessing the model’s experimental utility. In this work, we showed that the model performs well with 40 sessions for the MAP dataset and 20 sessions for the IBL dataset, but we have not systematically explored the minimal number of sessions or neurons required for effective training. However, we believe that the amount of data needed to train the model from scratch depends on several factors specific to each dataset, including the complexity of the behavioral task, the selection of brain areas of interest, and the animal species. For example, if the behavioral task is fairly complicated with the animal demonstrating highly diverse behaviors across trials, then the data demand would be larger compared to the case where the animals exhibit stereotypical behaviors. As a result, we believe this question is best addressed empirically in each use case, rather than by reporting a fixed threshold. \n\n>And using some transfer learning strategy? Would it be possible to train a foundation model of sorts that generalizes across regions, animals and/or species?\n\nYes, we believe it is possible to train a foundation model using our proposed architecture. One promising direction would be to pretrain the model on a diverse set of datasets that cover similar brain areas and behavioral task conditions. This pretrained model could then be fine-tuned on new datasets with limited data, enabling generalization across animals, regions, or even species (e.g. generalize across non-human primates and humans or across mice and rats). While we have not yet explored this direction, our framework is well-suited for such an extension and we see it as an exciting avenue for future work.\n\n>Does the model generalize to other recording methods like calcium recordings?\n\nYes, our method can be extended to calcium imaging data with minimal architectural changes. Specifically, the GLM-based stitcher can be replaced with a linear stitcher, and the reconstruction loss can be modified from a Poisson negative log-likelihood to mean squared error (MSE), which is more appropriate for continuous-valued calcium signals.\nThat said, the core motivation for our method—integrating data across sessions with partially-overlapping recorded areas—is less common in calcium imaging datasets. In typical calcium imaging experiments, especially wide-field imaging, the same set of brain areas are usually recorded across sessions. \n\nNonetheless, an interesting future direction would be to apply our approach to stitch together high temporal resolution, small field-of-view recordings with overlapping brain areas to infer wide-field activity with improved temporal resolution. This could help mitigate the limitations of wide-field calcium imaging, which often suffers from low temporal resolution due to technical constraints. \n\n>Regarding information content. How many latent dimensions were used for each brain region? Is that enough to capture intrinsic, stimulus and behavioral variability? Does one have to manually adjust this hyperparameter if not adequate?\n\nThe number of latent dimensions used in NeuroPaint for each dataset is detailed in Appendix section A.5. Rather than setting this hyperparameter manually, we used a principled procedure based on the participation ratio of the spike data across sessions to determine the number of latent factors (see A.5). This procedure is designed to ensure that the latent space is large enough to capture the variability in the data. \n\nAs shown by the strong reconstruction performance evaluated by deviance fraction explained (DFE; defined in A.7) in Figure 3C,G, the inferred latent dimensions are sufficient to capture intrinsic variability. To evaluate whether the latent factors capture the stimulus and behavioral variability, we performed decoding analysis on the MAP dataset. Specifically, we used logistic regression with cross-validation to decode either stimulus (high/low auditory tone) or choice (lick left/lick right/no lick) from the latent factors for each held-out area (**bottom line** in each cell of the table, mean+/-std over sessions), and we compared the **balanced accuracy** to that decoding from held-out spike data (**top line** in each cell of the table, mean+/-std over sessions). Stimulus was decoded using summed activity over the last 100 ms of the stimulus period; choice was decoded using summed activity over the last 100 ms of the delay period. \n\n|      | ALM        | lOrb       | vlOrb      | Pallidum  | Striatum  | MRN       | SC        |\n|------------------------|------------|------------|------------|-----------|-----------|-----------|-----------|\n| Stimulus (chance level =0.5)  | 0.60±0.12  | 0.58±0.08  | 0.53±0.08  | 0.65±0.07 | 0.53±0.04 | 0.62±0.08 | 0.66±0.08 |\n|                        | 0.71±0.08  | 0.50±0.13  | 0.61±0.13  | 0.80±0.13 | 0.73±0.06 | 0.68±0.12 | 0.76±0.09 |\n| Choice (chance level =0.33)   | 0.40±0.15  | 0.41±0.13  | 0.37±0.12  | 0.46±0.17 | 0.58±0.13 | 0.45±0.09 | 0.52±0.21 |\n|                        | 0.50±0.14  | 0.51±0.13  | 0.57±0.14  | 0.68±0.12 | 0.63±0.11 | 0.63±0.16 | 0.70±0.16 |\n\nNote that due to randomness, no data from VM-VAL was held out in the MAP dataset. As a result, we are unable to report results for this area.\n\nWe found that decoding accuracy for inferred latent factors is typically slightly higher than that from held-out spike data, suggesting that the inferred latent factors successfully capture the stimulus/behavioral variability in the brain area. This decoding improvement also suggests that the inferred latent factors provide a denoised and more complete representation than the held-out population spike data subsampled from the area of interest."}, {"review_id": "pQVv3wY2eN", "reviewer": "Reviewer_Jqe4", "summary": "In this work, the authors introduce NeuroPaint, a approach for inferring the latent dynamics of unobserved brain areas. Specifically, the model is a transformer-based that is use brain-area masking to allow the model to predict the dynamics of the masked brain area given the  other observed brain area.", "questions": "All my concerns are listed above.", "limitations": "yes", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "# Strengths\n\nI love the motivation behind the approach and think this is an interesting and important direction for computation neuroscience.\n\n# Weaknesses\n\nI have a number of qualms with the paper. To start, I don't understand the claim that multi-area state space models can't infer the dynamics of unknown brain areas. This is a very straight forward extension of most (if not all) state-space models in both training and inference. Training a state-space model---for simplicity, I will focus on deep SSMs trained via the ELBO---would be exactly the same as the proposed procedure: the input to the encoder would be the observed brain areas but the decoder would still try and predict all of the brain areas. Inference would be straight forward as well: I would pass the observed brain areas into my encoder and my decoder would automatically predict all of the brain areas.\n\nNext, there are a number of design choices that don't make sense to me. Firstly, the paper relies on the observation high-dimensional neural recordings can be explained by low-dimensional dynamics. Now, low and high dimensional are relative but in general the low dimensions should be significantly lower than the high dimensional neural recordings. But the authors don't seem to follow this. For the synthetic data, the authors state in lines 183-184 that each area contains 20-60 neurons per session but they choose the dimensionality of the latent factors for each dimension to be 24 (line 680 in the appendix). For the IBL dataset, most area and sessions have less than 100-150 neurons yet the dimensionality of the latent factors are quite large (84 for LP and 72 for DG, table 1 in the appendix); the dimensionality of the latent factors (relative to the observed number of neurons) is a little too large which seems to go against the author's hypothesis.\n\nLastly, I don't understand why after training NeuroPaint (which already has a linear Poisson decoder) the authors train another one! The original NeuroPaint Poisson decoder is a part of the model and should be used for decoding. The fact that a separate decoder needs to be trained afterwards makes me questions the usefulness of the approach.", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for pointing out several confusing parts of the paper, and we are willing to address these concerns as follows.\n\n>To start, I don't understand the claim that multi-area state space models can't infer the dynamics of unknown brain areas. This is a very straight forward extension of most (if not all) state-space models in both training and inference. Training a state-space model---for simplicity, I will focus on deep SSMs trained via the ELBO---would be exactly the same as the proposed procedure: the input to the encoder would be the observed brain areas but the decoder would still try and predict all of the brain areas. Inference would be straight forward as well: I would pass the observed brain areas into my encoder and my decoder would automatically predict all of the brain areas.\n\nWe agree with the reviewer’s comment that, in principle, other architectures such as state-space models can be extended to infer neural activity in unobserved brain areas. However, we would like to clarify that our contribution is to explicitly design and evaluate a paradigm tailored for this setting. Our paradigm includes several key innovations: 1) we introduce area-specific latent factors for both recorded and unrecorded areas; 2) we employ a mask training strategy where both unrecorded and masked areas are represented by mask tokens; 3) we incorporate consistency loss to encourage our model to generalize to unrecorded areas. \n\nWhile it is certainly possible to implement a similar model using other backbones, such as deep SSM, we chose transformer layers due to their maturity and wide support, which made implementation and integration much more straightforward. As a proof of concept, we implemented a linear version of the NeuroPaint in Appendix A.12, where we replace the transformer layers with linear layers. And we showed that this variant of NeuroPaint can perform comparably with LFADS, indicating that the paradigm we designed is broadly applicable across architectural backbones.  \n\nExtending NeuroPaint with a deep SSMs backbone would be an interesting future direction, as deep SSMs have shown potential in capturing long-range dependencies and enabling online inference. \n\n>Next, there are a number of design choices that don't make sense to me. Firstly, the paper relies on the observation high-dimensional neural recordings can be explained by low-dimensional dynamics. Now, low and high dimensional are relative but in general the low dimensions should be significantly lower than the high dimensional neural recordings. But the authors don't seem to follow this. For the synthetic data, the authors state in lines 183-184 that each area contains 20-60 neurons per session but they choose the dimensionality of the latent factors for each dimension to be 24 (line 680 in the appendix). For the IBL dataset, most area and sessions have less than 100-150 neurons yet the dimensionality of the latent factors are quite large (84 for LP and 72 for DG, table 1 in the appendix); the dimensionality of the latent factors (relative to the observed number of neurons) is a little too large which seems to go against the author's hypothesis.\n\nWe appreciate the reviewer’s concern and the opportunity to clarify our design choice regarding the number of latent dimensions. While the number of latent factors may appear high relative to the number of recorded neurons in a given session, it is low compared to the total number of neurons in a brain area, which is often on the order of 100,000 in the mouse brain. The latent factors shared across sessions are intended to capture neural dynamics common to the entire area, rather than being tailored to each subset of recorded neurons in each session. \n\nWhile the ambient/embedding dimensionality of the latent factors may be moderately large (sometimes over 50 as the reviewer pointed out), their intrinsic dimensionality is often much lower. This is expected, as we do not impose orthogonality constraints on the latent factors. Empirically, we observe strong correlations among inferred latent dimensions (see Fig. 4C), suggesting low intrinsic dimensionality (see table below for ambient/embedding dimension versus intrinsic dimension for each brain area of interest in the MAP dataset). This observation is consistent with the core hypothesis that each brain area exhibits low-dimensional dynamics. \n\nWe agree with the reviewer that the number of latent factors we chose may be higher than necessary. In future work, we plan to investigate how reducing the number of latent factors affects performance, with the goal of identifying the minimal dimensionality required to achieve optimal results.\n\nIn the table below, we report the intrinsic dimension of the reconstructed neural activity by the inferred latent factors for each recorded area in the MAP dataset. Here we evaluated the intrinsic dimension by calculating the **participation ratio** of the predicted firing rates based on the latent factors for each area separately, where the participation ratio reflects the effective number of linear dimensions contributing to the variance.    \n\n|                | ALM         | lOrb        | vlOrb       | Pallidum    | Striatum    | VAL-VM      | MRN         | SC          |\n|----------------|-------------|-------------|-------------|-------------|-------------|-------------|-------------|-------------|\n| **Ambient/embedding dimension** (number of latent factors) | 29          | 31          | 37          | 23          | 31          | 49          | 22          | 21          |\n| **Intrinsic dimension** (participation ratio, i.e. number of effective dimension; mean +/- std over sessions) | 4.71 ± 1.45 | 4.82 ± 1.40 | 4.21 ± 1.43 | 3.51 ± 1.36 | 4.19 ± 1.53 | 5.63 ± 1.40 | 3.90 ± 0.98 | 4.50 ± 1.05 |\n\n>Lastly, I don't understand why after training NeuroPaint (which already has a linear Poisson decoder) the authors train another one! The original NeuroPaint Poisson decoder is a part of the model and should be used for decoding. The fact that a separate decoder needs to be trained afterwards makes me questions the usefulness of the approach.\n\nWe appreciate the reviewer’s question and the chance to clarify this point. In the trained NeuroPaint model, we learn a separate linear Poisson decoder for each recorded brain area in each session. However, for unrecorded areas, by definition, we do not have spike data and therefore we can not train decoders for them. To approximate performance on unrecorded areas, we hold out certain areas during training, train NeuroPaint on the remaining data, and then test whether the inferred latents for the held-out areas can support accurate activity reconstruction using a separately trained linear Poisson decoder. We will clarify this point in the revised manuscript.\n\nImportantly, our goal is not to infer neural activity in the unrecorded areas at single-neuron level, but to infer the neural dynamics at the level of latent factors. These area-specific inferred latent factors offer opportunities to address key neuroscience questions, such as how neural dynamics differ across areas and how brain areas interact during behavior."}, {"review_id": "NFLEFjTJ5Y", "reviewer": "Reviewer_fqAe", "summary": "Most neural recordings sample only a few brain areas. The authors propose a model named NeuroPaint to address this by treating each unrecorded area as a masked token and learning a low-dimensional latent trajectory for every area through a shared transformer. More specifically, a cross-attention read-in embeds spikes, the transformer captures cross-area interactions, and a GLM-style read-out plus smoothness and consistency losses rebuild the spiking activity. On synthetic RNN data and large Neuropixels mouse datasets, the model predicts spikes in held-out areas far better than GLM and LFADS baselines.", "questions": "See above.", "limitations": "The authors discuss the limitations in Section 6, and I agree with their statements.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**:\n\n1. The model can predict spikes for brain areas that were not recorded by treating those areas as masked tokens, and it clearly outperforms both GLM and multi-session LFADS on synthetic data and on the IBL and MAP Neuropixels datasets.\n\n2. A clear architecture of this model: cross-attention read-in, shared transformer, GLM read-out, with most weights shared across sessions helps the model generalize across animals and sessions.\n\n**Weaknesses**:\n\n1. Section 5.3 shows that the latent factors are smooth across trials and that their representational-similarity structure shifts between stimulus, delay, and response periods. These findings are still descriptive: correlations only tell us the factors repeat, and RSA only tells us areas become more similar during the response, without showing which task variables the latents encode. A straightforward suggestion is to add a decoding experiment. For each area, train a simple linear classifier to predict trial variables from that area’s latents, and report its accuracy alongside classifiers trained on LFADS or GLM latents of the same dimensionality. If the proposed latents truly capture context-dependent variability, they should outperform these baselines. The authors are free to design any quantitative decoding analysis they see fit, so long as it clarifies what information the latents encode and enables a direct comparison with competing models.", "was_revised": true, "final_justification": "This paper addresses a fascinating and fundamentally important problem in neuroscience: inferring the dynamics of unobserved brain areas. Several reviewers, including myself, requested additional results to support the proposed method. In their rebuttal, the authors successfully addressed most of our concerns by providing the requested additional analyses.", "rebuttal": "We thank the reviewer for the overall positive comments and we would like to address the weakness as follows.\n\n>Section 5.3 shows that the latent factors are smooth across trials and that their representational-similarity structure shifts between stimulus, delay, and response periods. These findings are still descriptive: correlations only tell us the factors repeat, and RSA only tells us areas become more similar during the response, without showing which task variables the latents encode. A straightforward suggestion is to add a decoding experiment. For each area, train a simple linear classifier to predict trial variables from that area’s latents, and report its accuracy alongside classifiers trained on LFADS or GLM latents of the same dimensionality. If the proposed latents truly capture context-dependent variability, they should outperform these baselines. The authors are free to design any quantitative decoding analysis they see fit, so long as it clarifies what information the latents encode and enables a direct comparison with competing models.\n\nAs also noted by Reviewer EKaW, we performed decoding analyses to assess what task variables are captured by the inferred latent factors. Specifically, we trained linear classifiers (logistic regression) to decode either stimulus (high vs. low auditory tone) or choice (lick left/right/no lick) from the latent factors of each held-out brain area. We found that the latent factors in most areas reliably support decoding for both stimulus and behavioral choice, demonstrating that they encode meaningful, task-relevant information. \n\nStimulus was decoded using summed activity over the last 100 ms of the stimulus period, choice was decoded using summed activity over the last 100 ms of the delay period. Here we report the balanced accuracy (mean +/- std over sessions) as the decoding performance metrics, separately for each held-out area in the MAP dataset. The **bottom line** in each cell shows the balanced accuracy for decoding from inferred latent factors. As a comparison, we show the balanced accuracy for decoding from the held-out spike data in the **top line**. We found that decoding accuracy for inferred latent factors is typically slightly higher than that from held-out spike data, suggesting that the inferred latent factors successfully capture the stimulus/behavioral variability in the brain area. This decoding improvement also suggests that the inferred latent factors provide a denoised and more complete representation than the held-out population spike data subsampled from the area of interest.\n\n|      | ALM        | lOrb       | vlOrb      | Pallidum  | Striatum  | MRN       | SC        |\n|------------------------|------------|------------|------------|-----------|-----------|-----------|-----------|\n| Stimulus (chance level =0.5)  | 0.60±0.12  | 0.58±0.08  | 0.53±0.08  | 0.65±0.07 | 0.53±0.04 | 0.62±0.08 | 0.66±0.08 |\n|                        | 0.71±0.08  | 0.50±0.13  | 0.61±0.13  | 0.80±0.13 | 0.73±0.06 | 0.68±0.12 | 0.76±0.09 |\n| Choice (chance level =0.33)   | 0.40±0.15  | 0.41±0.13  | 0.37±0.12  | 0.46±0.17 | 0.58±0.13 | 0.45±0.09 | 0.52±0.21 |\n|                        | 0.50±0.14  | 0.51±0.13  | 0.57±0.14  | 0.68±0.12 | 0.63±0.11 | 0.63±0.16 | 0.70±0.16 |\n\nWe would like to clarify that a direct comparison with LFADS and GLM latents is not feasible. LFADS does not produce area-specific latent factors, so decoding from LFADS latents cannot be fairly compared to decoding from NeuroPaint’s area-specific factors, especially considering that the decoding performance varies a lot across brain areas. The GLM baseline in our study is designed to predict spikes in a held-out area based on spikes in recorded areas; it does not involve latent factors, and therefore does not support a comparable decoding analysis."}, {"review_id": "h44Tid8WmV", "reviewer": "Reviewer_Xqnt", "summary": "This paper proposes a multi-session, multi-area masked Transformer model of large-scale neural activity with the stated goal of inferring unobserved neural activity from unrecorded brain areas. In general, through an architecture with area-specific components and training across sessions with variable recorded brain areas, the model infers a \"brain wide\" set of dynamics for all specified brain areas on each trial. With the addition of two losses that encourage capturing cross-area correlations and temporally smooth factors, the model outperforms a multi-session stitched LFADS model in heldout-neuron prediction. This evaluated was done on a simulated dataset as well as two larger-scale multi-area, multi-session neuropixel recordings. The authors show how the resulting model with area-specific factors enabled post-hoc analysis of neural representations.", "questions": "* Can the authors provide ablation results for NeuroPaint for the different losses on the IBL and MAP datasets? I understand these experiments are too expensive to run in this time window, but I'm wondering if they have been done and not reported. I think it will be important to report these results for a final version of this manuscript to better understand what components were important for achieving the best predictions. For example, does the model without the consistency and regularization losses also outperform LFADS? \n\n* What is the relative weighting of the different loss terms? How was this chosen, and are the results sensitive to the relative weighting. \n\n* Can the authors explore a higher-dimensional LFADS model with a larger encoder, controller, and generator, alongside higher dimensional latent factors that match the dimensions of NeuroPaint? My understanding is that the LFADS latent factors were 28, 11, or 27 which is significantly lower than NeuroPaints latent factor dimensions. Next, in related work I have seen LFADS models with 256 for modeling a single neuropixels session and a 512 dimensional generator for modeling stitched neuropixel sessions. I would encourage at least bumping up to 256 dimensions for the generator, controller, and encoder for the IBL and MAP datasets. The potentially limited capacity of LFADS here could explain why it does not adequately explain some neurons while providing strong predictions for others. \n\nOverall, my evaluation score could increase with an improved reporting or discussion of how the model performs with and without the additional loss terms on the real datasets and/or by seeing that NeuroPaint also outperforms a larger LFADS model with ~256 dimensional RNNs and latent factors with larger dimensionality.", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 4, "strengths_and_weaknesses": "Strengths\n* The proposed NeuroPaint architecture appears to be a very compelling approach to encode multi-area structure into the model, which addresses an important problem in the field. In particular, the ability to reason about putative latent factors across all brain areas on individual trials (shown in Fig. 4) is very nice. \n\n* The evaluation across synthetic, IBL, and MAP datasets is strong and compelling. It is great to see the validation of the approach on the synthetic dataset in addition to testing on two difficult, large-scale neural datasets. In addition, the authors did a great job of clearly describing the model and providing many relevant details. \n\n* The proposed approach provided improvement prediction on a difficult problem in two real neural datasets, which is an important contribution.\n\nLimitations\n* The consistency loss appears to have been an important part of the approach, enabling it to improve on LFADS in the synthetic dataset and potentially as well for the real datasets. However, this approach seems difficult to scale and compute, as it is computed for each pair of brain areas. It additionally makes a strong assumption that the correlation structure between each pair of areas should be fixed (within each trial type). The authors appear to allude to this limitation when discussing innovations needed to train across hundreds of brain areas. \n\n* The language about what is being predicted about held-out areas could be improved. Initially, it sounds like spiking activity of heldout areas is being predicted unseen on new sessions; however importantly all models need to be aligned to a new heldout area on a new session with the GLM cross-validation. Instead, NeuroPaint provides a delineated set of latent factors for this new heldout area in a manner that the stitched LFADS architecture that mixes all regions together does not. For example, the final sentence on page 6 (\"It is important...\") should be altered, as it appears all models rely on post-hoc prediction. Instead, an important difference is NeuroPaint has separated the latent factors by region while LFADS has not. \n\n* The proposed model requires a lot more compute for benefits that have not yet scaled with compute cost. For example, for the 40-session MAP models, LFADS requires 16 GPU hours (1 for 16 hours) whereas NeuroPaint requires 240 GPU hours (12 for 20 hours), so 15x compute cost compared to a DFE approvement of 1.25x for NeuroPaint.  \n\n* I understand it is time-consuming to tune LFADS. I appreciate the thorough comparison and reporting that the authors have already done. However, I expect the LFADS model is somewhat limited in capacity, and a larger model may close the gap (see questions).", "was_revised": true, "final_justification": "The authors' have addressed my questions and concerns, and I have raised my score by a point to reflect that. The paper will be improved with a greater emphasis on the importance of the consistency loss along with discussion of its potential limitations.", "rebuttal": "Thank you very much for recognizing the originality and strengths of our work. We address each point raised as follows. \n\n>The consistency loss appears to have been an important part of the approach, enabling it to improve on LFADS in the synthetic dataset and potentially as well for the real datasets. However, this approach seems difficult to scale and compute, as it is computed for each pair of brain areas. It additionally makes a strong assumption that the correlation structure between each pair of areas should be fixed (within each trial type). The authors appear to allude to this limitation when discussing innovations needed to train across hundreds of brain areas.\n\nWe appreciate the reviewer’s insightful comments about the consistency loss. We would like to clarify a few important aspects regarding scalability and the assumptions involved. \n\nFirst, while the consistency loss has a computational cost that scales quadratically with the number of areas, we do not view this as a major limitation in practice. The total number of annotated areas in the mouse brain is on the order of hundreds. Importantly, even as the dataset grows larger in the future, the number of behaviorally relevant areas per experiment remains relatively limited. In contrast, the number of sessions in large-scale datasets can continue to grow – potentially reaching thousands in the future – so a method that scaled quadratically with session count would be far more problematic. Our design deliberately avoids such scaling, making it more suitable for future large-scale applications. We plan to explore alternative loss and model designs to reduce the computational cost, which we see as an important next step. \n\nSecond, the consistency loss is applied not to the raw spike data or area-specific latent factors, but to the embedding factors (see Fig. 1C for definition), which are related to the neural activity through highly nonlinear transformations. In other words, the assumption about stable correlations between areas is imposed only at the level of these embeddings, making it less restrictive than it may appear. We will clarify this point in the revised manuscript. \n\nWe agree that in more complex settings – such as neural recordings during trial-unaligned, naturalistic behaviors – the design of consistency loss might not be applicable. However, for trial-aligned data considered in our work, we believe the consistency constraint is reasonable and effective. \n\n>The language about what is being predicted about held-out areas could be improved ... For example, the final sentence on page 6 (\"It is important...\") should be altered, as it appears all models rely on post-hoc prediction. Instead, an important difference is NeuroPaint has separated the latent factors by region while LFADS has not.\n\nWe agree with the reviewer that the language needs to be improved. We will revise the sentences to clarify that the important difference is that NeuroPaint has separate latent factors for each area, including the unrecorded ones. \n\n>The proposed model requires a lot more compute for benefits that have not yet scaled with compute cost. For example, for the 40-session MAP models, LFADS requires 16 GPU hours (1 for 16 hours) whereas NeuroPaint requires 240 GPU hours (12 for 20 hours), so 15x compute cost compared to a DFE improvement of 1.25x for NeuroPaint.\n\nWe would like to emphasize that NeuroPaint’s contribution goes beyond improved reconstruction performance. A key advantage of NeuroPaint is its use of area-specific latent factors, which enables novel analysis of inter-area interactions and allow us to characterize how single-trial neural dynamics vary across brain areas (even for the unrecorded ones) during behavior. These capabilities are not readily supported by LFADS or similar methods. \n\nFurthermore, while the performance gain in DFE may appear modest, we note that LFADS is already a strong baseline, and improving upon it—especially across many sessions—is inherently challenging. In such cases, even incremental gains can reflect meaningful advances.\n \nTo address the concerns about computational efficiency, in Appendix A.12, we also implemented a linear version of NeuroPaint. This variant achieves performance comparable to LFADS with much lower computational cost  (see Supplemental Figure 2, 3 for performances, and A.11 for its computational cost), offering a more lightweight alternative for scenarios where computational resources are limited. We agree that reducing the computational cost without sacrificing performance is an important direction for future work. \n\n>I understand it is time-consuming to tune LFADS. I appreciate the thorough comparison and reporting that the authors have already done. However, I expect the LFADS model is somewhat limited in capacity, and a larger model may close the gap (see questions)\n\n>Can the authors explore a higher-dimensional LFADS model with a larger encoder, controller, and generator, alongside higher dimensional latent factors that match the dimensions of NeuroPaint? ... I would encourage at least bumping up to 256 dimensions for the generator, controller, and encoder for the IBL and MAP datasets. The potentially limited capacity of LFADS here could explain why it does not adequately explain some neurons while providing strong predictions for others.\n\nWe appreciate the reviewer’s thoughtful suggestion and agree that it is important to assess whether the performance gap is due to model capacity differences. To this end, we conducted additional experiments where we trained LFADS models on the MAP dataset using larger architectures: we increased the number of units in the encoder, controller, and generator to 256, and explored two latent dimensionalities—243 (matching the total number of NeuroPaint latent factors pooled across areas) and 49 (matching the maximum number of factors across areas in NeuroPaint). Additionally, to prevent the loss of LFADS from diverging to NaN, we reduced the learning rate from 3e-4 to 2e-4. With this adjustment, we confirmed that training remains stable and converged successfully under this lower learning rate. \nHere we report the deviance fraction explained (DFE, defined in A.7) metrics for neurons in the held-out areas pooled over 40 sessions. \n\n| Metric                                                                                   | LFADS (27 factors, used in Fig. 3G,H) | LFADS (256 units; 49 factors) | LFADS (256 units; 243 factors) | **NeuroPaint**      |\n|------------------------------------------------------------------------------------------|---------------------------------------|-------------------------------|-------------------------------|----------------------|\n| DFE (mean ± ste)                                                                         | 0.08 ± 0.006                          | -2.93e20                      | -9.12e20                      | **0.10 ± 0.003**     |\n| DFE (the 5% neurons with the poorest predictions were excluded; mean ± ste)                                              | 0.10 ± 0.002                          | 0.08 ± 0.002                  | -9.31 ± 1.31                  | **0.12 ± 0.002**     |\n\nIn both cases, we found that NeuroPaint continued to outperform LFADS in terms of reconstruction accuracy. Interestingly, the higher-capacity LFADS models performed worse than the smaller model used in our main experiments, potentially due to overfitting given the increased number of parameters. These results suggest that the performance gap is not solely attributable to architectural capacity, and that the paradigm introduced in NeuroPaint is the key to its improved performance. \n\n>Can the authors provide ablation results for NeuroPaint for the different losses on the IBL and MAP datasets? ... does the model without the consistency and regularization losses also outperform LFADS?\n\nWe thank the reviewer for highlighting the importance of understanding the contribution of different loss terms of our model. We have the ablation results for the MAP dataset in Appendix A.14 and Supplemental Figure 3, as well as for an additional synthetic dataset in Supplemental Figure 2. These analyses isolate the effects of the consistency and regularization losses. Our results show that both terms contribute to improved performance, with the consistency loss being particularly important—it is essential for NeuroPaint to outperform LFADS.\n\n>What is the relative weighting of the different loss terms? How was this chosen, and are the results sensitive to the relative weighting.\n\nWe thank the reviewer for pointing out this missing detail. The relative weighting of the loss terms was selected to ensure that the reconstruction loss remains the primary driver during training, while the consistency and regularization terms serve as auxiliary constraints. Specifically:\n\n- The **reconstruction loss** is scaled by a weight of 1 and normalized by the number of timesteps and neurons.\n- The **consistency loss** is scaled by a weight of 1 and normalized by the number of area pairs.\n- The **regularization loss** is scaled by a weight of 0.1 and normalized by the number of timesteps, and latent factors.\n\nThese scalings were chosen so that both the consistency and regularization losses remain at least an order of magnitude smaller than the reconstruction loss throughout training. We will clarify this point in the revised manuscript.\n\nWe have not yet conducted a systematic sensitivity analysis on these weightings, and we agree this is an important direction for future work—especially to further improve robustness and generalization across diverse datasets."}]

Author rebuttal:

Meta-review: This paper presents a deep learning-based technique for inferring the activity of missing brain areas in recorded neural datasets by leveraging multi-animal datasets with overlapping recordings across brain regions. This addresses an important practical challenge in neuroscience where complete brain-wide recordings are often not feasible from single animals.

The weaknesses initially identified by most reviewers have been adequately addressed, as reflected in their updated final assessments. One reviewer did not engage further in the discussion process, preventing me from confirming whether their specific concerns were resolved from their perspective. However, my own review of the authors' rebuttals indicates that the major concerns raised were at least partially clarified and addressed.

Given the importance of the problem this work tackles and the positive evaluations from the majority of reviewers, I recommend acceptance for this paper.
Decision: Accept (poster)