POCO: Scalable Neural Forecasting through Population Conditioning
Authors: Yu Duan
Hamza Tahir Chaudhry
Misha B. Ahrens
Christopher D Harvey
Matthew G Perich
Karl Deisseroth
Kanaka Rajan
Venue: NeurIPS 2025 poster ()
Keywords: Neural Dynamics
Neural Foundation Models
Time-Series Forecasting
Primary area: neuroscience_and_cognitive_science

Abstract: Predicting future neural activity is a core challenge in modeling brain dynamics, with applications ranging from scientific investigation to closed-loop neurotechnology. While recent models of population activity emphasize interpretability and behavioral decoding, neural forecasting—particularly across multi-session, spontaneous recordings—remains underexplored. We introduce POCO, a unified forecasting model that combines a lightweight univariate forecaster with a population-level encoder to capture both neuron-specific and brain-wide dynamics. Trained across five calcium imaging datasets spanning zebrafish, mice, and *C. elegans*, POCO achieves state-of-the-art accuracy at cellular resolution in spontaneous behaviors. After pre-training, POCO rapidly adapts to new recordings with minimal fine-tuning. Notably, POCO's learned unit embeddings recover biologically meaningful structure—such as brain region clustering—without any anatomical labels. Our comprehensive analysis reveals several key factors influencing performance, including context length, session diversity, and preprocessing. Together, these results position POCO as a scalable and adaptable approach for cross-session neural forecasting and offer actionable insights for future model design. By enabling accurate, generalizable forecasting models of neural dynamics across individuals and species, POCO lays the groundwork for adaptive neurotechnologies and large-scale efforts for neural foundation models. Code is available at [https://github.com/yuvenduan/POCO](https://github.com/yuvenduan/POCO).

POCO: Scalable Neural Forecasting through Population Conditioning
Yu Duan 1.5 Hamza Tahir Chaudhry? Misha B. Ahrens Christopher D. Harvey
Matthew G. Perich 6,7 Karl Deisseroth S*
Kanaka Rajan 4,5x
EECS, MIT SEAS, Harvard University 3 JJanelia Research Campus, HHMI
Harvard Medical School
Université de Montréal Stanford University *Corresponding Authors deissero@stanf tad edu, kanaka raj an@hms. harvard. edu
SKempner Institute
Mila Quebec AI Institute
Abstract
Predicting future neural activity core challenge in modeling brain dynamics, with applications ranging from scientific investigation to closed-loop neurotech- nology. While recent models of population activity emphasize interpretability and behavioral decoding, neural cenncaatinn-pprtiiuuuei spontaneous calcium weordings-remane unified forecasting model that combines a lightweight univariate forecaster with a population- encoder to capture both neuron- neuron-specific and brain-wide dynam- ics in calcium imaging recordings. Trained across five calcium imaging datasets spanning zebrafish, mice, and C. elegans, POCO achieves state-of-the-ar art accuracy at cellular resolution in spontaneous behaviors. After pre-training, POCO rapidly adapts to new recordings with minimal fine-tuning. Notably, POCO's 's learned unit embeddings recover biologically meaningful structure-such as brain region clus- tering-without wwithuut any anatomical labels. Our comprehensive analysis reveals several key factors influencing performance, including context length, session diversity, and preprocessing. Together, these results position POCO as a scalable and adapt- able approach for cross-session neural forecasting and offer actionable insights for future model design. By enabling accurate, generalizable forecasting models neural dynamics across individuals and species, POCO lays the groundwork for adaptive neurotechnologies and large-scale efforts for neural foundation models. comb yuvenduan/POC
across multi-session, underexplored. We introduce POCO, a
Code is available at
Introduction
The ability predict future states from the past is critical benchmark for models of complex systems such as the brain (32). Models capable of rapidly and accurately forecasting future neural activity across large spatial-temporal scales-rather than merely fitting historical data-are critical for applied technologies such as closed-loop optogenetic control [19. Here, we focus on a time-series forecasting (TSF) setup: Given recent history of measured neural activity, can we predict the neural population dynamics in the near future?
Recently, the increasing ability record from large populations of neurons has inspired a wide range of models for neural population dynamics [5]
27 46 60). However, much of this work has focused on interpreting latent features decoding behavior, while forward prediction of neural activity remains relatively underexplored. While some recent work has begun to explore forward modeling of spiking data [17 2 (56) [38], these approaches are not designed for calcium imaging and have been systematically compared with state-of-the-art models from time-series forecasting. Moreover, existing studies also tend focus short, trial-based data from individual animals in controlled tasks, but a comprehensive understanding of neural dynamics benefits from whole-brain recordings
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

during spontaneous, task-free behaviors. Furthermore, growing evidence for shared neural motifss across individuals [42
T2 (31, along with the rise of large-scale, multi-animal datasets, motivates trained across individuals that generalize to unseen subjects [ 6 2 However, classical models of population dynamics predominantly focus on fitting 400, [7 15]. limiting their ability to utilize larger datasets and capture common motif's shared across animals.
the development of foundation models-
single-session recordings
To address these gaps, we developed POCO (POpulation-COndiiionee forecaster), a unified predictive model for forecasting spontaneous, brain-wide neural activity. Trained on multi-animal calcium imaging datasets, POCO predicts combines simple univariate forecaster for individual neuron dynamics with a population encoder that models the influence of global brain state on each neuron, using Feature-wist Linear Modulation (FiLM) 339 to condition forecasts on population-level structure. For the population encoder, we adapt POYO 5 -riiinally developed for behavioral decoding in primates to summarize high- dimensional population activity across sessions. We benchmark POCO against standard baselines and state-of-the-art TSF models trained on five datasets from zebrafish, mice, and C. elegans.
dynamics up to 155 seconds into the future. It
In sum, our key contributions are: (1) We introduce POCO, novel architecture that combines local forecaster with a population encoder, for cellular-level neural dynamics forecasting. (2) We benchmark the forecasting performance of a wide range of models on five diverse calcium imaging datasets spanning different species, with a focus on neural recording during spontaneous behaviors. (3) We demonstrate that POCO scales effectively with longer recordings and additional sessions, and pre-trained POCO can quickly adapt to new sessions. (4) We conduct extensive analyses on factors that affect performance, including context length, dataset pre- prepproesssing multi-dataset training, and similarity between individuals. These analyses provide critical insights for future work on multi-session neural forecasting.
Related Work
Neural Foundation Models. A growing body of work aims to develop unified models trained on
neural data across multiple subjects, tasks, and datasets. This foundation model approach has been
applied to spiking data in primates (5) 52, 36 []. human EEG and fMRI recordings [ [0] (8).
and calcium imaging in mice 6 While much of this work emphasizes improving behavioral
decoding performance [
53,, some studies have explored forward prediction [7
2 5 38]
However, these efforts are largely confined spiking data recorded during short, base- motor or
mecision-making tasks. Neural prediction has also been explored in C. elegans 44] but the setup is limited to next-step prediction using autoregressive models.
Models of Population Dynamics. To understand nigh-dimmnsiooal population dynamics, one line of work has focused on inferring oow-dimensional latent representations from observations with models including RNNs [15
35 40
switching linear dynamical systems [18] 27], sequential variational
autoencoder ( (VAE) [46] 57 37 60), and latent diffusion models [22]. While some of these models
could, in theory, be adapted for forecasting, their focus to date has been to gain interpretable insights, especially in constrained neuroscience tasks.
Time-Series Forecasting. Time series forecasting is a general problem that emerges in a variety of domains [51 29 58 32). Deep learning has led a wide range of TSF architectures, including RNNs
[43. temporal convolutional networks (TCNs) [28 33] and Transformer-basee models 58 59 47
Simpler models such as MLP-based architectures (9] 55 [3 and even linear models 49 53 often
perform competitively or even outperform more complex alternatives. In neuroscience, Zapbench
B2 recent benchmark for forecasting, though recordings are limited to single larval zebrafish.
3 Method
3.1
Problem Setup
In this work, we consider multi-session TSF problem. For session j E [S], we use x denote the neural activity at time step where N, is the number of neurons recorded in session j.
R to
Given population activity of the last C' time steps x ca := Cx......tL E RCNN, we hope find

---

Features (M
FP M
Features (M)
F" FOy
Neurons (N)
Past neural signal
Prediction
Linear with
Linear
Neurons
ReLU
Neurons (N)
Neurons (N)
W
Time: 1
FILM Condilionng
Time: 64
Population Encoder (POYO)
7.8 g RA
Learnable Latents
B
Latents
16 steps form one token
Embedded Tokens
Neurons (N)
Embedding (d)
UnitEmbed(i, D)
Conditioning
K, V
L Self-Atten- tion
Q
Parameters
MM
Attention
K. V
Attention
Embedding (d)
EG, k) Wx A.k.
UnitEmbed(i, j) SebsionEmbed()
Figure 1: Model Architecture. (A) POCO combines univariate MLP forecaster (orange part) and a population encoder that conditions the MLP (blue part). This is a schematic for illustration only; traces and feature maps shown are not actual model input, output, or embedding. (B) The population encoder is adapted from POYO (S) We split the trace of each neuron into several tokens, encode the tokens with POYO, and then unit embedding query the conditioning parameters. See the Method section for more details.
predictor f that forecasts the next P steps and minimizes the mean squared error L:
f (x((2).))))
( LI
L(f) E,,
PN, kE+P x
(1)
where we use ||. || denotes Frobenius norm. most experiments, we use C' 48 and P 16. Importantly, the number of neurons N, varies across sessions, and neurons in different animals do not have one-to-one correspondences; yet, the forecasting problems for different sessions are closely linked due to the similarity in neural dynamics between animals [5 6 2 which distinguishes this setting from standard multivariate TSF setups.
3.2 Aopulation-Conditionn Forecaster
We first introduce the overall framework of POCO (Figure TA). Consider an MLP forecaster with hidden size M 1024 that takes past population activity x xi-Cit RRXN, input:
SMIP
( Cxt Wont ReLU i- bin) bouts
())
where W R MCC and W out E IR PxM are weight matrices. The MLP forecaster univariate, meaning that the prediction for neuron only depends on its own history, capturing individual auto-correlative properties and simple temporal patterns. Prior work has shown that these simple
univariate forecasters perform surprisingly well even for multivariate data 47 13 54 55].
Building on the MLP forecaster, add population encoder g that modulates the prediction of the MLP through Feature-wise Linear Modulation (FiLM) [39]. Specifically, the population encoder
gives the conditioning parameters g(x)2 () (y,9). f). where 7,B E MXN, are of the same shape as the hidden activations in MLP. We then define POCO as
SPoCo
(x (Pac) W out
ReLLU(W,,, in xn + bio B bout:
where @ denotes clement-wise multiplication. Intuitively, the FiLM conditioning allows the popu- lation encoder modulate how each neuron's 's past activity is interpreted, effectively tailoring the MLP forecaster to the broader brain state at each time point. This enables the model to account for context-dependent dynamics while maintaining neuron- -eeriin- predictions.
3.3 Population Encoder
We then need to define a population encoder y capable of modeling how the population state influences each neuron. To this end, we adapt recent architecture, POYO []]6), which combines the Perceiver-

---

I0 architecture [2] and a tokenization scheme for neural data (Figure B3. Specifically, for each neuron we partition the trace into segments of length Tc 16 and each segment forms a token, creating C/Tc = 3 tokens for each neuron. Then for each token k E [C/Tc] of each neuron i E [Ng]] we define the embedding Eii, k) E I2 as
()) kii, Wa Wa-- Terra + b + UnitEmbed(i,.) SessionEmbed(()),
where W E RaxTe is a linear projection, token, and both UnitEmbed (i. j) and SessionEmbed(j) are learnable embeddings in R Intuitively, after learning, the unit embedding can define the dynamical or finctiiona properties of the neuron, while the session embedding can account for different recording conditions (e.g., sampling rates, raw fluorescence magnitude) in different sessions. These token embeddings are arranged into a matrix E E CNN/Te xd which is then processed by Perceiver-IO [21]. Specifically, we use N = 8 learnable latents Lo N. xd as the query for the first layer. Lo is shared across sessions. After L self-attention layers, the final attention layer uses the unit embedding U, = UnitEmbed (+, E R N xd as queries to extract conditioning parameters (7, 3):
= kfc defines the temporal boundary of
L1 = Attentiono (Q = Lo; K V = E) E R
NNNLX
L Attention; (Q, (, Li) E IR Nexd xd
LE [L],
(5)
L L++ Attention (Q U
LL+1) RNxx
B WaL b3, 7 W, W,,L++2 by:
where Attention (Qtiion((,,4 . K, V) is a multi-head attention layer [50]. W., Wg E Mxd are learned weight matrices. We have L+ 2 attention layers in total, where I. = 1 in most experiments. Following POYO, we use rotary position embedding (55 (details are omitted above for simplicity). One advantage of the Perceiver-IO architecture that the time complexity only scales linearly with the number of neurons, allowing the model to efficiently scale to recordings of large neural populations. Although refer to individual neurons throughout, the same framework can also be applied to reduced representations of neural activity, such as principal components (PCs).
Our population encoder has two notable differences from the recent POYO+ model, which is also designed for calcium data (6) First, POYO+ is designed for behavioral decoding and thus the unit embedding is only used for tokenization. In contrast, here, unit embedding is reused in the last layer to query how the population drives each neuron. Second, in POYO+, Tc is always fixed to 1, which creates massive number of tokens when the context length C and the number of neurons N, large. We discuss the effect of Tc in Figure S15
4 Benchmark
4.1
Datasets
Table 1: Overview of the five datasets. fa is the sampling frequency. The number of neurons, recording length, and sampling frequency vary by session, the approximate average is shown here.
Species
Lab
#Sessions
#Neurons
#Steps
fs
Ca Indicator
Larval Zebrafish
Deisseroth
19
11K
4.3K
1.1Hz
GCaMP6s
Larval Zebrafish
Ahrens 11
15
77K
3.9K
2.1Hz
GCaMP66
Mice
Harvey 40 3
12
1.6K
15K
5.4Hz
GCaMP6s
C. elegans
Zimmer 23
5
126
3.1K
2.8Hz
GCaMP5K
C. elegans
Flavell
40
136
1.6K
1.7Hz
GCaMP7F
To comprehensively test the predictive capability of the models, we used five different datasets from different labs (Table Most recordings were collected during task-free free spontaneous behavior, though the Ahrens zebrafish dataset involves responses visual stimuli [II]. Details of the segmentation pipelines used to extract fluorescence traces are described in the original dataset publications. We z-scored fluorescence traces to zero mean and unit variance. For experiments involving predicting PCs, we computed PCs after 7-scoring individual neurons, and the magnitudes of PCs were preserved. We first cut each session into I K-step segments, then partitioned each segment into training, validation, and test sets by 3:1:1. See the Appendix for more details.

---

4.2 Baselines
We compared POCO against diverse set of baselines, from basic linear and regressiee models to state-of-the-art methods for time-series forecasting and dynamical system reconstruction. (1) MIP is POCO model without conditioning (Equation 2). For multi-session modeling, we use larger MLP (MLP_IF with similar parameter count as POCO. (2) NLinear, DI DLinear[55] are variants of univariate linear models that use linear projection from the context to predictions. (3) Latent PLRNN uses picce-wise linear RNN to model underlying dynamical states linked to observations through linear projection [24 35.. (4) TSMixer [66] is an all-MLP architecture for
TSF based mixing modules for the time and feature dimensions [16] (5) TexFilter [54] learns context-dependent frequency filters for time-series processing. (6) AR Transformer is a basic autoregressive Transformer [50]. (7) Netformer
31 infers inter-neuron connection strength via an attention layer, learning dynamic interaction graph. Here, we add softmax to attention weights for more stable training for multi-step prediction. (8) TCN denotes ModernTCN (3), recent multivariate pure-convolution architecture for TSF. More details about architectures and training can be found in the Appendix.
4.3 Copy Baseline and Prediction Score
Lastly, we considered a naive baseline that copies the last observation
feopy (x)))
(i)
(repeat for P = 16 steps).
(6)
Although extremely simple, due the slow dynamics of calcium traces, fcupy is strong baseline, especially in short-term forecasting 44 As a more intuitive metric than the raw MSE loss, we define the prediction score as the relative performance improvement compared to the copy baseline, i.e.,
Prediction Score (Smadel)
I[[fmodet) ( [[ ((ooopy)
which is similar to R2, but we use the last time step to replace the sample mean.
5 Results
Table 2: POCO achieves highest prediction scores across species and datasets. Prediction scores across five datasets show that POCO consistently outperforms baselines, especially in the multi- session setting. Zebrafish data are reduced to 512 PCs. 95% CI from seeds.
Zebrafish, 512 PCs
Mice
C-elegans
Model
Deisseroth
Ahrens
Zimmer
Flavell
Single-Session Models
POCO
0.466 #0.019
0.433 #0.088
0.415
#0.001
0.329
20.009
0.079 #0.017
MLP
0.399 +0.001
0.388 =0.001
0.409 +0.000
0.336 =0.001
0.236 #0.002
NLinear
0.167 40.000
0.211 20.000
0.348
20.000
0,250 20.000
0.217 20.001
Latent PLRNN
0,064 20.024
0.212 20.002
0.335 20.001
0.143 20.015
0.170 20.005
TexFilter
0.419 20.006
0.378 20.003
0.389 20.000
0,333 20,005
0.230 20.002
NetFormer
0.204 +0.013
0.208 +0.008
0.329
0.145 #0.004
0.168
AR_ Transformer
-0,875 +0.027
-0.054 +0.0ns
0.312 #0.005
-0.320 #0.063
-1.024 +0.038
DLinear
0.211
0.290 +0.000
0.394
0,267
0.221
TCN
0.153 #0.010
0.240 =0.004
0.360 #0.000
0.305 #0.003
0.226 #0.004
TSMixer
-0.550 +0.036
0.016 =0.009
0.390 #0.001
0.129 20.012
-0.199 +0.039
Multi-Session Models
MS_ PS_CO
0.525 +0.004
0.440 #0.003
0.420
#0.002
0.364
#0.005
0.213 #0.030
MS MS_MIL SS_MP__L
0.466 +0.001
0.418 +0.001
0.412 +0.000
0.360 #0.001
0.276 +0.007
MS OS_NLineer
0.165 +0.000
0.202 +0.000
0.347 +0.000
0.253 =0.000
().21 +0.000
MS Latent_PLRNN PLRNN
0.149 10.002
0.248 20.007
0.355 20.000
0,118 20.011
0.183 20.006
MS TexFilter
0,440 40.005
0.349 20.000
0.389 20.000
0,346 20.000
0.256 20.002
MS NetFormer
0.221 40 008
0.220 #0.002
0.331 +0.000
0.150 =0.002
0.217 #0.001
MS_ Trranforrrre
-0.777 #0.005
0.002 #0.012
0.317
-0.333 20.043
-0.675 #0.022
5

---

Table 3: Predicting at single-cell resolution in zebrafish. Prediction scores on raw neural traces (not PCA-reduced) from zebrafish datasets. All models are multi-session. 95% CI from 4 seeds.
Model
Zebrafish (Ahrens)
Zebrafish (Deisseroth)
MS POCO
0.429 #0.003
0.251 I 20.004
MS MS_MLP_L
0.423 #0.001
0.264 =0.000
WS__Nineerr
0.367 +0.000
0.172 +0.001
MS_TexFitter
0.398 +0.001
0.232 +0.004
Multi-Session POCO Outperforms Baselines. We tested POCO against baselines on five calcium imaging datasets (Table 2). Sample prediction traces for POCO are shown in Figure _ For zebrafish datasets, we first considered the more tractable problem of predicting the first 512 principal components (PCs) due to the large number of neurons. We compared training a different model for each session (single-session models) and training shared model for all sessions (multi-session models, MS). POCO consistently benefited from multi- training, outperforming all baselines on four out of five datasets. Other models such PLRNN, TexFilter, and NetFormer- also show performance gains from multi-session training. We obtained similar results measuring prediction errors by MSE and MAE (mean absolute error), as shown in Table 56 and S7
We further evaluated a subset efficient multi-session models on two zebrafish datasets at single-cell resolution (Table 33. Multi-session POCO achieved the best performance Ahrens' dataset and nest on Deisseroth's s, indicating its ability to model both raw neural activity and PCA-reduced signals. Nonetheless, its advantage over other models was smaller than in PC space, highlighting the difficulty of modeling large neuronal populations with sparse, noisy recordings.
A Zebrafion (Deissoroth), 512 PCs
Mics
B
Zebrafish (Delsswooh), 512 PGs
Mics
0.5-
Moin Stpornal
FOCO NLlneer Teofiber
pnod
TexHiter
12
36
Predicioo
Context [C]
Zobrafisht:
Sossion 1, NcNuron 18
Coologans:
Session Neuron
2. PC
M.F.L
Toffllrr
Sossion Neuron
Sccdion Neumn
gELFE
10
25
46
Figure 2: POCO maintains accuracy advantage over time and benefits from longer context. (A) MSE increases when forecasting longer into the future. Results are shown for two different datasets, see Figure [ygure/S6forr for additional datasets. Error bars show SEM of 3 random seeds. (B) Model performance improves with longer context.
(C) Sample prediction traces produced by POCO, large MLP, and Texfilter. The first C' = 48 steps given to the model as context.
Effect of Context and Prediction Length. We observed that prediction error gradually increases over = 16 prediction steps, but POCO generated relatively accurate predictions across all time steps (Figure 2)). To evaluate the impact of context length, we also tested on shorter context lengths Ce (1,3, 12, 24). and adjusted POCO's s token length Te to {1, 3, 6} to control the number of tokens. We found that univariate models perform poorly with short contexts, consistent with results in recent work (32). POCO outperformed baselines in different context lengths (Figure B). The prediction accuracy increased with context length but plateaus beyond C > 12, suggesting that most of the predictive information is contained within a relatively short temporal window.
POCO Performance Improvements Scale with Recording Length. We next tested how POCO performance scales with dataset size. First, instead of using the full training partition in each session, we tested the model performance when only the first Tirain steps are used (Figure 3 S7). We found that POCO shows steady improvement when longer recordings are used for training, whereas TexFilter shows slower improvements, and NLinear shows no apparent improvement. Second, we split sessions one dataset into several approximately even partitions and train one model for each partition (Figure 3 B, S8 We found that POCO consistently benefits from training on more sessions. Taken together, these results suggest that POCO effectively leverages longer recordings across sessions to learn complex neural dynamics.

---

Zebrafish (Deisceroth), 512 PCs
Mice
B
Zebrafich (Deiseerodh), PCs
Mice
0,425
0.52
D.422
W.aattn Sgore
0.-
0-2
9.375-
0.350
0.50
0.43
0.55
Jono modet for ench seceion)
(ore mode for of 19 sesaiona
Prediciinn Sicre
2.412
0.405
0.402
0:415
-4- POCD
- Toofttr
0.300
0.44
0.355
og2ttaalinng recording longth
0,275
leg2ttraniigg recording
76
24
Number stlts
12
Number spits
Figure 3: POCO performance improves with longer recordings and more sessions. (A) Prediction score vs. training recording length (x-axis in log scale) for two different datasets, Models were trained using increasing portions of each session's 's data. Error bars show SEM across 3 random seeds. (B) We split all sessions in one dataset into n approximately equal partitions and train one model on each partition, then take the average of model prediction scores across all partitions. Average prediction scores vs the number of splits n is shown for two datasets.
Table 4: POCO benefits from multi-sessson, but not multi-species, training. Comparing Angle-session, multi-session multi-species and zebrafish-only POCO variants. Within-species training yields the best performance. 95% CI from 4 seeds.
Zebrafish, 512 PCs
Mice
C-elegans
Model
Deisseroth
Ahrens
Zimmer
Flavell
Single-Session POCO
0,466 20.019
0.433 20.008
0.415 20.001
0.329 20.009
0,079 20.017
Multi-Session POCO
0,525 20.004
0.440 20.003
0.420 20.002
0.364 20.005
0.213 20.030
Multi-Species POCO
0.499 20.003
0.441 20.003
0.403 #0.000
0.330 #0,009
0.252 #0.011
Zebrafish POCO
0.500 #0.005
0.442 =0.004
POCO Does Not Consistently Benefit from Multi-Species Training. To see datasets from multiple species improves performance, we trained POCO on different datasets simultaneously. Specifically, in each model update step, we aggregated the loss computed from one random batch of data in each dataset. We found that the model generally does not benefit from multi-species training (Table 4 This may be due to differences across datasets in pre-processing pipelines, recording conditions, and, perhaps more importantly, differences in the underlying neural dynamics of recorded species, animals' behavioral states, and specific brain regions (Table [ Given this result and a recent report that pre-training on other specimens does not improve neural forecasting performance in zebrafish (20), we hypothesize that the model performance only significantly benefits from more sessions when the modeled systems are similar enough. We explored this hypothesis in the following simulation experiment.
A
0.10
paco POCD
B Zebrafish (Deisserolh)) 512
0.08-
6.11
POCO jwwbeeaing siriiinggg
0.05
0.32
1500
0.22
0.04
24
0.93
1.54
0.59
0.00
0.02
0.5
30
330
150
200
50
350
203
Thalking Stop
Taaning Stop
Figure 4: Multi-session POCO improves when individuals similar; POCO can quickly adapt to new sessions (A) Performance gain of multi-session POCO compared to single-sessing POCO on synthetic data for different values of 17. Larger 17 means individuals are less similar. We randomly generated 16 cohorts, each with 16 individuals. Each blue cross represents a cohort. Error bars SEM. (B) Validation loss curve of fine-tuning pre-trained POCO (Pre-POCO), pre-trained MLP (Pre-MLP), and training POCO, MLP from scratch. We also compared full-finetuning with only tuning the embedding, Error shades represent SEM for 3 random seeds. Two sample sessions from two datasets are shown here; see Figure S12 for more sessions.
Simulation. Here, we used simulated data to test how similarity between individuals influences multi-session model performance. To generate neural data from a synthesized cohort, we first randomly sample a template connectivity matrix Jo I) e guxa, where we use 300
7

---

neurons. Then for each synthesized individual i [16], we set
J = V 1 17 + ne,, ~~~ ((0,
where G is random deviation from the template connectivity matrix, E [0. 1| controls the similarity between individuals. We set the coefficient for Jo to be /1- 12² so that J, keeps unit variance. We then used each J as the connectivity of a noisy spontaneous RNN to generate synthetic neural data (see the Appendix for details) and trained POCO as on the multi-session neural data above (Figure 4A). POCO showed greater benefit from multi-session training when individuals shared similar connectivity patterns I 0.05), compared to when their connectivity was independent (1) 1).
We also compared POCO with baselines on single-session simulation data. Surprisingly, models including PLRNN, auto-regressive Transformer, and TSMixer performed significantly better than POCO, despite their relatively poor performance on real neural datasets (Figure Real neural data likely exhibits greater non-stationarity, multi-scale dependencies, and heterogeneous noise profiles than our current simulations, properties which POCO's architecture may be better suited to handle than models excelling in the simulated regime.
Finetuning on New Sessions. A core benefit of the foundation model approach is that it allows rapid adaptation to new sessions. We pre-trained POCO on 80% of sessions and fine-tuned this model on each of the remaining ones (see the Appendix for details). For finetuning, we compared full finetuning with only finetuning the unit and session embedding. We found that pre-trained models achieved reasonable predictive performance in only tens of training steps (Figure B). In addition, full finetuning leads to limited or no improvements compared to only fine-tuning the unit embedding (Table S9h. Rapid adaptation is crucial for real-time, closed-loop applications: in our setup, fine-tuning the embedding for 200 training steps takes less than 15 seconds, and the forecasting inference time is only 3.5 ms (see the Appendix for details). Consistent with previous results on multi-dataset training, pre-training different datasets does not improve performance (Table S1O
Zobrafish (Deisseroth), 512 PCs
Mice
C Zobrofish (Delsseratt)
Mice Similarity
3.3
Uni: Embedcing (JPAA)
Untt Embedting (UMAP)
Narmalized Cosine Sinilarity by
500
03
400
0.5
0.4
Figh
0.
Figure 5: POCO learns meaningful unit embeddings without supervision. (A) UMAP [34 visu- alization of unit embeddings after training POCO on the first 512 PCs in zebrafish dataset. (B) Visualization of unit embedding after training POCO on the mice dataset, where neurons are colored by the brain region. One sample session shown for (A) and (B), see Figure (S9 for more sessions. (C) Normalized average cosine similarity of unit embeddings between each pair of regions. Each row is normalized to [0, 1] and then averaged across runs, Patterns are consistent for different seeds (Figure STO See the Appendix for more details.
Analyzing Unit Embedding, Recent work shows that unit embeddings in POYO learns region and cell-type-specific information when trained for behavioral decoding (6] We analyzed the trained multi-session POCO model to determine if similar structure emerges. We found that when trained on PCs, unit embeddings are consistently distributed according to the order of the PCs (FigureBAA) When trained on neurons, we found significant clustering of neurons by brain region, indicated by the average pair-wise cosine similarity (Figure 5CC). In particular, retrosplenial cortex (RSP) neurons in mice form particularly distinct cluster (Figure 5B. Thus, POCO learns to encode functional and dynamical properties in the unit embedding when trained for forecasting, even when no prior knowledge (e.g., neuron location) is given to the model.
Effect of Filters. Filtering high or low frequency components is common preprocessing technique in calcium imaging, used to remove slow drifts or fast noise that are not directly related to neural dynamies 23 [6]. By default, we used no temporal filter to maximally preserve neural signals. However, found that with now-filtering. POCO still outperforms baselines except C. elegans
8

---

datasets (Table|S8). Low-pas pass filters improved models' performance compared to the copy baseline in most datasets, suggesting that high-frequence components are generally harder to predict (Figure S13 Low-pass filtering also helped POCO to benefit more from multi-session training (Figure (8))
Zaphench Evaluation. A Although the main focus of this work is on multi-session datasets, also tested our model on a recent neural population forecasting benchmark, Zapbench (32), which contains light-sheet microscopy recordings of 71721 neurons over 7879 time steps for one zebrafish. We followed the setup in Zapbench to partition the dataset and evaluate our model with short (C 4) or long (C = 256) context, and prediction length P 32. We found that when = 4, POCO outperforms other tracebbasdd methods and performs comparably to UNet, computationally expensive model operating directly on raw volumetric videos rather than segmented neural traces At longer contexts (C = 256). POCO underperforms relative to UNet and TSMixer (Figure See the Appendix for more details.
20
S14
Table 5: Both MLP forecaster and population encoder are necessary for POCO. Ablation study shows performance drops when either component is removed or simplified. 95% CI from 4 seeds.
Zebrafish, 512 PCs
Mice
Model
Deisseroth
Ahrens
Full POCO
0.525 #0,004
0.440
=0.003
0.420 #0.002
POYO only
-0.971 1 =0.015
-0.057
=0.001
0.332 #0.001
MLP only
0,417 #0.002
0.370
=0.001
0,409
#0.000
MLP conditioned by univariate Transformer
0,463 =0.001
0.408
20.002
0,411 20,000
MLP conditioned by unit embedding
0.481 20.004
0.434 20.004
0.413 20,000
Ablation Study. To test the necessity of the components of POCO, we removed or replaced some parts of the model and compared performance. Specifically, we tested (1) directly using the POYO model to generate 16-steps prediction instead of generating conditioning parameters (POYO only); (2) MLP without conditioning; (3) MLP conditioned by univariate Transformer that takes the calcium trace of a single neuron instead of encoding the whole population: (4) 3-layer MLP directly conditioned by unit embedding (see the Appendix for more details). Both the MLP forecaster and the population encoder were necessary for full performance (Table5)). We also tested how key model hyperparameters influence performance, including the length of the token (Figure S15). embedding dimension (Figure (T6)), number of layers (Figure S17). the number of latents (Figure SS ST8). learning rate (Figure ST9) and weight decay (Figure $20 We found that POCO performance is relatively stable for a range of hyperparameter settings, and even a small POYO encoder is sufficient.
6 Discussion
In this work, we introduced POCO, a population-conditionee forecaster that combines a local univariate predictor with a global population encoder to capture both neuron-level dynamics and shared brain-state structure. Across five calcium- calcium-imaging datasets in zebrafish, mice, and C. elegans, POCO achieves state-of-the-art overall forecasting accuracy. Beyond raw performance, we show that POCO rapidly adapts to new sessions with only tens of fine-tuning steps of its embeddings, making feasible for real-time adaptation during live recordings. Our analysis of POCO's learned unit embeddings demonstrates that the model autonomously uncovers meaningful population structure such as brain regions, even though no anatomical labels were provided. These findings underscore POCO's dual strength in accurate prediction and in learning interpretable representations of units. Finally, our results extend the recent progress on scaling up models for behavioral decoding [S] to the realm of neural prediction on spontaneous neural recordings in different species.
We note a few limitations that can be opportunities for future research. First, our results indicate that factors such as calcium indicator dynamics, preprocessing pipelines, and species differences can significantly affect model performance, yet a systematic understanding of their influence remains lacking (see Appendix for more additional discussion). Second, our findings highlight the difficulty of multi-dataset and multi-species training a challenge that may be mitigated by improved architectures or alignment strategies. Third, while POCO learns biologically meaningful unit embeddings within datasets, is unclear whether these representations are comparable across species or generalize to unseen brain regions. Finally, while we focus on calcium imaging during spontaneous behavior, extending POCO to spiking data and standardized behavioral tasks could enable the use of larger datasets and support modeling of neural dynamics in more structured settings [14 26).

---

Acknowledgments and Disclosure of Funding
This work was supported by the NIH (RF1DA0054033, James S. McDonnell Foundation (220020466), Simons Foundation (Pilot Extension-000003322000) McKnight Endowment Fund, CIFAR Azrieli Global Scholar Program, and NSF (2046583).
We would like to thank Sabera Talukder, Krystal Xuejing Pan, and Viren Jain for helpful discussions.
References
[1] A. S. Andalman, V. M. Burns, M. Lovett-Barron, M. Broxton, B. Poole, S. J. Yang. L. Grosenick, T. N. Lerner, R. Chen, T. Benster, et al. Neuronal dynamics regulating brain and behavioral state transitions. Cell, 177(4)9700-88,, 2019.
[2] A. Antoniades, Y. Yu, J. Canzano, W. Wang, and S. L. Smith. Neuroformer: Multimodal and multitask generative pretraining for brain data. arXiv preprint arXiv:2311.00136, 2023.
[3] C. Arlt, R. Barroso-Luque, S. Kira, C. A. Bruno, N. Xia, S. N. Chettih, S. Soares, N. L. Pettit, and C. D. Harvey. Cognitive experience alters cortical involvement in goal-directed navigation. Elife, 11:e76051, 2022.
[4] A. A. Atanas, J. Kim, Z. Wang, E. Bueno, M. Becker, D. Kang, J. Park, T. S. Kramer, F. K. Wan, S. Baskoylu, et al. Brain-wide representations of behavior spanning multiple timescales 2023.
and states in c. elegans. Cell, 186(19):4134-44
[5] M. Azabou, V. Arora, V. Ganesh, Mao, S. Nachimuthu, M. Mendelson, B. Richards, M. Perich, G. Lajoie, and E. Dyer. A unified, scalable framework for neural population decoding. Advances in Neural Information Processing Systems, 36:44937-44956, 2023.
[6] M. Azabou, K. X. Pan, V. Arora, I. J. Knight, E. L. Dyer, and B. A. Richards. Multi-session multi-task neural decoding from distinct cell-types and brain regions. In The Thirteenth International Conference on Learning Representations, 2025.
[7] M. Brenner, F. Hess, J. M. Mikhaeil, L. F. Bereska, Z. Monfared, P.-.. Kuo, and D. Durstewitz. Tractable dendritic rnns for reconstructing nonlinear dynamical systems. In International conference on machine learning, pages 2292-2320. Pmir, 2022.
[8] J. O. Caro, A. H. d. O. Fonseca, C. Averill, S. A. Rizvi, M. Rosati, J. L. Cross, P. Mittal, E. Zappala, D. Levine, R. M. Dhodapkar, et Brainlm: A foundation model for brain activity recordings. bioRxiv, pages 2023-09, 2023.
[9] C. Challu, K. G. Olivares, B. N. Oreshkin, F. G. Ramirez, M. M. Canseco, and A. Dubrawski. Nhits: Neural hierarchical interpolation for time series forecasting. In Proceedings of the AAAI conference on artificial intelligence, volume 37, pages 6989-6997, 2023.
[10] G. Chau, C. Wang. S. Talukder, V. Subramaniam, S. Soedarmadji, Y. Yue, B. Katz, and A. Barbu. Population transformer: Learning population-level representations of neural activity. ArXiv, pages arXiv-2406, 2024.
[11] X. Chen, Y. Mu, Y. Hu, A. T. Kuan, M. Nikitchenko, O. Randlett, A. B. Chen, J. P. Gavornik, H. Sompolinsky, F. Engert, et al. Brain-wide organization of neuronal activity and convergent sensorimotor transformations in larval zebrafish. Neuron, 100(4):876-89)) 2018
[12] M. M. Churchland, J. Cunningham, M. T. Kaufman, J. D. Foster, P. Nuyujukian, S. Ryu, and K. V. Shenoy. Neural population dynamics during reaching. Nature, 487(7405):51-56, 2012.
[13] A. Das, W. Kong, A. Leach, S. Mathur, R. Sen, and R. Yu. Long-term forecasting with tide: Time-series dense encoder. arXiv preprint 2rXiv:2304.08444 2023.
[14] S. E. de Vries, J. A. Lecoq, M. A. Buice, P. A. Groblewski, G. K. Ocker, M. Oliver, D. Feng, N. Cain, P. Ledochowitsch, D. Millman, et al. A large-scale standardized physiological survey reveals functional organization of the mouse visual cortex, Nature neuroscience, (3(1):33-111 2020.
10

---

[15] D. Durstewitz. A state space approach for piecewise-linear recurrent neural networks for identifying computational dynamics from neural measurements. PLoS computational biology. 13(6))e1005542, 2017.
[16] V. Ekambaram, A. Jati, N. Nguyen, P. Sinthong, and J. Kalagnanam. Tsmixer: Lightweight mip-mixer model for multivariate time series forecasting. In Proceedings of the 29th ACM SIGKDD conference on knowledge discovery and data mining, pages 459-469, 2023.
[17] A. C. Filipe and M. Park. One model to train them all: A unified diffusion framework for multi-context neural population forecasting, 2025. URL http:///ppeerreiii...uooo. net/forum? id=R9febbYRG7
[18] J. Glaser, M. Whiteway. J. P. Cunningham, L. Paninski, and S. Linderman. Recurrent switching dynamical systems models for multiple interacting neural populations. Advances in neural information processing systems, 33: 13:14487-14878, 2020.
[19] L. Grosenick, J. H. Marshel, and K. Deisseroth. Closed-loop and activity- giiiiyy-guddddd optogenetic control. Neuron, 86(1):106-33,, 2015.
[20] A. Immer, J.-M. Lueckmann, A. B.-Y. Chen, Li, M. D. Petkova, N. A. Iyer, A. Dev, G. Ihrke, W. Park, A. Petruncio, et al. Forecasting whole-brain neuronal activity from volumetric video. arXiv preprini arXiv:2503.00073. 2025.
[21] A. Jaegle, S. Borgeaud, J.-B. Alayrac C. Doersch, Ionescu, D. Ding, Koppula, D. Zoran, A. Brock, E. Shelhamer, et al. Perceiver io: A general architecture for structured inputs & outputs. arXiv preprint arXiv:2107. 14795, 2021.
[22] J. Kapoor, A. Schulz, J. Vetter, F. Pei, R. Gao, and H. Macke. Latent diffusion for neural spiking data. Advances in Neural Information Processing Systems, 37:118119 19199118154, 2024.
[23] S. Kato, H. S. Kaplan, T. Schrödel, S. Skora, T. H. Lindsay, E. Yemini, S. Lockery, and M. Zimmer. Global brain dynamics embed the motor command sequence of caenorhabditis elegans. Cell, 163(3):656-669, 2015.
[24] G. Koppe, H. Toutounji, P. Kirsch, S. Lis, and D. Durstewitz. Identifying nonlinear dynamical systems via generative recurrent neural networks with applications fmri. PLoS computational biology, 15(8)=e100728) 2019.
[25] A. Kumar, A. Gilra, M. Gonzalez-Soto A. Meunier, and M. Grosse-Wentrup Bunddle-net: Neuronal manifold learning meets behaviour. bioRxiv, pages 2023-08, 2023.
[26] I. B. Laboratory, B. Benson, J. Benson, D. Birman, N. Bonacchi, K. Bougrova, S. A. Bruijns, M. Carandini, J. Catarino, G. A. Chapuis, et al. A brain-wide map of neural activity during complex behaviour. biorxiv, pages 2023-07, 2023.
[27] S. W. Linderman, A. C. Miller, R. P. Adams, D. M. Blei, L. Paninski, and M. J. Johnson. Recurrent switching linear dynamical systems. arXiv preprint
2016.
[28] M. Liu, A. Zeng, M. Chen, 7. Xu, Q. Lai, L. Ma, and Q. Xu. Scinet: Time series modeling and forecasting with sample convolution and interaction. Advances in Neural Information Processing Systems, 55:5816-5822 2022.
[29] X. Liu, Y. Xia, Y. Liang, Hu, Y. Wang. L. Bai, Huang, Z. Liu, B. Hooi, and R. Zimmermann. Largest: A benchmark dataset for large-scale traffic forecasting. Advances in Neural Information Processing Systems, 36:75354-75371. 2023.
[30] I. Loshchilov and F. Hutter. Decoupled weight decay regularization. arXiv preprint 1711v:1711.051011 2017.
[31] Z. Lu, W. Zhang, T. Le, H. Wang, U. Sümbül, T. SheaBrown, and L. Mi. Netformer: An interpretable model for recovering dynamical connectivity in neuronal population dynamics. In The Thirteenth International Conference on Learning Representations, 2025. URL httpss
/ //openreview wwt.erewwwnnneeeeeeeeeeeeeee .wtt://orumiidddcccjiasssss

---

[32] J.-M. Lueckmann, A. Immer, A. B.-Y. Y. Chen, P. Li, M. D. Petkova, N. A. Iyer, L. W. Hesselink, A. Dev, G. Ihrke, W. Park, et al. Zapbench: benchmark for whole-brain activity prediction in zebrafish. arXiv preprint
2025.
[33] D. Luo and X. Wang. Modernten A modern pure convolution structure for general time series analysis. In The twelfth international conference on learning representations, pages 1-43, 2024.
[34] L. McInnes, J. Healy, and J. Melville. Umap: Uniform manifold approximation and projection for dimension reduction. arXiv preprint arXiv:1802.03426, 2018.
[35] J. Mikhaeil, Z. Monfared, and D. Durstewitz. On the difficulty of learning chaotic dynamics with mnns. Advances in neural information processing systems, 35:1 1297-11312, 2022.
[36] M. Pachitariu, C. Stringer, S. Schröder, M. Dipoppa, L. F. Rossi, M. Carandini, and K. D. Harris. Suite2p: beyond 10,000 neurons with standard two-photon microscopy. BioRxiv, page 061507, 2016.
[37] C. Pandarinath, D. J. O'Shea, J. Collins, R. Jozefowicz, S. D. Stavisky, J. C. Kao, E. M. Trautmann, M. T. Kaufman, S. I. Ryu, L. R. Hochberg, et al. Inferring single-trial trial neural population dynamics using sequential auto-encoders. Nature methods, (5(00))805-815, 2018.
[38] F. Pei, J. Ye, D. Zoltowski, A. Wu, R. H. Chowdhury, H. Sohn, J. E. O'Doherty, K. V. Shenoy, M. T. Kaufman, M. Churchland, et al. Neural latents benchmark 21: Evaluating latent variable models of neural population activity. arXiv preprint arXiv:2109.0444
2021.
[39] E. Perez, F. Strub, H. De Vries, V. Dumoulin, and A. Courville. Film: Visual reasoning with general conditioning layer. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018.
[40] M. G. Perich, C. Arlt, S. Soares, M. E. Young, C. P. Mosher, J. Minxha, E. Carter, U. Rutishauser, P. H. Rudebeck, C. D. Harvey, et al. Inferring brain-wide interactions using data-constrained recurrent neural network models. BioRxiv, pages 2020-12, 2020.
[41] E. A. Pnevmatikakis, D. Soudry, Y. Gao, T. A. Machado, J. Merel, D. Pfau, T. Reardon, Y. Mu, C. Lacefield, W. Yang, al. Simultaneous denoising, deconvolution, and demixing of calcium imaging data. Neuron, 89(2):285-295 2016.
[42] M. Safaie, J. C. Chang, J. Park, L. E. Miller, J. T. Dudman, M. G. Perich, and J. A. Gallego. Preserved neural dynamics across animals performing similar behaviour. Nature, 623(7988): 765-771, 2023.
[43] D. Salinas, V. Flunkert, J. Gasthaus, and Januschowski. Deepar: Probabilistic forecasting with autoregressive recurrent networks. International journal of forecasting, 36(3):1181-1191, 2020.
[44] Q. Simeon, L. Venâncio, M. A. Skuhersky, A. Nayebi, E. S. Boyden, and G. R. Yang. Scaling properties for artificial neural network models of a small nervous system. In SoutheastCon 2024, pages 516-524. IEEE, 2024.
[45] J. Su, M. Ahmed, Y. Lu, S. Pan, W. Bo, and Y. Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputingg 568:127063, 2024.
[46] D. Sussillo, R. Jozefowicz, L. Abbott, and C. Pandarinath. Lfads-latent factor analysis via dynamical systems. arXiv preprint
2016.
[47] S. Talukder, Y. Yue, and G. Gkioxari. Totem: Tokenized time series embeddings for general time series analysis. arXiv preprint arXiv:2402.16412, 2024.
[48] A. Thomas, C. Ré, and R. Poldrack. Self-supervised learning of brain dynamics from broad neuroimaging data, Advances in neural information processing systems, 35:21255-21261 2022.
[49] W. Toner and L.. Darlow. An analysis of linear time series forecasting models. arXiv preprint 2xXi:22403.14587 2024.
12

---

[50] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. Attention is all you need. Advances neural information processing systems, 30, 2017.
[51] H. Wu, H. Zhou, M. Long, and J. Wang. Interpretable weather forecasting for worldwide stations with a unified deep model. Nature Machine Intelligence, 5(6):602 61116022611,, 2023.
[52] J. Ye and C. Pandarinath. Representation learning for neural population activity with neural data transformers. arXiv preprint
2021.
[53] Ye, J. Collinger, L. Wehbe, and R. Gaunt. Neural data transformer 2: multi-context pretraining for neural spiking activity. Advances in Neural Information Processing Systems, 36:80352- 80374, 2023.
[54] K. Yi, J. Fei, Q. Zhang. He, Hao, D. Lian, and Fan. Filternet: Harnessing frequency filters for time series forecasting. Advances in Neural Information Processing Systems, 37: 55115-55140, 2024.
[55] A. Zeng, M. Chen, L. Zhang, and Q. Xu. Are transformers effective for time series forecasting? In Proceedings of the AAAl conference on artificial intelligence, volume 37, pages 11121-11128, 2023.
[56] Y. Zhang, Y. Wang, D. Jiménez-Benetó, Z. Wang, M. Azabou, B. Richards, R. Tung, O. Winter, E. Dyer, L. Paninski, al. Towards a" universal translator" for neural dynamics single-cell, single-spike resolution. Advances in Neural Information Processing Systems, 37:80495-80521, 2024.
[57] D. Zhou and X.-.. Wei. Learning identifiable and interpretable latent models of high- dimensional neural activity using pi-vae. Advances in neural information processing systems, 33:7234-7247, 2020.
[58] H. Zhou, S. Zhang, Peng. Zhang, J. Li, H. Xiong, and W. Zhang. Informer: Beyond efficient transformer for long sequence time-series forecasting. In Proceedings of the AAAI conference on artificial intelligence, volume 35, pages 11106-11115, 2021.
[59] T. Zhou, Z. Ma, Q. Wen, X. Wang, L. Sun, and R. Jin. Fedformer: Frequency enhanced decomposed transformer for long-term series forecasting. In International conference on machine learming, pages 27268-27288 PMLR, 2022.
[60] F. Zhu, H. A. Grier, R. Tandon, C. Cai, A. Agarwal, A. Giovannucci, M. T. Kaufman, and
C. Pandarinath. A deep learning framework for inference of single-trial neural population dynamics from calcium imaging with subframe temporal resolution. Nature neuroscience, 25 1733 2022.
13

---

A Supplementary Results
Table S6: MSE, comparison across datasets. Mean squared error (MSE) across five datasets, using the same model configurations as [2 Different from Table where the score computed by averaging across all sessions, here sessions are weighted by the size of the test set (N; x Dtest where Drest is the number sequences in the test set), so longer recording sessions and sessions with more neurons recorded will receive larger weight.
Zebrafish, 512 PCs
Mice
C-elegans
Model
Deisseroth
Ahrens
Zimmer
Flavell
Single-Session Models
POCO
6.022 +0.083
40.118 #0.473
0.845 #0.002
0.369
+0.005
0.696 #0.011
ML_
7.082 #0.021
43.335
#0,024
0.853 #0,000
0.365
40.001
0.582 +0.001
NLinear
9.587 #0.005
55,742 20.016
0.944 #0.000
0,411
#0.000
0.597 #0.000
Latent_ PLRNN
10,423 =0. 102
55,047 #0.189
0,964 #0.001
0.470 #0.007
0,633 #0.005
TexFilter
6,590 #0.066
44.040
#0.215
0.882 #0.000
0.368 +0.003
0.588 #0.001
NetFormer
8,877 10.129
56,169 40.527
0.973 20.000
0.466 40.002
0,635 a0.001
AR. Transformer
18.929 20.005
73.523
20.632
1.008 20.012
0.688 20.032
1.517 20.029
DLinear
8,986 20.003
49.993
20.018
0.876 20,000
0.401 20.000
0.594
20.000
TCN
9.575 +0.076
53.725
#0.254
0.926
+0.001
0.383 +0.002
0.591
+0.003
TSMixer
15.516 +0.354
783
40.682
0.882
+0,002
0)468
+0.004
0..1 1 +0.027
Mulri-Session Models
MS PPCO
5,498 #0.063
39.678
#0.198
0.839
#0.003
0.350
#0.003
0.600
#0.023
MS MLP L
6.108 #0.011
41.294
#0.110
0.850
20,000
0.352 +0,000
0.552 #0.005
MS WS_NLinear
9.565 #0.015
56,477
#0.006
0.946
+0,000
0.409
#0.000
0.594 =0.000
MS
9.597 #0.026
52.571
#0.435
0.936
+0.000
0.476 +0.005
0.621 #0.004
MS_ TexFilter
6.377 #0.053
46.221
#0.055
0.882
+0.001
0.361
+0.000
0,568
#0.001
MS NetFormer
8.803 #0.076
55.351
#0,210
0.970
#0.001
0.461 +0.001
0.597 +0.000
MS AR_ Transffrrmer
18.268 20.025
70.082
20.767
0.997 20.004
0.696 10.015
1.261 20.017
Worms (Zimmer)
(Flavell)
0.55
0.8
Meen Squared Ener
0.50
Meen Squared Enerr
0.7-
NLinear
0.6
NLinear
0.40
0.5
0.35
0.4
0.30
0.3
0.2
8
12
166
12
16
Prediction
Prediction Stap
Rish PC (Ahrens)
Rih (Deisteroth)
0.8
80
Mean Squared Errot
Mean Squared Error Error
0.6
NUnear TexFitter
NLinear
0.4
20
0.2
8
12
16
4
12
16
Prodiction
Prodiction Stap
Figure S6: Forecasting error increases with prediction horizon across datasets. Mean squared eиror (MSE) plotted over 16 prediction steps for four additional datasets. POCO maintains better performance over time compared to NLinear and TexFilter. Error bars show SEM across 3 seeds.
14

---

Table S7: Mean absolute error (MAE) comparison across datasets. Same as Table 56 reporting mean absolute error (MAE) instead of prediction score. 95% confidence intervals estimated with random seeds.
Zebrafish, 512 PCs
Mice
C-elegans
Model
Deisseroth
Ahrens
Zimmer
Flavell
Single-Session Models
POCO
1.310 20.005
4.116 #0.006
0.645 20.001
0,418 20.004
0,604 =0.007
MLP
1.442 #0.002
4.322 40.001
0.648 #0.000
0,413 #0.001
0,540 =0.001
NLinear
1,818 #0.000
4.872 +0.001
0.686 +0.000
0,438 =0.000
0.545 =0.000
Latent PLRNN
1.965 =0.009
5.092 20.008
0.709 +0.000
0.484 20.004
0.559 #0.002
TexFilter
1.443 10.010
4.416 #0.012
0.657 20.000
0.409 =0.002
0.537 #0.001
NetFormer
1.757 20.013
4.852 20.007
0.693 20.000
0.471 20.001
0,560 =0.001
AR Transformer
2,465
5.500
0.712 +0.006
0.606 =0.014
0.941
DLinear
1.761 +0.000
4.645
+0.001
0.658 +0.000
0.440 +0.000
0.547 +0.000
TCN
1.847 +0.016
4.822
0.674 +0.000
0.421
=0.001
0.539
TSMixer
2.026 +0.039
5.021 #0.035
0.665 =0.001
0.493
20.001
0.702 =0.010
Multi-Sessiee Models
MS POCO
1.262 =0.004
4.096 #0.006
0.643
=0.002
0.402
+0.003
0,556 #0.011
MS 3__MP__L
1.358 #0.002
4.191
+0.002
0.647
#0.001
0.403 =0.003
0.522 #0.004
MS NLinear
1.824 #0.001
4.868 #0.003
0.687 +0.000
0.437 #0.000
0,545 =0.000
MS Latent PLRNN
1.892 20.004
4.966 20.005
0.694 20.000
0.488 20.002
0.561 20.003
MS TexFilter
1.418 20.011
4.444
20.007
0.658 20.001
0.404 20.000
0,526 =0.002
MS NetFormer
1.766 10.006
4.849
20.004
0.692 20.000
0.469
20.001
0.544 00.000
MS AR Transformer
2.454 20.003
5.421
20.065
0.715 #0.002
0.615 20.007
0,862 #0.006
Table S8: Low-pass filtering improves prediction in most datasets. Performance across five
datasets after applying low-pass filter with cutoff of 0.1 x fs. where fa is the sampling frequency. Filtering improves POCO's performance relative to the copy baseline in most settings. Results are averaged across sessions; 95% confidence intervals from 4 seeds.
7ebrafish, 512 PCs
Mice
C-elegans
Model
Ahrens
Zimmer
Flavell
Single-Session Models
POCO
0.60 0.02
0.50 0.00
0.21 +.02
-0.04 0.03
MLP
0.59 + 0.00
0.50 = 0.00
0.31 # 4.000
0.35 = 0.00
NLinear
0.33 0.00
0.33 0.00
0.16 0.00
0.27 0.00
Latent_PLRNN PLRNN
0.21 + 0.00
0.19 = 0.00
0.13 + 0.02
0.16 = 0.01
TexFilter
0.61 +0.0
0.50 0.01
0.37 +.01
0.34 0.01
NetFormer
0,27 = 0.00
0.31 = 0.00
0.01 + 0.01
0.09 2 0.00
AR Transformer
-0.20 0.02
-0,14 0.01
-1.37 0.13
-1.48 1.003
DLinear
0.39 + 0.00
0.41 0.00
0.17 + 0.00
0.26 = 0.00
TCN
0.32 0.01
0.31
0.19 0.03
0.21 0.02
TSMixer
-0.05 0.03
0.32 = 0.00
-0.45 0.03
-0.46 0.06
Multi-Session Models
MS POCO
0.65 + 0.01
0.55 = 0.01
0.32 a 0.02
0.13 a 0.05
MSP
0.57 0.00
0.51 0.00
0.35 a 0.00
0.40 0.00
MS MS_NL NLinear
0.32 + 0.00
0.35 = 0.00
0.19 + 0.00
0.30 + 0.00
MS LS__etten__ PLRNN
Latent_
0.24 0.01
0.21 0.00
0.15 +.0r
0.19 0.01
MS TexFilter
0.60 + 0.01
0.51 = 0.01
0.41 + 0.01
0.42 = 0.01
MS NetFormer
0.31 1001
0.33 0.00
0.11 0.00
0.23 0.00
MS MS. Transformer
-0.12 +
0.01 0.01
-1.59 0.10
-1.07 +0002
15

---

Table S9: Fine-tuning POCO embeddings enables rapid adaptation. Test performance of fine- tuning a pre-trained POCO (Pre-POCO) versus training from scratch, and also compared pre-trained MLP and other ablation models. In particular, TMLP denotes an MI.P conditioned by a univariate Transformer, and UMLP denotes an MLP conditioned by unit embedding (see Ablation Study section and Appendix for details). For POCO, we compared full fine-tuning, embedding-one joint MLP+ MeP+embedding tuning. Notably, embedding-on tuning achieved performance comparable to full fine-tuning. Reported values are means with 95% confidence intervals over 3 seeds.
tuning, and
Zebrafish, 512 PCs
Mice
Model
Deisseroth
Ahrens
Pre-POCO (full finetune)
0.532 =0.014
0.424 =0.017
0.406 #0.001
Pre-POCO (embedding only)
0.539 20.008
0.392 20.062
0.405 10.001
Pre-POCO (unit embedding MLP)
0.534 20.009
0.412 20.015
0,405 20.000
Pre-TMLP
0,470 20.002
0,423 20.017
0,400 20.000
Pre-MLP (large)
0.470 #0.006
0.432 #0.007
0,401 40.000
Pre-UMLP (large, full finetune)
0.495 #0.003
0.427 20,013
0.404 40,000
Pre-UMLP (large, embedding only)
0.499 #0.001
0.400 #0.007
0,404 40.000
POCO
0,497 20.019
0,424 20,020
0.404 10.001
NLinear
0.153 #0.001
0. 097 #0.000
0.327 #0.002
MLP
0.386 #0.003
0.349 #0.005
0.395 +0.000
Table S10: Pre-training on mismatched datasets hurts performance. POCO models pre-trained on one zebrafish dataset and fine-tuned on another perform worse than models trained from scratch on the target dataset. Even joint pre-training on both datasets underperforms single-dataset training. These results highlight the importance of within-distribution pre-training. 95% confidence intervals estimated with 3 seeds.
Zebrafish, 512 PCs
Mice
Model
Deisseroth
Ahrens
Pre-POCO(Ahrem)
0.464 40.005
0.392 20.062
0.290 40.014
(reessoO(Deisserooth)
0.539 20.005
0.349
#0,031
0.338
20.005
Pre-POCO(Both Datasets) POCO
0.517 20.015
0.410 20.015
0.304 20.007
0.497 20.019
0.424 #0.020
0.398 #0.014
16

---

Fish (Ahrens)
Fish (Ahrens), Low-pass Filtered
0.45
0.65
Validation Score
0.40
Validation Score
0.60
0.55
0.35
NLinear TexFilter
0.50
NLinear
TexFiiter
0.45
0.35
0.20
10
11
9
10
(traniign recordi dinng length)
log2(training recording length)
Mice
Fish (Delsseroth)
0.425
0.400
Validation Score
0.22
0.375
POCO
NLinear
0.20
NLinear TexFilter
0.350
TexFilter
0.18
0.325
0.16
0.300
0.14
0.275
10
12
14
9
10
11
log2(training recording lengthn
log2ltraining recording length)
Worm (Flavell)
Worm (Flavell), Law pass Filtered
Validation Score
validation Score
POCO NLinear TexFilter
0.2
POCO
NLinear
0.0
0.0
TexFilter
0.1
-0.2
0.2
-0.4
8.0
8.5
9.0
9.5 10.0 10.5
8.0
8.5
9.0
9.5
10.0
10.5
log2(training recording length)
log2(training recording length)
Figure S7: Longer training recordings improve POCO performance across datasets. Prediction score increases with log-scaled training duration. Low-pass filtering further enhances gains some datasets. Error bars show SEM across 2 seeds.
17

---

Mice lowpass (12 Sessions)
Fish (Delsseroth) (19 Sessions)
POCO
0.250
POCO
Prediction Score
0.54
TexFilter
Prediction Score
TexFilter
0.245
0.53
0.240
0.51
0.235
0.50
0.230
12
4
2
19
10
3
2
Number of splits
Number of splits
Worm (Flavell) Sessions)
Worm (Flavell) lowpass (40 Sessions)
0.4
Prediction Score
Prediction Score
0.20
0.3
0.2
0.15
TexFilter
0.1
0.10
POCO
TexFilter
0.0
Number of splits
Number splits
Fish (Ahrens) PC (15 Sessions)
Fish (Ahrens) lowpass PC (15 Sessions)
0.44
POCO
Prediction Score
TexFilter
0.42
Prediction
0.40
POCO
TexFilter
0.38
0.36
15
5
15
5
Number of splits
Number of splits
Figure S8: Multi-session POCO benefits from training on more sessions. Prediction score improves sessions aggregated. Results are shown for both raw and low-pass filtered datasets. Error bars show SEM across seeds.
18

---

Und Enrbeedin HUMAN
Unit Oribddding IJWAN
Und Eribedting (UMA)
Unt Driowdd (UWAT)
490
499
-99
209
130
139
37%
Emoedding (UMAP)
UntErbbadiigg (UHAA)
Smoodd (UM59)
ng (UMAP)
335
0.53
Dv5
Linkt Emreedding (UMA]]
Unit E'm Emmddiing (UMAP)
Unit Embedding (UBAPP
Unit Embedding (UMAP)
12-
HQ
10
Ban
Dml
Dim
Figure S9: POCO unit embeddings reflect meaningful structure across sessions. UMAP projec- tions of unit embeddings from four sessions across zebrafish (Deisseroth and Ahrens) and mouse datasets. Embedding structure is consistent within datasets but varies across sessions.
19

---

3.4
Nomeleee Budidea- Dillnce
Nor lest Ccaine Siill
Normalzed Eudidea- Critance
Nor narma bad Cesine Siilarty
18
Homalzes Tucidaa tuctoen-Cieess Cienes
Norma bed Coeree Snivly
3.4
Nomaizes Eucidean
bed Cedine Sinvity
Figure S10: Unit embeddings cluster by brain region across runs and species. Normalized cosine similarity and Euclidean distance matrices computed on unit embeddings across zebrafish and mouse sessions. As in Figure DC, each row is normalized to [0, 1]. From top to bottom are 4 different runs of POCO.
20

---

1.0
POCO
0.8
MLP
Prediction Score
NLinear
0.6
Latent PLRNN
TexFilter
NetFormer
0.4
AR, Transformer
DLinear
TCN
0.2
TSMixer
0.0
150
300
Dataset
Figure S11: Baseline models perform well on simulated dynamics. On single-session synthetic data, PLRNN and TSMixer outperform POCO, unlike real data. Each point represents a random network instance. Error bars show SEM across seeds.
21

---

0,80
0.93
19403ion Laga
Voltdatoon locs
POCO (hill fineuune) POCO (umbudding only)
00.76
Pre Pre-RCCO rineuare
Po-P (ambudding only) (lare))
0.90
(arga)
PDCD
0.89
HLP
0.72
MLF
50
150
50
Training Stop
3.0
solidaaton Locs
2.5-
Pre-POCO (lul finaturn)
Pre-POCO [fuil firetunel
2.0
Pre POCO (embedding only) Pre-MLP (erge)
Pre POCO lembedtiing only)
Pre-MLP (lerge)
FOCO
FO:C0
1.5
50
100
150
200
50
1000
150
200
Thiining Step
Thiining Step
14
4.0
13163tion 1255
varostton lass
3.0
(uul finetune) Pre-POCO (umbedding only)
20-
fireture)
2.5-
Pre-POCO jumbedding only)
Pro MLP (arge)
Pro MLP (larget
2.0
PDCD
POCO
1.5
MLP
MLP
100
100
Triniing Stop
Training Stop
40-
80
28-
Lost
75-
Vallsatton
36-
PaeO Hull fineture) POCO lembedding only) (lrge))
PrePaCO fial finetare)
655
POCO Jembedding only)
(large)
POCO
60
32-
MLP
55-
MLP
50
1.00
200
50
100
150
Thaining
Thairing
36_
26-
Lateatoniets
valegttonons 5
32-
34
30
POCO Mul finetunc) Pre-POCO jembedding only)
Fre POCO Hull fincturel
22-
28
Pre-POCO jambedding only)
20-
PDCD
Pro MLP (arge)
26
PDCO
Pro MLP (larget
24
50
1.0
150
200
50
300
150
200
Training Step
Training Step
Figure S12: Fine-tuned POCO rapidly adapts to new sessions with minimal updates. Validation loss over 200 training steps for various fine-tuning strategies across 10 sessions. The top row contains the results of 2 sessions in the mice dataset. The second and third rows contain results of 4 sessions for Deisseroth's 's zebrafish dataset, where we are predicting the first 512 PCs. The last rows contain results of 4 sessions for Ahren's 's zebrafish dataset, where we are also predicting the first 512 PCs. Error shades: SEM across seeds.
22

---

Compars flter Eataset Werna Canmer
Concare Fitora Dataner: Worms (ftavel)
0.6
5.6
FOCO
0.4-
Nihsa
FL
NetRomer
Netlormer
62
MUBAR Turstarme
MulMAn Thrtsfor
8.0
5.0
Type
Fil Tape
Compars Fibert Datnent:
Compane Ribertorr Sacaset
0.6
5.5
TOCO
FoCo
MiP
0.4
NLanar
2.4
62-
M.BSAR Taratorme
Mi MuBin Thrnffor
6.0
0.0
Hax Tape
+Ro 1yp0
Fiherr Datacet (ohren))
Fibers (Abrenct
8.6-
0.6-
PCCD
FoCo
HLF
sur
0.4-
T
62
12
MumtR, Tareforme
6.0
0.0
File Trpe
Fiee Typa
Figure S13: Low-pass filtering improves POCO forecasting in most datasets. Prediction score under different filtering regimes (none, low-pass, band-pass) across datasets. Cutoff frequency for low-pass filter is 0.1 x f., where Sa the sampling frequency. A band-pass filter additionally removes frequency components lower than 10 x fa. Error bars: SEM over seeds.
MAE for C
MAE for C 256
0.028
0.0325
0.0300
0.026
0.0275
0.024
MAE
MAE
0.0250
linear
0.022
linear
0.0225
unet
0.0200
0.020
tsmixer
poco
poco
0.0175
0
10
15
20
25
30
0
10
15
20
25
30
Steps ahead
Steps ahead
Figure $14: POCO matches or outperforms volumetric models on Zapbench at short context. Mean absolute eтror over 32 prediction steps on Zapbench with C' = and C' 256. POCO matches UNet at short context but underperforms with longer context. Performance data for models other than POCO are from 32)
23

---

Fish (Ahrens)
Fish (Deisseroth)
0,450
0.530
POCO
POCO
0.445
0.525
Prediction Score
Prediction Score
0.440
0.520
0.435
0.515
0.430
0.510
0.425
0.505
0.500
30
10
30
Token Length
Token Length
Mice
Mice PC
0.430
0.390
POCO
POCO
Prediction Score
0.425
Prediction Scare
0.385
0.420
0.380
0.415
0.375
0.410
0.370
10
20
30
40
50
10
20
30
40
50
Token Length
Token Length
Worms (Zimmer)
Worms (Flavell)
0.40
0.30
POCO
POCO
Prediction Score
0.38
Prediction Score
0.20
0.36
0.15
0.32
0.05
0.30
0.00
10
20
30
40
50
10
20
30
40
50
Token Length
Token Length
Figure S15: Token length To influences POCO performance across datasets. Using Te is suboptimal and computationally expensive. Te averaged over 3 seeds.
16 balances performance and cost, Results
24

---

Fish PC (Ahrens)
Fish PC (Deisseroth)
0,450
0.530
POCO
POCO
0.445
0.525
0.440
0.520
Prediction
0.435
Prediction
0.515
0.430
0.510
0.425
0.505
250
500
750
1000 1250 1500
250
500
750
1000 1250
Hidden Size
Hidden Size
Mice
Mice PC
0.430
0.390
POCO
POCO
Prediction Score
0.425
Prediction Scare
0.385
0.420
0.380
0.415
0.375
0.410
0.370
250
500
750
1000 1250 1500
250
500
750
1000 1250 1500
Hidden Size
Hidden Size
Worms (Zimmer)
Worms (Flavell)
0.40
0.30
POCO
POCO
Prediction Score
0.38
Prediction Score
0.20
0.36
0.15
0.32
0.05
0.30
0.00
0
250
500 750 1000 1250 1500
0
250
500
750 1000 1250 1500 Hidden Size
Hidden Size
Figure S16: Embedding size moderately affects POCO performance. Validation performance shown across datasets as a function of embedding dimension d. Default value 128 achieves near-peak performance. SEM across 3 seeds.
25

---

Fish PC (Ahrens)
Fish PC (Delsseroth)
0.530
POCO
POCO
0.445
0.525
Prediction Score
0.440
0.520
Prediction
0.435
0.515
0.430
0.510
0.425
0.505
0.420
0.500
6
8
6
8
Number Layers Mice
Number of Layers
Mice
0.430
0.390
POCO
POCO
Prediction Score
0.425
Prediction Score
0.385
0.420
0.380
0.415
0.375
0.410
0.370
Number of Layers
Number of Layers
Worms (Zimmer)
Worms (Flavell)
0.40
0.30
POCO
POCO
0.25
Prediction Score
Prediction Score
0.36
0.15
0.34
0.10
0.32
0.30
0.00
Number of Layers
Number of Layers
Figure S17: POCO performs robustly across number of encoder layers. Increasing Perceiver-ID layers from to 8 shows minimal gain, suggesting layer suffices for most datasets. Error bars: SEM across 3 seeds.
26

---

Fish PC (Ahrens)
Fish (Deisseroth)
0,450
0.530
POCO
POCO
0.445
0.525
Prediction Score
0.440
0.520
Prediction
0.430
0.510
0.425
0.505
0.420
0.500
2.5
5.0
7.5
10.0 12.5 15.0
2.5
5.0
7.5 10.0 12.5 15.0
Latents
Number Latents
Mice
Mice PC
0.430
0.390
POCO
POCO
Prediction Score
0.425
Prediction Score
0.385
0.420
0.380
0.415
0.375
0.410
0.370
2.5
5.0
7.5 10.0 12.5 15.0 Number of Latents
2.5
5.0
7.5 10.0 12.5 15.0 Number of Latents
Worms (Zimmer)
Worms (Flavell)
0.40
0.30
POCO
POCO
Prediction Score
Prediction Score
0.20
0.36
0.15
0.34
0.32
0.05
0.30
0.00
2.5
5.0
7.5
10.0
12.5
15.0
2.5
5.0
7.5
10.0 12.5 15.0
Number of Latents
Number of Latents
Figure S18: Number latents affects POCO performance marginally. Validation score shown for different latent counts in the encoder. Performance saturates near 8 latents. SEM across 3 seeds.
27

---

Fish PC (Ahrens)
Fish PC (Deisseroth)
0.45
0.5
Validation Score
0.40
Validation Score
POCO
POCO
NLinear
0.4
NLinear
TexFilter
TexFilter
0.30
0.3
0.25
0.20
-4.0
-3.5
-3,0
-2.
-4.0
-3.5
-3.0
-2.5
log10(Leaming Rate)
logi O(Leaming Rate)
Mice
Mice PC
0.42
0.38
Validation Scote
Validation Score
0.36
0.40
POCO
POCO
NLinear TexFilter
0.34
NLinear
0.38
0.32
0.36
0.30
0.28
-2.5
-2.5
log10(Leaming Rate)
log1 logamnng Rate)
Worms (Flavell)
Worms (Zimmer)
0.26
0.38
Validation Score
0.24
Valldation Score
0.36
0.22
POCO
0.34
POCO
NLinear
TexFilter
0.32
TexFiiter
0.30
0.18
0.28
0.16
0.26
-3.5
-3.0
-2.5
logaO(L Rate)
log10(Leaming Rate)
Figure S19: POCO is robust to range of learning rates. Validation scores for different learning rates across datasets. Best performance usually occurs near 0.0003. SEM across 3 seeds.
28

---

Fish (Ahrens)
Fish (Delsseroth)
0.5
Validation Score
0.40
validation Score
0.35
NLinear
0.4
NLinear
TexFilter
TexFiiter
0.30
0.2
0.20
-5
-4
-3
-2
-1
loglo(Weight Decayl Mice
log1 Weeggtt Decay)
Mice PC
0.42
0.38
Validation Score
Validation Score
0.36
0.40
POCO
NLinear
0.34
NLinear TexFilter
0.38
TexFilter
0.32
0.36
0.34
0.28
-2
-4
-2
log10(Weight Decayl
log10(Welght Decay)
Worms (Flavell)
Worms (Zimmer)
Validation Scote
0.25
vanaauoon score
POCO
0.325
POCO
0.24
NLinear TexFilter
Niinear
ter
0.23
0.275
0.22
logl0(Weight Decay)
log10(Weight Decay)
Figure S20: POCO is relatively insensitive to weight decay settings. Prediction score across datasets with varied weight decay values. Performance stable over several orders of magnitude. SEM across seeds.
29

---

B Datasets
B.1
Dataset Details
Here we summarize the datasets used in this work. Additional details on the datasets can be found in the original dataset publications.
Zebrafish (Deisseroth). Parts of the dataset are published in [T. Larval zebrafish expressing nuclear-localizer GCaMP6s were used for two-photon calcium imaging. Fish were head-fixed for whole-brain imaging while retaining the ability to perform tail movements. Neural activity was recorded at about Hz. yielding volumetric data from 8,000-22,000 neurons per fish. The imaging data were motion-correcter converted into AF/F. ROIs were assigned brain region labels based anatomical landmarks.
segmented into nuclei, and fluorescence traces were extracted and
Experiments were conducted across four cohorts. The Spontaneous cohort was imaged during natural, unstimulated behavior. The Shocked cohort received randomly timed mild electric shocks during imaging. The Reshocked cohort was pre-exposed to a behavioral challenge before imaging, then imaged under the same shock protocol. The Ketamine cohort received ketamine exposure prior to shock delivery during imaging. There are 5 fish for each cohort, except that the Ketamine cohort has 4 fish. These conditions allowed us to examine brain-wide neural dynamics under varying levels of behavioral and pharmacological perturbation.
Zebrafish (Ahrens). The dataset is published in [I]] A light-sheet imaging setup is used to record whole-brain neural activity from larval zebrafish expressing nuclearloccalzedd GCaMP6f. Imaging was performed at ~ 2.1 Hz for ~ 50 minutes across 18 fish, resulting in ~ 6, 800 time points and 80, 000 segmented neurons per animal. Subject 8, 9, and 11 are excluded due to file incompatibility issues, thus, only 155 fish are used. During imaging, fish were exposed to battery of visual stimuli, including phototaxis cues, moving gratings (optomotor response), looming dises (escape), dark flashes, and spontaneous illumination blocks. Although behavioral data were also collected, we used only the neural activity traces for modeling, and no stimulus or behavioral labels were provided to the model.
Mice (Harvey). The dataset is partly described in 44 and the mesoscopic calcium imaging technique is detailed in [3]. It includes multi-session two-photon calcium imaging recordings from layer 2/3 neurons in four cortical regions: primary visual cortex (VI), secondary motor cortex (M2), posterior parietal cortex (PPC), and retrosplenial cortex (RSC). Mice expressing GCaMP6s were head-fixed and allowed to run spontaneously on an air-supported spherical treadmill in complete darkness. Imaging was performed using a large field-of-viewy two- two-photon microscope, capturing neural activity at about 5.4 Hz from two planes per region. Data were collected across multiple sessions and animals, with each session lasting 45-60 minutes. Four mice were recorded, with 1, 5, 4, and 2 sessions respectively. For the first two mice, all four regions were recorded. For the third mouse, two sessions included simultancous recordings of PPC, RSC, and V1, while the other two sessions included PPC, RSC, and M2. For the fourth mouse, only PPC, RSC, and VI were recorded. Motion correction and ROI extraction were performed using standard pipelines. Note that no behavioral or anatomical labels were provided to the model.
C. elegans (Zimmer). The dataset is described in 23 Code for data loading and preprocessing is partly based on (25) This dataset contains whole-brain calcium imaging recordings from C. elegans, using a pan-neuronally expressed nuclear calcium indicator GCamp5K. Neural activity was recorded from 5 animals at 2.85 volumes per second for minutes per animal. The imaging volume covered the head ganglia, including most sensory neurons, interneurons, all head motor neurons, and the anterior ventral cord motor neurons. Each recording included 107-131 neurons, with most active neurons identifiable by cell class. Animals were recorded either under constant oxygen during alternating oxygen stimuli. Only neural activity traces were used in our experiments; stimulus and cell identity labels were not provided to the model.
C. elegans (Flavell). The dataset is published in 4.. This dataset includes brain-wide calcium imaging recordings from freely moving C. elegans using a strain expressing nuclear- localized GCaMP7f and mNeetune2.55 in all neurons. Recordings were conducted on a custom dual-light-path microscope with closed-loop tracking to maintain the worm's head within the imaging field. Neural signals were extracted using automated segmentation and tracking based on 3D U-Net and registration pipelines. We used only the 40 sessions with NeuroPAL neuron identity labels-2 -21 recorded under baseline
30

---

conditions and 19 with noxious heat stimulus that induced a persistent behavioral state change. Only the neural activity traces were used in our experiments; stimulus timing and neuron identity labels were not provided to the model.
Zaphench. The benchmark is described in 22 This dataset consists of whole-brain calcium imaging from single head-fixed larval zebrafish recorded during fictive behavior in a virtual reality environment. Over the course of a two-hour session, the fish was exposed nine structured visual stimulus conditions designed to elicit range of sensorimotor responses. Neural activity was recorded at cellular resolution using light-sheet fluorescence microscopy, resulting in a 4D volumetric dataset spanning 71,721 segmented neurons. Postprocessing included motion correction, custom elastic alignment, and manual neuron segmentation using Flood- Food--illingg Network. Only the extracted per-neuron calcium traces were used in our experiments; stimulus information was not provided to the model.
B.2 Data Processing
For Zapbench, we follow the code provided with the benchmark to directly load the processed data 32.. For other datasets, we begin by z-scoring the raw fluorescence values for each neuron so that each calcium trace has zero mcan and unit variance. This normalization ensures that all neurons cqually weighted in the model an active neuron with a high baseline activity contributes similarly to less active one. For Deisscroth's 's zebrafish dataset, where raw calcium traces are unavailable, we instead apply z-scoring to the denoised activity extracted using constrained nonnegative matrix factorization (CNMF) [41]. For experiments involving frequency filtering, we apply phrse fourth-order Butterworth filter. The default low-pass filter removes frequency components above 0.1 fs. where fa is the sampling frequency. A band-pass filter additionally removes components below 5 x 10-3 3 x fa. After filtering, we optionally reduce dimensionality using principal component analysis (PCA), applied to the z-scored (and optionally filtered) neural activity matrix. The magnitudes of the resulting principal components are preserved, meaning that dominant components naturally carry more weight in the loss function.
B.3 Data Partitioning
We first divide each session into segments of 1,000 time steps. Each segment is then split into training, validation, and test partitions using 3:1:1 ratio. The final segment of a session may contain more than 1, 1,000 steps to maximize use of the available data. To construct the training set, we extract all possible consecutive subsequences of length C + P (64 by default) from the training partition using sliding window with stride 1. For example, if a session contains 2,500 steps, the training partitions would cover steps [1, 600) and [1001, 1900), and the training subsequences would include X1:++++P X ... X600-C- -100---++1-00 P+1000 and similarly for the second partition. Validation and test sets are constructed in the same way. However, because evaluation is computationally expensive for single-cell resolution zebrafish datasets, we use larger stride when extracting subsequences from the validation set: stride 32 for Ahrens' dataset and stride 8 for Deisseroth's 's dataset. Note that striding is applied only to the validation set in the context of single-cell level prediction.
For Zapbench, we instead use the provided tools load the dataset [32]. One notable difference is that in Zapbench, the dataset is first divided by the stimulus condition then divide each condition into train, validation and test sets.
C
Additional Model Details
C.1
POCO
In POCO, we use Rotary Position Embeddings (RoPE) in all attention layers 45 In RoPE, timestamp t assigned to each query token qi and key token k,, and the attention score is computed as
aij = softmax ((R(t,) 9T) (t,)))
(8)
where R(() is a time-dependent rotation matrix. We follow POYO 5 to construct R()., defined as T
a composition of 2 x rotation matrices with periods T; logarithmically spaced between Tmin and
Toax. In our experiments, we use Tmin = 10-3 x Tmax and set T Taxx 100 by default.
31

---

We also need to define the timestamps. For the first attention layer.
L Attentiono (Q = Lo; K, V = E) E B.
(9)
we assign timestamps the trace tokens E(i, as li,1 (k- 1)Te, which corresponds to the starting timestep of the token. For the latent tokens Lu(i), we set
---cc j [N NL
1
(10)
so that the timestamps uniformly span the input context length C. Timestamp for latents in later attention layers L, are defined in the same way. For the final layer, the query tokens are given a constant time step L C. Intuitively, the latents can be viewed as encoding the global population state at different moments in the past, and in the final layer, we query how those past states influence the future.
For Zapbench experiments, we modify these parameters to accommodate shorter or longer context lengths: specifically, we use maax 50, Tc for C' = 4, and Tiaxx 400, Tc = 64 for C' 256. The default token length is Tc = 16, resulting in 3 tokens per neuron when C = 48.
For the conditioning layer, we initialize the weights WB, Wy and biases bg, by of the linear mappings (from the final attention layer output to FiLM conditioning parameters) to zero, ensuring that POCO produces flat output at initialization. We follow POYO on additional model details: We use an additional feedforward layer following each attention layer. Residual connections are used attention head for the
for both the attention layers and the feedforward layers that follow. We first and last attention layer, and 8 heads for intermediate self-attention layers. We set the embedding and latent size to d 128; in rotary attention. each dimension of each head is 64. We use a dropout of 0.2 in the feed-forward layers following the attention layers and a dropout of 0.4 on the output of intermediate self-attention layers and the accompanying feedforward layers. We initialize the embeddings from (O, of) with du = 0.02.
C.2 Baselines
NLinear. In NLinear 555 the last step of the input context is subtracted from the input sequence, and the residual is passed through a linear layer to generate predictions (TRC -> PR.)) The last step is then added back to the output. This design is effective in time series forecasting (TSF) tasks it helps address distributional shifts between training and test sets. NLinear is a univariate model-each neuron's prediction depends only on its own history-which also makes it naturally capable of processing multi-session data, as it can handle an arbitrary number of neurons.
DLinear. We adopt the implementation from [55 DLinear first decomposes each time series into trend and a seasonal component. The trend component is obtained by applying a moving average filter with kernel size 2 2[C/4] + 1, and the seasonal component is the residual obtained by subtracting the trend from the original input. Both components are predicted independently using linear projections, and the final output is their sum. Like NLinear, DLinear is also univariate.
MLP. For single-session experiments, we use two-layer MLP with ReLU activation and hidden size 1024. This model is equivalent to POCO without FiLM conditioning. For multi- experiments, we use a 3-layer MLP with hidden size 2048, denoted as MLP L. MLP is also univariate.
PLRNN. We include PLRNN as baseline due to its demonstrated ability to reconstruct chaotic dynamical systems. We use the implementation from 77. The model defines a dynamical system over d d-dimensional latent state vector Z, as:
Z, Az W (Zz +h,
where A E Rdxd is diagonal matrix encoding self-connections, W E Bddd contains off-diagonal weights for inter-unit interactions, and h E Rd is a bias term. The activation function á is ReLU. We set d = 512. In our setup, due to the slow dynamics of calcium traces, we the alternative formulation
z, = a (Az.11 W ((z,1)) + h) + (1 a)
where = 0.05. To generate predictions, we first map the last step from the context to the latent state:
zc
W Winx + bin
32

---

then evolve the latent state for P = 16 steps and map it back to the observation space:
XC+ = Wlcczc+ - bm), te [P],
where W denotes the pseudo-inverse of Win- PLRNN is multivariate, but it only uses the last in time step of the context to generate prediction. To fully utilize the input during training, instead initialize the latent state 21 from X and evolve the latent dynamics model for C + P steps. We apply sparse teacher foreing (STF) [35]. injecting the ground-truth state every four steps reduce the drift from the groundtruth for effective training; STF is disabled during the final P prediction steps to simulate free prediction. For multi-session training, the latent dynamics (PLRNN parameters) are shared across sessions, while the input mapping Win is -ession-specific to capture individual variability.
AR Transformer. We also include an autoregressive Transformer [50] as a baseline to assess the performance of classical autoregressive architectures in neural population modeling. A linear layer (R + R maps each time step's population activity to a d-dimensional embedding, producing C tokens. Here we use d 512. These tokens are passed through 4-layer Transformer with causal attention masks. A final linear projection (I& + R²) maps the output embeddings to next-step predictions. To forecast P 16 steps, the model is applied autoregressively: at each step, the predicted activity is appended and used as input for the next prediction. As with PLRNN, for multi- session training. the Transformer backbone is shared, while the input and output projection layers are session- speecffiee
TSMixer. TSMixer [16] is all-MLP architecture for time series forecasting. It consists of stacked mixer blocks, each comprising a time mixer (a linear projection operating along the time dimension) and feature mixer (an MLP with hidden size 64 that operates across the neuron dimension). The inclusion of the feature mixer makes TSMixer a multivariate model. We use the default hyperparameter settings: two mixer blocks and a dropout rate of 0.1.
Fixfil TexFilter [54] is a recent MLP-based method that incorporates frequency-domaii filtering. It applies context-dependent filter on the franser-rransformed input, allowing the model to selectively enhance or suppress specific frequency components. In addition to the MLP layer, TexFilter uses learnable complex embeddings that modulate the frequency representation. TexFilter is univariate. In our setup, we set the embedding size to 128 and initialize the embeddings from (0),a%) To = 0.02, with dropout rate of 0.3.
TCN. ModernTCN 33 employs modernized convolutional blocks for time series modeling. The input is first divided into patches of size P = 4. which are embedded into D 64-dimensional vectors. These embeddings are processed using convolutional blocks that capture both temporal and cross-neuron dependencies, followed by a linear readout for prediction. This makes ModernTNN multivariate model. We follow the original work's 's hyperparameters for forecasting, except for the convolutional kernel sizes, which are adjusted to 17 (large) and 5 (small) to better suit shorter context window.
Netformer. Netformer 31 is recent approach for modeling dynamical connectivity in neural population activity. It embeds each neuron's past activity trace and uses an attention layer compute interaction weights A across neurons. The next-step prediction is given by:
X Ax, Xt-
Although the original work only evaluated next-step prediction, we extend it to multi-step forecasting by recursively applying the same update. However, we observed instability when training on long horizons and therefore applied a softmax to the attention weights to stabilize learning. Furthermore, to further improve stability, we also applied an instance normalization layer on the input and later unnormalized the prediction as in recent TSF works
(54 47 As in the original Netformer, we apply layer normalization along the time dimension, and use embedding size of 30. Netformer is multivariate model but naturally compatible with multi-session training, as the attention mechanism is independent of the number of neurons.
33

---

C.3
Ablation Study
In the ablation study, we considered several alternative architectures. First, directly use the POYO model to generate 16-steps prediction, we use a linear projection on top of the output of the last layer
L Attention I (Q - U, -I+1) RRN
(11)
RCEPP- oua T.+2 + bout:
where W out E ExP
Another alternative we tested is replacing the population encoder with univariate Transformer, which takes the calcium trace of a single neuron instead of encoding the whole population. Specifically, for each neuron i, we still use Tc 16 steps to form one token, but the token embedding no longer involves the unit and session embedding:
E(k) = Wx W TTemee + b.
(12)
We then apply a standard 1-layer transformer encoder used in recent time-series forecasting work 477. Note that the Transformer is applied separately to each neuron. We concatenate hidden states of the last encoder layer, resulting in a dC/ /Tc 128 3-dimensional vector for each neuron. Finally, we generate 16-step prediction by applying linear projection on the concatenated hidden state.
Finally, we tested whether POCO's performance could be achieved without the population encoder by relying solely on the unit embedding design. Specifically, we computed the unit embeddings as in POCO, but instead of passing them through the encoder, we directly used linear layers to generate y and B, which were then used to condition the MLP as in Equation 3. To ensure fair comparison, used a larger 3-layer MLP with hidden size 2048 (matching MLP L), giving it more parameters than POCO, and applied conditioning the last MLP layer.
D Experiments
D.1 Training
For multi-session training, we train each model for 10a steps. Single-session models are instead trained for 10 steps considering the smaller training set size. For all models, we used Adam 330 optimizer with learning rate 0.0003 and weight decay 10-4, 4, At each training step, a batch of 64 sequences is sampled from the training sets of all sessions. For single-cell prediction in larval zebrafish, we reduce the batch size to 8 for Deisseroth's 's zebrafish dataset and to 4 for Ahrens' zebrafish dataset due to memory constraints. Similarly, for the MLP conditioned by univariate Transformer, we reduce the batch size to 16. Gradient clipping is applied to limit the gradient norm to 5. Most models are trained using mean squared error (MSE) loss averaged over the P prediction steps. However, for PLRNN, the autoregressive Transformer, and Netformer-model. that generate one-step predictions- the loss is computed across all P steps. Models are evaluated on the validation set every 100 training steps, and we report test-set performance using the checkpoint with the lowest validation loss.
D.2 Simulation
Code for simulation is adapted from [40]. We generate synthetic neural data from chaotic RNN with a tanh nonlincarity, governed by the following dynamical equation:
dh
r = thnh(h),
T
- gJr + V2TO2G. h(0) ~ 1[,1, d
where h is the internal (pre-activation) state, r = tanh(h) is the firing rate, J is recurrent weight matrix, g = 2.0 is gain factor, & are N independent Gaussian white noise processes with zero mean and unit variance, = 0.1 controls the noise variance, and r = 0.1 is the time constant. We discretize the dynamics using Euler's method with time step At 0.01:
At (-h, gJ tanh(hi)) + 202
At
h hh +
T E.r rt tanh(h.),
34

---

where ç, (O,, at each step. We z-score the firing rates rt, and to simulate the slower sampling rate calcium imaging, we average r, over every f 5 time steps. The simulation runs from t to 1 4096 x /r, producing 4096 time steps of synthetic data comparable in length to real neural recordings.
D.3 Finetuning
For pretraining and finetuning experiments, we partition each dataset and use approximately 80% of the sessions for pretraining. Specifically, for Deisseroth's 's zebrafish dataset, we reserve the last session from each cohort for finetuning, yielding 15 sessions for pretraining. For Ahrens' zebrafish dataset, finetune the last 4 sessions and pretrain the remaining 11. For the mice dataset from Harvey et al., we finetune using the sessions from the last mouse. All models are finetuned for 2,000 steps and evaluated every 20 steps.
D.4 Unit Embedding
Although multi-session POCO learns a shared unit embedding space across sessions, the additional session embedding may introduce cession-specine offsets. To avoid this confound, we analyze and visualize unit embeddings separately for each session. Let (i) denote embedding of neuron i, and 5(²) the set of neurons in region u from session j. The cosine similarity between regions and computed as:
U(ia)TU(i.)
C(u, u)
Σ
Σ
for u +
E, |St |
ieE ies!
U(L)) (Il
Clu, u)
E
E
E, (
Ru pfin
To highlight relative similarity patterns, we compute a row-wise normalized similarity:
Clu,v) Clu, v) min. C(u, v')
max,' C(u, 0') min, C(u,v")
This normalized similarity more clearly reveals which brain regions are functionally closer further from a given region u. The same analysis can be applied using other distance metrics, such as Euclidean distance.
D.5 Multi-Dataset Training
To support multi-dataset training, we assign a separate dataloader each dataset. At each training step, we sample one batch from each dataset, compute the loss independently, and sum the losses before performing a single backward pass. To help POCO handle cross-dataset differences, we add dataset-specific embedding to each input token:
Eli, k)) Wx rr-r-TTre i
UnitEmbed(i,,) j. u) SessionEmbed(j.uu u) + DatasetEmbed uu),
where u E [D] is the dataset index and D is the total number of datasets.
D.6 Experiments on Zapbench
For Zapbench, follow the provided script for dataloading and testing 32). We train POCO for 25 epochs with a batch size of 8, validating after every epoch. We report test-set performance using the checkpoint with the lowest validation loss. Following the benchmark protocol, error (MAE) as the loss function. The TAXIS condition is excluded during training, and performance evaluated on the remaining stimulus conditions.
absolute
E
Additional Discussion
While POCO generally outperforms baseline models, its gains are smaller in certain settings. We believe these variations reflect differences in dataset properties such species, dignal-to-noise ratio
35

---

(SNR), recording duration, and population size. Datasets with lower SNR impose an effective noise ceiling on performance, reducing the apparent gap between models since no method can explain the irreducible variance. In such cases, simpler models like MLPs may appear relatively stronger as they are less prone to overfitting.
For example, the C. elegans datasets involve small populations and slow, graded dynamics, which likely limit the benefits of population-level conditioning. The performance drop after low-pass filtering may occur because C. elegans activity already evolves slowly-further filtering can remove subtle but informative temporal variations. Similarly, the smaller gain of POCO in single-neuron zebrafish data compared to PCA-reduced activity likely reflects higher noise levels and sparser activations at the single-cell level.
Another possible factor is the calcium indicator itself, as fluorescence dynamics differ across indicators (e.g., GCaMP6s, 6f, 7f). In the current implementation, deliberately chose not explicitly model calcium kinetics or deconvolved spikes, aiming to maintain flexibility across datasets with different indicators and preprocessing pipelines. Future work could explore incorporating indicator-speciffe dynamics or explicit deconvolution to improve generalization and interpretability across modalities.
More broadly, there are theoretical limits to predictability in neural forecasting. Even with large datasets and expressive models, many factors- such as stochastic spiking, unobserved inputs, and measurement noise introduce intrinsic uncertainty in future activity. Under MSE objective. models tend to output the average of possible futures, which can flatten predictions in regions of high uncertainty. Modeling uncertainty explicitly, for instance via probabilistic or diffusion-based forecasting approaches, could help capture this variability.
Finally, we note that in the time-series forecasting literature, new architectures rarely achieve top performance across all datasets [33 16 54], suggesting that variability in model ranking is a broader characteristic of modeling complex, noisy systems rather than a limitation unique to POCO. We hope that the remaining performance gaps observed here will motivate future research on how dataset properties and noise structures influence the limits of neural forecasting.
F Compute Resources
All models are trained on NVIDIA A100 or H100 GPUs paired with AMD EPYC 9454 CPUs. Only single CPU thread is used throughout training. Data preprocessing for all datasets takes less than one CPU day in total, and generating all synthetic data requires under two CPU days.
For multi-session prediction of the first 512 principal components (PCs) from Deisseroth's 's zebrafish dataset, training POCO for 10* steps takes less than 30 minutes, including data loading and validation time. Despite using high-end GPU, this training setup consumes under 4GB of GPU memory. Computational cost increases for datasets with more neurons: for single-cell prediction on Ahrens' zebrafish dataset, training consumes up to 48GB of GPU memory and completes in under one hour.
While training single model is fast, some experiments involve training many models. For example, training single-session model for each session in each dataset across 4 random seeds involves training over 300 models in total. The estimated total GPU time across all experiments approximately 1,500 hours.
The same hardware setup (H100 GPU) is used to measure finetuning and inference speed. When finetuning POCO on the first 512 PCs of Deisseroth's 's dataset, we find that adapting the embedding for 200 training steps takes less than 15 seconds, and forecasting a single sequence takes only 3.5 milliseconds.
36

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: All claims and stated contributions are supported by experiments.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Limitations of the current work are discussed at the end of the paper, in the Discussion section.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA]
Justification: The paper does not include theoretical results.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: Details on model and experiments can be found in the Appendix. Codes and instructions on processing public datasets used the paper will made available on Github.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification: We use mix of publicly available and collaboration-only datasets. Three datasets (Ahrens zebrafish, Zimmer and Flavell C. elegans) are publicly available; the remaining two (Deisseroth zebrafish and Harvey mouse) are available upon request or under collaboration agreements. Code to reproduce all results and instructions for accessing public datasets will be released on GitHub.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Details of experiments are specified in the Appendix.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: We included error bars for all applicable plots, where mostly standard eπror the mean (SEM) as stated the caption of each figure. For tables, we include 95% confidence intervals, but the Normality of emrrrs is not verified.
8. Experiments compute resources
37

---

For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: We provide the amount of compute used and the approximate run time for all experiments in the Appendix.
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: We make sure the work conforms with the NeurIPS Code of Ethics.
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: While this work focuses on modeling spontaneous neural activity in animal datasets, the modeling approach introduced here here-particularly POCO's ability to generalize across individuals and support rapid adaptation- brain-computer interfaces, closed-loop control systems, and real-time neuroprosthetics. As such applications emerge, considerations around safety, robustness, and clinically useful deployment will become increasingly relevant.
inform future neurotechnologies such
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification: The paper does not release data or models with high risks of misuse.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We use existing datasets and models with proper citation. Public datasets are cited in the manuscript with corresponding references. Where licenses are available (e.g.. Creative Commons), we follow them; for datasets not publicly licensed, access was obtained via collaboration and all terms of use have been respected.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: Key usage and functions will be documented in the released code.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects. Institutional review board (IRB) approvals or equivalent for research with human subjects
38

---

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, originality of the research, declaration is not required.
Answer: [NA]
Justification: Core method development does not involve LLMs; they are only used as coding and writing aid.
39


Reviews:
[{"review_id": "sWHgZ7IcYJ", "reviewer": "Reviewer_9twB", "summary": "The paper proposes a neural forecasting model called POCO that includes a forecaster and an encoder based on POYO to predict future calcium imaging activity based on past calcium imaging activity. The model is tailored to spontaneous activity and is evaluated on multiple datasets recordings from different species, where it beats a wide range of baselines. The authors also show that the learned embeddings reveal brain regions and investigate factors influencing performance.", "questions": "These are discussion points only. I would not expect the author to actually assess any of the below points in their rebuttal.\n\n1. The forecaster output is 1-dimensional, taking into account temporal patterns of individual neurons only. The authors point to previous work that has shown this structure to perform well. But the focus of that work was different. Would the authors expect improvements for higher dimensional forecasters?\n\n2. While the models are evaluated using a range of performance metrics (MSE, MAE, Prediction Score), the predictor $f$ is optimized to minimize the MSE (Eq. 1). Did the authors consider different objectives? This might be particularly important for generalizing the approach to other recording types, as mentioned in the Discussion.\n\n3. In the model, calcium imaging dynamics are not explicitly separated from underlying neuron dynamics. Was this a deliberate design decision?", "limitations": "The authors discussed some limitations including problems with session differences and generalization, as well as recording types beyond calcium imaging. In my view, this covers the essential points.", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 4, "originality": 3, "strengths_and_weaknesses": "Quality:\nThe paper introduces a novel model architecture, which consistently outperforms competing models on spontaneous calcium activity predictions. Comparisons include a large range of baselines, evaluated on several datasets including different species and recording parameters. These comparisons look fair to me. There is also a comparison on synthetic data where the proposed model does not outperform baselines. The authors attribute this to more complex characteristics of natural data but do not explore this in detail. This would have been possible with synthetic data. The authors also find good performance on PCA reduced calcium imaging data further suggesting versatility of the proposed model. Moreover, there is an ablation study, showing the necessity of the different POCO components.\n\nClarity:\nThe manuscript is very well written, motivating and formalizing the problem well. I could not find any mistake in the equations. Some very minor points: For equation lines where two equations are separated by a comma, please increase the space after the comma to avoid confusion about the meaning. Also in Fig. 1, squares suggest the same number of elements, i.e. $M=N$ and $d=N*3$, which I believe is not the case. It would be less confusing to instead use rectangles. Moreover, exactly the same squares are used, suggesting no changes in parameters.\n\nSignificance:\nThe goal of this work is to develop a neural prediction foundation model that generalizes across subjects. The authors mention utility in terms of benchmark models, closed-loop optogenetic control, and whole-brain multi-animal applications. No doubt, a general neural prediction model has broad applicability in many neuroscience settings.\n\nOriginality:\nThe proposed model adapts POYO as a model component but goes beyond that previous model by combining POYO with a univariate forecasting model. Properties of individual neurons, as well as session properties are dealt with using embeddings from a POYO component of the model. Unlike previous approaches, the emphasis of this work is on spontaneous, task-free calcium activity. The closest previous work might be the cited reference [29] which, however, is limited to a single subject, single species analysis. The proposed POCO model does not benefit from multiple species, though (Table 4). So the main benefit is its generalization performance across subjects.", "was_revised": true, "final_justification": "I thank the authors for answering all of my questions. I had minor concerns to begin with and don't have any major doubts, also reading the other reviewer comments. I recommend acceptance.", "rebuttal": "Thanks for your positive feedback and insightful questions! As you suggested, we will increase the space after the comma in equations, change the shapes in model illustration for better clarity\n\n> The forecaster output is 1-dimensional, taking into account temporal patterns of individual neurons only. The authors point to previous work that has shown this structure to perform well. But the focus of that work was different. Would the authors expect improvements for higher dimensional forecasters?\n\nIn the field of time-series forecasting, many univariate models have been shown to perform on par with or even better than multivariate models [1, 2]. However, as you rightly pointed out, these prior methods focus on different domains and were not evaluated on neural data. A key difference between neural recordings and time-series from other domains (e.g., finance, energy) is that neural channels often exhibit stronger causal interactions, potentially due to direct or indirect synaptic connections. For this reason, we expect a multivariate model to outperform simple univariate forecasters in the neural setting. This motivated our use of a population encoder to modulate otherwise simple MLP forecasters, allowing the model to capture the influence of global brain state on individual neurons.\n\nAn alternative approach would be to make the MLP forecaster itself multivariate (e.g., by using the entire population as input). However, a naive implementation of this could significantly increase the number of parameters and make the model more susceptible to overfitting. A promising direction for future work could be to incorporate summary statistics (e.g., average activity of nearby neurons or neurons within the same region) as additional input to the MLP forecaster, striking a better balance between expressivity and generalization.\n\n> While the models are evaluated using a range of performance metrics (MSE, MAE, Prediction Score), the predictor is optimized to minimize the MSE (Eq. 1). Did the authors consider different objectives? This might be particularly important for generalizing the approach to other recording types, as mentioned in the Discussion.\n\nIn this work, we optimized all models using mean squared error (MSE). This choice aligns with prior work in time-series forecasting and offers a stable training objective for multi-step prediction [1, 2]. However, we agree that alternative objectives—such as minimizing MAE or optimizing frequency-domain criteria—may be valuable, especially when adapting models to other types of neural recordings with different signal characteristics (e.g., spiking data or voltage imaging). Exploring alternative loss functions tailored to downstream goals or specific signal modalities (e.g., emphasizing timing accuracy or spectral fidelity) is a promising direction for future work.\n\n> In the model, calcium imaging dynamics are not explicitly separated from underlying neuron dynamics. Was this a deliberate design decision?\n\nThank you for this insightful question. Indeed, we did not explicitly separate calcium indicator dynamics from the underlying neuronal activity. This was a deliberate design choice aimed at keeping the model architecture broadly applicable across datasets with varying sampling rates, indicators (e.g., GCaMP6s, GCaMP6f, GCaMP7f), and preprocessing pipelines. Instead of modeling deconvolved spikes or assuming a fixed calcium convolution kernel, we train directly on the observed fluorescence traces, allowing the model to implicitly learn and adapt to the dynamics present in each dataset.\n\nThat said, incorporating explicit models of calcium kinetics may enhance generalization across recording modalities and possibly mitigate the challenge we found when doing multi-dataset training. This would be an interesting direction for future work, especially for bridging calcium imaging recordings with spiking datasets.\n\n[1] Yi et al. Filternet: Harnessing frequency filters for time series forecasting. [2] Ekambaram et al. Tsmixer: Lightweight mlp-mixer model for multivariate time series forecasting."}, {"review_id": "dEz2l7deDe", "reviewer": "Reviewer_xBNB", "summary": "This paper presents a novel neural forecasting framework that integrates a univariate forecaster with a population encoder, trained jointly across multiple sessions of *spontaneous calcium imaging* data. The authors demonstrate that it achieves state-of-the-art predictive accuracy compared to existing baselines for multiple species.", "questions": "- Prediction Visualization: To better understand the differences among the models, it would be useful to include a visual comparison of the predictions from POCO, MLP, and TexFilter—either by extending Fig. 2C or by adding a supplementary figure. This could help illustrate the qualitative distinctions that are not fully captured by the prediction metrics.\n- Context Length Saturation: I am also puzzled by the observation that POCO’s performance saturates rapidly with increasing context length. Could the authors provide some insight into why this might be happening?\n- Learnable Latents ($L_0$): Could the authors clarify the learnable latent variables $L_0$? Are these latents shared across all time slices and sessions, or are they session-specific or time-varying in any way?", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Overall, I find the paper well-written and easy to follow. The proposed approach is clearly described, and the empirical analysis is fairly comprehensive.\n\nHowever, while the method appears to work, I remain unconvinced about several key design choices. In particular, it is unclear why the authors chose to model population-level influences via feature-wise linear modulation. Could the authors provide more intuition and justification for this architectural decision?\n\nMoreover, the reported performance gains of POCO over the MLP baseline appear limited in some settings. Specifically:\n(a) On the two C. elegans datasets (Table 2), POCO shows only marginal or no improvement.\n(b) On the two Zebrafish datasets (Table 3), the performance advantage of POCO when using PCA-reduced activity diminishes or vanishes when using single-neuron activity.\n\nIt would strengthen the paper to include a more detailed qualitative comparison—e.g., visualizing representative predictions from POCO and MLP—to better illustrate the differences in their predictions and substantiate the proposed architecture's claimed benefits.", "was_revised": true, "final_justification": "The proposed method appears to be effective, as demonstrated by its performance on benchmark datasets (mouse and two zebrafish in the PC forecasting setting).  the mouse and two zebrafish datasets in the PC forecasting setting). The paper is clearly written, and the empirical analysis is fairly comprehensive, though not entirely satisfying. Based on these factors, I recommend acceptance.\nHowever, the performance gains of the proposed method are limited and, in many cases, not consistent. Notably, it often performs comparably to a simple MLP, which raises concerns about the method's overall contribution. Additional experiments and deeper analysis are necessary to better understand the strengths and limitations of the approach and to convincingly demonstrate its advantages. \nOverall, I recommend borderline acceptance.", "rebuttal": "Thanks for your constructive feedback and thoughtful suggestions!\n\n> Prediction Visualization: To better understand the differences among the models, it would be useful to include a visual comparison of the predictions from POCO, MLP, and TexFilter—either by extending Fig. 2C or by adding a supplementary figure. This could help illustrate the qualitative distinctions that are not fully captured by the prediction metrics.\n\nThank you for the suggestion regarding additional visualizations. We do find that these plots help clarify when POCO outperforms other models. While we are currently unable to update the figures due to conference policy, we describe the findings in text below.\n\nIn many trials, we observe that the predictions of POCO, MLP, and TexFilter are highly similar. For instance, at the single-neuron level, when calcium activity fluctuates around the baseline, all models tend to produce relatively flat predictions, and none predict the small fluctuations accurately—likely because these reflect noise rather than meaningful structure. Similarly, for both PC and raw neuron predictions, all models tend to perform similarly when the context exhibits a clear trend (e.g., a slow decay following a recent calcium peak). We believe this is due to the slow dynamics of calcium signals, which all models can exploit.\n\nHowever, when we selectively examine trials and neurons where model predictions differ, POCO often performs better in cases where the post-context activity deviates from the baseline or preceding trend. For example, when calcium activity is flat in the context window but rises or falls afterward, both MLP and TexFilter often produce flat predictions—similar to the copy baseline—whereas POCO often more accurately captures the deviation. This suggests that POCO is more capable of leveraging population-level information to detect subtle changes in neural dynamics, rather than simply extrapolating trends from the individual neuron's past activity.\n\n> Moreover, the reported performance gains of POCO over the MLP baseline appear limited in some settings. Specifically: (a) On the two C. elegans datasets (Table 2), POCO shows only marginal or no improvement. (b) On the two Zebrafish datasets (Table 3), the performance advantage of POCO when using PCA-reduced activity diminishes or vanishes when using single-neuron activity.\n\nThank you for pointing this out. We agree that POCO’s gains are smaller in some settings. As the five datasets come from different labs and use distinct experimental pipelines, the signal-to-noise ratio (SNR) varies across datasets. In general, simpler models may appear to perform relatively better on low-SNR recordings, as they are less prone to overfitting noise.\n\nIn particular:  \n(a) *C. elegans* datasets involve relatively small populations, shorter recordings, and potentially simpler dynamics, all of which can limit the benefit of population-level modeling.  \n(b) For the zebrafish datasets, single-neuron activity is likely noisier than PCA-reduced signals, as PCA effectively filters out small, uncorrelated noise. Moreover, as discussed earlier, models tend to predict similarly when calcium activity fluctuates around baseline. Since neural activity is sparser in single-neuron space than in PC space—and performance is averaged across all neurons—this can numerically reduce the observed performance gap between models.\n\nWhile we believe these are plausible explanations, we acknowledge in the Discussion that a systematic understanding of how dataset properties influence performance remains lacking and is an important direction for future work. Notably, most recent work on neural forecasting and foundation models does not include multi-species datasets [4, 5, 6]. We see it as a strength of our study to evaluate models across diverse datasets, as this exposes both their strengths and limitations—even if it makes the overall results less “clean.”\n\nWe also note that in the time-series forecasting literature, new models rarely achieve top performance across all datasets [1, 2, 3], suggesting this challenge is not unique to our setting, but reflects a broader difficulty of modeling time-series from complex systems.\n\n[1] Luo et al., *ModernTCN: A Modern Pure Convolution Structure for General Time Series Analysis*.  \n[2] Yi et al., *FilterNet: Harnessing Frequency Filters for Time Series Forecasting*.  \n[3] Ekambaram et al., *TSMixer: Lightweight MLP-Mixer Model for Multivariate Time Series Forecasting*.  \n[4] Lueckmann et al., *ZAPBench: A Benchmark for Whole-Brain Activity Prediction in Zebrafish*.  \n[5] Azabou et al., *A Unified, Scalable Framework for Neural Population Decoding*.  \n[6] Zhang et al., *Towards a “Universal Translator” for Neural Dynamics at Single-Cell, Single-Spike Resolution*.\n\n> In particular, it is unclear why the authors chose to model population-level influences via feature-wise linear modulation. Could the authors provide more intuition and justification for this architectural decision?\n\nThank you for the question. Indeed, in neuroscience, population dynamics are often modeled using recurrent architectures such as latent dynamical systems (e.g., Latent PLRNN, which we included in our benchmarks). However, we found empirically that such models are difficult to scale across heterogeneous sessions and large populations, and are also prone to overfitting.\n\nIn contrast, FiLM offers a simple yet expressive mechanism to condition the forecaster by modulating its hidden activations based on the global population state. Recall that $M$ is the hidden layer size in MLP and $P = 16$ is the number of time steps we hope to predict. The final prediction $\\hat{\\mathbf{x}}_{t:t+P}$ can be viewed as a linear combination of $M$ temporal modes $\\mathbf{v}\\_1, \\mathbf{v}\\_2, \\dots, \\mathbf{v}\\_M \\in \\mathbb{R}^{P}$, where each $\\mathbf{v}\\_i$ is a column of the output weight matrix $W\\_{\\text{out}}$. The hidden layer activations $\\mathbf{h} \\in \\mathbb{R}^M$ control the contribution of each mode:\n\n$\\hat{\\mathbf{x}}\\_{t:t+P} = \\mathbf{b}\\_{\\text{out}} + \\sum\\_{i=1}^M \\mathbf{h}\\_i \\mathbf{v}\\_i$\n\nBy using FiLM to condition $\\mathbf{h}$, the population encoder effectively adjusts the contribution of each mode, tailoring each neuron’s forecast based on the broader brain state. Moreover, FiLM is computationally lightweight and scales efficiently to large neural populations since it avoids explicitly modeling pairwise interactions.\n\n> Context Length Saturation: I am also puzzled by the observation that POCO’s performance saturates rapidly with increasing context length. Could the authors provide some insight into why this might be happening?\n\nOne plausible explanation is that much of the predictive information in calcium imaging traces is contained within relatively short temporal windows. From a dynamical systems perspective, the future evolution of a neural population depends on its current state. While calcium activity at a single time step may not fully reflect the underlying neural state—due to the slow decay and integration properties of calcium signals—it is likely that a relatively short history is sufficient to approximate the relevant state information for forecasting.\n\n> Learnable Latents (L0): Could the authors clarify the learnable latent variables L0? Are these latents shared across all time slices and sessions, or are they session-specific or time-varying in any way?\n\nYes, these are shared across all time slices and sessions. In early experiments, we did try making them session-specific by adding a session embedding to L0, but it didn’t seem to result in a clear difference. We will update the manuscript to avoid confusion."}, {"review_id": "VwBiAPKot8", "reviewer": "Reviewer_pkKL", "summary": "This paper proposes an efficient new model, POCO, for predicting future neural activity, by combining single-neuron forecasting with a light-weight model of interaction. This emphasizes applications to longer recordings of freely behaving animals. POCO is evaluated experimental on five calcium imaging datasets of free behavior in three different species where it outperforms all baselines, generalizes to new recordings, and recovers physiologically-relevant structure in its representations.", "questions": "Q1. The sampling rate varies by a factor of 5 across the different datasets, which using a 16 step prediction window would imply that only about 2ms of activity need to be predicted for mouse versus 10ms of activity of zebrafish. Is this being controlled for somehow? If not, why is it unimportant?\n\nQ2. I understand that given resource limitations it's reasonable to do much of the model's evaluations on relatively small datasets. However, it would be very helpful to see a few evaluations on a significantly longer dataset given this is an important part of the gap the paper intends to address.\n\nQ3. Did you compare against models like those of Azabou et al. [1] or Zhang et al. [2]? I am willing to raise my score if there is a convincing case that these models fall short in the paradigm considered here.\n\nQ4. Is the same-sized model used in the multi-species training setup? Conceivably, the larger aggregate dataset could support a larger model.\n\n[1] Azabou et al. \"Multi-task, multi-session neural decoding...\" ICLR 2025\n[2] Zhang et al. \"Towards a 'universal translator'...\" NeurIPS 2024", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths\n\nS1. Foundation models, generative ones in particular, are a fascinating new direction for neuroscience that may dramatically alter the field, and basic research on them is important and timely.\n\nS2. The experimental evaluation is comprehensive, thoughtful, and carefully explained.\n\nS3. The paper is well-written and clear throughout.\n\nWeaknesses\n\nW1. The gap in the related work is not entirely clear to me. The motivation given for the particular model examined in this paper emphasizes longer freely behaving recordings in contrast to short task-focused ones, but is this actually important? Do foundation models trained on the one really fail to generalize to the other? It's plausible, but it's an empirical question that needs investigation. Notably, those foundation models are not compared against in this paper, and the datasets examined here represent only a few seconds of neural activity -- far from long periods of free behavior. Also, why do we need a different architecture for these paradigms?\n\nW2. It would be nice to have a demonstration, even a small one, where forecasting is useful. In general predicting neural activity is an interesting problem but a solution to it should either develop our understanding of the brain or demonstrably improve the performance of an applied neuro technology.\n\nW3. The proposed method is a very basic extension of existing ideas and models which are already being used for closely related tasks.", "was_revised": true, "final_justification": "The core problem addressed in this paper -- forecasting in low-frequency calcium imaging datasets -- is an understudied one, if not particularly interesting, and this paper contributes both an effective model using well-established ideas and a comprehensive evaluation of different methods for this problem. The evaluation is especially rigorous and thorough and in my view is this paper's main contribution.", "rebuttal": "Thanks for your careful and constructive review!\n\n> the datasets examined here represent only a few seconds of neural activity -- far from long periods of free behavior.\n\nFirst, we would like to clarify the timescale of the datasets used in this work. In Table 1, the sampling frequency $f\\_s$ denotes the number of time steps (i.e., imaging frames) per second. Each session thus contains tens of minutes of recording, and 16-step prediction in the Deisseroth zebrafish dataset (with $f\\_s = 1.1$ Hz) corresponds to forecasting approximately 15 seconds into the future.\n\n> It would be very helpful to see a few evaluations on a significantly longer dataset given this is an important part of the gap the paper intends to address.\n\nWe hope the clarification above addresses this concern. In total, the datasets used in our study comprise nearly 50 hours of recordings, which we believe provide a substantial basis for testing and benchmarking forecasting models. That said, as noted in the Discussion, scaling and refining our approach for even larger datasets remains an important direction for future work.\n\n> The sampling rate varies by a factor of 5 across the different datasets, which using a 16 step prediction window would imply that only about 2ms of activity need to be predicted for mouse versus 10ms of activity of zebrafish. Is this being controlled for somehow? If not, why is it unimportant?\n\nAs you correctly pointed out, the sampling rates across datasets differ significantly. We chose not to downsample the data to a uniform frequency in order to make maximal use of the available data. In Figure 2A, we show that POCO outperforms baseline models across all 16 future steps, suggesting that the specific prediction length is not the primary factor determining relative model performance.\n\nHowever, we agree that downsampling to a consistent rate could offer useful insight. To investigate this, we downsampled Ahrens’ zebrafish dataset by a factor of 2 and the mouse dataset by a factor of 5, so that the effective sampling rate for all three datasets is approximately 1 Hz, matching Deisseroth’s zebrafish dataset.\n\n|Model|Fish PC(Deisseroth)|Fish PC(Ahrens)|Mice|\n|---|---|---|---|\n|POCO|**0.525±0.004**|**0.452±0.005**|0.455±0.001|\n|MLP|0.417±0.002|0.398±0.002|**0.459±0.000**|\n|NLinear|0.165±0.000|0.265±0.001|0.415±0.000|\n|TexFilter|0.440±0.005|0.384±0.006|0.434±0.000|\n|NetFormer|0.221±0.008|0.264±0.005|0.416±0.000|\n\nWe found that overall model performance remains consistent, though POCO performs slightly worse than MLP on the mouse dataset—likely because downsampling by a factor of 5 reduces the effective dataset size. This aligns with our findings in Figure 3A, where simpler models like MLP tend to perform better in low-data regimes.\n\n> The gap in the related work is not entirely clear to me. The motivation given for the particular model examined in this paper emphasizes longer freely behaving recordings in contrast to short task-focused ones, but is this actually important? … Also, why do we need a different architecture for these paradigms?\n> Did you compare against models like those of Azabou et al. [1] or Zhang et al. [2]?\n\nThank you for pointing out this potentially confusing aspect of our motivation. There are two key differences between our work and prior studies on neural foundation models:  \n(1) We focus on forward prediction of calcium imaging data, whereas most previous work has focused on spiking activity and/or behavioral decoding.  \n(2) Our datasets consist primarily of long recordings in freely behaving animals, while prior work often uses task-driven settings.\n\nImportantly, the need for a different model architecture in our case is not due to (2)—as you correctly noted, models can often be applied to both task-focused and freely behaving datasets. Rather, the main challenge stems from (1): models developed for spiking data or behavioral decoding are not directly applicable to the problem of predicting high-dimensional calcium traces.\n\nYou mentioned two recent works [1, 2], but it is not straightforward to adapt a behavioral decoding model [1] for forward forecasting, nor to apply a model designed for spiking data [2] to calcium imaging. Specifically, [2] trains on data with high temporal precision (20 ms bins) and predicts ~200 ms, whereas calcium imaging has much lower temporal resolution. Additionally, their spatial chunking strategy was designed for ~600 neurons per session for spiking data. Scaling this to tens of thousands of calcium traces across sessions is nontrivial – the time complexity of methods [2] grows quadratically with the number of neurons. The spatial chunking strategy is also hard to interpret if we aim to model PCs. In contrast, our task is more naturally aligned with time-series forecasting, which motivates our comparisons to general-purpose forecasting models. \n\nThat said, we agree that adapting more models for comparison is worthwhile, and we have already included an adaptation of [1] in our ablation study. Specifically, we tested a POYO-only variant where predictions are generated by querying the latent embedding directly, instead of using FiLM and MLP (Table 5). We found that this version produces predictions that are often disconnected from the context, which we believe is because the model cannot retain detailed information about current activity in each channel. Its performance does not consistently outperform the copy baseline, suggesting that while POYO may be effective for modeling global brain state or low-dimensional behavioral variables, it struggles with the fine-grained dynamics of large neural populations.\n\n[1] Azabou et al. \"Multi-session, multi-task neural decoding from distinct cell-types and brain regions\n\" ICLR 2025 [2] Zhang et al. \"Towards a “universal translator” for neural dynamics at single-cell, single-spike resolution\" NeurIPS 2024\n\n> Is the same-sized model used in the multi-species training setup? \n\nIn the paper, we reported results using a model of fixed size. However, we also tested larger POCO variants in the multi-species setting and did not observe significant performance improvements: \n\n|Model(layers,hidden size)|Fish PC(Deisseroth)|Fish PC(Ahrens)|Mice|Worms(Zimmer)|Worms(Flavell)|\n|---|---|---|---|---|---|\n|POCO(1,128)|0.500±0.001|0.441±0.007|0.402±0.002|0.324±0.010|0.250±0.011|\n|POCO(2,256)|0.497±0.009|0.440±0.002|0.402±0.003|0.318±0.020|0.252±0.019|\n|POCO(4,512)|0.494±0.009|0.442±0.007|0.401±0.002|0.321±0.008|0.250±0.025|\n\nThese results further support our view that multi-species forecasting challenges arise from biological and dataset heterogeneity, rather than insufficient model capacity.\n\n> It would be nice to have a demonstration, even a small one, where forecasting is useful. In general predicting neural activity is an interesting problem but a solution to it should either develop our understanding of the brain or demonstrably improve the performance of an applied neuro technology.\n\nWe agree that showcasing additional use cases could further strengthen the paper. However, we also believe that neural prediction is a sufficiently fundamental problem to be studied on its own (e.g., consider the line of work following Brain-Score [1] in vision neuroscience or the recent Zapbench [2]). Prior studies have shown that predictive models can be used to infer effective connectivity [3], and our analysis of unit embeddings suggests that POCO may also help interpret the contributions of individual neurons to population dynamics.\n\nForward models of neural activity are also critical for closed-loop or activity-guided optogenetic control, as they can be used to simulate the effects of stimulation [4]. They are particularly valuable in systems aiming to suppress seizures, where accurate forecasting can enable early intervention [5]. However, implementing such systems requires advanced imaging and hardware infrastructure, which is beyond the scope of this work.\n\n[1] Schrimpf, Martin, et al. \"Brain-score: Which artificial neural network for object recognition is most brain-like?.\" [2] Lueckmann, Jan-Matthis, et al. \"ZAPBench: A Benchmark for Whole-Brain Activity Prediction in Zebrafish. [3] Luo, Zixiang, et al. \"Mapping effective connectivity by virtually perturbing a surrogate brain.\" Nature Methods [4] Grosenick, Logan, James H. Marshel, and Karl Deisseroth. \"Closed-loop and activity-guided optogenetic control.\" Neuron [5] Kang, Wonok, et al. \"Closed-loop direct control of seizure focus in a rodent model of temporal lobe epilepsy via localized electric fields applied sequentially.\" Nature Communications\n\n> The proposed method is a very basic extension of existing ideas and models which are already being used for closely related tasks.\n\nWhile POCO builds on existing components (e.g., MLPs, FiLM, and Perceiver-based encoders), the core architectural idea—using population-level context to modulate per-neuron univariate forecasters for multi-step calcium activity prediction—is, to our knowledge, novel and tailored to the unique challenges of neural forecasting across sessions.\n\nWe also respectfully note that the proposed method is only one part of our contribution. The paper systematically explores neural prediction by using models from multiple research fields. Importantly, we evaluate all models on five diverse datasets spanning three species, including nearly 50 hours of recording in total. The scale of the datasets as well as the number of models tested are much larger than recently published benchmarking work on neural prediction [2]. We believe this breadth of experimentation, along with detailed analyses of generalization, adaptation, and interpretability, constitutes a valuable contribution to this emerging field which lacks controlled comparisons of different methods."}, {"review_id": "xWo59P2xYs", "reviewer": "Reviewer_KEzN", "summary": "The authors propose a new model POCO for forecasting neural activity (specifically from calcium imaging), building on POYO and FiLM. They demonstrate its SOTA performance in single- and multi-session settings across several organisms (zebrafish, mice, C. elegans). They further perform analyses on effects of hyperparameters and architectural choices to justify and inform their model design.", "questions": "1. As noted above, POCO gains over MLP don't seem that substantial outside of the two zebrafish PC forecasting tasks. Can simply scaling the MLP size further (to e.g., match parameter count of full POCO) make the simple MLP match or outperform POCO? Do you have any explanation for why the conditioning helps more in some cases than others?\n2. What hyperparameter tuning if any was done for TSMixer (and other baselines)? It appears the architecture described in the Appendix is smaller (hidden dimension of 64) compared to the optimal architecture found by Lueckmann et al. for their ZAPBench baselines.\n3. Do you have any explanation for the much poorer performance of POCO on C. elegans data after low-pass filtering?\n4. Are the reported MLP results for the finetuning evaluation (Figure S12) MLPs trained from scratch? If yes, can you report these curves if you used pre-trained multi-session MLPs?\n\nThis is a bit much to ask in the rebuttal period, but if there is time I would appreciate seeing this addressed too:\n\n5. Is it possible to add a new baseline that is just the MLP but with learnable unit embeddings as an additional input (either concatenated with the neural activity or used as modulation in FiLM)? I would be especially curious to see this in the pre-training -> fine-tuning unit embedding evaluations. Along with this, I would be curious to see the zero-shot and finetuned performance of the transformer-conditioned MLP from the ablation study. For me, these extra baselines could strengthen the argument for the use of the POYO encoder over something that simpler/more parallelizable does not consider the rest of the population.", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strengths:**\n* Text is clearly written and quite easy to understand.\n* The architecture itself is simple and reasonable, building on two established architectures (POYO encoder and FiLM).\n* The ablations and analyses of how performance varies with multi-session/subject training, dataset size, and architecture and hyperparameter settings are thorough and much appreciated.\n\n**Weaknesses:**\n* I worry that baselines were not tuned enough, though I understand that is a huge burden for the authors. But, in particular, the poor performance of TSMixer (worse than the copy baseline in many cases) is a bit strange when it performs well in ZAPBench. In fact, it substantially outperforms POCO in the long context setting, a fact I think should be more clearly stated in the text---it currently reads as if only the UNet bests POCO.\n* POCO does not seem to substantially outperform the simple MLP except for the two zebrafish datasets in the PC forecasting setting. In addition, when the C. elegans data are low-pass filtered, POCO performance drops sharply and significantly underperforms MLP, which raises doubts about POCO's robustness.\n* There is (to my understanding) no comparison in the finetuning evaluation to a pre-trained MLP, only to an MLP trained from scratch, but I think POCO needs to demonstrably outperform a pre-trained univariate MLP to prove useful in this setting.\n* The architecture draws heavily from prior work, and the results on structure in unit embeddings, multi-session/subject training, pre-training benefits, etc. seem generally consistent with prior work as well (POYO, NDT1-3). So, while the application to calcium trace forecasting is certainly new, the overall novelty of those results is a bit limited.\n\nIn summary, I appreciate the extensive evaluations and analyses in the paper, but without some further details/evaluations, I remain somewhat unconvinced of POCO's utility over even just a univariate MLP, both for conventional forecasting setting as well as the pre-training/finetuning setting.", "was_revised": true, "final_justification": "The additional experiments described in the rebuttal strengthen the authors' case that the complexities of POCO do lead to a fairly reliable performance benefit. Still, POCO in many cases doesn't seem to be that much better than just an MLP, and some of the variance in performance across datasets is a bit concerning. However, I really appreciate the overall thoroughness of the work and think the extensive evaluations in the study are worth sharing with the community and can lead to further work that may give us better understanding of when population encoding is beneficial to forecasting and what architectures are better suited to different datasets. For this reason, I recommend acceptance.", "rebuttal": "Thanks for your careful review and constructive feedback! \n\n> Can simply scaling the MLP size further (to e.g., match parameter count of full POCO) make the simple MLP match or outperform POCO?\n\nThank you for the question. In the current manuscript, the MLP baseline is identical to the MLP component used within POCO (2 layers, hidden size = 1024), which allows us to isolate the effect of population conditioning. However, we agree that comparing against larger MLPs with parameter counts similar to POCO is informative, so we evaluated deeper and wider MLPs:\n\n|Model (MLP layers,MLP hidden size)|Parameter Count|Fish PC(Deisseroth)|Fish PC(Ahrens)|Mice|\n|---|---|---|---|---|\n|MLP(2,1024)|0.07M|0.418±0.003|0.370±0.002|0.409±0.000|\n|MLP(2,2048)|0.14M|0.421±0.002|0.370±0.001|0.409±0.000|\n|MLP(3,2048)|4.3M|0.466±0.001|0.418±0.001|0.412±0.000|\n|MLP(4,2048)|8.5M|0.471±0.002|0.417±0.006|0.412±0.000|\n|POCO(2,1024)|2.6M–4.0M|**0.525±0.004**|0.440±0.003|**0.420±0.002**|\n|POCO(4,1024)|4.6M–6.1M|**0.525±0.012**|**0.449±0.001**|**0.420±0.001**|\n\nWe find that increasing MLP depth does improve its performance, particularly on the zebrafish datasets. However, POCO still consistently outperforms MLPs with comparable or larger parameter counts, indicating that model size alone does not explain the performance gap. Based on these findings, we plan to include a larger MLP baseline (3 layers, hidden size = 2048) in the final paper and include the table in appendix. We also acknowledge that further exploration may even yield stronger baselines. However, the same architectural enhancements could also be applied to POCO’s MLP forecaster, likely improving its performance as well. We view this as a valuable direction for future work.\n\n> Is it possible to add a new baseline that is just the MLP but with learnable unit embeddings as an additional input (either concatenated with the neural activity or used as modulation in FiLM)? \n\nThank you for the suggestion. We agree that including these additional baselines strengthens the evaluation and more rigorously demonstrates the advantage of POCO. As you proposed, we tested a model where an MLP is directly conditioned on unit embeddings using FiLM (\"UMLP\"). We also added session embedding so the embedding scheme is exactly the same as POCO. We used the large MLP in earlier experiments (3 layers, hidden size = 2048).\n\n| Model | Fish PC (Deisseroth) | Fish PC (Ahrens) | Mice |\n| --- | --- | --- | --- |\n| POCO | **0.525 ± 0.004** | **0.440 ± 0.003** | **0.420 ± 0.002** |\n| MLP (large) | 0.466 ± 0.001 | 0.418 ± 0.001 | 0.412 ± 0.000 |\n| UMLP (large) | 0.481 ± 0.004 | 0.434 ± 0.004 | 0.413 ± 0.000 |\n\nWe find that adding unit embeddings improves the MLP’s performance slightly, particularly when combined with a larger network. However, POCO still outperforms these variants, suggesting that the unit embedding can only account for part of the performance gain of POCO. We will add these results to the ablation study section in the updated manuscript.\n\n> Are the reported MLP results trained from scratch? If yes, can you report these curves if you used pre-trained multi-session MLPs?\n> Along with this, I would be curious to see the zero-shot and finetuned performance of the transformer-conditioned MLP from the ablation study. For me, these extra baselines could strengthen the argument for the use of the POYO encoder over something that simpler/more parallelizable does not consider the rest of the population\n\nThank you for pointing this out. Indeed, the MLP models in Figure S12 were trained from scratch. We agree that evaluating fine-tuned versions of MLP and the new baselines would further clarify POCO’s utility in this setting. As uploading new figures is not permitted, we summarize the final fine-tuned results below as Table S9. We also include zero-shot performance for transformer-conditioned MLPs (TMLP) and standard MLPs.\n\n|Model| Fish PC (Deisseroth) | Fish PC (Ahrens) | Mice |\n|---|---|---|---|\n|Pre-POCO(full finetune)|**0.532±0.014**|**0.424±0.017**|**0.406±0.001**|\n|Pre-POCO(embedding only)|**0.539±0.008**|**0.392±0.062**|**0.405±0.001**|\n|Pre-MLP|0.446±0.002|0.385±0.003|0.398±0.000|\n|Pre-TMLP|0.470±0.002|**0.423±0.017**|0.400±0.000|\n|Pre-MLP(large)|0.470±0.006|**0.432±0.007**|0.401±0.000|\n|Pre-UMLP(large,full finetune)|0.495±0.003|**0.427±0.013**|0.404±0.000|\n|Pre-UMLP(large,embedding only)|0.499±0.001|0.400±0.007|0.404±0.000|\n|Pre-TMLP(zero-shot)|0.471±0.001|0.362±0.007|0.397±0.000|\n|Pre-MLP(large,zero-shot)|0.474±0.001|0.369±0.002|0.399±0.001|\n\nOverall, pre-trained POCO achieves the best performance across datasets, although some models perform similarly on Ahrens’ dataset. We will update these results in the paper.\n\nWhile the performance gains may appear modest, we emphasize that predictive accuracy is not the only motivation for using POCO over a simple univariate forecaster. A major goal of neural prediction is to obtain models that can support downstream tasks such as studying effective connectivity [1] or enabling closed-loop optogenetic control [2]. Univariate forecasters lack population-level modeling and are not applicable to these use cases.\n\n[1] Luo, Zixiang, et al. \"Mapping effective connectivity by virtually perturbing a surrogate brain.\" *Nature Methods*  \n[2] Grosenick, Logan, James H. Marshel, and Karl Deisseroth. \"Closed-loop and activity-guided optogenetic control.\" *Neuron*\n\n> I worry that baselines were not tuned enough, though I understand that is a huge burden for the authors… TSMixer substantially outperforms POCO in the long context setting, a fact I think should be more clearly stated in the text… What hyperparameter tuning if any was done for TSMixer (and other baselines)? It appears the architecture described in the Appendix is smaller (hidden dimension of 64) compared to the optimal architecture found by Lueckmann et al. for their ZAPBench baselines.\n\nThank you for pointing this out. We will revise the text to more clearly acknowledge POCO’s limitations on ZAPBench. However, we note that ZAPBench includes only a single session, and POCO’s performance can vary across sessions even within the same dataset.\n\nAs described in the Appendix, we started from default hyperparameters in the original papers and made necessary adaptations to ensure baselines ran properly on our datasets (particularly for ModernTCN, NetFormer, and PLRNN). In Figures S18 and S19, we explored learning rate and weight decay effects and found model performance to be generally robust to these choices.\n\nGiven the large hyperparameter space and the number of datasets, sessions, and baselines, we did not extensively tune all parameters—especially model size. This is particularly challenging for single-session models like TSMixer, which require training a separate model per session, increasing computational cost.\n\nThat said, we agree model size can be sensitive. To address this, we ran additional TSMixer experiments and found that larger models tended to overfit and perform worse:\n\n|Model|Fish PC(Deisseroth)|Fish PC(Ahrens)|Mice|\n|---|---|---|---|\n|TSMixer 64|**-0.546±0.066**|**0.013±0.009**|**0.391±0.001**|\n|TSMixer 128|-0.688±0.014|-0.017±0.010|0.384±0.003|\n|TSMixer 256|-0.788±0.050|-0.031±0.012|0.376±0.001|\n\nWe also tested larger model sizes for PLRNN, AR_Transformer, NetFormer, and TexFilter, and did not observe substantial performance improvements. \n\n> POCO does not seem to substantially outperform the simple MLP except for the two zebrafish datasets in the PC forecasting setting. … when the C. elegans data are low-pass filtered, POCO performance drops sharply\n\nThank you for the comment. We agree that POCO’s gains appear smaller in some settings, which is likely due to differences in signal quality and dataset characteristics. Unpredictable noise in the recordings effectively imposes a noise ceiling on performance, pulling the prediction scores of different models closer together, as no model can explain the irreducible variance. We also find that POCO’s advantage is more pronounced in PCA-reduced space, where uncorrelated noise is suppressed. In contrast, single-neuron activity is sparser and noisier, and performance is averaged across neurons, which reduces measured gains.\n\nRegarding the performance drop after low-pass filtering on *C. elegans*, we believe this is because *C. elegans* neurons already exhibit slow, sparse, and graded dynamics as opposed to spike trains. Further filtering may remove subtle but informative temporal variations, making POCO easier to overfit to the training set, and thus flattening out any advantage over simpler models.\n\nDue to space constraints, please see the second part of our response to the second Reviewer (xBNB) for further discussion on why we believe this does not diminish the contribution of our work.\n\n> The architecture draws heavily from prior work, and the results seem generally consistent with prior work … while the application to calcium trace forecasting is certainly new, the overall novelty of those results is a bit limited.\n\nWhile POCO builds on existing components (e.g., MLPs, FiLM, Perceiver-based encoders), its core architectural idea—modulating univariate forecasters with population-level context for multi-step calcium prediction—is, to our knowledge, novel and specifically designed for the challenges of neural forecasting across sessions.\n\nWe respectfully note that the method itself is only part of our contribution. We evaluate all models on five diverse datasets spanning three species and nearly 50 hours of recording. To our knowledge, the multi-species training setup and accompanying simulation are also novel. Moreover, the scale of datasets and breadth of baselines far exceeds recent benchmarks like ZapBench. We believe this comprehensive evaluation—along with analyses of generalization, adaptation, and interpretability—adds significant value to the field."}]

Author rebuttal:

Meta-review: The paper introduces a new model for forecasting calcium activity, evaluated on four custom datasets and the ZAPBench benchmark. On the custom datasets, the model outperforms several baselines, while on ZAPBench, its performance is comparable to existing methods. The authors release their code and two of the four datasets, providing a valuable benchmark setting for neural forecasting research.

Reviewers are broadly positive about the paper’s contribution. Reviewer 9twB highlighted cross-subject generalization and asked mostly clarification questions. Reviewer xBNB was more critical, questioning architectural choices and the modest improvements over baselines, which the authors acknowledged. Reviewer KEzN raised issues about baseline tuning and experiments, but was convinced by the authors’ additional results, while also noting limited novelty relative to prior work (e.g., POYO, NDT). Reviewer pkKL emphasized concerns about clarity, particularly in how the contribution is framed in the abstract, introduction, and discussion.

Post-review, the paper received borderline acceptance recommendations. Remaining concerns include limited and inconsistent performance gains, with a simple MLP baseline not consistently surpassed. Reviewers requested further experiments and analysis; while some were provided, they were only partially satisfactory.

The central question for acceptance is whether the problem formulation and benchmarking setup justify publication. Reviewers noted that forecasting in low-frequency calcium imaging is understudied, and the paper’s main value lies in applying established methods with a thorough comparative evaluation in this domain.

Requested changes include, but are not limited to:

- "discussion on how dataset properties (e.g., species, SNR, recording length, recording size, etc.) influence model performance" (xBNB)
- include additional experimental results discussed with Reviewer KEzN; tables critical to the discussion should be added to the appendix, and referenced/interpreted in the paper (it is not acceptable to "bury" them in the appendix given the importance to the questions about baseline tuning etc)
Decision: Accept (poster)