Decoding Dynamic Visual Experience from Calcium Imaging via Cell-Pattern-Aware Pretraining
Authors: Sangyoon Bae
Mehdi Azabou
Blake Aaron Richards
Jiook Cha
Venue: ICLR 2026 Poster (2026)
Keywords: Computational Neuroscience
Machine Learning for Science
Neural Decoding
Calcium Imaging
Neural Population Heterogeneity
Visual Reconstruction
Scaling
Self-Supervised Learning
Representation Learning
Primary area: applications to neuroscience & cognitive science

Abstract: Neural recordings exhibit a distinctive form of heterogeneity rooted in differences in cell types, intrinsic circuit dynamics, and stochastic stimulus–response variability that goes beyond ordinary dataset variability, mixing statistically regular neurons with highly stochastic, stimulus-contingent ones within the same dataset. This heterogeneity poses a challenge for self-supervised learning (SSL)—learnable statistical regularity—thereby destabilizing representation learning and limiting reliable scaling.
We introduce POYO-CAP (Cell-pattern Aware Pretraining), a biologically grounded hybrid pretraining strategy that first trains with masked reconstruction plus lightweight auxiliary supervision on statistically regular neurons—identified via skewness and kurtosis—and then fine-tunes on more stochastic populations.
On the Allen Brain Observatory dataset, this curriculum yields 12–13\% relative improvements over from-scratch training and enables smooth, monotonic scaling with model size, whereas baselines trained on mixed populations plateau or destabilize. By making statistical predictability an explicit data-selection criterion, POYO-CAP turns neural heterogeneity into a scalable learning advantage for robust neural decoding.

Published as a conference paper at ICLR 2026
DECODING DYNAMIC VISUAL EXPERIENCE FROM CALCIUM IMAGING VIA CELL-PATTERN-AWARE PRE-
TRAINING
Sangyoon Bac
Mehdi Azabou
Interdisciplinary Program in Artificial Intelligence Scoul National University
NSF AI Institute for Artificial and Natural Intelligence (ARNI)
Columbia University
Scoul, 08826, South Korea
New York City, 10027, United States
{stel (stellasyber lasybae @nnu. ac.
{mehdi azabou gma i com
Blake Richards
Jiook Cha
Mila (Quebec AI Institute)
Department of Psychology
Dept. of Neurology & Neurosurgery, McGill University
Departunent of Brain and Cognitive Sciences
Montreal Neurological Institute, McGill University
Interdisciplinary Program in Artificial Intelligence Seoul National University Seoul, 08826, South Korea {connect anu ac kr
School of Computer Science, McGill University Learning in Machines and Brains Program, CIFAR
Montreal, H2S 3H1, Canada
(blake. ri chards @mila. quebec
ABSTRACT
Neural recordings exhibit a distinctive form of heterogeneity rooted in differ- ences cell types, intrinsic circuit dynamics, and stochastic responss variability that goes beyond ordinary dataset variability, mixing statistically regular neurons with highly stochastic, stimulus-cottngent ones within the same dataset. This heterogeneity poses challenge for self-supervised learn- ing (SSL)- statistical regularity-theree thereby destabilizing representation learning and limiting reliable scaling. We introduce POYO-CAP (Cell-patter Aware Pretraining), a biologically grounded hybrid pretraining strategy that first trains with masked reconstruction plus lightweight auxiliary supervision tistically regular neurons-identifies via skewness and kurtosis-and then fine- tunes more stochastic populations. On the Allen Brain Observatory dataset, this curriculum yields 12-13% relative improvements over from-scratch training and enables smooth, monotonic scaling with model size, whereas baselines trained mixed populations plateau or destabilize. By making statistical predictability explicit data-selection criterion, POYO-CAP turns neural heterogeneity into a scalable learning advantage for robust neural decoding.
INTRODUCTION
Learning useful representations from neural data poses a fundamental challenge for machine learn- ing, as datasets from varied lab settings are not only small-scale but the signals themselves are com- plex, highly-dimensional, and only-partially observed (limitation of recording technology), while available labels are typically too scarce and weak for effective supervision. Self-supervised learning (SSL.) offers powerful paradigm to address this data scarcity, as provides a way to learn from large amounts of data with limited access labels, thereby allowing many datasets to be combined. This could be particularly useful for reconstructing perceptions or intentions directly from neural activity, e.g. for Brain-Conme Computer Interfaces (BCIs).
However, successful SSL relies on statistical regularities in data, as evidenced by masked model- ing and sequence prediction in structured domains such as language (Harris) 1954: Tenney al.)
2019 Sinha et al., 2021]) Yu et al.) 2024 Lan et al.) 2019: Li Jurafsky] (017). Neural decoding, contrast, poses unique challenge this prerequisite of predictability. We only record small,

---

Published as a conference paper at ICLR 2026
biased subset of neurons from the full circuit, creating heterogeneous sample where predictability not uniform. This unpredietability often correlates with cell type: inhibitory and corticothalamic neurons tend to exhibit more regular dynamies, while excitatory pyramidal cells appear sparser and more stochastic in isolation, partly because we lack access to the broader network signals that drive them. Training SSL models indiscriminately on this mixed-signal data is therefore counterproduc- tive, as the loss becomes dominated by the unpredictable neurons, pulling the model's focus from the relevant and regular patterns should be learning.
We test the Statistical Regularity Hypothesis: that representation learning efficiency scales with the statistical regularity of the selected neural subset. This principle is motivated by the observation that different neural populations, such as inhibitory interneurons and modulatory neurons exhibit fundamentally distinct statistical dynamies. Our hypothesis leads a "data diet" approach for neu- roscience SSL, where, unlike conventional methods that rely on task difficulty, we propose that the intrinsic statistical properties of neurons should guide the learning curriculum.
To validate this, we introduce POYO-CAP, cell-pattern-wwe hybrid pretraining framework that uses higher-order statistics (skewness and kurtosis) as proxies for regularity to first pretrain on sta- tistically stable neural populations, combining masked reconstruction with lightweight auxiliary supervision to overcome prior methods' homogeneous treatment of heterogeneous data. Our results confirm the hypothesis: by transforming neural heterogeneity from challenge into asset, this ap- proach improves data efficiency by 1. 1.98x and enables high-fidelity movie frame reconstruction from calcium recordings under controlled stimulus setting, offering principled, cooogically-groundde recipe for scalable neural decoding.
Our contributions are threefold:
We introduce a ioongically-groundee pretraining paradigm that uses statistical regularity (rather than task-based difficulty) to guide data selection, selectively learning from neurons with highly regular responses first before training on more stochastic neurons.
We present an end-to-end decoder architecture that transforms neural population activity into high- fidelity visual reconstructions, operating independently of external stimulus information.
We demonstrate that functional heterogeneity, when properly leveraged through our regularity- based data diet, enables robust model scaling unlike conventional approaches that plateau with increased capacity.
Terminology 1. We refer to setup hybrid objective, simple form of curriculum learning (Bengio et al. (2009)). The primary objective is masked reconstruction on neural dynamics, while supervised auxiliary cross-entropy primitive stimuli serves an "easy" initial step to stabilize training and prevent representational collapse. Importantly, no downstream labels are used during this pretraining phase.
We define a neuron population as predictable from a self-supervised learning (SSL) perspective: its activity must contain sufficient statistical regularity for model to successfully construct masked portions of its signal. We empirically link this SSL-defined predictability to low skewness and kurtosis in calcium traces. Thus, while our definition aligns with the neuroscientific concept of stable firing patterns, it remains a fundamentally operational one, tied to the success of the masked reconstruction task.
Terminology 2.
2 RELATED WORK
Decoding Models for Neuroscience Recent neural decoding models span diverse architectures and learning paradigms. Transformer-base approaches such as POYO (Azabou et al. 2023) and POYO+ (Azabou et al. 2024) enable multi-session learning but depend on full supervision, lim- iting scalability to unlabeled data. Self-supervised methods like CEBRA (Schneider al.) (023) relax label requirements for single-session training but require labels for multi-session training. In visual reconstruction, fMRI-based frameworks have reached high fidelity citepchen2023cinematic
joo2024brain through masked modeling and large generative models, but rely on indirect stimulus- to-brain mappings from fMRI's slow hemodynamic signal. While these approaches set benchmarks for fMRI, direct comparison is challenging due modality differences. In contrast, method learns directly from neural recordings using intrinsic population dynamics with minimal auxiliary supervision.
2

---

Published as a conference paper at ICLR 2026
SSL in Neuroscience Most self-supervised approaches to neural data assume population homo- geneity and ignore functional specialization. Models such as Neuro- Neuro-BERT Wu et al. (2022) treat all neurons equally, while contrastive task-aware methods Song et al. (222 Zhao et al. (2024) de- pend on external supervision rather than intrinsic circuit structure. These frameworks overlook that predictable neurons (inhibitory interneurons and modulatory pathways) differ fundamentally from stimulus-encoding neurons in computational role and temporal dynamics. Recent work by JJohnson al. (2022) characterized such heterogeneity through in vivo imaging, while our predictability- based selection offers distinct computational advantages by identifying and pretraining on regulatory neurons, enabling SSL to capture circuit-level dynamics and improving scalability beyond uniform population models.
Data-Centric SSL and Neural Heterogeneity Our approach aligns with the emerging "data dier" perspective in machine learning, which posits that the quality of pre-training data is as critical as its quantity (Paul et al. 2021: Zhuang et al.) 2025). However, we distinguish our framework from these methods a fundamental way: while standard approaches prune training samples (e.g., specific images or text), our strategy selects neurons (feature sources). In neural recordings, heterogeneity is intrinsic to the sensor array itself, not just the examples. We demonstrate that adding more neurons can paradoxically lead to a "scaling collapse" a failure mode unique to heterogeneous neural pop- ulations. By selecting neurons based on statistical regularity, we resolve this collapse and transform heterogeneity from liability into an asset for scaling.
METHODS
3.1 DATASET AND PARTITION
We use the Allen Brain Observatory (BO) calcium imaging dataset, featuring recordings from 13 Cre driver lines (de Vries al. 2020). We partition the dataset into pretraining and finetuning sets at the Cre-line level. To form the pretraining set, we identified "predictable" subset by apply- ing knee-detection algorithm (Algorithm (ST) the per-line skewness and kurtosis distributions. This a priori process selected four lines (SST, VIP, PVALB, and NTSR1) that fell below the sta- tistical core-ccorespondingg to major inhibitory interneuron classes and one modulatory excitatory line. The remaining Cre lines were reserved for finetuning and downstream evaluation. For down- stream tasks (movie reconstruction and drifting grating decoding), train/validationntess splits were performed at the trial level within each session. Specifically, stimulus trials from each recording session (i.e., each animal) were divided into non-overlapping temporal segments for training, vali- dation, and testing. Thus, the same animal may appear across splits, but distinct trials were used for evaluation, preventing temporal leakage while assessing generalization across neural responses to unseen stimulus segments. Importantly, pretraining and finetuning datasets were strictly separated at the animal level due to the disjoint Cre-line partitioning: animals used for pretraining (predictable lines) do not overlap with those used for finetuning (unpredictable lines). This design ensures cross- subject transfer at the pretraining stage, while downstream evaluation focuses on generalization across trials within each biological subject. To guarantee fair comparison, we verified that all models (including baselines and ablations) were evaluated on identical held-out trial splits. Details are described Appendix A
3.2 CELL-PATTERN PRETRAINING
3.2. 1 DATA-EFFICIENT SELECTION CRITERIA
We hypothesize that neurons showing statistical regularity are ideal for effective SSL pretrain- ing. Within our framework, we operationally define this predictabilit- the inherent structure enabling effective masked reconstruction. To identify these neurons without labels, we leverage per- neuron skewness and kurtosis. We refer to the selected subset as exhibiting near-Gaussian activity (mean skewness 1.87, kurtosis 7.32), characterized by symmetric, thin-tailed distributions suitable for learning general features. In stark contrast, excluded neurons exhibit heavy-tailed, sparse burst- ing (mean kurtosis 148.51), better reserved for spec-spe fine-tuning. For rigorous empirical validation of these metries, see Appendix C
3

---

Published as a conference paper at ICLR 2026
Pretraining
Data Size
134 sessions, 80,146 samples skeuness 3.51, kurtosis S 22.62 Vx0000 (KISTI cas v100 4)
Selection Criteria
Hardware
Fine-tuning (Movic decoding, Drifting Gratings)
Selected Data
299 sessions, 1,170,931 samples skewness > 3.51, kurtosis > 22.62 900
Selection Criteria
Frames (movie decoding)
Hardware
4x V100 (KISTI cas_v100nv_4)
Computationally--Riiii Efffiieet Pretraining Summary of dataset scale (sessions and samples), predictable-neurgl selection criteria (skewness and kurtosis computed on per-neuron AF/F traces over the full recording). and computational setup for pretraining and fine-tuning.
Table 1:
Notes. (1) Selected Data number of predictable (pretraining) unpredictable (finetuning) sessions / samples after skewness/krtoois filtering. (2) For movie decoding, training batches preserved temporal order, whereas validation and test batches were randomly shuffled to evaluate generalization beyond temporal continuity.
To objectively partition the data, we applied a knee-detection algorithm (Satopaa et al. (201 Ih) to find a data-driven threshold across the 13 discrete CRE lines. Specifically, we identified the knee point on the sorted distribution of per-line mean statistics, establishing cutoff based on cell-type categories rather than individual neuron scores. While this approach failed for lower-order statis- ties like event rate and Fano factor, it revealed a clear breakpoint for both skewness and kurtosis, providing principled basis for our split. The resulting data-driven thresholds (skewness < 3.51, kurtosis I 22.62) identified a "predictable" subset comprising four CRE lines: SST, VIP, PVALB, and NTSR1. This statistically derived group is also biologically coherent, consisting of three ma- jor inhibitory interneuron classes and one regulatory corticothalamic excitatory line (NTSR1) 1), all of which are crucial for stabilizing neural circuits. This convergence of statistical and biological criteria validates that our method effectively captures neurons showing statistically regular firing pattern. Crucially, these thresholds were determined a priori as a single, fixed criterion partition the dataset, not as a tunable hyperparameter, which is why a sensitivity analysis was not performed.
3.2.2 MODEL FRAMEWORK
Predictable Neuron Pretraining with Auxiliary Classification We introduce a latent masked modeling approach train our model: masked and an unmasked views of the same sample fed independently through the encoder, the latent representation of the unmasked view is then used as target for the latent representation of the masked variant. To avoid representational collapse (Grill et al. )0 Chen et al. (2020)), we use supervised auxiliary loss. This auxiliary loss bootstraps early selectivity while masking- based reconstruction shapes representations for downstream decod- ing. The primitive labels also serve as guidance to stabilize early optimization.
Our architecture is based on the POYO+ (Azabou et al. (0244) architecture: calcium traces are tokenized into sequence of input tokens that are then compressed, using cross-attention
block, into sequence of latent tokens, which we note Z ( ...),,,()), where L is
the number of latent tokens and (i) E Ied is the latent embedding. Each latent token z has an associated timestamp relative to the context window. We introduce the following tem- poral masking scheme: causally mask a percentage of the latent tokens to form Z Zunaseed
(i)
{2(2), .... 2 (L- < MASKED > ..... < MASKED >}. We selected a masking ratio of 50% empirically, i.c. the second half of the context window is masked. We use a siamese network (see Figure l) to feed both Z, and Zmmakeed through the same self-attention blocks which yields E₁ and Z. Zuuskkek respectively. Finally,
Z, the target for Zypakeed
During pre-traie vre-training, the model is trained on a joint objective, consisting of self-supervised masked reconstruction and fully-supervised classification of drifting grating orientations. This auxiliary classification task stabilizes the early training dynamics before the model focuses on the complex downstream movie decoding task.
4

---

Published as a conference paper at ICLR 2026
POYO+
maxter
Crosa
Self
Croea
Drifting Gratings Ortentation
POYO+ Encoder
Decoder
b
POYO-UNet
PDYO- Encoder
Connection Docoder
POYO+
up
PDYO Brcodd
20rt Doooder
801
Drifling Gratings Onietaaton
> upsampli + conV llncar -..COTT
Neural Embedeing
Unpradictable CRE-line calcium traces
Figure Overall Framework of POYO-CAP. (a) Pretraining strategy using predictable calcium traces with masked reconstruction learning (50% masking temporal dimension), (b) Task specifpe finetuning with unpredictable traces using either skip-connection UNet decoder (complex tasks) or original POYO+ decoder (simple tasks). (c) Skip-Connece Connection UNet Decoder architecture replacing traditional encoder skip connections with neural embedding projections.
The pre- rre-trainigg loss is as follows:
LoSSpretrain LOss L (mmasked Zz) + A LoSSCnnssinngyy (Depredicee:: DG
where À is a loss weight that we empirically found = 0.01 to be optimal through grid search (A 0.001, 0.01, 0.1, with performance degrading by 7-11% for < 0.001 À 0.1. We keep the cross-entropy weight small so CE accelerates convergence while masking drives representation formation. This hybrid objective operationalizes curriculum learning strategy, where the simple auxiliary task provides a stable foundation for the more demanding masked reconstruction objective. Details provided in Appendix
Task-Specific Fine-tuning on Unpredictable Neurons Finetuning uses unpredictable CRE-line traces with task-specific decoders. For classification and simple regression tasks such as drifting- grating orientation prediction, we use the POYO+ multi-task decoder, and for complex movie frame reconstruction we employ dedicated vision-specialized Skin-Connection U-Net decoder.
The finetuning loss is as follows:
Loss movic 50 Loss focal + 50 Loss 50 Lossyrr + Loss 1.08ppercpphur + 0.1 Loss (2)
Losspa LossCroe=Entrop/ (DG predictedd DG
Loss weights in Eq. were determined through a systematic grid search over [0.1-100] using SSIM validation score. The different loss terms in the movie reconstruction loss corresponds to special- ized components (Focal (Lin et al. (2017)), FFT (Fast Fourier Transform, (Zhao et al. (2016))), Perceptual (Johnson al. (2016)), SSIM (Wang et al. (2004)) that ensure high-fidelity image reconstruction. See Appendix for details on each loss term.
Skip-Connection U-Net Decoder To address the challenge of reconstructing high-resolution movie frames, we designed specializaed decoder, this dense prediction task requires custom vision modules that were not designed the POYO+ decoder. Our new U-Net-inspired decoder generates frames from a single neural embedding. In each upsampling stage, direct projection of the latent vector (e.g., to 128 2 x 2, 64 x 4) is concatenated with the upsampled feature map and fused with x convolution. These repeated latent injections crucial for maintaining mantic information across all scales, enabling the faithful reconstruction fine visual details from compact neural representation. See Appendix H for more details.

---

Published as a conference paper at ICLR 2026
3.3 NUMERICAL ANALYSIS
3.3.1 LOSS LANDSCAPE ANALYSIS
To understand the challenge of optimizing representation learning models neural data, pro- jected neural activity onto its first two principal components (PCs) and approximated the recon- struction loss landscape. The loss each grid point in the PC space was estimated using a k-nearest neighbor approach (k: = 5), which considered the local variance of nearby data points and a distance penalty term. Landscapes were smoothed for visualization via Gaussian filter (a 1.0) (Li et al. (018)).
3.3.2 INFORMATION THEORY ANALYSIS
We used Fisher Information as a metric for data quality, where I(0) - E [ log p(x[))) quan- tifies the amount of information each data point provides about underlying model parameters Amari (1998,, with higher values indicating better parameter estimation and convergence. For a quasi- Gaussian signal, this be approximated the inverse of the signal variance (I 1/o²). Based on this, we defined the Effective Dataset Size (Desy) )) as the raw data size weighted by its quality, where a higher Fisher Information value corresponds a larger effective size. This allows for more accurate comparison of dataset utility beyond simple data point counts (Kaplan et al. (2020)).
3.4 REPRESENTATION ANALYSIS
We quantified the properties of the learned latent spaces using several metrics, including t-SNE for visualization, Intrinsic Dimension (ID) for efficiency (Levina & Bickel (2004), and metries to assess geometric dissimilarity and structural integrity. To assess dissimilarity between latent spaces learned by different models, we used Procrustes disparity (Dryden & Mardia (2016)) and Centered Kernel Alignment (CKA) (Kornblith et al. (2019)). To evaluate local structure, we used Temporal Neighborhood Preservation score (Venna Kaski (2001)) (see Appendix x for all definitions).
4 RESULTS
4.1 EXPERIMENTAL SETUPS AND BASELINES
To isolate the benefits of our cell-pattem-aawe aware pre-training, we compare our main model, POYO- CAP, against a crucial baseline:
Supervised Baseline (From-Scratch): To rigorously quantify the performance gains from our SSL stage, we compare against a baseline sharing an identical uucoder-decodee architecture but trained end-to-end on the downstream tasks without pre- traitriin
Architecture Ablation Studies: To disentangle the contributions of encoder representations and decoder capacity, we include three capacity- matched variants. Capacity matched means total pa- rameters are within +3% of our model.: (i) MLP Encoder - MLP Decoder, which maps neurons directly to pixels through deep fully-connected network with no spatial inductive bias; (ii) POYO Encoder + MLP Decoder, which retains our SSL encoder but replaces the U-Net decoder with a purely linear decoder to test whether learned representations alone can drive performance; (iii) POYO Encoder U-Net Decoder without skip connections, which preserves the U-Net hierarchy but removes lateral skip pathways to assess the importance of multiscale feature fusion.
We compare to POYO+ (Azabou et al., 2024) which is state-of-the-arr model. To benchmark against external SSL methods, we evaluated an adapted CEBRA baseline (Schneider et al.. 2023) by training its encoder and feeding representations to vision decoder. This yielded an SSIM of ~0.48, confirming that contrastive latent spaces optimized for behavioral alignment do not transfer effectively to high-fidelity pixel generation. For CEBRA, report the best performance between training from scratch and fine-tuning strategies. Regarding Neuro- BERT (Wu et al.) 2022). the lack of official implementation prevented reproducible adaptation, and thus it was excluded.

---

Published as a conference paper at ICLR 2026
4.2 EFFECT OF AWARE PRETRAINING
4.2.
CELL-PAITERN-AWARE PRETRAINING ENABLES SMOOTHI LOSS LANDSCAPE
Landscape: Neurons
Landscape: Unpredictable local FOV-2x)
(5mooth, convex-like, FOV=2xl
PC2 15.20
7330
2030
-18
16
30
-W0
15. 200
PCI
Figure Loss Landscape Topology Reveals a Dichotomy Optimization Difficulty. Masked recon- struction loss landscapes for predictable and unpredictable neurons, projected onto their first two principal components (PCs) with an expanded field of view (FOV=2x). (a) The landscape from predictable neurons is smooth and convex-like clearly revealing high-loss boundaries that enclose data points (red dots) in a single basin, indicating a well-posed optimization problem. (b) contrast, the landscape from unpredictable neurons rugged and non-convex, characterized by numerous local minima, which presents challenging, ill-posed problem.
Our analysis of the masked reconstruction loss landscape elucidates fundamental dichotomy in the nature of the optimization problems presented by the two neural populations. Predictable neu- rons induce geometrically well-posed landscape characterized by a smooth, convex-like surface (roughness øg = 14.8544)), which is highly amenable to gradient-basee based optimization methods. In stark contrast, unpredictable neurons give rise to a treacherous, non-convex landscape (roughness = 2048.477 4712) plagued by multitude of spurious local minima. Crucially, the quantitative contrast remains striking even with the expanded FOV: despite the inclusion of steep basin walls, the "unpredictable" landscape remains 1338 rougher than the "predictable" one. This confirms that our conclusion is robust to the choice scale: the structural optimization gap between the two populations is massive, regardless of the field of view. This topological difference explains why the pre-train tre-training task transforms from a simple optimization challenge to complex, ill- ill-posed problem, thereby providing rigorous geometric basis for the superior performance of the predictable- first pre-training curriculum.
4.2.2 PREDICTABLE NEURONS CONTAINS RICHER REPRESENTATION
Metric
Predictable
Unpredictable
Ratio (Urre./Unpred.)
Fisher Information (Data Quality)
64.51 +..11+0.500
0.555/0.66
33.47+0.46//0.33
1.93x 1.98x
Data Quality Ratio (Efficiency)
34.41
17.39
Effective Dataset Size
71.5 M
227.5 M
Table 2: Informattion-hhered Analysis of Data Quality. quantitative comparison predictable and unpredictable neural populations. The analysis reveals that predictable data information-thcorrttialll theoretically supe- rior, providing basis for its enhanced performance and scalability. Values are reported as mean + 95% CI.
Our analysis revealed that predictable neural data information-theorreiaallyy richer, which trans- lates directly to greater data efficiency. We quantified this using Fisher Information, finding that the predictable dataset had value of 64.5 compared 33.5 for the unpredictable dataset, indicating that each predictable data point contains 1.93 times more information for model training (Table 2) Consequently, while the raw dataset sizes were comparable, the quality-ajjusted Effective Dataset Size (Dafs) was significantly larger for the predictable population, making each of its data points 1.98 times more efficient for training.
4.2.3
PRETRAINING ACHIEVES HIGH PERFORMANCE
As shown in Table 3 our cell-pattern pretraining yields consistent performance gains across downstream tasks. On the movie decoding task, our approach achieves an SSIM of 0.593 for neural- to-visual reconstruction (Figure B. We note that training and test frames same stimulus set: thus, the task evaluates generalization of neural-to- nuuralto-framee mappings across tem- poral segments and pretraining transfer, rather than reconstruction of entirely novel stimuli. On the
drawn from the

---

Published as a conference paper at ICLR 2026
Method
Pretrain Data
Finetune Data
Movic SSIM
DG Accuracyt
POYO-CAP (Ours)
Predictable
Unpredictable All (Pred. Unpred.)
0.593100.013 0.528=00023
0.555.10.022
Baseline: Train All
N/A (From Scratch)
0.492+0.004
Architecture Ablation Studies
MLP EMC Dec.
Predictable Predictable Predictable
Unpredictable
0.449:0.002
POYO+ EnC. Dec.
Unpredictable
0.503+100019
POYO+ Ene.+
Unpredictable
0.466:+0.047
Dec. without skip
connection
CEBRA UNett
Predictable
Unpredictable
0.481=1.0010
Dec.
Dula-Selection Ablation Studies
Inhibitory-only SSL
Inhibitory Unpredictable
Excitatory
0.544+0.030
0.537=+0.02
Reverse SSL
Predictable
0.48911.0332
0.213110003
Mixed SSL
Unpred. + partial Pred. Random (Size-matched)
Unpredictable
0.543
0.313110.012
Random subset SSL
Remaining
0.532:10.044
0.254110001
Pretraining Objective Ablation Studies
Random Masking
Predictable
Unpredictable
0.54010.017
0.54810.028
Masking Loss only
Predictable
Unpredictable
0.496:10.000
0.099110019
Large CE weight (0.1)
Predictahle
Unpredictable 0.552+0.052
0.482+0.033
Small CF weight (0.001)
Predictahle
Unpredictable 0.532++0.042
0.469++0.015
Cross-Entropy Loss only
Predictahle
Unpredictable 0.506++0.057
0.452++0.026
Table 3: Performance comparison across multiple visual decoding tasks. Our proposed framework, POYO- CAP, consistently outperforms baseline models, demonstrating the effectiveness and generalizability of cell- pattern-awar sule-supe learning. Best results shown in bold. Movie decoding task is denoted as movie, drifting gratings decoding task is denoted as DG. Values are depicted mean 95% CI across three seeds (with 0.05. (paired t-test)). Dashes indicate tasks not applicable to image-only decoders.
drifting-gratie grrtinggggatiggs classification task, our model reaches 55.5% accuracy, outperforming the from- scratch baseline (49.2%)
Figure 3: End-to-end neural-to-vision decoding. (a-e) depict movie frames presented to the mouse, (f-j) depict reconstructed frames. Our model captures subtle tame-to-frame variations (red boxes), demonstrating faithful reconstruction of stimulus-driven visual structure rather than trivial averaging.
Ablation studies highlight the benefits of our approach, indicating that both the architecture and the learning objective tailored to the data's statistics are important factors. The superior performance of temporal masking over random masking underscores the value of the objective and lends functional support to our selection criteria. Temporal masking preserves local temporal dependencies critical for neural dynamics (lypically 50-100ms receptive fields in V1 neurons), while random masking disrupts these patterns. This result suggests the curated neurons ("predictable" neurons) indeed pos- sess the predictable temporal structure that specialized task can effectively exploit. Furthermore, data-selection ablations indicate that data quality can outweigh quantity; reversing the curriculum pretrain unpredictable neurons leads to worse performance than training from scratch, sug-
8

---

Published as a conference paper at ICLR 2026
gesting that pretraining highly stochastic data may establish less effective inductive bias for downstream learning. Overall, our approach of selectively pretraining on neurons with regular firing patterns leverages population heterogeneity to enable stable and scalable representation learning.
4.2.4 THE REPRESENTATIONAL ADVANTAGE OF
AWARE PRETRAINING
6.0
Figure 4: POYO-CAP learns a more efficient and structured latent manifold. (a-c) t-SNE visualization of latent spaces from our POYO-CAP model (a), a reverse SSL model (b), and from-scratch model (c). Point color reflects the temporal progression of frames. (d) Quantitative comparison of the Intrinsic Dimension (ID) for each model.
Analysis of the learned representations reveals a stark contrast between the strategies (Figure 4 Qualitatively, t-SNE visualizations show that our POYO-CAP model learns well-structured man- ifold that captures the data's temporal continuity, while baseline approaches like reverse SSL and from-scratch training yield disorganized or collapsed representations. This visual observation validated by multiple quantitative metrics. Our model's s latent space is more efficient, with signifi- cantly lower intrinsic dimension (ID) of 4.14 compared to from-scratch (4.97) and reverse SSL (7.82) models. It also better preserves local temporal structure, evidenced by a higher Temporal Neighborhood Preservation score (0.2355 vs. 0.1584 and 0.0960). Furthermore, high Procrustes disparity (>0.98) and low Centered Kernel Alignment (CKA,.00.13 confirm that the methods learn fundamentally different feature spaces. Taken together, these results demonstrate that our selective pre-training crucial for learning concise and structured representation of the neural code.
4.3 POYO-CAP ENABLES STABLE MODEL SCALING
A key advantage of our framework is its ability to enable stable model scaling, a critical property for building more powerful decoders. To quantify this, we performed a bootstrap regression analysis (N -10, 000). While models trained from scratch (gray) and those pretrained on only unpredictable neurons (cyan) exhibit erratic or flat scaling (slopes z 0.005 0.005-0.013), our main approach (red) un- locks consistent performance gains as model capacity increases, achieving a statistically significant positive slope (0.018, < 0.01). This represents ~40% steeper scaling trajectory compared to the from-scratch baseline. This demonstrates that well-designed pre-training strategy is prerequisite for effective scaling.
Furthermore, comparing pre-training data mixtures reveals what constitutes a good pre-training set. The model pretrained mixed predictable and unpredictable neurons (yellow) excels at smaller scales but fails to improve at larger capacities (slope=0.006). This suggests that the quality, not merely the quantity, of pre-training data is the key determinant for scalability. In view, the noisy signals from unpredictable neurons function as an information bottleneck, limiting the formation of a robust and smoothly scalable representation. Conversely, pre-training on the "clean" signal

---

testable Published as a conference paper at ICLR 2026
b
Dffect of Model Size
130
AALA
Teat SAM4
90-
320
422
2405
Model 3tce ONL
Medd sizz (M)
Figure 5: Pre-training with predictable neurons is crucial for effective model scaling, (a) Test SSIM performance versus model size for different pre-training strategies. Only the model pretrained exclusively on predictable neurons (red) demonstrates robust, positive scaling with model capacity (slope=0.018, p < 0.01 under bootstrap analysis). In contrast, training from scratch (gray) or including unpredictable ncurons in pre- training (cyan, ycllow) leads to flat or erratic scaling (slopes = 0,005-0.013) 0.013). (b) Corresponding validation loss during pre- rre-training the predictable set, showing general downward trend that indicates successful learning.
from predictable neurons (red) builds superior foundation that larger models can exploit, leading to significant performance gains. This successful scaling is corroborated by the general decrease in validation loss during the pre- pre-training stage, as shown in Figure
4.4 MECHANISTIC ANALYSIS OF TRANSFER
To understand the transfer from predictable to unpredictable neurons, we analyzed parameter dy- namics during fine-tuning. Pretraining establishes a stable "representational scaffold" that is largely preserved: encoder weights change by only ~0.18% (norm ~ 222,909), whereas the readout layer adapts substantially, with bias magnitudes increasing 12.4x (p < 0.01). This suggests that the encoder provides pre-optimized latent manifold (Figure 2). while the readout layer adjusts task- specific decision boundaries without destabilizing the representation (see Appendix F for details).
5 CONCLUSION
We introduce POYO-CAP, a cell-pattern-aware hybrid pretraining framework that makes self- supervised learning (SSL) robust to the brain's statistical complexity. Neural recordings combine multiple dynamical regimes- statistically regular activity and heavy-tailed bursts and indiscrimi- nate SSL on this mixture can ill-condition optimization and limit reliable scaling.
POYO-CAP treats statistical predictability as an explicit data-selection principle. Using skewness and kurtosis, we identify predictable subset and pretrain with temporal masked reconstruction plus lightweight auxiliary supervision, then fine-tune on more irregular populations. On the Allen Brain Observatory calcium imaging dataset, this predictable-fie reconstruction SSIM = 0.593; drifting gratings accuracy = 0.555), increases effective data efficiency by 1.98x, and yields smooth, monotonic gains with model capacity where baselines plateau or desta- bilize. The skewness/kurtosis thresholds serve as computational proxies consistent with known bio- logical distinctions without implying a causal mechanism, and our evaluation focuses on mouse vi- sual cortex under controlled stimulus conditions. These results frame predictability-fis curriculum signal for scaling neural SSL beyond this setting.
eurriculum improves decoding (movie
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGEMENTS
This work was supported by the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) (No. 202IRICIC1006503, RS-2023-000266777, RS-2023- 00265406, 25-2022-00421266,, RS-2024-00332331, RS-2024-00435722, RS RS-2025-25457239 RS- 2021-NR061 370, searchers Program through Seoul National University (No. 200-20250071, 200-20250049, 200- 20250116, 200-202501 15, 200-20250113). Additional support was provided by the Institute of Information & Communications Technology Planning & Evaluation (IITP) grant funded by the Ko- rea government (MSIT) [No.
and NRF-202 01A222021SS1A22220000007777 by the Re-
Artificial Intelligence Graduate School Program, Seoul National University] and by the Global Research Support Program in the Digital Field (RS- 2024-00421268. This work was also supported by the Artificial Intelligence Industrial Convergence Cluster Development Project funded by the Ministry of Science and ICT and Gwangju Metropoli- tan City, by the Korea Brain Research Institute (KBRI) basic research program (25-BR-05-01), by the Korea Health Industry Development Institute (KHIDI) and the Ministry of Health and Welfare, Republic of Korea (HR22C1605), and by the Korea Basic Science Institute (National Research Fa- cilities and Equipment Center) grant funded by the Ministry of Education (RS-2024-00435727), We acknowledge the National Supercomputing Center for providing supercomputing resources and technical support (KSC-2023CRR--5555 KSC-2024-CRE-0198 KSC-2025-CRE-03400
An award for computer time was provided by the U.S. Department of Energy's (DOE) ASCR Lead- ership Computing Challenge (ALCC). This research used resources the National Energy Research Scientific Computing Center (NERSC), a DOE Office of Science User Facility, under ALCC award m4750-2024, and supporting resources the Argonne and Oak Ridge Leadership Computing Fa- cilities, U.S. DOE Office of Science user facilities at Argonne National Laboratory and Oak Ridge National Laboratory.
REFERENCES
Shun-Ichi Amari. Natural gradient works efficiently in learning. Neural computation, 10(2):251- 276. 1998.
Mehdi Azabou, Vinam Arora, Venkataramana Ganesh, Ximeng Mao, Santosh Nachimuthu, Michael Mendelson, Blake Richards, Matthew Perich, Guillaume Lajoie, and Eva Dyer. A unified, scalable framework for neural population decoding. Advances in Neural Information Processing Systems, 36:44937-44955 2023.
Mehdi Azabou, Krystal Xuejing Pan, Vinam Arora, Ian Jarratt Knight, Eva Dyer, and Blake Aaron Richards. Multi-session, multi-task neural decoding from distinct cell-types and brain regions. In The Thirteenth International Conference on Learning Representations, 2024.
Yoshua Bengio. Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning, In Proceedings of the 26th annual international conference on machine learning, pp. 41-48, 2009.
Gyorgy Buzsaki and Andreas Draguhn. Neuronal oscillations in cortical networks. science, 304 (5679):1926-199) 1929, 2004.
Ting Chen, Simon Korbliihh, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. International conference on machine learning, pp. 1597-1607. PmLR, 2020.
Saskia EJ de Vries, Jerome A Lecoq, Michael A Buice, Peter A Groblewski, Gabriel K Ocker, Michael Oliver, David Feng, Nicholas Cain, Peter Ledochowitsch, Daniel Millman, et al. A large-scale standardized physiological survey reveals functional organization of the mouse visual cortex. Nature neuroscience, 23(1):138-151, 2020.
Ian I. Dryden and Kanti V Mardia. Statistical shape analysis: with applications in R. John Wiley & Sons, 2016.
Timur Garipov, Pavel Izmailov, Dmitrii Podoprikhin, Dmitry P Vetrov, and Andrew G Wilson. Loss surfaces, mode connectivity, and fast ensembling of dnns. Advances in neural information pro- cessing systems, 31, 2018.
11

---

Published as a conference paper at ICLR 2026
Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach self-supervised learning. Advances in neural information processing systems, 33:21271-212844 2020.
Mert Gurbuzbalaban, Umut Simsekli, and Lingjiong Zhu. The heavy-tail phenomenon in sgd. In International Conference on Machine Learning, pp. 3964-3975. PMLR, 2021.
Zellig S Harris. Distributional structure. Word, 10(2-3)1146-16 1954.
Connor Johnson, Lisa N Kretsge, William W Yen, Balaji Sriram, Alexandra O'Connor, Ruichen Sky Liu, Jessica Jimenez, Rhushikesh A Phadke, Kelly Wingfield, Charlotte Yeung, et al. Highly unstable heterogeneous representations in vip interneurons of the anterior cingulate cortex. Molec- ular psychiatry, 2765):2602-2618, 2022.
Justin Johnson, Alexandre Alahi, and Li Fei-Fei. Perceptual losses for real-time style transfer and super-resolution. In European conference on computer vision, pp. 694-711. Springer, 2016.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.0888 2020.
Steven M Kay, Fundamentals of statistical signal processing: estimation theory. Prentice-Hall, Inc., 1993.
Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of neural network representations revisited. In International conference on machine learning, pp. 3519- 3529. PMIR, 2019.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton. Imagenet classification with deep convo- lutional neural networks. In Advances in neural information processing systems 25, 2012.
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Sori- cut. Albert: A lite bert for self-supervised learning of language representations. arXiv preprint arXiv: 1942, 2019.
Elizaveta Levina and Peter Bickel. Maximum likelihood estimation of intrinsic dimension. Advances in neural information processing systems, 17, 2004.
Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, and Tom Goldstein. Visualizing the loss land- scape of neural nets. Advances in neural information processing systems, 31, 2018.
Jiwei Li and Dan Jurafsky. Neural net models of open-domain discourse coherence. Martha Palmer, Rebecca Hwa, and Sebastian Riedel (eds.). Proceedings of the 2017 Conference on Em- pirical Methods Natural Language Processing, pp. 198-209, Copenhagen, Denmark, Septem- ber 2017. Association for Computational Linguistics. doi: 10.18653/11D11--00199 URL
Tsung- Yi Lin, Priya Goyal, Ross Girshick, Kaiming He. and Piotr Dollár. Focal loss for dense object detection. Proceedings of the IEEE international conference on computer vision, pp. 2980-2988, 2017.
Behnam Neyshabur, Hanie Sedghi, and Chiyuan Zhang. What is being transferred in transfer learn- ing? Advances in neural information processing systems, 33:512-523, 2020.
Jorge Nocedal and Stephen J Wright. Numerical optimization. Springer, 2006.
Aaron den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding. arXiv preprint arXiv:1807.03744 2018
Mansheej Paul, Surya Ganguli, and Gintare Karolina Dziugaite. Deep learning data diet: Finding important examples early in training. Advances neural information processing systems, 34:20596-20677, 2021.
12

---

Published as a conference paper at ICLR 2026
Ville Satopaa, Jeannie Albrecht, David Irwin, and Barath Raghavan. Finding a" kneedle" in haystack: Detecting knee points in system behavior. In 2011 31st international conference on distributed computing systems workshops, pp. 166-171. IEEE, 2011.
Steffen Schneider, Jin Hwa Lee, and Mackenzie Weygandt Mathis. Learnable latent embeddings for joint behavioural and neural analysis. Nature, May 2023. ISSN 1476-4687. doi: 10.1038/ 141586-023-060311 URL/tttps://aol
org/ 00t.///455860022-000000000
Koustuv Sinha, Robin Jia, Dieuwke Hupkes, Joelle Pineau, Adina Williams, and Douwe Kiela. Masked language modeling and the distributional hypothesis: Order word matters pre-training for little. Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (eds.), Proceedings of the 2021 Conference Empirical Methods in Natural Language Pro- cessing, pp. 2888-2913, Online and Punta Cana, Dominican Republic, November 2021. Asso- ciation for Computational Linguistics. doi: (0.18853//1/20111 http://222....pmaii......m 2300. URL https: //aclantholooy.ccrrt.
emn p ma in. 230/
Yonghao Song, Bingchuan Liu, Xiang Li, Nanlin Shi, Yijun Wang. and Xiaorong Gao. Decoding natural images from eeg for object recognition. arXiv preprint arXiv:2308. 13234, 2023.
Ian Tenney, Dipanjan Das, and Ellie Pavlick. BERT rediscovers the classical NLP pipeline. In Anna Korhonen, David Traum, and Lluís Mårquez (eds.). Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. pp. 4601, Florence, Italy, July 1452. URL https:
2019. Association for Computational Linguisties. doi: 10.
Jarkko Venna and Samuel Kaski. Neighborhood preservation in nonlinear projection methods: An experimental study. In International conference on artificial neural networks, pp. 485-491. Springer, 2001.
Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE transactions on imuge processing, 13(4):600- 612, 2004.
Di Wu, Siyuan Li, Jie Yang, and Mohamad Sawan. Neuro-bert: Rethinking masked autoencoding for self-supervised neurological pretraining. arXiv preprint arXiv:2204. 12440, 2022.
Shaoyun Yu, Chanyuan Gu, Kexin Huang, and Ping Li. Predicting the next sentence (not word) in large language models: What model-brain alignment tells about discourse comprehension. Science advances, 10(21) :ead))ead,77 2024.
Hang Zhao, Orazio Gallo, luri Frosio, and Jan Kautz. Loss functions for image restoration with neural networks. IEEE Transactions on computational imaging, 3(1):47-57, 2016.
Yunxi Zhao Dong Nie, Geng Chen, Xia Wu, Daoqiang Zhang, and Xuyun Wen. TARDRL: Task- Aware Reconstruction for Dynamic Representation Learning of fMRI. In proceedings of Medical Image Computing and Computer Assisted Intervention - MICCAI 2024, volume LNCS 15011. Springer Nature Switzerland, October 2024.
Xinlin Zhuang, Jiahui Peng. Ren Ma, Yinfan Wang, Tianyi Bai, Xingjian Wei, Qiu Jiantao, Chi Zhang, Ying Qian, and Conghui He. Meta-rater: A multi-dimensional data selection method for pre-training language models. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. (0856-10894) 2025.
13

---

Published as a conference paper at ICLR 2026
A DETAILED DATA SPLITTING AND EMBEDDING INITIALIZATION
A. TRIAL-LEVEL TRAINIVALIDATIONTTESS SPLITS
For downstream tasks (movie reconstruction and drifting gratings decoding), train/validation/test splits were performed within each session (i.e., within each animal recording) at the trial level.
Specifically, stimulus presentations were temporally divided into non-overlapping trial segments independently for each session:
Natural Movie One: 80% train / 10% validation / 10% test (shuffle = False; temporal order preserved)
Drifting Gratings: 70% train / 10% validation / 20% test (shuffle True)
Thus, the animal may appear in train/validationntest splits, but distinct, non-overlapping stim- ulus trials are used. This design prevents temporal leakage while evaluating generalization across unseen stimulus segments within each biological subject. Importantly, downstream evaluation does not involve animal-level train/test separation.
A.2 PRETRAINING-FIETUUING CROSS-ANINA ANIMAL SEPARATION
In contrast to downstream splits, pretraining and finetuning datasets are strictly separated at the animal level.
In the Allen Brain Observatory dataset, each animal belongs to a single Cre driver line. Pretraining uses sessions from inhibitory/moduuatory Cre lines (SST, VIP, PVALB, NTSRI), whereas finetuning uses sessions from excitatory Cre lines. Because Cre lines are disjoint at the animal level, the pretraining and finetuning animal sets are completely non -veroaell....ng.
This Cre-line-based partition corresponds to the "held-out animals" referenced in our reviewer re- sponses: animals used during finetuning were
during pretraining.
A.3 UNIT EMBEDDING INITIALIZATION AND TRANSFER
Because pretraining and finetuning use disjoint animal sets, their recorded neurons do not overlap. Consequently, pretrained unit embeddings cannot be reused during finetuning.
During finetuning, all unit embeddings are randomly initialized from (O, 0.02).
Only the encoder (Perceiver) weights are transferred from pretraining. During finetuning, all pa- rameters numeters-iinlludng the newly initialized unit embeddings and the transferred encoder weights- are trained end-to-end.
Thus, cross-animal transfer arises solely from shared encoder representations rather than reused neuron-specific embeddings.
B DESIGN CHOICES AND BASELINE SELECTION
We focus our evaluation on architectures with comparable capacity for high-dimensional visual reconstruction. Many recent SSL methods in neuroscience are designed for different objectives. For instance, while contrastive methods like CEBRA (Schneider al. (2033) are effective for behavioral alignment, our empirical evaluation confirmed that their low-dimensional embeddings are suboptimal for direct pixel-level generation. Similarly, masked autoencoding methods such Neuro-BERY (Wu et al. (2022)) were excluded due to the lack of an official implementation and insufficient architectural capacity for high-resolution image generation. therefore selected POYO+ (Azabou et al. (2024)) our primary comparative model for its flexible architecture that can be scaled for dense prediction tasks.
To support visual reconstruction objective (304 608 pixel images), scaled the architec-
ture to use 1024-dimensional embeddings, a substantial increase from the 64 dimensions used the original work for classification. This architectural parity ensures fair comparison: both our method
14

---

Published as a conference paper at ICLR 2026
and the from-scratch baseline operate with identical encoder-decoder capacity. This design choice allows us to isolate the contribution of our cell sell-pattern-award SSL approach from architectural ad- vantages, providing rigorous evaluation of our core hypothesis.
C JUSTIFICATION FOR DATA PARTITIONING CRITERIA
C.I DETAILED DESCRIPTION ON CRE LINES
Cre Line
Type Excitatory
Functional Role
EMX1
Pan-excitatory, broad cortical excitatory neurons Pan-excitatory, glutamatergic projection neurons Upper layer excitatory, intracortical connections Layer 4 excitatory, thalamic input recipients
SLC17A7
Excitatory
CUX2
Excitatory
RORB
Excitatory
SCNNIA
Excitatory
Layer 4 excitatory, primary sensory processing
NRSAI
Excitatory
Layer 4 excitatory, sensory feature detection
RBP4
Excitatory
Layer 5 excitatory, subcortical projections
FEZF2
Excitatory
Deep layer excitatory, long-range projections
TLX3
Excitatory
Layer 5 excitatory, corticotectal projections
NTSRI
Excitatory
Layer 6 excitatory, corticothalamic feedback
VIP
Inhibitory
Disinhibitory interneurons, modulate inhibition Somatostatin interneurons, lateral inhibition
SST
Inhibitory
PVALB
Inhibitory Parvalbumin interneurons, fast spiking, timing
Table S1: Cre driver lines in the Allen Brain Observatory dataset
This table provides detailed information the 13 Cre driver lines from the Allen Brain Observa- tory dataset used this study. A central premise of our work is that the heterogeneous nature of neural populations is critical factor for self-supervised learning. This table offers comprehensive overview of this heterogeneity by detailing the specific functional roles and types of the neuronal subpopulations available in the dataset.
Each Cre Line targets specific type of neuron based on the expression a particular gene, allowing for cell- speeciii measurements. These are broadly categorized into two main Types:
Excitatory neurons: These neurons, such as Emx1 and Slcl 7a7. typically release neu- rotransmitters like glutamate that increase the likelihood of postsynaptic neuron firing. As detailed in the Functional Role column, they are involved in wide range of activi- ties, from broad cortical activation to specific roles in sensory processing (e.g., Scnnla in Layer 4) and forming long-range projections to other brain areas (c.g., Rbp4 4 in Layer 5),
Inhibitory neurons: These interneurons, such
and Pvalb, typically release neuro- transmitters like GABA that decrease the likelihood of a postsynaptic neuron firing. Their functional roles are often modulatory, involved in processes like lateral inhibition (SST), network disinhibition (Vip), and regulating the precise timing of neural activity (Pvalb).
As described in the main text, data-driven selection method identified four lines (SST, VIP, PVALE, and NTSR1) from this diverse catalog as having the "predictable' dynamics suitable for our pre-training objectives. This table provides the full context for that selection, detailing the characteristics of all potential cell types considered in this work.
C.2 VALIDATION OF SKEWNESS AND KURTOSIS AS PREDICTABILITY INDICATORS
Justification for Higher-Ordee Order Statistics Our central hypothesis is that neurons with different functional roles exhibit distinct statistical signatures their activity patterns. To create a principled data partition for curriculum, sought metrics that could reliably separate these populations. This analysis provides the empirical justification for our choice skewness and kurtosis over sim- pler, lower-order statistics.
15

---

Published as a conference paper at ICLR 2026
b
Preliruue 2en Unpediostie
6:39
Density
6:35
50
Skowness (log scale)
Kurtosis llng scale)
Figure S1: Statistical distributions of predictable and unpredictable neural subpopulations. Kernel Density Estimate (KDE) plots for (a) skewness and (b) kurtosis of calcium traces. The "Predictable" group (pink), selected for pre-training, exhibits distributions sharply concentrated at low values for both metrics. contrast, the "Unpredictable" group (cyan) shows broad, heavy-tailed distribu- tions. This clear statistical separation validates data-driven criteria for identifying stable neuron populations suitable for self-supervised learning. This statistical separation captures sub-types of neurons that may have more regulatory functions: inhibitory interneurons (SST, VIP, PVALB) and modulatory excitatory neurons (NTSRI), which all may be more involved in network stabilization rather than atimulus-specifie responses.
Interpreting the Metrics Neuroscience Context In the context of calcium imaging traces, skewness and kurtosis serve powerful proxies for the temporal dynamics of neuron's 's activity:
Skewness measures the asymmetry of the activity distribution. A low skewness (close to zero) implies symmetrie, quasi-Gaussian distribution, characteristic of neurons with stable baseline activity that fluctuates evenly. contrast, high positive skewness indicates a distribution with long right tail, the statistical fingerprint of neuron that is mostly quiescent but fires in sparse, high-amplitude positive bursts.
Kurtosis measures the "tailedness" of the distribution, or the prevalence of extreme out- liers. Low kurtosis is characteristic of Gaussian-like activity. High kurtosis indicates a "spiky" or leptokurtic distribution, where extreme events (large calcium transients) are far more common than would be expected from random noise. This is hallmark of event- driven, stimulus-encoding neurons.
Empirical Validation of Statistical Separation The distributions shown in Figure SI confirm that these metries provide a clear and robust separation between our two target populations.
Panel (a) shows that the 'Predictable' group (pink) has a skewness distribution sharply peaked low values, consistent with symmetric activity patterns. The 'Unpredictable" group (cyan), however, is broadly distributed across much higher skewness values, con- firming a burst-like firing pattern.
Panel (b) reveals even starker separation for kurtosis. The 'Predictable' group's dis- tribution is almost entirely concentrated at low values, indicating a near-total absence of extreme outlier events. This provides strong evidence that these neurons exhibit highly regular and constrained dynamics.
Functional Interpretation This clear statistical separation aligns directly with the known func- tional roles of the underlying neuron types. The low-skew, low-kurtosis profile is the statistical signature neurons engaged in network stabilization and modulation-the very neurons we iden- tify as 'predictable' (SST, VIP, PVALB, NTSR1). Conversely, the high-skew, high-kurtosis profile is the classic signature of sparse, stimulus stimulus-encoding neurons that fire selectively and powerfully. This strong correspondence between data-driven statistical signature and known biological function validates our selection criteria as principled method for identifying ideal neuron candidates for self-supervised me-training.
16

---

Published as a conference paper at ICLR 2026
How predictable lines were chosen. For each of the 13 CRE lines, skewness and kurtosis were computed from its neural activity distribution before training. A single knee (NTSRI) was estimated on the per-line statistic distribution, yiclding four predictable lines used entirely for pretraining; the remaining lines were reserved exclusively for split; mimmak/sessions/eeurons overlap across partitions.
This is a line-level dataset
Number of Cells
Event Rate
Fano Value
CRE Line
Median
Std
Median
Std
EXMI IRES CRE
7537
1.021
0.122
103869.88 8977
1611.701
SLC17A7 IRES2 CRE
7736
1.046
0.149
103676.897
2260.452
CUX2 CREERT2
10275
1.034
0.182
103686.88 8.8
2314.886
RORB IRES2 CRE
5009
1.055
0.291
103464.8 896
3461.491
SCNNIA TG3 CRE
1200
1.078
0.221
103217,894
2426.047
NR5AI CRE
2135
1.125
0.361
102710.88*
4346.653
RBP4 CRE KL100
1611
1.121
0.237
102770.890
2962.409
FEZF2 CREER
587
1.079
0.142
103497.88 .896
2182.647
TLX3 CRE PL566
1524
1.075
0.126
103190.893
1473.551
NTSRI CRE GN220
1239
1.041
0.0981
103566.899
1149.701
VIP IRES CRE
639
1.379
0.309
99914.863
3951.127
SST IRES CRE
573
1.183
0.240
101967.88.
2844.855
PVALB IRES CRE
245
1.332
0.308
100983.849
3995.032
Table S2: Event rate and Fano value statistics for each CRE line
In this section, we provide the empirical justification for selecting skewness and kurtosis as the pri- mary statistical indicators for identifying predictable neural subpopulations. We conducted parative statistical analysis the calcium trace signals between the predictable and unpredictable neuron groups, defined by the criteria in the main text. The results are summarized in Table S22
As shown in Table first and second-order statistics, namely the mean and variance the activity, showed statistically significant differences between the populations (p=0.347 and p=0.281, respectively). This suggests that simpler metrics related to the overall magnitude or spread of neural activity are insufficient to distinguish between neurons with different response pattern regularities.
Number of Cells
Skewness
Kurtosis
CRE Line
Median
Std
Median
Std
EXM1 IRES CRE
7537
5.637
6.169
88.966
887.759
SLC17A7 IRES2 CRE
7736
5.132
4.380
63.847
132.297
CUX2 CREERT2
10275
5.504
4.644
79.245
186.898
RORB IRES2 CRE
5009
6.283
5.300
88.748
443.990
SCNNIA TG3 CRE
1200
7.240
15.235
103.458
3027.682
NR5AI CRE
2135
6.159
8.254
69.922
1286. 154
RBP4 CRE KL100
1611
7.395
14.528
94.758
2377.199
FEZF2 CREER
587
5,108
3.763
55.862
96.430
TLX3 CRE PL56
1524
6.133
3.910
76.118
105.617
NTSRI CRE GN220
1239
2.453
3.579
22.616
83,209
VIP IRES CRE
639
3.507
1.770
19,145
22.122
SST IRES CRE
573
2.075
3.007
12.932
259.785
PVALB IRES CRE
245
1.991
1.525
8.258
17.978
Table S3: Skewness and kurtosis statistics for each CRE line
In stark contrast, higher-order statistics (Table S3 revealed dramatic and highly significant differ- ences. The predictable subpopulation exhibited low average skewness (1.87) and kurtosis (7.32), characteristic of more symmetric and less outlier-prone signal distributions. Conversely, the un- predictable subpopulation showed extremely high average skewness (9.84) and kurtosis (148.51),
17

---

Published as a conference paper at ICLR 2026
indicating heavily right-tailed and sparse, spiky activity patterns. These differences statisti- cally significant to a very high degree (p < 0.001).
This analysis empirically confirms that skewness and kurtosis exceptionally effective and reliable indicators for diffcrentiating neural populations based on their activity patterns, far more so than lower-order statistics. This provides strong validation for methodological choice use these metrics as the core selection criteria within the POYO-CAP framework.
Show
tacted StanTH
CRT Lines Extrted by kuttosn)
Figure S2: Data-driven threshold determination for predictable neuron selection. (a) Distri- bution of mean skewness values across CRE lines, sorted in ascending order. The knee detection algorithm identified natural breakpoint at skewness = 3.51 (red dashed line), corresponding to the NTSRI CRE line (red circle). CRE lines below this threshold exhibit stable, near-Gaussian activ- ity patterns suitable for self-supervised pretraining. (b) Distribution of mean kurtosis values across CRE lines, showing a similar elbow kurtosis 22.62 (red dashed line), again the NTSR1 bound- ary. The sharp increases beyond these breakpoints indicate the transition from predictable regulatory neurons to highly variable, Himmulu--oontngen populations. This objective approach ensures bio- logically grounded selection criteria rather than arbitrary thresholds.
Algorithm S1 Find Elbow (Knee) Point by Maximum Gradient
Input: Vector of values y [y1, ... ya
Output: Knee index
Compute consecutive gradients di +- M641 y: for 1,...,
Find index k 4- arg max; Si b position of largest gradient return &
knee the point before the sharpest rise
To objectively determine the threshold values for predictable neuron selection, we employed a knee detection algorithm on the distribution skewness and kurtosis values across CRE lines. For each metric, we calculated the gradient between consecutive CRE lines (sorted by their respective mean values) and identified the point preceding the sharpest increase as the elbow point (See algorithm ST). This approach revealed natural breakpoints at skewness < 3.51 and kurtosis < 22.62, corre- sponding to the NTSR1 CRE line as the boundary case (Figure S2). CRE lines below these thresh- olds (SST, VIP, PVALB, and NTSRI) exhibited consistently low and stable activity statistics, while those above showed sharp increases indicative of more variable, stimulus-drive responses. This data-driven approach ensures that selection criteria grounded in the natural distribution of neural activity patterns rather than arbitrary cutoffs, providing an objective foundation for distin- guishing predictable from unpredictable neural subpopulations.
Note: CRE line labels were only used to define the domain-level split (which lines go to pretraining finetuning) and were not used inside training losses, model selection, or evaluation.
18

---

Published as a conference paper at ICLR 2026
D THEORETICAL JUSTIFICATION FOR PRIORITIZING PREDICTABLE NEURONS IN TRAININGN
To understand the mechanisms behind the improved performance of our SSL methodology, we con- ducted a theoretical and empirical analysis comparing the properties of two representative neural populations: 'Predictable' (VIP inhibitory neurons) and "Unpredictable" (Sennla excitatory rons). This analysis reveals that the statistical and temporal characteristics of 'Predictable" neurons create more favorable learning scenario for SSL models.
D.1 ENHANCED TEMPORAL STRUCTURE AND INFORMATION CONTENT
Self-supervised learning on time-series data fundamentally relies on exploiting temporal regulari- ties. Our analysis shows that 'Predictable' neurons possess a much richer and more stable temporal structure.
Temporal Predictability As shown in the autocorrelation plot (Fig. S3d), the signal from pre- dictable neurons maintains a stronger correlation with its recent past compared to unpredictable neurons. This slower decay indicates that each time point contains more information about its neigh- bors, providing more robust signal for temporal contrastive learning tasks (Oord et al. (2018)).
Reconstruction Fidelity From an information theory perspective, signals that are easier com- press and reconstruct are more amenable to representation learning. We quantified this using the Cramér-Rao Lower Bound (CRLB), a theoretical minimum for estimator variance (Kay (I993). The analysis (Fig. S3c) shows that the mean CRLB for predictable neurons is 0.0476, while it is 1444 for unpredictable neurons. This suggests that predictable neurons can be reconstructed with 3.03 times greater theoretical efficiency, providing more reliable learning signal with lower intrinsic noise.
Signal Dynamies The power spectrum (Fig. $3
reveals that predictable signals have their power concentrated in low-frequency bands, indicative of smooth and continuous dynamics (Buzsaki & Draguhn (2004)). In contrast, unpredictable signals have flatter power spectrum, closer to white noise, signifying less discernible temporal structure.
D.2 FAVORABLE STATISTICAL DISTRIBUTIONS AND LEARNING STABILITY
Beyond temporal structure, the underlying statistical distribution of the data dramatically impacts the stability and efficiency of the learning process, particularly for gradient-based optimization.
The activity of unpredictable neurons follows sparse, heavy-tailed distribution, as visualized in the histogram (Fig. S3p). This is quantitatively confirmed in Fig, S3h, where these neurons exhibit extreme skewness (mean: 10.65) and kurtosis (mean: 475.93). Such distributions, with rare but high-amplitude events, can lead to unstable gradients and cause the model be overly influenced by outliers (Gurbuzbalaban al. (202T)). In contrast, the predictable neurons are
Distributional Properties
(mean skewness: 2.56, mean kurtosis: 12.98), providing a more well bell-behavee statistical foundation for learning.
Learning Stability We analyzed the stability of the learning problem by computing the condi- tion number of the data's covariance matrix, which reflects the curvature of the loss landscape. A high condition number implies landscape with sharp, narrow valleys, making it difficult for optimizers to converge (Nocedal & Wright (2006)). The condition number for unpredictable neu- 627.49, whereas it only 67.15 for predictable neurons (Fig, S3). This demonstrates that the learning problem posed by unpredictable neurons approximately 9.34 times more ill-conditioned, harder to optimize, than that of predictable neurons.
D.3 SYNTHESIS: PREDICTED SSL EFFICIENCY
Methodology for Composite Score To synthesize these multifaceted properties into a single met- ric, formulated composite score for predicted SSL efficiennc This score is weighted average
19

---

Published as a conference paper at ICLR 2026
a
Example Neural
Activity Distribuzions (lo scale)
C
Reconstruction (CRLB)
3.0
Redksacl unpreskrate
121
Poodctatee
2.3
2012
20"
30.
10
d
Average on
Power Spectral Dons by
f
Temporal Regu lanty Scores
Redictable Unpradirhbbie
Wedicmble
ieeicttbte
10n1
Lip Inprwocetable
00
Roser
00200
7300
9200
2300
1.0
JU
0.64
Frequincy 0:21
Regularit
Statistical Selection Criteria
Learning Stabiitty iCondition Numberi
Predicted SS. Efficiency
Doweiiti Marn Mec
Skow thoechold
Kart Uhwddoidd
0.0
2.5
5.0
7.5 SaRATIEE
12.5
15.0
Predciable
Unpred:catre
Fredic Ungrelstatie Ris)
Figure S3: Theoretical Analysis of Neural Signal Properties for Self-Supervised Learning (SSL) Efficiency. This figure provides comprehensive comparison between two distinct types of neural activity: 'Predictable' signals derived from inhibitory VIP neurons, which exhibit quasi- Gaussian distributions, and "Unpredictable" signals from excitatory Scnnla a neurons, characterized by sparse, skewed distributions. The analysis dissects why 'Predictable' neurons serve as a more effective dataset for SSL pre-training. (a) Example calcium signal traces (AF/F) over 350 sec- onds. The predictable trace (blue) shows smoother fluctuations, while the unpredictable trace (red) characterized by sparse, high-amplitude bursts. (b) Log-scale histograms of signal activity distri- butions, highlighting the heavy-tailed, skewed nature unpredictable signals compared to the more centered predictable signals. (c) Boxplot of the theoretical reconstruction
bounds (Cramér-Ran Lower Bound, CRLB). Predictable neurons show a significantly lower and tighter error distribution, indicating they are more reliably encoded. (d) Average autocorrelation functions. Predictable sig- nals exhibit slower decay in autocorrelation, signifying more persistent temporal structure. (e) Power spectral density (PSD) analysis. Predictable signals have more power concentrated at lower frequencies, consistent with smoother dynamics. (f) Distribution of temporal regularity scores. (h) Scatter plot kurtosis versus skewness for individual neurons. Predictable neurons (blue) largely fall within the statistical selection criteria (red dashed lines), whereas unpredictable neurons (orange) do not. (i) Learning stability, quantified by the condition number of the data covariance matrix. The much higher condition number for unpredictable data indicates more ill-conditioned and unstable learning problem. (j) A composite score predicting overall SSL pre- pre-trrainin training efficiency, integrating metrics from the preceding panels. Pre-training with predictable data first is predicted to be substan- tially more efficient.
20

---

Published as a conference paper at ICLR 2026
of five key factors derived from our preceding analyses: (1) Reconstruction Fidelity, based on the inverse of the theoretical error bound (CRLB); (2) Learning Stability, derived from the inverse of the learning problem's condition number; (3) Temporal Regularity, measured by the signal's autocorrelation and consistency; (4) Information Content, based on signal entropy; and (5) Fa- vorable Statistical Properties, rewarding low skewness and kurtosis. These factors weighted (0.3, 0.25, 0.2, 0.15, and 0.1, respectively) to reflect their relative importance in creating learnable, information-rich dataset.
Predicted Efficiency and Rationale The resulting composite score (Fig, predicts that pre- training on a dataset of predictable neurons first is 2.85 times more efficient than starting with unpredictable neurons. This theoretical result strongly supports our empirical findings and provides a clear rationale for our pre-training strategy: by first learning from the stable, information-rice and well-conditioned 'predictable' neurons, the model establish robust foundational representation before being fine-tuned on more complex, sparse signals.
E THEORETICAL JUSTIFICATION FOR CURRICULUM LEARNING
To provide a theoretical basis for our hybrid pre-training objective, particularly the use of a simple auxiliary task (drifting gratings) warm-up, we conducted simulation of currieulum learning principles. We defined sample difficulty based on local variance, distance to the manifold center, and local density, and simulated four training strategies: Easy-to-Hard, Hard-to-Easy, Random, and Mixed.
The results, summarized in Figure S4 unequivocally support an Easy-to-Hard curriculum. This strategy led to the highest final performance for both data types, achieving 1.43x better results for predictable data and 1.19x for unpredictable data compared ensuring superior training stability. Notably, the absolute performance and stability achieved on predictable data (0.9967 performance, 0.9998 stability) were substantially higher than on the more volatile unpredictable data (0.6945 performance, 0.9340 stability). This highlights that while an optimal curriculum is always beneficial, the intrinsic quality of the "easy" examples ultimately de- termines the robustness of the learned foundation. This analysis provides a principled foundation for our training methodology, where the simple DG task serves as the initial "easy" stage that stabilizes the model, and the broader predictable-fiis pre-training represents a macro-level application of the same principle.
random ordering, while also
F DETAILED WEIGHT DYNAMICS ANALYSIS
To empirically validate the "representational scaffold" hypothesis, we analyzed the model parame- ters before and after fine-tuning. We computed the relative L2 norm of the weight changes in the PerceiverIO encoder versus the task-specific readout heads.
Encoder Stability. The encoder, responsible for mapping neural activity the latent space, showed minimal change during the fine-tuning phase on unpredictable neurons. The relative weight change was 183%, indicating that the features learned from the predictable subset are robust and gener- alizable to the broader population. The stability of encoder norms (z>222.909) suggests the model stays within the same optimization basin found during pre-training ((arripov et al. 2018)).
Readout Adaptation. Conversely, the readout layer demonstrated dramatic specialization. The magnitude of the readout biases increased x2.4x. This confirms that transfer occurs through opti- mization geometry (Neyshabur al.) 2020)): the encoder maintains the stable manifold, while the readout adapts to the specific statistics and noise profile of the unpredictable neurons.
G A UNIFIED THEORETICAL FRAMEWORK FOR POYO-CAP
Our empirical results, particularly the successful scaling of our model, are underpinned by cohesive theoretical framework derived from the principles of representation and curriculum learning. This framework explains why the strategic use neural heterogeneity is not merely an effective heuristic but a principled approach to building scalable models of neural dynamics.
21

---

Published as a conference paper at ICLR 2026
Lewrlkg Oarve Pedditadi Duts
Leardiig Carsee Leppeiieele Dats
Teaiielerwwe by Stottny
Bay Sohar
Texbm
Sandes
19030 -300 6080
1830 x000 13/033
20120
20er2 USC
0
EX-
H4. To Bay
M2-
94
Figure S4: Theoretical Simulation of Curriculum Learning Strategies. This figure simulates the effect of different data ordering strategies model performance and training stability for both pre- dictable and unpredictable neural data. (a, b, c) Learning curves and final performance comparison. The "Easy Hard" curriculum (green) achieves the fastest convergence and highest final perfor- mance. (d, e, f) Training stability analysis. The "Easy to Hard" strategy maintains high stability, from significant initial instability. These results pro- vide strong theoretical justification for our predictable-first, curriculum-base pre-training strategy.
while the "Hard to Easy" approach (red) suffers
The Representational Advantage of Predictable Data The success of any learning algorithm is contingent on the quality of the data representation. Our analysis reveals that predictable neu- rons provide a fundamentally superior substrate for representation learning. They induce a smooth, convex-like loss landscape (Fig. 2). which makes optimization well-posed problem. Furthermore, the representations learned from this data are more efficient and structured, evidenced by their sig- nificantly lower intrinsic dimension and more organized latent manifold (Fig. (S3). This efficiency is rooted in their higher information content, as quantified by Fisher Information (Table 2) allowing the model to learn robust representation from smaller effective dataset size.
The Optimization Advantage of a Predictable-First Curriculum Beyond the static quality of the data, the order of presentation is critical. Our theoretical simulations of curriculum learning (Fig. S4) demonstrate that an "Easy-to-Hard Hard" strategy is optimal, maximizing both final performance and training stability. Starting with easy examples- those with clear, low-variance signals-alle allows the model to establish a stable foundational representation. The predictable neurons, with their inherent statistical regularity, serve as the ideal "easy" examples in the context of neural data.
The Synergy of Representation and Curriculum The remarkable success and scal- ability of PΟYO-CAP can be understood as direct result of the synergy between these two prin- ciples. Our method does not merely use a better curriculum; it applies the optimal curriculum to the optimal data. By starting with predictable neurons, we solve a well-posed representation learning problem in maximally stable manner. This establishes robust initial model that is well- prepared to subsequently learn the -reigidi complex features from the unpredictable data during fine-tuning. This unified view provides rigorous mathematical and conceptual foundation for empirical scaling results (Fig. 5) explaining why POYO-CAP unlocks consistent performance gains with increasing model capacity while other approaches stagnate or fail.
Synthesis:
22

---

Published as a conference paper at ICLR 2026
Algorithm S2 UNet Decoder with Latent Injection
Input: Latent vector 2 E I
Output: Reconstructed frame & E 12 128
< reshape(z, [d, 1, 1]
for each upsampling stage i - 1....., do
8/ f- Linear(z) - reshape([c,, his wal)
f- Upsample(z, scale 2)
I 4 (2)
concat( [z. s:])
Σ f 1
< ExtraUp(x) 32 64 x 128
H SKI--CONNECTION UNET DECODER ARCHITECTURE
I SPECIALIZED Loss COMPONENTS
To ensure high-fidelity image reconstruction, we employ composite loss function with several spe- cialized components. We adapt Focal Loss to regression task to emphasize challenging pixels and refine fine details (Eq. [ a and 7y are set as empirically. To preserve high-frequency structure, we introduce a frequency-domait loss using the Fast Fourier Transform (Eq. 5). Perceptual simi- larity is further promoted through both an SSIM loss (Eq, (6 and a perceptual loss computed as the mean-squared error (MSE) between feature maps of Imaeainet-reerraied AlexNet.
Specifically, we extract activations from the first four convolutional blocks of the AlexNet (Krizhevsky al. (2012) feature extractor ('layer=3' in the PyTorch implementation) after Ima- geNet normalization (mean [0.485, 0.456, 0.406], standard deviation [0.229, 0.224, 0.225)) (Eq. 7
REPRESENTATION ANALYSIS
For qualitative visualization, nigh-dimensione latent embeddings were projected into two- dimensional space using t-SNE (t-distributed Stochastic Neighbor Embedding). We then quantified the global properties of these spaces using three metrics: (1) Intrinsic Dimension (ID) to measure the efficiency of the representation, (2) Procrustes disparity, and (3) Centered Kernel Alignment (CKA) to assess the geometric dissimilarity between latent spaces learned by different models. Finally, to specifically quantify the preservation of local temporal structure, we implemented Temporal Neighborhood Preservation analysis. For each data point, we identified its k=10 nearest neighbors in the temporal domain (by frame index) and its k=10 nearest neighbors in the t-SNE latent space (by Euclidean distance). The similarity between these two sets neighbors was measured using the Jaccard index, and the score was averaged across all points in the sequence.
L088toral a(1 1y
LosSpFT |||F(}) - [F
(5)
LOSSSSIM SSIM(y.
(6)
LosSpercephnal (b(()) <
K DECODER ARCHITECTURE SELECTION
To validate the architectural choice of our visual decoder, we conducted a comparative analysis between proposed U-Net decoder and standard Transformer-based decoder. To ensure fair
23

---

Published as a conference paper at ICLR 2026
comparison, the Transformer decoder was capacity-matched maached (i.e., approximately equal number of total parameters) to our U-Net implementation.
The results revealed significant performance gap: the Transformer decoder achieved Movie SSIM of 0.48, substantially lower than the 0.593 achieved by our U-Net decoder. This performance difference highlights the importance of the spatial inductive bias inherent in convolutional architec- tures (U-Net) for dense pixel prediction tasks. While Transformers excel at modeling long-range dependencies, they lack the intrinsic local connectivity required for high-fidelity image reconstruc- tion from sparse neural embeddings, particularly in the limited-data regime of biological recordings. Consequently, we adopted the U-Net architecture as the optimal choice for our decoding framework.
L TASK-SPECIFIC NEURAL REPRESENTATION ANALYSIS
a
b
70
-20
-40
340
299
-50
-40
-20
20
10 60 60 -0.62
-0.50
-0.56
-0
-0.52
Componortt 2
Cumponceet
Figure S5: Task-specific adaptation of the learned latent manifold. Visualization of the final latent representations from our fine-tuned model For this classification task, the model learns geometrically structured representation with distinct, well-separated clusters corresponding the 8 stimulus directions. (b) Movie Decoding: For this reconstruction task, the model learns a continuous, non-linear manifold that captures the temporal trajectory of the movie frames.
different tasks. (a) Drifting Gratings:
The learned representations for the two main downstream tasks exhibit fundamentally different ge- ometries, confirmed by high Procrustes disparity (0.95) and low Centered Kernel Alignment (CKA. 0.18). This demonstrates that our pretrained model does not use a rigid, one-size-fits-all representation, but rather adapts its internal structure the specific demands of each task. For the drifting gratings classification task (Figure [S5,,, the model learns to organize its representations into discrete, maximally separated clusters to optimize for classification. In contrast, for the movie decoding reconstruction task (Figure (S5)). it learns a continuous, non-linear manifold that effec- tively represents the temporal flow of the visual experience. This adaptability highlights the model's ability to learn the true underlying structure of given neural decoding problem.
M PRETRAINED KNOWLEDGE IS DISTRIBUTED ALONG ENCODER COMPONENTS
To investigate where pretrained information is stored, we conducted ablation experiments by selec- tively freezing encoder components during finetuning, Our results reveal that the learned represen- tation distributed, not localized. Partially freezing any single component led to catastrophic recon- struction failures, whereas surprisingly, freezing the entire encoder better preserved spatial content (Figure (S6). This suggests that the pretrained representation relies on coordinated interactions across the entire encoder and requires holistic, rather than modular, adaptation during fine-tuning.
24

---

Published as a conference paper at ICLR 2026
Figure S6: Encoder freezing analysis. Freezing any subset of encoder layers degrades high- frequency detail, indicating pretrained knowledge is distributed across the entire encoder. (a) Ground truth, (h) ours (did not freeze encoder), (c) full encoder freezing, (d) partial encoder freezing (former layers only). partial encoder freezing (middle layers only). (f) partial encoder freezing (latter layers only).
N
USE OF LARGE LANGUAGE MODELS IN MANUSCRIPT PREPARATION
We acknowledge the use of a large language model (Google's Gemini) for language editing and refinement during the preparation this manuscript. The model was employed to improve gram- mar, clarity, and conciseness. The authors meticulously reviewed and revised all model-generater suggestions ensure scientific accuracy and preserve the original meaning. All conceptual work, experimental results, and scientific conclusions presented herein are entirely the work of the authors.
25


Reviews:
[{"review_id": "5v28pYkrrC", "reviewer": "Reviewer_xiui", "summary": "This paper introduces POYO-SSL, a self-supervised learning framework that addresses neural data heterogeneity by pre-training on statistically predictable neurons identified via skewness and kurtosis metrics before fine-tuning on unpredictable populations. Applied to calcium imaging data from the Allen Brain Observatory, the method achieves 12–13% relative gains in decoding dynamic visual experiences, enables high-fidelity movie reconstruction without external stimuli, and demonstrates stable scaling with model size, turning neural variability into an asset for robust representation learning.", "questions": "Weaknesses", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Biologically Grounded Data Selection: The method innovatively leverages higher-order statistics to prioritize predictable neurons (e.g., inhibitory interneurons like SST/VIP), transforming neural heterogeneity from a challenge into an advantage for stable pre-training and improved data efficiency (1.98×gain).  \n\n2. Scalable and Task-Adaptive Architecture: POYO-SSL enables monotonic performance scaling with model size and supports diverse decoders (e.g., Skip-Connection U-Net for movie reconstruction), achieving high-fidelity results without task-specific labels.", "weaknesses": "1. The method uses skewness ≤ 3.51 and kurtosis ≤ 22.62 as thresholds to partition neurons into predictable and unpredictable groups via a knee-detection algorithm. However, these thresholds are applied as a fixed, universal criterion without sensitivity analysis or validation across diverse datasets. The paper states: \"These thresholds were determined a priori as a single, fixed criterion to partition the dataset, not as a tunable hyperparameter, which is why a sensitivity analysis was not performed\". This risks overfitting to the Allen Brain Observatory dataset and limits generalizability. A robustness analysis (e.g., varying thresholds) would strengthen the approach.\n\n2. The skip-connection U-Net decoder is introduced for high-fidelity movie reconstruction but lacks ablation studies comparing it to other decoder designs (e.g., transformers). The description focuses on architectural choices without quantifying their individual contributions: \"Our new U-Net-inspired decoder generates frames from a single neural embedding... See Appendix F for more details\". Without isolating the decoder’s impact, it is unclear whether gains stem from the architecture or the pre-training strategy.\n\n3. The experiments compare POYO-SSL to a from-scratch baseline and POYO+ but omit broader comparisons with state-of-the-art methods like CEBRA or Neuro-BERT, arguing their architectures are not suited for direct high-fidelity visual reconstruction. However, this justification appears in an appendix, and the main text does not discuss adaptations or partial comparisons (e.g., feature extraction). This narrow scope may overstate POYO-SSL’s advantages.\n\n4. The scaling analysis (Figure 5) shows performance gains with model size but uses a limited range of capacities. The paper notes: \"Our main approach (red) unlocks consistent performance gains as model capacity increases\" , yet no details are provided about the maximum size tested or computational constraints. Expanding the scale range would better validate the claimed monotonic scaling.\n\n5. The theoretical analysis (e.g., loss landscape, Fisher Information) relies on projections and approximations without uncertainty quantification. For instance, the loss landscape roughness metrics ($\\sigma_L$) are derived from smoothed visualizations , which may hide variability.\n\n6. The paper emphasizes biological grounding but does not validate neuronal predictability against ground-truth cell-type properties beyond statistical correlations. While skewness/kurtosis align with inhibitory/excitatory roles, causal experiments are absent.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KD8PQsmUKm", "reviewer": "Reviewer_jGwv", "summary": "●\tThis paper studies how to improve the benefits derived from pre-training for downstream decoding. The authors discover that curating the type of data used for pre-training is crucial to obtaining good scaling. Specifically, they propose to pre-train on neural data that can be said to be more regular, and then fine-tuning is done on data that is less regular. Here, statistical regularity is defined by the authors in terms of skewness and kurtosis. The pretraining objective is masked reconstruction. They find that this pretraining results in better downstream performance on a movie reconstruction task and in classifying a drifting grating.", "questions": "●\tLine 146: This is a basic question, but what are the skewness and kurtosis being taken with respect to? The distribution of calcium values? Across what period of time?\n●\tSection 4.1: When results are reported, for example in Table 3, what neurons are being decoded from? Only the unpredictable neurons? What I'm trying to get at is this: Is the same evaluation data used across all experiments? If not, how can performances between experiments be compared as in Table 3?\n●\tI wonder, if regular activity is more beneficial for pre-training, would it be even better to produce synthetic calcium traces for pre-training that have even more regularity? Do you think this would further improve performance?\n\nMinor points:\n●\tline 190: is this a typo? Should it read: \"the latent representation of the unmasked view is then used as a target for the latent representation of the unmasked variant.\" ? That would make it fit with line 204.\n●\tFigure 1: What should the axis titles be on the calcium trace plots?\n●\tFigure 5: The caption mentions \"orange\", but the corresponding line is yellow.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "●\tThis answers a question about why scaling sometimes stalls even as more pretraining data is added. This is useful for the community to know.\n●\tThis paper provides an actionable lesson for anyone doing self-supervised learning: train on the easy-to-model data first.", "weaknesses": "●\tI'm missing something very basic: why should this method work at all? If pre-training is done on the regular data first, how can the model ever learn good representations of the irregular data? Moving from one neuron type to another represents a distribution shift. What's the explanation for how the model is able to handle this shift?\n●\tIf possible, could the authors please discuss connections to other domains where this strategy would be useful. For example, in language modeling, would it be better to pre-train on regular strings first?\n●\tThe writing could be clearer. See questions below. The methods section, particularly 3.2.2. could be more clear in many places if plainer language were used. For example, take line 205: \"We employ a curriculum learning approach combining masked reconstruction with weak supervision for stable representation learning. Our weakly-supervised auxiliary loss relies on simple visual primitives (drifting gratings) as a curriculum warm-up before moving to a complex downstream movie decoding task.\" In my opinion, it would be easier to understand the following sentence: \"During pre-training, the model is trained on a joint objective, consisting of self-supervised masked reconstruction and fully-supervised classification.\" I don't believe it is correct to use the term \"weak-supervision\" here, since labels are available for all the training examples. And I think it's more common to simply refer to this type of training as \"pre-training and fine-tuning\" rather than \"curriculum learning\".", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "iu7HAGtAP8", "reviewer": "Reviewer_ccVq", "summary": "The paper proposes POYO-SSL, a cell-pattern-aware self-supervised pretraining scheme for decoding dynamic visual experience from mouse calcium imaging. The key idea is to pretrain only on “predictable” neurons, identified a-priori via low skewness/kurtosis of calcium traces (knee thresholds: skew≤3.51, kurt≤22.62; CRE lines SST/VIP/PVALB/NTSR1), then fine-tune on the remaining “unpredictable” neurons for downstream tasks (movie reconstruction; drifting-grating orientation). On the Allen Brain Observatory, the method reports SSIM 0.593 for direct neural-to-movie reconstruction and 55.5% accuracy on gratings, outperforming a strong from-scratch POYO+ baseline with identical capacity. Ablations (architecture, data selection, masking/aux loss) and scaling plots are provided.", "questions": "You define “predictable” neurons via a knee-detection on skewness/kurtosis and set fixed thresholds (skew ≤ 3.51, kurt ≤ 22.62). How sensitive are results to these cutoffs? Please provide a sweep (±10–20%) or cross-validated thresholds.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "A priori selection of predictable neurons using skewness/kurtosis is transparent (knee-based thresholds) and applied consistently without leakage (animals/sessions/neurons disjoint).\n\nArchitecture capacity controls; data-diet variants (inhibitory-only, reverse, mixed); objective variants (temporal vs random masking; CE-only; weight sweeps). These help attribute where gains come from.", "weaknesses": "This paper relies mainly on POYO+-from-scratch leaves room for skepticism about general SOTA claims. Even if CEBRA/Neuro-BERT aren’t pixel decoders, an adapted masked-autoencoding baseline over calcium with the same U-Net decoder, or a temporal contrastive baseline (e.g., CPC-style) would help. At minimum, include a “random neuron subset (size-matched)” pretraining control to show skew/kurtosis selection matters beyond sample count and CRE composition (the paper has “mixed” and “reverse” but not “random size-matched”). \n\nThe authors state that SSIM = 0.593 is the highest reported to date for direct visual reconstruction from cellular-resolution neural recordings. Note they contrast with fMRI works (SSIM 0.19/0.365) which are different modalities and not directly comparable.\n\nIn addition, the comparison to a capacity-matched POYO+ trained from scratch is fair and clean, and there are thorough ablations (encoder/decoder variants; inhibitory-only / reverse / mixed data diets; masking vs random masking; CE-weight sweeps). However, external SSL or generative baselines adapted to calcium-to-image decoding are not included (the paper argues popular SSL methods like CEBRA/Neuro-BERT don’t target pixel-level generation). For a flagship result, adding one or two adapted published methods (or a strong masked-autoencoder baseline over calcium with the same U-Net decoder).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ATopgOZah4", "reviewer": "Reviewer_J6ke", "summary": "- This paper considers the problem of decoding relevant signals from neuronal activity recordings. The studied tasks aim to decode the presented visual input or its statistics from concurrent recordings of neuronal activity from the mouse visual cortex.\n\n- The paper argues that pre-sorting of neurons into 'predictable' and 'unpredictable' sets is useful for curriculum learning: the proposed deep model is first trained on a simple task and using the predictable subset. Then, the unpredictable subset is used to learn a harder task. A neuron is labeled as predictable if the 3rd and 4th order moments of its temporal activity statistics are low.\n\n- The paper proceeds to show that this way of curriculum training improves task performance and conducts ablation studies to support this finding.", "questions": "- Could you please expand the field of view of the two plots in Figure 2 so the loss at the boundaries reaches high values?\n\n- In Figure 2, does the loss correspond to the task reconstruction loss (drifting gratings? movie frame reconstruction?) or the masked loss (self-supervised)?\n\n- Was the knee-detection algorithm applied for each Cre-line? If so, why not apply it to individual neurons and admit neurons into the predictable set based on their own scores rather than based on the Cre-line they belong to?\n\n- (line 147) What does \"near-Gaussian activity\" mean?\n\n- How would the proposed POYO-SSL model perform if 'Finetune Data' is set to 'All' in Table 3?", "limitations": "", "rating": 2, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 1, "strengths": "- Curriculum training is not a new idea. Beyond a curriculum of tasks (easier to harder), the manuscript also argues for a curriculum of samples during training. This is a nuanced point and can perhaps still be considered within the classical curriculum training idea. However, its application to large-scale neural data analysis is novel.\n\n- The manuscript classifies a neuron as predictable and unpredictable based on moments of its own activity, which is easy to compute. It also shows that it corresponds to certain neuronal subsets with molecular underpinnings.\n\n- The paper suggests that it will be harder to decode the signal of interest from neurons whose activity is more variable (high 3rd, 4th moments), both because the task loss is larger (I didn't understand why only the first quadrant of top-2 PCs is shown in Fig. 2.) and those neurons have higher Fisher information for the task.", "weaknesses": "- A main weakness of the manuscript is its claim on high-fidelity movie reconstruction from neural recordings. This would indeed be a major advance. However, my understanding is that the reconstruction is for previously seen video fragments. For such a claim, reconstruction should be demonstrated on previously unseen scenes. As is, I believe the manuscript significantly over-claims. (If my understanding is wrong, I am willing to substantially increase my score.)\n\n- The nuance introduced to curriculum training in this manuscript (a curriculum of data samples) is not a significant enough contribution that warrants publication.\n    - The generalizability of this approach is not tested. It is not clear whether this would be helpful in analyzing other neuroscience datasets (e.g., non-visual) or accomplishing benchmark machine learning tasks.\n    - It can be considered as a natural variant although I don't know if this curriculum was explicitly studied before. This does not strike me as a significant enough contribution to the curriculum training paradigm.\n\n- I don't think the scalability argument surrounding Figure 5 is established in a convincing way. (All plots trend upwards and the y-axis is zoomed-in.) I think there is insufficient evidence to conclude that the slope of the linear fit will be larger for any one model. (e.g., the conclusion depends strongly on the set of points used for such a fit.)\n\n- Statistical Regularity Hypothesis: I believe this is a hypothesis that the authors are putting forth. (Please clarify.) Self-supervised learning will obviously and clearly work better with data with statistical regularity. (e.g., it is not possible to learn noise.) That is, masked reconstruction accuracy will be higher for more predictable neurons. However, whether this extends to supervised (task) performance following masked learning is not clear and I believe that is what the authors want to propose. Predicting the orientation of the drifting grating or the reconstruction of movie frames are both supervised tasks. I think this hypothesis needs a major rethinking and multiple qualifying statements may need to be added.\n\n- Section 3.3 presents various numerical analyses, not theoretical analyses. Please consider renaming.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)