ZAPBench: A Benchmark for Whole-Brain Activity Prediction in Zebrafish
Authors: Jan-Matthis Lueckmann
Alexander Immer
Alex Bo-Yuan Chen
Peter H. Li
Mariela D Petkova
Nirmala A Iyer
Luuk Willem Hesselink
Aparna Dev
Gudrun Ihrke
Woohyun Park
Alyson Petruncio
Aubrey Weigel
Wyatt Korff
Florian Engert
Jeff Lichtman
Misha Ahrens
Michal Januszewski
Viren Jain
Venue: ICLR 2025 Spotlight (2025)
Keywords: neuroscience
zebrafish
forecasting
benchmark
timeseries
lightsheet microscopy
calcium imaging
Primary area: datasets and benchmarks
TL;DR: ZAPBench evaluates how well different models can predict the activity of over 70,000 neurons in a novel larval zebrafish dataset.

Abstract: Data-driven benchmarks have led to significant progress in key scientific modeling domains including weather and structural biology. Here, we introduce the Zebrafish Activity Prediction Benchmark (ZAPBench) to measure progress on the problem of predicting cellular-resolution neural activity throughout an entire vertebrate brain. The benchmark is based on a novel dataset containing 4d light-sheet microscopy recordings of over 70,000 neurons in a larval zebrafish brain, along with motion stabilized and voxel-level cell segmentations of these data that facilitate development of a variety of forecasting methods. Initial results from a selection of time series and volumetric video modeling approaches achieve better performance than naive baseline methods, but also show room for further improvement. The specific brain used in the activity recording is also undergoing synaptic-level anatomical mapping, which will enable future integration of detailed structural information into forecasting methods.

Published as a conference paper at ICLR 2025
ZAPBENCH: A BENCHMARK FOR WHOLE-BRAIN ACTIVITY PREDICTION IN ZEBRAFISH
Jan-Matthis Lueckmann's Alexander Immer'. Alex Bo- Yuan Chen', Peter H. Li', Mariela D. Petkova", Nirmala A. Iyer", Luuk Willem Hesselink', Aparna Dev Gudrun Thrke?, Woohyun Park², Alyson Petruncio", Aubrey Weigel', Wyatt Korff Florian Engert?, Jeff W. Lichtman?, Misha B. Ahrens Michal Januszewskil. & Viren Jain
!Google Research, Harvard University, *HHMI Janelia, 'Radboud University "Correspondence to (mjanusz, vi ren @goog e. com, ahrensm@ jane ia hhmi. org.
ABSTRACT
Data-driven benchmarks have led to significant progress in key scientific modeling domains including weather and structural biology. Here, we introduce the Zebrafish Activity Prediction Benchmark (ZAPBench) to measure progress on the problem of predicting cellular-resolution neural activity throughout an entire vertebrate brain. The benchmark is based on novel dataset containing 4d light-sheet microscopy recordings of over 70,000 neurons in a larval zebrafish brain, along with motion stabilized and voxel- cell segmentations of these data that facilitate develop- ment of a variety forecasting methods. Initial results from a selection of time series and volumetric video modeling approaches achieve better performance than naive baseline methods, but also show room for further improvement. The specific brain used in the activity recording is also undergoing synaptic- -enet anatomical mapping, which will enable future integration of detailed structural information into forecasting methods.
INTRODUCTION
In the natural sciences, a fundamental test of understanding a system is the ability to predict its future behavior based on past observations. This principle has driven progress in fields ranging from celestial mechanics to meteorology. We propose to apply the same rigorous standard to the vertebrate brain, by posing simple yet fundamental question: given C' seconds of observed neuronal activity context, how accurately what are the fundamental limits of predictability in this complex system? To encourage exploration of these questions, introduce dataset and associated benchmark, the Zebrafish Activity Prediction Benchmark (ZAPBenchh. which enables rigorous evaluation of any simulation or forecasting method that produces neural activity predictions at single-cell resolution.
predict the subsequent ~30 seconds? More generally,
Formal benchmarks that quantify progress on prediction tasks have served a critical dual purpose in applied computer science by both identifying broadly superior prediction techniques compared to previous state of the art (Krizhevsky et al. 2012) as well as driving landmark domain-specific results that catalyze scientific discovery (Jumper al.) 202T). Advances have generally been driven by machine learning techniques, which themselves rely on several key ingredients: significant quantities of data; a formal metric to quantitatively compare techniques; and computing power sufficient to efficiently utilize the underlying data. Neuroscience has long been at the forefront of the collection, curation, and analysis of large-scale datasets (e.g., Ahrens al.] 2013 Nguyen al. 2016 Aimon al. 2019; MICrONS Consortium al. 2021 1 International Brain Laboratory al. (203). Here, we show that recent progress has enabled a prediction-oriente vertebrates. Specifically, the transparent zebrafish larva (Danio rerio) permits simultaneous optical recording of all neural activity within the brain at single-neuron resolution over multiple hours.
view of whole-brain neural activity in
Larval zebrafish the only vertebrate species in which whole-brain activity at cellular resolution can currently be obtained. We recorded such activity during nine behavioral tasks, and then extensively postprocessed the resulting 4d video dataset by mitigating motion artifacts, segmenting individual

---

Published as a conference paper at ICLR 2025
A
EM
GT
PA MAE
Figure 1: Dataset and Benchmark. A. Whole brain activity of a larval zebrafish at cellular resolution was recorded with light-sheet mieroscopy setup, while the fish experienced range of visual stimuli (Vladimirov et al. 2014). In addition to the light-sheet (LS) 4d-dataset, a synapse-resolution electron microscopy (EM) 3d-dataset was acquired from the same animal. B. We propose a novel forecasting benchmark in which neural activity is predicted from past activity, using both time series and volumetric video models. Predicted activity (PA) is compared ground truth (GT), and performance is scored by computing the mean absolute error (MAE) between both.
neuron somas using supervised deep learning methods, and mapping each neuron's 's activity to 1d time series (" ("activity trace"). We established a training/validationn//e split within the data, and defined and implemented reference evaluation scheme. Z.PBBench provides the full activity data and associated code to make computational modeling of this data highly accessible. See Fig. for a high-level overview of dataset and benchmark.
A major goal of this effort is to enable comparison of a wide variety of modeling approaches, As an initial step in this direction, we provide of baseline results using both time-series methods that exclusively focus on 1d representations, as well as a neural network trained to directly input and output scrge-scale 3d video frames, capable of exploiting information lost in the conversion to id time series. We do not place any constraints on the types of models allowed in the benchmark, and initially focus on purely data-driven "black box" models prioritizing accuracy over biological plausibility or interpretability. We look forward to the development and evaluation of additional techniques on ZAPBench, including novel ML methods, "white box" biophysically grounded approaches (Kunert et al. 2014 Hines & Carnevale, 1997 Deistler et al. 2024), and more recent "grey box" hybrid schemes Lappalainen et al. 2024 Mi et al. 2022).
Among forecasting benchmarks. example of recent successful effort is WeatherBench (Rasp cct al. 2020] 2024). which accelerated the development of machine learning weather prediction. Benchmarks on neuron activity data have also been proposed before, but ZAPBench marks the first time a forecasting challenge has been posed with this level of coverage for a vertebrate brain. The Sensorium competition (Turishcheva et al. 2023). BrainScore (Schrimpf et al.) 2020), and the Neural Latents Benchmark (Pei et al. (2021) are prior related works of particular note. Crucially, all of these previous efforts analyzed only a small fraction of the brain they studied. For example, the Sensorium competition dataset covers less than 0.1 % of the neurons in the mouse brain from which it was acquired.
Finally, unique aspect of ZAPBench is that the underlying physical wiring diagram (connectome) of the specific animal analyzed in the benchmark under reconstruction and will ultimately be available to augment modeling efforts in the future. This is the first time a whole- brain activity recording will be paired with whole-braie structural reconstruction vertebrate.
In summary, we make the following contributions:
1. A dataset of whole-brain activity of larval zebrafish recorded at single cell resolution, with extensive postprocessing including alignment and segmentation.
2. A forecasting benchmark with clearly defined metrics on multivariate activity traces with much higher dimensionality than typical for time-series forecasting contests (e.g.. Makri- dakis et al., 2022). and which, for the first time, introduces a four-dimensional volumetric dataset (i.e., 3d images+ + time) in the biomedical domain for forecasting purposes.
3. Forecasting results spanning simple parameerrii basclines, representative time-series models, and volumetric video prediction model. Our comparisons show that models
2

---

Published as a conference paper at ICLR 2025
A
calcium activity
Uree death
tima
deptt
depth
B
flow fields
degrth
death
ce2th
segmentation
Aeeoth
deegth
Figure 2: Lightsheet data and postprocessing. A. Frames of the raw calcium activity at different Z-depths and time points. Brightness encodes fluorescence, i.e., activity. Last two panels compare beginning and end of experiment at the same depth, which is misaligned at cellular resolution. B. Flow fields mated correct for deformations in the volume. Color encodes magnitude of flow field in Y-direction, with more saturated colors indicating larger magnitude. There is more deformation at the end the session relative to the beginning. C. Segmentation at different depths, which we used to extract activity traces from the aligned volume.
generally outperform naive baselines, but there is room for improvement. Among other insights, we find that the time series models we benchmarked may underutilize cross-ncuron information, while video models demonstrate the best overall performance. Qualitative analyses reveal that model errors are not uniformly distributed across the brain.
4. Public release of all relevant code, including a web-based viewer for interactively visualizing whole-brain activity single cell resolution, and training code for all discussed models, available at: rpogle-researchh research, gi tiuu....... io/ zapbench
2 DATASET
2.1 EXPERIMENTAL SETUP
Data was collected from a larval zebrafish (Danio rerio, 6 days post fertilization) placed in virtual reality environment. During the experimental session, lasting about two hours, the fish was subject to nine different visual stimulus conditions (GAIN, DOTS, FLASH, TAXIS, TURNING, POSITION, OPEN LOOP, ROTATION, and DARK) designed to probe a range of different behaviors. For example, in the first ten minutes of the experiment, visuomotor gain adaptation was probed: a drifting forward moving sine grating was projected underneath the fish, to which the fish responded by swimming- as if trying to move against water current. Throughout the entire experiment, the fish was head fixed but able to move its tail. By recording electrical signals from the tail and using that to estimate the strength of the swimming activity, the velocity of the stimulus real-time based on the behavior of the fish. The fish adapted its swim strength according to a stimulus gain factor which varied in this condition.
coupled to and manipulated in

---

Published as a conference paper at ICLR 2025
GAIN
FLASH
TAXIS
TURNING
POSITION
OPEN LOOP
ROTATION
71,721 neurons
min
7,879 time steps
6F/F
B
Figure 3: Activity traces. A. Time series for 71,721 neurons extracted from whole-brain calcium recording lasting two hours. Color represents normalized activity (AF/F) with brighter colors indicating higher activity. White lines denote changes stimulus condition, the short name which on top. Neurons ordered by similarity, using rastermap (Stringer al. (023). Note that this representation squeezes the neuron dimension relative to the time dimension. The original aspect ratio of neurons timesteps is approximately 9: 1, whereas in the figure 1:2 for presentation purposes. B. Per-condition training/validationnness set splits.
During such fictive behavior, whole-brain activity was recorded cellular resolution using light-sheet fluorescence microscope (LSFM) containing two laser beams and an overhead cam- era (Vladimirov et al. 2014). Briefly, light-sheet calcium imaging genetically modified zebrafish expressing GCaMP (Dana al. 2019) works by illuminating thin plane of tissue with laser light sheet that rapidly moves across the brain in the axial (Z) direction. When neurons in this plane are active, calcium influx triggers the GCaMP protein to fluoresce. Note that this is indirect measure- ment of brain activity, as changes calcium levels are a proxy for neuronal spiking, compared to directly recording the membrane voltage.
Full details regarding the experiment, including light-sheet imaging, virtual reality, and stimulus conditions in Appendix A
2.2 LIGHTSHEET DATASET POSTPROCESSING
The raw video of whole-brain activity during fictive behavior 4-dimensional volume of 2048 x x 7879 voxels in XYZT imaged at 406 nm 406 nmx4 4 4mmmmmm11 ms resolution. Frames different depth and time points are shown Fig. 2A.. We extensively postprocessed this dataset: aligned it to correct for elastic deformation, segmented the neuron cell bodies, and extracted per-neuron activity traces. We briefly comment on these aspects below. Full technical details are in Appendix B
Alignment. Over the recording period, the volume deformed such that the spatial location neurons in the beginning of the session did not match their location at the end. We found that neither global nor piece-wise 2D affine transformations were sufficient to correct for the misalignment. Therefore, developed custom alignment pipeline in which dense optical flow fields were estimated per time step, with example frames shown in Fig. 28. Flow fields were regularized with an elastic spring mesh and used to warp each frame.
4

---

Published as a conference paper at ICLR 2025
Segmentation. The goal of ZAPBench is accurately predict per-neuron activity traces, for which a voxel-level cell segmentation is required. We found neither simple heuristic methods (e.g.. thresholded correlation maps) nor pre-trained segmentation approaches (such as Strinner al. 2021) to be sufficiently accurate. Instead, we manually annotated around 2,000 neurons as training data using a custom protocol (Petruncio & The CellMap Project Team (023) for customized segmentation pipeline based on one-shot Flood Filling Networks (FFNs: Januszewski et al. 2018). We segmented total of 71,721 putative neurons (see Fig. 2C for examples).
Activity traces. Using the segmentation and a standard normalization of the activity in which baseline fluorescence was subtracted (AF/F), we obtained per-neuron traces, shown in Fig. 3/.. Over the course of the experiment, the fish was exposed to nine different visual stimulus conditions. For most of the conditions, repetitive structure is apparent. This can be explained, in part, by the periodicity of the stimuli. For instance, in the FLASS condition, light and dark images were shown alternately every 30s for total of 10min.
2.3 STRUCTURAL RECONSTRUCTION IN PROGRESS
After conclusion of the activity recording session, a synapse-resolution (4 x 4 x 30 nm& voxel size) structural volume of the brain of the fish was acquired using volume electron microscopy (EM). The EM volume, once fully analyzed, will make it possible to see detailed morphology of all cells within the brain of the fish together with their chemical synapses, and to match these structural reconstructions to the activity traces. The initial stages of the automated analysis of the EM volume have been completed and currently undergoing manual proofreading by team of specialists.
3 BENCHMARK
We propose novel forecasting benchmark: the Zebrafish Activity Prediction Benchmark (ZAP- Bench), the goal of which is to predict future neural activity from past activity. Specifically, we consider the task of predicting the next half minute of activity in two regimes: given either short (few seconds) or a long (several minutes) context window of immediately preceding activity. We use the Mean Absolute Error (MAF) measure how close predictions are the actual recorded values. We consider forecasting models taking either activity traces (time series) or volumetric video as input, but always use trace predictions compute errors. We also define two naive baselines for calibration.
3.1 SETUP
The goal of the benchmark is to predict held-out snippets future neural activity Å over a prediction horizon H given past neural activity A over context window C. This can be formalized as finding functions that map:
f(Ai.c) -AC+IC+H
where At denotes time slice of neural activity in the range [41, 12). For time series forecasting, A is matrix with time and neuron dimensions (7879x71721). For volumetric video models, A is 4d tensor with time and spatial dimensions XYZ.
The prediction horizon is H 32 steps, meaning that snippets of 0.9 ss 0.5 min duration of future activity are to predicted, The context size is C 4 or C 256 steps, to which we refer as short and long context, respectively.
Forecasting models can make use of covariate information: available covariates include information about the stimulus shown in the context window (past covariates, Pl.c), the stimulus shown in the prediction horizon, which follows the context (future covariates, Fc+10+11). as well as spatial locations of neurons (static covariates, S):
f(A.c. PI.C, FoH1C+H. ACT1C+H.
(2)
We divide the dataset by stimulus condition, splitting each condition into 70% training data, 10% validation data for model selection, and 20% test data for evaluation. We completely hold-out one condition, TAXIS, and only use it for testing (see Fig. 3
5

---

Published as a conference paper at ICLR 2025
3.2 EVALUATION
For evaluation, models make predictions of length H on test data, which are compared against ground truth activity. We denote snippet of H steps of predicted activity starting at some absolute time point in the experiment by Ai. snippets of length H in its respective test time range:
H. For each condition, a of predictions is made, for all contiguous
Poond. {Ai.1 with E T
(3)
where Tood contains all time indices of a condition's 's test set starting from which full snippet of H steps can be predicted. Similarly, we define set Geood. containing corresponding ground truth activity snippets Ai.H-
We calculate the mean absolute error (MAE) to quantify model performance:
MAE cond. (Pouns.: Geomd.) [Toond.
A
(4)
N
TE
1=1
where N is the number of neurons, and h E {1... H}, that MAEs are calculated per condition and per step h predicted ahead in the horizon.
3.3 MODELS
We evaluated a representative set of models on ZAPBench, covering time-series models, volumetric video models, as well as naive baselines. Our selection was biased towards simple but state-of-the-art models and exploration of different classes of input-output mappings. We briefly introduce the models here and provide full details including our hyperparameter choices in AAppendix C
3.3.1 TIME-SERIES FORECASTING
Linear. Zeng et al. 2023 found that simple linear models can outperform state-of-the-ar transformer architectures on multiple forecasting benchmarks. Inspired by these results, we included linear forecasting models, which map
fLinear (a1:C.n Φ) = âC+1C+H..
(5)
where :22 denotes time slice of neural activity in the range [t1, t2] of a single neuron out of N total neurons. Note that a single set of parameters & is used to predict all time series, which is why this type of model is also referred to a global univariate model. The model uses single linear transformation to map C inputs to H outputs. We optionally use the normalization proposed in ZZng al. (023) by which values step C' are subtracted from the input and added back to the output.
TiDE. Proposed by Das et al. 2023), Time-series Dense Enoder (TiDE) is a global univariate Multi-Layer Perceptron (MLP) architecture, with nonlinearities and covariates,
SNDE (aI.C.a, P Pl..; Fc+1.++1: HI S, x) âc+L.C+H,.
For past and future covariates, P and F, we encoded information about the stimulus, as detailed in Appendix B. For static covariates S, used sine-cosine embeddings to encode the spatial locations of neurons.
TSMixer. Rather than making univariate predictions per neuron, TSMixer (Chen al. 2023 is multivariate forecasting model,
FTSMixer (A1.c, Φ) AC+LIC+H
TSMixer processes time and neuron dimensions in an alternating fashion: a block includes an MLP applied along the time dimension (time-mixing), followed by transposition and two MLPs applied along the feature, i.c. neuron, dimension (feature-mixing).
Time-Mix. A variation of TSMixer without feature-mixing. Since individual time series in the activity matrix cannot influence each other, Time-Mix, can be viewed as mapping
Time Mix (a1:C.n 3) ac+1:C
(8)
i.e., global univariate model, similar to fLinea in Equation 5

---

Published as a conference paper at ICLR 2025
3.3.2 VOLUMETRIC VIDEO FORECASTING
Instead of extracting the traces from the volumetric video, we can also apply models directly and use the segmentation mask for the loss and output. In particular, for video models we have four-dimensional A where each time step has spatial dimensions 2048 x152x72 in XYZ. We used the volume from which traces were extracted in Sec. 2.2 but slightly cropped it in the Y-axis for computational reasons.
U-Net. To construct the function,
fu-Net (A Φ) Acth.
we use a variation of the U-Net (Ronneberger et al. 2015) with modifications for scalability to the volumetric case with 18M weights Φ. This model naturally multivariate. Down- and upsampling schemes are described in Appendix C We condition every block on a lead time h between 1 and 32, as proposed by Andrychowicz al. (2023) for weather forecasting, using FiLM layers (Perez et al., 2018). Therefore, the output layer maps to single sire-dimensional frame that forecasts the video for the given lead time. We use the neuron mask compute and optimize the same MAE loss as on the traces. Similarly, we use traces extracted from forecasts for evaluation. Extensive model selection and pretraining results for this model are in Immer al. (2025).
3.3.3 NAIVE BASELINES
To calibrate performance of time series and video models, we define naive, parameter-free baselines which do not require any training.
Mean. The mean baseline makes predictions based on averaging past activity of each neuron
MMean (ac- (ac-w:c.)) ac+1:++H..
(10)
where the hyper-parameter W defines the length of the window of past activity that SMean averages over: the average is repeated for all H steps in the prediction horizon. Evaluating performance for different choices of W s C, find that short W are better when predicting few steps ahead, and longer W are better when predicting steps later in the horizon (supplementary Fig. ST). Based on validation performance,
W 4 for C 4. For C = 256, we model that W = 4 for steps 1... 10 in the prediction horizon, and W = 128 for steps 12.
Stimulus. As described in Sec. 2.2] some of the repetitive structure in the trace matrix can be attributed to stimuli that are presented multiple times. We formulate stimulus-evoked baseline to capture this:
fstmulus (F (Fc+1.++)) = Ac+IC++
(11)
where /stimulus describes the lookup based on stimulus phase test time. More specifically, we chunked conditions by stimulus repeats, aligned repeats, and computed the average response per neuron and return this stimulus-evoked response (Fig. S2 as prediction.
4 RESULTS
From our experiments, we gain number of insights, and identify opportunities for future work. First, we analyse performance relative to naive baselines:
#1: Models largely outperform naive baselines. Comparing models across short and long context for different horizons Fig. we find that models mostly outperform naive baselines terms of grand average MAE (taken across all conditions except the held-out one).
#2: Baselines facilitate comparisons across conditions. more nuanced view of results provided in Fig. S3 and Fig. S4 for short and long context. respectively: rather than calculating the grand average, we report MAE per condition. Comparing conditions, we observe that performance greatly varies in terms of absolute MAE. Different conditions activate different fractions of neurons in the brain, which, in turn, translates to changes in mean activity levels and error magnitude. Naive baselines such as mean and stimulus baseline thus help calibrate performance across conditions.
#3: Stimulus baseline can be competitive. Although simple, we find that the stimulus baseline can be competitive in some settings. This is reflected in the grand average results for short context, 32
7

---

Published as a conference paper at ICLR 2025
step ahead
steps ahead
steps ahead
16 steps ahead
32 steps ahead
short context
0.030
MAE
0.025
0.020
0.05
long context
0.030
MAE
0.025
0.15
Inwer
time-mx-
unal
Inoar
sdc
trmaee
fmme-mi
undt
sdc
unes
Innor
knnes
smmemx
Sio
tim
Figure 4: Grand average results for short and long context. To compare overall performance, take the grand average MAE (lower is better) across conditions for short (C = 4) and long context (C 256). Error bars indicate variability due random number generator seeding, excluding variability across conditions (95%--cofidence intervals; random seeds). Values are clipped to axis limits. The dotted black line indicates performance of the mean baseline, the solid line is the stimulus baseline. Per-condition results are reported in the supplement.
steps ahead (Fig. 4). Considering per-condition short context results, we e.g. find that less than half of the models outperform the stimulus baseline at 32 steps predicted ahead (53 out of 120 runs), and that the stimulus baseline is particularly strong on conditions FLASH, TURNING, and ROTATION (ig. S3). This suggests room for improvement since stimulus covariates can be used by forecasting models. Note that this finding does not apply to long context (Fig. S4)
Contrasting different context lengths, we find that:
#4: Long context improves predictions further ahead. Models given long context tend to perform better than ones given short context. The difference greater with more steps predicted ahead (Fig. 4p. Note that long context is not always advantageous, in particular, when predicting only single step ahead we find that U-Net performs worse with long context. We speculate that this is due effectively having more examples in the training dataset when = 4.
#5: Differences between models are minor in long context setting. For short context, we see more variability between models than for long context. For long context, performance of TiDE, TSMixer, Time-Mix, and U-Net is very similar, while Linear is worse (Fig. 4 Fig. S4). Time-Mix the simplest architecture performing well long context, global univariate model without covariates.
Finally, we highlight three key findings with opportunities for future work:
#6: Cross-neuron information may be underutilized for time-series models. For TiDE, found that a variant without static covariates, i.e., spatial positions of neurons, performed best Fig. S7). Contrasting TSMixer with global univariate models (TiDE and Time-Mix), pattern suggesting that mixing time-series to utilize cross-neuron information helps. Difficulty in utilizing nigh-dimensional multivariate information has also been a point of discussion in recent forecasting literature (e.g., Zeng et al.) 2023: Nie et al.) 2023). We view this an opportunity to explore new approaches. Future availability the connectome will also enable the use of additional model classes, e.g., graph-based ones.
do see clear
#7: Video model ranks best for short context. Overall, the U-Net model ranks best on short context, as evidenced by grand average and per-condition results. The relative difference other models is largest for single step ahead predictions in the short context setting. Looking at the hold-out condition (Fig. 5). also find advantage across different context lengths for single step ahead predictions, but not for longer horizons. Attempting to find time-series models that match or outperform these results is an interesting avenue for future work.
8

---

Published as a conference paper at ICLR 2025
step ahead
steps ahead
8 steps ahead
16 steps ahead
32 slops ahead
0.035
short context
0.030
MAE
0.025
0.020
0.018
context
0.030
MAE
0.025
0.020
0.018
Inser
tamborr
umm-mmx
unat
Inser
tamiaar
unit-
Innerr
tembor
unst
Ineer
lamber
Smo-mix
unvt
Inner
tsmoer
timombx
Umm
smo
Figure 5: Hold-out condition results for short and long context. Performance on TAXIS, held-out from training, measured by MAE (lower is better). Points individual runs (3 random seeds per model). Values shown are clipped to axis limits. The dotted black line indicates performance of the mean baseline. Note that the poor performance of TiDE can be explained by its reliance stimulus covariates, which are out-of-distributio for this condition.
#8: Errors are not evenly distributed across the brain. To facilitate qualitative com- parisons between models,
developed web-based 3d visualization tool available at: wtw.ssearrr................ git.ub..t....... 10/ za zapbenchi (see Fig. S11 for a screenshot and deseription, and uploaded supplementary material for video demonstration). Visualizing the spatial location of error,, we observed that they tend to cluster, rather than being evenly distributed across the brain. For instance, observed tendency for higher MAEs in the -onteir part of the brain (pallium), which is considered homologous to the mammalian hippocampus (Yññez et al., (022) and thus likely plays a role learning and memory formation, and for which there some evidence supporting quantity discrimination (Messina et al., 2022). Additional analyses of MAE variations may benefit from incorporating data from standard zebrafish atlases (Randlett al. 2015 Kunst et al. 2019).
google-researe rch.
5 CONCLUSIONS AND FUTURE WORK
We presented ZAPBench, a novel whole-brain activity prediction benchmark. The dataset used in the benchmark represents the current state-of-the-art in experimental and postprocessing techniques. Potential limitations remain, which we discuss in more detail in Appendix B
Our initial results show that while selected approaches based recent literature are sufficient outperform naive baselines, significant room for improvement remains. We see multiple avenues for exploration. Our results suggest that cross-neuronal information may currently be underutilized. Graph-based approaches and latent variable models that explicitly account for multivariateness and correlations between activity traces may be promising directions to explore. Models may also benefit from the incorporation of known biological inductive biases and from probabilistic approaches which quantify forecast uncertainty and allow sampling multiple future trajectories.
MAE, the benchmark metric, is commonly used for evaluating forecasting models, but it provides only a very limited measure of activity distribution characteristics. Scientifically useful models of brain activity should not only accurately predict immediate future states but also realistically represent the underlying generative processes and their statistical distributions. In the future, it might therefore be necessary incorporate additional metries such as Continuous Ranked Probability Scores (CRPS; Gneiting & Raftery| 2007). as well as metrics that assess the "physical realism" of predicted brain activity, such measures of cuto-correlltion,
power spectral density.
Because the brain is a stochastic system, there is also a minimal non-zero level of error beyond which perfect forecasting model could not improve. The numerical value corresponding this

---

Published as a conference paper at ICLR 2025
"performance ceiling" is not known, and there is gold standard forecasting model compare to. Stochasticity of brain activity arises from inherent biophysical and biochemical sources randomness (e.g. probabilistie vesicle release in synaptic transmission, random binding of neurotransmitters to postsynaptic receptors), unobserved processes (e.g. neuromodulation, intrinsic cell excitability varying over time) and sensory stimuli other than visual that olfactory, auditory, mechanoception).
under precise control (e.g.
The future availability of the connectome will help mitigate the latter sources, as sensory cells could then be distinguished from cells focused on internal computations and therefore used as input (covariates) rather than prediction targets in the benchmark. More importantly, the connectome will enable more approaches, such as white- and grey-box models that directly incorporate the structural connectivity information and combine mechanistic insights with data-driven modeling. will also enable additional investigations, such as inference of structure from function.
We hope ZAPBenchh will serve as catalyst for the development of increasingly accurate and sophisticated models of brain activity and stimulate innovation in predictive modeling.
REPRODUCIBILIY STATEMENT
Our experimental setup is described in full detail in Appendix A] data acquisition and postprocessing in Appendix BJ and technical details models and hyperparameters are in Appendix C Datasets, all relevant code, and interactive visualizations are available through a companion website (see Appendix D for additional information): google-research, waarurchh.ittttt www.uitthhh.... 1o/ zapbench
ACKNOWLEDGMENTS
We thank Marc Coram, Stephan Hoyer, Thomas Kipf, Urs Köster, Ian Langmore, Martyna B. Plomecka, Stephan Rasp, Srinivas C. Turaga, Sven Dorkenwald and the Connectomics team at Google Research for technical discussions, and Michael P. Brenner, Lizzie Dorfman, Elise Kleeman, and John Platt, for project support. We also thank Greg M. Fleishman for help with cross-modal alignment, and Shawn Stanley and Vijaya Teja Rayavarapu for exploratory analyses.
Some of this work was supported by the FishEM Project Team at Janelia Research Campus which generated the ground truth data in collaboration with the CelIMap Project Team and Project Technical Resources.
This article is subject to HHMI's Open Access to Publications policy. HHMI lab heads and project team leads have previously granted nonexclusive CC BY 4.0 license to the public and a sublicensable license to HHMI in their research articles.
REFERENCES
Misha B. Ahrens, Michael B. Orger, Drew N. Robson, Jennifer M. Li, and Philipp J. Keller. Whole- brain functional imaging cellular resolution using light-sheet microscopy. Nature Methods, 2013.
doi: 10. 1..000/mmtth.4555 2434
Sophie Aimon, Takeo Katsuki, Tongqiu Jía, Logan Grosenick, Michael Broxton, Karl Deisseroth, Terrence J. Sejnowski, and Ralph J. Greenspan. Fast near-whole- imaging in adult Drosophila during responses to stimuli and behavior. PLOS Biology, 2019. doi: 10.1371/journal.phr 2207 2006732
Marcin Andrychowicz, Lasse Espeholt, Di Li, Samier Merchant, Alexander Merose, Fred Zyda, Shreya Agrawal, and Nal Kalchbrenner. Deep learning for day forecasts from sparse observations.
arXiv,, 2023. doi:
Si-An Chen, Chun-Liang Li, Nate Yoder, Sercan O Arik, and Tomas Pfister. TSMixer: An All-MLP architecture for time series forecasting. Transactions on Machine Learning Research, 2023.
Hod Dana, Yi Sun, Boaz Mohar, Brad K Hulse, Aaron M Kerlin, Jeremy P Hasseman, Getahun Tsegaye, Arthur Tsang, Allan Wong, Ronak Patel, John J. Macklin, Yang Chen, Arthur Konnerth, Vivek Jayaraman, Loren L. Looger, Eric R. Schreiter, and Karel Svoboda. high-peeformance calcium sensors for imaging activity in neuronal populations and microcompartments. Nature Methods, 2019. doi: 10. 1011033/4415922211994333.
10

---

Published as a conference paper at ICLR 2025
Abhimanyu Das, Weihao Kong, Andrew Leach, Shaan Mathur, Rajat Sen, and Rose Yu. Long- term Forecasting with TiDE: Time-series Dense Encoder. Transactions on Machine Learning Research, 2023.
Michael Deistler, Kyra L. Kadhim, Matthijs Pals, Jonas Beck, Ziwei Huang, Manuel Gloeckler, Janne K. Lappalainen, Comelius Schröder, Philipp Berens, Pedro J. Gonçalves, and Jakob H. Macke. Differentiable simulation enables large-scale training of detailed biophysical models of neural dynamics. bioRxiv, 2024.
doi: 10.1101/2024.08.01 .608999
Tilmann Gneiting and Adrian E. Raftery. Strictly proper scoring rules, prediction, and estimation. Journal of the American Statistical Association, 2007. doi: 10. (1198/066115560000000
Micheal L. Hines and Nicholas T. Carnevale. The NEURON Simulation Environment. Neural Computation, 1997. doi: wttp///cco..99...6......
Alexander Immer, Jan-Matthis Lueckmann, Alex Yu-Y Chen, Peter H. Li, Mariela D. Petkova, Nirmala A. Iyer, Aparna Dev, Gudrun Ihrke, Woohyun Park, Alyson Petruncio, Aubrey Weigel, Wyatt Korff, Florian Engert, Jeff W. Lichtman, Misha B. Ahrens, Viren Jain, and Michal Januszewski. Forecasting whole-brain neural activity from volumetric video. arXiv, 2025. doi 1044850/arXi,2255.000777
International Brain Laboratory, Brandon Benson, Julius Benson, Daniel Birman, Niccolò Bonacchi, Matteo Carandini, Joana A. Catarino, Gaelle A. Chapuis, Anne K. Churchland, Yang Dan, Peter Dayan, Eric EJ Witt, Tatiana A. Engel, Michele Fabbri, Mayo Faulkner, Ila Rani Fiete, Charles Findling, Laura Freitas Silva, Berk Gerçek, Kenneth Harris. Michael Häusser, Sonja B. Hofer, Fei Hu, Félix Hubert, Julia M. Huntenburg. Anup Khanal, Christopher Krasniak, Christopher Langdon, Petrina Y. Lau, Zachary Mainen, Guido T. Meijer, Nathaniel J. Miska, Thomas D. Mrsic-Flogel, Jean- Paul Noel. Kai Nylund, Alejandro Pan- Vazquez, Alexandre Pouget, Cyrille Rossant, Noam Roth, Rylan Schaeffer, Michael Schartner, Yanliang Shi, Karolina Z. Socha, Nicholas A. Steinmetz, Karel Svoboda, Anne E. Urai, Miles J. Wells, Steven Jon West, Matthew R. Whiteway, Olivier Winter, and Ilana B. Witten A Brain-Wide Map of Neural Activity during Complex Behaviour. bioRxiv, 2023. doi: 10.1101/202300.066 .7.4477688
Michal Januszewski, Jörgen Kornfeld, Peter H. Li, Art Pope, Tim Blakely, Larry Lindsey, Jeremy Maitin-Shepard, Mike Tyka, Winfried Denk, and Viren Jain. High-precision automated reconstruc- tion of neurons with flood-filling networks. Nature Methods, 2018. doi: 10.1038//41592.018/
0049-4]
John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ronneberger, Kathryn Tunyasuvunakool, Russ Bates, Augustin Židek, Anna Potapenko, Alex Bridgland, Clemens Meyer, Simon A. A. Kohl, Andrew J. Ballard, Andrew Cowie, Bernardino Romera- Paredes, Stanislav Nikolov, Rishub Jain, Jonas Adler, Trevor Back, Stig Petersen, David Reiman, Ellen Clancy, Michal Zielinski, Martin Steinegger, Michalina Pacholska, Tamas Berghammer, Sebastian Bodenstein, David Silver, Oriol Vinyals, Andrew W. Senior, Koray Kavukcuoglu, Push- meet Kohli, and Demis Hassabis. Highly accurate protein structure prediction with AlphaFold. Nature, 2021. doi:
Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton. ImageNet Classification with Deep Convolutional Neural Networks. In Advances in Neural Information Processing Systems, 2012.
James Kunert, Eli Shlizerman, and J. Nathan Kutz. Low-dimensional functionality of complex network dynamics: Neurosensory integration in the Caenorhabditis elegans connectome. Phys. Rev. E, 2014. doi: 10. 10.100XPPhysReeE.589..555
Michael Kunst, Eva Laurell, Nouwar Mokayes, Anna Kramer, Fumi Kubo, António M. Fernandes, Dominique Förster, Marco Dal Maschio, and Herwig Baier. A Iatuulaar--eoouttioon Atlas of the Larval Zebrafish Brain. Neuron, 2019. doi: (..1016/j.neuroon 2tt.:oo....nooo...004...... 2019.00.034
Janne K. Lappalainen, Fabian D. Tschopp, Sridhama Prakhya, Mason McGill, Aljoscha Nern, Kazunori Shinomiya, Shin-ya Takemura, Eyal Gruntman, Jakob H. Macke, and Srinivas C. Turaga. Connectome constrannd networks predict neural activity across the fly visual system. Nature, 2024. doi: 556-444413--02477333993

---

Published as a conference paper at ICLR 2025
Spyros Makridakis, Evangelos Spiliotis, and Vassilios Assimakopoulos. M5 accuracy compe- tition: Results, findings. and conclusions. International Journal of Forecasting, 2022. doi
10.0016/.fforcass.2211...100
Andrea Messina, Davide Potrich, Ilaria Schiona, Valeria Anna Sovrano, Scott E. Fraser, Caroline H Brennan, and Giorgio Vallortigara. Neurons in the dorso-central division of zebrafish pallium respond to change in visual numerosity. Cerebral Cortex, 2022. doi: 10. 10933//ccrcor//haa2288
Lu Mi, Richard Xu, Sridhama Prakhya, Albert Lin, Nir Shavit, Aravinthan Samuel, and Srinivas C. Turaga. caanectome-constraine Latent Variable Model of Whole-Brain Neural Activity. In International Conference on Learning Representations, 2022.
MICiONS Consortium, J. Alexander Bae, Mahaly Baptiste, Caitlyn A. Bishop, Agnes L. Bodor, Derrick Brittain, JoAnn Buchanan, Daniel Bumbarger, Manuel A. Castro, Brendan Celii, Erick Cobos, Forrest Collman, Nuno Maçarico da Costa, Sven Dorkenwald, Leila Elabbady, Paul G. Fahey, Tim Fliss, Emmanouil Froudarakis, Jay Gager, Clare Gamlin, William Gray- Roncal, Akhilesh Halageri, James Hebditch, Zhen Jia, Emily Joyce, Justin Joyce, Chris Jordan, Daniel Kapner, Nico Kemnitz, Sam Kinn, Lindsey M. Kitchell, Selden Koolman, Kai Kuehner, Kisuk Lee. Kai Li, Ran Lu, Thomas Macrina, Gayathri Mahalingam, Jordan Matelsky. Sarah McReynolds, Elanine Miranda, Eric Mitchell, Shanka Subhra Mondal, Merlin Moore, Shang Mu, Taliah Muhammad, Barak Nehoran, Oluwaseun Ogedengbe, Christos Papadopoulos, Stelios Papadopoulos, Saumil Patel, Xaq Pitkow, Sergiy Popovych, Anthony Ramos, R. Clay Reid, Jacob Reimer, Patricia K. Rivlin, Victoria Rose, Casey M. Schneider- Mizell, H. Sebastian Seung, Ben Silverman, William Silversmith, Amy Sterling, Fabian H. Sinz, Cameron I.. Smith, Shelby Suckow, Marc Takeno, Zheng H. Tan, Andreas S. Tolias, Russel Torres, Nicholas I.. Turner, Edgar Y. Walker, Tianyu Wang, Adrian Wanner, Brock A. Wester, Grace Williams, Sarah Williams, Kyle Willie, Ryan Willie, William Wong, Jingpeng Wu, Chris Xu, Runzhe Yang, Dimitri Yatsenko, Fei Ye, Wenjing Yin, Rob Young, Szi-chieh Yu, Daniel Xenes, and Chi Zhang. Functional connectomics spanning multiple areas of mouse visual cortex. bioRxiv, 2021.
doi: 10.1101/2021.0% 0ttp//2221..7.2..4444 4554022
Jeffrey P. Nguyen, Frederick B. Shipley, Ashley N. Linder, George S. Plummer, Mochi Liu, Sagar U. Setru, Joshua W. Shaevitz, and Andrew M. Leifer. Whole-brain calcium imaging with cellular resolution in freely behaving Caenorhabditis elegans. Proceedings of the National Academy of doi: 10. 100073/paa.11071100
Sciences, 2016.
Yuqi Nic. Nam H. Nguyen, Phanwadee Sinthong, and Jayant Kalagnanam. A Time Series is Worth 64 Words: Long-term Forecasting with Transformers. In International Conference on Learning Representations, 2023.
Felix Pei, Joel Ye, David Zoltowski, Anqi Wu, Raced H. Chowdhury, Hansem Sohn, Joseph E. O'Doherty, Krishna V. Shenoy, Matthew T. Kaufman, Mark Churchland, Mehrdad Jazayeri, Lee E. Miller, Jonathan Pillow, II Memming Park, Eva L. Dyer, and Chethan Pandarinath. Neural Latents Benchmark '21: Evaluating latent variable models of neural population activity. In Neural Information Processing Systems, Track on Datasets and Benchmarks, 2021.
Ethan Perez, Florian Strub, Harm De Vries, Vincent Dumoulin, and Aaron Courville. Film: Visual reasoning with general conditioning layer. In Proceedings of the AAAI conference on artificial intelligence, 2018
Alyson Petruncio and The CellMap Project Team. Labeling of dense light data in amira 3d 2022.1 classic segmentation workroom. 2023. doi: 10. 10.17504/pootccols..o io. dm6ggii6pp77pvv
Owen Randlett, Caroline L. Wee, Eva A. Naumann, Onyeka Nnaemeka, David Schoppik, James E. Fitzgerald, Ruben Portugues, Alix M. B. Lacoste, Clemens Riegler, Florian Engert, and Alexander F. Schier. Whole-brain activity mapping onto a zebrafish brain atlas. Nature Methods, 2015. doi: 10.11038/meth.3581
Stephan Rasp, Peter D Dueben, Sebastian Scher, Jonathan A Weyn, Soukayna Mouatadid, and Nils Thuerey. WeatherBench: A benchmark data set for data-driven weather forecasting. Journal of Advances in Modeling Earth Systems, 2020. doi: 10.1029/2000MSS022220
12

---

Published as a conference paper at ICLR 2025
Stephan Rasp, Stephan Hoyer, Alexander Merose, lan Langmore, Peter Battaglia, Tyler Russel, Vivian Yang. Rob Carver, Shreya Agrawal, Matthew Chantry, Zied Ben Bouallegue, Peter Dueben, Carla Bromberg, Jared Sisk, Luke Barrington, Aaron Bell, and Fei Sha. WeatherBench 2: A benchmark for the next generation of data-driven global weather models. arXiv, 2024.
Alvaro Sanchez-
doi: 10.448550/axX..230....55 15560)
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical image computing and computter-asiisee intervention, 2015.
Martin Schrimpf, Jonas Kubilius, Michael J. Lee, N. Apurva Ratan Murty, Robert Ajemian, and James J. DiCarlo. Integrative Benchmarking to Advance Neurally Mechanistic Models of Human Intelligence. Neuron, 2020. doi: wtt.l///..eurom.2220....
Carsen Stringer, Tim Wang, Michalis Michaelos, and Marius Pachitariu. Cellpose: A generalist algorithm for cellular segmentation. Nature Methods, 2021. doi: 10:008/6441922200000001111
Carsen Stringer, Lin Zhong, Atika Syeda, Fengtong Du, Maria Kesa, and Marius Pachitariu. Rastermap: discovery method for neural population recordings. bioRxiv, 2023. doi
10.1101/2023300.2555550 25,50077
Polina Turishcheva. Paul G. Fahey, Laura Hansel, Rachel Froebe, Kayla Ponder, Michaela Vystrčilová, Konstantin Willeke, Mohammad Bashiri, Eric Wang. Zhiwei Ding, Andreas S. Tolias, Fabian H. Sinz, and Alexander S. Ecker. The dynamic sensorium competition for predicting large-scale mouse visual cortex activity from videos. arXiv, 2023. doi: 104485500arXx122335.5666
Nikita Vladimirov, Yu Mu, Takashi Kawashima, Davis V. Bennett, Chao-Tsung Yang, Loren L. Looger, Philipp Keller, Jeremy Freeman, and Misha B. Ahrens. Light-sheet functional imaging in fictively behaving zebrafish. Nature Methods, 2014. doi: 10.1038/nmeth.300
Julián Yáñez, Mónica Folgueira, Ihán Lamas, and Ramón Anadón. The organization of the ze- brafish pallium from a hodological perspective. Journal of Comparative Neurology, 2022. doi
00.1000nee.25266
Ailing Zeng, Muxi Chen, Lei Zhang, and Qiang Xu. Are Transformers Effective for Time Series Forecasting? AAAI Conference on Artificial Intelligence, 2023, doi: 10.
13

---

Published as a conference paper at ICLR 2025
Appendices
Experimental setup
Experimental model and subject details
A.2
Light-sheet imaging
A.3
Virtual reality setup and fictive behavior
A.4
Visual stimuli for virtual-reality behavior
Datasets and postprocessing
B.
Acquired light-sheet datasets
Alignment
B,
Normalization
B.
4
Segmentation
B 5
Trace extraction
B.6
Stimulus feature encoding
7
Limitations
B.8
Progress electron microscopy volume reconstruction
Models
Linear
TiDE
TSMixer
4 Time-Mix
C.5 U-Net
D Data and code release
D.I Datasets
D.2 Benchmark
Supplementary Tables
10
Supplementary Figures
11
References
23

---

Published as a conference paper at ICLR 2025
A EXPERIMENTAL SETUP
A.1 EXPERIMENTAL MODEL AND SUBJECT DETAILS
Experiments were conducted in accordance with the guidelines of the National Institutes of Health. Animals were handled according to IACUC protocol 22-0216 (Ahrens lab). We used a 6 dpf zebrafish with terdear-targeted GCaMP variant for our experiments (Tg(clavI3:H2BB [[fclavB::BBB-CaaMPT) The sex of the fish is indeterminate this age. Fish were raised in shallow Petri dishes, on 14 h light, 10 dark cycle at around 27 'C and fed ad libitum with paramecia after 4 dpf. The experiment was done during daylight hours (4 h4-14h h after lights on). All protocols and procedures were approved by the Janelia Institutional Animal Care and Use Committee.
TRGeCap: Tg(elav13:Gal4- Cra.:l::SGG4-V111...U CU
A.2 LIGHT-SHEET IMAGING
To perform light-sheet imaging of whole- brain activity, we used a light-sheet microscope described previously (Vladimirov et al.) 2014). We used two 488 nm laser beams that were scanned through low-NA objectives in the horizontal and vertical directions to generate the light sheet and move dorsally and ventrally in the brain. One light sheet (side laser) entered the brain from the left side of the head, while the other (front laser) entered the brain from the front of the head, between the eyes. The front laser was swept such that illumination was restricted the region between the eyes. The side laser was switched off when it swept in front of the eyes to prevent excitation of the retina. For every plane, an GCaMP7f fluorescence image was acquired using a 6x/0..8 NA detection objective (Nikon), Nikon tube lens, 525/ 550 nm detection filter (Semrock), and camera (Orca Flash 4.0 v2, Hamamatsu). After each plane was acquired, the detection and illumination objectives were moved dorsally by 4 um to collect the next plane, until the entire span of the brain was imaged at around volume/second.
A.3 VIRTUAL REALITY SETUP AND FICTIVE BEHAVIOR
We used previously published protocol for virtual reality fictive behavior (Vladimirov et al., 2014).
Briefly, the larval zebrafish were paralyzed with the nicotinic acetylcholine receptor blocker a- bungarotoxin Aldric-Addrich #203980) (immersion in 25 μM toxin in external solution for 10 30000 s). The fish was then embedded on an acrylic platform in custom square behavioral chamber. Agarose was removed around the head of the fish and over the dorsal part of the fish 's tail. Large-barrel glass pipettes (tip diameter ~60 jm) were attached to the left and right dorsal sides of the fish's tail via gentle suction to record motor nerve electrical activity. The electrical activity was amplified and filtered using a MultiClamp 700B amplifier. The 10 ms-rolling standard deviation of the signal was calculated and defined to be the swim signal used to quantify behavior and provide visual feedback.
Visual stimuli were projected underneath the fish using video projector (MP-CD1, Sony). In closed loop configurations, swim signals above 2.5 standard deviations of baseline triggered backwards movement of visual stimuli with speed proportional to swim power (stimulus velocity drift speed gain X swim power).
A.4
VISUAL STIMULI FOR RERTUAL-REALIVE BEHAVIOR
We used the following list of stimuli during our experiments:
1.
To probe gain adaptation (Ahrens et al. 2012: Kawashima et al.) 2016), animals were subject to periods of two different visuomotor gains, low gain and high gain, that alternated every 30 seconds, for total of 10 minutes (i.e. 10 trials each of high and low gain). During these periods, the animal was subject to a sine-grating visual stimulus that would drift forward with constant velocity when the animal was not swimming, in order to evoke the optomotor response (OMR; Orger et al., 2000; Naumann et al., 2016). When the animal emitted swim, the sine grating would move backwards with velocity proportional to the vigor and to the visuomotor gain. The gain during the high gain periods was two times stronger than during low gain periods.
Gain:
2

---

Published as a conference paper at ICLR 2025
2. Dots:
To probe evidence accumulation and decision-main making (Bahl & Engert 2020), animals were subjected to a series of cchheeee random dot, open-loop optic flow stimuli. Briefly, during baseline periods, animals were shown small flickering dots (dot lifetime 200 ms). During three stimulus periods, 100% of the dots moved rightward for 20s.
To probe animals' light and dark flash-evoked startle responses (Burgess Granato 2007), we alternated delivery of whole-field light and dark stimuli every 30s for a total of 10 minutes.
Flash:
4. Taxis: To probe animals' phototactic behavior (Brockerhoff al., 1995 Wolf al. 2017
Chen et (20T], we alternated the left and right visual hemifields to be either light or dark, cycling over the following (left, right) combinations: (light, light), (dark, light), (dark, dark), (light, dark), (light, light), (light, dark). Each combination lasted for 20s and was repeated total of 5 times.
5. Turning:
To drive turning behavior, we delivered drifting sine gratings in an open loop configuration, cycling between forward, leftward, rightward, and backwards motion. Motion lasted for 30s with 30s of stationary gratings in between trials. Each direction was repeated 5 times.
6. Position: We probed positional homeostasis as previously described Yang et al. 2022) Briefly, we delivered s long forward pulse of sine grating, followed by delay period of various lengths (3s, 6s, or 9s), followed by 30s period of open loop forward grating. We performed 3 trials of each delay period.
7. Open loop:
To probe futility- inndueed passivity (Mu al., 2019), we delivered constant- velocity, open-loop forward drifting sine gratings for 15 min. The animal's 's swimming did not result in any visual feedback.
8. Rotation: To probe neural activity response rotational stimuli, delivered sine grating stimuli that rotated with constant angular velocity. We cycled between clockwise and counterclockwise rotations, with each rotation direction lasting for 30s for total of 10 min.
9. Dark:
To measure spontaneous brain activity, all visual stimuli turned off in this condition.
3

---

Published as a conference paper at ICLR 2025
B DATASETS AND POSTPROCESSING
B.I ACQUIRED LIGHT-SHEET DATASETS
Using the experimental setup detailed in Appendix A lightsheet functional imaging was used to acquire whole-brain activity and anatomy datasets.
Functional activity volume. Whole-brain activity during fictive behavior was im- aged at 406 nmx406 nm pm1914 ms resolution in XYZT, yielding volume of size 4048x1328-0 72x voxels.
Functional anatomy volume. In addition, volume with higher Z-resolution and longer exposure was captured at the beginning of the session. This functional anatomy dataset was imaged at 406 nm 406 nmx 1 jmm umx6.82s resolution in XYZT. Ten frames were taken, yielding volume of size 2048 x 3328 x30 x 10 voxels.
For our analyses, discarded the first 13 frames in Z of the functional anatomy dataset, so that this volume has four times more frames in Z. as compared to the activity dataset.
B.2 ALIGNMENT
Since the acquired activity data shows significant spatio-tempont deformation, we developed a pipeline to align it-using elastic alignment preceeded by translational pre-alignment.
Static reference. The anatomy light-sheet volume was postprocessed to serve as a static reference for alignment: rescaled its intensity values for each frame to double precision, computed the temporal average over frames, and applied Contrast Limited Adaptive Histogram Equalization (CLAHE), using kernel of size 128x 128x 16 in XYZ, and 1024 bins.
Translational pre-alignment. For pre-alignment, we estimated per-timestep translational offsets in XYZ. align the activity dataset against static reference, using phase cross-correlation as implemented in scikit-image (van der Walt et al. (2014). For this, we linearly upsampled the activity volume four-fold in Z so that its size matched the reference. We temporally filtered the resulting offsets with median filter of size 128 (to avoid jitter in the estimates), and warped with a spline interpolation order of three.
Elastic alignment. To address remaining misalignment, we estimated per-timestep optical flows between the translationally pre-aligned volume against the static reference using the SOFIMA toolbox (Januszewski et al. 2024). We performed two rounds of elastic alignment at varying granularity, each of which consisted of the following consecutive steps:
1. Estimation: Flow fields were estimated using cross-correlation between patches while masking out background, Background was estimated heuristically using an active contours model and morphological operations. For round 1, patches of size 128 128 64 extracted with step sizes 32 x2x32xx 16 in XYZ were used. These sizes were halved for round 2.
2. Filtering: We filtered the resulting flow fields for local consistency. More specifically, we discarded patches that exceeded our threshold for absolute maximum movement magnitude (more than 20 voxels), absolute deviation from the 3x3 window median (more than 10 voxels), as well as ones for which the correlation peak did not fulfill our sharpness criterion (lower than 1.0).
3. Relaxation:
We modeled each frame of the functional activity volume cuboid 3d mass-spring mesh system, with the nodes of the mesh separated by 32 x2x 16 voxels in virtual O-length Hookean spring with a rest position at the location indicated by the flow field vector. We then allowed the system to relax using the FIRE algorithm (Bitzek et al.) 2006) and the following settings in SOFIMA: ko = 0.05, k = 0.1, di = 0.001. We used the relaxed mesh coordinate map representation of the transform warping the frames into alignment with the anatomical reference volume.
XYZ. We used the filtered flow field to connect every mesh node with at most
For final alignment, we combined all transforms into a single coordinate map and warped the raw data with spline interpolation order three. Alignment for example frame is illustrated in Fig. S8
4

---

Published as a conference paper at ICLR 2025
B.3 NORMALIZATION
We estimated baseline fluorescence in the aligned dataset by calculating the 8th percentile of each voxel using a temporal window 400 steps (~ min). We spatially median filtered the percentile volume with a 3 x 3 kernel, yielding the baseline volume Fo of the same size the aligned volume F. We then normalized each voxel in the aligned fluorescence data by calculating AF AF/F. FF, i.e.:
F F- AF/F
Fo
We used clipping to eliminate extreme outliers in the resulting value, with lower bound -0.25, and upper bound 1.5, obtain the normalized activity dataset.
B.4 SEGMENTATION
To segment individual neurons in the dataset, we used modified versions of flood-filling networks (FFNs; Januszewski et al. 2018). We applied FFNs to the functional anatomy data, training them on subvolumes of densely annotated neurons that were labelled manually, using the software Amira.
Preprocessing. The anatomy light-sheet volume was preprocessed for segmentation: as for alignment, we rescaled its intensity values for each frame to double precision, computed the temporal average over frames, and applied Contrast Limited Adaptive Histogram Equalization (CLAHE), using kernel of size 128 128x 16 XYZ, and 1024 bins. For FFN training, we additionally normalized this volume by 1) applying In(x + 1), 2) z-scoring. and 3) scaling each voxel by 0.5.
Training data. We manually labelled neurons three subvolumes taken from the postprocessed functional anatomy data, where each subvolume has a size of 100x x08 voxels in XYZ. A total of 2,176 neurons were labelled across the three subvolumes, selected to cover dense regions of varying data quality (e.g.. different amount of blurriness), following the protocol described in Petruncio & The CellMap Project Team (2023).
We used the volumetric annotations to generate examples for FFN training follows: positive example center points were generated for all labeled (non-zero) voxels after applying binary erosion twice the annotated neuron segments; negative example center points were taken for all unlabeled (background) voxels after dilating the annotated neurons twice.
Network architecture. We used residual convstack architecture (Januszewski al. 2018) with
33x33x33-voxx field of view (FOV), 8 residual modules, 32 feature maps in all internal convolu- tional layers, and layer normalization after the first convolution, at the beginning of every residual module, and before the final layer.
Training. We trained the FFN using the AdamW optimizer (Loshchilov & Hutter 2017 with learning rate of 0.01, batch size of 64. Training examples were formed by loading 33 x3x33x33 3333x33vvo0 3oxel subvolumes centered at the locations described above. We chose a smaller training example size and did not use FOV movement policy ("one-shot training") unlike Januszewski et al. (0088) because single FOV was sufficiently large fully contain even the largest somas.
Inference. We selected the checkpoint with the lowest variation of information between segmentation and ground truth annotations. We created binary mask consisting of voxels with normalized intensity values 0.5, eroded it twice and considered all its voxels as seed point locations for FFN inference. This yielded a segmentation of size 2048 x 1328 2288, with a total 82,536 putative neurons.
Postprocessing. We heuristically filtered out neurons, excluding any fragments that were smaller than 250 voxels. We excluded neurons whose bounding boxes intersected the first 17 sections because of rendering artifacts in the corresponding layers of the aligned functional data. In addition, we manually screened out 11 113 segments that were located outside of the brain. The postprocessed segmentation contains total of 71,721 putative neurons.
To evaluate the quality of our segmentation, we manually annotated 1,358 cells across eight evaluation subvolumes (held out from training: Fig. $9) and report metrics in Table ST
5

---

Published as a conference paper at ICLR 2025
B.5 TRACE EXTRACTION
To extract traces, we 4x-downsampled the segmentation to the Z-resolution of the normalized activity data. We then extracted time series for each neuron by averaging voxels falling into the respective segmentation mask, yielding a 7879x71721 trace matrix (time x neurons))
We sorted traces by similarity using rastermap (Stringer et 2023) for visualization purposes. We used 100 clusters, 200 PCs, locality parameter of 0.1, and time lag window parameter of 5,
B.6 STIMULUS FEATURE ENCODING
The visual stimulus shown to the fish is recorded in two channels during the entire experiment. For models to utilize this information, we encode these channels into a covariate matrix X of shape 7879 x26, where individual conditions are separated so they can be distinguished. All stimuli are sufficiently different, and, hence, shared encoding of variables is possible. After every encoding, there a single binary dimension that indicates the condition identity. For example, dimension is if the GAIN condition is active and otherwise 0.
1. Gain: encode in the first dimension as either low with -1 or high with 1.
2. Dots: encode in the third dimension as either - or for two present settings of orientation and coherence.
3. Flash: encode in the fifth dimension as -1 for dark and for bright.
4. Taxis:
encode in dimensions and as for dark and for bright on the left and right
side, respectively.
5. Turning:
encode velocity in dimension 10 and a sine-cosine encoding of the direction of the sine gratings in dimensions 11 and 12.
6. Position: dimensions 14 to 16 encode the grating type in one-hot fashion, and dimension encodes the delay in the range [0, 0.9].
7. Open loop: fixed stimulus so no encoding except the condition indicator in dimension 19,
8. Rotation: encode the direction of the rotation in dimension 20 and 1 for right and
leftward rotation, respectively.
9. Dark: only indicator variable in dimension 22 for open loop.
The last four variables of the covariate matrix are used to track specimen identity and are not meaningful in the benchmark context.
B.7 LIMITATIONS
Our dataset reflects current best practices in experimental design and data postprocessing, however, does have a number of limitations.
Acquisition. Constraints of phototoxicity limited the duration of experiment, and thus the total amount of activity data recorded. Because of the microscope's 's finite voxel throughput budget, there a fundamental trade off between the hignal-to-noise-ratio, and the temporal and spatial resolutions (particularly in the axial direction). We optimized the latter to be able to resolve individual cells, leaving us with volume scan rate of ~~ Hz, two orders of magnitude lower than the highest known action potential frequencies in zebrafish. We attempted to mitigate this by using a nuclear-targeted targeted GCaMP variant, optimized for the slower kinetics of calcium in the nucleus. side benefit this choice is that the somas, which in zebrafish predominantly cluster in dense groups, were easier to segment and to later match to the reference structural EM volume. Overall, our dataset should be considered to represent low-pass filtered version of the underlying high frequency electrical activity within the brain. Optical depth limitations of the imaging system also meant we could not image the complete brain, and some cells its ventral side thus not part of the LSFM volume. Finally, the substantial cost and effort necessary to process the volume EM data resulted in only a single specimen being imaged in both modalities.
Postprocessing. While elastic alignment of all frames to the anatomical reference volume elim- inated any visually noticeable large-scale deformation, occasional spatially and temporally local

---

Published as a conference paper at ICLR 2025
misalignments might remain the experiment done with an animal whose brain was still actively developing (we observed rare events of cell birth and migration). Since no proofreading was performed the cell segmentation segments are expected to suffer from low rate of split and merge errors (Table ST). These should he resolved once the EM volume reconstruction is available and aligned the LSFM data, since somas can be unambiguously detected in EM images.
using to extract activity traces, our putative soma
Signal. Visual inspection of the LSFM data reveals widely present structured noise in the form of vertical stripes oscillating at a high frequency, comparable or higher than that of the volume acquisition rate. The stripe widths are on the order of typical soma diameter. The genesis of these stripes is thought to have to do with instabilities in the illumination system in the microscope. They can distort the extracted activity trace signals, though we expect their impact to be limited due the disparity between their frequency and the characteristic timescales of the calcium dynamics.
We have attempted to filter them out with heuristic and machine learning- -ased approaches, as well as simple frequency band filtering, but were not able obtain a visually satisfying result for which we were also confident that all relevant underlying signal was retained.
In addition to the spurious oscillations and correlations induced by the stripes, scattered fluorescence from adjacent cells might cause some amount of signal mixing, even though our segmentation assigns voxels to cells in binary manner.
Generalizability. We used standard behavioral assays for fictive behavior, as outlined in subsec tion A.4 These do not probe the full behavioral repertoire of larval zebrafish, but were selected to accommodate the limits of the experimental setup. Specifically, only visually evoked behaviors were tested over limited time period of about 2h. The reflexes tested in the experiment universal for zebrafish, but might not apply to other fish species. The zebrafish neural architecture is not fully stereotyped and is expected differ between specimens. Neuronal activity forecasting models trained for the ZAPBench data are therefore likely to lose predictive power when directly applied to recordings from other specimens.
B.8 PROGRESS ON ELECTRON MICROSCOPY VOLUME RECONSTRUCTION
The electron microscopy volume has been aligned and segmented, and is currently undergoing proofreading (manual correction of reconstruction mistakes) by an expert team. The volume covered by the EM images is superset of that recorded by the lightsheet microscope. We identified about 190,000 putative somas within the volume, which includes both neurons and other cell types, like glia. We performed a preliminary registration of the two volumes, obtaining good quality matches (within 10 um of a semi-automnn aonitautomati placed correspondence point) for about 45,000 cells (Fig. $10..
7

---

Published as a conference paper at ICLR 2025
C MODELS
C.I
LINEAR
The linear model consists of single dense layer. We also implemented the normalization proposed for the NLinear model in Zeng et al. (203)) but did not find this to increase performance. We use the AdamW W (Loshchilov Hutter 2017) optimizer with a learning rate of 10-3, weight decay of 10- and early stopping the validation set loss.
C.2 TIDE
For the TiDE model (Das et al. layers, decoder output dimensionality of 32, and layer reversible instance norm. We the AdamW (Loshchilov & Hutter 2017) optimizer with a learning rate of 10-3, weight decay of 10-1, and early stopping the validation set loss.
2023 we use a hidden layer size of 128, 2 encoder and decoder
Based on the ablations in Fig. S7 we report TiDE without past and static covariates in the main text, as this variation showed best overall performance.
C.3 TSMIXER
For TSMixer (Chen et al. 2023), we compared different numbers of blocks, MLP dimensions, and normalization schemes on the validation set. Based on these results, we selected an architecture with 2 blocks, MLP dimension of 256, and no instance norm for C 4, and 2 blocks, MLP dimension 128, and reversible instance norm for C = 256. We use the Adam (Loshchilov & Hutter 2017) optimizer with learning rate of 10-3, weight decay of 10-4, and early stopping on the validation set loss.
C.4 TIME-MIX
Time-Mix is a version of TSMixer in which feature mixing modules are ablated. Similar to TSMixer, we performed hyperparameter selection based on the validation set. For C' = 4 and C' = 256 we used an architecture with blocks. We used reversible instance norm for = 256 but not for C = 4. We use the A (Loshchilov & Hutter, 2017) optimizer with learning rate of 10-3, weight decay of 10 10-4, and early stopping on the validation set loss.
C.5 U-NET
The U-Net downsamples the input video by a factor of in XY using area-averaging 512x288 which greatly reduces computational requirements and statistically performs equally well as models with full resolution input. We use diree-dimnsiioal convolutions throughout the network, and treat the temporal context C' as input features instead of the grayscale channel, The first resampling block uses factor 2 in XY and does not downsample 7 to achieve roughly isotropic resolution in the three dimensions. We then use three further resampling blocks with factor 2 in all three spatial dimensions down to a voxel resolution of roughly 26 yum and shape 32 x 9x We use three residual blocks at each resolution, except at the lowest resolution where we use four, and fix 128 features throughout the U-Net. Each block uses a pre-activation design (He et al. 2016), with each two group normalization layers (Wu & He 2018) using 16 groups. Swish activations (Ramachandran et al. 2017), and 3 convolutions. For the output, we upsample twice to obtain the original resolution, and use residual block per resolution, but with reduced feature dimension of 32.
We condition every block on lead time between and 32, as proposed by Andrychowicz et al. (2023) for weather forecasting, using FiLM layers (Perez et al. 2018). Therefore, the output layer maps to a single three-dimensional frame that forecasts the video for the given lead time.
We use the neuron mask to compute and optimize the same MAE loss on the traces, and the AdamW (Loshchilov & Hutter 2017) optimizer with a learning rate of 10 decayed to 10 over 500, 000 steps for C' 4 and 250, 000 steps for C 256. For C' 256, we use only the initial downsampling and no further U-Net blocks because we found that these lead to overfitting.
8

---

Published as a conference paper at ICLR 2025
D DATA AND CODE RELEASE
Datasets, all relevant code for the benchmark, and interactive visualizations are available through dedicated project website: ptw. for zapbench is at: gi rithub. com www.ooootgoooeeeeooooooooeoe research/zapbeantee
The code repository
D.1 DATASETS
Raw datasets as well as postprocessed versions are each terabyte-sized. We host them on cloud storage in format that allows streaming access. This enables visualization with Neuroglancer (Maitin- Shepard et al. 2021), web-based viewer for volumetric datasets. A screenshot is in Fig. S12 The project website has links to browser-based views of volumes, including the aligned data, segmentation, and trace matrix.
To enable distributed volumetric data postprocessing, developed custom framework based on TensorStore (TensorStore developers 2024). More specifically, we used TensorStore's 's virtual views feature in combination with Beam (Apache Beam developers) 2024). We open-sourced this framework as part of Blakely et al. (202)). Configuration files are released as part of the zapbench repository.
D.2 BENCHMARK
All forecasting models used in this paper were implemented in jax (Bradbury et al. 2018). We implemented custom data loaders on top Grain (Grain developers, 2024), which are easily usable with frameworks such jax and PyTorch (Ansel et al. 2024). We open-sourced all relevant code the zapbenchhrepository,
To interactively visualize forecasts, developed custom web-based viewer top of three. js (Three.js js developers) 2024). An example view is in Fig. STT and a video demonstration is uploaded as supplementary material along with the submission. Links to visualizations for all individual models that have been benchmarked are made accessible through our project website.

---

Published as a conference paper at ICLR 2025
E
SUPPLEMENTARY TABLES
Table S1: Evaluation of segmentation on eight manually annotated subvolumes in terms of counts and variation of information (VOI) metrics. See Fig. S9 for spatial locations and extent of subvolumes.
Subvolume
Annotated Count
Predicted Count
VOI Split
VOI Merge
VOI Total
E1
112
158
0.719
1.342
2.060
E2
258
324
0.359
0.685
1.044
E3
214
131
0.548
4.137
4.685
E4
168
181
0.648
1.331
1.979
E5
166
168
0.502
2.085
2.587
E6
150
176
0.644
1.733
2.377
E7
128
123
0.603
1.519
2.121
E8
162
189
0.758
1.884
2.642
10

---

Published as a conference paper at ICLR 2025
F SUPPLEMENTARY FIGURES
step ahead
steps ahead
steps ahead
steps ahead
32 steps ahead
0.035
val set MAE
0.026
0.020
0.015
mean W-1
W-2
mean Wid
mean W=8
mesn W/-32
Wape
mean W-n -12
mean W- W-258
mean W=1
mean W-2
Widd
mean WF8
mean We 18
mean W-
mean WEBe
mean 128
mean W-256
mean WF
mean W--2
mean Wid
mean W
mean Wi 16
mean WeBe
meen W- 128
meen W-296
mean WSt
mean W-2
mean W--d
mean Way
mean Wi W-18
mean V
mean W 128
W-236
mean We1
mean W-2
mean Wid
mean WF8
mean We
mean W=
mean W=04
mean W- W-128
mean W=296
mean
mean
mean
Figure S1: Naive baseline. Mean baseline with varying window length W evaluated on validation
set MAE across conditions (lower is better).

---

Published as a conference paper at ICLR 2025
A
71,721 neurons
HOLD-OUT CONDITION (TANIS)
7,879 steps
7,879 steps
AFF
Figure S2: Stimulus-evoked response. A. For each condition, we computed the stimulus-evoker response by aligning activity according stimulus phase and computing the average (excluding periods in the test set). Here, we repeat this evoked response for each repetition in the condition. Color represents normalized activity (AF/F) with brighter colors indicating higher activity. The original aspect ratio of neurons to timesteps is approximately 9:1. Note that this representation squeezes the neuron dimension relative the time dimension. Conditions are separated by white vertical lines. Alternating colors on horizontal line below traces indicate stimulus repeats for a given condition. B. Activity traces for comparison the evoked response. Unlike Fig. 3 neurons sorted by similarity.
12

---

Published as a conference paper at ICLR 2025
step ahead
steps ahead
8 steps ahead
16 stops ahead
32 slops ahead
0.030
gain
MAE
0.025
0.020
0.018
0.030
dots
MAE
0.025
0.020
6
0.018
flassh
MAE
0.025
0.020
0.015
MAE
0.025
0.000
0.015
position
MAE
0.025
6
0.020
6
0.016
MAE
0.025
0.020
0.015
rolation
MAE
0.025
0.000
0.055
dark
MAE
0.020
0.015
Ineer
unat
Ineer
unat-
Ineear
unet-
Ineer
unet-
Ineer
time
time
tme
tme
Figure S3: Short context results per condition. Points are individual runs (3 seeds per model). Performance of the naive baseline is marked by the black line. MAE values are clipped axis limits. The dotted black line indicates performance of the mean baseline, the solid line is the stimulus baseline. Short context results evaluated on mean squared error (MSE) are reported in Fig.
13

---

Published as a conference paper at ICLR 2025
step ahead
steps ahead
8 steps ahead
16 stops ahead
32 slops ahead
0.035
0.030
MAE
0.025
0.020
0.015
0.085
0.080
dots
MAE
0.025
0.020
0.015
0.035
flash
MAE
0.020
0.015
0.036
0.000
luming
MAE
0.020
0.015
0.055
0.080
position
MAE
0
0.015
0.055
0.000
MAE
0.015
0.035
0.000
rotation
MAE
0.035
0.000
MAE
0
0
0.018
Ineer
timemix
undt
Ineer-
unit
Ineer
uni
Inear
unit
Inear
timemmi
undt
time-mm
Figure S4: Long context results per condition. Points individual runs (3 random seeds per model). Performance of the naive baseline is marked by the black line. MAE values are clipped axis limits. The dotted black line indicates performance of the mean baseline, the solid line is the stimulus baseline. Long context results evaluated on mean squared error (MSE) are reported in Fig. S6
14

---

Published as a conference paper at ICLR 2025
step ahead
steps ahead
sieps ahead
steps ahead
32 steps ahead
0.0035
0.0027
MSE
0.0018
0.0001
0.0098
dots
MSE
1
0.0010
0.0001
0.0035
0.0027
flash
MSE
0.0018
0.0010
0.0001
0.0027
turning
MSE
0.0018
0.0010
0.0001
0.0035
0.0027
position
MSE
0.0018
0.0005
open loop
MSE
0.0018
0.0010
0.0095
-..-..------....... -8.-..-...................
rotation
MSE
0.0010
0.0001
0.0035
0.0027
darkk
MSE
0.0018
0.0010
0.0001
Ihear
unat
unat
tamber
unet
50c
tomoer
time-mic mix
linear
Nenberr
time
time
time
Figure S5: Short context results per condition by mean squared error (MSE). Points are indi- vidual runs (3 random seeds per model). Performance of the naive baseline is marked by the black line. MSE values are clipped to axis limits. The dotted black line indicates performance of the mean baseline, the solid line is the stimulus baseline.
15

---

Published as a conference paper at ICLR 2025
step ahead
4 steps ahead
sleps ahead
16 steps ahead
32 steps ahead
0.0080
MSE
0.0010
0.0001
0.0036
0.0027
dots
MSE
0.0018
0.0010
0.0001
0.0027
flash
MSE
0.0018
0.0010
0.0001
0.0035
0.0027
turning
MSE
0.0018
0.0035
0.0027
position
MSE
0.0018
0.0010
0.0001
0.0035
open loop
MSE
0.0010
0
0.0001
0.0035
0.0027
rotation
MSE
0.0018
0.0010
0.0001
0.0027
dafk
MSE
0.0018
""""""""""""""
tnar
rde-
unat
Inwar
Ummo-mi
unat
timter
Umma-mik
unet-
Inser
Ide
banker
lime-mik-
Inear
rdc
bankerr
Figure S6: Long context results per condition by mean squared error (MSE). Points are indi- vidual runs (3 random seeds per model). Performance of the naive baseline is marked by the black line. MSE values are clipped axis limits. The dotted black line indicates performance of the mean baseline, the solid line is the stimulus baseline.
16

---

Published as a conference paper at ICLR 2025
step ahead
steps ahead
steps ahead
16 steps ahead
32 steps ahead
0.035
short context
0.030
MAE
0.025
0.020
0.035
long context
0.080
MAE
0.015
ful model
without past
model
without pest
model
without past
ful model
without past
ful model
without stalio.
without stalic
without stalic.
without stalic
without stalic.
without stalic.
without stalic.
without stalic.
fature.
without
without stalic.
without static.
withou
Figure S7: TiDE ablations. We focused model selection variants without past stimulus covariates.
We skipped the full model for = 256, as C = results did not suggest superiority of the full model.
17

---

Published as a conference paper at ICLR 2025
50mm
Figure S8: Alignment for example frame. Left: Final frame (in time) of raw calcium activity at single Z-depth in green, overlaid with anatomical reference in blue. Right: Same frame after alignment, same colors.
18

---

Published as a conference paper at ICLR 2025
E3
E6
E5
E2
E4
E7
3amm
Figure 89: Eyaluation subvolumes. Locations eight evaluation subvolumes selected for manual annotation. Cells in these volumes annotated across the entire 7-depth (1358 cells total). Results are reported in Table SI
19

---

Published as a conference paper at ICLR 2025
50 pm
Figure S10: Registration and reconstruction of the electron microscopy volume. Top two panels: cross sections through LSFM data overlaid on top of EM data. Bottom panel: sample neuron reconstructions throughout the brain.
20

---

Published as a conference paper at ICLR 2025
thangloner
sottinge
Figure $11: Fluroglancer screenshot. Predictions of all benchmarked models are stored such that they can be visualized interactively using custom browser-based viewer we built for this purpose. Trace-based predictions displayed as a circle with color indicating activity. Panels show ground-truth, predicted activity, and MAE respectively. The viewer is accessible at: google le research. github. io/ zapbench
projected on the location of neurons in 3 dimensions, where each neuron
21

---

Published as a conference paper at ICLR 2025
Teargeenno
Incigils
1622
(405mn]
664
(4jm) 36
(P14.1633:3mg
3939
foctislyy 2000
@
D
100 pm
Sections
Figure $12: Neuroglancer screenshot. Volumes are stored such that they can be browsed interac- tively, using neuroglancer, WebGL-based viewer for volumetric data -heparr et al. 2021).
Views can be accessed at: google aessercc..iithee io/ zapbench
22

---

Published as a conference paper at ICLR 2025
REFERENCES
Misha B. Ahrens, Jennifer M. Li, Michael B. Orger, Drew N. Robson, Alexander F. Schier, Florian Engert, and Ruben Portugues. Brain-wide neuronal dynamics during motor adaptation in zebrafish. Nature, 2012. doi: 10. 1011000/tturrl 1057
Marcin Andrychowiez, Lasse Espeholt, Di Li, Samier Merchant, Alexander Merose, Fred Zyda, Shreya Agrawal, and Nal Kalchbrenner. Deep learning for day forecasts from sparse observations. arXiv, 2023.
doi: (0448550aaxx..22066.60777
Jason Ansel, Edward Yang, Horace He, Natalia Gimelshein, Animesh Jain, Michael Voznesensky, Bin Bao, Peter Bell, David Berard, Evgeni Burovski, Geeta Chauhan, Anjali Chourdia, Will Constable, Alban Desmaison, Zachary Vito,o Elias Ellison, Will Feng. Jiong Gong, Michael Gschwind, Brian Hirsh, Sherlock Huang, Kshiteej Kalambarkar, Laurent Kirsch, Michael Lazos, Mario Lezcano. Yanbo Liang, Jason Liang, Yinghai Lu, C. K. Luk, Bert Maher, Yunjie Pan, Christian Puhrsch, Matthias Reso, Mark Saroufim, Marcos Yukio Siraichi, Helen Suk, Shunting Zhang, Michael Suo, Phil Tillet, Xu Zhao, Eikan Wang, Keren Zhou, Richard Zou, Xiaodong Wang, Ajit Mathews, William Wen, Gregory Chanan, Peng Wu, and Soumith Chintala. Pytorch 2: Faster machine learning through dynamic python bytecode transformation and graph compilation. In ACM International Conference on Architectural Support for Programming Languages and Operating Systems, 2024. doi: 10. 10.114576200665334400666
Apache Beam developers. Apache beam: Unified programming model for batch and streaming data processing.. 2024. https:/::////
c apache/
Armin Bahl and Florian Engert. Neural circuits for evidence accumulation and decision making in larval zebrafish. Nature Neuroscience, 2020. doi: 10. 10.10388.44559991944553333
Erik Bitzek, Pekka Koskinen, Franz Gähler, Michael Moseler, and Peter Gumbsch. Structural relaxation made simple. Phys. Rev. Lett., 2006. doi: 10.1103PhyyskeeLee.....110.
Tim Blakely, Peter H. Li, Jan-Matthis Lueckmann, Michal Januszewski, and Alexander Immer. Google connectomics utilities. Zenodo, 2024. doi: 10.5281//zendoo.11577777
James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Milnn and Qiao Zhang. JAX: composable transformations of Python+NumPy programs, 2018. URL
htttp://gitee com/ jammmajjamm/jax
Susan E. Brockerhoff, James B. Hurley, Ulrike Janssen-Bienhold, Stephan C. F. Neuhauss, Wolf- gang Driever, and James E. Dowling. behavioral screen for isolating zebrafish mutants with visual system defects. Proceedings of the National Academy of Sciences, 1995. doi: 10.1073/p nas, nas.92.22 Nis.92.23.105455
Harold A. Burgess and Michael Granato. Modulation of locomotor activity in larval zebrafish during light adaptation. Journal of Experimental Biology, 2007. doi: 10.1242//eb.0033999
Alex B. Chen, Diptodip Deb, Armin Bahl, and Florian Engert. Algorithms underlying flexible phototaxis in larval zebrafish. Journal of Experimental Biology, 2021. doi: 10.1242//eb.2383366
Si-An Chen, Chun-Liang Li, Nate Yoder, Sercan O Arik, and Tomas Pfister. TSMixer: An All-MLP architecture for time series forecasting. Transactions on Machine Learning Research, 2023.
Abhimanyu Das, Weihao Kong, Andrew Leach, Shaan Mathur, Rajat Sen, and Rose Yu. Long-ten term Forecasting with TiDE; Time-series Dense Encoder. Transactions on Machine Learning Research, 2023.
Grain developers. Grain: Library for reading data for training and evaluating jax models., 2024. URL https: thw..
e/ e/grain
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In European Conference on Computer Vision, 2016.
23

---

Published as a conference paper at ICLR 2025
Michat Januszewski, Jörgen Kornfeld, Peter H. Li, Art Pope, Tim Blakely, Larry Lindsey, Jeremy Maitin-Shar Sheepadd, tion of neurons with flood-filling networks. Nature Methods, 2018. doi: 10.1038/441592-018
Mike Tyka. Winfried Denk, and Viren Jain. high-precision automated reconstruc-
0049-4
Michal Januszewski, Tim Blakely, and Jan-Matthis Lueckmann. SOFIMA: Scalable optical flow- based image montaging and alignment. Zenodo, 2024. doi: 7.5288//zenod..1053342
Takashi Kawashima, Maarten F. Zwart, Chao-Tsung Yang, Brett D. Mensh, and Misha B. Ahrens. The Serotonergic System Tracks the Outcomes of Actions Mediate Short-TTem Motor Learning. Cell, 2016. doi: 10. /011///.eel.221600090
Ilya Loshchilov and Frank Hutter. Fixing weight decay regularization in adam. arXiv, 2017. doi: 01.48500/rXii.1111..55
Jeremy Maitin-Shepane Alex Baden, William Silversmith, Eric Perlman, Forrest Collman, Tim Blakely, Jan Funke, Chris Jordan, Ben Falk, Nico Kemnitz, tingzhao, Chris Roat, Manuel Castro, Sridhar Jagannathan, moenigin, Jody Clements, Austin Hoag, Bill Katz, Dave Parsons, Jingpeng Wu, Lee Kamentsky, Pavel Chervakov, Philip Hubbard, Stuart Berg, John Hoffer, Akhilesh Halageri, Christian Machacek, Kevin Mader, Lutz Roeder, and Peter H. Li. googleeneurrolancee. Zenodo, 2021. doi: 10.528 1772ood.5.00
Yu Mu, Davis Bennett, Mikail Rubinov, Sujatha Narayan, Chao-Tsung Yang. Masashi Tanimoto, Brett D. Mensh, Loren L. Looger, and Misha B. Ahrens. Glia Accumulate Evidence that Actions Are Futile and Suppress Unsuccessful Behavior. Cell, 2019. doi: ce.10116).el1.2201005.....
Eva A. Naumann, James E. Fitzgerald, Timothy W. Dunn, Jason Rihel, Haim Sompolinsky, and Florian Engert. From Whole-Bran Data to Functional Circuit Models: The Zebrafish Optomotor Response. Cell, 2016. doi: 10.10166f.e11..2166100000 10.019
Michael B. Orger, Matthew C. Smear, Stuart M. Anstis, and Herwig Baier. Perception of Fourier and non-Fourier motion by larval zebrafish. Nature Neuroscience, 2000. doi: 10. (0.1038/80649)
Ethan Perez, Florian Strub, Harm De Vries, Vincent Dumoulin, and Aaron Courville. Film: Visual reasoning with general conditioning layer. In Proceedings of the AAAI conference on artificial intelligence, 2018.
Alyson Petruncio and The CellMap Project Team. Labeling of dense light data in amira 3d 2022.1 classic segmentation workroom. 2023, doi: 10. 10.17504/pottccols. io dm6ggix6pgzz/vv
Prajit Ramachandran, Barret Zoph, and Quoc V Le. Searching for activation functions. arXiv, 2017. doi: 10.48550/arXiv. 1iv.:/XXX..1111055...
Carsen Stringer, Lin Zhong, Atika Syeda, Fengtong Du, Maria Kesa, and Marius Pachitariu. Rastermap: A discovery method for neural population recordings. bioRxiv, 2023. doi
10.1101//002300.225555
TensorStore developers. Tensorstore: Library for reading and writing large multi-dimenssonall arrays,
2024. WtlL/httss:/githhute com /t/g///olllgggorereeeeeoe
Three.js developers. three.js: Javascript 3d library, 2024. URL https://github.com mnrdoob/three..
Stéfan van der Walt, Johannes L. Schönberger, Juan Nunez-Iglesias, François Boulogne, Joshua D. Warner, Neil Yager, Emmanuelle Gouillart, Tony Yu, and the scikit-image contributors. Scikit-
image: Image processing in Python. PeerJ, 2014. doi: 10.7717/peerj
Nikita Vladimirov, Yu MU, Takashi Kawashima, Davis V. Bennett, Chao-Tsung Yang, Loren L. Looger, Philipp Keller, Jeremy Freeman, and Misha B. Ahrens. Light-sheet functional imaging in fictively behaving zebrafish. Nature Methods, 2014. doi: 10. 1038/mmmthh.3400
Sébastien Wolf, Alexis M. Dubreuil, Tommaso Bertoni, Urs Lucas Böhm, Volker Bormuth, Raphaël Candelier, Sophia Karpenko, David G. C. Hildebrand, Isaac H. Bianco, Rémi Monasson, and Georges Debrégeas. Sensorimotor computation underlying phototaxis zebrafish. Nature Communications, 2017. doi: 10. 00.10388441477777-003000.3.
24

---

Published as a conference paper at ICLR 2025
Yuxin Wu and Kaiming He. Group normalization. In European Conference Computer Vision, 2018.
En Yang, Maarten F. Zwart, Ben James, Mikail Rubinov, Ziqiang Wei, Sujatha Narayan, Nikita Vladimirov, Brett D. Mensh, James E. Fitzgerald, and Misha B. Ahrens. A brainstem inte- grator for self-location memory and positional homeostasis in zebrafish. Cell, 2022. doi [0.10166j.ce12222211..222
Ailing Zeng. Muxi Chen, Lei Zhang, and Qiang Xu. Are Transformers Effective for Time Series Forecasting? In AAAI Conference Artificial Intelligence, 2023, doi: 10.1609/aaii 2337
25


Reviews:
[{"review_id": "rfNCAfNklw", "reviewer": "Reviewer_Q3aS", "summary": "In this paper, the authors built a benchmark dataset for whole-brain activity prediction in zebrafish. This dataset contains a 4d light-sheet microscopy recordings of over 70,000 neurons in a larval zebrafish brain and corresponding segmentations of the neurons. To illustrate how well current time-series prediction models do, they benchmarked several models on this dataset.", "questions": "1. For the 2000 neurons manually annotated, how accurate they are? Are there any metrics to measure the segmentation accuracy?\n2. The authors are encouraged to discuss how likely the results abtained from a single test zebrafish to other live zebrafishes.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Building such kind of dataset for whole-brain activity prediction and understanding is quite valuable.\n2. Solid study with detailed procedures described, e.g. 2000 neurons were manually labeled as training data.", "weaknesses": "1. If I understand correctly, this dataset is built based on a single zebrafish. So benchmarks acquired from this dataset and evaluations may not be well generalized to other zebrafishes.\n\n2. Regarding time-series predictions, there are no sota methods employed in this study, e.g. temporal fusion transformer, informer, n-beats, and deepar to name a few.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4urp284nhH", "reviewer": "Reviewer_tKPW", "summary": "The paper focuses on understanding the predictability of neuronal activity using zebrafish as a model organism. The study introduces the Zebrafish Activity Prediction Benchmark (ZAPBench), a structured framework designed to forecast neuronal behavior at single-cell resolution. The primary research question is centered around how accurately future neuronal activity can be predicted based on prior observations. This investigation aims to uncover fundamental predictability limits in complex neural systems, providing a formal benchmark that can guide and standardize future research in this domain.\n\nKey contributions of the paper include:\n\nThe creation of ZAPBench, a dataset tailored for predictive neuroscience, focusing on single-cell activity in zebrafish.\nA comprehensive methodology to assess prediction accuracy across various models.\nFoundational insights into the neural predictability that could enhance brain function understanding and improve modeling standards in neuroscience.\n\nThe framework is intended to encourage reproducibility and provide a standard for benchmarking advancements in neuronal activity prediction methods.", "questions": "How did you select the specific prediction models tested in the benchmark?\n\nIt would be helpful to understand the rationale behind the chosen models. Are they meant to represent a range of prediction approaches, or were they selected based on prior performance in related studies? This clarification can provide insight into how comprehensive the benchmark is in capturing predictive capabilities across model types and whether the authors plan to expand the set of models in future work.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "Originality:\nThe paper demonstrates originality by addressing the challenge of predicting neuronal activity with a new focus on zebrafish, providing a dataset that enables prediction at single-cell resolution. By framing a well-defined benchmark (ZAPBench) and emphasizing zebrafish as a model organism, the study introduces a valuable structure for exploring neuronal predictability. This originality lies in applying prediction models to a high-resolution, single-cell level dataset, which pushes beyond the coarser frameworks commonly used in neural forecasting.\n\nQuality:\nThe research is grounded in solid methodology, with a detailed approach to constructing the ZAPBench framework and rigorously evaluating prediction models. By providing an extensive dataset and clear benchmark criteria, the authors lay a high-quality foundation for reproducible research. The paper's methodology allows for systematic assessment and comparison of various forecasting methods, thereby contributing significantly to the reliability and comprehensiveness of the research.\n\nClarity:\nThe paper’s clarity is evident in its systematic presentation of the problem, methodology, and dataset. It carefully articulates the goals, including the potential limits of predictability in neuronal systems, and provides transparent descriptions of the benchmarks and data structures. Additionally, the focus on single-cell resolution is well-explained, helping readers grasp the significance of such granular predictability in the context of neuroscience.\n\nSignificance:\nThe significance of this work is substantial, given the pressing need for robust frameworks in neuroscience to predict and understand brain function. By creating ZAPBench, the authors lay the groundwork for a standardized method to evaluate neuronal predictability, likely stimulating future research and practical applications in brain modeling and potentially even in clinical neuroscience. The dataset and benchmark fill a critical gap in neural forecasting studies, advancing the field's understanding of predictability within complex biological systems and establishing a model that could inspire cross-disciplinary innovations.", "weaknesses": "1. Limited Scope of Evaluation Models:\nWhile the paper establishes ZAPBench as a benchmark for predicting neuronal activity, it would benefit from a broader exploration of prediction models. Presently, if only a limited selection of forecasting models is tested, it may not fully illustrate the benchmark's potential to evaluate diverse approaches across neural network architectures, classical machine learning algorithms, or even emerging time-series models. Including additional model categories or hybrid approaches could better demonstrate ZAPBench’s versatility and the applicability of its predictive insights across various methodologies.\n\n2. Lack of Real-World Validation:\nAlthough ZAPBench provides a controlled dataset for benchmarking, validation in diverse environments or settings would strengthen the applicability of its findings. For instance, if the framework could be tested or at least hypothetically mapped to real-time or less controlled environments, this would add value by showing the benchmark’s robustness and adaptability. Additionally, introducing comparisons with other neural prediction benchmarks, if available, could highlight the strengths and limitations of ZAPBench in relation to existing datasets and benchmarks, increasing confidence in its practical utility.\n\n3. Limited Analysis on Predictability Boundaries:\nWhile the paper raises the important question of the fundamental limits of predictability, it could benefit from a deeper analysis or even a dedicated section on the boundaries of predictability within neural systems. Specifically, the authors could include additional metrics or scenarios that highlight instances where predictability fails or reaches theoretical limitations. This would provide researchers with a clearer understanding of where model improvements might be focused, enhancing the practical impact of the benchmark.\n\n4. Insufficient Discussion on Potential Applications:\nWhile the framework is designed as a benchmark, an expanded discussion on its practical applications could add significant value. For instance, the authors could discuss how ZAPBench might be used in specific fields such as neuroprosthetics, disease modeling, or behavioral neuroscience. By laying out clear, actionable scenarios in which this benchmark might influence real-world applications or interdisciplinary studies, the authors could better contextualize the benchmark's significance and foster broader adoption.\n\n5. Dataset Generalization and Potential Biases:\nIt’s unclear if the zebrafish dataset is representative of broader neuronal predictability patterns, especially for application to other species or contexts. Providing a more explicit discussion on the generalizability of the zebrafish model—or any limitations it may present—could offer readers a clearer sense of the benchmark's scope. This could also include the authors' considerations or guidelines on potential dataset biases and how they might affect downstream applications or interpretations of the benchmark's results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "l44zTIekO1", "reviewer": "Reviewer_v2hF", "summary": "The authors will share a dataset of neuronal activity (both 1D traces and 4D movies), a benchmark metric to rank models by their predictive power, and a comparison of different classes of models using this metric.\n\nThis dataset and benchmark contribution can accelerate the development of predictive models of the dynamics of neuronal systems. \n\nSuch datasets and benchmarks are timely for the field and can spur innovations in representation learning and predictive modeling (fully data-driven, fully physics-based, and hybrid models). In turn, the models can accelerate the analysis of time series data acquired in neuroscience and potentially be helpful for other time-series modeling problems, such as weather prediction.", "questions": "* Why does the paper report data only from one zebrafish brain? What are the key challenges, and given the work you have done so far, can you collect data from more brains?\n* Why does the paper not report estimates of action potential? I am not an expert in the properties of GCaMP. Can you clarify the limits of accuracy with which action potential be estimated from GCaMP activity?\n* JAX's adoption is growing, but PyTorch is still more widely used. What can you do so that the community can reuse your code, e.g., wrap it with CLI and containerize the code?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "* The dataset is comprehensive, and the tooling to preprocess and explore the dataset is well-developed.\n* The choice of sharing both 1D traces and 4D movies is sound because it encourages the development of various models with the same dataset.\n* The authors report useful naive baselines and performance with useful classes of models.\n* The relative strengths of the models and areas of improvement are articulated.", "weaknesses": "* Insufficient data to evaluate model generalization across brains: The data is from only one fish brain, which poses challenges in training models that generalize across multiple brains. The authors point out that the data took ~2 hours to acquire but much longer to preprocess. Now that they have the pipeline established, I think they should image more fish (2-8) and process them together.  Given the technological challenges, I understand that connectome can be mapped only in one or two fish. However, the live imaging data across many fish can guide which connectomes should be built.\n* The MAE (mean absolute error) metric is insufficient to develop probabilistic models and models of underlying biology:  An important class of models is the ones that predict the probability of underlying biology activity, e.g., the probability of action potential from GCaMP activity or estimating functional connection between pairs of neurons from synchrony of activity.  The mean absolute error metric per neuron is unlikely useful for ranking such models. The authors should introduce a metric that enables modeling of the statistical distribution of activity per neuron or pair of neurons.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bZZrmIjUCL", "reviewer": "Reviewer_8vrJ", "summary": "The authors provide an image dataset of neuron activity in an almost entire zebrafish brain under a variety of visual stimuli. The activity dataset is acquired at a resolution of 406nm x 406nm x 4um x 914ms using calcium imaging. The image volumes are spatially aligned and segmented using machine learning to produce activity traces for 71,721 putative neurons. The authors split the data into training/validation/test sets then evaluate several time-series forecasting methods in their ability to predict neural activity. Parametric models generally show modest improvements over naive models, especially with longer context.", "questions": "I encourage the authors to include answers to these questions in the paper, to the extent possible:\n- Do any other whole-brain, cellular-resolution activity datasets exist publicly for zebrafish?\n- Are there any preliminary results in the EM analysis? e.g. Can you show preliminary results of registering cell bodies across EM and calcium imaging? Can you show the reconstructed morphology of one of the neurons in the calcium data? I think some results like this are necessary to justify inclusion in this paper.\n- Why do you choose MAE over MSE, which might have advantages in optimization? Zeng et al., 2023 includes both MAE and MSE.\n- Doesn't the output of the stimulus model also depend on past covariates and past activity? The equation only shows future covariates as an input.\n- As someone new to calcium signal analysis, an MAE of < 0.03 seems surprisingly good for the naive models - why do you think this is? I think computing the global variance of the data, or the variances of the individual neurons could help contextualize whether a 3% average error is good or not. This is related to my earlier comment about colorbars on the trace heatmaps.\n- \"Variability due to seeding\" - does this mean initialization of the model parameters? The term \"seed\" is also used in FFN inference, which potentially overloads the term.\n- How does the stimulus baseline work for the TAXIS dataset, where there is no training dataset available for lookup?\n- The flood-filling network approach was designed for tracking neuronal processes, but the segmentation task here is cell body instance segmentation", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 4, "strengths": "The authors provide a comprehensive, cellular-resolution calcium-imaging dataset which they process for alignment and cell segmentation which, to my knowledge, is a unique contribution to publicly available neuroscience datasets. Their data allows computational researchers to test neural activity forecasting models with little to no preprocessing, lowering barriers to this kind of research. The authors also test a diverse collection of forecasting algorithms to establish performance standards.", "weaknesses": "Introduction\n- The authors could do more to describe which cellular-resolution calcium imaging datasets already exist. While the authors argue that their contribution is to the forecasting community, I think some publicly available datasets could be trivially processed for this goal as well. Specifically, Nguyen et al., 2016 (which they reference), and the MICrONS dataset arguably serve offer similar information as the authors' dataset, in other organisms.\n\nFigures\n- It is hard to see much detail in several of panels with whole-brain views. Consider enlarging panels or adding more insets to, especially, Fig 1B, Fig 2C\n- It is hard to interpret Fig 2B, I encourage alternative registration visualizations e.g. showing deformed gridlines, overlaid images, and/or differences between overlaid images.\n- Trace heatmaps (Fig. 3 and S2) could use colorbars, and, if possible, a graphic (aligned in time) showing the treatment conditions e.g. marking the light vs. dark conditions in the FLASH phase.\n\nSegmentation\n- It is hard to determine the accuracy of segmentation, a better visualization of the segmentation results and/or performance on a validation set would help evaluate how well the segmentations can be trusted.\n\nEM\n- The authors mention future plans of registering EM images, but in the absence of any concrete results, I don't think this should be included in the paper beyond mentioning it in the Future Work section.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Spotlight)