Dynamical Modeling of Behaviorally Relevant Spatiotemporal Patterns in Neural Imaging Data
Authors: Sayed Mohammad Hosseini
Maryam Shanechi
Venue: ICML 2025 poster (2025)
Keywords: Deep Learning
Dynamical Modeling
Neural Imaging
Behavior
Neuroscience
Primary area: applications->neuroscience_cognitive_science
TL;DR: We propose a method to model spatiotemporal dynamics in neural imaging data and their relationship to behavior.

Abstract: High-dimensional imaging of neural activity, such as widefield calcium and functional ultrasound imaging, provide a rich source of information for understanding the relationship between brain activity and behavior. Accurately modeling neural dynamics in these modalities is crucial for understanding this relationship but is hindered by the high-dimensionality, complex spatiotemporal dependencies, and prevalent behaviorally irrelevant dynamics in these modalities. Existing dynamical models often employ preprocessing steps to obtain low-dimensional representations from neural image modalities. However, this process can discard behaviorally relevant information and miss spatiotemporal structure. We propose SBIND, a novel data-driven deep learning framework to model spatiotemporal dependencies in neural images and disentangle their behaviorally relevant dynamics from other neural dynamics. We validate SBIND on widefield imaging datasets, and show its extension to functional ultrasound imaging, a recent modality whose dynamical modeling has largely remained unexplored. We find that our model effectively identifies both local and long-range spatial dependencies across the brain while also dissociating behaviorally relevant neural dynamics. Doing so, SBIND outperforms existing models in neural-behavioral prediction. Overall, SBIND provides a versatile tool for investigating the neural mechanisms underlying behavior using imaging modalities.

Dynamical Modeling of Behaviorally Relevant Spatiotemporal Patterns in Neural Imaging Data
Mohammad Hosseini
Maryam M. Shanechi 123
Abstract
1. Introduction
Recent advances in neuroimaging techniques, such as wide- field calcium and functional ultrasound imaging, have pro- vided unprecedented access to high-dimensional neural data that can measure the brain at larger spatial scales than tra- ditional eicctrophysiolggical modalities (Macé et al., 2011; Musall et al., 2019; Benisty et al., 2022). These imag- ing techniques are increasingly central to modern neuro- science, offering complementary insights electrophys- iological recordings by enabling the study of large-scale network dynamics, distributed neural representations, and functional connectivity, which is critical for understand- ing cognition and complex behavior (Cardin et al., 2020; Mace et al., 2018). Widefield calcium imaging utilizes flu- orescent indicators to monitor calcium influx in neurons, capturing mesoscale neural activity across large expanse of the cortical surface with relatively high temporal resolu- tion compared to other neural imaging modalities (Musall et al., 2019; Ren & Komiyama, 2021; Nietz et al., 2023). Functional ultrasound, on the other hand, detects changes in cerebral blood volume, which are correlated with both single-ncuron activity and local field potentials, offering wider spatial coverage and a less invasive approach com- pared to electrophysiological implants, with great potential for brain-computer interfaces (BCIs) Nunez-Elizalde et al., 2022; Griggs et al., 2024; Rabut et al., 2024).
High- sigh--dimenional imaging of neural activity, such as widefield calcium and functional ultrasound imaging, provide a rich source of information for understanding the relationship between brain ac- tivity and behavior. Accurately modeling neural dynamics in these modalities is erucial for under- standing this relationship but is hindered by the sigh-iiennsionaltty, complex spatiotemporal de- pendencies, and prevalent behaviorally irreeevant dynamics in these modalities. Existing dynamical models often employ preprocessing steps to ob- tain fow-dimensional representations from neural image modalities. However, this process can dis- card behaviorally relevant information and miss spatiotemporal structure. We propose SBIND, novel data-driven deep learning framework to model spatiotemporal dependencies in neural im- ages and disentangle their behaviorally relevant dynamics from other neural dynamics. We vali- date SBIND on widefield imaging datasets, and show its extension functional ultrasound imag- ing, recent modality whose dynamical modeling has largely remained unexplored. We find that our model effectively identifies both local and long- range spatial dependencies across the brain while also dissociating behaviorally relevant neural dy- namics. Doing so, SBIND outperforms existing models in behrao-behavioral prediction. Overall, SBIND provides a versatile tool for investigating the neural mechanisms underlying behavior using imaging modalities.
Despite the rich spatial and temporal information that these imaging modalities can provide as well as their potential application in BCIs, fully capturing their complex spatiotem- poral dynamics and their link observed behavior has re- mained elusive due several challenges (Dinsdale et al., 2022). Specifically, these modalities are high-dimensional and have spatiotemporal patterns that are complex and in- clude both local and global dependencies. Furthermore, these patterns also contain prevalent behavior-irrelevan components. Hence, developing new methods that address the distinct challenges of neural imaging data to accurately model such datasets is crucial both investigate brain- behavior links and enable their BCIs (Norman et al., 2021; Griggs et al., 2024).
Ellectrical and Computer Engineering, Viterbi School of Engi- neering, University of Southern California (USC). Los Angeles, CA, USA *Computer Science, Viterbi School of Engineering, USC, Los Angeles, CA. USA Biomedical Engineering, Viterbi School of Engineering, USC, Los Angeles, CA, USA. Correspondence to: Maryam M. Shanechi <shanechi @uucc.cuu>.
new modalities in
Proceedings of the 42nd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 20255 by the author(s).
To analyze these high-dimensional neural imaging datasets, existing work often employ an initial dimensionality re-

---

Dynamical Modeling of Behaviorally Relevant Spatiotemporal Patterns in Neural Imaging Data
duction step in the form of preprocessing before behavior decoding or latent state modeling. This step may utilize unsupervised methods such as principal component analysis (PCA) (Musall et al., 2019). or rely on pre-defined regions of interest (ROIs) in the brain (Saxena et al., 2020; Wang et al., 2020). These preprocessed low-dimensional features are then used in modeling, for example, to investigate neural- behavioral relationships in widefield calcium imaging data (Batty et al., 2019; Whiteway et al., 2021; Benisty et al., 2024; Wang et al., 2024) or to decode movement intentions from functional ultrasound data (Griggs et al., 2024). While computationally efficient, these preprocessing approaches may discard spatiotemporal information and inadvertently remove behaviorally relevant dynamics, limiting the ability to fully capture the complex spatiotemporal patterns that link neural activity to behavior.
we devise a two-phase learning approach, where one Con- vRNN first learns the hehaviorally relevant dynamics, and then a subsequent ConvRNN captures other neural dynam- ics. To our knowledge, SBIND is the first neural-behavions dynamical model to learn directly from raw widefield and functional ultrasound imaging data, without relying on pre- processing. Also, our work demonstrates the first dynamical latent modeling of functional ultrasound modality. We show that SBIND achieves superior performance in both behav- ior decoding and neural prediction for widefield imaging and functional ultrasound imaging data compared to other neural-behavion models. Also, SBIND can learn neural dynamics in datasets with various behavior distributions, such as continuous, categorical, and intermittently recorded behavior.
Further complicating this problem, neural recordings often contain a vast amount of information that does not relate to specific behavior of interest (Sani et al., 2021; Stringer et al., 2019; de Vries et al., 2020; Hasnain et al., 2023; Vahidi et al., 2024). This is especially the case for neural imaging modalities because they cover a large spatial scale in the brain, including many regions (Musall et al., 2019; Whiteway et al., 2021). Thus, a challenge in modeling neu- ral activity is to disentangle behaviorally relevant neural dynamics from other ongoing processes in the brain (Sani et al., 2021). Indeed, traditional unsupervised learning meth- ods for modeling neural activity may not optimally extract the behaviorally relevant neural components and may mix them with other neural components. To address this chal- lenge, recent studies have jointly used neural and behavioral data during model training, leading to more accurate infer- ence of behaviorally relevant neural dynamics (Sani et al., 2021; Hurwitz et al., 2021; Schneider et al., 2023; Gondur et al., 2024; Sani et al., 2024; Wang et al., 2024; Oganesian et al., 2024; Vahidi et al., 2025). Some of these works have employed dynamical models, which model the temporal evolution of time-series data in terms of latent state. How- ever, ceural-behavion models have either not focused on neural imaging modalities or used a preprocessing step as noted above.
2. Related Work
When working with widefield imaging data, preprocessing in the form of dimensionality reduction is almost always employed to obtain cow-dimeniional representations from raw widefield images. These methods can be broadly cate- gorized into two families: (1) Unsupervised dimensionality reduction, where methods like PCA or Independent Com- ponent Analysis are used to reduce the data dimensionality before further neural modeling (Musall et al., 2019; Nietz et al., 2023; West et al., 2024; Scaglione et al., 2024). How- ever, these unsupervised techniques can discard the spatial dependencies between brain regions inherent in widefield data. (2) ROI-based methods, which leverage brain atlases to incorporate spatial information during dimensionality re- duction (Mishne al., 2018; Liu et al., 2019; Wang et al., 2020). A popular example in the second family is LocaNMF (Saxena et al., 2020), which is frequently used in modeling widefield data (Batty et al., 2019; Wang et al., 2024). In functional ultrasound imaging (fUSI), while the literature more limited, unsupervised techniques such as PCA have been used before decoding movement intentions from these neural images (Norman et al., 2021; Griggs et al., 2024).
These dimensionality reduction methods are often optimized to extract features that capture maximum variance in neural images, sometimes incorporating auxiliary losses to account for spatial information, such as LocaNME. Current meth- ods then utilize these extracted features for dynamical mod- eling (Batty et al., 2019; Wang al., 2024; Benisty et al., 2024; Karniol-Tambour et al., 2024). However, common drawback of this approach for dynamical modeling is that spatial information may be lost during feature extraction; furthermore, dimensionality reduction is performed without considering the behavior of interest, which can lead to miss- ing crucial behavior-relatee information in neural imaging data. Our approach addresses these limitations by 1) directly modeling the raw neural image data, capturing both local
Contributions To address the above limitations, we pro- pose SBIND (Spatiotemporal modeling of Behavior in Imag- ing Neural Data), a novel deep learning framework for dy- namical modeling of complex local and global spatiotempo- ral patterns in neural imaging data and disentangling their behaviorally relevant components. SBIND utilizes convo- lutional recurrent neural networks (ConvRNNs) to capture local short-range spatiotemporal dependencies and com- bines them with self-attention that is integrated into the dynamics to capture global spatiotemporal dependencies in the original image data. Moreover, to disentangle behav- iorally relevant dynamics in these local and global patterns,
2

---

Dynamical Modeling of Behaviorally Relevant Spatiotemporal Patterns in Neural Imaging Data
and global spatiotemporal dependencies through ConvRNN and self-attention mechanisms, and 2) learning the dynami- cal model jointly with neural images and behavioral data to disentangle behaviorally relevant image dynamics.
neural imaging modalities.
3. Methods
3.1. SBIND Model
Outside the neural imaging domain and primarily for mod- eling electrophysiological neural modalities, various unsu- pervised learning methods have been developed to capture neural dynamics agnostic behavior (Pandarinath et al., 2018; Le & Shlizerman, 2022; Wang et al., 2023; Li et al., 2024; Lu et al., 2025; Abbaspourazad et al., 2024). For example, STNDT (Le & Shlizerman, 2022) models spa- tiotemporal structure in Poisson-diitributed spiking activity using a Transformer architecture. However, these unsuper- vised methods neither incorporate image priors nor aim to dissociate behaviorally relevant dynamics.
Problem Formulation. Figure demonstrates the SBIND architecture. Neural activity (Y₁ E [R=xxX*W) and si- multancously recorded behavior (2ş E 1)* are modeled as observations generated by dynamical system with la- tent states (X) E (R=xX''W'', ). The generative model is
defined as:
X
K+I = fA + WR
K = (X + VE
(I)
Zk = (* + EK
where X represents the latent state at time k, modeled as a spatiotemporal representation in the form of a 3 di- mensional (3D) latent volume characterized by its depth (number of channels, n2),, height (H'), and width (W"). W& E R x",'''' represents the noise affecting the latent state dynamics, and Vı E x x and eg E R are the observation noises for neural images and behavior, respec- tively. Here, H and W are the height and width of the input neural images, ny is the number of input image channels (ny = in all the experiments, but here included for gener- ality of the formulation), and n. is the dimensionality of the behavior vector.
To learn behaviorally relevant neural dynamics, recent stud- ies have developed neural-behavioral models that jointly consider neural activity and behavior during learning (Sani et al., 2021; Hurwitz et al., 2021; Schneider et al., 2023; Sani et al., 2024; Wang et al., 2024; Oganesian et al., 2024). method termed PSID (Sani et al., 2021) dissociates behav- iorally relevant neural dynamics within a linear dynamical system model. A recent method named CEBRA (Schneider et al., 2023) extracts latent embeddings using learning framework that incorporates behavioral supervision through a contrastive loss. Another recent method termed DPAD (Sani et al., 2024) learns a dynamical model in the form of a evo-section RNN that dissociates behaviorally rele- vant dynamies by incorporating an optimization stage fo- cused solely behavior prediction, while having subse- quent stages learn other neural dynamics. Other methods such as BeNeDiff (Wang et al., 2024), TNDM (Hurwitz et al., 2021), and DFINE (A (Abbaspourazad et al., 2024) use a combined loss that incorporates both behavior and neural re- construction to fit their dynamics. There are also multimodal methods such as Ham-G--VAE (Gondur et al., 2024) that fuse neural and behavioral data for behavior reconstruction, unlike the above -ehavi-be methods that only use the neural data for latent and behavior inference. Although effective, these methods either do not explicitly consider an image prior for the observed neural activity or do not directly address neural imaging data. Therefore, they may struggle in learning spatial information when raw image data is passed to the model.
Given this dynamical system, we can infer the latent state from neural observations Yv using an RNN as follows:
X = fA(Xx) + +({)))
& = C(X+)
(2)
2B = D(Xx)
The RNN is parameterized by fA (recurrence) and K (en- coder); it estimates the latent state X given the past neural images (Yi,..., Y&J. all observed information up to time k. The predicted neural image at time k, Y,, is derived from X via the neural decoder C(-). while the decoded behavior 2 is obtained using the behavior decoder D(-).
This latent state encapsulates
Model Parameterization and Design. Our model is con- structed using four key mappings that serve distinct roles in capturing the relationship between neural images and behavior: f.()) describes the latent state recursion. the encoder, (-), maps the observed neural images into this latent representation. The decoders, C(-) and D(-). map the latent state to the corresponding neural and behavior observation spaces, respectively.
Our method also jointly considers the neural-behavioral data during model training and disentangles behaviorally relevant neural dynamics. However, unlike the above neural- behavioral approaches that are not designed for raw neural images, our method integrates spatial priors directly into the model to capture both local and global information from raw neural images with image distributions. We show that this leads to more accurate neural-behavion prediction for
The mappings C. D, and K are all parameterized by con- volutional layers. This choice is motivated by the inherent spatial structure in the neural image data. Convolutional lay- ers are well-suited for hierarchically capturing local spatial
3

---

Dynamical Modeling of Behaviorally Relevant Spatiotemporal Patterns in Neural Imaging Data
(a)
ConyRNN1
(b)
A+G)
Neural Images
Mr wht +
S
wiw Sosneiiii
(1I) k+1
Behavior Decoder (11)
7 k++1
Yk
ConvRNNZ
Self-Attention
Neural Encoders ((t,,K0)) K(2)
x
fa (x²)
* Convolution
S
Add
Concatenation
Neural Decoders cO)) c(22
9 2k41
Patching/Merging
xx2)
3D feature embedding
Figure 1. (a) schematic representation of SBIND for jointly modeling neural image and behavior data. SBIND uses two ConvRNNs learn hehaviorally relevant neural dynamics (ConvRNN1) and behaviorally irrelevant neural dynamics (ComyRNN2) by separating the (b) The recurrence function, f(l()) (-) in ConvRNNI captures both spatial and temporal dependencies by applying convolutional layer that captures local information followed by self-attention patches of the latent state images learn global spatial information. f/ (-) applies similar function to X (2) using different set of parameters. The neural encoders are shallow convolutional networks designed to locally process the input images and downsample them into lower-dimensional latent representations. The behavior decoder predicts the behavior time-series based on the behaviorally relevant latent x while the
latent states into two subsets, x and x (2) respectively.
neural decoder uses both the behaviorally relevant and irrelevant latent states, (!) and X (2) k+1
k+1 to predict the neural image time-serics.
dependencies and features in such data (LeCun et al., 1998). Specifically, K utilizes few nonlinear convolutional layers to downsample the neural images and extract local features. The neural decoder C' uses convolutional layers to upsample and transform the latent representation back to the original image space to predict the neural images one-step into the future. For D, convolutional layers decode behavior from the latent representation, with additional fully connected layers incorporated to decode continuous behavioral data that may not have a spatial structure.
patch Xx. has dimensions n, X P x P and approximately represents features from specific region of the brain (Fig- ure lb). Multi-head self-attention is then applied across these patches, allowing the model to learn spatial relation- ships between different regions (See details in Appendix 1.1.2). Thus, as part of fA. the self-attention mechanism calculates spatial dependencies within each time step on the latent images, while the temporal dependencies are handled by the recurrent application of the entire fa function (Equa- tion 2). This combination of convolutional and self-attentin layers constitutes the recursion function, (Xx), which is summed with the encoded neural image from the current step, K K(Y),, ), to obtain X This enables the model to learn temporal dynamics as well as both local and global spatial patterns in neural image time-series data.
The recursion function, f-). is formulated as fa(-) GlobalAttn (A)))), where GlobalAtro represents the self- attention mechanism, * indicates the convolution operator, and A is set of convolutional kernels. f-(-) designed to capture spatiotemporal dependencies in the latent state dynamics not only locally but also globally. To do so, it utilizes a convolutional layer, A, to aggregate local features in the latent states. To additionally capture global context, a self-attention mechanism is incorporated (Vaswani et al., 2017). Within each time step, we divide the latent state im- age. Xg. into patches, {xe.1, X un Xk.M J. where each
3.2. Neural and Behavioral Loss Functions and Distributions
We use one-step-ahead behavior decoding and neural image prediction losses to fit the parameters of the model. For neu- ral prediction, we use combination of LI, L2, and gradient

---

Dynamical Modeling of Behaviorally Relevant Spatiotemporal Patterns in Neural Imaging Data
difference loss (GDL) (Mathieu al., 2016). The GDL loss encourages the preservation of local image structure and im- proves the accuracy and structural fidelity of the predicted neural images (See formulation in Appendix A.1.3).
ConvRNN, x(", as fixed values. In the second phase of learning, fA (.), (-), and (-) optimized to minimize the error in predicting the neural image dynamics from both x and x Doing so allows x to capture
SBIND is designed to handle various behavioral data types, including continuous, categorical, and intermittently recorded behavior. This is achieved by modifying the train- ing process and behavior loss. For Gaussian and categorical distributions, mean squared error (MSE) and cross-entropy losses are used, respectively, across time points. Moreover, in cases of intermittently recorded behavior, where behav- ior observations are sparse or missing, during training, our model utilizes a masked behavior loss to account for the missing observations while learning the latent representa- tions from neural images as usual. Note, during inference, only neural images are used for inference of latents and decoding of behavior. Further details regarding the specific loss functions employed for each scenario are provided in Appendix A.1.3.
neural image patterns not already captured by X This process cleanly separates the behaviorally relevant latents X (1) from other latents X {
The full inference model and learning details are formulated in Appendix A.1.1 and Appendix A.1.4, respectively.
3.4. Metrics and Evaluation
After training the model, we compute the one-step-ahead predictions of neural images and behavior on the test set using Equation 2 based only on neural image data. We use 5- fold vross-valiittion for widefield calcium imaging datasets and 10-fold vross-validtiion for each fUSI session. We report several metrics depending on the nature of the task. For neural prediction, compute the MSE and coefficient of determination (R²) between the predicted and observed neural images for one-step-ahead prediction. For behavior decoding, we use different metrics based on the type of behavioral data. We compute the MSE for one-step-aae one-step-ahead ahead decoding of continuous behavioral data. For categorical data, we report accuracy and Area Under the Curve (AUC), and for imbalanced datasets, we use the Fl-score.
3.3. Dynamical Model Architecture and Learning
To effectively disentangle behaviorally relevant neural dy- namics from other neural dynamics, we construct model architecture consisting of two ConvRNNs, each integrated with self-attention mechanisms. To learn the model, we dedicate the latents of ConvRNNI capturing the behav- iorally relevant dynamics optimize behavior decoding and the latents of ConvRNN2 to finding the other neural dynamics to optimize neural prediction. We learn these two parts of our model sequentially for simpler interpreta- tion and separation of the latents (Sani et al., 2024), though is straightforward to learn them simultancously with a combined ceural-behavioral loss.
(a)
(b)
Behaviorally Relevant Dynamics. The first ConvRNN in- tegrated with self-attention mechanisms focuses finding the behaviorally relevant latents, decoding behavior, and capturing the corresponding neural dynamics. This Con- vRNN is parameterized by (.), (-),,,, C+)(), and DO).) In the first phase of learning, f(a() (), (-),)), and D()() are optimized to minimize the error in predicting behavior from the neural activity. This allows the model to learn a latent state representation, denoted as ( that captures the behaviorally relevant neural dynamics. Also, CC)(() is optimized to reconstruct the neural images from the learned behaviorally relevant states, X (1)
(c)
(d)
nose
M
left handle
eye
right handle
paw
spout
face
M wmw
right spout
0.0
1.7
3.3
5.0
0.0 1.7 3.3 5.0 Time (s)
Time
Figure 2. (a) Behavior videos recorded from a head-fixee mouse reporting the spatial position of visual or auditory stimuli. (b) Ex- ample widefield neural image. (c) Extracted continuous behavior from ROIs in behavior videos for dataset WFCI 1. (d) Behavior as binary traces for dataset WFCI
Other Neural Dynamics. The second ConvRNN integrated with self-attention mechanisms focuses on learning the re- maining neural dynamics that are not captured by the first ConvRNN. This ConvRNN is parameterized by f (22 (-), (-), and (-).. It takes as input the neural images, as well as the behaviorally relevant states from the first
5

---

Dynamical Modeling of Behaviorally Relevant Spatiotemporal Patterns in Neural Imaging Data
the target and as missing during other periods of the trial. This provides an intermittently recorded behavior type for validation. Each session has 154.77 + 93.75 successful tri- als, and each trial has length of 15 seconds, corresponding to 30 image frames (see Appendix A.4.2 for more details).
(a)
(b)
4.1.2. IMPLEMENTATION DETAILS
5.0 0.4
5.0
1.5
We used three convolutional layers for both the encoder, K, and the decoder, C, to capture local spatiotemporal features and downsample the feature maps in the image dimension to 32x32. The transition function, A, is parameterized by single convolutional layer, and f(-) is parameterized by multi-head self-attention (Vaswani et al., 2017) with 8 heads and an embedding dimension of 256, applied to patches of size n x x 4 (ConvRNN1) and n2 x x 4 (ConvRNN2). For behavior decoding, D uses a few convolutional layers followed by fully connected layer. A single inference step of the SBIND model takes 17.9 ms average on an NVIDIA RTX 6000 Ada Generation GPU, which is shorter than the effective sampling rate of up to 10 Hz used in fUSI (Rabut et al., 2024; Macé et al., 2011) 1) and sampling rate of 30 Hz used in WFCI, suggesting potential feasibility for real-time applications such as BCIs. See Appendix A.1.6 and Table A.1 for details training information, hyperparameter tuning, and the choice of hyperparmeters for each of the datasets.
fixation cue memory
target
reward hold
Figure 3. (a) samples frames fUSI dataset. (b) Memory- guided saccade task timeline. The monkey fixated, viewed brief periph- cral cue (2 or locations), maintained fixation during a memory period, and then made a saccade to the remembered target location.
4. Experiments
4.1. Experimental Details
4.1.1. DATASETS
We evaluate our model on three different neural- -ehavioral datasets: two publicly-vvilable widefield calcium imaging (WFCI) datasets (Churchland et al., 2019) (Figure 2) and one publicly-availabe fUST dataset (Griggs et al., 2023) (Figure 3).
WFCI 1: In this dataset, neural activity across the mouse dorsal cortex was optically recorded (Figure 2b), prepro- cessed, and downsampled to 1x128x128 pixel images. This dataset consists of 248 trials (49008 neural image frames) in which mice indicated the perceived spatial location (left or right) of an auditory or visual stimulus by licking the corresponding spout. Concurrently, behavior videos were recorded using two cameras (Figure 2a). We extracted 14 dimensions of continuous behavior from ROIs in the videos (Figure 2c). This continuous behavioral data was used for behavior decoding (see Section A.4.1 1 for details).
4.2. All SBIND Model Components Contribute to Accurate Neural Image Modeling
Here, we perform ablation studies by systematically remov- ing or modifying specific parts of the model and comparing the resulting performances using relevant behavior decoding and neural prediction metrics (Table 1). We find that each component in our SBIND contributes to accurate neural- behavioral prediction as follows.
First, we assess the importance of using convolutional layers compared to Multi-Layer Perceptrons (MLPs) by construct- ing MLP-SBIND. MLP-SBIND parameterizes all the map- pings with MLPs, with an option to also incorporate com- monly used preprocessing methods for WFCI data. We show that even using this preprocessing whether LocaNMF or PCA as is done in current neural image models, this ap- proach underperforms SBIND in both neural and behavior predictions due to lack of inductive bias for image data, un- like SBIND, which has convolutional layers as a component (Table 1).
WFCI 2:
This dataset comprises 412 trials (38927 neu- ral image frames) with the same trial structure and neural recordings as WFCI 1. However, instead of behavior videos, four binary sensor traces were recorded from handles and spouts, providing categorical behavioral data for decoding (Figure 2d).
This dataset consists of 13 sessions of functional ul- trasound imaging recordings from a non-human primate performing memory-guided saccade or reach movements to one of 2 or 8 peripheral targets (Figure 3b). Functional ultrasound images were recorded at 2 Hz (Figure 3a). Be- havior for this dataset consists of the directions the monkey saccaded to in successful trials. Thus, we use categorical distribution for the behavior. Also, in this case, since there is one target per trial, we take behavior as available only during the period when the monkey was actually fixating on
FUSI:
Next, we assess the importance of our disentangled model architecture and neural-behavioral losses during learning. We construct SBIND-Unsup to train only the second phase of our model, i.c., learning neural dynamics in unsuper- vised manner, which leads to just single set of latents. We find that this ablated model exhibits inferior behavior de-
6

---

Dynamical Modeling of Behaviorally Relevant Spatiotemporal Patterns in Neural Imaging Data
Table One-step-ahead hehavior decoding and neural prediction performances (Mean + SEM) for various ablations of SBIND across 5 folds for widefield (WFCI) datasets in terms of meean-squree error (MSE) and/or Fl -ccoe as appropriate. As indicated by the arrows. lower better MSE and higher better for Fl-score. Tables A.2 for additional comparisons.
WFCI 1
WFCI 2
MODEL
PREPROCESSING
ВEН. MSE
NEUR.
MSE
BEH. FI-SCORE 1
NEUR. MSE 1
MLP-SBIND
FLATTEN
0.6383 + 0.0281
0.1239 0.0040
0.3066 + 0.0108
0.1889 + 0.0052
MLP-SBIND
LOCANMF
0.5980 + 0.0253
0.0539 0.0016
0.3804 + 0.0072
0.2467 + 0.0106
MLP-SBIND
PCA
0.6067 = 0.0245
0.0589 0.0029
0.2998 = 0.0289
0.2834 + 0.0241
SBIND-UNSUP
0.5413 + 0.0185
0.0403
0.0020
0.3985 + 0.0194
0.1497 1 0.0116
SBIND NOATT
0.5392 + 0.0203
0.0552 0.0013
0.4039 + 0.0191
0.1948 + 0.0047
SBIND w/o FA
0.7866 0.0441
0.0812 0.0033
0.36-45 = 0.0197
0.2150 + 0.0032
SBIND
0.4955 + 0.0254
0.0414 0.0029
0.4569 0.0036
0.1644 + 0.0090
tions, we employed Integrated Gradient implementation in Captum framework (Sundararajan et al., 2017; Kokhlikyan et al., 2020) to assess the contribution of brain-wide activity to the prediction of a specific target location in the brain. We averaged the attribution all brain regions in predicting the target location across all frames of the test set WFCI data. The resulting attribution maps (Figure demon- strate that SBIND, unlike SBIND NoAtt, leverages more global information for neural prediction. This confirms that the celf-attention mechanism enables the model to capture dependencies beyond the local receptive fields of the convo- lutional layers, highlighting the importance of self-attention in understanding whole-brain dynamies.
(a)
(b)
-0.9
Actrr Meeee SeMMe o160
2.0
4.0
Figure 4. Mcan contribution of all brain regions to predicting the activity of the pixel marked by x in the brain map, using (a) SBIND NoAu vs. (b) SBIND. See Figure A.4 for more examples.
4.3. Comparison with Existing Houral-Behavioral Models
coding compared SBIND with the same number of latent states (Table 1), highlighting the importance of our method for disentangling behaviorally relevant dynamics in neural image data.
We also find that SBIND outperforms recent neural- behavioral models on the same two WFCI datasets as the previous section (Table 2). First, we compared SBIND with DPAD (Sani al., 2024), which has nonlinearity options in its RNN and its encoder-decoder architecture. Despite this flexibility, DPAD performs worse in behavior decoding and neural prediction for neural image modalities, whether taking flattened widefield images as input or using prepro- cessing on the images in the form of LocaNMF or PCA (Table 2). This highlights the benefits of incorporating spa- tial priors for neural images in the model architecture, as in SBIND. Using common preprocessing techniques improves DPAD's s neural prediction but does not change its behavior decoding performance, both of which still remain inferior to SBIND.
Next, we investigate the recurrence unit by constructing SBIND w/o fA that removes the recurrence (see Equation A.13). SBIND w/o fA fails to capture long-term temporal predic-
dependencies. leading to inferior ceural-behavinn tions than SBIND (Table 1). Finally, to show the importance of the self-attention mechanism, we build SBIND NoAr that removes the self-attention layer in the recurrence mapping, simplifying the recurrence function to local convolutional layer. SBIND NoAtt has inferior performance in both be- havioral and neural predictions (Table 1), showing the im- portance of self-attention for capturing global dependencies. To further explore this result, we increased the patch size used in the self-attention mechanism and observed improved neural prediction performance (Figure A.6). This suggests that larger patches allow the model to focus on more global information, complementing the local processing of the con- volutional layers.
Next, we find that SBIND outperforms CEBRA (Schneider et al., 2023) in neural-behavioral prediction, regardless of whether CEBRA uses common preprocessing techniques, such as LocaNMF or PCA, or not (Table 2). CEBRA's low neural prediction performance (Table 2) can be attributed to the fact that it does not learn any residual dynamics for
To pinpoint the brain regions that drive the neural predic-
7

---

Dynamical Modeling of Behaviorally Relevant Spatiotemporal Patterns in Neural Imaging Data
Table 2. Behavior decoding MSE or Fi-score and neural prediction MSE (Mean + SEM) across folds for each dataset. As indicated by the arrows, lower is better for MSE and higher is better for Fl-score. More detailed comparisons are provided in Table A.3 and Table A.4.
WFCI 1
WFCI
MODEL
PREPROCESSING
ВEН. MSE 4
NEUR. MSE 4
BEH. F1-SCORE 1
NEUR. MSE
DPAD
LOCANMF
0.5877 = 0.0226
0.0543 = 0.0009
0.3613 = 0.0145
0.2483 + 0.0092
DPAD
PCA
0,6164 0.0254
0.0628 = 0.0030
0.2688 = 0.0299
0,2550 + 0.0063
DPAD
FLATTEN
0.6179 0.0270
0.1302 = 0.0013
0.3177 = 0.0091
0.2211 + 0.0071
CEBRA
LOCANMF
0.6250 0.0194
0.4976 = 0.0241
0.3113 = 0.0183
0.4363 + 0.0168
CEBRA
PCA
0.6312 0. 0.0239
0.6856 = 0.0222
0.3005 = 0.0235
0.5323 + 0.0245
CEBRA
FLATTEN
0.5995 + 0. 0.0228
0.7032 + 0.0201
0.2909 + 0.0116
0.2795 + 0.0109
SBIND
0.4955 + 0.0254
0.0414 = 0.0029
0.4569 = 0.0036
0.1644 + 0.0090
neural prediction and relies on the embeddings guided by behavior. Figures A.2 illustrates that CEBRA primarily cap- tures neural activity related to the observed behavior, with limited ability to predict activity in other brain regions. This highlights that neural activity encodes more information than just the studied behavior (Musall et al., 2019), and it is important to model residual neural dynamics to gain a more complete understanding of brain function.
we used a multi-decoder approach in the decoder mapping, ( to predict the vertical and horizontal directions (left- right-stationarr) (Appendix A.4.2). During evaluation, we used the latent state at the last time-step in the trial to predict the direction of movement. We first performed ablation studies on all sessions. The ablated models underperformed SBIND in ceural-behavions prediction, again showing the importance of each component in our model (Table A.7 and Table A.8).
Finally, in two additional baseline comparisons, SBIND again demonstrated its superiority for joint neural image and behavioral modeling by outperforming adapted versions of STNDT (Le& & Shlizerman, 2022) and TNDM (Hurwitz et al., 2021). Originally developed for spiking clectrophys- iology data, we adapted these methods for our widefield imaging data as follows: we used LocaNMF features as input, placed a Gaussian prior over these inputs, and trained the models using an MSE loss instead of their original Pois- son likelihood. Even with these adaptations, STNDT, which employs two separate sets of Transformers to capture spatial and temporal information, still underperforms in both neu- ral and behavioral predictions compared to SBIND (Table A.5). Similarly, SBIND surpassed the adapted TNDM, a sequential variational autoencoder that learns two sets of latent factors for behaviorally relevant and irrelevant dy- namics using a mixed neural-behavioral objective. These results consistently highlight the strength of SBIND's s model design in modeling raw spatiotemporal neural images and disentangling behaviorally relevant neural dynamics.
We then compared SBIND with other deep learning base- lines as well as the original method used in (Griggs et al., 2024) for this fUSI dataset i.e., PCA + linear discriminant analysis (LDA). For DPAD, we made similar decoder mod- ifications to classify both 2-directional and 8-directional movements. For CEBRA, we found that labeling the entire trial with the target location yielded better embeddings for the decoding task than using only the samples during tar- get fixation (See Appendix A.4.2 for details). As shown in Table 3, all models are inferior to SBIND in behavior de- coding and neural prediction, again showing the importance of capturing image spatiotemporal structures through our CNnvRNN and self-attention mechanisms.
5. Discussion
We develop SBIND, a novel approach for modeling dynam- ies in neural image data and their relationship to behavior, and demonstrate its success for two diverse neural imaging modalities: optical widefield and focused ultrasound imag- ing. We address the challenges of high-dimensionality and complex spatiotemporal patterns in these image modalities by designing a convolutional recurrent architecture com- bined with a self- eelf-attention mechanism. This enables us to learn both local and long-range spatiotemporal dynamics directly from raw neural image data without the need for preprocessing. Additionally, we tackle the challenge of prevalent behaviorally irrelevant dynamics in these images by disentangling the behavior- dynamics while
4.4. Functional Ultrasound Imaging Data Results
For the fUSI dataset, behavior consisted of the target the monkey reached or saccaded to for each trial; thus, behavior was only taken as available for the time-steps in the trial during which the monkey was fixated on the target. This gave us a categorical and intermittently recorded behavior time-series for modeling. In the 2-directional tasks, we used binary target classification with binary cross-entropy loss. In 8-directional tasks, similar (Griggs et al., 2024),
8

---

Dynamical Modeling of Behaviorally Relevant Spatiotemporal Patterns in Neural Imaging Data
Table Behavior decoding accuracy (quantified proportion of trials whose target was correctly decoded) and neural prediction MSF (Mean + SEM) across 10 folds and all sessions in the IUSI dataset. As indicated by the arrows. lower is better for MSE and higher is better for accuracy. See Table A.7 and Table A.8 for comparison with ablated variants of SBIND. See Table A.9 for more comparisons.
2-DIRECTIONAL SESSIONS
8-DIRECTIONAL SESSIONS
MODEL
PREPROCESSING
ВEH. ACCURACY 1
NEUR. MSE 4
BEH. ACCURACY 1
NEUR. MSE 4
LDA
PCA
0.7001 1 0.0177
0.2806 + 0.0204
DPAD
FLATTEN
0.5631 + 0.0191
0.8409 + 0.0072
0.1840 + 0.0131
0.9236 + 0.0083
DPAD
PCA
0,6783 + 0.0167
0.6391 = 0.0056
0.2864 + 0.0183
0.7109 + 0.0034
CEBRA
FLAITEN
0.6813 1 0.0193
1.6647 + 0.0128
0.2705 + 0.0165
1.6966 1 0.0144
CEBRA
PCA
0.7276 + 0.0203
1.5832 + 0.0088
0.2633 + 0.0163
1.6984 + 0.0190
SBIND
0.7300 + 0.0191
0.4725 + 0.0165
0.3521 + 0.0201
0.3919 + 0.0107
simultaneously capturing other ongoing neural processes. In contrast to other ceural-behavioral models that rely on preprocessing, SBIND assumes an image prior the in- put, enabling the learning of spatiotemporal information for modeling neural images and predicting behavior. Our model outperforms existing neural-behavioral models in predicting behavior, regardless of the prior distribution of the behav- ior of interest. Furthermore, to our knowledge, our work presents the first dynamical latent state model for fUSI data.
SBIND offers significant practical advantages for BCIs (Shanechi, 2019; Shenoy & Carmena, 2014; Oganesian & Shanechi, 2024). SBIND's ' inference time is faster than typ- ical sampling rates of WFCI and FUSI (Rabut et al., 2024; Macé et al., 2011) This property, coupled with SBIND's re- cursive inference, can enable real-time and computationally efficient modeling of neural images and behavior decoding. Also, SBIND's recurrent architecture inherently supports neural forecasting of behavior several steps into the future, without need for model retraining. These capabilities, com- bined with SBIND's 's successful demonstration on diverse imaging modalities, could help enable future non-invasive BCIs using fUSI, which is a recent promising modality whose dynamical modeling was previously unexplored.
In this study, we focused on neural imaging modalities with relatively high temporal resolution, namely widefield calcium imaging and functional ultrasound imaging that also has potential for BCIs. This is in contrast to some imaging modalities such as functional magnetic resonance imaging (IMRI) that have lower temporal resolution and are largely not usable for portable BCIs. Indeed, for this reason, prior deep learning approaches for fMRI are different in goals compared to our work. These fMRI models largely focus on classifying task conditions or participant demographics (e.g., age, sex) (Gadgil et al., 2020; Kan et al., 2022; Malkiel et al., 2022; Li et al., 2023) rather than modeling behaviorally relevant temporal dynamics and disentangling them from other neural temporal dynamics, which is our goal here.
Impact Statement
This work can enable the decoding of non-invasive fUSI neural modalities, which may have positive societal impact by facilitating the design of non-invasive BCIs that restore lost movement in paralyzed patients. Other than this, this paper also presents work whose goal is to advance the field of Machine Learning. In this regard, there are many poten- tial societal consequences of work, none which we feel must be specifically highlighted here.
Our work focused on learning the model using data from single session or animal. An important direction for future work is to extend SBIND to enable multi-session learning, for example by adding session specific read-in and read- out layers while keeping the model core uniform across sessions. Doing so may both improve performance and allow for investigating how whole-brain local and global spatiotemporal dynamics change across sessions and an- imals using SBIND. Indeed, for training their PCA-LDA decoder on fUSI data, Griggs et al. (2024) showed that using multi-session data for pretraining the decoder can be helpful since day-to-day recordings have similarities their neural representations. Finally, tracking time-varying dynamics through adaptive learning be another interesting future direction (Ahmadipour et al., 2021; Yang et al., 2021).
Acknowledgment
This work was supported, in part, by National Institutes of Health (NIH) grants RO1MH123770, R61MH135407, and RF1DA056402, NIH Director's New Innovator Award DP2- MH126378 and National Science Foundation (NSF) CR- CNS program award IIS- 21 13271. We thank Simon Musall and Anne Churchland for making the widefield imaging datasets publicly available and providing guidance on their usage. We also thank the Andersen and Shapiro Labs for making the fUSI dataset publicly available. Finally, we thank Parsa Vahidi for his help with the manuscript and bascline comparisons.
9

---

Dynamical Modeling of Behaviorally Relevant Spatiotemporal Patterns in Neural Imaging Data
Reproducibility Statement
Dinsdale, N. K., Bluemke, E., Sundaresan, V., Jenkinson, M., Smith, S. M., and Namburete, A. I. L.. Challenges for machine learning in clinical translation of big data imaging studies. Neuron, 110(23)3866--33 3881, 2022.
To ensure the reproducibility our work, we are shar- ing the code for SBIND at https: ptttps://qitinn //ithub..bb com/ SharechiLab/see /BIDDD We also provide model hyper- parameters and training details in Appendix A.1.6. The Widefield Calcium (Churchland et al., 2019) and functional Ultrasound Imaging (Griggs et al., 2023) datasets used in our main experiments are publicly available; details of the preprocessing applied to these datasets are provided in Ap- pendix A.4, and our preprocessing framework for these datasets also shared in our repository for those interested in reproducing the main experiments presented in this work.
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations 2021.
Gadgil, S., Zhao, Q., Pfefferbaum, A., Sullivan, E., Adeli, E., and Pohl, K. Spatio-temporal graph convolution for resting-stat statte fmni analysis. In Medical Image Computing and Computer-Assister Intervention, volume 12267, PP. 528-538. Springer, 2020.
References
Abbaspourazad, H., Erturk, E., Pesaran, B., and Shanechi, M. M. Dynamical flexible inference of nonlinear latent factors and structures in neural population activity. Nature Biomedical Engineering, 8:85-108, 2024. doi: https: //doi.org/10 wtt.g///...00/////1 10010000411511000--1100000
Gondur, R., Schaffer, E. S., Keeley, S., Bin Sikandar, U., and Aoi, M. Multi-modal gaussian process variational au- toencoders for neural and behavioral data. In Proceedings of the International Conference on Learning Representa- tions (ICLR). ICLR, 2024.
Ahmadipour, P., Yang, Y., Chang, E. F., and Shanechi, M. M. Adaptive tracking of human ECoG network dynamics. Journal of Neural Engineering, 18(1):016011, 2021.
Griggs, W., Norman, S., Deffieux, T., Segura, F., Osmanski, B.-F. Chau, G., Christopoulos, V., Liu, C., Tanter, M., Shapiro, M. G., Andersen, R. A., and of Technology, C. I. Dataset for: Decoding motor plans using a closed-loop ul- trasonic brain-machin interface (1.0.0), 2023. [Dataset].
Batty, E., Whiteway, M., Saxena, S., Biderman, D., Abe, T., Musall, S., Gillis, W., Markowitz, J., Churchland, A., Cunningham, J. P., et al. Behavenet: nonlinear embedding and bayesian neural decoding of behavioral videos. In Advances in Neural Information Processing Systems, pp. 15680-15691, 2019.
Griggs, W. S., Norman, S. I.., Deffieux, T., Rogers, N. J., Sheth, S. A., Oxley, T. J., Martinet, L. E., Liu, Y., Pesaran, B., and Connolly, A. T. Decoding motor plans using a closed-loop ultrasonic brain-machine interface. Nature Neuroscience, 27:196-207, 2024.
Benisty, H., Song, A., Mishne, G., and Charles, A. S. Re- view of data processing of functional optical microscopy for neuroscience. Neurophotonics, 9(4):041402-, August 2022.
Hasnain, M. A., Birnbau,, J. E.,, Nunez, J. L. U., Hartman, E. K., Chandrasekaran, C., and Economo, M. N. Separat- ing cognitive and motor processes in the behaving mouse. bioRxiv, 2023. doi: 10. 10.1101/2023.8..3..55
Benisty, H., Barson, D., Moberly, A. H., Matsumoto, H., Wu, L.. Tseng, H.-Y., Kaeser, P. S., and Svoboda, K. Rapid fluctuations in functional connectivity of cortical networks encode spontaneous behavior. Nature Neuro- science, 27:148-158 2024.
Hurwitz, C., Srivastava, A., Xu, K., Jude, J., Perich, M., Miller, L., and Hennig, M. Targeted neural dynamical modeling. Advances Neural Information Processing Systems,
Cardin, J. A., Crair, M. C., and Higley, M. J. Mesoscopic imaging: Shining wide light on large-scale neural dy- namics. Neuron, 108(1) (1:(1)333444 43, 2020.
2021.
Kan, X., Dai, W., Cui, H., Zhang, Z., Guo, Y., and Yang. C. Brain network transformer. In Advances in Neural Information Processing Systems, 2022.
Churchland, A. K., Musall, S., Kaufman, M. T., Juavinett, A. L.. and Gluf, S. Dataset from: Single-trial neural dynamics are dominated by richly varied movements, October 2019. [Dataset].
Tamiour, O.,, Zoltowski, D. M., Diamanti, E. M., Pinto, I.., Brody, C. D., Tank, D. W., and Pillow, J. W. Modeling state-dependent communication between brain regions with switching nonlinear dynamical systems. In The Twelfth International Conference on Learning Repre- sentations, 2024.
de Vries, S., Lecoq. J., Buice, M., et al. A large-scale stan- dardized physiological survey reveals functional organi- zation of the mouse visual cortex. Nature Neuroscience, 2020.
10

---

Oganesian, L. L., Sani, O. G., and Shanechi, M. M. Spectral learning of shared dynamics between generalized-linea processes. In Advances in Neural Information Process- Curran Asso- Dynamical Modeling of Behaviorally Relevant Spatiotemporal Patterns in Neural Imaging Data
Kokhlikyan, N., Miglani, V., Martin, M., Wang, E., Al- sallakh, B., Reynolds, J., Melnikov, A., Kliushkina, N., Araya, C., Yan, S., and Reblitz-Richarddse tum: A unified and generic model interpretability library for pytorch, 2020. URL http:ttss://// /ttpas///aavvvv. orw.v.orrr//ssss 2009.00 07896.
with Deep Learning, volume 172 of Proceedings of Ma- chine Learning Research, pp. 895-913, PMLR, 06-08 Jul 2022.
O Cap-
Mathieu, M., Couprie, C., and LeCun, Y. Deep multi-scale video prediction beyond mean square error. In Interna- tional Conference Learning Representations, 2016.
Le, T. and Shlizerman, E. STNDT: Modeling neural popu- lation activity with spatiotemporal transformers. In Oh, A. H., Agarwal, A., Belgrave, D., and Cho, K. (eds.), Ad- vances Neural Information Processing Systems, 2022.
Mishne, G., Coifman, R., Lavzin, M., and Schiller, Auto- mated cellular structure extraction in biological images with applications to calcium imaging data. bioRxiv, 05 2018.
LeCun, Y., Bottou, L., Bengio, Y., and Haffner, P. Gradient- based learning applied to document recognition. Proceed- ings of the IEEE, 86( 1):22)8-22333 23244, 1998.
Musall, S., Kaufman, M. T., Juavinett, A. L., Gluf, S., and Churchland, A. K. Single-trial neural dynamics are domi- nated by richly varied movements. Nature Neuroscience, 22:1677-1686, 2019.
Li, H., Zhu, J., Chen, J., Liu, G., and Calhoun, V. D. Swift: Swin transformer for fmrr analysis. In Advances in Neural Information Processing Systems, 2023.
Nietz, A. K., Streng, M. I.., Popa, L., S., Carter, R. E., Flaherty, E. B., Aronson, J. D., and Ebner, J. To be and not to be: wide-field ca2+ imaging reveals neocortical functional segmentation combines stability and flexibility. Cerebral Cortex, 33(11 (3(11):6543-665 6558,, 02 2023. ISSN 1047-3211.
Li, W., Li, C., Wang. Y., and Wu, A. Multi-region marko- vian gaussian process: An efficient method to discover directional communications across multiple brain regions. In Forty-first International Conference on Machine Learn- ing, 2024.
Norman, S. L., Maresca, D., Christopoulos, V. N., Griggs, W. S., Demene, C., Tanter, M., Shapiro, M. G., and Ander- sen, R. A. Single-trial decoding of movement intentions using funetional ultrasound neuroimaging. Neuron, 109 (9):1554-1566.e4, 2021. ISSN 0896-6273.
Lin, T.-Y., Goyal, P., Girshick, R., He, K., and Dollar, P. Focal loss for dense object detection. In Proceedings of the IEEE International Conference on Computer Vision, Oct 2017.
Liu, J., Whiteway, M. R., Sheikhattar, A., Butts, D. A., Babadi, B., and Kanold, P. O. Parallel processing of sound dynamics across mouse auditory cortex via spa- tially patterned thalamic inputs and distinet areal intra- cortical circuits. Cell Reports, 27(3):872-885.44 2019. ISSN 2211-.
Nunez-Elizalde, A. O., Krumin, M., Reddy, C. B., Montaldo, G., Urban, A., Harris, K. D., and Carandini, M. Neural correlates of blood flow measured by ultrasound. Neuron, 110(10):6331116400 2022.
Oganesian, L. L. and Shanechi, M. M. Brain-computer interfaces for neuropsychiatric disorders. Nature Reviews Bioengineering, 2:653-670, 2024.
Lu, Z., Zhang, W., Le, T., Wang, H., Sümbül, U., SheaBrown, E. T., and Mi, L. Netformer: An inter- pretable model for recovering dynamical connectivity in neuronal population dynamics. In The Thirteenth Inter- national Conference on Learning Representations, 2025.
ing Systems, volume 37, pp. 19150-89189 ciates, Inc., 2024.
Macé, E., Montaldo, G., Cohen, I., Baulac, M., Fink, M., and Tanter, M. Functional ultrasound imaging of the brain. Nature Methods, 8(8):662-664, 2011.
Pandarinath, C., O'Shea, D. J., Collins, J., Jozefowicz, R., Stavisky, S. D., Kao, J. C., Trautmann, E. M., Kaufman, M. T., Ryu, S. I., Hochberg, L. R., et al. Inferring single- trial neural population dynamics using sequential auto- encoders. Nature methods, 15(10))805-81, 2018.
Mace, E., Montaldo, G.. Trenholm, S., Cowan, C., Brignall, A., Urban, A., and Roska, B. Whole-brai functional ultrasound imaging reveals brain modules for visuomotor 2018
integration. Neuron, 100(5):
Rabut, C., Norman, S. L., Griggs, W. S., Russin, J. J., Jann, K., Christopoulos, V., Liu, C., Andersen, R. A., and Shapiro, M. G. Functional ultrasound imaging of human brain activity through an acoustically transparent cranial window. Science Translational Medicine, 16(749); eadj3143 2024.
Malkiel, I., Rosenman, G., Wolf, L., and Hendler, T. Self-supervised transformers for fimi representation. In Konukoglu, E., Menze, B., Venkataraman, A., Baumgart- ner, C., Dou, Q., and Albarqouni, S. (eds.), Proceedings The 5th International Conference on Medical Imaging

---

Dynamical Modeling of Behaviorally Relevant Spatiotemporal Patterns in Neural Imaging Data
Ren, C. and Komiyama, T. Wide-field calcium imaging of cortex-wide activity in awake, head-fixed mice. STAR Protocols, 2(4):100973, 2021.
Vahidi, P., Sani, O. G., and Shanechi, M. BRAID: Input-driven nonlinear dynamical modeling of neural- behavioral data. In The Thirteenth International Confer- ence on Learning Representations, 2025.
Sani, O. G., Abbaspourazad, H., Wong, Y. T., Pesaran, B., and Shanechi, M. M. Modeling behaviorally relevant neural dynamics enabled by preferential subspace identi- fication. Nature Neuroscience, (4(1::140-14) 2021.
Valley, M. T., Moore, M. G., Zhuang, J., Mesa, N., Castelli,
D., Sullivan, D., Reimers, M., and Waters, J. Separa- tion of hemodynamin signals from geamp fluorescence measured with wide-field imaging. Journal of Neurophys- iology, (33)))33563366, January 2020.
Sani, O. G., Pesaran, B., and Shanechi, M. M. Dissociative and prioritized modeling of behaviorally relevant neu- ral dynamics using recurrent neural networks. Nature Neuroscience, 27:2033-2045, 2024.
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L.,, and Polosukhin, I. Atten- tion is all you need. In Advances in Neural Information Processing Systems, volume 30, 2017.
Saxena, S., Kinsella, I,, Musall, S., Kim, S. H., Meszaros, J., Thibodeaux, D. N., Kim, C., Cunningham, J., Hillman, E. M. C., Churchland, A., and Paninski, L. Localized semi- nonnegative matrix factorization (locanmf) of widefield calcium imaging data. PLoS Computational Biology, 16 1007710 2020.
Wang, Q., Ding, S.-L., Li, Y., Royall, J., Feng, D., Lesnar, P., Graddis, N., Naeemi, M., Facer, B., Ho, A., Dolbeare, T., Blanchard, B., Dee, N., Wakeman, W., Hirokawa, K. E., Szafer, A., Sunkin, S. M., Oh, S. W., Bernard, A., Phillips, J. W., Hawrylycz, M., Koch, C., Zeng, H., Harris, J. A., and Ng. L. The allen mouse brain common coordinate framework: A 3d reference atlas. Cell, 181(4): 936-953.e20, 2020. ISSN 0092-8674.
Scaglione, A., Resta, F., Goretti, F,, and Pavone, S. Group ica of wide-field calcium imaging data reveals the retros- plenial cortex as a major contributor cortical activity during anesthesia. Frontiers in Cellular Neuroscience, 18, 2024. ISSN 1662-5102.
Wang, Y., Wu, Z., Li, C., and Wu, A. Extraction and re- covery of nattioltemporl structure in latent dynamics alignment with diffusion model. In Thirty-seventh Con- ference on Neural Information Processing Systems, 2023.
Schneider, S., Lee, J. H., and Mathis, M. W. Learnable latent embeddings for joint behavioural and neural analysis. Nature, 617:360-368, 2023.
Wang, Y., Li, C., Li, W., and Wu, A. Exploring behavior- relevant and disentangled neural dynamics with genera- tive diffusion models. In The Thirty-eighth Annual Con- ference on Neural Information Processing Systems, 2024.
Shanechi, M. M. macine interfaces from motor to mood. Nature neuroscience, 22(10):554-1164, 2019.
Shenoy, K. V. and Carmena, J. M. Combining decoder design and neural adaptation in brain-machine interfaces. Neuron, 84(4):665-680, 2014.
West, S., Gerhart, M., and Ebner, T. Wide-field calcium imaging of cortical activation and functional connectivity in externally- and internally-driven locomotion. Nature Communications, 15:7792, 2024.
Stringer, C., Pachitariu, M., Steinmetz, N., Reddy, C. B., Carandini, M., and Harris, K. D. Spontaneous behaviors drive multidimensional, brainwide activity. Science, 364 (6437))aavv7893, 2019.
Whiteway, M. R., Biderman, D., Friedman, Y., Dipoppa, M., Buchanan, F. K., Wu, A., Zhou, J., Bonacchi, N., Miska, N. J., Noel, P., Rodriguez, E., Schartner, M., Socha, K., Urai, A. E.,, Salzman, C. D., Laboratory, I. B., Cunningham, J. P., and Paninski, L.. Partitioning vari- ability in animal behavioral videos using semi-supervised variational autoencoders. PLoS Computational Biology, (70)xx1009433 2021.
Sundararajan, M., Taly, A., and Yan, Q. Axiomatic attribu- tion for deep networks. In International conference on machine learning, pp. 3319-3328. PMLR, 2017.
Tompson, J., Goroshin, R., Jain, A., LeCun, Y., and Bregler, C. Efficient object localization using convolutional net- works. In Proceedings of the 32nd International Confer- ence on Machine Learning (ICML), pp. 656,, 2015.
Xu, B., Wang, N., Chen, T., and Li, M. Empirical evaluation of rectified activations in convolutional network. CoRR, abs/1 1 305.00853, 2015.
Vahidi, P., Sani, O. G., and Shanechi, M. M. Modeling and dissociation of intrinsic and input-driven neural pop- ulation dynamics underlying behavior. Proceedings of the National Academy of Sciences, 121(7):22112871121, 2024.
Yang, Y., Ahmadipour, P., and Shanechi, M. M. Adaptive latent state modeling of brain network dynamics with real-time learning rate optimization. Journal of Neural Engineering, 18(3):036013, 2021.
12

---

Dynamical Modeling of Behaviorally Relevant Spatiotemporal Patterns in Neural Imaging Data
A. Appendix
A.1. Methods Supplementary Details
A.1.1. FORMULATION OF THE FULL INFERENCE MODEL
Our model employs a two-RNN architecture to effectively capture and disentangle behaviorally relevant neural dynamics from other neural dynamies. The first RNN (ConvRNNI) focuses on behaviorally relevant dynamies before the second RNN (ConvRNN2) learns the remaining dynamics.
/ parameterized by f()())))))) (.), (.), ()), and (.) It takes as input the neural images, Ys. and outputs latent state state representation, x RBixH'>W that captures the behaviorally relevant neural dynamics.
ComvNNN2 is parameterized by f (.. (-). and (+). It takes as input the neural images, Yx, as well as the latent states from ConvRNNI, _ and outputs a latent state representation, x (2) E R xW" ,that captures the residual neural dynamics not captured by ComRRNNI.
In Equation 2, both latent states combined for simplicity. The full inference model (Figure la) can be formulated as follows:
( (x)) + (Y,)
X (2) F K (Y,
x kt1
(A.1)
Yx = -Cu(X C (x
2% (X, (1))
where Y XXx is the predicted neural images and 2n R= is the predicted behavior time index, k.
The full set of states can be denoted X₃ x xW' where nu n1 + 122, and is achieved by concatenating the image latent states in the channel dimensions. As seen Equation A. X x()) is calculated independently x(2) As discussed in Section 1.1.4, x( (1) learned to decode behavior, they essentially capture behaviorally relevant neural dynamics. X (2) learn other neural dynamics by optimizing for neural prediction. This is achieved by passing the neural images and states from COnvRNN/ as residuals in the calculation of the second set of states. The above formulation can be written in
combined form as Equation 2, where:
X!O x x
X
fA(Xa) = (x 91)
GlobalAttn ((Ab))))tt(((A())) * *x(")
fA
GlobalAttn (2) (A((). * x(22)
(Y,)
K(Y.) = KP2 x (1))
C(X.) = C (x(")) (() + (22) (2)
D(X2) (x(())
These notations connect the full inference model (Equation A. and the combined form in Equation 2. Also, fa(.) GlobalAttn(A * (.)), where GlobalAttn represents the self-attention mechanism, and A represents the convolutional kernels applied on the states prior to self-attention.
A.1.2. DETAILS OF SELF-ATTENTOO OPERATION
The recurrence function, f( (.), is designed to capture spatiotemporal dependencies in the latent state representation of ComvRNNI both locally and globally (Figure A. For simplicity, we explain details of applying the recurrence function
O ()), on the latent state time index k x(") However, the same function applied at all other time indices recurrently.
13

---

Dynamical Modeling of Behaviorally Relevant Spatiotemporal Patterns in Neural Imaging Data
X ""
c.X. H' x W
Self- Aoff-Attention
1x1
1x1
A
Patching
Merging
conv
(
Figure A.I. Details self-attention mechanism P. Self-attention used to capture global spatial relationships within the latent state. (2 applies the same self-attention mechanism using different set parameters
(1) a single convolutional layer to capture local dependencies latent state. Additionally, recurrence functions incorporate self-attention mechanism capture global context and long-range dependencies the latent state dynamics. Here we explain the details of applying self-attention to x
Reduction in channel dimension. For f( (-), the process begins by passing X through 1x1 convolutional layer to reduce the channel size from nı C, resulting in representation with dimensions H' x W' This reduction in channel size is optional and makes subsequent self-attentio computation more efficient. In experiments, we found that reducing the channel dimension to = 2 provided good balance between computational and parameter efficiency, and model performance.
Patching. Next, the latent state representation is divided into patches, {x x 2 X }. similar to the approach used in (Dosovitskiy et al., 2021). The total number of patches, M, is calculated as M (H' x W')/(P x P), where P is the (1) patch size. Each patch, X has dimensions c P P and can be interpreted as representation of features from roughly specific brain region.
(1)
Self-attention block. The patches then flattened into vectors of shape cP2 and treated tokens for the self-attention mechanism. A one-dimensional learnable embedding is added each of the tokens (patches), so the self-attention layer is informed of the position of the embedding in the latent state images. Multi-head self-attention is applied to these tokens (Vaswani et al., 2017), allowing the model to learn spatial relationships between different patches. layer normalization applied before and after the self-attention.
Projecting back the latent state representation. After applying self-attention to the embedded patches, the patches reshaped and rearranged into H' W' to get to the original spatial dimensions, effectively reversing the patching operation. A 1 x convolutional layer then projects the x H' W latents to nı H' x W dimensional space. The resulting representation is added to the output of A (1), forming the final output of the recurrence function.
This combination of convolutional layers and self-attention layers in the recurrence function enables the model to effectively capture both local and global spatial dependencies. The same recurrence function is applied x((), which has dimensions n2 HI' x W', across all time points using different set of parameters to learn separate set of dynamics for the behaviorally irrelevant component of the neural images.
A. 1.3. LOSS FUNCTIONS
Neural Prediction Loss For neural prediction, we use a combination of L1, L2, and gradient difference loss (GDL) (Mathieu et al., 2016). The GDL loss encourages the preservation of local image structure by penalizing differences between
14

---

Dynamical Modeling of Behaviorally Relevant Spatiotemporal Patterns in Neural Imaging Data
the gradients of the predicted and ground-truth images. This combined loss function aims improve the accuracy and structural fidelity of the predicted neural images. The L1, L2, and GDL functions are defined follows:
CLi(Y,, 1k) Σ
1 - -Y2
(A.2)
LIa(Ye, Yx) Σ
(Drn - -Y
(A.3)
bj
Larad(Yrs Yx)
Σ
|llx - - -111 1122
- + |||
||
||
(A.4)
where i.j index the spatial dimensions of the image. The total loss for neural image reconstruction given by:
Cy = Cr, AL Ya) + AgraaLgrad (&x, Yk);
(A.5)
where ALI and Agrad are hyperparameters that control the relative weights of the LI and GDL losses.
Behavior Decoding Loss For behavior decoding, the loss function, Lz, is chosen based on the distribution and availability of data each time point.
Continuous Behavior: Assuming isotropic Gaussian distribution for this type of behavior, the MSE loss.
Categorical Behavior: We class-weighted cross-entropy loss for categorically distributed behavior address potential class imbalance (Lin et al., 2017). For instance, in the WFCI 2 dataset, where the mouse is not touching the sensors over 90% of the time, we assign weight of 0.9 to class 1 (indicating when the mouse is touching sensor-left handle, right handle, left spout, or right spout) and 0.1 to class 0 (indicating when the mouse is not touching sensor).
We utilize a masking strategy to handle intermittently recorded behavior. The behavior loss is calculated only at the sparse time points where the behavior observed.
Intermittently Recorded Behavior:
A. 1.4. TWO-PHASE LEARNING DETAILS
To disentangle behaviorally relevant neural dynamics from other neural dynamics, we design a model architecture with two ConvRNNs, each incorporating self-attention mechanisms. The parameters of these two ConvRNNs are learned in two sequential phases to achieve the disentanglement.
Phase 1: Learning Behaviorally Relevant Dynamics
First, the parameters ConvRNNI i.e. ()) (+) and (-) - and the behaviorally relevant latent states, x( leamedd minimize the error in predicting behavior from the neural images. The following recurrent formula used predict behavior, 2( one-step into the future:
x (1)
= (x()) (Ys)
Etl (x) (1)
(A.6)
The optimization is formulated as:
min 12 1.0 KO,
Ca 2()),
(A.7)
where Lg the loss function for behavior decoding chosen based on the distribution of behavior (Appendix A.1.3). E [1, 2,.., T] where T is the total number of samples. This optimization ensures that ConvRNNI learns neural dynamics that are relevant to behavior.
15

---

Dynamical Modeling of Behaviorally Relevant Spatiotemporal Patterns in Neural Imaging Data
Once ConvRNNI is learned, its parameters fixed, and the decoder (-) is optimized to predict neural images one step into the future from the learned latent states, X (1)
k This can be formulated as:
= cx)
(A.8)
and the optimization is formulated as:
min
Σ (Y CC)
(A.9)
where Ly the loss function for neural reconstruction, defined Equation A.5.
Phase Learning Residual Neural Dynamics
In this phase, the parameters of ComRRNN2 - i.e. f (.). (.). and (.) are learned to minimize the error in predicting the residual neural images, i..., the part of neural images not predicted by the behaviorally relevant states of ConvRNNI. This is achieved by training ConYRNN2 to predict the difference between the observed neural images and the neural images predicted by ConvRNNI. The residual neural predictions calculated using the following recurrent formulation:
x (2)
(2) + ( (x
(Y,, X
(1) 841
R+1
C
(A.10)
This step learns the residual neural dynamics and the latent states, X (2)
The optimization is formulated
min Evr(Yn - (2)
(A.11)
1 RER)CC k
This concludes learning the ConvRNNs and the total latent states X;. Note that in the two optimization steps in Equations A.9, and A, I, the optimization only controls and learns the parameters in the current optimization, and the parameters from previous optimizations are fixed.
A.1. ALTERNATIVE FORMULATION FOR THE INFERENCE MODEL
We can optionally concatenate the mappings from the encoder, K (YY:), with the states of the current ConvRNN before feeding the latent states into the recurrence funetion. This can be formulated as:
(
f()) (x (Y,)))
S+1
X X
( S (X,
() (1)
K41 K ((((
(A.12)
Y, C( (x) (3)
2
D((X
This can be thought of as including the information from the neural images at the current time index within the mapping. This does not change the dimensions of the latent states, but the input the convolutional layer within the recurrence uses more kernels. This more general form of Equation I, and for the first datasets, this form achieves slightly better performance (See Table A.6).
A. SBIND ARCHITECTURE AND IMPLEMENTATION DETAILS
The Neural Encoders, K (1) and (2), each consist of three convolutional layers that downsample the input neural images to a 32 x 32 spatial resolution. These layers process the images statically and locally. To ensure stable learning, each convolutional layer is followed by batch normalization to normalize activations, and Leaky ReLU (Xu et al., 2015) is used as the activation function. Padding is applied to preserve spatial dimensions during convolutions.
16

---

Dynamical Modeling of Behaviorally Relevant Spatiotemporal Patterns in Neural Imaging Data
The neural decoders, C(1) and C consist of three transposed convolutional layers, which are designed upsample the latent states and project them back into the neural image observation space. The same activation function and normalization are applied to these decoders.
The behavior decoder, D, begins by further downsampling the 32 x 32 latent state to a 4 x 4 spatial resolution using three convolutional layers, each with stride 2, batch normalization, Leaky ReLU activation, and channel dropouts (Tompson et al., 2015). This reduction is followed by a fully connected layer to project the latent state into the behavior observation space,
The recurrence functions, f and fA each use a single convolutional layer with 3 x 3 kernels to process the latent states locally. Afterward, the model applies multi-head self-attention to patches of the latent state images to capture long-range dependencies across the latent space. Each latent patch is mapped to 356-dimensional embedding space, which is then used compute the self-attention.
Training details, including the learning rate, optimizer choice, and hyperparameters of the mappings, are summarized in Table A.I.
Hyperparameter Tuning:
We use latent states with dimensions Wh x 32 x 32 across all experiments. The first n channels of the latent states are used in ConvRNNZ to model behaviorally relevant dynamics. Any additional channels are used for ConvRNN2 to learn residual neural dynamics (M2 = max(ng n1, 0)). We use n1 = 8 across all experiments. The number of latent channels, f22 was tuned in the set {1, 2, 4, 8, 16, 32). where for nu S 8 only ComvRNN/ is trained, and for nu > 8 ConvRNN2 is learned in the second phase in addition to COmRNNNI. The patch size for the self-attention layer in f ((1) and ( was tuned in the set {1,2, 4, 8, 16).
Implementation Details: For WFCI dataset with 39200 neural image samples used for training, takes 2445 seconds on average run all 3 optimization steps for 80 epochs on NVIDIA RTX 6000 Ada Generation GPU, and the inference takes 13.5 seconds on 9800 sequential samples.
A.2. Ablation Details
1. MLP (PARAMETERIZED)-SBINDD
This ablation explores the importance of convolutional layers in our model by replacing them with MLPs. maintains the same two-phase learning scheme as SBIND, but uses MLPs to parameterize all the mappings in both phases and relies on
(2) are vectors of shape In x1x1 and MLPs to learn spatiotemporal structure in neural image data. The states x and x k Ra2x1x1 respectively, disregarding the spatial distribution in neural images.
Optionally, we use commonly used preprocessing techniques on widefield and ultrasound data to obtain low-dimensional representation (as vector) for the neural images. This representation is then used as input for training and inference the model. After learning the model and obtaining predictions in the oww-iiensional space, we project the prediction back to the neural image space to compare the performance in neural prediction with SBIND.
A.2.2. SBIND w/o fA
This ablation investigates the importance of recurrent neural networks in our model by removing the recurrence mapping, f-). This is essentially a Convolutional Autoencoder (CAE) that takes neural images input and attempts to predict neural images one-step into the future. This is equivalent our model without the recurrence function, just optimizing for K, C, and D. This ablation is used to pinpoint the importance of using information from neural images more than one sample in the past for modeling. A behavior decoder mapping, D is trained downstream to project the latent representation of the CAE X5, to the behavior of interest. The same loss functions from SBIND are used for neural and behavioral prediction, The inference of this model can be formulated as:
X = K(Y:)
Ye41 c(X+1))
(A.13)
2 = D(X++))
17

---

Dynamical Modeling of Behaviorally Relevant Spatiotemporal Patterns in Neural Imaging Data
Table A.I. SBIND Model and Training Details and Hyperparameters Across All Datasets
Component
Hyperparameter
WFCI
WFCI2 2
FUSI
Input Dimensions
1x128x128
1x128x128
1x128x125
8
8
24
24
General
Batch Size
30
Sequence Length
30
Learning Rate
le-3
le-3
le-3
LR Schedule
StepLR
StepLR
StepLR
Max Training Epochs
80
80
80
Weight Decay
le-6
le-6
le-6
Layers
3
3
3
Kerne Size
5x5
5x5
5x5
Strides
2,
2, 2, 1
2, 2, 1
Neural Encoders (()) (K(2))
Channel Dropout
0
Num Kernels
32, 32, n1 (n2)
32, 32, 71 (n2)
32. 32, n1 (n2)
Layers
2
Kernel Size
5x5
5x5
5x5
Neural Decoder C (C(2))
Strides
2
2
Channel Dropout
0
0
0
Num Kernels
32, 32,
32, 32,
32, 32,
Aza
2.0
2.0
2.0
Agrad
0.3
0.3
0.3
Conv Layers
4
Kernel Size
5x5
5x5
5x5
Strides
2
2
Behavior Decoder (D())
Channel Dropout
0.4
0.4
0.4
Num Kernels
64,
64, 64
64, 64, 64
16. 16, 16, 16 1
FCN Layers
FCN hidden units
64
64
16
Conv. Layers
I
Kernel Size
3x3
3x3
3x3
Strides
Recurrence function f( (f(²
Channel Dropout
0
0
Hidden Dim
48
48
48
Num Kernels
Ti (n2)
(122)
n1 (n2)
Self-Attention Heads
8
8
Patch Size
8
Embedding Dim
256
256
256
Positional Embedding
Learnable
Learnable 16
Learnable 16
Num Patches
16
A.2.3. SBIND MSE LY
This ablation explores the effect of the neural loss function in Equation A.5 by removing the GDL and LI loss components. This ablation the same model architecture and training procedure SBIND, but trains the model using only the MSE loss for neural prediction in the optimizations of Equations A.9 and A.II.
A.2.4. SBIND-UNSUP (UNSUPERVISED)
This ablation investigates the importance of disentangling behaviorally relevant dynamics by removing the first phase of our algorithm. This forces the model to learn all neural dynamics without considering their relevance to behavior, as
predicts neural data one step the future without using behavior information. Effectively, this ablation sets n and
18

---

Dynamical Modeling of Behaviorally Relevant Spatiotemporal Patterns in Neural Imaging Data
fs = T22 as it learns X & and ConvRNN2 parameters while still using self-attention the recurrence and convolutional layers. Because SBIND-Unsup only learns a single ConvRNN for neural prediction, may learn behaviorally irrelevant neural dynamics in neural images, potentially resulting in inferior behavioral prediction performance.
We use the same latent state dimensions, fz x H' x W whenever we compare the performance of SBIND with this variant.
A.2.5. SBIND NOATT
This ablation assesses the impact of the self-attention mechanism by removing it from the recurrence function, fa()) This model still disentangles behaviorally relevant neural dynamics using the two-phase learning scheme. also uses the same hyperparameters for all other mappings (K, C, and D) and the same loss functions as SBIND. However, it does not utilize self-attention to capture long-range spatial information in the latent space, and consequently, in the neural image space. By removing self-attention, (.) simplifies to local convolutional layer. This prevents the model from capturing dependencies between distant brain regions and using these dependencies for neural and behavioral prediction (Figure 4),
A.3. Baseline Neural-behavioral Models and Preprocessing Methods
First, we list the two dimensionality reduction methods that commonly used when working with widefield calcium and functional ultrasound imaging data. We use these as optional preprocessing steps to obtain a low-dimensional representation for neural- behavioral baselines. We compare our model with CEBRA (Schneider et al., 2023), which extracts latent embeddings informed by behavior and fits decoders for neural and behavior observations. Moreover, we compare our model performance with DPAD (Sani et al., 2024), which a nonlinear dynamical model that learns behaviorally relevant neural dynamics.
A.3.1. PCA
PCA is often performed on widefield calcium imaging data dimensionality reduction technique before performing modeling (Musall et al., 2019). Enough principal components are extracted to explain a sufficient amount of variance in the neural images. For functional ultrasound imaging, PCA serves the same purpose and is also employed to decode movement intentions (Griggs et al., 2024; Norman et al., 2021).
When using PCA for preprocessing the baseline models, we tune the number of principal components (PCs) used to represent the neural images, treating it as a hyperparameter. We select the number of PCs from the set of values (25, 50, 100, 200, 400}. After training baseline models with PCA preprocessing and obtaining predictions of PCs in the low-dimensional space, project the predictions back to the neural image space to compare the performance in neural prediction with SBIND.
A.3.2. LOCANME
Localized semi- cemi-nonnegative matrix factorization (LocaNMF) (Saxena et al., 2020) is a dimensionality reduction method that decomposes widefield imaging data into localized spatial components and corresponding temporal components. The temporal component is used as a low-dimensional representation of widefield calcium imaging data. LocaNMF leverages the Allen brain atlas (Wang et al., 2020) to initialize spatial components and encourages localization by limiting their spread, while still allowing contributions from neighboring regions to capture relevant variance. The result is more interpretable decomposition, where each temporal component primarily corresponds to a specific brain region
When using LocaNMF for preprocessing, we tune its hyperparameters. The number of components for Singular Value Decomposition is varied within the set of values (100, 200, 400, 1000). The "minrank" hyperparameter is selected from the set {1, 5). Other hyperparameters are set to their default values.
A.3.3. CEBRA
CEBRA (Schneider al., 2023) is a non-dynamic model that uses convolutional neural network encoders in its architecture to process neighboring time points of the data within small, fixed window length. It uses a contrastive loss to extract latent embeddings informed by either simultaneous behavior labels or time information. CEBRA-Behavion uses an objective that aligns neural activity in the embedding space such that time points with similar behavior have similar embeddings. CEBRA-Time uses time information to extract the embeddings. After learning the embeddings, it fits decoders to map the embeddings from each time point to the observation space (i.c., behavior or neural).
19

---

Dynamical Modeling of Behaviorally Relevant Spatiotemporal Patterns in Neural Imaging Data
We performed hyperparameter tuning for CEBRA. We used the default "KNN-Decoder" for categorical behavior prediction in WFCI 2 and fUSI datasets. For neural prediction and continuous behavior, we tested the default "KNN- "L1 Linear Regressor", and Ridge regressor decoder, with the latter achieving superior performance. We report neural reconstruction pero--tep-ahheale across folds and all the pixels within the brain areas for CEBRA. For continuous behavioral data, we report same-step decoding performance, and for categorical behavioral data, we report accuracy, auc, or Fi-score as appropriate. The embedding dimension was explored from 1 to 256 in powers of 2. For widefield datasets, the "time-offset" was selected from {5, 10,20), 20), while for the functional ultrasound imaging dataset, it was chosen from {3,5, 10). The "temperature" hyperparameter was varied within the set {0.01, 0.101,0.1,1, 10). The best-performing model across folds is reported for all experiments.
A.3.4. DPAD
DPAD (Dissociative Prioritized Analysis of Dynamics) (Sani et al., 2024) is nonlinear dynamical model that focuses on learning behaviorally relevant neural dynamics and dissociating them from other dynamics in neural activity. It achieves this by fitting two dynamical models, formulated as a two-section RNN, one for behaviorally relevant neural dynamics and another for the remaining neural dynamies. DPAD also replaces linear mappings in the dynamical models with MLPs to flexibly learn the source of nonlinearity in the data. However, it not specifically designed for image data and thus does not explicitly account for the spatial structure in the image-distributed neural data.
To compare with DPAD, we first identified the source of nonlinearity by using MLPs for each of the parameters of the model. We identified behavior readout parameter, C,, as source of nonlinearity and used an MLP with I or 2 hidden layers for this parameter. We tuned DPAD hyperparameters by varying the latent state dimension from 1 to 256 in powers of 2. For neural prediction and continuous behavioral data, we report one-step-ahead prediction for comparison. For categorical data, we report accuracy, AUC, or Fi-score as appropriate.
A.3.5. STNDT
STNDT (Le & Shlizerman, 2022) utilizes Transformer architecture for spatiotemporal modeling of neural population spiking activity. Originally designed for spiking data, we adapted STNDT to accept preprocessed LocaNMF features as input, as its direct application to raw images is computationally prohibitive due to the quadratic complexity of its spatial self-attention mechanism over large number of pixels. For these LocaNMF features, we placed a Gaussian prior on the components and employed an MSE loss instead of the model's original Poisson likelihood. STNDT was trained using its original objectives, including masked reconstruction and a contrastive loss. For behavior decoding, we followed the approach discussed on STNDT's OpenReview forum, which is to use ridge regression decode behavior from the learned latent states.
For hyperparameter tuning when using STNDT with LocaNMF features, we used the default hyperparameter choices such as number of transformers, masking ratio, etc., and varied the number of LocaNMF components provided as input (i.c., embedding dimension for STNDT),, exploring values in the set (55, 123, 270}. Table A.5 reports performance of the adapted STNDT model which achieves the best behavior decoding.
A.3.6. TNDM
TNDM (Hurwitz al., 2021) is sequential autoencoder-based model designed to learn two distinct sets of latent factors from spiking data, with dimensionalities n1 and n22, corresponding to behaviorally relevant and behaviorally irrelevant dynamics, respectively. It achieves this by optimizing a combined neurao-behavioral reconstruction loss. Given TNDM's original design for Poisson-distribueed spiking data, we adapted it for our widefield imaging datasets. This involved using preprocessed LocaNMF features as input, assuming a Gaussian distribution for these input features, and changing TNDM's neural reconstruction loss to MSE.
Hyperparameter tuning for TNDM involved sweeping the dimensionalities for the behaviorally relevant latent factors, n1. selected from {8, 16, 32, 64), and the behaviorally irrelevant latent factors, n2,, selected from (0, 16, 64). Table A.5 reports performance of the adapted TNDM model which achieves the best behavior decoding.
20

---

Dynamical Modeling of Behaviorally Relevant Spatiotemporal Patterns in Neural Imaging Data
A.4. Datasets Details
A.4.1. WIDEFIELD CALCIUM IMAGING (WFCT) DATASETS
The WFCI datasets were collected from head-fixed mice performing decision-making making task, where they reported the spatial position of auditory visual stimuli by licking the corresponding spout (Churchland et al., 2019). Neural activity across the dorsal cortex was optically recorded at 30 and 15 Hz for WFCI and WFCI 2 datasets, respectively. We preprocessed the neural images remove hemodynamic artifacts using linear regression method (Musall et al., 2019; Valley et al., 2020). The raw neural images, with dimensions 540x640 pixels, were cropped to include only the brain regions and downsampled to 128x123 128 pixels. A pixel-wise temporal causal filter (0.1 Hz, 2nd order Butterworth high-pass) was applied to remove drift in the time series.
WFCI 1:
This dataset consists of 248 trials with variable lengths 1.55 seconds). Concurrently with neural recordings. behavior videos were recorded from two viewpoints (face and bottom; see Figure 2a). Following similar procedure to (Musall et al., 2019), 14 dimensions of continuous behavior were extracted from seven regions of interest (eye, nose, whisker, paw, chest, body, and mouth) in the videos (see Figure 2c). For each region, the first principal component of both the original video frames and the motion video frames (computed as the absolute temporal derivative of frames) was extracted and used for behavioral prediction (see Figure A.5).
WFCI 2:
This dataset comprises 412 trials with variable lengths (6.30 + 0.37 seconds) with the same trial structure and neural recordings as WFCI 1. However, instead of behavior videos, four binary sensors detected contact with the animal's forepaws (handles) and tongue (spouts), providing categorical behavioral data for decoding (see Figure 2d), where 1 I's in any of the binary traces represents the time samples where the mouse was touching the corresponding sensor.
A.4.2. FUNCTIONAL ULTRASOUND IMAGING (FUSI) DATASET
The fUSI dataset consists of recordings from a non-human primate performing a nemory-guidee saccade or reach task to either or 8 peripheral targets (Griggs et al., 2023). Trials began with a 5 + second fixation period, followed by a 400 ms presentation of peripheral cue. After the cue disappeared, there was a 5 + second memory period before the monkey executed saccade or reach to the remembered target location. Successful trials were followed by 1.5 + 0.5 second hold period and then a reward. Each trial was followed by an inter-trial period before the next trial started.
Preprocessing:
We applied causal temporal voxel-wise filter (0.02 Hz high-pass Butterworth, 2nd order) to remove drift during the sessions. Similar to (Griggs et al., 2024), we z-scored the data voxel-wise over a rolling 60-frame buffer. Next, a pillbox spatial filter with a radius of 2 pixels was applied to each frame. The images were originally 128 x 132 pixels and cropped to 128x x 128 pixels.
Decoding: In this dataset, behavior consisted of the target the monkey reached or fixated on for each trial. Thus, we considered behavior as available only during the 1.5-second period (equivalent to 3 samples) before reward period when the monkey was fixating on the target. This gave us categorical and intermittently recorded behavior time-series for modeling, We used these 3 samples as the only samples in the trials of length 30 where we have intermittent behavior available. To fit all variants of SBIND, we masked out other time points in the trial and optimized the parameters only for those 3 specific samples, effectively implementing intermittent behavior decoding during training. In the 2-directional tasks, we used binary target classification. In 8-directional tasks, similar to (Griggs et al., 2024), we used a multi-decoder approach in the decoder mapping to predict the vertical and horizontal directions. Thus, the decoder (()) has 6 softmaxed output dimensions: for probabilities of left-right-center summing to 1, and for probabilities of up-center-down summing 1. For training PCA+LDA, we used either the 3 samples before the reward period, as in (Griggs et al., 2024), all samples of the trial to decode directions. For training CEBRA, we used either the 3 samples before the reward period or all the samples (default choice for target classification task) of the trials for learning the embeddings, with the latter proving more effective for the target classification task. To fit DPAD, we used the 3 samples before reward period to fit the first RNN. During evaluation, we used the latent embedding at the last time-step in the trial to predict the direction of movement.
A.5. Supplementary Experiments
21

---

Dynamical Modeling of Behaviorally Relevant Spatiotemporal Patterns in Neural Imaging Data
Table A.I. One-step-alee step-ahead behavior decoding and neural prediction performances for various ablations of SBIND across folds for WFCI 1 dataset in terms of R². As indicated by the arrows, higher is better for R². For neural prediction, R² (Mean = SEM) is reported across folds and pixels within the brain areas. For behavior decoding, R2 (Mean SEM) is reported across folds and 14 dimensions behavior.
MODEL
PREPROCESSING
BEН. R² t
NEUR. R² 1
MLP-SBIND
FLATTEN
0.3620 1 0.0194
0.8209 1 0.0033
MLP-SBIND
LOCANME
0.4025 + 0.0152
0.8702 + 0.0015
MLP-SBIND
PCA
0.3934 0.0145
0.8926 0.0016
SBIND-UNSUS
0.4589 + 0.0108
0.8724 0.0039
SBIND NOATT
0,4612 1 0.0107
0.8652 0.0032
SBIND w/o FA
0.2080 + 0.0453
0.8543 + 0.0037
SBIND MSE Ly
0.5030 + 0.0179
0.8217 + 0.0133
SBIND
0.5059 1 0.0166
0.8724 1 0.0069
Table A.2. One-step step-ahead behavior decoding and neural prediction performances for various ablations of SBIND across 5 folds for WFCI 2 dataset in terms R and AUC. As indicated by the arrows, higher is better for R and AUC. For neural prediction, R (Mean t SEM) is reported across 5 folds and all pixels in the brain areas. For behavior decoding, AUC (Mean + SEM) is reported across 5 folds and 4 classification tasks left handle, right handle, left spout, right spout.
MODEL
PREPROCESSING
BEH. AUCT
NEUR. R²T
MLP-SBIND
FLATTEN
0.8706 + 0.0038
0.7503 + 0.0076
MLP-SBIND
LOCANMF
0.8823 + 0.0037
0.6402 + 0.0041
MLP-SBIND
PCA
0.8120 0.0325
0.6703 0.0239
SBIND-UNSUS
0.9182 0.0039
0.7970 0.0133
SBIND NOATT
0.9071 +: 0.0060
0.7451 0.0043
SBIND w/o FA
0.8934 + 0.0045
0.7339 + 0.0022
SBIND MSE LY
0.9299 + 0.0029
0.7418 + 0.0049
SBIND
0.9282 + 0.0020
0.7749 + 0.0074
Table A.3. Behavior decoding and neural prediction R (Mean + SEM) across folds WFCI datasct. As indicated by the arrows, higher is better for R².. For neural prediction, It (Mean 1 SEM) is reported across 5 folds and all pixels in the brain areas. For behavior decoding, IR? (Mean SEM) is reported across folds and 14 dimensions of behavior.
MODEL
PREPROCESSING
BEH. R²
NEUR. R² 1
DPAD
FLATTEN
0.3826 + 0.0189
0.8434 + 0.0022
DPAD
LOCANME
0.4128 = 0.0133
0.8697 = 0.0011
DPAD
PCA
0.3839 + 0.0157
0.8902 1 0.0008
CEBRA
FLATTEN
0.4001 + 0.0132
0.5957 + 0.0216
CEBRA
LOCANMF
0.3745 = 0.0081
0.4453 = 0.0099
CEBRA
PCA
0.3686 + 0.0127
0.4638 1 0.0079
SBIND
0.5059 0.0166
0.8724 0.0069
22

---

Dynamical Modeling of Behaviorally Relevant Spatiotemporal Patterns in Neural Imaging Data
Table A.4. Behavior decoding AUC and neural prediction It (Mean + SEM) across folds for WFCI 2 dataset. As indicated by the arrows, higher better for Rº and AUC. For neural prediction, Itº (Mean 1 SEM) is reported across 5 folds and all pixels the brain areas. For hehavior decoding, AUC (Mean + SEM) is reported across folds and 4 classification tasks for left handle, right handle, left spout, and right spout. For CEBRA, "KNNDecoder" is used for decoding which does directly report AUC.
MODEL
PREPROCESSING
BEН. AUC1
NEUR. R²²
DPAD
FLATTEN
0.8782 = 0.0059
0.7440 = 0.0049
DPAD
LOCANMF
0.8888 0.0057
0.6374 + 0.0033
DPAD
PCA
0.8039 + 0.0090
0.7182 + 0.0038
CEBRA
FLATTEN
0.7228 = 0.0052
CEBRA
LOCANMF
0.4971 = 0.0045
CEBRA
PCA
0.4913 + 0.0131
SBIND
0.9282
0. 0.0020
0.7749 0.0074
Table A.5. Comparison of baselines including adapted STNDT and TNDM on WFCII dataset. Behavior decoding and neural prediction MSE and R² (Mean SEM) across folds.
MODEL
PREPROCESSING
NEUR. MSE 1
NEUR. H² f
MSE
BEH. R²
DPAD
LOCANMF
0.0543 = 0.0009
0.8697 = 0.0011
0.5877 + 0.0226
0.4128 0.0133
CEBRA
LOCANMF
0.4976 = 0.0241
0.4453 + 0.0099
0.6250 + 0.0194
0.3745 0.0081
STNDT
LOCANMF
0.0685 0.0090
0.8376 = 0.0088
0.6033 0.0240
0.3951 + 0.0156
TNDM
LOCANMF
0.7912 0.0290
0.5022 1 0.0081
0.7749 1: 0.0240
0.2233 1 0.0109
SBIND-UNSUP
0.0403 = 0.0020
0.8724 + 0.0039
0.5413 + 0.0185
0.4589 + 0.0108
SBIND
0.0414 = 0.0029
0.8724 = 0.0069
0.4955 + 0.0254
0.5059 + 0.0166
Table A.6. Performance comparison of SBIND WFCII datasets using two recurrent update formulations for integrating current neural image information (Y +). The table contrasts the concatenation approach as in Eq. A.12, where the encoded input K is concatenated with the latent state X₃ before the recurrent function. against the summation approach as Eq. A.1. Results demonstrate that the concatenation method Eq. A.12 yields improved neural prediction MSE, neural R², behavioral MSE, and behavioral R?a for WFCII dataset.
Model Formulation
Neural MSE
Neural R²
Beh MSE
Beh R²
SBIND w. Recurrent Eq. A.12
0.0414 + 0.0029
0.8724 + 0.0069
0.4955 0.0254
0.5059 + 0,0166
SBIND W. Recurrent Eq. A.1
0.0664 + 0.0013
0.8545 + 0.0027
0.5306 0.0198 0.4680 + 0.0182
Table A.7. Comparison of various ablations behavior decoding accuracy (quantificd as proportion of trials whose target was correctly decoded) and AUC across 10 folds and all sessions of [USI Data. For 8-directional sessions multi-decoder approach used with decoders to predict vertical and horizontal directions [left-right-saationnll directions, 4 sessions and 10 folds reported. As indicated by the arrows, higher is hetter for accuracy and AUC.
Multi-class AUC averaged over two vertical and horizontal
2-DIRECTIONAL SESSIONS
8-DIRECTIONAL SESSIONS
MODEL.
PREPROCESSING FLATTEN PCA
BEH. ACCURACY!
BEH. AUC1
BEH. ACCURACY?
BEn. AUC1
SBIND
0.5984 1 0.0162
0.6285 1 0.0216
0.2175 1 0.0169
0.5833 1 0.0196
MLP-SBIND
0.6565 + 0.0178
0.7296 + 0.0214
0.2966 + 0.0183
0.6823 + 0.0170
SBIND- -UNUUP
0.7030 + 0.0180
0.7859 + 0.0194
0.3411 + 0.0187
0.7304 + 0.0176
SBIND NOATT
0.6889 + 0.0184
0.7480 + 0.0237
0.2893 = 0.0179
0.6975 + 0.0145
SBIND
0.7300 1 0.0191
0.8067 0.0180
0.3521 0.0201
0.7393 0.0169
23

---

Dynamical Modeling of Behaviorally Relevant Spatiotemporal Patterns in Neural Imaging Data
Table A.8. Comparison various ablations of SBIND one-step-ahead neural prediction performance (MSE and R²)) across folds and all sessions of IUSI dataset. As indicated by the arrows, lower is better for MSE and higher is better for I??.
2- 2-DIRECTIONAL SESSIONS
8-DIRECTIONAL SESSIONS
MODEL
PREPROCESSING
NEUR. MSEL
NEUR. Rº²T
NEUR. MSEJ
NEUR. R²²T
MLP-SBIND
FLATTEN
0.7885 = 0.0067
0.2641 + 0.0060
0.8743 0.0065
0.2274 + 0.0071
MLP-SBIND
PCA
0.6322 = 0.0054
0.3985 + 0.0043
0.7058 + 0.0030
0.3586 + 0.0038
SBIND UNSUPP
0.4453 + 0.0127
0.6029 0.0112
0.3827 0.0086
0.6683 1 0.0073
SBIND NOATT
0.5036 + 0.0155
0.5545 + 0.0133
0.4230 0.0109
0.6344 + 0.0090
SBIND MSE LY
0.4900 = 0.0108
0.5555 + 0.0094
0.4225 + 0.0076
0.6287 + 0.0064
SBIND
0.4725 1 0.0165
0.5736 1 0.0144
0.3919 1 0.0107
0.6558 1 0.0094
Table A.9. Comparison of baselines in behavior decoding AUC and neural prediction R² across 10 folds and all sessions (Mean + SEM). For DPAD and SBIND, in 2-directional sessions the AUC for binary classification reported over 10 folds and 9 sessions of fUSI Data. In 8-directional sessions, multi-dccoder approach is used with two decoders predict vertical and horizontal directions (lelt-rigtt-stationary) Multi-class AUC averaged over two vertical and horizontal directions, 4 sessions, and 10 folds are reported. For CEBRA, "KNNDecoder" is used for decoding which does directly report AUC. As indicated by the arrows, higher is better for AUC and R².
2-DIRECTIONAL SESSIONS
S-DIRECTIONAL SESSIONS
MODEL
PREPROCESSING
BEH. AUC1
NEUR. RºT
BEH. AUCT
NEUR. R
LDA
PCA
0.7130 + 0.0242
0.6987 = 0.0164
DPAD
FLATTEN
0.5940 0.0221
0.2355 = 0.0066
0.6060 1 0.0157
0.2011 4 0.0079
DPAD
PCA
0.7399 0.0208
0.3938 = 0.0045
0.6752 = 0.0179
0.3554 0.0040
CEBRA
FLATTEN
-0.3899 + 0.0128
-0.5188 0.0085
CEBRA
PCA
00.231 + 0.0069
0.3076 0.0103
SBIND
0.8067 0.0180
0.5736 0.0144
0.7393 1 0.0169
0.6558 0.0094
(a)
(b)
(c)
(d)
10
-01
Figure A.2. Pixel-wise neural prediction H² values across brain regions. Neural prediction maps for the WFCI dataset depicted for (a) CEBRA, (b) SBIND, (c) SBIND NoAtt, and (d) DPAD. SBIND produces more detailed neural predictions compared to its variant without the self-attention mechanism. CEBRA poorly predicts neural activity because it uses latent embeddings guided by behavior and lacks extra embedding dimensions for residual neural activity unrelated to behavior. This is consistent with the observation that widefield calcium imaging datasets often contain significant neural activity unrelated to behavior (Musall al., 2019). Higher is better for I?a.
24

---

Dynamical Modeling of Behaviorally Relevant Spatiotemporal Patterns in Neural Imaging Data
(a)
(b)
(c)
(d)
00
NRMSE
0.0
Figure A.3. Pixel-wise neural prediction NRMSE values across brain regions. Neural prediction maps for the WFCI dataset are
depicted for (a) CEBRA, (b) SBIND, (c) SBIND NoAtt, and (d) DPAD. Lower is better for NRMSE.
(a)
(b)
(c)
SBIND NoAtt
-2.0
Log10 Mean Attribution
(d)
(e)
(f)
-3.0
-4.0
-5
Figure A.4. Mean contribution of all brain regions to predicting the activity of three different pixels marked by in the brain map. The plots display the mean attribution of whole-brain activity to the neural prediction of specific points, derived using the Captum framework. We calculated the attribution of cach input image across all time points the neural prediction of the specified points in different plots. These attribution maps were then averaged across time for different neural images to find the mean attribution. This analysis was performed on the data from the WFCI dataset after training models, (a-c) SBIND NoAtt. (d-f SBIND.
25

---

Dynamical Modeling of Behaviorally Relevant Spatiotemporal Patterns in Neural Imaging Data
framees (PC1)
True
Pred
100
300
400
500
600
700
800
frames (PC2)
Ao Aarlle
bp Vor
Pred
200
200
300
400
500
600
700
800
Truc
100
200
300
400
500
600
700
800
(PC2)
Pred
100
400
600
800
(PC2)
True
100
200
300
400
500
600
700
80o
frames (PC22
True
Pred
100
300
400
500
600
700
800
Time (samples)
Figure A.5. SBIND example behavior decoding. Predictions over dimensions continuous behavior extracted from behavior videos for WFCI dataset. (See Appendix A.4 and Figure 2 for details of behavior extraction.)
26

---

Dynamical Modeling of Behaviorally Relevant Spatiotemporal Patterns in Neural Imaging Data
0.06
0.05
Mean MSE
0.04
0.03
0.02
0.01
0.00
No Attention
16
Figure A.6. Larger Self-Attention Patch Sizes Lead Better Neural Prediction. Neural prediction MSE across different patch sizes for the WFCI dataset across 5 folds and 2 runs.
27


Reviews:
[{"review_id": "9wPmDlaVq4", "reviewer": "Reviewer_UYoU", "summary": "This work proposes SBIND, a dynamical model for neural imaging data designed to extract behaviorally relevant spatiotemporal patterns. The model mainly uses a double-RNN technique to disentangle behaviorally relevant neural dynamics from other covariates of high-dimensional neural activity. The first RNN captures the behaviorally relevant latent states, while the second RNN accounts for remaining neural dynamics. The study demonstrates that SBIND outperforms existing models on several benchmarks.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### [1]: Neuroscientific meaning\nWe thank the reviewer for raising this key point regarding the scientific utility of neural imaging data compared to electrophysiology (EP) data, which we will now clarify in the manuscript. Brain function relies on diverse spatial and temporal scales from single neurons to large-scale networks of neuronal populations. To understand how the brain generates behavior, we need to model neural activity across all these scales. While EP recordings measure the small-scale spiking activity of a group of single neurons, they do not measure large-scale networks that are thought to be a key basis for cognition and complex behavior (Cardin et al, 2020). Neural imaging can measure these large-scale networks by providing **large-scale spatial coverage** across cortical areas or even whole-brain access (Macé et al., 2018). As such, neural imaging data are complementary to EP data and play an increasingly central role in modern neuroscience by allowing the study of **large-scale network dynamics, functional connectivity, and mesoscale neural analyses**, which are all crucial for a full understanding of how the brain generates behavior and task performance (Musall et al., 2019; Nietz et al., 2023). Additionally, functional ultrasound imaging (fUSI) may offer a less invasive approach compared to EP for developing brain-computer interfaces (BCIs) as recently demonstrated (Griggs et al. 2024; Rabut et al., 2024).\n\nFinally, we agree with the reviewer that neural imaging and EP data have distinct statistical characteristics. Precisely for this reason, we developed SBIND to directly learn such specific spatiotemporal structure from raw neural images, to enable extracting more precise dynamical neural information. We recognize that our initial motivation for focusing on these imaging modalities could have been more clearly articulated and appreciate the reviewer’s insight.\n### [2]: Neural-behavioral models\nFirst, as explained above, neural imaging data play a critical role in understanding large-scale network dynamics that are key to complex behavior and cognition. Furthermore, we demonstrated SBIND's application on two fundamentally distinct neural imaging modalities that are increasingly employed: an optical modality based on widefield calcium imaging (WFCI), and an acoustic modality based on fUSI.\n\nSecond, while we agree that the general concept of joint neural-behavioral modeling has been explored, particularly for EP data, we emphasize that SBIND's novelty and contribution lie in doing so for neural imaging modalities by developing a novel architectural design tailored for the unique challenges of neural imaging data. These challenges, as highlighted in the introduction and also noted by the reviewer, include **high dimensionality, complex spatiotemporal dependencies, and prevalence of behaviorally irrelevant dynamics**. Our model differs significantly from prior joint neural-behavioral models that typically operate on lower-dimensional EP recordings that have a much smaller spatial scale than neural imaging (Sani et al., 2024), or on preprocessed time series extracted from neural imaging data (Wang et al., 2024).\n\nFinally, we demonstrated that SBIND outperforms SOTA methods, such as DPAD and CEBRA. We now also add a new baseline, STNDT (Table B.2 in our response to Reviewer 9wsc) to our results, showing SBIND’s superior performance in both behavior decoding and neural prediction compared with all baselines. Furthermore, as Reviewer z1x2 pointed out, \"methods that can be applied to fUSI are relatively sparse\", and to our knowledge, SBIND is the first dynamical latent state model for fUSI, whose successful application to fUSI underscores its potential to **facilitate the design of non-invasive BCIs**.\n### [3]: References\nWe thank the reviewer for bringing these relevant papers to our attention and will incorporate a discussion of them into our revised manuscript. While these represent important advancements, they differ significantly from SBIND. They all address electrophysiology data and focus on distinct goals such as cross-session latent alignment (Wang et al., 2023), modeling inter-regional communication (Li et al., 2024), capturing spatial and temporal dependencies in population activity (Le et al., 2022), or recovering interpretable inter-neuron connectivity (Zhang et al., 2024). Crucially, none are designed for joint neural-behavioral modeling, and thus they do not disentangle behaviorally relevant dynamics, nor do they employ image-specific architectural priors like SBIND's ConvRNNs with integrated self-attention. SBIND's focus thus remains distinct in leveraging the spatiotemporal structure of imaging data for robust, disentangled neural-behavioral modeling.\n\n**References:**\n\nCardin et al., Shining a Wide Light on Large-Scale Neural Dynamics, Neuron 2020\n\nMacé et al., Whole-Brain Functional Ultrasound Imaging Reveals Brain Modules, Neuron 2018\n\nAlso see Lines 452, 524, 529, 542, 559", "claims_and_evidence": "The authors shows that the proposed SBIND improves neural-behavioral decoding performance owns to effectively disentangling behavior-related and remaining neural dynamics. The results are generalizable across datasets as well as detailed ablation studies.", "methods_and_evaluation_criteria": "For the method, I am a bit skeptical about the neuroscience meaning of learning with neural imaging data, which is highly noisy. By comparison, may be some electrophysiology data, like Poisson spiking counts, are much more scientific meaningful. Meanwhile, the model structure of decoding neural and behavior at the same time is a bit too common actually.", "theoretical_claims": "No theoretical claims within this work.", "experimental_designs_or_analyses": "The metric of decoding for behavior prediction provides reasonable experimental validation.", "relation_to_broader_scientific_literature": "I don't really know the contribution of this paper to broader neuroscience field given that it's only processing a certain kind of imaging data format which is hard to truly interpret.", "essential_references_not_discussed": "For the neural data integration:\n* Extraction and Recovery of Spatio-Temporal Structure in Latent Dynamics Alignment with Diffusion Models. NeurIPS 2023, Wang et al.\n* Multi-Region Markovian Gaussian Process: An Efficient Method to Discover Directional Communications Across Multiple Brain Regions. ICML 2024, Li et al.\nFor the neural representation learning:\n* STNDT: Modeling Neural Population Activity with a Spatiotemporal Transformer. NeurIPS 2022, Le et al.\n* NetFormer: An interpretable model for recovering identity and structure in neural population dynamics. 2024, Zhang et al.", "other_strengths_and_weaknesses": "Please refer to my Methods And Evaluation Criteria section.", "other_comments_or_suggestions": "Please refer to my Methods And Evaluation Criteria section.", "questions_for_authors": "No more Questions. Please refer to my Methods And Evaluation Criteria section.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VYaNYKnVsY", "reviewer": "Reviewer_9wsc", "summary": "This work propose SBIND, a data-driven deep learning framework to model the spatiotemporal dependencies in the neural image data and behavior data. Existing methods fail to model the dependencies of behaviors and neural dynamics. This work allows modeling the complex local and global spatial temporal patterns, and achieving better performance in predicting dynamics and behavior decoding. This model includes a neural encoder with two separate modules for behavior and neural decoding, and outperforms baselines such as CEBRA and DPAD without the need for preprocessing.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### [1]: New Baselines\n\nWe thank the reviewer for their constructive feedback. We agree that comparing SBIND with more baselines strengthens our contribution. In response, we have added new baseline comparisons.\n\n**Comparison with STNDT:**\n\nSTNDT uses a Transformer architecture for spatiotemporal modeling of neural population spiking activity. Unlike SBIND, which processes raw images, STNDT was originally designed for Poisson-distributed spiking data. **We adapted STNDT to accept preprocessed imaging data extracted using LocaNMF**, which is a widely used approach (Wang et al, 2024). We put a Gaussian prior on LocaNMF (instead of the original Poisson prior for spikes), and trained with STNDT's original objectives (masked reconstruction and contrastive loss). Following the original work, behavior decoding was performed via ridge regression on the learned latents. We swept the number of latents for STNDT. The results (**Table B.1**), show that SBIND significantly outperforms STNDT on WFCI1 dataset.\n\n\n**Table B.1: Comparison with STNDT.**\n|Model|Num latents|Neur. R2|Beh. R2|\n|---|---|---|---|\n|STNDT|55|0.8326±0.0015|0.3529±0.0164|\n|STNDT|123|0.8408±0.0012|0.3711±0.0153|\n|STNDT|270|0.8376±0.0088|0.3951±0.0156|\n|SBIND|$n_x=16$|0.8724±0.0069|0.5059±0.0166|\n\n**Comparison with TNDM:**\n\nTNDM (Hurwitz et al., 2021) uses a sequential variational autoencoder (SVAE) to learn two sets of latent factors in spiking data with dimensionality $n_1$ and $n_2$ for behaviorally relevant and irrelevant dynamics, respectively. TNDM was originally designed for Poisson-distributed spiking data. To provide a meaningful comparison on our WFCI dataset, we adapted TNDM to accept preprocessed imaging data extracted using LocaNMF. We assumed a Gaussian distribution for these inputs and changed TNDM's neural reconstruction loss to MSE. We also swept $n_1$ and $n_2$ values for TNDM. As shown in the **newly added Table B.2**, SBIND achieves superior performance compared to this adapted TNDM in both neural prediction and behavior decoding, demonstrating the benefit of SBIND's architecture that is specifically designed for spatiotemporal image data.\n\n**Table B.2: Comparison with TNDM on WFCI1 Dataset.**\n|Model|$n_1$|$n_2$|Neur. R2|Beh. R2|\n|---|---|---|---|---|\n|TNDM|8|8|0.5442±0.0073|0.1718±0.0173|\n|TNDM|16|16|0.5464±0.0102|0.1683±0.0203|\n|TNDM|16|64|0.5022±0.0081|0.2233±0.0109|\n|SBIND|8|8|0.8724±0.0069|0.5059±0.0166|\n\nThe reviewer also mentions two other methods, but direct empirical comparison to these was not feasible as described below:\n\n* **PoYo (Azabou et al., 2023)** is a foundation model for spiking activity, which builds an unsupervised, large-scale foundation model to generalize across different subjects and tasks. This is a different goal compared to SBIND’s goal of joint neural-behavioral modeling to disentangle behaviorally relevant neural dynamics. PoYo uses a tokenization scheme specific to spikes, whereas SBIND focuses on neural imaging modalities. Furthermore, the code for PoYo is not available.\n\n* **BeNeDiff:** The code for BeNeDiff is also not publicly available. BeNeDiff employs SVAEs similar to TNDM (which we now compared to). A key distinction is that SBIND processes raw imaging data, while BeNeDiff uses **LocaNMF**-preprocessed data. SBIND captures local and global spatiotemporal dependencies in images, which may be lost after preprocessing with methods such as LocaNMF.\n\n---\n### [2]: Neural Latent Benchmark (NLB) Comparison Request\n\nWe thank the reviewer for raising this point. Unfortunately, however, all NLB datasets consist exclusively of **electrophysiological (EP) recordings**. SBIND, in contrast, is specifically designed to model the spatiotemporal grid structure of *neural imaging data* (WFCI and fUSI). Our core contribution lies in developing a method tailored to leverage this structure from raw image sequences. Therefore, while the NLB is valuable for benchmarking models designed for EP data, it is not directly applicable for evaluating the specific contributions and application of SBIND.\n\n---\n### [3]: Question on Raw vs Preprocessed Data\n\nThis is another great point about modeling raw versus preprocessed neural images. Methods like LocaNMF extract latent components roughly corresponding to brain regions. The adapted STNDT we compared against uses learnable positional embeddings to encode spatial relationships within LocaNMF components. However, as shown in our comparison (Table B.2), this method, even with mechanisms for spatial encoding on preprocessed data, yields inferior performance compared to SBIND.\n\nIn terms of computational efficiency, we acknowledge that training models on preprocessed data is generally faster than training on raw images. However, as shown in Tables 2 and 3, baselines using various preprocessing methods consistently perform worse than SBIND. Therefore, SBIND represents a trade-off that prioritizes higher predictive accuracy and the ability to learn directly from the neural images.", "claims_and_evidence": "This model demonstrates the benefits of integrating behavior and neural activity into one unified framework, which allows modeling the dependencies between behavior and neural dynamics. This model demonstrate superior performance in both behavior decoding and neural activity prediction. The claim is well-supported.", "methods_and_evaluation_criteria": "It demonstrates on two publicly available neural imaging and behavior datasets. This model demonstrated superior performance in both decoding behavior as well as predicting neural dynamics. While there are multiple related works that are not directly compared with, such as BeNeDiff, TNDM and mm-GP-VAE mentioned in the related works. Moreover, there are also variants of approaches that are studied in the datasets from Neural Latent Benchmark Challenges, which this work is not evaluated on, previous works such as [1][2] that decoding behaviors that not compared with. \n\n[1] STNDT: Modeling Neural Population Activity with a Spatiotemporal Transformer.\n\n[2] A Unified, Scalable Framework for Neural Population Decoding.", "theoretical_claims": "There are no theoretical claims or proofs.", "experimental_designs_or_analyses": "The model is evaluated on two publicly available neural datasets, and multiple well-established baselines including CEBRA, LDA, DPAD are compared with. The model has demonstrated superior performance, while another neural latent benchmark could also be covered in this study.", "relation_to_broader_scientific_literature": "This paper worked on an important and challenging problem in brain behavior decoding. It outperforms existing well-established works such as CEBRA, and achieving higher predictive accuracy, which provides an effective method to the neuroscience community.", "essential_references_not_discussed": "Variants of approaches that are studied in the datasets from Neural Latent Benchmark Challenges, which this work is not evaluated on, previous works such as [1][2] that decoding behaviors that not compared with.\n\n[1] STNDT: Modeling Neural Population Activity with a Spatiotemporal Transformer.\n\n[2] A Unified, Scalable Framework for Neural Population Decoding.", "other_strengths_and_weaknesses": "The paper presents an effective solution to predict neural activities as well as behavioral decoding, it demonstrates better accuracy compared to existing approaches. While this model is not evaluated on a comprehensive Neural Latent benchmark and other existing SOTA methods.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "How will the model perform using neural imaging data directly, compared to using the extracted neural activities and spatial locations? Will that provide additional computational efficiency while without sacrifice of accuracy?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_concerns": "N/A"}}, {"review_id": "OPykqcjV5m", "reviewer": "Reviewer_z1x2", "summary": "The authors propose SBIND, which learns behaviorally relevant and irrelevant neural dynamics directly from high-dimensional imaging data without preprocessing. The authors apply SBIND to widefield imaging datasets and functional ultrasound, and find that SBIND outperforms existing methods that involve preprocessing in predicting one-step-ahead behavior and neural activity.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### [1]: Comments & Suggestions\nThank you for these helpful suggestions for improving the clarity of our work.\n\n* We will fix the typo on Line 87 and ensure all symbols are clearly defined upon first use. Specifically, $n_y$ is the number of neural images in $Y_k$ at time k, and H and W are the height and width. $n_x$ is the number of dimensions of the latent states, and $n_z$ represents the dimension of the behavior vector, $z_k$. Across all datasets, $n_y=1$, but we include $n_y$ for generality of the formulation.\n* We agree with the suggestion regarding notation consistency and will amend Line 162 to clarify the RNN estimates $X_{k+1}$ given $Y_{1:k}$.\n* **Inference using the full sequence $Y_{1:K}$**: Our model performs inference causally (prediction at time k+1 uses observations up to time k), given its importance for causal neuroscience investigations and real-time BCI applications. Using the full sequence would correspond to non-causal inference (i.e., smoothing), which was not the goal of this work but can be achieved using bidirectional RNNs in future studies.\n* Finally, we have verified that the anonymous link to the code works; the previous issue may have been a temporary glitch.\n\n---\n### [2]: Combined Loss\nWe adopted the two-stage approach because it allows us to explicitly disentangle the behaviorally relevant latent states $X^{(1)}$. The ConvRNN in Stage 1 is optimized *specifically* for behavior prediction, allowing the first latent state, $X^{(1)}$, to focus on capturing the behaviorally relevant dynamics. Stage 2 then focuses on modeling the *residual* neural dynamics, which are not predictable from $X^{(1)}$. This provides a clear separation/disentanglement of states.\n\nWhile one-stage training with a mixed neural-behavioral loss is possible, doing so poses a challenge for clear disentanglement. Specifically, in this case, there is no guarantee that any state dimension will just focus on behaviorally relevant or just the residual dynamics. As such, states may actually be mixed up rather than disentangled. Doing so also may create practical challenges. First, the mixed loss optimization will heavily rely on **tuning the relative weights** of the neural and behavior loss terms, which can be difficult and sensitive to hyperparameters. Second, achieving disentanglement with a single combined loss may necessitate incorporating additional specialized loss terms or architectural constraints specifically designed for separation –e.g., KL divergence terms in TNDM (Hurwitz et al., 2021) that we now compare to and show that SBIND outperforms it (see Table B.2); finding the optimal balance for these specialized terms typically requires a potentially sensitive hyperparameter tuning process. In contrast, our two-phase approach avoids this sensitive hyperparameter tuning.\n\n---\n### [3]: Multi-step Decoding & Inference Speed\nWe thank the reviewer for this important question. We will add these points to the manuscript.\n\n**Multi-step Prediction:** SBIND's recurrent architecture inherently allows for **multi-step** prediction. We can do so with no additional optimization or retraining by performing **recursive forecasting** during inference as follows: instead of feeding the neural observation at the next time step into the neural encoder ($K^{(1)}$), we can feed the model's own one-step-ahead neural image prediction, $\\hat{Y}_{k+1}$.\n\nThis allows the model to predict the subsequent state $X_{k+2|k}$, behavior $z_{k+2}$, and neural image $Y_{k+2}$ without having the observation at time k+1, and this same process can be iterated further into the future.\n\n**Inference Speed:** A single inference step of the SBIND model takes **17.9 ms** on an NVIDIA RTX 6000 GPU, which is faster than the sampling intervals of WFCI (\\~33-67 ms) and fUSI (\\~100-500 ms), suggesting potential feasibility for real-time BCI applications (Rabut et al, 2024; Mace et al, 2011).\n\n---\n### [4]: Equation 2\nThis is another great question. Because $X_k$ and $Y_k$ have **different spatial dimensions** ($H' \\times W'$ vs. $H \\times W$), we must first *encode* $Y_k$ using $K(\\cdot)$ to obtain a representation $K(Y_k)$ with the same spatial dimensions as $X_k$. Afterwards, there are two main options: either **sum** $K(Y_k)$ with the processed state $f_A(X_k)$ (as formulated in Eq. 2, or equivalently, Eq. A.1), or **concatenate** $K(Y_k)$ with the state $X_k$ before applying the recurrent function $f_A(\\cdot)$ (as discussed in Appendix A.1.5). For the WFCI datasets reported, we utilized the **concatenation approach** (Eq. A.12), and it yielded better performance as seen in the **new Table B.3**.\n\n**Table B.3: Comparison of SBIND Variants on WFCI1 Dataset.**\n||Neur. R2|Beh. R2|\n|-|-|-|\n|SBIND w. Recurrent Eq. A.12|0.8724±0.0069|0.5059±0.0166|\n|SBIND w. Recurrent Eq. A.1|0.8545±0.0027|0.4680±0.0182|\n\n**References**:\n\nRabut et al., Functional ultrasound imaging of human brain activity through transparent. Sci Transl Med. 2024.\n\nAlso see Lines 502, 462.", "claims_and_evidence": "The authors claim that SBIND outperforms existing methods, and this is well demonstrated by Tables 1-3, where the authors compared SBIND not just to other methods across multiple datasets but also to SBIND that has different architectures. This analysis allows identifying which components of the model contribute to good performance. The authors show that using both the local and long-range spatial dependencies in the images for the behavioral and neural activity prediction is important for performance. This is demonstrated in e.g., Figure 4 and Appendix A.4.", "methods_and_evaluation_criteria": "The SBIND method involves training of two different ConvRNNs in two stages. In the first phase, a ConvRNN with a self-attention mechanism is trained so that it is optimized to perform behavior prediction on the next timestep along with neural activity prediction. This allows learning a low-dimensional representation that is behaviorally relevant. In the second phase, another ConvRNN with a similar architecture tries to predict only neural activity, but from both the latent representation it learns and the latent representation from the first ConvRNN. This allows the second ConvRNN to learn a low-dimensional representation that is behaviorally irrelevant. This disentangling of representations is shown to be important in SBIND’s performance (Table 1) and makes sense for this application of decoding behavior from neural activity.\n\nInstead of simply using MSE, the authors use the gradient difference loss (GDL) with L1 and L2 functions, which, the authors note, have been shown to preserve local image structures (Mathieu et al., 2016). This makes sense for this application.", "theoretical_claims": "There were no theoretical claims in this paper.", "experimental_designs_or_analyses": "The experiments are sound. The authors have done rigorous ablation studies in Tables 1, A.1 and A.2 for calcium images and Tables A.5 and A.6 for ultrasound. The authors also report the hyperparameters used in SBIND in Table A.1.", "relation_to_broader_scientific_literature": "In contrast to previous papers that involve preprocessing steps (e.g., PCA) and pre-defined ROI (e.g., LocaNMF), SBIND takes the calcium imaging and ultrasound imaging data directly to perform neural activity and behavior prediction. As far as I know, methods that can be applied to fUSI are relatively sparse in neuroscience, and because SBIND is a general model that can be applied to both calcium images and ultrasound, this method may have wide applications.", "essential_references_not_discussed": "I couldn’t think of essential references not cited in this paper.", "other_strengths_and_weaknesses": "A major strength of this work is that it is widely applicable to both calcium imaging and functional ultrasound data. It uses convolutional RNN and attention mechanism to take into account local and global spatial information when doing the prediction of the next image and behavior, which has not been done previously. The clarity of the paper can be improved.", "other_comments_or_suggestions": "- Line 87 space typo. \n- What do H and W mean in Equation (1)? Please define all symbols used in equations.\n- Also, for consistency maybe it could say X_{k+1} given {Y_1, …, Y_k}. For inference, couldn’t you have used all Y_{1:K}?\n- The anonymous link to the repository didn’t seem to work for me. Is this an error on my end?", "questions_for_authors": "1. How is the performance affected if the model is trained not in two stages but all at once? Is it possible to do, and still achieve disentangling of latents? Will this potentially improve training time and also performance?\n\n2. Can the method be used to perform behavioral decoding not just for the next timestep, but also for much later future timesteps? How fast is the inference step? I think mentioning these in the text might be helpful in the context of being useful for BCI.\n\n3. In Equation (2), why is it that X and Y go into different networks and sumed? Could you have made the model so that X and Y both go into f_A?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hKrwDvqNWB", "reviewer": "Reviewer_gpPL", "summary": "This work proposed a novel learning framework, SBIND, to model spatiotemporal neural imaging. SBIND is able to model the spatiotemporal neural dynamics of neural activity, at the same time SBIND can disentangle the behavioral relevant neural dynamics. The authors experimented with both calcium and ultrasound imaging datasets. Further model comparison with baselines demonstrates the proposed method is able to decode the behavior data and recover the neural activity from the latent space with reasonable accuracy. Ablation studies were also conducted to prove the effectiveness of the proposed method.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### [1]: motivation for ConvRNNs\nWe thank the reviewer for this insightful question. There are two reasons we need to pass the ConvRNN1 states to ConvRNN2. First, ConvRNN2 aims to capture **residual neural dynamics** not explained by behaviorally relevant states, $X_k^{(1)}$. To ensure $X_k^{(2)}$ learns dynamics not included in $X_k^{(1)}$, we pass $X_k^{(1)}$ to ConvRNN2. Thus, this connection functions similarly to a **residual connection** which avoids re-learning the dynamics of ConvRNN1. Second, Sani et al. 2021 have proven that for linear dynamical systems, to get a general disentangled linear state-space model, one needs a link from $X_{k+1}^{(1)}$ to inform the recursion of $X_k^{(2)}$ (i.e. how $X_{k+1}^{(2)}$ is derived from $X_k^{(2)}$). Although SBIND uses nonlinear ConvRNNs, this linear theory provides another motivation for passing the behaviorally relevant states from ConvRNN1 to ConvRNN2.\n\n### [2]: Hyperparameter (HP) Search\nPrimarily, we swept the latent dimensions $n_x$ and patch size for all datasets, as these were found to be the most critical HPs for performance. Other major HPs (e.g., learning rate, sequence length, convolutional layers were fixed based on the best validation performance on the WFCI datasets. For the fUSI dataset these HPs were picked based on best behavior prediction in the validation set using one representative session and then fixed at these values across all sessions.\n\n### [3]: Applying to Electrophysiological (EP) Data\nWe thank the reviewer for the insightful suggestion to assess SBIND's robustness on EP data. While SBIND is primarily designed for the unique challenges of neural imaging, we agree this is a valuable comparison. We applied SBIND to an **EP dataset** (O'Doherty et al., 2020) containing smoothed spike counts from 42 channels and 4D behavioral kinematics (horizontal and vertical velocity and position) from two sessions of NHP reaching movements. We did not have access to the spatial coordinates of the electrodes, so we could not form an image based on true coordinates. Instead, for comparison purposes, we formed an **arbitrary** 6x7 2D pseudo-image from the 42 channels.\n\nDespite this the pseudo-image input lacking true spatial structure, SBIND performed comparably to DPAD (**new Table B.4**), suggesting the robustness in SBIND's dynamical modeling and disentanglement approach. Finally, as the reviewer noted, we agree SBIND holds potential for other imaging modalities or modalities that naturally possess a grid structure, such as EEG recordings.\n\n**Table B.4: Comparison with DPAD on EP Recordings**\n|Model|Neur. R2|Beh. R2|\n|-|-|-|\n|DPAD|0.8321±0.0089|0.5053±0.0201|\n|SBIND|0.8247±0.0079|0.4972±0.0212|\n\n---\n### [4]: Comments & Suggestions\nWe thank the reviewer for the helpful suggestions to improve clarity:\n\n* **ConvRNN naming**: SBIND employs an **attention-augmented ConvRNN** architecture. While the self-attention (SA) module captures *spatial* dependencies within each time step, the architecture remains fundamentally *recurrent* because the state equation ensures that $X_{k+1}$ depends on $X_{k}$. Therefore, we used \"ConvRNN\" to describe the overall model class, with the understanding that it incorporates SA. We will clarify this in the manuscript and ensure \"ConvRNN1\" and \"ConvRNN2\" are used consistently.\n\n* **$n_y, n_x, n_z$ definitions**: Please refer to Reviewer z1x2 response [1].\n\n### [5]: Dataset Size\nWFCI 1 and WFCI 2 contain 49k, and 39k frames. While the fUSI dataset has ~5k frames per session, the availability of 13 sessions provides enough data overall for evaluation. To mitigate overfitting, we employed standard techniques including L2 weight decay, dropouts, and early stopping. Example loss curves are provided via [this anonymous link](https://anonymous.4open.science/r/sbind-88FF/loss.png), showing that overfitting was not an issue.\n\n### [6]: Main Contributing Components\nAblation studies (Sec 4.2; Tables 1, A.1, A.2; Fig 4) show SBIND's performance benefits stem from its ConvRNN architecture with integrated SA and use of GDL loss for more precise neural dynamical modeling. Also, learning behaviorally relevant dynamics directly from raw images leads to better behavior decoding.\n\n### [7]: $f_A$ function\nThe SA operates purely in the *spatial* domain within a single time step, k. At each time, the latent image $X_{k}$ is divided into spatial patches, and the multi-head SA computes the relationships between these patches *at that specific time* (not across different time steps). The **temporal dependencies** are handled by the outer *recurrent structure* of the ConvRNN, where $X_{k+1}$ is computed based on $X_{k}$. The effect of patch size (which influences the *spatial scope* of attention) was investigated in our ablation study (Figure A.6), showing that larger patch sizes result in better self-prediction. Also, the sequence length HP was chosen to balance sufficient temporal context with the computational limits.\n\n**References:** Line 495", "claims_and_evidence": "The claims are reasonable.", "methods_and_evaluation_criteria": "The motivation behind taking ConvRNN1 outputs as inputs to ConvRNN2 is unclear.", "theoretical_claims": "No theoretical claims were made in the submission.", "experimental_designs_or_analyses": "The experimental designs are reasonable. How the hyperparameter search is performed?", "relation_to_broader_scientific_literature": "Previous literature learns the behavioral relevant neural dynamics and behavioral-irrelevant neural dynamics in two stages, while SBIND can learn them simultaneously in one stage. Also the", "essential_references_not_discussed": "The submission includes sufficient references as far as I know.", "other_strengths_and_weaknesses": "The authors proposed a novel method to learn behavioral-relevant neural representations and neural dynamics at the same time. The method can be applied to a general neural imaging dataset. The performance improvement is significant compared to previous methods. Besides, the model architecture designed simultaneously learns the behavioral relevant dynamics and the general dynamics which can help with understanding of neural functions. \n\n\nIt will also be good to see how the proposed method can be applied to electrode-based neural recordings and compare the results with the baseline methods. I believe technically, the approach can be applied to various data formats.", "other_comments_or_suggestions": "On lines 88-88, the name, CovRNN, is confusing: \"utilized convolutional recurrent neural networks (CovRNN),\" while the model temporal relationship is learned with attention instead of RNN. \n\nIn section 2.2, when mentioning ConvRNN1 it is better to use ConvRNN1 or ConvRNN2 to avoid confusion.", "questions_for_authors": "In the method section, what is $n_y$, $n_x$, $n_z$ corresponding to?\n\nIs that necessary to add ConvRNN1 outputs to ConvRNN2, I think there will be redundant information. \n\nHow large the current dataset is? It seems the number of available trials is limited. Did the authors see any overfitting issues?\n\nWhat do the authors think are the main components that contribute most to performance improvement?\n\nFor the $f_A$ model authors mentioned the function is for learning long temporal relationships, how authors concatenate image patches with the temporal steps. How the number of patches and the length of temporal window will affect the results.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The draft is about a new model of spatiotemporal relations in multivariate data for the specific domain of application of neural activity.
The scores were mostly positive but with two 3 a bit on the border, and I agree that marginal acceptance is the best recommendation here.
As it is usually the case with borderline papers, the balance of merits and weaknesses is subtle. Among the merits are the goodness of some results (but I'll never get tired of repeating the same major weakness in ICML; virtually no one supports their claims with proper inferential statistics and this draft is as flawed in this regard as many others) -this is especially critical in this paper for which the main target seems to be simply increasing accuracy at all costs-, the reasonable experimentation and the potential wide applicability. On the weaknesses, reviewer UYoU and myself coincide on the difficulty to interpret the model which in biomedical applications is often regarded as an important aspect. Also, other criticisms have been expressed such as the lack of clarity or the fact that there is a lot of previous work in the field that is oversighted here suggesting that the topic may be a bit saturated now. Indeed, the departing premise is only a half-truth; while certainly some models miss or discard spatiotemporal information but there is a wealthiness of other approaches that do consider such relations. And it does not help either that there is no clear theoretical contribution to make the draft stand out loud and clear. So in summary, taking a resultist stand the paper deserves publication but the balance could have gone the other way around easily...
Decision: Accept (poster)