Exploring Behavior-Relevant and Disentangled Neural Dynamics with Generative Diffusion Models
Authors: Yule Wang
Chengrui Li
Weihan Li
Anqi Wu
Venue: NeurIPS 2024 poster (2024)
Keywords: Neural Latent Discovery
Neural Behavior Analysis
Diffusion Models
Neuroscience
Primary area: neuroscience_and_cognitive_science
TL;DR: We propose BeNeDiff, a method for revealing disentangled neural dynamics associated with behaviors via generative diffusion models.

Abstract: Understanding the neural basis of behavior is a fundamental goal in neuroscience. Current research in large-scale neuro-behavioral data analysis often relies on decoding models, which quantify behavioral information in neural data but lack details on behavior encoding. This raises an intriguing scientific question: "how can we enable in-depth exploration of neural representations in behavioral tasks, revealing interpretable neural dynamics associated with behaviors". However, addressing this issue is challenging due to the varied behavioral encoding across different brain regions and mixed selectivity at the population level. To tackle this limitation, our approach, named ("BeNeDiff"), first identifies a fine-grained and disentangled neural subspace using a behavior-informed latent variable model. It then employs state-of-the-art generative diffusion models to synthesize behavior videos that interpret the neural dynamics of each latent factor. We validate the method on multi-session datasets containing widefield calcium imaging recordings across the dorsal cortex. Through guiding the diffusion model to activate individual latent factors, we verify that the neural dynamics of latent factors in the disentangled neural subspace provide interpretable quantifications of the behaviors of interest. At the same time, the neural subspace in BeNeDiff demonstrates high disentanglement and neural reconstruction quality.

Exploring Behavior-Relevant and Disentangled Neural Dynamics with Generative Diffusion Models
Yule Wang Georgia Institute of Technology Atlanta, GA, 30332 USA
Chengrui Li
Georgia Institute of Technology Atlanta, GA, 30332 USA cnli mmlichnngrui@gatech edu
yulewang@gatech.o edu
Weihan Li Georgia Institute of Technology Atlanta, GA, 30332 USA
Anqi Wu Georgia Institute Technology Atlanta, GA, 30332 USA anqivu@gatech. edu
weihanli@gatech.c edu
Abstract
Understanding the neural basis of behavior is a fundamental goal in neuroscience. Current research in large-scale neuro-hehavioral data analysis often relies decoding models, which quantify behavioral information in neural data but lack details on behavior encoding. This raises an intriguing scientific question: "how can we enable in-depth exploration of neural representations in behavioral tasks, revealing interpretable neural dynamics associated with behaviors". However, addressing this issue is challenging due to the varied behavioral encoding across different brain regions and mixed selectivity the population level. To tackle this limitation, our approach, named "BeNeDiff", first identifies fine-grained and disentangled neural subspace using a behavior-informed latent variable model. It then employs state-of-the-an generative diffusion models to synthesize behavior videos that interpret the neural dynamics of each latent factor. We validate the method multi-session datasets containing wide-field calcium imaging recordings across multiple brain regions of the dorsal cortex. By guiding the diffusion model to activate individual latent factors, we verify that the neural dynamics of latent factors the
in the disentangled neural subspace provide interpretable quantifications behaviors of interest across multiple brain regions. Mcanwhile, the neural subspace in BeNeDiff demonstrates high disentanglement and neural reconstruction quality. Our codes are available at https: //g thub. com Hwt./BBAAMMLGGG////M GT/BeNeeDBBe..B
1 Introduction
Understanding and elucidating the complex interrelationships between behavioral data and neural population activity long-standing goal systems neuroscience [Batty et al., 2019; Gomez- Marin et al., 2014; Krakauer et al., 2017; Berman, 2018]. Exploring the neural basis of behavior not only deepens our basic knowledge of brain functions but also establishes a foundation for developing improved treatments for psychiatric and neurological conditions [Vieira et al., 2017: Ibáñez et al., 2018). Significant progress has been achieved in developing computational toolkits for neuro- betrr--bhavirrall decoding by using behavior video data [Whiteway al., 2021; 1; Batty al., 2019; Musall et al., 2019; Stringer et al., 2019]. These methods perform region-based behavior decoding to map neural activity across multiple brain regions of the dorsal cortex the behaviors from the videos. However, these methods only quantify how much behavioral information is encoded neural populations, but do not reveal the details of such encoding. There has been markedly less focus,
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

(A)
MOs
(B)
55p
MOs
VI5
60
60
60
SSp
CVR*%
CVR
30-
30
30
20-
Levers
Spouts
Paw (x) Juw
Levers
Spouts
Paw-Ux)
Jan
Levers
Spouts
Pau-(xi) Juw
VIS
Figure 1: Empirical study across multiple brain regions of dorsal cortex neural recordings of mouse in a visual crcision-makingg task. (A) The Brain Atlas map [Lein et al., 2007]. (B) Neural signals in various brain regions (SSp, MOs, and VIS) exhibit mixed selectivity in behavior of interest decoding. "Levers", "Spouts". "Paw-(x)", and "Jaw" are four behaviors of interest. cvR? 2 is short for cross-validation coefficient of determination. The higher, the better.
with cordex-wide signals, on enabling in-depth exploration of neural activities during behavioral tasks, where specific neural patterns reveal dynamic evolutions corresponding to distinct behaviors of interest. However, empirically addressing this scientific question challenging due to neural population activities in various brain regions exhibiting mixed selectivity [Sani et al., 2021; Hasnain et al., 2023], responding robustly to multiple behaviors of interest. We further verify this finding through empirical study across three brain regions on the dorsal cortex of head-fixce mice [Musall et al., 2019] (shown in Figure 1).
To tackle this issue, we propose method - Exploring Behavior-Relevane and Interpretable Neural Dynamics with Generative Diffusion Models ("BeNeDDiff".. We first employ neural latent variable model (LVM) to identify orthogonal and disentangled neural latent subspace. This is achieved through a semi-supervised variational autoencoder, which integrates behavioral labels to rotate the subspace. Subsequently, our main idea is to explore the neural dynamics of each latent factor in the learned subspace for distinct quantifications of the behaviors of interest. However, such a workflow is non-trivial since naive latent manipulation produces samples not conform to the original distribution, leading to mapped video-based behavioral data that loses its validity (we further detail this part in Method Section 3.2.1).
Notably, we aim to investigate the rehaaioral-speciffcit of neural latent factors in a generative fashion. We leverage -fate-of-thhear video diffusion models (VDMs) to generate behavior videos predicted to activate individual latent factors along the single-trial trajectory. Technically, the VDMs are capable of capturing the overall temporal dynamics and synthesizing behavior videos in classifier-guidde manner [Dhariwal and Nichol, 2021]. Inspired by Noise-Contrastive Estimation [Gutmann and Hyvärinen, 2010], the guidance objective is formulated to amplify the variance of the selected latent factor along its neural trajectory while suppressing the variance of the neural trajectories of the other latent factors.
We conduct experiments to verify the efficacy of BeNeDiff on a widefield calcium imaging dataset, where a hexd-fixed mouse performs a visual decision decision-making making task across multiple sessions [Musall et al., 2018: 2019]. The neural subspace in BeNeDiff exhibits high levels of disentanglement and neural reconstruction quality, as evidenced by multiple quantitative metrics. By guiding the diffusion model to activate individual latent factors, we verify that the neural dynamics within the disentangled subspace provide interpretable and selective quantifications of the behaviors of interest (c.g., paw movements) across multiple brain regions. These results advance our understanding of neuro-bchavioral relationships through the identification of fine- ane-ggrined behavioral subspaces and the uncovering of disentangled neural dynamics.
To highlight our major contributions: (1) This is the first work to explore wide-field imaging across multiple brain regions of the dorsal cortex of head-fixed mice during a decision-making makinn task using neural subspace analysis, rather than merely performing -euro-be decoding. We uncover disentangled neural representations for various behaviors.
(2) To visualize the behavior dynamics within a disentangled neural subspace of each brain region, we develop a novel VDM-based interpretation tool that faithfully reflects behavior-related neural dynamics. It essential to interpret the meaning of each neural latent dimension well the behavior dynamics it encodes.

---

Activate (d)
Guidance Gradient Vy log ((c Y)
Linear
Neural
Diffusion Denoiser
Encoder
Behavior Video
Y
Disentangled Neural Subspace
Figure 2: Schematic diagram neural dynamics interpretation with BeNeDiff. We first employ neural LVM identify a disentangled neural latent subspace (the left part). Then, we train linear neural encoder to map behavior video frames to neural trajectories. We use video diffusion models (VDMs) to generate behavior videos guided by the neural encoder, based on the objective of activating the variance of individual latent factors along the single-trial trajectory. This approach provides interpretable quantifications of neural dynamics in relation to the behaviors of interest.
Preliminaries
2.1 Problem Formulation
We first provide the notations of the paired neuro-behavioral observations. The single-trial neural population activities are denoted as X = [x1,..., x1] RR where L is the trial length (i.e, number of time bins). N is the number of observed neural signals. The behavioral video frames denoted as Y ......, Y X where H, W are the height and width of the compressed behavior video frames. We extract behavior labels = [ul..... uz] E L from the video frames using behavior LVM [Whiteway al., 2021]. B is the number of the behavior.
We build a variational autoencoder (VAE) [Kingma and Welling, 2013] infer the neural latent trajectories Z [z.....,ze .2]] E I.X,, which are also informed by behavioral labels. is the latent factor number. We denote its probabilistic encoder and decoder qo (Z X, U) and P& (X, U Z), respectively. We denote the neural trajectory of single latent factor as ( Z, where d E f1.2..... D}. Our primary goal to investigate the neural dynamics of (d) through selectivity quantifications of its corresponding single-trie trial behavioral video data Y.
2.2 Generative Video Diffusson Models
Diffusion models have also achieved impressive results in video synthesis over recent years [Ho et al., 2022b;a; Harvey al., 2022]. VDMs process fixed number of frames and factorize them over the temporal dimension via deep neural network [Ho al., 2022a; Harvey et al., 2022]. The training of VDMs starts from a forward process with variance schedule {BI...., Br the noised sample Y, follows the Gaussian conditional: q (Y, Yo) N (Y:: Vo, Yo. (1 - ā,) I), where
0z Br and a₁ II Og- denoising model ēo(-) is trained reverse the forward process
using a weighted mean squared error loss:
LVDM(0) = Ee-N(o, E (w(At)) (le)))le,,, -
(Y,, ().)))))
in which time-steps are uniformly sampled and w(Ar) the weighting ratio. This loss function can be justified as optimizing a weighted variational lower bound on the data log-likelihodd In the sampling phase, we start from Y, No, Ixxxxw) and perform step-by-step denoising,
1
1
Y
Vai VT ai čo (Y.,t) ocCL:
Y,
(2)
where random noise perturbation e [10,Iw for timesteps t 1, e, 0 when t 1. and of
1-
1-as Br.
3 Methods
Then, we train linear neural encoder from the behavior video frames the neural trajectories. We leverage video diffusion models (VDMs) generate behavior videos guided by the neural encoder,

---

based on the objective of activating the variance of individual latent factors along the single-trial trajectory, providing interpretable quantifications of neural dynamics with respect to the behaviors of interest.
In this section, first detail the process by which BeNeDiff infers disentangled neural latent subspace. We then discuss the approach that BeNeDiff interprets the selectivity of neural dynamics of latent factors using the video diffusion model.
3.1 Behavior-Relevani and Disentangled Neural Latent Subspace Learning
Drawing inspiration from recent progress in the field neural LVMs [Kingma et al., 2014; Klys et al., 2018], employ VAE learn disentangled neural subspace. The neural data X usually contains a good amount of information other than behavior [Hasnain et al., 2023], thus an unsupervised disentangled VAE won't t effectively discover disentangled subspace with behavior only. Therefore, we introduce behavior labels U to inform the VAE to learn a latent subspace that better accounts for the variance related to behavior. We note that this technique widely adopted in previous neuro-behavioral analysis works [Wang et al., 2024; Schneider et al., 2023; Gondur et al., 2023]. Notably, enforce the disentanglement in the latent subspace, we incorporate a lotal-correlation (TC) penalty term [Chen et al., 2018] to enforce the VAE to find statistically independent latent factors in the supervipd setting. The VAE optimizes the following evidence lower bound (ELBO) [MacKay, 2003]:
log Pa (X, U) > g (Z) log Z)
+ log P ( - DKL I (Z x, UU)
Neural Reconstraction
Behavior Info.
Regularization
- B DKL (z
X, X.U (z)
X.U)
Total Correlation
in which (4)) denotes the neural trajectory of the d-th latent factor, and the value of 33 controls the strength of disentanglement penalty. However, the factorial density in this term is untractable in practice, so here we use the minibatch-weighted sampling estimator [Chen et al., 2018] to approximate the TC penalty term. We note that the variational autoencoder employs sequential architecture [Fabius and Van Amersfoort, 2014] to capture the overall temporal dynamics along the single-trial trajectory {x plugging bi-directional recurrent units [Schuster and Paliwal, 1997] into both the probabilistic encoder uo (-) and decoder (-)..
3.2 Diffusion Guided Video Generation for Neural Dynamics Interpretation
3.2.1
Downside of Latent Manipulation for Interpreting Neural Dynamics
As for testifying the neural dynamics of a single disentangled latent factor (d)) on the behavioral videos Y, straightforward attempt is train neural-net model approximate the posterior distribution p(Y Z) and then perform latent manipulation on each single latent factor. There are two major techniques to perform latent manipulation. The first is naïve manipulation. This method manipulates single subspace (dd) while keeping the non-target latent factors fixed at arbitrary values. then observes how the manipulation affects Y. The induced changes in the videos reveal the dynamics encoded by (d)), The second method uses classifier-free guidance [Ho and Salimans, 2022], where we allow the activated latent factor z (d) evolve while fixing non-target latent factors to arbitrary values. However, setting arbitrary values without knowing the true distributions of non-target subspaces can lead to unnatural distortions in generated videos, complicating the visualization and interpretation of genuine animal behavioral dynamics.
3.2.2 Rehavioral Video Generation for Neural Dynamics Interpretation
So here we employ the video diffusion models (VDMs) to explore factor-wise neural dynamics through a generative manner, which is capable of maintaining temporal consistency and behavioral dynamies across frames. The primary goal to perform behavior data generation conditioned on activating single latent factor along the neural trajectory. Thus the resulting behavior video can

---

provide interpretable quantifications of the neural dynamics of factor (d) Specifically, we implement classifier guidance [Kawar et al., 2022]. By Bayes rule, we obtain the following posterior density and gradient [Mardani et al., 2023]:
P (Y, c) po P) (c) Y.) p(c),
Yy, Po. (Y, c) = Vy, log Po (Y:) + Vy, log (c Y,).
(5)
Unconditional Gradient
Guidance Gradiont
in which 0, A are the parameter sets for the classifier and the denoising model, respectively. Note that t indicates the time step in the diffusion model. Our goal is to estimate the two terms on the RHS of Eq. (5) to perform conditional denoising in each step. We first approximate the density of the behavior video data through a standard denoising model ég (Y,,)) according to Eq. (I) since the first unconditional gradient term can be derived through it:
Vr, log po (Y.)
dz ēg (Y,,)).
(6)
VI
For the calculation of the guidance term, first train linear neural encoder the classifier from the behavior video data to the neural latent variables of the learned -upi-super VAE subspace. We denote the estimated neural latent trajectories as 2, [2,,1,..... 22, E RLDD, in which:
2 W (Yc(Y,,))) + q q (0,
where S ( S L, 2,,1 RD denotes the estimated value of latent factors time bin / and diffusion step L. The parameter set of the linear encoder X {W, Q). W E DxM is the linear transformation matrix, Q E DxD is the covariance matrix and vec(-) represents vectorizing the two-dimensional video frame into column vector. After training the encoder, we fix all parameters and use it to construct the density PA (c Y.).
The class labels {c(*), _,.,),...,, c(D) J. in which c(d) is one-hot column vector with a one at the d-th dimension and zeros elsewhere. Drawing inspiration from Noise-Contrastive Estimation [Gutmann and Hyvärinen, 2010], our guidance objective of the activation of latent factor d-th is formulated as maximizing the variance of the trajectory ((4) while minimizing the variance of the other latent factor trajectories in Z:
exp 1 2, cla)) /T) + = exp (sx (2, ((,, ( 
exp (sx 2, (l(0))))
logpx (cl()) x.) log
(8)
where 12 (2,c(d) -Var Z ((d) calculates the variance of the selected latent factor and calculates the variance of an- other sampled latent factor's 's trajectory. Var i E I is a row vector where each element the variance of every latent factor along the neural trajectory. is the temperature parameter. K is hyperparameter controlling the number of sampled negative samples at each iteration.
fā (2 c()) Var 2 cli), j (1,,..
The gradient Vv, log px (c(d) Y.) is computed using automatic differentiation [Paszke al., 2017]. Algorithm describes the guided behavior video generation steps of our proposed framework BeNeDiff.
Algorithm 1: Generative Video Diffusion Model for Neural Dynamics Interpretation
Input: Condition label ((d) for interpreting the neural dynamics of the d-th latent factor Initiate Yr (O, IxxHwW)
for t = T 1 do
E (Y,,t) t) = Eo (Y,,t) - -i Vv, logpA (cld) Y.)
G ~ (O, I) ift 1, else E, = 0
Y Var Y,
I-a É (Y,;t)) + o,C,
VI-a.
end
Output: Generated behavior video Yo
5

---

4 Related Works
Disentangled Latent Subspace Learning. Neural LVMs is a fundamental framework which posits that single-trial neural population activities rely on low-dimensional "neural manifolds" [Gallego et al., 2018; Mitchell-Hegge et al., 2023; Li et al., 2023a; Hurwitz et al., 2021] and their extracted latent variables are successful in describing single-trial neural actívities [Li et al., 2024c; 2022; 2024d: Liu et al., 2021; 2022: Li et al., 2024a]. Learning disentangled latent variables that uncover statistically independent latent factors [Chen al., 2018] can provide enhanced robustness, interpretability, and controllability. Typically, this type of work involves adding auxiliary regularizer terms to enhance orthogonality [Mathieu et al., 2019] and reduce the total correlation [Chen al., 2018] among the latent factors. In neuroscience, there have been studies focusing on the disentanglement of latent subspace within rich behavioral data [Whiteway et al., 2021; Shi et al., 2021]. However, our work is the first to discover interpretable and disentangled latent subspaces of wide-field imaging data.
Generative Diffusion Models. In recent years, diffusion models have achieved great success in generating high-quality images due to their expressivity and flexibility [Ho et al., 2020; Song et al., 2020a;b; Vahdat et al., 2021]. Moreover, for the more challenging task of video generation, there have been several explorations using diffusion models to address it. From a modeling perspective, the key concern how to maintain temporal dynamics and consistency across frames. Most existing works [Ho et al., 2022b;a] extend the 2D U-Net architecture [Ronneberger et al., 2015; Song et al., 2024] to a 3D framework by considering the time axis. this 3D framework, convolutions are performed in both spatial and temporal dimensions. Additionally, recent studies in neural computation have leveraged generative diffusion models to tackle -emaii--pecific tasks, such as neural distribution alignment (Wang et al., 2024] and decoding visual stimulus from brain activities [Sun et al., 2024b;a]. Our work the first to employ generative diffusion models for analyzing neuro-behavioral data relationships.
Experimental Results
5.1 Dataset Description
Side View
Bottom View
A head-fixed mouse performed visual decision-making task while neural activity across the dorsal cortex was optically recorded using widefield calcium imaging [Musall et al., 2019; Churchland et al., 2019]. The mouse's ' behavior included both instructed and uninstructed movements. For behavioral data acquisition, two cameras captured video frames from both a side view and bottom view. The dataset comprises 1126 trials conducted over two sessions, with 189 frames per trial at a frame rate of 30 Hz. Concurrently, neural activity was recorded at the same frame rate. The grayscale video frames were down- sampled to 128x 128 pixels. We extract 275 dimensions of neural signals from the high-dimensional widefield imaging data using the open-sourced LocaNMF decomposition toolkit [Saxena et al., 2020]. As shown in Figure 3, the behaviors of interest include the moving lick spouts, moving levers, the single visible right paw trajectories, and the movement of the jaw and chest, all tracked using DeepLabCut [Mathis et al., 2018].
Jaw
Spout
R Paw (X)
Chest
Lever
R Paw (Y)
Trial
Spout
Trial
Start
End
0 (sec)
2.8
57
63
Figure 3: Widefield Calcium
Imaging Dataset. The head- fixed mouse is performing a vi- sual decision-making task, with the behaviors of interest and the trial structure illustrated.
5.2 Disentangled Neural Latent Subspace Investigation
We note that we train a unique neural LVM for each individual brain region (single-region), and we evaluate both the behavior decoding and neural reconstruction performance of each brain region- specific neural latent trajectories.
Single Latent Factor Behavior Decoding, In order to verify the disentanglement of the learned neural subspace BeNeDiff, we evaluate the behavior label decoding performance of each individual latent factor. Specifically, we train a unique linear regressor for each latent factor from the VAE and plot the decoding accuracy as the R-squared value (R2%). The results of VIS-Right region (the right visual region) are shown in Figure 4. The main observation is that each latent factor is specific to

---

Latent Factor-1
Latent Factor-2
Latent Factor-3
0.6-
0.4
0.4
0.2
0.2
0.2-
0.0
Levers
Spout
Paw (e)
Paw (V)
Chest
jaw
0.0
Levers
Spout
Paw ix)
Puw S
Chest
jaw
0.0
Levers
Spout
Paw (x)
Paw (S)
Chest
jaw
Latent Factor-4
Latent Factor-5
Latent Factor-6
1.0
1.0
0.8
0.0
0.6
0.6
0.6
0.4
0.4
0.4
0.2
0.2
0.2
0.0
0.0
0.0
Levers
Spouit
Paw Dd
Faw (v)
Chest
jaw
Levers
Spoit
Paw bi)
Paw (t)
Chest
Jaw
Levers
Spout
Paw
Faw (s)
Chrest
jaw
Figure 4: Behavior decoding results of the disentangled neural latent variables of the ViS-Right region. We observe that the decoding capability of each latent factor is specified to the corresponding behavior of interest, exhibiting a single-mode shape. In contrast, the original neural signals exhibit mixed selectivity to the behaviors, shown in Figure I (B). Each experiment condition is repeated times, with the mean represented by the bar plot and the standard deviations shown error bars.
unique behavior of interest, confirming the orthogonality and clear disentanglement of the inferred latent trajectories from quantitative perspective.
Neural Observation Signals Reconstruction. To prevent the VAE from overfitting the behavioral labels, BeNeDiff also aims to maintain a low reconstruction error for neural activity. Table presents the quantitative reconstruction results compared to baseline methods, including Semi-Supervised Learning (SSL) [Kingma et al., 2014], CEBRA [Schneider et al., 2023], and Pi-VAE [Zhou and Wei, 2020]. The table records the R-squared values (R², in %) and RMSE for each method, Additionally, we plot the ground-truth neural signals and the reconstructed signals of several methods in a single well-preserved given the
trial in Figure 5. The main observation that the neural reconstruction behavioral priors. One possible explanation is that the behavioral labels rotate the latent subspace while preserving the necessary information for reconstructing the neural data. The neural signals can be hardly recovered from the behavior labels only. It indicates that the behavior-informed latent does encode significant neural information that is not contained in the behavior labels. Furthermore, we evaluate the disentanglement quality of the latent subspace using the widely-adopter MIG (Mutual Information Gap) metric [Chen et al., 2018), also listed Table We observe that the learned latent subspace of BeNeDiff significantly enhances disentanglement compared to the vanilla VAE.
Table Baseline Comparison of the neural LVM on two brain regions of Session-l. The boldface denotes the highest score of the MIG metric. Each experiment condition is repeated with 5 runs, and their mean and standard deviations are listed.
Region
Metrics
SSL
CEBRA
pi- VAE
Ours
(%)
81.10 (=:0226
79.60 (+0.22)
74.37 (#0.24)
75.41 (+0.24)
VIS-Left
RMSE
32.77 (=0.17)
33.07
(+0.18)
36.74 (+0.22)
35.50
(7)
MIG(%)
37.50 (+0.20)
40.12
(+0.24)
43.98 (+0.29)
55.87
(=0.26)
R (7)
76.65 (#0.30)
72.63
(10228)
70.73 (#0.23)
69.59
(+0.22)
MOs-Left
RMSE J
30.64 (+0.21)
32.14 (+0.17))
35.69
(+00.19)
36.91
(+0.118)
MIG(%)
36.89 (+0.23)
37.94
(+0.233)
42.20 (+0028)
58.56
(+0.29)

---

Trial 321; Dim
Trial 321; Dim
Nomabed GCANP TeaNd fuoreacaao
1.0
Cooourdh Trutt Selfis specsiddd BuNkDD!
Ground- Irch
0.5
BohcOI
0.0
-0.5
-0.5
-1.0
-1.0
Time-Bins
Time Bins
Normalaed fuarescence
Trial 33; Dim 17
Nornalzud OCaNP fuorescano
1.0
Trial 33; Dim
Ground Trush
Ground Bruh
Sell-e DeN
Sekeuperdoos
0.5
0.5
Bela
-0.5
-0.5
100
150
Time-Biat Bins
Time-Bins
Figure 5: Neural signal reconstruction performance evaluation of the VIS-Right region. We observe that the neural reconstruction quality from the latent subspace of BeNeDiff is maintained given the behavioral labels. SupfrSSupervseed" denotes the VAE w/o behavior labels.
5.3 Neural Dynamics Exploration of Disentangled Latent Factors
From the quantitative experiments in the previous subsection, we obtained information about the decoding and disentanglement quality within the subspace. However, these metrics have limitations in interpreting single-trial neural dynamics, especially the complex temporal structures over time. Here, we visualize the generated videos from BeNeDiff and the baseline latent manipulation methods, demonstrating that BeNeDiff provides interpretable quantifications of the behaviors of interest.
Latent Manipulation Methods for Comparison. We compare the neural dynamics exploration
performance of BeNeDiff against the following two latent manipulation methods:
Naïve Latent Manipulation: the standard manipulation method discussed in Section 3.2.1, which approximates the posterior of behavioral videos given the neural latent trajectories p( p(Y Z), using a neural network that incorporates recurrent units and spatio-temporal convolutional layers. Classifier-free Guidance [Ho and Salimans, 2022]: a method that approximates the posterior p(Y Z) with diffusion models. It co-trains conditional and an unconditional diffusion model together, combining the resulting conditional and unconditional scores at each diffusion step. In the conditional model, the entire neural latent trajectory Z is set the condition, formulating the denoiser as ? (Y,, Z, t). For the manipulation of the latent, we keep the activated latent factor (d) to evolve while setting the values of the other latent factors to those in the first frame of the trial.
Setup. To verify the neural dynamics interpretation capability of BeNeDiff, we generate behavioral video data given the activation of each behavior of interest (generated trials with the activation of Jaw and Paw-(y) are shown in Figure 6 and Figure 9, respectively). For visualization and video analysis, we plot frames at intervals of five and compute their frame differences. The conditional module the massifier--ree guidance method is trained with auxiliary convolutional head. Compared to general video synthesis [Harvey et al., 2022; Esser et al., 2023], our behavioral video data are more focused on maintaining the temporal dynamics and consistency across video frames, thus in BeNeDiff, tailor the standard 3D U-Net architecture [Çiçek et al., 2016] from temporal self-attention layers to temporal convolutions layers [Li et al., 2023b; 2024b] to maintain local temporal consistency. While we keep the spatial self-attention layers the same. The diffusion model is trained on an Nvidia V100, using approximately 20 computer hours.
Results Analyses of the Generated Videos. shown in Figure 6, for the naive latent manipulation method, the distribution of neural signals often falls outside the original distribution after manipulation, resulting in blurred generated frames. The frame differences are entangled, and the "Jaw" latent factor affects the entire head movement of the mouse, particularly in the first four frames shown.

---

Naive Latent
Manipulation
Inter-Frame Difference
Classifier-Free Guidance
Inter-Frame
Difference
BeNeDiff
(Ours)
Inter-Frame
Difference
Figure 6: Generated Single-trial Behavioral Videos with Latent Factor Guidance from the side view. Compared to baseline methods, we observe that the neural dynamics of latent factor in the results of BeNeDiff show specificity to the "Jaw" movements.
On the other hand, for classifier-free guidance, the generated videos maintain coherent consistency between frames. However, it does not interpret neural dynamies well in this context, resulting in trajectory with small movements in the "Jaw". This is because the overall latent trajectory is used as the input to the model and the other latent factors are kept fixed, making it difficult to discriminate the evolution of a single factor effectively. In contrast, the results of BefeDiff show more specificity to the targeted behavior of interest. The inter-frame differences in BeNeDiff's results are clearly specified to the "Jaw" movements, and the structure of the neural dynamics is wree--preerred and consistent with ground-trut truth "Jaw" behavior trajectories. A similar pattern is evident with the other latent factors, as shown in Figures 9, 10, and 11 in the appendix.
5.4 Neural Dynamics Exploration of Disentangled Latent Factors Across Brain Regions
Besides the capability of revealing interpretable neural dynamics of each latent factor associated with behaviors, here we further investigate the neural dynamics differences across brain regions through BeNeDiff. As shown in Figure and Figure 12 in the appendix, we present the 2D neural latent trajectories of two latent factors, specifically related to "Paw-(x)" and "Paw-(y)", across six brain regions for two randomly selected trials. From the starting point of the trial, we observe that the latent trajectories corresponding to the left and right hemispheres of the VIS both show noticeable change starting earlier. Next, the SSp regions show a large shift in activity, followed by a similar change in the MOs regions. However, it is difficult to clearly visualize the specific motion encoded by each region and to distinguish how different the motions are encoded solely based on neural trajectory plots. This further highlights the need for using video diffusion model for visualization and interpretation.
In contrast, in the generated behavior video samples of BeNeDiff (as illustrated by the frame differences in Figure and Figure 13 in the appendix), where the "Paw-(x)" and "Paw-(y)" latent factors are activated, the behavioral dynamics encoded by these two latent factors are observed across different brain regions. First, paw movements are detected the VIS regions before the "Levers" come in. This early activity in VIS could reflect its role in the predictive coding of behaviors, indicating that this region may predict motor movements before they happen. Next, the SSp regions exhibit paw movements that are synchronized with the onset of the "Levers". indicating potential role

---

for SSp in processing somatosensory feedback. Subsequently, in the MOs regions, paw movements are observed following the "Levers" onset, which is consistent with MOs' role in motor execution and control, occurring slightly after SSp.
Left Hemisphere Regions |Fetched Trial Id: 143)
Right Hemisphere Regions (Fetched Trial Id: 143)
66
Y-axis
-1.3
VSL Letanta Tauddry
VS-At Letaraa
-25
SSp Laterts Wrjectery
-1.
S5p Lacerts Traectury
M3s L Latore: Tapeetory
Nos Latrons Tegectory
Stati Part
Start toomt
"Lexer VSL.
-20 Lever stvS
Stp-l
Lover
Figure 7: Learnt Neural Latent Trajectories of BeNeDiff across various brain regions. It is difficult to clearly visualize the specific motion encoded by each region and to distinguish how different the motions are encoded across brain regions.
"Levers" Coming In Point
VIS-Right
Inter-Frame
Difference
SSp-Right
Inter-Frame
Difference
MOs-Right
Inter-Frame
Difference
Figure 8: Generated video frame differences across the right hemisphere regions. The red dots in the figure indicate paw appearances.
To sum up. although neural trajectory plots provide clear temporal sequence of activations across regions, is challenging to directly visualize the specific behavioral dynamics encoded by each region and to discriminate how they differ. This limitation highlights the necessity for video diffusion model BeNeDiff, to better visualize and interpret the encoded behavioral dynamics of each neural latent factor. By synthesizing realistic behavior videos in generative fashion, BeNeDiff enables us to better understand the unique neural dynamics each brain region and their corresponding behavioral dynamics.
10

---

References
Fleanor Batty, Matthew Whiteway, Shreya Saxena, Dan Biderman, Taiga Abe, Simon Musall, Winthrop Gillis, Jeffrey Markowitz, Anne Churchland. John P Cunningham, et al. Behavenet: nonlinear embedding and bayesian neural decoding of behavioral videos. Advances in Neural Information Processing Systems, 32, 2019.
Gordon J Berman. Measuring behavior across scales. BMC biology, 16:1-112 2018.
Ricky TQ Chen, Xuechen Li, Roger B Grosse, and David K Duvenaud. Isolating sources of disentanglement in variational autoencoders. Advances neural information processing systems, 31, 2018.
Anne K Churchland, Simon Musall, Matthew T Kaufman, Ashley L. Juavinett, and Steven Gluf. Dataset from: Single-trial neural dynamics are dominated by richly varied movements. 2019.
Özgün Çiçek, Ahmed Abdulkadir, Soeren S Lienkamp. Thomas Brox, and Olaf Ronneberger. 3d u-net: learning dense volumetric segmentation from sparse annotation. In Medical Image Computing and Computer-Assioe -ssisted InterventionMMICAA 20/6: 19th International Conference, Athens, Greece, October 17-21, 2016, Proceedings, Part II 19, pages 424 432. Springer, 2016.
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans image synthesis. Advances in neural information processing systems, 34:8780-8799 -77944 2021.
Patrick Esser, Johnathan Chiu, Parmida Atighehchian, Jonathan Granskog, and Anastasis Germanidis. Structure and content-guided video synthesis with diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7346 7356, 2023.
Otto Fabius and Joost R Van Amersfoort. Variational recurrent auto-encoders. arXiv preprint arXiv:1412.6581, 2014.
Juan A Gallego, Matthew G Perich, Stephanie Naufel, Christian Ethier, Sara A Solla, and Lee Miller. Cortical population activity within a preserved neural manifold underlies multiple motor behaviors. Nature communications, 9(1):4233, 2018.
Alex Gomez Marin, Joseph Paton, Adam R Kampff, Rui M Costa, and Zachary Mainen. Big behavioral data: psychology, ethology and the foundations of neuroscience. Nature neuroscience, 17(11):1455-146) 2014.
Rabia Gondur, Usama Bin Sikandar, Evan Schaffer, Mikio Christian Aoi, and Stephen L Keeley. multi-modal gaussian process variational autoencoders for neural and behavioral data. arXiv preprint arXiv:2310.03111 2023.
Michael Gutmann and Aapo Hyvärinen. Noise-contrastive estimation: A new estimation principle for unnormalized statistical models. In Proceedings of the thirteenth international conference artifiical intelligence and statistics, pages 297-304. JMLR Workshop and Conference Proceedings, 2010.
William Harvey, Saeid Naderiparizi, Vaden Masrani, Christian Weilbach, and Frank Wood. Flexible diffusion modeling of long videos. Advances in Neural Information Processing Systems, 35: 27953-27963 2022.
Munib A Hasnain, Jaclyn E Birnbaum, Juan Luis Ugarte Nunez, Emma K Hartman, Chandramouli Chandrasekaran, and Michael N Economo. Separating cognitive and motor processes in the behaving mouse. bioRxiv, 2023.
Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12598. 2022.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6851. 2020.
Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederik P Kingma, Ben Poole, Mohammad Norouzi, David Fleet, et al. Imagen video: High definition video generation with diffusion models. arXiv preprint
2022a.
11

---

Jonathan Ho. Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J Fleet. Video diffusion models. Advances in Neural Information Processing Systems, 0488633-8666 2022b.
Cole Hurwitz, Akash Srivastava, Kai Xu, Justin Jude, Matthew Perich, Lee Miller, and Matthias Hennig. Targeted neural dynamical modeling. Advances in Neural Information Processing Systems, 34:29379- 29392, 2021.
Agustín Ibáñez, Adolfo M Gareía, Sol Esteves, Adrián Yoris, Edinson Muñoz, Lucila Reynaldo, Marcos Luis Pietto, Federico Adolfi, and Facundo Manes. Social neuroscience: undoing the schism between neurology and psychiatry. Social Neuroscience, 13(1):1-39, 2018.
Bahjat Kawar, Roy Ganz, and Michael Elad. Enhancing diffusion- based image synthesis with robust classifier guidance. arXiv preprint arXiv:2208.0866 086664 2022.
Diederik P Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint 14X1::1412.6980,, 2014.
Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv:1312.6114 2013.
arXiv preprint
Durk P Kingma, Shakir Mohamed, Danilo Jimenez Rezende, and Max Welling. Semi-supervised learning with deep generative models. Advances in neural information processing systems, 27, 2014.
Jack Klys, Jake Snell, and Richard Zemel. Learning latent subspaces in variational autoencoders. Advances in neural information processing systems, 31, 2018.
John W Krakauer, Asif A Ghazanfar, Alex Gomez-Marin, Malcolm A Maclver, and David Poeppel. Neuroscience needs behavior: correcting a reductionist bias. Neuron, 93(3):480-490, 2017.
Ed Lein, Michael J Hawrylyez, Nancy Ao, Mikael Ayres, Amy Bensinger, Amy Bernard, Andrew F Boe, Mark S Boguski, Kevin S Brockway, Emi Byrnes, et al. Genome-wide atlas of gene expression in the adult mouse brain. Nature, 445(7124):168-176, 2007.
Chengrui Li, Yule Wang, Weihan Li, and Angi Wu. Forward chii divergence based variational importance sampling. arXiv preprint arXiv:2311.02566, 2023a.
Chengrui Li, Weihan Li, Yule Wang, and Anqi Wu. A differentiable partially observable generalized linear model with forward-backwart message passing. In Forty-first International Conference on Machine Learning, 2024a.
Panfeng Li, Mohamed Abouelenien, and Rada Mihaleea. Deception detection from linguistic and physiological data streams using bimodal convolutional neural networks. arXiv preprint arXiv:2311. 10944, 2023b.
Panfeng Li, Mohamed Abouelenien, Rada Mihalcea, Zhicheng Ding, Qikai Yang, and Yiming Zhou. Deception detection from linguistic and physiological data streams using bimodal convolutional neural networks. In 2024 5th International Conference on Information Science, Parallel and Distributed Systems (ISPDS), pages 263-267. IEEE, 2024b. doi: 00.1109/SPDD62279.20044 10667569.
Weihan Li, Yu Qi, and Gang Pan. Online neural sequence detection with hierarchical dirichlet point process. Advances in Neurul Information Processing Systems, 35:6654-6665, 2022.
Weihan Li, Chengrui Li, Yule Wang, and Anqi Wu. Multi-region markovian gaussian process: An efficient method to discover directional communications across multiple brain regions. arXiv preprint S00v:2400.006888 2024c.
Weihan Li, Yule Wang, Chengrui Li, and Angi Wu. Markovian gaussian process: A universal state- space representation for stationary temporal gaussian process. arXiv preprint arXiv:2407.00399, 2024d.
12

---

Ran Liu, Mehdi Azabou, Max Dabagia, Chi-Heng Lin, Mohammad Gheshlaghi Azar, Keith Hengen, Michal Valko, and Eva Dyer. Drop, swap, and generate: self-supervised approach for generating neural activity. Advances in neural information processing systems, 34:10587-10599 2021.
Ran Liu, Mehdi Azabou, Max Dabagia, Jingyun Xiao, and Eva Dyer. Seeing the forest and the tree: Building representations of both individual and collective dynamics with transformers. Advances in neural information processing systems, 35:2377-2391, 2022.
David JC MacKay. Information theory, inference and learning algorithms. Cambridge university
press, 2003.
Morteza Mardani, Jiaming Song, Jan Kautz, and Arash Vahdat. A variational perspective on solving inverse problems with diffusion models. arXiv preprint arXiv:2305.043: 04391, 2023.
Emile Mathieu, Tom Rainforth, Nana Siddharth, and Yee Whye Teh. Disentangling disentanglement in variational autoencoders, In International conference on machine learning, pages 4402-4412.
PMLR, 2019.
Alexander Mathis, Pranav Mamidanna, Kevin M Cury, Taiga Abe, Venkatesh N Murthy, Macken- zie Weygandt Mathis, and Matthias Bethge. Deeplabcut: markerless pose estimation of user-defined body parts with deep learning. Nature neuroscience, 21( 1281 1-12-9 2018
Rufus Mitchell- Heggs, Seigfred Prado, Giuseppe P Gava, Mary Ann Go, and Simon R Schultz. Neural manifold analysis of brain circuit dynamics in health and disease. Journal of Computational Neuroscience, 51(1):1-21, 2023.
Simon Musall, Matthew T Kaufman, Steven Gluf, and Anne K Churchland. Movement-related activity dominates cortex during sensory-guide decision making. BioRxiv, page 308288, 2018.
Simon Musall, Matthew T Kaufman, Ashley I. Juavinett, Steven Gluf, and Anne K Churchland. Single-trial neural dynamics are dominated by richly varied movements. Nature neuroscience, 22 (10):1677-1666, 2019.
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary De Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. 2017.
Andrinandrasana David Rasamoelina, Fouzia Adjailia, and Peter Sinčák. A review of activation function for artificial neural network. In 2020 IEEE 18th World Symposium on Applied Machine Intelligence and Informatics (SAMI), pages 281 286. IEEE, 2020.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Medicul image computing and computer-assisted interventionMMICCA. 2015: 18th international conference, Munich, Germany, October 5-9, 2015, proceedings, part III 18, pages 234-241. Springer, 2015.
Omid G Sani, Hamidreza Abbaspourazad, Yan T Wong, Bijan Pesaran, and Maryam M Shanechi. Modeling behaviorally relevant neural dynamics enabled by preferential subspace identification. Nature Neuroscience, 11:::4401999 2021.
Shreya Saxena, Ian Kinsella, Simon Musall, Sharon H Kim, Jozsef Meszaros, David N Thibodeaux, Carla Kim, John Cunningham, Elizabeth MC Hillman, Anne Churchland, et al. Localized semi- nonnegative matrix factorization (locanmf) of widefield calcium imaging data. PLoS computational biology, 16(4):e 1007991, 2020.
Steffen Schneider, Jin Hwa Lee, and Mackenzie Weygandt Mathis. Learnable latent embeddings for joint behavioural and neural analysis. Nature, 617(7960):360-366 2023.
Mike Schuster and Kuldip K Paliwal. Bidirectional recurrent neural networks. IEEE transactions on Signal Processing, (5(11)):673-266 1997.
Changhao Shi, Sivan Schwartz, Shahar Levy, Shay Achvat, Maisan Abboud, Amir Ghanayim, Jackie Schiller, and Gal Mishne Learning disentangled behavior embeddings. Advances neural information processing systems, 14:22562-225773 2021.
13

---

Han Song, Cong Liu, and Huafeng Dai. Bundledslam: An accurate visual slam system using multiple cameras. In 2024 IEEE 7th Advanced Information Technology, Electronic and Automation Control Conference (IAEAC), volume 7, pages 106-111, 11, 2024. doi: 10. 11991AAEACS9433222244000000
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502, 2020a.
Yang Song, Jascha Sohl-Dickstein, Diederik Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:201 1,3456,, 2020b.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: simple way prevent neural networks from overfitting. The journal of machine learning research, 15(1):1929-131) 1958,, 2014.
Carsen Stringer, Marius Pachitariu, Nicholas Steinmetz, Charu Bai Reddy, Matteo Carandini, and Kenneth Harris. Spontaneous behaviors drive multidimensional, brainwide activity. Science, 364(6437)xcav78993 2019.
Jingyuan Sun, Mingxiao Li, Zijiao Chen, and Marie-Francinn Moens. Neurocine: Decoding vivid video sequences from human brain activties. arXiv preprint arXiv:2402.01590, 2024a.
Jingyuan Sun, Mingxiao Li, Zijiao Chen, Yunhao Zhang, Shaonan Wang, and Marie-Francine Moens. Contrast, attend and diffuse decode high-resolution images from brain activities. Advances in Neural Information Processing Systems, 36, 2024b.
Arash Vahdat, Karsten Kreis, and Jan Kautz. Score- generative modeling in latent space. Advances in neural information processing systems, 14:11287-11302, 2021.
Sandra Vieira, Walter HL Pinaya, and Andrea Mechelli. Using deep learning to investigate the neuroimaging correlates of psychiatric and neurological disorders: Methods and applications. Neuroscience & Biobehavioral Reviews, 74:58-75, 2017.
Yule Wang, Zijing Wu, Chengrui Li, and Anqi Wu. Extraction and recovery of spatio-temporal structure in latent dynamics alignment with diffusion model. Advances in Neural Information Processing Systems, 36, 2024.
Matthew R Whiteway, Dan Biderman, Yoni Friedman, Mario Dipoppa, E Kelly Buchanan, Anqi Wu, John Zhou, Niccoló Bonacchi, Nathaniel J Miska, Jean-Paul Noel, et Partitioning variability in animal behavioral videos using semi-supervised variational autoencoders. PLoS computational biology, 17(9)z 009439. 2021.
Ding Zhou and Xue-Xin Wei. Learning identifiable and interpretable latent models of high- dimensional neural activity using pi-vae. Advances in Neural Information Processing Systems, 33: 7234-7247, 2020.
14

---

Appendix to "Exploring Schavior-Relevee and Disentangled Neural Dynamics with Generative Diffusion Models"
A Methodology details
Broader Impacts and Future Work. Our results highlight the method's ability to reveal fine- ane-ggrinedd neuro-behavioral relationships, advancing our understanding of how neural dynamics encode behavior. These results demonstrate how BeNeDiff can elucidate interpretable quantifications of behaviors of interest, making it a promising machine learning tool for explainable neuroscience. Future work will explore extending this approach to more neural datasets and further refining the generative models for more theoretical interpretability and utility in neuroscience research.
Training Details of Neural LVM. The neural signal dimensions for the brain regions
follows: MOs_L: 14 dimensions, MOs_ __: 14 dimensions, VIS_I. 24 dimensions, VIS_R- 21 dimensions, SSp_L.: 23 dimensions, and SSp R: 22 dimensions. Both the probabilistic encoder and decoder of the neural LVM are based on an RNN architecture Fabius and Van Amersfoort [2014]. Mean squared error (MSE) is used for both the neural reconstruction and behavior decoding loss. We use the Adam Optimizer Kingma and Ba 24] for optimization and the learning rate is set as 0.001. The batch size uniformly set 32. The latent subspace factor number is fixed at 6, which is the same as the number of behaviors of interest. We employ the dropout technique Srivastava et al. [2014] and the ReLU activation function Rasamoelina al. [2020] between layers in our probabilistic encoder and decoder neural networks.
Training Details of Video Diffusion Models. We adopt the architecture of the VDM of 3D-UNet [Ho et al., 2022b] with the e-parameterization. We use both spatial attention and spatial convolutions. The temporal convolutions are used maintain consistency between frames. The embedding input size to the UNet architecture is set as 32 and the UNet has three downsampling and upsampling layers. The diffusion timestep is set 200. The training batch size is 64, with learning rate of 0.001. We use Group Normalization.
B In-depth Investigation on the neural LVM module across brain regions
Table 2: The R2% and RMSE of the neural reconstruction, and the disentanglement MIG of the latent subspace the Vis-Right region data. The boldface denotes the highest score of the MIG metric. Each experiment condition is repeated with 5 runs, and their mean and standard deviations are listed.
Session-1
Session-2
Metries Method
Standard VAE
Ours
Standard VAE
Ours
R²(%) RMSE
77.79 (+0.00)
73.74 (#0.24)
78.68 (+0.21)
71.13 ( 0.299
48.94 (#0.18)
55.17 (#0.19)
49.54 (=.222)
54.27 (=0.19)
MIG(%) f
34.61 (+0.30)
56.36 (#0.29
33.20 (+0.27)
59.05 (-.26)
Table 3: Ablation Study of the neural LVM module. The boldface denotes the highest score of the MIG metric. Each experiment condition is repeated with 5 runs, and their mean and standard deviations are listed.
Region
Metrics
Standard VAE
w/o Beha
w/o TC
Ours
(²)%)
83.66 (=0021)
77.74 (+0.23)
79.82 (+0.29)
75.41
(+0.24)
VIS-Left
RMSE |
30.96 (00188)
34.86 (+0.20)
34.71 (+0.13)
35.50
(7)
MIG(%)
33.13 (#0.24)
48.54 (+0.23)
38.13 (+0.27)
55.87
(#0.26)
(²(%)
84.70
(+0.244)
76.08
(+-0.20)
75.49 (+-0222)
69.59
(+-0222)
MOs-Left
RMSE +
31.41
(+0.22)
34.14 (+0.25)
34.92 (+0016)
36.91
(+00118)
MIG(%)
32.96 (+0.21)
49.79
(+0.23)
40.74 (+0.23)
58.56
(+0029)
15

---

C Video Generation Results on Various Behaviors of Interests
Using Figure 9 as an example, for the naive latent manipulation method, the generated frames are in reasonable form. Nevertheless, the frame differences are still intertwined, and the latent factor of "Paw-(y)" heavily affects the "Spout" movement. Meanwhile, for classifier-fre guidance, the trajectories focus on the mouse movements, but they are still entangled with the "Chest" movements. In contrast, the results of BeNeDiff show more specificity to the targeted behavior of interest. The inter-frame differences in BeNeDiff's results are clearly specified to the "Paw-(y)" movements, and the temporal evolution of the neural dynamics is coherent with real-world mouse paw trajectories. The generated results in Figures 10 and 11 show similar trend, demonstrating specificity to the Paw-(x)" and Spout" factors.
Naive Latent
Manipulation
Inter-Frame Difference
Classifier-Free Guidance
Inter-Frame Difference
Bife
(Ours)
Inter-Frame
Difference
Figure 9: Generated Single-trial Behavioral Videos with Latent Factor Guidance from the bottom view. Compared to baseline methods, we observe that the neural dynamics of a latent factor in the results of BeNeDiff show specificity the "Paw-(y)" movements.
16

---

Naive Latent
Manipulation
Inter-Frame Difference
Classifier-Free
Guidance
Inter-Frame Difference
BeNeDiff
(Ours)
Inter-Frame
Difference
Figure 10: Generated Single-trial Behavioral Videos with Latent Factor Guidance from the bottom view. Compared to baseline methods, we observe that the neural dynamics of a latent factor in the results of BeNeDiff show specificity to the "Paw-(x)" movements.
Naïve Latent
Manipulation
Inter-Frame Difference
Classifier-Freo Guidance
Inter-Frame Difference
BeNeDiff
(Ours)
Inter-Frame
Difference
Figure 11: Generated Single-trial Behavioral Videos with Latent Factor Guidance from the bottom view. Compared to baseline methods, we observe that the neural dynamics of latent factor the results of BeNeDiff show specificity to the "Spout" movements.
17

---

D Learnt Neural Latent Trajectories of BeNeDiff across various brain regions
Left Hemisphere Regions (Fetched Trial Id: 72)
Right Hemisphere Regions (Fetched Tria Id: 72)
16
Vaxis
0.
Vi-lt Laients traesslory
Vbstt Laients traestory
55p & Labents Trjectory
Sipt Sip-R Latents Trjjettory
MCa Labenta Trjectory
HD Latenta Trajectory
-10
Pait
Start Poin
20p4.
X-axis
X-axis
Figure 12: Learnt Neural Latent Trajectories of BeNeDiff across various brain regions. It difficult to clearly visualize the specific motion encoded by each region and distinguish how different the motions are encoded across brain regions.
E
Video Generation Results on Various Brain Regions of the Left Hemisphere
"Levers" Coming Point
VIS-Left
Inter-Frame Difference
SSp-Left
Infer-Frame
Difference
MOs-Left
Inter-Frame
Difference
Figure 13: Generated video frame differences across the left hemisphere regions. The red dots in the figure indicate paw appearances.
18

---

Discussion and Limitation
Our study introduces BeNeDiff, novel approach leveraging behavior-informed latent variable models and generative diffusion models to uncover and interpret neural dynamics. Through empirical validation, we demonstrate that BeNeDiff effectively identifies disentangled neural subspace and synthesizes behavior videos that provide interpretable insights into neural activities associated with distinct behaviors of interest. However, for the neural latent variable model (LVM) module, there exists balance between disentangling the neural subspace with behavior semantics and maintaining neural reconstruction performance. For each brain region and session, at this stage, a careful hyper- parameter search is necessary to balance the weight between these two components. For the generative video diffusion module, we implement the neural encoder (classifier for guidance) as linear regressor for interpretability. This linear assumption can be relaxed later for improved guidance performance.
19

---

NeurIPS Paper Checklist
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification:
In Section Introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
In Section Discussion.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
20

---

Justification: [NA]
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: In Section 5 Experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
21

---

Answer:
[Yes]
Justification:
In Section 5 Experiments.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.. cc/
wwwpammssssooooioioooooooooooooe icy for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https:
/ //nips.coo. wttpccccpppccceeeeececceccccccct Sttp:///gg.iieeeeeeeuuubiiiiiiii /Cooeeeooiiesseeeeeeeeeee iitommmsssion sswssnsiloll. ceiiinoyoccy) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: In Section 5 Experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: In Section 5 Experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
22

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: In Section 5 Experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that NeurIPS Code of Ethics https: : //wuurrrrs....// cc htcpppiccc/thhhccsseeeeeeee
didn't make into the paper).
9. Code Of Ethies
Question: Does the research conducted in the paper conform, in every respect, with the
Answer: [Yes]
Justification: [Yes]
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: In Appendix.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to
23

---

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative socictal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi sepporswithoode thcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's is creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
24

---

Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/coddemoodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
25


Reviews:
[{"review_id": "6x5qaODEiY", "reviewer": "Reviewer_o3cJ", "summary": "The authors implement a VAE with behavioral labels to identify behavior-specific latent variables. They use these latents to then create a generative model for experimental video. They focus on the application of their model to a wide-field cortical recordings of a mouse during an experimental visual task. They show that their model identifies disentangled behaviorally relevant latents and can create experimental video where identifiable behavioral elements are generated from specific neural latents.", "questions": "NA", "limitations": "The authors only minimally discussed the limits of the ability of their model.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "I see two primary contributions in this paper that make it appropriate for a Neurips audience. 1) The use of Total correlation measures to disentangle latents in a VAE framework in neuroscience. 2) Using generative video diffusion models to identify what video features relate to individual neural latents. Both of these are interesting applications of existing methods but they are applied in a clever way to neural data and I think these would be of interest to a computational neuroscience audience.", "weaknesses": "Each of the primary contributions to neural LVM type models (the use of total correlation and the diffusion modeling) are not sufficiently benchmarked against competing approaches. It is unclear how this model compares to the others used in neuroscience. If the authors wanted to strengthen the case that their model is an important novel contribution for behaviorally-relevant latent disentangling, they could compare how total correlation VAE compares to other methods where behavioral labels are used to help disentangle the latent states. One important comparison that I would be curious to see that isn't cited would be the pi-VAE1, but the others the authors cite could be used. Specifically, these other models could be added to figures 4 and 5.\n\nThe authors could also focus on how using video diffusion is a novel application in this setting. Again, some additional benchmarking with existing deep LVMs in neuroscience and some further discussion and emphasis as to how the diffusion methodology solves an important problem in neuroscience that perhaps existing methods cannot solve would strengthen this paper.\n\nThough the authors highlight the application of the model to this specific dataset as a contribution, this alone would be more appropriate for a more experimentally-focused venue. For neurips, the focus should be on the model advancements and their applications, not the significance of the dataset.\n\n1 - Learning identifiable and interpretable latent models of high-dimensional neural activity using pi-VAE - Zhou and Wei", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer o3cJ,\n\nThank you for your detailed and constructive comments. We would like to make the following clarifications. Hopefully these will resolve most of your concerns, and they can be taken into account when deciding the final review score.\n\n> Each of the primary contributions to neural LVM type models (the use of total correlation and the diffusion modeling) are not sufficiently benchmarked against competing approaches. \n\nWe would like to clarify that the contribution of BeNeDiff is two-fold: (1) The use of the total correlation term in the behavior-informed neural latent variable model (LVM) to induce a disentangled neural latent subspace. While this module is not our main modeling focus, it is effective compared to baselines (please refer to Table 1 of the attached PDF for the results). (2) The main novelty and contribution of our paper is the video diffusion modeling (VDM) module, which serves as an interpretation tool for visualizing the neural dynamics of each learned disentangled latent factor. The generated behavioral videos of VDM conditioned on activating one learned latent factor over time demonstrate that each factor shows specificity to a ground-truth behavior of interest (e.g., paw-x-axis movement, jaw movement) and aligns with ground-truth behavioral trajectories. The qualitative performance comparison of this VDM module against baselines is presented in Figures 6 and 7 of the manuscript, with detailed analyses in Sections 3.2.1 and 5.3.\n\n> If the authors wanted to strengthen the case that their model is an important novel contribution for behaviorally-relevant latent disentangling, they could compare how total correlation VAE compares to other methods where behavioral labels are used to help disentangle the latents. One important comparison that I would be curious to see would be the pi-VAE [1]. \n\nIncluding the suggested pi-VAE [1], we have added the following three baselines for extensive comparisons. Please refer to Section 1.3 of the global response for the baseline descriptions and tables with results. We also note that the VDM module of BeNeDiff can generalize to interpret the neural dynamics of the latent factors learned by all the neural LVM baselines mentioned above.\n\n> The authors could also focus on how using video diffusion is a novel application in this setting. Again, some further discussion and emphasis as to how the diffusion methodology solves an important problem in neuroscience that perhaps existing methods cannot solve would strengthen this paper.\n\nThis is a highly valid point. We would like to emphasize that the *scientific question* in neuroscience we aim to answer in this paper is \"*how can we enable in-depth exploration of neural latent factors with video behavior recorded, revealing interpretable neural dynamics associated with behaviors*\". Therefore, the modeling goal is to develop a machine learning tool for visualizing neural dynamics encoded by each learned latent factor. This involves mapping the neural latent factors $\\mathbf{Z}$ to behavior videos $\\mathbf{Y}$, activating a single latent factor $\\mathbf{z}^{(d)}$, and observing how the manipulation affects $\\mathbf{Y}$. The induced changes in the videos reveal the dynamics encoded by $\\mathbf{z}^{(d)}$. \n\nPrevious manipulation methods [2, 3] often change the target $\\mathbf{z}^{(d)}$ while fixing non-target subspaces to arbitrary values. However, setting arbitrary values without knowing the true distributions of non-target subspaces can lead to unnatural distortions in generated videos, complicating the interpretation and visualization of genuine animal behavioral dynamics. \n\nBeNeDiff proposes using a video diffusion model (VDM) module to explore disentangled neural dynamics in a generative manner. Specifically, the VDM module employs a neural encoder (Eq. (7) of the manuscript) to guide the generation of videos that maximize variance in the neural trajectory of target latent factor $\\mathbf{z}^{(d)}$ while minimizing variance in other factors' trajectories. This approach ensures that the generated behavioral videos predominantly reflect the neural dynamics of the target latent factor. It maintains naturalness by avoiding assumptions about specific values in non-target subspaces, thereby preventing the generation of videos with unnatural distortion, which manifests the modeling-wise novelty of BeNeDiff. As shown in Figures 6 and 7 of the manuscript, for each latent factor, the VDM module provides interpretable quantifications of its neural dynamics with specificity to the behaviors of interest. These generated video results are also available in the 'BeNeDiff-Generated-Video' folder of the supplementary materials.\n\n> Though the authors highlight the application of the model to this specific dataset as a contribution, this alone would be more appropriate for a more experimentally-focused venue. For neurips, the focus should be on the model advancements and their applications.\n\nWe would like to politely disagree with the statement that the application of our proposed method is specific to this dataset. Please refer to Section 1.1 of the global response for details on how BeNeDiff can be generalized to multiple datasets. For the model advancements, please refer to the answer of the above point. We also note that the diffusion model has been employed as an interpretation tool in previous neuroscience works, resulting in publication [4] at NeurIPS.\n\n> The authors only minimally discussed the limits of the ability of their model.\n\nDue to space limitations, please refer to Section 1.5 of the global response for detail.\n\n\n\nWe look forward to further discussion, and are happy to answer any questions that may arise.\n\n\n\nRefs:\n\n[1] pi-VAE. (Zhou et al., 2020)\n\n[2] Partitioning variability in animal behavioral videos using ss-vaes. (Whiteway et al., 2021)\n\n[3] Classifier-free diffusion guidance. (Ho et al., 2022)\n\n[4] Cortical discovery using large scale generative models. (Luo et al., 2023)", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8K0UoAXu08", "reviewer": "Reviewer_CLKJ", "summary": "The paper proposed BeNeDiff to learn disentangled neural trajectories together with a generative diffusion mode for behavior. BeNeDiff leverages beta-VAE for learning of disentangled space space Additional behavior generation module is applied for interpretation of the neural subspace. It is interesting to see that the learned latent latent trajectory is separated for different behavior types. The qualitative results and quantitative results demonstrate the superior of the proposed method.", "questions": "1.How training and testing data are split? Are they session wise or random? How the method can be generalized to different sessions or animals?\n\n2.Does the learned latent trajectory show some clustering property, or somehow organized, could you visualize it?\n\n3.For session 3.2, $c$ is better to be described as it is first mentioned in eq.5. Does the number of class is equals to feature dimension D and the same as the number of the behavior classes\"? \n\n4.To get figure 4, which latent variable do you use, $\\hat{Z}$ or $Z$?\n\n5.For fig.6, and fig.7, could you show the difference of the ground truth frame with the same behavior type as comparison.", "limitations": "It could be better the paper include more analyze on the learned latent trajectory and see if it could provide some scientific insight.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1.The disentangle property of neural latent space is significant compared to previous methods based on the quantitative metric.\n\n2.The idea leveraging diffusion model to interpret learned neural subspace is interesting.", "weaknesses": "1.It is unclear to me if the behavior video generation part is trained together with the disentangled subspace learning. It seems true based on the figure, but didn't see how they are optimized together from text.\n\n2.it is interesting to learn the disentangled neural space. Some visualization of the learned latent space is beneficial. \n\n3.Additional ablation study on different contribution of the loss function could be performed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer CLKJ,\n\nThank you for your valuable comments. We would like to make the following clarifications. Hopefully these will resolve most of your concerns, and they can be taken into account when deciding the final review score.\n\n> It is unclear to me if the behavior video generation part is trained together with the disentangled subspace learning. It seems true based on the figure, but didn't see how they are optimized together from text.\n\nWe would like to note that given the learned disentangled neural subspace and trajectories (from the neural latent variable model), the behavioral video (generation) diffusion model (VDM) module serves as an expressive ML tool for interpreting and visualizing complex factor-wise neural dynamics. Hence, the behavioral VDM is trained after the neural LVM in two steps, as we need high-quality neural latents for training the classifier (neural encoder) in VDM. This procedure is also in line with the conventions of previous studies [1] for scientific interpretation. \n\nRegarding Figure 2, it describes the neural dynamics interpretation phase of BeNeDiff rather than the model training phase. In this phase, behavior videos are generated with guidance from the neural encoder. In the revised manuscript, we will also enhance Figure 2 for clearer visualization.\n\n> It is interesting to learn the disentangled neural space. Some visualization of the learned latent space is beneficial. Does the learned latent trajectory show some clustering property, or somehow organized, could you visualize it? It could be better the paper include more analyze on the learned latent trajectory and see if it could provide some scientific insight.\n\nWe thank the reviewer for this practical suggestion. In Figure 1(B) of the attached PDF, in each subfigure, we select two neural latent factors at a time and plot their 2D trajectories. Latent-factor-3 is informed by the \"Paw-x\" behavior-of-interest. We observe that the learned dimension axes are disentangled, but the behavior dynamics are not readable from the trajectories. Figure 1(C) depicts the temporal evolution of all six latent factors, which is also difficult to interpret in terms of the behavior dynamics encoded by the neural population. In Figure 1(D) of the attached PDF, the full neural latent trajectories of trials (dimension reduction by PCA to 3D) are also hard to directly interpret or identify any clustering structure. \n\nTo sum up, the learned neural subspace exhibits certain disentangled properties, but the visualization results are not intuitive or readable enough for experimentalists. Therefore, the VDM module in BeNeDiff is crucial for clearly interpreting the neural dynamics of each latent factor, which demonstrates specificity to behaviors of interest and aligns with ground-truth behavioral trajectories (e.g., Latent-factor-3 is aligned with \"Paw-x\" in Figure 1(E)). Please refer to Section 1.2 of the global response for detail.\n\n> Additional ablation study on different contribution of the loss function could be performed.\n\nPlease refer to Section 1.4 of the global response for a detailed discussion of this point.\n\n> (1) How training and testing data are split? Are they session wise or random? (2) How the method can be generalized to different sessions or animals?\n\n(1) We train separate models for each session independently. Within each session, we use an 80/20 split for training and testing, employing cross-validation to ensure model robustness. \n\n(2) This is an interesting point. The generalizability of each method is a vital consideration. For the neural data part, the difficulty arises from the varying number of observed neurons across different sessions. Additionally, the same brain region can exhibit varying encoding for the same behavior of interest across different animals. One possible solution to mitigate the issues is to add a linear probing layer to perform neural align [2, 3] from various sessions and animals into a unified subspace, then can be adpated to the BeNeDiff framework. Meanwhile, the VDM module can generalize well to multi-sessions and multi-animals settings by fitting a unified neural encoder for the aligned neural data.\n\n> (1) For session 3.2, 𝑐 is better to be described as it is first mentioned in eq.5. (2) Does the number of class is equals to feature dimension D and the same as the number of the behavior classes\"?\n\n(1) Thank you for this practical suggestion. We will move the description of class label $\\mathbf{c}$ from line 147 to after Eq. (5).\n\n(2) Yes. The number of class labels $\\mathbf{c}$ is the same as the latent factor dimension $D$, as well as the number of behaviors of interest.\n\n>  To get figure 4, which latent variable do you use, 𝑍^ or 𝑍?\n\nWe use each latent factor of $\\mathbf{Z}$ for behavior decoding, and get the results in Figure 4.\n\n>  For fig.6, and fig.7, could you show the difference of the ground truth frame with the same behavior type as comparison.\n\nWe apologize for the confusion and would like to clarify that the results in Figures 6 and 7 are all derived from analyzing the behavior-related neural dynamics of one target neural latent factor using two baseline latent manipulation methods and BeNeDiff. The ground truth frames from behavior trials in the dataset do not specifically activate one neural latent factor, so the frame differences between consecutive images result in entangled behaviors. Including them would not provide a fair comparison. We have, however, provided several ground-truth behavior trials in the supplementary materials for reference.\n\n\n\nWe look forward to further discussion, and are happy to answer any questions that may arise.\n\n\n\nRefs: \n\n[1] Cortical discovery using large scale generative models. (Luo et al., 2023)\n\n[2] Stabilizing brain-computer interfaces through alignment of latent dynamics. (Karpowicz et al., 2022)\n\n[3] Using adversarial networks to extend brain computer interface decoding accuracy over time. (Ma et al., 2023)", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3b8rzkCC2p", "reviewer": "Reviewer_HCYS", "summary": "This study presents BeNeDiff that integrates behavior-informed latent variable models with state-of-the-art generative diffusion models to investigate neural dynamics during behavioral tasks. The methodology involves identifying fine-grained and disentangled neural subspaces and synthesizing behavior videos to provide interpretable quantifications of neural activity related to specific behaviors. The authors validate their approach using multi-session widefield calcium imaging datasets, demonstrating high levels of neural disentanglement and reconstruction quality.", "questions": "I have no question.", "limitations": "see WeaKnesses part.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. Using wide-field calcium imaging for neural-behavior analysis is both innovative and interesting.\n\n2. The approach of employing a behavior LVM for latent factor disentanglement, followed by a video diffusion model for behavior encoding, is both clear and impressive.\n\n3. Especially, using $Z$ produced by LVM as learning target for the classifier in video diffusion model is quiet important, avoiding the issue of \"label leakage\" which maybe introduced by classifier-free guidance generation.\n\n4. The paper is well-organized, and the disentanglement results and behavior encoding results are both meaningful and useful.", "weaknesses": "1. This work primarily relies on a behavior LVM for neural activity disentanglement, which can be influenced by the newly introduced total correlation (TC) penalty term with $\\beta$ weighted. The impact of this penalty should be considered through ablation studies.\n\n\n**Not a Weakness**: There is significant research combining behavioral labels with latent variables of neural activity [1,2], considering the effects of different disentanglement algorithms on behavior encoding would further enhance this study.\n\n[1] Zhou, Ding, and Xue-Xin Wei. \"Learning identifiable and interpretable latent models of high-dimensional neural activity using pi-VAE.\" Advances in Neural Information Processing Systems 33 (2020): 7234-7247.\n\n[2] Schneider, Steffen, Jin Hwa Lee, and Mackenzie Weygandt Mathis. \"Learnable latent embeddings for joint behavioural and neural analysis.\" Nature 617.7960 (2023): 360-368.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer HCYS,\n\nThank you for your recognition of the work and insightful questions. We provide clarifications and new results that we have generated to address your questions below.\n\n> This work primarily relies on a behavior LVM for neural activity disentanglement, which can be influenced by the newly introduced total correlation (TC) penalty term with 𝛽 weighted. The impact of this penalty should be considered through ablation studies.\n\nWe appreciate your attention to this aspect. The following are our ablation studies with various $\\beta$ values. Each experiment condition is repeated with 5 runs, and their mean and variance are listed.\n\n|              Metrics               |    $\\beta = 1$     |    $\\beta = 2$     | $\\beta = 4$ (Ours) |    $\\beta = 8$     |\n| :--------------------------------: | :----------------: | :----------------: | :----------------: | :----------------: |\n|        $R^2$(%) $\\uparrow$         | $79.24 (\\pm 0.27)$ | $77.97 (\\pm 0.19)$ | $75.41 (\\pm 0.24)$ | $71.44 (\\pm 0.28)$ |\n|      RMSE (x100) $\\downarrow$      | $33.90 (\\pm 0.21)$ | $34.65 (\\pm 0.16)$ | $35.50 (\\pm 0.17)$ | $38.84 (\\pm 0.18)$ |\n| $\\operatorname{MIG}$(%) $\\uparrow$ | $48.75 (\\pm 0.26)$ | $51.37 (\\pm 0.28)$ | $55.87 (\\pm 0.26)$ | $58.85 (\\pm 0.26)$ |\n\n*Table 1: Hyper-parameter $\\beta$ Investigation on the SSp-Left region of Session-1.* \n\n|              Metrics               |    $\\beta = 1$     |    $\\beta = 2$     | $\\beta = 4$ (Ours) |    $\\beta = 8$     |\n| :--------------------------------: | :----------------: | :----------------: | :----------------: | :----------------: |\n|        $R^2$(%) $\\uparrow$         | $74.93 (\\pm 0.24)$ | $72.50 (\\pm 0.20)$ | $69.59 (\\pm 0.22)$ | $66.00 (\\pm 0.27)$ |\n|      RMSE (x100) $\\downarrow$      | $33.80 (\\pm 0.16)$ | $35.08 (\\pm 0.22)$ | $36.91 (\\pm 0.18)$ | $39.99 (\\pm 0.18)$ |\n| $\\operatorname{MIG}$(%) $\\uparrow$ | $49.26 (\\pm 0.28)$ | $52.76 (\\pm 0.29)$ | $58.56 (\\pm 0.29)$ | $61.06 (\\pm 0.27)$ |\n\n*Table 2: Hyper-parameter $\\beta$ Investigation on the MOs-Left region of Session-1.*\n\nFrom the results, we observed that setting $\\beta = 1$ or $\\beta = 2$ resulted in lower disentanglement of the latent subspace. However, increasing $\\beta = 8$ led to substantial degradation in neural reconstruction quality. Hence, we determined that $\\beta = 4$ offers a balanced trade-off between disentanglement and reconstruction. These Tables will be added in the Appendix of the revised manuscript.\n\n> There is significant research combining behavioral labels with latent variables of neural activity [1,2], considering the effects of different disentanglement algorithms on behavior encoding would further enhance this study.\n\nThis is an insightful point. We would like to note that the listed two research works [1, 2] combining behavioral labels for neural latent variable learning do not focus on the disentanglement of the neural subspace. The total-correlation (TC) term proposed here is promising to be combined with these two approaches [1, 2] to achieve a more disentangled latent subspace.  We added the TC term to their loss functions, and the results are reported in the following Tables 3 and 4. Each experiment condition is repeated with 5 runs, and their mean and variance are listed.\n\n|          Metrics \\ Method          |       pi-VAE       |    pi-VAE w/ TC    |       CEBRA        |    CEBRA w/ TC     |\n| :--------------------------------: | :----------------: | :----------------: | :----------------: | :----------------: |\n|        $R^2$(%) $\\uparrow$         | $79.60 (\\pm 0.22)$ | $72.10 (\\pm 0.25)$ | $74.37 (\\pm 0.24)$ | $70.51 (\\pm 0.28)$ |\n|      RMSE (x100) $\\downarrow$      | $33.07 (\\pm 0.18)$ | $36.20 (\\pm 0.20)$ | $36.74 (\\pm 0.22)$ | $37.82 (\\pm 0.24)$ |\n| $\\operatorname{MIG}$(%) $\\uparrow$ | $40.12 (\\pm 0.24)$ | $57.27 (\\pm 0.23)$ | $43.98 (\\pm 0.29)$ | $49.66 (\\pm 0.27)$ |\n\n*Table 3: Total-correlation term effect analyses on the SSp-Left region of Session-1.*\n\n|          Metrics \\ Method          |       pi-VAE       |    pi-VAE w/ TC    |       CEBRA        |    CEBRA w/ TC     |\n| :--------------------------------: | :----------------: | :----------------: | :----------------: | :----------------: |\n|        $R^2$(%) $\\uparrow$         | $72.63 (\\pm 0.28)$ | $66.99 (\\pm 0.29)$ | $70.73 (\\pm 0.23)$ | $67.65 (\\pm 0.23)$ |\n|      RMSE (x100) $\\downarrow$      | $32.14 (\\pm 0.17)$ | $37.12 (\\pm 0.18)$ | $35.69 (\\pm 0.19)$ | $37.75 (\\pm 0.25)$ |\n| $\\operatorname{MIG}$(%) $\\uparrow$ | $37.94 (\\pm 0.23)$ | $60.14 (\\pm 0.24)$ | $42.20 (\\pm 0.28)$ | $50.21 (\\pm 0.31)$ |\n\n*Table 4: Total-correlation term effect analyses on the MOs-Left region of Session-1.*\n\nAfter incorporating the total-correlation term,  we observe that the disentanglement performance of these baseline neural LVMs improves. These results will also be added to the Appendix of the revised manuscript. We would like to emphasize that the disentangled neural LVM module is not our primary contribution. Instead, the main modeling and scientific contribution of our work both lies in the video diffusion modeling (VDM) module of BeNeDiff, which interprets the neural dynamics of each disentangled latent factor in a generative manner. The VDM module is crucial for clearly interpreting the neural dynamics of each latent factor, demonstrating specificity to behaviors of interest (e.g., paw-x-axis movement, jaw movement) and aligning with ground-truth behavioral trajectories.  Moreover, the VDM module can be generalized to all the aforementioned neural LVM baselines with the total-correlation term for the discovery of neural dynamics in the learned neural subspace. We can also include more generated video results in an anonymized link upon request.\n\n\n\nRefs:\n\n[1] Learning identifiable and interpretable latent models of high-dimensional neural activity using pi-VAE. (Zhou et al., 2020)\n\n[2] Learnable latent embeddings for joint behavioural and neural analysis. (Schneider et al., 2023)", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We would like to express our sincere gratitude to all the reviewers for their insightful feedback and suggestions. We appreciate the positive comments which characterized our work as having an `"interesting and clever"` idea for leveraging video diffusion models to interpret inferred neural subspaces  (o3cJ, CLKJ), `"significant and impressive"` methodological advancement (CLKJ, HCYS), being `"appropriate for NeurIPS and neuroscience audiences"` (o3cJ), with a `"well-organized presentation"` (tfQ5, 2uTS), and recognizing the experimental results of our work to be `"meaningful and useful"` (HCYS). 

In this place, we would like to first provide several general clarifications to enhance overall understanding of our work.

**1.1 Generalizability of BeNeDiff to Datasets**

We would like to clarify that our framework, BeNeDiff, can generalize to datasets without constraints on the format of neural data, while the behavioral data is in the format of timely-aligned video frames. Dataset with these properties are readily accessible in modern experimental settings. Due to the scientific fact that various brain regions, such as SSp and MOs, exhibit different neural latent trajectories related to behaviors of interest, in the paper, we selected the Wide-Field Calcium Imaging dataset [1] to investigate neural dynamics within each region and to analyze discrepancies across different brain regions. Notably, our VDM module is also compatible with single-region Neuropixels data [2] and multi-region voltage imaging data [3]. We will provide the visualization results for these datasets in the appendix of our revised manuscript. We can also include the results in an anonymized link upon request.

**1.2 Main Contribution**

The main modeling and scientific contribution of BeNeDiff both lies in the video diffusion modeling (VDM) module, which interprets the neural dynamics of each disentangled latent factor in a generative manner. Specifically, the VDM module employs a neural encoder (Eq. (7) of the manuscript) to guide the generation of videos that maximize variance in the neural trajectory of target latent factor while minimizing variance in other factors' trajectories. The generation results clearly interprets the neural dynamics of each latent factor, demonstrating specificity to behaviors of interest (e.g., paw-x-axis movement, jaw movement) and aligning with ground-truth behavioral trajectories. This approach ensures that the generated behavioral videos predominantly reflect the neural dynamics of the target latent factor.

**1.3 Baselines Comparison for Neural LVM module**

The following are our comparisons focusing on behaviorally-relevant latent disentangling of the neural LVM module of BeNeDiff. Based on Table 1 of the manuscript, we have added the following three baseline methods for comparison: SSL [4], pi-VAE [5], and CEBRA [6], in which:

* Semi-Supervised Learning (SSL) [4]: a deep generative modeling method that extends the standard variational bound with behavior labels corresponding to each data point.
* pi-VAE [5]: an identifiable variational auto-encoder conditioned on task variables (e.g., motor observable states) for interpretable neural latent discovery.
* CEBRA [6]: a deep neural encoding method that jointly uses behavioural and neural data with constrastive loss for nonlinear neural dynamics discovery. 

Please refer to Table 1 of the attached PDF for the results. We observe that, compared to other behavior-informed baseline methods, the neural LVM module of our proposed BeNeDiff achieves the highest disentanglement performance (MIG) while maintaining a high neural reconstruction rate. As the neural LVM module is not our primary contribution, we did not allocate much text and space for it in the manuscript. We will incorporate these comparisons into the appendix of our revised manuscript.  We will also include the corresponding results of these baselines into Figures 4 and 5 of the manuscript.  

**1.4 Ablation Study for Neural LVM module**

Please refer to Table 2 of the attached PDF for the results of the variants of our method, demonstrating the contributions of the behavior-informed loss term and the total-correlation penalty loss term. We observe that both terms improve the disentanglement of the neural subspace while generally maintaining neural reconstruction rate. Meanwhile, the neural reconstruction expectation term and the KL regularizer term are basic components of the variational bound and removing them for an ablation study would result in a loss of mathematical integrity. We will incorporate these studies into the appendix of our revised manuscript.

**1.5 Limitation Discussion**

(1) For the neural latent variable model (LVM) module, there exists a balance between disentangling the neural subspace with behavior semantics and maintaining neural reconstruction performance. For each brain region and session, at this stage, a careful hyper-parameter search is necessary to balance the weight between these two components.

(2) For the generative video diffusion module, we implement the neural encoder (classifier for guidance) in Eq. (7) of the manuscript as a linear regressor for interpretability. This linear assumption can be relaxed later for improved guidance performance.

These points will be included in Section 6, 'Discussion', of the revised manuscript.



Refs:

[1] Single-trial neural dynamics are dominated by richly varied movements. (Musall et al., 2019)

[2] Eight-probe Neuropixels recordings during spontaneous behaviors. (Steinmetz et al., 2019)

[3] Widefield imaging of cortical voltage dynamics with an indicator evolved for one-photon microscopy. (Lu et al., 2023)

[4] Semi-supervised learning with deep generative models. (Kingma et al., 2014)

[5] Learning identifiable and interpretable latent models of neural activity using pi-VAE. (Zhou et al., 2020)

[6] Learnable latent embeddings for joint behavioural and neural analysis. (Schneider et al., 2023)

Meta-review: The paper presents a two stage approach for investigating latent representations derived from neural activity.

First, the authors present a TC-VAE with additional supervision (akin to pi-VAE), which they claim provides additional disentanglement of the feature space. Reviewers agree that the claims made regarding disentanglement are too strong, and **should be tuned down in the revision of the paper**. In particular, the standards of empirical evaluation are below what would be usually expected at NeurIPS on a standardized ML benchmark, and the authors should put care into the wording for the camera-ready version.

Regarding experiments, the additional evaluations made during the rebuttal phase are great (especially adding more methods like pi-VAE and CEBRA, and adding the TC regularization there) and should be added to the main manuscript and discussed in the context of disentanglement. Ideally, more disentanglement metrics beyond MIG should be reported in the tables to get a more holistic impression of the model performance.

Finally, the authors should make it clearer that the system is a combination of multiple potentially independent methods, vs. an end-to-end learnable system where the diffusion model informs the latent space -- this seems to be suggested by wording and e.g. the diagram in Figure 2. Better highlighting that the method of estimated the latents is separate from the "visualization" method would improve the clarity.

I recommend acceptance on the paper conditional on these modifications for the camera-ready revision of the paper.
Decision: Accept (poster)