Shared-AE: Automatic Identification of Shared Subspaces in High-dimensional Neural and Behavioral Activity
Authors: Daiyao Yi
Hao Dong
Michael James Higley
Anne Churchland
Shreya Saxena
Venue: ICLR 2025 Poster (2025)
Keywords: Computational neuroscience
Multimodal
Social behavior
Primary area: applications to neuroscience & cognitive science

Abstract: Understanding the relationship between behavior and neural activity is crucial for understanding brain function. An effective method is to learn embeddings for interconnected modalities. For simple behavioral tasks, neural features can be learned based on labels. However, complex behaviors, such as social interactions, require the joint extraction of behavioral and neural characteristics. In this paper, we present an autoencoder (AE) framework, called Shared-AE, which includes a novel regularization term that automatically identifies features shared between neural activity and behavior, while simultaneously capturing the unique private features specific to each modality. We apply Shared-AE to large-scale neural activity recorded across the entire dorsal cortex of the mouse, during two very different behaviors: (i) head-fixed mice performing a self-initiated decision-making task, and (ii) freely-moving social behavior amongst two mice. Our model successfully captures both `shared features', shared across neural and behavioral activity, and `private features', unique to each modality, significantly enhancing our understanding of the alignment between neural activity and complex behaviors. The original code for the entire Shared-AE framework on Pytorch has been made publicly available at: \url{https://github.com/saxenalab-neuro/Shared-AE}.

Published as a conference paper at ICLR 2025
SHARED-AE: AUTOMATIC IDENTIFICATION OF SHARED SUBSPACES IN HIGH-DIMENSIONAL NEU- RAL AND BEHAVIORAL ACTIVITY
Daiyao
Hao Dong, Michael James Higley Department of Neuroscience Yale University
Department of Electrical & Computer Engineering Yale University
New Haven, CT 06520, USA da yao. ei eiiiyall..... edu
New Haven, CT 06520, USA
{hao Thao.dong, m. m.higley @ealey)@allleeeee....
Anne Churchland
Shreya Saxena
Department of Neurobiology University of California, Los Angeles
Department of Biomedical Engineering Yale University
Los Angeles, CA 90095, USA
New Haven, CT 06520, USA sh shreya. swx.na@yale. edu
Achurchland@@eddee annd@leddeett ucla a.la.. edu
ABSTRACT
Understanding the relationship between behavior and neural activity crucial for understanding brain function. An effective method to learn embeddings for interconnected modalities. For simple behavioral tasks, neural features can be learned based on labels. However, complex behaviors, such as social inter- actions, require the joint extraction of behavioral and neural characteristics. In this paper, we present an autoencoder (AE) framework, called Shared-AE, which includes novel regularization term that automatically identifies features shared between neural activity and behavior, while simultaneously capturing the unique private features specific to each modality. We apply Shared-AE to large-scale neural activity recorded across the entire dorsal cortex of the mouse, during two very different behaviors: (i) head-fixed fixedfiie mice performing self-initiated decision- making task, and (ii) freely-moving social behavior amongst two mice. Our model successfully captures both 'shared features', shared across neural and behavioral activity, and 'private features', unique to each modality, significantly enhancing understanding of the alignment between neural activity and complex behav- iors. The original code for the entire Shared-AE framework on Pytorch has been made publicly available at: htttps://iithust com/ saxenalal neuro/ Shared-AF
INTRODUCTION
Recent advances in hardware and storage capabilities enable us to obtain comprehensive behavioral recordings of the subject along with the corresponding neural activity from large parts of the brain. It is now widely recognized that understanding the relationship between complex neural activity and high-dimensional behavior is crucial step in brain research that has historically been under- estimated (Pereira et al. (2020); Whiteway et al. (22T)). Understanding this relationship provides insight into how the brain processes information during different behaviors and tasks. An effective approach to achieve this is to learn embeddings for these interconnected modalities, which allows for the identification of patterns within complex datasets.
Current research on learning neural embeddings focuses simple tasks and is largely based on behavioral labels (Pandarinath et al. (2017); [Schneider et al. (2023); Zhou & Wei (2020)). How- ever, when it comes to more complex task-related behaviors and social interactions which cannot be captured by simple labels, understanding relevant features becomes significantly more challenging. Moreover, effectively aligning the features of the behavioral and neural modalities requires care- ful consideration. Integrating these features requires preserving the unique characteristics of each modality while extracting the shared aspects.

---

Published as a conference paper at ICLR 2025
In this article, we propose Autoencoder- (AE-based) framework termed Shared-AE, incor- porating a novel regularization term designed to identify features common to behavior and neural activity. Autoencoders
particularly well-suited for such tasks, they capture the underlying structure of the data while ensuring that latent variables retain meaningful features from the input. In our approach, utilize the Cauchy-Schwar (CS) divergence enhance the model's capability to extract shared information across modalities. Additionally, we apply the inverse CS divergence to enable private latent variables to capture features unique to each modality.
Our model effectively captures both 'shared features', which are common across modalities, and 'private features', which are unique to each modality. By clearly partitioning these types of features, our approach provides a comprehensive understanding of which aspects of neural activity align with behavior. Compared with other models such as (Shi et al. (2019). Gondur et al. (2024). ) Sani et al. (202T), and [Singh Alvarado et al. (2021), our framework successfully addresses the issue of modality leakage- where features from one modality undesirably influence the latent representations of another. This ensures that the shared features truly reflect across-modality insights, while the private features maintain the distinct characteristics of each data source.
We apply the Shared-AE framework towards understanding two very different datasets: (i) head- fixed mouse performing a self-initiated decision decision-making task, and (ii) a freely-moving mouse en- gaged in social behavior. In both cases, neural activity is recorded using widefield calcium imaging (WFCI) capturing many regions across the dorsal cortex. We find that the model successfully captures shared subspaces across individual and social behavior, identifying brain regions that are most aligned with the recorded behavior. Additionally, we identify aligned motif's between the neural and behavioral modalities using a Hidden Markov Model (HMM). Furthermore, Shared-AE allows us to compare neural alignment when using raw video data as compared to pose estimation and related extracted features, revealing that the activity of certain brain regions aligns more closely with raw behavioral videos. Lastly, Shared-AE can generalize beyond two simultaneously recorded modalities to examine the common subspace across multiple modalities. With the increase in naturalistic behavioral recordings and large-scale neural data, Shared-AF of- fers an automated and interpretable framework to identify behaviorally-relcaan neural activity and neurally-relevant behavioral patterns.
We highlight the novelty of Shared-AE through the following key contributions: (i) Enhanced Interpretability via Latent Subspace Separation: Shared-AE introduces novel approach that explicitly separates shared and private latent spaces, improving interpretability by reducing infor- mation leakage between modalities. This design allows robust inference even when data from one modality is unavailable or corrupted during testing. (ii) Improved Performance on Paired and Unpaired Tasks: Shared-AF maintains the integrity of latent representations in data from different modalities, even when data from one of the modalities is shuffled during evaluation (" ('unpaired' task). This robustness is demonstrated by its superior performance on the 2AFC dataset, where Shared-AE outperforms existing models such as Gondur et al. (2024): Sani et al. (221 2024); Shi et al. 2011)) (iii) Flexibility with Multiple Modalities and Image Data: Unlike previous methods, Shared-AE is able to handle complex data types such as raw image data, and is also able to extend to more than three modalities. This capability significantly expands its applicability compared to previously published methods, enabling richer behavioral representations than pose estimation alone. (iv) Min- imizing Distribution Distance Instead of Fitting to Predefined Priors: Rather than conforming to predefined priors, done in models such as Yi et al. (2022). Shared-AE minimizes the distances between distributions learned from data. This approach yields more flexible and meaningful latent representations. (v) Utility for Downstream Tasks and Improved Variance Explained: The sep- aration of shared and private latent variables ensures that the learned representations are robust and suitable for diverse downstream tasks, while enabling insights into -ehavioe relationships. A detailed discussion on the technical and scientific novelty of Shared-AE can be found in Appendix
A.3
RELATED WORKS
2.1 MULTI-MODAL INTEGRATION IN NEUROSCIENCE
Multimodal integration is a rapidly growing area of research within artificial intelligence (AD) and machine learning Baltrušaitis et al. (2017); Steyaert et al. (2023); : Brenner et al. (2024); Radford
et al. (202T): Shi et al. (2021): Tian et al. (2020): Schneider et al. (2023) Zhang et al. (2020); Lake
& Higley| (2022); Cardin et al. (2020): Lake al. (2020); [Singh Alvarado et al. (2021 Liu et al.
2

---

Published as a conference paper at ICLR 2025
2021): Shi et al. (2019): Gondur et al. (2024): Sani et al. (2024). This field aims to combine and analyze data from multiple sources to improve the understanding and performance of AI systems. In neuroscience, multimodal data often refers to different types of recordings, such as fMRI and PET (Zhang et al.] (2020); [Steyaert et al. 2023)), which provide complementary information about brain activity and function. Additionally, multimodality can encompass both behavioral data and corresponding neural activity, providing more comprehensive view of brain function.
Recent research, such as Sani al. 2021 2024), focuses on using dynamical models to generate Sehaviorally-releaan and behaviorally- wir-wiiiaally-rreevvannt neural latent variables. While introducing private latent spaces enhances interpretability, these methods struggle when faced with spurious temporal correlations and high-dinge -igh-dimensional data. In contrast, our approach extracts modelity-specific shared subspaces for mcdoss-moaality relationships while maintaining interpretability through an autoen- coder framework that reconstructs each modality, enabling scalability to complex behaviors. In Singh Alvarado al. (202T). the authors introduced a multi-encoder model that fused different modalities using Multimodal Variational Autoencoder (M-VAE) with product-of-experts (PoE) approach for modality fusion. Similarly, in the Multimodal Mixture-of-Experts VAE (MM-VAE) (Shi et al. (2019)), the authors employed a mixtuuee-of-expers (MoE) strategy instead of PoE to fuse modalities. However, both these methods result in information leakage between modalities, leading to ambiguities as the origin of the data in the latent space, drastically reducing the interpretabil- ity of the latent space. In Gondur et al.] (2024), the authors introduced a Gaussian Process (GP) framework to handle temporal relationships and designed separate latent spaces to capture private features for each modality. However, their approach falls short when dealing with unpaired tasks, as it requires both modalities be present and aligned during inference. This limitation reduces the model's 's flexibility and robustness in scenarios where data from one modality may be unavailable, shuffled, or otherwise corrupted.
2.2 LEARNING EMBEDDINGS IN NEUROSCIENCE
Recent advancements in hardware and storage capabilities have significantly enhanced the quality and capacity of behavioral and neural recordings. Consequently, substantial body of research has focused on extracting lower-dimensional features from these high-dimensional datasets. Learning embeddings involves extracting lower-dimensional features for both behavior and neural activity. In the field of learning behavioral embeddings, pose estimation tools such as Laue et al., (202T) (Q0TT): PPeeeira et al. (2022) have been broadly applied to track keypoint positions from the behavioral videos, and methods like Luxem et al. (2022); Wiltschko et al. (2013): Berman et al. (2014) utilize these keypoint behavioral features. Other works generate behavioral fea- tures directly from videos, for example, [Batty et al. (2019) applied VAEs for capturing the animal's 's behavioral features. Furthermore, WWhitwway et al. (2021) and et al. (2022); Klys et al. (2018) pro- duce interpretable latent spaces by constraining the latent distribution in various ways. Due to the nigh-dimensional nature of neural activity, learning lower-dimensional neural representations is also crucial for uncovering neural dynamics. Models such as Churchland et al.) (2012); Sani et al. (202T) apply linear methods learn interpretable embeddings from neural activity. Nonlinear models like Pandarinath et al. (2017); Zhou & Wei (2020) adopt VAE-based approaches to project behavior onto neural activity. Furthermore, [Schneider al. (2023) uses an encoder- based model and constrains the latent space during behavioral tasks using contrastive learning. However, none of these models have effectively addressed more complex behavioral tasks.
positions to generate lower-dimensien
3 METHODS
3.1 OVERVIEW
Our goal is to develop representations that capture information shared between different modali- ties, such as behavior and neural activity. We employ an AE-based model with dual encoders to extract features from each modality independently (Fig. D. The latent space of each modality is subsequently divided into two subspaces: shared and private. To promote commonality across the modalities for shared latent variables, we regularize these subspaces using the Cauchy-Schwer (CS) divergence that encourages alignment between different subspaces. Moreover, the private latent variables are constrained using the inverse CS divergence regularization that encourages distinctive subspaces. Finally, the corresponding private latent variables are combined with the shared latent variables to reconstruct the original data in each modality using modality-specffic decoders (Fig. I.
3

---

Published as a conference paper at ICLR 2025
Neural Activity
Behavior
Private
Shared
Private
(3)
Behavioral Encoder
Behavioral Decoder
Keypoint positions or raw images
Neural
mymm
LocaNMF
Neural activity Encoder
activity Decoder
5; Shared behavioral
5: Shared
Inverse CS-divergence
p. Privoie behavioral latents
p. Privatae neural latents
CS-divergence
Figure 1: Shared-AE architecture: (1) Encoder: each modality is encoded separately. (2) Sepa- ration into private vs. shared: after encoding, the latents are separated into private and pre-shared latent variables through linear dense layers. Cauchy-Schwart (CS) divergence is applied to encour- age alignment and the inverse form applied to encourage orthogonality. (3) Decoder: the latents for each modality are decoded by separate decoders for reconstruction.
3.2 MODEL STRUCTURE
We assume that we can record from C different modalities: i E {1, 21,2,.... CJ. Let modality i consist of recorded data X* over T time points X {-f 22,,,.2. ..Z.r We use a sliding window approach based on the recorded data to capture temporal information in each sample of the input; we define Y {yi .2.....ff 3 as the network input for modality where 3 = [z)_w:<t -/-==+1--. 1.... x(),, with w being the window size. To extract useful representations from each input modality, we apply separate encoders to each modality fo with set of learnable parameters 0'.
Let 2 = fo- (vi) represent the encoded representation of the t-th sample for modality i. 2f maps onto two subspaces: the shared latent subspace sf, where sj = W + b,, and the private latent subspace p', where pi P + bp. Here, W. and W, are weights while b, and 6, are bias terms. Finally, the shared latents and private latents for each modality are concatenated to form a combined latent space [s", s',p', which is then decoded back to reconstruct the original input y ([s),, Ptl) using separate decoders for each modality. Here, s/ is the reconstruction and fx= is the decoder with parameters X².
In this study, we consider the two (or more) modalities to consist of simultaneously recorded neural activity and behavior. We assume that the behavior is directly recorded using behavioral video camera, with either the raw video being considered the behavioral modality, or poses that are cap- tured by pose estimation methods such Lauer al. (202T): Pereira al. (2022).
3.3 REGULARIZATION ENCOURAGING SHARED VERSUS PRIVATE LATENT VARIABLES
Two modalities: To encourage shared structure in the 'shared' latent variables from different modal- ities, we regularize these using the CS-divergence between s; and s² (Santana et al. (2016): Kampa al. (201T)). Moreover, to encourage distinct representations in the shared versus private latent variables for each modality, regularize these using the inverse CS-divergence between the shared latents and the private latents for distinctiveness.
For two probability distribution functions (PDFs) (x) and fa(),, given the CS inequality (see Appendix A.4), CS-divergence measures the distance between the two distributions (Jenssen et al. 2006 and given by: Des(fi,fa = log ((())
J fi(x)dxx f f2(x))
Dcs( fi, f2) equals zero if and only if the two distributions fi (z()) and f2(x) are the same. For two given latent subspace with dimension d, the 1s and 2nd modalities, sf and s? the PDF of each distribution can be approximated by its Parzen window estimator:
N,
N
ja(s) = N
Σ (8, t=1
(2)
4

---

Published as a conference paper at ICLR 2025
where i 1, Goz is the Gaussian kernel with kernel size o (Santana et al. (2016) and is expressed J. By replacing the actual densities in Eq. the numerator
as Go² ()) (2x02) can be rewritten
exp(
|--x1l 20
N Σ J,K=1
J [ f ((o))a(o)ds |
1
Gor (
(3)
NIN2
According to the convolution theorem for Gaussian functions, the above equation be simplified
as fi (2()ss NyN, 2j.k=1 G (/ (s). st). Here we denote _j (v ($)
as ((1,, f2). Similarly, by replacing the f2(s) by f1(s), the expression becomes 1 [f) Ny 2j.e (V)2nj(s)), sz.), then the denominator can be written (V) V(2, f2)). for
simplification, we put V(Ji, f() as V(Ji) where i E {1,2].
Finally, equation can be expressed as:
V(f.
Los D2)( log (V(fiV((f)
(4)
Here, fi(st) and f2(s²) represents the distribution of shared latent space for modality 1 and 2. In Equation (4), minimizing V(fi) would result in the spreading out of (x), while maximizing V(fi, f2) would make the samples in both distributions closer together (Yi et al. (2022)). Thus, minimize Les for shared latent spaces and maximize it to reduce the similarity between shared and private latent spaces within the same modality. Additionally, by maximizing this value, we promote distinctiveness between the private latent spaces of different modalities. There is close relationship between the VAE and the CS CS-regularized AE, as detailed in Appendix
3.4 OBJECTIVE FUNCTION FOR TWo MODALITIES Overall, the objective function can be expressed as:
LMSE, + LMSSE, aLes,je + BB esspp + +y + 8L
(5)
Here, the terms LMSE, and LMSE, represent the reconstruction loss of the two input modalities, respectively. The Les.,uz' Lessper Losuggy' and Lesp1p2 represents the CS-divergence loss between different latent subspaces. a is introduced to control the similarity between the shared latent space.
3.%, 7, and are adopted to assist the model in producing independent latent subspaces.
3.5 GENERALIZATION TO MORE THAN TWO MODALITIES
The CS "S-divergence can also be extended to measure the distance between multiple distributions. For C' number of PDFs, the CS-divergence can be written follows:
C-1 Emulti-cs = Dos(fi, fc) = - log ΣΣ
v(faff)
(6)
i=1 iSi VVVVVV
Here, c EC-1 c. Similarly, Des f2,, equals zero if and only if the C' distributions f1()), f2(x), .... fc(x) are the same. Thus, the objective function for modalities can be written as:
LMSE, +...+ + LMSEG + C Cx
8L-
+ (86 co.pes +... 13
multi
(7)
Here, Lmultiice represents the CS loss across different modalities. For the above equation, the num- ber of MSE loss terms and the CS loss terms, which encourage independence between shared and private latent spaces within the same modality, remain consistent with the total number of modali- ties. To simplify, the weight for promoting independence is set the same value across different modalities.
4 RESULTS
We evaluated our model using three datasets: one simulated dataset and two experimental datasets. All evaluations were conducted on held-out data. The training details are provided in Appendix A.7
5

---

Published as a conference paper at ICLR 2025
4.1 SIMULATED DATASET: 3DSHAPE
We evaluated our model using simulated dataset consisting of multiple sessions, each containing recordings from modalities. The "image" modality consists of 3D shapes that varied in orien- lation, scale, shape, and color, with procedural changes applied to these features over time (Burgess) & Kim (2018)). We also simulate the corresponding non-linear time series encoding of the scale and orientation, with temporally-structtne Further details can be found in Appendix A.2.1
noise that periodically changes value between 1 and 4.
Shared latent: time series
Shared latent: image
scale, orantattion shape 4hape classes
Image
Latent dimension Private latent: time series
Latent dimension
Scale Orientations
Private latent: image
scale: classes
Lonssiii suaeey
orientation: 10 classes
C-E-0
Noise: randomly assigned 4 classos
Time series
Shared features
Private image feature
Noise
Shape
time feature
Latent dimension
Latent dimension.....
Figure 2: 3Dshape: A. The image modality consists of 3D shapes with different scales, orienta- tions, and shapes, while the time series modality encodes the scales, orientations, and temporally- structured noise; the shared information across modalities is the scales and orientations. B. Shared latent spaces for each modality, each point represents single frame, color-coded by orientations and scales. C. Private latent for each modality, color-coded by the corresponding private features: the private latent spaces retains the private features.
Our goal is to capture the shared information present in both image and time series data within the shared latent spaces, while ensuring that the private latent spaces are specific to each modality, retaining only the modality-speciff features. By doing so, we can effectively separate the shared time series and image features from those that are unique to either domain. This approach allows for clearer understanding of the underlying relationships between time series and images, and how the two modalities align or diverge in capturing the intricacies of the data.
To evaluate the success of our model, compare the correlation matrices across different latent spaces, specifically focusing on the shared and private latents. The shared latent spaces exhibit significantly higher correlation scores compared to the private latent spaces (as detailed in Appendix I. This higher correlation in the shared spaces suggests that the model effectively captures the common structure between time series and image data, while the private spaces remain distinct, as intended.
We further visualize the latent spaces, with the latents color-coded by orientations and scales (Fig. 2B). The latent spaces show clear and well-aligned separation between different orientations and scales, further validating the ability of the model to organize key features the shared latent space. The alignment in the shared space demonstrates that the model was able to identify the common underlying patterns between the two modalities
way that enhances interpretability.
We apply linear decoding model to each latent space for feature decoding after binning the data (Fig. 3 Appendix 1..222). The shared latent spaces demonstrate high accuracy in decoding scales and orientations, highlighting their ability to capture common features across both modalities. The private latent spaces perform well in isolating modality-specine information. For example, the pri- vate image latents retain more shape-related details, leading to better decoding accuracy for shape features as compared to the shared latents. Similarly, the private time series latents excel in decoding temporally-structured noise, confirming their success in capturing modality-speciff content.
Next, we demonstrate the necessity of applying the CS-divergence to the latent spaces: we train model using the same architecture but without applying the CS-divergence. As shown in Ap- Pandix[A.12.33 the results indicate a lack of separation between the private and shared latent spaces, highlighting the necessity of constraints to effectively distinguish shared and private features.

---

Published as a conference paper at ICLR 2025
In comparison to previously explored models such as MM-VAE Shi et al. (2019 and joint en- coding models (Singh Alvarado et al. (202)), the shared time series and image latents are able to represent the shared features more accurately (Fig. Appendix A. In comparison models,
comparrion Buntteanes Priata Shood sabee Nosts
Shared-AL AE private time series latents
1.0
qraditsur
Steeeittess Pivzaa
Shared- AE privase mage lasents
Shared- AE shared latents
0.8
MM4-IAA shared lotents
Joint Encoding shared
0.6
R
0.4
0.2
Kod
bawnn Medi decider
0.0
Time sorve
dncoding inated Featones
image lasanes
fhow latentss decioaiin pricte stries Aution
Image latends decediing privnee image feature
Mad
Med Zineet
deoode
rookll be-snit Put prodeete
Figure 3: Comparison against baseline models while having unpaired the data from the modal- ities. The Shared-AB shared latent results in better decoding accuracy of shared features, while private latents retain more modality-specifie information. The drop in decoding accuracy from the shared latents for baseline models clearly indicates that there is information leakage between modal- ities during training.
information leakage may result in the blending of modalities, thereby yielding unreliable outcomes in the analysis of relationships between them. For example, in Section 4.2 we aimed assess the variance explained by each latent subspace. However, if one latent subspace contains features from other modalities (as is prone to happen with the presence of a 'global' latent), it becomes unclear where specific features originate from, leading ambiguities and confusion in interpreting the latent spaces. Here, we conducted a crucial test to ensure no information leakage between the different modalities. First, we shuffled the time series data while keeping the image data unchanged. We then applied this unpaired data to the trained model to generate new latent representations for both modalities. Using these latent representations, that the decoding results from the latent representation remained unchanged for the unpaired data, confirming that no information leakage occurred between the modalities in Shared-AE (Table (4) This robust performance validates the integrity of the model in maintaining clear boundaries be- tween the shared and private latent spaces, ensuring that shared information is genuinely mutual between the two modalities, and not due to unwanted information transfer or contamination.
proceeded with the decoding task. We observed
In contrast, baseline models demonstrated information leakage, as indicated by noticeable drop in decoding accuracy when applied to similar tasks (Table [] Appendix 12.2). This performance decline highlights the strength of the Shared-AE model in preventing information leakage between modalities, making it more reliable choice for multi-modal integration. Additionally, the baseline model fails to differentiate between private and shared features, which results in a lack of inter- pretability in the latent space. Ensuring clear separation between shared and private latent vari- ables is essential in multi-modal scenarios, especially when handling complex, high-dimensional data. This separation guarantees that the insights derived from the model accurately reflect the true underlying data patterns, rather than being influenced by data contamination. Furthermore, as shown in Appendix 12.2.22 when the shared features across both modalities are weak, our model outperforms the rest.
4.2 HEAD-FIXED BEHAVIOR: ALO-ALTERNATIVE FORCED CHOICE TASK (2AFC)
We apply Shared-AE to an experimental dataset involving a fexe-fixed mouse performing a self- initiated visual discrimination task, with behavioral video recordings from two views (face and body), that included the mouse and experimental equipment. We labeled the paws, spouts, and levers using Deepl abbCut (DIC). Simultaneously. WFCI across the mouse dorsal cortex was recorded. Fur- ther details the experimental setup are in Appendix A.2.2 and recording / preprocessing details in Musall et al. (2019); Saxena et al. (2020).
Two Modalities: keypoint positions and WFCI To explore shared information between pose- estimated behavioral variables and large-scale neural activity (WFCI), we trained Shared-AE with chose shared latent
these two modalities. Based on reconstruction results (Appendix A.9.2), dimension of 50. Post- Post-training, we extracted latent spaces from the held-out dataset for downstream analyses. We first compared the correlation between the shared latent spaces between the two modal-

---

Published as a conference paper at ICLR 2025
A
Correlation coefficint:
B
LR
Shared behavioral lotents vs. Shared
left paw
MOpp 155p
Shared neural latents
right paw
left spout
RSP
lever
right spout
Right
Shared behaviora Intents
Shared neura Latents
Shared behavioral latents
Variance explained in shared behavioral
E
Lever space
Left spout
Right spout regions
Intent for different keypoint postiions
explained
10
0.
visual area
somatosensory area other brain regions
motor area
MA
02
430 1.00
CAN 6,27
4158
50
rgit
2011
part 1000
Reypoint positions
Brain regions
visualizing shared latents: (A) Correlation coefficients between the shared latent spaces: the matrix has a strong diagonal component, indicating that the shared latents are well aligned. (B) In the shared behavioral latent space, we can identify the latent variables that are most helpful in reconstructing each keypoint, and then identify the brain regions that these reconstruct. (C) We identify the regions involved in modulating each keypoint using the procedure in (B). (D) Variance explained for the shared behavioral latents, which indicates how the shared space contributes to the reconstruction of each behavioral variable (equipment in blue and body parts in yellow). Variance explained for the shared neural latents, which captures most features from the primary somatosensory and the primary motor area (Abbreviation list in Appendix(A.]]
Figure 4: Head-fixed mouse:
ities. As shown in Fig. 4A the shared latent spaces of behavioral and neural activities show a high correlation, whereas the private latent spaces do not exhibit this trait (Fig. [A). Given the high corre- lation among the shared latents, we next examined the correspondence between the latent spaces and the different features in each modality, i.c., the keypoints and brain regions BB. Specifically, we decoded body position and neural activity using each latent in the shared space, establishing one-to-many correspondence between the latent variables and the underlying data. As illustrated in Fig. 4B, this process allows us to identify the corresponding brain regions for each body posi- tion. We see in Fig, 4C the primary brain regions involved with each behavioral variable; here, the contralateral side of the brain is shown to be related to each behavior.
Next, focus the role of shared latent variables by isolating them from private latents and employing them to reconstruct the original data using a frozen decoder. This analysis reveals that the equipment in the experiment has considerable shared information with the neural activity since the mouse is performing a task, as compared to the mouse's 's body parts (Fig. AD). Notably, the somatosensory and motor cortices have substantial shared information with the behavior (Fig. 4E).
A Shared neural latents
Shared behavioral latents
B
LR
MOp
MO:
15p
RSP
Left paw raise
Right paw raise
UMAP: Color coded by HMM motifs
Spout in: response left
Left paw raise
Spout in: response right
Right paw raise
Spout
response left
response right
Figure 5: Head-fixed mouse: shared neuro-behavioral motif's: A. UMAP representation of the shared neural and behavioral latent subspaces, color-coded by motifs learnt using Hidden Markov Models (HMMs) applied to the shared behavioral subspace; we see that the shared latent success- fully captures behavioral motifs, and affords interpretability the neural subspace.. B. The corre- spondence between each shared behavioral HMM state and brain regions (details in text).
8

---

Published as a conference paper at ICLR 2025
Finally, we examine whether the shared latents are able to generate reasonable shared neuro- behavioral motifs. To understand this, we apply a Hidden Markov Model (HMM) to the shared behavioral subspace and evaluate its generalization to the shared neural latent in Fig. [A.. In con- trast, the private neural latent space was unable to capture the patterns observed in the behavior, highlighting the importance of the shared latent space in identifying cross-modality relationships (Appendix A. 13.3). We further calculate the correspondence between various HMM states and brain regions (Fig. 5B: for each HMM state, we compute the variance explained by the shared latent spaces towards each brain region; we only consider those brain regions that are represented with R² > 0.3. This variance-ex explained analysis revealed important insights into how different brain regions were involved in distinct behavioral states. For instance, the visual cortex displayed a stronger response to equipment movement, likely reflecting the mouse's reliance on visual cues during task performance. Meanwhile, the somatosensory cortex was associated with nearly all of the identified motifs, suggesting that it played a central role in the task across multiple behaviors.
In comparison with other multi-modal models such as MM-GP- VAE Gondur et al. (2024). PSID Sani et al.] (202T), DPAD Sani et al. (2024), MM-VAE Shi et al. (2019), and Joint Encoding Model Singh Alvarado et al. (202I), we emphasize that Shared-AE has better decoding accuracy and of- fers significantly improved interpretability of the latent space, due to the capability of Shared-AE to form modality-specifiic shared subspaces. The configuration of each model be found in Ap- [..3.T] To evaluate decoding accuracy, we trained a linear regression model on the training dataset to predict body position using the neural latent representations. In cases where a decoder was readily available during training, such as in PSID and DPAD, we directly used it to generate these Appendix A3.2. Additionally, we compared our model with the baseline models on unpaired datasets, where Shared-AE has better decoding accuracy ( (Appendix (ppeenix(A.1..2)) AA13.2)
predictions (Table !
Table 1: Behavioral decoding accuracy with paired modalities for 2AFC body positions dataset
Subspace
PSID
DPAD
MM-VAE
MM-GP-VA GP-VEE
Shared-AE
Private latents
NA
NA 0.27 + 0.02
NA
0.37 E 0.00
0.22 E 0.03
Shared latents
0.20 + 0.03
0.41 + 0.07 0.36 + 0.01
0.41 + 0.05
Two Modalities: behavioral video and neural activity To test the generalization of our model, we apply Shared-AE to the same head-fixed dataset, while changing the behavioral modality from keypoints to nigh-dimensional behavioral images. We see high correlation between the shared behavioral and neural latents (Appendix (C). Using latent traversals, which systematically vary the latent variables to reveal their influence on the model's output and help identify specific behavioral features encoded in the latent space, we demonstrate that the shared behavioral latent captures the movements of the paw, spout, and lever (Fig. 15A).. The private latent however captures the ap- pearance of the mouse such as the shape of the eyes (Appendix A. Additionally, we compare the neural encoding results using shared behavioral latents with (i) the behavioral videos, and (ii) the keypoint positions being the behavioral modality. We found that during lnrellndel behav- iors, such as raising the left and right paws, the images' shared behavioral latents show significantly higher accuracy for encoding the visual and motor areas (Fig. [15], Appendix [T.IT). The behav- ioral image captures more features than just keypoint positions, which is consistent with the findings reported in the original paper. (Musall et al. (2019)). Moreover, unlike models such as PSID, Shared- AE can handle image data, providing greater flexibility for complex multimodal tasks. As shown in Appendix A.13.2] Shared-AE outperformed the baseline models.
Three Modalities: body position, behavioral video, and neural activity Shared AE can be ex- tended to more than two modalities; here, we show this capability using body positions, behavioral videos, and neural activity from the head-fixed mouse dataset. The shared latent spaces of these three modalities are highly correlated (Appendix B). To test whether the shared neural latents capture details about the behavior, we predicted future keypoint positions and compared this to predictions using shared neural latents from two modalities. Results show higher accuracy in predicting right paw movement with the three-modality model, especially for potentially task aask-unreltted states such as 'right paw raise', while task-related predictions remain comparable the wwo-mollyty model.
4.3 SOCIAL BEHAVIOR
To further evaluate the performance of our model in a more complex behavioral setting, we con- ducted experiments on social behavior dataset. In this scenario, two mice, referred to the agent (ml) and subject (m2), were engaged in social interactions, while mesoscopic imaging was simulta- neously performed using a large field-of-view miniscope on the subject. For more information,

---

Published as a conference paper at ICLR 2025
Appendix A.2.3 The body positions of the two mice were labeled by DLC Lauer et al. (202)
while the neural activity was preprocessed by LocaNMF (Saxena et al. (220)). In addition to tracking the body positions, nose-to-nose distance between the two mice, the relative angle between their body orientations, and
extracted several socially-relevant behavioral features, such as the
other key interaction metrics. These behavioral features allowed to quantify the complexity of social behaviors and understand their relationships with neural activity.
Based on the reconstruction accuracy (Appendix (.9.3), we set the latent dimension be 60. We apply principal component analysis (PCA) to the shared latent spaces and show high correlations across the dominant PCs of the different shared latent subspaces (Fig. [oA). The private latent spaces do not exhibit this trait (Appendix [A. 10.3). The latents effectively capture the temporal features, and the neural and behavioral latents are well aligned (Fig. ()
Next, we perform latent traversals to visualize the contribution of each latent space. As expected, as shown in Fig. GCC. the shared neural latents capture a higher R² for the subject's behavior as compared to the agent's. This supports the idea that neural encoding primarily reflects the subject's s perspective and interaction within the environment. However, the shared behavioral latents also capture the agent's behavior and many social features, such as the nose-to-tail distance, suggesting that proximity to the agent and the agent's 's behavior itself plays a significant role in modulating neural activity. Furthermore, the shared neural latents show widespread activation across regions, including the somatosensory and motor areas (Fig. 6 p). These areas are critical for movement and sensory integration, further validating the model's capacity to identify relevant neural substrates that underpin behavior. This alignment between neural activity and behavior underscores the robustness of the shared latent representations in capturing social dynamics in multi-agent settings.
A
Variance explained in shared behavioral latent space for behavioral features
WFC Social behavior
1.8
ageni
Pestume
5.8
6.6
Comelation confficient
4129
Shared latents
nome mit suddece
right svcal
nz sot seey)
mo lent swrts
nott ariyn
noce micance
neseco hallecess
mt mi sere
right camss
est nunnts
halleceton
mn saccph
hail sicancc
nut eelrto hullaceee
rece angle betweeen right
speed
Keypoint positions
Variance explained in shared neural latent space for brain regions
B
Shaied benoviraa
8.6
Neural
Rehavioral
www.aaa 201
metor cornE
0.4
440
44
475
8.07
200
850
N
590
F89t
M
Brain Regions
Figure 6: Socially behaving mice: A. Example neural and behavioral data (top); correlation be- tween the shared latent spaces (bottom). B. Lower dimensional embedding for each modality found using PCA. color-coded by the time spent in the arena. C-D: Variance explained by each shared latent subspace: the shared behavioral latents capture features related to subject and agent, as well as some social features, represented the shared neural subspace with distributed neural coding.
5 CONCLUSION AND LIMITATIONS
This paper introduces Shared-AE, an AE-based framework using CS regularization to identify fea- tures common to both behavior and neural activity, especially useful in settings where both modal- ities are high-dimensional and represent complex behavior. By utilizing CS divergence and its in- verse, Shared-AE captures both shared and unique features across modalities, enhancing our under- standing of the relationship between neural activity and behavior. Despite its numerous hyperpa- rameters, the model remains robust when well-trained. Limitations include its requirement for equal latent subspace (Appendix A.5)) dimensions, which can be infiexible, and its generalizability across multiple subjects is uncertain. Overall, Shared-AE is a robust tool for multi-modal research. Fu- ture work will explore more complex encoder models and pre- trained networks to improve training efficiency and the ability to capture informative features. Here, we aim to achieve neuroscientific insight, and do not note any negative societal impact.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
We gratefully acknowledge support from the NIH Brain Initiative IR34DA059718-01 and NSF NCS 2350329. We thank Nancy Padilla- Padilla-Coreano for helpful discussions.
REFERENCES
Allen Institute for Cell Science
Allen
Institute for Brain Science. Technical
white
paper: Allen mouse common coordinate framework and reference at- Technical report. URL https: hettps://comuunty... wec.ly.briinnmamnn org/t/
las.
Tadas Baltrušaitis, Chaitanya Ahuja, and Louis-Philippe Morency, Multimodal machine learning: A survey and taxonomy, 2017.
Eleanor Batty, Matthew Whiteway, Shreya Saxena, Dan Biderman, Taiga Abe, Simon Musall, Winthrop Gillis, Jeffrey Markowitz, Anne Churchland, John P Cunningham, Sandeep R Datta, Scott Linderman, and Liam Paninski. Behavenet: nonlinear embedding and bayesian neural decoding of behavioral videos. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché- Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems, vol- ume 32. Curran Associates, Inc., 2019. URL http::://prcceeedaal http://pooeeeeeiiggeeeeeee neurips.co/
Gordon J. Berman, Daniel M. Choi, William Bialek, and Joshua W. Shaevitz. Map- ping the stereotyped behaviour of freely moving fruit flies. Journal of The Royal So- ciery Interface, 11(99):0010072, 2014. doi: 10. 10.1098/siif22014.0622 royal
URL https://
www.yppwwliiiig....... wttp:/hhii..gg/oo//o///o///////a /atp:///o////////aa/aa// /10.
Manuel Brenner, Florian Hess, Georgia Koppe, and Daniel Durstewitz. Integrating multimodal data for joint generative modeling of complex dynamics, 2024.
Chris Burgess and Hyunjik Kim. 3d shapes dataset. http://gihhbbooooeeemmmmlll/hhas 2018.
Jessica A. Cardin, Michael C. Crair, and Michael Higley. Mesoscopic imaging: Shining wide light on large-scale neural dynamics. Neuron, 108:33-43, 2020. URL https://api. semant escho lar http:/Corruutt::::2000
Mark M. Churchland, John Cunningham, Matthew T. Kaufman, Justin D. Foster, Paul Nuyujukian, Stephen I. Ryu, and Krishna V. Shenoy. Neural population dynamics during reaching. Nature, 487:51 - 56, 2012. TRLhttps: api semant www.sttttttnccnccceeee www.wtaa...cccccccccocooooc....e wwt.:hooo.....ooooopaaeeeee
Rabia Gondur. Usama Bin Sikandar, Evan Schaffer, Mikio Christian Aoi. and Stephen L Kee- ley. Multi-modal gaussian process variational autoencoders for neural and behavioral data. In The Twelfth International Conference on Learning Representations, 2024. URL https: 7 //openreview. net. wett/foromiidaaiiitt.... joe4
Robert Jenssen, Jose C. Principe, Deniz Erdogmus, and Torbjørn Eltoft. The cauchy-schwarz divergence and parzen windowing: Connections to graph theory and mercer kernels. Jour- nal of the Franklin Institute, 343(6):61--62,, 2006. ISSN 0016-0032. doi: https://doi.rg// 10.101 ifraankiin20000 03.018 URL https:://// pestpa:/wweee /wwww//wwww/... sci scceeneedirect com/ /cimne/iince////// aatiicle/pii/5011600
767 Winners of the student paper competition at the 2005 IEEE International Conference Acoustics Speech and Signal Processing (ICASSP) held in Philadelphia PA, provide the ofate-of-art their fields research.
Kittipat Kampa, Erion Hasanbelliu, and Jose C. Principe. Closed-form cauchy-schwan pdf diver- gence for mixture of gaussians. In The 2011 International Joint Conference on Neural Nenworks, pp. 2578-2588 2011. doi: 10.1 109//CCN 166335553
Jack Klys, Jake Snell, and Richard Zemel. Learning latent subspaces in variational autoencoders. Advances in neural information processing systems, 31, 2018

---

Published as a conference paper at ICLR 2025
Evelyn Lake and Michael Higley. Building bridges: simultancous multimodal neuroimaging ap- proaches for exploring the organization of brain networks. Neurophotonics, 9, 09 2022. doi: 10.1117/1.NPh..3.0322.
Evelyn Lake, Xinxin Ge, Xilin Shen, Peter Herman, Fahmeed Hyder, Jessica Cardin, Michael Higley, Dustin Scheinost, Xenophon Papademetris, Michael Crair, and Robert Constable. Si- multaneous cortex-wide fluorescence ca imaging and whole-brain fmri. Nature Methods, 17: 1-10, 12 2020. doi:
00088/441522-02-009999
Jessy Lauer. Mu Zhou, Shaokai Yc, William Menegas, Tanmay Nath, Mohammed Mostafizur Rah- man, Valentina Di Santo, Daniel Soberanes, Guoping Feng, Venkatesh N. Murthy, George V. Lauder, Catherine Dulac, Mackenzie W. Mathis, and Alexander Mathis. Multi-animal pose esti- mation and tracking with deeplabeut. bioRxiv, 2021.
Ran Liu, Mehdi Azabou, Max Dabagia, Chi Heng Lin, Mohammad Gheshlaghi Azar, Keith B. Hen- gen, Michal Valko, and Eva L. Dyer. Drop, swap, and generate: self-supervised approach for generating neural activity. In Marc' Aurelio Ranzato, Alina Beygelzimer, Yann Dauphin, Percy S. Liang, and Jenn Wortman Vaughan (eds.), Advances in Neural Information Processing Systems 34 35th Conference Neural Information Processing Systems, NeurlPS 2021, Advances in Neural Information Processing Systems, pp. 10587-10599. Neural information processing systems foun- dation, 2021. Publisher Copyright: @ 2021 Neural information processing systems foundation. All rights reserved.; 35th Conference on Neural Information Processing Systems, NeurlPS 2021 Through 14-12-2021.
Conference date: 06-12-2021
Kevin Luxem, Petra Mocellin, Falko Fuhrmann, Johannes Kürsch, Stefan Remy, and Pavol Bauer. Identifying behavioral structure from deep variational embeddings of animal motion. bioRxiv, 2022. doi: 10. 10.1101/2020.05. 4.095430. Barly/2022/01/1/4222
05. 095430
Simon Musall, Matthew T. Kaufman, Ashley L Juavinett, Steven Gluf, and Anne K. Churchland. Single-trial neural dynamics are dominated by richly varied movements. Nature neuroscience, 22:167 1686, 2019.
Chethan Pandarinath, Daniel O'Shea, Jasmine Collins, Rafal Józefowicz, Sergey D. Stavisky, Jonathan C. Kao, Eric M. Trautmann, Matthew T. Kaufman, Stephen I. Ryu, Leigh R. Hochberg, Jaimie M. Henderson, Krishna Shenoy, L. F. Abbott, and David Sussillo. Inferring single-trial neural population dynamics using sequential anto-en. Nature methods, 15:805 815, 2017. URL http::ttt////c.t //api. semant sew.n www.aappp....oo.ooo.o.o.. 1122815
Talmo Pereira, Joshua Shaevitz, and Mala Murthy. Quantifying behavior to understand the brain. Nature Neuroscience, 23: 559- 12 2020. doi: 10.10888/411993-0000007444
Talmo Pereira, Nathaniel Tabris, Arie Matsliah, David Turner, Junyu Li, Shruthi Ravindranath, Eleni Papadoyannis, Edna Normand, David Deutsch, Z. Wang, Grace McKenzie-Smith, Catalin Mite- lut, Marielisa Castro, John DvUva, Mikhail Kislin, Dan Sanes, Sarah Kocher, Samuel Wang, An- negret Falkner, and Mala Murthy. Sleap: A deep learning system for multi- maiti-animall pose tracking. Nature Methods, 19:1-10, 04 2022. doi: 10. 10.11038--41192202--01011
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision, 2021.
Omid Sani, Bijan Pesaran, and Maryam Shanechi. Dissociative and prioritized modeling of be- haviorally relevant neural dynamics using recurrent neural networks. Nature Neuroscience, 27: 2033-2045, 09 2024. doi: 111033//41133-02--0001
Omid G Sani, Hamidreza Abbaspourazad, Yan T Wong, Bijan Pesaran, and Maryam M Shanechi. Modeling behaviorally relevant neural dynamics enabled by preferential subspace identifica- tion. Nature neuroscience, 24(1):140 149, January 2021. ISSN 1097-6256. doi: 10.
1038/4441593-000--7733-
URL https: http:://wwwww.. www.bbioxxi..cmm htt.://wwww..oooeeotee/////a///a
earl /ttp:///1////
early/2019/1//17//
full.ullllle pud.
12

---

Published as a conference paper at ICLR 2025
Eder Santana, Matthew Emigh, and Jose Principe. Information theoretic-learning auto-encoder. 03 2016.
Shreya Saxena, Ian Kinsella, Simon Musall, Sharon H. Kim, Jozsef Meszaros, David N. Thi- bodeaux, Carla Kim, John Cunningham, Elizabeth M. C. Hillman, Anne Churchland, and Liam Paninski. Localized semi-nonnegative matrix factorization (locanmf) of widefield calcium imag- ing data. PLOS Computationnal Biology, 16(4): 04 2020. doi: ht..33//joural.ppp. 1007791. jouraalouraa...am www.pccii........ 100.1 1007791
URL
Steffen Schneider, Jin Hwa Lee, and Mackenzie Weygandt Mathis. Learnable latent embeddings for joint behavioural and neural analysis. Nature, May 2023. ISSN 1476-4687. doi: 10. 1031/ s4158 1886-023-0003--6 BtRhhttps://dd http://oooorrgg/g//// 0033///41180--03-066000633
Yuge Shi, N. Siddharth, Brooks Paige, and Philip H. S. Torr. Variational ofxtere autoen- coders for multi-modal deep generative models, 2019. URL https://azxiv.crg//
191 063331
Yuge Shi, Brooks Paige, Philip H. Torr, and N. Siddharth. Relating by contrasting: A data-efficient framework for multimodal generative models, 2021.
Jonnathan Singh Alvarado, Jack Goffinet, Valerie Michael, William Liberti, Jordan Hatfield, Timo- thy Gardner, John Pearson, and Richard Mooney. Neural dynamics underlying birdsong practice and performance. Nature, 599(7886):635- 639, November 2021. ISSN 0028-0836. doi: 10.
1088641156-001--400441 URI
o www.rrrrratrrrr..eeeeoe PMCc9es//MMM191/0
Sandra Steyaert, Marija Pizurica, Divya Nagaraj. Priya Khandelwal, Tina Hernandez- Boushsrrd- An- drew Gentles, and Olivier Gevaert. Multimodal data fusion for cancer biomarker discovery with deep learning. Nature Machine Intelligence, 5, 04 2023. doi: 10.1038/s42256-023 023-006335..
Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding, 2020.
Linh Tran, Maja Pantic, and Marc Peter Deisenroth. Cauchy-schwarz regularized autoencoder, 2021.
URL htttps://zzxiv org/ abs/ 02149
Matthew R Whiteway, Dan Biderman, Yoni Friedman, Mario Dipoppa, E Kelly Buchanan, Anqi Wu, John Zhou, Niccolò Bonacchi, Nathaniel Miska, Jean-Paul Noel, Erica Rodriguez, Michael Schartner, Karolina Socha, Anne E Urai, C Daniel Salzman, The International Brain Labora- tory, John Cunningham, and Liam Paninski. Partitioning variability in animal behavioral videos using semi- variational autoencoders. bioRxiv, 2021. doi: 10. 10.1101/2021.22222 432309. URL https:: htttp::/www.bbooxivv.ooo http://wwwwaaaiooooooc///////ta/
www.:/iixxivv.oo
02.22. 432309
Alexander B. Wiltschko, Matthew J. Johnson, Giuliano Iurilli, Ralph E. Peterson, Jesse M. Katon, Stan I.. Pashkovski, Victoria E. Abraira, Ryan Adams, and Sandeep Robert Datta. Mapping sub-second structure in mouse behavior. Neuron, 8866):121--113: 2015. ISSN 0896-6273. doi: 10.
https///bi.ogg//O.000oj...mumo 2015. .031. URL www. scwencedi com/ Bccience/atticle//////8632
Daiyao Yi, Simon Musall, Anne Churchland, Nancy Padilla- Madilla-Coreano, and Shreya Saxena. Disen- tangled multi-subject and social behavioral representations through constrained subspace vari- ational autoencoder (cs-vae). bioRxiv, 2022. doi: 10. 101100000222000000000000000 URL https:
7/ wwww www.bbiorxiv... http://oggooooooooo ////////.ccxii.oooooooottttee http://oonneeeeea.l..y.emm /ttp://eeneaaaalll/////////////0
09. 01 506091
Yu- Yu-Dong Zhang, Zhengchao Dong, Shui-Hua Wang. Xiang Yu, Xujing Yao, Qinghua Zhou, Hua Hu, Min Li, Carmen Jiménez-Mest Javier Ramirez, Francisco J. Martinez, and Juan Manuel Gorriz. Advances in multimodal data fusion in neuroimaging: Overview, challenges, and novel orien-
tation. Information Fusion, 64: 14:1499187, 2020. ISSN 1566-2535. doi: https://doi.org/10.10111
infffus.2220.0..000 URL hattps::/wwee www. www.wseeiieec..cccommmm com/ http://ccieccccccoocccceee/erraa wttmm//ceennc//rrcllee pii 0111/6666663323333
0155/116662332030331
Ding Zhou and Xue-Xin Wei. Learning identifiable and interpretable latent mod-
els of high-dimensional neural activity using pi-vae. In H. Larochelle, M. Ranzato,
13

---

Published as a conference paper at ICLR 2025
R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Pro-
cessing Systems, volume 33, pp. 7234-7247. Curran Associates, Inc.. 2020. URL
https: http://cccceeeennoooooooo.o....a coppper files/paper/202//EL
14

---

Published as a conference paper at ICLR 2025
A
APPENDIX
A.1
ABBREVIATION LIST
detailed abbreviation list for different brain regions can be found at Allen Institute for Brain Sci-
eice
A.2 DATASET
A.2.1 DATASET: 3DSHAPE
The 3DShapps dataset is composed of procedurally generated 3D objects, each defined by six in- dependent ground truth latent factors: floor color, wall color, object color, scale, shape, and orien- tation. Every possible combination of these factors is uniquely represented, resulting in dataset of 480,000 images. For our task, we converted the images to grayscale to remove the color-related features, leaving three key attributes: (1) scale, with 8 values linearly spaced between 0 and (2) shape, with 4 distinct categories [O, 1, 2, 3], and (3) orientation, with 15 values linearly spaced be- tween -30 and 30 degrees. For the labels, we retained the scale and orientation values and introduced a cemporally-stracturer noise that periodically changes between one and four. For time efficiency, we applied 8000 images for training and 2000 images for testing.
A.2.2 DATASET: AWO-ALTERNATIVE FORCED CHOICE TASK (2AFC)
We employed a subset of the behavioral dataset from Musall et al. 2019). The task involved mice pressing lever to start, displaying visual stimulus to the left right, and then making decision by licking a spout corresponding to the stimulus direction after a delay. Correct choices were re- warded with juice. Behavior and neural activity were recorded 30 Hz. The training set consists of 388 trials, and the test set contains 128 trials, each comprising 189 frames.
A.2.3 DATASET: SOCIAL BEHAVIOR
Our study used a social behavior dataset involving simultancous brain mesoscopic imaging and natural behavior recording in mice. Mesoscopic imaging was performed using a large field-of- view miniscope, while behavior was captured by three cameras at different angles. Two GCaMP6 expressing mice (slc17a7- x ail62) were observed in a cubic box arena, with one mouse equipeed with miniscope. The video, around 18 minutes long, was split into 326 chunks, with a 3:1 training distance, tail-to-nose distance, speed, and
to test data ratio. Social features such as angle between mice were calculated, resulting in 37 features in the behavioral datasets.
A.3 TECHNICAL AND SCIENTIFIC NOVELTY OF SHARED-AE
Shared-AE introduces a novel approach to joint neural and behavioral modeling, addressing key limitations in existing methods and offering several distinct advantages:
(a) Enhanced Interpretability through Latent Subspace Separation: Unlike the model by Gondur et al. (2024), which combines neural and behavioral latents into a single subspace, Shared- AE explicitly separates shared and private latent spaces. This separation preserves interpretability by reducing information leakage across modalities, allowing us to better disentangle distinct modality-specific features. Indeed, having common shared subspaces can be detrimental towards interpretability, since it is unclear which modality is leading to the performance of the shared latent. As an additional key consequence of modality-specinn shared subspaces, if we only have data from one modality during inference, we do not require data from the other modalities to generate robust and meaningful representations.
(b) Improved Performance on Paired and Unpaired Tasks: We designed an 'unpaired' task to ensure there no information leakage between different modalities. Critically, after training the model, if one modality's is data is corrupted during inference, this should not affect the other modality's 's latent representation or decodability. Here, during inference only, one modality was shuffled across time, while the other modality remained unchanged. The resulting 'unpaired' data was then input into the trained model to generate new latent representations for the unshuffled modality. This approach allowed to assess whether the model could maintain the integrity of the
15

---

Published as a conference paper at ICLR 2025
unshuffled modality's latent features, ensuring that the latent representations unaffected by the shuffled modality. We compared Shared-AE against existing approaches on the 2AFC dataset (as requested by the reviewers), demonstrating superior performance in both paired and unpaired tasks (Table T1 12). Shared-AE outperforms other models by effectively capturing complex relationships across modalities, and preserves information in modality even when the other modality may be corrupted during inference. This robustness is crucial for practical applications where complete data may not always be available.
(c) Flexibility with Multiple Modalities and Image Data: Shared-AE is designed to handle more than three modalities, including image data, significantly broadening its application scope from previous research; in fact, this capability has not previously been shown in any multi-modal neuroscience study. shown in Figure 6B, using raw behavioral images as well as pose estimation leads to better prediction accuracy of neural activity compared to using pose estimation alone. This finding aligns with Musall et al. (211)), indicating that raw image data provides a richer representation of behavior. In contrast, existing models such PSID are limited in processing data from more than two modalities effectively.
(d) Minimizing Distribution Distance Instead of Predefined Priors: Previous works such as Yi et al. 2022) and Tran et al. (2021) do indeed use CS-divergence, but in drastically different way than in our study: they CS CS-divergence fit the latent distribution a predefined prior, whereas Shared-AE minimizes the distance between two learned distributions instead. This approach avoids the limitations of predefined priors and allows for more flexible and meaningful representations.
(e) Utility in Downstream Tasks and Enhanced Variance Explained: By separating the latent features into distinct shared and private subspaces, Shared-AE provides representations that can be effectively used for multiple downstream tasks.
A.4 CS INEQUALITY
For two functions (xx) and g(x)), the Cauchy-Schwarz inqquality is expressed
/ (M)os(o)an $ f
1o(x))
(8)
with equality holding if and only if the two functions are linearly dependent.
A.5 EQUAL LATENT SUBSPACE DIMENSIONS
Mathematically, the CS-divergence requires both distributions to have the same dimensionality, al- lowing for the calculation of cosine similarity after kemnllizing the latent representations. In practice, as illustrated in Fig. 2, the dimensionality of the latent space often exceeds the actual number of fea- tures. This implies that when the latent dimension is large, there may be redundancy in each latent subspace. To overcome this limitation in practice, we perform dimensionality reduction on the latent subspaces after training, especially for visualization purposes.
A.6 RELATIONSHIP BETWEEN VAE AND CS REGULARIZED AE
For standard VAE with a Gaussian prior, the Evidence Lower Bound (ELBO) is defined as:
ELBO = E (= |lo pe (x|2) + (o)g H(go(()z)
(9)
= E Eg log Pa (x: [)] + 12 1e po } + H (1s
(10)
= (x Plo [x] - (x)))(()
(11)
Where x is the input, = (N 1), the learned latent, and pa (x|²) = N (Nnn (nn, During training, the goal is to maximize the ELBO. Therefore, the objective function is written as:
(x)()))
Cx = Drz - E E== [log pe (z|2)
(12)
In contrast, our model, Shared-AE, utilizes the CS-divergence for regularization. Unlike the VAE, which employs KI. divergence measure the difference between the approximate posterior and
16

---

Published as a conference paper at ICLR 2025
the prior, Shared-AE leverages the CS divergence to encourage the alignment of the approximate posterior with the prior, enhancing flexibility in capturing the underlying structure of the data.
To place Shared-AE in a probabilistic framework, we consider the objective to maximize the log marginal likelihood of the model, below (Tran al. (2021)).
max E Pe pe (I)] = max E p(x) [og (pt= [z]
(13)
Using Jensen's 's inequality, we can obtain a lower bound to the log-marginal likelihood as follows:
log pogp)(z) log (ppppss)poopppaaaaee) 2 1B (oo(=)>)]lg))) [log (=)
(14)
Similarly, as in the VAE framework, define mapping qo (2|2²) which transforms some input x: to (probabilistic) features 2. By adding regularization R. we penalize any deviation between (=(xz)
from p(=). Ideally, this regularization is metric function for which
when g * p and
if and only if q
max Be po (x[22] 8,0
(15)
subject R((o)) <.
(16)
In this formulation, specifies the magnitude of the applied constraint. If R is defined as the KL divergence, we have the original ELBO formulation. We diverge from this principle and use the SShwaa-Schwarz divergence for regularization to match an approximate posterior to a prior, with the advantage added flexibility and expressiveness. Thus, the objective function given by: (x(]=)]
max Ep(=) 0.6
(17)
subject to Dcs ( (z(x))p(()) <c
(18)
Rewriting this as Lagrangian, we obtain:
F(r;0, 6. A) = E: - (=|=) Pa [z| A(Des (Esl))) - c),
(19)
where À is the regularization coefficient ensuring that the posterior distribution close to the prior p(z). We can rewrite this as:
F(z; A, o, A) 2 Ez-as(=)a)) [lo pa [=] ADos Ip())) =:
A).
(20)
While VAEs rely on KL divergence regularize the latent space and align the approximate posterior with the Gaussian prior, CS-AE uses CS divergence, allowing for more flexible and potentially richer latent space representations. Moreover, our formulation in Shared-AE further defines specific set of latent variables from one modality to regularize using the latent variables obtained from a different modality, to elucidate the shared structure in the data.
A.7 TRAINING DETAILS
All models were trained and tested single NVIDIA A100 using PyTorch 2.0.1. For our dataset, the runtime per batch varied based on the modalities used: approximately 0.398 seconds for keypoint positions and neural activity, 4.523 seconds for behavioral images and neural activity, and 5.575 seconds for keypoint positions, behavioral images, and neural activity. Each model was trained for 100 epochs with a batch size of 256, using the Adam optimizer with learning rate of le- 4.
A.7.1 3DSHAPE
For each input, we applied a window size of one. A 2D ResNet-18 backbone was used for image input, and a ID R8 backbone for time series data. A 2-layer 2D convolutional decoder was applied to image data and a 2-layer ID convolutional decoder to time series data. For classification tasks, scale and orientation are the shared features between image and time series data. The shape is the image-only feature while the temporally strrccuue noise is the time series-only feature.
A.7.2 2AFC
A sequence of 9 frames was stacked together, based on the HMM output of keypoint positions. Each sequence spans approximately 0.3 seconds. For keypoint positions and neural activity, ID ResNet- 18 backbone was used for encoding, and 4-layer 2D convolutional decoder for behavioral images and 2-layer 1D convolutional decoder for keypoint positions and neural activity.
17

---

Published as a conference paper at ICLR 2025
A.7.3 SOCIAL BEHAVIOR
sequence of 8 frames was stacked together based on the HMM output of keypoint positions. A 1D ResNet-18 18 encoder and a 2-layer ID convolutional decoder were used for both modalities.
A.8 STATISTIC TESTS
We performed t-tests the test dataset and used the p-value to determine the significance of our results. The p-auu annotation legend is:
ns: 5.00 X 10 < 2 1.00 x 10"
1.00 10-² 5.00 10-2
1.00 x <p= 1.00 x 10-2
1.00 x 10 <ps 1.00 x 10-3
#*8*;
p S 1.00 x 10
A.9 HYPERPARAMETER SEARCH
This section describes the procedures for selecting parameters. Due to the nature of the loss, each subspace should have the same number of latent dimensions.
A.9.1 3DSHAPE
The original dataset includes 6 different features. To simplify the dataset, grayscaled the images to have only 3 features. For better reconstruction accuracy, we choose the latent dimensions equal 5. The mean MSE loss for the image and the mean R2 for the time series 0.23e 5 and 0.98 0.007, respectively. The kernel size is set to 15 and the weight for all the CS and inverse loss terms 10.
The final training CS loss for the shared latent space is 0.0009; the inverse CS loss is 9.78 + 0.03 for the individual shared and private latent spaces; the inverse CS loss for the private latent space of different modalities is 9.78.
A.9.2 2AFC
We choose the latent dimension based on the reconstruction accuracy using 5-fold cross-validation: the smallest value when the MSE loss converged. For keypoint positions, different parts are encountered, and show the mean and maximum R² values for all keypoints. Similarly, for neural activity with 21 different regions, we show the mean and maximum R22 values across all brain regions (Figure 7 We evaluated the decoding accuracy of body positions using shared neural latents with varying latent dimensions: 50, 80, and 100. As shown in Table 2 the decoding accuracy remains relatively consistent across these dimensions, indicating that the model's performance is robust to changes in latent dimensionality. For this dataset, we chose latent dimension equal to 50. The held-out data served as the test set on which all the results are reported. For two-modality
Table 2: Behavioral decoding accuracy with various latent dimensions using Shared-AE for 2AFC dataset
50
80
100
Shared neural latent 0.41 + 0.05 0.43 = 0.03 0.42 + 0.04
task with the image the behavioral input, the reconstruction result for the image is evaluated by MSE. We choose 85 as the latent dimension. For the three modalities tasks, we choose the latent dimension to be 85 for comparison. The mean MSE loss and the mean R² are comparable to the two modality results being 0.1e - 5 and 0.65 + 0.2, respectively.
For other hyperparameters, simplification, we set a, Y, and to be the same, all of them equal 5. The results are robust to changes in these hyperparameters, as long the losses converge certain range. In Table 3
set the kernel size for this dataset to be 15 (Yi et al. (0222 For
18

---

Published as a conference paper at ICLR 2025
Two modalities for headfixed Keypoint and neural activity
Max: R2
mena
Latent Dimensions vs Reconstruction Accuarcy: Positions
Latent Dimensions vs Reconstruction Accuancy: neuraal activity
0.9
0.6
2
2 0.5
0.7
0.4
0.6
0.3
0.5
40
60
30
dimensions
B
mouse: Behavioral image
activity
Latent Dimensions Reconstruction Accuarey: image
1e-5
latent dimensions different brain
0.7
0.6
0.4
30
60
30
60
70
80
40
50
60
70
80
latant dimensions
intent dimensions
C
Two modalities for social behavior: behavior:Keypointt position and neural activity
Latent Dimensions Reconstruction Accuarcy: Keypoint Positions
Latent Dimensions vs Reconstruction Accuarey: neural activity
1.00
0.95
0.8
0.90
2
02
0.8
0.85
0.4
1.0
20
40
50 70
80
10 20
40
50
70
80
90
Wheet dimens ons.
Figure 7: Reconstruction accuracy for different datasets with different latent dimensions. The mean R represents the average accuracy across all keypoint positions and neural channels, while the max R indicates the maximum accuracy observed among these keypoints and channels. This distinction helps capture both the overall performance comprehensive evaluation of its predictive capabilities: A. 2AFC: modal tasks with keypoint positions and neural activity B. 2AFC: 2 modal tasks with behavioral image and neural activity C. social behavior: 2 modal tasks with keypoint positions and neural activity
the peak accuracy of our model across different fea-
tures, providing
include the after-training CS divergence loss for each task. For simplicity, we calculated the mean value of the inverse CS loss for the same modality (reported before inversion).
19

---

Published as a conference paper at ICLR 2025
Table 3: CS loss and inverse CS loss values for head fixed dataset
cs loss for shared latents
inverse cs loss for inverse es loss for
tasks
the same modal
different modal
keypoint positions+ neural activity
0.01
8.65 0.05
6.77
image+ neural activity
0.019
8.59 + 0.007
5.16
A.9.3 SOCIAL BEНAAIIO
According to the reconstruction plot in Fig 7C. we chose 60 as the number of latent dimensions. The kernel size is set to and the weight for all the and inverse loss terms is 10. The final training CS loss for the shared latent space is 0.006; the inverse CS loss is 1:35+:0:: for the individual shared and private latent spaces; the inverse CS loss for the private latent space different modalities 6.99.
A.
CORRELATION COEFFICIENT FOR DIFFERENT LATENT SUBSPACES
A. 10.1
3DSHAPE
The correlation coefficient matrices for different latent subspaces shown in Fig. Despite some correlations between the shared and private image latent spaces, the other subspaces are well separated.
Correlation confficient between different latent subspaces
latent dimenion-5
Shared image latents Private image latents
Shared image latents vs Shared time series latents
Private image latents vs shared time series latents
Shared time series latents
Shared time senies latents
Private image latents
Shared image latents
Shared image latents
Private image latents
Figure 8: 3Dshape: Correlation coefficient for different latent subspaces.
A.1 2AFC
The correlation coefficient matrices for different latent subspaces shown in Fig. 9 As expected, the shared latents exhibit high correlations, while the shared and private latents for the same modality show lower correlations.
A.10.3 SOCIAL BEHAVIOR
The correlation coefficient matrices for different latent subspaces are shown in Fig. T0 Despite some correlations between the shared and private social latent spaces, the other subspaces are well separated.
A. PREDICTION TASKS FOR DIFFERENT MODALITIES
We compared the prediction accuracy between image behavioral latents and keypoint positions be- havioral latents for different HMM states (Figs. II and 12.
We applicd the shared neural latent generated by the three-modal tasks for behavior prediction (Fig.
13
20

---

Published as a conference paper at ICLR 2025
A
Correlation iillionio cient betmeen diffrrent abent subspaces
modall: keypoint positions activity
Shared behavicral atents vs Private behaviora latents Shared neural lotents Private neursl labents Private behavioral atents vs Private neural Iocents
Private behavioral atents
Private neural atents
neuut abarts
04
1.4
44-
24233320 3529434444
-90
0.3. 3nssse2ssar00oo0heee 4544
Shared behavioral latents
latents
Private behaviora latents
Correlation coefficiant between
subspaces
meodels
keypoint positions
imaee activit
Shared neural latents
Shared image behavioral
Shared
image behavioral
Shared keypoint behavioral latents latents
keypoint
Shared keypoint
Shared neural latents
00
11
Shared
Shared image behavioral latents
Shared image behavioral latents
keypoint behavioral latents
Corre stion coefficient belween shared and private latent subspaces
m Imagen neural activty
Shared neura latents
Private neural latents
behavioral
Private image behavioral latents
03
11
-04
Shared
eoneor AARP coce apee Shared image behavioral latents
an conde copEare1 Private image behaviora latents
Correlation coefficient for different latent subspaces. A. Comparison between different latent spaces for the 2-modal task: keypoint positions and neural activity. (Correlation should low for shared vs. private and private vs. private) B. Comparison between shared latent spaces for the 2-modal task: behavioral image and neural activity. C. Comparison between different latent spaces for the 3-modal task: behavioral image, keypoint positions, and neural activity.
Figure 9: 2AFC:
Correlation confficient between different latent subspaces social behavior (body positions + social features) +neural activity
Shared sacial behaviara Wntents Priste sccial benavicral istents meura litents lacents
Private behavioral
12
Privte dxcla banysiona bovett
Figure 10: Social behavior: Correlation coefficient for different latent subspaces.
21

---

Published as a conference paper at ICLR 2025
Prediction of Neural Activity using Behavior
Image zettuioral letents
keypoitt posttion nebaviorsl Intentss
HOPI_
0.35
0.10
0.05
0.25
0.00
0.2
0.1
0.15
0.0
IPM PISSS
IPM math
ACAd1LL
M.J.J.
0.30
0.30
0.20
0.
20
0.20
0.15
0.10
0.05
0.05
0.00
HFV
ASP
VIS_L
MOBLL
0.30
0:35
n5
0.25
0.3
0.20
0.15
0.00
0.10
05
0.10
10
2.05
0.05
05
-0.10
Reppend the right spout
tevil
Respand to left spout
Left pave, raice
Right prowe neise
Respond the right spout
lever
Respond to the feft spout
Left sisse, ralse
Right poww ruise
Respond the right spoit
leval in
Respond to lnst sposi
Lest paw raise
Right sree, paice
Figure 1I: Prediction accuracy of neural activity (A): comparison between the image behavioral latents and key point positions behavioral latents.
A.
ADDITIONAL RESULTS: 3Dshape dataset
A.12.1
PCA EMBEDDINGS OF THE SHARED AND PRIVATE LATENT SUBSPACE
Fig. 14 illustrates that the shared latent does not capture private features, while the private latent exhibits lower decoding accuracy for shared features. This further demonstrates that our model effectively generates distinct shared and private latents.
1.12 BASELINE COMPARISON WITH PSID AND DPAD
The key advantage of the Shared-AE model over other approaches lies in ability effectively disentangle shared and private features, resulting in a clearer separation of modality infor- mation. This distinct separation not only enhances the interpretability of the latent representations but also supports a wider range of downstream tasks across different data modalities. As shown in T when the shared features across both modalities are weak, both PSID and DPAD struggle to effectively separate the private features from the shared latent spaces. Here, both models resulted in an erroneously high decoding accuracy of a private feature from the shared latent space. Shared-AE was able successfully decouple the shared private features in its latent subspaces.
22

---

Published as a conference paper at ICLR 2025
Prediction from behavioral latents neural activity
keypsint
MDSTL
FAPI_A
MOp1A
0.3
0.5
D2
0.2
0.5
0.1
0.3
0.0
0.2
0.1
0.30
11
0.50
0.25
0.4
0.20
F1
0.35
0.15
0.30
0.10
0.25
0.20
0.1
0.00
ACAe1_R
PLLR
RSP_R
0.30
0.20
0.25
0.30
D5
0.25
0.15
0.10
0.05
0.05
0.15
0.05
-0.05
0.
V5.
MOB
NO.1
0.25
0.40
08
0.20
0.25
0.15
0.30
0.20
0.10
0.25
0,20
0.15
0.05
0.15
6.10
0 00
-0.05
Respond to right spout
Never in
Respand the spout
Left paw, raine
Right, pare mide
Respond to the right spoit
lever
Respend left spout
paae, race
Right pave ruse
Respand right spous
lever, in
Respond the lott spout
Left naw. ralse
Right, pawe raise
Prediction accuracy of neural activity (B): comparison between the image behavioral latents and key point positions behavioral latents
Figure 12:
A.12.3 BASELINE COMPARISON WITH BASELINE MODEL WITHOUT CSEDIVERGENCE
As baseline, we trained the model on simulated dataset using the same architecture but without applying CS-divergence. As shown in Table 7 the results indicate lack of separation between the private and shared latent spaces, highlighting the necessity of constraints to effectively distinguish shared and private features. Importantly, the shared subspaces in the Shared-AE are not in fact reconstructing both modalities; they remain completely separate from each other, with purely the CS CS-divergence regularize them to be similar.
23

---

Published as a conference paper at ICLR 2025
Prediction of Behavior using Neural Activity
0.8
12-
10-
0.6
0.4
0.2
04-
02-
0.0
-0.2
-02
[yl
Inf
0.6
1.2
75
1.0
0.-
0.4
0.4
0.0
0.2
0.0
Lever
Spout in:
response right
Spout Int
respomse feft
Left paw raise
Right patel ralle
Lever in
Spout in:
response right
Spout in:
respanse left
Left pawe raiie
Right pase
rever (x)
postion activty)
took: modslwwth semovioral activity
latent back: modsluwth keypolint postion, behavloaal and activity
0.25
14
0.00
-0.25
IPM noths
Figure 13: Prediction of keypoint positions using shared neural latents shows that incorporating the third modality (behavioral image) enhances the neural latents' ability to capture intricate features.
Shared latent: time series
Shared latent: image
Decoding sccuracy: 0.173x 0.04
Decoding accuracy. 0.112-1.001
Latent dimension 2
Latent dimension
Noise Shape
Latent dimension 1
Latent dimension 1
Private latent: time series
Private latent: image
Decoding accuracy: 0.640 0.24
Decoding accuraty 0.503
Latent dimension
Latent dimension
50
100
118
Latent dimension
1
Latent dimension
Scale Orientations
Figure 14: PCA embeddings for shared and private behavioral and neural latent subspace: A. Shared latent spaces for each modality, each point represents single frame, color-coded by the correspond- ing private features: the private latent spaces retains the private features. B. Private latent for each modality, color-coded by orientations and scales.
24

---

Published as a conference paper at ICLR 2025
Table 4: Shared-AE unpaired vs paired results: The shared latent results in better classification accuracy in shared features while private latent retained more modality-ppecifie information. 2. Shufiling one of the modalities during test time does affect latent classification accuracy which indicates that there is no information leakage between modalities.
Shared-AE (unpaired)
Shared-AE (paired)
Private time series latents on scale and o Shared time series latents scale and orientationst
0.647 + 0.05
0.640 E 0.04
0.980 + 0.09
0.981 + 0.01
Private time series latents on shapel.
0.017 + 0.001
0.008 + 0.001
Shared time series latents shapel
-0.012 + 0.001
-0.015 + 0.01
Private time series latents on semporally-structreed noise
0.732 + 0.012
0.745 + 0.005
Shared time series latents on temporally noise.
0.183 + 0.01
0.173 0.04
Private image latents on scale and orientations.
0.523 = 0.02
0.53 = 0.05
Shared image latents on scale and orientationst
0.757 0.08
0.751 0.07
Private image latents shapet
0.768 = 0.03
0.767 + 0.09
Shared image latents on shapel
0,288 + 0.01
0.289 + 0.008
Private image latents temporally--strucurred noisel
0.012 1 0.05
0.011 + 0.08
Shared image latents on temporally-structuree noise
0.015 + 0.01
0.015 + 0.007
Table 5: Baseline model performance on unpaired and paired datasets: the drop on classification indicates that there information leakage during training
Baseline comparison for unpaired dataset
MM-VAE
Joint encoding model
Shared time series latents on scale and orientations Shared time series latents shape
0.52 + 0.21
0.58 + 0.1
-0.01 0.04
0.013 + 0.04
Shared time series latents on cemporally-structur noise
-0.001 + 0.007
0.034 + 0.08
Shared image latents on scale and orientations
0.42 0.11
0.55 0.11
Shared image latents on shape
0.146 + 0.001
0.010 + 0.01
Shared image latents on temporally- structured ennporall-struccturedd
noise
-0.003 + 0.007
0.042 0.07
Baseline comparison for paired dataset
MM-VAE 0.90 = 0.06
Joint encoding model
Shared time series latents scale and orientations
0.975 + 0.01
Shared time series latents on shape
0.22 + 0.03
-0.012 + 0.01
Shared time series latents lemporally-structured noise
0.0005 + 0.000
0.024 + 0.001
Shared image latents on scale and orientations
0.90 = 0.06
0.646 + 0.001
Shared image latents on shape
0.154 = 0.003
0.861 0.003
Shared image latents temporally-structure noise
0.003 + 0.000
0.071 + 0.003
Table 6: Decoding accuracy for temporally-structureed noise (private time-series feature) simu-
lated dataset
Subspace
PSID
DPAD
Shared-AE
Shared latent..
0.99 + 0.00
0.99
0.00
0.015 + 0.01
Private image latent|
NA
NA
0.012 + 0.05
Private time-series latent
NA
NA
0.732 + 0.00
A.13 ADDITIONAL RESULTS: headfixed dataset
A.13.1 BASELINE MODEL CONFIGURATION
In MM-GG-VVE the dimensions of both the shared and private latent spaces are set to 50, similar to configurations in MM-VAE VAE and the Joint Encoding model. For PSID: m1 5, n.r 50, 2. For DPAD: nl = 5, næ 50 and the method code is Mc These consistent latent space dimensions across models provide standardized basis for comparison, ensuring that the
25

---

Published as a conference paper at ICLR 2025
Table 7: Baseline model without CS-divergence vs. Shared-AE: performance simulated paired datasets
Decoding latents and targets
Baseline
Shared-AE
Shared time series latents scale and orientationst
-0.036 + 0.01
0.981 + 0.01
Shared time series latents on shape 4
-0.012 0.01
-0.015 0.01
Shared time series latents
-trpuotulyy noise +
-0.018 0.005
0.173 0.004
Shared image latents on scale and orientations?
0.67 0.04
0.751
Shared image latents shape L Shared image latents on remporally-strcctur
0.52 = 0.005
0.289 0.008
noise.
0.035 + 0.003
0.015 + 0.007
Private time series latents on scale and orientations, Private time series latents on shape & Private time series latents on rmmpraall-structuree noise f
-0.030 0.01
0.674 :.05
-0.023 + 0.01
0.017 0.001
-0.004 0.001
0.732 0.0012
Private image latents scale and orientations.,
0.70 0.02
0.523 + 0.02
Private image latents on shape f
0.78 0.005
0.768 + 0.03
Private image latents camporaaly-sstrucuure noise.
0.075 + 0.006
0.012 + 0.05
Reconstructed image while changing B
Prediction of Neural Activity using Behavion: Visual area: keypoint position behavioral later Image behavioral latents
Original image the value of shared behavioral latent space
0.35
movement
0,30
3BW m
RF 0,20
0,15
eot movemen
0.10
0.05
novement
Lever in
response right
Spout in:
Spout in:
eepoossele left
Left paw raise
Right paw raise
Minimum shared latent Maximum movement
HMM
Figure 15: Head-fixed mouse: behavioral video and WFCI A. Latent traversal for the shared behavioral latents shows automatic identification of neurally reelevant features such jaw and paw movement. B. Neural activity prediction: comparison between the video-based behavioral latents and the keypoint behavioral latents shows that there is videos than poses in select brain regions. Full list shown in Appendix A.II
shared information about behavioral
evaluation focuses on differences in model design and methodology rather than variations in latent space capacity.
1.1 BASELINE COMPARISONS
Five baseline models are applied to the 2AFC simultancously collected neural WFCI and behavioral video data for comparison: PSID Sani et al. (2021). DPAD Sani et al.] (2024). MM- VAE Shi et al. (209)), the Joint Encoding Model /Singh Alvarado et al. (021), and MM-GP- Gondur et al. 2024). Importantly, the first four baseline models failed to separate the latent space into distinct shared and private latent subspaces. In order to quantify the amount of information represented in each latent subspace, we compared the ability of each model to reconstruct one of the modalities, the body positions, in Tables I and 12 Shared-AB achieves higher decoding accuracy, particularly on unpaired datasets, compared to existing models. This demonstrates that it does not suffer from modality leakage, ensuring that during inference, robust representations can be generated even only one modality's data is available. This is particularly advantageous in situations where acquir- ing all modalities simultaneously is challenging. The ability to produce meaningful latents without the need for all modalities underscores the practical utility and ficxibility of approach. We em- phasize that Shared-AE offers significantly improved interpretability of the latent space. Moreover, unlike PSID, Shared-AE is capable of handling image data and can accommodate more than three modalities, providing greater flexibility for complex multimodal tasks.
The model proposed by Gondur al. (2024) combines the behavioral and the neural latent into common subspace. This can be detrimental to interpretability since it is unclear which modality is leading to the performance of the shared latent. As detailed in Appendix A.3 the combined shared
26

---

Published as a conference paper at ICLR 2025
latent causes information bleeding between different modalities. We implemented MM-GP-VA. EAE based on Gondur et al. (2024) as well as a similar model by incorporating a single shared latent space for both modalities and tested it the 2AFC dataset, referring to this implementation as MM-nonGP- VAF. These models were used as baselines to compare against our proposed approach. As shown in Table 8 Shared-AF outperforms both of these models both paired and unpaired tasks. Critically, Shared-AE has the ability to generate reasonable latents despite corrupted data in the other modality during inference.
Table 8: Behavioral decoding accuracy with unpaired modalities for 2AFC body position dataset MM-nonGP-VVAE Shared-AE
tasks
MM-GP- GA-
Private latent
0.01 0.00
0.25 = 0.02
0.22 - 0.03
Shared latent
0.006 = 0.00
0.23 = 0.03
0.41 = 0.05
Table 9: Behavioral decoding aceuracy with paired modalities for 2AFC body position dataset
tasks
MM-GP T-M-GP-VR VAE MM-nonGP-VAN Shared-AE
Private latent
0.37 + 0.00
0.24 + 0.02
0.22 E 0.03
Shared latent
0.36 + 0.01
0.32 + 0.03
0.41 + 0.05
For comparison with PSID, we further validate that PSID and DPAD cannot effectively handle im- age data, we conducted an experiment where the flattened behavioral image was used as the input behavioral data for PSID and DPAD (Table TOp. We then used the generated neural latent represen- tations for body position decoding and compared the results with those obtained from Shared-AE, which directly utilizes the behavioral image as input. The decoding results obtained by Shared-AE outperformed the rest.
Table 10: Behavioral decoding accuracy with paired modalities for 2AFC image dataset
Subspace
PSID
DPAD
Shared-AE
Shared latents
0.32 0.02
0.31 + 0.00
0.38 + 0.01
Table 11: Behavioral decoding accuracy with unpaired modalities for 2AFC body position dataset
Subspace
MM-VAF
Joint Encoding Model MM-GP-VAL
Shared-AF.
Private latents
NA
NA
0.01 0.00
0.22 + 0.03
Shared latents
0.22 + 0.03
0.20 + 0.02
0.006 + 0.00
0.41 + 0.05
Table 12: Behavioral decoding accuracy with paired modalities for 2AFC body position dataset
Subspace
PSID
DPAD
MM-VAL
Joint Encoding Model
MM-GP-VAL
Shared-AE
Private latents
NA
NA
NA
NA
0.37 + 0.00
0.22 + 0.03
Shared latents
0.20 + 0.03
0.27 + 0.02
0.41 0.07
0.40 + 0.06
0.36 + 0.01
0.41 + 0.05
A.13.3 UMAP EMBEDDINGS OF THE PRIVATE LATENT SUBSPACE
We show that the private latent subspaces cannot capture different HMM states inferred by behav- ioral latents (Fig. 16).
A. 13.4 INFLUENCE OF THE PRIVATE BEHAVIORAL. LATENT SUBSPACE ON IMAGE RECONSTRUCTION
We performed the same task in Sec. 4.2 varying the private behavioral latent from its minimum value to its maximum value. (Fig.
27

---

Published as a conference paper at ICLR 2025
Privts reser intert spaces
Petate batrrdere spocns
UMAP: UMlor Colorroooddd coded by HMM motils
latent embettling dim
Istere embeldiing dim
Lewer in
Spouti in resparse right
5 ppout in response left
Left para raiee
pow raise
Figure 16: UMAP embeddings for private behavioral and neural latent subspace for the 2AFC dataset.
Reconstructed image while chaning the value of shared latent space
Original image
shape
image defective
Minimum shared latent Maximum shared latent
Figure 17: Changing the value of the private latent space from its minimum its maximum results in
a defection in the image. The private latent captures more about the behaviorally unrelated features.
28


Reviews:
[{"review_id": "HMLEmBkMum", "reviewer": "Reviewer_8fed", "summary": "This paper develops a method for learning shared and private\nembeddings between two or more sources of information (modalities).\nThey apply the algorithm to two problems with behavioral and neural\ndata as well as an illustrative artificial data problem.", "questions": "Questions\n\nHow does this work compare to the ICLR 2024 paper of Gondur, Sikandar,\nSchaffer, Aoi, and Keeley (Multi-modal Gaussian Process Variational\nAutoencoders for neural and behavioral data)?  That paper also has a\nshared multi-modal embedding and separate (within modality) embeddings\nand has also been used for complex behavioral tasks ( hawkmoth\ntracking a moving flower and limb movement of drosophila with\nsimultaneous neural recordings).\n\nThe paper says that you\texamined the influence of latent dimensions on reconstruction\naccuracy.  Was that using the test data\tor some\tseparate data?\t(If the\ttest data, how do\nyou justify that?)   How are the other parameters set? -\tthe paper is vague on this.\n\nPlease elaborate on why equal latent subspace dimensions are required.", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "Strengths\n- The paper is pretty clearly written\n\n- The artificial data problem helps to clarify how the algorithm works\n\n- The real world experiments are\timportant to demonstrate practicality in this important area", "weaknesses": "Weaknesses\n\n- Missing comparison to an important related ICLR24 paper (see questions below).    If the authors can compare to results from that algorithm, or otherwise justify the superiority of this method (or superiority in some settings/situations),  I would likely improve my rating.  \n\n- Unclear how parameters were set  (see questions below).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SVk1wutSMi", "reviewer": "Reviewer_QBbN", "summary": "The paper proposes an autoencoder-based framework for finding shared and private subspaces of multimodal neural and behavioral data. Their goal is to separate the shared subspace from the modality-specific subspaces. To do so, constraints (Cauchy-Schwarz divergence) are added on the distribution of subspaces to encourage/discourage their alignment for this particular goal. This method is evaluated on one simulated dataset as well as two distinct experimental datasets from mice.", "questions": "My questions are the ones raised in weaknesses and Minor weaknesses/questions.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1.\tThe paper is well-presented. The goal is clear and many analyses including both simulated datasets and two experimental datasets with different behavioral complexities are analyzed. \n2.\tVarious analyses such as connecting behavioral variates to corresponding brain area through shared latents, and investigating the difference between modeling behavior as markers or raw videos in the learned shared features.", "weaknesses": "1.\tMethodological novelty seems minimal. The authors note “a novel regularization term designed to identify features common to both behavior and neural activity” as their main methodological novelty. However, a very similar regularization scheme has been previously proposed by Yi et al. (2022). The difference between Shared-AE and this work is not adequately discussed making the methodological novelty unclear. Also, the idea of using CS-divergence instead of standard VAE with KL-divergence is not novel either as previously proposed by Tran et al. (2021).\n\n2.\tThere are numerous methods on neural-behavioral modeling and finding shared vs. private subspaces, none of which are compared to and many which are not discussed. In general, the manuscript seems to mix up unsupervised latent variable models of neural data with latent variable models of neural-behavioral data in its discussions and writeup. The only neural-behavioral data discussed in Related work (but not compared to) is Schneider et al 2023. Another neural-behavioral model that is cited is Sani et al 2021, but it is not discussed or compared to, and is instead grouped with an unsupervised model of neural data. Another neural-behavioral model in Zhou and Wei 2020 is also simply cited but not compared to. The authors need to separate the neural vs. neural-behavioral models in their manuscript and provide sufficient discussion of differences between other neural-behavioral models with theirs. Comparison to these neural-behavioral models is also needed to show the advantage of this method. In addition to the above cited works, there are also some other very relevant neural-behavioral models that are not cited, for example:\n\nGondur et al. 2024: This work appeared in the previous ICLR and has a very similar architecture designed for the same purpose using GP-VAE. However, it is not cited, and the key differences are not discussed. Given how closely related this method is to the authors’ work, it can serve as a baseline. \n\nHurwitz et al. 2021: This work proposes a sequential VAE for modeling neural-behavioral data. This needs to be cited and discussed.\n\nSani et al. 2024: This work proposes an RNN-based architecture that separates shared/private subspaces in neural-behavioral data and needs to be cited and discussed. \n\n3.\tEffect of novel terms in the loss i.e., the CS-divergence and their inverses are not assessed. As this is the main addition to a standard multimodal AE architecture in this work, it is crucial to evaluate whether presence of each term contributes to current results or not. Even without these constraints, the reconstruction loss itself can enforce shared vs private subspaces (at least to some extent) as the shared ones reconstruct both modalities whereas private ones reconstruct the specific modality alone.\n\n\n4.\tLack of baseline comparison in real data analysis. The same baselines used in simulated dataset (Shi et al (2019), Singh Alvarado et al. (2021)) are not shown in real data. Additionally, there are several relevant works on neural-behavioral modeling some of which could be used as baseline to better assess what benefits Shared-AE adds as mentioned in item 2 above. \n\n5.\tThe authors claim that their framework is better for more complex/social behavior types than Schneider et al. (2023). But what about all the other neural-behavioral models? Is shared-AE more suitable for complex behavior than others and if so why? This claim does not seem convincing without further comparisons. \n\n6.\tI find calling this method unsupervised very misleading. In the context of neural-behavioral modeling, supervision typically means use of behavior for guiding behavior-related features of neural activity. In this sense, the proposed approach is fully supervised not only during learning but also during inference, putting it in the multimodal family. The manuscript refers to this method as unsupervised throughout the paper including in the title. This needs to be corrected.  \n\n7.\tThe model uses hyperparameters $(\\alpha, \\beta, \\gamma, \\delta)$ to control the contribution of regularizations to the overall loss. However, the effect of these hyperparameters on the results are not investigated. Authors note that the results are robust to changes in the hyperparameters, but I did not find the results that show this robustness. Please provide these.\n\n\nMinor weaknesses/questions \n\n1.\tWhy does the method need to learn two separate shared latents? It seems these two should ideally correspond to the same thing. Why not have a single shared latent which is used in both decoders? \n\n2.\tIn the unpaired analysis, is shuffling happening across time? Why does maintaining performance in this scenario indicate avoiding modality leakage?\n\n3.\tWhat is the basis for choosing the state dimension based on Fig. 8? Why are the reconstruction performance vs dimension so noisy in Fig. 8?\n\n4.\tWhat does min/max R2 refer to in Fig. 8?\n\ntypographical errors:\n\n-\tLine 302: Fig. 4.1 => Fig. 4? \n-\tLine 302: missing space between “data” and “(Fig”\n-\tFig. 7 caption includes panels E-F while the results are missing.  “E-F: Prediction accuracy for neural activity and behavior under different distance groups.” It seems these panels are not included.\n-\tCaptions for panels B and C of Fig. 10 do not match. It seems the order is wrong.\n-\tFig 11 has very tiny titles\n-\tLine 1002: reference to Fig. A.9.3 is incorrect.\n\nReferences:\n\nDaiyao Yi, Simon Musall, Anne Churchland, Nancy Padilla-Coreano, and Shreya Saxena. Disentangled multi-subject and social behavioral representations through a constrained subspace variational autoencoder (cs-vae). bioRxiv, 2022. doi: 10.1101/2022.09.01.506091. URL https: //www.biorxiv.org/content/early/2022/09/05/2022.09.01.506091.\n\nLinh Tran, Maja Pantic, and Marc Peter Deisenroth. Cauchy-schwarz regularized autoencoder, 2021. URL https://arxiv.org/abs/2101.02149.\n\nYuge Shi, N. Siddharth, Brooks Paige, and Philip H. S. Torr. Variational mixture-of-experts autoencoder for multi-modal deep generative models, 2019. URL https://arxiv.org/abs/1911.03393.\n\nJonnathan Singh Alvarado, Jack Goffinet, Valerie Michael, William Liberti, Jordan Hatfield, Timothy Gardner, John Pearson, and Richard Mooney. Neural dynamics underlying birdsong practice and performance. Nature, 599(7886):635—639, November 2021. ISSN 0028-0836.\n\nSteffen Schneider, Jin Hwa Lee, and Mackenzie Weygandt Mathis. Learnable latent embeddings for joint behavioural and neural analysis. Nature, 617: 360–368 May 2023. ISSN 1476-4687.\n\nRabia Gondur, Usama Bin Sikandar, Evan Schaffer, Mikio Christian Aoi, and Stephen L Keeley. Multi-modal gaussian process variational autoencoders for neural and behavioral data. In International Conference on Learning Representations, 2024.\n\nCole Hurwitz, Akash Srivastava, Kai Xu, Justin Jude, Matthew Perich, Lee Miller, and Matthias Hennig. Targeted neural dynamical modeling. Advances in Neural Information Processing Systems, 34:29379–29392, 2021.\n\nOmid G Sani, Hamidreza Abbaspourazad, Yan T Wong, Bijan Pesaran, and Maryam M Shanechi. Modeling behaviorally relevant neural dynamics enabled by preferential subspace identification. Nature Neuroscience, 24(1):140–149, 2021.\n\nDing Zhou and Xue-Xin Wei. Learning identifiable and interpretable latent models of high-dimensional neural activity using pi-vae.  Advances in Neural Information Processing Systems, volume 33, pp. 7234–7247,  2020. \n\nOmid G Sani, Bijan Pesaran, and Maryam M Shanechi. Dissociative and prioritized modeling of behaviorally relevant neural dynamics using recurrent neural networks. Nature Neuroscience, 27: 2033–2045, 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tNhrlWMWkD", "reviewer": "Reviewer_h2Dv", "summary": "This paper contributes to a growing literature on learning shared representations for multimodal data in neuroscience, where many researchers are interested in learning joint representations of brain data and behavior. Whereas many previous methods have focused on learning shared latent representations by combining latents learned from individual modalities, this work further partitions each modality's latent space into a private latent spaces and a shared latent space, which is linearly mixed with other modalities' shared latents. This separation is engendered by the use of a Cauchy-Schwarz Divergence for aligning shared latents and separating shared from private latents. Experiments on one synthetic and two real data sets show suggestive links between brain data and behavior, though what one is to make of these is a bit unclear. Moreover, the technical contribution of the paper is perhaps small when considered in light of other work cited.\n\nIn all, this is a solid paper that is, in my view, below the bar for acceptance without a more substantial technical contribution.", "questions": "## My key question:\n- What is the technical innovation between this and the two other Yi papers? Is it just a different application?\n\n## Small points of clarification:\n- ll. 108-120: the ending here is a bit vague; would help to clarify what these sorts of models would miss in more complex tasks (and hopefully show in experiments)\n- ll. 141-152: Model scales as number of pairs of modalities; probably not a limitation in practice, but a few words about scaling might help.\n_ l. 165: what is $s'_t$ here? Is it the same as $s_t^{pre}$?\n- l. 171: what is $y'$? Is the prime a typo?\n- ll. 162-168: It would be nice to have a diagram of this, since one could easily lose track of the different linear models: If I understand correctly: $s^{pre}$ is a linear function of each modality's latents, and $s^i$ is a linear function of both modalities' _shared_ latents.\n- l. 196 In Eq 2, how well does this estimator do in moderate-sized latent spaces? Is it a reasonable estimator? One is effectively saying that the density belongs to a reproducing kernel Hilbert space, right?\n- ll. 259-264: This description is a bit terse and may be hard to follow for readers (like me) who were not familiar with this dataset. I realize details are in the supplement, but the main text should be a bit more self-contained.\n- ll. 318-321: Why, exactly, do we need a strong separation between modalities? What is the use case? I realize it affects decoding performance (e.g., Figure 3) but what might we use this analysis to conclude in an experiment?\n- Figure 4A: Sorting the rows and columns by some sort of biclustering algorithm might make the correlation structure more apparent. This matrix plot is not very compelling as presented.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- Principled approach to structuring latent spaces based on a desired semantics: some information in each latent space is common to all modes, some is private.\n- The need for interpretable joint encodings is of high interest in neuroscience.\n- CS-Divergence is a reasonable means of effecting the separation of subspaces, and the authors have chosen a pretty reasonable-seeming method of approximating this quantity.\n- The experiments on real data use challenging datasets that encapsulate many challenges faced by the community.", "weaknesses": "- It is somewhat unclear what the technical innovation in this paper is beyond the Yi et al. preprint cited, as well as a similar paper by Yi and Saxena at EMBC in 2022 [1]. Both of those works use the same CS divergence setup as here, and I am struggling to see where the technical innovation is (though the application is somewhat different). I don't see the strength of the experimental results here being novel or interesting enough on their own to justify acceptance without an additional technical advance.\n- The authors use a latent space partition that is distinct from the Whiteway et al. paper but somewhat related to the Sani et al. work they cite. I realize that the PSID paper is linear, but the Shanechi group also has work on nonlinear methods that preserve this kind of partition (the most recent of which was likely unpublished when this work was submitted) that should probably come in for a fuller discussion.\n- In the framing of this work, I don't believe I fully understood the authors' rationale for needing shared vs. private subspaces. It's conceptually interesting, but the experiments simply focus on decoding. In what circumstances do we need such a partition, and what is the sign that not having it is hurting us scientifically? If this were clearer, I think it would be easier for readers to judge the success of the experiments. \n- The figures are all quite small and cramped, making them somewhat hard to parse. It's not always clear what the \"win\" is with the experiments.\n\n[1]: D. Yi and S. Saxena, \"Modeling the behavior of multiple subjects using a Cauchy-Schwarz regularized Partitioned Subspace Variational AutoEncoder (CS-PS-VAE),\" 2022 44th Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC), Glasgow, Scotland, United Kingdom, 2022, pp. 497-503, doi: 10.1109/EMBC48229.2022.9871466.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZBybjklixb", "reviewer": "Reviewer_HB99", "summary": "This paper introduces a latent-space disentangling autoencoder to identify shared and private latent features from multi-modal neural and behavioral data. Proposed disentanglement is based on a Cauchy-Schwarz divergence based regularizer applied between different components (private and shared features) of the latent representations obtained via behavioral and neural encoders. Both inter and intra modality regularization losses are combined in addition to the standard autoencoder loss. Experimental analyses are performed first on a simulated dataset, and then on different complex behavioral datasets with neural recordings from mice.", "questions": "- The overall idea of disentangling the latent representation space using inter- and intra-modality loss regularizers has been previously explored in several works. There are also actually works proposing a similar autoencoder regularization framework in other settings [Tran et al. \"Cauchy–Schwarz Regularized Autoencoder\", JMLR 2022]. Perhaps one question that the authors should clarify with a clear statement is their methodological ML novelty (i.e., if the proposed regularized training scheme is completely novel, or if the paper only contains a strong empirical novelty).\n\n- Majority of the results show strong consistency between the disentangled latent features extracted from behavioral and neural data. Regarding the latent space visualizations (UMAP etc), how did the authors determine the latent dimensionality in each experiment? How consistent are these results with respect to changing this dimensionality?\n\n- The dataset retrieved for the 2AFC experiments seems rather small in terms of the number of trials. Also it seems to be divided only once into a train/test split. Therefore, I would ask if the authors performed any CV of the model training process, and evaluate the significance of their results in that sense?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Consistent experimental findings based on a simple disentangled representation learning model with a tailored training objective.\n- It has a unique empirical strength with a focus on neural data analysis from complex multi-modal social behavior experiments.", "weaknesses": "- Some of the empirical results need further validation, considering the details present in the Appendix.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)