Disentangling 3D Animal Pose Dynamics with Scrubbed Conditional Latent Variables
Authors: Joshua Huang Wu
Hari Koneru
James Russell Ravenel
Anshuman Sabath
James Michael Roach
Shaun Sze-Xian Lim
MICHAEL R TADROSS
Alex H Williams
Timothy W DUNN
Venue: ICLR 2025 Poster (2025)
Keywords: behavioral neuroscience
systems neuroscience
Primary area: applications to neuroscience & cognitive science

Abstract: Methods for tracking lab animal movements in unconstrained environments have become increasingly common and powerful tools for neuroscience. The prevailing hypothesis is that animal behavior in these environments comprises sequences of discrete stereotyped body movements ("motifs" or "actions"). However, the same action can occur at different speeds or heading directions, and the same action may manifest slightly differently across subjects due to, for example, variation in body size. These and other forms of nuisance variability complicate attempts to quantify animal behavior in terms of discrete action sequences and draw meaningful comparisons across individual subjects. To address this, we present a framework for motion analysis that uses conditional variational autoencoders in conjunction with adversarial learning paradigms to disentangle behavioral factors. We demonstrate the utility of this approach in downstream tasks such as clustering, decodability, and motion synthesis. Further, we apply our technique to improve disease detection in a Parkinsonian mouse model.

Published as a conference paper at ICLR 2025
DISENTANGLING 3D ANIMAL POSE DYNAMICS WITH SCRUBBED CONDITIONAL LATENT VARIABLES
Joshua H Wu
Hari Koneru' J Russell Ravenel' Anshuman Sabath' James M Roach
Shaun SX Lim
Michael Tadross' Alex H Williams Timothy W Dumn 1*
Duke University
NNw York University Flatiron Institute
ABSTRACT
Methods for tracking lab animal movements in unconstrained environments have become increasingly common and powerful tools for neuroscience. The prevailing hypothesis is that animal behavior in these environments comprises sequences of discrete stereotyped body movements (" ("motifs" "actions"). However, the same action can occur at different speeds or heading directions, and the same action may manifest slightly differently across subjects due to, for example, variation body size. These and other forms of nuisance variability complicate attempts quantify animal behavior in terms of discrete action sequences and draw meaningful comparisons across individual subjects. To address this, present framework for motion analysis that uses conditional variational autoencoders in conjunction with adversarial learning paradigms disentangle behavioral factors. We demonstrate the utility of this approach in downstream tasks such as clustering, decodability, and motion synthesis. Further, we apply our technique improve disease detection in a Parkinsonian mouse model.
INTRODUCTION
Animal behavior is largely expressed through movement (Musall et al., 2019; Bernstein, 1967). Thus, the ability to identify variation in animal movement is critical for understanding neural (dys) (dys)ffnnttion (McCullough & Goodhill, 2021). In recent years, pose estimation has become popular for behavioral measurement (Dunn et al., 2021; Nath et al., 2019; Zimmermann et al., 2020; Bala et al., 2020; Pereira et al., 2022), enabling more objective and comprehensive kinematic profiling. Given the increasing throughput and dimensionality of behavioral measurement assays (Nourizonoz et al., 2020; Bialek, 2022; Marshall al., 2022), the ability to learn meaningful and interpretable representations of behavior is essential (Datta et al., 2019).
common analysis goal is to cluster or segment recurring and stereotyped pose sequences- as running, turning, and rearing- and compare their frequencies and transition statistics across experimental conditions (Berman et al., 2014; Weinreb et al., 2023; Wiltschko al., 2020; 2015b; Marshall et al., 2022). Currently, most approaches are purely unsupervised and can be sensitive to nuisance variability, such as animal individuality, or over-segment due to continuous variability, such as speed (Van Dam et al., 2023; Costacurta et al., 2022). As result, these methods often produce action spaces in which true biological signals are uninterpretably entangled with confounding variables uninformative features. For example, the same fundamental action (c.g., walking) can be performed with different levels of vigor (fast vs. slow) and with different directionality (walking straight vs. slightly turning left right). All of these variants are typically split into different action clusters (see Marshall et al., 2022), introducing extrancous statistical structure that may obscure more meaningful biological signals.
In principle, action spaces could be made invariant to nuisance variables via disentanglement of specified factors (Shu al., 2019), thus permitting the identification and analysis of meaningful behavioral dynamics (Shi et al., 2021; Whiteway et al., 2021; Yi & Saxena, 2022). However, existing disentanglement approaches have shortcomings. The increasingly common conditional variational autoencoder (C-VAE) approach learss accurate disentangled representations of the conditional
"Correspondence (joshua.we wu, timothy. motthy.dunn] @čuke. edu, alex. h.w. 1 aama@nyuuee

---

Published as a conference paper at ICLR 2025
variable, such that changing it while holding the other latent variables fixed generates outputs varying only in the conditional factor (Petrovich et al., 2021b; Guo et al., 2020; Khemakhem et al., 2020). However, we will show that the other latent variables are not necessarily, to the conditional factor. Adversarial approaches can be used to learn disentangled subspaces with specific invariances (Ding al., 2020; Sanchez al., 2020; Zhao et al., 2021; Brakel & Bengio, 2017), but these methods rely on secondary neural networks that can be difficult to train and sensitive architectural and hyperparameter choices (Arjovsky & Bottou, 2017: Radford, 2015). Furthermore, adversarial neural networks often target fully disentangled representations, e.g.. via mutual information minimization (Belghazi et al., 2018: Cheng et al., 2020), but certain nuisance variables, like speed, have some dependencies with action understanding (e.g., subject will always move farther during locomotion than when grooming). Fully disentangling such variables into invariant subspaces would destroy relevant behavioral variation. Ideally, the degree of disentanglement could be easily specified and controlled.
typically, invariant
To address these challenges, present scrubbed C-VAEs (SC-VAE) for learning disentangled motion features from 3D pose sequences (Fig. Ia). Given a variable to disentangle (calculated deterministically
known priorí), SC-VAE seeks to learn latent representation in which the variable cannot be decoded by specified linear or nonlinear functions, called "scrubbers". SC- VAE achieves this by conditioning VAE on disentanglement variables while adversarially penalizing the encoder based on a scrubber's ss ability to estimate variable information from the latent representation. As result, we obtain variational distribution with direct control over the level of contribution from nuisance variability (Fig. lb).
We show that scrubbing enhances latent representations by encouraging nuisance-invarian clusters and improving the consistency of the conditional generative model. We also demonstrate SC-VAE's ability to enhance and disentangle biological phenotypes from nuisance factors in mouse model of Parkinson's 's disease, illustrating how VA-VAE can be used to further neurobehavioral inquiry. When examining different choices of scrubbing, we find that: 1) neural network scrubbers built from existing adversarial disentanglement methods are less reliable than scrubbers with explicit parametric assumptions, and 2) assumptions of linear or nonlinear disentanglement can impact downstream analyses due to the level of dependence between disentangled variables and behavioral understanding.
a.
Scrubbed C-VAE Overview
b. Scrubber Removes Unwanted Latent Structure
Linearly Scrubbed
Noniinearly Scrubbed
3D Pose Sequence
X
Broser
Representation
so00 Goog
PCA Visualization of Latents
Kinemotio
Z
Neypore Foottnn
PO1
PO1
FC1
vatlable
concer
Scnubbe
Sacobir
PC
Tirs
PC1
PC1
Figure 1: Overview of SC-VAE. (a) SC- VAE encourages disentangled representations by directly concatenating known factors to latent variables in conditional VAE. (b) A "scrubber" removes information about these factors from the latent variables, such that factors are unable be explained by specified linear or nonlinear functions.
2 PRIOR WORK
Models of animal movement Finescale manual annotation (e.g., rears, turns) of animal behaviors time-intensive and subjective. Further, many experimental manipulations-suchh as new phar- macological treatments-induus unknown behavioral phenotypes (Nestler & Hyman, 2010). Thus, most research has gravitated towards unsupervised, data-driven methods. Popular approaches apply dimensionality reduction and clustering to
pose sequences (Berman et al., 2014; Marshall et al., 2021) or auto-regressive hidden Markov models (ARHMMs) to model action motifs discrete states with intra-state linear dynamics (Wiltschko al., 2015a; Weinreb al., 2023).
2

---

Published as a conference paper at ICLR 2025
More recently, deep neural networks have been leveraged to learn low-dimensional representations of behavior. Due to challenging perspective and illumination effects in videos, many behavioral analysis methods use pose trajectories- and increasingly 3D poses (Marshall et al., 2022)-222 imput (Dunn et al., 2021; Nath et al., 2019; Zimmermann et al., 2020; Bala et al., 2020; Pereira et al., 2022). For example, VAME. generates single-frane frame pose embeddings from variational autoencoders nonlinear srature-learning step before fitting an ARHMM (Luxem et al., 2022). BAMS, on the other hand, generates representations by predicting distributions of future kinematies (Azabou et al., 2024).
Our study builds on these works by applying constraints on representational spaces to reduce the contribution of nuisance effects and provide control over these behavioral representations. Our motivation is perhaps most similar to previous work by Costacurta et al. (2022), who developed time-warped extensions to the ARHMM to separate changes in speed from other behavioral factors. Despite similar motivations, our method aims to be more general tool for disentanglement which learns -ow-iimesiiona feature spaces robust to a wider range of factors and that could be used for additional tasks beyond time series clustering.
Disentangled representations Although deep networks have the capacity to capture complex nonlinear dynamics, the feature spaces identified by deep networks are not always immediately interpretable and prone to learning undesirable representations due to dataset-specific idiosyncrasies. This unpredictability in learning deep representations has limited their use in the exploratory manner required by experimental research (von Ziegler et al., 2021).
A broad literature on the use of deep networks to extract disentangled representations of complex, high-dimensional datasets seeks to address these challenges (Gilpin et al., 2018). While many have investigated the possibility of fully unsupervised disentanglement with VAE-like architectures (e.g.. BAV and its variants) (Higgins et al., 2017: Chen et al., 2018; Esmaeili et al., 2019), doing so challenging in practice. Further assumptions are usually needed to theoretically guarantee identifiability and disentanglement (Locatello et al., 2019), leading to increasing popularity in weakly supervised disentanglement (Shu et al., 2019; Khemakhem et al., 2020).
Neuroscientists have applied and adapted ideas from this literature to account for nuisance variability in raw video data, such as changes in illumination or slight differences in camera angles over days (Whiteway et al., 2021; Shi et al., 2021; Yi & Saxena, 2022). Here we focus on 3D pose data and aim to disentangle additional forms of nuisance variation, such as speed and angular heading, which are cheaply computable from the inferred pose tracks, and animal identity (a type of batch effect). While we focus on these specific variables here, our approach should generalize to other nuisance factors in the future. Prior neuroscience applications of disentanglement have shown success in identifying subspaces for factors. However, they have not explored the invariance over those factors in opposing subspaces.
Our approach is in line with the formalization of disentanglement by Shu et al. (2019) as both the ability of: 1) a disentangled dimension to accurately represent factor, and 2) the corresponding null space to be invariant to that factor. Although there exist disentanglement methods in other application fields that apply both properties (Zhao et al., 2021; Sanchez al., 2020; Ding al., 2020), these methods rely on the specification of secondary networks to remove residual factor information in representation space. often via adversarial training (Ganin & Lempitsky. 2015b; Brakel & Bengio, 2017; Cheng et al., 2020). Here, we contribute novel strategies that: 1) bypass the need for specifying architectures or hyperparameters for adversarial networks, and 2) are more specific about the degree of disentanglement when variables are not perfectly independent of behavioral understanding.
METHODS
3.1
POSE PREPROCESSING
We apply our method to 3D body pose sequences extracted from raw videos. This preprocessing step removes several confounding variables, such as differences in illumination across recording sessions. However, some residual sources variability remain, such as differences in body size across subjects. Typical methods of pose preprocessing involve translating and rotating all postures to a root position and direction (i.e., "pointing north") ") (Marshall et al., 2021; Luxem et al., 2022). However, important trajectory and turning information are lost when this alignment is performed on
3

---

Published as a conference paper at ICLR 2025
each frame independently. To avoid this loss of information, apply single rotation and translation to all frames in a pose sequence to align the pose at a central time point. That is, given a window translate and rotate each posture such that the central
of time around central reference frame, frame is at a specified position and orientation (Fig. 4). In a subset of experiments, we refrained from rotating the pose entirely to demonstrate the ability of our method disentangle the heading direction and recapitulate a representation commensurate with the fully processed data.
In contrast to some existing methods (e.g.. Luxem al., 2022), we represent pose using joint angles between keypoints positioned at fixed distances from their neighbors. Intuitively, this representation forces any pose dynamics model obey the physical constraints of the body skeleton, leading to cleaner generative samples (see discussion in Marshall et al., 2021; Huang et al., 2017; Vemulapalli et al., 2014). Optimizing over raw angular variables is complicated by discontinuities; thus, we follow well-established protocol from the computer vision community for lifting 31D rotation matrices into a continuous 6D space (Zhou et al., 2019). Each posture composed of K joint angles is then represented as a (6K + 3)-dimensional vector, where the three additional dimensions come from the global position of the root keypoint. Altogether, each pose sequence is denoted x, E IR L +3) for a sequence over L frames centered at frame t.
3.2 DISENTANGLED REPRESENTATIONS AND CONDITIONAL VAE ARCHITECTURE
Disentangled variables Our core goal is to learn a low-dimensional representation of behavior where simple factors (e.g., speed, heading direction, and animal identity) are put into orthogonal dimensions from factors that learned in data-driven, unsupervised fashion. In essence, this would separate the semantic content of actihns-whether whether the animal is running, digging, rearing, etc.-from the character of these differently when turning.
the animal is running fast vs. slow oriented
We formalize this goal as follows. We assume that for any pose sequence x, we can compute vector of nuisance factors v, E IR-, Loosely, we will say that latent representation 2, taking values in R² disentangled with respect to v, when one can identify two orthogonal subspaces 2, and z such that can be reliably inferred from z but not reliably inferred from Z,. In other words, the goal is to have z capture known behavioral factors, while z, reflects unknown behavioral factors that are learned from the data with minimal redundancies to z!-
Variational autoencoders (VAEs) The backbone of our method is VAE with ID convolutional layers (Kingma & Welling, 2013). Adopting standard notation, we use an encoder network go to map pose sequences onto a distribution over latent variables sampled via reparameterization trick, Z; (. x.). The pose sequence is then reconstructed by a decoding network which evaluates the likelihood po(x 2.). In a standard VAE, the parameters ( 0} are adjusted with respect to the evidence lower bound (ELBO):
LELBO (Ø, 0) = -Ex, [Ez,ngs(-Ixe) [log (x., 2:) - log (z xx]]]
(1)
Conditional VAEs (C-VAEs) An established way to promote disentanglement in supervised fashion is to use a conditional VAE (Sohn et al., 2015; Khemakhem et al., 2020). C-VAEs are a popular approach for controllable human motion synthesis, as they can generate realistic body pose sequences that correspond to categorical actions (e.g., throwing. kicking, or lunging, Guo et al., 2020; Petrovich et al., 2021a; Gu et al., 2024). C-VAEs have also been exploited within neuroscience (Wu et al., 2020), but to our knowledge, have not been leveraged for behavioral representation learning.
Here, latent variables are sampled from conditional prior distribution that depends on another variable Vt. which is observed separately or derived deterministically from x₁, In our case, V: will correspond to a vector representation of the animal's running speed, heading direction, encoding of the animal's identity. Exact implementations for C- VAES vary. We choose a simple approach of concatenating V, onto the sampled output of the encoder network. That is, sample
one hot
2: ~ (- xx) while z is strictly defined by vt. A new vector ă, = [z, vz] is passed to the decoding network. The C-VAE is then trained with respect to the same ELBO loss, which is now interpreted as a lower bound of the expected conditional marginal likelihood, conditioning with respect to v,, For full details on model architectures, see C.I.
Intuitively, since z, and v, are both passed to the decoder, any information about v, that is encoded by 2, is redundant and inefficient. Thus, if the latent dimension D is small, one expects that the C-VAE
4

---

Published as a conference paper at ICLR 2025
will learn disentangled representation. While this intuition is bomn out to certain extent in practice, our experience (documented below in Sec. 4) shows that a C-VAE will often learn a distribution over z that is only partly disentangled from v,. Further, choosing a smaller latent dimension D can be expected to result in more efficient coding and disentanglement but less expressivity and trainability. Thus, a vanilla C- C-V seems impose trade-off between faithful modeling of the data and learning disentangied latents.
3.3 SCRUBBING OUT RESIDUAL INFORMATION
A core component of our approach is to augment the C-VAE loss function to reduce depen- dence between 2, and Vs. This is analogous to minimizing mutual information (MI), [(z,, V.) Ez.vv [log(p(z,, v=)/p(z)))) ( [v()))]] Although MI maximization is commonly used to generate repre- sentations with shared features between paired samples (Hjelm et al., 2018; Bachman et al., 2019; Zhao et al., 2021; Sanchez et al., 2020), there are generally no tractable upper bounds for minimization (Poole et al., 2019). In the context of behavioral analysis, our objective is to create a representation space in which v, is "hidden" from downstream analyses with constrained expressiveness (c.g.. linear regression or clustering via a Gaussian mixture model). We therefore substitute MI with predictive information (Xu et al., 2020b):
( vo) max [Exx,v (Ezergo())ss) [logp(ve [x(
where (+) is potentially nonlinear function family parameterized by v. and log p(v. [s)())) is the log-likelihood of v; conditioned on sufficient statistics f (zz). Given an approximate maximizer ei of eq. (2), we promote disentanglement by introducing an adversarial penalty Lscuu (Ø) = L (zz, Vz). We then seek to jointly minimize LELBO (1) 0) + S (ω) over the model parameters {0) 0} for
some user-specified hyperparameter À > 0.
Thus, (x) is adversarial decoder that aims to maximize the log-likelihood of va given z, This penalty term is minimized when the encoder network, parameterized by Ø. encodes a variational dis- tribution over z, with no decodable information about v,. This would imply perfect disentanglement with respect to the class of functions parameterized by th. Mutual information is recovered in the limit that (.) defines an arbitrarily complex probability density (see Xu et al., 2020b), but we will see that it can be advantageous to constrain f to a more limited class of functions (e.g., linear).
The effect of the penalty term AL (Ø) is to "scrub out" information about v, contained in 2,, and we therefore call (-) a "sccubber." The overall model architecture is diagrammed in Fig. la. However, is challenging to efficiently solve the inner maximization problem over si in eq. (2). We now describe several strategies to overcome this challenge.
3.4 SCRUBBING WITH ADVERSARIAL NEURAL NETWORKS
A classic approach to adversarial training is to define fg to be a neural network trained jointly with the C-VAR. We implement two such methods here for disentangling behavioral representations.
Scrubbing with gradient reversal layers (CC-VAE-GR First, we apply an existing method that popular the context of domain adaptation in deep networks (Ganin & Lempitsky, 2015a) and applied to disentanglement (Ding et al., 2020). Where (+) is multi-layer perception (MLP)
ensemble, the idea is to estimate the gradient of:
Ex,,y. [Ba-gg(-)xx) [logp(v; [z:)))
with respect to e and ó over a minibatch of pose sequences. Then, we simultancously perform gradient ascent and descent updates ú and o, respectively. This is akin to jointly training the model over {1/, Φ. 0} but with the gradients reversed for tb.
Scrubbing with neural discriminators (SC-VAE-ND) An alternative approach investigated by Sanchez et al. (2020), Brakel & Bengio (2017), and Kim & Mnih (2018) draws inspiration from generative adversarial networks (GANs; Goodfellow et al., 2020). Given latent sample [2; v:]" T and define an adversarial discriminator, fe(.), to classify
we generate a "fake" sample [z; V/Hs] real from fake representations. By minimizing the cross-entropy loss of this classifier, the mutual
5

---

Published as a conference paper at ICLR 2025
information between Zy and Ve is minimized (Brakel & Bengio, 2017). Unlike for the rest of the methods in this paper, (t) and the C-VAE trained in alternation for SC-VAE-ND.
Challenges These strategies can work if one carefully tunes separate learning rates for ub and f 0}. However, found that fine-tuning these hyperparameters was, perhaps unsurprisingly, challenging in practice (Shen et al., 2020; Scaramuzzino et al., 2023). When not delicately tuned, & fails to track the distribution of z, induced by parameter updates to the encoder qs. Thus, (.) will be suboptimal adversary, and (.) will not be tight to (.). Furthermore, these approaches have strong assumptions of nonlinear disentanglement, which may not be appropriate in cases where the scrubbed variable is not independent (see Sec. 4.2 and 4.4). In the following methods, we navigate these challenges by: adaptively tuning learning rates, 2) applying more direct estimations of &b or (-)- which do not require specifying network architectures and their hyperparameters, and 3) introducing scrubbers with more constrained (e.g.. linear or quadratic) assumptions of scrubbing.
3.5 SCRUBBING UNDER PARAMETRIC CONSTRAINTS
Linear scrubbing by moving average least squares (SC-VAE-MALS) If we constrain f to be a linear function and use a Gaussian likelihood to model vr. then the maximization over tb is simply an ordinary least squares problem. The optimal prediction be calculated in closed-form by the normal equations (see e.g., Murphy, 2012, Ch. 7):
(w) = Elvz ] (F[zz) [ + 3I) 31) 2₁
where 3 is the 12-regularization constant. Thus, to evaluate (ω) for a linear model, we simply need to keep a running estimate of two matrices: v/0 = E[vz' and vpr E|zz"] J. We do this by computing exponential moving averages (EMA) of these quantities, replacing expectations with empirical averages. Given a single data instance xt, we sample v,, Zz ~ g-(- xt), and update our estimate of the sufficient statistics accordingly:
30 f a)v + +a09)
(5)
C1 [11 ()22)) + ano2)
(6)
where is the smoothing factor of this EMA filter. In practice, compute these updates over minibatch with multiple data samples. We forgo manual tuning of a by running two EMA filters in parallel with separate smoothing factors small, fixed difference. On each minibatch, we increment
or decrement the smoothing factors based on which filter provides a better fit to the minibatch statisties (see A.2 for details).
Extension to scrubbing by polynomial regression The method above immediately extends to polynomial regression by augmenting z₁ to include higher-order polynomial features obtain quadratic (-MAQS) and cubic (-MACS) scrubbing. This enables progressive nonlinear scrubbing.
Categorical serubbing with adaptive quadratic discriminators (SC-VAE-QD) The EMA filter can also be readily applied to other scrubbing models beyond ordinary least squares. When v, categorical variable encoding animal identity, we treat fz,) as a prediction of the class label. A simple approach is to keep rumning estimate of the class-conditional means, Ev=e[2] and covariances, Covv=e [2, z). and then use a quadratic discriminant analysis (QDA) classifier (z) = p(v, = =//z)) (see A.2). We update the mean and covariance estimates in the same way as eq. (5) and eq. (6) with the self-tuning smoothing factor.
Nonlinear scrubbing with kernel density mutual information estimators (SC-VAE-MI) The constrained adversarial maximization over (-) is related to classical quantities in information theory. In particular, if the class of functions parameterized by 1/ contains all possible models, then minimizing LScrub is equivalent to minimizing the mutual information between Z, and V; (Xu et al., 2020a). Above, we built scrubbers assuming a parametric form for fe(.)) (e.g.. linear or quadratic), but we can also develop a scrubber that estimates the mutual information in a nonparametric fashion. We investigated simple approach of using Gaussian kernel density estimation to approximate the joint probability density p(z,, vt). Given this approximation of the density, provides an estimate of the mutual information (Moon
ī(z. vt) = Ex.,, [log(p(z, vt) /(z( (pv:))) (t)(p(z))pvv.)) et al., 1995). It is well-known that mutual information estimators suffer curse of dimensionality;

---

Published as a conference paper at ICLR 2025
to combat this, large batch size (2048 pose sequences) and tune the bandwidth parameter of the Gaussian kernel to compromise between estimator bias and variance. This results in a biased estimate of mutual information, which we identify with Lserah(
4 RESULTS
4.1
DATA
We used data from 4 C57BL/6 mice freely exploring a 30 cm² open field (324k frames per mouse. 1 hour at 90 fps). For applications, we used n = 36 C57BL/6 mice recorded in 1-hour sessions (90 fps) in this same open field before and after Parkinson's 's discase (PD) induction (6-OHDA unilateral injection model). 3D poses were tracked with DANNCE (Dunn et al., 2021). In each animal, we also performed fluorescent immunostaining of the striatum bilaterally and quantified the extent of dopaminergic neurodegeneration as the ratio of integrated fluorescence between the lesioned to non-lesioned hemispheres (more details in D).
4.2 DISENTANGLING NUTSANCE VARIABLES
For all results, we trained separate models to isolate the effects of each scrubber on each nuisance variable. For implementation details, see and
Heading direction We first tested heading direction disentanglement in models trained without rotationally processed data as described in Section 3.1. As the optimal representation can be derived by fitting VAE to fully processed pose sequences (VAE Processed), heading disentanglement serves as an ideal baseline to validate our scrubbing methods. When examining a vanilla VAE trained on raw (not rotationally processed) ) data, heading direction was, unsurprisingly, linearly decodable from the latents (Fig. 2a). Conditioning the latent variable was not sufficient to disentangle heading direction from other latent factors; C-VAE conditioned heading direction formed representations that were structured with heading direction in a similar way to the vanilla VAF, despite being redundant with the conditioned variable. Although OC-VAE-GR and -ND nonlinear adversaries (i.e., an MLP), we found they were unreliable in producing representations that were even linearly invariant heading. SC-VAE-MALS was found to more reliable in achieving linear disentanglement by directly estimating a linear adversary.
Although SC-VAE-MALS was successful in linear disentanglement, an optimal representation arguably should have near complete disentanglement (beyond linear) of heading direction since the facing direction of behavior does not affect semantic label.' Scrubbing by quadratic regression (SC-VAE-MAQS) or mutual information estimation (SC-VAE-MI) reached nonlinear decodability commensurate with the rotationally preprocessed optimum (VAE Processed). Thus, both SC- VAE- MI and -MAQS exhibit strong linear and nonlinear disentanglement, We then tested SC-VAE disentanglement on speed and animal identity: nuisance variables whose influences on behavioral representations cannot be trivially removed via data preprocessing.
Heading Direction
Average Speed
Animal Identity
Livear
Chence
Logaac
QDA
VIR
CAR
VAE-GR
MARGR
SCHRRSMM MRE-MAL MALS
SCAAE MAGE
VAR-A A0
VIE
CAAE
SCAAEOR
SCV4E-0
SC MAGS
3C MAR-4
VAS.
C-MAE
SCAMEIOR
SC
SCAAE-D
Figure 2: Decoding nuisance variables from scrubbed latents. R2 for linear and nonlinear (MLP) regression on (a) speed and (b) heading from latents. (c) Classification accuracy for animal identity using logistic regression and quadratic discriminants.
IImportanll, we will argue below that this logic does not apply to some other behavioral variables, such as the animal's speed, which are nor independent of action categories. In these cases, fully nonlinear scrubbing may lead to undesirable performance, but constrained (e.g., linear) scrubber can provide partial disentanglement.
7

---

Published as a conference paper at ICLR 2025
Speed Unlike with heading direction, speed is naturally entangled with behavioral factors e.g.. rearing and rumning exhibit inherently different speed profiles. The generative model defined by the VAE ought to reflect this dependence, some nonlinear dependence between speed and the unsupervised latents should be preserved and not "scrubbed out". On the other hand, the same action be performed different speeds, so it would be desirable least partially disentangle speed from other variables. Trends for linear and nonlinear disentanglement across models for speed scrubbing were similar to our heading results (Fig. 2b). SC-VAE-MAQS was the exception where it did not affect speed disentanglement. Although SC-VAE-MI continued to exhibit strong disentanglement, this came at the expense of having interpretable clusters of actions within the latent space. We further explore the subtleties of linear vs. nonlinear speed disentanglement in Section 4.4.
Animal identity Small differences in subject limb proportions or sizes can affect behavioral kine- matics and postures in complex nonlinear ways, even in pose preprocessing without explicit notions of limb lengths (e.g., ours described in Sec. 3.1). These differences cause trivial separations of the same behaviors across individuals. When scrubbing animal identity, SC-VAE representations were also the most linearly and nonlinearly invariant (Fig. 2c). As with heading direction and speed, neural network scrubbers (SC-VAE-GR and -ND) were less effective than using a specified nonlinear scrubber (SC-VAE VAE-QD) for identity disentanglement.
Scrubbing effects evident when visualizing latent representations (Fig. Ib, 6). Vanilla VAE and C-VAE representations were highly entangled with nuisance variables across principal axes, with regions dominated by specific speeds, heading directions, animal identities. The scrubbing models showed representations with more mixing across nuisance variables. SC-VAE-MI, for heading and speed, and SC-VAE-QD, for identity, produced the most "salt and pepper" representations.
Table 1: Effects of scrubbing on motion synthesis. Rº of heading or speed of generated behaviors with respect to random heading or speed conditioned inputs to the decoder.
Model
Heading Direction R
Average Speed R
C-VÆE
0.502
0.635
SC-VAE-GR
0.957
0.654
SC-VAE-ND
0,079
0.626
SC-VAE-MALS
0.983
0.683
SC-VAE-MAQS
0.989
0,648
SC-VAE-MI
0.994
0.607
4.3 CONSISTENT MOTION SYNTHESIS
To evaluate whether our models in Section 4.2 maintained valid representations when scrubbing heading or speed decodability, we conditioned their decoders on random speeds and heading directions and evaluated the resulting speeds and headings of the generated sequences (Table 1). We found that scrubbing improved the consistency of conditionally generated sequences, suggesting new strategies for improving conditional motion synthesis.
4.4 CLUSTERING DISENTANGLED REPRESENTATIONS
When clustering with Gaussian mixture models (GMMs), found that the anisotropies due to nuisance variability in vanilla VAE and C-VAE representations determined the makeup of behavioral clusters, as they were comprised largely of behaviors with similar speeds or heading directions (Fig. 3a, e) as opposed to typical behaviorally relevant clusters (Fig. 7). Thus, these nuisance variables drive trivial segmentation of behavioral space that obscures biologically meaningful behavioral identification. Scrubbing the C-VAE representations progressively reduced the influence of nuisance variables, increasing the variance of heading direction and speed within GMM clusters (Fig. 3b, f top). In the heading direction test case, SC-VAE-M completely climinated nuisance clustering and produced fully neading-mixet behavioral clusters. (Fig. 3d).
However, we found that MI scrubbing for speed was too strong and produced clusters longer associated with typical coarse behavioral labels (e.g.. walking, rearing, grooming). To further investi- gate, we trained second vanilla VAE and labeled six walking clusters with differing distributions of speed (Fig. 3e). When comparing the latents in other models associated with these labeled walking clusters, the distributions of walking behaviors in the SC-VAE-MALS and SC- VAE--E- representa-
8

---

Published as a conference paper at ICLR 2025
a. Heading Wallong Clusters from Vanilia VAE
b.
Heading GMM Cluster Clicular Variance
Log Liceroood
CAMl
SC-ANF-BR
SCAWE-DD
SC SCAME-MAOS
SCUME-
Wi Pryersss
1-Component Gauesian Log Likelhood
Heading
Heading from Clustering Walking Behaviors
SCAREWALLS
SCHAE--4
WI Processed
Headlng (nwd)
Walking Clusters Vanilia VAE
SC-VAEMI
Speed Walking Clusters Vanila VAE
Speed GMM Cluster Variance
0.2
a1-
Waiing
Log Lebooo
1 J-Component Gaussian Log Likelihood
Speed (mnefrmne)
Cuperr 9.
Spead from Clustering Waiking Rehaviors
Speed mowhtmme
Figure 3: Examining the effects of scrubbing on clustering. (a) Heading direction distribution in walking GMM (k = =50) clusters identified from a separate VAE trained not rotationally preprocessed data. (b) (top) Weighted average circular variance of heading by cluster. (bottom) 5-fold cross log-likel liglliill of diagonal Gaussian component the normalized latents of each model corresponding to the clusters in (a). (c) Histograms of heading in GMM clusters of VAE, SC-VAE-MALS, -MI, and VAE Processed latents corresponding to the clusters in (a). (d) Example walking clusters in VAE (left) and SC- VAE-VI (right) after scrubbing heading. (e) Speed distribution in walking GMM (k 50) clusters identified from a separate VAE. (f) (top) Weighted average variance of speed by cluster. (bottom) 5-fold cross log-liielihood of a diagonal Gaussian component on the normalized latents per model corresponding to the clusters in
(e). (g) Histograms of average
speed in GMM (k- 3) clusters of VAE, SC-VAE-GR, -MALS, and -MI latents corresponding to the clusters in (c).
tions were better captured by a single cluster than other models (Fig. 3f bottom), demonstrating how scrubbing can combine behaviors previously distinguished only by nuisance variation (see B.6 for details this metric). Further clustering of these walking behaviors in SC- completely speed-mixed clusters, whereas the separation was still clearly present in vanilla VAE. Repeating this with heading scrubbing (separately identifying six walking clusters with different heading distributions). SC-VAT VAE-MI was found to have the most ideal walking representation (Fig. 3c, d). These results suggest the need for disentanglement methods with weaker (-MALS) and stronger (-MI) assumptions for disentanglement which reflect the level of independence between nuisance variables and behavioral understanding.
revealed
4.5 NEUROLOGICAL DISEASE ANALYSIS
Subject- idiosyncrasies increase variability in behavioral expression across animals, potentially obscuring the effects of neurohehavioral perturbations. In our PDD dataset, found that serubbing
9

---

Published as a conference paper at ICLR 2025
animal identity attenuated this variability, enhancing the detection of behavioral shifts in disease (PD) sessions. Using maximum mean discrepancy (MMD; 6.6; Gretton et al., 2012; Goffinet et al., 2021) to measure behavioral distribution similarity between recording sessions, we observed large increase in similarity within conditions and a decrease between conditions for the SC-V OC-VAE-QD model (Fig. 9). That is, given disease session moved closer to the other disease sessions and farther from the healthy sessions, suggesting improved discriminability of disease-associated behavioral changes. We quantified this using an effect size metric, calculated as the MMD between subject's healthy and disease sessions relative to the MMD between healthy sessions across animals (Table 2 left, B.6). We also used k-NN classifier predict the type of session (healthy vs. disease) given behavioral representation (Table 2 middle). SC-VAE-QD outperformed other models on these metrics. As control, we also scrubbed the healthy vs. disease session label itself ('Reverse Control'), which resulted in significant decrease in all disease discriminability metrics.
Striatal immunostaining revealed another potential source of PD phenotype variability: not all animals experienced the same degree of dopaminergic denervation. More intact dopaminergic circuity should be associated with fewer behavioral changes between healthy and PD sessions (Boix et al., 2015; Slézia et al., 2023). Thus, we expect behavioral representations better capturing these changes to be more correlated with the extent of denervation. While the healthy- -iieease MMD was correlated for all models, SC-VAB-QD strengthened the correlation (Table 2 right).
Table 2: Scrubbing identity in a Parkinsonian mouse dataset. Impact of identity scrubbing
quantified as effect size (d) mean + SEM across animals, healthy vs. disease classification accuracy
(%), and the correlation (Pearson r) of MMD behavioral change to dopamine denervation extent.
Model
Effect Size d' f
Accuracy (%) f
Denervation correlation (r) f
VAE
1.07 = 0.05
81.3
0.208
C-VAE
1.11 0.05
82.2
0.221
SC-VAE-GR
1.17 = 0.06
83.3
0,249
SC-VAE-QD
1.71 = 0.11
85.8
0.406
Reverse Control
0.68 = 0.03
65.2
0.086
5 CONCLUSION AND LIMITATIONS
This study presents a novel framework for disentangling nuisance behavioral or experimental variables by removing (i.e., "scrubbing") variable information from latent spaces via adversarial learning objectives. Our adaptive moving average, quadratic, and mutual information scrubbers are successful in producing behavioral representations that are linearly or nonlinearly invariant to nuisance variables, reducing behavioral over-segmentation and enhancing the interpretability of latent spaces. Existing approaches based on gradient reversal and adversarial networks generally only assume nonlinear serubbing and, in experience, difficult to fine-tune. In contrast, our scrubbing approach using exponential moving averages of sufficient statistics is hyperparameter-free.
A limitation of convolutional encoder-decoder architecture is that it represents fixed-length pose sequences as a single latent code, which constrains feature timescales and does not scale efficiently to long sequences. This issue could be addressed in future work by adopting a dynamic latent variable model. Additionally, our nuisance variable scope can be expanded to include factors on finer spatial and temporal scales, such as per frame or individual body part kinematies. Although all of the models presented in this paper were trained while scrubbing one nuisance variable at time, SC-VAE can, in principle, be trained to jointly condition on multiple nuisance variables. This could increase the purity of behavioral representations, enable multifactor motion synthesis, and permit direct comparisons of nuisance variable effects within common space. Future work should explore the effects of scrubbing multiple nuisance variables, either jointly or individually.
Finally, we have focused on scrubbing out nuisance variables that are easily available (e.g., speed, heading direction, and animal identity), but it may not be obvious how to identify analogous variables or determine the correct level of disentanglement in more complex datasets (e.g., social). Future work could focus scrubbing in exploratory analyses, such as fine-tune uulle-ensupervised models. This work and future work seek to address the growing necessity for predictable and interpretable deep representation learning methods for neuroscientific research.
10

---

Published as a conference paper at ICLR 2025
6 ACKNOWLEDGEMENTS
J.H.W. W. acknowledges support from the National Science Foundation Graduate Research Fellowship under Grant No. DGE 2139754 and the supplemental INTERN Grant No. DGS 2139754 Amendment 007. M.R.T. acknowledges support from the National Institutes of Health (NIH) grant R34DA059512, and W.D.D. from NIH grants R34DA059512 and ROIGM136972.
ETHICS STATEMENT
Animal care and use: The care, procedures, and experimental manipulation of all animals were reviewed and approved by Duke University Institutional Animal Care and Use Committee. See D for further details. Declaration of interests statement: T.W.D. is co-founder of dannce.ai.
REFERENCES
Martin Arjovsky and Léon Bottou. Towards principled methods for training generative adversarial networks. arXiv preprint 10Xiv:1701004800 2017.
Mehdi Azabou, Michael Mendelson, Nauman Ahad, Maks Sorokin, Shantanu Thakoor, Carolina Urzay, and Eva Dyer. Relax, it doesn't matter how you get there: A new self-supervised approach for multi-timescale behavior analysis. Advances in Neural Information Processing Systems, 36, 2024.
Philip Bachman, R Devon Hjelm, and William Buchwalter. Learning representations by maximizing mutual information across views. Advances in neural information processing systems, 32, 2019.
Prancct C Bala, Benjamin R Eisenreich, Seng Bum Michael Yoo, Benjamin Hayden, Hyun Soo Park, and Jan Zimmermann. Automated markerless pose estimation in freely moving macaques with openmonkeystudio. Nature communications, 11(1):4560, 2020.
Mohamed Ishmael Belghazi, Aristide Baratin, Sai Rajeswar, Sherjil Ozair, Yoshua Bengio, Aaron Courville, and R Devon Hjelm. Mine: mutual information neural estimation. arXiv preprint 1oXiv:/80100000 2018.
Gordon Berman, Daniel M Choi, William Bialek, and Joshua W Shaevitz. Mapping the stereotyped behaviour of freely moving fruit flies. Journal of The Royal Society Interface, 21099)20014062. 2014.
Nicholas Bernstein. The coordination and regulation movements. (No Title), 1967.
William Bialck. On the dimensionality of behavior. Proceedings of the National Academy of Sciences, 119(18):c2021 18002188111011 19, May 2022. Publisher: Proceedings of the National Academy of Sciences.
Jordi Boix, Thomas Padel, and Gesine Paul. A partial lesion model of parkinson's 's disease in mice- characterization of a scohda-induces medial forebrain bundle lesion. Behavioural brain research, 284:196-208 2015.
Philemon Brakel and Yoshua Bengio. Learning independent features with adversarial nets for non-linear ica. arXiv preprint arXiv: 17X1v:1100.05050, 2017.
Ricky TQ Chen, Xuechen Li, Roger Grosse, and David K Duvenaud. Isolating sources of disentanglement in variational autoencoders. Advances in neural information processing systems, 31, 2018.
Pengyu Cheng, Weituo Hao, Shuyang Dai, Jiachang Liu, Zhe Gan, and Lawrence Carin. Club: A contrastive log-ratio upper bound of mutual information. In International conference machine learning, pp. 1779-1788. PMLR, 2020.
Julia Costacurta, Lea Duncker, Blue Sheffer, Winthrop Gillis, Caleb Weinreb, Jeffrey Markowitz, Sandeep R Datta, Alex Williams, and Scott Linderman. Distinguishing discrete and continuous behavioral variability using warped autoregressive hmms. In Advances in Neural Information Processing Systems, volume 35, pp. 23838-23850 Curran Associates, Inc., 2022.

---

Published as a conference paper at ICLR 2025
Sandeep Robert Datta, David J Anderson, Kristin Branson, Pietro Perona, and Andrew Leifer. Computational neuroethology: a call to action. Neuron, 104(1):11-24, 2019.
Zheng Ding, Yifan Xu, Weijian Xu, Gaurav Parmar, Yang Yang, Max Welling, and Zhuowen Tu. Guided variational autoencoder for disentanglement learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 7920-7929, 2020.
Timothy W Dunn, Jesse D Marshall, Kyle S Severson, Diego E. Aldarondo, David GC Hildebrand, Selmaan N Chettih, William Wang, Amanda Gellis, David Carlson, Dmitriy Aronov, et al. Geometric deep learning enables 3d kinematic profiling across species and environments. Nature methods, 18(5):564-55 473,, 2021.
Babak Esmaeili, Hao Wu, Sarthak Jain, Alican Bozkurt, Narayanaswamy Siddharth, Brooks Paige, Dana H Brooks, Jennifer Dy, and Jan-Willem Meent. Structured disentangled representations, In The 22nd International Conference on Artificial Intelligence and Statistics, pp. 2525-2534. PMLR, 2019.
Yaroslav Ganin and Victor Lempitsky. Unsupervised domain adaptation by backpropagation. In Francis Bach and David Blei (eds.), Proceedings of the 32nd International Conference on Machine Learning, volume 37 of Proceedings of Machine Learning Research, pp. 1180-1189, Lille, France, 07-09 Jul 201 5a. PMLR.
Yaroslav Ganin and Victor Lempitsky. Unsupervised domain adaptation by backpropagation. In International conference on machine learning, pp. 180-1189. PMLR, 2015b.
Leilani H Gilpin, David Bau, Ben Z Yuan, Ayesha Bajwa, Michael Specter, and Lalana Kagal. Explaining explanations: An overview of interpretability of machine learning. In 2018 IEEE Sth International Conference on data science and advanced analytics (DSAA), pp. 80-89. IEEE, 2018.
Jack Goffinet, Samuel Brudner, Richard Mooney, and John Pearson. Low-dimensional learned feature spaces quantify individual and group differences in vocal repertoires. Elife, 10:e67855, 2021.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Communications of the ACM, 63(11):139-14) 144, 2020.
Arthur Gretton, Karsten M Borgwardt, Malte J Rasch, Bernhard Schölkopf, and Alexander Smola. A kernel two-sample test. The Journal of Machine Learning Research, 13(1):723-773, 2012.
Chunzhi Gu, Jun Yu, and Chao Zhang. Learning disentangled representations for controllable human motion prediction. Puttern Recognition, 146: 146:199998, 2024.
Chuan Guo. Xinxin Zuo, Sen Wang. Shihao Zou, Qingyao Sun, Annan Deng, Minglun Gong, and Li Cheng. Action2motion: Conditioned generation of 3d human motions. In Proceedings of the 28th ACM International Conference Multimedia, pp. 2021-2029, 2020.
Chengan He, Jun Saito, James Zachary, Holly Rushmeier, and Yi Zhou. Nemf: Neural motion fields for kinematic animation. Advances Neural Information Processing Systems, 35:4244 4256, 2022.
Irina Higgins, Loic Matthey, Arka Pal, Christopher P Burgess, Xavier Glorot, Matthew M Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with constrained variational framework. ICLR (Poster), 3, 2017.
Devon Hjelm, Alex Fedorov, Samuel Lavoie-Marchilee Karan Grewal, Phil Bachman, Adam Trischler, and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization. arXiv preprint arXiv: 1888, 06670, 2018.
Zhiwu Huang, Chengde Wan, Thomas Probst, and Lue Van Gool. Deep learning lie groups for skeleton-base action recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 6099-6108, 2017.
12

---

Published as a conference paper at ICLR 2025
llye Khemakhem, Diederik Kingma, Ricardo Monti, and Aapo Hyvarinen. Variational autoencoders and nonlinear ica: A unifying framework. In International Conference on Artificial Intelligence and Statistics, pp. 2207-2217. PMLR. 2020.
Hyunjik Kim and Andriy Mnih. Disentangling by factorising, In International conference on machine learning, pp.
PMLR, 2018.
Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv: 14, 2013.
arXiv preprint
Francesco Locatello, Stefan Bauer, Mario Lucic, Gunnar Raetsch, Sylvain Gelly, Bernhard Schölkopf, and Olivier Bachem. Challenging common assumptions in the unsupervised learning of disentan- gled representations. international conference on machine learning, pp. 4114-4124. PMLR, 2019.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint 17Xiv:1711.05000 2017.
Kevin Luxem, Petra Mocellin, Falko Fuhrmann, Johannes Kürsch, Stephanie R Miller, Jorge Palop, Stefan Remy, and Pavol Bauer. Identifying behavioral structure from deep variational embeddings of animal motion. Communications Biology, 5(1):1267, 2022.
Jesse D Marshall, Diego E Aldarondo, Timothy W Dunn, William L Wang, Gordon Berman, and Bence P Ölveczky. Continuous whole-body 3d kinematic recordings across the rodent behavioral repertoire. Neuron, 109(3))420-437) 2021.
Jesse D. Marshall, Tianqing Li, Joshua H. Wu, and Timothy W. Dunn. Leaving flatland: Advances in 3D behavioral measurement. Current Opinion in Neurobiology, 73:102522, April 2022. ISSN 1873-6882.
Michael H McCullough and Geoffrey J Goodhill. Unsupervised quantification of naturalistic animal behaviors for gaining insight into the brain. Current Opinion in Neurobiology, 70:89-100, 2021.
Young-Il Moon, Balaji Rajagopalan, and Upmanu Lall. Estimation of mutual information using kernel density estimators. Physical Review E, 52(3):2318, 1995.
Kevin P Murphy. Machine learning: a probabilistic perspective. MIT press, 2012.
Simon Musall, Matthew T. Kaufmmn, Ashley L. Juavinett, Steven Gluf, and Anne K. Churchland. Single-trial neural dynamies are dominated by richly varied movements. Nature Neuroscience, 12100):66771666, October 2019. ISSN 1546-1726 Group.
Number: 10 Publisher: Nature Publishing
Tanmay Nath, Alexander Mathis, An Chi Chen, Amir Patel, Matthias Bethge, and Mackenzie Wey- gandt Mathis. Using DeepLabCut for 3D markerless pose estimation across species and behaviors. Nature Protocols, 14(7):2152-2117) July 2019, ISSN 1750-2799.
Eric Nestler and Steven Hyman. Animal models neuropsychiatric disorders. Nature neuro- science, 13110): 2010.
Ali Nourizonoz, Robert Zimmermann, Chun Lum Andy Ho, Sebastien Pellat, Yannick Ormen, Clément Prévost-Solié, Gilles Reymond, Fabien Pifferi, Fabienne Aujard, Anthony Herrel, and Daniel Huber. EthoLoop: automated closed-loop neuroethology in naturalistic environments. Nature Methods, 17510):1052-100 October 2020. ISSN 1548-7105.
Talmo D Pereira, Nathaniel Tabris, Arie Matsliah, David M Turner, Junyu Li, Shruthi Ravindranath, Eleni S Papadoyannis, Edna Normand, David S Deutsch, Z Yan Wang, al. Sleap: A deep learning system for multi-animal pose tracking. Nature methods, 19(4):886-49) 2022.
Mathis Petrovich, Michael J. Black, and Gül Varol. Action-conditioned 3d human motion synthesis with transformer vae. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 10985-10995, October 2021a.
13

---

Published as a conference paper at ICLR 2025
Mathis Petrovich, Michael J Black, and Güi Varol. Action-conditiones 3d human motion synthesis with transformer vae. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 10985-10995, 2021b.
Ben Poole, Sherjil Ozair, Aaron Van Den Oord, Alex Alemi, and George Tucker. On variational bounds of mutual information. In International Conference on Machine Learning. pp. 5171-5180.
PMLR, 2019.
Alec Radford.
Unsupervised representation learning with deep convolutional generative adversarial 2015.
networks. arXiv preprint
Eduardo Hugo Sanchez, Mathicu Serrurier, and Mathias Ortner. Learning disentangled representations via mutual information estimation. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXII 16, pp. 205-221. Springer, 2020.
Giovanna Scaramuzzino, Federico Becattini, and Alberto Del Bimbo. Auribute disentanglement with gradient reversal for interactive fashion retrieval. Pattern Recognition Letters, 172:203-212, 2023.
Jiayi Shen, Xiaohan Chen, Howard Heaton, Tianlong Chen, Jialin Liu, Wotao Yin, and Zhangyang Wang. Learning minimax optimizer: pilot study. In International Conference on Learning Representations, 2020.
Changhao Shi, Sivan Schwartz, Shahar Levy, Shay Achvat, Maisan Abboud, Amir Ghanayim, Jackie Schiller, and Gal Mishne. Learning disentangled behavior embeddings. Advances in neural information processing systems, 34:22562-22573, 2021.
Rui Shu, Yining Chen, Abhishek Kumar, Stefano Ermon, and Ben Poole. Weakly supervised disentanglement with guarantees. arXiv preprint arXiv:1910.09772, 2019.
Andrea Slézia, Panna Hegedüs, Evgeniia Rusina, Katalin Lengyel, Nicola Solari, Attila Kaszas, Diána Balázsfi, Boris Botzanowski, Emma Acerbo, Florian Missey, et al. Behavioral, neural and ultrastructural alterations a graded-dose 6-ohda mouse model of early-stage parkinson's 's disease. Scientific Reports, 19478, 2023.
Kihyuk Sohn, Honglak Lee, and Xinchen Yan. Learning structured output representation using deep conditional generative models. Advances in neural information processing systems, 28, 2015.
Elsbeth A. Van Dam, Lucas P. J. J. Noldus, and Marcel A. J. Van Gerven. Disentangling rodent behaviors to improve automated behavior recognition. Frontiers in Neuroscience, 17, July 2023. ISSN 1662-453X. Publisher: Frontiers.
Raviteja Vemulapalli, Felipe Arrate, and Rama Chellappa. Human action recognition by representing 3d skeletons as points in a lie group. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 588-595, 2014.
Lukas von Ziegler. Oliver Sturman, and Johannes Bohacek. Big behavior: challenges and op- portunities in a new era of deep behavior profiling. Neurapsychopharmacoooss 46(1):33 -4. 2021.
Caleb Weinreb, Jonah Pearl, Sherry Lin, Mohammed Abdal Monium Osman, Libby Zhang, Sidharth Annapragada, Eli Conlin, Red Hoffman, Sofia Makowska, Winthrop Gillis, et al. Keypoint- moseq: parsing behavior by linking point tracking to pose dynamics. BioRxiv, 2023.
Matthew R Whiteway, Dan Biderman, Yoni Friedman, Mario Dipoppa, E Kelly Buchanan, Anqi Wu, John Zhou, Niccolò Bonacchi, Nathaniel J Miska, Jean-Paul Noel, et Partitioning variability in animal behavioral videos using semi sami-supervised variational autoencoders. PLoS computational biology, 17(9)e 1009439, 2021.
Alexander B. Wiltschko, Matthew J. Johnson, Giuliano Iurilli, Ralph E. Peterson, Jesse M. Katon, Stan L. Pashkovski. Victoria Abraira, Ryan Adams, and Sandeep Robert Datta. Mapping sub-second structure in mouse behavior. Neuron, 18(6):1111-11 2015a. ISSN 0889-66733.
14

---

Published as a conference paper at ICLR 2025
Alexander B. Wiltschko, Matthew J. Johnson, Giuliano Iurilli, Ralph E. Peterson, Jesse M. Katon, Stan L. Pashkovski, Victoria Abraira, Ryan Adams, and Sandeep Robert Datta. Mapping Sub-Second Structure in Mouse Behavior. Neuron, 1216)11 December 2015b. ISSN 0896-6273.
Alexander B. Wiltschko, Tatsuya Tsukahara, Ayman Zeine, Rockwell Anyoha, Winthrop F. Gillis, Jeffrey Markowitz, Ralph Peterson, Jesse Katon, Matthew J. Johnson, and Sandeep Robert Datta. Revealing the structure of pharmacobehavioral space through motion sequencing. Nature Neuroscience, 23 1): 4333-1443, November 2020. ISSN 1546-1726.
Anqi Wu, Estefany Kelly Buchanan, Matthew Whiteway, Michael Schartner, Guido Meijer, Jean-Paul Noel, Erica Rodriguez, Claire Everett, Amy Norovich, Evan Schaffer, et al. Deep graph pose: semi-supervised deep graphical model for improved animal pose tracking. Advances in Neural Information Processing Systems, 33:6040-6052, 2020.
Yilun Xu, Shengjia Zhao, Jiaming Song, Russell Stewart, and Stefano Ermon. A theory of usable infor- mation under computational constraints. In International Conference on Learning Representations, 2020a.
Yilun Xu, Shengjia Zhao, Jiaming Song, Russell Stewart, and Stefano Ermon. A theory of usable information under computational constraints. arXiv preprint 20t::20002.1688. 2020b.
Daiyao Yi and Shreya Saxena. Modeling the behavior of multiple subjects using cauchy-schwarz regularized partitioned subspace variational autoencoder (cs-ps-vae). In 2022 44th Annual Interna- fional Conference of the IEEE Engineering in Medicine & Biology Society (EMBC), pp. 497-503. IEEE, 2022.
Long Zhao, Yuxiao Wang. Jiaping Zhao, Liangzhe Yuan, Jennifer J Sun, Florian Schroff, Hartwig Adam, Xi Peng, Dimitris Metaxas, and Ting Liu. Learning dew-disentangled human pose representation by contrastive cross-view mutual information maximization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12793- 2021.
Yi Zhou, Connelly Barnes, Jingwan Lu, Jimei Yang, and Hao Li. On the continuity of rotation representations in neural networks. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 7745-5753, 2019.
Christian Zimmermann, Artur Schneider, Mansour Alyahyay, Thomas Brox, and Ilka Diester. Freipose: deep learning framework for precise animal motion capture in 3d spaces. BioRxiv, pp. 2020-02, 2020.
15

---

Published as a conference paper at ICLR 2025
A METHOD DETAILS
A.1
POSE PREPROCESSING
behavior 1
behavior
processing
behavior 2
behavior 2
input to model
Figure 4: Aligning pose trajectories. All pose sequences translated and aligned that the central frame is globally centered and rotated to face in a standard direction. This representation preserves important trajectory information. We decompose global root position as is commonly done in 3D deep pose methods (e.g., Petrovich et al., 2021b; He al., 2022).
pose sequences into local joint rotations and
A.2 SCRUBBERS
Scrubbing by adaptive moving average least square (SC-VAE-MALS)
Consider linear decoder:
fo (z) = uiz where 1 = (xb² + + which can be evaluated using the loss:
(7)
L(z.v; = ||v fv. (z)1l
(8)
Algorithm 1 SC-VAE-MALS
Initialize network parameters
6.0
Initialize linear estimator parameters
& E NxD
Initialize forgetting factor
a E (0, 1)
Initialize L-2reeuuarrization coefficient
3 3.e. R repeat
Draw batch with K samples
{xk, Vi R.V. 2k (.(. xx) RDJK 1k=1
Calculate decoder loss:
Lscrub * Lk-1 L(Zk, Vki
Update v based on the normal equations for ordinary least squares regression:
30 +++ (1 a) ΣK kk (Vx*k T) + and
$L = (1-a) Sk (Z T) + ari)
Update network parameters:
+- @ + L.semb + LALBO + TRecon]
0 V LRccon]
until convergence
16

---

Published as a conference paper at ICLR 2025
Scrubbing by adaptive quadratic discriminators (SC-VAE-QD)
Consider the class-conditional Bayes classifier:
fa(z) p(u = clz)
(9)
with likelihood:
p(=/u= = N22 x
(10)
For multi-class problems, maintain C). This estimator can be evaluated based on the Gaussian log-likelihood
estimator per class, & (ee, E ,e E Yce
L(z, u; (b) elpe E E Elp Σ 2 u cc)
(11)
Algorithm 2 SC-VAE-QD
Initialize network parameters
6,0
Initialize parameters of two classifers
Va, 1/6
+ al c. a (0,1)
repeat
Draw minibatch of {xx; Vk: A1c as in Alg
T.semb =0
for c do
Evaluate the Gaussian log-likelihood ratio for each classifier and average scrubbing loss:
Liceubb = Lscuub + # Ex L(z,, Uyi ()i())) v)
Update class means and covariances for both estimators: fe (1- aa -v
+ a
E Cov mo (2k, 2kk
# (1 á F [x] a 40
D'' = Cov Cov 12k 2k]
end for
Update network parameters as Alg 1 until convergence
Automatically tuning forgetting factor
One the core features of the moving average scrubbing methods (SC-VAE-MALS and -QD) is the automatic tuning of the forgetting factor, X. Given discriminator function, (z), and
minimization objective, L(z, , 8; ()). we simultaneously estimate two discriminators from the same function family, fv. (z) and fer (z). The estimators use distinct forgetting factors, a, and as with fixed offset e. We tune both Oa and og in the direction of the better-perfone netter-performig discriminator.
17

---

Published as a conference paper at ICLR 2025
Algorithm 3 Automatically tuning forgetting factor (A)
Va. 4b Initialize parameters of two discriminators Initialize forgetting factors with fixed offset: Qa (0)1-c) as 0l 6
repeat
Draw batch with K samples:
(Xk: Vk), Zk ~ go(- xx) E RDx
Average the losses of the two discriminators obtain the scrubbing loss
Iscnub [T Vk: tha) + [(Zk, v/a]
Forgetting factors step by A in the direction of the better decoder between fv. and fes if L(zk, Vki (Zkk, Vki vh) then
Ola marraa A)
Cg = +
else
Og = min(as + 1)
Oka abb -
end if
Continue with updating chas t/b, Φ, and as described in Algorithms and until convergence
Scrubbing by kernel density mutual information estimation (SC-VAE-MI)
In the previous objectives, we used adversarial parametric functions f-) as proxies to estimate and minimize the maximum predictive information in eq. (2). Here we consider a nonparametric estimator of the mutual information between and mutual information between these random variables is defined as:
alternative scrubbing objective. The
[(z, v) = Ez,v log p(z. v)
Ev.z [log z) - logp(v) p())
(12)
p(v)p(z)
We approximate the densities of the joint and marginal distributions using a kernel density estimator, as done prior work (e.g., Moon et al., 1995). Given a set of samples from the joint distribution, {zj, vi)/=1- approximate the joint density function as being proportional to:
p(v.z) Rd p(v, x
Σ k[z, v). [2, vz])
(13)
5=1
where [z. v] denotes vector concatenation, and .) denotes Gaussian kernel with a user-defined bandwidth parameter f:
1
||x
x'11
x(xxx') exp
(14)
2h
We approximate the marginal distributions over z and v in a similar fashion:
p(v) R2 p(v) x Σ k(v.v,)
(15)
j=1
p(z) p(z) k(z,z,) zz)
(16)
j=1
Therefore, we can approximate the term appearing inside the expectation eq. (12) as:
log p(z,v) v) p(v)p(z)
2 log 1(2z, v), [zj, vyl)
log
k(v.
log Σ x(2,2,) +
(17)
where C is constant that does not depend on the distribution of and v. Now, given second batch of samples from the joint distribution {īg. v > we can approximate the expectation appearing in eq. (12) with the empirical mean. Thus, obtain:
18

---

Published as a conference paper at ICLR 2025
pkZ V&) ((v){(z)
1(z v) ~ ((z,
log
(18)
K
log
ke (Z Vkl, vyl)
(19)
K
Rm
K
K log Σ k(z,, +,)
log
v5) K
(20)
K
2=1
as our estimate of the mutual information (up to an additive constant C). Importantly, this estimator differentiable with respect to 2,...., kk. allowing us to compute gradients to update the network's learned representations. We caution that this estimator suffers well-known curse of dimensionality and must be used with care. We expect it to work best when the VAE representation low-dimensional and large batches of data (i.e., large K and J) used to form the estimate. The choice of bandwidth parameter in the Gaussian kernel function is also critical; smaller choices h will require larger batches of data to form stable estimate.
Algorithm 4 SC-VAE-MI
Initialize network parameters
6,0 Initialize bandwidth parameter h R
repeat
Draw batch with K samples
{xk, Vk E R.N Zk ~ (o(:
xk) E RDIK k==
Draw batch with J samples
{x), E RV,, Zj (- x1) E BD) Jj=1
Using eq. (18)
Toserub f(z, v)
Update network parameters as in Alg 1
until convergence
19

---

Published as a conference paper at ICLR 2025
B
EXPERIMENT DETAILS
B.1
CONDITIONAL MOTION GENERATION
Walking
Walk Rear
Rear Down to Walk
Left Turn
Fast
Medium
Slow
Figure Enabling conditional motion generation. By manually adjusting the value of the conditioned variable, Vz, enable granular control this variable in the generated output without changing the gross behavioral category. Here we change the conditioned speed and are able to adjust the speed of behavior.
B.2 LATENT VISUALIZATIONS
Hewting Scuitiing
WAZ
FOS
Scrubiing
WAZ
CAME
SCAWEGRR
SCANEAD
SCANCWALL
SCANDMMACE
SCAMCAN
POS
Aniral Soutling
wes
WA--Q
Adimal
AYRUA
Figure 6: PCA visualizations. Projections of vanilla and scrubbed model representations the top principal component dimensions, colored by heading (top), speed (mid), and identity (bottom).
20

---

Published as a conference paper at ICLR 2025
B.3 OBTAINING BEHAVIORAL MOTIFS
Behavioral Motifs from GMM Clustering on Latents
Walk
Right Turn
Rear/Sit Into Walk
High Rear
ldle
Left Body Groom
Figure Example motifs from GMM clustering. Motifs extracted from our models' deep representations qualitatively reveal interpretable behavioral motifs.
B.4 NUISANCE VARIABLE DETAILS
Average speed Average speed is 3-dimensional vector corresponding to: (1) the average displace- ment of the root keypoint during a window (51 frames here), (2) the average speed of the spine and head relative to the root keypoint, and (3) the average speed of the limbs relative to the attached spine keypoint.
Heading direction Given pose sequence, we calculate the yaw, of the mid-spine + front-spine segment for the central frame of a pose sequence (i.e., the 26th frame of a 51 frame sequence this paper). The final heading direction used by the scrubber is [sin a cos ai T.
Animal identity We represent all categorical nuisance variables with one-hot encoding vectors.
B.5 SCRUBBING IDENTITY IN VAE WITHOUT CONDITIONING
a.
Animel ID Decoding from Latents
b.
Animal ID Maximum Mean Discrepancy
C30
VAE
CAREE
VAE-GH
SCHAE-OG SCHAE-NO VAL-QU SC-WAL-QU
VAE
C-VEE
SCHAE-GR SC-IAE-D
VAS-20
5CAME-gD
Figure 8: Decoding identity from scrubbed latents. (a) Classification accuracy for animal identity
using logistic regression and quadratic discriminates. (b) Paired inter-individual MMD.
Given the need to maintain a reasonable reconstruction error when learning a scrubbed variational distribution, it follows that conditioning would be required when disentangling variables such speed or heading. However, conditioning does not seem to be as intuitively necessary when disentangling variables, such as animal identity, which do not directly contribute the reconstruction error. Despite
21

---

Published as a conference paper at ICLR 2025
this, find that conditioning with one-hot encodings animal identity improves the scrubbing performance, as the linear and quadratic decodability of identity is reduced compared to scrubbing identity in standard VAE (VAE-GR and VAE-QD in Fig. 8a). SC-VAE representations also reduced the paired maximum mean discrepancy (MMD) (Gretton et al., 2012) between individuals (Fig. 8h).
B.6 METRIC DEFINITIONS
One-component Gaussian log-likelihood Given mean-centered and standardized latent represen- tation over a dataset, we split the walking behaviors Z into Ziroin and Zieat We use the mean I and diagonal covariance I²I of Zrak to calculate the Gaussian log-likelihood of Lest
Inp(z μt o² = [d 2nT + In + (z- (0o')) 1 (2-14) -4]
(21)
Thus, we quantify the level at which walking clusters of different speeds or heading directions can be represented by single, merged Gaussian distribution. We valds-eal this over five folds.
VAE
C-VAE
SC-VAE-GR
0.12
0.1
0.12
Holfy
0.00
0.00
0.26
0.06
0.00
0.00
Park karts
0.02
0.02
0.02
0.00
0.50
Paninoo's
Heiltyy
Paninson's
Healtty
Punkmons
Reverse Control
SC-VAE-QD
0.10
0.25
Minin Dlocregarp
0.15
Prnddoon
0.10
0.00
Figure Pairwise maximum mean discrepancy. For each session (healthy or diseased) of each animal, we calculated the maximum mean discrepancy between their behavioral distribution. Qualitatively, the difference in the cross-condition MMD (top-right and bottom-left quadrants) vs. the in-condition MMD (top-left and bottom-right quadrants) is more pronounced in SC-VAE-QU compared to other models. This demonstrates the enhanced disease discrininability in our scrubbed models.
Maximum mean discrepancy We use maximum mean discrepancy (MMD) (Gretton et al., 2012) to define a distance between two latent behavioral distributions with M and N samples, X E R Mx and Y E NxD, Given a radial basis kernel function with bandwidth h:
x' [2
k(z, z') = exp
(22)
MMD is defined as:
M
N
DMMD(X, Y) M(M 1)
xxx,,x))
k(y,,,))
N(N 1)
1=1 1
(23)
2 M
Σ
Σ k(x,,y))
MN
22

---

Published as a conference paper at ICLR 2025
Following the heuristic in Gretton al. (2012). we define h to be the median Euclidean distance in the aggregate sample [X, YJ. When visualizing the paired MMD between all available sessions (72 total) in Fig. 9, qualitatively find increased differences across conditions compared to within conditions. We describe our quantification of this effect below.
In Sec. 4.5, compare the healthy-diseas, discrepancy the inter-individual discrepancy. Given a dataset of individuals where H. is the healthy behavioral distribution and D, is the diseased behavioral distribution of animal we calculate metrics akin statistical effect size:
[dH]: = (I-1) * (M, D.)
(24)
Ejet (H, H2)
[dp]: = (1 (1-1) 1) DMMD (H; D.)
(25)
Σjti DM Dj)
Cohen's effect size is more similar to the cross-condition distance normalized to the variance within the control condition (healthy). Thus, we report dæ in Table 2. Both metrics are reported in Table 3. Finally, construct k-nearest neighbor classifier (k 35) to classify behavioral distributions using our MMD distance to quantify the changes in decoding accuracy of the disease label due to scrubbing.
Table 3: Maximum mean discrepancy metries. We quantify the difference in animal's cross- condition representations compared to the average in in-condition difference.
Effect Size
Model
du
dp 1
VAE
1.07 + 0.05
2.69 0.20
C-VAE
1.11 + 0.05
2.82 + 0.22
SC-VAE-GR
1.17 + 0.06
3.07 + 0.24
SC-VAE-QD
1.71 + 0.11
5.55 0.44
Reverse Control
0.68 + 0.03
1.51 + 0.14
23

---

Published as a conference paper at ICLR 2025
C MODEL TRAINING DETAILS
C.1 MODEL ARCHITECTURES
Table 4: Residual block architecture.
Layer
Kernel
Stride
Padding
Activation
Normalization
Channels out
Input
None
None
in cecannell
Conv ID 1
3
2
PReLU
Batch
out_ out_channel//2
Conv 2
1
1
None
None
out channel
ConvID skip
3
2
1
None
None
out cut_channel
Add
PReLU
Batch
out min_chanel
Table 5: Residual block transpose architecture.
Layer
Kernel
Stride
Padding
Activation
Normalization
Channels out
Input
None
None
in ea_channel
Conv ID
3
1
PReLU
Batch
in_channel//2
ConviD
2
None
None
out channel
Upsample by 2
None
None
out_ out_channee
Cony1D skip
None
None
out_channel
Add
PReLU
Batch
out out_chaate not_chanel
Table 6: Representation encoder architecture.
Layer
Kernel
Stride
Padding
Activation
Normalization
Output
Input
None
None
51x11I
ConvlD ID
7
1
3
PReLU
None
51x128
Residual
3
2
PReLU
Batch
26x256
Residual 2
3
2
1
PReLU
Batch
13
Flatten
None
None
6656
Dense (μ)
PReLU
None
64
Dense (a)
PReLU
None
64
24

---

Published as a conference paper at ICLR 2025
Table Representation decoder architecture. c the dimension of the disentanglement variable
Layer
Kernel
Stride
Padding
Activation
Normalization
Output
Encoded input
None
None
64 c
Dense
PReLU
None
6656
Unflatten
None
None
13x512
Residual 1
2
PReLU
Batch
x5x256
Residual 2
3
2
PReLU
Batch
x9 128
Cony 1D
Tanh
None
x1x111
Table 8: Gradient reversal scrubber. Uses ReI.U activation for all dense layers. e is the dimension of the disentanglement variable
Layer
Input
Output
Gradient reversal
(before MI.P Ensemble)
MLP
Dense 1
64
64
Dense 2
64
64
Dense 3
64
MLP
Dense 1
64
64
Dense 2
64
MLP 3
Dense 1
64
64
Dense 2
32
Dense 3
32
MLP
Dense 1
64
128
Dense 2
128
128
Dense 3
128
C.2 TRAINING DETAILS AND HYPERPARAMETER
We standardized training such that the following true across all models.
Models were trained on NVIDIA A100 and V100 GPUs.
Models were trained using AdamW optimizer (Loshchilov & Hutter, 2017) with learning rate of 0.0001 and gradient clipping at 10,000.
Losses models the same weights for joint position error the VAE ELBO loss.
Models were trained for 400 iterations over the training dataset with batch sizes of 2048.
The model architecture described in Section C.1 was not changed across tasks.
We used a pose sequence window size 51 frames.
The size of the latent dimension z₁ was 64 for the results in Sections 4.4-4.2 and for the results in Section 4.5.
To ensure fair comparison across models, we limited hyperparameter-tunii to only method-relevan hyperparameters. General hyperparameters and optimizers were chosen improve the joint position emror vanilla VAE reconstructions. These hyperparameters, as described above, were held constant across all VAE, C-VAE, and SC-VAE models.
Parameter tuning for SC-VAE scrubbers largely revolved around tuning the of the Lseuh (Φ) loss. In most cases, increasing lambda increased the strength of the scrubber up its linear or nonlinear assumptions. However, when À was too large, we found some cases posterior collapse or mode collapse, primarily in the neural network scrubbing models (SC-VAE-GR and -ND). For SC-VAE-GB we were additionally required to tune the scaling factor of the gradient reversal layer a. For SC- VAE- ND, found performance changes due the number of discriminator training iterations in between
25

---

Published as a conference paper at ICLR 2025
C-VAE iterations. If too few iterations were taken, the discriminator would underfit and vice versa. Due to our increment strategy for forgetting factors (Algorithm A.2), training SC-VAE-MALS and SC-VAE-QD, À was the only required parameter to tune. In the polynomial extensions to the moving average algorithm (SC-VA SC--AE-MAQS), we found better performance with non-zero 1.2 regularization, B. Finally, for SC-VAE-M we additionally tuned the h parameter determining the spherical width of the kernel.
selected based the level of nonlinear invariance in the model latent representations to the scrubbed nuisance variable (i.e., the "MLP and "QDA" metries in Fig. 2), filtering out models with posterior collapse. All reported metrics are calculated on test set separate from the training and validation dataset. The results in Sections 4.4-4.2 were calculated on the same set of models. Consequently, there were models not selected with better performance in the conditional motion synthesis task in Table 4.2. While increased invariance usually led to better performance motion synthesis, we observed an inverse correlation in SC- SC-VAE-ND models trained to scrub heading direction where the least invariant models had an R² of up to 0.859, Models were re-trained from scratch on the Parkinsonian dataset to obtain the results in Section 4.5.
Models were evaluated held-out validation set. Hyperparameters
Table Hyperparameter search. We list the hyperparameters and values over which we trained models for our experiments. We applied grid search for methods with more than one hyperparameter. Hyperparameters
Model
Values
À
10, 20, 50, 100
SC-VAE-GR
a
0.8, 1.0, 1.2
X
10, 50, 100
SC-VAE-ND
n_iter
1, 10, 50, 100
SC-VAE-MALS
À
10, 20,
5, 10, 20
SC-VAE-MAQS
3
0.25, 0.5, 0.75
À
250, 500, 750
SC-VAE-MI
h
0,25, 0.5, 0.75
À
5, 10, 20, 50
26

---

Published as a conference paper at ICLR 2025
D PARKINSON'S DATASET
D.1 IMPACT AND ANIMAL CARE STATEMENT
Phenotyping animal behavior is an early and essential step in drug discovery, and analyses of high- resolution 3D pose measurements provide sensitive, reproducible, and scalable avenue for such preclinical assays. By releasing datasets and benchmarks for 3D pose analyses, continued development and acceleration of new phenotyping methods. We thus are facilitating the development of new therapeuties for neuropsychiatric disorders. All animal data AAALAC-accredited animal facilities. The care and experimental manipulation of all animals were reviewed and approved by Duke University Institutional Animal Care and Use Committee.
supporting the
collected
D.2 LICENSES
Our dataset is released under CC BY 4.0 Attribution International license, which permits sharing and adapting the datasets without restrictions except for providing attribution to the dataset when used adapted in new medium.
D.3 DATASET SPLITS AND ACCESS
We used a 50-25-25 train-validation-lest split for our datasets. All hour-long videos were split into 1-minute long sections and randomized across each dataset while ensuring equal distribution of animal identity. Additionally, as animal behavior can change across the length ensured that each dataset received even distribution for each third of the recording session. Our datasets, train- validation-tes splits, Python code, and documentation for running all benchmarks can be found Github repository. See http://gggtubb..ommcmmmaaaannret
recording session,
D.4 ETHICS
Please D.I for a discussion of animal ethics. We acknowledge that by aiding the development of behavioral activity analysis approaches, bad actors could potentially use developed approaches for unethical purposes. However, we believe that the benefits behavioral analysis approaches offer for society, in terms of the potential to alleviate human disease burden, outweigh the (minimal) risks of misuse.
Complete Lesion
Incomplete Lesion
Figure 10: Dopamine denervation in PD mouse brain. Example fluorescent immunostaining for tyrosine hydroxylase (TH) in two mice after inducing Parkinson's 's discase via 6-hydroxy dopamine (6-OHDA) lesioning. TH (bright green) indicates the presence dopamine neurons. (left) Example complete lesion. (right) Example incomplete lesion.
27

---

Published as a conference paper at ICLR 2025
D.5 DATASET COLLECTION
Adult Drdla-Cre (8 females, 8 males, 18-20 weeks old) and Adora2a-Cre (10 females, 10 males, 18-20 weeks old) mice were anesthetized and implanted with a metal cannula (P1 tech) targeting the left medial forebrain bundle (from bregma: -0.7mm AP, -1.2mm MI, -4.75mm DV). Custom-built headbars were then affixed to the skull using UV glue and dental cement.
After allowing week for the mice to recover. mice were habituated to head fixation and to the open field (30 em x 30 cm) arena, then a I-hour baseline recording was performed in the arena another 1 week later. Immediately following the "baseline" session, mice were head-fixed, and 0.9 μLL of a freshly made solution of 3.6mg/ml 6-hydroxy dopamine (6-OHDA) and mmg/ml ascorbic acid was infused through the cranial cannula at rate of 0.1 jLL/min, allowing an additional 6 minutes for diffusion before removing the infusion cannula. 6-OHDA lesioned mice were provided with supportive care including daily subcutaneous saline injections and hand feeding with mash made from high-caloric diet supplement (Stat, ground-up macal) and ground up rodent feed (LabDiet 5001) until their weight stabilized. Two weeks after the lesion, mice were again placed in the open-field arena for hour to collect their 'lesion' behavioral data.
At the conclusion of the experiment, mice were deeply anesthetized with isofturane and underwent transcardial perfusion with 15 mL of 0.1M phosphate buffered saline (PBS) followed by 50 mL of cold 4% pareeormaldehyde (PFA) in O. I. phosphate buffer (PB), PH 7.4 for fixation. Brains were then dissected from the skull and soaked overnight at 4 in the PFA solution, followed by 3 washes with O. IM PBS. Each brain was then embedded mold in 5% agarose in order to collect 50 μM coronal sections with a vibratome (Leica VTI200S) into wells containing O. IM PBS.
The next day, brain sections were immunostained for tyrosine hydroxylase (TH), a marker for dopamine neurons. Briefly, sections were washed in PBS before 2-hour incubation in blocking solution consisting of 5% goat serum, 3% bovine serum albumin, and 0.3% trition-x. Sections were then transferred to a half-block solution containing 1:1000 Rabbit T-TH (PelFreez, P40101) overnight at 4 with agitation. Sections were then washed in 0.1M PBS containing 0. % tween before a 4-hour incubation in a half-block solution containing 1:1000 goat o-rabbit 488 (Invitrogen, A1 A11008). Finally, sections were washed in PBS containing tween, then PBS prior to mounting on glass slides (VWR) and coverslipping with Vectashield Vibrance with DAPI (Vector Labs).
Fluorescent images of each immunostained coronal section were then collected on a VS200 slide scanner. Analysis of pixel intensity of the fluorescent signal for TH was performed using custom MATLAB script. For each brain, sections were analyzed that evenly span the anterior to posterior axis of the striatum. On each section, the striatum manually segmented in both hemispheres, and the fluorescent signal was summed across the sections for both the healthy and lesioned hemispheres. The value for the fluorescence in the lesioned hemisphere was then divided by the value for the fluorescence in the healthy hemisphere to yield the 'integrated fluorescence' for each animal.
28


Reviews:
[{"review_id": "d6VvoffXlI", "reviewer": "Reviewer_nAGW", "summary": "The authors present a framework for motion analysis that uses conditional variational autoencoders to disentangle desired behavioral variables seen in animals and to remove nuisance confounds. They augment the C-VAE loss function to reduce dependence between the VAE latent variables and the behavioral variables of interest. They explore different methods to scrub out disentanglement including linear, quadratic, cubic, MLP, and MI based approaches. They thoroughly analyze their model in a simulated setting and apply their technique to improve disease detection in a Parkinsonian mouse model.", "questions": "Why do the authors think that the linear scrubbing improved the conditioned sequence best? This is seemingly the most limited way to represent z in equation 2? In principle, shouldn't the more sophisticated approaches (like quadratic and cubic) also be able to capture the linear approach? A bit more discussion on this point might be interesting to include.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "The paper is very clearly written, the results are compelling and thorough, and the model addresses a need in the neuroscience community as there are increasingly common adaptations of VAE style models to behavioral data. I appreciate the overview of C-VAEs as well as their motivation and clear explanation of their scrubbing methodology. The analysis on the two real datasets demonstrate the utility of the approach and I appreciate the clear visualizations demonstration appropriate disentanglement using the various SCVAES.", "weaknesses": "Some more discussion of the parkinsonian dataset would be appreciated. It is unclear what details are similar with this particular dataset and the previous one (e.g. are the behavioral variables scrubbed/conditioned the same way? are the model architectures and hyperparameters the same? ) This is a small point but it might be nice to point to supplemental information in this brief final results paragraph.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fwXzM60jvY", "reviewer": "Reviewer_p9qK", "summary": "The authors propose scrubbed conditional variational autoencoder (SC-VAE), a novel framework for disentangling nuisance factors by removing variable information from latent spaces by using an adversarial learning objective. They demonstrate the utility of SC-VAE using 2 mouse behavior datasets.", "questions": "- In terms of pose estimation, I often hear about DeepLabCut. I know that framework doesn’t use a VAE backbone but in general how does SC-VAE compare to DeepLabCut?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- This framework extends C-VAEs into a more interpretable realm by introducing scrubbing. The idea of scrubbing the nuisance variables out of data can allow researchers to focus on the variable of interest without the unnecessary/irrelevant information. \n- The idea of isolating the semantic information from the character of the action has the potential to be applied to many fields other than neuroscience such as speech recognition or emotion classification.\n- Clear flow throughout the paper.", "weaknesses": "- Generalizability: Since there is the need for specific assumptions and constraints to guarantee disentanglement, e.g.  picking the known factors, SC-VAE might be inflexible across datasets with unknown structures. I know that the unsupervised methods were briefly mentioned at the beginning and the most prominent weakness to them seemed to be the sensitivity to nuisance variability but this then brings the flexibility in mind once again in terms of more complex datasets with less prior knowledge on the structure. I would be interested to know how SC-VAE could be adapted to work in cases where nuisance factors are hard to determine.\n- Figures: Figures can be more polished. Right now many of them look like default matplotlib figures (See minor points for improvement suggestions).\n\nMinor points\n\n- For each plot in Fig 2.a,b,c,d, the top and rightmost axis can be removed, and for Fig2.a.b the legend doesn't need to be repeated. Same for fig4.\n- Fig3.f and fig3.i have overlapping histograms, and in general the axis labels are illegible in fig3.\n- l398 there might be extra space in ‘Fig. 2e’.\n- Fig4.a the second plot y axis label is too close to the title.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Z3zxdiBgVE", "reviewer": "Reviewer_qJjc", "summary": "The authors propose a framework for modeling 3D pose dynamics which is capable of disentangling both continuous and discrete nuisance factors (such as speed and animal identity). They propose to use a conditional variational autoencoder, where the conditioning variables are the nuisance factors to disentangle. In order to properly ensure disentanglement of the nuisance factors from the VAE latents, the authors then propose a series of strategies for \"scrubbing\" the nuisance factors from the latents. They then demonstrate the effectiveness of these various strategies using 3D poses from a mouse behavioral experiment in a series of well-designed control studies. Finally, they apply these strategies to a larger dataset of healthy and diseased mice and demonstrate their ability to accurately discriminate between these two groups.", "questions": "the sentence starting on L76 is very dense and difficult to parse: \"we show that simple linear estimators perform favorably to traditional neural adversarial paradigms while introducing strategies for nonlinear estimators which bypass the need for specifying adversarial neural networks and their hyperparameters.\" can this be clarified?\n\nfig 1 contains a lot of information but is difficult to see at its current size (especially 1c), can this be enlarged?\n\nL293: \"we increment or decrement the forgetting factors based on which filter provides a better fit to the minibatch statistic\" - this seems to imply either/both forgetting factors can change, which contradicts the previous statement that one factor is \"fixed\" - please clarify\n\nsec 4.4: the authors find that different scrubbing strategies are appropriate for heading vs speed. This is an interesting finding, but what does it mean for the practical applicability of this method?\n\ndo the authors have any intuition for what features are being removed by scrubbing subject identity? is it just subject size, or something else? if the 3D poses were normalized within each subject by, say, distance from tailbase to neck (or some other such pair of points), does the subject id scrubbing still result in disentangling?\n\nas a control experiment, if the authors scrubbed disease state instead of animal identity in the final analysis, does the MMD drop to a chance level?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "the problem is well-motivated appropriate for the ICLR audience, and the paper is generally well-written\n\nthe solution proposed by the authors is a creative synthesis of the existing literature, as well as some extensions necessary to make their solution work in practice\n\nthe validation experiments shown in figs 2a-c are strong, with appropriate baselines (I like the use of VAE Processed as a baseline in the heading direction experiments)\n\nconsistent motion synthesis (sec 4.3) is also a neat way to test disentangling", "weaknesses": "the authors test a wide range of models, but there is a lack of comparison to other established disentangling methods. while this literature is large, it would, for example, be useful to compare speed disentangling against the model in costacurta et al 2022. this model builds invariance to a specific nuisance variable (speed) and was designed for this exact type of data; if the authors could show that one of their models performs on par with the costacurta model, and can disentangle other generic variables besides, it would strengthen the argument for the scrubbing approach with pose data.\n\nsome details of the experiments are not clear. in sec 4.2, is there a separate model fit for each of the nuisance variables? if so, what happens if a single model is trained to scrub both speed and heading direction? in general, it is not clear if this approach is robust to scrubbing multiple variables at once.\n\nthe MMD analysis in sec 4.5 should be expanded upon, at the moment it is hard to understand. A simple decoding analysis (healthy vs diseased) would be a nice complement, and could lead to a punchier conclusion (\"we could decode disease state x% of the time in the VAE, and y% of the time in the SC-VAE-QD\"). I am also somewhat surprised that scrubbing leads to _improved_ discriminability between healthy and diseased conditions; I suppose this is because there are systematic differences between the two conditions that are obscured by animal identity? There are many such healthy vs diseased experiments with transgenic lines where a single subject will fall into one or the other category, but never both. In this case _not_ scrubbing subject ID would likely lead to higher discriminability (though perhaps for uninteresting reasons, such as diseased subjects are slightly smaller on average, etc.)\n\nunsupervised clustering of pose data is becoming more ubiquitous in large-scale drug screens and disease research. the differences between two disease models, or the effects of a drug, can manifest in subtle differences in animal behavior. one of the potential drawbacks of this approach is that, by scrubbing certain information from the latent representation, these subtle differences may also be lost. while I think this work is super interesting and useful, I think a fuller discussion of its potential drawbacks (and possibly how these can be controlled) should be included in the Conclusions section.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Vcy8Oc12Rx", "reviewer": "Reviewer_ZpEQ", "summary": "The paper, \"Disentangling 3D Pose Dynamics with Scrubbed Conditional Latent Variables,\" introduces the Scrubbed Conditional Variational Autoencoder (SC-VAE) as a framework for analyzing 3D pose data to extract behavior-relevant signals from nuisance factors like speed, direction, and individual traits. While the approach could offer valuable tools for behavioral research, several fundamental issues weaken the rigor, interpretability, and reproducibility of the work, limiting its potential impact.", "questions": "What evidence supports the claim that nuisance factors like speed do not affect behavior representation (Line 60)? Has this assumption been empirically tested?\n\nCould you offer more insight into how parameter choices, especially the selection of λ for weighting loss terms, affect the model's performance?\n\nGiven that SC-VAE optimizes latent variables directly, would you consider including other adversarial methods for a more balanced comparison? How might this impact the conclusions drawn about SC-VAE’s performance?\n\nWould you consider adding comparisons with other adversarial baselines designed to disentangle nuisance variables? This would provide a clearer assessment of SC-VAE's robustness in comparison to similar methods.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The SC-VAE framework could provide useful tools in fields like neuroscience, where disentangling complex behaviors is essential. By addressing nuisance factors, SC-VAE theoretically enhances clustering and behavioral analysis. However, without stronger empirical validation, its practical contributions remain speculative.", "weaknesses": "-Key claims lack adequate empirical or theoretical support. For example, Line 52-53 states that \"the other latent variables are not necessarily, nor typically, invariant to the conditional factor\" without any justification or citation. Similarly, Line 60 lacks evidence to support the assertion that nuisance factors, like speed, do not impact behavior representation. This weakens confidence in the model’s purported advantages.\n\n-Several statements are overly vague, such as Line 222-223: \"if the latent dimension D is small, one expects the C-VAE to learn a disentangled representation.\" Without specifying a relative scale for 𝐷, it is difficult to understand or reproduce the model setup. Clearer definitions and justifications for parameters are necessary to enhance reproducibility and interoperability.\n\n-The model is evaluated against only a single baseline, despite references to multiple adversarial models. This limited comparison restricts the ability to gauge SC-VAE’s robustness and leaves the evaluation incomplete, particularly given the lack of consideration for other adversarial baselines aimed at disentangling nuisance variables.\n\n-Line 65 refers to \"weak supervision\" while the model receives the full ground truth for certain variables. This contradicts conventional definitions of weak supervision, potentially misleading readers regarding the model’s true level of supervision.\n\n-The comparison between SC-VAE and C-VAE is not entirely fair, as SC-VAE benefits from direct optimization of latent variables, giving it an advantage. It would be fairer to include other adversarial methods in the comparison to better contextualize SC-VAE’s performance.\n\n-Key implementation details, such as model architecture, hyperparameters, and balancing of loss components (particularly λ for term weighting), are missing. Without these, it is challenging to reproduce the results or determine if they arise from design choices or hyperparameter tuning.\n\n-Visualizations referenced in Line 397 are cluttered and lack clear captions, making them difficult to interpret and detracting from the study’s overall clarity. Simplifying these figures and providing self-explanatory captions would improve readability and support the model’s claims more effectively.\n\n-The log-likelihood (LL) calculation presented in Figure 3.C is not well explained, making it hard to interpret the metric’s relevance to the conclusions. A more detailed description of the calculation process would improve transparency.\n\n-The authors claim SC-VAE can extend to other species and behaviors, but no additional experiments support this assertion. Expanding validation to other datasets is crucial for establishing generalizability.\n\n\n\nIn summary, the paper presents a potentially valuable framework for disentangling behavior-relevant signals from nuisance factors in 3D pose data. However, critical gaps in methodology, inadequate baseline comparisons, and insufficient experimental details limit the work’s robustness and reliability. Addressing these issues—through clearer claims, more thorough benchmarks, and comprehensive experimental details—would be essential for making a strong contribution to the field. As it stands, the paper falls short of the standards necessary for acceptance in its current form.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)