Disentangling 3D Animal Pose Dynamics with Scrubbed Conditional Latent Variables

Authors: Joshua Huang Wu, Hari Koneru, James Russell Ravenel, Anshuman Sabath, James Michael Roach, Shaun Sze-Xian Lim, MICHAEL R TADROSS, Alex H Williams, Timothy W DUNN
Primary area: applications to neuroscience & cognitive science
Keywords: behavioral neuroscience, systems neuroscience

Abstract:
Methods for tracking lab animal movements in unconstrained environments have become increasingly common and powerful tools for neuroscience. The prevailing hypothesis is that animal behavior in these environments comprises sequences of discrete stereotyped body movements ("motifs" or "actions"). However, the same action can occur at different speeds or heading directions, and the same action may manifest slightly differently across subjects due to, for example, variation in body size. These and other forms of nuisance variability complicate attempts to quantify animal behavior in terms of discrete action sequences and draw meaningful comparisons across individual subjects. To address this, we present a framework for motion analysis that uses conditional variational autoencoders in conjunction with adversarial learning paradigms to disentangle behavioral factors. We demonstrate the utility of this approach in downstream tasks such as clustering, decodability, and motion synthesis. Further, we apply our technique to improve disease detection in a Parkinsonian mouse model.

Decision: accept (poster)
Reviewer scores: 8, 6, 8, 6

Reviews and discussion:
<thread>
creation date:
1730340991985

thread_type:
official_review

<thread_object>
creator: reviewer_zpeq

<content>
summary:
The paper, "Disentangling 3D Pose Dynamics with Scrubbed Conditional Latent Variables," introduces the Scrubbed Conditional Variational Autoencoder (SC-VAE) as a framework for analyzing 3D pose data to extract behavior-relevant signals from nuisance factors like speed, direction, and individual traits. While the approach could offer valuable tools for behavioral research, several fundamental issues weaken the rigor, interpretability, and reproducibility of the work, limiting its potential impact.

strengths:
The SC-VAE framework could provide useful tools in fields like neuroscience, where disentangling complex behaviors is essential. By addressing nuisance factors, SC-VAE theoretically enhances clustering and behavioral analysis. However, without stronger empirical validation, its practical contributions remain speculative.

weaknesses:
-Key claims lack adequate empirical or theoretical support. For example, Line 52-53 states that "the other latent variables are not necessarily, nor typically, invariant to the conditional factor" without any justification or citation. Similarly, Line 60 lacks evidence to support the assertion that nuisance factors, like speed, do not impact behavior representation. This weakens confidence in the model’s purported advantages.

-Several statements are overly vague, such as Line 222-223: "if the latent dimension D is small, one expects the C-VAE to learn a disentangled representation." Without specifying a relative scale for 𝐷, it is difficult to understand or reproduce the model setup. Clearer definitions and justifications for parameters are necessary to enhance reproducibility and interoperability.

-The model is evaluated against only a single baseline, despite references to multiple adversarial models. This limited comparison restricts the ability to gauge SC-VAE’s robustness and leaves the evaluation incomplete, particularly given the lack of consideration for other adversarial baselines aimed at disentangling nuisance variables.

-Line 65 refers to "weak supervision" while the model receives the full ground truth for certain variables. This contradicts conventional definitions of weak supervision, potentially misleading readers regarding the model’s true level of supervision.

-The comparison between SC-VAE and C-VAE is not entirely fair, as SC-VAE benefits from direct optimization of latent variables, giving it an advantage. It would be fairer to include other adversarial methods in the comparison to better contextualize SC-VAE’s performance.

-Key implementation details, such as model architecture, hyperparameters, and balancing of loss components (particularly λ for term weighting), are missing. Without these, it is challenging to reproduce the results or determine if they arise from design choices or hyperparameter tuning.

-Visualizations referenced in Line 397 are cluttered and lack clear captions, making them difficult to interpret and detracting from the study’s overall clarity. Simplifying these figures and providing self-explanatory captions would improve readability and support the model’s claims more effectively.

-The log-likelihood (LL) calculation presented in Figure 3.C is not well explained, making it hard to interpret the metric’s relevance to the conclusions. A more detailed description of the calculation process would improve transparency.

-The authors claim SC-VAE can extend to other species and behaviors, but no additional experiments support this assertion. Expanding validation to other datasets is crucial for establishing generalizability.



In summary, the paper presents a potentially valuable framework for disentangling behavior-relevant signals from nuisance factors in 3D pose data. However, critical gaps in methodology, inadequate baseline comparisons, and insufficient experimental details limit the work’s robustness and reliability. Addressing these issues—through clearer claims, more thorough benchmarks, and comprehensive experimental details—would be essential for making a strong contribution to the field. As it stands, the paper falls short of the standards necessary for acceptance in its current form.

questions:
What evidence supports the claim that nuisance factors like speed do not affect behavior representation (Line 60)? Has this assumption been empirically tested?

Could you offer more insight into how parameter choices, especially the selection of λ for weighting loss terms, affect the model's performance?

Given that SC-VAE optimizes latent variables directly, would you consider including other adversarial methods for a more balanced comparison? How might this impact the conclusions drawn about SC-VAE’s performance?

Would you consider adding comparisons with other adversarial baselines designed to disentangle nuisance variables? This would provide a clearer assessment of SC-VAE's robustness in comparison to similar methods.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730652164072

thread_type:
official_review

<thread_object>
creator: reviewer_qjjc

<content>
summary:
The authors propose a framework for modeling 3D pose dynamics which is capable of disentangling both continuous and discrete nuisance factors (such as speed and animal identity). They propose to use a conditional variational autoencoder, where the conditioning variables are the nuisance factors to disentangle. In order to properly ensure disentanglement of the nuisance factors from the VAE latents, the authors then propose a series of strategies for "scrubbing" the nuisance factors from the latents. They then demonstrate the effectiveness of these various strategies using 3D poses from a mouse behavioral experiment in a series of well-designed control studies. Finally, they apply these strategies to a larger dataset of healthy and diseased mice and demonstrate their ability to accurately discriminate between these two groups.

strengths:
the problem is well-motivated appropriate for the ICLR audience, and the paper is generally well-written

the solution proposed by the authors is a creative synthesis of the existing literature, as well as some extensions necessary to make their solution work in practice

the validation experiments shown in figs 2a-c are strong, with appropriate baselines (I like the use of VAE Processed as a baseline in the heading direction experiments)

consistent motion synthesis (sec 4.3) is also a neat way to test disentangling

weaknesses:
the authors test a wide range of models, but there is a lack of comparison to other established disentangling methods. while this literature is large, it would, for example, be useful to compare speed disentangling against the model in costacurta et al 2022. this model builds invariance to a specific nuisance variable (speed) and was designed for this exact type of data; if the authors could show that one of their models performs on par with the costacurta model, and can disentangle other generic variables besides, it would strengthen the argument for the scrubbing approach with pose data.

some details of the experiments are not clear. in sec 4.2, is there a separate model fit for each of the nuisance variables? if so, what happens if a single model is trained to scrub both speed and heading direction? in general, it is not clear if this approach is robust to scrubbing multiple variables at once.

the MMD analysis in sec 4.5 should be expanded upon, at the moment it is hard to understand. A simple decoding analysis (healthy vs diseased) would be a nice complement, and could lead to a punchier conclusion ("we could decode disease state x% of the time in the VAE, and y% of the time in the SC-VAE-QD"). I am also somewhat surprised that scrubbing leads to _improved_ discriminability between healthy and diseased conditions; I suppose this is because there are systematic differences between the two conditions that are obscured by animal identity? There are many such healthy vs diseased experiments with transgenic lines where a single subject will fall into one or the other category, but never both. In this case _not_ scrubbing subject ID would likely lead to higher discriminability (though perhaps for uninteresting reasons, such as diseased subjects are slightly smaller on average, etc.)

unsupervised clustering of pose data is becoming more ubiquitous in large-scale drug screens and disease research. the differences between two disease models, or the effects of a drug, can manifest in subtle differences in animal behavior. one of the potential drawbacks of this approach is that, by scrubbing certain information from the latent representation, these subtle differences may also be lost. while I think this work is super interesting and useful, I think a fuller discussion of its potential drawbacks (and possibly how these can be controlled) should be included in the Conclusions section.

questions:
the sentence starting on L76 is very dense and difficult to parse: "we show that simple linear estimators perform favorably to traditional neural adversarial paradigms while introducing strategies for nonlinear estimators which bypass the need for specifying adversarial neural networks and their hyperparameters." can this be clarified?

fig 1 contains a lot of information but is difficult to see at its current size (especially 1c), can this be enlarged?

L293: "we increment or decrement the forgetting factors based on which filter provides a better fit to the minibatch statistic" - this seems to imply either/both forgetting factors can change, which contradicts the previous statement that one factor is "fixed" - please clarify

sec 4.4: the authors find that different scrubbing strategies are appropriate for heading vs speed. This is an interesting finding, but what does it mean for the practical applicability of this method?

do the authors have any intuition for what features are being removed by scrubbing subject identity? is it just subject size, or something else? if the 3D poses were normalized within each subject by, say, distance from tailbase to neck (or some other such pair of points), does the subject id scrubbing still result in disentangling?

as a control experiment, if the authors scrubbed disease state instead of animal identity in the final analysis, does the MMD drop to a chance level?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730654844303

thread_type:
official_review

<thread_object>
creator: reviewer_p9qk

<content>
summary:
The authors propose scrubbed conditional variational autoencoder (SC-VAE), a novel framework for disentangling nuisance factors by removing variable information from latent spaces by using an adversarial learning objective. They demonstrate the utility of SC-VAE using 2 mouse behavior datasets.

strengths:
- This framework extends C-VAEs into a more interpretable realm by introducing scrubbing. The idea of scrubbing the nuisance variables out of data can allow researchers to focus on the variable of interest without the unnecessary/irrelevant information. 
- The idea of isolating the semantic information from the character of the action has the potential to be applied to many fields other than neuroscience such as speech recognition or emotion classification.
- Clear flow throughout the paper.

weaknesses:
- Generalizability: Since there is the need for specific assumptions and constraints to guarantee disentanglement, e.g.  picking the known factors, SC-VAE might be inflexible across datasets with unknown structures. I know that the unsupervised methods were briefly mentioned at the beginning and the most prominent weakness to them seemed to be the sensitivity to nuisance variability but this then brings the flexibility in mind once again in terms of more complex datasets with less prior knowledge on the structure. I would be interested to know how SC-VAE could be adapted to work in cases where nuisance factors are hard to determine.
- Figures: Figures can be more polished. Right now many of them look like default matplotlib figures (See minor points for improvement suggestions).

Minor points

- For each plot in Fig 2.a,b,c,d, the top and rightmost axis can be removed, and for Fig2.a.b the legend doesn't need to be repeated. Same for fig4.
- Fig3.f and fig3.i have overlapping histograms, and in general the axis labels are illegible in fig3.
- l398 there might be extra space in ‘Fig. 2e’.
- Fig4.a the second plot y axis label is too close to the title.

questions:
- In terms of pose estimation, I often hear about DeepLabCut. I know that framework doesn’t use a VAE backbone but in general how does SC-VAE compare to DeepLabCut?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730738658642

thread_type:
official_review

<thread_object>
creator: reviewer_nagw

<content>
summary:
The authors present a framework for motion analysis that uses conditional variational autoencoders to disentangle desired behavioral variables seen in animals and to remove nuisance confounds. They augment the C-VAE loss function to reduce dependence between the VAE latent variables and the behavioral variables of interest. They explore different methods to scrub out disentanglement including linear, quadratic, cubic, MLP, and MI based approaches. They thoroughly analyze their model in a simulated setting and apply their technique to improve disease detection in a Parkinsonian mouse model.

strengths:
The paper is very clearly written, the results are compelling and thorough, and the model addresses a need in the neuroscience community as there are increasingly common adaptations of VAE style models to behavioral data. I appreciate the overview of C-VAEs as well as their motivation and clear explanation of their scrubbing methodology. The analysis on the two real datasets demonstrate the utility of the approach and I appreciate the clear visualizations demonstration appropriate disentanglement using the various SCVAES.

weaknesses:
Some more discussion of the parkinsonian dataset would be appreciated. It is unclear what details are similar with this particular dataset and the previous one (e.g. are the behavioral variables scrubbed/conditioned the same way? are the model architectures and hyperparameters the same? ) This is a small point but it might be nice to point to supplemental information in this brief final results paragraph.

questions:
Why do the authors think that the linear scrubbing improved the conditioned sequence best? This is seemingly the most limited way to represent z in equation 2? In principle, shouldn't the more sophisticated approaches (like quadratic and cubic) also be able to capture the linear approach? A bit more discussion on this point might be interesting to include.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732742462560

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Summary of Revised Manuscript

comment:
We thank the reviewers for their time and insights. We are encouraged to see that all reviewers found our methods well-motivated and essential for our application in behavioral neuroscience. In response, we have uploaded an updated paper incorporating all the suggestions in the reviews. Our revision includes requested results from additional experiments and performance metrics, improvements to figures, edits to the text, and an expanded Appendix. We believe that our paper is much improved, and we thank the reviewers for this.

Here we highlight key improvements. A full itemized list of changes can be found on pages 27-28 of the revised PDF.

> New baseline - scrubbing with adversarial examples (SC-VAE-ND).

Reviewers qJjc and ZpeQ asked for additional baselines, specifically adversarial methods. Please note that there was a miscommunication in the initial paper, where we did not clearly convey that SC-VAE-GR was already included as an adversarial baseline. This has been clarified in the text.

However, we have also implemented an additional adversarial disentanglement method based on (Sanchez et al. 2020 “Learning disentangled representations via mutual information estimation.”) and have titled it SC-VAE-ND in the revised PDF. On all metrics except heading-conditioned motion synthesis, SC-VAE-ND performed worse than the previous adversarial SC-VAE-GR baseline. 

We would like to re-emphasize that one of our primary contributions is to introduce scrubbers that: (1) do not require training a secondary network, and (2) can vary in assumptions on the level of scrubbing desired. These are our SC-VAE-MALS, -MAQS, -QD, and -MI scrubbers. To help clarify these contributions, we have updated some language at the end of our Introduction section, in Sec 4.4, and have reformatted our methods section to have a dedicated section (Sec 3.5) to discuss neural network scrubbers (SC-VAE-GR and -ND) and their associated challenges.

> Improving the presentation of figures and tables

We thank the reviewers for their suggestions on improving the presentation of our results. We have increased the size of all fonts and figure elements and polished the figure stylings. We have moved our former Figs. 1c and 2e to the Appendix and reformatted the former Fig. 4 into a table. Fig. 2a-c, Fig. 3b, Fig. 3f, and Table 1 have also been updated with results from the new SC-VAE-ND model. Table 2 (former Fig. 4) has been updated to better communicate how metrics relate to PD discriminability, and we have added new disease classification results. We have addressed all requested changes, and we are happy to make further changes for the camera-ready version if anything new is requested during the discussion period.

> Expanded Appendices

As suggested by reviewers nAGW and ZpEQ, we have expanded our Appendices to include more details on our calculated metrics, methods, model architectures, training, hyperparameter tuning, and datasets.  

> A new control scrubbing model, new healthy/disease decoding metrics, and clearer writing for the Parkinson’s disease results.

In response to excellent suggestions from qJjc, we have made major improvements to the PD analysis section. In addition to rewriting the section in the paper to better motivate and interpret the analysis, we also added a new control model (scrubbing disease state) and a new healthy/disease classification accuracy metric (Table 2). We also distilled the MMD differences into a single effect size measure and simplified our partial lesion terminology (Table 2). We also further expanded on MMD and its definition in the Appendix. The Appendix also includes a new figure (Fig. 9) visualizing the MMD across animals and conditions for all models. We think this visualization elevates intuition for how MMD is changing when scrubbing animal ID and why this would increase healthy vs. disease discriminability.

> Clarifications to text and expanded discussion

Edits for clarity were made throughout (see pages 27-28 of the revised PDF for a detailed list). These edits included changes to better qualify statements made in the introduction and better explain the utility and novelty of parametric scrubbers and their different linear and nonlinear functional forms. We also expanded our discussion of limitations, addressing multivariate scrubbing and situations where nuisance variability may not be known.  

Again, we sincerely thank the reviewers for helping us improve our paper. We believe we have addressed all reviewer concerns and look forward to any further discussion.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper proposes a framework for modeling 3D pose dynamics capable of disentangling both continuous and discrete nuisance factors, such as speed and animal identity. The authors employ a conditional variational autoencoder (VAE), where the conditioning variables correspond to the nuisance factors to be disentangled. To ensure proper disentanglement of these nuisance factors from the VAE latents, the authors introduce a series of strategies for "scrubbing" the nuisance factors from the latent representations. They demonstrate the utility of this approach in various downstream tasks, including clustering, decodability, and motion synthesis. Additionally, the technique is applied to enhance disease detection in a Parkinsonian mouse model, showcasing its broader applicability.

The paper is well-motivated, well-organized, well-written, and presents appealing results. The AC recognizes the contribution of the proposed framework and its promising results. After the rebuttal, all reviewers leaned toward accepting the paper. The AC concurs with the reviewers and recommends the paper for acceptance. To further enhance the quality of the paper, the AC encourages the authors to incorporate the reviewers' suggestions in the final revision.