Learning Disentangled Behavior Embeddings
Authors: Changhao Shi
Sivan Schwartz
Shahar Levy
Shay Achvat
Maisan Abboud
Amir Ghanayim
Jackie Schiller
Gal Mishne
Venue: NeurIPS 2021 Spotlight (2021)
Keywords: Neuroscience
Animal Behavior
Behavioral Videos
Generative Models

Abstract: To understand the relationship between behavior and neural activity, experiments in neuroscience often include an animal performing a repeated behavior such as a motor task. Recent progress in computer vision and deep learning has shown great potential in the automated analysis of behavior by leveraging large and high-quality video datasets. In this paper, we design Disentangled Behavior Embedding (DBE) to learn robust behavioral embeddings from unlabeled, multi-view, high-resolution behavioral videos across different animals and multiple sessions. We further combine DBE with a stochastic temporal model to propose Variational Disentangled Behavior Embedding (VDBE), an end-to-end approach that learns meaningful discrete behavior representations and generates interpretable behavioral videos. Our models learn consistent behavior representations by explicitly disentangling the dynamic behavioral factors (pose) from time-invariant, non-behavioral nuisance factors (context) in a deep autoencoder, and exploit the temporal structures of pose dynamics. Compared to competing approaches, DBE and VDBE enjoy superior performance on downstream tasks such as fine-grained behavioral motif generation and behavior decoding.

Learning Disentangled Behavior Embeddings
Changhao Shi"
University of California, San Diego cshi bucsd.eddu
Sivan Schwartz, Shahar Levy, Shay Achvat, Maisan Abboud, Amir Ghanayim
Technion Israel Institute Technology @campus........... wwwhniecc.nn.eee ac
(sivan. (sivan.s,shahar55, shay. aww.acchaatma.. mai.anaabbodd, amir gh @
Jackie Schiller
Gal Mishne
Technion Israel Institute of Technology jackie@technion.c ac.il
University of California, San Diego gmi shne@ucsd. edu
Abstract
To understand the relationship between behavior and neural activity, experiments in neuroscience often include an animal performing a repeated behavior such as a motor task. Recent progress in computer vision and deep learning has shown great potential in the automated analysis of behavior by leveraging large and high-quality video datasets. In this paper, we design Disentangled Behavior Embedding (DBE) learn robust behavioral embeddings from unlabeled, multi-view, high-resolution behavioral videos across different animals and multiple sessions. We further combine DBE with stochastic temporal model to propose Variational Disentangled Behavior Embedding (VDBE), an end-to-end approach that learns meaningful discrete behavior representations and generates interpretable behavioral videos. Our models learn consistent behavior representations by explicitly disentangling the dynamic behavioral factors (pose) from lime-invariant, non-behavioral nuisance factors (context) in deep autoencoder, and exploit the temporal structures of pose dynamics. Compared to competing approaches, DBE and VDBE enjoy superior performance downstream tasks such as fine- fine-grained behavioral motif generation and behavior decoding.
Introduction
Understanding the relationship between animal behavior and neural activity is a long-standing goal in neuroscience. To this end, recent advances in deep learning and computer vision has led to significant progress in the essential task of automatic analysis of high-resolution behavioral videos.
To extract behavioral information from rich video recordings, two avenues of research relying on deep learning have been proposed: landmark-based pose estimation methods [22 26 32 and autoencoder-baase dimensionality reduction methods (2). Pose estimation methods characterize animal behavior with the trajectories of body-part landmarks. Such methods train deep neural networks to predict manually labeled landmarks from raw video frames, then use the trained models to generate trajectories for new videos. While these methods have been highly successful in behavior analysis, they 1) require manual labeling which is often expensive; 2) heavily rely on the manual selection of the landmarks, which may differ across human annotators; and 3) struggle with subtle behavior that is hard to track, such as facial movements.
"qqual contributions
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

The complementary approach seeks to reduce the high-dimensional video data to low-dimensional latent factors through non-linear dimensionality reduction. These methods [2] rely on deep autoen- coders to encode video frames by solving an unsupervised reconstruction task. The learned latent embeddings can then be used for downstream tasks, such as clustering, neural decoding, etc. While unsupervised learning does not rely on manual labeling, existing approaches currently suffer from certain drawbacks. First of all, canonical autoencoders can be easily biased by the varying visual nui- sances across videos, such as lighting, distance to camera and physical attributes of animals, resulting in substantial distributional shifts among the learned behavior embeddings. While non-behavioral visual variability is usually negligible within single session videos. it makes downstream analysis difficult when applied to more than one session and to different animals. Secondly, end-to- and training videos becomes challenging when considering temporal structures, traditional temporal modeling methods are mostly designed for low-dimensional data. How to better exploit temporal structures of nigh-dimensional videos for action recognition, motion planning. etc., remains an open problem for computer vision research.
In this paper, we design Disentangled Behavior Embedding (DBE) to learn robust behavior embed- dings from large, unlabeled, multi-session videos. Inspired by previous works (33), DBE mitigates the distributional gaps in multi-session videos by explicitly disentangling the dynamic behavioral factors (pose) from time-invariant, non- non-behvvooral factors (context). A video frame is thus repre- sented by a pair of disentangled pose and context components in contrast to single entangled representation. For a given video, the context component is designed to be time- invariiar to exclude any behavioral dynamics, whereas the pose component is bottlenecked to keep it from duplicating context information. DBE latents can be used in conjunction with temporal analysis models, e.g., Variational Animal Motion Embedding (VAME) 21. to perform downstream analysis. We also propose Variational Disentangled Behavior Embedding (VDBE), a fully end-to-end trainable model, which further exploits the temporal structures of the pose components using a stochastic dynamic model. Using variational inference, a Gaussian mixture prior is trained to capture the multi-modality of the transition between consecutive pose components and generate both continuous and discrete representations of the underlying animal behaviors in videos. Our methods are not only fully unsuper- vised, but also enjoy superior performances on downstream tasks such as behavioral state estimation and arniggraindd behavioral motif generation. To summarize, the main contributions of this paper are:
1. We develop DBE to tackle the distributional shift across multi-session videos with adaptation to the behavioral neuroscience setting, c.g., extension to multi-view videos, proper design of context embedding with standard behavioral neuroscience paradigms.
2. On top of DBE, we design VDBE to learn both continuous and discrete latent representations for simultancous embedding and segmentation within the same model. VDBE is end-to-end trainable in an unsupervised fashion, alleviating the need to train a second post-hoc model applied to latent embeddings.
2 Related Works
2.1 Video disentanglement
Disentanglement of the pose and context in a video is area of increasing interest in computer vision. The explicit separation of the pose from the context not only facilitates the analysis of the temporal dynamics but also makes video manipulation easier. As a pioneer in this field, MCNet 30) is an end-to-end framework that models motion (pose) and content (context) independently. MCNet uses a convolutional LSTM to model temporal dynamics and shows good performance on pixel-level video prediction on real-world datasets. Sharing similar ideas, DRNet [5] also separates the pose from the content but models temporal dynamics directly on low-dimensional pose embeddings. also utilizes an explicit adversarial loss for disentanglement. Although DRNet shows superior performance in video prediction, it is not end-to-end trainable. DDPAE [33 tackles videos of multiple targets on the moving MNIST dataset and performs disentanglement on each individual target. Yingzhen and Mandt 33 proposed a disentangled VAE and shows controllable video generation on the synthetic Sprite dataset. MoCoGAN (29 employs generative adversarial network that decomposes the random noises into a content part and motion part for video generation. S3VAE R4 learns disentangled context and pose representations of videos with self-supervised regularization. In our approach

---

build a disentanglement mechanism in a similar paradigm as [33], by fixing context embeddings across time and constraining the dimension of pose embeddings.
2.2 Stochastic dynamic modeling
Latent state- space models (SSM) are widely used to model the temporal structures for sequential data [
17 20, 71. While these models have achieved fruitful results in many areas such as natural language processing and speech recognition, adapting these models high-dimensional, non- synthetic videos is still challenging to this date. Hafner et al. [11] proposed PlaNet, with a recurrent state-space model at its core, to learn environment dynamics from high-dimensional, synthetic videos and auxiliary action variables for planning. Franceschi et al. [9] proposed latent dynamical model enhanced by residual connections for stochastic video prediction. Although there are other papers on stochastic video generation and prediction on low-dimensional frame embeddings 6 [8), these methods tackle the problems in an auto- auto-regressive fashion and thus offer no single latent state vector that captures all the dynamics, as opposite to state-space models. Our approach closely relates to
[T] [9] but differs by using a multi-modal distribution for the transition model and the ability to generate both continuous and discrete representations.
2.3 Behavioral video analysis
Recently, supervised deep pose estimation methods have been widely used for behavior analysis for many animal models. Methods such as DeepLabCut (DLC) [22), DeepPoseKit [TO] and LEAP (26) train deep neural networks to produce confidence maps for predefined landmark annotations, all in a similar fashion. Leveraging the success of DeepLabCut, Wu et al. 32 improved the quality of pose estimation by imposing additional spatial and temporal constraints on multiple landmarks during training. Luxem et al. [21] further developed recurrent variational autoencoder (VAME) encode the pose estimation from DLC to a higher-level behavioral space. Prior to deep learning approaches, Kabra al. (15] provided supervised pipeline automatically annotate animal behavior based on user-defined labeling.
On the other hand, unsupervised dimensionality reduction methods like PCA have been used to analyze behavior for decades [28 4 3I. More recently, BehaveNet (2 presented meaningful results of behavior segmentation and neural decoding. BehaveNet first uses deep convolutional autoencoder to compress behavioral videos, then fits an autoregressive hidden Markov (ARHMM) model to exploits the temporal structures of the learned frame embeddings. However, BehaveNet was evaluated only on videos in single experimental session and cannot handle the distributional shift across multiple sessions. Our approach, which falls into this second category, not only inherits the advantage of being unsupervised but also handles the distributional gaps across multiple sessions by explicitly disentangling poses from contexts.
3 Methods
3.1 Problem formulation
Formally, given a behavioral video F1. of length T, we are interested in disentangling the dynamic behavioral factors from the time-invariant non-behavioral nuisances in the video in an unsupervised fashion. Our goal is to learn meaningful emhedding =i.r that capture common behavioral structures across multiple animals in multiple recording sessions. We assume that the animal performing repetitive behavioral protocol, such as mouse performing a forepaw reach or lever press [19] 24], and not freely moving. Additionally, in behavior experiments, videos are often recorded from different angles, resulting in multi-view data, e.g. (Ia, F66 166)1: 1.r where a and b denote the two views. For milti-view videos, we treat each view as a separate image and fuse the outputs of encoders of each view as the representation of this timestamp, whereas for mono-view videos the embedding of the frame is the natural representation. For the rest of this paper, we will slightly abuse the notations and omit the view subscripts, e.g., Fi. refer to either mono-view frames or multi-view frames (ra, 2&)) 31)1.r I. and ž1. refer to either the embeddings of mono-view frames or the fused embeddings of multi-vier frames.
3

---

Video Clip
Enc
Dynamicc Model
Continzous Vartabies
RNN
SSM
Pose
Discrete Varlables
Pose
Context Frame
Enc
Reconstruction
Context
Concatenation
Figure 1: Overview of VDBE mono-view video. The pose encoder encodes individual frames into a sequence frame embeddings, which are then passed through stochastic dynamic model to generate pose embeddings. The context embeddings are generated from single conditional context frame by the context encoder. Both embeddings are concatenated at each timestamp as inputs to decoder which reconstructs the original video clip. The dynamic model (top right) uses an RNN to infer both latent continuous embeddings and discrete states that capture the transition of the frame embeddings, and uses a linear sate- model for generation. When the dynamic model is not used, the full VDBE model becomes DBE. The input video can be either single view or multi-view For multi-vien videos, encoders and decoders used for each view separately, and the pose and context components are concatenations of the corresponding encoder outputs from each view.
3.2 Disentanglement
In order factor out the non-behavioral factors, approach utilizes separate lines of encoders, context encoder E and pose encoder E", for each view of the videos. Assuming the non-behavioral factors time-invariant within each video, the convolutional context encoder E infers video-level context embedding 2" from a single frame the video. For multi-view videos, the context of each view is further concatenated into single context embedding 2", then used for the reconstruction of all the frames in this video. The pose encoder E. shares the same architecture with E., and encodes the behavioral dynamics of the animal from all video frames T1:T into the pose embeddings = For multi-view videos, the outputs of pose encoders are concatenated to get =f.7. The temporal structures of the pose component are exploited by a stochastic dynamic model stacked top of the pose encoders, which we will detail in the following. To reconstruct the input video T1.7, the context embedding z" is duplicated and concatenated with each pose embedding 2f-r.. then for each view these concatenated embeddings are further decoded into frame reconstructions by the corresponding convolutional decoders D.
Because the context embedding =f is time-invariant and therefore incapable of capturing dynamics, ensure that it contains behavioral information about the animal. Thus, the key question about the disentanglement is how to ensure that the pose embeddings contain no context information. To enforce disentanglement, we create bottleneck on the pose embeddings inspired by (33). We choose the dimension of the pose embeddings to be as small as possible in the premise of good reconstruction. This ensures that the capacity of pose embeddings only sufficient for capturing the temporal dynamics without any redundant context information. As we will show in the Sec 4 the disentangled pose embeddings of different sessions are well-alie well-aligned with each other, in sharp contrast against the results from canonical autoencoders where the embeddings of different sessions form distinct clusters. The disentangled context embedding, on the other hand, capture these non-behavioral nuisances and form into clusters by sessions as expected. The full disentanglement mechanism is shown in Figure []
3.3 Dynamic model
model the temporal dynamics of the pose embeddings ~Κ. using a sequential variational au- toencoder. For the sake of brevity, we will use to refer to z² in this section. Similar abbreviations are applied for other variables as well. We assume that the pose components govern the latent state of the behaviors and we want to model the temporal transition of the states. In typical stochastic

---

91
92
93
94
91
92
9s
94
2
22
Z3
22
#3
F
J/2
T2
J3
(a) Generation
(b) Inference
Figure 2: The generation (a) and inference (b) process of stochastic dynamic model. Round nodes represent stochastic variables and squared nodes represent deterministic variables. Note that in this illustration, 21 is inferred from two conditional frames ø and z2; and for c₁ and St only the paths from their immediate past frames are shown for the sake of clarity.
state-space model, the transition from the previous state ž to the current state is modeled stochastically. However, a purely stochastic transition makes hard for the model to keep long-term information [TT]. Thus,
factor out the stochasticity of the dynamics into separate component This means that at a given timestamp, the stochastic latent variable 8 is first sampled from prior distribution, usually Gaussian distribution, then the current state 2 is generated from the previous state 2 and the stochasticity s deterministically.
While this strategy has been proved to be effective in many scenarios 6 11 9.. the uni-modal Gaussian transition distribution limited for behavior analysis because it does not capture the multi-modality of behavior. For example, in the video dataset we will show in Section the transition of the animal poses is usually caused by set of distinct, prototypical actions, such as lift, reach and grab. A uni-modal Gaussian transition distribution is inadequate for modeling such multi-modal transitions. Furthermore, for the purpose of motif segmentation it offers no discrete states which are highly desired.
more suitable choice for capturing such multi-modality a Gaussian mixture model 27 Shu et al. 27
design the transition distribution in the generative model to be a Gaussian mixture while keeping posterior inference approximation to uni-modal Gaussian. A closed-form approximation of the exact KL. divergence is then used in evidence lower bound for optimization. In this paper, we use an alternative approach to introduce the Gaussian mixture. Instead using a single stochastic latent variable with a Gaussian mixture prior, we decompose it into two stochastic variables, the mixture index e and Gaussian noise g. This Gaussian mixture prior also enables the model to perform embedding (2)) and segmentation (ct) simultaneously. With all these, we can dive into the details of our variational autoencoder, as illustrated in the zoomed-in region in Figure]
Generation The generative procedure shown in Figure 2a We model the transition between the previous state 2t-1 and the current state 2t using Gaussian mixture. The generative model is:
21 ~ NN,, I), 241 So( St), G ~ cat(TT((()) 9: ~ (O0, I), 8, Pe. to, @ Og It (Deco zc));
(2)
where wa and fo are linear projection layers. At each timestamp, stochastic variables o, and gi model the discrete state and the additional stochasticity, then the continuous state 2 is generated with deterministic path from 2;-1, C; and gt- Note that although both C and 9i capture the stochasticity of the transition, only c is dependent on the previous state 2/-1- is sampled from categorical distribution parameterized by 2 and gt is simply sampled from a standard Gaussian distribution.
The inferred e is used for for motif segmentation in the test stage. We assume the covariance matrix of each Gaussian component is diagonal and @ denotes glement-wise multiplication (Hadamard product).
5

---

Inference Because of the non-linearity in the generative model, the true posterior distributions of the latent variables are intractable. As shown in Figure 2b we use variational inference to approximate these posteriors. The inferred posterior is factorized as qo(CI:T, 91.T, =1) = ((=1g(e/221:) ((
where q(c((21:)) is conditional categorical distribution, and ((9 (21:7) as well as [1(e)) is conditional Gaussian distribution. We use shared GRU network followed by separate projection heads for g(c)(æ1:)) and q(9 (T1:)) For (22 [z1,),, we use an MLP whose inputs are the first C' conditional frames in the sequence. At test-time, we also use these inference networks to generate behavior representation, i.e. we use the mean of the Gaussian distribution and the highest category of the categorical distribution as the continuous and discrete representations of the behavior of each frame.
Optimization
We use the following evidence lower bound (ELBO) for optimization:
(o. Basler
(4)
KDxI (g ( (9o(0))) (00 yD (
where a, 33 and yy are the trade-off parameters that control the information flow to each stochastic latent variables C1:T, 91.T and respectively [12). The first log-likelihood term corresponds to the frame reconstruction and the other KL-divergence terms are the regularization of the inferred stochastic latent variables to their prior knowledge.
The above expectation estimated with single sample from the approximated posterior distribution. To allow gradients to back propagaee through the non- non-differentiable sampling operation, we use the re- p--aammeterrzation tricks. For the Gaussian posterior distributions g(gh (a1:T) and 7(2) |=1:c), we re-parameterize the distribution into their estimated mean and standard deviation with standard Gaussian noise [16]. For the categorical posterior distribution (((())1:)) we use the Gumbel-Softmax re- parameterization trick [14] the sampling process is re- p-rrrmmtterized into the estimated discrete gee-robability and a Gumbel noise, following by a softmax normalization. The outputs of softmax are also discretized to one-hot encodings to ensure consistency between training and testing. Note that theoretically one can compute the exact expectation over the discrete latent variable o without using Monte Carlo estimation, but experiments show that one- time sampling can generate satisfying results with the advantage of being more efficient.
4 Results
We evaluate our methods on two multi-session video datasets: the Hand-reach dataset [99 we collected, and the Wide-Field Calcium Imaging (WFCI) dataset 24 25). The Hand-reach dataset contains vidcos of head-fixed mice who were trained to reach from their perch to food pellet. The datasets consists of 263 suc- cessful trials of expert animals (who have learned the motor task) in 8 sessions, and includes fine-grained be- havior. In successful execution of the task, the animal will lift its paw from the perch, reach and grab the pellet and bring it to its mouth and eat it. The animal might perform multiple grab attempts in single trial before successfully grabbing the pellet. An illustration of the Hand-reach dataset is shown in Figure 3 The videos were acquired on multiple dates, within which the ex- perimental layout is potentially different, e.g., location of the table. Two cameras simultaneously record from the front and right side the mouse. Each video is 12 seconds long, with a frame rate of 200Hz. The video begins with 4-seconds inter-trial interval (ITI), the end of which we set as the origin (0) of our time axis. A cue of 0.2 seconds is shortly followed by table-turning which brings the food in front of
Tane
0.2 sec
Rotation
0.2
Time (sec)
-0.25 b
1.75
Figure 3: Hand-reach dataset. Mice were head-fixed and trained to grab food pel- let. Trial structure: 4 seconds ITT period followed by the tone and table turning. We analyze the [-0.25, 1.75] segment which contains most of the significant behaviors for expert mice.

---

the mouse. Since the mice in this dataset are experts, and thus mostly stationary before the cue and chewing the pellet after successful execution of the hand-reach, we use a 2-second clip of the main dynamic behavior of each full video to train the model. Each clip starts at 0.25 second before the cue and ends at 1.75 seconds after, which contains most of the motions of interest. The context frame for each video is selected from the ITT period, sampled at 1 second before the cue, when most of the subjects are stationary.
In the WFCI dataset (an extended version of the dataset in (2]., head-fixed mice perform a visual decision making task, using levers to initiate trials and reporting choices by licking from spouts. The behavioral videos are recorded from a side and a bottom view of the subjects, with length of 189 frames and frame rate of 30Hz. The dataset contains 3708 videos from 8 sessions of 4 animals in total. We randomly select 80% of the videos as training set and use the remaining 20% as test set.
We train both DBE and VDBE, without and with the stochastic dynamic modeling component. We compare our methods against two behavior embedding methods: BehaveNet (2) and VAME (21]. with implementation details of each provided Appendix [ and Appendix [ respectively. In Section [4.] and 4.2, we demonstrate the efficacy of our disentanglement component. In Section (.3 we show both DBE (combined with motif segmentation model) and VDBE achieve superior results for motif segmentation compared to the original models.
4.1 Disentanglement
To demonstrate the efficacy of disentanglement, we focus on session recognition: given the behavior embedding of specific video frame, can we identify which session (animal and date) this frame comes from? We train a simple linear classifier to classify the experimental session from different pose embeddings, and ideal disentanglement should generate session-agnostic pose embeddings. The accuracy of the session classification for each model is reported in Table [ VDBE model generates session- -agnossi pose embeddings, compared the plain CAE autoencoder used in Behavenet. Although DLC also provides session-agnosta coordinates, selected markers do not encode enough useful information for extracting robust behavioral motif's across sessions (Section 4.3). The context embedding of our methods (DBE-context and VDBE-context) accurately classify the session, further demonstrating disentanglement of context from pose.
DBE as well as our full
We further visualize the advantage disentanglement in Figure We plot the 2D PCA applied to the BehaveNet and DBE embeddings of two videos that show similar behaviors but are recorded in different sessions. The plain CAE of BehaveNet generates separate pose embeddings even though the behaviors are similar. This distributional shift will be problematic when wants transfer the knowledge learned in one session to another. This means that one needs to learn a new model every time when there are new sessions, and learn a mapping across animals relate the embeddings of the same behavior. In contrast to BehaveNet, our DBE nicely aligns the two videos in the pose embedding space. As will show in the later section, this consistency is crucial to motif segmentation and behavior decoding. More results comparing BehaveNet and DBE embeddings, as well as ethograms of both the Hand-reach and the WFCI dataset, are provided in Appendix and G
BehaveNet
DBE
Model
Hand-reach
WFCI
BehaveNet
100.00
99.99
Mouse]
DLC
21.19
DBE VDBE
28.21
35.49
17.94
19.54
DBE-context
88.59
100.00
VDBE-context
96.19
100.00
Figure 4: Embeddings of 2 similar videos (Hand-reach)
Table I: Session classification
4.2
Trajectory decoding
DLC is robust method used to track manually selected landmarks in behavioral videos. A good behavioral embedding should contain the information encoded by DLCC To illustrate that our approach learns meaningful behavior embeddings, we perform simple linear regression task from the DBE
7

---

embeddings to the DLC coordinates in the Hand-reach dataset. A good fit of the linear regression model suggests that our learned embedding contains the DLC coordinate information.
We place 5 markers on the center and 4 digits of the right paw of the mouse in each view of the videos, and train DLC model to track the coordinates of those 10 markers. Due the differences in spatial layout of the sessions (e.g., table location), we fit separate linear regression models to each marker for each individual experimental session. Only DLC coordinates that have confidence score larger than 0.95 are used for the training of linear regression.
0.85
0.00
0:75
0:70
0.65
0.60
1-Hevt5
1-Havis
1-tevzo
M
rezz
rebza
Ha
(a) DLC trajectories prediction
(b) Goodness-offfin of linear regression
Figure 5: Predicting DLC trajectories from disentangled embeddings on the Hand-reach dataset. (a) Example video overlaid with true DLC trajectories (red) and the linear regression model predictions (green). (b) Ri averaged across each coordinate for each individual session.
An example of DLC predictions and the average R of each individual linear regression model are shown in Figure 5 capturing the right paw of the mouse moving from the perch to the pellet on the table. The high R² scores in Figure 5b|show show that the linear regression models are well-fitted to the data, which indicates that the DBE embeddings manifest meaningful information about the selected markers.
Figure [a] the predictions are well aligned with the true DLC trajectories,
4.3 Motif segmentation
Learning fine-grained behavior motifs from videos is an essential but challenging task for analyzing animal behavior, especially when supervision is available. In this section, compare our approach with other competing methods for motif segmentation on the Hand-reach dataset. To evaluate the quality of the generated motifs, we select 10% of the videos (25 videos) and perform expert manual labeling of the selected videos with a set of seven behaviors of interest: On-Perch, Lift, Reach, Grab, To-mouth, At-mouth, and Back-To-Periee (4730 labeled frames). The rest of the videos are used as the training set for motif segmentation methods, i.e. clustering. For each method, we assign the learned motifs to the labeled set of behavior, where the mapping is selected maximize the clustering accuracy of holdout videos. Thus, each motif corresponds to a "mega-cluster" that may or may not contain more than one inferred discrete state. Note that we set the number of the discrete states & to be larger than the number of target labels for two reasons. The first reason is that only part of the frames are labeled and the target labels do not cover all the possible behavior of the mice. Extra clusters are necessary to capture these additional behavior. The second reason is that setting larger number of motifs rather than the exact number of labels accounts for inter-class versus intra-class variability. For example, this enables separating highly variable behavioral label like Reach into more than one motif without having to compensate for that by merging At-mouth and To-mouth into a single motif. We evaluate the performance of motif segmentation on three clustering metrics, accuracy, Normalized Mutual Information (NMI), and Adjusted Rand Index (ARI).
We use the DBE embeddings in two variations for motif segmentation: DBE+ARHMM which replaces the CAE embeddings of the BehaveNet model with DBE embeddings, and VBE+V which replaces the DLC trajectories with DBE embeddings as inputs to VAME. As shown in Table and Figure 6] disentanglement is the key to achieve good behavioral clusters across different animals and multiple sessions. Combining DBE for frame embedding and 30-cluster VAME (DBE+VAME) for temporal dynamics and segmentation achieves the best accuracy, NMI and ARI. Our full VDBE model has very similar performance, with both models exhibiting ~ 30% higher accuracy than BehaveNet and over 20% higher accuracy than the original VAME framework (relying on DLC inputs). These results also show that using a more powerful temporal model such as VAME can
8

---

Table 2:
motif segmentation for the Hand-reach dataset
Model
Unsup.
Dis.
End2End
Accuracy
NMI
ARI
BehaveNet (k=15)
X
50.52
30.89
20.98
BehaveNet (k=30)
x
X
53.09
33.44
25,09
DBE + ARHMM (k=15)
/
X
64.04
42.06
36.08
DBE + ARHMM (k=30)
V
X
71.49
48.96
45.76
DLC + VAME (k=30)
X
X
57.17
42.07
44.09
V
DBE + VAME (k=15)
X
71.31
59.47
51.46
DBE + VAME (k=30)
X
81.25
67.31
67.33
VDBE (k=30)
V
79.53
64.85
66.94
hed
08
Crab
Mouth
ToMouth
#tNouth
AlMouth
02
Back
Back
Readh
Grab
Tottotth
Aotounn
Back
Perch
Reaid
ubab
Tollouth
Back
Herch
Tithoolf
Back
Prcckiee
Pheciled
(a) BehaveNet (k=30)
(b) DBE+VAME (k=30)
VDBE (k=30)
Figure 6: Confusion matrices of BehaveNet, DBE+VAME VAME and VDBE for the Hand-reach dataset.
benefit motif segmentation. With disentanglement, the confusion matrices Fig. 6 are more diagonal which indicates better clustering. Figure S shows examples of random segments from VDBE motif's corresponding to 4 different labels. Note that VDBE successfully clusters the behaviors even when the context varying. Additional results can be found in supplementary materials.
4.4
Controllable behavior generation
One advantage of VDBE and ARHMM-bassd models compared to VAME is that we can decode synthetic videos from behavior embeddings, which demonstrates better interpretability and allows potential applications to neural decoding. In this section, we further investigate behavior decoding by manipulating the context components. In order to generate synthetic videos that replicate source behavior with new context, we use the inferred discrete behavior representations C1:7 from the source video and the context embedding from the target session, while g1.7 and 2 are sampled from their prior distributions. If the target session is the same as the source session, the synthetic video should be similar to the source video. The results are shown in Figure & Note that the behavior in the source video is replicated in both generated videos with the same and different contexts. This indicates that our model successfully learns the dynamics of the behavior.
Conclusions
In this paper, we tackle the problem of unsupervised behavior representation learning from multi- session videos. A disentanglement mechanism is designed to separate session-wiss wise non-behavioral factors from the useful behavioral information, and a stochastic dynamic model is jointly trained to model the temporal dynamics of behaviors. Compared with competing methods, our method shows superior performance multiple tasks such fine-grained motif segmentation, meanwhile also enjoys the advantages of being unsupervised, end-to-end trainable, and more interpretable.
Despite the above improvements, couple of open questions still remain and will be explored in future work. Firstly, the current model which relies on a predefined number of states is limited in generalization of the motif assignment to videos which include novel unseen behavior. Secondly, our disentanglement assumes the non-behavioral nuisances are stationary through video, which is not always true. Future work should benefit from allowing slow dynamics in context embeddings. Thirdly,

---

(a) Lift
(b) Reach
(c) Grab
(d) At-Mouth
Figure 7: Motifs learned by VDBE. The blue channel indicates the start of the motif and the red channel indicates the end. Note that in each column the behavior is consistent but the context is varying (e.g., location of the table and identity of the mouse).
Source- Source-Mroic Mrovel
Gencaaed Monsaand--onnee Monee
Generated Maniee
Figure 8: Source (top row) and generated videos of the same discrete behavior sequence, with the same (middle row) and different (bottom row) contexts. Each row is plotted at intervals of 20 frames.
with the distributional shift in videos confirmed, it would be interesting to study disentangling other visual behavior representations such as DLC for multi-session analysis. Finally, with the learned behavioral embeddings consistent across sessions, future works can leverage this representation relate behavior to neural activity
Regarding socictal impacts of our study well behavior analysis research in general, although there a significant gap between the head-fixed animal behavior studied here and naturalistic human behavior, we should always be careful about the negative consequences for human monitoring.
Code is available http:://tttaaaammmmmiiiiaaaaaaaa Betprettannnnneeeai-----oo-eeoe
Acknowledgement
This work was supported by in part by the National Institutes Health grant No. RO1EB026936. thank Simon Musall and Matthew Whiteway for providing the WFCI dataset and their generous help.
10

---

References
[1] M. Babacizadch, C. Finn, D. Erhan, R. H. Campbell, and S. Levine, Stochastic variational video prediction. arXiv preprint arXiv:
2017.
[2] E. Batty, M. Whiteway, S. Saxena, D. Biderman, T. Abe, S. Musall, W. Gillis, J. Markowitz,
A. Churchland, P. Cunningham, et Behavenet: nonlinear embedding and bayesian neural decoding of behavioral videos. Advances in Neural Information Processing Systems, 32: 15706-15717, 2019.
[3] Bayer and C. Osendorfer. Learning stochastic recurrent networks. arXiv:1411.7666 2014.
arXiv preprint
[4] G. J. Berman, D. M. Choi, W. Bialek, and J. W. Shaevitz. Mapping the stereotyped behaviour of freely moving fruit flies. Journal of The Royal Society Interface, 1 11(99):0010677 2014.
[5] E. Denton and V. Birodkar. Unsupervised learning of disentangled representations from video. In Proceedings of the 31 International Conference on Neural Information Processing Systems, pages 4417-4426, 2017.
[6] E. Denton and R. Fergus. Stochastic video generation with a learned prior. In International Conference on Machine Learning, pages 1174-1183. PMLR, 2018.
[7] Z. Dong, B. Seybold, K. Murphy, and H. Bui. Collapsed amortized variational inference for switching nonlinear dynamical systems. In International Conference on Machine Learning, pages 2638 2647.267..22 PMLR, 2020.
[8] M. Fraccaro, S. Kamronn, U. Paquet, and O. Winther. A disentangled recognition and nonlinear dynamics model for unsupervised learning. arXiv preprint 17Xiv:1710.05741, 2017.
[9] J.-Y. Franceschi, E. Delasalles, M. Chen, Lamprier, and P. Gallinari. Stochastic latent residual video prediction. In International Conference on Machine Learning, pages 3233-3246. PMLR, 2020.
[10] M. Graving, D. Chae, H. Naik, L. Li, B. Koger, B. R. Costelloe, and I. D. Couzin. Deepposekit, a software toolkit for fast and robust animal pose estimation using deep learning. Elife, c47994, 2019.
[11] D. Hafner, T. Lillicrap, Fischer, R. Villegas, D. Ha, H. Lee, and J. Davidson. Learning latent dynamics for planning from pixels. International Conference on Machine Learning, pages PMLR, 2019.
[12] I. Higgins, L. Matthey, A. Pal, C. Burgess, X. Glorot, M. Botvinick, S. Mohamed, and A. Ler- chner. beta-vae: Learning basic visual concepts with a constrained variational framework. 2016.
[13] J. Hsich, B. Liu, D.-A. Huang, L. Fei-Fei, and J. C. Niebles. Learning to decompose and disentangle representations for video prediction. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, pages 515-524, 2018.
[14] E. Jang, Gu, and B. Poole. Categorical reparameterization with gumbel-softmax, arXiv preprint arXiv: 1.01 144, 2016.
[15] M. Kabra, A. A. Robie, M. Kivera-Alba, S. Branson, and K. Branson. Jaaba: interactive machine learning for automatic annotation of animal behavior. Nature methods, 10(1):64, 2013.
[16] D. P. Kingma and M. Welling. Auto-encoding variational bayes. arXiv preprint arXiv: 1312.6114, 2013.
[17] R. Krishnan, U. Shalit, and D. Sontag. Structured inference networks for nonlinear state space models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 31, 2017.
[18] A. X. Lee, R. Zhang, F. Ebert, P. Abbeel, C. Finn, and S. Levine. Stochastic adversarial video prediction. arXiv preprint arXiv:1804.00 0,553,,, 2018.
[19] S. Levy, M. Lavzin, H. Benisty, A. Ghanayim. U. Dubin, S. Achvat, Z. Brosh, F. Aeed, B. D. Mensh, Y. Schiller, et al. Cell-type--pecific outcome representation the primary motor cortex. Neuron, 107(5):55--97,, 2020.
[20] S. Linderman, M. Johnson, A. Miller, R. Adams, D. Blei, and L. Paninski. Bayesian learning and inference in recurrent switching linear dynamical systems, Artificial Intelligence and Statistics, pages 914-922. PMLR, 2017.

---

[21] K. Luxem, F. Fuhrmann, J. Kürsch, S. Remy, and P. Bauer. Identifying behavioral structure from deep variational embeddings of animal motion. bioRxiv, 2020.
[22] A. Mathis, P. Mamidanna, K. M. Cury, T. Abe, V. N. Murthy, M. W. Mathis, and M. Bethge. Deeplabcut: markerless pose estimation of user-defined body parts with deep learning. Nature neuroscience, 21): 1291))228811199 2018.
[23] G. Mishne, R. Talmon, R. Meir, J. Schiller, M. Lavzin, U. Dubin, and R. R. Coifman. Hierar- chical coupled geomeerry analysis for neuronal structure and activity pattern discovery. IEEE Journal of Selected Topics in Signal Processing, (0(7):1238-12533 2016.
[24] S. Musall, M. T. Kaufman, S. Gluf, and A. K. Churchland. Movement-related activity dominates cortex during sensory-guided decision making. BioRxiv, page 308288, 2018.
[25] S. Musall, M. T. Kaufman, A. L. Juavinett, S. Gluf, and A. K. Churchland. Single-trial neural dynamics are dominated by richly varied movements, Nature neuroscience, 22(10):1677-1666, 2019.
[26] T. D. Percira, D. E. Aldarondo, L. Willmore, M. Kislin, S. S.H.. Wang. M. Murthy, and J. W. Shaevitz. Fast animal pose estimation using deep neural networks. Nature methods, 16(1):
117-125,
[27] R. Shu, J. Brofos, F. Zhang, H. H. Bui, M. Ghavamzadeh, and M. Kochenderfer. Stochastic video prediction with conditional density estimation. In ECCV Workshop on Action and Anticipation for Visual Learning, volume 2, page 2, 2016.
[28] G. J. Stephens, B. Johnson-Kerner, W. Bialek, and W. S. Ryu. Dimensionality and dynamics in the behavior of c. elegans. PLoS Comput Biol, ((4)e100002) 2008.
(29] S. Tulyakov, M.-Y. Liu, X. Yang, and J. Kautz. Mocogan: Decomposing motion and content for video generation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1526-1535, 2018.
[30] R. Villegas, J. Yang. S. Hong. X. Lin, and H. Lee. Decomposing motion and content for natural video sequence prediction. In Sth International Conference on Learning Representations, ICLR 2017. International Conference on Learning Representations, ICLR, 2017.
[31] A. B. Wiltschko, M. J. Johnson, G. Turilli, R. E. Peterson, J. M. Katon, S. L. Pashkovski, V. E. Abraira, R. Adams, and S. R. Datta. Mapping sub-second structure in mouse behavior.
Neuron, 88(6): 122-1135, 2015.
[32] A. Wu, E. K. Buchanan, M. Whiteway, M. Schartner, G. Meijer, J.-P. Noel, E. Rodriguez, C. Everett, A. Norovich, E. Schaffer, al. Deep graph pose: semi-supervised deep graphical model for improved animal pose tracking. bioRxiv, 2020.
[33] L. Yingzhen and S. Mandt. Disentangled sequential autoencoder. In International Conference on Machine Learning, pages 1670-56797 PMLR, 2018.
[34] Y. Zhu, M. R. Min, A. Kadav, and H. P. Graf. S3vae: Self-supervised sequential vae for repre- sentation disentanglement and data generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6538-6547, 2020.
12


Reviews:
[{"review_id": "whBQOHrQ5f", "reviewer": "Reviewer_c1eX", "summary": "The authors propose a model for learning representations from videos of animal behavior, such that temporally-variant factors are disentangled from temporally-invariant factors. The method is trained in an unsupervised fashion based on video reconstruction, with one branch that contains pose dynamics and another branch that contains context information. The pose branch also includes a stochastic dynamic model. Experiments are on two mouse datasets, Hand-Reach and WFCI, with most of the results being on Hand-Reach. The model is able to reconstruct videos of mouse hand-reach actions, and in addition be applied to a few tasks, such as motif segmentation.", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "1. Most of the experiments are only on a single dataset collected by the authors (with a specific task and organism - mouse hand-reach). The second, more commonly used dataset WFCI, seemed to have only been used for one sub-experiment (Table 1). It is difficult to know if the proposed model will generalize to other behavioral video settings, past a single reach-grab action. \n2. The generated videos look very shaky (the paw of the mouse flicks back and forth a lot in many samples) in the supplementary materials, compared to what the authors presented in the main paper (Figure 8). Could the authors comment on this? It is unclear to me if the learned dynamics are reasonable.\n3. The consistency of the method is unclear as the authors did not do multiple runs (Checklist 3c).\n4. The related works and baselines seem incomplete - there are some other works disentangling time-variant from time-invariant factors from video, such as [A], [B]. The author may also want to consider including some other methods for learning disentangled representations as baseline comparisons. Since parts of the model are based on [31], at a minimum, I feel the proposed method should be compared to [31] as a baseline.\n5. Minor point: the names chosen by the authors seem too general (ex: “Disentangled Autoencoders”). I’m not sure if this is appropriate since large parts of the model seem to be based on [31] “Disentangled Sequential Autoencoder” and this work is still sequential.\n6. I like the motivation of the paper (better understand the relationship between neural activity and behavior), but the authors have left connecting to neural activity for future work.\n7. The authors emphasize interpretability a lot (ex: line 55 – “more interpretable”) but I feel the experiments don’t really demonstrate this aspect of the model. The model is still a black-box autoencoder - I’m not sure if the model is “more interpretable” than baselines as claimed. Could the authors comment more on why this model is interpretable? \n\n[A] Zhu, Yizhe, et al. \"S3VAE: Self-supervised sequential VAE for representation disentanglement and data generation.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020.\n\n[B] Tulyakov, Sergey, et al. \"Mocogan: Decomposing motion and content for video generation.\" Proceedings of the IEEE conference on computer vision and pattern recognition. 2018.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors could expand a bit more on the societal impact section in Section 5 (stating \"potential negative consequences for human monitoring\" seems very vague). Are there more concrete societal impacts of behavior modeling that you could discuss in a few sentences?", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "txuRdZ0GjMM", "reviewer": "Reviewer_U58y", "summary": "This paper proposes a model to learn embeddings for videos of animal behaviors. The model is based on an autoencoder trained to disentangle time-invariant context and motion (pose) in a video by reconstructing each frame given the encoding of a single per-video context frame, and a per-frame low-dimensional encoding, forcing it to represent dynamic pose and ignore static appearence cues. The model also proposes a recurrent VAE to decode pose in videos, allowing to decode multi-modal behaviors that are consistent with some given context. The proposed model is tested on two non-released mouse behavior datasets, showing videos of trained mice performing behaviors across different training sessions. The learned latent pose embeddings are invariant to different session given the same behavior, can be used to predict pose keypoints, cluster different behavioral motifs and generate multi-modal consistent mouse behaviors.       ", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\n## Strengths\nThis work focuses on learning behavioral representations in videos, which is a relevant and open problem. The proposed model does not assume access to annotations on the video data, and could be general enough to be applied to multiple kinds of behaviors, even though it is only studied on datasets of trained mice behaviors. The model is end-to-end trainable and modular, such that different components can be combined with prior work. The work is generally transparent about the limitations and assumptions of the model, stating in lines 320-324 the assumption that the content is static and therefore that a single context frame is enough to encode non-pose views. The work cites relevant work on disentangled video representation learning, dynamic modeling and animal behavior analysis. Given that the paper talks about keypoint-based methods for behavioral analysis, and the limitation on these models requiring manual annotation, I am missing a reference to unsupervised keypoint discovery works [1,2], which are relevant here. The proposed evaluation tasks are well designed, and show different aspects of a desireable behavior representation:\n\n1. Disentangled video representations, consistent for similar behaviors even when the context changes\n\n2. The learned representations can separate different kinds of behaviors, as shown when using unsupervised clustering to segment differnt kinds of behavioral motifs\n3. Keypoint prediction: while learned unsupervisedly, the representations can linearly decode annotated keypoints, showing that they contain relevant dynamic information\n4. Behavior generation: the representations can generate different behaviors, consistent with a given video context.\n\nFinally, the paper is generally well written, the motivation and experimental setup is generally clear, even though I would appreciate better clarification in the explanation of the method and discussion of results, as I mention below.\n\n## Weaknesses:\nEven though I appreciate this work's take on learning unsupervised representations that disentangle content and motion, I have multiple concerns about the paper's model and the results, which make me lean towards rejection at the current state.\n\n\n**Method:**\nMy first concern is in the proposed method, both in the design and in how it is described. One key in the two datasets is the fact that each video is recorded from multiple views, which is exploited by the model by concatenating the views both in the contextual and the positional encoding. \n\nFirstly, there is no explanation on why the two views provide better contextual or pose information, what is the effect of removing these views, or why concatenating views instead of averaging and decoding. This design is also a limitation since it makes the model only applicable in videos that have multi-view recordings. It might be that having multiple views is essential in learning representations that capture pose information, but because this is not ablated it is difficult to measure it's importance. In that sense, the paper is missing a relevant reference [3] that heavily relies in access to multiviews to build meaningful representations in video. Besides no further mentions to the multi-view aspect, this choice is not reflected in Figure 1, which makes it easier for the reader to ignore a potentially important design. The representation of context is not too inspired, taking a single frame to encode it. While the paper states the assumption that the videos will have non-dynamic background, it requires manually selecting a content frame that is representative enough, which makes it hard to scale. Some design aspects of the variational pose predictor are also not totally clear, like why taking an approach different than (26) and predicting a single mixture index c. From the paper, it seems like this index c could be used for clustering behavioral motifsin section 4.3, but this is definitely not clear from the text. Overall, I don't find the model particularly original or inspiring: the dimensionality bottleneck which seems to be one of the key components is already exploited in previous work (31), and the model misses an opportunity to use methods in existing works (5) to ensure desirable properties in the context and pose vectors, for instance making sure that the context representation is adversarially indistinguishable across frames in a given view, or the pose for different views in the same frame.\n\n\n**Experiments/Results:**\nEven though the proposed experiments are good measures of good behavioral representations, as well as potential advantages of the proposed method, they actually cast more doubts about how valuable is the proposed approach. Starting by the disentangled experiments, while it is promising to see that the pose embeddings make it hard to predict session to which they correspond, it should also be reported the classification performance of the context embeddings, which should be recovering this information in order to claim that the embeddings are really disentangled. By looking at some of the figures, it seems like the videos are generally similar looking context-wise, and it could be that the context embedding is really discarding this information, where the generator is powerful enough to generate a context that is really similar across all the dataset. In that sense it would be interesting to explore a dataset where there are larger visual changes from session to session, I am not totally convinced that each session represents such a large context shift. The trajectory decoding experiments are similarly valuable, but it is hard to understand how powerful are the dynamic embeddings without seeing the performance of other models, or any baseline to compare with. The motif segmentation casts further doubts into the method, since the best performing baselines are the ones not using the variational encoder component (DisAE). Finally, it is hard to understand the advantages of the variational decoder in the controllable behavior generation: without further explanation I cannot see how diverse are the behaviors and how the model would compare if the pose decoder was deterministic. For the different experiments it would be great to have ablations of the different components of the model to understand the contribution (multiview, variational encoder, context vector)\n\n**Clarity:**\nThere are multiple points in the paper where it is difficult to follow the experimental setup, details of the model or even motivation. To start, the introduction starts by talking about relationship with neural activity of animal behavior, but this is not exploited in the paper. As previously mentioned, it is really difficult to understand what is the contribution of the discrete and continuous representation in the variational encoder. From the description of the model it seems like the continuous representation should be directly based on the discrete class c_{t}, but this is not clear from figure 1 or the equations (1,2,3). Figure 1 is in fact never refered in the paper, and the lack of matching notation of the text or the omission of important aspects (such as how the multiviews are combined) makes the model difficult to understand. There also seem to be issues in equstions 1,2,3, with multiple assignments for c_{t}. It is not clear at all if the c_{t} here is used for the motif segmentation, or that experiment is based on some unsupervised clustering algorithm. The experimental section could also be largely improved: besides the mentioned issues in the motif segmentation experiment, authors should clarify on which dataset is section 4.2 evaluated. Furthermore, it is hard to understand the nature of the dataset studied, given figure 3, which also contains an insufficient caption. How is it relevant to define the origin (0) at 4 seconds after the start of the trial if the video that is considered goes from -0.25 to 1.75. It is misleading to talk about 12 second videos if only 2 seconds are used of those. Finally it seems like the context frames can be sampled 1 second before the origin, which means that the context could potentially be not part of the video: is it the case that, depending on the sampling, the context will be part of the frames encoded in the latent behavior or not?\n\n**Reproducibility:**\nEven though the paper contains details about how the models are trained, the fact that it is only tested on non-released datasets makes it impossible to reproduce.\n\n\n\n[1] Kulkarni, Tejas D., et al. \"Unsupervised Learning of Object Keypoints for Perception and Control.\" Advances in Neural Information Processing Systems 32 (2019): 10724-10734.\n\n[2] Minderer, Matthias, et al. \"Unsupervised Learning of Object Structure and Dynamics from Videos.\"\n\n[3] Sermanet, Pierre, et al. \"Time-contrastive networks: Self-supervised learning from video.\" 2018 IEEE international conference on robotics and automation (ICRA). IEEE, 2018.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Societal impact is very slightly addressed, but I agree with the authors on the gap between the experiments conducted here and potential experiments that could be conducted with humans. I would maybe make some reference so the importance of privacy and building embeddings that do not preserve identity, but it is more a suggestion than a criticism. The limitations of the work are addressed.", "review_text": "", "extra_scores": {"ethical_concerns": "No foreseeable ethical concerns in the current state. See my societal ipact section.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "hSlFj5o_oVi", "reviewer": "Reviewer_poVh", "summary": "In “Learning Disentangled Behavior Embeddings”, the authors propose a new method for automatically quantifying the structure of head-fixed animal behavior found in video footage. Specifically, they propose using a disentangled autoencoder, which explicitly attempts to factorize time-varying structure from time-invariant structure (i.e. context). ", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The disentangled autoencoder proposed by the authors addresses a key problem in the field: dealing with variation in video footage across experiments. The model is also trainable end-to-end, and it appears to perform better than the published state-of-the-art.  However, I have some concerns first about the novelty of the work, given that the autoencoder follows the work of Yingzhen and Mandt. Second, it is difficult to know how the autoencoder achieves its apparent performance gains without additional analysis.\n\nMajor points:\n\n1)\tThe disentanglement mechanism proposed by the authors is very similar to that employed by Yingzhen and Mandt.  The authors should provide a more explicit comparison between the work presented here and the sequential autoencoder previously proposed by Yingzhen and Mandt.\na.\tI mention this because the largest performance gains appear to come from using the DisAE, rather than the stochastic dynamic modeling component.\n2)\tThe distributional shifts from experiment to experiment should be analyzed more explicitly.  In Figure 4, the BehaveNet distributions appear to be mean-shifted, yet their dynamics look quite similar to one another. This raises the possibility that the differences from experiment to experiment are trivial, e.g. differences in lighting or the coat color of the animal.\na.\tFirst, the authors need to provide examples of what time-invariant “context” is being accounted for by the model. \nb.\tSecondly, the Appendix mentions that the videos were normalized prior to feeding them into BehaveNet.  What normalization was applied here? The details are important here to understand exactly what is introducing the distribution shift shown in Figure 4.\n3)\tIs any information lost in the disentangled autoencoder on a session-by-session basis? In other words, the authors should compare models tried on a single experiment only, to see if the disentangled autoencoder performs as well as BehaveNet and/or DLC + VAME. If the disentangled autoencoder performs just as well using single-experiment datasets and generalizes well to other experiments, it would be extremely useful to know.   \n4)\tHow did the authors tune the hyperparameters of the competing methods? Re: the ARHMM, it is not clear what “latent dimension” refers to here. The authors should try a variety of lags, and stickiness to prevent the model from over-switching. Same question re: tuning the hyperparameters of VAME. In general, these competing methods need to be optimized for performance to provide a meaningful comparison.\n\nMinor points:\n\n1)\tTypos throughout.\n2)\tIn the examples shown in Figure 12, why k=10 for BehaveNet and DisAE + ARHMM?  The authors should show the best performing variant of each model for comparison here. \n3)\tCan the authors provide more information about the generated videos? What do they correspond to exactly?\n4)\tIt would be interesting to see the results of using the DisAE on the DLC coordinates.  I wonder if most of the useful information in the video is present in the DLC data as well.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes, no comment here.", "review_text": "", "extra_scores": {"ethical_concerns": "I found no ethical concerns.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers and the AC appreciate the amount and quality of the rebuttal. From clarifying several points raised by the reviewers, to incorporating baselines (Yingzhen & Mandt) and ablations (multiview). All reviewers and the AC appreciate as well as their constructive spirit during the discussion period.

Having said that, Reviewers poVh and c1eX consider that the work is not yet ready for publication while Reviewer poVh considers it borderline.

The main points of criticism are that it is still unclear why the full DBE is necessarily more interpretable than applying a clustering method post-hoc. This is a central aspect of the work and should be clarified (e.g. it is one of the main arguments to justify the use of stochastic dynamic modelling). Additionally, the performance benefit of the DBE appears limited. While still supporting the paper, Reviewer poVh agreed with this criticism in the discussion not visible to the authors.
There is consensus among the reviewers that the work is promising and will certainly lead to a valuable contribution. The reviewers encourage the authors to reformulate the work as a study of different representations of animal embeddings in videos, incorporating the ablations described in the response and adding comparisons to at least another baseline. In sum, the changes needed to bring the paper up to the NeurIPS standards are too significant at this stage to recommend acceptance.

All reviewers and the AC hope to see a revised version of the approach to be published soon in a future venue.

Decision: Accept (Spotlight)