Learning Disentangled Behavior Embeddings
Authors: Changhao Shi
Sivan Schwartz
Shahar Levy
Shay Achvat
Maisan Abboud
Amir Ghanayim
Jackie Schiller
Gal Mishne
Venue: NeurIPS 2021 Spotlight (2021)
Keywords: Neuroscience
Animal Behavior
Behavioral Videos
Generative Models

Abstract: To understand the relationship between behavior and neural activity, experiments in neuroscience often include an animal performing a repeated behavior such as a motor task. Recent progress in computer vision and deep learning has shown great potential in the automated analysis of behavior by leveraging large and high-quality video datasets. In this paper, we design Disentangled Behavior Embedding (DBE) to learn robust behavioral embeddings from unlabeled, multi-view, high-resolution behavioral videos across different animals and multiple sessions. We further combine DBE with a stochastic temporal model to propose Variational Disentangled Behavior Embedding (VDBE), an end-to-end approach that learns meaningful discrete behavior representations and generates interpretable behavioral videos. Our models learn consistent behavior representations by explicitly disentangling the dynamic behavioral factors (pose) from time-invariant, non-behavioral nuisance factors (context) in a deep autoencoder, and exploit the temporal structures of pose dynamics. Compared to competing approaches, DBE and VDBE enjoy superior performance on downstream tasks such as fine-grained behavioral motif generation and behavior decoding.

Learning Disentangled Behavior Embeddings
Changhao Shi"
University of California, San Diego cshi bucsd.eddu
Sivan Schwartz, Shahar Levy, Shay Achvat, Maisan Abboud, Amir Ghanayim
Technion Israel Institute Technology
(sivan. (sivan.s,shahar55, shay. aww.acchaatma.. maisanaabbodd, amir gh @ @campus........... wwwhniecc.nn.eee ac
Jackie Schiller
Gal Mishne
Technion Israel Institute of Technology jackie@technion.c ac.il
University of California, San Diego gmi shne@ucsd. edu
Abstract
To understand the relationship between behavior and neural activity, experiments in neuroscience often include an animal performing a repeated behavior such as a motor task. Recent progress in computer vision and deep learning has shown great potential in the automated analysis of behavior by leveraging large and high-quality video datasets. In this paper, we design Disentangled Behavior Embedding (DBE) learn robust behavioral embeddings from unlabeled, multi-view, high-resolution behavioral videos across different animals and multiple sessions. We further combine DBE with stochastic temporal model to propose Variational Disentangled Behavior Embedding (VDBE), an end-to-end approach that learns meaningful discrete behavior representations and generates interpretable behavioral videos. Our models learn consistent behavior representations by explicitly disentangling the dynamic behavioral factors (pose) from lime-invariant, non-behavioral nuisance factors (context) in deep autoencoder, and exploit the temporal structures of pose dynamics. Compared to competing approaches, DBE and VDBE enjoy superior performance downstream tasks such as fine- fine-grained behavioral motif generation and behavior decoding.
Introduction
Understanding the relationship between animal behavior and neural activity is a long-standing goal in neuroscience. To this end, recent advances in deep learning and computer vision has led to significant progress in the essential task of automatic analysis of high-resolution behavioral videos.
To extract behavioral information from rich video recordings, two avenues of research relying on deep learning have been proposed: landmark-based pose estimation methods [22 26 32 and autoencoder-baase dimensionality reduction methods (2). Pose estimation methods characterize animal behavior with the trajectories of body-part landmarks. Such methods train deep neural networks to predict manually labeled landmarks from raw video frames, then use the trained models to generate trajectories for new videos. While these methods have been highly successful in behavior analysis, they 1) require manual labeling which is often expensive; 2) heavily rely on the manual selection of the landmarks, which may differ across human annotators; and 3) struggle with subtle behavior that is hard to track, such as facial movements.
"qqual contributions
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

The complementary approach seeks to reduce the high-dimensional video data to low-dimensional latent factors through non-linear dimensionality reduction. These methods [2] rely on deep autoen- coders to encode video frames by solving an unsupervised reconstruction task. The learned latent embeddings can then be used for downstream tasks, such as clustering, neural decoding, etc. While unsupervised learning does not rely on manual labeling, existing approaches currently suffer from certain drawbacks. First of all, canonical autoencoders can be easily biased by the varying visual nui- sances across videos, such as lighting, distance to camera and physical attributes of animals, resulting in substantial distributional shifts among the learned behavior embeddings. While non-behavioral visual variability is usually negligible within single session videos. it makes downstream analysis difficult when applied to more than one session and to different animals. Secondly, end-to- and training videos becomes challenging when considering temporal structures, traditional temporal modeling methods are mostly designed for low-dimensional data. How to better exploit temporal structures of nigh-dimensional videos for action recognition, motion planning. etc., remains an open problem for computer vision research.
In this paper, we design Disentangled Behavior Embedding (DBE) to learn robust behavior embed- dings from large, unlabeled, multi-session videos. Inspired by previous works (33), DBE mitigates the distributional gaps in multi-session videos by explicitly disentangling the dynamic behavioral factors (pose) from time-invariant, non- nen-behvvooral factors (context). A video frame is thus repre- sented by a pair of disentangled pose and context components in contrast to single entangled representation. For a given video, the context component is designed to be time- invariiar to exclude any behavioral dynamics, whereas the pose component is bottlenecked to keep it from duplicating context information. DBE latents can be used in conjunction with temporal analysis models, e.g., Variational Animal Motion Embedding (VAME) 21. to perform downstream analysis. We also propose Variational Disentangled Behavior Embedding (VDBE), a fully end-to-end trainable model, which further exploits the temporal structures of the pose components using a stochastic dynamic model. Using variational inference, a Gaussian mixture prior is trained to capture the multi-modality of the transition between consecutive pose components and generate both continuous and discrete representations of the underlying animal behaviors in videos. Our methods are not only fully unsuper- vised, but also enjoy superior performances on downstream tasks such as behavioral state estimation and -rniggraindd behavioral motif generation. To summarize, the main contributions of this paper are:
1. We develop DBE to tackle the distributional shift across multi-session videos with adaptation to the behavioral neuroscience setting, c.g., extension to multi-view videos, proper design of context embedding with standard behavioral neuroscience paradigms.
2. On top of DBE, we design VDBE to learn both continuous and discrete latent representations for simultancous embedding and segmentation within the same model. VDBE is end-to-end trainable in an unsupervised fashion, alleviating the need to train a second post-hoc model applied to latent embeddings.
2 Related Works
2.1 Video disentanglement
Disentanglement of the pose and context in a video is area of increasing interest in computer vision. The explicit separation of the pose from the context not only facilitates the analysis of the temporal dynamics but also makes video manipulation easier. As a pioneer in this field, MCNet 30) is an end-to-end framework that models motion (pose) and content (context) independently. MCNet uses a convolutional LSTM to model temporal dynamics and shows good performance on pixel-level video prediction on real-world datasets. Sharing similar ideas, DRNet [5] also separates the pose from the content but models temporal dynamics directly on low-dimensional pose embeddings. also utilizes an explicit adversarial loss for disentanglement. Although DRNet shows superior performance in video prediction, it is not end-to-end trainable. DDPAE [33 tackles videos of multiple targets on the moving MNIST dataset and performs disentanglement on each individual target. Yingzhen and Mandt 33 proposed a disentangled VAE and shows controllable video generation on the synthetic Sprite dataset. MoCoGAN (29 employs generative adversarial network that decomposes the random noises into a content part and motion part for video generation. S3VAE R4 learns disentangled context and pose representations of videos with self-supervised regularization. In our approach

---

build a disentanglement mechanism in a similar paradigm as [33], by fixing context embeddings across time and constraining the dimension of pose embeddings.
2.2 Stochastic dynamic modeling
Latent state- space models (SSM) are widely used to model the temporal structures for sequential data [
17 20, 71. While these models have achieved fruitful results in many areas such as natural language processing and speech recognition, adapting these models high-dimensional, non- synthetic videos is still challenging to this date. Hafner et al. [11] proposed PlaNet, with a recurrent state-space model at its core, to learn environment dynamics from high-dimensional, synthetic videos and auxiliary action variables for planning. Franceschi et al. [9] proposed latent dynamical model enhanced by residual connections for stochastic video prediction. Although there are other papers on stochastic video generation and prediction on low-dimensional frame embeddings 6 [8), these methods tackle the problems in an auto- auto-regressive fashion and thus offer no single latent state vector that captures all the dynamics, as opposite to state-space models. Our approach closely relates to
[T] [9] but differs by using a multi-modal distribution for the transition model and the ability to generate both continuous and discrete representations.
2.3 Behavioral video analysis
Recently, supervised deep pose estimation methods have been widely used for behavior analysis for many animal models. Methods such as DeepLabCut (DLC) [22), DeepPoseKit [TO] and LEAP (26) train deep neural networks to produce confidence maps for predefined landmark annotations, all in a similar fashion. Leveraging the success of DeepLabCut, Wu et al. 32 improved the quality of pose estimation by imposing additional spatial and temporal constraints on multiple landmarks during training. Luxem et al. [21] further developed recurrent variational autoencoder (VAME) encode the pose estimation from DLC to a higher-level behavioral space. Prior to deep learning approaches, Kabra al. (15] provided supervised pipeline automatically annotate animal behavior based on user-defined labeling.
On the other hand, unsupervised dimensionality reduction methods like PCA have been used to analyze behavior for decades [28 4 3I. More recently, BehaveNet (2 presented meaningful results of behavior segmentation and neural decoding. BehaveNet first uses deep convolutional autoencoder to compress behavioral videos, then fits an autoregressive hidden Markov (ARHMM) model to exploits the temporal structures of the learned frame embeddings. However, BehaveNet was evaluated only on videos in single experimental session and cannot handle the distributional shift across multiple sessions. Our approach, which falls into this second category, not only inherits the advantage of being unsupervised but also handles the distributional gaps across multiple sessions by explicitly disentangling poses from contexts.
3 Methods
3.1 Problem formulation
Formally, given a behavioral video F1. of length T, we are interested in disentangling the dynamic behavioral factors from the time-invariant non-behavioral nuisances in the video in an unsupervised fashion. Our goal is to learn meaningful emhedding =i.r that capture common behavioral structures across multiple animals in multiple recording sessions. We assume that the animal performing repetitive behavioral protocol, such as mouse performing a forepaw reach or lever press [19] 24], and not freely moving. Additionally, in behavior experiments, videos are often recorded from different angles, resulting in multi-view data, e.g. (Ia, F66 166)1: 1.r where a and b denote the two views. For milti-view videos, we treat each view as a separate image and fuse the outputs of encoders of each view as the representation of this timestamp, whereas for mono-view videos the embedding of the frame is the natural representation. For the rest of this paper, we will slightly abuse the notations and omit the view subscripts, e.g., Fi. refer to either mono-view frames or multi-view frames (ra, 2&)) 31)1.r I. and ž1. refer to either the embeddings of mono-view frames or the fused embeddings of multi-vier frames.
3

---

Video Clip
Enc
Dynamicc Model
Continzous Vartabies
RNN
SSM
Pose
Discrete Varlables
Pose
Context Frame
Enc
Reconstruction
Context
Concatenation
Figure 1: Overview of VDBE mono-view video. The pose encoder encodes individual frames into a sequence frame embeddings, which are then passed through stochastic dynamic model to generate pose embeddings. The context embeddings are generated from single conditional context frame by the context encoder. Both embeddings are concatenated at each timestamp as inputs to decoder which reconstructs the original video clip. The dynamic model (top right) uses an RNN to infer both latent continuous embeddings and discrete states that capture the transition of the frame embeddings, and uses a linear sate- model for generation. When the dynamic model is not used, the full VDBE model becomes DBE. The input video can be either single view or multi-view For multi-vien videos, encoders and decoders used for each view separately, and the pose and context components are concatenations of the corresponding encoder outputs from each view.
3.2 Disentanglement
In order factor out the non-behavioral factors, approach utilizes separate lines of encoders, context encoder E and pose encoder E", for each view of the videos. Assuming the non-behavioral factors time-invariant within each video, the convolutional context encoder E infers video-level context embedding 2" from a single frame the video. For multi-view videos, the context of each view is further concatenated into single context embedding 2", then used for the reconstruction of all the frames in this video. The pose encoder E. shares the same architecture with E., and encodes the behavioral dynamics of the animal from all video frames T1:T into the pose embeddings = For multi-view videos, the outputs of pose encoders are concatenated to get =f.7. The temporal structures of the pose component are exploited by a stochastic dynamic model stacked top of the pose encoders, which we will detail in the following. To reconstruct the input video T1.7, the context embedding z" is duplicated and concatenated with each pose embedding 2f-r.. then for each view these concatenated embeddings are further decoded into frame reconstructions by the corresponding convolutional decoders D.
Because the context embedding =f is time-invariant and therefore incapable of capturing dynamics, ensure that it contains behavioral information about the animal. Thus, the key question about the disentanglement is how to ensure that the pose embeddings contain no context information. To enforce disentanglement, we create bottleneck on the pose embeddings inspired by (33). We choose the dimension of the pose embeddings to be as small as possible in the premise of good reconstruction. This ensures that the capacity of pose embeddings only sufficient for capturing the temporal dynamics without any redundant context information. As we will show in the Sec 4 the disentangled pose embeddings of different sessions are well-alia well-aligned with each other, in sharp contrast against the results from canonical autoencoders where the embeddings of different sessions form distinct clusters. The disentangled context embedding, on the other hand, capture these non-behavioral nuisances and form into clusters by sessions as expected. The full disentanglement mechanism is shown in Figure []
3.3 Dynamic model
model the temporal dynamics of the pose embeddings ~Κ. using a sequential variational au- toencoder. For the sake of brevity, we will use to refer to z² in this section. Similar abbreviations are applied for other variables as well. We assume that the pose components govern the latent state of the behaviors and we want to model the temporal transition of the states. In typical stochastic

---

91
92
93
94
91
92
9s
94
2
22
Z3
22
#3
F
J/2
T2
J3
(a) Generation
(b) Inference
Figure 2: The generation (a) and inference (b) process of stochastic dynamic model. Round nodes represent stochastic variables and squared nodes represent deterministic variables. Note that in this illustration, 21 is inferred from two conditional frames ø and z2; and for c₁ and St only the paths from their immediate past frames are shown for the sake of clarity.
state-space model, the transition from the previous state ž to the current state is modeled stochastically. However, a purely stochastic transition makes hard for the model to keep long-term information [TT]. Thus,
factor out the stochasticity of the dynamics into separate component This means that at a given timestamp, the stochastic latent variable 8 is first sampled from prior distribution, usually Gaussian distribution, then the current state 2 is generated from the previous state 2 and the stochasticity s deterministically.
While this strategy has been proved to be effective in many scenarios 6 11 9.. the uni-modal Gaussian transition distribution limited for behavior analysis because it does not capture the multi-modality of behavior. For example, in the video dataset we will show in Section the transition of the animal poses is usually caused by set of distinct, prototypical actions, such as lift, reach and grab. A uni-modal Gaussian transition distribution is inadequate for modeling such multi-modal transitions. Furthermore, for the purpose of motif segmentation it offers no discrete states which are highly desired.
more suitable choice for capturing such multi-modality a Gaussian mixture model 27 Shu et al. 27
design the transition distribution in the generative model to be a Gaussian mixture while keeping posterior inference approximation to uni-modal Gaussian. A closed-form approximation of the exact KL. divergence is then used in evidence lower bound for optimization. In this paper, we use an alternative approach to introduce the Gaussian mixture. Instead using a single stochastic latent variable with a Gaussian mixture prior, we decompose it into two stochastic variables, the mixture index e and Gaussian noise g. This Gaussian mixture prior also enables the model to perform embedding (2)) and segmentation (ct) simultaneously. With all these, we can dive into the details of our variational autoencoder, as illustrated in the zoomed-in region in Figure]
Generation The generative procedure shown in Figure 2a We model the transition between the previous state 2t-1 and the current state 2t using Gaussian mixture. The generative model is:
21 ~ NN,, I), 241 So( St), G ~ cat(TT((()) 9: ~ (O0, I), 8, Pe. to, @ Og It (Deco zc));
(2)
where wa and fo are linear projection layers. At each timestamp, stochastic variables o, and gi model the discrete state and the additional stochasticity, then the continuous state 2 is generated with deterministic path from 2;-1, C; and gt- Note that although both C and 9i capture the stochasticity of the transition, only c is dependent on the previous state 2/-1- is sampled from categorical distribution parameterized by 2 and gt is simply sampled from a standard Gaussian distribution.
The inferred e is used for for motif segmentation in the test stage. We assume the covariance matrix of each Gaussian component is diagonal and @ denotes glement-wise multiplication (Hadamard product).
5

---

Inference Because of the non-linearity in the generative model, the true posterior distributions of the latent variables are intractable. As shown in Figure 2b we use variational inference to approximate these posteriors. The inferred posterior is factorized as qo(CI:T, 91.T, =1) = ((=1g(e/221:) ((
where q(c((21:)) is conditional categorical distribution, and ((9 (21:7) as well as [1(e)) is conditional Gaussian distribution. We use shared GRU network followed by separate projection heads for g(c)(æ1:)) and q(9 (T1:)) For (22 [z1,),, we use an MLP whose inputs are the first C' conditional frames in the sequence. At test-time, we also use these inference networks to generate behavior representation, i.e. we use the mean of the Gaussian distribution and the highest category of the categorical distribution as the continuous and discrete representations of the behavior of each frame.
Optimization
We use the following evidence lower bound (ELBO) for optimization:
(o. Basler
(4)
KDxI (g ( (9o(0))) (00 yD (
where a, 33 and yy are the trade-off parameters that control the information flow to each stochastic latent variables C1:T, 91.T and respectively [12). The first log-likelihood term corresponds to the frame reconstruction and the other KL-divergence terms are the regularization of the inferred stochastic latent variables to their prior knowledge.
The above expectation estimated with single sample from the approximated posterior distribution. To allow gradients to back propagaee through the non- non-differentiable sampling operation, we use the re- p--aammeterrzation tricks. For the Gaussian posterior distributions g(gh (a1:T) and 7(2) |=1:c), we re-parameterize the distribution into their estimated mean and standard deviation with standard Gaussian noise [16]. For the categorical posterior distribution (((())1:)) we use the Gumbel-Softmax re- parameterization trick [14] the sampling process is re- p-rrrmmtterized into the estimated discrete gee-robability and a Gumbel noise, following by a softmax normalization. The outputs of softmax are also discretized to one-hot encodings to ensure consistency between training and testing. Note that theoretically one can compute the exact expectation over the discrete latent variable o without using Monte Carlo estimation, but experiments show that one- time sampling can generate satisfying results with the advantage of being more efficient.
4 Results
We evaluate our methods on two multi-session video datasets: the Hand-reach dataset [99 we collected, and the Wide-Field Calcium Imaging (WFCI) dataset 24 25). The Hand-reach dataset contains vidcos of head-fixed mice who were trained to reach from their perch to food pellet. The datasets consists of 263 suc- cessful trials of expert animals (who have learned the
Tane
motor task) in 8 sessions, and includes fine-grained be- havior. In successful execution of the task, the animal will lift its paw from the perch, reach and grab the pellet and bring it to its mouth and eat it. The animal might perform multiple grab attempts in single trial before successfully grabbing the pellet. An illustration of the Hand-reach dataset is shown in Figure 3 The videos were acquired on multiple dates, within which the ex- perimental layout is potentially different, e.g., location of the table. Two cameras simultaneously record from the front and right side the mouse. Each video is 12 seconds long, with a frame rate of 200Hz. The video begins with 4-seconds inter-trial interval (ITI), the end of which we set as the origin (0) of our time axis. A cue of 0.2 seconds is shortly followed by table-turning which brings the food in front of
0.2 sec
Rotation
0.2
Time (sec)
-0.25 b
1.75
Figure 3: Hand-reach dataset. Mice were head-fixed and trained to grab food pel- let. Trial structure: 4 seconds ITT period followed by the tone and table turning. We analyze the [-0.25, 1.75] segment which contains most of the significant behaviors for expert mice.

---

the mouse. Since the mice in this dataset are experts, and thus mostly stationary before the cue and chewing the pellet after successful execution of the hand-reach, we use a 2-second clip of the main dynamic behavior of each full video to train the model. Each clip starts at 0.25 second before the cue and ends at 1.75 seconds after, which contains most of the motions of interest. The context frame for each video is selected from the ITT period, sampled at 1 second before the cue, when most of the subjects are stationary.
In the WFCI dataset (an extended version of the dataset in (2]., head-fixed mice perform a visual decision making task, using levers to initiate trials and reporting choices by licking from spouts. The behavioral videos are recorded from a side and a bottom view of the subjects, with length of 189 frames and frame rate of 30Hz. The dataset contains 3708 videos from 8 sessions of 4 animals in total. We randomly select 80% of the videos as training set and use the remaining 20% as test set.
We train both DBE and VDBE, without and with the stochastic dynamic modeling component. We compare our methods against two behavior embedding methods: BehaveNet (2) and VAME (21]. with implementation details of each provided Appendix [ and Appendix [ respectively. In Section [4.] and 4.2, we demonstrate the efficacy of our disentanglement component. In Section (.3 we show both DBE (combined with motif segmentation model) and VDBE achieve superior results for motif segmentation compared to the original models.
4.1 Disentanglement
To demonstrate the efficacy of disentanglement, we focus on session recognition: given the behavior embedding of specific video frame, can we identify which session (animal and date) this frame comes from? We train a simple linear classifier to classify the experimental session from different pose embeddings, and ideal disentanglement should generate session-agnostic pose embeddings. The accuracy of the session classification for each model is reported in Table [ VDBE model generates session- -agnossi pose embeddings, compared the plain CAE autoencoder used in Behavenet. Although DLC also provides session-agnosta coordinates, selected markers do not encode enough useful information for extracting robust behavioral motif's across sessions (Section 4.3). The context embedding of our methods (DBE-context and VDBE-context) accurately classify the session, further demonstrating disentanglement of context from pose.
DBE as well as our full
We further visualize the advantage disentanglement in Figure We plot the 2D PCA applied to the BehaveNet and DBE embeddings of two videos that show similar behaviors but are recorded in different sessions. The plain CAE of BehaveNet generates separate pose embeddings even though the behaviors are similar. This distributional shift will be problematic when wants transfer the knowledge learned in one session to another. This means that one needs to learn a new model every time when there are new sessions, and learn a mapping across animals relate the embeddings of the same behavior. In contrast to BehaveNet, our DBE nicely aligns the two videos in the pose embedding space. As will show in the later section, this consistency is crucial to motif segmentation and behavior decoding. More results comparing BehaveNet and DBE embeddings, as well as ethograms of both the Hand-reach and the WFCI dataset, are provided in Appendix and G
BehaveNet
DBE
Model
Hand-reach
WFCI
BehaveNet
100.00
99.99
Mouse]
DLC
21.19
DBE VDBE
28.21
35.49
17.94
19.54
DBE-context
88.59
100.00
VDBE-context
96.19
100.00
Figure 4: Embeddings of 2 similar videos (Hand-reach)
Table I: Session classification
4.2
Trajectory decoding
DLC is robust method used to track manually selected landmarks in behavioral videos. A good behavioral embedding should contain the information encoded by DLCC To illustrate that our approach learns meaningful behavior embeddings, we perform simple linear regression task from the DBE
7

---

embeddings to the DLC coordinates in the Hand-reach dataset. A good fit of the linear regression model suggests that our learned embedding contains the DLC coordinate information.
We place 5 markers on the center and 4 digits of the right paw of the mouse in each view of the videos, and train DLC model to track the coordinates of those 10 markers. Due the differences in spatial layout of the sessions (e.g., table location), we fit separate linear regression models to each marker for each individual experimental session. Only DLC coordinates that have confidence score larger than 0.95 are used for the training of linear regression.
0.85
0.00
0:75
0:70
0.65
0.60
1-Hevt5
1-Havis
1-tevzo
M
rezz
rebza
Ha
(a) DLC trajectories prediction
(b) Goodness-offfin of linear regression
Figure 5: Predicting DLC trajectories from disentangled embeddings on the Hand-reach dataset. (a) Example video overlaid with true DLC trajectories (red) and the linear regression model predictions (green). (b) Ri averaged across each coordinate for each individual session.
An example of DLC predictions and the average R of each individual linear regression model are shown in Figure 5 capturing the right paw of the mouse moving from the perch to the pellet on the table. The high R² scores in Figure 5b|show show that the linear regression models are well-fitted to the data, which indicates that the DBE embeddings manifest meaningful information about the selected markers.
Figure [a] the predictions are well aligned with the true DLC trajectories,
4.3 Motif segmentation
Learning fine-grained behavior motifs from videos is an essential but challenging task for analyzing animal behavior, especially when supervision is available. In this section, compare our approach with other competing methods for motif segmentation on the Hand-reach dataset. To evaluate the quality of the generated motifs, we select 10% of the videos (25 videos) and perform expert manual labeling of the selected videos with a set of seven behaviors of interest: On-Perch, Lift, Reach, Grab, To-mouth, At-mouth, and Back-To-Perie (4730 labeled frames). The rest of the videos are used as the training set for motif segmentation methods, i.e. clustering. For each method, we assign the learned motifs to the labeled set of behavior, where the mapping is selected maximize the clustering accuracy of holdout videos. Thus, each motif corresponds to a "mega-cluster" that may or may not contain more than one inferred discrete state. Note that we set the number of the discrete states & to be larger than the number of target labels for two reasons. The first reason is that only part of the frames are labeled and the target labels do not cover all the possible behavior of the mice. Extra clusters are necessary to capture these additional behavior. The second reason is that setting larger number of motifs rather than the exact number of labels accounts for inter-class versus intra-class variability. For example, this enables separating highly variable behavioral label like Reach into more than one motif without having to compensate for that by merging At-mouth and To-mouth into a single motif. We evaluate the performance of motif segmentation on three clustering metrics, accuracy, Normalized Mutual Information (NMI), and Adjusted Rand Index (ARI).
We use the DBE embeddings in two variations for motif segmentation: DBE+ARHMM which replaces the CAE embeddings of the BehaveNet model with DBE embeddings, and VBE+V which replaces the DLC trajectories with DBE embeddings as inputs to VAME. As shown in Table and Figure 6] disentanglement is the key to achieve good behavioral clusters across different animals and multiple sessions. Combining DBE for frame embedding and 30-cluster VAME (DBE+VAME) for temporal dynamics and segmentation achieves the best accuracy, NMI and ARI. Our full VDBE model has very similar performance, with both models exhibiting ~ 30% higher accuracy than BehaveNet and over 20% higher accuracy than the original VAME framework (relying on DLC inputs). These results also show that using a more powerful temporal model such as VAME can
8

---

Table 2:
motif segmentation for the Hand-reach dataset
Model
Unsup.
Dis.
End2End
Accuracy
NMI
ARI
BehaveNet (k=15)
X
50.52
30.89
20.98
BehaveNet (k=30)
x
X
53.09
33.44
25,09
DBE + ARHMM (k=15)
/
X
64.04
42.06
36.08
DBE + ARHMM (k=30)
V
X
71.49
48.96
45.76
DLC + VAME (k=30)
X
X
57.17
42.07
44.09
DBE + VAME (k=15)
V
X
71.31
59.47
51.46
DBE + VAME (k=30)
X
81.25
67.31
67.33
VDBE (k=30)
V
79.53
64.85
66.94
hed
08
Crab
Mouth
ToMouth
ToMou
#tNouth
AlMouth
02
Back
Back
Readh
Grab
Tottotth
Aotounn
Back
Perch
Reaid
ubab
Tollouth
Back
Herch
Tithoolf
Back
Prcckiee
Pheciled
(a) BehaveNet (k=30)
(b) DBE+VAME (k=30)
VDBE (k=30)
Figure 6: Confusion matrices of BehaveNet, DBE+VAME VAME and VDBE for the Hand-reach dataset.
benefit motif segmentation. With disentanglement, the confusion matrices Fig. 6 are more diagonal which indicates better clustering. Figure S shows examples of random segments from VDBE motif's corresponding to 4 different labels. Note that VDBE successfully clusters the behaviors even when the context varying. Additional results can be found in supplementary materials.
4.4
Controllable behavior generation
One advantage of VDBE and ARHMM-bassd models compared to VAME is that we can decode synthetic videos from behavior embeddings, which demonstrates better interpretability and allows potential applications to neural decoding. In this section, we further investigate behavior decoding by manipulating the context components. In order to generate synthetic videos that replicate source behavior with new context, we use the inferred discrete behavior representations C1:7 from the source video and the context embedding from the target session, while g1.7 and 2 are sampled from their prior distributions. If the target session is the same as the source session, the synthetic video should be similar to the source video. The results are shown in Figure & Note that the behavior in the source video is replicated in both generated videos with the same and different contexts. This indicates that our model successfully learns the dynamics of the behavior.
Conclusions
In this paper, we tackle the problem of unsupervised behavior representation learning from multi- session videos. A disentanglement mechanism is designed to separate session-wiss wise non-behavioral factors from the useful behavioral information, and a stochastic dynamic model is jointly trained to model the temporal dynamics of behaviors. Compared with competing methods, our method shows superior performance multiple tasks such fine-grained motif segmentation, meanwhile also enjoys the advantages of being unsupervised, end-to-end trainable, and more interpretable.
Despite the above improvements, couple of open questions still remain and will be explored in future work. Firstly, the current model which relies on a predefined number of states is limited in generalization of the motif assignment to videos which include novel unseen behavior. Secondly, our disentanglement assumes the non-behavioral nuisances are stationary through video, which is not always true. Future work should benefit from allowing slow dynamics in context embeddings. Thirdly,

---

(a) Lift
(b) Reach
(c) Grab
(d) At-Mouth
Figure 7: Motifs learned by VDBE. The blue channel indicates the start of the motif and the red channel indicates the end. Note that in each column the behavior is consistent but the context is varying (e.g., location of the table and identity of the mouse).
Source- Source-Mroic Mrovel
Gencaaed Monsaand--onnee Monee
Generated Maniee
Figure 8: Source (top row) and generated videos of the same discrete behavior sequence, with the same (middle row) and different (bottom row) contexts. Each row is plotted at intervals of 20 frames.
with the distributional shift in videos confirmed, it would be interesting to study disentangling other visual behavior representations such as DLC for multi-session analysis. Finally, with the learned behavioral embeddings consistent across sessions, future works can leverage this representation relate behavior to neural activity
Regarding socictal impacts of our study well behavior analysis research in general, although there a significant gap between the head-fixed animal behavior studied here and naturalistic human behavior, we should always be careful about the negative consequences for human monitoring.
Code is available http:://tttaaaaommmmmiiiaaaaaaaa Betprettannnnneeeai-----oooeeoe
Acknowledgement
This work was supported by in part by the National Institutes Health grant No. RO1EB026936. thank Simon Musall and Matthew Whiteway for providing the WFCI dataset and their generous help.
10

---

References
[1] M. Babacizadch, C. Finn, D. Erhan, R. H. Campbell, and S. Levine, Stochastic variational video prediction. arXiv preprint arXiv:
2017.
[2] E. Batty, M. Whiteway, S. Saxena, D. Biderman, T. Abe, S. Musall, W. Gillis, J. Markowitz,
A. Churchland, P. Cunningham, et Behavenet: nonlinear embedding and bayesian neural decoding of behavioral videos. Advances in Neural Information Processing Systems, 32: 15706-15717, 2019.
[3] Bayer and C. Osendorfer. Learning stochastic recurrent networks. arXiv:1411.7666 2014.
arXiv preprint
[4] G. J. Berman, D. M. Choi, W. Bialek, and J. W. Shaevitz. Mapping the stereotyped behaviour of freely moving fruit flies. Journal of The Royal Society Interface, 1 11(99):0010677 2014.
[5] E. Denton and V. Birodkar. Unsupervised learning of disentangled representations from video. In Proceedings of the 31 International Conference on Neural Information Processing Systems, pages 4417-4426, 2017.
[6] E. Denton and R. Fergus. Stochastic video generation with a learned prior. In International Conference on Machine Learning, pages 1174-1183. PMLR, 2018.
[7] Z. Dong, B. Seybold, K. Murphy, and H. Bui. Collapsed amortized variational inference for switching nonlinear dynamical systems. In International Conference on Machine Learning, pages 2638 2647.267..22 PMLR, 2020.
[8] M. Fraccaro, S. Kamronn, U. Paquet, and O. Winther. A disentangled recognition and nonlinear dynamics model for unsupervised learning. arXiv preprint 17Xiv:1710.05741, 2017.
[9] J.-Y. Franceschi, E. Delasalles, M. Chen, Lamprier, and P. Gallinari. Stochastic latent residual video prediction. In International Conference on Machine Learning, pages 3233-3246. PMLR, 2020.
[10] M. Graving, D. Chae, H. Naik, L. Li, B. Koger, B. R. Costelloe, and I. D. Couzin. Deepposekit, a software toolkit for fast and robust animal pose estimation using deep learning. Elife, c47994, 2019.
[11] D. Hafner, T. Lillicrap, Fischer, R. Villegas, D. Ha, H. Lee, and J. Davidson. Learning latent dynamics for planning from pixels. International Conference on Machine Learning, pages PMLR, 2019.
[12] I. Higgins, L. Matthey, A. Pal, C. Burgess, X. Glorot, M. Botvinick, S. Mohamed, and A. Ler- chner. beta-vae: Learning basic visual concepts with a constrained variational framework. 2016.
[13] J. Hsich, B. Liu, D.-A. Huang, L. Fei-Fei, and J. C. Niebles. Learning to decompose and disentangle representations for video prediction. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, pages 515-524, 2018.
[14] E. Jang, Gu, and B. Poole. Categorical reparameterization with gumbel-softmax, arXiv preprint arXiv: 1.01 144, 2016.
[15] M. Kabra, A. A. Robie, M. Kivera-Alba, S. Branson, and K. Branson. Jaaba: interactive machine learning for automatic annotation of animal behavior. Nature methods, 10(1):64, 2013.
[16] D. P. Kingma and M. Welling. Auto-encoding variational bayes. arXiv preprint arXiv: 1312.6114, 2013.
[17] R. Krishnan, U. Shalit, and D. Sontag. Structured inference networks for nonlinear state space models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 31, 2017.
[18] A. X. Lee, R. Zhang, F. Ebert, P. Abbeel, C. Finn, and S. Levine. Stochastic adversarial video prediction. arXiv preprint arXiv:1804.00 0,553,,, 2018.
[19] S. Levy, M. Lavzin, H. Benisty, A. Ghanayim. U. Dubin, S. Achvat, Z. Brosh, F. Aeed, B. D. Mensh, Y. Schiller, et al. Cell-type--pecific outcome representation the primary motor cortex. Neuron, 107(5):55--97,, 2020.
[20] S. Linderman, M. Johnson, A. Miller, R. Adams, D. Blei, and L. Paninski. Bayesian learning and inference in recurrent switching linear dynamical systems, Artificial Intelligence and Statistics, pages 914-922. PMLR, 2017.

---

[21] K. Luxem, F. Fuhrmann, J. Kürsch, S. Remy, and P. Bauer. Identifying behavioral structure from deep variational embeddings of animal motion. bioRxiv, 2020.
[22] A. Mathis, P. Mamidanna, K. M. Cury, T. Abe, V. N. Murthy, M. W. Mathis, and M. Bethge. Deeplabcut: markerless pose estimation of user-defined body parts with deep learning. Nature neuroscience, 21): 1291))288811199 2018.
[23] G. Mishne, R. Talmon, R. Meir, J. Schiller, M. Lavzin, U. Dubin, and R. R. Coifman. Hierar- chical coupled coupled-gcometry analysis for neuronal structure and activity pattern discovery. IEEE Journal of Selected Topics in Signal Processing, (0(7):1238-12533 2016.
[24] S. Musall, M. T. Kaufman, S. Gluf, and A. K. Churchland. Movement-related activity dominates cortex during sensory-guided decision making. BioRxiv, page 308288, 2018.
[25] S. Musall, M. T. Kaufman, A. L. Juavinett, S. Gluf, and A. K. Churchland. Single-trial neural dynamics are dominated by richly varied movements, Nature neuroscience, 22(10):1677-1666, 2019.
[26] T. D. Percira, D. E. Aldarondo, L. Willmore, M. Kislin, S. S.H.. Wang. M. Murthy, and J. W. Shaevitz. Fast animal pose estimation using deep neural networks. Nature methods, 16(1):
117-125,
[27] R. Shu, J. Brofos, F. Zhang, H. H. Bui, M. Ghavamzadeh, and M. Kochenderfer. Stochastic video prediction with conditional density estimation. In ECCV Workshop on Action and Anticipation for Visual Learning, volume 2, page 2, 2016.
[28] G. J. Stephens, B. Johnson-Kerner, W. Bialek, and W. S. Ryu. Dimensionality and dynamics in the behavior of c. elegans. PLoS Comput Biol, ((4)e100002) 2008.
(29] S. Tulyakov, M.-Y. Liu, X. Yang, and J. Kautz. Mocogan: Decomposing motion and content for video generation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1526-1535, 2018.
[30] R. Villegas, J. Yang. S. Hong. X. Lin, and H. Lee. Decomposing motion and content for natural video sequence prediction. In Sth International Conference on Learning Representations, ICLR 2017. International Conference on Learning Representations, ICLR, 2017.
[31] A. B. Wiltschko, M. J. Johnson, G. Turilli, R. E. Peterson, J. M. Katon, S. L. Pashkovski, V. E. Abraira, R. Adams, and S. R. Datta. Mapping sub-second structure in mouse behavior.
Neuron, 88(6): 122-1135, 2015.
[32] A. Wu, E. K. Buchanan, M. Whiteway, M. Schartner, G. Meijer, J.-P. Noel, E. Rodriguez, C. Everett, A. Norovich, E. Schaffer, al. Deep graph pose: semi-supervised deep graphical model for improved animal pose tracking. bioRxiv, 2020.
[33] L. Yingzhen and S. Mandt. Disentangled sequential autoencoder. In International Conference on Machine Learning, pages 1670-56797 PMLR, 2018.
[34] Y. Zhu, M. R. Min, A. Kadav, and H. P. Graf. S3vae: Self-supervised sequential vae for repre- sentation disentanglement and data generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6538-6547, 2020.
12


Reviews:
[{"review_id": "ytv92r7P9G4", "reviewer": "Reviewer_bsYQ", "summary": "Animal behavior can be a huge space full of behavioral motifs. Hand-labeling allows for algorithms to much better extract behavioral information, but it takes a lot of time. This paper tries to address this problem by building an algorithm that extracts this information from unlabeled behavioral videos. The authors evaluate the ability of the algorithm to generalize across sessions, as well as the quality of the various behavioral motifs identified by DisAE. \n", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "In this paper the authors develop an unsupervised algorithm to extract behavioral motifs from either a hand reach task that they collected or a decision-making task from an already published work. The authors are properly motivating the question and cite relevant work adequately. They present their results with clarity and the paper is well organized.  \n\nThe proposed approach (DisAE algorithm) shows great generalization performance across sessions when compared to BehaveNet. Moreover, it shows quite good performance in predicting forelimb trajectories from disentangled embeddings showing that the DisAE embeddings can carry information about various behavioral markers. Finally, when coupled with either ARHMM or VAME (or just the DBE) it has great ability to capture various behavioral motifs.  \n\nIt would be nice to evaluate how the algorithm handles the time-variant non-behavioral factors such as moving bedding, visual stimulus reflections etc. In some of the example videos it can be seen that the DBE captures the movement of the food pellet disk which I expect would influence the ability of the algorithm to separate the motifs. Also, the context identification is automatic but how does it ensure that the context embedding does not contain any behavioral information? While in the text (4.4 section) it is claimed that the model successfully learns the dynamics of the behaviors based on the ability to generate the behavioral output, the included videos in the supplementary data show relatively poor performance of the DBE algorithm in generating a behavior. It would be important to provide a proper evaluation of the DBE algorithm performance by showing the various motifs that can be generated. \n\nOverall, I find this paper good and while there are several algorithms that can be used to dissect behavioral motifs, I find that the DisAE algorithm shows great performance and once combined with other temporal analysis models it can be useful for analysis of behavioral motifs in unlabeled videos in an unsupervised manner. \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "In section 5 the authors discuss two important limitations of their work, the ability of the algorithm to generalize on unobserved behavioral motifs and the sensitivity to the stationarity of the non-behavioral factors. I would include as a limitation the fact that even if unsupervised, this algorithm needs supervision for the identification and further classification of various behavioral motifs. ", "review_text": "", "extra_scores": {"ethical_concerns": "N/A", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "ttIHsxaMSt", "reviewer": "Reviewer_Yzq9", "summary": "In this work, the authors propose a new model for analyzing “unlabeled, multi-view, high-resolution behavioral videos across different animals and multiple sessions”. This model (Disentangled Autoencoder, DisAE) learns robust behavior embeddings that are disentangled from the constant context in the experiments. The authors further add a stochastic dynamic model to DisAE to get an end-to-end approach (DBE) that both learns meaning representations and generates videos. After comparing the proposed models to previously proposed ones, the authors show that DisAE and DBE outperform earlier models on encoding less information about the experiment session and also on motif segmentation tasks. However, this paper lacks ablation studies supporting the specific designs in the models. Overall, the new models proposed in this work seem to be stronger than others and are therefore worth being known to the community.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This work proposes a neural network model for behavior analysis on videos recorded from behavioral experiments from different animals and multiple sessions. The model, called DisAE, is designed to produce embeddings only encoding animal behaviors but not the experiment context. This is done through having one embedding that varies across different frames to encode the animal pose information and another embedding that is only determined by one frame to encode the session-specific context information. Both embeddings are then concatenated to get the final embedding that is used to reconstruct the input frames. Based on this DisAE model, the authors further propose a stochastic dynamic model to learn and infer how the pose embeddings evolve through time. This model is then added to DisAE, yielding the DBE model, which is trained end-to-end to learn behavior representations and generate new videos. To illustrate the power of these two proposed models, the authors first compare them with BehaveNet and DLC on how much the session-specific information can be decoded from the animal pose embeddings and show that DisAE and DBE indeed have less information about sessions in their pose-specific embeddings, just as designed. These two models are then compared to other models on how well they can do motif segmentation, which also yields better performance. The authors also show that DBE successfully reconstructs reasonable videos even for a different context. \nAlthough the better performance of DisAE and DBE compared to other models is indeed significant, the paper can greatly benefit from more ablation studies to justify its design. For example, the authors have mentioned that the pose embeddings are designed to be smaller dimensional to make them only contain the dynamic information. But there are no empirical results supporting this design. The authors could have compared the performance of models with different dimensions on the relevant task (like the session information prediction task) to show this. Another design is the choice of using Gaussian mixture model instead of single Gaussian distribution in the stochastic dynamic model. Though it is theoretically reasonable to imagine how different actions could require different Gaussian distributions, an empirical experiment to support this hypothesis can still be useful.\nIn general, the new models proposed in this work are indeed models with better performance and therefore worth being known by the community. The paper itself is also well-written and easy to read.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes, both of these are briefly addressed in the discussion section.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4.5"}, "extra_text": {}}, {"review_id": "szL4Ur3tid_", "reviewer": "Reviewer_88jd", "summary": "The authors present an end-to-end trainable framework for the unsupervised analysis of animal behavioral videos. Their model produces a latent representation for each frame which separates pose information from context (background) information, as well as models the temporal evolution of the pose information. This model can be applied to multi-view videos with varying background information.", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "originality: This paper combines ideas from several previous works in a seemingly novel way. It would be useful in the Related Works section if the authors could comment on how their approach differs from DRNet (DBE does not use an adversarial loss for disentanglement; are dynamics models similar?), DDPAE (not clear what similarities/differences are here), and Yingzhen and Mandt (do they model temporal dynamics?). The paper provides clear advances over the references cited in the behavioral video analysis section.\n\nquality: The method is sufficiently tested and rigorously compared to relevant baselines.\n\nclarity: The paper is very clearly written. Table 2 could benefit from a caption if space allows.\n\nsignificance: The authors have adapted several more general video analysis models to address a specific (but important) video analysis task: the modeling of animal behavioral videos from neuroscience experiments. The results demonstrate this method clearly outperforms relevant baselines from the neuroscience literature (DLC, Behavenet, VAME) but does not compare to the more general models (DRNet, DDPAE, etc).", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "This model was thoroughly tested on two head-fixed mouse datasets, but it is not clear how well the model will work when analyzing videos of freely moving animals. I don't think the authors need to analyze a new dataset, but they should mention this limitation (of their analyses, not necessarily the model) in the discussion.", "review_text": "", "extra_scores": {"ethical_concerns": "None", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "1"}, "extra_text": {}}, {"review_id": "PH1UTH_EZM", "reviewer": "Reviewer_d5ns", "summary": "This paper introduces a method named Disentangled Autoencoder (DisAE) to learn disentangled pose and context representations from videos. By keeping the pose component static based on a key frame and by extracting temporal context components, the proposed DBE model starts to generate both continuous and discrete representations from the input video. Temporal dynamics of the pose components were learned using sequential variational autoencoder under a stochastic dynamic model which utilizes a gaussian mixture prior considering two stochasitic variables for the mixture index and a gaussian noise. The method was evaluated on controlled multi-session video datasets, one which the authors collected (Hand-reach) and another one is WFCI. Associated code and sample videos were also shared. However, Hand-reach dataset was not openly available.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper highlights an interesting and important work in learning disentangled representations of animal behavior. The collected Hand-reach dataset is much appreciated; however, the dataset is not open-sourced as far as I can tell. Some comments and concerns are as below:\n\n1.\tThe manuscript is fairly well written. However, the use of abbreviations and mathematical terms without proper explanations affect the readability. For example, in line 50, VAME is not defined; in line 99, ARHMM is not defined; in line 108, Z^p_(1:T) is not defined.\n\n2.\tFigure 1 is not referred in the manuscript.\n\n3.\tFigure 1 omitted the multi-view nature of the input dataset. This brings confusion on how the context and pose embeddings are generated from multiple views. There are notable differences between Section 3.2 Disentanglement and Figure 1, which makes it hard for the readers to connect the overview of DBE (Figure 1) and the methodology.\n\n4.\tIn the code shared, models/nets.py presents the class definitions for BehaveNet, DisAE, and DBE. IN DisAE and DBE network class definitions, authors define a self.first_frame variable (default = True) taking input from config file. While going through the training code, there is no apparent updates provided to this variable during the optimization. If that is the case, then the if condition in lines 161, 255, and 326 will always be taken. Was this deliberate or an error in the code? From the code it looks like the embeddings are getting fused here. Am I missing something?\n\n5.\tIn Figure 5, the inter quartile range associated with the R^2 score for first few sessions are considerably larger than the later sessions. Does this have any relations to how the DisAE model was trained and the session classification performance?\n\n6.\tI would like to hear the authors thoughts on how the proposed disentanglement methodology might work on large behavior datasets such as CAlMS21 (https://data.caltech.edu/records/1991) which was recently published. Multi-session multi-subject data of large scale could affect the performance of the proposed stochastic dynamic models while generating long videos. I am curious to see how the length of input videos play a part in the video generation over multiple states.\n\n7.\tIt would be better if the data descriptions and hardware and software experimental setup are presented as subsections instead of as paragraphs under Section 4 - Results. \n\n8.\tIt would also be better if the main hyperparameters chosen for the training are included in the main manuscript rather than in the appendix.\n\n9.\tSome information about the motif transitions could have been included in Figure 8 to compare based on the source and generated videos.\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Have the authors adequately addressed the limitations and potential negative societal impact of their work? Yes", "review_text": "", "extra_scores": {"ethical_concerns": "None", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "7 hrs"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes to learn disentagled embeddings of behavioural video---between time-dependent dynamic factors (called pose) and other factors (called context) in an unsupervised manner.

The motivation, model setup, objective, and experimental setup are all very well described, evaluated, and analysed. The model in particular is motivated very well given the specific application in consideration (behavioural video), and the ablations performed serve to highlight the advantages of the main components. The analyses also serve to highlight the effects of the disentanglement provided by the model.

I would strongly encourage the authors to carefully consider the reviewers' comments about edits and corrections, and ensure that these are incorporated in the updated manuscript.

Overall, the reviewers agree that this is a very good piece of work and based on these merits, the paper should be accepted for publication.

Decision: Accept (Spotlight)