Relax, it doesn’t matter how you get there: A new self-supervised approach for multi-timescale behavior analysis
Authors: Mehdi Azabou
Michael Jacob Mendelson
Nauman Ahad
Maks Sorokin
Shantanu Thakoor
Carolina Urzay
Eva L Dyer
Venue: NeurIPS 2023 spotlight (2023)
Keywords: animal behavior
behavioral neuroscience
self-supervised learning
multi-timescale
TL;DR: This paper presents a self-supervised learning model to analyze animal tracking data and to reveal the factors that underlie animal behavior across different timescales.

Abstract: Unconstrained and natural  behavior consists of dynamics that are complex and  unpredictable, especially when trying to predict what will happen  multiple steps into the future. While some success has been found in building representations of animal behavior under constrained or simplified task-based conditions, many of these models cannot be applied to free and naturalistic settings where behavior becomes increasingly hard to model. In this work, we develop a multi-task representation learning model for animal behavior that combines two novel components: (i) an action-prediction objective that aims to predict the  distribution of actions over future timesteps, and (ii) a multi-scale architecture that builds separate latent spaces to accommodate short- and long-term dynamics. After demonstrating the ability of the method to build representations of both local and global dynamics in robots in varying environments and terrains, we apply our method to the MABe 2022 Multi-Agent Behavior challenge, where our model ranks first overall on both mice and fly benchmarks. In all of these cases, we show that our model can build representations that capture the many different factors that drive behavior and solve a wide range of downstream tasks.

Relax, it doesn't 't matter how you get there: A new self-supervised approach for multi-timescale behavior analysis
Mehdi Azabou" Georgia Tech
Michael J. Mendelson Georgia Tech
Nauman Ahad Georgia Tech
Maks Sorokin Georgia Tech
Shantanu Thakoor
Carolina Urzay
Eva L. Dyer'
DeepMind
Georgia Tech
Georgia Tech
Abstract
Unconstrained and natural behavior consists of dynamics that are complex and unpredictable, especially when trying to predict what will happen multiple steps into the future. While some success has been found in building representations animal behavior under constrained or simplified task-based conditions, many of these models cannot be applied to free and naturalistic settings where behavior becomes increasingly hard to model. In this work, we develop a multi-task repre- sentation learning model for animal behavior that combines (i) an action-predee actioon-rediction objective that aims to predict the distribution of actions over future timesteps, and (ii) multi-scale architecture that builds separate latent spaces to accommodate short- and long-term dynamics. After demonstrating the ability of the method build representations of both local and global dynamics in robots in varying environments and terrains, we apply our method to the MABe 2022 Multi-Agent Behavior challenge, where our model ranks first overall on both mice and fly benchmarks. In all of these cases, we show that our model can build representations that capture the many different factors that drive behavior and solve wide range of downstream tasks.
novel components:
Introduction
Behavior is shaped by various factors operating across different timescales. Immediate motivations can drive moment-to-momett interactions, while long-term experiences even the time of day can influence behavior on broader scales. Analyzing these dynamics, particularly in complex and naturalistic contexts (34; 28), has now become critical component in many modern studies in neuroscience (28), cognitive science, and in social behavior and decision making (38; 52; 56; 5). Additionally, monitoring and tracking systems allow for modeling of multi-agent interactions (27; 11; 16) and social behaviors (52; 1), providing valuable insights into dynamics across many individuals.
In order to learn latent factors that may influence behavioral patterns, a promising solution is to build models of behavior in unsupervised manner (25; 68). Unsupervised models are of particular interest in this domain as becomes hard to identify complex behaviors which can composed of many "syllables" of movement (65), and are thus hard and tedious annotate. Recent work in this direction build such representations using generative modeling and reconstruction. based objectives,
Contact: [mazabou,evadyer] @ gatech.edu. Project page and code: https://multiacat behavior.
github.ion io/
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

time (secondd
target
predicted
State Action
BMD"los
bins
learned represeettionnnn
receptive field
Long-term TCN
Shart-term
TCN
Predictor MLP
Figure 1: Overview of our opproach. Bootstrap Across Multiple Scales (BAMS) uses two temporal convolu- tional networks with two latent spaces, each with their own receptive field sizes. The model is trained on a novel learning objective that consists in predicting future action distributions instead of future action sequences. In this figure, sample data from the MABe dataset, only subset of the channels shown.
typically by performing open loop (58; 13; 10) closed loop (13) prediction of observations actions multiple timesteps into the future.
However, when using a reconstruction or prediction objective to analyze behavior, future actions become hard to predict and models can start become myopie, focusing only on short-range interactions in the data (65). To circumvent this overly local learning of dynamics, there have been number of efforts build models of long-term behavioral style (41; 6), where instance-level learning methods are used to extract single representation for entire sequence. However, these models then lose their ability to provide time-varying representations that capture the dynamic nature of different behaviors. It is still an outstanding challenge to build representations that can capture both short-term behavioral dynamics along with longer-term trends and global structure.
In this work, we develop a new self- approach for learning multiscale representations of behavior. Our method consists two core innovations: (i) novel action- preddictonnnionn approach that aims to predict the distribution of actions over future timesteps, without modeling exactly when each action taken, and (ii) novel multi-scale architecture that builds separate latent spaces to accommodate short- and long-term dynamics. We combine both of these innovations and show that our approach can capture both long-term and short-term attributes of behavior and work flexibly to solve variety of different downstream tasks.
To our approach, utilize behavioral datasets that contain multiple tasks that vary in complexity and contain distinct synthetic dataset of the multi-limb kinematics from quadruped robots by varying the robot's morphological properties and the environment's 's terrain type and difficulty. Using this robot behavior data, we demonstrate that our method can effectively build dynamical models of behavior that accurately elicit both the robot and environment properties, without any explicit training signal encouraging the learning of this information.
dynamics. Using NVIDIA's Isaac Gym (37), we generate
Having established that our approach can successfully predict the complex behavior of an artificial creature, we apply it to two multi-agent behavior benchmarks (61) and challenges with multiple tasks that vary in their frame-level (local) vs. sequence-level (global) labels and properties. On the mouse triplet benchmark, we rank first overall on the leaderboard (averaged across 13 tasks), first on all of the 4 global tasks, and are in the top-3 on all the 9 frame-level local subtasks. In one of the global tasks (decoding the strain of the mouse), we achieve impressive performance over the other methods, with 10% gap over the next best performing method. On the fruit fly groups benchmark, we also rank first overall (averaged across 50 tasks), and outperform other methods on both average frame-level and sequence-level subtasks. Our results demonstrate that our approach can provide representations that can be used to decode meaningful information from behavior that spans many timescales (longer approach interactions, grooming, etc.) well as global attributes like the time of day or the strain the mouse.
nicrowd www/challenggss/mmiiiiiooo htwp://www..mmii.iaaeeeeeeiiinne www.hhhhhhhhhllllloeeeee htt.://ww...../oo/oooo//aamaaeea
http://wwwnoooooooocoeeeeeeeee challenge-true
aicrowd. wwwplh/wweeaaaaaaaaaaaaccleeeeee htw.:///wwooaoooooooooooooooooaa
dyy.gooouscccdddrbbrddd/ post chellenge-true

---

The contributions of this work include:
A superruuee framework that learns representations in two separate long-term and short- term embedding spaces. Bootstrapping is performed within each timescale, using latent predictive loss across positive views only, and hence can process much longer sequences than contrastive methods that require negative views.
A novel prediction task for behavioral analysis and cloning called HoA (histogram of actions), which aims to predict the future distribution of keypoints instead of the precise ordering of future states. We use an efficient implementation of the 2- 2-Wasserstein divergence as a measure of distributional fit between the true movement distribution and the predicted movements.
New stateoffthhearr performance: When applied to the Multi-agent (MABe) Benchmark, model ranked #1 overall and achieves top scores on all of the sequence-level (global) tasks. In aggregate, we achieve the top rank on the leaderboard.
A procedurally generated dataset of walking quadruped robots with ground truth annotations and multiple frame- and sequence-leve tasks. We believe that this dataset can provide a robust benchmark for future work in this direction.
2 Method
Our approach addresses two critical challenges of modeling naturalistic behavior (Figure 1). In Section 2.2, introduce a novel distributional-relaattoo of the eecontruuction-based learning objective. Section 2.3, we introduce an architecture and self-supervised learning objectives that support the learning of behavior at different levels of temporal granularity.
2.1 Problem setup
We assume a fixed dataset of trajectories, each comprised of a sequence observations x; and/or actions yr. Where actions are not explicitly provided, in many cases we can infer actions based on the difference between consecutive observations. Our goal is to learn, for each timestep, behavioral representations z, that capture both global-information such as the strain of the mouse the time of day, as well as temporally loccllisd representations such as the activity each mouse is engaged in at given point in time. As obtaining labeled datasets for realistically-use useful scales of agent population and diversity of behavior is impractical, we aim to learn representations in an unsupervised manner.
2.2 Histogram of Actions (HoA): A novel objective for predicting future
Modeling behavior dynamics can be done by training model to predict future actions. This reconstruction-base. objective becomes challenging when behavior is complex and non-stereotyped. Let us consider the example of a mouse scanning the room, rotating its head from one side to the other. It is possible to extrapolate the trajectory of the head over a few milliseconds, but prediction quickly becomes impossible, not because this particular behavior complex but because any temporal misalignment in the prediction leads to increasing errors.
We propose to predict the distribution of future actions rather than their sequence. The motivation behind this distributional-reaxaatoo of the reconstruction objective lies in blurring the exact temporal unfolding of the actions while preserving their behavioral fingerprint.
Predicting histograms of future actions. Let y, E N be the action vector at time Each feature in the action vector can, for example, represent the linear velocity of a joint or the angular velocity of the head. Given observations "u.....,,, of the behavior at timesteps through f, the objective is to predict the distribution of future actions over the next L timesteps. For each i-th element of the action vector, we compute -iedimmnsional normalized histogram of the values it takes between timesteps + and L. We pre-partition the space of action values into K equally spaced bins, resulting in -ddimensional histogram that we denotes as h,,,, for all keypoints S S N.
We introduce a predictor that, given the extracted representation 2; predicts all feature-wise histograms of future actions. The predictor a multi-layer perceptron (MLP) with an output space in Ig x The output is split into N vectors, which are normalized using the softmax operator. We henl. each estimating the histogram of the i-th action feature following timestep t. obtain [h.......
3

---

EMD² loss for histograms. To measure the loss between the predicted and target histograms, we use the Discrete Wasserstein distance, also known the Earth Mover's Distance (EMD). This distance is obtained by solving an optimal transport problem that consists in moving mass from one distribution to the other while incurring the lowest transport cost. In our case, the cost of moving mass from one bin to another is equal to the number of steps between the two bins.
Because our histogram has equally-sized bins, the EMD is equivalent to the Mallows distance which has a closed-form solution (31; 20). In particular we use EMD which has been shown to be easier to optimize and converge faster (53). The loss is defined follows:
K
DEMD (he,s: h,,))
(CDF&
&DF (h,,)),
k=1
where CDF (h) is the k-th element of the cumulative distribution function of h.
The total loss is obtained by summing over all features of the action vector, which leaves us with the following loss time t:
N D DEMD (hr,, hy.s)
Le
2.3 Multi-timescale bootstrapping in temporally-diverse architecture
In order to form richer and multí-scale representations of behavior, we use self-supervised learning objective. We introduce new approach using latent predictive losses to build representations across different scales while preserving the granularity in each. We achieve this by explicitly separating the short-term and long-term representations, then bootstrapping within each representation space. This approach enables us to learn from otherwise incompatible representation learning objectives (55; 62).
2.3.1 Two latent spaces are better than one
Our goal is to capture and separate short-term and long-term dynamics in two different spaces. We use the Temporal Convolutional Network (TCN) (4) as our building block. The TCN produces representation at time / that only depends on the past observations (46).
We design architecture that separates the different timescales by using TCN encoders: A short-term encoder, fs. that captures short-term dynamics and targets momentary behaviors such as drinking, running or chasing; long-term encoder, fi, that captures long-term dynamics and targets longstanding factors that modulate behavior (strain of mouse, time of day). Architecturally, the difference between the two is that increase the number of layers and use larger dilation rates (8) for the long-term encoder, thus effectively covering a larger receptive field (more history) in the input sequence. All feature embeddings extracted by the TCNs are concatenated, to produce the final embedding, = concat iz,, z/].
2.3.2 Bootstrapping Across Multiple Scales
We draw inspiration from recent work (17; 50; 18) that uses latent bootstrapping to learn latent space where "positive" views are mapped close to each other. Unlike other contrastive methods (69), bootstrapping does not require negative examples.
positive range
In the context of temporal representation learning, a common as- sumption is that points that are nearby in time are positive views of each other and can be constrained to lie nearby in the latent space (69; 3). In our case, we can bootstrap and find positive views at both the short-term and also at a more long-term scale, as illustrated Figure 2.
positive range
Visualization of the short-term and long-term windows used build multi-scale similar- ity. Positives are selected within a window. The window is small for short-term embeddings and can be as large the entire sequence for long-term embeddings.
Figure 2:
Bootstrapping short-term representations. We randomly select samples, both future or past, that are within small window A of the current timestep other words, d E [-A, A]. Bootstrapping involves using shallow network to predict the representation of one
view from the other (17). We

---

use predictor 4s that takes in the short-term embedding z and learns to regress 2/+6 using the loss:
9. (,()) [g:
Z
Lr.short
sgg
where sg[:] denotes the stop gradient operator. Unlike (17). we do not use an exponential moving average of the model, but simply increase the learning rate of the predictor in (50).
Bootstrapping long-term representations. For long-term behavior embeddings which should be stable the level of sequence, sample any other time point in the same sequence, i.e. l' E [0, T7. We use a similar setup for the long-term behavior embedding, where predictor qi is trained over longer time periods in the limit, over the entire sequence.
(1(2) (z))l2 2
Lr.long =
sg
(4)
2.4 Putting it all together
Finally, we optimize the proposed multi-task architecture with a combined loss:
L: + afCr.hhort + Lr,tong)
(5)
where scalar that used to weigh the contribution of the short- and long-term contrastive losses. In practice, we find that we simply need to choose that re-scales the contrastive losses to the same order of magnitude as the Hoλ prediction loss.
3 Experiments
3.1 Simulated Quadrupeds Experiment
3.1.1
A synthetic dataset of simulated legged robots
To test our model's ability to separate behavioral factors that vary in complexity and contain distinct multi-timescale dynamics, we introduce new dataset generated from heterogeneous population of quadrupeds traversing different terrains. Simulation enables access information that generally inaccessible or hard to acquire in real-world setting and provides ground-truth information about the agent and the world state.
Agents. We use advanced robotic systems (33) that imitates 4-legged creatures capable of various locomotion skills. These robots are trained walk on challenging terrains using reinforcement learning (51). We use two robots that differ by their morphology, ANY mal B and ANYmal mal C. To create heterogeneity in the population, we randomize the body mass of the robot as well as the target traversal velocity. We track set of 24 proprioceptive features including linear and angular velocities of the robots' joints.
Procedurally generated environments. Using NVIDIA's Isaac Gym (33) simulation environment, we procedurally generate maps composed of multiple segments different terrains types (Figure 3). We consider five different terrains including flat surfaces, pits, hills, ascending and descending stairs. also vary the roughness and slope of the terrain to control the difficulty of terrain traversal.
Stairs
Flat
Stairs
Flat
Robot Type
Robot Velocity
www.mmm
WwAMAN WUU www
0.75
8.50
incrasiing velocity
Anymal
-0.5
Anymal
-225
-0.50
3.50
-1.0
-2.5
0.0
0.5
3.0
-3.0
-0.5
0.0
0.5
Figure 3: Quadrupeds walking on procedurally generated map. (A) Illustration showing the two robots LANYmal and mNY C) walking on the procedurally generated map, with segments of different terrain types. As the robots traverse different terrains, the velocity of their joints is tracked and visualized. (B) Long-term embeddings learned by BAMS. On the left, overlay the labels for the type of robot and on the right we overlay the velocity of the robot; we observe clear organization of the latents terms of their velocity and robot type.
5

---

Sequence-level Tasks
Frame-level Tasks
Model
Robot Type* (1) Linear velocity (L)
Terrain type* (t)
Terrain slope ())
Terrain difficulty (L)
PCA
99.72
0.069
08.83
0.037
0.790
TCN
99.93
0.102
33.03
0.037
0.080
BAMS
99,96 19,96
0.038
39.89
0.033
0.078
short-term 100.00
0.094
34.86
0.036
0.079
long-term 99.88
0.020
32.39
0.036
0.078
Table I: Linear readouts of robot behavior: For each task, we report the linear decoding performance on sequence-level and frame-level tasks. Tasks marked with * are classification tasks for which the Fl-score reported, for remaining tasks, we report the mean-squared error.
Experimental setup. We collect 5182 trajectories of robots walking through terrains. We record for minutes at frequency of 50Hz. For evaluation, we split the dataset into train and test sets (80/20 split) and use multi-task probes that correspond to different long-term and short-term behavioral factors. More details can be found in Appendix A.
3.1.2 Results
Results in Table I suggest that our model performs well on these diverse prediction tasks, A major advantage of our method is the separation of the short-term and long- long-term dynamics, which enables us to more clearly identify the multi-scale factors. While some of the tasks are represented best in the mixed model, we find that the linear velocity is more decodable in the long- serg-termm embedding. while terrain type is more decodable in the short-term embedding. We further analyze the formed representations by visualizing the embeddings in the different spaces. Figure 3-B, we visualize the long-term embedding space and find that our model is able to capture the main factors of variance in the dataset, corresponding to the robot type and the velocity at which the robots are moving. This suggest that in the absence of labels, the learned embedding can provide valuable insights into how the recorded population distributed without the need for annotations. In Appendix A, we visualize the extracted embeddings of single sequence over time. We find that the long-term embedding is more stable and smooth, while the short term embeddings reveal different blocks of behavior that change more frequently.
3.2 Experiments on Mouse Triplet Dataset
3.2.1 Experimental Setup and Tasks
Learned representation
Dataset description. The mouse triplet dataset (61) is part of the Multi-Agent Behavior Challenge (MABe 2022), hosted at CVPR 2022. This large-scale dataset was introduced to address the lack of standardized benchmarks for representation learning of animal behavior. consists of a set trajectories from three mice interacting an open-field arena. total of 5336 one-minute clips, recorded from top-view camera, were curated and processed to track twelve anatomically defined keypoints on each mouse, shown in Figure
Strain
Chasing 2e
Time of day
Figure 4: Multi-Agent Behavior (MABe) Mouse Triplets Challenge. (A) Keypoint tracking approaches used extract keypoints from many positions the mouse body in video. (B) Methods are evaluated across 13 different tasks.
As part of this benchmark, a set of 13 common behavior analysis tasks were identified, and are used evaluate the performance of representation learning methods. Over the course of these sequences, the mice might exhibit individual and social behaviors. Some might unfold at the frame level, like chasing being chased, others at the sequence level, like light cycles affecting the behavior of the mice or mouse strains that inherently differentiate behavior.
Integrating features across multiple animals. We process the trajectory data to extract 36 features characterizing each mouse individually, including head orientation, body velocity and joint angles. We construct short-term TCN and long-term TCN encoders to have representation size 32 each, and receptive fields approximated to be 60 and 1200 frames respectively. We compute the histogram of actions over second, i.e. T. = 30 and use K 32 bins. We build representations for each mouse independently, which means that at time and for each frame t, produce embeddings 2r,1, Z1,2

---

Sequence-level subtusks
Frame-fevel subiarks
Model
Day (1)
Time (1)
Strain
Lights
Approach
Chase
Close
Contact
Huddle
0/G
Q/O
Watching
PCA
0,09416
0,09445
$1.60
54.65
0.86
0.14
49.27
57.87
12.71
0.21
0.60
0.53
6.65
TVAE
0.09403
0.09442
52.98
59.33
44.77
21.96
0.83
10:20
T-BYOL
0.09362
0.093733
60.95
65.05
1.68
0.72
62.48
48.19
18.52
0.35
0.96
0.82
17.77
T-BERT
0.09262
0.09276
78.63
70.22
55.84
1,40
TS2Vee
0.09380
0.09422
57.12
65.60
1.29
0.66
59.53
46.13
24.74
0.35
1.09
0.74
12.37
T-Porceiver
0.09322
0.09223
69.81
69.68
1.57
1.27
60.84
47.81
28.32
0.41
1.16
0.8G
16.42
T-GPT
0.09269
0.09384
64.45
65.39
1.73
0.64
69.05
55.78
23.80
0.46
1.12
1.05
17.86
T- PoiniNet
0,09275
0.09320
66.01
67.15
2.56
4.57
70.68
55.96
21.23
0.84
2.79
2.32
15.61
BAMS
0.09099
0,08989
88.23
72.00
2.74
1,89
67.22
53.43
31.43
0.59
1.61
1.57
18.15
short-term
0.09288
0.09294
61.33
66.
1,80
1.15
66.58
52.60
25.34
0.39
1.09
0.98
16.74
long-term
0.09174
0.09037
86.49
70.91
2.10
1.06
61.99
49.12
29.09
0.45
1.32
1.08
13.98
Table 2: Linear readouts of mouse behavior: We report the BAMS against the top performing models in the MABe 2022 challenge. All numbers are reported in (59) except for TS2Vec and T-BYOL, which we produce. The scores show the performance of the linear readouts across 13 different tasks, Mean-squared error (MSE) is used in the case of tasks and 2, since they are continuously labeled. In the rest of the subtasks, which are binary (yes/no), Fl-scores are used. The best- performing models are those with low MSE scores and high F1 Fl-scores, are highlighted in bold.
and 3r,3, for each mouse respectively and concatentate the embeddings for all three individuals to build a joint embedding for evaluation. The model is trained for 500 epochs using the Adam optimizer with learning rate of 10-3, More details can be found in Appendix B.
Interaction loss. To learn additional features that are useful in predicting animal interactions for multi-agent settings, we introduce simple auxiliary loss to predict the distances between the trio at time f. Our input features do not include any information about the global position of the mice in the arena, so the model can only rely on the inherent behavior and movement of each individual mouse to draw conclusions about their proximity. Thus, we build network h that takes in the embeddings of two mice i and j and predicts the distance di between them as follows,
Laxx = [ Z.,)) di
(6)
This penalty is added the loss in Equation 5 to encourage learning of shared features across the different individual embeddings.
Evaluation protocol. To evaluate the performance of the model in detecting frame- and sequence- level behaviors, we compute representations 14,1 21,2 and 4,, for each mouse respectively, which we then aggregate into single mouse triplet embedding using two different pooling strategies. First, apply average pooling to get Zt.. Second, we apply max pooling and min pooling, then compute the difference to get Zz mi.mixx max min Both aggregated embeddings into a 128-dim embedding for each frame in the sequence. Evaluation of each one of the 13 tasks, is performed by training linear layer on top of the frozen representations, producing final F1 score or a mean squared error depending on the task.
concatenated
3.2.2 Results
We compare our model against PCA, trajectory VAE (TVAE) (13), TS2Vec (69), BYOL (17), and the top performing models in the MABe2022 challenge (61) which are adapted respectively from Perceiver (23), GPT (7), PointNet (49) and BERT (14). All methods are trained using some form of reconstruction-based objective, with both T-Perceiver and T-BERT using masked modeling. Both T-BERT and T-PointNet also supplement their training with a contrastive learning objective using positives from the same sequence. It is also important to note that the training set labels from two tasks (Lights and Chase) are made publicly available, and are used as additional supervision in the T-Perceiver and T-BERT models. We do not use any supervision for BAMS.
Our model achieves new state-of-the-an result on the MABe Multi-Agent Behavior 2022- Mouse Triplets Challenge, as can be seen in Table 2. We rank first overall based upon our performance on all tasks, and show impressive boosts in performance on global sequence-level tasks where we find a 22% improvement in the Strain task and 5% improvement in the Light task. In the frame-level tasks, we remain competitive with other approaches, and rank 1st on 3 out of 9 of the frame-level subtasks; this is in contrast to the other top performing model that explicitly models the interaction between mice by introducing hand-crafted pairwise features. This outside the scope of this work, as we do not focus social interactions beyond predicting the distance between mice.
7

---

We observe that our proposed method results in significant gaps on sequence-level tasks. In particular, we observe a marked umprovement on the prediction of strain over the second-place model 78 78.63%, with BAMS yielding a 10% improvement in accuracy at 88. 28.23%. These big improvements suggest that we might have identified and addressed a critical problem in behavior modeling. the next section, we conduct series of ablations to further dissect our model's performance.
Multi-timescale embedding separation also enables us to probe our model for timescale-specifte features. In Table 2, we report the decoding performance with short-term embeddings and long-term embeddings respectively. We find that sequence-leve behavioral factors are better revealed in the long-term space, while the frame-level factors are more distinct in the short-term space. That being said, decoding performance is still best when using both timescales.
By default, BAMS is pre-trained with all available trajectory data. We also test BAMS in the inductive setting, where we only pre-train using the training split (only 1800 out of 5336) of the dataset. Results are reported in Appendix B.5. We find that the performance drop is modest, and that BAMS trained in this setting still beats all other methods.
3.2.3 Ablations
To understand the role of the different proposed com- ponents in enabling to achieve state-of-the-art per- formance, we conduct a series of ablations on the MABe benchmark that we report in Table 3. First, we compare the performance of BAMS when trained with the traditional reconstruction based objective (multi-step sequential prediction). BAMS without the НоA prediction objective, performs comparably with many of the top-entries, though performance ultimately improved when using this novel learning objective. We note that we had to reduce the number of prediction frames to 10, as the training fails if we go beyond. With our Ноλ objective, we can use 30 frames, which strongly suggests that this loss can be stably applied over longer time horizons when compared to other losses.
Hist. of actions
Muli- Muli- tinesiti
Bontstrapiig
Seq MSE
Seq
Frame F1
0.090415
80.12
19.85
0,093100
68.30
19.46
0.090717
78.11
19.45
0.092483
73.37
19.52
Table 3:
Ablations on the Mouse Triplet Dataset. We report the average sequence-level MSF and F1- score, and the average frame-level F1-score.
Next, we analyze the role of the multi-timeseal bootstrapping in improving the quality of the representation. When removing the bootstrapping objective, we find a 2% drop in the sequence-level averaged F1-score, as well modest drops in frame-level performance. This suggests that this learning objective may help to resolve global and intermediate-scal features. We also ablate the multi-timescale component, i.e. we perform bootstrapping but in the same space, by using a single TCN that spans the receptive field of the two used originally. This results in sub-optimal performance, which emphasizes the idea that having different objectives in different spaces critical prevent interference and provide ideal performance. We report additional ablations in Appendix B.4, including an ablation of the interaction module where we find that it is not critical for good performance.
3.3 Experiments on Fruit Fly Groups Dataset
3.3.1 Experimental Setup and Tasks
The fruit fly groups dataset is the second dataset the MABe benchmark. It consists of tracking data of group of 9 to 11 flies interacting in a small dish (Figure 5). Tracking data consists of 19 keypoints on each fly body and wings, and is recorded much higher frame rate compared to the mice (150 vs. 30 fps). Precise neural activity manipulations are performed on certain neurons which, when activated, induce certain types of behavior including courtship, avoidance and female aggression (61). Additionally, the groups of flies are differentiated by various genetic mutations and tagged by sex. This along with other behavioral factors provide us with 50 different subtasks, both fevel-level and sequence-level, that can be
Figure 5: Sample frame from the Fruit Fly Groups Dataset.
evaluate the learned representations.
In this set of experiments, we are interested in testing whether our proposed method generalizes to dataset from another organism, and whether it can work out-of-the-bee box with minimal expert
8

---

intervention. In particular we chose to not extract any features manually and only use the provided data, we use the default hyperparameters found in the previous experiment and do not perform hyperparameter tuning. and finally we do not use an interaction module. Note that we follow the same evaluation procedure established previously.
3.3.2 Results
We compare our model against PCA, TVAE. TS2Vec, and the top performing models in the fly challenge: T-GPT, and T-Perceiver. Our model achieves state- of-the-art performance on the fly dataset, as seen in Table 4. BAMS outperforms other models on both frame-level and sequence-level sub-tasks, and we note significant boost in the average frame-level F1 score. This result further demonstrates the generaliz- ability of our approach to new datasets and scaling to an even larger numbers of animals.
Model
All
Seq F1
Frame
PCA
42.5
23.0
45.2
TVAE
37.0
22.2
39.0
T-Perceiver
44.8
19.7
48.2
T-GPT
45.8
24.5
48.7
BAMS
48.2
25.4
51.1
Table 4: Linear readouts of fly behavior We report the average performance of various models on both frame-level, and sequence- subtasks,
4 Related Work
4.1 Animal behavior analysis
Pose estimation and animal tracking. Recently, there has been a recent democratization of automated methods for pose estimation and animal tracking that has made it possible to conduct large scale behavioral studies in many scientific domains. These tools abstract behavior trajectories from video recordings, and facilitate the modeling of behavioral dynamics. Most pipelines for analyzing animal behavior consist of three key steps (36): (1) pose estimation (60; 67; 15; 48; 40), (2) spatial-temporal feature extraction (35; 10), and (3) quantification and phenotyping of behavior (43; 39; 66). In our work, we consider the analysis of behavior after pose estimation performed. However, one could imagine using our multi-timescale bootstrapping approach for representation learning in video analysis, where eelf-supervised losses have been proposed recently for keypoint discovery (60; 64; 24).
Disentanglement of animal behavior in videos. In recent work (54; 32; 63), distinct disentangled behaviorial embeddings are learned from video, separating non- -on-behhvioral features (context, recording condition, etc.) from the dynamic behavioral factors (pose). This has even been applied to situations with multiple individuals, performing disentanglement on each individual (21). This is performed by training two encoders, typically variational autoencoders (VAEs) (29), on either multi-view dynamic information or single image. These encoders create explicitly separated embeddings of behavioral and context features. In comparison to this work, rather than seperating behavior from context, our model considers the explicit separation of behavioral embeddings across multiple timescales, and considers the construction of a global embedding that consistent over long timescales.
Modeling social behavior. For social and multi-animal datasets, there are a number of other challenges that arise. Simba (45) and MARS (52) have similar overall workflows for detecting keypoints and pose of many animals and classifying social behaviors. More recently, semi- supervised approach TREBA has been introduced (59) for building behavior embeddings using task programming. TREBA is built on top of the trajectory VAE (13), a variational generative model for learning representations of physical trajectories in space. In our work, we do not consider a reconstruction objective but a future prediction objective, in addition to bootstrapping the behavior representations at different timescales.
4.2 Representation learning for sequential data
The self-supervised learning (SSL) framework has gained lot of popularity recently due to its impressive performance in many domains (14; 9; 12). Many SSL methods are built based on the concept of instance-specife aligmment loss: Different views of each datapoint are created based on pre-selected augmentations, and the views that are produced from the same datapoint are treated as positive examples; while the views that are produced from different datapoints are treated as negative examples. While contrastive methods like SimCLR (9) utilize both positive examples and negative examples to guide the learning, BYOL (17) proposes framework in which augmentations of

---

sample are brought closer together in the representation space through predictive regression loss. Recent work (18; 50; 44; 2) applies BYOL to learn representations of sequential data. In such cases, neighboring samples in time are considered to be positive examples of each other, assuming temporal smoothness of the semantics underlying the sequence. The model is trained such that neighboring samples in time are mapped close to each other in the representation space. However, these methods use a single scale to define similarities unlike our method.
Recently, self-supervised methods such as TS2vec (69) learn self-supervised representations for sequential data by generating positive sub- seuusee views through more complex temporal augmenta- tions that can be integrated at instance or local level. Positive sub-sequences are temporally contrasted with other representations within the same sequence as well contrasted with representations of other available sequences. However, unlike our method, the same space of representations are used for learning these multi-scale representations. We learn separate space of representations for different time scales to encourage disentanglement.
The idea of using multi-scale feature extractors can be found in representation learning. In (50), a video representation learning framework, two different encoders process a narrow view and a broad view respectively. The narrow view corresponds broad view spans a larger timescale. The objective, however, is different to ours, as the narrow and broad representations are brought closer to each other, in the goal of encoding their mutual information. This strategy is also used in graph representation learning where a local-neighborhood of node is compared its global neighborhood (19; 22). Our method differs that we bootstrap the embeddings at different timescales separately, this is important to maintain the fine granularity specific to each timescale, thus revealing richer information about behavioral dynamics.
video clip of a few seconds, while the
5 Conclusion
Behavior is likely to be driven by a number of factors that can unfold over different timescales. Thus, having ways model behavior and discover differences in behavioral repertoires actions many scales could provide insights into individual differences, and help, for example, detect signatures of cognitive impairment (66). We make steps towards addressing these needs by proposing BAMS, a novel celf-supervised approach that learns representations for behavioral data at different timescales.
Our analysis on realworld datasets centers on two different datasets in multi-agent benchmark, both with very different dimensionalities and variable tasks. Despite this, our approach is designed to model wide range of behaviors, whether it's individual, social, structured or naturalistic. More multi-scale and multi-task benchmarks like MABe are needed in order properly evaluate and guide the development of tools for general behavior analysis.
Currently, our model has a relatively simple way of (optionally) modeling interactions between animals in the multi-agent setting. Despite this, we show really competitive performance in multi- agent analysis without using handcrafted interaction features like T-PointNet or T-BERT (61). Moving forward, it would be interesting to develop new ways to learn features of multi-animal interactions, especially in open environments where animals might come in and out of frame or get occluded.
Our experiments highlight the truly multi scale nature of BAMS and show that our method can learn to distinguish global as well as temporally local behaviors. Currently, we seperate short-term from long-term dynamics through contrastive loss and separation of the information into two latent spaces for each scale. Although this approach appears to be effective, we don't modify the loss explicitly to disentangle the two latent spaces. By providing additional incentives for the model to separate short from long-term dynamics we hope to improve the interpretability of the model.
When extending our model from mice to flies, we found that it was possible to use the same overall model architecture and hyperparameters, despite the major differences in datasets and underlying tasks. Thus, given the robustness of the method, we imagine that it can be utilized in the analysis of other species and even more diverse types of behavior (42; 26). By combining our modeling approach with methods for self-supervised video keypoint discovery (57), we could further extend BAMS to raw video data without needing the intermediate step of pose estimation. We also look forward to the opportunities provided by simultaneous recordings of the brain and behavior (30; 47) and other multi-modal sources of input that could be leveraged to further study how behavior unfolds across different timescales.
10

---

Acknowledgements
We would like to thank Mohammad Gheshlaghi Azar and Remi Munos for their feedback on the work. This project was supported by NIH award 1 IR01EB029852, NSF awards IIS-2039741 and IIS-2146072, as well as generous gifts from the Alfred Sloan Foundation, the McKnight Foundation, and the CIFAR Azrieli Global Scholars Program.
References
[1] David J Anderson and Pietro Perona. Toward a science of computational ethology. Neuron, 84(1):18-31,
2014.
[2] Mehdi Azabou, Mohammad Gheshlaghi Azar, Ran Liu, Chi-Heng Lin, Erik C. Johnson, Kiran Bhaskaran- Nair, Max Dabagia, Bernardo Avila-Pires, Lindsey Kitchell, Keith B. Hengen, William Gray-Ronead Michal Valko, and Eva L. Dyer. Mine your own view: Self-supervised learning through across- across-sample
prediction, 2021. 10
[3] Mehdi Azabou, Max Dabagia, Ran Liu, Chi-Heng Lin, Keith B Hengen, and Eva L. Dyer. Using self- supervision and augmentations to build insights into neural coding. NeurlPS 2021 Workshop on Self-
supervised Learning: Theory and Practice, 2021. 4
[4] Shaojic Bai, J. Zico Kolter, and Vladlen Koltun. An empirical evaluation of generic convolutional and
recurrent networks for sequence modeling, 2018. 4, 16
[5] Kathleen Bates, Kim N Le, and Hang Lu. Deep learning for robust and flcxible tracking in behavioral studies for c. clegans. PLOS Computational Biology, 18(4))c1009942, 2022. 1
[6] Jane Bromley, Isabelle Guyon, Yann LeCun, Eduard Säckinger, and Roopak Shah. Signature verification using a" siamese" time delay neural network. Advances in neural information processing systems, 6, 1993.
[7] Tom Brown, Benjamin Mann, Nick Ryder, Melanic Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, ct al. Language models are fow-shot leamers. Advances in neural information processing systems, 33:1877-1901, 2020. 7
[8]
Liang-Chieh Chen, Yukun Zhu, George Papandreou, Florian Schroff, and Hartwig Adam. Encoder-decoder with atrous separable convolution for semantic image segmentation. Proceedings of the European conference on computer vision (ECCV), pages 801-818, 2018. 4
[9] Ting Chen, Simon Korblithh Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pages
1597-1607. PMLR. 2020. 9
[10] Xinyu Chen, Jiajic Xu, Rui Zhou, Wei Chen, Junhua Fang, and Chengfei Liu. Trajvae: A variational
autoencoder model for trajectory generation. Neurocomputing, 428:332 339, 2021. 2, 9
[11] Zexin Chen, Ruihan Zhang, Yu Eva Zhang, Haowen Zhou, Hao-Shu Fang, Rachel R Rock, Aneesh Bal, Nancy Corila- Laurel Keyes, Kay M Tye, et al. Alphatracker: multi-anina tracking and
behavioral analysis tool. biorxiv, 2020.
[12] Yu-An Chung and James Glass. Specch2vce: scquence-to-sequeeee framework for learning word embeddings from speech. arXiv preprint 1803.
2018. 9
[13] John Co- Re-Reyes YuXuan Liu, Abhishek Gupta. Benjamin Eysenbach, Pieter Abbeel, and Sergey Levine. Self consistent trajectory autoencoder: Hierarchical reinforcement learning with trajectory embeddings. In International conference on machine learning. pages
PMLR, 2018. 2, 7.9
[14] Jacob Devlin, Ming-Wei Chang, Kenton Lec, and Kristina Toutanova. Bert: Pre-training of deep bidirec-
tional transformers for language understanding, arXiv preprini 1810v 04805, 2018. 7.
[15] Brandon J Forys, Dongsheng Xiao, Pankaj Gupta, and Timothy H Murphy. Real-time seleetive markerless
tracking of forepaws of head fixed mice using deep neural networks. Eneuro. 7(3), 2020. 9
[16] Keisuke Fujii, Naoya Takeishi, Kazushi Tsutsui, Emyo Fujioka, Nozomi Nishiumi, Ryoya Tanaka, Mika Fukushiro, Kaoru Ide, Hiroyoshi Kohno, Ken Yoda, et al. Learning interaction rules from multi-animal trajectories via augmented behavioral models. Advances in Neural Information Processing Systems. 2021. 1
[17] lean-Bastion Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent: A new approach to self-supervised learning. arXiv preprini arXiv:2006.07733 2020. 4, 5,
7.9
[18] Daniel Guo, Bernardo Avila Pires, Bilal Piot, Jean-bastien Grill, Florent Altché, Rémi Munos, and Mohammad Gheshlaghi Azar. Bootstrap latent-predictive representations for multitask reinforcement learning. arXiv preprint arXiv:2004. 14646, 2020. 4, 10
[19] Kaveh Hassani and Amir Hosein Khasahmadi. Contrastive multi- representation learning on graphs. In International Conference on Machine Learning. pages 41 16-411.. PMLR, 2020. 10
[20] Le Hou, Chen-Ping Yu, and Dimitris Samaras. Squared earth mover's distance-based loss for training deep neural networks. arXiv preprint arXiv: 161.:6911.,, 2016. 4
[21] Jun ln-Ting Hsich, Bingbin Liu, De-An Huang, Li F Fei-Fei, and Juan Carlos Niebles. Learning to decompose and disentangle representations for video prediction. Advances in neural information processing systems.
31, 2018.
11

---

[22] Yifei Hu and Ya Zhang. Graph contrastive learning with local and global mutual information maximization.
In 2020 The 8th International Conference on Information Technology: loT and Smart City, ICIT 2020,
page 74-78. New York, NY, USA. 2020, Association for Computing Machinery. 10
[23] Andrew Jaegle, Felix Gimeno, Andy Brock, Oriol Vinyals, Andrew Zisserman, and Joao Carreira. Perceiver:
General perception with iterative attention. In International conference on machine learning, pages 4651-
4664. PMLR, 2021. 7
[24] Tomas Jakab, Ankush Gupta, Hakan Bilen, and Andrea Vedaldi. Self-supervised learning of interpretable keypoints from unlabelled vidcos. In Proceedings of the IEEE/CVF Conference on Computer Vision and
Pattern Recognition, pages 8787-8797, 2020. 9
[25] Yinjun Jia, Shuaishuai Li, Xuan Guo, Bo Lei. Jungiang Hu, Xiao-Hong Xu, and Wei Zhang. Selfee,
self- self-supervised features extraction of animal behaviors. Elife, 11:e76218, 2022. 1
[26] Zachary V. Johnson, Manu Tej Sharma Arrojwala, Vineeth Aljapur, Tyrone Lee, Tucker J. Lancaster, Mark C. Lowder, Karen Gu, Joseph Stockert, Rachel I.. Lecesne, Jean M. Moorman, Jeffrey T. Streelman, and Patrick T. McGrath, Automated measurement of long-term hower hehaviors in lake malawi cichlids
using depth sensing and action recognition. Scientific Reports, 10(1):2057)) 2020. 10
[27] Mayank Kabra, Alice A Robie, Marta livera-Alba, Steven Branson, and Kristin Branson. Jaaba: interactive machine learning for automatic annotation of animal behavior. Nature methods, 10(1) (01))64-67, 2013. 1
[28] Ann Kennedy. The what, how, and why of naturalistic behavior. Current Opinion in Neurobiology,
74:102549, 2022. 1
[29] Diederik P Kingma, Max Welling, et al. An introduction to variational autoencoders. Foundations and Trendst in Machine Learning, 1 12(4):307-392 2019. 9
[30] The International Brain Laboratory. Valeria gguillon-Rodrigueee Dora Angelaki, Hannah Bayer, Niccolo Bonacchi, Matteo Carandini, Fanny Cazeltes, Gaelle Chapuis, Anne K Churchland, Yang Dan, Eric Dewitt, Mayo Faulkner, Hamish Forrest, Laura Haetzel, Michael Häusser, Sonja B Hofer, Fei Hu, Anup Khanal, Christopher Krasniak, Ines Laranjeira, Zachary F Mainen, Guido Meijer, Nathanicl J Miska, Thomas D Mrsic- Mrsic-Flogel, Masayoshi Murakami, Jean-Pain Noel, Alejandro Pan- Vazquez, Cyrille Rossant, Joshua Sanders, Karolina Socha, Rebecca Terry Anne E Urai, Hernando Vergara, Miles Wells, Christian J Wilson, llana Witten, Lauren E Wool, and Anthony M Zador. Standardized and reproducible measurement of decision-making in mice, ilfe, 10:e63713 1, may 2021. 10
[31] E. Levina and Bickel. The earth mover's 's distance is the mallows distance: some insights from statistics. In Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001, volume 2, pages 251-256 vol.2, 2001. 4
[32] Yingzhen Li and Stephan Mandt. Disentangled sequential autoencoder. arXiv preprint 103.v01803.0291,
2018. 9
[33] Jacky Liang. Viktor Makoviychuk, Ankur Handa, Nuttapong Chentanez, Miles Macklin, and Dieter Fox. Gpu-accelerated rohotic simulation for distributed reinforcement learning, 2018. 5
[34] Lijiang Long. Zachary V Johnson. Junyu Li, Tucker J Lancaster, Vineeth Aljapur, Jeffrey T Streelman, and Patrick T McGrath. Automatic classification of cichlid behaviors using 3d convolutional residual networks. Iscience, 23(10): 10159,,, 2020. 1
[35] Kevin Luxem, Falko Fuhrmann, Johannes Kürsch, Stefan Remy, and Pavol Bauer. Identifying behavioral structure from deep variational embeddings of animal motion. BioRxiv, 2020. 9
[36] Kevin Luxem, Jennifer J. Sun, Sean Bradley, Keerthi Krishnan, Talmo D. Pereira, Eric A. Yttri, Jan Zimmermann, and Mark Laubach. Open-source tools for behavioral video analysis: Setup, methods, and development, 2022. 9
[37] Viktor Makoviychuk, Lukasz Wawrzyniak, Yunrong Guo, Michelle Lu, Kier Storey, Miles Macklin, David Hoeller, Nikita Rudin, Arthur Allshire, Ankur Handa, et al. Isaac gym: High performance gpu-based physics simulation for robot learning. arXiv preprint arXiv:2108. 10470, 2021.
[38] Markus Marks, Qiuhan Jin, Oliver Sturman, Lukas von Ziegler, Sepp Kollmorgen, Wolfger von der Behrens, Valerio Mante, Johannes Bohacek, and Mehmet Fatih Yanik. beep-learning-hasee identification, tracking, pose estimation and hehaviour classification of interacting primates and mice in complex environments.
Nature Machine Intelligence, 4(4):331-33 --40, 2022. 1
[39] Jesse D. Marshall, Tianqing Li, Joshua H. Wu, and Timothy W. Dunn. Leaving flatland: Advances in 3d behavioral measurement. Current Opinion in Neurobiology, 73:102522, 2022. 9
[40] Alexander Mathis, Pranav Mamidanna, Kevin M. Cury, Taiga Abe, Venkatesh N. Murthy, Mackenzie Wey- gandt Mathis, and Matthias Bethge. Deeplabcut: markerless pose estimation of user-defined body parts
with deep learning. Nature Newroscience, 21:1281-1288 2018 9
[41] Reid Mcllroy- Young, Yu Wang. Siddhartha Sen, Jon Kleinberg, and Ashton Anderson. Detecting individual dacision-making style: Exploring behavioral stylometry in chess. Advances in Neural Information
Processing Systems, 34:24482-24497, 2021. 2
[42] Michael J. Mendelson, Mehdi Azabou, Suma Jacob, Nicola Grissom, David Darrow, Becket Ebitz, Alexander Herman, and Eva L. Dyer. Learning signatures of decision making from many individuals playing the same game. In 2023 11th International IEEE/EMBS Conference on Neural Engineering (NER), pages 1- 1-5, 1-5.2011 2023. 10
[43] Aditya Nair, Tomomi Karigo. Bin Yang, Scott W Linderman, David J Anderson, and Ann Kennedy. An approximate line attractor the hypothalamus that encodes an aggressive internal state. bioRxiv. 2022. [44] Daisuke Nizzumi, Daiki Takeuchi, Yasunori Ohishi, Noboru Harada, and Kunio Kashino. Byol for audio: Self- supervised leamnngg for purpose audio representation, 2021. 10
12

---

[45] Simon RO Nilsson, Nastacia L Goodwin, Jia Jie Choong, Sophia Hwang, Hayden R Wright, Zane C Norville, Xiaoyu Tong. Dayu Lin. Brandon S Bentzley, Neir Eshel, et al. Simple behavioral analysis (simba)-an open source toolkit for computer classification complex social behaviors in experimental animals, BioRxiv, 2020. 9
[46] Aaron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, and Koray Kavukcuoglu. Wavenet: A generative model for raw audio.
arXiv preprint 1rXiv:1609,0333 000990939999.. 2016. 4
[47] David F, Parks, Aidan M. Schneider, Yifan Xu, Samuel J. Brunwasser, Samuel Funderburk, Danilo Thurber, Tim Blanche, Eva L. Dyer, David Haussler, and Keith B. Hengen. A non-oscillatory, millisccondssaale embedding of brain state provides insight into behavior. bioRxiv, 2023, 10
[48] Talmo D Pereira, Diego Aldarondo, Lindsay Willmore, Mikhail Kislin, Samuel S-H Wang. Mala Murthy, and Joshua W Shaevitz. Fast animal pose estimation using deep neural networks. Nature methods,
16(1):117 125, 2019. 9
[49] Charles Qi, Hao Su, Kaichun Mo, and Lconidas Guibas. Pointnet:
Deep learning on point sets for 3d classification and segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 652-660) 2017. 7
[50] Adria Recasens, Pauline Luc, Jean--aatiistee Baptiste Alayrac, Luyu Wang. Ross Hemsley, Florian Strub, Corentin Tallec, Mateusz Malinowski, Viorica Patrauccan, Florent Altché, Michal Valko, bean-Bastien Grill, Aäron van den Oord, and Andrew Zisserman. Broaden your views for self-supervised video learning, 2021. 4, 5,
[51] Nikita Rudin. David Hoeller, Philipp Reist, and Marco Hutter. Learingg to walk in minutes using massively parallel deep reinforcement learning. In Conference on Robot Learning, pages 91-100. PMLR, 2022. 5 Cristina Segalin, Jalani Williams, Tomomi Karigo, May Hui, Moriel Zelikowsky, Jennifer J Sun, Pictro Perona, David J Anderson, and Ann Kennedy. The mouse action recognition system (mars) software pipeline for automated analysis of social behaviors in mice. Elife. 10:e63720, 2021. 1, 9. 16
[52]
[53] Shai Shalev-Shwards and Ambuj Tewari. Stochastic methods for 1 1 regularized loss minimization. In Proceedings of the 26th Annual International Conference Machine Learning, pages 929 936, 2009. 4 [54] Changhao Shi, Sivan Sehwartz. Shahar Levy, Shay Achvat, Maisan Abboud, Amir Ghanayim. Jackie Schiller, and Cal Mishne. Learning disentangled behavior embeddings. Advances in Neural Information Processing Systems, 34, 2021.
[55] Tom F Sterkenburg and Peter Grünwald. The no-free-lunch thcorems of supervised learning. Synthese, 19903)9997,11001,, 2021.
[56] Oliver Sturman. Lukas von Ziegler. Christa Schläppi. Furkan Akyol, Mattia Privitera, Daria Slominski. Christina Grimm, Lactitia Thieren, Valerio Zerbi, Benjamin Grewe, et al. Deep learning- based behavioral analysis reaches human accuracy and is capable of outperforming commercial solutions. Neuropsychophar- macology, 45(1 )): 1952, 2020. 1
[57] Jennifer J. Sun. Lili Karashchuk, Amil Dravid, Serim Ryou, Sonia Fereidooni, John C. Tuthill, Aggelos Katsaggelos, Bingni W. Brunton, Georgia Gkioxari, Ann Kennedy, Yisong Yue, and Pietro Perona. Bkind- 3d: Self-supervised 3d keypoint discovery from multi-view videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 9001-9010 June 2023. 10 [58] Jennifer Sun, Ann Kennedy, Eric Zhan, David J Anderson, Yisong Yue, and Pietro Perona. Task program- ming: Learning data efficient behavior representations. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 2876-2885, 2021. 2
[59] Jennifer J Sun, Ann Kennedy, Eric Zhan, David J Anderson, Yisong Yuc, and Pictro Perona. Task program- Learning data efficient behavior representations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2876-288- 2875, 2021. 7,9
ming:
[60]
Jennifer Sun, Serim Ryou, Roni Goldshmid, Brandon Weissbourd, John Dabiri, David Anderson, Ann Kennedy, Yisong Yue, and Pictro Perona. Self-supervised keypoint discovery in behavioral videos. arXiv
preprint XrXi Jennifer Sun, Andrew Ulmer, Dipam Chakraborty, Brian Geuther, Edward Hayes, Heng Jia, Vivek Kumar, Zachary Partridge, Alice Robic, Catherine H. Schretter, et al. The mabe22 henchmarks for representation learning of multi-
2021. 9
[61]
behavior. arXiv preprini 2.Xiv:2207.10553 2022. 2. 6. 7, 8. 10, 16
[62] Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding. In European conference
computer vision, pages 776-794, Springer, 2020. 4
[63] Ruben Villegas, Jimei Yang, Seunghoon Hong, Xunyu Lin, and Honglak Lee. Decomposing motion and content for natural video sequence prediction. arXiv preprint arXiv:1706. 080333 2017. 9
[64]
Chengde Wan, Thomas Probst, Lue Van Gool, and Angela Yao. Self-supervised 3d hand pose estimation through training by fitting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pottern
Recognition, pages 10853-10862, 2019. 9
[65] Alexander B Wiltschko, Matthew Johnson, Giuliano Iurilli, Ralph Peterson, Jesse M Katon, Stan Pashkovski, Victoria E Abraira, Ryan P Adams, and Sandeep Robert Datta. Mapping sub-second structure
in mouse behavior. Neuron, 88(6):1 121-11355,1.1 2015. 1,2
[66] Alexander B Wiltschko, Tatsuya Tsukahara, Ayman Zeine, Rockwell Anyoha, Winthrop F Gillis, Jeffrey E Markowitz, Ralph F. Peterson, Jesse Katon, Matthew J Johnson, and Sandeep Robert Datta. Revealing the structure of pharnacobehavioral space through motion sequencing. Nature neuroscience, 23(11):1433- 1443, 2020. 10
13

---

[67]
Anqi Wu, Estefany Kelly Buchanan, Matthew Whiteway. Michael Schartner, Guido Meijer, Jean-Pan Paul Noel. Erica Rodriguez. Claire Everett, Amy Norovich, Evan Schaffer, al. Deep graph pose: semi-supervised deep graphical model for improved animal pose tracking. Advances Neural Information Processing Systems, 33:6040-60555 2020. 9
[68]
Yuzhe Yang. Xin Liu, Jiang Wu, Silviu Borac, Dina Katabi, Ming-Zher Poh, Daniel McDuff. Simper: Simple self-supervised learning of periodic targets. arXiv preprint arXiv:22 2022.
[69] Zhihan Yue, Yujing Wang. Juanyong Duan, Tianmeng Yang, Congrui Huang, Yunhai Tong, and Bixiong
Xu. Ts2vec: Towards universal representation of time series. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 1980-8987, 2022. 4, 7, 10, 18
14

---

A Experimental details:
Simulated Quadrupeds
A.1 Data generation
Simulation details. We record total of 5182 trajectories. 2756 were generated for robots of type ANYmal B and 2426 sequences were generated for robots of type ANYmal C. These are quadruped robots, which means that they have four legs. Each leg has 3 degrees of freedoms hip, shank and thigh. The position and velocities these degrees of freedom for all 4 legs were recorded. This results in 24 features for each robot. Robots are generated while traversing an procedurally generated environment with different terrain types and traversal difficulty, as show in Figure 6. We only keep trajectories that correspond successful traversal.
Flat
Flat 2
HII
Ascunding stars Descunding stars
Paps aatuary dhull
Figure 6: Visualization of different simulated environments. (A) Screenshots from the simulator showing robot walking down some stairs, and view of the terrain landscape. (B) Visualization of the different terrain sections, characterized by terrain type and different levels of difficulty. Terrain are made more difficult to traverse by either making them more rough or have steeper slopes.
Tasks. To evaluate the representation quality of our model, we use multi-task probes that correspond to different long-term and short-term behavioral factors.
Robot type: the robot either be of type "ANYmal B" "ANYmal C". These robots have the same degrees of freedom and tracked joints but differ by their morphology. This is sequence-level task.
Linear velocity: the command of the robot is constant velocity vector. The amplitude of the velocity dictates how fast the robot is commanded to traverse the environment. A higher velocity would translate into more clumpsy and more risk-taking behavior. This is sequence-level task.
the environment is generated with multiple segments of five terrain types that are categorized as: flat surfaces, pits, hills, ascending and descending stairs. This is frame-level task.
Terrain type:
the slope of the surface the robot is walking on. This is a frame-level task. Terrain difficulty: the different terrain segments have different difficulty levels based on terrain roughness steepness of the surface. This a frame-level task.
Terrain slope:
Why this dataset. Simulation-based data collection enables access to information that is generally inaccessible or hard to acquire real-world setting. Unlike noisy measurements coming from the camera-based feature extractor in the case of the mouse dataset, physics engines do not suffer from the problem of noise. Instead, they provide accurate ground-truit truth information about the creature and the world state free of charge. Access to such information is at times critical for scrutinizing the capabilities of the learning algorithms.
A.2 Visualizing differences between short-term and long-term embeddings
In Figure 7, we visualize how the short-term and long-term embeddings evolve over time, for a single sample sequence. We note a clear difference in the smoothness in the two timescales. In the short-term embeddings, we note a clear block structure corresponding to different blocks of behavior that span few seconds, while in the long-term embeddings the representation is more stable over
15

---

time. This suggests that, as expected, the bootstrapping objectives are forming representations with different levels of granularity.
Short term embedding
Long term embedding
Eowil
Figure 7: Visualization the short- and long-term embeddings. We visualize for single sequence how the short thort-term long- term embeddings evolve over time.
B Experimental details: Mouse Triplet
B.1 Feature extraction
Each mouse in the arena is tracked using 12 anatomically defined keypoints. We process these keypoints to extract 36 different features characterizing each mouse individually, similar to (52). We separate the keypoints into two different areas, the head and the body, for each we extract different measures of displacement, that we express the frame of the mouse, i.e. these features are invariant to the pose of the mouse relative to the arena. These features include:
Head linear velocity vector that we express using polar coordinates.
Head angular velocity denoting the change in the heading direction in the arena.
Body linear velocity vector that we express using polar coordinates.
Body angular velocity denoting the change in the direction of the body with respect to the arena.
Angular and linear velocities of the fore paws and the hind paws.
Spine length change, depicting the expansion and contraction of the mouse's ' body. Angles formed the tail with respect the body.
We normalize all features before training. We also use cosine and sinus of the angles instead of the angles. During training, we did use any form of augmentation.
Noise in the data. Because of errors in pose estimation and tracking, there are sometimes errors in the tracking data, notably some identity swap issues (61). To address this, we simply zero out all of the corresponding features and flag the frame invalid. A binary feature is also add to the input features indicating whether or not the frame valid. When predicting future actions, we only compute the error over windows in which at least 80% of the frames are valid.
B.2 Difference between Histogram of Actions and previous objectives
Our novel objective consists predicting the future histogram actions instead of predicting the future sequence of actions. In Figure 8, show what the target is for sample from the MABe Mouse Triplet dataset. Note that the time dimension collapsed, blurring the exact unrolling of the future events, but preserving the set of values that these actions will sweep. Note that the loss (EMD) applied for each action feature.
We show that by relaxing our future prediction loss to HoA loss, we benefit in terms of the representations that are learned by the model, especially for sequence-level (global tasks). Thus, in many ways, we show that directly forecasting, which is what many previous approaches have used for representation learning, can actually lead to representations that capture less of the task structure. While the model doesn't predict future timesteps directly, we can visualize the histogram prediction for the model and ground truth (Figure 9).
B.3 Training details
Architecture. We use two TCNs (4). Each TCN is built using multiple residual blocks, each residual block is composed of two convolutional layers, and use PReI.U activation, dropout and weight normalization. All convolutions are dilated with a rate that increases after each residual
16

---

Previous work
Ours
Target:
Sequence of actions Loss: Mean Squared Error (MSE)
Target: Histogram of actions
Loss: Earth Mover's Distance (EMD)
m
0
0.6
166
00
100
0.0
0.0
0.
19
time
bins
Figure 8: Prediction target for sample of the MABe Mouse Triplet dataset.
block. The formula r' where is the index of the residual block. The first TCN is the short-term encoder, which uses 4 blocks with output sizes [64, 64, 32, 32] and dilation rate = 2. The second TCN is the long-term encoder, which uses 5 blocks with output size [64, 64, 64, 32, 32 and a dilation rate 4. The output of both encoders are concatenated to form 64d embedding. The predictor is a multi-layer perceptron (MLP) that has hidden layers.
Training. We train the model for 500 epochs using the Adam optimizer with learning rate of 10-3 and weight decay 4 10-5 decrease the learning rate to 10 after 100 epochs. We use a batch size of 96, and compute the future histogram action prediction error for each timestep t starting at 5 seconds after the start of each sequence, in order to allow the model to aggregate enough context. We set the learning rate of the predictors used for bootstrapping to be 10 times higher than the learning rate used for the rest of the weights.
Evaluation. During the development of the model (Figure 10), we test our model on the public test splits, and only look the performance on the private set after finishing any hyperparameter tuning. We repeat the training and evaluation 5 times and report the average performance
B.4 Additional ablations
In addition to those mentioned in the main text, we perform two additional ablations to BAMS (Table 5). In our first experiment, we removed the interaction loss from the model, meaning that the
Ground
pine
Pradiction
birs
timee
Ground Truth
bins
19
UKK
Lef
Prediction
time
Figure 9: Visualization of histograms of future actions, for two random action features. For timestamp I on the time axis, we show the 32 dimensional histogram of future actions, which is the target of prediction for BAMS.
17

---

linear layer
Frame-level
Sequence-level
freeze representation
Z
label
Zz
label
Figure 10: Linear evaluation protocol. The model is frozen, and for each task, a single linear layer is trained predict the corresponding labels.
dynamics of each mouse modeled completely independently from each other. The ablated model secs a small drop in performance, but continues to outperform all other methods on average.
B.5
BAMS in the inductive setting
The mouse triplet dataset (5336 sequences) has three different sets, a training set (1800 sequences), private test set and public test set. During training of the representation learning model, we can either pre-train on all of the available data (transductive setting) or on the training set only (inductive setting). During linear evaluation, the different linear layers are trained using labels from the training set and the performance reported on the public test set (during the challenge) and then on the private test set (to rank models).
We train BAMS in the inductive setting and report the performance in Table 5. We find that even when BAMS is trained with approximately one third of the data, the drop in performance is modest. More importantly, BAMS preserves its ranking compared other methods, and still achieves state-of-the-art performance.
Table 5: Linear readouts of mouse behavior: The best-performing models are those with low MSE scores and high F1-scores.
Sequence-leve
Model PCA
Day ())
Time ())
Smain
Lights
Approach
Chase
Close
Contact
Huddle
OFF
OKG
(N)
Watching
0009416
0.09445
31.60
54.65
0.86
0.14
49.27
37.57
12.71
0.21
0.60
0.53
6.65
TVAE
0.09442
T-BERT
0.09262
0.09276
78.63
1.80
0.87
70.22
30.24
0.51
1.10
1.12
17.27
TS2Vee
0.09300
0.094222
57.12
65.60
1.29
0.56
59.53
46.13
24.74
0.35
1.09
0.74
12.37
T-Perceiver
DA03222
0.09323
69.81
69.68
1.57
1.27
60.4
47.81
28.32
0.41
1.16
0.86
16.42
T-PTT
T-PointNet
D.09275
0.09320
66.01
67.15
2.56
4,57
70.68
$5.96
21.23
0.84
2.79
2.32
15.61
BAMS no interaction
0.09164
0.09154
83.47
71.23
2.55
2.03
63.63
50.97
31.15
0.58
1.47
1.37
15.10
BAMS inductive
0.09112
0.09132
83.44
70.39
2.62
1.40
65.95
$2.39
31.08
0.60
1.51
1.40
18.14
Table 6: TS2Vec Linear readouts of mouse behavior.
Sequence-Jeve subtooks
Frame-Ieuel zubtacks
Model
Day (1)
Time (1)
Strain
Lights
Approach
Chase
Close
Contact
Huddle
OMT
OIG
0/0
Watching
TS2Vec-
0.35
TS2Vee-T
0.09882
1.0252
45.82
46.69
0.72
0.14
43.19
34.93
9.38
0.186
0.38
0.38
05.31
TS
0.09846
1.01646
46.67
44.28
0.67
0.13
44.56
33.87
9.79
0.178
0.42
0.42
04.58
B.6
Notes on TS2Vec experiments
TS2Vec (69) employs two types of contrastive losses to learn representations. The first of these losses is an instance contrastive loss which contrasts a sequence with all other sequences in a batch which treated as negative examples, while two subsequences extracted from the same sequence are treated as positive examples. The second loss is temporal contrastive loss which acts along single time series. Temporal representations of nearby time points are taken as positive examples, while the rest of the time points within the same sequence are taken as negative examples. The results for the three versions of TS2vec, namely TS2Vec-I, which uses only instance contrastive loss, TS2Vec-T, which uses only temporal contrastive loss,and TS2Vec IT, which uses both instance and temporal contrastive losses, are listed Table Our TS2Vec experiments on the mouse dataset showed that using temporal contrastive loss resulted in worse performance across all tasks compared to only
18

---

using instance contrastive loss. For this reason, we only report results for TS2vec that only employs
instance contrastive loss.
We note that for both Ve2 and TS2Vec-IT, we ran into out-of-memory errors when creating instance-level or global contrast. Contrastive learning methods usually incur high computational costs, we find that our method, which doesn't I rely on negative examples, can scale better when working with longer sequences and larger datasets.
19


Reviews:
[{"review_id": "dV2yEekmFu", "reviewer": "Reviewer_6S6e", "summary": "The authors propose a multi-task representation learning model for animal behavior consisting of an action-prediction objective and a multi-scale architecture with separate short and long term dynamics. The authors use this model to achieve state of the art performance on the MABe2022 Challenge. Furthermore, they generate a dataset of varying maps for quadruped robots and use the proposed method for behavior predictions in these environments.", "questions": "To my understanding, the MABe challenge was held during a fixed timeline, and there may be many papers on this topic which do not treat the dataset as a benchmark. Could you explain if you think this is true, and could you also explain regarding the challenge time?", "limitations": "Limitations are discussed, but broader impacts are not. An insight to the authors view of broader impacts of connecting animal behavior models and robotics as they do in their work would be interesting to hear. Also, additional details regarding limitations should be discussed if there are specific robot morphologies you expect this approach to not work well for.", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "The paper is well-written with a clear motivation, organization, discussion of mathematical details and experimental results. Diagrams in the method section make the approach particularly clear.\n\nInteresting and novel methodological design with the distinct latent spaces for short and long term dynamics as well as the histogram of actions objective. Also, these methodological components are methodologically simple and increase model interpretability.\n\nState of the art performance on MABe2022 challenge for sequence level tasks and some frame level tasks.\n\nConnection between animal behavior modeling solution and robotics application.\n\nNew robot learning dataset of quadrupeds leveraging behavior model for which tasks will be released as a community benchmark.\n\nCode and implementation details are released, making the results easily reproducible.\n", "weaknesses": "While the connection between mouse and robot behavior is interesting, the connection should be made a bit more strongly. What would these behavior models do to aid robotics? This question should be explicitly addressed in the text.\n\nThere is a distinct pattern in the results where the frame level results are less competitive than the sequence level. An appropriate discussion and explanation in the text should be added for this performance difference.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your comments and questions! We provide point-by-point responses below.\n\n> 1. While the connection between mouse and robot behavior is interesting, the connection should be made a bit more strongly. What would these behavior models do to aid robotics? This question should be explicitly addressed in the text.\n\n**Reply:** In the realm of robotics, the approach for learning robot policies (controllers) from human or animal demonstrations is gaining traction. For instance, to learn complex tasks such as object manipulation or navigation, the data is often gathered by various human tele-operators [1]. Similarly, the animal behavior data can be leveraged as a reference for learning agile and more natural robot locomotion capabilities [2]. Understanding the unique tele-operator or animal traits, biases, and tendencies can help eliminate or harness them for improved robot learning. We plan to add a discussion on these connections in the Discussion section of the paper. \n\n[1] https://learning-from-play.github.io/\n\n[2] https://xbpeng.github.io/projects/Robotic_Imitation/index.html\n\n\n> 2. There is a distinct pattern in the results where the frame level results are less competitive than the sequence level. An appropriate discussion and explanation in the text should be added for this performance difference.\n\n**Reply:** Thanks for the suggestion. Many of the frame-level behaviors in the Mabe benchmark are social behaviors that involve pairs of mice. Because we don’t explicitly model features related to these social behaviors like other approaches (T-PointNet, T-BERT), BAMS doesn’t achieve the state-of-the-art but is still very competitive. At the same time, because we build representations of the long-term and sequence-level dynamics in a separate latent space, our approach really shines in many of the global tasks. We plan to add more discussion of this point in the revised paper.\n\n> 3. The MABe challenge was held during a fixed timeline, and there may be many papers on this topic which do not treat the dataset as a benchmark. Could you explain if you think this is true, and could you also explain regarding the challenge time?\n\n**Reply:** We believe that the most relevant works on animal behavior as well as self-supervised learning methods for sequential data, have been either evaluated by the original authors (like T-VAE [2]) or contributed by the community as part of the challenge (like T-BERT [1]). An extensive benchmark has been released alongside the MABe datasets [1], and we compare against all of the included methods. We benchmark additional self-supervised learning methods like TS2Vec [3], and we have added BYOL [4] in response to reviewer Uc6V. We do not think that the challenge window is a concern, we believe that the benchmark itself is more important than the challenge, because it presents challenging and well rounded tasks for evaluating animal behavior representation methods, which have otherwise been limited to testing on in-house datasets, with limited comparisons against other work.\n\n> 4. An insight to the authors view of broader impacts of connecting animal behavior models and robotics as they do in their work would be interesting to hear. \n\n**Reply:** As we advance towards more fine-grained and naturalistic long-term recordings of behavior in animals and humans, analyzing fine-grained behavior can enhance both robots and human-robot interfaces. In the medical field, these models could be used to develop a more naturally moving robotic controller for prosthetics or exoskeletons. The universality of the controller can be achieved from better understanding of long & short term subtleties of human behaviors, as well as rapidly adapted for the comfort and preference of the final user. Unsupervised data driven approaches can also be powerful for detecting and diagnosing the abnormalities of the complex robot/agent behaviors.\n\n> 5. Additional details regarding limitations should be discussed if there are specific robot morphologies you expect this approach to not work well for.\n\n**Reply:** Currently, our investigations only focus on joint-based robots, and do not have inherent limitations for more complex joint-based morphology designs. There are subfields of robotics that we have yet to explore such as wheeled, buoyancy or soft robotics. While there are no theoretical reasons to believe our approach wouldn’t suit, an investigation and empirical evaluation is needed. Furthermore, our robot experiments currently solely rely on proprioceptive information (robot’s perception of its own body) to extract the behavior models. The integration of exteroceptive sensing will expose the model to a richer information space, which could enable the model to further improve the behavior modeling by observing connections between the robots and its surroundings.\n\n---\n\n[1] Sun, Jennifer J., et al. \"MABe22: A Multi-Species Multi-Task Benchmark for Learned Representations of Behavior.\" ICML 2023. \n\n[2] Sun, Jennifer J., et al. \"Task programming: Learning data efficient behavior representations.\" CVPR 2021.\n\n[3] Yue, Zhihan, et al. \"Ts2vec: Towards universal representation of time series.\" AAAI 2022\n\n[4] Grill, Jean-Bastien, et al. \"Bootstrap your own latent-a new approach to self-supervised learning.\" NeurIPS 2023\n"}, {"review_id": "aXKdMxmPqD", "reviewer": "Reviewer_hofo", "summary": "The paper introduces a self-supervised (autoregressive) learning system that processes behavioral data on two fixed time scales via TCNs. Combined, the TCN output it then used to predict behavioral histograms over a fixed time window. The innovative histogram encoding encourages the learning of behavioral distribution rather than exact behavioral sequence patterns, thus making the system independent of exact temporal dynamics. Additionally, a bootstrapping technique is used to encourage prediction locality (slightly unclear how exactly, see below). The results are quite impressive seeing that the system is not trained to actually classify or distinguish behavioral patterns or agent properties. The system indeed reaches state-of-the-art performance outperforming all other approaches in a recent mouse behavior analysis benchmark. Another artificial robot behavioral benchmark also yields great performance. In both cases the authors show that the emergent latent hidden state representations z learn to encode short- and long-term behavioral characteristics.", "questions": "I had a hard time to understand the multi-scale bootstrapping methodology. I am not familiar with the term “bootstrapping” in this respect – please specify more accurately how it is implemented. It is also a pitty that the two L_r losses are not visualized in Figure 1. What does “positive views” mean here? How are the predictors q_s and q_l implemented. \n\nCan you run further ablations to prove the robustness of your approach? \n\nFor which tasks is your approach not suited?", "limitations": "Limitations are rather brief and do not really discuss the inducative biases / data assumptions implemented in the system. In fact, the only limitation that is mentioned is the lack of further latent state disentanglement, which I am not quite sure I understand the point here – you very clearly disentangle short- from longer-term behavioral pattern prediction (or rather behavioral histogram prediction). I would rather mentioned the other limitations, which I have mentioned above – including crucial system design choices and critical system parameters.\n\nPlease correct the references to tables.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "An autoregressive approach that learns to characterize behavioral patterns, solely by post-training a linear classifier given the emergent latent codes. \n\nSystem outperforms state-of-the-art systems on a recent benchmark without supervised training. \n\nLatent structures emerge that are very well-suited to characterize behavioral patterns (short-term dynamics) as well as more global behavioral tendencies (agent properties / time of day). They are – at least to a certain extent – directly interpretable. \n\nAn innovative histogram-based loss component that does most of the trick and should be applicable in other tasks and domains. (I wonder if you also attempted to use the more common KL divergence… does it make a difference?).", "weaknesses": "System does not seem to be generally applicable. It focuses on analyzing repetitive behavioral patterns and general, global, behavioral-influencing agent properties. It seems to me that the tasks indeed are a perfect fit for the presented system. \n\nIn the mouse task the distance prediction loss comes out of the blue. An ablation of this one should be conducted. Also the minimal statistics mean, max, min come out of the blue. What is their impact here? \n\nThe short- and long-term horizons are fixed and chosen by the system designer. I wonder how robust the system is when varying the two parameters in a reasonable range. \n\nThe system assumes separate patterns within the short-term and long-term behavioral dynamics and  the benchmarks offer exactly this. \n\nThe authors talk about “realistic robot behavior” – I would remove “realistic” here.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your comments and questions! We provide point-by-point responses below.\n\n> 1. System does not seem to be generally applicable. It focuses on analyzing repetitive behavioral patterns and general, global, behavioral-influencing agent properties. It seems to me that the tasks indeed are a perfect fit for the presented system.\n\n**Reply:** Thank you for the positive feedback that the tasks that we selected are a perfect fit. While the focus of our current work has been on animal behavior, we do believe that the histogram of actions loss and some aspects of our multi-scale loss should be more generally applicable. We hope to explore more applications of HoA in general time-series analysis in the future.\n\n> 2. In the mouse task the distance prediction loss comes out of the blue. An ablation of this one should be conducted. Also the minimal statistics mean, max, min come out of the blue. What is their impact here?\n\n**Reply:** Thanks for your questions. We agree that the introduction of the mouse distance prediction task and pooling could be better explained. We will revise the text to discuss the rationale and provide better context in a revised version.\n\nTo address your question about the impact of these different components, we conducted a number of ablations, which we report in Table 1 of the accompanying pdf. In our first experiment, we removed the interaction loss from the model, meaning that the dynamics of each mouse are modeled completely independently from each other. The ablated model sees a small drop in performance, but continues to outperform all other methods on average. We also ablate the different pooling functions and replace them with a single average pooling operation, where again we find that the drop in performance is minimal. Note that the motivation behind the use of average pooling is to get the average embedding of the mouse triplet, while the max and min help estimate the spread of the embeddings, capturing the difference between their states. We plan to include these ablations in a revised version. Thanks for your suggestions!\n\n> 3. The authors talk about “realistic robot behavior” – I would remove “realistic” here.\n\n**Reply:** We can make that change.\n\n > 4. “I had a hard time understanding the multi-scale bootstrapping methodology. I am not familiar with the term “bootstrapping” in this respect – please specify more accurately how it is implemented. It is also a pity that the two L_r losses are not visualized in Figure 1.” \n\n**Reply:** The idea of bootstrapping for contrastive learning was originally introduced in Bootstrap Your Own Latent (BYOL) [1]. BYOL learns an embedding of samples (images, in the original paper) that map augmentations of the same sample to similar points in the latent space, and thus needs to start with the notion of “positive views” or a definition of which points should be close in the latent space. In the application of BYOL to temporal data, a positive range of samples or window in time is defined over which all the nearby points in time are considered positive samples. In our model, we have two different windows over which we define positive samples: in the short term space, points within a short distance are mapped to nearby points in the latent space (according to Equation 3), and in the long-term space, all points that are within a larger window get mapped close to each other in their latent space. We will provide more context and details on the loss and related background on bootstrapping in a revised version. \n\n> 5. What does “positive views” mean here? How are the predictors q_s and q_l implemented?\n\n**Reply:** As described in the last response, positive views are defined based upon whether two points in time are within L samples from one another. Views are selected in this window uniformly at random during training. The predictors are small MLPs that aim to predict the representation of one view (called target view) from the other (called online view), we encourage the reviewer to refer to [1]. Both timescales have a separate projector and separate set of latents to build contrast over. We will add this discussion and details on the projectors in the revised paper.\n\n> 6. For which tasks is your approach not suited?\n\n**Reply:** BAMS is designed to build representations that can both capture frame-level dynamics as well as global trends in behavior that occur throughout a sequence. Thus, BAMS is not well suited to work with very short sequences that don’t provide sufficient long-term information. We will provide discussion on limitations and situations in which BAMS may not work well in the discussion section of the revised work. \n\n---\n\n[1] Grill, Jean-Bastien, et al. \"Bootstrap your own latent-a new approach to self-supervised learning.\" NeurIPS 2023\n"}, {"review_id": "F3VFM2dnOf", "reviewer": "Reviewer_Uc6V", "summary": "The authors propose a model called BAMS for unsupervised behavioral analysis. It combines two temporal convolutional neural networks with different timescales used to learn latents, as well as a loss to predicting a distribution of future actions. BAMS reaches strong performance on a novel, small-scale synthetic dataset as well as one of the MABE datasets.", "questions": "I wonder if the authors also ran BAMS on the flies in MABE?", "limitations": "In the abstract the authors state: “While some success has been found in building representations of animal behavior under constrained or simplified task-based conditions, many of these models cannot be applied to free and naturalistic settings where behavior becomes increasingly hard to model.” \n\nI think this sentence should be removed, as this paper also does not deal with naturalistic behavior.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1)\tPaper well written and easy to follow.\n2)\tSimple combination of known features (TCN, distribution matching, BYOL) to reach strong performance for unsupervised behavioral analysis on two datasets. \n3)\tAblation analysis is good, and gives insights into design choices.", "weaknesses": "1)\tWhile the overall pipeline is novel, each component is rather well known and thus the overall novelty is somewhat limited. \n2)\tBAMS is only validated on one of two MABE datasets (the benchmark also contains flies), as well as one novel small synthetic dataset. While the results on the mouse MABE dataset are strong, how the method is adapted to multiple mice is rather ad-hoc. Thus, ideally it should be tested on single-animal behavioral datasets. \n3)\tFor MABE the baselines are rather strong (as they are from the MABE competition), but they are mostly self-supervised methods using transformers (GPT, BERT, etc. style). It’s nice that the BAMS with TCNs can actually beat those methods for some of the behaviors – However, I think it would be great if a vanilla BYOL method could also be run by the authors to assess the novelty of BAMS.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your comments and questions! We provide point-by-point responses below.\n\n> 1. “While the overall pipeline is novel, each component is rather well known and thus the overall novelty is somewhat limited.” \n\n**Reply:** The method consists of two core components: (1) a HoA predictive loss, (2) an architecture and contrastive loss to encourage multi-timescale disentanglement. While it is true that bootstrapping and temporal contrastive learning have been applied to sequential data (TS2Vec for example), HoA is a novel objective and we have not seen this idea applied to time series modeling. Through a series of ablations, we show how each component contributes to the overall accuracy, and these experiments point to the significance of our novel HoA loss formulation in providing critical boosts in performance.\n\n> 2. “While the results on the mouse MABE dataset are strong, how the method is adapted to multiple mice is rather ad-hoc. Thus, ideally it should be tested on single-animal behavioral datasets.”\n\n**Reply:** \n- Rationale behind selecting MABe:  One key innovation and motivation behind BAMS is its ability to build representations of behavior that allow for short-term (frame-level) classification as well as more global (sequence-level) classification. The Mabe dataset and benchmark provided a perfect fit for evaluation of diverse behaviors and behavioral traits that span different timescales. It is unique in that it contains both frame-level and sequence-level labels for behavior, and has sequences that are long enough to support separation of factors across timescales. For this reason, we found MABe to be a good fit for our analysis and few other datasets to meet the multitask criteria.\n- Ablations of the multi-mouse features: Because other methods we compare against use multi-mouse features, we use a simple interaction loss to capture some of the triplet-level behavior. To address your concern, we completely remove this component, and model each mouse independently: BAMS is trained on single animals in this case.   During evaluation we still need to use pooling to compare against other methods. We report the results in Table 1 in the accompanying pdf. We show that the model still ranks first despite not modeling multi-animal interactions.  We appreciate your comments and feel that with this ablation, the paper is stronger and the results are even more compelling. \n\n\n> 3. “For MABE the baselines are rather strong (as they are from the MABE competition), but they are mostly self-supervised methods using transformers (GPT, BERT, etc. style). [...] I think it would be great if a vanilla BYOL method could also be run by the authors to assess the novelty of BAMS.”\n\n**Reply:** Thanks for your suggestion. In addition to transformer-based architectures that are currently well ranked on the leaderboard, we also included a more recent time-series contrastive learning model, TS2Vec, as a baseline for Mabe, which uses both frame-level and sequence-level contrast. We thought that this was a strong contrastive baseline given that it was designed for time series data and also has an inherent mechanism for building local and global contrast. To address your concern, we ran a temporal version of BYOL [1] using a TCN backbone and  a single positivity window of around a minute.\n\n> 4. Question: I wonder if the authors tested the method on flies?\n\n**Reply:**   We haven’t yet tested on the flies dataset but think this would be an interesting dataset for future investigations.\n\n---\n\n[1] Grill, Jean-Bastien, et al. \"Bootstrap your own latent-a new approach to self-supervised learning.\" NeurIPS 2023\n"}, {"review_id": "qpwv8BqTkB", "reviewer": "Reviewer_7X95", "summary": "The authors propose BAMS, an architecture for learning agent behaviors by mapping observations to actions. BAMS utilizes two Temporal Convolutional Networks (TCNs) operating at different time scales, enabling modeling of both short-term and long-term trends in data through varying dilation rates. The short-term and long-term representations obtained from these networks are further enhanced using self-supervised bootstrapping objectives inspired by previous research. In addition, the authors introduce a novel decoder that predicts histograms to capture distributions instead of discrete predictions during training, which they believe aids in long-term forecasting. For deployment, a linear layer is fitted on the learned representations, followed by local training on the linear layer prior to evaluation. Experimental results on mobile robotics and animal behavior datasets demonstrate promising predictions compared to baseline methods.", "questions": "Applicability to General Time Series/Dynamics Modelling: Do you believe that your method is applicable to a wide range of time series/dynamics modelling tasks, thus increasing its relevance to the broader machine learning community? It would be valuable to understand the potential generalizability and versatility of your method beyond the specific datasets and scenarios discussed in the paper.\n\nDifferentiation from Existing Multi-Time Scale Modelling Literature [2][3]: How does your method stand out when compared to the existing literature on multi-time scale modelling? It would be insightful to highlight the unique contributions or advantages that set your approach apart from other techniques designed for similar purposes.\n\nLong-Horizon Predictions and Forecasting: How accurately can BAMS make long-horizon or multistep-ahead predictions/forecasting, especially for the robotic dataset? This information was not very clear / lacking in the paper. If long-horizon predictions are feasible, what is the reliable prediction horizon (e.g., number of seconds into the future) that your method achieves on both datasets without performance degradation? Did you make a plot of the predictions vs ground truths for both datasets, what did this plot convey ?? This information would provide valuable insights into the method's capability for long-term forecasting.\n\n**References**\n\n[2] Chung, Junyoung, Sungjin Ahn, and Yoshua Bengio. \"Hierarchical multiscale recurrent neural networks.\" arXiv preprint arXiv:1609.01704 (2016).\n\n[3] Mujika, Asier, Florian Meier, and Angelika Steger. \"Fast-slow recurrent neural networks.\" Advances in Neural Information Processing Systems 30 (2017).", "limitations": "The authors were upfront about some limitations of the work. There was no discussion on broader impacts.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "These are the strengths of the paper in my opinion:\n\nClear and Coherent Writing: The paper is well-written and maintains a clear and organized structure, ensuring a pleasant reading experience. \n\nRelevance to the Research Community: The problem of learning representations for multi-time scale behaviour addressed in the paper is very relevant to the research community. \n\nNovel Approach Of predicting Histograms: The paper introduces an interesting innovative approach by utilizing histograms for predictions to capture distributions which are accompanied by the introduction of compatible loss functions. I think this idea would help with capturing multi-modal behaviours/predictions in a better manner for future applications, though further analysis needs to be done in this regard.\n\nStrong Empirical Performance: The proposed method demonstrates strong empirical performance on two challenging benchmarks. By achieving promising results in these real-world scenarios, the paper highlights the effectiveness and potential practical application of the approach.", "weaknesses": "The weakness of the paper under review is listed below:\n\nLack of Well-Founded Design Principles: The design choices in the proposed method appear to be based on arbitrary experimentation rather than being guided by well-established principles. This absence of a solid theoretical foundation raises concerns about the method's reliability and may compromise its technical rigour/novelty. \n\nLimited Applicability to General Cases: I doubt the method's suitability for the general case of learning and predicting long-term behaviours in time series or sequential data, which is more relevant to the machine learning community. The reliance on specific datasets and the lack of comprehensive exploration across diverse scenarios raises questions about the method's ability to generalize its performance beyond the limited contexts it has been tested on. More empirical evidence is needed to establish its applicability in broader settings.\n\nInsufficient Discussion of Related Works: The paper fails to adequately address and discuss related works on hierarchical neural network architectures [2][3] that are specifically designed to model multi-time scale behaviour. This omission prevents a comprehensive understanding of the method's positioning and effectiveness compared to existing \"generic\" approaches. It would be valuable to include discussions and comparisons with these alternative methods to provide a more complete perspective.\n\nCumbersome Use of Multiple Loss Functions and Architecture Modification: The method's reliance on multiple loss functions and the need to modify the architecture during deployment by introducing and training a linear decoder may be perceived as cumbersome and lacking elegance. This complexity raises concerns about the practicality and efficiency of the proposed approach, as simpler and more streamlined alternatives exist. A more streamlined and efficient design would enhance the method's overall appeal.\n\n**References**\n\n[2] Chung, Junyoung, Sungjin Ahn, and Yoshua Bengio. \"Hierarchical multiscale recurrent neural networks.\" arXiv preprint arXiv:1609.01704 (2016).\n\n[3] Mujika, Asier, Florian Meier, and Angelika Steger. \"Fast-slow recurrent neural networks.\" Advances in Neural Information Processing Systems 30 (2017).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your comments and questions! We provide point-by-point responses below.\n\n> 1. “The design choices in the proposed method appear to be based on arbitrary experimentation rather than being guided by well-established principles. This absence of a solid theoretical foundation raises concerns about the method's reliability and may compromise its technical rigour/novelty.”\n\n**Reply:** We believe that our design choices were built from insights about the unique challenges of animal behavior analysis. We have extensively experimented with different representation learning objectives, and made critical observations about what can be improved in the current landscape of behavior representation learning methods. In particular we identify limitations of time series future prediction, as well as, the importance of multi-timescales modeling in multi-task settings. To tackle both of these challenges, we introduce a histogram prediction task for “relaxing” multi-step prediction, and (2) a multi-scale disentanglement loss to help separate the latents. We study the contribution of these different components through various ablations (Table 3), and find that both objectives are critical for state-of-the-art performance.\n\n> 2. “The method's reliance on multiple loss functions and the need to modify the architecture during deployment by introducing and training a linear decoder may be perceived as cumbersome and lacking elegance. [...] A more streamlined and efficient design would enhance the method's overall appeal.\n\n**Reply:** Throughout the paper, we utilize standard procedures commonly used in self-supervised learning [1,2], where we train the model without any labels and then use a linear decoder to evaluate the separability of the feature space from only a few labeled instances. Indeed, this is the evaluation strategy that is used in MABe, and all of the models are trained without labels and then a post-eval strategy is used to rank the methods in the competition and in the benchmark. \n\nPlease note that Reviewer hofo notes that our post-hoc evaluation is a strength, and that BAMS “learns to characterize behavioral patterns, solely by post-training a linear classifier given the emergent latent codes” (hofo).” With this clarification over our evaluation procedure, we hope that you consider raising your score.\n\n> 3. “It would be valuable to understand the potential generalizability and versatility of your method beyond the specific datasets and scenarios discussed in the paper.”\n\n**Reply:** Analysis of behavior is becoming an increasingly important task in neuroscience, as well as advertising, marketing, and robotics. While we have not yet studied the application of BAMS in other time-series domains, we believe that many aspects of our approach could be more applied to a wide range of different datasets. In particular, the idea of relaxing the MSE or next-step prediction task to a HoA task could also be applied in other architectures and in different data domains. We think this is an exciting direction for future work.\n\n> 4. “Long-Horizon Predictions and Forecasting: How accurately can BAMS make long-horizon or multistep-ahead predictions/forecasting, especially for the robotic dataset? This information was not very clear / lacking in the paper. [...] Did you make a plot of the predictions vs ground truths for both datasets?.”\n\n**Reply:** While time series forecasting is an important task in general, in this work we are challenging the use of such an approach in cases where the downstream task is not forecasting. The focus of the paper is on representation learning and thus we use SSL and representation learning benchmarks and datasets rather than forecasting. We show that by relaxing our future prediction loss to a HoA loss, we benefit in terms of the representations that are learned by the model, especially for sequence-level (global tasks). Thus, in many ways, we show that directly forecasting, which is what many previous approaches have used for representation learning, can actually lead to representations that capture less of the task structure.\nWhile the model doesn’t predict future timesteps directly, we can visualize the histogram prediction for the model and ground truth (see the PDF uploaded to OpenReview). \n\n> 5. “The paper fails to adequately address and discuss related works on hierarchical neural network architectures [2][3] that are specifically designed to model multi-time scale behaviour”. \n\n**Reply:** In our discussion of related work, we focused more on the recent work in behavior modeling, and in temporal contrastive learning and bootstrapping which has been explored in recent years like for video [3] as an example, as we believed these papers to be most relevant to our current work and innovations.  In our revised paper, we plan to add more discussion of other architectures for multiscale sequence modeling and will include the references provided by the reviewer. Thank you for your suggestions! \n\n---\n\n[1] Chen, Ting, et al. \"A simple framework for contrastive learning of visual representations.\" ICML, 2020.\n\n[2] Sun, Jennifer J., et al. \"MABe22: A Multi-Species Multi-Task Benchmark for Learned Representations of Behavior.\" ICML 2023. \n\n[3] Recasens, Adria, et al. \"Broaden your views for self-supervised video learning.\" CVPR, 2021\n"}]

Author rebuttal:
We would like to thank all of the reviewers for their thoughtful feedback and helpful suggestions! We are delighted to hear that all of the reviewers appreciated the work and motivation, and were impressed with the fact that BAMS achieves state-of-the-art performance on a complex realworld multitask benchmark.

Strengths highlighted by the reviewers:
- **Innovations of new distributional loss for forward prediction:**  (7X95) “an interesting innovative approach by utilizing histograms for predictions” (hofo) “An innovative histogram-based loss component”
Method and approach:  (6S6e) “Interesting and novel methodological design with the distinct latent spaces for short and long term dynamics as well as the histogram of actions objective.” (hofo) “learns to characterize behavioral patterns, solely by post-training a linear classifier given the emergent latent codes” (7X95) “very relevant to the research community” (Uc6v) “Ablation analysis is good, and gives insights into design choices.”
- **Writing and presentation:** (7X95) “The paper is well-written and maintains a clear and organized structure, ensuring a pleasant reading experience.”  (Uc6v) “Paper well written and easy to follow.” (6S6e) “The paper is well-written with a clear motivation, [...] Diagrams in the method section make the approach particularly clear.”
- **State-of-the-art performance on a challenging real world benchmark:**  (6S6e) “State of the art performance on MABe2022 challenge” (7X95) “the paper highlights the effectiveness and potential practical application of the approach.”

**Main revisions and new experiments:**

To address reviewer suggestions and concerns, we ran three sets of experiments: 
- A new BYOL baseline to address Reviewer Uc6V’s comment regarding the inclusion of additional contrastive learning baselines for Table 2,
- Ablations of the multi-mouse interaction loss and pooling operations as suggested by Uc6V and hofo, to address questions about the impact of these components on final performance, 

Meta-review: This paper present a new approach to multi-scale modeling of dynamical systems (i.e., time-series).
This is a very important topic, and this contribution is timely and likely to be of interest to a wide audience.

All reviewers agree that the approach proposed in novel and interesting.
The experimental results are thorough and include a diverse set of datasets.
Decision: Accept (spotlight)