SlowGuess/NeurIPS_2023_OCR huggingface dataset:
- Head linear velocity vector that we express using polar coordinates.
- Head angular velocity denoting the change in the heading direction in the arena.
- Body linear velocity vector that we express using polar coordinates.
- Body angular velocity denoting the change in the direction of the body with respect to the arena.
- Angular and linear velocities of the fore paws and the hind paws.
- Spine length change, depicting the expansion and contraction of the mouse's body.
- Angles formed by the tail with respect to the body.
We normalize all features before training. We also use cosine and sinus of the angles instead of the angles. During training, we did not use any form of augmentation.
Noise in the data. Because of errors in pose estimation and tracking, there are sometimes errors in the tracking data, notably some identity swap issues (61). To address this, we simply zero out all of the corresponding features and flag the frame as invalid. A binary feature is also add to the input features indicating whether or not the frame is valid. When predicting future actions, we only compute the error over windows in which at least  $80\%$  of the frames are valid.
B.2 Difference between Histogram of Actions and previous objectives
Our novel objective consists in predicting the future histogram of actions instead of predicting the future sequence of actions. In Figure 8, we show what the target is for a sample from the MABe Mouse Triplet dataset. Note that the time dimension is collapsed, blurring the exact unrolling of the future events, but preserving the set of values that these actions will sweep. Note that the loss (EMD) is applied for each action feature.
We show that by relaxing our future prediction loss to a HoA loss, we benefit in terms of the representations that are learned by the model, especially for sequence-level (global tasks). Thus, in many ways, we show that directly forecasting, which is what many previous approaches have used for representation learning, can actually lead to representations that capture less of the task structure. While the model doesn't predict future timesteps directly, we can visualize the histogram prediction for the model and ground truth (Figure 9).
B.3 Training details
Architecture. We use two TCNs (4). Each TCN is built using multiple residual blocks, each residual block is composed of two convolutional layers, and use PReLU activation, dropout and weight normalization. All convolutions are dilated with a rate  $r$ , that increases after each residual
Figure 8: Prediction target for a sample of the MABe Mouse Triplet dataset.
block. The formula is  $r^i$  where  $i$  is the index of the residual block. The first TCN is the short-term encoder, which uses 4 blocks with output sizes [64, 64, 32, 32] and a dilation rate  $r = 2$ . The second TCN is the long-term encoder, which uses 5 blocks with output size [64, 64, 64, 32, 32] and a dilation rate  $r = 4$ . The output of both encoders are concatenated to form a 64d embedding. The predictor is a multi-layer perceptron (MLP) that has 4 hidden layers.
Training. We train the model for 500 epochs using the Adam optimizer with a learning rate of  $10^{-3}$  and weight decay  $4 \cdot 10^{-5}$ , we decrease the learning rate to  $10^{-4}$  after 100 epochs. We use a batch size of 96, and compute the future histogram of action prediction error for each timestep  $t$  starting at 5 seconds after the start of each sequence, in order to allow the model to aggregate enough context. We set the learning rate of the predictors used for bootstrapping to be 10 times higher than the learning rate used for the rest of the weights.
Evaluation. During the development of the model (Figure 10), we test our model on the public test splits, and only look at the performance on the private set after finishing any hyperparameter tuning. We repeat the training and evaluation 5 times and report the average performance
B.4 Additional ablations
In addition to those mentioned in the main text, we perform two additional ablations to BAMS (Table 5). In our first experiment, we removed the interaction loss from the model, meaning that the
Figure 9: Visualization of histograms of future actions, for two random action features. For a timestamp  $t$  on the time axis, we show the 32 dimensional histogram of future actions, which is the target of prediction for BAMS.
freeze representation
Figure 10: Linear evaluation protocol. The model is frozen, and for each task, a single linear layer is trained to predict the corresponding labels.
dynamics of each mouse are modeled completely independently from each other. The ablated model sees a small drop in performance, but continues to outperform all other methods on average.
B.5 BAMS in the inductive setting
The mouse triplet dataset (5336 sequences) has three different sets, a training set (1800 sequences), a private test set and a public test set. During training of the representation learning model, we can either pre-train on all of the available data (transductive setting) or on the training set only (inductive setting). During linear evaluation, the different linear layers are trained using labels from the training set and the performance is reported on the public test set (during the challenge) and then on the private test set (to rank models).
We train BAMS in the inductive setting and report the performance in Table 5. We find that even when BAMS is trained with approximately one third of the data, the drop in performance is modest. More importantly, BAMS preserves its ranking compared to other methods, and still achieves state-of-the-art performance.
Table 5: Linear readouts of mouse behavior. The best-performing models are those with low MSE scores and high F1-scores.
Model	Sequence-level subtasks	Frame-level subtasks
Day (↓)	Time (↓)	Strain	Lights	Approach	Chase	Close	Contact	Huddle	O/E	O/G	O/O	Watching
PCA	0.09416	0.09445	51.60	54.65	0.86	0.14	49.27	37.87	12.71	0.21	0.60	0.53	6.65
TVAE	0.09403	0.09442	52.98	56.80	1.07	0.45	59.33	44.77	21.96	0.27	0.83	0.62	10.20
T-BERT	0.09262	0.09276	78.63	68.84	1.80	0.87	70.22	55.84	30.24	0.51	1.40	1.12	17.27
TS2Vec	0.09380	0.09422	57.12	65.60	1.29	0.66	59.53	46.13	24.74	0.35	1.09	0.74	12.37
T-Perceiver	0.09322	0.09323	69.81	69.68	1.57	1.27	60.84	47.81	28.32	0.41	1.16	0.86	16.42
T-GPT	0.09269	0.09384	64.45	65.39	1.73	0.64	69.05	55.78	23.80	0.46	1.12	1.05	17.86
T-PointNet	0.09275	0.09320	66.01	67.15	2.56	4.57	70.68	55.96	21.23	0.84	2.79	2.32	15.61
BAMS - no interaction	0.09164	0.09154	83.47	71.23	2.55	2.03	63.63	50.97	31.15	0.58	1.47	1.37	15.10
BAMS - inductive	0.09112	0.09132	83.44	70.39	2.62	1.40	65.98	52.39	31.08	0.60	1.54	1.40	18.14
BAMS - transductive	0.09094	0.08989	88.23	72.00	2.74	1.89	67.22	53.43	31.43	0.59	1.61	1.57	18.15
Table 6: TS2Vec Linear readouts of mouse behavior.
Model	Sequence-level subtasks	Frame-level subtasks
Day (↓)	Time (↓)	Strain	Lights	Approach	Chase	Close	Contact	Huddle	O/E	O/G	O/O	Watching
TS2Vec-I	0.09380	0.09422	57.12	65.60	1.29	0.66	59.53	46.13	24.74	0.35	1.09	0.74	12.37
TS2Vec-T	0.09882	1.0252	45.82	46.69	0.72	0.14	45.19	34.93	9.38	0.186	0.38	0.38	05.31
TS2Vec-IT	0.09846	1.01646	46.67	44.28	0.67	0.13	44.56	33.87	9.79	0.178	0.42	0.42	04.58
B.6 Notes on TS2Vec experiments
TS2Vec (69) employs two types of contrastive losses to learn representations. The first of these losses is an instance contrastive loss which contrasts a sequence with all other sequences in a batch which are treated as negative examples, while two subsequences extracted from the same sequence are treated as positive examples. The second loss is a temporal contrastive loss which acts along a single time series. Temporal representations of nearby time points are taken as positive examples, while the rest of the time points within the same sequence are taken as negative examples. The results for the three versions of TS2vec, namely TS2Vec-I, which uses only instance contrastive loss, TS2Vec-T, which uses only temporal contrastive loss, and TS2Vec IT, which uses both instance and temporal contrastive losses, are listed in Table 6. Our TS2Vec experiments on the mouse dataset showed that using temporal contrastive loss resulted in worse performance across all tasks as compared to only
using instance contrastive loss. For this reason, we only report results for TS2vec that only employs instance contrastive loss.
We note that for both TS2Vec and TS2Vec-IT, we ran into out-of-memory errors when creating instance-level or global contrast. Contrastive learning methods usually incur high computational costs, we find that our method, which doesn't rely on negative examples, can scale better when working with longer sequences and larger datasets.