Animal behavioral analysis and neural encoding with transformer-based self-supervised pretraining
Authors: Yanchen Wang
Han Yu
Ari Blau
Yizi Zhang
International Brain Laboratory
Liam Paninski
Cole Lincoln Hurwitz
Matthew R Whiteway
Venue: ICLR 2026 Poster (2026)
Keywords: behavior analysis
neuroscience
neural encoding
electrophysiology
Neuropixels
pose estimation
action segmentation
Primary area: applications to neuroscience & cognitive science

Abstract: The brain can only be fully understood through the lens of the behavior it generates--a guiding principle in modern neuroscience research that nevertheless presents significant technical challenges. Many studies capture behavior with cameras, but video analysis approaches typically rely on specialized models requiring extensive labeled data. We address this limitation with BEAST (BEhavioral Analysis via Self-supervised pretraining of Transformers),  a novel and scalable framework that pretrains experiment-specific vision transformers for diverse neuro-behavior analyses. BEAST combines masked autoencoding with temporal contrastive learning to effectively leverage unlabeled video data. Through comprehensive evaluation across multiple species, we demonstrate improved performance in three critical neuro-behavioral tasks: extracting behavioral features that correlate with neural activity, and pose estimation and action segmentation in both the single- and multi-animal settings. Our method establishes a powerful and versatile backbone model that accelerates behavioral analysis in scenarios where labeled data remains scarce.

Published as a conference paper at ICLR 2026
ANIMAL BEHAVIORAL ANALYSIS AND NEURAL ENCOD- ING WITH TRANSFORMER-BE -BSED PRETRAINING
Yanchen Wang**, Han Yu Ari Blau', Yizi Zhang', International Brain Laboratory, Liam Paninski Cole Hurwitz & Matthew R. Whiteway?
CColumbia University, New York, NY USA "Equal contribution
1yw4503,hy2562, bsb2144, y24123, 1mp2107, ch3676, mm3323 (ea3322))ccllllll @columbie a, edu
ABSTRACT
The brain can only fully understood through the lens of the behavior it generates- a guiding principle in modern neuroscience research that nevertheless presents significant technical challenges. Many studies capture behavior with cameras, but video analysis approaches typically rely on specialized models requiring extensive labeled data. We address this limitation with BEAST (BEhavioral Analysis via Self-supervised pretraining of Transformers), novel and scalable framework that pretrains experiment-sper specific vision transformers for diverse neuro-behavioe anal- yses. BEAST combines masked autoencoding with temporal contrastive learning to effectively leverage unlabeled video data. Through comprehensive evaluation across multiple species, demonstrate improved performance in three critical neuro-behavioral tasks: extracting behavioral features that correlate with neural activity, and pose estimation and action segmentation in both the single- and multi- animal settings. Our method establishes a powerful and versatile backbone model that accelerates behavioral analysis in scenarios where labeled data remains scarce.
INTRODUCTION
Understanding the relationship between brain and behavior is fundamental challenge across wide range of medical and scientific disciplines (Krakauer al. 2017 Datta et al. 2019). Precise methods for extracting meaningful information from behavioral videos are essential for advancing these fields (Pereira et al., 2020). self-supervised learning has revolutionized image and video understanding through large-scale foundation models (Chen et al. 2220 Caron et al. [02 He et al. 2022), offering powerful tools that are beginning to transform scientific analyses (Huang et al. 2023a Lastufka et al. 2024). However, these models have yet to be effectively translated specialized domains like animal behavior analysis, creating a significant opportunity for methods that bridge cutting-edge machine learning with the specific demands of neuroscience and behavioral research.
Animal behavior videos present unique characteristics and challenges distinct from general video understanding. Controlled experiments generate large quantities of videos with static backgrounds and consistent camera angles, where the primary variation arises from animal movements and interactions. These videos enable numerous downstream analyses, and here we focus three fundamentally different applications that collectively address large proportion of behavioral neuroscience use cases: (1) neural activity prediction (or "neural encoding")). which requires extracting behavioral features that correlate with simultancously recorded brain activity (Datta et al.) 2019 Pereira et al.) [2020] Urai
al. 2022): ) (2) pose estimation, which tracks specific anatomical landmarks for quantitative analysis of movement patterns (Mathis and Mathis, 2020 |Pereira et al. 2020); and (3) action segmentation, which classifies distinct behavioral states like grooming, rearing or social interactions on every frame (Datta et al. 2019: Pereira et al. 2020). Each task demands different representations of the same underlying behavioral data, and current approaches typically require task-specific models and extensive labeled datasets (von Ziegler et al. (021). Furthermore, most approaches fail to leverage the vast amounts unlabeled data generated by behavior experiments, a significant untapped resource that, if harnessed properly, could substantially improve performance these downstream tasks.

---

Published as a conference paper at ICLR 2026
We address these challenges through novel self-vupe pretraining framework for raw videos that produces a robust backbone for multiple downstream neuro- behavioral tasks. BEAST (BEhavioral Analysis via Self-supervised pretraining of Transformers) leverages the unique properties of exper- imental videos by combining masked autoencoding (He et al.) 2022) to capture rich frame-level appearance information with temporal contrastive learning (Hyvarinen and Morioka 2016) to model behavioral dynamics. We introduce a novel frame sampling strategy for the contrastive loss, designed to focus on learning representations of animal behavior against statie backgrounds. BEAST trains on videos from a single experimental setup, creating tailored, versatile models that can be fine-tuned for multiple analytical needs specific to that experimental context. We demonstrate the value of this approach through comprehensive evaluation on three downstream tasks: (1) neural encoding in three mouse datasets; (2) pose estimation across four datasets spanning two species and single- and multi-view setups; and (3) action segmentation in both single- and multi-animal setups. BEAST achieves competitive or superior performance for the neural encoding and action segmentation tasks- outperforming baselines such DINOv2 (Oquab et al. (023)-while eliminating the need for the pose estimation step typically required by existing methods, dramatically reducing manual labeling effort. At the same time, pose estimation remains valuable for producing interpretable features critical to understanding movement dynamics, and BEAST enables significantly improved pose estimation for given labeling budget. These results establish BEAST as a simple yet powerful foundation that can accelerate behavioral understanding across disciplines where fine- the-grained analysis is essential.
2 RELATED WORK
Neural encoding models. Neural encoding measures how observable signals predict neural activity, and provide a quantitative framework for interrogating neural representations. Earlier approaches applied generalized linear models to single neurons using controlled stimuli such visual or auditory inputs (Paninski] 2004; Truccolo et al.) 2005] Pillow et al.) 2008; McFarland et al. (001)). More
recently, deep learning methods have shown great promise in predicting neural population responses to sensory stimuli, including visual (Yamins et al., 2014 Sccrimmp et al. 2018 Wang et al. (002a)).
auditory (Kell et al. 2018 [i et al., 2023), and tactile (Zhuang et al.) (2.7] inputs. The widespread
adoption of video monitoring during experiments has demonstrated that vidco-based behavioral
covariates explain significant neural variability in both spontaneous (Stringer et al.
2019 Syeda et al.
2024) and task-driver behaviors (Musall et al. [2019 IBL et al. [025a] Wang et al. 2025b Chen et al.
2024 Zhang et al.] 2025b). For example, Musall et al. (2019) showed that uninstructed movements
explain substantial fraction of cortical neural variance, and the International Brain Lab leveraged large-scale, region-resolved encoding analyses to chart the distribution of task-related information
across the brain (IBL et al. 2025a). However, extracting rich spatiotemporal information from video remains challenging. Most studies rely on either a small set of keypoints (Syeda et a 2024| [BL
ct al. 2025a Wang et al.
2025b; Chen al. (024) or latent dimensions using PCA (Stringer et al. 2019 Musall et al. 2019) or autoencoders (Batty et al., 2019 Wang et al., 2025b Chen et al. 2024).
with limited eftorts to predict neural activity directly from raw video (but see Wang et al. (2025b)).
Large-scale models for behavioral video analysis. Large-scale models for animal behavior analysis have predominantly focused on single tasks. For pose estimation, methods differ in how they balance flexibility and labeling requirements. DeepLabCut (Mathis et al. 2018) leverages ImageNet- pretrained backbones for fine-tuning on experiment-speeiffic labeled datasets, offering flexibility but requiring more manual labeling. This work inspired a range of other general-purport animal pose estimation tools including LEAP (Pereira et al.] 2019)), DeepPoseKit (Graving et al.] 2019). TRex (Walter and Couzin 2021), SLEAP (Pereira et al. 2022), and Lighnining Pose (Biderman et al. 2024). In contrast, several specialized pose estimation tools provide tailored solutions for common experimental setups, such as top-down views of freely moving mice (Ye al., 2024) and facial analysis of head-fixed rodents (Syeda et al. 2024, Daruwalla et al.) 2024), sigmificantly reducing labeling requirements. Similarly, in action segmentation, specialized systems developed for resident-intrude assays (Segalin et (201) Goodwin et al. (024) achieve high performance but remain limited to a specific experimental paradigm. While VideoPrism (Zhao et al. 2024) offers general foundation model supporting multiple behavioral tasks (Sun et al.) 2024). it relies on frozen backbone trained on generic internet data rather than domain-specific content. Despite these advances, accessible solutions exist for creating general behavior analysis models that leverage unlabeled data across multiple tasks. BEAST addresses this gap by enabling individual labs to develop experiment-specific models from their own unlabeled videos for diverse analyses.
2

---

Published as a conference paper at ICLR 2026
Self-supervised learning for images and videos. Contrastive learning has emerged as powerful self-supervised learning (SSL) framework; among its many predecessors and variants, SimCLR (Chen) et al. 2020) popularized a simple and effective recipe that maximizes agreement between differently augmented views of the same sample via a contrastive loss in latent space. The contrastive method has also been extended to the temporal (Hyvarinen and Morioka 2016) and video domains (Qian ct al. 2021; Recasens et al., [021] Dave et al.) 2022). Another line of self-supervised approaches uses knowledge distillation, where a student network learns to match the outputs of a teacher network, such as DINO (Caron et al., 2021, Oquab et al., 2023] Siméoni et al. 2025). Masked modeling is a complementary approach that has demonstrated remarkable success, particularly masked autoencoding (MAE) (He et al. 2022), which revolutionized visual self-supervised learning by adapting BERT-style masked prediction to images using Vision Transformers (Dosovitskiy et al., 2020). VideoMAE (Tong et al.) (022) and BEVT (Wang et al., 2022) extended this approach to video data by leveraging spatiotemporal dependencies. Various works have combined contrastive and MAE objectives more efficient alternative for capturing spatiotemporal dependencies, as video models can require much more compute for training and inference (Mishra et al., [022]] Huang et al.) 2023b Lu et al.] 2023 Lehner et al., 2024). Of note is VIC-MAE (Hernandez al. (2024), which uses patch-based features for masked autoencoding loss. The local features are also pooled into a global feature vector which is used with contrastive loss computed across frames from multiple videos. The efficiency of contrastive bandaattive-baee methods compared to native video models is of particular interest in our application domain, where labs often do not have access to extensive compute resources. SSL techniques from computer vision have increasingly been adapted to the study of animal behavior. These approaches use SSL to extract useful feature representations from pose, image, or video data, which are then applied to downstream tasks such as action segmentation (Appendix F).
3 METHODS
BEAST uses a combination of an satage-based masked autoencoding (MAE) loss-which excels at capturing per-frame appearance details-and temporal contrastive loss-which captures dependencies across frames (Fig. [A). This integration enables single backbone to excel across diverse downstream tasks, from precise keypoint localization to predicting complex structure in neural activity (Fig. [B)..
BEAST builds upon VIC-MAE (Hernandez et al., 2024), which combines masked autoencoding and contrastive losses, but introduces key adaptations for neuroscience applications. The most significant modification is how frames are sampled for the contrastive loss. VIC-MAE allows any two frames from the same video to be a positive pair, and frames from different videos are negative pairs (Xu and Wang 202I. While this may be appropriate for benchmark datasets with short clips, animal behavior experiments generate long- ong-duration recordings where behaviors repeat across time. We instead define positive frames within narrow temporal window around the anchor (+1 frame), while allowing negative frames to be either distant and dissimilar frames in the same video, or from different videos. Crucially, this strategy outperforms that of VIC-MAE (Table See Appendix [ for more details on our frame selection strategy and additional training and architecture simplifications of VIC-MAE.
Vision transformer (VTT), The standard image VIT (Dosovitskiy et al., (020) data pipeline starts with 2D image x E #x (H, W, C are height, width, channels) and splits it into 2D patches, each with shape (P x P x C)., where the patch size P is typically 16. Each patch is reshaped to a vector of length C²C, and all patches are concatenated into sequence of the N flattened 2D patches Xp E NxP*C), Each flattened patch is mapped with a trainable linear projection to a "patch token," vector of size D. We add ID position embeddings to the patch tokens to retain patch location information. We add a learnable CLS token to the patch token sequence, which serves as a global representation for the image. The resulting patch tokens augmented with position embeddings (t E IR NxD) D) and the concatenated CLS token serve as the input to the standard VIT encoder.
Masked autoencoding loss. The masked autoencoding (MAF) loss randomly masks out a high proportion of the patch tokens (here, 0.75 (He et al. 2022).. We call the resulting unmasked tokens tun E L where L = 0.25 x N is the number of unmasked tokens. The unmasked tokens processed by the VIT to produce embeddings Zum (VT((tmm) The masked embeddings Zm E (N-..)) L.) (consisting of all zeros) are then combined with the unmasked embeddings processed by VIT to form a complete patch sequence z E NND, which is passed through a transformer decoder to produce a reconstruction Xp E NNPPCC trained via mean square error:
LMSE N Epen( x2 We refer to the model trained only with this MAE loss as VIT-M.
3

---

Published as a conference paper at ICLR 2026
A
Decoder
Video
VIT
Negaliva
Decoder
Inference
Video
Embedding Space
(+N
Limie
Video
Decoder
Attraci
Ropet
Reconstruction Loss
Contrastive Loss
B
Neural Encoding
Pose Estimation
Action Segmentation
Figure BEAST framework. A: Our self-supervised pretraining framework BEAST combines masked autoencoding (He al. 2022) temporal contrastive learning (Chen al. 2020). anchor frame at time r paired with positive frame from + 1, while more distant frames from the same vidco, or frames from other vidcos, serve as negative examples. Frames are divided into patches, with most patches randomly masked, A vision transformer (VIT) processes the remaining patches, which must reconstruct all patches. The VIT CLS tokens, which serve global representation of each frame, nonlinearly projected to new space where the contrastive loss pulls anchor-positive pairs together and pushes anchor-negative pairs apart. BEAST supports various downstream neuro-hehavioral tasks including neural encoding, pose estimation, and action segmentation.
Temporal contrastive loss. The masked autoencoding loss is sufficient for reconstructing low-level features on individual frames. To imbue embeddings with temporal information (which may be required for certain downstream tasks), we employ a contrastive loss that produces similar embeddings for frames close in time, and distinct embeddings for frames far apart time from different videos. To achieve this, each batch with B samples contains B anchor frames, and each anchor frame x (from time & and video v) has randomly chosen positive frame from x All remaining frames are treated as negative frames. Note this approach differs from other temporal contrastive losses that allow any frame from the anchor frame's video to positive frame (Xu and Wang 202T Hernandez. et al. 2024), which does not perform well with temporally-extencee (T. To improve the robustness of this approach, we select the initial set of anchor frames from given video to be visually distinct from each other as possible (Table 4). We utilize the InfoNCE loss (Oord et al. 2018) computed on nonlinear projections of the CLS embeddings (which outperform other frame aggregation methods, Table 7 that output by the VIT. The projector outputs {2,,) exp(z, z",)
behavioral videos
used for the contrastive learning, calculated LIMONNEE - EieA log E,s, exp(x" x where is the positive example associated with and the set of B/2 anchor frames. We refer the model trained with both the masked autoencoding and contrastive losses as BEAST.
Training and finetuning. We initialize our models with pretrained ImageNet weights (Deng et al. 2009 He et al. 2022). Details of dataset construction, data augmentations, and batch construction provided in Appendix B We define the loss as LMSE + X LInTANCE where À balances the two losses and is selected using the validation sets of the various datasets. Models trained for 800 epochs using the Adam W optimizer (Loshchilov and Hutter) 2017) with a cosine annealing learning rate scheduler (Loshchilov and Hutter 2016), taking approximately 25 hours Nvidia A40 GPUs.
4

---

Published as a conference paper at ICLR 2026
SRR
ICN
3.45 BPST
94
0:45
0.25
0.32
Reypaints
PCA
CEBRA
0.4 0.35
0:45
2:35
1,85
Regporni
and
1.39
5.2
0.48
Wilsker
0.13 0.33
Eneryy
4.24
Vedion Ewrrg
1.13
1.01
BENT
Versence Explained
Samuacc Toglerkee We
0.6
0.83
Facomap
0.19
Kayparts
KCA
9.55
0,33
3.14
Figure 2: BEAST improves neural encoding. A: Example vidco frame from each dataset. B: Encoding performance is evaluated across multiple baseline features with both linear models (hatched bars: reduced rank regression, RRR) and nonlinear models (solid bars; temporal convolution network. TCN). CEBRA uses contrastive loss to embed video frames in latent feature space. "Motion energy" for the IBL-whisker dataset a ID estimate movement calculated as the sum of the absolute pixel differences between successive frames. BEAST features outperform all baselines in both linear and nonlinear regimes. Error bars show standard error of the mean (S E.M. of Bits per Spike (BPS) across = neurons from five test sessions (IBL and IBL-whisker) S.E.M. of variance explained of the principal components of the neural activity across five test sessions (Facemap; see text). C: Scatterplot comparison of BEAST vs keypoint-based model performance in example session. Each dot corresponds to an individual neuron. The values the bottom-right corner represent session-averaged BPS. D: Top, middle: comparison of the predicted trial-averaged firing rates for BEAST and keypoints (lines) and single-trial variability obtained by subtracting the neuron's average firing rate on each trial (heatmaps). Bottom: comparison of predicted neural principal components for the Facemap dataset.
4 RESULTS
We demonstrate the versatility of BEAST through comprehensive evaluation across three downstream couro-behavions tasks: (1) neural encoding, which challenges the model to extract spatiotemporal features that can predict patterns neural activity; (2) pose estimation, which assesses the model's ability to extract fine- fine-graine grained appearance details; and (3) action segmentation, which evaluates the model's 's capacity to extract spatiotemporal features required for predicting behavioral sequences. Throughout these evaluations, we present systematic ablation experiments that demonstrate the critical importance of our combined loss functions, and explore various adaptation strategies, including the use of CLS tokens patch embeddings from frozen backbone, as well as end-to-end fine-tuning.
4.1 NEURAL ENCODING
Predicting neural activity from behavior videos represents significant challenge with promising implications for understanding the relationship between brain and behavior (Musall et al., 2019 Stringer et al. 2019 Wang et al., 2025b). Traditional approaches often rely on keypoints (IBL et al.
2025b Syeda et al. 2024), potentially missing critical behavioral features that are not included in tracking or are obscured by fur or feathers. Studies have employed Principal Component Analysis on frames (Musall et al. 2019 Stringer et al. 2019), but this linear technique may inadequately capture subtle behavioral nuances. Transformer embeddings offer a compelling alternative, potentially out- performing linear approaches without being constrained by predefined keypoints, thereby providing richer representations that could reveal previously undetectable neuro-hehavioral correlations.
Datasets. We present results on three high-quality neuro- behuv--b datasets employing diverse neural recording technologies (Fig. 2 The first dataset is head-fixee mouse performing a decision-makin task from the International Brain Laboratory (IBL et al.) (025a). This dataset, "IBL," features simultaneous behavioral video and neural activity monitoring at single-cell, single-spike resolution
5

---

Published as a conference paper at ICLR 2026
2017) spanning multiple brain regions (average of 168 neurons using Neuropixels probes (Jun et al.
per session). The second dataset, whiskeiisker." " reuses the same sessions but utilizes a cropped area around the whisker pad in the video, a particularly salient behavioral feature for predicting neural activity (Stringer et al. 2019 Whiteway et al.) 2021] Syeda et al. 2024). The third dataset comes from the Facemap study (Syeda al.] 2024). where neural activity is captured through two-photon calcium imaging, technique capable of resolving a large number of individual cells, but unable to detect individual spikes. Following the authors' approach, we predict the principal components of the neural activity to capture predominant variance patterns across the large recorded neural populations.
Models. We first descrihe various feature representations used for neural encoding, followed by two models (linear and nonlinear) that fit each representation. The first representation for IBI. and Facemap are keypoints tracked across the face and body (11 for IBL, 12 for Facemap). For the IBL-whisker dataset, which lacks keypoints, we instead utilize a ID estimate of whisker pad motion energy (Appendix C. The second representation utilizes PCA applied to raw video frames, while the third leverages CEBRA (Schneider et al. (023) (which employs a contrastive loss to embed inputs in a latent feature space) applied to raw video frames. Finally, we present results (using the CLS token) from BEAST, which is initialized with ImageNet weights then fine-tuned separately on each test session. Table 13 shows additional baselines that use frozen features from pretrained MAE (He et al., 2022). DINOv2 (Oquab et al., 2023), and CLIP (Radford et al.) 2021) models. We train two encoders: linear encoders, which reveal how directly accessible information within the features, and nonlinear encoders, which better determine the upper bounds of information content in the features. The linear encoder is reduced rank regression model (Zhang al. (022a)). The nonlinear encoder is the temporal convolution network (TCN) proposed in the Facemap study (Syeda et al.) 2024).
Evaluation. All model hyperparameters are tuned to ensure robust baseline performance (Ap- pendix C). To evaluate neural encoding approaches, we utilize the Bits Per Spike (BPS) met- ric PPei et al. 2021) on the spike-resolved IBL dataset (higher values better), and the R2 metric on the neural principal components in the Facemap dataset. All models are evaluated five test sessions.
Results. We find that nonlinear encoders consistently outperform their linear counterparts across all datasets and feature representations (Fig- and Table T3). Notably, non-keypoint representations surpass keypoint-baselt approaches in both IBL and Facemap datasets, confirming our hypothe- sis that behavior videos contain richer information than what pose estimation typically captures. BEAST shows consistent improvements in neural encoding quality across all datasets and range of dimensionalities (Fig. (). indicating that BEAST'S exceptional performance is not limited to high- dimensional embedding spaces. Interestingly, the comparable BPS values for BEAST in both IBL and IBL-whisker datasets suggest that a substantial portion of the neurally-relevant behavior information captured by the whisker pad activity, least in the recorded brain regions.
The ViT-based models in Fig. 2 are fine-tuned individually for each ses- sion to enable direct comparison with baseline approaches. We investigated whether pretraining provides additional benefits (Table [T]. Strikingly, mod- els pretrained on ImageNet using only the MAE loss. "VIT-M (IN)". outper- form previous baselines without any fine-tuning. Further pretraining on 77 IBL sessions "VIT-M (IN+PT))" improves performance both IBL datasets, validating the impor- tance of domain-specific pretraining. By incorporating the contrastive objective, BEAST achieves superior zero-shot performance: BEAST (IN+PT) outperforms the MAE-only variant, as well as contrastive-only variant VIT-C. Session-specific fine-tuning, "BEAST (IN+PT+FT)", provides addi- tional significant gains, reaching performance levels comparable models fine-tuned directly from ImageNet weights (Table [3). Notably, even without fine-tuning, domain-specinf pretrained models remain highly competitive, offering researchers a practical option when computational resources for fine-tuning are limited. Finally, we experimented with using the patch embeddings as input to the neural encoder, but found superior performance with the CLS tokens (Table 12
Table 1: Zero-shot neural encoding performance (BPS stan- dard error of mean over N-842 neurons across 5 test sessions).
Method (TCN)
IBL.
Whi--hiskee
VIT-M (IN)
0.321 + 0.013
0.301 + 0.012
VIT-M (IN+PT)
0.331 0.013
0,311 : 0.013
VIT-C (IN+PT)
0.314 1: 0.013
0.283 + 0.011
BEAST (IN+PT)
0.292 1 0.012
0.309 1 0.013
BEAST (IN+PT+FT)
0.347 + 0.014
0.326 = 0.013

---

Published as a conference paper at ICLR 2026
Mouse perseptual decision-making 2025) keypoinss
Mouse locomation (Wamen al. 2021) keypoints
Residenn-itr (Burgas-Artim 2012) keypoints animmatts)
Freely swimming mormyrdd (Bideman 2024) keypoints views)
BO pixets
20 pixels
50 pixels
100 pizess
MMM0M MYNNANNNN
year
0
ResNett50 (AP10K) VIT-M (IM)
VIT-C (N++P) VIT-M PT)+ BEAST
100-Frame Tinetune
Mimemmuuee 100- Brabune
CRIM13 fineture
Mirntthh 100-Frame linetura
10
Indeels ser compuiiion
Ensemble des
Ersemble 1nd
Enserniee dev
Ensembble
"Hartoor" Kaypolnts
Figure 3: BEAST improves pose estimation. A: Example frame from each dataset overlaid with ground truth annotations. Green stars indicate the highlighted keypoint in panel B: Example traces from the ResNet-50 (gray) and BEAST (green) models for single keypoint in a held-out video. BEAST traces evolve more smoothly in time and do not contain erroneous jumps like the ResNet-50 baseline. C: Pixel error as function of keypoint difficulty (see main text; smaller is better): left-hand side shows performance across all keypoints; moving the right drops the easier keypoints defined by inter-seed and -model prediction variance. Vertical dashed lines indicate the percentage of data used for the pixel error computation. VIT-M (IN) is a VIT backbone pretrained on ImageNet with masked autoencoding loss; VIT-M (IN+PT) uses the same architecture and loss but is initialized with ImageNet pretrained pretrried weights then further pretrained experiment- specific unlabeled frames. VrT-C (IN+PT) performs the experiment-specift pretraining using the temporal contrastive loss only.
4.2
POSE ESTIMATION
Pose estimation is fundamental technique animal behavior analysis (Pereira al. 2020). enabling precise quantification of posture and movement. Unlike human pose estimation, which benefits from extensive labeled datasets and standardized anatomy, animal pose estimation presents unique challenges such scarcity of large annotated datasets and significant morphological diversity across species. Pretraining models on large volumes of unlabeled behavior videos can potentially reduce the labeled data requirements for accurate keypoint localization in various experimental paradigms.
Datasets. We present results on four distinct datasets (Fig. 30: (1) head-fixed mouse performing
a decision-making task (IBL. et al. 2025a); (2) head-fixed mouse running on a treadmill, seen from two views (Warren et al. [021] I); (3) the Caltech Resident-Intrudee Mouse (CRIM13) dataset, consisting of two freely interacting mice (Burgos-Artizzu et al.
2012); and (4) a freely moving
weakly electric fish, seen from three views (Biderman et al. 2024
Pedraja et al. 2025)
Models. We implemented pose estimation models using Lightnnin Pose (Biderman et al. 2024). We established strong baseline utilizing a ResNet-50 backbone pretrained on AP-10K Yuu et al. 2021). which outperforms a DeepLabCut baseline (ImageNet-pretrree pretrained ResNet-50) on all but the CRIM13 dataset (Fig. [T]. Our second baseline is a Vision Transformer (VIT-B/16 pretrained on ImageNet (He et al. 2022) using our own implementation of ViTPose (Xu et al. 2022), enabling assessment of potential improvements when transitioning from convolutional- to transformer based architectures. Our own VIT based models utilize this same architecture. In the Appendix additional baselines that use fine-tuned DINO (Caron et al., (22T), DINOv2 (Oquab et al. (202).. and Segment Anything (Kirillov et al. 2023) encoders (which BEAST consistently outperforms;
provide
7

---

Published as a conference paper at ICLR 2026
Fig. 9 For consistency across all model variants, we employ identical pose estimation head that transforms backbone features into keypoint heatmaps. Given the spatial nature of the task, we use patch embeddings rather than CLS tokens in the transformers, and train all models end-to-end.
Evaluation. To rigorously evaluate our pose estimation models, we designed challenging limited- data scenario with only 100 labeled training frames, a realistic constraint for many research settings where extensive annotation is impractical. We measured pixel error between predicted keypoints and ground truth on a test set of novel subjects. For each backbone, we fit three models on different 100-frame subsets. Results are presented as pixel eπror relative to ensemble standard deviation (e.s.d.) across all seeds and backbones following Biderran et al. (2024), with error curves showing perfor- mance varying difficulty thresholds. Each point corresponds to keypoints with e.s.d. exceeding the threshold value, with the leftmost portion showing eгror across all keypoints and rightward movement including only increasingly challenging keypoints (those with higher inter-model variability).
Results. We find robust improvements in pose estimation quality across all datasets when utilizing BEAST (Fig.B) 3. The all datasets except the fish, demonstrating the effectiveness of transformers even with limited labels. Notably, pretraining the transformer with the MAE objective on experiment-specific data yields substantial performance gains across all datasets, including the challenging fish dataset. Augmenting MAE with the contrastive objective (BEAST) produces additional performance improvements for some datasets, with particularly pronounced benefits observed in the IBL. dataset. However, pretraining only on the contrastive objective leads to significantly worse results, consistent with the different learning objectives: the temporal contrastive loss emphasizes high-level temporal structure, whereas the MAE loss emphasizes low-level, pixel-level features. Consequently, MAE-pretrained representations are better suited for pixel-level prediction tasks like pose estimation, though addition of the contrastive loss in BEAST still provides complementary benefits. We also find BEAST's performance advantages persist when scaling to larger training datasets (Fig. 10).
VrT outperforms the AP-10K AP-1OK-pretrained ResNet-50 on
4.3 ACTION SEGMENTATION
Action segmentation classifies discrete behaviors using spatiotemporal video features (Pereira et al. 2020). Similar to pose estimation, a central challenge in animal action segmentation is the lack of large annotated datasets, as behaviors of interest often vary across species and experimental contexts. Many current approaches rely on keypoints (Branson et al.] 2009 Kabra et al.] 2013] (Seaaiin et al., 2021 Gabriel et al., 2022 Goodwin al., 2024), requiring an initial labor-intensive and error-prone preprocessing step. Vision transformer embeddings eliminate this preprocessing requirement and provide an attractive alternative they match or exceed the performance of keypoint- based methods.
Datasets. We present results on datasets (Fig. 4 (1) the "IBL." dataset (IBL. et al. 2025a), which contains four behavior classes for the paw nearest the camera; and (2) the Caltech Mouse Social Interactions (CalMS2I) dataset (Sun al. (021a), a resident-intruddr assay of interacting mouse pairs that contains four social behavior classes.
Models. We implemented models using two types of embeddings from frozen VIT models: (1) CLS tokens and (2) per-patch embeddings. For CLS embeddings, tested both a linear model and TCN (Lea et af. 2016). enriching the input with inter-frame differences (Blau et al., (022).. We
used sliding window over this feature sequence to predict the action class of the central frame.
For patch embeddings, we applied multi-head attention pooling (Lee et al. 2019 Yu et al. 2022
Sun et al. 2024) integrate information across patches, then concatenated the resulting frame-level embeddings with their inter- frrme differences before processing them through a TCN (Fig. 12
For IBL, we compared against three baseline features: (1) a single paw keypoint, obtained using five pose estimation networks (cach trained with 7,000 labeled frames) post-processed with an Ensemble Kalman Smoother (Biderman et al. 2024 ; (2) principal components of the video frames; and (3) the CLS and patch embeddings extracted from a frozen-weight VIT pretrained on ImageNet with
MAE loss (which outperformed patch embeddings from DINOv2 in both datasets; Table [5]. For CaIMS21, we compared against four baseline features: (1) Trajectory Embedding for Behavior Analysis (TREBA) (unn et al., 2021b), a self-supervised feature extraction method for keypoint trajectories; (2) Simple Behavioral Analysis (SimBA) (Goodwin et al. 2024), which extracts hundreds of crafted features from the keypoint trajectories; (3) principal components of video frames; and (4) the CLS and patch embeddings from a frozen-weight VIT pretrained ImageNet with MAE
8

---

Published as a conference paper at ICLR 2026
A
TCN
Enserbble
TCN
Ensemble
Mouse perceptual decision making (IBL 2023)
1.0
CalMS21: resident- intruder assay (Sun al., 2021)
0.9
0.88
0.8r ouy
0.70
0740777 0.810
0.8
0.8
0.71
0.71
Ft
0.7
0.5
0.1
0.18..6
0.5
0.4
Keypoinns
(putones))
BEAST
(patshea))
0.0
PCR
Keyponts
(SimB44
(patchne)
BEAST
(patches)
WiTH
VIT
Keypointe (SimBA)
B
Keypoints
BEAST
1.0
1.0
1.0
0.02 0.01
8.8
0.01
0.66
0.29 0.01
0.05
0.83
0:14
0.01
0.02
08
True facd
tegget
0.86
8.8
invenigation
0.02
0.16
06
0.05
0.16
06
wheat Jam
0.00
0.28 0.72 0.00
0.4
0.01
0.27 0.72
0.00
moink
0.00
0.14
0.82
0.08
0.4
0.01
0.00
0.92
0.02
04
grasmn
0.00
0.01
0.00
0.99
0.2
0.00 0.01
0:99
0.9
0.00
0.05
0.00
0.94
0.2
0.00
0.04
0.94
02
lospel jum
grasm
tdgeet
asheed tum
grosm
artech
attach
Investigttonn
mound
Predicted
C
Dosttion
position
SMBBA
SMBBAPO22
cless
MITH BOW
gound
200
1000
1000
Uie [tames]
time (tamms)
BEAST-PC-1
BEAST-PC PC-2
BEAST.PC-1
BEAST-PC
class
BEAST
gound
200
400
600
800
1000
200
400
600
800
1000
Umo (homes)
tim (hames))
ndget
wheel
attack
Invectigatio
other
Figure 4: BEAST improves action segmentation. A: Example frame from each dataset; performance evaluated across multiple baseline features with both TCN (solid) and ensembled (hatched) models. Error bars represent standard error the mean across five random initializations. B: Confusion matrices for TCN models based on keypoints and BEAST patch embeddings. C: Example behavior sequences with feature traces (single seed shown for BEAST models), ensemble probabilities, ensembled model cthograms, ground truth ethograms, and crror frames. PCs of SimBA and BEAST features are shown for illustration, but the models utilize the full feature set.
The pose estimator used for TREBA and SimBA was trained with 15,000 labeled frames (un ct al. 2021a). For all baselines except SimBA, we also concatenated inter-frame differences.
loss.
Evaluation. All model hyperparameters are tuned to ensure robust baseline performance (Ap- pendix FD. We evaluate performance on held-out animals using the macro-averaged F1 score. For the CalMS2I dataset, following [unn al. (2021a), we average the F1 score over the attack, investigation and mount classes. For all models we train five networks using different random initializations. We also report the results of model ensembles by averaging logits across seeds before applying softmax.
Results. BEAST demonstrates strong action segmentation performance across all datasets (Fig, 4 Remarkably, ImageNet- preeraiid VrT-M patch embeddings nearly match keypoint-bassed methods despite utilizing frozen, general-purpor backbone. This establishes competitive baseline without requiring the thousands of labels needed to train pose estimation networks. On IBL., BEAST improves upon ImageNet baselines. The keypoint-basee model excels here due to action classes corresponding to paw movements easily captured by pose estimation, but this advantage disappears with ensembling: BEAST ensemble Fl matches the keypoint ensemble. CalMS21 better demonstrates BEAST'S abilities, which surpasses the SimBA baseline and substantially outperforms the TREBA baseline

---

Published as a conference paper at ICLR 2026
(Table The ensembled F1 score of 0.84 places result in the top 15 of the Multi-Agent Behavior Challenge on Alcrowd.com (top score of 0.89). Additional experiments confirm domain- specific pretraining benefits: BEAST CLS tokens consistently outperform their saaggeett-pretriinn counterparts ( though patch-based models perform significantly better due to their enhanced spatial resolution and multi-headed attention pooling. An ablation experiment on BRAST'S loss terms show that backbones pretrained with a contrastive-only (VIT-C) or MAE-only (VIT-M) loss do not perform as well their combination (Table [5]. Across all experiments, nonlinear models consistently outperform their linear counterparts, except for PCA features on CalMS21 (Table [5)... All evaluations use frozen backbones with only INN heads fine-tuned, suggesting further gains may be possible through full backbone fine-tuning.
BEAST'S advantages extend beyond abso- lute F1 improvements. Pose estimation- based approaches require both extensive labeling and iterative training and vali- dation of pose estimation models before action segmentation, often months- even years-long process (IBL et al. (0022). BEAST eliminates this entire pipeline, achieving competitive or superior performance using only unlabeled video for pretraining.
Table 2: Action segmentation performance (FI S...MM..
Method (TCN)
IBL.
CaIMS21
TREBA
0.72 + 0.01
VIT-M (IN) (CLS)
0.79 + 0.00
0.60
VIT-M (IN) (patch)
0.84 -: 0.00
0.74
BEAST (IN+PT) (CLS)
0.81 + 0.00
0.63
BEAST (IN+PT) (patch)
0.87 + 0.01
0.81 0.01
5 DISCUSSION
This work introduces BEAST, framework for self-supervised vision transformer pretraining lever- aging domain-specific video data. We demonstrated BEAST'S significant benefits across neural encoding, pose estimation, and action segmentation tasks. Notably, BEAST outperforms DINOv2 and other computer vision foundation models across all tasks (Table 13), demonstrating the power of
pretraining. Our frame- based approach is efficient alternative native video models like VIDEOMAE (Tong et al.) 2022). which require significantly more compute for training and inference (Table 8): however BEAST still outperforms frozen VIDEOMAR on the neural encoding task (Table 13
Our work establishes foundation for several promising future directions. Investigation of transformer attention and learned features could clarify how BEAST operates across different tasks. The black-box nature of VIT embeddings presents interpretability challenges in scientific contexts where transparent representations like pose estimates are often preferred. Visualization methods provide initial insights (Fig. 7. but systematic analysis of what features drive performance on different tasks would strengthen our understanding of when and why BEAST succeeds.
While we have demonstrated BEAST'S performance across diverse experimental contexts-including head-fixed fixed and freely moving animals, single- and multi-view setups, and solitary or social behaviors- validation across more environments and species needed. The success of masked autoencoding and contrastive losses in general computer vision suggests BEAST should adapt well to naturalistic settings (e.g., home cages, zoos, field studies). The primary challenge will be adjusting the frame sampling strategy to accommodate different visual statistics and behavioral distributions in these less controlled environments.
two complementary paths toward making powerful self-supervised models more accessible to individual labs. First, using smaller transformer architectures will reduce pretraining, fine-tuning, and inference costs, but may sacrifice performance. Second, BEAST'S framework could enable foundation models of animal behavior trained across diverse datasets, rather than the dataset-specific pretraining we present here. Such foundation models would allow labs to finetune already-powerful pretrained models rather than pretraining themselves. Together, these approaches would lower barriers to adoption and enable wider application of supervieer learning across the neuroscience community.
Finally,
Acknowledgmentt and Disclosure of Funding
This work supported by Gatsby Charitable Foundation GAT3708, NIH U19NS123716 and R50NS145433, NSF 1707398, the National Science Foundation and DoD OUSD (R&F) under Cooperative Agreement PHY-2229929 (The NSF AI Institute for Artificial and Natural Intelligence). Simons Foundation, Wellcome Trust 216324, and funds from Zuckerman Institute Team Science.
10

---

Published as a conference paper at ICLR 2026
REPRODUCIBILITY STATEMENT
We have attempted to provide sufficient detail to foster reproducibility of all analyses presented in
this manuscript.
Data availability. All datasets are sourced from public repositories, with links and accompanying licenses provided in Appendix A.
Model availability. Pretrained BEAST models for each dataset are available https:// http://.......oo..e...oooo/aaaaa
under the CC-BY 4.0 license.
Code availability.
BEAST frame selection, pretraining, and inference:
https://github.com
Neural encoding:
Reduced Rank Regression for IBL: https://githubb.commeeesalsss
bra inw encco I
Reduced Rank Regression for Facemap: httpaa//ggttuub.cc///ouusaaaaa
facemap/blob/wv 0..1..0.7/ //wcmmaa///////aaa htt.a/////.aoaaaa.eeer.oeoooo/ra
utils. utils.pyfL110
Temporal Convolution
Network:
https://gtthbb..omm/bussssstti
model.p1
Pose estimation: teleases/tag/v2.022
Action
http:uuir..atttoo//eeelsse/aaa/p bttp://gtttub.com/aaanakkk.
segmentation:
Analysis details.
Pretraining (Appendix B): BEAST pretraining procedure and frame selection strategy.
Neural encoding (Appendix C Model architectures, training procedures, and hyperparame- ter tuning details.
Pose estimation (Appendix D): Training procedures and hyperparameter selection.
Action segmentation (Appendix E}: Model architectures, training procedures, and hyperpa- rameter tuning details.

---

Published as a conference paper at ICLR 2026
REFERENCES
Mehdi Azabou, Michael Mendelson, Nauman Ahad, Maks Sorokin, Shantanu Thakoor, Carolina Urzay, and Eva Dyer. Relax, it doesn't matter how you get there: A new self-supervised approach for multi-timescale behavior analysis. Advances in Neural Information Processing Systems, 36: 28491-28509, 2023.
Adrien Bardes, Quentin Garrido, Jean Ponce, Xinlei Chen, Michael Rabbat, Yann LeCun, Mido Assran, and Nicolas Ballas. V-jepa: Latent video prediction for visual representation learning. Openreview.net, 2023.
Eleanor Batty, Matthew Whiteway. Shreya Saxena, Dan Biderman, Taiga Abe, Simon Musall, Winthrop Gillis, Jeffrey Markowitz, Anne Churchland, John P Cunningham, et al. Behavenet: nonlinear embedding and bayesian neural decoding of behavioral videos. Advances neural information processing systems, 32, 2019.
Dan Biderman, Matthew R Whiteway, Cole Hurwitz, Nicholas Greenspan, Robert S Lee, Ankit Vish- nubhotla, Richard Warren, Federico Pedraja, Dillon Noone, Michael M Schartner, et al. Lightning pose: improved animal pose estimation via supervise learning. bayesian ensembling and cloud-native open-source tools. Nature Methods, 21(7):1316-333 2024.
Ari Blau, Evan S Schaffer, Neeli Mishra, Nathaniel Miska, International Brain Laboratory, Liam Paninski, and Matthew R Whiteway. A study of animal action segmentation algorithms across supervised, unsupervised, and semi-supervised learning paradigms. Neurons, Behavior, Data Analysis, and Theory, 2024. doi: 10.51628/00cc.1 127770.
Kristin Branson, Alice A Robie, John Bender, Pietro Perona, and Michael H Dickinson. High- throughput ethomics in large groups of drosophila. Nature methods, 6(6):451-457, 2009.
Xavier P Burgos-Artizzu, Piotr Dollár, Dayu Lin, David Anderson, and Pietro Perona, Social behavior recognition in continuous video. In 20/2 IEEE conference computer vision and pattern recognition, pages 1322-1329. IEEE, 2012.
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. Advances in neural information processing systems, 33:99 12-9924. 2020.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. Proceedings of the IEEE/CVF international conference on computer vision, pages 9650-9660, 2021.
Susu Chen, Yi Liu, Ziyue Aiden Wang, Jennifer Colonell, Liu Liu, Han Hou, Nai-Wen Tien, Tim Wang, Timothy Harris, Shaul Druckmann, et al. Brain-wide neural activity underlying memory-guided movement. Cell, 187(3):67--611, 2024.
Ting Chen, Simon Korbllith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. International conference on machine learning, pages 1597-1607. PmLR, 2020.
John Co-Reyes, YuXuan Liu, Abhishek Gupta, Benjamin Eysenbach, Pieter Abbeel, and Sergey Levine. Self-consistent trajectory autoencoder: Hierarchical reinforcement learning with trajectory embeddings. In International conference on machine learning, pages 1009-1018, PMLR, 2018.
Kyle Daruwalla, Irene Nozal Martin, Linghua Zhang, Diana Naglië, Andrew Frankel, Catherine Rasgaitis, Xinyan Zhang, Zainab Ahmad, Jeremy C Borniger, and Xun Helen Hou. Cheese3d: Sensitive detection and analysis of whole-face movement in mice. bioRxiv, pages 2024-05, 2024.
Sandeep Robert Datta, David J Anderson, Kristin Branson, Pietro Perona, and Andrew Leifer. Computational neuroethology: call to action. Neuron, 104(1):11-22) 2019.
Ishan Dave, Rohit Gupta, Mamshad Nayeem Rizve, and Mubarak Shah. Telr: Temporal contrastive learning for video representation. Computer Vision and Image Understanding, 219:103406, 2022.
12

---

Published as a conference paper at ICLR 2026
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 JEEE conference on computer vision and pattern recognition, pages 248-255. Iece, 2009.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition scale. arXiv preprint 1rX1::2010.11929, 2020.
Christopher Gabriel, Zachary Zeidler, Benita Jin, Changliang Guo, Caitlin M Goodpaster, Adri- enne Q Kashay, Anna Wu, Molly Delaney, Jovian Cheung, Lauren E DiFazio, et al. Behaviordepot is simple, flexible tool for automated behavioral detection based on markerless pose tracking. Elife, 11:74414, 2022.
Nastacia L Goodwin, Jia J Choong, Sophia Hwang, Kayla Pitts, Liana Bloom, Aasiya Islam, Yizhe Y Zhang, Eric Szelenyi, Xiaoyu Tong, Emily L Newman, et al. Simple behavioral analysis (simba) as a platform for explainable machine learning in behavioral neuroscience. Nature neuroscience, 27(7):1411-1244 2024.
Jacob M Graving, Daniel Chae, Hemal Naik, Liang Li, Benjamin Koger, Blair Costelloe, and lain D Couzin. Deepposekit. software toolkit for fast and robust animal pose estimation using deep learning. elife, 8te47994, 2019.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 16000-16009, 2022.
Jefferson Hernandez, Ruben Villegas, and Vicente Ordonez. Vic-mae: Self-supervised representation learning from images and video with contrastive masked autoencoders. In European Conference on Computer Vision, pages 444-463. Springer, 2024.
Shih- Shih-Cheng Huang, Anuj Pareek, Malte Jensen, Matthew Lungren, Serena Yeung, and Akshay Chaudhari. Self-supervised learning for medical image classification: systematic review and implementation guidelines. NPJ Digital Medicine, 6(1):74, 2023a.
Zhicheng Huang. Xiaojie Jin, Chengze Lu, Qibin Hou, Ming-Mine Cheng, Dongmei Fu, Xiaohui Shen, and Jiashi Feng. Contrastive masked autoencoders stronger vision learners. IEEE Transactions on Pattern Analysis and Machine Intelligence, 46(4):2506 2517, 2023b.
Aapo Hyvarinen and Hiroshi Morioka. Unsupervised feature extraction by time-contrastive learning and nonlinear ica. Advances in neural information processing systems, 29, 2016.
IBL. Data release Brainwide map Q4 2022. Figshare, 2023. doi: 10.6084/m9 figshare.21400815.
URL https: httpp:::///igggareeee cttp:///ceeceeeaeceeecee/c//eeee http:iiceeeesreerrn////ttt... rel galease_-ao Leaase__Baase Brai Arainwider nwwde_mapp____ 24_2022/21400811
IBL. Valeria
Dora Angelaki, Hannah Bayer, Niccolo Bonacchi, Matteo Caran- dini, Fanny Cazettes, Gaelle Chapuis, Anne K Churchland, Yang Dan, et al. Standardized and reproducible measurement of decision-making in mice. Elife, 10:e63711 1, 2021.
IBL, Dan Birman, Niccolo Bonacchi, Kelly Buchanan, Gaelle Chapuis, Julia Huntenburg, Guido Meijer, Liam Paninski, Michael Schartner, Karel Svoboda, Matthew Whiteway, Miles Wells, and Olivier Winter. Video hardware and software for the international brain laboratory, figshare, 2022. doi: 10.6084/m9. (tt.mmm...gghaae.1911 19694452. URL httpt:ttp//... //o////////////////// /trg/10.0 6084 / f gahare 19694452]
IBL, Dora Angelaki, Brandon Benson, Julius Benson, Daniel Birman, Niccolò Bonacchi, Kcénia Bougrova, Sebastian A Bruijns, Matteo Carandini, Joana A Catarino, et al. brain-wide map of neural activity during complex behaviour. Nature, 645(8079):177, 2025a.
13

---

Published as a conference paper at ICLR 2026
IBL, Kush Banga, Julius Benson, Jai Bhagat, Dan Biderman, Daniel Birman, Niccolò Bonacchi, Sebastian A Bruijns, Kelly Buchanan, Robert AA Campbell, et al. Reproducibility of vivo electrophysiolggiaal measurements in mice. Elife, 13:RP100840, 4 2025b.
Yinjun Jia, Shuaishuai Li, Xuan Guo, Bo Lei, Junqiang Hu, Xiao-Hong Xu, and Wei Zhang. Selfee, self-supervised features extraction of animal behaviors. Elife, 11:e76218, 2022.
James J Jun, Nicholas A Steinmetz, Joshua H Siegle, Daniel J Denman, Marius Bauza, Brian Barbarits, Albert K Lec, Costas Anastassiou, Alexandru Andrei, Çağatay Aydın, et al. Fully integrated silicon probes for high-density recording of neural activity, Nature, 551(7679):232-236 2017.
Mayank Kabra, Alice A Robie, Marta Rivera-Alba, Steven Branson, and Kristin Branson. Jaaba: interactive machine learning for automatic annotation of animal behavior. Nature methods, 10(1): 64 67, 2013.
Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijayanaaaiimhnn, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, et al. The kinetics human action video dataset. arXiv preprint arXiv: arXiv:1705.0669 170.106550000 2017.
Alexander JE Kell, Daniel LK Yamins, Erica N Shook, Sam V Norman-Haiinnere, and Josh H McDermott, A task-optimized neural network replicates human auditory behavior, predicts brain responses, and reveals a cortical processing hierarchy. Neuron, 98(3):630-00) 6304644,.444 2018.
Diederik P Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, al. Segment anything. In Proceedings of the IEEE/CVF international conference on computer vision, pages 4015-4026, 2023.
John W Krakauer, Asif A Ghazanfar, Alex Gomez-Marin, Malcolm Maclver, and David Poeppel. Neuroscience needs behavior: correcting reductionist bias. Neuron, 93(3):480-490, 2017.
Erica Lastufka, Mariia Drozdova, Vitaliy Kinakh, Davide Piras, and Voloshynovskyy. Vision foundation models: can they be applied astrophysics data? arXiv preprint 11X.v:2409.11175. 2024.
Colin Lea, Rene Vidal, Austin Reiter, and Gregory Hager. Temporal convolutional networks: unified approach to action segmentation. In Computer vision-ECCV 2016 workshops: Amsterdam, the Netherlands, October 8-10 and 15-16, 2016, proceedings, part III 14, pages 47-54. Springer, 2016.
Juho Lee, Yoonho Lee, Jungtaek Kim, Adam Kosiorek, Seungjin Choi, and Yee Whye Teh. Set trans- former: A framework for attention- based permutation-invariee neural networks. In International conference on machine learning, pages 3744-3753. PMLR, 2019.
Johannes Lehner, Benedikt Alkin, Andreas Fürst, Elisabeth Rumetshofer, Lukas Miklautz, and Sepp Hochreiter. Contrastive tuning: A little help to make masked autoencoders forget. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 2965-2973, 2024.
Yuanning Li, Gopala K Anumanchipalli, Abdelrahman Mohamed, Peili Chen, Laurel H Camey, Junfeng Lu, Jinsong Wu, and Edward Chang. Dissecting neural computations in the human auditory pathway using deep neural networks for speech. Nature Neuroscience, 26(12):2213-2225 2023.
Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint arXiv:1608.039:: 033883 2016.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv: 2017.
14

---

Published as a conference paper at ICLR 2026
Cheng-Ze -Ze Lu, Xiaojie Jin, Zhicheng Huang, Qibin Hou, Ming-Ming Cheng, and Jiashi Feng. Cmae-v: contrastive masked autoencoders for video action recognition. arXiv preprint Stt::2301.00011. 2023.
Kevin Luxem, Petra Mocellin, Falko Fuhrmann, Johannes Kürsch, Stephanie R Miller, Jorge Palop. Stefan Remy, and Pavol Bauer. Identifying behavioral structure from deep variational embeddings of animal motion. Communications Biology, 5(1):1267, 2022.
Alexander Mathis, Pranav Mamidanna, Kevin M Cury, Taiga Abe, Venkatesh N Murthy, Macken- zie Weygandt Mathis, and Matthias Bethge. Deeplabcut: markerless pose estimation of user-defined body parts with deep learning. Nature neuroscience, 21(9): 1281-998, 2018.
Mackenzie Weygandt Mathis and Alexander Mathis. Deep learning tools for the measurement of animal behavior in neuroscience. Current opinion in neurobiology, 60:1-11, 2020.
James M McFarland, Yuwei Cui, and Daniel A Butts. Inferring nonlinear neuronal computation based on physiologically plausible inputs. PLoS computational biology. 9(7):e0033133, 2013.
Leland McInnes, John Healy, and James Melville. Umap: Uniform manifold approximation and 2018.
projection for dimension reduction. arXiv preprint
Shlok Mishra, Joshua Robinson, Huiwen Chang. David Jacobs, Aaron Sarna, Aaron Maschinot, and Dilip Krishnan. A simple, efficient and scalable contrastive masked autoencoder for learning visual representations. arXiv preprint arXiv:2210.16877, 2022.
Felix B Mueller, Timo Lueddecke, Richard Vogg, and Alexander Ecker. Domain-adaptive pretrain- ing improves primate behavior recognition. arXiv preprint 1XXi::2509.12993 2025.
Simon Musall, Matthew T Kaufman, Ashley I. Juavinett, Steven Gluf, and Anne Churchland. Single-trial neural dynamics are dominated by richly varied movements. Nature neuroscience, 22 (10):1677-1686, 2019.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint 1rXXiv1807.000 03748,, 2018
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07/999 2023.
Liam Paninski. Maximum likelihood estimation of cascade point-process neural encoding models. Network: Computation Neural Systems, 15(:)2243, 2004.
Federico Pedraja, Michael Genecin, Dillon Noone, Dan Biderman, Philip Cho, Richard A Warren, David Ehrlich, and Nathaniel B Sawtell. Direct cerebellar control over motor production in species with extreme cerebellar enlargement. Current Biology, 2025.
Felix Pei, Joel Ye, David M. Zoltowski, Anqi Wu, Raeed H. Chowdhury, Hansem Sohn, Joseph E. O' D'Dhherty, Krishna V. Shenoy, Matthew T. Kaufman, Mark Churchland, Mehrdad Jazayeri, Lee Miller, Jonathan Pillow, Il Memming Park, Eva L. Dyer, and Chethan Pandarinath. Neural latents benchmark '21: Evaluating latent variable models of neural population activity. In Advances in Neural Information P Processing Systems (NeurIPS), Track Datasets and Benchmarks, 2021. URL /tttps://arxivve org/ abs/ 22//2219..0 04463
Talmo D Pereira, Diego E Aldarondo, Lindsay Willmore, Mikhail Kislin, Samuel S-H Wang, Mala Murthy, and Joshua W Shaevitz. Fast animal pose estimation using deep neural networks. Nature methods, 16(1):117-125, 2019.
Talmo D Pereira, Joshua W Shaevitz, and Mala Murthy. Quantifying behavior to understand the brain. Nature neuroscience, 23(12):1537-1549 2020.
Talmo D Pereira, Nathaniel Tabris, Arie Matsliah. David M Turner, Junyu Li, Shruthi Ravindranath, Eleni Papadoyannis, Edna Normand, David S Deutsch, Yan Wang, et al. Sleap: A deep learning system for multi-animal pose tracking. Nature methods, 1944):88-495, 2022.
15

---

Published as a conference paper at ICLR 2026
Jonathan W Pillow, Jonathon Shlens, Liam Paninski, Alexander Sher, Alan M Litke, EJ Chichilnisky, and Eero P Simoncelli. apatio-tempordl correlations and visual signalling in a complete neuronal population. Nature, 454(7207):995 999, 2008
Lorenzo Posani, Shuqi Wang, Samuel P Muscinelli, Liam Paninski, and Stefano Fusi. Rarely categorical, always high-dimensional: how the neural code changes along the cortical hierarchy. bioRxiv, pages 2024-11, 2025.
Rui Qian, Tianjian Meng, Boqing Gong, Ming-Hsuan Yang. Huisheng Wang, Serge Belongie, and Yin Cui. Spatiotemporal contrastive video representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6964-6974, 2021.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages PmLR, 2021.
Adria Recasens, Pauline Lue, Jean- Hann-Bapiiste Alayrac, Luyu Wang, Florian Strub, Corentin Tallec, Mateusz Malinowski, Viorica Pätrăucean, Florent Altché, Michal Valko, et al. Broaden your views for sell-supervised video learning. In Proceedings of the IEEE/CVE international conference on computer vision, pages 1255-1265, 2021.
Steffen Schneider, Jin Hwa Lee, and Mackenzie Weygandt Mathis. Learnable latent embeddings for joint behavioural and neural analysis. Nature, 617(7960):360-22
2023.
Martin Schrimpf. Jonas Kubilius, Ha Hong, Najib Majaj, Rishi Rajalingham. Elias B Issa, Kohitij Kar, Pouya Bashivan, Jonathan Prescott-Roy. Franziska Geiger, et al. Brain-score: Which artificial neural network for object recognition most brain-like? BioRxiv, page 407007, 2018.
Cristina Segalin, Jalani Williams, Tomomi Karigo, May Hui, Moriel Zelikowsky, Jennifer J Sun, Pietro Perona, David Anderson, and Ann Kennedy. The mouse action recognition system (mars) software pipeline for automated analysis of social behaviors in mice. Elife, 10:e63720, 2021.
Oriane Siméoni, Huy V Vo, Maximilian Seitzer, Federico Baldassarre, Maxime Oquab, Cijo Jose, Vasil Khalidov, Mare Szafraniec, Seungeun Yi, Michaël Ramamonjisoa, et al. Dinov3, arXiv preprint a50iv:2508. 10104, 2025.
Carsen Stringer, Marius Pachitariu, Nicholas Steinmetz, Charu Bai Reddy, Matteo Carandini, and Kenneth D Harris. Spontaneous behaviors drive multidimensional, brainwide activity. Science, 364(6447))ccav78%)) 7893, 2019.
Jennifer J Sun, Tomomi Karigo, Dipam Chakraborty, Sharada P Mohanty, Benjamin Wild, Quan Sun, Chen Chen, David Anderson, Pietro Perona, Yisong Yue, et al. The multi-agent behavior dataset: Mouse dyadic social interactions. Advances in neural information processing systems, 202I(DBI): 1, 2021a.
Jennifer Sun, Ann Kennedy, Eric Zhan, David J Anderson, Yisong Yue, and Pietro Perona. Task programming: Learning data efficient behavior representations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2876-2885, 2021b.
Jennifer J Sun, Markus Marks, Andrew Wesley Ulmer, Dipam Chakraborty, Brian Geuther, Edward Hayes, Heng Jia, Vivek Kumar, Sebastian Oleszko, Zachary Partridge, al. Mabe22: A multi- species multi-task benchmark for learned representations of behavior. In International Conference on Machine Learning, pages 22996. PMLR, 2023.
Jennifer Sun, Hao Zhou, Long Zhao, Liangzhe Yuan, Bryan Seybold, David Hendon, Florian Schroff, David A Ross, Hartwig Adam, Bo Hu, et al. Video foundation models for animal behavior analysis. bioRxiv, pages 2024-07, 2024.
Atika Syeda, Lin Zhong, Renee Tung. Will Long, Marius Pachitariu, and Carsen Stringer. Facemap: a framework for modeling neural activity based on orofacial tracking. Nature neuroscience, 27(1): 187-195, 2024.
16

---

Published as a conference paper at ICLR 2026
Zhan Tong, Yibing Song, Jue Wang, and Limin Wang. Videomae: Masked autoencoders data- efficient learners for self-supervised video pre-training. Advances in neural information processing systems,
2022.
Wilson Truccolo, Uri T Eden, Matthew R Fellows, John P Donoghue, and Emery N Brown. A point process framework for relating neural spiking activity to spiking history, neural ensemble, and extrinsic covariate effects. Journal of neurophysiology, 93(2):1074-1089, 2005.
Anne E Urai, Brent Doiron, Andrew M Leifer, and Anne K Churchland. Large-scale neural recordings call for new insights to link brain and behavior. Nature neuroscience, 25(1):11-19, 2022.
Lukas von Ziegler, Oliver Sturman, and Jobannes Bohacek. Big behavior: challenges and op- portunities in a new era of deep behavior profiling. Neuropyychopharmacooyyy 46(1):33 44, 2021.
Tristan Walter and Iain D Couzin. Trex, fast multi-animal tracking system with markerless identification, and 2d estimation of posture and visual fields. Elife, 10:e64000, 2021.
Eric Y Wang, Paul G Fahey, Zhuokun Ding, Stelios Papadopoulos, Kayla Ponder, Marissa Weis, Andersen Chang, Taliah Muhammad, Saumil Patel, Zhiwei Ding, et al. Foundation model of neural activity predicts response to new stimulus types. Nature, 640(8058):4704444 477, 2025a.
Rui Wang, Dongdong Chen, Zuxuan Wu, Yinpeng Chen, Xiyang Dai, Mengchen Liu, Yu-Gang Jiang, Luowei Zhou, and Lu Yuan. Bevt: Bert pretraining of video transformers. In Proceedings the IEEE/CVF conference on computer vision and patlern recognition, pages 14733-14743, 2022.
Ziyue Aiden Wang, Balint Kurgyis, Susu Chen, Byungwoo Kang, Feng Chen, Yi Liu, Dave Liu, Karel Svoboda, Nuo Li, and Shaul Druckmann. Brain-wide analysis reveals movement encoding structured across and within brain areas. Nature Neuroscience, pages 1-12, 2025b.
Richard A Warren, Qianyun Zhang, Judah R Hoffman, Edward Y Li, Y Kate Hong, Randy M Bruno, and Nathaniel Sawtell. A rapid whisker- based decision underlying skilled locomotion in mice. Elife, 10:e63596, 2021.
Matthew R Whiteway, Dan Biderman, Yoni Friedman, Mario Dipoppa, E Kelly Buchanan, Anqi Wu, John Zhou, Niccolò Bonacchi, Nathaniel J Miska, Jean-Paul Noel, et Partitioning variability in animal behavioral videos using semi-supervised variational autoencoders. PLoS computational biology, 17(9); 1009439, 2021.
Jiarui Xu and Xiaolong Wang. Rethinking self-supervised correspondence learning: A video frame-level similarity perspective. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 10075-10085, 2021.
Yufei Xu, Jing Zhang, Qiming Zhang, and Dacheng Tao. Vitpose: Simple vision transformer baselines for human pose estimation. Advances in neural information processing systems, 35:38571-38584, 2022.
Daniel LK Yamins, Ha Hong, Charles F Cadieu, Ethan A Solomon, Darren Seibert, and James J DiCarlo. performance-optimizze hierarchical models predict neural responses in higher visual cortex. Proceedings of the national academy of sciences, 1 111(2)):66198822 2014.
Shaokai Ye, Anastasiia Filippova, Jessy Lauer, Steffen Schneider, Maxime Vidal, Tian Qiu, Alexander Mathis, and Mackenzie Weygandt Mathis. Superanimal pretrained pose estimation models for behavioral analysis. Nature communications, 15(1):5165, 2024.
Fisher Yu and Vladlen Koltun. Multi-scale context aggregation by dilated convolutions. arXiv preprint arXiv:1511.07111 2015.
Hang Yu, Yufei Xu, Jing Zhang, Wei Zhao, Ziyu Guan, and Dacheng Tao. Ap-10k: A benchmark for animal pose estimation in the wild. arXiv preprint arXiv:2108 12617, 2021.
Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini. and Yonghui Wu. Coca: Contrastive captioners are image-text foundation models. arXiv preprint axX1v:2005.01917, 2022.
17

---

Published as a conference paper at ICLR 2026
Eric Zhan, Albert Tseng, Yisong Yue, Adith Swaminathan, and Matthew Hausknecht. Learning cali- bratable policies using programmatic style-consistency. International Conference on Machine Learning, pages 11001-11011 PMLR, 2020.
Yizi Zhang, Hanrui Lyu, Cole Hurwitz, Shuqi Wang, Charles Findling, Yanchen Wang, Felix Hubert, Alexandre Pouget, Erdem Varol, and Liam Paninski. Exploiting correlations across trials and behavioral sessions improve neural decoding. Neuron, 2025a.
Yizi Zhang, Yanchen Wang, Mehdi Azabou, Alexandre Andre, Zixuan Wang, Hanrui Lyu, The International Brain Laboratory, Eva Dyer, Liam Paninski. and Cole Hurwitz. Neural encoding and decoding scale. arXiv preprint 1rix::2500408201,,,, 2025b.
Long Zhao, Nitesh B Gundavarapu, Liangzhe Yuan, Hao Zhou, Shen Yan, Jennifer J Sun, Luke Friedman, Rui Qian, Tobias Weyand, Yue Zhao, et al. Videoprism: A foundational visual encoder for video understanding. arXiv preprint 2301 2024.
Chengjie Zheng. Tewodros Mulugeta Dagnew, Liuyue Yang. Wei Ding, Shiqian Shen, Changning Wang, and Ping Chen. Animal-jepa: Advancing animal behavior studies through joint embedding predictive architecture in video analysis. In 2024 IEEE International Conference on Big Data (BigDuta), pages 1909-1918. IEEE, 2024.
Tianxun Zhou, Calvin Chee Hoe Cheah, Eunice Wei Mun Chin, Jie Chen, Hui Jia Farm, Eyleen Lay Keow Goh, and Keng Hwee Chiam. Constrastivepose: A contrastive learning approach for self- supervised feature engineering for pose estimation and behavorial classification of interacting animals. BioRxiv, pages 2022-11, 2022.
Chengxu Zhuang, Jonas Kubilius, Mitra Hartmann, and Daniel L Yamins. Toward goal-driven neural network models for the rodent whisker-trigeming system. Advances in Neural Information Processing Systems, 30, 2017.
18

---

Published as a conference paper at ICLR 2026
Appendix
A DATASETS
All datasets used for this study were collected in compliance with the relevant ethical regulations (see the references for each dataset).
A.I IBL
This dataset (BLL 2023) from the International Brain Lab (IBL) and consists of head-fixed mice performing decision-making task (IBL et al. 2021 a). Two cameras-'left (60 Hz) and 'right' (150 captuue roughly orthogonal side views of the mouse's face and up- per trunk during each session. Frames are downsampled to 256 320 pixels for labeling and video storage. We accessed the raw videos and neural activity under the CC-BY 4.0 license using these instructions: tttt::://inttbraiiine external/data http://aaaaaaeeeeeaae release http:///ootttarreee.eeeeeaaaaaaa hwtp://....iiaaaaeeeeeeaama.eeee 31310144 under the CC-BY 4.0 license.
html] Frames for pretraining are available at
Pose estimation. Frames were reshaped during training els. Two keypoints were labeled per view, one for each paw. We accessed
256 X 256 pix-
the initial pose estimation labels from the public repository /2023
https://
a https///////aagrratttttteeee Q1 Biderman Whiteway et al/ ibl wtt..//iwooaaa......aaaaanaaaaa. 7e79e865 E2ff- 4709 17770002200 b203- (7133-77cdccc55 7wc.acc4666...... zip under the CC-BY 4.0 license. This dataset contains 6,071 labeled train frames from 35 animals and 1,446 labeled test frames from 10 animals, all at 128 x 102 pixel resolution. Despite the large number of labeled frames, we observed poor performance in sessions with bright lights or other distractors. Additionally, the low resolution often obscured fine details, for example complicating discrimination of individual paws when close together. To address these limitations, we retrieved the full-resolution frames (1280 x 1024 for left view, 640 x 512 for right) from the raw videos and downsampled them to 320 256 pixels. This higher resolution revealed occasional labeling errors, which we manually corrected. We then added 1,437 newly labeled frames from 15 additional animals, creating an expanded training set of 7,609 frames from 50 animals. The updated pose estimation dataset available https: :ttps://bbt //ibl brain wide map publicc s3. ama zonaws com www.cooowgggggg.eceee Tags/ 2026 Q1 Wang
ib. wwweeTraackin..trainn...
zip
under the CC-BY 4.0 license.
Action segmentation. Four action classes are labeled for the paw closest the camera: (1) still; (2) grooming: (3) turning the wheel; and (4) fidget (any movement that is not grooming or wheel turning). We accessed the initial action segmentation labels from/https: //doi. org/10 6084/ 4/ m9. f99ghag 27479760 v1 under the CC-BY 4.0 license. This dataset contains 1,000,000 frames from 10 animals, of which 14,107 are labeled.
We expanded this dataset as the original study (Blau et al. 2024) only used five animals each for train- ing and testing. First, we trained an ensemble of five TCN- based action segmentation models on all 10 existing animals. We applied these models to new batch of 53 sessions and calculated the variance predicted probabilities across all models for each frame. The 19 sessions with the highest average ensemble variance (indicating where models disagreed most) were selected for further labeling. We then labeled 36,009 additional frames from these sessions. For the analyses in this paper, we selected the subset of all labeled sessions that are included in the BEAST pretraining sessions, and split these into train (32,521 frames from 18 animals) and test sets (7,786 frames from 5 animals). The updated action segmentation dataset is available https: 2ttFaaggg22222011___2___
-ww.-ddeemapppp---p Yu ________ll ib1 zig under CC-BY 4.0 license.
13 httaaaaccnwwsscccco/oa/eeeeeeeee vi deodctt iwt.oActtinSeeeee www.iionntteeeennnnnnnnmn http://ttttttttttaaaannnnaaaaaae
Neural encoding, For the neural analysis we use subset of the IBL repeated site dataset (IBL et al. 2025b). This dataset consists of Neuropixels recordings collected from 10 labs with standardized experimental pipelines. The recordings target the same five brain regions across all mice: VISa (primary visual cortex), CA1 and DG (hippocampus), and LP and PO (thalamic nuclei). We evaluate neural encoding models five randomly selected sessions. Moreover, used the trial-aligned
19

---

Published as a conference paper at ICLR 2026
neural activity data, taking 2 seconds of activity aligned wheel movement onset. We binned the spikes every 20 ms to get a total of 100 bins per trial. We also filtered out the low firing rate neurons by setting minimum threshold of 2 Hz. For each trial we randomly select 100 video frames (out of a possible 120) to fit the downstream neural encoding models, resulting in an effective sampling rate of 50 Hz to match the neural data.
A.2 IBL- WHISKER
We localize the whisker pad using anchor keypoints on the nose and eye, following the procedure in IBL et al. 2022). We use the same sessions and neural activity the "IBL" dataset.
A.3
MIRROR-MOUSE
Head-fixed mice
circular treadmill while avoiding a moving obstacle (Warren et al. 2021).
The treadmill had a transparent floor and a mirror mounted inside at 45%, allowing a single camera to capture two roughly orthogonal views (side view and bottom view via the mirror) at 250 Hz. The camera was positioned at a large distance from the subject (~11.1 m) to minimize perspective distortion. Frames are 406 396 pixels and reshaped during pose estimation training 256 256 pixels. Seventeen keypoints were labeled across the two views including seven keypoints on the mouse's 's body per view, plus three keypoints on the moving obstacle. The full training dataset consists of 789 labeled frames across 10 animals; the test dataset consists of 253 labeled frames across three animals. We accessed the labeled pose estimation dataset from https: /doi. / org/100 O. 6084 4 m9. www.figgarre.cm 24 249933151555 v1 under the CC-BY 4.0 license. Frames for pretraining are available at ut/ttps://dol/ org/ 6084 m9. figshare. attaarr...33... 31320529 under the CC-BY 4.0 license.
A.4 MIRROR-FISH
Mormyrid fish of the species Gnathonemus petersii swam freely in and out of an experimental tank, capturing worms from a well (Biderman al., 2024; Pedraja al.] 2025). The tank had side mirror and a top mirror, both at 45 providing three different views seen from single camera at 300 Hz. The camera was placed ~1.7 m away from the center of the fish tank to reduce distortions. Frames are 384 x 512 pixels and reshaped during training to 256 x 384 pixels. Seventeen body parts were labeled across each of three views for a total of keypoints. The full training dataset consists of 373 frames across three animals; the test dataset consists of 94 frames across three animals. We accessed the labeled pose estimation dataset from https: / /doi. org/10. 608 / m9. www.ffiggee.a.........o.o.r. 249 w1 under the CC-BY 4.0 license. Frames for pretraining available at
6084 l.......sshee www.ffggggee.... 313207773133 under the CC-BY 4.0 license.
A.5 CRIM13
The Caltech Eesident-Intruder Mouse (CRIM13) dataset (Burgos- et al. 2012) consists of two mice interacting in an enclosed arena, captured by a top-view camera at 30 Hz. Frames are 480 x 640 pixels and reshaped during pose estimation training to 256 x 256 pixels. Seven keypoints were labeled on each mouse for total of 14 keypoints (Segalin et al. 2021). The full training dataset consists of 3,986 frames across four resident mice; the test dataset consists of 1,274 frames across the same four resident mice but a different set of intruder mice. The original dataset is available at (httpss 4emt 10t10 We accessed the labeled pose estimation 24w.:rree.2999888 v under the CC-
7/data.
dataset from http: :ttp::://doooooo http:///oo.oorg/////// org/10, 6084/m9 BY 4.0 license. Frames for pretraining are available at /https:://datt org 10 6084/00440000 m/m9 figshare. 31321213 under the CC-BY 4.0 license.
A.6 CALMS21
The Caltech Mouse Social Interactions (CalMS21) dataset (Sun et al. 2021a), like CRIM13, consists of two mice interacting in an enclosed arena, captured by top-view camera at 30 Hz. The dataset consists of videos with tracked poses and frame-level behavior annotations. Four behavior classes are labeled: attack, investigation, mount, and other (i.e., none of the above). The full training dataset consists of 506,668 frames across 68 videos; the test dataset consists of 262, frames across 19 videos. We accessed the pose estimates, TREBA features (Sun et al. 2021b), and behavior annotations from hotps: /oi. http://oooooogg/////// wwt.oo/g//oooo. 22000 1991 under the CC-BY 4.0 license. Frames for pretraining are available at httph:tppss.tt doi org 6084/ m9. figshare... 31337953 under the CC-BY 4.0 license.
20

---

Published as a conference paper at ICLR 2026
A.7 FACEMAP
Head-fixed mice were free run on an air-floating ball in darkness (Syeda et al., 2024). single infrared camera captured one of several side or front views of the mouse's s face and upper trunk during each session at 50 Hz. Fifteen keypoints were labeled across the face (mouse, nose, whiskers, eyes) and paw. Neural activity was recorded across visual and sensorimotor areas using two- photon calcium imaging at Hz. Approximately 30,000 to 50,000 cells were recorded in a given session. In our encoding task, we predict the 128 neural principal components following Syeda et al. (2024). We evaluate neural encoding models on five randomly selected sessions. The publicly available data did not contain additional videos, we only fine-tuned neural encoding models with this dataset. We accessed the raw videos, pose estimates, and neural activity from https
wtt.eaaa............. 23712957 under the CC-BY-NC 4.0 license.
Table 3: Number of
frames utilized across tasks, with number of source videos in parentheses. Pretraining frames unlabeled, and the number includes both anchor frames and temporal context frames. Pose estimation and action segmentation frames are labeled; these models are trained across multiple videos. Neural encoding frames have matched neural activity (output) for each time point of behavior (input); these models are trained on single videos, since the neural populations change from one session to the next.
Pose estimation
Action segmentattinn
Neural encoding
Pretraining
Irain/val
train/val
train/val
lest
IBL IBL ctal
2025a
270,025 (77)
7,609 (128)
(19)
35,521 (18)
7,786 (5)
338,760 (5)
42,720 (5)
Mirre-mouse Warren et
2021
94,272 (17)
789 (17)
253 (5)
Misor.issh
Biderman et
2024
47,943 (28)
373 (28)
94 (10)
CRIM13
Burgos-Aniin Arizzu et al
2012
99,926 (36)
3,986 (37)
1,274 (19)
CaIMS2I (Sn et
202
91,087 (70)
506,668 (68)
262,107 (19)
Facemmp Syeda
2024
1,790,200 (5) 447,550 (5)
B BEAST IMPLEMENTATION
BEAST utilizes a standard VIT-B/16 architecture (Dosovitskiy et al.) 2020), and combines a masked autoencoding and temporal contrastive learning loss. This approach is also taken by VIC-MAE (Her) nandez et al. 2024), and we introduce key adaptations and simplifications to make BEAST suitable for applications behavioral neuroscience, which we elaborate on more in the following sections.
B. ARCHITECTURE
We selected the "base" VIT-B/16 architecture over other VIT variants because it is expressive enough to capture rich frame-level information while remaining computationally efficient for training and inference on long videos. While VIC-MAE employs a pooled attention layer to transform patch embeddings into a fi8-dimensional vector that serves the global image representation for the temporal contrastive loss, we use the standard CLS token (Table 7). This CLS token is passed through nonlinear projector consisting of four components: linear layer, Batch Norm (necessary for stable training. Fig. 5). ReLU activation, and a final linear layer.
The BEAST backbone initialized using weights pretrained on ImageNet with a masked autoencoding loss. This model has exceptional zero-shot performance on neural encoding, outperforming all other baselines, and is further improved with domain-specific pretraining (Table We find other pretrained Saabonnes-INOvy (Oquab et al. 2023) and CLIP (Radford et al. 2021 also have strong zero-shot performance, but do not significantly improve upon MAE+ MaaleeemaeNNe et (Table indicating this is a reasonable pretrained backbone from which to start our own domain-specine pretraining (see Fig. 9
and Table 15 for similar results on pose estimation and action segmentation, respectively).
BEAST incorporates time through its temporal contrastive loss, which efficiently captures information across frames. We explored the performance of VIDEOMAE (Tong et al. 2022), a related video model pretrained on Kinetics-400 (Kay et al. 2017). We found that the frozen VIDEOMAE backbone does outperform the nume-based VIT-MAE on the neural encoding task (Table [3). Interestingly BEAST, which applies additional domain -sencfc pretraining to VIT-MAE, still outperforms VIDEOMAE. This raises the intriguing question of whether further domain -spciffc pretraining of VIDEOMAE could surpass BEAAS performance. We view this as important direction for future work.
21

---

Published as a conference paper at ICLR 2026
Training Contrastive Accuracy
0.6
Accuracy
0.5
0.4
0.3
0.2
0.1
Step
10k
20k
30k
40k
w/o Batch Norm
w/ BatchNorm
Figure 5: Effect of Batch Normalization on contrastive training accuracy. Training contrastive accuracy improves significantly with the use of Batch Normalization (BatchNorm) in the nonlinear projection head. Models trained with BatchNorm exhibit smoother learning curves and achieve higher final accuracy compared those without BatchNorm. "Accuracy" defined the fraction of anchor frames in batch where the corresponding positive frame has logit score higher than all other negative frames.
B.2 TRAINING
We discuss frame selection and sampling strategies, data augmentations, and global pooling strategies below. We apply the MAE loss uniformly across all frame types (anchor, positive, negative), whereas VIC-MAE only applies the MAE loss to anchor frames. The global batch size is to 2048, distributed across 8 Nvidia A40 GPUs. We use the AdamW optimizer with a weight decay of 0.05. The learning rate scheduled using PyTorch's OneCycleLR scheduler, with base learning rate of 5 10-3 The maximum learning rate is computed max_Ir Ir base_Ir Ir global batch size with pet_start set to 0.15 and div_d factor set to 10. We train all models for 800 epochs.
256
Frame selection strategy. Animal behavior videos often contain extended periods of inactivity repetitive behaviors. Pretraining VIT models all available frames would capture redundant information and increase computation time. Instead, we extract diverse frames exhibiting distinct poses (to optimize the masked autoencoding loss) while preserving local temporal relationships (to leverage the temporal contrastive loss). We first downsample all frames to 32 x 32 pixels and calculate motion energy, defined as the absolute pixel-wise differences between consccutive frames. We climinate frames in the bottom 50th percentile of motion energy, retaining only those with significant movement. We then apply k-meaan clustering to the remaining downsampled frames, with the number of clusters matching our target number of anchor frames per video (e.g., 600). For each cluster, we select the frame closest to the cluster center, along with its immediate predecessor and successor in time, which serve positive examples for the contrastive loss (for total of, e.g., 1800 frames per video). This methodology ensures a high-quality, diverse dataset for efficient pretraining and outperforms random frame selection in the neural encoding task (Table
Table 4: Frame selection strategy ablation. We pretrain models using either random selection ("Random") " or the PCA+k-means strategy deseribed above ("Selected"). We evaluate the representations of these models using zero-shot performance on the neural encoding task using bits per spike (BPS) metric. report mean and standard deviation of BPS across five test sessions.
IBI.
LBL-whisker
Method
TCN
RRR
TCN
RRR
ViT-M (Random)
0.311 + 0.107
0.168 + 0.091
0.319 + 0.084
0.147 + 0.026
BEAST (Random)
0.319 0.1004
0.176 0.094
0.319 + 0.081
0.138 + 0.035
BEAST (Selected)
0.337 0.103
0.177
0.076
0.317 0.083
0.138
0.029
22

---

Published as a conference paper at ICLR 2026
Frame sampling strategy during training. Once have diverse set of training frames, must construct batches during training. As stated in Sec. 3 for a batch of size B we randomly select B/2 anchor frames, which can originate from any and all videos. For BEAST, each anchor frame x is paired with a positive frame randomly selected from x All other frames serve as negative frames, including frames from the same video. Due the frame selection strategy described above, frames from the same video will be visually distinct and not interfere with the contrastive loss (Fig. 6). This batch construction procedure is distinct from VIC-MAE, which allows any two frames from the same video to be positive pair, while only frames from different videos are negative pairs. We find our approach outperforms the VIC-MAE approach in the neural encoding task (Table ( This sampling strategy only applies to BEAST; the VIT-M models do not contain the contrastive loss, and we only train them with the anchor frames.
Table Frame sampling strategy ablation. We pretrain models using either the VIC-MAE or BEAST frame sampling strategy; both models use the superior "Selected" frame selection strategy. evaluate the representa- tions of these models using zero-shot performance on the neural encoding task using the bits per spike (BPS) metric. We report the mean and standard deviation of BPS across five test sessions.
IBL
IBL-whise whisker
Method
TCN
RRR
TCN
RRR
VIC-MAE (IN+PT)
0.331 0.103
0.141 0.080
0.289 0.055
0.127 + 0.033
BEAST (IN+PT)
0.337 0.103
0.177 + 0.076
0.317 + 0.083
0.138 + 0.029
cross-session
Embeddings colored by session
same session
Figure 6: Temporal contrastive visualization. UMAP embeddings (MeInnes et al. 2018 of 32 principal components (PCs) all anchor frames used for pretraining on the IBI. datasct. Fach anchor frame colored by the session it was sampled from. The left and right columns show example frames drawn from the same session and from different sessions, respectively. The sampled frames largely visually distinct, even frames next each other UMAP space (e.g.. frames B and C, or D and E, the left-hand side).
Data augmentation. The default data augmentation procedure (He et al. 2022) applies random resized crop to 244 x 244 pixels with a crop ratio between 0.2 and 1.0, followed by random
23

---

Published as a conference paper at ICLR 2026
horizontal flip with probability 50%. We explored extended augmentation strategy that adds further transformations: rotation up to 45 degrees and color jittering (brightness=O.4, contrast=0.4, saturation=0.4, hue=0.1), in addition to the crop and flip. We compare the performance of BEAST using both the default and the extended augmentation strategy. The additional augmentations achieve performance similar the default setting (Table 6 so use the default throughout the paper.
Table 6: Data augmentation ablation. We pretrain models using either default extended data augmentations. We evaluate the representations these models using zero-shot performance on the neural encoding task using the bits per spike (BPS) metric. We report the mean and standard deviation of BPS across five test sessions.
IBL
IBL IBL-whisker
Method
TCN
RRR
TCN
RRR
BEAST (default data aug)
0.337 + 0.103
0.177 + 0.076
0.317 +: 0.083
0.138 +: 0.029
BEAST (extend data aug)
0.328 + 0.102
0.163 = 0.081
0.314 0.077
0.150 + 0.042
Pooling strategy. The CLS token serves as global frame representation, effectively pooling infor- mation across all spatial positions into single latent vector. To explore alternative pooling strategies during pretraining, we conducted additional experiments by pretraining models using mean pooling and attention pooling of the patch embeddings, then evaluating performance on the ncural encoding task (Table 77.. The ablation results clearly demonstrate that the CLS token is the most effective aggregation method for pretraining.
Table 7: Pooling ablation. We pretrain models using either the CLS token, or mean or attention pooling of the patch embeddings, to aggregate information across the image for the temporal contrastive loss. We evaluate the representations of these models using zero-shot performance the neural encoding task using the hits per spike (BPS) metric. We report the mean and standard deviation of BPS across five test sessions.
IBL
Whisweesker
Method
TCN
RRR
TCN
RRR
BEAST (mean pooling)
0.321 0.104
0.159
0.080
0.302 + 0.082
0.128 + 0.025
BEAST (attention pooling)
0.323 + 0.100
0.141 4 0.076
0.307 1 0.072
0.130 + 0.031
BEAST (CLS token)
0.337 + 0.103
0.277 + 0.076
0.317 + 0.083
0.138 + 0.029
B.3
HYPERPARAMETRA DETAILS
The BEAST objective combines reconstruction and contrastive losses. During the early stages of training, important for the model to focus on accurately reconstructing the input, so the reconstruction loss should dominate. As the model learns to capture low-level pixel structure, the contrastive loss gradually becomes more important. It acts as a regularizer, encouraging the model to learn higher-level temporal representations rather than overfitting local pixel patterns.
The weighting factor for the contrastive loss, X, plays a crucial role during pretraining: too large and the model will not reconstruct the input well; too small and the model does not reap its regularizing benefits. Through hyperparameter tuning based on neural encoding performance (on the validation set), we set X 0.01 for all datasets, except IBL and mirror-mouse, where we set À 0.02. This choice ensures the reconstruction loss is emphasized in the early phases of training, while the contrastive loss naturally takes over as reconstruction error decreases toward the end, eliminating the need for an annealing schedule.
The masking strategy for the reconstruction loss also significantly impacts performance. We found that an aggressive mask ratio of 0.75 works effectively across various tasks, for both VrT-M and BEAST. When fine-tuning BEAST for neural encoding models (IN+FT or IN+PT+FT) we tested mask ratios of 0.75 and 0.9, with 0.9 performing better on validation data. These 0.9-ratio models are used for the final fine-tuning results presented our tables and figures.
B.4 COMPUTATIONAL EFFICIENCY COMPARISON
To compare the computational efficiency of image (VIT-M, BEAST) versus video models (VIDEOMAE). recorded three metrics for each model: runtime (ms per batch), Giga Float-
24

---

Published as a conference paper at ICLR 2026
ing Point Operations per Second (GFLOPS), and memory required for forward pass with batch size one. All experiments were run using the fvcore library on a single A100 GPU. Since runtime varies across batches while GFLOPS and memory are deterministic, deviation of runtime across 32 batches. We benchmark two modes: "pretrain", which uses patch masking (0.75 for image models, 0.9 for VideoMAE, following optimal settings from the respective papers); and "finetune", which omits patch masking as relevant for our downstream tasks.
report mean and standard
BEAST and VIT-M show comparable performance across all metries, with BEAST having slightly longer runtime and larger memory footprint due to the nonlinear projector used in the contrastive loss (not substantial enough affect GFLOPS). VIDEOMAE requires substantially more resources due to processing 16 consecutive frames batch element: during pretraining, it requires 2.5x runtime, 3.5x GFLOPS, and >2x memory compared to BEAST. These differences are even more pronounced during finetuning when patches are not masked: VIDEOMAE, requires 11 x runtime, 11 x GFLOPS, and 7x memory.
Table Computational efficiency comparison. We benchmark runtime (mean and standard deviation across 32 batches), GFLOPS, and memory usage for BEAST, VIT-M, and VIDEOMAE during pretraining (with patch masking) and finetuning (without patch masking). All measurements are for forward pass with batch size of single A100 GPU.
Runtime (ms/batch)
GFLOPS
Memory (MB)
Model
Finetune
Pretrain
Finetune
Pretrain
Finetune
Pretrain
ViT-M
5.43 0.70
7.6 + 2.43
17.59
9.80
369.0
332.72
BEAST
6.21 = 0.94
7.9 = 2.26
17.59
9.80
409.1
333.40 40
VideoMAE
71.51 + 3.50
20.16 + 2.47
199.49
35.24
2955.1
831.61
B.5 FEATURE VISUALIZATIONS
We computed PCA patch embeddings from 100 randomly sampled frames in the test set and visualized the first three components as RGB channels of two example frames. Compared DINOv2- whose embeddings emphasize broader semantic structures- the BEAST pre-trained model captures finer-graines details that are critical for neural encoding, pose estimation and behavior segmentation. In contrast, the ViT-M model pretrained only on ImageNet produces patch embeddings that appear noisier and less structurally coherent.
Mouse perceplusl decision-msking (IBL 2025)
Mouse locomotion (Warren et al., 2021)
Resident-inttuder assey (Burgoe-Artizzu et al., 2012)
Freely swimming momyrid fish (Bidernan al., 2024)
orignalranee
Figure Visualization the first three PCA components across models.
25

---

Published as a conference paper at ICLR 2026
C NEURAL ENCODING
C.I FEATURE REPRESENTATIONS
11 keypoints in the publicty available dataset: left and right paws, two edges of the tongue, two edges of the lick spout, nose, and four edges of the pupil. For the Facemap dataset we use keypoints in the publicly available dataset: three whiskers, four points on the nose, four corners of the eye, and the one visible paw.
Keypoints. For the IBL dataset
Whisker pad motion energy. We localize the whisker pad in the IBL dataset using anchor keypoints on the nose and eye. We then compute the motion energy of the whisker pad the absolute pixel-wise differences between consecutive frames, resulting in a one-dimensional representation at each time point (IBL et al. 2022).
Principal Component Analysis. We compute PCA per-session basis using all frames in the video. A subset of the resulting PCs are used for neural encoding. See Sec. C.6 for information on dimensionality ablation experiment.
CEBRA. CEBRA (Schneider et al. 2023) is contrastive learning approach that provides baseline which is complementary to the VIT-M models pretrained on ImageNet with a masked autoencoding objective. Similar to our PCA approach, train individual unsupervised CEBRA model for each session using the convolutional neural network option and the default of fset 1 mmodel.
DINOv2. For the DINOv2 model (Oquab et al. 2023), we extract the CLS embedding for each frame using frozen pretrained backbone, and use these input to the encoding models. We did not pretrain this model ourselves, but rather used the model checkpoint available at https //huggingface.ccm co/
VIT-M variants. For the VrT-M models, we extract the CLS embedding for each frame using frozen pretrained backbone, and use these as input to the encoding models. Alternative approaches could include: (I) using patch embeddings with a multi-head attention pooling layer (as in action segmentation work), or (2) fine-tuning the backbone itself while using either CLS or patch approach for pose estimation). We expect these alternative approaches would improve performance and plan to explore them in future work.
embeddings (similar
VrT-M (IN): VIT-M model pretrained ImageNet using a masked autoencoding loss. We did not pretrain this model ourselves, but rather used the model checkpoint available at bttps://hugginggf.t
VIT-M (IN+PT): Initialized with the ImageNet- weights, then further pretrained on dataset-specific frames.
VIT-M (IN+PT+FT): Initialized with the dataset-specific pretrained weights (IN+PT), and then further fine-tuned on a single session.
All training (except VIT-M (IN)) is performed as described in Appendix B
ViT-C variants. For the VIT-C (IN+PT) model, start with the VIT-M (IN) weights then continue pretraining on -ataset-specific frames with the temporal contrastive loss only. Therefore there is no VrT-C (IN) variant. We did not perform additional session-specific fine-tuning, there is no ViT-C (IN+PT+FT) variant.
The BEAST model variants follow the same naming pattern as VIT-M, with one exception: there is "BEAST (IN)" variant, BEAST requires video frames rather than just static images for pretraining.
BEAST variants.
C.2 REDUCED RANK REGRESSION
IBI.. For the IBL dataset, followed the Reduced Rank Regression (RRR) setup described in (Posani et al. (025).. We trained all models using the I.-BFGS optimizer and set the rank constraint to 3. To denoise the neural signals, we applied 1-dimensional smoothing filter to the neural activity. The
hyperparameter search (Sec. C.4 was conducted over the ranges specified in Table
Facemap. For the Facemap dataset, we followed the setup described in (Syeda et al. 2024). using the implementation provided in the official Facemap repository To deal with different neural and
26

---

Published as a conference paper at ICLR 2026
Table 9: RRR model hyperparameters for IBL dataset.
Hyperparameter
Value Range
Output Dim
Number of Neurons
Rank
3
Optimizer
L-BFGS
Learning Rate
Log (ngf-oroomm.... 2)
L2
100
behavioral sampling rates, this model first resamples the behavioral timestamps to match the neural timestamps, and then fits a Reduced Rank Regression model using low-rank SVD. We adopted the default rank of 32 as used in the original implementation. The Lambda parameter refers to the regularization strength, which we set to a relatively low value to avoid over wver-penalizing the weights. The output dimensionality was set to 128, corresponding to the number of neural principal components used in the model. Model parameters estimated via a closed-form least squares approach. The hyperparameters we used are specified Table 10
Table 10: RRR model hyperparameters for Facemap dataset.
Hyperparameter
Value Range
Output Dim
128
Rank
32
Lambda
le-6
C.3 TEMPORAL CONVOLUTION NETWORK
We used the same implementation of the Temporal Convolution Network (TCN) to process frame embeddings for both the IBL and Facemap datasets, based on the official Facemap repository The convolutional kernel operates along the temporal dimension of the input (behavioral) data. To deal with different neural and behavioral sampling rates, this model resamples the resulting latent representation at neural timestamps using nearest-neighbor indexing. The TCN model was trained for 300 epochs using the AdamW optimizer, with learning rate decimation (multiplied by 0.1) applied at epochs 120 and 200. The hyperparameter search (Sec. was conducted over the ranges specified in Table
Table 11: TCN model hyperparameters:
'ILL dataset; Facemap dataset
Hyperparameter
Value Range
Output Dim
Number of Neurons". 128*
Learning Rate
Log- Nog-Uniform(5e-5 2c-3)
Optimizer Weight Decay
AdamW
le-4
C.4 HYPERPARAMETER SELECTION
For the IBL data, we divided the trials into train (80%), validation (10%), and test (10%) sets. For the Facemap data we followed the experimental setup described the original study (Syeda et al.) 2024]: the session is split into ten blocks; the first 75% of each block is assigned to the training set; the following 3 seconds are excluded to remove data leakage due to autocorrelation in behavior and neural activity; and the final set of frames from the block are assigned to the test set. There is no validation set. For both the RRR and TCN models, we conducted hyperparameter searches separately for each feature type identify the best-performing configurations. Specifically, we performed 30 runs of randomly selected byperparameters per model type, evaluating performance on the validation set (IBL) or test set (Facemap) using evaluation metric specific to each dataset: bits per spike (BPS) for IBL and variance explained for Facemap. The only exception was the RRR model for Facemap, where the parameters were fixed according the original implementation. We select the
27

---

Published as a conference paper at ICLR 2026
model with the best performance on the validation (IBL) test (Facemap) set, and report results on the test set.
C.5 PATCH EMBEDDINGS
The output of the VIT encoder consists of CLS token embedding RCLS E R and patch token embeddings 2 E I2 xD where D = 768 the embedding dimension and is the total number of image patches (N 196 for 224 x 224 input frame). We compare the CLS token and attention- pooled patch embeddings as inputs for neural encoding (see implementation details in E.4). and find that the CLS token outperforms the patch embeddings (Table (2). Given its lower dimensionality and superior results, we adopt the CLS token representation for all subsequent encoding tasks.
Table 12: Comparison of CLS and patch embeddings with TCN encoder. We report the mean and standard deviation of BPS across five test sessions.
Method
IBL.
IBL-whisker
BEAST (CLS)
0.337 0.103
0.317 0.083
BEAST (patch)
0.278 + 0.065
0.288 0.070
C.6 DIMENSIONALITY ABLATION EXPERIMENTS
One of the most important hyperparameters for the PCA, CEBRA, and VIT models is the la- tent/embedding dimensionality. To thoroughly explore performance across this parameter, we tested these models using various dimensionality values. For a given dimensionality k. we used different approaches: (1) for PCA, selected the top k principal components; (2) for CEBRA, retrained the model with & latent dimensions; and (3) for VIT models, due to computational constraints, we first trained the full fi8-dimensional models, then applied PCA to the embedding space and selected the top VIT principal components. For each feature, model type, and dimensionality k, we fit downstream neural encoding models using the complete hyperparameter search described previously. Figure 2 reports the best result for each model, though notably BEAST outperforms all baselines across all dimensionality values (Fig.
A
Carrea Enceding Dirmmsionally
atteseer CNN Encoding Dimersionalit
Pacamap CAN Encoding Symensionally
PCA
04-
622
TESAA
0.35
CERILL
#5UST
BRAS-
0.0
0.20
VC.4
0.10
Erabodding
Indecding Cimeaaoon
Enoudding
B
a troocligg gimendoraaity
encoding L'ieensiraaly
Pacorrop 9tN encoding sinnaiii
IA
FCA
0.2
025
0.35
Dl Ber pP0
(N-T7)
0.00
625
Hilitt
0:55
0.0
C5854
6110
8/n
0730
006
0.00
Inbesteg Dieenoa
Emelddingg Dinersion
Figure 8: Encoding performance as a function embedding dimension. BEAST outperforms all other baselines over embedding dimensions spanning several orders of magnitude, demonstrating the superiority of its representations for any given dimensionality. Results for Keypoints and Motion Energy are included at their respective dimensionalities for each dataset. The Facemap encoding results for I-dimensional data performed poorly and included NaN values some sessions, so we excluded them from figure.
28

---

Published as a conference paper at ICLR 2026
C.7 EXTENDED NEURAL ENCODING RESULTS
We collect all neural encoding results in Table 13 The values for PCA and CEBRA correspond to the 100-dimensional results in Fig. 8 the values for VIT-M and BEAST correspond to the full 768-dimensional models.
Table 13: Neural encoding results across feature types and models. All ViT-hased models use CLSS tokens from a frozen backhone. "IN" refers to a model pretrained with ImageNet weights; "IN+PT" refers to models that are initialized with ImageNet--retrained weights then further pretrained on experiment-speecfic data;" "+FT" refers to models that are initialized with pretrained weights based on what comes before "+" then fine-tuned on individual sessions. For IBL and IBL-whisker we report the mean and standard enror of the mean of BPS over N=842 neurons across five test sessions. For Facemap we report the mean and standard deviation Variance Explained of the first 128 Principal Components of neural activity across five test sessions.
TBL. (BPS)
whiskrr (BPS)
Facemap (Var Exp)
Features
TCN
RRR
TCN
RRR
TCN
Keypaints
0.189 0.008
0.260 + 0.011
0.224 + 0.047
0.403 + 0.077
Motion energy
0.086 = 0.006
0.113 + 0.007
PCA
0,260 0.010
0.309 = 0.012
0.212 0.009
0.272 + 0.011
0.177 = 0.061
0,407 = 0.090
CEBRA
0.239 + 0.010
0.293 0.012
0.204 0.009
0.260 + 0.011
0.143 + 0.046
0.278 0.064
DINOv2 (LVD-142M)
0,183 0.008
0.294 1 0.013
0.138 0.007
0,289 1 0.012
CLIP (WIT-400M)
0,182 0.008
0.292 + 0.012
0. 0,131 = 0.006
0.268 + 0.012
VidcoMAE (Kinetics-100)
0,189 0.010
0.318 = 0.013
0.133 0.007
0,298 + 0.013
VIT-M (IN)
0,192 0.009
0.321 0.013
0.129 0.007
0.301 1 0.012
0.254 a 0.061
0.446 1: 0.100
VIT-M (IN-PT)
0.172 + 0.008
0.331 + 0.013
0.148 0.007
0.311 + 0.013
VYT-M (IN-FT)
0,233 0.000
0.345 + 0.014
0 = 0.010
0.330 + 0.011
0.200 + 0.051
0.461 + 0.000
VIT-M (IN-PT+FT)
0,285 0.011
0.347 - 0.014
0.235 - 0.010
0.328 1 0.013
VIT-C (IN+PT)
0,137 0.008
0.314.0.013
0.120 0.006
0,283 1 0.011
BEAST (IN+PT)
0,206
0.012
0.133 0.006
0.309 0.013
BEAST (I++F))
0.261
0.347
0.240 = 0.010
0.331 + 0.013
0.281 = 0.051
0.461 = 0.089
BEAST (IN+P+FFT)
0,252 0.011
0.347 0.014
0.234 1. 0.010
0.326 1 0.013
D POSE ESTIMATION
D.1 MODELS
The pose estimation models consist of backbone and a head. The backbone is either a ResNet-50 (He al.) 2016) or a VIT-B/16 (Dosovitskiy et al. 2020), both producing feature maps of shape [N, H, W for a given image, where N denotes the feature dimension and H, W denote the height and width of the feature maps. All models employ an identical linear upsampling head that begins with a P Pixelse Shuffle layer, reshaping the feature maps to [N/4, 2H, 2w]. These reshaped features then pass through two consecutive 2D convolutional transpose layers with kernel size (3, 3) and stride (2, 2). doubling the spatial resolution after each layer. The head architecture omits batch normalization and nonlincarities between these layers. The output passes through a 2D softmax function, generating a normalized heatmap for each keypoint.
D.2 TRAINING
We divided the labeled data into training (95% and validation (5%) sets, with test frames coming from entirely held-out videos. We used a batch size of eight frames. Data augmentations include random crops, rotations, motion blur and histogram equalization. Models were trained for 300 epochs, with validation loss recorded every five epochs. For final evaluation, we selected the model with the lowest validation loss. Training utilized an Adam optimizer (Kinmma and Ba 2014) with an initial learning rate of 0.001 for the ResNet and 5 for the transformers, which was halved at epochs 150, 200, and 250. To facilitate feature learning, we kept the backbone frozen during the first 20 epochs of training before allowing for full end-to-end optimization. The loss function is the mean square error between each predicted heatmap and a ground truth heatmap constructed from labeled data.
D.3 PRETRAINED BACKBONES
We test several additional VIT backbones pose estimation to validate our BEAST results:
29

---

Published as a conference paper at ICLR 2026
Segment Anything (SAM) (Kirillov et al.) 2023): huggingface wwt.aa/aco.///aaeccooccoooooaoaa bbsse
checkpoint from
https://
DINO (Caron et al.
2020):
checkpoint from
https:/hugginffacce.
DINOv2 (Qqua et al.
Tacebookk/dnov2-bo 2023); checkpoint from
https:/hhuginggacee. co/
These backbones are trained using the same procedure as the ResNet-50 and BEAST models (see above). We find other pretrained backbones mostly outperform the ResNet-50 baseline (Fig. ( SAM is generally the least performant backbone. DINOv2 consistently outperforms DINO across all datasets, and BEAST achieves the lowest pixel error in most cases (only outperformed by DINOv2 in the Mirror-fie fish dataset). These results demonstrate BEAST'S experiment-spee specifcc pretraining framework can surpass state-of- state-of-the-art general purpose vision foundation models for pose estimation.
Reshies-00 (AP10KO
BAM
DINO
DINOV2
BEAST
IBL 100-Frame
100-Frame
ChIM1s
Mirtorfish
19%
20.
3
Porl Eiror
Ensemble std
Ensemble
Ensembtee
Ensemble
Figure 9: Pose estimation of fine-tuned vision foundation model backbones. evaluated ResNet-50 (pretrained on AP10K), SAM, DINO, DINOv2, and BEAST hackbones on pose estimation datasets. We evaluate the these models using pixel error at various ensemble standard deviation thresholds, with values in the table representing the percentage of keypoints the chosen threshold. Smaller values indicate smaller but more challenging subsets of keypoints (see main text).
Reshies-0
VIT-M
BEAST
IBL 1000 Frame Anetune
Minor 789 Frame Finetune
CRIN13 1000 Frama Finetune
Minon 373 Frame Fineture
g8
1..0%
22
0
Bnsemble std dov
Figure 10: Pose estimation performance with more training frames. The results in Fig. demonstrate pose estimation performance of various models using just 100 labeled frames. To ensure those comparisons hold on larger datasets we trained models (three random seeds per backhone) using the maximum number frames the training dataset or 1000 frames, whichever is smaller. We still see consistent gains for both the transformer architecture pretrained on ImageNet (red) and our pretraining strategy (blue, green).
D.4 DEEPLABCUT BASELINE
For the DeepLabCut baseline (version 3.0.0) we trained models using an ImageNet-pretraine ResNet- 50 backbone. To properly isolate differences between DeepLabCut and Lightning Pose algorithms, we matched training frames, batch size, learning rate schedule, and number of epochs. For all other hyperparameters
used the DeepLabCut package defaults (c.g., data augmentation). We train models using three different train/val data splits, and ensure these splits exactly match those used for the Lightning Pose models.
30

---

Published as a conference paper at ICLR 2026
Lightning Pose outperforms DeepLabCut on the IBL, Mirror- and Mirror-fist fish datasets, while DeepLabCut outperforms Lightning Pose on CRIM13 (Fig. D... Notably, our pretrained BEAST backbones outperform both DeepLabCut and Lightning Pose across all four datasets.
Reshiet-00 (AP10K) DespLabbCtt (Roshett-8) BEAST
IBL 100-Frame
Miroe
100-Frame
Mirrorfich 100-Frame Resr
20.8% 20.8%
40
5
Pow Eroe
Ensemble std
Ensemble
Ensembbe
Ensemble
Figure 11: Pose estimation of DeepLabCut (DLC). Figure conventions as Fig.
E ACTION SEGMENTATION
For action segmentation we consider variety of input feature types and modeling approaches. We first consider range input features: keypoints, PCA the raw video frames (fit on the same frames as ViT pretraining; we use 768 PCs for downstream models), and VIT-based CLS tokens. For each of these feature types fit both linear and nonlinear (temporal convolution network, TCN) models. We also train a TCN model on the VIT patch embeddings, with a multi-head attention pooling layer to reduce the dimensionality of the features before entering the TCN.
E.1 MODELS
Linear model. The linear action segmentation model uses ID temporal convolution layer, followed by a linear layer that maps from the number of features the the number of action classes, followed by a softmax. There are no other forms of nonlinearity in the model.
Temporal convolution network. The nonlinear action segmentation model is dilated TCN (Lea et al. 2016) with 2 dilation blocks. Each dilation block consists of a sequence of 2 sub-blocks (ID) convolution layer + Icaky ReLU nonlinearity - dropout with probability-0.10) as well as a residual connection between the input and output of the dilation block. The dilation of the convolutional filters starts with for the first dilation block, then increases by a factor of for each additional dilation block. This results in a larger temporal receptive field the model gets deeper, allowing for learning of longer range dependencies (Yu and Koltun 2015).
Both models utilize weighted cross entropy loss function, with class weights inversely proportional to the class frequency the training data.
E.2 TRAINING
Each video is split into sequences of 500 (IBL.) 100 (CalMS21) frames. We divide the data into training (90%) and validation (10%) sets, with test frames coming from entirely held-out videos. We use batch size of 16 sequences. Models were trained for 500 epochs using the Adam optimizer (Kingma and Ba, 2014).
E.3 HYPERPARAMETER DETAILS
For each model type-linear and nonlinear-and each feature type, we run a hyperparameter search across all combinations of parameters Table 14 using three random weight initializations. The hyperparameter combination with the best Fl score on the validation data, averaged across the three seeds, is selected for evaluation on the test set. For this hyperparameter combination, we train with two additional seeds and report results in the figures and tables averaged across five seeds.
E.4 TEMPORAL CONVOLUTION NETWORK WITH MULTI-HEAD ATTENTION POOLING
We aggregate the patch embeddings from the BEAST encoder using a multi-head attention pooling layer (Lee et al. 2019), which produces single pooled embedding per frame input to the TCN.
31

---

Published as a conference paper at ICLR 2026
Table 14: Action segmentation hyperparameters. "TCN only
Hyperparameter
Value Range
Learning rate
le-3, le-4, le-5 0.1 9. 17, 33 16, 32, 64, 96 2
Dropout
Temporal filter length
Number of hidden units"
Number of hidden layers"
This layer uses a learnable query S R X with patch embeddings 2 E RND as keys and values. To capture motion-related features, we further concatenate the frame-to-frame difference of the pooled embeddings as additional input to the TCN (Fig. 12). We fixed the number of attention heads in the pooling layer to 8 for all models.
Our previous model utilized CLS embeddings, which allowed for an efficient workflow: we processed videos through the transformer backbone and saved the CLS embedding from each frame as separate file. These pre-computed embeddings could then be directly loaded to train the downstream TCN classifier without requiring video reading during training. However, patch embeddings present significantly larger memory requirements, making disk storage infeasible. To address this challenge, developed a data loading pipeline that performs end-to-end processing: it loads video frames, passes them through the transformer backbone, and feeds the resulting patch embeddings directly to the TCN model within the same training loop. This integrated approach, while computationally more intensive, eliminates the need for intermediate storage. Due to these increased computational demands, we modified the training procedure for the multi-head attention pooling models follows.
The pooling layer and TCN classifier were trained jointly for 200 epochs on CalMS21 and 100 epochs on IBL using the Adam optimizer (Kingma and Ba| (2014) with an initial learning rate of 1c-3. The epoch counts were determined through separate experiment that withheld a subset of validation videos and monitored the validation Fl score until convergence. Training followed cosine-annealing schedule with warm restarts (Loshchilov and Hutter] 2016) configured with To = 34, Tmult 2 and mmmin = 5.-5. We used 6 or 8 NVIDIA A40 GPUs for the training, each with a batch size of 2. giving effective batch size of 12 or 16. The sequence length was fixed at 500. Due to the increased compute required to train these models, we fixed the TCN hyperparameters to be those found for the ELS--baed model for each dataset.
Time
VIT
lsamable
K
temporal convolutional network
embedsing
Mubi head attention
pooling
behavior class
pootec embedding
Figure 12: Multi-head attention pooling TCN head for action segmentation. A: Per-patch embeddings from the BEAST encoder are pooled using a multi-head attention layer, where a learnable query token attends to the patch embeddings to produce a single pooled embedding for each frame. B: Temporal differences between consecutive pooled embeddings class of the center frame the sliding window, window size of shown for illustration; the actual window size was tuned as a hyperparameter.
concatenated as additional input the TCN, which predicts the behavior
32

---

Published as a conference paper at ICLR 2026
Table 15: Action segmentation results IBL and CalMS21 datasets across feature types and models. All ViT-based models use frozen a frozen backbone; "CLS" indicates models trained on the global CLS embeddings. while "patch" indicates models trained with a multi-head attention pooling layer applied to the patch embeddings. "IN" refers to a model pretrained with ImageNet weights, "IN+PT" refers to models that are initialized with ImageNet-pretraanes weights then further pretrained experiment-specffic data. We report the mean and standard deviation of Fl on test data across five random train/val splits.
Linear
TCN
Dataset
Features
Features
Features, Features
Features
Features, A Features
IBL
Keypoints
0.54 = 1
0.55 L.5--
0.86 1.4e-3
0.88 2
PCA
0.54 = 4 3
0.55 = 6.3e-3
0.61 + 1.0c-2 2
0.71 + 2.8e-3
VIT-M (IN) (CTSS
0.68 + 1.4e-3
0.68 7.0e-4
0.78 7.4e-3
0.79 2.7e-3
VIEM (IN+PT) (C1.8)
0.74 + 2. 2
0.72 2.0c-3
0.78 4.0e-3
0.80 + 4.6r-8
BEAST (CT.S)
0.70 + 7.9e-3
0.69 2.5e-3
0.80 3.0e-3
0.81 6.9e-4
DINOv2 (putch)
0.77 2.7e
VII-M (IN) (patch)
0.81 3.7e-3
VIT-M (IN+PT) (patch)
0.85 3.6e
VIT-C (IN+PT) (patch)
0.79 4.6e
BEAST (IN-PT) (patch)
0.87 5.1e-3
CaIMS21
SimBA
Goodwin al.
2024
0.75 5.10
0.53 8.
0.78 8e
2.99-3
TREBA Sun et al. 202
0.29 = 1
0.30 1.5e-3
0.70 4.6e-
0.
PCA
0.10 0 3.1e-
0.10 = 3.2e-3
5.
0.18 0.18:4.5 1.5e-
VIE-M (IN) (CI.S)
0.50 + 2.2e-2
0.52 + 1.3e-3
0.53 1.2e-2
0.60 + 2.8r-8
VIT-M (IN+PT) (CLS)
0.60 +: 5.5e-3
0.60 + 1 1
0.60 9.1e-3
0.65 + 2.2e-3
BEAST (IN-PT) (CLB)
0.53 +: 4.6e-3
0.51 + 4.0e-2
0.58 5.2e-3
0.63 2.7e-3
DINOv2 (putch)
0.68 4
VIT-M (IN) (patch)
0.74 2 2.9e-
VIT-M (IN+PT) (patch)
0.82 + 9.5e
VYT-C (IN+PT) (patch)
0.53 2.9c 2
BEAST (IN-PT) (patch)
0.81 + 7.7e-3
F SELF-SUPERVISED LEARNING METHODS FOR ANIMAL BEHAVIOR
Self-superviser learning (SSL) techniques from computer vision have increasingly been adapted to the study of animal behavior. These approaches use SSL to extract useful feature representations from pose, image, or video data, which are then applied to downstream tasks such action segmentation.
Pose-based approaches. Trajectory Embedding for Behavior Analysis (TREBA) (Sun et al. 202 employs a multi-task self-supervised framework that uses trajectory reconstruction as its primary objective through Trajectory Variational Autoencoders (Co-Reyes al.) 2018 Zhan al. 2020). TREBA additionally requires the TVAE embedding to decode various auxiliary tasks consisting of simple data transformations designed by domain experts, with the resulting embeddings serv- ing as input to downstream action segmentation models. Variational Animal Motion Embedding (VAME) (Luxem et al.) 2022) uses a sequential variational autoencoder embed pose sequences into a latent space by reconstructing both current and subsequent time steps. Unlike TREBA, VAME applies clustering to the learned embeddings, creating fully unsupervised action segmentation pipeline. ContrastivePose (Zhou et al., (022) leverages geometric augmentations (flipping, rota- tion, translation) of pose coordinates to generate positive pairs for contrastive learning. followed by fine-tuning action segmentation tasks. Bootstrap Across Multiple Scales (BAMS) (Azabou et al. 2023) employs dual temporal convolutional networks with different receptive field sizes to create complementary short- and long-term embedding spaces. BAMS introduces novel training objective requiring prediction of future action distributions rather than specific action sequences, with validation the MABe benchmark (Sun al. 2023) across multiple tasks including action segmentation and mouse strain classification.
Image and video-based approaches. Selfee (Jia et al. 2022 constructs composite RGB frames from 3-frame grayscale video sequences (assigning each frame to separate color channel) and applies standard image-base based contrastive learning techniques, demonstrating effectiveness on action segmen- tation and anomaly detection. Mueller et al. (2025) adapt pretrained V-JEPA model (Bardes et al.)
33

---

Published as a conference paper at ICLR 2026
2023), SSL approach specialized for video understanding, through domain- adaptive pretraining primate behavior datasets, validating their approach on behavior recognition tasks. Similarly, Animal- JEPA Zheng al. 2024) modifies the V-JEPA training strategy with domain-specifie masking techniques and validates on mouse behavior classification tasks.
Distinction from prior work. While these methods share conceptual similarities with BEAST, our approach is distinguished by its general frame-based training objectives and comprehensive evaluation across neural activity prediction and pose estimation tasks, in addition to the standard action segmentation task. Furthermore, since the pose-based methods described above rely on pose estimates as input, BEAST could potentially enhance their performance by providing higher-quality pose estimation preprocessing step.
G BEAST WORKFLOW
We give an overview of the BEAST workflow for a new user, highlighting steps where BEAST enhances traditional workflow.
STAGE DATA COLLECTION
Collect behavioral videos
Optionally collect simultaneous neural recordings
The use of BEAST does not affect this step of the workflow.
STAGE 2: BEAST PRETRAINING
This step will be skipped traditional workflow.
Extract frames from unlabeled videos (typically 100K frames)
Train BEAST transformer backbone these frames (~112 hours 8 GPUs)
This step does not require manual annotation
STAGE 3: DOWNSTREAM APPLICATIONS
Option A: Pose estimation
Annotate 100-1000 frames from 5-50 videos with keypoints (can be different from pretrain- ing set)
Fine-tune pose estimation model using BEAST backbone
Key advantage: improved performance over existing backbones (Fig.
Run inference on new videos extract keypoints
Option Action segmentation
Annotate 1000-5000 frames per behavior, idcally across 5-10 videos, with action labels (independent of pose annotation)
Fine-tune action segmentation model using BEAST backbone
Key advantage 1; skip pose estimation pipeline entirely
Key advantage 2: equivalent or better performance compared to pose estimates (Fig
Run inference new videos to extract frame-by-fran actions
Option C: Neural encoding
No manual annotation (neural activity provides the "labels")
Fine-tune neural encoding model using BEAST backbone (video input, neural activity output)
Key advantage 1: skip pose estimation pipeline entirely
Key advantage 2: improved performance over existing behavioral features (Fig
Predict neural activity from video
34

---

Published as a conference paper at ICLR 2026
H BROADER IMPACTS
The BEAST framework enables more efficient extraction of meaningful information from video data, potentially accelerating behavioral neuroscience research with several beneficial outcomes. By reducing the need for extensive human labeling while improving accuracy, BEAST can democratize advanced video analysis capabilities for laboratories with limited resources. This efficiency could accelerate basic science discoveries that underlie advances in biomedical applications, neurological disorder treatments, and improved understanding of brain function.
While BEAST is developed primarily for behavioral neuroscience studies using animal subjects, the underlying technology could potentially be repurposed for human video analysis, raising several concerns:
Surveillance capabilities: The improved ability to track and categorize behaviors could enhance surveillance technologies, potentially infringing on privacy rights if deployed without appropriate oversight.
Bias and faimess: As with any Al system trained on specific datasets, BEAST-derived models may perform differently across demographic groups if applied to human subjects, potentially perpetuating biases in downstream applications.
Resource inequality: While pretrained BEAST model can improve the efficiency of downstream tasks, the computational requirements for pretraining itself may limit access to this technology for under-resourced institutions, potentially widening existing disparities in research capabilities.
35


Reviews:
[{"review_id": "7mJKGwJ3Ee", "reviewer": "Reviewer_6xX2", "summary": "The authors propose BEAST, a transformer model for behavioral analysis with neuroscientific applications. The model uses a combination of a masked and contrastive objective on raw video frames, and can be used for three downstream tasks (neural encoding, action segmentation, pose estimation). The authors claim benefits in neural encoding over baseline methods for video/behavior encoding, improved zero shot neural encoding performance over a ViT baseline, improvements in keypoint estimation over a ViT baseline, and in action segmentation vs. the SimBA model. The supplementary material contains a number of additional experiments, variations and ablations of the proposed approach.", "questions": "1. I might have missed this in the numerous appendix results, but is there an ablation on the role of the two training objectives (masking vs. contrastive learning)? Are both required for getting high performance?\n2. The method seems incremental in terms of the pre-training objective, which is also transparently communicated in the appendix. Could the authors elaborate again what the most conceptually close prior algorithm to BEAST is, and clearly delineate what the changes are they added, and what the effect of this change is? This could also be made a bit clearer in the main paper.\n3. Judging from the std/confidence intervals in most tables, the difference between ViT variants and BEAST is slim. Could the authors run stats on the different tables and communicate transparently which advances are actually signficant?\n4. Is the keypoint benchmark, especially for the baseline model, in Figure 3 grounded in the literature? Same question for Figure 4.\n5. For neural encoding, Table 4 suggests that frozen backbones perform almost on par with BEAST. What happens if e.g. VideoMAE would be finetuned on the same dataset? I find it likely that this might already surpass BEAST performance. Was such an experiment (using an existing state of the art pre-training paradigm on the same data) done?\n6. There is no mentioning that the code (training and/or inference) and/or the model weights are planned to be open sourced. Is this planned?\n\nHappy to discuss further.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 2, "strengths": "- Ablations and comparisions are very extensive and clearly outlined\n- Paper is very well written. Dense, but very informative and to the point.\n- The figures have high quality and are far above the average of ICLR papers\n- The model has convincing performance across the different tasks.", "weaknesses": "While the method presented seems powerful for a variety of tasks, the evaluation as-is is currently too weak; it seems like BEAST is largely building on existing video-pre-training schemes. This in itself might be fine, but from e.g. Table 1 and 4 it seems that even frozen backbone models are suitable for solving the neural encoding tasks.\n\nThe authors need to better delineate what their methodological contribution is, and what it adds over a strong baseline model. I would e.g. consider doing a bit of finetuning with DINOv2, or VideoMAE, or TCN, on the authors' dataset, and check if this might surpass the performance of BEAST on one of multiple of the tasks; the tables suggest that this is likely. The paper might still be a nice contributions, but should be rephrased slightly; right now it appears like the exact combination of training objective is causing a performance gains, which I am critical about.\n\nSome weaknesses I see are:\n\n1. Figure 2: Choice of baselines is questionable. Motion energy and and PCA are good as \"naive\" baselines, but the CEBRA result feels obvious given that the backbone for BEAST is a ViT model, and for CEBRA the authors used a small convnet. The better comparison would be to use the same backbone, and only vary the training objective. The authors should also pull the baseline model (VIT-M) in Table 1 into this figure/barplot for better contextualization what their particular training objective contributes to this task.\n2. Figure 3: Same concerns regarding baselines. BEAST is compared against a ResNet-50 model. What about other pose estimation algorithms that are widely used, e.g. DeepLabCut, which is also referenced in the text? These would be relevant comparisons for practitioners.\n3. More baselines for action segmentation could be discussed and benchmarked; right now the result seems anecdotal.\n4. While the results in the figures are nice, it seems like the authors created these benchmark situations and evaluated; it would be good to discuss how the authors validated their baselines, and if there are any previously published benchmarks BEAST can be applied on (and please excuse if there is a supplementary result that I overlooked already doing this)\n5. The authors cite \"temporal contrastive learning (Chen et al., 2020)\", which is not the right reference for temporal contrastive learning. TCL and PCL might be better algorithms to cite, or even CPC (2018).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tphLAi2vot", "reviewer": "Reviewer_gTWQ", "summary": "The paper introduces BEAST (BEhavioral Analysis via Self-supervised pretraining of Transformers), a domain-tailored framework that pretrains experiment-specific ViTs on unlabeled animal-behavior videos by combining masked autoencoding with a temporal contrastive objective. A key ingredient is a frame selection/sampling strategy designed for static backgrounds with movement-driven variation. The pretrained backbone is evaluated on three tasks across multiple datasets/species. BEAST delivers competitive or superior results to strong baselines while reducing reliance on large labeled sets or keypoint-first pipelines.", "questions": "see weakness", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Cleverly tailors self-supervised pretraining to behavioral videos (static background, movement-driven variation) via selection/sampling strategies. \n3. Evaluates frozen-backbone features on segmentation and zero-/few-shot neural encoding, with systematic ablations.\n3. Clear writing.\n4. On action segmentation, BEAST bypasses pose-estimation training while matching or exceeding keypoint-based systems, reducing months-long annotation pipelines.", "weaknesses": "1. Cross-domain generalization. The approach emphasizes single-paradigm pretraining; comprehensive tests across cameras/environments/species would strengthen claims of generality. \n2. Efficiency. The paper positions BEAST as more efficient than native video models but does not report controlled FLOPs/memory/runtime.\n3. Interpretability. Beyond showing CLS superiority for pretraining, adding feature visualizations or sensitivity analyses linking learned features to anatomical/behavioral elements would aid scientific adoption.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "p1QW10QrI6", "reviewer": "Reviewer_ys7M", "summary": "This paper introduces BEAST, a self-supervised learning framework that combines masked autoencoding (MAE) and temporal contrastive learning to pretrain vision transformers on animal behavior videos. The authors demonstrate the utility of BEAST across three key neuro-behavioral tasks: neural encoding, pose estimation, and action segmentation. The method is evaluated on multiple datasets and species, showing consistent improvements over existing baselines, especially in settings with limited labeled data.", "questions": "1. Why does BEAST largely underperform compared to DINOv2 on the IBL-whisker dataset (in terms of RRR)? Would pretraining DINOv2 on IBL data improve its performance further?\n\n2. Have you considered using only patches containing the animal (via segmentation or attention) for behavior classification? This may help reduce background noise.\n\n3. Why was DINOv2 not included as a baseline for behavior classification and pose estimation? Its strong performance in other domains warrants its inclusion.\n\n4. How does BEAST compare to DINOv3, which was released after this submission?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "Comprehensive Evaluation: The paper thoroughly evaluates BEAST across multiple tasks (neural encoding, pose estimation, action segmentation) and datasets (mice, fish), demonstrating its versatility and robustness.\n\nNeural Encoding Focus: The inclusion of neural encoding as a downstream task is a significant and compelling contribution, linking behavioral video analysis directly to neural activity—a fundamental goal in neuroscience.\n\nEfficient Use of Unlabeled Data: BEAST effectively leverages unlabeled video data through self-supervised pretraining, reducing the need for extensive manual annotation.\n\nAblation Studies: The paper includes extensive ablations (e.g., frame selection, pooling strategies, loss weighting) that provide valuable insights into the design choices.", "weaknesses": "1. Limited Discussion of Related Self-Supervised Methods: While the paper discusses general self-supervised learning (SSL) methods like MAE and contrastive learning, it does not adequately address existing SSL approaches specifically designed for animal behavior. For example:\n\n    ConstrastivePose: A contrastive learning approach for self-supervised feature engineering for pose estimation and behavorial   classification of interacting animals.\n    Tianxun Zhou, Calvin Chee Hoe Cheah, Eunice Wei Mun Chin, Jie Chen, Hui Jia Farm, Eyleen Lay Keow Goh, Keng Hwee Chiam.\n    bioRxiv 2022.11.09.515746; doi: https://doi.org/10.1101/2022.11.09.515746\n\n    Selfee: Self-supervised Features Extraction of animal behaviors.\n    Yinjun Jia, Shuaishuai Li, Xuan Guo, Junqiang Hu, Xiao-Hong Xu, Wei Zhang.\n    bioRxiv 2021.12.24.474120; doi: https://doi.org/10.1101/2021.12.24.474120\n\n    Domain-Adaptive Pretraining Improves Primate Behavior Recognition.\n    Felix B. Mueller, Timo Lueddecke, Richard Vogg, Alexander S. Ecker.\n    arXiv:2509.12193; doi: https://doi.org/10.48550/arXiv.2509.12193\n\n    A more thorough discussion of these works would better contextualize BEAST’s contributions.\n\n2. Performance on IBL-Whisker Dataset: The results show that BEAST does not outperform DINOv2 on the IBL-whisker dataset (Table 4). The authors should discuss why this might be the case (e.g., DINOv2’s stronger pretraining or architectural advantages?). Also, whether further pretraining DINOv2 on IBL data would close or widen the gap.\n\n3. Missing DINOv2 Baselines for some tasks: While DINO is included in pose estimation and DINO v2 is evaluated in neural encoding, DINOv2 is not evaluated for behavior classification or pose estimation. It is confusing. Given its strong performance in other domains, it should be included as a baseline for a fair comparison.\n\n4. Technical Novelty: The core components of BEAST (MAE + contrastive learning) are based on ViC-MAE, which is not novel. The main contributions lie in the application-specific adaptations (e.g., frame sampling, domain-specific pretraining) rather than the underlying architecture.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "D24NnKPGmq", "reviewer": "Reviewer_2yb7", "summary": "The paper suggests to use InfoNCE contrastive loss to learn better representations from the video, where similar and different embeddings are defined based on the time proximity.\nThe model in pre-trained jointly with InfoNCE and reconstruction (demasking)  loss (MSE) and later fine-tuned for a specific task, such as a)neural encoding; b) pose estimation; c)action segmentation.", "questions": "1. *(most crucial)* Could you provide some qualitative experiments showing that the temporal contrastive loss actually selects different enough negative pairs? Have you explored any alternative sampling strategies for the frames ?\n2. In lines 482-484 you write *\"The black-box nature of VIT embeddings presents interpretability challenges in scientific contexts where transparent representations like pose estimates are often preferred.\"*. Could you please clarify how exactly is your model more interpretable?\n3. How do you match frequencies between behaviour and neuronal recordings for Fig2? Spiking neuronal data should be much higher frequency than behavioural timeseries and calcium data should be slower, especially for Facemap dataset, where behavioural videos should have been 50Hz to catch whiskers motions.\n4. What is the intuition/ interpretation behind Bits Per Spike metric, especially in the calcium imaging settings?\n5. Why SLEAP and DeepLabCut are not used as baselines for pose estimations? (DeepLabCut might be seen as pretrained ResNet version but having simple UNet comparison is missing, which both of them do, is missing). If you have tried it and it did not work on 100 frames, I think it should be mentioned in the appendix given how common these tools are in the community.\n6. How would you expect you method to work for animal tracking for in the wild (e.g. monkeys tracking in the jungles), where the motion is much less periodic and more various but the background could be stationary as well (e.g. wind would move the leaves)? \n\nMinor:\n1. You report training *\"taking approximately 25 hours on 8 Nvidia A40 GPUs\"* in line 202. This is without ImageNet pretraining and simultaneously on all datasets (e.g. mice and fish together)? Why does it take so long if you start from the weights after ImageNet pretraining? Do you have a warmup stage before cosine annealing?\n2. Why do you train CEBRA models not jointly across sessions ? (as mentioned in App. C)\n3. For Fig.4 - could you show non-normalized confusion matrixes (panel B) because it looks like for the middle just the majority class gets switched? Could you also please provide the weighted accuracy ?\n4. For Fig.4 panel A - what is ensembled for PCA? should not it be deterministic if only seeds are varied? If yes - how is there a difference for \"CalMS21\" dataset?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper is very well-written and nice to read\n2. The paper brings recent advances in self-supervised learning toward animal tracking community and tests the proposed method on various datasets from multiple labs.\n3. On some tasks like Action segmentation (table 3) the method shows a clear improvement \n4. The authors consistently provides error bars (SEM or std) in all plots and tables\n5. Realistic evaluation: The limited labeling settings (100 labeled frames as described in page 7 lines 362-363) is close to real life evaluation scenarios.", "weaknesses": "1. A lot of references are missing, I would highly inspire the authors to do another iteration on the related works\nThe paper basically introduced contrastive learning on spatiotemporal video chunck, which have already been done in [1] also using InfoNCE. Additionally, Triplet loss [2] another conceptually close (to attract and repel to the anchor point) and is worth mentioning and maybe benchmarking.\nOn the animal tracking / pose estimation side, SLEAP [3], DeepPoseKit [4], TRex [5] are missing.\nCorresponding baselines (or justification if they are not needed) is missing as well.\n\n2. Its not clear why contrastive temporal loss even works for highly repetitive behaviour - e.g. for the dataset of a mouse walking paws - this is supposed to be very close to each other no matter is the points are close in time or not, as far as the paws are arranges same (\"left or right paw step\"). A bit of this discussion together with the frame selection process is is presented in the Appendix B2 but I do not find it clear enough and Table 4 and 5 often do not show statistically significant improvement (eg the improvement is 3rd digit after the dot, while the std is variable in the 1st digit after the dot, as in Table 4 1st column).\n\n3. The improvements in tables seems to be borderline statistically significant (within std error bars, e.g. table 1 reports the improvement in the 2nd digit behind the dot while $\\pm \\text{std} \\approx 0.1$, same comments for tables 4 and 5 was already done above).\n\n\nReferences:  \n[1] Qian, Rui, et al. \"Spatiotemporal contrastive video representation learning.\" Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2021.  \n[2] Schroff, Florian, Dmitry Kalenichenko, and James Philbin. \"Facenet: A unified embedding for face recognition and clustering.\" Proceedings of the IEEE conference on computer vision and pattern recognition. 2015.  \n[3] Pereira, Talmo D., et al. \"SLEAP: A deep learning system for multi-animal pose tracking.\" Nature methods 19.4 (2022): 486-495.  \n[4] Graving, J. M. et al. DeepPoseKit, a software toolkit for fast and robust animal pose estimation using deep learning. eLife 8, e47994 (2019).  \n[5] Walter, T. & Couzin, I. D. TRex, a fast multi-animal tracking system with markerless identification, and 2D estimation of posture and visual fields. eLife 10, e64000 (2021).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)