Uncovering the Hidden Dynamics of Video Self-supervised Learning under Distribution Shifts
Authors: Pritam Sarkar
Ahmad Beirami
Ali Etemad
Venue: NeurIPS 2023 spotlight (2023)
Keywords: computer vision
self-supervised learning
video self-supervised learning
natural distribution shift
video learning
out-of-distribution generalization
TL;DR: We comprehensively study the behaviour of several popular video self-supervised methods in response to various forms of natural distribution shift, uncovering a series of intriguing findings and interesting behaviors.

Abstract: Video self-supervised learning (VSSL) has made significant progress in recent years. However, the exact behavior and dynamics of these models under different forms of distribution shift are not yet known. In this paper, we comprehensively study the behavior of six popular self-supervised methods (v-SimCLR, v-MoCo, v-BYOL, v-SimSiam, v-DINO, v-MAE) in response to various forms of natural distribution shift, i.e., (i) context shift, (ii) viewpoint shift, (iii) actor shift, (iv) source shift, (v) generalizability to unknown classes (zero-shot), and (vi) open-set recognition. To perform this extensive study, we carefully craft a test bed consisting of 17 in-distribution and out-of-distribution benchmark pairs using available public datasets and a series of evaluation protocols to stress-test the different methods under the intended shifts. Our study uncovers a series of intriguing findings and interesting behaviors of VSSL methods. For instance, we observe that while video models generally struggle with context shifts, v-MAE and supervised learning exhibit more robustness. Moreover, our study shows that v-MAE is a strong temporal learner, whereas contrastive methods, v-SimCLR and v-MoCo, exhibit strong performances against viewpoint shifts. When studying the notion of open-set recognition, we notice a trade-off between closed-set and open-set recognition performance if the pretrained VSSL encoders are used without finetuning. We hope that our work will contribute to the development of robust video representation learning frameworks for various real-world scenarios. The project page and code are available at: https://pritamqu.github.io/OOD-VSSL.

Uncovering the Hidden Dynamics of Video Self-supervised Learning under Distribution Shifts
Pritam Sarkar Ahmad Beirami
Ali Etemad 1.3
Queen's University
2 Vector Institute
Gooole Research beirami@google.co com
(pritam.sarkan ali. eqummad
Abstract
Video self-supervised learning (VSSL) has made significant progress in recent years. However, the exact behavior and dynamics of these models under different forms of distribution shift not yet known. In this paper, we comprehensively study the behavior of six popular self-supervised methods (u-SimCLR, v-MoCo, v-BYOL, u-SimSiam, v-DINO, v-MAE) in response various forms of natural distribution shift, i.e., (i) context shift, (ii) viewpoint shift, (ii) actor shift, (iv) source shift, (v) generalizability to unknown classes (zero-shot), and (vi) open-set recognition. To perform this extensive study, we carefully craft test bed consisting of 17 in-distribution and cuitof-distribution benchmark pairs using available public datasets and a series of evaluation protocols to stress-test the different methods under the intended shifts. Our study uncovers series of intriguing findings and interesting behaviors of VSSL methods. For instance, we observe that while video models generally struggle with context shifts, e-MAE and supervised learning exhibit more robustness. Moreover, our study shows that v-MAE is strong temporal learner, whereas contrastive methods, e-SimCLR and v-MoCo, exhibit strong performances against viewpoint shifts. When studying the notion of open-set recognition, we notice a trade-off between closed-set and open-set recognition performance if the pretrained VSSL encoders used without finetuning. We hope that our work will contribute to the development of robust video representation learning frameworks for various real-world scenarios. The project page and code are available at: https:/ //prii/riii tamqu. gi github. 20/0D--VSSS
Introduction
Self-supervised learning has achieved tremendous success in learning strong and meaningful represen- tations in various video domains, such as action recognition video summarization [9], and video captioning [10, 11, 12). Considering the diversity and complexity of the video domain, real-world deployment of video-based intelligent systems requires to understand the model performance under distribution shifts. Distribution shifts may occur due differences in contextual information, viewpoint, geographical location, and the presence of unknown classes with respect the training data, among others.
3, 5, 5, 6, 7]. action localization [8],
Despite the vast amount of work on VSSL [13, 14, 15, 16, 17, 18, 1, 19, 8, 12], a number of fundamental questions regarding the duito--distribution behavior and dynamics of VSSL methods remain unanswered. To date, there have been no comprehensive studies of these aspects, which we attempt to address in this paper. Specifically, pose and answer the following questions:
How do the learned spatial and temporal representations vary based on different VSSL pretrain- Q1.
ing methodologies? How robust are these representations to different distribution shifts?
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Q2. Considering recent findings about the robustness of finetuning on the generalizability of large language models (LLMs) [20, 21], we pose the question: How does finetuning influence the out-of-distribution (OoD) generalization and zero-shot performance of VSSL.?
Q3. How VSSL. methods perform on open-set problems (where test samples can be from classes previously unknown to the model)? And what is the relationship between performance in closed-set vs, open-set recognition?
Q4. Do different VSSL methods exhibit comparable decision-making patterns ('decision similarity") given the same training conditions? And how is this impacted by different distribution shifts?
To address these questions, we consider six dif- ferent supervuee learning algorithms, (i) SimCLR [22]. (i)) MOCO-v3 [23], (ii)) BYOL
[24], (iv) SimSiam [25], (v) DINO [26], and
0.96
1.00
0.34
6.9
(vi) MAE [27]. along with fully supervised learning, and analyze their behaviors in vari- ous OoD settings. We select these methods to cover three key categories self-supervision, namely contrastive (SimCLR, MoCo-v3), non- contrastive YBOO SimSiam, DINO), and gen- erative (MAE), approaches. In particular, these methods represent fundamental approaches on which numerous variants have been built and therefore represent many existing nuanced solu- tions in the area [2, 3, 4, 5, 6, 1, 19, 8, 12]. For distribution shifts, we study series of different possibilities which occur in videos in real-world settings due to changes in context (e.g.. real vs. mime actions) [28, 29, 30, 31, 32], view- point (e.g,
+M0C0
0.80
2.48
0.99
6.6
288
140
Refurmnno ad coto
VMMMSI
0.96
0.31
687
0.92
VONO
608
6.4
0.83
682
698
Conrert
Whsport
SOATT
Figure We present high-level overview of different video learning methods depicting their performance under different distribution shifts. We normalized the eval. metric of each method to that of the highest performing method in each OoD scenario and averaged the results over multiple OoD datasets. See Appendix D.6 for more details.
vs. ego-centric view) [33, 34, 35, 36], actor (e.g.. (e.g..real-we real-world vs. syn- thetic) [37, 38], and data sources (c.g., different datasets) [39, 40]. Moreover, we evaluate the generalizability of the VSSL methods on unseen classes, i.e., zero-shot recognition [41, 42, 43] and open-set recognition [44, 45, 46, 47]. To perform this investigation, we design a comprehensive study consisting 17 in-distribution and out-of- iitto--distribution (InD-OoD) dataset pairs and examine the dynamics of VSSL under different distribution shifts using a variety of evaluation protocols including linear evaluation, finetuning, unsupervised clustering, and zero-shot recognition. Moreover, for fair comparison, the VSSL methods are pretrained in identical experimental setups. We provide a high-level overview comparing the performance of all methods across all shifts in Figure 1. To the best of our knowledge, this is the first work to study VSSL under distribution shift to this depth.
In summary, our contributions are as follows:
We present the first comprehensive and large-scale systematic investigation of real-world distribu- tion shifts on VSSL methods. Our study encompasses 2 large-scale pretraining datasets, video learning algorithms, 17 InD-OoD dataset pairs, as well as 3 toy datasets. Our thorough evalua- tion involves total of 269 experiments, covering various evaluation protocols including linear evaluation, finetuning, unsupervised clustering, zero-shot recognition, and open-set recognition.
Our study uncovers series of intriguing behaviors and relationships of various VSSL methods, shedding light on the strengths and weaknesses that often go unnoticed in InD validation. Moreover, our investigation provides a comprehensive and impartial perspective on the effectiveness of supervised vs. seff-super pretraining.
2 Related work
The analysis of robustness is well-established area in computer vision, with an extensive body of research dedicated to image-based models [48, 49, 50, 51, 52]. Despite the growing popularity of video models in different domains, detailed comparative studies on their robustness remains

---

under-explored. We come across two recent works [53, 54] that study the behavior of video models (supervised) against synthetic perturbations. An initial study [54] explores the performance of video models against spatial corruptions like noise, blur, color jittering and others. In subsequent work, [53] extends the analysis of video models on temporal perturbations like frame reversal, jumbling, and freezing among others. In particular, our study focuses on real-world distribution shifts, which we find crucial as synthetic perturbations yield little or no consistency to the natural distribution shifts of real data [49, 48]. Moreover, none of the prior works attempts to understand the behavior of video self-supervised methods under various forms of distribution shift.
3 Preliminaries
Contrastive methods. We study two popular contrastive methods: e-SimCLR' and U-MoCo. These methods learn representations by maximizing the similarity of positive pairs while minimizing the similarity of negative pairs using the InfoNCE loss [22, 23, 55], where the similarity scores are calculated using 1.2-normalized cosine distance. The implementation of our v-MoCo is based on MoCo-v3 [23], and it uses momentum target encoder to fetch the key embeddings for the corresponding query embeddings. The momentum encoder is progressively updated from the online encoder using an exponential moving average (EMA) technique [56]. In contrast, u-SimCLR directly uses the online encoder to compute the similarity scores from the embeddings of the given views. Additionally, v-MoCo employs a predictor head, unlike u-SimCLR.
Non-contrastive methods, We study three popular non-contrastive methods v-B v-SimSiam, and D-DINO. These methods learn meaningful representations by minimizing the distance between positive views of given sample, without requiring any negative pairs. e-BYOL uses an architecture similar to e-MoCo, but instead of optimizing an InfoNCE loss, it minimizes the 1.2-normalized cosine distance of the positive views [24, 25]. e-SimSiam is an ablation variant of BOB that does not use a target encoder and directly obtains the embeddings corresponding to the views from the online encoder. The setup of u-DINO is also similar to BOL, but it introduces 'Centering' [26] to prevent against model collapse, instead of using a predictor head like v-BYOL and v-SimSiam.
Generative method. Finally, we also study e-MAE which aims to learn representations through reconstructions of heavily masked inputs. e-MAE employs an autoencoder architecture [27, 1. 57] where the encoder compresses the input information, which is then reconstructed by the decoder while minimizing the I.2-reconstruetion error [27].
We provide additional details and design specifics for each method in Appendix B.
4 Distribution Shifts
Let Py denote a probability distribution over labels. Let Pxly denote a class conditional density of input x given labels y. We draw y from Py and then x from (xly( In the case of InD, there is no distribution shift and the validation set is drawn from the exact same distribution as above. We consider two types of distribution shifts in this paper: input-based and output-based. For Input shift, assume that the class conditional density shifts from Paly to 9zly- To measure the OoD performance, we choose 4xly such that there is no overlap between the distributions of the unlabelled pretraining data and OoD validation sets. For Output shift, we assume that the label distribution shifts from Py to qy- Note that we are particularly interested in cases where the support set of q might be different from that of py: i.e., novel classes appear at test time.
We study four different input shifts, (i) context shift, (ii) viewpoint shift, (ii)) actor shift, and (iv) source shift. Context shift or out-of-context refers to when the scenic background contextual information is misleading or absent, e.g., mime actions. Humans possess a deep understanding of actions and can thus recognize actions without much context. However, vision models tend to rely heavily on contextual cues and background objects to make predictions [28, 58, 59]. As a result, models may generalize poorly when contextual information is absent or different from what the model has been trained on. Viewpoint shift refers to the change in the viewpoint or perspective of the camera that captures the scene. Some of the popular viewpoints in the video domains include third- third-person view, egocentric view, bird's- s-eye view, top-down view, and surveillance camera view,
denotes the video variant of the amage--ased self supervised method.
3

---

Context shift
Actor shift (animal)
Actor + view shift (syn.+-down)
Driving shift (surv.) low res.
Opening door
Reading Source
View shift (ego.)
Carrying Unknown n action (zero-shot)
Holding picture Close set
Ride horse
Open-sel
hummer watermelon
Soccer
Basketball
Badminton
Hockey
Figure 2: Sample video frames of distribution shifts. In these examples, the left frames of each category represent an InD sample and the right frames represent an OoD sample.
among few others. In this work, we examine the generalizability of models trained third-person views to other viewpoints including egocentric, surveillance camera, and top-down views, Further, we study generalizability under actor shift, which occurs when the 'type' of actors changes between training and test sets. These actor-type shifts can include human-to-aninn -o-immal shifts synthetic-to-ret shifts. We consider human actions the real world as InD while animal actions and synthetic videos (e.g., video games, animation) are considered OoD. Lastly, we study the distribution shifts caused due to the changes the data sources, referred as source shift. discussed in [39, 40), datasets of similar classes also exhibit distribution shifts due to the difference in curation strategies, annotation schemes, demographics, geographical locations, and other factors, even when none of the specific shifts mentioned earlier apply. To assess the generalizability of video models under real-world distribution shifts, we also investigate their performance when faced with multiple forms of distribution shifts occurring simultaneously. For example, we evaluate the model's ability to handle 'top-down synthetic videos' which causes both viewpoint and actor shifts concurrently, A few examples presented in Figure 2.
We also investigate the dynamics of video models under output shifts. We perform zero-shot recognition on both regular actions and unusual or rare actions. An example of rare actions would be 'hammering watermelon' as opposed to 'cutting watermelon', as shown in Figure Lastly, we evaluate performance in open-set problems where models are required to distinguish between known vs. unknown classes while correctly predicting the known ones [60, 45, 44, 46]. Deep learning models are known to struggle in such scenarios due to their tendency towards over-confident predictions. We note that existing literature has only evaluated VSSL methods under closed-set scenarios, neglecting the importance of their performance in real-world open-set settings.
Experiment setup
Benchmarks. We use two large-scale video action datasets, Kinetics400 [61] and Kinetics700 [62], for pretraining the video models. The results presented in the main paper Kinetics400 for pre-training, while we provide additional results based on pretraining with Kinetics700 in Ap- pendix D. To evaluate the video models under distribution shifts,
use total of 12 real-world benchmarks, comprising: Mimetics10 and Mimetics50 [28] as the out-of-context validation sets; CharadesEgo [36], Tiny Virat-v2 [63], and Sims4Action [64] to investigate viewpoint shifts (ego- centric, surveillance camera, and top-down views, respectively); ActorShift [65] and Sims4action [64], for actor shifts (animal and synthetic domains, respectively); UCF101 [66] and HMDB51 [67] for source shift: UCF101 [66], HMDB51 [67], and RareAct [68] for zero-shot recognition; UCF101 and HMDB51 for open-set recognition while using Kinetics400 and UCF101 as closed-set. For each OoD validation set, we create an InD training and validation set measure the change in performance. We construct the InD splits using Kinetics400 [61], Kinetics700 [62], MiT-v2 [69], and CharadesEgo [36]. Finally, also use toy datasets to conduct experiments in controlled

---

setups, including ToyBox [70], COIL [34], STL-10 [71]. Additional details of the benchmarks can be found in Appendix C.
Pretraining. To ensure a fair comparison between the VSSL methods, we pretrain them in identical setups with necessary adjustments in hyperparameters. Although some of the methods studied in this work are already available in the literature [3, 5, 72, 11. they follow a variety of experiment setups including different architectures (e.g., R2+1D [73], R3D [72], TSM [74]. VIT [75]), inputs, and others. Therefore, they could not be directly adopted for our experiments and are instead re- re-implemented. Specifically, we keep the encoder, inputs, batch sizes, and maximum number of pretraining epochs the same for all the methods. Furthermore, VSSL methods are tuned based on the InD validation split, with no exposure to OoD validation sets. We use the ViT-Base [75, 76] as the encoder, with a patch size of 4 x 16 Amongst the 6 VSSL methods studied in this paper, all of them use Siamese [77] architecture other than e-MAF. Therefore, the contrastive and non-contrastive methods are fed with 2 random spatio-temporal augmented crops from the original videos, similar to [72]. For fair comparison, the e-MAF which requires single view as input, fed with 3x32x 1122 inputs, while the other VSSL methods are fed with 3 x 16 x 112 inputs per view. Additional details for VSSL pretraining can be found in Appendix B.
Evaluation. To study input-based distribution shifts, we perform linear evaluation and finetuning using the InD training splits, followed by evaluating on both InD and OoD validation splits. We follow the standard protocol used in [72, 2, 4. 6, 78. 79, 7] for linear evaluation and finetuning. To evaluate the models on zero-shot recognition, we follow [41, 42] and jointly train video-text encoders using the Word2Vee [80] word embeddings. Lastly, we follow [45] for open-set recognition. We report top-1 accuracy for multi-class classification and mean average precision (meanAP) for multi-label multi-class classification. Moreover, for open-set problems, we report the area under the ROC curve (AUC) for distinguishing known vs. unknown, and accuracy for measuring closed-set performance. Please find more details in Appendix C.
6 Findings
Q1: Dynamics of learned spatial and temporal representations under distribution shifts
Our experiments on context shift show that video models greatly suffer out-of-context generaliza- tion, as OoD performance significantly drops for all of the methods as presented in Table 1. Notice that o-Supervised achieves the best OoD performance under linear evaluation (Lin.) and w-MAE achieves the best when finetuned (FT), for both benchmarks. Intuitively speaking, the models need to learn strong temporal dynamics to generalize well under context shift, as the background or contextual information may be misleading or absent. v-MAE and o-Supervised show strong temporal learning capabilities as they learn time-variane representations. This is in contrast to the other (contrastive and con-contrastive) methods which encourage learning time-invarian by minimizing the embedding distance of the positive pairs sampled from two different timestamps. Additionally, our statistical analysis in Appendix D.7 confirms the higher robustness of e-Supervised and v- e-MAE against context shift (10 class) in both linear and finetuned setups. Moreover, in context shift with 50 classes, o-Supervised exhibits more robustness in linear evaluation, while e-MAE is more robust when finetuned.
representations
Table 1: Comparison under context shift.
50.
7600
40.0
Context (10 class)
Context (50 class)
35.0
Method
Lin.
FT
Lin.
FT
10.0
OoD
InD
OoD
InD
OoD
InD
OoD InD
10
28.0
r-Supervised 37.0 89.2...
41.2 92.7
15.0 66 3..
19.0 78.1
1--imCLRR
31.4..: 92.5...
35.3 94.4
14.9 71.7 19.1 81.8
(a) Transformation recog. (b) Object recog.
r- MoMooo
29.7
92. mm
39.0 94.2
15.0 74.0... 20,5 81.8
b-BYOL
31.6..r 89.3
41.2 94.6 14.4.aa 71 8.82 21.1 80.8
Figure 3: Comparing VSSL methods dis-
e-SimSiam
30.8. 89.
40.4 93.8
13.8
uss 19.0 78.2
--DINO
34.8.ma 90.4
40.4
93.6
13.0wa 68.7aa 19.0 78.8
entangled (a) temporal and (b) spatial repre- sentations.
M-B
33.3 82.9.t
41.2 95.2
12.3 56, 56.4 26.0 81.4
5

---

To further investigate the performance of the video models in learning temporal dynamics, we conduct a toy experiment
controlled setup using ToyBox [70]. ToyBox consists of distinct temporal transformations of 12 different toy objects and transformations include positive rotations, negative rotations, and translations across the z; y, and 2 axes. We use the frozen VSSL encoders to extract embeddings, which are then used to perform K-means clustering to distinguish the transformations. As the static spatial information is non-discriminative to the temporal transformations, the models have to understand the underlying temporal dynamics to correctly distinguish the applied transformations. The results, presented in Figure (a), show that v-MAE and v v-Supervised outperform the other methods by large margin on this task, confirming that they are better temporal learners. Additionally, to ensure that the superiority of v-MAE and e-Supervised are not due to spatial representations, we also test the frozen encoders in object recognition using the videos of the same static objects (i.e., no temporal information), and we find that e-MAE and o-Supervised are worse among all of them (see Figure (b)). This proves that e-MAE and u-Supervised are indeed strong temporal learners while being weak spatial learners.
These findings raise interesting question: if the contrastive and non-contrastive methods are not effective temporal learners, then how do they perform well on action recognition [72, 5, 81, 2. 79, 82, 3, 7J2 We believe that the strong performance of these methods action recognition can be attributed to the significance of spatial information in classifying human activities. The temporal sequence of representations might not be crucial for these tasks, which possibly allows contrastive and non-contrastive methods to perform well.
Table 2: Comparison of video models in viewpoint and actor shifts.
Viewpoint (ego.)
Viewpoint (surv.tlow res) View+Acl (l-downesyn.)
Actor (animal)
Method
Lin.
FT
Lin.
FT
Lin.
FT
Lin.
FT
OoD
InD
OoD
InD
OoD
InD
OoD
InD
OoD
InD
OoD
InD
OoD
InD
OoD InD
Supervisdd
11.4.: 14.3.m
13.6
17.8
23.1 33.4
24.5
43.4
28.5...
62.3...
44.3
76
64.8... 90.0
69.6 92.3
o-SimCLR
12.7
14.7.m
15.6
19.6
26.1.:. 39.1
28.0
47.5
42.4.:>
6..
8as
63.1 78.8
67.9.. 91.7
73.2 92.9
e-MoCo
13.3
15.0.ma
16.1
19.4
24.8
40.0.
27.6
48.6
41.1.a.
67.9
62.8 80.0
la. 92.2mm
71.4 92.9
--BYOL
12.0
14.4-m:
15.1 18.7
22.7
37.8
24.7
46.9
37.3
65.6...
56.2 78.2 68.3... 91.5mm
71.4
o-SimSiam
11.6iaz
14.
13.7
17.6
13.33.14433..
25.8
45.4
40.0
65. Sum
53.6 76.0 68.
L
71.4
v-DINO
12.0aaa
14.4.m
13.7
17.4
22.3 35.3
24.2
45.1
35.3
50.3
77.5 66.7.t
90,7
71.4 92.2
w-MAE
10.9 13.7-m:
14.2
21.4
23.5 32.0
29.1
48.6
37.8 58.0...
61.1 76.2 59.8..: 9m.
72.6 92.9
The results on viewpoint shifts, presented in Table 2 reveal that contrastive methods (v-SimCLR and v-MoCo) generally achieve superior performance in all three setups in both linear and finetuning schemes. Moreover, based on the statistical analysis in Appendix D.7, e-SimCLR exhibits more robustness all three viewpoint shifts, while v-MoCo exhibits robustness in egocentric and top- down viewpoint shifts. We believe the better performance of e-SimCLR and e-MoCo is due to the availability of negative samples during pretraining which improves viewpoint invariance compared to the other approaches. Note that while aggressive cropping also improves viewpoint invariance as discussed in [83]. we do not notice such improvements in non-contrastive methods although a similar aggressive cropping is applied. It is worth noting that while u-Supervised exhibits a comparable performance to VSSL methods in egocentric and surveillance camera viewpoint shifts, its performance decreases significantly when multiple shifts are applied concurrently, as evident in the case of synthetic top-down viewpoint shift. This suggests that supervised learning may not generalize well in scenarios with more complex and realistic distribution shifts.
For further investigation, we study viewpoint invariance in a controlled setup using the pre- trained VSSL encoders through unsupervised evaluation, avoiding any form of additional train- ing which may alter the learned representations. Moreover, to limit the influence of temporal rep- resentations we use an mage--ased toy dataset COIL100 [34] which is commonly used to test viewpoint invariance [84, 85), as it contains im- ages of similar objects from different angles. Our results presented in Figure (left) show that M-MoCo achieves the best performance, fol-
75.0
83.0
(a) Viewpoint invariance
(b) Low- wwwrresolution
Figure 4: Comparing models on viewpoint invari- ance (a) and low-resolution robustness (b).

---

lowed by D-SimRL.. Moreover, the results show that v-MAE is susceptible to viewpoint shifts as it performs worse. We believe that as v-MAE and D-Supervised are trained with a single view of frame sequences, they are more sensitive to viewpoint shifts. Additionally, amongst the non-contrastive methods, e-DINO shows a better performance. Notice that our results so far have revealed a trade-off between learning viewpoint invariance vs. strong temporal dynamics. While single-stream networks learn better temporal dynamics, Siamese frameworks learn better viewpoint invariance.
Notice that while contrastive methods dominate performance under viewpoint shifts (Table 2), e-MAE shows the best performance only in the case of low-resolution surveillance cameras. We hypothesize that this is due to the fact that in addition to v-MAE being strong temporal learner (see Figure 3), it is also inherently robust low-resolution inputs. This is likely since it learns strong pixel-level relations [86] through reconstruction of highly occluded inputs. To further investigate this, we perform a follow-up study in a controlled toy setup where we systematically decrease the resolution of input frames from 1 to 162 and measure the robustness as 1 (acc112 accx)/ /100, following [53]. Here acCn refers to accuracy at resolution N2. We use 112 as the maximum resolution as this is used during pretraining, and 16² is set as the lowest resolution as we use a patch size of 16?. To limit the interference of temporal representations with the outcome, we use an image dataset, STLI0 [71]. to perform unsupervised image classification using the frozen pretrained VSSL encoders. The results presented in Figure 4 (right) show that e-MAF achieves a more steady performance on low-resolution inputs compared to the other methods, which is in alignment with our hypothesis.
Highlights: (a) Video models generally struggle in out-of-context generalization, while e- Supervised and v-MAE exhibit better performance as they are strong temporal learners. (b) Contrastive methods (u-SimCLR, u-MoCo) exhibit better performance to viewpoint shifts. (c) v-MAE is robust against extremely low-resolution inputs. (d) e-Supervised shows extreme vulner- ability in complex scenarios when multiple distribution shifts are applied concurrently.
Table 3: Comparison under source shift.
Table 4: Comparison in zero-shot action recognition.
UCFRMDB
HMDRUCC
Method
Method
UCF
HMDB
RareAct K400 (InD)
OoDHH) (nD(
OoD(H) (nD(U))
OoD(U) (ob(H) OoD(U) (nIXH)
Lin.
FT
Lin.
FT
Lin.
FT
Lin.
FT
e- e-Supervised
45.8.ox 92.7.sa
56.3
96.8
50.3.ss
75.9
51.1
84.6
u-Supervised
n/n 37.4
n/a
19.0
wa 9.9
n/a 59.0
>-SinLLR
47,7
96.0..
53.7
97.7
55.1.s
82.0...
56.7
K5.6
D-SimCLK
37.2
4033 18.6.
24.9 7.
10.
56.8
69.8
e-MoCo
51.5
v-MoCo
19.5..
--BYOL
51.4w:
94.5..
59.4
97.5
63.
79.9..:
60.4
87.3
r-BYOL
33.0a.:1.0
24.7
7.5m.
10.3 57.4 69.3
-SimSiam
46.1as
92.6..
50.6
95.5
52.2
76.2...
53.2
83.0
v-SinSiam
34.0... 38.7
18....2100 7.7
11.1
64.4
-DINO
49.3.s
94.3
51.4
96.3
53
8:o
77.5
55.6
87.8
D-DINO
54.3 41.33..3 17.2...2.
8. 10.5
a3.4...6..7
MAB
39.5m= 89.2
39. 72.8
n-MAR
25. 14.2.-25.8
Q2: Effect of finetuning under distribution shifts
Our thorough empirical experiments presented in Tables 1, 2, and 3. show several interesting aspects of finetuning under distribution shifts. First, finetun- ing generally results in better InD and OoD performance compared to linear evaluation (see Table 5). Moreover, the benefits of finetuning vary between different VSSL methods; for example, M-MAE benefits the most from fine- tuning (see Figures 5 and 6). Second, the benefits of finetuning depend on the nature of the distribution shift. Specifically, by comparing OoD and InD performance, we observe that finetuning is more helpful for actor shift in both animal and synthetic domains (see Figure 5), whereas it is less beneficial for zero-shot and viewpoint shift in both egocentric and surveillance camera view. Moreover, finetuning often leads to in-distribution overfitting' where the InD validation improves with additional training, while leading poor OoD generalization (see Appendix D). Overall, the results of zero-shot recognition presented in Table 4 show that no single method dominates in all three benchmarks for pero-shot recognition. e-MAE, e-MoCo, and v-SimSiam achieve the best performances on HMDB, UCF, and RareAct, respectively. when finetuned. Third, the benefits of finetuning also depend on the InD training benchmark. For example, as shown in Figure 5, while finetuning is significantly beneficial under the source shift of UCF to HMDB, it interestingly hurts the performance when this shift is reversed (HMDB to UCF). This can be attributed to the limited utility of small-scale datasets, such as HMDB, in learning strong
InD
OoD
Lin. 63.2%
31.5%
71.9%
37.4%
FT.
(1 (.7)
(1.5.)
Table 5: Linear vs.
finetuned comparison sum- mary. Please see Appendix D.5 for additional details.
7

---

representations. Such datasets could potentially even diminish the effectiveness of robust pretrained models. For instance, in the case of v-BYOL, the performance of the frozen encoder exceeds that of the finetuned model, as seen in Table 3 (HMDB/UCFF.
In set buzin Out of bouibbbutttt
31.0
23.0
13.0
50-
dugrenttee anter and soaee
Figure 5: Linear vs. finetuned performance comparison under real-world distribution shifts. We measure the improvements as the difference between finetuned and linear evaluation accuracy.
To further investigate scenarios where finetun- ing may impair VSSL pretraining, we evalu- ate the video models temporally perturbed inputs (e.g., freeze frames, random sampling, reverse sampling) in controlled setup. As the VSSL methods are never trained with temporally perturbed inputs, this constitutes distribution shift. The results presented in Figure 6 show that end-to-end finetuning of the contrastive and non- contrastive video encoders diminishes the time- invariance of the learned representations, which decreases robustness to temporal perturbations. As a result, such cases, the frozen encoders perform better than the finetuned ones. More- over, the frozen v-MAE shows the worst perfor- mance under temporal shifts as it learns time- variant representations through self-supervised pretraining.
60.0
50.0
40.0
36.0
26.0
36.0
40
50.0
60.0
79.0
accuracy
V.MCCO
Froose
WEFOL
9-SIMCLI
sarnaing rundom
VDIINO
9-SIIGA
VMAE
V.SUEERVISED
Figure 6: Linear vs. finetuned performance com- parison under synthetic temporal perturbations.
Highlights: (a) As opposed to LLMs, finetuning generally helps VSSL both InD and OoD. (b) The benefits of finetuning largely vary between different VSSL methods and the type of distribution shifts. (c) Finetuning provides more benefits against actor shifts in both animal and synthetic domains in comparison viewpoint shifts like egocentric and surveillance camera views. (d) Finetuning degrades robustness to temporal perturbations as it impairs the time-invariant representations of contrastive and non-contrastive methods. It can also degrade performance under source shift depending on the quality of the training benchmark.
Q3: Closed-set vs. open-set performance
We perform open-set recognition in two setups: (1) we use the Kinetics400 for closed-set recognition, while the non-overlapping classes from the UCF101 and HMDB51 are used for open-set. (2) we use UCF101 and the non-overlapping classes of HMDB for closed-set and open-set recognition, respectively. To perform a comprehensive analysis on the performance of closed-set vs. open- set, we adopt two approaches, i.e, single-objective (closed-set only with cross-entropy error) and joint-objective (closed-set and open-set recognition simultaneously with DEAR loss [45]) variants.
8

---

Table Comparison in open-set recognition. e-Supervised trained from scratch with DEAR loss failed to converge in K/U and K/H. Here, closed/open-see pairs are denoted as K/H: Kinetics/HMDB, K/U: Kinetics/UCF, U/H: UCF/HMDB, CE: cross-entropy error.
(a.) End-to-end finetuned.
(b.) Linear evaluation.
Open-set (AUC)
Closed-set w/ DEAR (Acc.)
Chosed-st CE
Open-sel (AUC)
Closed-sel wr DEAR (Acc)
Clieed-el
Method
Method
w/ (Aec)
U/H
K400"
K400
U101"
U101
U101
e-Supervised
77.7
86.5
57.6
87.4
11 u-Supersised
77.
80....
81.7
e-SinCLR
n-SinCLR
85.2
70.1
n-MoCo
50.7
32.7
e-BYOL
60.5
60.1
81.7
69.4
89.4
72.5
00.2
B-BYOL
56.0..x
63.7
85.4
e-SimSiam
60.6
55.9
77.8
64.7
85.9
69.3
87.2
U. U-SimSiaam
73.7
79.5az:
82.5
e-DINO
60.0
58.1
81.7
65.9
87.0
71.3
88.5
v-DINO
79.4
$0.4.s
30.9
o-MAE
73.1
87.8
75
2020
380
115
(a) UCPHMMB
(b) Kineties00oUCCF
Kineti>s&WIIMMI
Relationship
open- performance,
Figure 7: (a-c) Comparing open macro-F1 scores of finetuned models vs. openness (openness is measured as the ratio of unknown to known classes). (d) The relationships between closed-set and open-set recognition performance of frozen pretrained encoders.
The results in Table 6 (a) demonstrate that the models struggle in open-set recognition when using Kinetics400 closed-set in comparison UCF101. This is due to the large number of known classes in Kinetics400 compared to the unknowns (400 known classes Kinetics400 vs. 31 and 22 unknown classes from UCF and HMDB, respectively). Therefore, the models tend to become over-confident and make false predictions. Evident from Table (a), contrastive methods are robust open-set recognition all setups. This is likely due to the auxiliary information contributed by the negative samples used in these methods. To verify this hypothesis, we analyze the open macro-F1 scores vs. openness of the video models following [45, 44] by incrementally adding more unknown classes. The results in Figure 7 (a-c) show that both v-SimCLR and v-MoCo consistently achieve better performances compared to others. An additional statistical analysis on the performance of contrastive methods in open-set recognition is presented in Appendix D.7, which confirms their better performance relative to other methods.
Next, instead of end-to-end finetuning, we use the frozen encoders to train a linear head for open-set recognition. The top 3 closed-set performers, e-BYOL, v-MoCo, and e-SimCLR (see Figure (d) or U101* in Table 6 (b)), show the worst performance in open-set recognition. However, the pretrained encoders, o-SimSiam, v-DINO, and u-Supervised, which considered weaker in closed- set, perform better in open-set recognition, which might be due to their lack of over-confidence. v-DINO outperforms other methods open-set recognition achieving 79.4% while the performance of o-SimCLR and e-MoCo drops to almost chance levels. In comparison, v-MoCo reports 84.9% in standard closed-set recognition, whereas D-DINO achieves 80.9%. The results presented in Figure (d) suggest that there is trade-off between closed-set and open-set recognition performances, particularly when frozen encoders are used without finetuning.
Highlights: (a) Contrastive methods demonstrate superior performance in open-set recognition when finetuned. (b) There is trade-off between closed-set and open-set recognition performance when frozen pretrained encoders are used. (c) Strong frozen encoders (w-MoCo, v-SimCLR) have open-set generalization performance due their over-confident predictions. (d) On the other hand, slightly weak VSSL frozen encoders (v-DINO, u-SimSiam) show better open-set performance, while e-MAE seems perform poorly in both settings.

---

Figure 8: The decision similarity between the video models in InD (top) vs. OoD (bottom). The lighter color indicates less similarity.
Q4: Decision similarity under distribution shifts
To measure 'decision similarity' between the video models, we evaluate whether the models make similar predictions, regardless of their correctness. The results presented in Figure demonstrate that decision similarity varies between the VSSL methods both InD and OoD, but is generally lower in OoD settings. Specifically, while we observe only a slight decrease in decision similarity in the case of animal domain actor shift, we observe significant drops in decision similarity in the case of context shifts and source shifts. Moreover, the results reveal that the decision similarity between supervised and self-supervised methods significantly drops under distribution shifts, indicating that under such shifts, VSSL methods make considerably different decisions than supervised learning. We also observe low decision similarity between v-MAE and other VSSL methods, which can be due to the generative nature of e-MAE vs. the others. Lastly, we note that contrastive methods tend to have greater decision similarities with each other than the similarities observed among non-contrastive methods. Additional results are presented in Appendix D.
Highlights: (a) The decision similarity of video models decreases under distribution shifts, which further varies based on the type of shift. (b) Context and source shifts cause the most dissimilarity between decisions. (c) Overall, the predictions between the supervised and self supervieed methods, as well as between e-MAE and other VSSI. methods exhibit the least similarity.
7 Discussion and Summary
Limitations. We consider contrastive, non-contrastive, and generative VSSL methods in our work, as they are well-established and have demonstrated strong performance in previous studies [72, 1, 3, 5] on various video benchmarks [1. 87, 72]. However, there exists another category of VSSL methods which uses pretext tasks for self-supervision, e.g., rotation prediction [88]. frame and clip order prediction [14, 89], motion prediction [90], and others [91, 92, 93, 94]. While these methods are not included in our study, they are worth exploring in future research. Moreover, our work primarily focuses on various self-supernu supervised methods as opposed to network architectures. It would valuable to further investigate VSSL under distribution shifts with larger Transformer [95] or convolutional [96] architectures, which we could not perform due to resource constraints. Nonetheless, our findings serve as a foundation for future studies. We discuss the broader impact of our work in Appendix A.
Summary. In this work, we thoroughly examine the behavior of VSSL methods under real-world distribution shifts that commonly occur in videos due to changes in context, viewpoint, actor, and source. Moreover, our study delves into investigating the generalizability of VSSL methods in zero- shot and open-set recognition. To rigorously evaluate the robustness of video models, we introduce a comprehensive OoD test bed curated from existing literature. This test bed is carefully designed to stress test the robustness of video models and provide a comprehensive evaluation of their capabilities. Our study uncovers a wide range of interesting dynamics of various VSSL methods under different distribution shifts, which can be instrumental in guiding future research and algorithm development in video representation learning. To the best of our knowledge, this the first work to systematically investigate VSSL under real-world distribution shifts.
10

---

Acknowledgments and Disclosure of Funding
PS and AE grateful the Bank of Montreal and Mitacs for funding their research. PS and AE are also thankful to SciNet HPC Consortium for helping with the computation resources. We thank Vandad Davoodnia (Queen's University) and Ananth Balashankar (Google Research) for fruitful discussions and feedback.
References
[1] Zhan Tong, Yibing Song, Jue Wang. and Limin Wang. Videomae: Masked autoencoders are data-efficient learners for self-supervised video pre-training. arXiv preprint 12X.::22233116022
2022. 1, 3, 5, 10, 21,
[2] Pedro Morgado, Nuno Vasconcelos, and Ishan Misra. Audio-visual instance discrimination with cross-modal agreement. In CVPR, pages 12475-12486, 2021. 2, 5, 6, 26
[3] Rui Qian, Tianjian Meng, Boqing Gong, Ming-Hsuan Yang, Huisheng Wang, Serge Belongie, and Yin Cui. Spatiotemporal contrastive video representation learning. In CVPR, pages 6964-6974, 2021. 1, 2, 5, 6, 10
[4] Humam Alwassel, Dhruv Mahajan, Bruno Korbar, Lorenzo Torresani, Bernard Ghanem, and Du Tran. Sule-supe learning by cross-modee audio-video clustering. NeurlPS, 33, 2020.
1, 2,
[5] Tian Pan, Yibing Song, Tianyu Yang, Wenhao Jiang, and Wei Liu. Videomoco: Contrastive video representation learning with temporally adversarial examples. In CVPR, pages I 11205- 11144, 2021. 1, 2, 5, 6,
[6] Pritam Sarkar and Ali Etemad. Xkd: Cross-modal knowledge distillation with domain alignment for video representation learning. arXiv preprint arXiv:2211.33929, 2022. 1, 2, 5, 26
[7] Pritam Sarkar and Ali Etemad. Self-supervised audio-visual representation learning with relaxed cross- temporal synchronicity, 2021. 1, 2, 5, 6, 26
[8] Limin Wang, Bingkun Huang, Zhiyu Zhao, Zhan Tong, Yinan He, Yi Wang, Yali Wang, and Yu Qiao. Videomae v2: Scaling video masked autoencoders with dual masking. arXiv preprint arXiv:2303. 2023. 1,2
[9] Li Haopeng, Ke Qiuhong. Gong Mingming, and Zhang Rui. Video summarization based on video-text modelling. arXiv preprint arXiv:2201. Ttt::2001.02494,, 2022.
[10] Karan Desai and Justin Johnson. Virtex: Learning visual representations from textual annota- tions, 2021.
[11] Lu Yuan, Dongdong Chen, Yi-Ling Chen, Noel Codella, Xiyang Dai, Jianfeng Gao, Houdong Hu, Xuedong Huang, Boxin Li, Chunyuan Li, Ce Liu, Mengchen Liu, Zicheng Liu, Yumao Lu, Yu Shi, Lijuan Wang, Jianfeng Wang, Bin Xiao, Zhen Xiao, Jianwei Yang, Michael Zeng, Luowei Zhou, and Pengchuan Zhang. Florence: A new foundation model for computer vision, 2021.
[12] Yi Wang. Kunchang Li, Yizhuo Li. Yinan He, Bingkun Huang, Zhiyu Zhao, Hongjic Zhang, Jilan Xu, Yi Liu, Zun Wang, et al. Internvideo: General video foundation models via generative and discriminative learning. arXiv preprint arXiv:2212.03191, 2022. 1,
[13] Uta Buchler, Biagio Brattoli, and Bjom Ommer. Improving spatiotemporal self-superrision by deep reinforcement learning. In ECCV, 2018.
[14] Hsin- Hsin-Ying Lee, Jia-Bin Huang, Mancesh Singh, and Ming- Yang. Unsupervised repre- sentation learning by sorting sequences. In CVPR, 2017. 1, 10
[15] Hyeon Cho, Taehoon Kim, Hyung Jin Chang, and Wonjun Hwang. Self-supervised spatio- temporal representation learning using variable playback speed prediction. arXiv preprint arXiv:2003.0266 2020.

---

[16] Dezhao Luo, Chang Liu, Yu Zhou, Dongbao Yang, Can Ma. Qixiang Ye, and Weiping Wang. Video cloze procedure for self supervised spatio-temporal learning. In AAAI, 2020.
[17] Dahun Kim, Donghyeon Cho, and In So Kweon. Self-supervised video representation learning with space-time cubic puzzles. In AAAI, 2019. 1
[18] Fei Wu, Qingzhong Wang, Jiang Bian, Ning Ding, Feixiang Lu, Jun Cheng, Dejing Dou, and Haoyi Xiong. A survey on video action recognition in sports: Datasets, methods and applications. IEE Transactions on Multimedia, 2022.
[19] Rui Wang, Dongdong Chen, Zuxuan Wu, Yinpeng Chen, Xiyang Dai, Mengchen Liu, Yu-Gang Jiang, Luowei Zhou, and Lu Yuan. Bevt: Bert pretraining of video transformers, arXiv preprint arXiv:2112.01529, 2021. 1, 2
[20] Beliz Gunel, Jingfei Du, Alexis Conneau, and Veselin Stoyanov. Supervised contrastive learning for pre-trained language model fine-tuning. In ICLR, 2021.
[21] Hyundong Cho, Chinnadhurai Sankar, Christopher Lin, Kaushik Sadagopan. Shahin Shayan- deh, Asli Celikyilmaz, Jonathan May, and Ahmad Beirami. Know thy strengths: Compre- hensive dialogue state tracking diagnostics. Findings of the ACL: EMNLP 2022, pages 5345-5359, 2022. 2
[22] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In ICMI., pages 1597-1607, 2020. 2, 3, 22
[23] Xinlei Chen, Saining Xie, and Kaiming He. An empirical study of training self-supervised vision transformers. In ICCV, pages
2021. 2.3
[24] Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Pires, Zhaohan Guo, Mohammad Azar, et al. Bootstrap your own latent: A new approach to self-supervised learning. In NeurlPS, 2020. 2, 3, 21
[25] Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In CVPR,
pages 15750-15758, 2021. 2, 21
[26] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In ICCV, pages
2021. 2, 3,
[27] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders scalable vision learners. arXiv preprint arXiv:2/11.06377, 2021. 3, 21
[28] Philippe Weinzaepfel and Grégory Rogez. Mimetics: Towards understanding human actions out of context. IJCV, 129(5), 2021. 2. 3.4, 25
[29] Jihoon Chung, Yu Wu, and Olga Russakovsky. Enabling detailed action recognition evaluation through video dataset augmentation. NeurIPS, 35:39020-39033, 2022.
[30] Jinpeng Wang, Yuting Gao, Ke Li, Yiqi Lin, Andy J Ma, Hao Cheng, Pai Peng, Feiyue Huang, Rongrong Ji, and Xing Sun. Removing the background by adding the background: Towards background robust self-supervised video representation learning. In CVPR, pages 11804-11813, 2021. 2
[31] Jinwoo Choi, Chen Gao, Joseph CE Messou, and fa-Bin Huang. Why can't dance in the mall? learning to mitigate scene bias in action recognition. NeurIPS, 32, 2019. 2
[32] Manlin Zhang, Jinpeng Wang, and Andy J Ma. Suppressing static visual cues via normalizing flows for self self-supervised video representation learning. In AAAI, volume 36, pages 3300-3308, 2022. 2
[33] Yong Jae Lee, Joydeep Ghosh, and Kristen Grauman. Discovering important people and objects for egocentric video summarization. In CVPR, pages 1346-1353, 2012. 2
[34] Sheila J. Nayar. Columbia object image library (coil100). 1996. 2, 5, 6, 28
12

---

[35] Ugur Demir, Yogesh S Rawat, and Mubarak Shah. Tinyvirat: Low-resolution video action recognition. In ICPR, pages 7387-7394, 2021.
[36] Gunnar A Sigurdsson, Abhinav Gupta, Cordelia Schmid, Ali Farhadi, and Karteek Alahari. Charades-ego: A large-scale dataset of paired third and first person videos. arXiv preprint arXiv:1804.09626, 2018. 2, 4, 25
[37] Xavier Puig, Kevin Ra, Marko Boben, Jiaman Li, Tingwu Wang, Sanja Fidler, and Antonio Torralba. Virtualhome: Simulating household activities via programs. In CVPR, pages 8494-8502, 2018. 2
[38] Gül Varol, Ivan Laptev, Cordelia Schmid, and Andrew Zisserman. Synthetic humans for action recognition from unseen viewpoints. IJCV, 129(7):2264 2287, 2021.
[39] Angelina Wang, Alexander Liu, Ryan Zhang. Anat Kleiman, Leslie Kim, Dora Zhao, Iroha Shirai, Arvind Narayanan, and Olga Russakovsky. Revise: A tool for measuring and mitigating bias in visual datasets. IJCV,
2022. 2, 4
[40] Antonio Torralba and Alexei A Efros. Unbiased look at dataset bias. In CVPR, pages 1521-1528, 2011. 2,
[41] Biagio Brattoli, Joseph Tighe, Fedor Zhdanov, Pietro Perona, and Krzysztof Chalupka. Re- thinking zero-shot video classification: End-to-end training for realistic applications. In CVPR, pages 4613-4623, 2020. 5.5
[42] Alec Kerrigan, Kevin Duarte, Yogesh Rawat, and Mubarak Shah. Reformulating zero-shot action recognition for multi-label actions. NeurIPS, 34:25566-25577 2021. 2, 21, 27
[43] Farhad Pourpanah, Moloud Abdar, Yuxuan Luo, Xinlei Zhou, Ran Wang, Chee Peng Lim, Xi-Zhao Wang, and QM Jonathan Wu, A review of generalized zero-shot learning methods. PAMI, 2022. 2
[44] Abhijit Bendale and Terrance E Boult. Towards open set deep networks. In CVPR, pages 1563-1572, 2016. 2, 4,
[45] Wentao Bao, Qi Yu, and Yu Kong. Evidential deep learning for open set action recognition. In ICCV, pages 13359-13358, 2021. 2, 4, 5, 8, 9,
[46] Walter Scheirer, Anderson Rezende Rocha, Archana Sapkota, and Terrance E Boult. Toward open set recognition. PAMI, 1577):1777-117 2012. 2,
[47] Chuanxing Geng, Sheng-jun Huang, and Songean Chen. Recent advances in open set recogni- tion: A survey. PAMI, 43(10):3614 3631, 2020. 2
[48] Florian Wenzel, Andrea Dittadi, Peter Vincent Gehler, Carl-Johann Simon-Gabriel, Max Horn, Dominik Zietlow, David Kernert, Chris Russell, Thomas Brox, Bernt Schiele, Bernhard Schölkopf. and Francesco Locatello. Assaying ouist--distribution generalization in transfer learning, 2022. 2,
[49] Rohan Taori, Achal Dave, Vaishaal Shankar, Nicholas Carlini, Benjamin Recht, and Ludwig Schmidt. Measuring robustness natural distribution shifts in image classification. NeurIPS, 33:18583-18599, 2020. 2,
[50] Olivia Wiles, Sven Gowal, Florian Stimberg, Sylvestre-Alvis Rebuffi, Ira Ktena, Krishna- murthy Dj Dvijotham, and Ali Taylan Cemgil. A fine-grained analysis distribution shift. In ICLR, 2022.
[51] Haotian Ye, Chuanlong Xie, Tianle Cai, Ruichen Li, Zhenguo Li, and Liwei Wang. Towards a theoretical framework of -ut-o--disti generalization. M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, NeurIPS, volume 34, pages 23519- 23531, 2021. 2
[52] Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common corruptions and perturbations, 2019.
13

---

[53] Madeline C Schiappa, Naman Biyani, Shruti Vyas, Hamid Palangi, Vibhav Vineet, and Yogesh Rawat. Large-scale scale robustness analysis of video action recognition models. arXiv preprint arXiv:2207.01398; 2022. 7,38
[54] Chenyu Yi, Siyuan Yang. Haoliang Li, Yap-peng Tan, and Alex Kot. Benchmarking the robustness of spatial-temporal models against corruptions. arXiv preprint 10X.v:2110006113
2021. 3, 38
[55] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint / 2018. 3, 26
[56] Antti Tarvainen and Harri Valpola. Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results. NeurlPS, 30, 2017. 3, 21
[57] Christoph Feichtenhofer, Haoqi Fan, Yanghao Li, and Kaiming He. Masked autoencoders as spatiotemporal learners. arXiv preprint 220Xv::200.09/13. 2022. 3, 21,
[58] Yi Li and Nuno Vasconcelos. Repair: Removing representation bias by dataset resampling. 2019. 3
[59] Yingwei Li, Yi Li, and Nuno Vasconcelos. Resound: Towards action recognition without representation bias. ECCV, Septemher 2018.
[60] Sagar Vaze, Kai Han, Andrea Vedaldi, and Andrew Zisserman. O recognition: A good closed-set classifier is all you need. arXiv preprint arXiv:2110.0620,, 2021. 4
[61] Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijaya- narasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, al. The kinetics human action video dataset. arXiv preprint 1705:009550950 2017. 4, 21
[62] Lucas Smaira, João Carreira, Eric Noland, Ellen Clancy, Amy Wu, and Andrew Zisserman. A short note on the kinetics-700- 2020. 4, 26,26 26
human action dataset. arXiv preprint arXiv:2010. 10864,
[63] Praveen Tirupattur, Aayush J Rana, Tushar Sangam, Shruti Vyas, Yogesh S Rawat, and Mubarak Shah. Tinyaction challenge: Recognizing real-world low-resolution activities in videos. arXiv preprint arXiv:2 207. 1494, 2021. 4, 25
[64] Alina Roitberg, David Schneider, Aulia Djamal, Constantin Seibold, Simon ReiB, and Rainer Stiefelhagen. Let's play for action: Recognizing activities of daily living by learning from life simulation video games. In IROS, pages 8563-85666 8569, 2021. 4, 25
[65] Yunhua Zhang, Hazel Doughty, Ling Shao, and Cees GM Snoek. Audio-adaptive activity recognition across video domains. CVPR, pages 13791-13800, 2022. 4, 26
[66] Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. Ucf101: A dataset of 101 human actions classes from videos in the wild, arXiv preprint arXiv: arXiv: 2012. 4,
[67] Hildegard Kuehne, Hueihan Jhuang, Estíbaliz Garrote, Tomaso Poggio, and Thomas Serre. Hmdb: a large video database for human motion recognition. In ICCV, pages 2556-2563,
2011. 21
[68] Antoine Miech, Jean-Baptiste Alayrac, Ivan Laptev, Josef Sivic, and Andrew Zisserman. Rareact: A video dataset of unusual interactions. arXiv preprint arXiv:2008.01018. 2020.
[69] Mathew Monfort, Alex Andonian, Bolei Zhou, Kandan Ramakrishnan, Sarah Adel Bargal, Tom Yan, Lisa Brown, Quanfu Fan, Dan Gutfreund, Carl Vondrick, al. Moments in time dataset: one million videos for event understanding. PAMI, 42(2):502-508, 2019, 4, 25
[70] Xiaohan Wang. Tengyu Ma, James Ainooson, Seunghwan Cha, Xiaotian Wang, Azhar Molla, and Maithilee Kunda. The toybox dataset of egocentric visual object transformations, arXiv preprint arXiv: 06034, 2018. 5, 6, 34
[71] Adam Coates, Andrew Ng, and Honglak Lee. An analysis of single-layer networks in unsupervised feature learning. In ICAIS, pages 215-223, PMLR, 2011. 5, 7.
14

---

[72] Christoph Feichtenhofer, Haoqi Fan, Bo Xiong, Ross Girshick, and Kaiming He. A large-scale study on unsupervised spatiotemporal representation learning. In CVPR, pages 3299-3309, 2021. 5, 6, 10, 22, 26
[73] Du Tran, Heng Wang, Lorenzo Torresani, Jamie Ray, Yann LeCun, and Manohar Paluri. A closer look at spatiotemporal convolutions for action recognition. In CVPR, pages 6450-6459, 2018. 5
[74] Ji Lin, Chuang Gan, and Song Han. Tsm: Temporal shift module for efficient video under- standing. In ICCV, pages 7083-7093, 2019.
[75] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghami, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition scale. arXiv preprint arXiv:2010. 1929, 2020. 5,
[76] Anurag Amab, Mostafa Dehghani, Georg Heigold, Chen Sun, Mario Lučié, and Cordelia Schmid. Vivit: A video vision transformer. In ICCV, pages 6836-6846, 2021.
[77] Jane Bromley, Isabelle Guyon, Yann LeCun, Eduard Säckinger, and Roopak Shah. Signature verification using a" siamese" time delay neural network. NeurIPS, 6, 1993. 5
[78] Hassan Akbari, Liangzhe Yuan, Rui Qian, Wei-Hong Chuang, Shih-Fu Chang, Yin Cui, and Boqing Gong. Vatt: Transformers for multimodal self-supervised learning from raw video, audio and text. NewrIPS, 34, 2021. 5, 26
[79] Pedro Morgado, Ishan Misra, and Nuno Vasconcelos. Robust audio-visual instance discrimina- tion. In CVPR, pages 12934 12945, 2021. 5, 6, 26
[80] Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word representations vector space. arXiv preprint arXiv: 33811, 2013. 5, 27
[81] Adrià Recasens, Pauline Luc, Jean-Baptiste Alayrac, Luyu Wang, Florian Strub, Corentin Tallec, Mateusz Malinowski, Viorica Patraucean, Florent Altché, Michal Valko, al. Broaden your views for self-supervised video learning. arXiv preprint arXiv:2 16559, 2021. 6
[82] Jean-Baptiste Alayrac, Adria Recasens, Rosalia Schneider, Relja Arandjelovic, Jason Rama- puram, Jeffrey De Fauw, Lucas Smaira, Sander Dieleman, and Andrew Zisserman. Self- supervised multimodal versatile networks. NeurIPS, 2(6):7, 2020. 6
[83] Senthil Purushwalkam and Abhinav Gupta. Demystifying contrastive supe-supe learning: Invariances, augmentations and dataset biases. NeuriPS, 0033407-3418, 2020. 6
[84] Jianwei Yang. Devi Parikh, and Dhruv Batra. Joint unsupervised learning of deep representa- tions and image clusters. In CVPR, 2016.
[85] Uno Fang, Man Li, Jianxin Li, Longxiang Gao, Tao Jia, and Yanchun Zhang. comprehensive survey on multi-view clustering. IEEE Transactions on Knowledge and Data Engineering, pages 1-20, 2023.
[86] Namuk Park, Wonjae Kim, Byeongho Heo, Tackyung Kim, and Sangdoo Yun. What do self-supe vision transformers learn? ICLR, 2023. 7
[87] Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, and Kaiming He. Slowfast networks for
video recognition. In ICCV, pages 2202-6211, 2019. 10
[88] Longlong Jing, Xiaodong Yang. Jingen Liu, and Yingli Tian. Self-supervised spatiotemporal feature learning via video rotation prediction. arXiv preprint 18/ 1387, 2018. 10
[89] Dejing Xu, Jun Xiao, Zhou Zhao, Jian Shao, Di Xie, and Yueting Zhuang. Self-supervised spatiotemporal learning via video clip order prediction. CVPR, pages 10334-10343, 2019. 10
15

---

[90] Jiangliu Wang, Jianbo Jiao, Linchao Bao. Shengfeng He, Yunhui Liu, and Wei Liu. Self- supervised spatio-temporaa representation learning for videos by predicting motion and ap- pearance statistics. In CVPR, pages 4005-4015, 2019. 10
[91] Haodong Duan, Nanxuan Zhao, Kai Chen, and Dahua Lin. Transrank: Sulf-supervised video representation learning via ranking-based transformation recognition. In CVPR, pages 3000-3010, 2022. 10
[92] Ishan Misra, Lawrence Zitnick, and Martial Hebert. Shuffle and learn: unsupervised learning using temporal order verification. In ECCV, pages 527-544, 2016. 10
[93] Tengda Han, Weidi Xie, and Andrew Zisserman. Video representation learning by dense predictive coding. In CVPRW, pages 0-0, 2019. 10
[94] Dahun Kim, Donghyeon Cho, and In So Kweon. Self-supervised video representation learning with space-time cubic puzzles. In AAAI, volume 33, pages 6545-8552, 2019. 10
[95] Mostafa Dehghani, Josip Djolonga, Basil Mustafa, Piotr Padlewski, Jonathan Heek, Justin Gilmer, Andreas Steiner, Mathilde Caron, Robert Geirhos, Ibrahim Alabdulmohsin, Rodolphe Jenatton, Lucas Beyer, Michael Tschannen, Anurag Arnb,, Xiao Wang, Carlos Riquelme, Matthias Minderer, Joan Puigcerver, Utku Evci, Manoj Kumar, Sjoerd van Steenkiste, Gamaleldin F. Elsayed, Aravindh Mahendran, Fisher Yu, Avital Oliver, Fantine Huot, Jasmijn Bastings, Mark Patrick Collier, Alexey Gritsenko, Vighnesh Birodkar, Cristina Vasconcelos, Yi Tay. Thomas Mensink, Alexander Kolesnikov, Filip Pavetić, Dustin Tran, Thomas Kipf, Mario Lučié, Xiaohua Zhai, Daniel Keysers, Jeremiah Harmsen, and Neil Houlsby. Scaling vision transformers to 22 billion parameters, 2023. 10
[96] Zhuang Liu, Hanzi Mao, Chao- Yuao-Yannnha Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11976-119866 2022. 10
[97] Biagio Brattoli, Joseph Tighe, Fedor Zhdanov, Pietro Perona, and Krzysztof Chalupka. Re- thinking zero-shot video classification: End-to-end training for realistic applications. In CVPR, pages 4623, 2020. 21
[98] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101,
21
[99] Sashank Reddi, Satyen Kale, and Sanjiy Kumar. On the convergence of adam and beyond. arXiv preprint arXiv: 09237, 2019. 21
[100] Zachary Teed and Jia Deng. Raft: Recurrent all-pairs field transforms for optical flow. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part I1 16, pages 402 419. Springer, 2020. 24
[101] Filip Ilie, Thomas Pock, and Richard Wildes. appearance free action recognition possible? In ECCV, pages 156-173. Springer, 2022. 24
[102] Hyojin Bahng, Sanghyuk Chun, Sangdoo Yun, Jaegul Choo, and Seong Joon Oh. Learning de-biased representations with biased representations. In International Conference on Machine Learning, pages 528-539, PMLR, 2020. 25
[103] Kevin Qinghong Lin, Jinpeng Wang, Mattia Soldan, Michael Wray, Rui Yan, Eric Z. XU, Difei Gao, Rong-Cheng Tu, Wenzhe Zhao, Weijie Kong, et al. Egocentric video-language pretraining. NeurlPS, 35:7575-7586 2022. 26
[104] Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint arXiv: 108, 2019. 26, 27
[105] Shreyank N Gowda, Laura Sevilla-Lara, Kiyoon Kim, Frank Keller, and Marcus Rohrbach. A new split for evaluating true zero-shot action recognition. Pattern Recognition, pages 191-205. Springer, 2022. 26
16

---

[106] Jacob Devlin, Ming Wei Chang, Kenton Lee. and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint 1XX.::8810004055. 2018. 27
[107] Harold W Kuhn. The hungarian method for the assignment problem. Naval research logistics quarterly, 1955.
[108] Xiaohan Wang, Tengyu Ma, James Ainooson, Seunghwan Cha, Xiaotian Wang. Azhar Molla, and Maithilee Kunda. Seeing neural networks through box of toys: The toybox dataset of visual object transformations, 2018. 27
17

---

Appendix
The organization of the supplementary material is as follows:
Appendix A: Broader impact
Appendix B: Video self-supervised learning
B.1 Methods
B.2 Implementation details
B.3 Additional insights
Appendix C: Distribution shifts
C.1 Overview
C.2 Context shift
C.3 Viewpoint shift
C,4 Actor shift
C.5 Source shift
C,6 Zero-shot recognition
C.7 Open-set recognition
C.8 Toy experiments
Appendix D: Additional experiments and results
D.1 Pretrained Kinetics700
D.2 Temporal dynamics
D.3 Comparing InD vs. OoD performance under natural distribution shifts
D.4 Effect of synthetic perturbations
D.5 Effect of finetuning in out-of-distribution generalization
D.6 High-level overview
D.7 Statistical analyses
D.8 Exploring performance variation of VSSL methods across action classes
18

---

A Broader impact
In this work, we conduct a thorough study on the behavior of Video Self-supervised Learning (VSSL) methods in the face of distribution shifts. We believe this work has far-reaching implications for advancing the field of video representation learning and its real world applications. This work is of utmost importance primarily for the following two reasons.
Real-world deployment: Video models have wide range of real-world applications across various domains. These applications include surveillance and security, autonomous vehicles, sports analysis, video content recommendation, human-computer interaction, video captioning and summarization, among many others. However, in real-world scenarios, distribution shifts pose significant challenge. where the distribution of the data that the system encounters may differ from the training data distribution. Understanding how VSSL algorithms behave under real-world distributional shifts crucial for ensuring their reliable performance and generalization capabilities for real-world applications. By studying VSSL methods under distribution shifts, we identify the limitations and challenges faced by the popular VSSL methods. We believe our work step towards enabling the development of robust and reliable solutions for real-world deployment.
Algorithmic robustness: Our study provides valuable insights into the algorithmic robustness of VSSI. methods, advancing our understanding of VSSL algorithms. By systematically exploring the behavior of these methods under distribution shifts, we have uncovered intriguing findings and observed interesting behaviors which are previously unknown. Our work not only highlights the strengths of VSSL methods but also identifies their weaknesses in different scenarios. We find that different VSSL methods exhibit varying levels of robustness when confronted with different types of distribution shifts. Some methods demonstrate resilience against certain shifts, while they prove be vulnerable to others. These findings pave the way for future research to develop robust frameworks that leverage the strengths of different approaches, thereby enhancing the overall performance of VSSL algorithms.
19

---

B
Video self-supervised learning
B.1
Methods
In the following section, provide comprehensive overview of our implementation of the VSSL methods explored in this study. We define an encoder f(-), which is composed Transformer backbone ((-) and an MLP projection head h(-). We further define the auxiliary network components, a target encoder f-(-) (utilized in v-MoCo, o-BYOL. and v-DINO), a decoder fa(.) (employed in u-MAE), and a predictor head f-(.) (utilized in e-BYOL, u-SimSiam, and e-MoCo). Note, fe(.) and fa(-) are built on the Transformer architecture, while fz(-) utilizes an MLP head. For given samples v, generate augmented views 1i and v2 where ii and U2 randomly sampled from different timestamps and differently augmented. Moreover, an augmented view v; IR TxHx can be expressed into sequence of spatio-temporal patches of size L w C, where, P = T/tx H/hx W/w. We project the spatio-temporal patches to linear layer followed by feeding the encoders. Following, we briefly summarize the training algorithms of these VSSL methods along with their design differences. An overview of these frameworks is presented in Figure S1.
21
Leimmle
22
Pt
Lmccc
ZZ2
Lense
v
sg
predictor
decoder
encoder
encoder
encoder
encoder
target
V
(a) u-SimCLR
(b) v-MoCo
(C) U-MAE
Layel
Z24
-igiiiint
Leino
+ sg
sg
sg
predictor
predictor
centering
7
encoder
ema
target
encoder
encoder
encoder
ema
target
V2
V2
(d) v-BYOL
(e) v-SimSiam
(f) w- D-DINO
Figure S1: A simplified version of the video self-super methods are presented.
e-SimCLR, The objective of v-SimCLR maximize the similarity between positive pairs and minimize the similarity between negative pairs using the InfoNCE loss function. Given two augmented views V1 and U2, we obtain 21 and 22 as f(1) and f(22). The L2 normalized cosine similarity scores, referred as logits, are then calculated sim (21, 22), where sim(21, 22)
x || The
pretraining loss is calculated as Lximetr = (nloNCE(logits) as per Equation $1, where k+ refers to the positive embeddings (i.e., embeddings of the augmented views) and k refers the negative embeddings (embeddings of other samples in minibatch).
InfoN CE(logiss -log Exe(et) exp(logits/r)
Σne(e exp(logits/r)) where T temperature.
(SI)
20

---

v-MoCo. Similar to v-SimCLR, e-MoCo also calculates InfoNCE loss for given query-key pairs. We obtain the query embeddings as pı = (f))) and its corresponding key embeddings are obtained as 222 = sg( (g2))((2))), where sg refers to stop-gradient operation. In Equation S1, the positive and negative key embeddings {k+, E 222 and the query embedding q E P1- To calculate symmetrized loss we also obtain P2 and 221. The training objective Lmoco is defined as InfoNCE(simm())) (pim(p), x22)) ) + InfoNCE
221)). The encoder f is updated using mmoco. and the target encoder f. is updated using the exponential moving average (EMA) [56] as ft a1+ where a is the EMA coefficient.
B-B. Different from v-SimCLR and v-MoCo, v-B YOL does not require any negative pairs and only relies on positive views. Therefore, we obtain p1 and 222 as ff((())) and sg(fi ((()) to calculate the similarity scores as sim(p), 222). Moreover, to obtain symmetrized loss, we also calculate spm,p2, 221) and define total loss Liyol = - (p1, 222) - sim(p2, 221). Please note, we use Lhyol to train the f. and the weights of Ja are updated using an EMA similar to D-MoCo.
o-SimSiam. The approach for u-SimSiam is very similar the e-BYOL, except it does not use fo- and directly uses the representations of the f as the target. We obtain pl and 22 as fn(((r)) and (g2) calculate the similarity scores as sim(pI, 22). Similarly, we also obtain sim(p2, 21) and define total loss Lssmeiam = sim(p1, z2) sim(p2, =1).
--DINO. Similar to v-BYOL and M-MoCo, v-DINO uses target encoder fi(-) but does not use a predictor head. We obtain 21 and z as f(11) and sg sg(f(()) - C) respectively, where C refers to the Centering operation which can be interpreted adding bias term, preventing the model from collapsing [26]. Next, we calculate the cross-entropy (CF) error between 21 and 222 as softmax /T)) log(softmax(z1 //t)), where Tr and T. denote the temperatures used to sharpen the representations of the target and base encoder respectively. Finally, we calculate loss Ldino CIE(2 + 222) + CE(22, 221) which is used to update f and the weights of fa are updated using EMA similar to b-BYOL and M-MoCo.
CF(21; x=2)
e-MAE. Unlike the methods mentioned above, v-MAE takes single view and performs masking and reconstruction using an autoencoder architecture [27. 1, 57]. Moreover, different from the other models, the encoder does not contain a projection head, i.c., f = 8, instead uses decoder fa for reconstruction. For a given sample we obtain masked tokens a imm as {v'Im' = 1) and the corrupted inputs as ê = {w|m² = 0} We calculate the reconstruction loss as mmee (fa(0(0)) - w/ml) which is used to train the 0 and fa-
After pretraining using the aforementioned methods, discard the auxiliary network components and utilize only 9 for downstream evaluations.
B.2 Implementation details
Datasets. We use 2 popular large-scale datasets Kinetics400 [61] and Kinetics700 [62] for pretraining. Kinetics400 consists of 240K samples including 400 action classes, whereas, Kinetics700 consists of 537K training samples spread over 700 action classes. Following [97,42], we discard the overlapping classes between Kintics700 and UCF101 [66]. HMDB51 [67] from pretraining, which results in approximately 480K samples spread over 663 action classes. We follow such a setup so that the pretrained encoder can be used for zero-shot recognition on full UCF101 and HMDB51.
Pretraining. To ensure a fair comparison between the VSSL methods, we pretrain them in identical setups with necessary adjustments in hyperparameters. Specifically, we keep the encoder, inputs, batch size, and maximum number of pretraining epochs identical for all the methods. However, we adjust the data augmentation strategy and auxiliary network components (e.g., predictor head, projector head, decoder) as required achieve their best performance on 2 validation sets Kinetics400 and UCFI01. For example, the projector head of v-DINO is based on the configuration proposed in [26], whereas, the projector head for the other Siamese frameworks made of an MLP head similar to [24, 25]. The details of the frameworks are presented in Table S1. Considering the widespread use of Transformers vision, VSSL methods are implemented using ViT-B [75] as the encoder with a patch size of 4 x 162. Videos are downsampled to 8 FPS and resized to 3 x 1122 frames. Except for e-MAE, all VSSL methods are fed with 16 frames in each view and 32 frames are fed as input to e-MAF. We use Adam [98, 99] optimizer with cosine learning rate scheduler and train all methods up to 800 epochs with batch size of 768. We present the hyperparameters in Table S3. All the methods are pretrained using 8 V100 32 GB GPUs in parallel. We present the computational specifics
21

---

in Table S2, which shows that e-MAE is the most computationally efficient framework amongst all the VSSL methods.
Table S1: Architecture details of the VSSL frameworks.
v-SimCLR v-MoCo v-BYOL v-SimSiam v-DINO e-MAE
Encoder
VIT-B
Target Encoder
X
/
X
/
Predictor (# layers)
X
1
2
1
X
X
Projector (# layers)
4
3
3
4
3
X
Decoder depth)
X
X
X
Table S2: Computational details of VSSL pretraining.
v-SimCLR e-MoCo v-BYOL. v-SimSiam e-DINO w-MAE
Encoder Param
88M
Total Params
106M
110M
114M
114M 36G
96M
96M
Pretraining Flops
36G
72G
72G
72G
10G
Time//00 epochs
23Hrs.
11 Hrs.
Table S3: Hyperparameters of VSSL pretraining.
e-SImCLR
a-MoCo
B-BYOL
e-SimSlam
R-DINO
=-MAE
Batch size
Crop
[0.08, 1.0]
0..2,0.766
00.5
Clip duration
2.0 [0.6, 0.6, 0.6, 0.15]
4.0
Color
Grayscale
0.2
Gaussian
Hroizontal
0.5
Epochs
800
Warmup epochs
30
Optimizer
Adam [0.9, 0.95]
Weight decay
0.05
0.05 0.5
Leurning rale (Base LR)
2c
3e-
3e
le
3e-
3c
Pred LR (x Base LR)
10x
10x
10x
EMA (cosine sch.)
0.998
0.997
-
0.997
Online temp. (fixed)
0.1
0.1
Target temp. (cosine sch.)
0.04-0.07
Centering momentum
0.9
Masking ratio
85%
B.3 Additional insights
Aggressive cropping. As discussed in the main paper, aggressive cropping leads to viewpoint invariance, therefore we experiment with various multi-crop ratios commonly used in the literature [22, 72] across all six VSSL methods. We present the best setups corresponding to individual methods Table S3. We observe that aggressive cropping negatively impacts the performance e-MAE. This is primarily because reconstructing a local crop serves as weak pseudo task due to limited pixel-level variation, resulting in suboptimal representation learning. Additionally, as depicted in Figure S2, the impact of cropping varies between contrastive and non-contrastive methods. Very high aggressive cropping ratios (0.08 1.0) prove beneficial for contrastive methods, while slightly less aggressive cropping ratios (0.2 0.766) yield better results for non-contrastive methods.
Pretraining epoch vs. accuracy. We track the performance of different videl learning methods on two downstream benchmarks, Kinetics400 and split-1 of UCF101. In particular, Kinetics400 is used for finetuning and UCF101 for linear evaluation. The results are presented in Figure S3 shows that while e-MAE achieves the lowest linear evaluation accuracy, it shows the best performance when finetuned. Moreover, e-BYOL, 0-SimCLR, and v-MoCo exhibit superior performance compared to e-DINO and e- e-SimSiam in both the linear evaluation and finetuning setups. Interestingly, contrastive
22

---

v-YOL [0.2-0.766]
V-SIMCLR [0.2-0.766]
vSYOL [0.08-1.0]
82.5
V-SIMCLR [0. 10.08-1.0]
VIDIO [0.2-0.766]
V-MOCO [0.2-0.766]
[0.08-1.0]
v.MOCO [0.08-1.0]
71.5
75.0
75.0
72.5
72.5
70.0
70.0
67.5
67.5
65.0
65.0
100.0
Epoch
Epoch
(a) Non-contrastive methods
(b) Contrastive methods
Figure S2: The impact of cropping strategies slightly differs between non-contrastive methods (b-BYOL, v-DINO) and contrastive methods (v-SimCI.R, r-MoCo). Aggressive cropping with crop ratio [0.08, 1.0]. enhances the performance of contrastive methods, while ratio of [0.2, 766 yields the best performance
non-contrastive methods. We show the performance with crop ratio of [0.2, 0. 76666 with solid lines and [0.08, 1.0] with dashed lines. Evaluation is carried out on UCF101 using a linear SVM.
methods u-SimCLR and e-MoCo show comparable performance among them, however, there is notable performance gap hetween the non-contrastive methods. Specifically, e-BYOL outperforms both u-SimSiam and v-DINO by significant margin in both the linear evaluation and finetuning setups. Moreover, Figure S3 (a) indicates minimal or no improvements between 600 to 800 epochs. Therefore, refrain from training the models beyond 800 epochs.
85.0
73.0
82.5
72.0
80.0
71.0
77.5
70.0
69.0
68.0
725
67.0
70.0
v-BYOL
V.MAE
V.SIMCLR
V-BOOL
V-MAE
V.SIMELR
66.0
67.
200.0
300.0
500.0
200.0
500.0
700.0
Epech
Epoch
(a) UCF101 (linear SVM)
(b) Kinetics400 (finetuned)
Figure S3: The pretraining epochs vs. top-1 accuracies are presented. VSSL methods are validated in two setups. Kinetics400 is used for finetuning and UCF101 is used for linear evaluation. e-BYOL, o-SimCLR, and v-MoCo show superior performance compared to a-DINO and v-SimSiam in both setups. Interestingly, v-MAE achieves the lowest linear evaluation accuracy, while performing the best when finetuned.
Spatial vs. temporal dependency. addition to the out-distribution tests discussed in the main paper, also conduct several interesting analyses to understand the dependency of these VSSL
23

---

No Temporal
Partial Appearance
No Appearance
Original Frames
Figure S4: Sample frames from UCF101 depicting different
dependency scenarios.
Table S4: Comparing video models on controlled temporal and appearance information.
No Temporal Partial Appe. No Appe.
Original
Method
Lin. /TT.
Lin. / FT.
Lin. / FT. Lin. /T.
t- Supervised
43.7 / 36.6
4.1 4,9
1.3 1.0
81.7 / 87.4
0-SimCLR
57.1 46.5
4.3/4.9
0.9
84.1 / 90.3
v-MoCo
56.0 48.8
4.7/6.7
1.7/1.1
84.9/ 90.7
v-BYOL
58.2/50.9
3,5/6.0
1.1/2.6
85,4/901
u-SimSiam
28.7
5.6/4.5
2.0/
82.5 87.2
u-DINO
38.9/38.6
3.2/7.7
1.2/
8 88.5
w-MAE
5.67
3.1/5.3
1.1227
76.2/90.
methods towards spatial vs. temporal representations. To explore the spatial vs. temporal dependency of the VSSL methods, we use UCF101 as the base validation dataset and create synthetic variant of it. (i) No Temporal:
completely remove the temporal information from the input and treat single frames as an input. In particular, we obtain 80 frames per video that represent 10 seconds of video sampled at 8 FPS. (ii) Partial Appearance: Next, to suppress the spatial information, we obtain the optical flow [100] from the original videos. (iii) No Appearance: Following [101], we also obtain the videos of no appearance, i.e., the static frames have meaning and actions are present in the temporal dimension. We present sample frames of these variants Figure S4.
The results are presented in Table S4 show several intriguing observations. temporal dimension from the videos leads to a significant drop in performance in the methods. Amongst all the video models, e-MAE suffers the worst in the absence of temporal information in both linear and finetune setups, which shows its overly dependent nature on temporal information and it is weak spatial learner (also discussed in the main paper). Second, in the absence of temporal information, the frozen encoder outperforms the finetuned variant even in the case of o-Supervised. We also notice the performance of the non-contrastive methods significantly varies under such setups, for example, while u-B YOL. achieves the best performance in linear evaluation, e-DINO performs worse other than e-MAE. Additionally, the video models is capable understanding actions just from the temporal information. In fact, the performance of video models drops near chance level 1%) in no appearance setup and achieve slightly better (3 - 7%) in partial appearance setup.
First, removing the
tests under partial
appearance reveal that none of
24

---

Distribution Shifts
C.1
Overview
We provide summary of our proposed out-of-distribution test bed in Table S5, and the details of our setup are described in the following subsections. Due to the large number of experiments conducted across various setups, will release configuration files for individual evaluation the project website rather than sharing them here.
Table S5: An overview of our outoo--distribution test-bed. #Samples are in following order training samples/InD test samples/OoD test samples; In zero-shot and open-set recognition, #Classes indicates the number of InD/OoD classes and for others the number of InD and OoD classes remains the same.
Distribution Shift
InD
OoD
#Classes #Samples
1. Context shift (10 classes)
Kinetics400 Kinetics400
Mimetics 10
10
5930/494/ /944/36
2. Context shift (50 classes)
Mimetics50
50
34K//2481 //713
3. Viewpoint shift (egocentric)
CharadesEgo
CharadesEgo
157
3/K/9386/991 /9145
4. Viewpoint shift (surveillance)
MiT-v2
TinaVirat-v2 v2
14
41K/1400/2644
5. Actor shift (animal)
Kinetics400
ActorShift
7
15K/1018/165
6. Viewpoint + Actor shift ((op-dowa+ssytheetic)
MiTv2
Sims4AAtion
19K/600/950
7. Source shift (UCF/HMDB)
UCF
HMDB
17
1877/746/511 /510
8. Source shift (HMDB/UCF)
HMDB
UCF
17
1190/510/749
9. Zero-shot (K400/UCF)
Kinetics400
UCF
400/31
240K 240K/20K 22////KK/366 /3965
10. Zero-shot (K400/HMDB))
Kinetics400
HMDB
400/22
240K BL0K/20K/330 /3288
11. Zero-shot (K400/RarreAct)
Kinetics400
RareAct
400/149
240K/ 240K/20K/
20K/1961
Zero-shot (K700/UCF)
Kinetics700
UCF
663/101 480K/
/13K
13. Zero-shot (K70//HMBB)
Kinetics700
HMDB
663/51
480K/
/6.7K
14. Zero-shot (K7000RarcAAct)
Kinetics700
RareAct
663/149
480K/
/1961
15. Open-set (K-400/UCF)
Kinetics400
UCF
400/31
240K 20K BOOK/3966 3966
16. Open-set (K400/HMMB)
Kinetics400
HMDB
400/22
240K 200//////// /3288
17. Open-set (U101/HMDB)
UCF101
HMDB
101/34
9537 2337/3783/4366 /366
C.2 Context shift
We use the two splits of Mimetics (i.e., Mimetics10 [102] and Mimetics50 [28]) the OoD validation benchmark to study video models under context shift. Mimetics consists of subset of action classes from Kinetics400, where the context the videos is either partial or misleading or completely absent. We train the pretrained encoders using videos of corresponding action classes from Kinetics400, and subsequently perform InD and OoD validations using Kinetics400 and Mimetics, respectively. For linear evaluation, we extract the fixed features by taking the mean of all the patches from the last layer of the encoder. We then employ an SVM with a linear kernel for classification. In the finetuning stage, we add a fully-connected layer on top of the encoder and train it end-to-end using the AdamW optimizer and cross-entropy error. The top-1 accuracy is reported for both InD and OoD validation. Please note, the different VSSL methods are individually tuned to achieve their best performance.
C.3 Viewpoint shift
We study three types of viewpoint shifts, (i) egocentric view, (ii) surveillance camera view, and (iii) top-down view. We use CharadesEgo [36], Tiny Virat-v2 [63], and Sims4Action [64] as the OoD benchmark for egocentric view, (ow-resolution surveillance camera view, and top-down view respectively. CharadesEgo a collection of paired third perd--pesonn and first-person videos of human activities. Therefore, we use the corresponding third-person person videas from CharadesEgo [36] for InD training and validation split while the egocentric view is used as OoD validation. We use MiT-v2 [69] to create the corresponding InD split for Tiny Virat-v2 and Sims4Action [64]. MiT-v2 is large-scale video benchmark of 1M videos spread over 339 different human actions. We use the videos of overlapping classes between MiT-v2 and Tiny Virat-v2 study surveillance camera viewpoint shift.
25

---

Similarly, the videos of overlapping classes between MīT-v2 and Sims4Action are used to study top-down viewpoint shifts. The class assignments between InD and OoD benchmarks are mentioned at the end of this subsection.
For training the models on CharadesEgo, adopt the approach proposed in [103]. In this setup, instead of using one-hot encoded labels, we utilize the text descriptions associated with the videos. We generate tokens from these descriptions and train a video-fext encoder using an InfoNCE objective function [55]. In particular, we use the DistilBERT base [104] as the text encoder along with the pretrained ViT-B is used as the video encoder. We redirect the readers to [103], to know more about the training scheme of CharadesEgo. In the case of linear evaluation, the above-mentionel setup remains the same, except we keep the video encoder frozen. Moreover, our training strategy with MīT-v2 to study surveillance camera viewpoint and top-down viewpoint shifts follow standard finetuning setup using cross-entropy error [72, 2, 4, 6, 78, 79, 7]. However, due to the difference in the nature of the datasets, as Tiny Virat contains multiple actions per video, whereas, MiT-v2 consists of single actions per video, we make an adjustment in the prediction layer. Instead of using a Softmax layer, we utilize Sigmoid layer to predict multiple possible outputs for given input video. We set the threshold for classification at 0.5. Similarly, the setup for linear evaluation on Tiny Virat remains the same, except the video encoder is kept frozen. Next, in the case of Sims4Action, we use SVM for linear evaluation. We report mean average precision and Fi-score for CharadesEgo and Tiny Virat, respectively, and top-1 accuracy for the rest of them.
Tiny Virat-v2 and MiT-v2. opening: opening, pull: pulling, activity_carrying carrying: carrying, entering: entering, exiting: exiting, loading: loading, talking: talking, riding, closing: closing, srtivitywwlking.... walking, push: pushing, activity_standing standing, specialized_ phoiin telephoning.
running: running. riding:
Sims4Action and MiT-v2. cook: cooking, drink: drinking, eat: eating, read_book- reading, use_phone: telephoning, walk: walking.
C.4 Actor shift
We study actor shift in two setups, i.e., animal domain and synthetic domain. We use the ActorShift [65] as the OoD validation set to test generalizability on animal actions, while using the video of overlapping classes from Kinetics700 [62] as InD training and validation. We use SVM for linear evaluation and follow standard techniques [72. 2, 4, 6, 78, 79, 7] as discussed earlier for finetuning. We report top-1 accuracy for both InD and OoD validation. We present the overlapping classes between Kinetics700 and ActorShift at the end of this subsection. The setup for synthetic domain shift is already discussed in the previous subsection (Viewpoint shift).
ActorShift and Kinetics700. sleeping, watching tv, eating, drinking, swimming, running, opening door.
C.5 Source shift
To study source shift, use the videos from the common classes of UCF101 and HMDB51. While using the UCF as the training split, we use the corresponding videos from HMDB as the OoD validation set and vice versa. Following standard protocol [72, 2, 6, 78, 79, 7]. we use SVM for linear evaluation and the pretrained encoder along with a fully-connected layer for finetuning. We report the top-1 accuracy for both setups. The class assignments are as follows.
UCF101 and HMDB51. FrisheeCatch: catch, RockClimbinglndoo:: climb, Diving: dive, (Basket- ballDuun,, Basketball): dribble, Fencing: fencing, GolfSwingg golf, (HandstandPushups, Handstand- Walking): handstand, (LongJump. JumpingJack): jump, SoccerPenalty: kick ball, PullUps: pullup, Punch: punch, PushUps: pushup, Biking: ride_ bike, HorseRiding: ride_horse, ThrowDiscus: throw, WalkingWithDog walk, Archery: bow,
C.6 Zero-shot
We perform zero-shot in two setups: (1) we train the video-text encoder on Kinetics400, while the noe-overlapping classes from UCF101 and HMDB51 are used for OoD validation [105]. (2) We train the models using the videos from Kinetics700 that do not share classes with UCF101 and HMDB51,
26

---

while the full UCFI01 and HMDB51 are used for OoD validation. Moreover, in both cases, we also use RareAct as the OoD benchmark, which consists of rare or unusual video actions.
Since, the pretraining of VSSL methods does not employ language, first, a video-text encoder is jointly trained learn the mapping between video and text representations. Following [42], use Vec [80] to extract the word embeddings from the labels, which are then fed to a text encoder consists of an MLP head. We measure the similarity between the video and text pairs as the dot product of L2 normalized video-text embeddings, referred to as logits, Next, the video text encoders are trained to minimize the Softmax cross-entrood eoos-yy error using the obtained logits and the class labels. We redirect readers to [42] for additional details of the training method. Please note, we also try with stronger language model (e.g., BERT[106] DistilBERT 104J) than Word2 however, it does not improve the performance, in fact, slightly worsens it. Since the labels in Kinetics are typically composed of only few words (mostly less than 2), the average pooled word embeddings work well in such cases. This finding is consistent with the observations of [42]. Moreover, we perform zero-shot in both setups when the pretrained video encoder is kept frozen and finetuned in an end-to-end manner.
C.7 Open-set
We perform open-set recognition in two setups: (1) We use the Kinetics400 for closed-set recognition, while the non non-overlapping classes from the UCF101 and HMDB51 are used for open-set, similar splits that have been used in zero-shot. (2) We use UCF101 and the non-overlapping classes of HMDB for closed-set and open-set recognition, respectively. We obtain the non-overlapping classes by manual inspection, presented at the end of this subsection.
To enable the models for open-set recognition, we follow [45] and train the pretrained encoders with the DEAR loss which aims to calibrate the uncertainty to ensure the models show high uncertainty towards the unknown and low uncertainty towards the known. We redirect readers to [45] for more information about open-set training. We conduct open-set recognition in both finetuned and linear evaluation setups. We observe that both supervised training and linear evaluation setups using VSSL encoders did not converge when Kinetics400 was used for closed set recognition. This is likely because the models tend to become over-confident due to the large number of known classes in Kinetics400 compared to the unknowns. To measure the models' performance we report the area under the ROC curve (AUC) for open-set recognition and accuracy for closed-set recognition.
HMDB51. The non-overlapping classes from HMDB51 used in open-set recognition while using UCFI01 as the closed-set: brush _air, cartwheel, chew, clap, climb buab__sttrrs,,s drink, eat, _foor, fla_f hit, hug, kick, kiss, laugh, pick, pour, push, run, shake shake_hands, hands, shoot_ ball, shoot_bow, shoot_ge gun, sit, situp, smile, smoke, somersault, stand, swing baseball, sword, sword _exercise, talk, turn, wave.
C.8
Toy experiments
In all of our toy experiments, we use the pretrained frozen encoders to extract the fixed representations, which are then used for K-means clustering". Additionally, we apply the Hungarian algorithm [107] for assigning the formed cluster to the targets and measure the accuracy. The details of the datasets used conduct the toy experiments are follows.
C.8.1 ToyBox
The ToyBox [108] consists of egocentric views of 9 distinct temporal transformations of different toy objects. The transformations include positive rotations, negative rotations, and translations across the y. and axis. A few representative frames illustrating such transformations are presented in Figure S5. Using the pretrained frozen encoders, we aim to correctly cluster different transformations across various objects. Moreover, as the static spatial information is non-discriminative to the temporal transformations, we aim to form clusters solely based on temporal representations. The main motivation behind this experiment is to disentangle spatial representations from the embedding space to independently evaluate the models on their ability to learn temporal dynamics.
2 skleean. cluster. KNeans
27

---

C.8.2 COIL100
We use COIL100 [34] to study viewpoint invariance which consists of images of different objects captured from different camera angles. We particularly choose an image-based based dataset for this experiment to restrict any discriminatory temporal features in the embedding space that may attribute to the outcome of the models' prediction. We present few representative frames in Figure S6
C.8.3 STL10
We use image dataset STL10 [71] to test the robustness of the pretrained encoder against low- resolution inputs. We systematically reduce the resolution from 112 to 162, please see a few examples in Figure S7. Similar to our viewpoint shift setup, we choose an image dataset to restrict any influence of temporal representations in the outcome.
LINE
185
positive rotations 2-axís
negative rotations --axis
positive rotations actoss 1-axis
negative rolations across 7/axis
KOU
positive rolations aCTOSS 2-aXiS
negative rolations acTosS z-axis
Figure S5: Samples from ToyBox showing rotations across different axes.
Figure S6: Samples from COIL.100 showing different viewpoints.
C.9 License of Dataset
In Table S6, we summarize the licensing terms for the datasets used in this study.
28

---

1:2
95
64
112
Figure S7: Samples from STL10 presenting low-resolution inputs.
Table S6: Licensing terms for the datasets used in this study.
Dataset
License
Link
CharadesFggo
Non-commercial use
https://prior. allenai org/ httg///ojetss//// charades-ego
Moments-in in- T2
Non-commercial use CC BY 4.0
http://nomente csail mit.edei
Kinetics
https://www.eeennt/ com/open-suurce/ kinetica
HMDB51 CC BY 4.0
https://serre-lab.cdpp
bromn.edu/ressorcc//
mad-a--lagge-uuann-moiii--
ToyBox Mimetics
CC BY 4.0 Open access
http:://aivalla..gittbb
computeer-visiiom/iiiatiiaa/
UCF101
Open access
https://ww.crrov.uee ucf
php
Tiny Virattr2
Open access
https://ww.crcr.uuc..ee
suuy=-ettoon-challlegg---oo00p1: #tabtwo
COIL100
Open access
sttpp:/www..cs.collbbaaaadddd/
sotrrare/sfft11b/co-1100ppp
STL-10
Open access
httpa://ee.staafooreee//aaaaas st110/ htte:r//wuaaaus.fggggaareaaaarrt
ActorShift
MIT
Batsstt/Accoo/hhhtt.zz///9999
Sims4Action
MIT
attpe///githuub.cooarrrrttbgr
sims4action
RareAct
Apache
http:://github..cmmmatitiii
RareAct
29

---

D Additional Experiments and Results
D.1 Pretrained on Kinetics700
In addition to the Kinetics400 used in the main paper. we further utilize the Kinetics700 700 to evaluate the benefits of incorporating more diverse data for out-of-distribution generalization. We use a subset of Kinetics700 comprising 480K samples, which doubles the size of Kinetics400. We use this subset to pretrain the VSSL methods for the same number of total iterations as Kinetics400. Unless stated otherwise, we compare the performance between Kinetics400 and Kinetics700 pretraining in linear evaluation, averaging over 3 trials. In Table S7 and Figure S8, we present high-level overview showing the impact of using more diverse data in pretraining. Table S7 highlights that non-contrastive methods (u-BYOL, o-SimSiam, v-DINO) benefit more from the inclusion of diverse data compared to the contrastive approaches (w-SimCL.R, M-MoCo) and e-MAF. Interestingly, we observe slight decrease in the performance of M-MAE in this setting. This could be attributed to several factors, such as the pretrained encoder VIT-B reaching saturation in the pretraining setup of v-MAE, resulting in no improvements in performance from the additional data. To address this, a larger backbone like ViT-L or ViT-H, used in prior works [57, 1], could be explored. Unfortunately, resource limitations prevented us from adopting a larger network to test this. Detailed results are presented in Tables S8, 89, S10, S11 and S12.
Table S7: Summary of the models' behavior with more diverse pretraining data is presented. In terms of overall performance, we report the average accuracy across all VSSL methods and distribution shifts, encompassing total of 66 experiments. Additionally, experiments for each individual VSSL method. The results demonstrate that non-contrastive methods (v-BYOL, o-SimSiam, v-DINO) benefit more from the inclusion of diverse data compared the contrastive approaches S--CLL,, v-MoCo) and M-MAE, Moreover, we observe a slight decrease in the performance of v-MAE when pretrained with more diverse videos from Kinetics700.
present results from 11
Methods
Pretraining
InD
OoD
Kinetics400
63.0
31.6
Overall
Kineties700
64.5 (1 1.5)
32.5 (1 0.9)
Kineties400
66.0
32.9
v-SimCLR
Kineties700
67.0 (f 1.0)
33.0 (1 0.1)
Kinetics400
67.1
33.1
u-MoCo
Kinetics700
67.7 (1 0.6)
33.9 (1 0.8)
Kinetics400
65.2
33.1
v-BYOL
Kinetics700
68.3 (1 3.1)
35.0 (1 1.9)
Kinetics400
62.0
31.5
u-SimSiam
Kinetics700
63.9
(1 1.9)
32.4 (1 0.9)
Kinetics400
63.1
31.5
D-DINO
Kinetics700
63.9 (1 0.8)
34.1 (1 2.6)
Kinetics400
54.4
27.4
0-MAE
Kinetics700
54.1 () 0.3)
26.6 (4 0.8)
30

---

Table S8: Comparison of video models under context shift when pretrained with Kinetics400 (#240K) vs. Kinetics700 (#480K). We highlight the best result in each scenario in bold. Interestingly, we observe consistent benefit from the inclusion more diverse data in the InD validation sets of both splits. However, the improvements in OoD are not consistently observed.
Context (10 class)
Context class)
Method
Kinetics400
Kinetics700
Kinetics400
Kinetics700
Ool
InD
OoD
InD
OoD
InD
OoD
InD
0-SimCLR
31.4
92.5
30.2
91.2
14.9
71.7
15.3
73.3
D-MoCo
29.7
92.2
27.7
93.1
15.0
74.0
14.1
75.0
v-BYOL
31.6
89.3
32.4
91.4
14.4
71.8
15.8
75.2
o-SimSiam
30.8
89.5
33.6
91.4
13.8
67.9
14.7
70.3
v-DINO
34.8
90.4
34,3
91.0
13.0
68.7
14.5
72.3
w-MAE
33.3
82.9
31.9
83.4
12.3
56.4
12.7
54.5
Table S9: Comparison of video models under viewpoint and actor shifts, when pretrained with Kinetics400 (#240K) vs. Kinetics700 (#480K). We highlight the best results in each scenario bold. The results demonstrate that adding diverse pretraining data significantly improves performance under egocentric viewpoint shifts and top-down synthetic domain shifts. However, less significant improvements are observed for surveillance viewpoint shifts and animal domain actor shifts. Moreover, the performance of v-MAE deteriorates when pretrained with Kinetics700.
View (ego.)
View (surv.+low res) View+Act (t-d+syn.)
Actor (animal)
Method
K400
K700
K400
K700
K400
K700
K400
K700
OoD InD
OoD InD
OoD InD
OoD InD
OoD
InD
OoD InD
OoD InD
OoD InD
0-SimCLR
12.7 14.7
13.7
15.7
26.1
39.1
24.9
40.2
42.4
67.8
43.1
71.2
67.9
91.7 68.7 92.4
v-MoCo
13.3
15.0
14.1
16.1
24.8
40.0
25.9
40.5
41.1
67.9
49.4
69.7
68.1
92.2
67.1 92.5
--BYOL
12.0
14.4
13.9
16.1
22.7
37.8
22.2
41.7
37.3
65.6 45.5
71.4
68.3
91.5 69.1 93.3
u-SimSiam
11.6
14.1
12.8
14.9
23.3
34.3
21.4
37.3
40.0
65.5 37.9
67.0
68.1 91.1 67.7 91.5
u-DINO
12.0
14.4
13.4
15.7
22.3
35.3
22.9
38.0
35.3
62.9
42.0
68.5
66.7
90.7 68.3 91.2
e-MAE
10.9
13.7
10.9
13.8
23.5
32.0
23.1
31.2
37.8
58.0
33.5
57.6
59.8
85.9
56.2 86.3
Table S10: Comparison of video models under source shift, when pretrained with Kinetics400 (#240K) vs. Kinetics700 (#480K). We highlight the best results each scenario in bold. DINO and v-SimSiam benefit the most from the inclusion of diverse pretraining data, while the other methods show the same or worse performance.
UCF/HMDB
HMDB/UCF
Method
Kineties400
Kinetics700
Kinetics400
Kinetics700
OoD(HI) InD(U) OoD(HI) InD(U) OoD(U) InD(HI)
OoD(U) InD(H)
D-SimCLR
47.7
96.0
45.1
96.5
55.1
82.0
55.1
83.6
3-MoCo
51.5
97.1
48.3
96.3
57.2
84.5
54.6
82.6
v-BYOL
51.4
94.5
50.5
96.8
63.1
79.9
61.1
82.8
v-SimSiam
46.1
92.6
48.0
92.5
52.2
76.2
53.1
80.5
e-DINO
49.3
94.3
50.3
95.8
53.8
77.5
58.2
81.8
v-MAE
39.5
89.2
39.5
89.3
39.1
72.8
37.5
72.4
31

---

Table S11: Comparison of video models in zero-shot recognition, when pretrained with Kinetics400 (#240K) vs. Kinetics700 (#480K). We highlight the best results in each scenario in bold. The results demonstrate that the inclusion of diverse data generally improves the zero-shot performance. We observe significant improvements in most cases, with the exception of v-MAE on UCF and e-BYOL HMDB. Moreover, we also notice significant improvements in InD performance, particularly in the case of non-contrastive methods and v-MAE
UCF
HMDB
RareAct
Kinetics400 (InD)
Method
K400
K700
K400
K700
K400
K700
K400
K700
0-SimCLR
37.2
39.0
18.6
19.0
7.7
9.4
56.8
57.8
2 o-MoCo
35.2
40.6
19.5
21.6
8.7
9.4
58.4
59.5
V- BYL
33.0
44.0
22.4
21.1
7.5
9.9
57.4
61.0
u-SimSiam
34.0
39.2
18.8
20.3
7.7
8.3
50.4
52.3
e-DINO
34.3
17.2
20.0
9.5
53.4
58.3
e-MAE
25.5
24.1
14.2
17.2
5.8
6.2
35.9
35.4
Table S12: Comparison of video models in open-set recognition, when pretrained with Kinetics400 (#240K) vs. Kinetics700 (#480K). We highlight the best results in each scenario in bold. The results exhibit very similar trend in both pretraining setups. For example, while v-SimCLR and v-MoCo perform fairly well closed-set, they show near chance-level performance in open-set. MAE performs poorly in both open-set and closed-set, in both pretraining setups. v-DINO achieves the best open-set performance in both setups while retaining decent closed-set performance.
Pretrained on Kinetics400
Pretrained on Kinetics700
Method
Open-set (AUC) U/H
Closed-set
Closed-set
Open-set (AUC)
Closed-set
Closed-set
w/ DEAR (Acc.) w/ CE (Acc.)
w/ DEAR (Acc.) w/ CE (Acc.)
U101
U101
U/H
U101*
U101
e-SimCLR
49.9
11.6
84.1
49.7
18.5
83.6
g-MoCo
50.7
32.7
84.9
53.1
53.6
84.7
o-BYOL
56.0
63.7
85.4
85.5
86.7
o-SimSiam
73.7
79.5
82.5
69.6
80.5
82.3
e-DINO
79.4
80.4
80.9
82.0
84.5
85.0
e-MAE
66.0
74.6
76.2
65.6
74.2
75.1
Table S13: Comparison of VSSL methods in zero-shot recognition, when pretrained with Kinetics700 (#480K). We highlight the best results in each scenario in bold. In addition to Table S11, we also finetune video-text encoders with Kinetics700 and perform zero-shot recognition using the full UCF101 and HMDB51, along with RareAct. Please note that during both pretraining and finetuning, we discard the videos from Kinetics700 that belong the overlapping classes from UCF101 and HMDB51. a-MAE clearly shows superiority when validated on UCF101 and HMDB51, while D-MoCo outperforms others when validated on RareAct.
Method
UCF
HMDB RareAct
e-SiCLL
45.7
34,0
15.3
e-MoCo
48.4
33,9
18.4
B-BYOL
47.3
33.4
15.5
e-SimSiam
37.9
29.1
13.3
8- a-DINO
46.6
32.9
16.2
0- e-MAE
50.1
37.3
16.0
32

---

100.0
80.0
Kinetics700 (480K)
60.0
40.0
20.0
40.0 Kinetics400 (240K)
y=x
Context shift
(U/H)
y=mx+c (InD)
shift cls)
v-BYOL
y=mx+c (OoD)
Zero-shot (RareAct)
V-DINO
In-distribution
view shift (syn.+t-down)
V-MAE
ditt-f--iitrrbbution
Zero-shot (UCF)
V-MOCO
Source shift (HJU)
View shift (surv.+low res.)
V-SIMCLR
Actor shift (animal)
View shift (ego.)
V-SIMSIAM
Zero-shot (HMDB)
Figure S8: We present a high-level depiction of the models' performance when pretrained with Kinetics400 vs. Kinetics700. When pretrained with Kinetics700, the trend line of InD performance (blue dashed line) is slightly higher than that of the OoD performance (brown dashed line).
33

---

D.2 Temporal dynamics
In this subsection, we provide the additional results of our experiment on the temporal dynamics of VSSL methods. We present the performance on temporal transformation recognition in Figure S9, Moreover, we present the performance of VSSL methods in classifying the same objects from static videos, in Figure $10.
Transformation recognition (Temporal)
50.0
50.0
40.0
40.0
40.0
40.0
Accuracy
Accuracy
Accuracy
Accuracy
30.0-
30.0
30.0
30.0
20.0
10.0
v-SYOL
--DINO
ESIMO V-SI4SSAAS SUPERVIEED
10.0
YMAA
10.0
v-rol
WMAE
SIM2LL
V-SISAA SUPONVISDD
10.0
SIMCLL
(a) Cat
(b) Duck
(c) Giraffe
Horse
50.0
50.0
50.0
50.0
Accurscy
Accuracy
Accuracy
Accuracy
30.0
30.0
20.0
20.0
20.0
20.0
10.0
v-81OL
DDO
L-SUPERVSEE
10.0
K-DNO
SiNcc u.SUPERVIstD
FSSIMSIAI
10.0
D-w
V-SMSSAM
I.SUPERVSEED
HeaaL vowo
MMCCOV +-SMMLLA
VSIMSIAM ASUPERVSED
(0) Ball
(0 Cup
(g) Mug
(h) Spoon
40.0
40.0
40.0
Accuarcy
fccuracy
Accuracy
30.0
30.0
30.0
30.0
20.0
20.0
10.0
D-DIOO
SUPERVISED
verok.
10.0
DINNO
WAME
wHocOV KISINCLR
L-SWSSAM
SUPERVISEED
10.0
varoot DiNOO
WHAt
HMocOVs
PSINCLA
KSMSSAM
SSiPERVSED
Airplane
Car
Helicopter
Truck
50.0
50.0
50.0
50.0
40.0
40.0
40.0
40.0
Accuraty
Accuracy
Accuracy
Accuracy
20.0
20.0
20.0
20.0
10.0
v-YOL
MAE
SSMCLAA
SUPERVIED
10.0
v-rol
kaiNd
ViMAE
MOCOv
R
SUPERVEED
10.0
VOWND
Hacors
S.SIILLA
S-SHAA SUPERVEED
10.0
H-vot
volla
V-MAE
A
SUPERVISED
(i) Animals (4 objects)
(k) Houscholds (4 objects)
Vehicles (4 objects)
(m) All (12 objects)
Figure 89: Evaluating performance of video models egocentric transformation recognition using ToyBox [70]. The models are used to classify total types of temporal transformations including positive rotations, negative rotations, and transformations across the y, and axes. In all of the cases, e-MAE consistently shows superior performance.
34

---

Visual object recognition (Spatial)
70.0
70.0
70.0
5000
60.0
40.0
40.0
4000
30.0
30.0
30.0
30.0
20.0
s-r voleo
Satttc -cattiie SIOPERVEED
20.0
-pot
20.0
wirck
[4] Animals
(b) Houscholds
(c) Vehieles
(d) All
Figure S10: Evaluating performance video models in egocentric object recognition. The models used to classify the videos of static objects including 4 types of animals, household items, cars, and all them together (a total of 12 classes). In all cases w-MAE consistently exhibits the lowest performance. Among all the categories, the models perform significantly better in classifying household objects than animals or vehicles. This likely because the pretraining data Kinetics400 consist of several human actions involving household objects e.g., kitchen utensils, etc.
35

---

D.3 Comparing InD vs. OoD performance under natural distribution shifts
To gain high-level understanding of the models' performance in InD vs. OoD, we visualize their accuracy in 2D space in Figure S11. It is noteworthy that superior InD performance does not necessarily translate to better OoD performance. We observe cases where models exhibit similar InD performance but significantly differ in their performance under distribution shifts. Ablation variants of Figure S11 are presented Figure $12. Statistical analyses on the robustness of video models are presented in Appendix D.7.
100.0-
80.0
Out-of-distribuuion accuracy
60.0
40.0
Drby
20.0
40.0
100.0
In-distribution accuracy
y-x
Context shift
v-8YOL
Linear (empty)
Zero-shot (RareAct)
V-DINO
Finetune (filled)
Domain+ view shift (syn.+t-down
V-MAE
Source shift (H/U)
Zero-shot (UCF)
V-MOCO
Domain shift (animal)
View shift (surv.+low res.)
V-SIMCLR
Zero-shot (HMDB)
View shift (ego.)
V-SIMSIAM
Context shift (10 cls)
Source shift (U/H)
V-SUPERVISED
Figure $11: We compare the InD OoD performance of video models under different distribution shifts. The filled markers indicate the finetuned results, while the empty markers represent the results from linear evaluations. In all cases, we observe decrease in performance in the OoD validation set, as indicated by the data points below the = x line.
36

---

50.0
36.
6..0
12.0
10.0
I-dinttukin acnurtey
d-ditrr-door scorcyy
Indiitaurionn actancy
ymd
yera
persiic
yma
Bhatune
2V SUPERV
Fretune
VSMS VIUUIE SED
Fratuno
p-rooed 1.7
y-msec LC7
y-m
V DRANE0
Context (10
Contest shift (50
Viewpoint (cgo.)
52.5
$5.0
30.0-
$1.0
47.5-
45.0
40.0
0
22.
00.0
$2.0
320
35.
450
35.3
263 700 3S0 1OL0
1001
85.3
Indiettttaamn ecorasy ay
r-db budte auareyy
Irclabtousee ectatecy
Linar
Linerr
6010L
Unver
-010L
Annure
Proture
Prstame
+-100
VEREAM
parserc 201
MMA
SUrEE SED
yonstci N07
VOUTE 5E0
yemsic 1LT1
+
WOUPEWWBED
(d) Viewpoint shift bear-tlow
(e) Viewpontsscctor shift (top aoppsteesyt
Actor shift (animal)
30.5
$1.0
65.00
35.0
40.5
95.0
1,15
8.00
31.0
25.0
25.0
55.0
66,5
I-d6rtbutum ecoa dyy
1-db B-dit uurg
y-18
(-
v-Mac
Viok
yox
y-mheec ITT)
perterc 221
yomrtc 103
yemetc (LT)
(2) Source shift
Uhi (IBU)
35.0
31.0
40.0-
2.0
31.0
15.0
11.0
15.0
i-timitutio artubey
dor acorsy
yad
yerr
yea
peroiic
VMDCO
Rretuee
VB80
Freture
VSMS
y-meoc 3.7
UMA
y-meecid7 LC7
VISUNERU
Zero-shot (HMDR)
(k) Zero-shot (RareAct)
An ablated view of InD vs. OoD performance under natural distribution shifts, utilizing both frozen (empty markers) and finetuned (filled markers) encoders. Here, y max + shows a linear fit for the given data points (i.e., projected InD and OoD performance metrics). The data points above linear fit indicate more robustness. (a and b) o-Supervised demonstrates superior performance in linear evaluation, while o-MAR achieves the best results when finetuned. On the other hand, although u-MoCo and v-SimCLR show strong performance in InD validation, they exhibit weaker generalization in out-of-contes, scenarios. (c, d, and e) Overall, v-SimCLR and e-MoCo perform better in viewpoint shifts in both linear and finetuning. M-MAE (finetuned) shows single instance of superior results for the specific case of low-resolution surveillance camera shift due to its robustness in now-resolution inputs. (f) In animal domain actor shift, v-BYOL achieves the best results in linear evaluation, whereas, D-SimCLR outperforms others when finetuned. (g and h) B-BYOL achieves the best performance under source shifts in all setups. and j) Frozen B-BYOL achieves the best zero-shot recognition on UCF but performs poorly HMDB. On the other hand, the frozen D-SimCL.R performs the best on HMDB, while generalizing poorly on UCR (k) Overall, video models generalize poorly in zero-shot recognition of unusual actions (RareAct).
Figure S12:
37

---

D.4 Effect of synthetic perturbations
In addition to the natural distribution shifts, we further extend our work investigating the performance of VSSL methods under synthetic perturbations. We follow the setup proposed in [53] to create the synthetic perturbation of varying severity on scale of 5. We apply total of 16 different synthetic perturbations belonging to the common spatio-temporal augmentation techniques such as noise addition, blurring, temporal perturbation, and camera motion. We test the robustness of the frozen encoders against synthetic perturbations on 2 benchmarks UCF101 and Kinetics400, presented in Figures S13 and $14 respectively. The results demonstrate that with increasing perturbation severity, the performance of video models deteriorates. However, it is important to acknowledge that certain visual augmentations, such as noise and blur, are already applied during the self-supervised pretraining phase. As result, they may not be considered as distribution shifts within the scope of our setup. Furthermore, it is neither possible to train sulf-suue methods without such augmentation techniques. Despite these considerations,
present these results for the sake of completeness.
Since spatial perturbations are extensively covered literature, we refrain from explaining them here and redirect readers to [53, 54, 49]. Instead, we briefly discuss the temporal perturbations [53]. 'Sampling': : using different frame rates than used training; 'Reverse sampling': reversing the temporal order with different sampling rates; 'Jumble": shuffling frames in segments; 'Box jumble': shuffling the segments instead of frames; 'Freezing': randomly freezing video frames; 'Random shuffle": randomly shuffle the frame orders, please note random shuffle has just one severity level.
38

---

2"
1241 Palve
Zoee
PRECO
PNTUU00O Soretty
2
250L
2200
14000
Petubaton
464
43 22
Figure S13: Spatio-temporal perturbations on UCF101 (linear evaluation). When subjected to extreme perturbations, the performance of VSSL methods experiences a significant decline, reaching near-chance levels. Notably, v-MAE exhibits the worst performance across all setups. Among all methods, BOBY demonstrates superiority in scenarios involving blur and temporal perturbations, while D-MoCo outperforms others in rotations and translations. Furthermore, e-DINO excels when tested on noisy videos.
39

---

LEND
30
State Sa120
2oom B67
Peloerie Svery
Seaett
Petubaal
12T24
Det KTO2
Sancen CheTs
Figure S14: Spatio-temporal perturbations Kinetics400 (linear evaluation). When subjected extreme perturbations, the performance of VSSL methods experiences significant decline, reaching near-chance levels. Notably, v-MAE exhibits the worst performance across all setups. Among all methods, v-MoCo demonstrates superiority in scenarios involving blur, rotations, and temporal perturbations, while B-BYOL. outperforming the others when tested on noisy videos.
40

---

D.5 Effect of finetuning in out-of-distribution generalization
D.5.1 Linear vs. finetuning performance comparison
We conduct thorough study evaluating the impact of finetuning across different VSSL methods. We present high-level overview in Table $14 and more detailed results in Figures S15, S16, and S17.
Table S14: A high-level summary of the impact of the models' performance when finetuned, both InD and OoD. Overall, finetuning improves performance in both Inl) and OoD. However, the performance varies based VSSL methods. Generally, the improvements in Inl performance tend to be more significant than those in OoD. We note that v-MAE benefits the most from finetuning due to its extremely poor generalizability when using frozen encoders.
Methods
Eval. setup
InD
OoD
Linear
63.2
31.5
Overall
Finetune
71.9 (1 8.7)
37.4 (f 5.9)
Linear
65.5
30.7
e-Supervised
Finetune
69.0 (1 3.5)
35.1 (.4)
Linear
66.0
32,9
v-SimCLR
Finetune
73.4 (1 7.4)
38.2 (1 5.9)
Linear
67.1
33.1
u-MoCo
Finetune
73.6 (1 6.5)
39.1 (1 6.0)
Linear
65.2
33.1
v-BYOL
Finetune
73.2 (f 8.0)
38.7 (f 4.6)
Linear
62.0
31.5
u-SimSiam
Finetune
70.4 (1 7.6)
36.2 4.7)
Linear
63.1
31.5
N-DINO
Finetune
71.0 (1 7.9)
36.4 (1 4.9)
Linear
54.4
27.4
U-MAE
Finetune
72.6 (1 7.2)
38.3 (1 10.9)
41

---

255
200
Figure S15: A detailed comparison of the impact of finetuning InD OoD. For optimal viewing, please rotate 90 degrees to the left D. The results demonstrate that the benefits finetuning highly dependent finetuning tends to be more advantageous in scenarios involving actor shifts (animal domain and synthetic domain). Conversely, its benefits relatively less pronounced under viewpoint shifts and zero-shot recognition. Moreover, the impact of finetuning in source shift and context shift is mixed. For example, while we notice significant benefits in UCF HMDB shift, finetuning in f'ct worsens the performance in the case of HMDB to UCF shift. Additionally, finetuning is more beneficial when evaluated on smaller benchmark with just 10 out-of-context classes. However, the improvements are relatively modest in a more challenging setup involving 50 out-of-context classes.
the VSSL. method and type of distribution shift. Our analysis reveals that
42

---

80.0
y=x
Finetune accuracy
y=mx+c +c (Bath)
y-mx+c (InD)
60.0
y=mx+c (OoD)
In-distribution (filled)
Out-of-distribution (empty)
Source shift (H/U)
Actor shift (animal)
Zero Zero-shot (HMDB)
Context shift (10 cis)
40.0
Context shift (50 cls)
shot (RareAct)
Actor+view shift (tyoo+):down)
Zero-shot (UCF)
Aa
View shift (surv.
shift (ego.)
shift (U/H)
20.0
V-BYOL
V-MAE
V-MOCO
V-SMMLLR
V-SIMSIAM
V.SUPERVISED
60.0
80.0
100.0
Linear accuracy
Figure S16: Comparing the performance of video models in linear evaluation vs. finetuning. In the plot, the filled markers represent InD results, while the empty markers represent the OoD results. Overall, we observe that finetuning consistently leads to improved performance, indicated by the data points lying above the diagonal line (y = x),, with only one exception. is worth noting that average improvement in InD performance (shown by the blue dashed line) is higher compared to the improvement OoD performance (shown by the brown dashed line).
43

---

13.0
1000
2440
190
481
50
260
180
280
18.0
203 000
Una sauutyy
Jeear aCCe90
Unea scane
PINGE
+0
WGPSHM
seewer
1030
1MMMM
Dx witommee
V
shift classes)
Contest classes)
Viewpoint shift (egn.)
00.0
$6.6
16.0
06.0
85
20.0
200
29.0
36.0
25.
303
203
!FC
305
400
300
805
TC5
000
122 EEC TTE 80
EXT XEX 15
18%
8- B-acoiauee thiee
Gddedaants kesttt
(d) Viewpoint shift (surv:tlow res.
(e) Viewpolnt shift (topdooaassytthekk)
(8) Actor shift (animal)
3000
76.0
200
10.3
40.0
12.0
0250
4800
26.0
300
43.0
303
X0
Uner scUrt)
+210
+SNC
you
Endiciiation (iadi
(g) Source shift (UUHI
(b) Source shift (H/U)
(i) Zereehot (UFCF)
703
532
233
40.2
32.0
20.3
233
23. 123
214
RED
150
480
5I0
16-
300
+992
+595.0 4M
Zeroscuu (HMDB)
(k) Zerr-shot (RareAct)
Figure S17: An ablated view of linear vs. finetuning performance comparison. Our results demonstrate that improved InD performance does not necessarily translate better OoD performance. We observe cases where models exhibit similar InD performance but significantly differ in their performance under distribution shifts, e.g., please see the linear results in d, f, and i.
44

---

D.5.2 In-distribution overfitting due to finetuning
As mentioned in the main paper, while finetuning enhances overall performance, it is susceptible to in-distribution overfitting In other words, while additional training leads to improved performance on InD validation, it adversely affects OoD performance. We present few examples of such instances Figure S18. In practice, we apply early stopping to tackle such overfitting.
08-
Epachs
Esochs
Epochs
b-BYOL
e SimCLR
V-DINO
Action shift (symhetic)
Actor shift (symihetic)
Actor shifi (yythetic)
40
00
Epochs
Epects
Epochs
Actor shift (animal)
Zero shot (OaD: HMDB, IaD: Kinetics400)
Zero shot (OoD: UCF, InD: Kinetics100)
Figure S18: A few examples showing in-distribution overfitting caused during finetuning.
45

---

D.6 High-level overview
The high-level overview presented Figure summarizes our findings from Tables 1, 2, 2, and 6(a) of the main paper. To create the overview plot, we first normalize the OoD performance (e.g., accuracy, mAP) of each experiment that the best-performing method. Thus, the best performing method gets 1.0 and all other methods get a score in [0, 11. Next, we group the experiments based on their super-categories and then take their average score. For example, in the case of source shift, we compute this average by grouping the normalized scores of 4 experiments, namely linear and finetuning evaluation of both UCF/HMDB and HMDB/UCF shifts. We follow similar steps for all the shifts (details are given below), resulting in 7 X 6 matrix, where 7 represents the number of video methods, and 6 represents the total number shifts. This final matrix then used to color-code and present high-level summary of all methods across all shifts.
Context shift: Table
Context shift (10 classes) linear, best performing method: v-Supervised Context shift (10 classes) finetune; best performing methods: u-Supervised, v-BYOL, w-MAE
Context shift (50 classes) linear; best performing methods: e-Supervised, e-MoCo
Context shift (50 classes) finetune; best performing method: v-MAE
Viewpoint shift: Table
Viewpoint shift (egocentric) linear; best performing method: v-MoCo
Viewpoint shift (egocentric) finetune; best performing method: v-MoCo
Viewpoint shift (surveillance+low resolution) lincar, best performing method:
SimCLR
Viewpoint shift (surveillance+low resolution) finetune; best performing method: v-
MAE
Viewpoint+actor shift (top-down+synthettic linear; best performing method:
SimCLR
Viewpoint+actor shift (top-down-synthetic) finetune; best performing method: v-
SimCLR
Actor shift: Table 2
Viewpoint+actor shift (lop-downssynthetic) linear; best performing method: v-
SimCLR
Viewpoint+actor shift (toyyyeintyyt finetune; best performing method:
SimCLR
Actor shift (animal) linear; best performing method: w-BYOL
Actor shift (animal) finetune; best performing method: v-SimCLR
Source shift: Table 3
Source shift (UCF/HMDB) linear; best performing method: v-MoCo
Source shift (UCF/HMDB) finetune; best performing method: e-BYOL
Source shift (HMDB/UCF) linear; best performing method: v-BYOL
Source shift (HMDB/UCF) finetune; best performing method: v-BYOL
Zero-shot recognition: Table
Zero-shot (UCF) linear, best performing method: v-SimCLR
Zero-shot (UCF) finetune; best performing method: e-MoCo
Zero-shot (HMDB) linear; best performing method: R-BYOL
Zero-shot (HMDB) finetune; best performing method: e-MAE
Zero-shot (RareAct) linear; best performing method: e-MoCo
Zero-shot (RareAct) finetune; best performing method: v-SimSiam
Open-set recognition: Table 6(a)
Open-set (K400/UCF) finetune; best performing method: e-SimCLR
- Open-set (K400/HMDB) finetune; best performing method: e-MoCo
Open-set (U101/HHDB) finetune; best performing method: e-MoCo
46

---

D.7
Statistical analysis
We present detailed statistical analyses to verify the statistical significance of all the major findings/- claims discussed in the main paper. In particular, we compare the robustness of the video models under different distribution shifts, after compensating their InD performance. The adjusted OoD performance difference between the two methods is calculated as:
Adjusted AODD = AO.D m X IND 1.96 x SEOoD
Here, m E [0, 1] is the slope linear fit projected InD and OoD performance measures all the models for particular shift, calculated as:
Σ (x+ B) 7)
(34 x)²
where z; and y refer to InD and OoD performance measures for 7 method and E [1. n], and is the total number of methods studied. Moreover, F and refer to the mean InD and OoD performances, respectively. Between Method and Method 2, we calculate Λo.D = y1 y/ and Ann = I1 22. while SEO.D is the standard error of Method I's OoD performance across several trials. We consider the standard error with statistical significance at 95% confidence. Finally, we accept Method is more robust than Method 2, if Adjusted AO.D 0. The results of our statistical analyses presented in Tables S15 through S26.
Table $15: Comparative statistical analysis of the robustness of e-MAE and u-Supervised under context shift (original numbers from Table I).
Adjusted AOOD
Distribution shift
Method 1
Method 2
Lin.
FT.
Supervissd
e-SimCLR
5.24
5,90
Supeervise
u-MoCo
6.96
2.20
SSpeervisedd
e-BYOL
5.00
0.00
Superrvised
u-SimSiam
5.73
0.80
e- Supervised
e-DINO
1.81
0.80
Context (10 class)
e-MAE
e-SimCLR
0.25
5.90
w-MAE
v-MoCo
1.97
2.20
e-MAE
v-BYOL
0.01
0.00
w-MAE
v-SimSiam
0.74
0.80
M-MAE
v-DINO
-3.17
0.80
o-Supervised
v-SimCLR
0.44
2.68
e- Supervvised
v-MoCo
0.66
1.28
Superviseed
u-BYOL
0.92
-0.07
e- Supervised
u-SimSiam
0.99
0.08
e Supervised
u-DINO
1.88
0.53
Context (50 class)
MMMAE
e-SimCLR
-0.82
7.20
MMMAE
u-MoCo
-0.59
5.80
e-MAE
v-BYOL
-0.34
4.45
--MAE
u-SimSiam
-0.27
4.59
D-MAE
v-DINO
0.63
5.04
47

---

Table $16: Comparative statistical analysis of the robustness of e-MAE and o-Supervised in learning
temporal dynamics (original numbers from Figure 3(a)).
Distribution shift Method
Method 2
Adjusted ΔOOD
u-Supervised
e-BYOL
0.06
v-Supervised
e-SimSiam
0.05
u-Supervised
v-DINO
0.05
o-Supervised
v-SimCLR
0.06
u-Supervised
v-MoCo
0.06
Temporal learner
e-MAE
v-BYOL
0.19
v-MAE
v-SimSiam
0.19
e-MAE
v-DINO
0.19
v-MAE
v-SimCLR
0.20
--MAE
e-MoCo
0.20
Table S17: Comparative statistical analysis of the robustness of contrastive methods (v-SimCLR and o-MoCo) under viewpoint shift (original numbers from Table 2).
Adjusted ÅooD
Distribution shift
Method 1
Method
Lin.
FT.
v-SimLLR
e-BYOL
0.21
0.20
--SimLLR
v-SimSiam
0.35
1.28
v-SimCLR
v-DINO
0.24
1.16
V-SimLLR
v-Supervised
0.71
1.44
v-SimCLR
v-MAE
0.65
2.01
Viewpoint (ego.)
v-MoCo
u-BYOL
0.56
0.69
v-MoCo
v-SimSiam
0.71
1.78
v-MoCo
v-DINO
0.60
1.66
*-MoCo
u-Supervised
1.06
1.94
M-MoCo
u-MAE
1.00
2.51
v-SimLLR
u-BYOL
2.50
2.77
SimCCR
u-SimSiam
0.98
0.51
S-SiCLR
u-DINO
2.22
1.84
--SiCLL
v-Supervised
0.89
0.10
0-SimCLR
v-MAE
0.21
-0.16
Viewpoint (surv.+low res)
e-MoCo
v-BYOL
0.49
1.44
v-MoCo
v-SimSiam
-1.03
-0.81
M-MoCo
v-DINO
0.22
0.52
v-MoCo
u-Supervised
-1.12
-1.22
M-MoCo
v-MAE
-1.79
-1.48
v-SimLLR
v-BYOL
1.87
6.30
v-SimCLR
v-SimSiam
-0.88
6.70
SimCCL
u-DINO
2.06
11.5
v SimCLR
u-Supervised
8.45
16.0
D-SimLLR
u-MAE
-3.93
-0.60
View+Act (t-down+syn.)
e-MoCo
B-BYOL
1.81
4.80
v-MoCo
u-SimSiam
-0.94
5.20
v-MoCo
D-DINO
2.00
10.0
v-MoCo
e-Supervised
8.39
14.5
U-MoCo
v-MAE
-3.99
-2.10
48

---

Table S18: Comparative statistical analysis of the robustness of contrastive methods (e-SimCLR and
o-MoCo) in learning viewpoint invariance (original numbers from Figure 4(a)).
Distribution shift
Method 1
Method 2
Adjusted AOOD
e-SimCLR
e-BYOL
0.05
v-SimCLR
v-SimSiam
0.04
v-SimCLR
v-DINO
0.02
v-SimCLR
o-Supervised
0.09
v-SimCLR
v-MAE
0.09
Viewpoint invariance
e-MoCo
v-BYOL
0.04
v-MoCo
S-SimSiam
0.03
e-MoCo
e-DINO
0.02
v-MoCo
r-Supervised
0.09
e-MoCo
e-MAE
0.09
Table S19: Comparative statistical analysis of the robustness of single stream networks (e-Supervised and e-MAE) in learning temporal dynamics and Siamese networks (v-BYOL, v-SimSiam, v-DINO, u-SimCLR, and u-MoCo) in learning viewpoint invariance (original numbers from Figures 3(a) and 4(a)).
Distribution shift
Method
Method
Adjusted AOoD
o-Supervised
w-BYOL
0.06
o-Supervised
u-SimSiam
0.05
2 o-Supervised
v-DINO
0.05
Supervisdd
v-SimCLR
0.06
Superrvsee
v-MoCo
0,06
Temporal learner
w-MAE
v-BYOL
0.19
e-MAE
v-SimSiam
0.19
e-MAE
v-DINO
0.19
4-MAE
v-SimCLR
0.20
e-MAE
e-MoCo
0.20
e-SimCLR
v-Supervised
0.09
v-SimCLR
v-MAE
0.09
e-MoCo
v-Supervised
0.09
e-MoCo
e-MAE
0.09
B-BYOL
o-Supervised
0.05
Viewpoint invariance
B-BYOL
v-MAE
0.05
v u-SimSiam
e-Supervised
0.06
v-SimSiam
v-MAE
0.06
w DINO
D-Superrised
0.07
D-DINO
e-MAE
0.07
Table S20: Comparative statistical analysis of the vulnerability (opposite of robustness) of v- Supervised under multiple shifts (original numbers from Table 2).
Adjusted AOoD
Distribution shift
Method
Method
Lin.
FT.
o-Supervised
u-BYOL
-7.11
-9.70
u-Supervised
v-SimSiam
-9.87
-9.30
Supervised
v-DINO
-6.92
4.50
View+Act (t-down+syn.)
v Superviiee
v-SimCLR
-10.6
-16.0
Supervised
e-MoCo
-9.26
-14.5
vー S-Sppevvieed
v-MAB
-12.9
-16.6
49

---

Table S21: Comparative statistical analysis of the robustness of e-BYOL under source shift (original numbers from Table 3).
Adjusted AOOD
Distribution shift
Method 1
Method 2
Lin.
FT.
v-BYOL
o-Supeevised
8.56
6.60
e-BYOL
e-SimSiam
6.96
2.90
e-BYOL
v-DINO
6.70
0.30
HMDB to UCF
e-BYOL
e-SimCLR
9.96
2.00
B-BYOL
w-MoCo
10.3
1.10
B-BYOL
a-MAE
16.7
10.8
B-BYOL
Supervisee
3.14
2.40
B-BYOL
v-SimSiam
2.72
6.80
B-BYOL
v-DINO
1.23
6.80
UCF to HMDB
B-BYOL
v-SimCLR
4.52
5.90
e-BYOL e-MoCo
1.89
5.10
e-BYOL
w-MAE
6.00
3.10
Table S22:
Comparative statistical analysis of the impacts of finetuning over frozen encoder under source shift (original numbers from Table 3).
Distribution shift
Method 1
Method 2
Adjusted AOOD
HMDB to UCF UCF to HMDB
Finetune
Linear
-8.67
Finetune
Linear
1.97
Table S23:
Comparative statistical analysis of the robustness of contrastive methods (v-SimCLR and w-MoCo) in open-set recognition when finetuned (original numbers from Table 6(a)).
Distribution shift
Method
Method 2
Adjusted A AOOD
v-SimCLR
e-BYOL
2.44
v-SimCLR
u-SimSiam
2.42
e-SimCLR
v-DINO
2.94
v-SimCLR
e-MAE
7.85
Kinetics400/UCF
v-MoCo
u-BYOL
0.72
u-MoCo
e-SimSiam
0.70
v-MoCo
v-DINO
1.22
M-MoCo
e-MAE
6.13
v-SimCLR
v-BYOL
0.04
v-SimCLR
u-SimSiam
2.12
v-SimCLR
v-DINO
1.03
u-SimCLR
v-MAE
5.09
Kinetic4400/MMBB
e-MoCo
e-BYOL
0.76
e-MoCo
u-SimSiam
2.84
M-MoCo
v-DINO
1.75
v-MoCo
e-MAE
5.82
e-SimCLR
e-BYOL
2.58
u-SimCLR
v-SimSiam
4.29
v-SimCLR
v-DINO
1.31
u-SimCLR
v-MAE
11.38
UCF101/HMDB
M-MoCo
v-BYOL
3.20
M-MoCo
v-SimSiam
4.91
M-Mo
v-DINO
1.94
M-MoCo
e-MAR
12.0
50

---

Table $24:
Comparative statistical analysis of the robustness slightly weak frozen encoders (v-
DINO, v-SimSiam, and n--uupervised) in open-set recognition in linear evaluation (the original numbers from Table 6(b)).
Distribution shift
Method 1
Method 2
Adjusted AOOD
e-DINO
v-BYOL
23.4
e-DINO
e-SimCLR
29.5
v-DINO
v-MoCo
28.7
v-DINO
e-MAE
13.4
U-SimSiam
e-BYOL
17.5
v-SimSiam
e-SimCLR
23.5
UCFI0L/HMDB
v SimSiam
v-MoCo
22.7
v- SimSiam
v-MAE
7,40
v Supervised
v-BYOL
21.2
Superrvssed
v-SimCLR
27.3
v -uSppeviseed
v-MoCo
26.5
v u-Suuervised
A-MAE
11.2
Table S25: Comparative statistical analysis of the robustness all methods in zero-shot recognition (original numbers from Table 4).
Adjusted ΔOoD
Distribution shift
Method
Method 2
Lin.
FT.
B-BYOL BYOL
u-SimSiam
-5.02
-0.24
U-DINO
-4.01
-2.13
c BYOL
u-MAE
-2.89
-1.56
BYOL
0-SimCLR
-5.37
0.95
t- B-BOL
e-MoCo
-2.70
-5.30
SimSiam
u-BYOL
2.31
0.24
r-SimSiam
u-DINO
-0.42
-1.89
o-Simiiam
v-MAE
0.70
-1.32
t-SimSiam
v-SimCLR
-1.79
1.19
--SimSiam
U-MoCo
0.89
-5.06
v- DINO
B-BYOL
1.78
2.13
=DDINO
S-Simmiam
-1.96
1.89
u-DINO
e-MAE
0.17
0.57
e-DINO
v-SimCLR
-2.31
3.08
u-DINO
D-MoCo
0.36
-3.17
sero-shot on UCF
0-MAE
b-BYOL
0.28
1.56
u-MAE
D-SimSiam
-3.46
1.32
e-MAE
v-DINO
-2.45
-0.57
e-MAE
v-SimCLR
-3,81
2.51
e-MAE
e-MoCo
-1.14
-3.74
D-SimLLR
u-BYOL
2.40
-0.95
V-SiCLLR
v-SimSiam
-1.34
-1.19
SimCLR
v-DINO
-0.32
-3.08
V-SimCLR
a-MAE
0.79
-2.51
v-SimCLR
e-MoCo
0.99
-6.25
B-MoCo
b-BYOL
-1.40
5.30
D-MoCo
u-Simmiam
-5.14
5.06
v-MoCo
v-DINO
-4.12
3.17
M-MoCo
e-MAE
-3,01
3.74
M-MoCo
e-SimCLR
-5.49
6.25
51

---

Table $25 Part 2. Comparative statistical analysis of the robustness of all methods in zero-shot
recognition (original numbers from Table 4).
Adjusted ΔOOD
Distribution shift
Method
Method
Lin.
FT.
u-BYOL
v-SimSiam
-0.18
1.05
u-BYOL
v-DINO
2.17
-0,01
B-BYOL
v-MAE
0.55
-1.58
B-BYOL
V-SimCLR
1.64
0.06
BBBOL
e-MoCo
1.27
2.40
--SimSiam
B-BYOL
-2.61
-1.05
SimSiam
e-DINO
1.48
-1.06
o-Simmiam
v-MAE
-0.14
-2.63
v- SimSiam
v-SimCLR
0.94
-0.99
+-Simiim
V-MoCo
0.58
1.35
v D-DINO
v-BYOL
-4.55
0.01
v-DINO
v-SimSiam
-2.11
1.06
U-DINO
w-MAE
-188
-1.57
u-DINO
e-SimCLR
-0.80
0.07
v-DINO
e-MoCo
-1.16
2.41
Zero-shot on HMDB
t-MAE
e-BYOL
-3.15
1.58
v-MAE
v-SimSiam
-1.42
2.63
v-MAE
v-DINO
0.94
1.57
v-MAE
v-SimCLR
0.40
1.64
v-MAE
v-MoCo
0.04
3.98
u-SimCLR
v-BYOL
-5.14
-0,06
D-SimCLR
v-SimSiam
-3.40
0.99
u-SimCLR
v-DINO
-1.05
-0.07
D-SimCLR
e-MAE
-2.67
-1.64
0-SimCLR
e-MoCo
-1.95
2.34
M-MoCo
b-BYOL
-5.55
-2.00
e-MoCo
v-SimSiam
-3.81
-1.35
U-MoCo
v-DINO
-1.46
-2.41
V-MoCo
w-MAE
-3.08
-3.98
M-MoCo
0-SimCLR
-1.99
-2.34
52

---

Table $25 Part 3. Comparative statistical analysis of the robustness of all methods in zero-shot
recognition (original numbers from Table 4).
Adjusted AOoD
Distribution shift
Method
Method 2
Lin.
FT.
e-BYOL
o-SimSiam
-1.19
-0.95
v-BYOL
v-DINO
-1.36
-0.31
B-BYOL
v-MAE
-0.82
-0.43
B-BYOL
V-SimCLR
-0.59
-0.09
B-BYOL
e-MoCo
-1.38
-0.20
n --SimSiam
e-BYOL
0.59
0.95
SimSiam
e-DINO
-0.43
0.64
e- SimSiam
v-MAE
0.11
0.52
SimSiam
v-SimCLR
0.33
0.86
SimSiam
v-MoCo
-0.45
0.75
--DINO
v-BYOL
0.79
0.31
e- D-NO
V-SimSiam
-0.08
-0.64
D-DINO
v-MAE
0.30
-0.12
R-DINO
e-SimCIR
0.53
0.22
R-DINO
v-MoCo
-0.26
0.11
Zero-shot on RareAct
8-MAE
e-BYOL
-0.39
0.43
0-MAE
v-SimSiam
-1.25
-0,52
M-MAE
v-DINO
-1.41
0.12
0-MAE
v-SimCLR
-0.65
0.34
V-MAB
u-MoCo
-1.43
0.23
V v-SimCLR
v-BYOL
-0.26
0.09
SimCLL
o--iimiam
-1.12
-0.86
V- SimCLR
v-DINO
-1.28
-0.22
0-SimCLR
e-MAE
-0.75
-0.34
V-SimCLR
e-MoCo
-1.3
-0.11
U-MoCo
b-BYOL
0.50
0.20
U-MoCo
v-SimSiam
-0.36
-0.75
e-MoCo
v-DINO
-0.52
-0.11
U-MoCo
v-MAE
0.01
-0.23
M-MoCo
V-SimCLR
0.24
0.11
53

---

Table S26: Comparative statistical analysis of the robustness of all methods under animal domain
actor shift (original numbers from Table 2).
Adjusted AOoD
Distribution shift
Method 1
Method 2
Lin.
FT.
Superrvieed
v-SimCLR
-1.86
-3.00
--Supervised
e-MoCo
-1.50
-1,20
-upervvsseed
v-BYOL
-2.66
-1.20
u-Supervised
o-SimSiam
-3.24
-2.00
o-Supervised
v-DINO
-1.70
-1.90
v-Supervised
v-MAE
0.39
-2.40
SimCLR
o-Supervised
0.74
3.00
D-SimCLR
e-MoCo
-0.2
1,80
SimCLR
v-BYOL
-1.16
1.80
D-SimCLR
e-SimSiam
-1.94
1,00
--SmCLR
e-DINO
-0.40
1.10
D-SimCLR
e-MAE
1.69
0.60
e-MoCo
n-Supervised
0.09
1.20
v-MoCo
v-SimCLR
-1.21
-1.00
v-MoCo
v-BYOL
-182
0.00
e-MoCo
v-SimSiam
-2.59
-0.80
e-MoCo
v-DINO
-1.05
-0.70
M-MoCo
v-MAE
1.04
-1.20
v-BYOL
o-Supervised
1.58
1.20
e-BYOL
v-SimCLR
0.28
-1.80
B-BYOL
e-MoCo
0.64
0.00
Animal domain actor shift
BYOL
v-SimSiam
-1.10
-0.80
b-BYOL
e-DINO
0.45
-0.70
--BYOL
v-MAE
2.54
-1.20
o-SimSiam
e-Supervised
1.27
2.00
--SimSiam
v-SimCLR
-0.03
-1.00
v SimSiam
M-MoCo
0.33
0.80
SimSiam
--BYOL
-0.63
0.80
e SimSiam
D-DINO
0.13
0.10
SimSiam
e-MAE
2.22
-0.40
DDDIOO
e-Supervised
0.67
1.90
D-DINO
v-SimCLR
-0.63
-1.10
w D-DINO
M-MoCo
-0.27
0.70
D-DINO
e-BYOL
-1.23
0.70
DDDINO
v-SimSiam
-2.01
-0.10
D-DINO
e-MAE
1.63
-0.50
e
Superrviseed
-1.6
2.40
R-MAE
v-SimCLR
-2.00
-0.60
e-MAE
v-MoCo
-2.55
1.20
w-MAE
e-BYOL
-3.51
1,20
8-MAE
v-SimSiam
.4.28
0.40
e-MAE
v-DINO
-2.74
0.50
54

---

D.8 Exploring performance variation of VSSL methods across action classes
To provide a deeper understanding, we conduct comprehensive analysis examining the impact of distribution shifts on the models' prediction across various action classes. Our investigation focuses on evaluating the performance of the finetuned models across different categories in both the InD and OoD. The findings are presented as follows:
Figure S19 Context shift (10 classes)
Figure S20 Context shift (50 classes)
Figure S21 Viewpoint shift (egocentric)
Figure S22 Viewpoint shift (surveillance camera+low resolution)
Figure S23 Viewpoint + actor shift (top-down synthetic)
Figure S24 Actor shift (animal)
Figure S25 Source shift (UCF to HMDB)
Figure S26 Source shift (HMDB to UCF)
Figure S28 Zero-shot (UCF)
Figure S27 Zero-shot (HMDB)
Figure S29 Zero-shot (RareAct)
Figure S30 Open-set (UCF/HMDB)
Figure S32 Open-set (Kinetics400/UCFF)
Figure S31 Open-set (Kinetics400//MMBBB
55

---

8.59
Sup
Figure $19: Context shift (10 classes): In-distribution vs. out-of-distribution performance compari- son per action class. For optimal viewing, please zoom in. All of the contrastive and non-contrastive methods fail identify 'canoeing' in OoD, whereas, u-Supervised and v-MAE make few correct predictions. In another extreme example, v-MAE makes few correct predictions for 'surfing water', while other methods completely fail. In particular, VSSL methods demonstrate relatively good performance in identifying certain action classes in out-of-context scenarios, such as 'golf driving' and "playing volleyball'.
56

---

Figure S20: Context shift (50 classes): In-distribution vs. distribbtito performance com- parison per action class. For optimal viewing, please zoom in. Some of the top-performing action classes under context shift include 'brushing teeth', 'catching or throwing frishee', 'juggling balls'. 'juggling soccer ball', and 'skipping rope". It is noteworthy that all of these action classes involve high temporal motion and eyelic patterns. This observation leads us to speculate that the presence of these attributes may assist video models in recognizing actions in out-of-context settings.
57

---

Figure S21: Viewpoint shift (egocentric): In-distribution vs. -iit----iitriuution performance
comparison per action class. For optimal viewing, please zoom in and rotate 90 degrees to the left ' few examples of poor OoD generalization include action classes such as: 'lying on a bed', 'fixing vacuum', and 'someone is closing something", among a few others. We also note few instances, where models show very similar InD and OoD performance, such as 'holding some food', 'holding boom', and 'taking food from somewhere", among a few others.
58

---

WNNDO
3.53
Irdilades Ouoolchtbuuttr
10
2.40
125
Dayes
VOVOL
372
1.50
339
222
K10
Indrtada1
1.00
329
139
332
3.7
112
1.40
Figure S22: Viewpoint shift (surveillance view + low resolution): In-distribution vs. out-of- distribution performance comparison per action class. For optimal viewing, please zoom in. Interest- ingly, both e-BYOL and v-SimSiam perform well in identifying 'running' but failed to identify 'riding'. On the other hand, v-SimCLR and u-MoCo correctly identify 'riding' but fail to predict "running'. Overall, models show poor performance in distinguishing actions with subtle differences like "coosing' vs. 'opening'. "entering" vs. "exiting', and 'pull' vs. 'push'.
59

---

V-SIMCLR
V-M0C0
V-BYOL
0008
In-ditibtion
In In-disibttion
1000
mn-lstittionn
Out of disibudion
Out of disibution
Out disibuttonn
0.80
0.85
0.50
0.60
0.20
0.20
0.20
dronk
read trook
phoaree
wastle
cok
drink
redd Book
usa phore
cool
drint
read Brokk
use phore
V-SIMSIAM
V-DINO
V-MAE
0008 In-distibution
0008
Indiasibutton
0008
In-disd button
Dul-o--iiiibulon
Cui-of-dis buiion
Out-af-dist bution
0.80
0.80
0.60
0.69
0.20
0.20
0.20
read brook
phome
cook
drokk
raad bock
une plane
orikk
drink
read book
ure phone
SED
In-disiibttion Dul-df-istibuion
0.80
0.00
0.40
0.20
read phane
Figure $23: Viewpoint actor shift (top-down synthetic): In-distribution out-of-distribution performance comparison per action class. For optimal viewing, please zoom in. The models exhibit particularly poor performance identifying the action classes 'eat", followed by 'drink'. Conversely, they demonstrate relatively better performance in identifying actions such as 'read book', 'use phone", and 'walk".
60

---

It-dsibutee
3.50
Indicibutin
1,50
In-di
Out dil-elide albution
out-l-istauaar
5.80
3.50
0.40
5.50
2.50
0.50
2.40
0.40
0.50
of Compoe
MSIMSIAM
ODNO
BMWAA
0.0
3.50
0.59
5.40
3.40
0.40
3.20
0.0
100
+ Compon
VSSUEEVIISE
1.00
1: 1-ds:b.itnn
idibution
125
6.20
5.00
Figure S24: Actor shift (animal): In-distribution vs. -ut-of-distributiin performance comparison per action class. For optimal viewing, please zoom The models exhibit particularly poor performance in identifying action classes 'drinking' and 'rumning'. Conversely, they perform well in 'predicting eating", 'opening door', 'swimming'. and watchhng tv'.
61

---

03 2aradrd
Figure $25: Source shift (UCF to HMDB): In-distribution out-of-distribution performance comparison per action class. For optimal viewing, please zoom in. The results demonstrate the challenges faced by the video models in generalizing across similar action classes from different datasets. For instance, the models trained on UCF videos which include 'soccer penalty',. struggle to generalize the 'kick ball' class in HMDB. Similarly, the 'walking with dog' class in UCF does not aid generalizing to 'walk' in HMDB
62

---

4.20
8.8
1BVO
3.50
0.05
6.40
070
Figure S26: Source shift (HMDB to UCF): In-distribution vs. duitof-distri performance comparison per action class. For optimal viewing, please zoom in. We notice large variability in performances across different methods. For example, o-Supervised performs very poorly in identifying the action 'dive', whereas. all the self-supervised methods perform reasonably well. Furthermore, while v-BYOL and r-DINO demonstrate strong performance in identifying the action 'walk', the other methods struggle to achieve similar level of accuracy. Overall, the VSSL methods tend to exhibit poor generalization capabilities for actions such as 'jump', 'throw", and 'dribble', among others.
63

---

SIMCRR
V. V.MOCO
0.70
0.50
0.60
0.40
616
Caty
cloc Nain
de TARNAAE
VSIISIAM
0.80
0.70
0.60
0.60
0.50
6:45
3.40
0.30
0.20
6.20
0.0
0.00
6i3 fox tuning
vourt libeed
thove
6.2
Spard cooee Cal the tre siore
co0l
0.60
0.70
0.00
0.30
020
628
take Cayy meily
cluch W0th
V.SUPENIIED
Caale
rour Hiidey tag fary
the
Figure $27: Zero-shot recognition (HMDB): In-distribution vs. cnttof-distribution performance comparison per action class. For optimal viewing, please zoom in. The video models overall generalize well on action classes such as 'chew', 'handstand', 'kick', and 'smile', whereas, they show poor generalizability other action classes such 'fencing', 'flie flac", 'pick', 'run', and 'turn' among others. Moreover, their performance largely varies across different actions.
64

---

XM000
114
<B80
con
Figure $28: Zero-shot recognition (UCF): In-distribution vs. out-of-distribution performance com- parison per action class. For optimal viewing, please zoom in. The video models overall generalize well on action classes such as "pplyy lipstick", 'rafting', 'ice dancing', and 'sumo wrestling' whereas, they show poor generalizability on action classes such as 'fencing', 'front crawl', "hammering,, "mixing', and 'rowing' among others. Morcover, their performance largely varies across different actions.
65

---

11771 11
Figure S29: Zero-shot recognition (RareAct): In-distribution vs. cut-of-distribution performance comparison per action class. For optimal viewing, please zoom in and rotate 90 degrees the left '.. The models tend to generalize poorly on unusual actions. Moreover, compared to other zero-shot setups, notice the maximum variability in model performance across different action classes.
66

---

u-SimCLR
V-MoCo
TOYBL
22
V-SimSiam
ONICIa
O-MAE
Figure S30: Confusion matrices for open-set recognition, using UCF101 closed-set and HMDB as open-set. The x-axis represents the ground truth and the y-axis represents the predicted labels. We highlight known classes with green and unknown classes with red. Among all the methods o-SimSiam and v-MAE show worse performance in identifying 'unknown' classes from HMDB as "unknown'.
67

---

u-SimCLR
V-MoCo
e-BYOL
v-SimSiam
CNICFT
e-MAE
Figure S31: Confusion matrices of open-set recognition, using Kinetics400 closed-set and HMDB as open-set. The x-axis represents the ground truth and the y-axis represents the predicted labels. We highlight known classes with green and unknown classes with red. The results exhibit that all the video models struggle in identifying "unknown' classes from HMDB as 'unknown' This is likely due to their over-confident predictions.
68

---

u-SimCLR
v-MoCo
B-BYOL
u-SimSiam
BuCC-a ONIG
e-MAE
Figure S32: Confusion matrices of open-set recognition, using Kinetics400 closed-set and UCF as open-set. The x-axis represents the ground truth and the y-axis represents the predicted labels. We highlight known classes with green and unknown classes with red. The results exhibit that all the video models struggle in identifying 'unknown' classes from UCF as 'unknown". This is likely due their over-confident predictions.
69


Reviews:
[{"review_id": "0Cosgng6GW", "reviewer": "Reviewer_iDRZ", "summary": "This works studies various video SSL methods under distribution shifts. 6 different video SSL methods (SiMCLR, MoCO, BYOL, SimSiam, DINO, MAE) are trained on Kinetics-400 under the same experiment settings (e.g. ViT-B, fixed number of epochs, etc) and then evaluated on various distribution shifts (e.g. viewpoint shift, context shift) and report the findings (e.g. contrastive methods more robust to viewpoint shifts, while MAE is a stronger temporal learner).", "questions": "Could the authors please discuss some of the limitations and caveats to their findings (e.g. some are listed above)?", "limitations": "Yes.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Writing and presentation is clear.\n2. Same experimental and test bed setup for all the baselines. The test bed and evaluation code could be useful to the community.\n3. Useful empirical observations characterizing the strengths and weakness of prominent approaches.\n", "weaknesses": "No experiments or analyses on model scaling or different architectures. \n\nSome findings are pretty much expected (e.g. contrastive methods not being good temporal learners happens by design), but it's nice have an easy test bed to quantify them.\n\nSome of the settings controlled for can be strong caveats, e.g. fixing the number of epochs can bias evaluations towards methods that converge faster. It is unclear how much of the results hold when models used their respective optimal settings - which is an important setting for evaluations.\n\nSome of the differences could stem not from core differences in pretext task, but in use of a teacher. e.g. MAE can also use an EMA teacher and the representations would converge much faster. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for providing such valuable feedback. We are happy to note that the reviewer finds our work useful to the community.\n\n\n> Experiments on different architectures or model scaling\n\nWe have now conducted experiments on video ResNet-50 given its strong performance in video SSL [72]. We conduct these experiments in a variety of OoD setups including context shift 10 and 50 classes, source shift in both *UCF to HMDB* and *HMDB to UCF*, and animal domain actor shift. The results are presented in **Table R1 (attached pdf)**. To conduct these experiments, we perform linear evaluations using pretrained weights released by [72] and compare them to the ViT-B. These results confirm that our general findings based on ViT are also applicable when tested using ResNet-based architecture. We anticipate similar trends to hold when scaling the backbone.\n\n\n> Some findings are pretty much expected, but it's nice to have an easy test bed to quantify them\n\nWe agree with the reviewer that some of these findings might be intuitive and as the reviewer pointed out there is no work that studies the robustness of video SSL models under real-world distribution shifts, in a unified setup. We would also like to point out that some of the findings are in fact quite counterintuitive. For example, we were surprised to find that:\n\n\n- *frozen contrastive encoders* perform poorly in open-set recognition, while achieving the best open-set performance when finetuned; \n- superior performance of v-BYOL frozen encoder compared to finetuned under source shift (HMDB/UCF);\n- consistently poor performance of v-MAE in linear evaluations.\n\n\nHowever, our additional investigation allowed us to further explore these findings and pinpoint the root causes of such intriguing behaviours.\n\n\n\n\n> VSSL pretraining hyperparameters and convergence\n\n\nWe would like to point out that we have swept a broad range of parameters for each method to ensure that the hyperparameters are thoroughly tuned for each method.\n\n\nBelow we provide a description for some of the key hyperparameter setups:\n\n\n- **Augmentation**: The optimal setup for augmentation varies amongst contrastive, non-contrastive, and masked autoencoder. Based on empirical findings, cropping ratios of [0.08, 1], [0.2, 0.766], and [0.5, 1] are used for contrastive, non-contrastive, and v-MAE respectively. Additionally, color-jittering, blur, or horizontal flip are not applied for v-MAE, but they are applied to other VSSL methods.\n- **Epoch**: Following [1, 6], We pretrain these methods *up to* 800 epochs and track their performance using InD validation sets (UCF101, Kinetics400) and notice that all the methods reach their optimal somewhere between 600 to 800 epochs. We use the best checkpoints of each method in the downstream tasks.\n- **Learning Rate (LR)**: We individually tune LR for all the methods through a grid search between 5e-5 to 1e-3. We find the optimal LR for v-MoCo, v-BYOL, and v-DINO as 3e-4, v-SimCLR as 2e-4, and v-SimSiam as 1e-4. Additionally, the predictor heads are trained with 10 times higher LR with respect to the base LR, to achieve optimal performance.\n- **Projector head**: The optimal configuration for the projector head also varies amongst VSSL methods, e.g., an MLP head of 4 layers is optimal for v-SimCLR and v-SimSiam, while an MLP head of 3 layers works best for v-MoCo, v-BYOL, and v-DINO.\n- **Predictor head**: the configuration of the predictor head is also adjusted, i.e., v-MoCo and v-SimSiam work best with a predictor head of just 1 layer while a 2-layer predictor head works best for v-BYOL.\n- **Others**: we also tune other hyperparameters like **weight-decay**, and **EMA**, among others to find the optimal configuration for each method.\n\n\nWe present the hyperparameters related to each method in **Tables S1 and S3** in **Appendix B**. \n\n\n\n\n\n\n> Discuss limitations/caveats\n\n\nAs discussed in Sec.7 of the paper, it would be of interest to further investigate VSSL methods under distribution shifts with larger Transformer or convolutional architectures, which we could not perform due to resource constraints. However, we believe our findings serve as a foundation for future works to study the behaviour of OoD robustness with model scalability.\n\n\nReferences:\n\n- [1] VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Training, Z. Tong et al., NeurIPS 2022\n- [6] XKD: Cross-modal Knowledge Distillation with Domain Alignment for Video Representation Learning, P. Sarkar et al., 2022\n- [72] A Large-Scale Study on Unsupervised Spatiotemporal Representation Learning, C Feichtenhofer et al, CVPR 2021\n\n\n"}, {"review_id": "rxtIAg2h3x", "reviewer": "Reviewer_D3xd", "summary": "The paper proposes a set of benchmarks to assess different robustness properties of video representation learning models, including contrastive, non-contrastive, and generative models. The paper trains multiple of these models using a common training protocol and reports multiple empirical findings on different forms of distribution shift.", "questions": "- If I read the methods correctly, you trained all models from scratch on the respective datasets (Kinetics400 and -700). Did the video models  you trained readily outperform the best \"static image\" models available for the respective methods?\n- Will you open source the model checkpoints and source code for all models you trained in this study? The paper only makes a statement regarding the code.\n- The results in Table 2, Viewpoint (ego) seem to be very bad (around 11%). Did you investigate possible causes for this performance drop? How do you justify to infer conclusions from this part of the table (e.g. in l. 246 etc) given the bad InD performance to begin with? ", "limitations": "Yes, limitations have been addressed (up to the additional comments I made above).\n\nHowever, given the amount of datasets used in the paper, I am missing license and copyright statements for these datasets, e.g. in the appendix, that go beyond the citations provided in section 5 in the main paper. This could e.g. be included into Appendix C. If the benchmark is intended for later release as outlined, I think it would be very useful to have such an overview directly in the paper for future reference by users of the benchmark.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "The paper outlines a very clear training and evaluation protocol and nicely outlines the results. I like the \"highlights\" section at the end of each addressed question section (up to comments below). The considered datasets and models are extensive. Benchmark and models could become a very valuable resource for further robustness benchmarking on video datasets.", "weaknesses": "- One major weakness of the paper seems to be the missing control for the baseline performance of the proposed methods. For instance, in Table 2, several claims are made with respect to the performance of contrastive vs. other models under viewpoint shifts (l. 246 etc). However, in Table 2 it is unclear whether these improvements are originating from the improved OoD, or in fact the difference in InD performance. Given that the authors computed error bars for all results, I propose to equip every statement in the paper with a suitable hypothesis test capable to test the influence of the different factors.\n- Another major weakness is the missing link to published results. All models in the paper are trained from scratch by the authors, and no numbers are reported by applying existing model checkpoints from the literature and confirming that the trained models reach a comparable performance level. What is the rationale for not verifying model performance e.g. against the released best models trained on image datasets?\n\n\nOther weaknesses:\n\n- In the training protocol, parameters like the batch size were kept constant across methods. However, it is known that methods like SimCLR depend on availability of large batch sizes, while models like MoCo have mechanisms build in to more effectively leverage data in small batchsize training setups. Hence, I disagree with the author's statement that the considered setting is \"fair\". Happy to discuss the rationale for these choices (vs. for example using the best available model configurations). Minimally, it would be good to discuss this point more in the paper, e.g. in the limitation section.\n- For a purely empirical paper, I would recommend to back up the fairly general statements at the end of each section with statistical tests.\n- The plots are unreadable without heavily zooming in, e.g. Figure 4 and Figure 6. Figure 5 is borderline.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for providing such valuable feedback and a thorough review. We are glad to note that the reviewer finds the 'Highlights' at the end of each section useful and finds our work extensive and valuable.\n\n> Do the improvements originate from the improved OoD or from the difference in InD performance, e.g., Table 2?\n\nWe thank the reviewer for raising the question. In **Table R4 (attached pdf)**,  we perform statistical tests (Pearson corr.) investigating if the improvements in OoD are due to the *OoD robustness* or from improved InD performance. \n\n- Overall, InD vs. OoD performance in linear eval. show a higher corr. compared to when finetuned, i.e., 0.71 vs. 0.55.\n- In the case of viewpoint shift (in Table 2), we notice a strong corr. in *linear eval. of egocentric view* and *finetuned eval. of surveillance camera view*. In all the other viewpoint shift setups, the corr. is fairly low.\n\nWe further study the relative OoD robustness (i.e., measuring the performance drop w.r.t InD) to compensate the effect of varying InD performance using the corr. coefficient between OoD and InD performance, and find that our overall conclusions regarding models' robustness are still valid. We note that this is not a perfect solution to compensate the effect but nevertheless, the best we could think of. We'd be happy to try any other suggestions the reviewer might have in mind.\n\n> Confirming that the VSSL implementations reach a comparable performance level to prior works\n\nIn **Table R5 (attached pdf)**, we compare the performance of v-SimCLR with prior works which are also based on (or inspired by) SimCLR, confirming that our implementation achieves comparable performance w.r.t the prior works. Please note that while these methods use a similar objective function to SimCLR, their implementations largely vary amongst each other. Considering such variations amongst the prior works, we choose to implement and train all the VSSL methods by ourselves and pretrain them in identical experiment setups with necessary hyperparameter tuning for a fair comparison.\n\n> What is the rationale for not using models trained on image datasets? Did the video models you trained readily outperform the best \"static image\" models available for the respective methods?\n\nIn **Table R2 (attached pdf)**, we compare image vs. video pretraining based on 3 SSL methods (MoCo-v3, DINO, MAE) on a variety of OoD setups. We particularly choose these 3 methods as they are originally proposed with ViT similar to our setup. The results presented in Table R3 exhibit up to 9.8% improvements when using video models compared to their image variants.\n\n> Batch size is kept constant; SimCLR works better w/ large batch but MoCo works better w/ small batch.\n\n- We follow a similar *video* pretraining setup of [72] and use the same batch size for all the variants.\n- We would also like to clarify that our v-MoCo is based on MoCo-v3 [23] (not *MoCo by K. He et al., CVPR'20*), and as discussed in [23] MoCo-v3 performs best with a similar batch size (2048) to SimCLR when pretrained with unlabelled *images* from ImageNet. \n- From [3] we observe that the performance of *video contrastive methods* shows a very stable performance when using a batch size between 512 to 1024, and below or above that range performance degrades. We believe the batch size of 768 is not a detrimental factor here as it is within a standard range and the LR is adjusted accordingly. \n\n> Hypothesis and statistical test to back up the summary statements at the end of each section\n\nThe Highlights mentioned in section 6 are a summary of the key findings based on our empirical study. To ensure our observations are statistically significant, we run these experiments 3 times with different seeds and report the average and standard deviation. Additionally, to strengthen our arguments observed from the evaluation on the real-world datasets, we also conduct a series of toy experiments in a controlled setup to further verify some of these hypotheses. \n\n> The plots are unreadable without zooming.\n\nWe thank the reviewer for pointing it out. We will enlarge the size of the figures in the final version as it allows for an additional page.\n\n> Will you open source the model checkpoints and source code?\n\nYes, we will open-source the model checkpoints and source codes upon publication.\n\n> Possible cause for poor results in egocentric viewpoint shift experiments and additional justifications\n\nThis is likely considering the challenging nature of the Charades-Ego dataset, which is comprised of videos from 157 fine-grained household activities, moreover, each video contains multiple labels. However, we would like to highlight that our results are in a similar range to prior works, e.g., [36] reports InD mAP of 23.3 vs. ours 21.4, and OoD mAP of 19.5 vs. ours 16.1. The 2-3% drop is likely since [36] uses  RGB *+ optical flow*, whereas our method only uses RGB frames. As for the linear eval. performance being around 11%, we would like to point out that to our knowledge, no prior works have performed linear eval. on this dataset.\n\n> How to make conclusions about OoD when InD performance is not strong, e.g., Table 2 Viewpoint (ego.)?\n\nWe acknowledge that in such a case, a strong claim about OoD could not be made. However, our conclusion about viewpoint shift is not only based on the performance on egocentric viewpoint shift but rather based on a trend noticed across all 3 viewpoint shifts in both linear and finetuning. Moreover, our claims are also validated through toy experiments for additional confirmation. \n\n> Dataset license\n\nWe thank the reviewer for this suggestion. Following, we provide the license statements for each dataset and will also add this in Appendix C.\n\n- CharadesEgo, MiT-v2: License for Non-Commercial Use\n- Kinetics, HMDB51, ToyBox: CC BY 4.0\n- Mimetics, UCF101, TinyVirat-v2, COIL100, STL-10: Open access\n- ActorShift, Sims4Action: MIT\n- RareAct: Apache\n"}, {"review_id": "zGs0DappjG", "reviewer": "Reviewer_jxy6", "summary": "The paper studies the generalizability of many self-supervised training approaches in the video domain. The paper uses 17 tasks to completely test different aspects of the models, including view-point change, temporal modeling, and open set generalizability.", "questions": "1. How is the v-supervised trained? Does it use kinetics' labeled data for pre-training?\n2. In lines 194 - 198 and Table 1, I didn't see the OoD performance of v-supervised and v-MAE as better than the others if they learn better time-variant information. Most of the time Table 1 shows mixed results in linear probing and fine-tuning (e.g. v-MAE performs better in OoD with FT but worse in Lin.).\n3. Maybe this paper can consider citing [1, 2] in lines 219 - 222.\n4. In Table 5, why the trend of fine-tuning and linear probing is very different?\n\n[1] Lei, Jie, Tamara L. Berg and Mohit Bansal. “Revealing Single Frame Bias for Video-and-Language Learning.”\n[2] Buch, S., Cristobal Eyzaguirre, Adrien Gaidon, Jiajun Wu, Li Fei-Fei and Juan Carlos Niebles. “Revisiting the “Video” in Video-Language Understanding.”\n\n---\n**Post-rebuttal**\nThank you for the authors' response. I have read it and it addressed my questions.", "limitations": "The limitation is adequate.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. Most of the cutting-edge video pre-training approaches are covered.\n2. Generalizability is an important problem for current large foundation models.", "weaknesses": "1. It's good to have a hyper-parameter table in the appendix. However, I am not sure if the authors have swept hyper-parameters for each method, and make sure each pre-training is complete. I understand the training for each approach is expensive, but the value will be reduced if the pre-training recipe hasn't been fully explored. \n2. Given the paper studies the pros and cons of each approach, I would expect there will be a proposed approach that performs better in most cases or a guideline that can help readers to develop such a method.\n3. Now the conclusion of each question scatters in different sections. This is fine but I think having a summary table to compare all approaches and show the findings at the beginning or at the end could help readers to understand the pros and cons of each approach better.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for sharing such thoughtful reviews. We are happy to find the overall positive feedback provided by the reviewer.\n\n\n> It's good to have a hyperparameter table in the appendix; if the authors have swept hyperparameters for each method?\n\n\nWe would like to point out that we have indeed swept a broad range of parameters for each method to ensure that the hyperparameters are thoroughly tuned for each method.\n\n\nBelow we provide a description for some of the key hyperparameter setups:\n\n\n- **Augmentation**: The optimal setup for augmentation varies amongst contrastive, non-contrastive, and masked autoencoder. Based on empirical findings, cropping ratios of [0.08, 1], [0.2, 0.766], and [0.5, 1] are used for contrastive, non-contrastive, and v-MAE respectively. Additionally, color-jittering, blur, or horizontal flip are not applied for v-MAE, but they are applied to other VSSL methods.\n- **Epoch**: Following [1, 6], we pretrain these methods *up to* 800 epochs and track their performance using InD validation sets (UCF101, Kinetics400) and notice that all the methods reach their optimal somewhere between 600 to 800 epochs. We use the best checkpoints of each method in the downstream tasks.\n- **Learning Rate (LR)**: We individually tune LR for all the methods through a grid search between 5e-5 to 1e-3. We find the optimal LR for v-MoCo, v-BYOL, and v-DINO as 3e-4, v-SimCLR as 2e-4, and v-SimSiam as 1e-4. Additionally, the predictor heads are trained with 10 times higher LR with respect to the base LR, to achieve optimal performance.\n- **Projector head**: The optimal configuration for the projector head also varies amongst the VSSL methods, e.g., an MLP head of 4 layers is optimal for v-SimCLR and v-SimSiam, while an MLP head of 3 layers works best for v-MoCo, v-BYOL, and v-DINO.\n- **Predictor head**: the configuration of the predictor head is also adjusted, i.e., v-MoCo and v-SimSiam work best with a predictor head of just 1 layer while a 2-layer predictor head works best for v-BYOL.\n- **Others**: we also tune other hyperparameters like **weight-decay**, and **EMA** coefficient, among others to find the optimal configuration for each method.\n\n\nWe present the hyperparameters related to each method in **Tables S1 and S3** in **Appendix B**.\n\n\n> Guidelines for future work\n\n\nWe thank the reviewer for this question.\nFollowing, we share some of our thoughts that can be used for future work. We will also include this in the final version of the paper. \n\n\nA general guideline could be to train video SSL frameworks to learn *local time-invariant representations* and *global time-variant representations*. \n\n\nOur intuition is that such models would be\n- robust against viewpoint shifts as it learns *view-invariance* through local time-invariant representations, similar to v-MoCo and v-SimCLR\n- robust against mere temporal perturbations as they learn locally time-invariant representations, similar to v-BYOL, v-DINO\n- robust to context shifts as it understands the global temporal dynamics well, similar to v-MAE.\n\n\nThe objective function can be designed as a combination of masked reconstruction and contrastive/non-contrastive methods. We will further investigate such approaches in future.\n\n\n> Summary table to compare all approaches\n\n\nWe thank the reviewer for sharing this great idea. Please see **Figure R1 (attached pdf)**, we will also add it in the final version.\n\n\n> How is the v-Supervised trained? Does it use kinetics' labeled data for pre-training?\n\n\nYes, the v-Supervised model is pretrained using the labels of the Kinetics dataset. We follow a similar recipe to [76] ViViT: A Video Vision Transformer.\n\n\n> In Table 1, v-MAE and v-Supervised show mixed results; v-MAE performs better in OoD with FT. but worse in Lin.\n\n\n- Yes, the reviewer’s observation is correct. However, our key takeaway from Table 1 is that *v-MAE consistently outperforms in both OoD setups when finetuned and v-Supervised shows the best performance in 3 out of 4 OoD setups*. It should be noted that the poor performance of v-MAE in Lin. is not specific to context shift, v-MAE shows poor Lin. performance in almost all the setups. Therefore, we conjecture v-MAE and v-Supervised are strong temporal learners based on their overall superior performance. \n- However, as the trend in real-world evaluation is noisy as correctly noted by the reviewer, we carry out tests in a controlled setup on a toy dataset. In particular, we aim to disentangle the spatial and temporal representations to accurately evaluate who learns the temporal dynamics better irrespective of their spatial representation learning capability. As presented in Figure 2a, both v-MAE and v-Supervised show superiority over other methods by a very large margin, confirming that they learn better temporal dynamics.\n\n\n> Suggested reference\n\n\nWe thank the reviewer for suggesting these refs, we will add them in the final version. They are indeed relevant to our study, \n- the work by Lei et. al. is related to context shift as it studies static appearance bias; \n- Buch et. al. introduces atemporal probe model which is relevant to our work, as we also discuss the ability of the video models in understanding temporal dynamics.\n\n\n\n\n\n\n> In Table 5, why is the trend of fine-tuning and linear probing very different?\n\n\nThe superiority of v-SimCLR and v-MoCo in open-set recognition when finetuned may be attributed to their better generalizability, which, while advantageous in linear probing, becomes a pitfall in open-set scenarios. These highly generalizable models, driven by their overconfidence, often misinterpret unknowns as known classes, leading to incorrect predictions. Conversely, *weak frozen encoders* avoid such misclassifications due to their limited generalizability and perform better in open-set conditions, as observed in models like v-DINO and v-Supervised. Interestingly, such a trade-off is only noticed in linear evaluation and not in finetuning.\n"}, {"review_id": "DeWewKlpgc", "reviewer": "Reviewer_E4vF", "summary": "The authors studied the behavior of six popular self-supervised methods in response to various forms of natural distribution shift. And the study uncovers a series of interesting findings and behaviors of video self-supervised learning (VSSL) methods. The experiments and results are beneficial for the video representation learning community.", "questions": "1. Any suggestions for the future research to design better DSSL method?\n\n2. Providing a table of summarisation of the 6 methods might be helpful for the readers. There are many experiments and data.\n\n3. The methods are pre-trained on K400 and K700 datasets, which might not be large enough. If the models are pre-trained on a much larger dataset, will the conclusions in this paper be changed?\n\n4. Does the model sizes effect the conclusions in this paper ?\n", "limitations": "Limitations have been discussed in the text. And no obvious negative societal impacts.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Extensive experiments are conducted. And sufficient experimental data and analysis are provided. It seems to the reviewer that all the conclusions are grounded.\n\n\n\n", "weaknesses": "1. It seems that the effects of data size and model size are not discussed.\n\n2. Many experiments are conducted and some conclusions are obtained. But no improved VSSL are provided, which might be more interesting.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for taking the time and providing such valuable feedback. We are happy to note that the reviewer finds our arguments well-grounded with the experimental results.\n\n\n\n\n> Effect of data and model size\n\n\n- In **Table S6 (Appendix D)**, we notice that using a larger dataset of more diverse videos generally helps in better performance in both InD and OoD, e.g., overall linear evaluation performance is improved by 1.5% and 0.9% in InD and OoD. Amongst the VSSL methods, v-BYOL benefits the most from the availability of diverse pretraining data, e.g., the overall performance of v-BYOL is improved by 3.1% and 1.9% in InD and OoD respectively. Please kindly refer to Appendix D1 where we discuss the effect of using larger data.\n\n\n- We acknowledge that the impact of model sizes would be another area for exploration, which we have mentioned in Section 7 under “Limitations”. While we could not perform experiments on a larger network due to resource constraints, we have now explored a different architecture, video ResNet50 (please see **Table R1 in the attached pdf**), which shows consistent trends with respect to the ViT backbones used in our paper. Given this consistency when changing the backbone, we anticipate similar observations should the model sizes also be reasonably changed. But we agree that further experiments are required to substantiate this. \n\n\n\n\n\n\n> No improved VSSL are provided; Suggestions for future research\n\n\nWe thank the reviewer for this question. As our overall goal in this work is to study the robustness of *existing* VSSL models under distribution shifts, introducing a new method is beyond the scope of this paper.\n\nWe do believe, however, that our work can be used to drive the design of future VSSL methods with improved performance. Following, we share some of our thoughts that can be used as a suggestion for future work, we will also include this in the final version. \n\nA general guideline could be to train video SSL frameworks to learn *local time-invariant representations* and *global time-variant representations*. \n\nOur intuition is that such models would be\n- robust against viewpoint shifts as it learns *view-invariance* through local time-invariant representations, similar to v-MoCo and v-SimCLR\n- robust against mere temporal perturbations as they learn locally time-invariant representations, similar to v-BYOL and v-DINO\n- robust to context shifts as it understands the global temporal dynamics well, similar to v-MAE.\n\n\nThe objective function can be designed as a combination of masked reconstruction and contrastive/non-contrastive methods. We will further investigate such approaches in future.\n\n\n> Summary table\n\n\nWe thank the reviewer for sharing this great idea. Please see **Figure R1 (attached pdf)**, we will also add it in the final version.\n\n\n> K400 and K700 datasets might not be large enough; If the models are pre-trained on a much larger dataset, will the conclusions in this paper be changed?\n\n\n- Kinetics700 (K700) consists of 0.5 million videos and is one of the widely used large-scale **open-source** pretraining benchmarks for video self-supervised learning [1,3,72].\n- Amongst the other open-source video datasets, a potential alternative is AudioSet 2M, which is popularly used in audiovisual self-supervised learning. We internally experiment with AudioSet as well and the results are added to **Table R3 (attached pdf)**. We find that AudioSet also shows a similar trend to our findings from K700, confirming that our findings are likely to be aligned even when pretrained on other large datasets.\n- Despite AudioSet being larger than Kinetics700, we do not notice a clear benefit as shown in **Table R3 (attached pdf)**. This is likely since the action classes present in Kinetics are more aligned with the downstream actions compared to AudioSet. \n- We note a few prior works that used even larger datasets like IG65M [98] which could have been beneficial in our study; however, they are not open-sourced, hence we could not use them.\n\n\n\n\nReferences:\n\n\n- [1] VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Training, Z Tong et al., NeurIPS 2022\n- [3] Spatiotemporal Contrastive Video Representation Learning, R Qian et al., CVPR 2021\n- [72] A Large-Scale Study on Unsupervised Spatiotemporal Representation Learning, C Feichtenhofer et al., CVPR 2021\n- [98] Large-scale weakly-supervised pre-training for video action recognition, D Ghadiyaram et al., CVPR 2019\n\n\n"}]

Author rebuttal:
We sincerely thank the review committee for their time and for providing constructive feedback. We are happy to see the overall engaging comments given by all the reviewers and glad to note that all reviewers find our work valuable to the community. We have carefully addressed all the concerns raised by the reviewers under the individual response section. Following, we provide a summary of our response.




- **VSSL pretraining hyperparameter search**: In response to individual reviewers, we clarified that we have strived to tune all SSL methods studied in this work to achieve their best performance,  involving best practices and/or grid search for augmentation setups, learning rates, batch size, weight decay, EMA, mask ratio, temperature, and the configuration of the projector heads, predictor heads, and decoder, among others. Please see the detailed response under **Reviewers iDRZ or jxy6** and a summary of pretraining setups is provided in Tables S1 and S3 in Appendix B. Additionally, all the models and codes will be released upon publication for reproducibility.


- **Experiment on existing model checkpoints with a different architecture**: To analyze the generalizability of our findings beyond ViTs, we have conducted experiments to study the OoD robustness of the video SSL methods using existing checkpoints of Video ResNet-50 (Slow only). The results presented in **Table R1 (attached pdf)** show that our findings based on ViT are aligned when evaluated on a video ResNet.


- **Verifying image vs. video pretraining**: In order to ascertain the benefits of video pretraining over models pretrained on static images, we have conducted additional experiments comparing the performance of image SSL vs. video SSL pretraining. Our comparison setups include 3 SSL methods MoCo-v3, DINO, and MAE across a variety of distribution shifts. The results presented in **Table R2 (attached pdf)** exhibit significant improvements when using video models compared to their image variants.


- **Relation between InD vs. OoD performance**: We have now conducted additional statistical analysis investigating the relation between the models' performance in InD vs. OoD. The results presented in **Table R4 (attached pdf)** show a higher corr. when directly using the pretrained frozen encoders (linear probing) compared to finetuned, i.e., average corr. of 0.71 vs. 0.55. This indicates that finetuning is less beneficial under distribution shifts compared to InD. Please see the detailed response under **Reviewer D3xd**.


- **Guidelines for future work**: To further help future research in developing robust and reliable video learning models, we shared some of our thoughts on future work based on the findings from this paper. In short, we find that a robust video representation framework should learn *local time-invariant* and *global time-variant* representations. Our intuition is that such representations are robust to context shift, viewpoint shift, and temporal perturbations among others. Moreover, such models can be designed by exploring a joint objective of masked reconstruction and contrastive/non-contrastive methods. Please see individual responses under **Reviewers jxy6 or E4vF** for more details.


- **Summary table**: In **Figure R1 (attached pdf)** we provide a high-level overview of different VSSL models depicting their robustness and vulnerability under different evaluation setups in both InD and OoD.


- **Pretraining with other large datasets**: We have now added results of VSSL methods pretrained on another large-scale dataset AudioSet consisting of 2M video. The results presented in **Table R3 (attached pdf)** exhibit a very similar trend to the pretraining of Kinetics700. Such strong relation confirms that our findings are likely to be aligned even when pretrained on other large datasets. Please see the detailed response under **Reviewer E4vF**.


We hope that our responses adequately address all the points raised by the reviewers. We would be more than happy to address any additional comments the reviewers may have during the discussion phase. 






Meta-review: The paper presents a thorough study of video self-supervised learning that uncovers a series of interesting findings that uncover the hidden dynamics of VSSL methods across a variety of distribution shifts.  There was an effective rebuttal and Four knowledgeable reviewers recommended accepting the paper: Accept, borderline Accept, Weak Accept, Weak Accept. Based on the clear positive consensus, it is our recommendation to accept the paper. No basis to overturn the reviews.   Authors should attend to the main points in the reviews and integrate the rebuttal material when preparing a final version.
Decision: Accept (spotlight)