SlowGuess/NeurIPS_2023_OCR huggingface dataset:
(a) Viewpoint invariance
(b) Low-resolution
Figure 4: Comparing models on viewpoint invariance (a) and low-resolution robustness (b).
lowed by  $v$ -SimCLR. Moreover, the results show that  $v$ -MAE is susceptible to viewpoint shifts as it performs worse. We believe that as  $v$ -MAE and  $v$ -Supervised are trained with a single view of frame sequences, they are more sensitive to viewpoint shifts. Additionally, amongst the non-contrastive methods,  $v$ -DINO shows a better performance. Notice that our results so far have revealed a trade-off between learning viewpoint invariance vs. strong temporal dynamics. While single-stream networks learn better temporal dynamics, Siamese frameworks learn better viewpoint invariance.
Notice that while contrastive methods dominate performance under viewpoint shifts (Table 2),  $v$ -MAE shows the best performance only in the case of low-resolution surveillance cameras. We hypothesize that this is due to the fact that in addition to  $v$ -MAE being a strong temporal learner (see Figure 3), it is also inherently robust to low-resolution inputs. This is likely since it learns strong pixel-level relations [86] through reconstruction of highly occluded inputs. To further investigate this, we perform a follow-up study in a controlled toy setup where we systematically decrease the resolution of input frames from  $112^2$  to  $16^2$  and measure the robustness as  $1 - (\mathrm{acc}_{112} - \mathrm{acc}_{\mathrm{N}}) / 100$ , following [53]. Here  $\mathrm{acc}_{\mathrm{N}}$  refers to accuracy at resolution  $\mathrm{N}^2$ . We use  $112^2$  as the maximum resolution as this is used during pretraining, and  $16^2$  is set as the lowest resolution as we use a patch size of  $16^2$ . To limit the interference of temporal representations with the outcome, we use an image dataset, STL10 [71], to perform unsupervised image classification using the frozen pretrained VSSL encoders. The results presented in Figure 4 (right) show that  $v$ -MAE achieves a more steady performance on low-resolution inputs compared to the other methods, which is in alignment with our hypothesis.
Highlights: (a) Video models generally struggle in out-of-context generalization, while  $v$ -Supervised and  $v$ -MAE exhibit better performance as they are strong temporal learners. (b) Contrastive methods ( $v$ -SimCLR,  $v$ -MoCo) exhibit better performance to viewpoint shifts. (c)  $v$ -MAE is robust against extremely low-resolution inputs. (d)  $v$ -Supervised shows extreme vulnerability in complex scenarios when multiple distribution shifts are applied concurrently.
Table 3: Comparison under source shift.
Method	UCF/HMDB	HMDB/UCF
Lin.	FT	Lin.	FT
OoD(H)	InD(U)	OoD(H)	InD(U)	OoD(U)	InD(H)	OoD(U)	InD(H)
v-Supervised	45.8±0.8	92.7±0.2	56.3	96.8	50.3±0.9	75.9±0.9	51.1	84.6
v-SimCLR	47.7±0.3	96.0±0.4	53.7	97.7	55.1±0.4	82.0±0.4	56.7	85.6
v-MoCo	51.5±0.6	97.1±0.5	55.7	98.9	57.2±0.4	84.5±0.4	58.3	86.3
v-BYOL	51.4±0.7	94.5±0.2	59.4	97.5	63.1±0.2	79.9±0.2	60.4	87.3
v-SimSiam	46.1±1.0	92.6±0.7	50.6	95.5	52.2±0.3	76.2±0.3	53.2	83.0
v-DINO	49.3±0.4	94.3±0.5	51.4	96.3	53.8±0.4	77.5±0.4	55.6	82.8
v-MAE	39.5±0.2	89.2±0.9	55.3	96.5	39.1±1.4	72.8±1.4	43.6	81.3
Table 4: Comparison in zero-shot action recognition.
Method	UCF	HMDB	RareAct	K400 (InD)
Lin.	FT	Lin.	FT	Lin.	FT	Lin.	FT
v-Supervised	n/a	37.4	n/a	19.0	n/a	9.9	n/a	59.0
v-SimCLR	37.2±1.8	40.3	18.6±1.7	24.9	7.7±0.6	10.4	56.8±0.2	69.8
v-MoCo	35.2±3.0	46.3	19.5±2.6	22.3	8.7±0.6	10.5	58.4±0.4	69.3
v-BYOL	33.0±1.4	41.0	22.4±2.1	24.7	7.5±0.4	10.3	57.4±0.1	69.3
v-SimSiam	34.0±1.6	38.7	18.8±1.0	21.0	7.7±0.3	11.1	50.4±0.0	64.3
v-DINO	34.3±1.0	41.3	17.2±0.3	22.8	8.1±0.3	10.5	53.4±0.2	65.7
v-MAE	25.5±1.4	42.1	14.2±0.7	25.8	5.8±0.9	10.7	35.9±0.0	68.4
Q2: Effect of finetuning under distribution shifts
Our thorough empirical experiments presented in Tables 1, 2, and 3, show several interesting aspects of finetuning under distribution shifts. First, finetuning generally results in better InD and OoD performance compared to linear evaluation (see Table 5). Moreover, the benefits of finetuning vary between different VSSL methods; for example,  $v$ -MAE benefits the most from finetuning (see Figures 5 and 6). Second, the benefits of finetuning depend on the nature of the distribution shift. Specifically, by comparing OoD and InD performance, we observe that finetuning is more helpful for actor shift in both animal and synthetic domains (see Figure 5), whereas it is less beneficial for zero-shot and viewpoint shift in both egocentric and surveillance camera view. Moreover, finetuning often leads to 'in-distribution overfitting' where the InD validation improves with additional training, while leading to poor OoD
generalization (see Appendix D). Overall, the results of zero-shot recognition presented in Table 4 show that no single method dominates in all three benchmarks for zero-shot recognition.  $v$ -MAE,  $v$ -MoCo, and  $v$ -SimSiam achieve the best performances on HMDB, UCF, and RareAct, respectively, when finetuned. Third, the benefits of finetuning also depend on the InD training benchmark. For example, as shown in Figure 5, while finetuning is significantly beneficial under the source shift of UCF to HMDB, it interestingly hurts the performance when this shift is reversed (HMDB to UCF). This can be attributed to the limited utility of small-scale datasets, such as HMDB, in learning strong
InD	OoD
Lin.	63.2%	31.5%
FT.	71.9% (↑ 8.7)	37.4% (↑ 5.9)
Table 5: Linear vs. finetuned comparison summary. Please see Appendix D.5 for additional details.
representations. Such datasets could potentially even diminish the effectiveness of robust pretrained models. For instance, in the case of  $v$ -BYOL, the performance of the frozen encoder exceeds that of the finetuned model, as seen in Table 3 (HMDB/UCF).
Figure 5: Linear vs. finetuned performance comparison under real-world distribution shifts. We measure the improvements as the difference between finetuned and linear evaluation accuracy.
To further investigate scenarios where finetuning may impair VSSL pretraining, we evaluate the video models on temporally perturbed inputs (e.g., freeze frames, random sampling, reverse sampling) in a controlled setup. As the VSSL methods are never trained with temporally perturbed inputs, this constitutes a distribution shift. The results presented in Figure 6 show that end-to-end finetuning of the contrastive and non-contrastive video encoders diminishes the time-invariance of the learned representations, which decreases robustness to temporal perturbations. As a result, in such cases, the frozen encoders perform better than the finetuned ones. Moreover, the frozen  $v$ -MAE shows the worst performance under temporal shifts as it learns time-variant representations through self-supervised pretraining.
Figure 6: Linear vs. finetuned performance comparison under synthetic temporal perturbations.
Highlights: (a) As opposed to LLMs, finetuning generally helps VSSL in both InD and OoD. (b) The benefits of finetuning largely vary between different VSSL methods and the type of distribution shifts. (c) Finetuning provides more benefits against actor shifts in both animal and synthetic domains in comparison to viewpoint shifts like egocentric and surveillance camera views. (d) Finetuning degrades robustness to temporal perturbations as it impairs the time-invariant representations of contrastive and non-contrastive methods. It can also degrade performance under source shift depending on the quality of the training benchmark.
Q3: Closed-set vs. open-set performance
We perform open-set recognition in two setups: (1) we use the Kinetics400 for closed-set recognition, while the non-overlapping classes from the UCF101 and HMDB51 are used for open-set. (2) we use UCF101 and the non-overlapping classes of HMDB for closed-set and open-set recognition, respectively. To perform a comprehensive analysis on the performance of closed-set vs. open-set, we adopt two approaches, i.e., single-objective (closed-set only with cross-entropy error) and joint-objective (closed-set and open-set recognition simultaneously with DEAR loss [45]) variants.
Table 6: Comparison in open-set recognition.  $v$ -Supervised trained from scratch with DEAR loss failed to converge in K/U and K/H. Here, closed/open-set pairs are denoted as K/H: Kinetics/HMDB, K/U: Kinetics/UCF, U/H: UCF/HMDB, CE: cross-entropy error.
(a.) End-to-end finetuned.
Method	Open-set (AUC)	Closed-set w/ DEAR (Acc.)	Closed-set w/ CE (Acc.)
K/U	K/H	U/H	K400*	U101*	K400*	U101*
v-Supervised	-	-	77.7	-	86.5	57.6	87.4
v-SimCLR	63.0	60.1	84.3	69.8	90.6	72.5	90.3
v-MoCo	61.3	61.0	85.2	70.1	90.4	72.8	90.7
v-BYOL	60.5	60.1	81.7	69.4	89.4	72.5	90.2
v-SimSiam	60.6	55.9	77.8	64.7	85.9	69.3	87.2
v-DINO	60.0	58.3	81.7	65.9	87.0	71.3	88.5
v-MAE	55.1	55.5	73.1	67.5	87.8	73.3	90.5
(b.) Linear evaluation.
Method	Open-set (AUC)	Closed-set w/ DEAR (Acc.)	Closed-set w/ CE (Acc.)
U/H	U101*	U101*
v-Supervised	77.3±0.1	80.4±0.1	81.7
v-SimCLR	49.9±0.3	11.6±1.0	84.1
v-MoCo	50.7±0.6	32.7±0.5	84.9
v-BYOL	56.0±1.5	63.7±0.8	85.4
v-SimSiam	73.7±0.3	79.5±0.2	82.5
v-DINO	79.4±0.0	80.4±0.2	80.9
v-MAE	66.0±0.0	74.6±0.2	76.2
(a)UCF/HMDB
(b) Kinetics400/UCF
(c) Kinetics400/HMDB
closed-set and open-set performance.
Figure 7: (a-c) Comparing open macro-F1 scores of finetuned models vs. openness (openness is measured as the ratio of unknown to known classes). (d) The relationships between closed-set and open-set recognition performance of frozen pretrained encoders.
(d) Relationship b/w
The results in Table 6 (a) demonstrate that the models struggle in open-set recognition when using Kinetics400 as closed-set in comparison to UCF101. This is due to the large number of known classes in Kinetics400 compared to the unknowns (400 known classes in Kinetics400 vs. 31 and 22 unknown classes from UCF and HMDB, respectively). Therefore, the models tend to become over-confident and make false predictions. Evident from Table 6 (a), contrastive methods are robust to open-set recognition in all setups. This is likely due to the auxiliary information contributed by the negative samples used in these methods. To verify this hypothesis, we analyze the open macro-F1 scores vs. openness of the video models following [45, 44] by incrementally adding more unknown classes. The results in Figure 7 (a-c) show that both  $v$ -SimCLR and  $v$ -MoCo consistently achieve better performances compared to others. An additional statistical analysis on the performance of contrastive methods in open-set recognition is presented in Appendix D.7, which confirms their better performance relative to other methods.
Next, instead of end-to-end finetuning, we use the frozen encoders to train a linear head for open-set recognition. The top 3 closed-set performers,  $v$ -BYOL,  $v$ -MoCo, and  $v$ -SimCLR (see Figure 7 (d) or U101* in Table 6 (b)), show the worst performance in open-set recognition. However, the pretrained encoders,  $v$ -SimSiam,  $v$ -DINO, and  $v$ -Supervised, which are considered weaker in closed-set, perform better in open-set recognition, which might be due to their lack of over-confidence.  $v$ -DINO outperforms other methods in open-set recognition achieving  $79.4\%$  while the performance of  $v$ -SimCLR and  $v$ -MoCo drops to almost chance levels. In comparison,  $v$ -MoCo reports  $84.9\%$  in standard closed-set recognition, whereas  $v$ -DINO achieves  $80.9\%$ . The results presented in Figure 7 (d) suggest that there is a trade-off between closed-set and open-set recognition performances, particularly when frozen encoders are used without finetuning.
Highlights: (a) Contrastive methods demonstrate superior performance in open-set recognition when finetuned. (b) There is a trade-off between closed-set and open-set recognition performance when frozen pretrained encoders are used. (c) Strong frozen encoders ( $v$ -MoCo,  $v$ -SimCLR) have no open-set generalization performance due to their over-confident predictions. (d) On the other hand, slightly weak VSSL frozen encoders ( $v$ -DINO,  $v$ -SimSiam) show better open-set performance, while  $v$ -MAE seems to perform poorly in both settings.
Figure 8: The decision similarity between the video models in InD (top) vs. OoD (bottom). The lighter color indicates less similarity.
Q4: Decision similarity under distribution shifts
To measure 'decision similarity' between the video models, we evaluate whether the models make similar predictions, regardless of their correctness. The results presented in Figure 8 demonstrate that decision similarity varies between the VSSL methods both InD and OoD, but is generally lower in OoD settings. Specifically, while we observe only a slight decrease in decision similarity in the case of animal domain actor shift, we observe significant drops in decision similarity in the case of context shifts and source shifts. Moreover, the results reveal that the decision similarity between supervised and self-supervised methods significantly drops under distribution shifts, indicating that under such shifts, VSSL methods make considerably different decisions than supervised learning. We also observe low decision similarity between  $v$ -MAE and other VSSL methods, which can be due to the generative nature of  $v$ -MAE vs. the others. Lastly, we note that contrastive methods tend to have greater decision similarities with each other than the similarities observed among non-contrastive methods. Additional results are presented in Appendix D.
Highlights: (a) The decision similarity of video models decreases under distribution shifts, which further varies based on the type of shift. (b) Context and source shifts cause the most dissimilarity between decisions. (c) Overall, the predictions between the supervised and self-supervised methods, as well as between  $\mathbf{v}$ -MAE and other VSSL methods exhibit the least similarity.
7 Discussion and Summary
Limitations. We consider contrastive, non-contrastive, and generative VSSL methods in our work, as they are well-established and have demonstrated strong performance in previous studies [72, 1, 3, 5] on various video benchmarks [1, 87, 72]. However, there exists another category of VSSL methods which uses pretext tasks for self-supervision, e.g., rotation prediction [88], frame and clip order prediction [14, 89], motion prediction [90], and others [91, 92, 93, 94]. While these methods are not included in our study, they are worth exploring in future research. Moreover, our work primarily focuses on various self-supervised methods as opposed to network architectures. It would be valuable to further investigate VSSL under distribution shifts with larger Transformer [95] or convolutional [96] architectures, which we could not perform due to resource constraints. Nonetheless, our findings serve as a foundation for future studies. We discuss the broader impact of our work in Appendix A.
Summary. In this work, we thoroughly examine the behavior of VSSL methods under real-world distribution shifts that commonly occur in videos due to changes in context, viewpoint, actor, and source. Moreover, our study delves into investigating the generalizability of VSSL methods in zero-shot and open-set recognition. To rigorously evaluate the robustness of video models, we introduce a comprehensive OoD test bed curated from existing literature. This test bed is carefully designed to stress test the robustness of video models and provide a comprehensive evaluation of their capabilities. Our study uncovers a wide range of interesting dynamics of various VSSL methods under different distribution shifts, which can be instrumental in guiding future research and algorithm development in video representation learning. To the best of our knowledge, this is the first work to systematically investigate VSSL under real-world distribution shifts.
PS and AE are grateful to the Bank of Montreal and Mitacs for funding their research. PS and AE are also thankful to SciNet HPC Consortium for helping with the computation resources. We thank Vandad Davoodnia (Queen's University) and Ananth Balashankar (Google Research) for fruitful discussions and feedback.
[1] Zhan Tong, Yibing Song, Jue Wang, and Limin Wang. Videomae: Masked autoencoders are data-efficient learners for self-supervised video pre-training. arXiv preprint arXiv:2203.12602, 2022. 1, 2, 3, 5, 10, 21, 30
[2] Pedro Morgado, Nuno Vasconcelos, and Ishan Misra. Audio-visual instance discrimination with cross-modal agreement. In CVPR, pages 12475–12486, 2021. 1, 2, 5, 6, 26
[3] Rui Qian, Tianjian Meng, Boqing Gong, Ming-Hsuan Yang, Huisheng Wang, Serge Belongie, and Yin Cui. Spatiotemporal contrastive video representation learning. In CVPR, pages 6964-6974, 2021. 1, 2, 5, 6, 10
[4] Humam Alwassel, Dhruv Mahajan, Bruno Korbar, Lorenzo Torresani, Bernard Ghanem, and Du Tran. Self-supervised learning by cross-modal audio-video clustering. NeurIPS, 33, 2020. 1, 2, 5, 26
[5] Tian Pan, Yibing Song, Tianyu Yang, Wenhao Jiang, and Wei Liu. Videomoco: Contrastive video representation learning with temporally adversarial examples. In CVPR, pages 11205-11214, 2021. 1, 2, 5, 6, 10
[6] Pritam Sarkar and Ali Etemad. Xkd: Cross-modal knowledge distillation with domain alignment for video representation learning. arXiv preprint arXiv:2211.13929, 2022. 1, 2, 5, 26
[7] Pritam Sarkar and Ali Etemad. Self-supervised audio-visual representation learning with relaxed cross-modal temporal synchronicity, 2021. 1, 2, 5, 6, 26
[8] Limin Wang, Bingkun Huang, Zhiyu Zhao, Zhan Tong, Yinan He, Yi Wang, Yali Wang, and Yu Qiao. Videomae v2: Scaling video masked autoencoders with dual masking. arXiv preprint arXiv:2303.16727, 2023. 1, 2
[9] Li Haopeng, Ke Qiuhong, Gong Mingming, and Zhang Rui. Video summarization based on video-text modelling. arXiv preprint arXiv:2201.02494, 2022. 1
[10] Karan Desai and Justin Johnson. Virtex: Learning visual representations from textual annotations, 2021. 1
[11] Lu Yuan, Dongdong Chen, Yi-Ling Chen, Noel Codella, Xiyang Dai, Jianfeng Gao, Houdong Hu, Xuedong Huang, Boxin Li, Chunyuan Li, Ce Liu, Mengchen Liu, Zicheng Liu, Yumao Lu, Yu Shi, Lijuan Wang, Jianfeng Wang, Bin Xiao, Zhen Xiao, Jianwei Yang, Michael Zeng, Luowei Zhou, and Pengchuan Zhang. Florence: A new foundation model for computer vision, 2021. 1
[12] Yi Wang, Kunchang Li, Yizhuo Li, Yinan He, Bingkun Huang, Zhiyu Zhao, Hongjie Zhang, Jilan Xu, Yi Liu, Zun Wang, et al. Internvideo: General video foundation models via generative and discriminative learning. arXiv preprint arXiv:2212.03191, 2022. 1, 2
[13] Uta Buchler, Biagio Brattoli, and Bjorn Ommer. Improving spatiotemporal self-supervision by deep reinforcement learning. In ECCV, 2018. 1
[14] Hsin-Ying Lee, Jia-Bin Huang, Maneesh Singh, and Ming-Hsuan Yang. Unsupervised representation learning by sorting sequences. In CVPR, 2017. 1, 10
[15] Hyeon Cho, Taehoon Kim, Hyung Jin Chang, and Wonjun Hwang. Self-supervised spatiotemporal representation learning using variable playback speed prediction. arXiv preprint arXiv:2003.02692, 2020. 1
[106] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018. 27
[107] Harold W Kuhn. The hungarian method for the assignment problem. Naval research logistics quarterly, 1955. 27
[108] Xiaohan Wang, Tengyu Ma, James Ainooson, Seunghwan Cha, Xiaotian Wang, Azhar Molla, and Maithilee Kunda. Seeing neural networks through a box of toys: The toybox dataset of visual object transformations, 2018. 27
The organization of the supplementary material is as follows:
Appendix A: Broader impact
Appendix B: Video self-supervised learning
B.1 Methods
B.2 Implementation details
B.3 Additional insights
Appendix C: Distribution shifts
C.1 Overview
C.2 Context shift
C.3 Viewpoint shift
C.4 Actor shift
C.5 Source shift
C.6 Zero-shot recognition
C.7 Open-set recognition
C.8 Toy experiments
Appendix D: Additional experiments and results
D.1 Pretrained on Kinetics700
D.2 Temporal dynamics
D.3 Comparing InD vs. OoD performance under natural distribution shifts
D.4 Effect of synthetic perturbations
D.5 Effect of finetuning in out-of-distribution generalization
D.6 High-level overview
D.7 Statistical analyses
D.8 Exploring performance variation of VSSL methods across action classes
In this work, we conduct a thorough study on the behavior of Video Self-supervised Learning (VSSL) methods in the face of distribution shifts. We believe this work has far-reaching implications for advancing the field of video representation learning and its real-world applications. This work is of utmost importance primarily for the following two reasons.
Real-world deployment: Video models have a wide range of real-world applications across various domains. These applications include surveillance and security, autonomous vehicles, sports analysis, video content recommendation, human-computer interaction, video captioning and summarization, among many others. However, in real-world scenarios, distribution shifts pose a significant challenge, where the distribution of the data that the system encounters may differ from the training data distribution. Understanding how VSSL algorithms behave under real-world distributional shifts is crucial for ensuring their reliable performance and generalization capabilities for real-world applications. By studying VSSL methods under distribution shifts, we identify the limitations and challenges faced by the popular VSSL methods. We believe our work is a step towards enabling the development of robust and reliable solutions for real-world deployment.
Algorithmic robustness: Our study provides valuable insights into the algorithmic robustness of VSSL methods, advancing our understanding of VSSL algorithms. By systematically exploring the behavior of these methods under distribution shifts, we have uncovered intriguing findings and observed interesting behaviors which are previously unknown. Our work not only highlights the strengths of VSSL methods but also identifies their weaknesses in different scenarios. We find that different VSSL methods exhibit varying levels of robustness when confronted with different types of distribution shifts. Some methods demonstrate resilience against certain shifts, while they prove to be vulnerable to others. These findings pave the way for future research to develop robust frameworks that leverage the strengths of different approaches, thereby enhancing the overall performance of VSSL algorithms.
B Video self-supervised learning
In the following section, we provide a comprehensive overview of our implementation of the VSSL methods explored in this study. We define an encoder  $f(\cdot)$ , which is composed of a Transformer backbone  $\theta(\cdot)$  and an MLP projection head  $h(\cdot)$ . We further define the auxiliary network components, a target encoder  $f_{t}(\cdot)$  (utilized in  $v$ -MoCo,  $v$ -BYOL, and  $v$ -DINO), a decoder  $f_{d}(\cdot)$  (employed in  $v$ -MAE), and a predictor head  $f_{p}(\cdot)$  (utilized in  $v$ -BYOL,  $v$ -SimSiam, and  $v$ -MoCo). Note,  $f_{t}(\cdot)$  and  $f_{d}(\cdot)$  are built on the Transformer architecture, while  $f_{p}(\cdot)$  utilizes an MLP head. For given samples  $v$ , we generate two augmented views as  $v_{1}$  and  $v_{2}$ , where  $v_{1}$  and  $v_{2}$  are randomly sampled from different timestamps and differently augmented. Moreover, an augmented view  $v_{i} \in \mathbb{R}^{T \times H \times W \times C}$  can be expressed into a sequence of P spatio-temporal patches of size  $t \times h \times w \times C$ , where,  $P = T / t \times H / h \times W / w$ . We project the spatio-temporal patches to a linear layer followed by feeding to the encoders. Following, we briefly summarize the training algorithms of these VSSL methods along with their design differences. An overview of these frameworks is presented in Figure S1.
(a)  $v$ -SimCLR
(b)  $v$ -MoCo
(c)  $v$ -MAE
(d)  $v$ -BYOL
(e)  $v$ -SimSiam
(f)  $v$ -DINO
Figure S1: A simplified version of the video self-supervised methods are presented.