SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Pritam Sarkar $^{1,2}$
Ahmad Beirami3
Ali Etemad<sup>1,3</sup>
$^{1}$ Queen's University
{pritam.sarkar,ali.etemad}@queensu.ca
beirami@google.com
Video self-supervised learning (VSSL) has made significant progress in recent years. However, the exact behavior and dynamics of these models under different forms of distribution shift are not yet known. In this paper, we comprehensively study the behavior of six popular self-supervised methods ( $v$ -SimCLR,  $v$ -MoCo,  $v$ -BYOL,  $v$ -SimSiam,  $v$ -DINO,  $v$ -MAE) in response to various forms of natural distribution shift, i.e., (i) context shift, (ii) viewpoint shift, (iii) actor shift, (iv) source shift, (v) generalizability to unknown classes (zero-shot), and (vi) open-set recognition. To perform this extensive study, we carefully craft a test bed consisting of 17 in-distribution and out-of-distribution benchmark pairs using available public datasets and a series of evaluation protocols to stress-test the different methods under the intended shifts. Our study uncovers a series of intriguing findings and interesting behaviors of VSSL methods. For instance, we observe that while video models generally struggle with context shifts,  $v$ -MAE and supervised learning exhibit more robustness. Moreover, our study shows that  $v$ -MAE is a strong temporal learner, whereas contrastive methods,  $v$ -SimCLR and  $v$ -MoCo, exhibit strong performances against viewpoint shifts. When studying the notion of open-set recognition, we notice a trade-off between closed-set and open-set recognition performance if the pretrained VSSL encoders are used without finetuning. We hope that our work will contribute to the development of robust video representation learning frameworks for various real-world scenarios. The project page and code are available at: https://pritamqu.github.io/00D-VSSL.
Self-supervised learning has achieved tremendous success in learning strong and meaningful representations in various video domains, such as action recognition [1, 2, 3, 4, 5, 6, 7], action localization [8], video summarization [9], and video captioning [10, 11, 12]. Considering the diversity and complexity of the video domain, real-world deployment of video-based intelligent systems requires to understand the model performance under distribution shifts. Distribution shifts may occur due to differences in contextual information, viewpoint, geographical location, and the presence of unknown classes with respect to the training data, among others.
Despite the vast amount of work on VSSL [13, 14, 15, 16, 17, 18, 5, 1, 19, 8, 12], a number of fundamental questions regarding the out-of-distribution behavior and dynamics of VSSL methods remain unanswered. To date, there have been no comprehensive studies of these aspects, which we attempt to address in this paper. Specifically, we pose and answer the following questions:
Q1. How do the learned spatial and temporal representations vary based on different VSSL pretraining methodologies? How robust are these representations to different distribution shifts?
Q2. Considering recent findings about the robustness of finetuning on the generalizability of large language models (LLMs) [20, 21], we pose the question: How does finetuning influence the out-of-distribution (OoD) generalization and zero-shot performance of VSSL?
Q3. How do VSSL methods perform on open-set problems (where test samples can be from classes previously unknown to the model)? And what is the relationship between performance in closed-set vs. open-set recognition?
Q4. Do different VSSL methods exhibit comparable decision-making patterns ('decision similarity') given the same training conditions? And how is this impacted by different distribution shifts?
To address these questions, we consider six different self-supervised learning algorithms, (i) SimCLR [22], (ii) MOCO-v3 [23], (iii) BYOL [24], (iv) SimSiam [25], (v) DINO [26], and (vi) MAE [27], along with fully supervised learning, and analyze their behaviors in various OoD settings. We select these methods to cover three key categories of self-supervision, namely contrastive (SimCLR, MoCo-v3), non-contrastive (BYOL, SimSiam, DINO), and generative (MAE), approaches. In particular, these methods represent fundamental approaches on which numerous variants have been built and therefore represent many existing nuanced solutions in the area [2, 3, 4, 5, 6, 7, 1, 19, 8, 12]. For distribution shifts, we study a series of different possibilities which occur in videos in real-world settings due to changes in context (e.g., real vs. mime actions) [28, 29, 30, 31, 32], viewpoint (e.g., third-person vs. ego-centric view) [33, 34, 35, 36], actor (e.g., real-world vs. synthetic) [37, 38], and data sources (e.g., different datasets) [39, 40]. Moreover, we evaluate the generalizability of the VSSL methods on unseen classes, i.e., zero-shot recognition [41, 42, 43] and open-set recognition [44, 45, 46, 47]. To
Figure 1: We present a high-level overview of different video learning methods depicting their performance under different distribution shifts. We normalized the eval. metric of each method to that of the highest performing method in each OoD scenario and averaged the results over multiple OoD datasets. See Appendix D.6 for more details.
perform this investigation, we design a comprehensive study consisting of 17 in-distribution and out-of-distribution (InD-OoD) dataset pairs and examine the dynamics of VSSL under different distribution shifts using a variety of evaluation protocols including linear evaluation, finetuning, unsupervised clustering, and zero-shot recognition. Moreover, for a fair comparison, the VSSL methods are pretrained in identical experimental setups. We provide a high-level overview comparing the performance of all methods across all shifts in Figure 1. To the best of our knowledge, this is the first work to study VSSL under distribution shift to this depth.
- We present the first comprehensive and large-scale systematic investigation of real-world distribution shifts on VSSL methods. Our study encompasses 2 large-scale pretraining datasets, 7 video learning algorithms, 17 InD-OoD dataset pairs, as well as 3 toy datasets. Our thorough evaluation involves a total of 269 experiments, covering various evaluation protocols including linear evaluation, finetuning, unsupervised clustering, zero-shot recognition, and open-set recognition.
- Our study uncovers a series of intriguing behaviors and relationships of various VSSL methods, shedding light on the strengths and weaknesses that often go unnoticed in InD validation. Moreover, our investigation provides a comprehensive and impartial perspective on the effectiveness of supervised vs. self-supervised pretraining.
The analysis of robustness is a well-established area in computer vision, with an extensive body of research dedicated to image-based models [48, 49, 50, 51, 52]. Despite the growing popularity of video models in different domains, detailed comparative studies on their robustness remains
under-explored. We come across two recent works [53, 54] that study the behavior of video models (supervised) against synthetic perturbations. An initial study [54] explores the performance of video models against spatial corruptions like noise, blur, color jittering and others. In a subsequent work, [53] extends the analysis of video models on temporal perturbations like frame reversal, jumbling, and freezing among others. In particular, our study focuses on real-world distribution shifts, which we find crucial as synthetic perturbations yield little or no consistency to the natural distribution shifts of real data [49, 48]. Moreover, none of the prior works attempts to understand the behavior of video self-supervised methods under various forms of distribution shift.
Contrastive methods. We study two popular contrastive methods:  $v$ -SimCLR $^1$  and  $v$ -MoCo. These methods learn representations by maximizing the similarity of positive pairs while minimizing the similarity of negative pairs using the InfoNCE loss [22, 23, 55], where the similarity scores are calculated using L2-normalized cosine distance. The implementation of our  $v$ -MoCo is based on MoCo-v3 [23], and it uses a momentum target encoder to fetch the key embeddings for the corresponding query embeddings. The momentum encoder is progressively updated from the online encoder using an exponential moving average (EMA) technique [56]. In contrast,  $v$ -SimCLR directly uses the online encoder to compute the similarity scores from the embeddings of the given views. Additionally,  $v$ -MoCo employs a predictor head, unlike  $v$ -SimCLR.
Non-contrastive methods. We study three popular non-contrastive methods  $v$ -BYOL,  $v$ -SimSiam, and  $v$ -DINO. These methods learn meaningful representations by minimizing the distance between positive views of a given sample, without requiring any negative pairs.  $v$ -BYOL uses an architecture similar to  $v$ -MoCo, but instead of optimizing an InfoNCE loss, it minimizes the L2-normalized cosine distance of the positive views [24, 25].  $v$ -SimSiam is an ablation variant of  $v$ -BYOL that does not use a target encoder and directly obtains the embeddings corresponding to the views from the online encoder. The setup of  $v$ -DINO is also similar to  $v$ -BYOL, but it introduces 'Centering' [26] to prevent against model collapse, instead of using a predictor head like  $v$ -BYOL and  $v$ -SimSiam.
Generative method. Finally, we also study  $v$ -MAE which aims to learn representations through reconstructions of heavily masked inputs.  $v$ -MAE employs an autoencoder architecture [27, 1, 57] where the encoder compresses the input information, which is then reconstructed by the decoder while minimizing the L2-reconstruction error [27].
We provide additional details and design specifics for each method in Appendix B.
4 Distribution Shifts
Let  $p_y$  denote a probability distribution over labels. Let  $p_{x|y}$  denote a class conditional density of input  $x$  given labels  $y$ . We draw  $y$  from  $p_y$  and then  $x$  from  $p_{x|y}(x|y)$ . In the case of InD, there is no distribution shift and the validation set is drawn from the exact same distribution as above. We consider two types of distribution shifts in this paper: input-based and output-based. For Input shift, we assume that the class conditional density shifts from  $p_{x|y}$  to  $q_{x|y}$ . To measure the OoD performance, we choose  $q_{x|y}$  such that there is no overlap between the distributions of the unlabelled pretraining data and OoD validation sets. For Output shift, we assume that the label distribution shifts from  $p_y$  to  $q_y$ . Note that we are particularly interested in cases where the support set of  $q_y$  might be different from that of  $p_y$ , i.e., novel classes appear at test time.
We study four different input shifts, (i) context shift, (ii) viewpoint shift, (iii) actor shift, and (iv) source shift. Context shift or out-of-context refers to when the scenic background or contextual information is misleading or absent, e.g., mime actions. Humans possess a deep understanding of actions and can thus recognize actions without much context. However, vision models tend to rely heavily on contextual cues and background objects to make predictions [28, 58, 59]. As a result, models may generalize poorly when contextual information is absent or different from what the model has been trained on. Viewpoint shift refers to the change in the viewpoint or perspective of the camera that captures the scene. Some of the popular viewpoints in the video domains include third-person view, egocentric view, bird's-eye view, top-down view, and surveillance camera view,
$^{1}\pmb{v}$  denotes the video variant of the image-based self-supervised method.
Cut vs. hammer watermelon
Soccer
Basketball
Figure 2: Sample video frames of distribution shifts. In these examples, the left frames of each category represent an InD sample and the right frames represent an OoD sample.
Badminton
Hockey
among a few others. In this work, we examine the generalizability of models trained on third-person views to other viewpoints including egocentric, surveillance camera, and top-down views. Further, we study generalizability under actor shift, which occurs when the 'type' of actors changes between training and test sets. These actor-type shifts can include human-to-animal shifts or synthetic-to-real shifts. We consider human actions in the real world as InD while animal actions and synthetic videos (e.g., video games, animation) are considered OoD. Lastly, we study the distribution shifts caused due to the changes in the data sources, referred to as source shift. As discussed in [39, 40], datasets of similar classes also exhibit distribution shifts due to the difference in curation strategies, annotation schemes, demographics, geographical locations, and other factors, even when none of the specific shifts mentioned earlier apply. To assess the generalizability of video models under real-world distribution shifts, we also investigate their performance when faced with multiple forms of distribution shifts occurring simultaneously. For example, we evaluate the model's ability to handle 'top-down synthetic videos' which causes both viewpoint and actor shifts concurrently. A few examples are presented in Figure 2.
We also investigate the dynamics of video models under output shifts. We perform zero-shot recognition on both regular actions and unusual or rare actions. An example of rare actions would be 'hammering watermelon' as opposed to 'cutting watermelon', as shown in Figure 2. Lastly, we evaluate performance in open-set problems where models are required to distinguish between known vs. unknown classes while correctly predicting the known ones [60, 45, 44, 46]. Deep learning models are known to struggle in such scenarios due to their tendency towards over-confident predictions. We note that existing literature has only evaluated VSSL methods under closed-set scenarios, neglecting the importance of their performance in real-world open-set settings.
5 Experiment setup
Benchmarks. We use two large-scale video action datasets, Kinetics400 [61] and Kinetics700 [62], for pretraining the video models. The results presented in the main paper use Kinetics400 for pre-training, while we provide additional results based on pretraining with Kinetics700 in Appendix D. To evaluate the video models under distribution shifts, we use a total of 12 real-world benchmarks, comprising: Mimetics10 and Mimetics50 [28] as the out-of-context validation sets; CharadesEgo [36], TinyVirat-v2 [63], and Sims4Action [64] to investigate viewpoint shifts (egocentric, surveillance camera, and top-down views, respectively); ActorShift [65] and Sims4action [64], for actor shifts (animal and synthetic domains, respectively); UCF101 [66] and HMDB51 [67] for source shift; UCF101 [66], HMDB51 [67], and RareAct [68] for zero-shot recognition; UCF101 and HMDB51 for open-set recognition while using Kinetics400 and UCF101 as closed-set. For each OoD validation set, we create an InD training and validation set to measure the change in performance. We construct the InD splits using Kinetics400 [61], Kinetics700 [62], MiT-v2 [69], and CharadesEgo [36]. Finally, we also use 3 toy datasets to conduct experiments in controlled
setups, including ToyBox [70], COIL [34], STL-10 [71]. Additional details of the benchmarks can be found in Appendix C.
Pretraining. To ensure a fair comparison between the VSSL methods, we pretrain them in identical setups with necessary adjustments in hyperparameters. Although some of the methods studied in this work are already available in the literature [3, 5, 72, 1], they follow a variety of experiment setups including different architectures (e.g., R2+1D [73], R3D [72], TSM [74], ViT [75]), inputs, and others. Therefore, they could not be directly adopted for our experiments and are instead re-implemented. Specifically, we keep the encoder, inputs, batch sizes, and maximum number of pretraining epochs the same for all the methods. Furthermore, VSSL methods are tuned based on the InD validation split, with no exposure to OoD validation sets. We use the ViT-Base [75, 76] as the encoder, with a patch size of  $4 \times 16^2$ . Amongst the 6 VSSL methods studied in this paper, all of them use a Siamese [77] architecture other than  $v$ -MAE. Therefore, the contrastive and non-contrastive methods are fed with 2 random spatio-temporal augmented crops from the original videos, similar to [72]. For a fair comparison, the  $v$ -MAE which requires a single view as input, is fed with  $3 \times 32 \times 112^2$  inputs, while the other VSSL methods are fed with  $3 \times 16 \times 112^2$  inputs per view. Additional details for VSSL pretraining can be found in Appendix B.
Evaluation. To study input-based distribution shifts, we perform linear evaluation and finetuning using the InD training splits, followed by evaluating on both InD and OoD validation splits. We follow the standard protocol used in [72, 2, 4, 6, 78, 79, 7] for linear evaluation and finetuning. To evaluate the models on zero-shot recognition, we follow [41, 42] and jointly train video-text encoders using the Word2Vec [80] word embeddings. Lastly, we follow [45] for open-set recognition. We report top-1 accuracy for multi-class classification and mean average precision (meanAP) for multi-label multi-class classification. Moreover, for open-set problems, we report the area under the ROC curve (AUC) for distinguishing known vs. unknown, and accuracy for measuring closed-set performance. Please find more details in Appendix C.
6 Findings
Q1: Dynamics of learned spatial and temporal representations under distribution shifts
Our experiments on context shift show that video models greatly suffer in out-of-context generalization, as OoD performance significantly drops for all of the methods as presented in Table 1. Notice that  $v$ -Supervised achieves the best OoD performance under linear evaluation (Lin.) and  $v$ -MAE achieves the best when finetuned (FT), for both benchmarks. Intuitively speaking, the models need to learn strong temporal dynamics to generalize well under context shift, as the background or contextual information may be misleading or absent.  $v$ -MAE and  $v$ -Supervised show strong temporal learning capabilities as they learn time-variant representations. This is in contrast to the other (contrastive and non-contrastive) methods which encourage learning time-invariant or time-persistent representations by minimizing the embedding distance of the positive pairs sampled from two different timestamps. Additionally, our statistical analysis in Appendix D.7 confirms the higher robustness of  $v$ -Supervised and  $v$ -MAE against context shift (10 class) in both linear and finetuned setups. Moreover, in context shift with 50 classes,  $v$ -Supervised exhibits more robustness in linear evaluation, while  $v$ -MAE is more robust when finetuned.
Table 1: Comparison under context shift.
Method	Context (10 class)	Context (50 class)
Lin.	FT	Lin.	FT
OoD	InD	OoD	InD	OoD	InD
v-Supervised	37.0±0.4	89.2±0.5	41.2	92.7	15.0±0.5	66.3±0.5
v-SimCLR	31.4±1.5	92.5±0.2	35.3	94.4	14.9±0.5	71.7±0.5
v-MoCo	29.7±0.4	92.2±0.6	39.0	94.2	15.0±0.2	74.0±0.2
v-BYOL	31.6±0.7	89.3±0.2	41.2	94.6	14.4±0.2	71.8±0.2
v-SimSiam	30.8±0.7	89.5±0.5	40.4	93.8	13.8±0.6	67.9±0.6
v-DINO	34.8±0.8	90.4±0.1	40.4	93.6	13.0±0.3	68.7±0.3
v-MAE	33.3±1.8	82.9±1.0	41.2	95.2	12.3±0.4	56.4±0.4
(a) Transformation recog.
Figure 3: Comparing VSSL methods on disentangled (a) temporal and (b) spatial representations.
(b) Object recog.
To further investigate the performance of the video models in learning temporal dynamics, we conduct a toy experiment in a controlled setup using ToyBox [70]. ToyBox consists of 9 distinct temporal transformations of 12 different toy objects and transformations include positive rotations, negative rotations, and translations across the  $x$ ,  $y$ , and  $z$  axes. We use the frozen VSSL encoders to extract embeddings, which are then used to perform a K-means clustering to distinguish the transformations. As the static spatial information is non-discriminative to the temporal transformations, the models have to understand the underlying temporal dynamics to correctly distinguish the applied transformations. The results, presented in Figure 3 (a), show that  $\mathbf{v}$ -MAE and  $\mathbf{v}$ -Supervised outperform the other methods by a large margin on this task, confirming that they are better temporal learners. Additionally, to ensure that the superiority of  $\mathbf{v}$ -MAE and  $\mathbf{v}$ -Supervised are not due to spatial representations, we also test the frozen encoders in object recognition using the videos of the same static objects (i.e., no temporal information), and we find that  $\mathbf{v}$ -MAE and  $\mathbf{v}$ -Supervised are worse among all of them (see Figure 3 (b)). This proves that  $\mathbf{v}$ -MAE and  $\mathbf{v}$ -Supervised are indeed strong temporal learners while being weak spatial learners.
These findings raise an interesting question: if the contrastive and non-contrastive methods are not effective temporal learners, then how do they perform well on action recognition [72, 5, 81, 2, 79, 82, 3, 7]? We believe that the strong performance of these methods on action recognition can be attributed to the significance of spatial information in classifying human activities. The temporal sequence of representations might not be as crucial for these tasks, which possibly allows contrastive and non-contrastive methods to perform well.
Table 2: Comparison of video models in viewpoint and actor shifts.
Method	Viewpoint (ego.)	Viewpoint (surv.+low res)	View+Act (t-down+syn.)	Actor (animal)
Lin.	FT	Lin.	FT	Lin.	FT	Lin.	FT
OoD	InD	OoD	InD	OoD	InD	OoD	InD	OoD	InD	OoD	InD
v-Supervised	11.4±0.1	14.3±0.1	13.6	17.8	23.1±0.2	33.4±0.2	24.5	43.4	28.5±0.6	62.3±0.6	44.3	76
v-SimCLR	12.7±0.2	14.7±0.1	15.6	19.6	26.1±0.6	39.1±0.6	28.0	47.5	42.4±1.8	67.8±1.8	63.1	78.8
v-MoCo	13.3±0.1	15.0±0.2	16.1	19.4	24.8±1.2	40.0±1.2	27.6	48.6	41.1±0.4	67.9±0.4	62.8	80.0
v-BYOL	12.0±0.1	14.4±0.1	15.1	18.7	22.7±0.7	37.8±0.7	24.7	46.9	37.3±0.2	65.0±0.2	56.2	78.2
v-SimSiam	11.6±0.2	14.1±0.1	13.7	17.6	23.3±0.4	34.3±0.4	25.8	45.4	40.0±1.0	65.5±1.0	53.6	76.0
v-DINO	12.0±0.2	14.4±0.1	13.7	17.4	22.3±0.9	35.3±0.9	24.2	45.1	35.3±0.4	62.9±0.4	50.3	77.5
v-MAE	10.9±0.0	13.7±0.1	14.2	21.4	23.5±0.9	32.0±0.9	29.1	48.6	37.8±3.0	58.0±3.0	61.1	76.2
The results on viewpoint shifts, presented in Table 2 reveal that contrastive methods ( $v$ -SimCLR and  $v$ -MoCo) generally achieve superior performance in all three setups in both linear and finetuning schemes. Moreover, based on the statistical analysis in Appendix D.7,  $v$ -SimCLR exhibits more robustness to all three viewpoint shifts, while  $v$ -MoCo exhibits robustness in egocentric and top-down viewpoint shifts. We believe the better performance of  $v$ -SimCLR and  $v$ -MoCo is due to the availability of negative samples during pretraining which improves viewpoint invariance compared to the other approaches. Note that while aggressive cropping also improves viewpoint invariance as discussed in [83], we do not notice such improvements in non-contrastive methods although a similar aggressive cropping is applied. It is worth noting that while  $v$ -Supervised exhibits a comparable performance to VSSL methods in egocentric and surveillance camera viewpoint shifts, its performance decreases significantly when multiple shifts are applied concurrently, as evident in the case of synthetic top-down viewpoint shift. This suggests that supervised learning may not generalize well in scenarios with more complex and realistic distribution shifts.
For further investigation, we study viewpoint invariance in a controlled setup using the pretrained VSSL encoders through unsupervised evaluation, avoiding any form of additional training which may alter the learned representations. Moreover, to limit the influence of temporal representations we use an image-based toy dataset COIL100 [34] which is commonly used to test viewpoint invariance [84, 85], as it contains images of similar objects from different angles. Our results presented in Figure 4 (left) show that  $v$ -MoCo achieves the best performance, fol