SlowGuess/NeurIPS_2023_OCR huggingface dataset:
To enable the models for open-set recognition, we follow [45] and train the pretrained encoders with the DEAR loss which aims to calibrate the uncertainty to ensure the models show high uncertainty towards the unknown and low uncertainty towards the known. We redirect readers to [45] for more information about open-set training. We conduct open-set recognition in both finetuned and linear evaluation setups. We observe that both supervised training and linear evaluation setups using VSSL encoders did not converge when Kinetics400 was used for closed set recognition. This is likely because the models tend to become over-confident due to the large number of known classes in Kinetics400 compared to the unknowns. To measure the models' performance we report the area under the ROC curve (AUC) for open-set recognition and accuracy for closed-set recognition.
HMDB51. The non-overlapping classes from HMDB51 used in open-set recognition while using UCF101 as the closed-set: brush_hair, cartwheel, chew, clap, climb_stairs, drink, eat, fall_floors, flic_FLAC, hit, hug, kick, kiss, laugh, pick, pour, push, run, shakeHands, shoot_ball, shoot_bow, shoot-gun, sit, situp, smile, smoke, somersault, stand, swing_baseball, sword, sword exercise, talk, turn, wave.
C.8 Toy experiments
In all of our toy experiments, we use the pretrained frozen encoders to extract the fixed representations, which are then used for K-means clustering. Additionally, we apply the Hungarian algorithm [107] for assigning the formed cluster to the targets and measure the accuracy. The details of the datasets used to conduct the toy experiments are as follows.
C.8.1 ToyBox
The ToyBox [108] consists of egocentric views of 9 distinct temporal transformations of different toy objects. The transformations include positive rotations, negative rotations, and translations across the  $x$ ,  $y$ , and  $z$  axis. A few representative frames illustrating such transformations are presented in Figure S5. Using the pretrained frozen encoders, we aim to correctly cluster different transformations across various objects. Moreover, as the static spatial information is non-discriminative to the temporal transformations, we aim to form clusters solely based on temporal representations. The main motivation behind this experiment is to disentangle spatial representations from the embedding space to independently evaluate the models on their ability to learn temporal dynamics.
$^{2}$ sklearn.cluster.KMeans
C.8.2 COIL100
We use COIL100 [34] to study viewpoint invariance which consists of images of different objects captured from different camera angles. We particularly choose an image-based dataset for this experiment to restrict any discriminatory temporal features in the embedding space that may attribute to the outcome of the models' prediction. We present a few representative frames in Figure S6
C.8.3 STL10
We use an image dataset STL10 [71] to test the robustness of the pretrained encoder against low-resolution inputs. We systematically reduce the resolution from  $112^{2}$  to  $16^{2}$ , please see a few examples in Figure S7. Similar to our viewpoint shift setup, we choose an image dataset to restrict any influence of temporal representations in the outcome.
positive rotations across  $x$ -axis
negative rotations across  $x$ -axis
positive rotations across  $y$  -axis
positive rotations across  $z$ -axis
negative rotations across  $y$  -axis
negative rotations across  $z$ -axis
Figure S5: Samples from ToyBox showing rotations across different axes.
Figure S6: Samples from COIL100 showing different viewpoints.
C.9 License of Dataset
In Table S6, we summarize the licensing terms for the datasets used in this study.
Figure S7: Samples from STL10 presenting low-resolution inputs.
Table S6: Licensing terms for the datasets used in this study.
Dataset	License	Link
CharadesEgo	Non-commercial use	https://prior.allenai.org/projects/charades-ego
Moments-in-Time-v2	Non-commercial use	http://moments.csail.mit.edu/
Kinetics	CC BY 4.0	https://www_deepmind.com/open-source/kinetics
HMDB51	CC BY 4.0	https://serre-lab.clps.brown.edu/resource/hmdb-a-large-human-motion-database/
ToyBox	CC BY 4.0	https://aivaslab.github.io/toxybox/
Mimetics	Open access	https://europe.naverlabs.com/research/computer-vision/mimetics/
UCF101	Open access	https://www.crcv.ucf.edu/data/UCF101.php
TinyVirat-v2	Open access	https://www.crcv.ucf.edu/tiny-actions-challenge-cvpr2021/#tabtwo
COIL100	Open access	https://www.cs.columbia.edu/CAVE/software/softlib/coil-100.php
STL-10	Open access	https://cs.stanford.edu/~acoates/stl10/
ActorShift	MIT	https://uvaauas.figshare.com/articles/dataset/ActorShift_zip/19387046
Sims4Action	MIT	https://github.com/aroitberg/sims4action
RareAct	Apache	https://github.com/antoine77340/RareAct
D Additional Experiments and Results
D.1 Pretrained on Kinetics700
In addition to the Kinetics400 used in the main paper, we further utilize the Kinetics700 to evaluate the benefits of incorporating more diverse data for out-of-distribution generalization. We use a subset of Kinetics700 comprising 480K samples, which doubles the size of Kinetics400. We use this subset to pretrain the VSSL methods for the same number of total iterations as Kinetics400. Unless stated otherwise, we compare the performance between Kinetics400 and Kinetics700 pretraining in linear evaluation, averaging over 3 trials. In Table S7 and Figure S8, we present a high-level overview showing the impact of using more diverse data in pretraining. Table S7 highlights that non-contrastive methods ( $v$ -BYOL,  $v$ -SimSiam,  $v$ -DINO) benefit more from the inclusion of diverse data compared to the contrastive approaches ( $v$ -SimCLR,  $v$ -MoCo) and  $v$ -MAE. Interestingly, we observe a slight decrease in the performance of  $v$ -MAE in this setting. This could be attributed to several factors, such as the pretrained encoder ViT-B reaching saturation in the pretraining setup of  $v$ -MAE, resulting in no improvements in performance from the additional data. To address this, a larger backbone like ViT-L or ViT-H, as used in prior works [57, 1], could be explored. Unfortunately, resource limitations prevented us from adopting a larger network to test this. Detailed results are presented in Tables S8, S9, S10, S11 and S12.
Table S7: Summary of the models' behavior with more diverse pretraining data is presented. In terms of overall performance, we report the average accuracy across all VSSL methods and distribution shifts, encompassing a total of 66 experiments. Additionally, we present results from 11 experiments for each individual VSSL method. The results demonstrate that non-contrastive methods (v-BYOL, v-SimSiam, v-DINO) benefit more from the inclusion of diverse data compared to the contrastive approaches (v-SimCLR, v-MoCo) and v-MAE. Moreover, we observe a slight decrease in the performance of v-MAE when pretrained with more diverse videos from Kinetics700.
Methods	Pretraining	InD	OoD
Overall	Kinetics400	63.0	31.6
Kinetics700	64.5 (↑ 1.5)	32.5 (↑ 0.9)
v-SimCLR	Kinetics400	66.0	32.9
Kinetics700	67.0 (↑ 1.0)	33.0 (↑ 0.1)
v-MoCo	Kinetics400	67.1	33.1
Kinetics700	67.7 (↑ 0.6)	33.9 (↑ 0.8)
v-BYOL	Kinetics400	65.2	33.1
Kinetics700	68.3 (↑ 3.1)	35.0 (↑ 1.9)
v-SimSiam	Kinetics400	62.0	31.5
Kinetics700	63.9 (↑ 1.9)	32.4 (↑ 0.9)
v-DINO	Kinetics400	63.1	31.5
Kinetics700	63.9 (↑ 0.8)	34.1 (↑ 2.6)
v-MAE	Kinetics400	54.4	27.4
Kinetics700	54.1 (↓ 0.3)	26.6 (↓ 0.8)
Table S8: Comparison of video models under context shift when pretrained with Kinetics400 (#240K) vs. Kinetics700 (#480K). We highlight the best result in each scenario in bold. Interestingly, we observe a consistent benefit from the inclusion of more diverse data in the InD validation sets of both splits. However, the improvements in OoD are not consistently observed.
Method	Context (10 class)	Context (50 class)
Kinetics400	Kinetics700	Kinetics400	Kinetics700
OoD	InD	OoD	InD	OoD	InD	OoD	InD
v-SimCLR	31.4	92.5	30.2	91.2	14.9	71.7	15.3	73.3
v-MoCo	29.7	92.2	27.7	93.1	15.0	74.0	14.1	75.0
v-BYOL	31.6	89.3	32.4	91.4	14.4	71.8	15.8	75.2
v-SimSiam	30.8	89.5	33.6	91.4	13.8	67.9	14.7	70.3
v-DINO	34.8	90.4	34.3	91.0	13.0	68.7	14.5	72.3
v-MAE	33.3	82.9	31.9	83.4	12.3	56.4	12.7	54.5
Table S9: Comparison of video models under viewpoint and actor shifts, when pretrained with Kinetics400 (#240K) vs. Kinetics700 (#480K). We highlight the best results in each scenario in bold. The results demonstrate that adding diverse pretraining data significantly improves the performance under egocentric viewpoint shifts and top-down synthetic domain shifts. However, less significant improvements are observed for surveillance viewpoint shifts and animal domain actor shifts. Moreover, the performance of  $v$ -MAE deteriorates when pretrained with Kinetics700.
Method	View (ego.)	View (surv.+low res)	View+Act (t-d+syn.)	Actor (animal)
K400	K700	K400	K700	K400	K700	K400	K700
OoD InD	OoD InD	OoD InD	OoD InD	OoD InD	OoD InD	OoD InD	OoD InD
v-SimCLR	12.7	13.7	26.1	39.1	24.9	40.2	42.4	67.8
v-MoCo	13.3	14.1	24.8	40.0	25.9	40.5	41.1	67.9
v-BYOL	12.0	13.9	22.7	37.8	22.2	41.7	37.3	65.6
v-SimSiam	11.6	12.8	23.3	34.3	21.4	37.3	40.0	65.5
v-DINO	12.0	13.4	22.3	35.3	22.9	38.0	35.3	62.9
v-MAE	10.9	13.7	23.5	32.0	23.1	31.2	37.8	58.0
Table S10: Comparison of video models under source shift, when pretrained with Kinetics400 (#240K) vs. Kinetics700 (#480K). We highlight the best results in each scenario in bold.  $\mathbf{v}$ -DINO and  $\mathbf{v}$ -SimSiam benefit the most from the inclusion of diverse pretraining data, while the other methods show the same or worse performance.
Method	UCF/HMDB	HMDB/UCF
Kinetics400	Kinetics700	Kinetics400	Kinetics700
OoD(H)	InD(U)	OoD(H)	InD(U)	OoD(U)	InD(H)	OoD(U)	InD(H)
v-SimCLR	47.7	96.0	45.1	96.5	55.1	82.0	55.1	83.6
v-MoCo	51.5	97.1	48.3	96.3	57.2	84.5	54.6	82.6
v-BYOL	51.4	94.5	50.5	96.8	63.1	79.9	61.1	82.8
v-SimSiam	46.1	92.6	48.0	92.5	52.2	76.2	53.1	80.5
v-DINO	49.3	94.3	50.3	95.8	53.8	77.5	58.2	81.8
v-MAE	39.5	89.2	39.5	89.3	39.1	72.8	37.5	72.4
Table S11: Comparison of video models in zero-shot recognition, when pretrained with Kinetics400 (#240K) vs. Kinetics700 (#480K). We highlight the best results in each scenario in bold. The results demonstrate that the inclusion of diverse data generally improves the zero-shot performance. We observe significant improvements in most cases, with the exception of  $v$ -MAE on UCF and  $v$ -BYOL on HMDB. Moreover, we also notice significant improvements in InD performance, particularly in the case of non-contrastive methods and  $v$ -MAE.
Method	UCF	HMDB	RareAct	Kinetics400 (InD)
K400	K700	K400	K700	K400	K700	K400	K700
v-SimCLR	37.2	39.0	18.6	19.0	7.7	9.4	56.8	57.8
v-MoCo	35.2	40.6	19.5	21.6	8.7	9.4	58.4	59.5
v-BYOL	33.0	44.0	22.4	21.1	7.5	9.9	57.4	61.0
v-SimSiam	34.0	39.2	18.8	20.3	7.7	8.3	50.4	52.3
v-DINO	34.3	41.1	17.2	20.0	8.1	9.5	53.4	58.3
v-MAE	25.5	24.1	14.2	17.2	5.8	6.2	35.9	35.4
Table S12: Comparison of video models in open-set recognition, when pretrained with Kinetics400 (#240K) vs. Kinetics700 (#480K). We highlight the best results in each scenario in bold. The results exhibit a very similar trend in both pretraining setups. For example, while  $v$ -SimCLR and  $v$ -MoCo perform fairly well in closed-set, they show near chance-level performance in open-set.  $v$ -MAE performs poorly in both open-set and closed-set, in both pretraining setups.  $v$ -DINO achieves the best open-set performance in both setups while retaining decent closed-set performance.
Method	Pretrained on Kinetics400	Pretrained on Kinetics700
Open-set (AUC) U/H	Closed-set w/ DEAR (Acc.) U101*	Closed-set w/ CE (Acc.) U101*	Open-set (AUC) U/H	Closed-set w/ DEAR (Acc.) U101*	Closed-set w/ CE (Acc.) U101*
v-SimCLR	49.9	11.6	84.1	49.7	18.5	83.6
v-MoCo	50.7	32.7	84.9	53.1	53.6	84.7
v-BYOL	56.0	63.7	85.4	74.5	85.5	86.7
v-SimSiam	73.7	79.5	82.5	69.6	80.5	82.3
v-DINO	79.4	80.4	80.9	82.0	84.5	85.0
v-MAE	66.0	74.6	76.2	65.6	74.2	75.1
Table S13: Comparison of VSSL methods in zero-shot recognition, when pretrained with Kinetics700 (#480K). We highlight the best results in each scenario in bold. In addition to Table S11, we also finetune video-text encoders with Kinetics700 and perform zero-shot recognition using the full UCF101 and HMDB51, along with RareAct. Please note that during both pretraining and finetuning, we discard the videos from Kinetics700 that belong to the overlapping classes from UCF101 and HMDB51.  $v$ -MAE clearly shows superiority when validated on UCF101 and HMDB51, while  $v$ -MoCo outperforms others when validated on RareAct.
Method	UCF	HMDB	RareAct
v-SimCLR	45.7	34.0	15.3
v-MoCo	48.4	33.9	18.4
v-BYOL	47.3	33.4	15.5
v-SimSiam	37.9	29.1	13.3
v-DINO	46.6	32.9	16.2
v-MAE	50.1	37.3	16.0
Figure S8: We present a high-level depiction of the models' performance when pretrained with Kinetics400 vs. Kinetics700. When pretrained with Kinetics700, the trend line of InD performance (blue dashed line) is slightly higher than that of the OoD performance (brown dashed line).
y=x	Context shift (10 cls)	Source shift (U/H)
y=mx+c (lnD)	Context shift (50 cls)	v-BYOL
y=mx+c (OoD)	Zero-shot (RareAct)	v-DINO
In-distribution	Actor+view shift (syn.+t-down)	v-MAE
Out-of-distribution	Zero-shot (UCF)	v-MOCO
Source shift (H/U)	View shift (surv.+low res.)	v-SIMCLR
Actor shift (animal)	View shift (ego.)	v-SIMSIAM
Zero-shot (HMDB)
D.2 Temporal dynamics
In this subsection, we provide the additional results of our experiment on the temporal dynamics of VSSL methods. We present the performance on temporal transformation recognition in Figure S9. Moreover, we present the performance of VSSL methods in classifying the same objects from static videos, in Figure S10.
Transformation recognition (Temporal)
(a) Cat
(e) Ball
(b) Duck
(c) Giraffe
(d) Horse
(f) Cup
(g) Mug
(h) Spoon
(i) Airplane
(j) Animals (4 objects)
Figure S9: Evaluating performance of video models on egocentric transformation recognition using ToyBox [70]. The models are used to classify a total of 9 types of temporal transformations including positive rotations, negative rotations, and transformations across the  $x$ ,  $y$ , and  $z$  axes. In all of the cases,  $\mathbf{v}$ -MAE consistently shows superior performance.
(j) Car
(k) Households (4 objects)
(k) Helicopter
(1) Vehicles (4 objects)
(1) Truck
(m) All (12 objects)
(a) Animals
(b) Households
Figure S10: Evaluating performance of video models in egocentric object recognition. The models are used to classify the videos of static objects including 4 types of animals, household items, cars, and all of them together (a total of 12 classes). In all cases  $v$ -MAE consistently exhibits the lowest performance. Among all the categories, the models perform significantly better in classifying household objects than animals or vehicles. This is likely because the pretraining data Kinetics400 consist of several human actions involving household objects e.g., kitchen utensils, etc.
(c) Vehicles
(d) All
D.3 Comparing InD vs. OoD performance under natural distribution shifts
To gain a high-level understanding of the models' performance in InD vs. OoD, we visualize their accuracy in a 2D space in Figure S11. It is noteworthy that superior InD performance does not necessarily translate to better OoD performance. We observe cases where models exhibit similar InD performance but significantly differ in their performance under distribution shifts. Ablation variants of Figure S11 are presented in Figure S12. Statistical analyses on the robustness of video models are presented in Appendix D.7.
Figure S11: We compare the InD vs. OoD performance of video models under different distribution shifts. The filled markers indicate the finetuned results, while the empty markers represent the results from linear evaluations. In all cases, we observe a decrease in performance in the OoD validation set, as indicated by the data points below the  $y = x$  line.
y=x	Φ	Context shift (50 cls)	v-BYOL
Linear (empty)	★	Zero-shot (RareAct)	v-DINO
Finetune (filled)	∇	Domain+view shift (syn.+t-down)	v-MAE
Source shift (H/U)	○	Zero-shot (UCF)	v-MOCO
Domain shift (animal)	▷	View shift (surv.+low res.)	v-SIMCLR
Zero-shot (HMDB)	◁	View shift (ego.)	v-SIMSIAM
Context shift (10 cls)	□	Source shift (U/H)	v-SUPERVISED
y=x	y=mx+c (FT)	v-MOCO
Linear	v-BYOL	v-SIMCLR
Finetune	v-DINO	v-SIMSIAM
y=mx+c (LT)	v-MAE	v-SUPERVISED
(a) Context shift (10 classes)
(b) Context shift (50 classes)
(c) Viewpoint shift (ego.)
(d) Viewpoint shift (surv.+low res)
(e) Viewpoint+actor shift (top-down+synthetic)
(f) Actor shift (animal)
(i) Zero-shot (UCF)
(g) Source shift  $(\mathbf{U} / \mathbf{H})$
(j) Zero-shot (HMDB)
Figure S12: An ablated view of InD vs. OoD performance under natural distribution shifts, utilizing both frozen (empty markers) and finetuned (filled markers) encoders. Here,  $y = mx + c$  shows a linear fit for the given data points (i.e., projected InD and OoD performance metrics). The data points above linear fit indicate more robustness. (a and b)  $v$ -Supervised demonstrates superior performance in linear evaluation, while  $v$ -MAE achieves the best results when finetuned. On the other hand, although  $v$ -MoCo and  $v$ -SimCLR show strong performance in InD validation, they exhibit weaker generalization in out-of-context scenarios. (c, d, and e) Overall,  $v$ -SimCLR and  $v$ -MoCo perform better in viewpoint shifts in both linear and finetuning.  $v$ -MAE (finetuned) shows a single instance of superior results for the specific case of low-resolution surveillance camera shift due to its robustness in low-resolution inputs. (f) In animal domain actor shift,  $v$ -BYOL achieves the best results in linear evaluation, whereas,  $v$ -SimCLR outperforms others when finetuned. (g and h)  $v$ -BYOL achieves the best performance under source shifts in all setups. (i and j) Frozen  $v$ -BYOL achieves the best zero-shot recognition on UCF but performs poorly on HMDB. On the other hand, the frozen  $v$ -SimCLR performs the best on HMDB, while generalizing poorly on UCF. (k) Overall, video models generalize poorly in zero-shot recognition of unusual actions (RareAct).
(h) Source shift  $(\mathbf{H} / \mathbf{U})$
(k) Zero-shot (RareAct)
y=x	y=mx+c (FT)	v-MOCO
Linear	v-BYOL	v-SIMCLR
◆ Finetune	v-DINO	v-SIMSIAM
y=mx+c (LT)	v-MAE	v-SUPERVISED
D.4 Effect of synthetic perturbations
In addition to the natural distribution shifts, we further extend our work investigating the performance of VSSL methods under synthetic perturbations. We follow the setup proposed in [53] to create the synthetic perturbation of varying severity on a scale of 1 to 5. We apply a total of 16 different synthetic perturbations belonging to the common spatio-temporal augmentation techniques such as noise addition, blurring, temporal perturbation, and camera motion. We test the robustness of the frozen encoders against synthetic perturbations on 2 benchmarks UCF101 and Kinetics400, presented in Figures S13 and S14 respectively. The results demonstrate that with increasing perturbation severity, the performance of video models deteriorates. However, it is important to acknowledge that certain visual augmentations, such as noise and blur, are already applied during the self-supervised pretraining phase. As a result, they may not be considered as distribution shifts within the scope of our setup. Furthermore, it is neither possible to train self-supervised methods without such augmentation techniques. Despite these considerations, we present these results for the sake of completeness.
Since spatial perturbations are extensively covered in literature, we refrain from explaining them here and redirect readers to [53, 54, 49]. Instead, we briefly discuss the temporal perturbations [53]. 'Sampling': using different frame rates than used in training; 'Reverse sampling': reversing the temporal order with different sampling rates; 'Jumble': shuffling frames in segments; 'Box jumble': shuffling the segments instead of frames; 'Freezing': randomly freezing video frames; 'Random shuffle': randomly shuffle the frame orders, please note random shuffle has just one severity level.
Figure S13: Spatio-temporal perturbations on UCF101 (linear evaluation). When subjected to extreme perturbations, the performance of VSSL methods experiences a significant decline, reaching near-chance levels. Notably,  $v$ -MAE exhibits the worst performance across all setups. Among all methods,  $v$ -BYOL demonstrates superiority in scenarios involving blur and temporal perturbations, while  $v$ -MoCo outperforms others in rotations and translations. Furthermore,  $v$ -DINO excels when tested on noisy videos.
Figure S14: Spatio-temporal perturbations on Kinetics400 (linear evaluation). When subjected to extreme perturbations, the performance of VSSL methods experiences a significant decline, reaching near-chance levels. Notably,  $v$ -MAE exhibits the worst performance across all setups. Among all methods,  $v$ -MoCo demonstrates superiority in scenarios involving blur, rotations, and temporal perturbations, while  $v$ -BYOL outperforming the others when tested on noisy videos.
D.5 Effect of finetuning in out-of-distribution generalization
D.5.1 Linear vs. finetuning performance comparison
We conduct a thorough study evaluating the impact of finetuning across different VSSL methods. We present a high-level overview in Table S14 and more detailed results in Figures S15, S16, and S17.
Table S14: A high-level summary of the impact of the models' performance when finetuned, both InD and OoD. Overall, finetuning improves performance in both InD and OoD. However, the performance varies based on VSSL methods. Generally, the improvements in InD performance tend to be more significant than those in OoD. We note that  $\mathbf{v}$ -MAE benefits the most from finetuning due to its extremely poor generalizability when using frozen encoders.
Methods	Eval. setup	InD	OoD
Overall	Linear	63.2	31.5
Finetune	71.9 (↑8.7)	37.4 (↑5.9)
v-Supervised	Linear	65.5	30.7
Finetune	69.0 (↑3.5)	35.1 (↑4.4)
v-SimCLR	Linear	66.0	32.9
Finetune	73.4 (↑7.4)	38.2 (↑5.9)
v-MoCo	Linear	67.1	33.1
Finetune	73.6 (↑6.5)	39.1 (↑6.0)
v-BYOL	Linear	65.2	33.1
Finetune	73.2 (↑8.0)	38.7 (↑4.6)
v-SimSiam	Linear	62.0	31.5
Finetune	70.4 (↑7.6)	36.2 (↑4.7)
v-DINO	Linear	63.1	31.5
Finetune	71.0 (↑7.9)	36.4 (↑4.9)
v-MAE	Linear	54.4	27.4
Finetune	72.6 (↑7.2)	38.3 (↑10.9)
Figure S15: A detailed comparison of the impact of finetuning on InD vs. OoD. For optimal viewing, please rotate 90 degrees to the left. The results demonstrate that the benefits of finetuning are highly dependent on the VSSL method and type of distribution shift. Our analysis reveals that finetuning tends to be more advantageous in scenarios involving actor shifts (animal domain and synthetic domain). Conversely, its benefits are relatively less pronounced under viewpoint shifts and zero-shot recognition. Moreover, the impact of finetuning in source shift and context shift is mixed. For example, while we notice significant benefits in UCF to HMDB shift, finetuning in fact worsens the performance in the case of HMDB to UCF shift. Additionally, finetuning is more beneficial when evaluated on a smaller benchmark with just 10 out-of-context classes. However, the improvements are relatively modest in a more challenging setup involving 50 out-of-context classes.
Figure S16: Comparing the performance of video models in linear evaluation vs. finetuning. In the plot, the filled markers represent InD results, while the empty markers represent the OoD results. Overall, we observe that finetuning consistently leads to improved performance, as indicated by the data points lying above the diagonal line ( $y = x$ ), with only one exception. It is worth noting that the average improvement in InD performance (shown by the blue dashed line) is higher compared to the improvement in OoD performance (shown by the brown dashed line).
y=x	v-BYOL	v-SIMCLR
y=mx+c	v-DINO	v-SIMSIAM
In-distribution (filled)	v-MAE	v-SUPERVISSED
Out-of-distribution (empty)	v-MOCO
y=x	v-BYOL	v-SIMCLR
y=m×c	v-DINO	v-SIMSIM
ln-distribution (filled)	v-MAE	v-SUPERVISED
Out-of-distribution (empty)	v-MOCO
y=x	v-BYOL	v-SIMCLR
y=mx+c	v-DINO	v-SIMIAM
ln-distribution (filled)	v-MAE	v-SUPERVISED
Out-of-distribution (empty)	v-MOCO