SlowGuess/NeurIPS_2023_OCR huggingface dataset:
y=x	v-BYOL	v-SIMCLR
y=mx+c	v-DINO	v-SIMIAM
ln-distribution (filed)	v-MAE	v-SUPERVISED
Out-of-distribution (empty)	v-MOCO
y=x	v-BYOL	v-SIMCLR
y=mx+c	v-DINO	v-SIMSIM
ln-distribution (filed)	v-MAE	v-SUPERVISED
Out-of-distribution (empty)	v-MOCO
y=x	v-BYOL	v-SIMCLR
y=m×c	v-DINO	v-SIMSIM
In-distribution (filed)	v-MAE	v-SUPERVISED
Out-of-distribution (empty)	v-MOCO
(d) Viewpoint shift (surv.+low res.)
(e) Viewpoint+actor shift (top-down+synthetic)
(f) Actor shift (animal)
(i) Zero-shot (UCF)
y=x	v-BYOL	v-SIMCLR
y=mx+c	v-DINO	v-SIMIAM
In-distribution (filled)	v-MAE	v-SUPERVISED
Out-of-distribution (empty)	v-MOOC
y=x	v-BYOL	v-SIMCLR
y=mx+c	v-DINO	v-SIMSIM
ln-distribution (filled)	v-MAE	v-SUPERVISED
a Out-of-distribution (empty)	v-MOCO
\( \underline{\underline{\mathbf{y}} = x} \)	v-BYOL	v-MOCO
\( \underline{\underline{\mathbf{y}} = mx + c} \)	v-DINO	v-SIMCLR
In-distribution (filled)	v-MAE	v-SIMSIAM
Out-of-distribution (empty)
(g) Source shift  $(\mathbf{U} / \mathbf{H})$
(j) Zero-shot (HMDB)
Figure S17: An ablated view of linear vs. finetuning performance comparison. Our results demonstrate that improved InD performance does not necessarily translate to better OoD performance. We observe cases where models exhibit similar InD performance but significantly differ in their performance under distribution shifts, e.g., please see the linear results in d, e, f, and i.
(h) Source shift (H/U)
(k) Zero-shot (RareAct)
y=x	v-BYOL	v-MOCO
y=mx+c	v-DINO	v-SIMCLRC
In-distribution (filled)	v-MAE	v-SIMSIAM
Out-of-distribution (empty)
y=x	v-BYOL	v-MOCO
y=m×c	v-DINO	v-SIMCLRC
■ In-distribution (filled)	v-MAE	v-SIMSIAM
■ Out-of-distribution (empty)
D.5.2 In-distribution overfitting due to finetuning
As mentioned in the main paper, while finetuning enhances overall performance, it is susceptible to in-distribution overfitting. In other words, while additional training leads to improved performance on InD validation, it adversely affects OoD performance. We present a few examples of such instances in Figure S18. In practice, we apply early stopping to tackle such overfitting.
$\pmb{v}$  -BYOL
$\pmb{v}$  -SimCLR
$\pmb{v}$  -DINO
Actor shift (synthetic)
$\pmb{v}$  -SimSiam
Actor shift (animal)
Figure S18: A few examples showing in-distribution overfitting caused during finetuning.
Actor shift (synthetic)
$\pmb{v}$  -BYOL
Zero-shot (OoD: HMDB, InD: Kinetics400)
Actor shift (synthetic)
$\pmb{v}$  -MoCo
Zero-shot (OoD: UCF, InD: Kinetics400)
D.6 High-level overview
The high-level overview presented in Figure 1 summarizes our findings from Tables 1, 2, 3, 4, and 6(a) of the main paper. To create the overview plot, we first normalize the OoD performance (e.g., accuracy, mAP) of each experiment to that of the best-performing method. Thus, the best performing method gets 1.0 and all other methods get a score in [0, 1]. Next, we group the experiments based on their super-categories and then take their average score. For example, in the case of source shift, we compute this average by grouping the normalized scores of 4 experiments, namely linear and finetuning evaluation of both UCF/HMDB and HMDB/UCF shifts. We follow similar steps for all the shifts (details are given below), resulting in a  $7 \times 6$  matrix, where 7 represents the number of video methods, and 6 represents the total number of shifts. This final matrix is then used to color-code and present a high-level summary of all methods across all shifts.
- Context shift: Table 1
- Context shift (10 classes) linear; best performing method:  $\mathbf{v}$ -Supervised
- Context shift (10 classes) finetune; best performing methods:  $v$ -Supervised,  $v$ -BYOL,  $v$ -MAE
- Context shift (50 classes) linear; best performing methods:  $v$ -Supervised,  $v$ -MoCo
- Context shift (50 classes) finetune; best performing method:  $\mathbf{v}$ -MAE
- Viewpoint shift: Table 2
- Viewpoint shift (egocentric) linear; best performing method:  $v$ -MoCo
- Viewpoint shift (egocentric) finetune; best performing method:  $v$ -MoCo
- Viewpoint shift (surveillance+low resolution) linear; best performing method:  $\mathbf{v}$ -SimCLR
- Viewpoint shift (surveillance+low resolution) finetune; best performing method:  $v$ -MAE
- Viewpoint+actor shift (top-down+synthetic) linear; best performing method:  $\mathbf{v}$ -SimCLR
- Viewpoint+actor shift (top-down+synthetic) finetune; best performing method:  $\mathbf{v}$ -SimCLR
- Actor shift: Table 2
- Viewpoint+actor shift (top-down+synthetic) linear; best performing method:  $v$ -SimCLR
- Viewpoint+actor shift (top-down+synthetic) finetune; best performing method:  $\mathbf{v}$ -SimCLR
- Actor shift (animal) linear; best performing method:  $v$ -BYOL
- Actor shift (animal) finetune; best performing method:  $v$ -SimCLR
- Source shift: Table 3
- Source shift (UCF/HMDB) linear; best performing method:  $v$ -MoCo
- Source shift (UCF/HMDB) finetune; best performing method:  $v$ -BYOL
- Source shift (HMDB/UCF) linear; best performing method:  $v$ -BYOL
- Source shift (HMDB/UCF) finetune; best performing method:  $v$ -BYOL
Zero-shot recognition: Table 4
- Zero-shot (UCF) linear; best performing method:  $v$ -SimCLR
- Zero-shot (UCF) finetune; best performing method:  $v$ -MoCo
- Zero-shot (HMDB) linear; best performing method:  $v$ -BYOL
- Zero-shot (HMDB) finetune; best performing method:  $\mathbf{v}$ -MAE
- Zero-shot (RareAct) linear; best performing method:  $v$ -MoCo
- Zero-shot (RareAct) finetune; best performing method:  $v$ -SimSiam
- Open-set recognition: Table 6(a)
- Open-set (K400/UCF) finetune; best performing method:  $v$ -SimCLR
- Open-set (K400/HMDB) finetune; best performing method:  $v$ -MoCo
- Open-set (U101/HHDB) finetune; best performing method:  $v$ -MoCo
D.7 Statistical analysis
We present detailed statistical analyses to verify the statistical significance of all the major findings/claims discussed in the main paper. In particular, we compare the robustness of the video models under different distribution shifts, after compensating their InD performance. The adjusted OoD performance difference between the two methods is calculated as:
$$
\text {A d j u s t e d} \Delta_ {\mathrm {O o D}} = \Delta_ {\mathrm {O o D}} - m \times \Delta_ {\mathrm {I n D}} - 1. 9 6 \times \mathrm {S E} _ {\mathrm {O o D}}
$$
Here,  $m \in [0,1]$  is the slope of a linear fit of projected InD and OoD performance measures of all the models for a particular shift, calculated as:
$$
m = \frac {\sum_ {i = 1} ^ {n} (x _ {i} - \bar {x}) (y _ {i} - \bar {y})}{\sum_ {i = 1} ^ {n} (x _ {i} - \bar {x}) ^ {2}},
$$
where  $x_{i}$  and  $y_{i}$  refer to InD and OoD performance measures for  $i^{\text{th}}$  method and  $i \in [1, n]$ , and  $n$  is the total number of methods studied. Moreover,  $\bar{x}$  and  $\bar{y}$  refer to the mean InD and OoD performances, respectively. Between Method 1 and Method 2, we calculate  $\Delta_{\mathrm{OoD}} = y_{1} - y_{2}$  and  $\Delta_{\mathrm{InD}} = x_{1} - x_{2}$ , while  $\mathrm{SE}_{\mathrm{OoD}}$  is the standard error of Method 1's OoD performance across several trials. We consider the standard error with statistical significance at 95% confidence. Finally, we accept Method 1 is more robust than Method 2, if Adjusted  $\Delta_{\mathrm{OoD}} > 0$ . The results of our statistical analyses are presented in Tables S15 through S26.
Table S15: Comparative statistical analysis of the robustness of  $v$ -MAE and  $v$ -Supervised under context shift (original numbers from Table 1).
Distribution shift	Method 1	Method 2	Adjusted ΔOOD
Lin.	FT.
Context (10 class)	v-Supervised	v-SimCLR	5.24	5.90
v-Supervised	v-MoCo	6.96	2.20
v-Supervised	v-BYOL	5.00	0.00
v-Supervised	v-SimSiam	5.73	0.80
v-Supervised	v-DINO	1.81	0.80
v-MAE	v-SimCLR	0.25	5.90
v-MAE	v-MoCo	1.97	2.20
v-MAE	v-BYOL	0.01	0.00
v-MAE	v-SimSiam	0.74	0.80
v-MAE	v-DINO	-3.17	0.80
Context (50 class)	v-Supervised	v-SimCLR	0.44	2.68
v-Supervised	v-MoCo	0.66	1.28
v-Supervised	v-BYOL	0.92	-0.07
v-Supervised	v-SimSiam	0.99	0.08
v-Supervised	v-DINO	1.88	0.53
v-MAE	v-SimCLR	-0.82	7.20
v-MAE	v-MoCo	-0.59	5.80
v-MAE	v-BYOL	-0.34	4.45
v-MAE	v-SimSiam	-0.27	4.59
v-MAE	v-DINO	0.63	5.04
Table S16: Comparative statistical analysis of the robustness of  $v$ -MAE and  $v$ -Supervised in learning temporal dynamics (original numbers from Figure 3(a)).
Distribution shift	Method 1	Method 2	Adjusted ΔOOD
Temporal learner	v-Supervised	v-BYOL	0.06
v-Supervised	v-SimSiam	0.05
v-Supervised	v-DINO	0.05
v-Supervised	v-SimCLR	0.06
v-Supervised	v-MoCo	0.06
v-MAE	v-BYOL	0.19
v-MAE	v-SimSiam	0.19
v-MAE	v-DINO	0.19
v-MAE	v-SimCLR	0.20
v-MAE	v-MoCo	0.20
Table S17: Comparative statistical analysis of the robustness of contrastive methods ( $v$ -SimCLR and  $v$ -MoCo) under viewpoint shift (original numbers from Table 2).
Distribution shift	Method 1	Method 2	Adjusted ΔOOD
Lin.	FT.
Viewpoint (ego.)	v-SimCLR	v-BYOL	0.21	0.20
v-SimCLR	v-SimSiam	0.35	1.28
v-SimCLR	v-DINO	0.24	1.16
v-SimCLR	v-Supervised	0.71	1.44
v-SimCLR	v-MAE	0.65	2.01
v-MoCo	v-BYOL	0.56	0.69
v-MoCo	v-SimSiam	0.71	1.78
v-MoCo	v-DINO	0.60	1.66
v-MoCo	v-Supervised	1.06	1.94
v-MoCo	v-MAE	1.00	2.51
Viewpoint (surv.+low res)	v-SimCLR	v-BYOL	2.50	2.77
v-SimCLR	v-SimSiam	0.98	0.51
v-SimCLR	v-DINO	2.22	1.84
v-SimCLR	v-Supervised	0.89	0.10
v-SimCLR	v-MAE	0.21	-0.16
v-MoCo	v-BYOL	0.49	1.44
v-MoCo	v-SimSiam	-1.03	-0.81
v-MoCo	v-DINO	0.22	0.52
v-MoCo	v-Supervised	-1.12	-1.22
v-MoCo	v-MAE	-1.79	-1.48
View+Act (t-down+syn.)	v-SimCLR	v-BYOL	1.87	6.30
v-SimCLR	v-SimSiam	-0.88	6.70
v-SimCLR	v-DINO	2.06	11.5
v-SimCLR	v-Supervised	8.45	16.0
v-SimCLR	v-MAE	-3.93	-0.60
v-MoCo	v-BYOL	1.81	4.80
v-MoCo	v-SimSiam	-0.94	5.20
v-MoCo	v-DINO	2.00	10.0
v-MoCo	v-Supervised	8.39	14.5
v-MoCo	v-MAE	-3.99	-2.10
Table S18: Comparative statistical analysis of the robustness of contrastive methods ( $v$ -SimCLR and  $v$ -MoCo) in learning viewpoint invariance (original numbers from Figure 4(a)).
Distribution shift	Method 1	Method 2	Adjusted ΔOOD
Viewpoint invariance	v-SimCLR	v-BYOL	0.05
v-SimCLR	v-SimSiam	0.04
v-SimCLR	v-DINO	0.02
v-SimCLR	v-Supervised	0.09
v-SimCLR	v-MAE	0.09
v-MoCo	v-BYOL	0.04
v-MoCo	v-SimSiam	0.03
v-MoCo	v-DINO	0.02
v-MoCo	v-Supervised	0.09
v-MoCo	v-MAE	0.09
Table S19: Comparative statistical analysis of the robustness of single stream networks ( $v$ -Supervised and  $v$ -MAE) in learning temporal dynamics and Siamese networks ( $v$ -BYOL,  $v$ -SimSiam,  $v$ -DINO,  $v$ -SimCLR, and  $v$ -MoCo) in learning viewpoint invariance (original numbers from Figures 3(a) and 4(a)).
Distribution shift	Method 1	Method 2	Adjusted ΔOOD
Temporal learner	v-Supervised	v-BYOL	0.06
v-Supervised	v-SimSiam	0.05
v-Supervised	v-DINO	0.05
v-Supervised	v-SimCLR	0.06
v-Supervised	v-MoCo	0.06
v-MAE	v-BYOL	0.19
v-MAE	v-SimSiam	0.19
v-MAE	v-DINO	0.19
v-MAE	v-SimCLR	0.20
v-MAE	v-MoCo	0.20
Viewpoint invariance	v-SimCLR	v-Supervised	0.09
v-SimCLR	v-MAE	0.09
v-MoCo	v-Supervised	0.09
v-MoCo	v-MAE	0.09
v-BYOL	v-Supervised	0.05
v-BYOL	v-MAE	0.05
v-SimSiam	v-Supervised	0.06
v-SimSiam	v-MAE	0.06
v-DINO	v-Supervised	0.07
v-DINO	v-MAE	0.07
Table S20: Comparative statistical analysis of the vulnerability (opposite of robustness) of  $v$ -Supervised under multiple shifts (original numbers from Table 2).
Distribution shift	Method 1	Method 2	Adjusted ΔOOD
Lin.	FT.
View+Act (t-down+syn.)	v-Supervised	v-BYOL	-7.11	-9.70
v-Supervised	v-SimSiam	-9.87	-9.30
v-Supervised	v-DINO	-6.92	-4.50
v-Supervised	v-SimCLR	-10.6	-16.0
v-Supervised	v-MoCo	-9.26	-14.5
v-Supervised	v-MAE	-12.9	-16.6
Table S21: Comparative statistical analysis of the robustness of  $v$ -BYOL under source shift (original numbers from Table 3).
Distribution shift	Method 1	Method 2	Adjusted ΔOOD
Lin.	FT.
HMDB to UCF	v-BYOL	v-Supervised	8.56	6.60
v-BYOL	v-SimSiam	6.96	2.90
v-BYOL	v-DINO	6.70	0.30
v-BYOL	v-SimCLR	9.96	2.00
v-BYOL	v-MoCo	10.3	1.10
v-BYOL	v-MAE	16.7	10.8
UCF to HMDB	v-BYOL	v-Supervised	3.14	2.40
v-BYOL	v-SimSiam	2.72	6.80
v-BYOL	v-DINO	1.23	6.80
v-BYOL	v-SimCLR	4.52	5.90
v-BYOL	v-MoCo	1.89	5.10
v-BYOL	v-MAE	6.00	3.10
Table S22: Comparative statistical analysis of the impacts of finetuning over frozen encoder under source shift (original numbers from Table 3).
Distribution shift	Method 1	Method 2	Adjusted ΔOOD
HMDB to UCF	Finetune	Linear	-8.67
UCF to HMDB	Finetune	Linear	1.97
Table S23: Comparative statistical analysis of the robustness of contrastive methods ( $v$ -SimCLR and  $v$ -MoCo) in open-set recognition when finetuned (original numbers from Table 6(a)).
Distribution shift	Method 1	Method 2	Adjusted ΔOOD
Kinetics400/UCF	v-SimCLR	v-BYOL	2.44
v-SimCLR	v-SimSiam	2.42
v-SimCLR	v-DINO	2.94
v-SimCLR	v-MAE	7.85
v-MoCo	v-BYOL	0.72
v-MoCo	v-SimSiam	0.70
v-MoCo	v-DINO	1.22
v-MoCo	v-MAE	6.13
Kinetics400/HMDB	v-SimCLR	v-BYOL	0.04
v-SimCLR	v-SimSiam	2.12
v-SimCLR	v-DINO	1.03
v-SimCLR	v-MAE	5.09
v-MoCo	v-BYOL	0.76
v-MoCo	v-SimSiam	2.84
v-MoCo	v-DINO	1.75
v-MoCo	v-MAE	5.82
UCF101/HMDB	v-SimCLR	v-BYOL	2.58
v-SimCLR	v-SimSiam	4.29
v-SimCLR	v-DINO	1.31
v-SimCLR	v-MAE	11.38
v-MoCo	v-BYOL	3.20
v-MoCo	v-SimSiam	4.91
v-MoCo	v-DINO	1.94
v-MoCo	v-MAE	12.0
Table S24: Comparative statistical analysis of the robustness of slightly weak frozen encoders ( $v$ -DINO,  $v$ -SimSiam, and  $v$ -Supervised) in open-set recognition in linear evaluation (the original numbers from Table 6(b)).
Distribution shift	Method 1	Method 2	Adjusted ΔOOD
UCF101/HMDB	v-DINO	v-BYOL	23.4
v-DINO	v-SimCLR	29.5
v-DINO	v-MoCo	28.7
v-DINO	v-MAE	13.4
v-SimSiam	v-BYOL	17.5
v-SimSiam	v-SimCLR	23.5
v-SimSiam	v-MoCo	22.7
v-SimSiam	v-MAE	7.40
v-Supervised	v-BYOL	21.2
v-Supervised	v-SimCLR	27.3
v-Supervised	v-MoCo	26.5
v-Supervised	v-MAE	11.2
Table S25: Part 1. Comparative statistical analysis of the robustness of all methods in zero-shot recognition (original numbers from Table 4).
Distribution shift	Method 1	Method 2	Adjusted ΔOOD
Lin.	FT.
Zero-shot on UCF	v-BYOL	v-SimSiam	-5.02	-0.24
v-BYOL	v-DINO	-4.01	-2.13
v-BYOL	v-MAE	-2.89	-1.56
v-BYOL	v-SimCLR	-5.37	0.95
v-BYOL	v-MoCo	-2.70	-5.30
v-SimSiam	v-BYOL	2.31	0.24
v-SimSiam	v-DINO	-0.42	-1.89
v-SimSiam	v-MAE	0.70	-1.32
v-SimSiam	v-SimCLR	-1.79	1.19
v-SimSiam	v-MoCo	0.89	-5.06
v-DINO	v-BYOL	1.78	2.13
v-DINO	v-SimSiam	-1.96	1.89
v-DINO	v-MAE	0.17	0.57
v-DINO	v-SimCLR	-2.31	3.08
v-DINO	v-MoCo	0.36	-3.17
v-MAE	v-BYOL	0.28	1.56
v-MAE	v-SimSiam	-3.46	1.32
v-MAE	v-DINO	-2.45	-0.57
v-MAE	v-SimCLR	-3.81	2.51
v-MAE	v-MoCo	-1.14	-3.74
v-SimCLR	v-BYOL	2.40	-0.95
v-SimCLR	v-SimSiam	-1.34	-1.19
v-SimCLR	v-DINO	-0.32	-3.08
v-SimCLR	v-MAE	0.79	-2.51
v-SimCLR	v-MoCo	0.99	-6.25
v-MoCo	v-BYOL	-1.40	5.30
v-MoCo	v-SimSiam	-5.14	5.06
v-MoCo	v-DINO	-4.12	3.17
v-MoCo	v-MAE	-3.01	3.74
v-MoCo	v-SimCLR	-5.49	6.25
Table S25 Part 2. Comparative statistical analysis of the robustness of all methods in zero-shot recognition (original numbers from Table 4).
Distribution shift	Method 1	Method 2	Adjusted ΔOOD
Lin.	FT.
Zero-shot on HMDB	v-BYOL	v-SimSiam	-0.18	1.05
v-BYOL	v-DINO	2.17	-0.01
v-BYOL	v-MAE	0.55	-1.58
v-BYOL	v-SimCLR	1.64	0.06
v-BYOL	v-MoCo	1.27	2.40
v-SimSiam	v-BYOL	-2.61	-1.05
v-SimSiam	v-DINO	1.48	-1.06
v-SimSiam	v-MAE	-0.14	-2.63
v-SimSiam	v-SimCLR	0.94	-0.99
v-SimSiam	v-MoCo	0.58	1.35
v-DINO	v-BYOL	-4.35	0.01
v-DINO	v-SimSiam	-2.61	1.06
v-DINO	v-MAE	-1.88	-1.57
v-DINO	v-SimCLR	-0.80	0.07
v-DINO	v-MoCo	-1.16	2.41
v-MAE	v-BYOL	-3.15	1.58
v-MAE	v-SimSiam	-1.42	2.63
v-MAE	v-DINO	0.94	1.57
v-MAE	v-SimCLR	0.40	1.64
v-MAE	v-MoCo	0.04	3.98
v-SimCLR	v-BYOL	-5.14	-0.06
v-SimCLR	v-SimSiam	-3.40	0.99
v-SimCLR	v-DINO	-1.05	-0.07
v-SimCLR	v-MAE	-2.67	-1.64
v-SimCLR	v-MoCo	-1.95	2.34
v-MoCo	v-BYOL	-5.55	-2.40
v-MoCo	v-SimSiam	-3.81	-1.35
v-MoCo	v-DINO	-1.46	-2.41
v-MoCo	v-MAE	-3.08	-3.98
v-MoCo	v-SimCLR	-1.99	-2.34
Table S25 Part 3. Comparative statistical analysis of the robustness of all methods in zero-shot recognition (original numbers from Table 4).
Distribution shift	Method 1	Method 2	Adjusted ΔOOD
Lin.	FT.
Zero-shot on RareAct	v-BYOL	v-SimSiam	-1.19	-0.95
v-BYOL	v-DINO	-1.36	-0.31
v-BYOL	v-MAE	-0.82	-0.43
v-BYOL	v-SimCLR	-0.59	-0.09
v-BYOL	v-MoCo	-1.38	-0.20
v-SimSiam	v-BYOL	0.59	0.95
v-SimSiam	v-DINO	-0.43	0.64
v-SimSiam	v-MAE	0.11	0.52
v-SimSiam	v-SimCLR	0.33	0.86
v-SimSiam	v-MoCo	-0.45	0.75
v-DINO	v-BYOL	0.79	0.31
v-DINO	v-SimSiam	-0.08	-0.64
v-DINO	v-MAE	0.30	-0.12
v-DINO	v-SimCLR	0.53	0.22
v-DINO	v-MoCo	-0.26	0.11
v-MAE	v-BYOL	-0.39	0.43
v-MAE	v-SimSiam	-1.25	-0.52
v-MAE	v-DINO	-1.41	0.12
v-MAE	v-SimCLR	-0.65	0.34
v-MAE	v-MoCo	-1.43	0.23
v-SimCLR	v-BYOL	-0.26	0.09
v-SimCLR	v-SimSiam	-1.12	-0.86
v-SimCLR	v-DINO	-1.28	-0.22
v-SimCLR	v-MAE	-0.75	-0.34
v-SimCLR	v-MoCo	-1.3	-0.11
v-MoCo	v-BYOL	0.50	0.20
v-MoCo	v-SimSiam	-0.36	-0.75
v-MoCo	v-DINO	-0.52	-0.11
v-MoCo	v-MAE	0.01	-0.23
v-MoCo	v-SimCLR	0.24	0.11
Table S26: Comparative statistical analysis of the robustness of all methods under animal domain actor shift (original numbers from Table 2).
Distribution shift	Method 1	Method 2	Adjusted ΔOOD
Lin.	FT.
Animal domain actor shift	v-Supervised	v-SimCLR	-1.86	-3.00
v-Supervised	v-MoCo	-1.50	-1.20
v-Supervised	v-BYOL	-2.46	-1.20
v-Supervised	v-SimSiam	-3.24	-2.00
v-Supervised	v-DINO	-1.70	-1.90
v-Supervised	v-MAE	0.39	-2.40
v-SimCLR	v-Supervised	0.74	3.00
v-SimCLR	v-MoCo	-0.2	1.80
v-SimCLR	v-BYOL	-1.16	1.80
v-SimCLR	v-SimSiam	-1.94	1.00
v-SimCLR	v-DINO	-0.40	1.10
v-SimCLR	v-MAE	1.69	0.60
v-MoCo	v-Supervised	0.09	1.20
v-MoCo	v-SimCLR	-1.21	-1.80
v-MoCo	v-BYOL	-1.82	0.00
v-MoCo	v-SimSiam	-2.59	-0.80
v-MoCo	v-DINO	-1.05	-0.70
v-MoCo	v-MAE	1.04	-1.20
v-BYOL	v-Supervised	1.58	1.20
v-BYOL	v-SimCLR	0.28	-1.80
v-BYOL	v-MoCo	0.64	0.00
v-BYOL	v-SimSiam	-1.10	-0.80
v-BYOL	v-DINO	0.45	-0.70
v-BYOL	v-MAE	2.54	-1.20
v-SimSiam	v-Supervised	1.27	2.00
v-SimSiam	v-SimCLR	-0.03	-1.00
v-SimSiam	v-MoCo	0.33	0.80
v-SimSiam	v-BYOL	-0.63	0.80
v-SimSiam	v-DINO	0.13	0.10
v-SimSiam	v-MAE	2.22	-0.40
v-DINO	v-Supervised	0.67	1.90
v-DINO	v-SimCLR	-0.63	-1.10
v-DINO	v-MoCo	-0.27	0.70
v-DINO	v-BYOL	-1.23	0.70
v-DINO	v-SimSiam	-2.01	-0.10
v-DINO	v-MAE	1.63	-0.50
v-MAE	v-Supervised	-1.60	2.40
v-MAE	v-SimCLR	-2.90	-0.60
v-MAE	v-MoCo	-2.55	1.20
v-MAE	v-BYOL	-3.51	1.20
v-MAE	v-SimSiam	-4.28	0.40
v-MAE	v-DINO	-2.74	0.50
D.8 Exploring performance variation of VSSL methods across action classes
To provide a deeper understanding, we conduct a comprehensive analysis examining the impact of distribution shifts on the models' prediction across various action classes. Our investigation focuses on evaluating the performance of the finetuned models across different categories in both the InD and OoD. The findings are presented as follows:
Figure S19 Context shift (10 classes)
Figure S20 Context shift (50 classes)
Figure S21 Viewpoint shift (egocentric)
Figure S22 Viewpoint shift (surveillance camera+low resolution)
Figure S23 Viewpoint + actor shift (top-down + synthetic)
Figure S24 Actor shift (animal)
Figure S25 Source shift (UCF to HMDB)
Figure S26 Source shift (HMDB to UCF)
Figure S28 Zero-shot (UCF)
Figure S27 Zero-shot (HMDB)
Figure S29 Zero-shot (RareAct)
Figure S30 Open-set (UCF/HMDB)
Figure S32 Open-set (Kinetics400/UCF)
Figure S31 Open-set (Kinetics400/HMDB)
Figure S19: Context shift (10 classes): In-distribution vs. out-of-distribution performance comparison per action class. For optimal viewing, please zoom in. All of the contrastive and non-contrastive methods fail to identify 'canoeing' in OoD, whereas,  $v$ -Supervised and  $v$ -MAE make a few correct predictions. In another extreme example,  $v$ -MAE makes a few correct predictions for 'surfing water', while other methods completely fail. In particular, VSSL methods demonstrate relatively good performance in identifying certain action classes in out-of-context scenarios, such as 'golf driving' and 'playing volleyball'.
Figure S20: Context shift (50 classes): In-distribution vs. out-of-distribution performance comparison per action class. For optimal viewing, please zoom in. Some of the top-performing action classes under context shift include 'brushing teeth', 'catching or throwing frisbee', 'juggling balls', 'juggling soccer ball', and 'skipping rope'. It is noteworthy that all of these action classes involve high temporal motion and cyclic patterns. This observation leads us to speculate that the presence of these attributes may assist video models in recognizing actions in out-of-context settings.
Figure S21: **Viewpoint shift (egocentric):** In-distribution vs. out-of-distribution performance comparison per action class. For optimal viewing, please zoom in and rotate 90 degrees to the left. A few examples of poor OoD generalization include action classes such as: 'lying on a bed', 'fixing a vacuum', and 'someone is closing something', among a few others. We also note a few instances, where models show very similar InD and OoD performance, such as 'holding some food', 'holding a boom', and 'taking food from somewhere', among a few others.
Figure S22: Viewpoint shift (surveillance view + low resolution): In-distribution vs. out-of-distribution performance comparison per action class. For optimal viewing, please zoom in. Interestingly, both  $v$ -BYOL and  $v$ -SimSiam perform well in identifying 'running' but failed to identify 'riding'. On the other hand,  $v$ -SimCLR and  $v$ -MoCo correctly identify 'riding' but fail to predict 'running'. Overall, models show poor performance in distinguishing actions with subtle differences like 'closing' vs. 'opening', 'entering' vs. 'exiting', and 'pull' vs. 'push'.
Figure S23: **Viewpoint + actor shift (top-down + synthetic): In-distribution vs. out-of-distribution performance comparison per action class. For optimal viewing, please zoom in. The models exhibit particularly poor performance in identifying the action classes 'eat', followed by 'drink'. Conversely, they demonstrate relatively better performance in identifying actions such as 'read book', 'use phone', and 'walk'.
Figure S24: Actor shift (animal): In-distribution vs. out-of-distribution performance comparison per action class. For optimal viewing, please zoom in. The models exhibit particularly poor performance in identifying action classes 'drinking' and 'running'. Conversely, they perform well in 'predicting eating', 'opening door', 'swimming', and 'watching tv'.
Figure S25: Source shift (UCF to HMDB): In-distribution vs. out-of-distribution performance comparison per action class. For optimal viewing, please zoom in. The results demonstrate the challenges faced by the video models in generalizing across similar action classes from different datasets. For instance, the models trained on UCF videos which include 'soccer penalty', struggle to generalize to the 'kick ball' class in HMDB. Similarly, the 'walking with dog' class in UCF does not aid in generalizing to 'walk' in HMDB.
Figure S26: Source shift (HMDB to UCF): In-distribution vs. out-of-distribution performance comparison per action class. For optimal viewing, please zoom in. We notice large variability in performances across different methods. For example,  $v$ -Supervised performs very poorly in identifying the action 'dive', whereas, all the self-supervised methods perform reasonably well. Furthermore, while  $v$ -BYOL and  $v$ -DINO demonstrate strong performance in identifying the action 'walk', the other methods struggle to achieve a similar level of accuracy. Overall, the VSSL methods tend to exhibit poor generalization capabilities for actions such as 'jump', 'throw', and 'dribble', among others.
Figure S27: Zero-shot recognition (HMDB): In-distribution vs. out-of-distribution performance comparison per action class. For optimal viewing, please zoom in. The video models overall generalize well on action classes such as 'chew', 'handstand', 'kick', and 'smile', whereas, they show poor generalizability on other action classes such as 'fencing', 'flic flac', 'pick', 'run', and 'turn' among others. Moreover, their performance largely varies across different actions.
Figure S28: Zero-shot recognition (UCF): In-distribution vs. out-of-distribution performance comparison per action class. For optimal viewing, please zoom in. The video models overall generalize well on action classes such as 'apply lipstick', 'rafting', 'ice dancing', and 'sumo wrestling', whereas, they show poor generalizability on action classes such as 'fencing', 'front crawl', 'hammering', 'mixing', and 'rowing' among others. Moreover, their performance largely varies across different actions.