SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$\pmb{v}$  -SimCLR. The objective of  $\pmb{v}$  -SimCLR is to maximize the similarity between positive pairs and minimize the similarity between negative pairs using the InfoNCE loss function. Given two augmented views  $v_{1}$  and  $v_{2}$ , we obtain  $z_{1}$  and  $z_{2}$  as  $f(v_{1})$  and  $f(v_{2})$ . The L2 normalized cosine similarity scores, referred to as logits, are then calculated as  $\mathrm{sim}(z_1,z_2)$ , where  $\mathrm{sim}(z_1,z_2) = \frac{z_1^\top z_2}{||z_1||||z_2||}$ . The pretraining loss is calculated as  $\mathcal{L}_{simcr} = \mathrm{InfoNCE}(\logitss)$  as per Equation S1, where  $k+$  refers to the positive embeddings (i.e., embeddings of the augmented views) and  $k-$  refers to the negative embeddings (embeddings of other samples in a minibatch).
$$
\operatorname {I n f o N C E} (\log_ {t}) = - \log \frac {\sum_ {k \in \{k ^ {+} \}} \exp (\log_ {t} / \tau)}{\sum_ {k \in \{k ^ {+} , k ^ {-} \}} \exp (\log_ {t} / \tau)}, \text {w h e r e} \tau \text {i s t e m p e r a t u r e .} \tag {S1}
$$
$\pmb{v}$  -MoCo. Similar to  $\pmb{v}$  -SimCLR,  $\pmb{v}$  -MoCo also calculates InfoNCE loss for given query-key pairs. We obtain the query embeddings as  $p_1 = f_p(f(\pmb{v}_1))$  and its corresponding key embeddings are obtained as  $zz_{2} = \mathrm{sg}(f_{t}(\pmb{v}_{2}))$ , where  $\mathrm{sg}$  refers to stop-gradient operation. In Equation S1, the positive and negative key embeddings  $\{k+, k- \} \in zz_{2}$  and the query embedding  $q \in p_1$ . To calculate a symmetrized loss we also obtain  $p_2$  and  $zz_{1}$ . The training objective  $\mathcal{L}_{moco}$  is defined as InfoNCE  $(\mathrm{sim}(p_1,zz_2)) + \mathrm{InfoNCE}(\mathrm{sim}(p_2,zz_1))$ . The encoder  $f$  is updated using  $\mathcal{L}_{moco}$ , and the target encoder  $f_{t}$  is updated using the exponential moving average (EMA) [56] as  $f_{t} \gets \alpha f + (1 - \alpha)f$ , where  $\alpha$  is the EMA coefficient.
$\pmb{v}$ -BYOL. Different from  $\pmb{v}$ -SimCLR and  $\pmb{v}$ -MoCo,  $\pmb{v}$ -BYOL does not require any negative pairs and only relies on positive views. Therefore, we obtain  $p_1$  and  $zz_2$  as  $f_p(f(\pmb{v}_1))$  and  $\mathrm{sg}(f_t(\pmb{v}_2))$  to calculate the similarity scores as  $\mathrm{sim}(p_1,zz_2)$ . Moreover, to obtain a symmetrized loss, we also calculate  $\mathrm{sim}(p_2,zz_1)$  and define total loss  $\mathcal{L}_{byol} = -\mathrm{sim}(p_1,zz_2) - \mathrm{sim}(p_2,zz_1)$ . Please note, we use  $\mathcal{L}_{byol}$  to train the  $f$ , and the weights of  $f_t$  are updated using an EMA similar to  $\pmb{v}$ -MoCo.
$\pmb{v}$  -SimSiam. The approach for  $\pmb{v}$  -SimSiam is very similar to the  $\pmb{v}$  -BYOL, except it does not use  $f_{t}$  and directly uses the representations of the  $f$  as the target. We obtain  $p_1$  and  $z_{2}$  as  $f_{p}(f(\pmb{v}_{1}))$  and  $\mathrm{sg}(f(\pmb{v}_2))$  to calculate the similarity scores as  $\mathrm{sim}(p_1,z_2)$ . Similarly, we also obtain  $\mathrm{sim}(p_2,z_1)$  and define total loss  $\mathcal{L}_{\text{simpSiam}} = -\mathrm{sim}(p_1,z_2) - \mathrm{sim}(p_2,z_1)$ .
$\pmb{v}$  -DINO. Similar to  $\pmb{v}$  -BYOL and  $\pmb{v}$  -MoCo,  $\pmb{v}$  -DINO uses a target encoder  $f_{t}(\cdot)$  but does not use a predictor head. We obtain  $z_{1}$  and  $zz_{2}$  as  $f(\pmb{v}_{1})$  and  $\mathrm{sg}(f_t(\pmb{v}_2) - \mathbf{C})$  respectively, where  $\mathbf{C}$  refers to the Centering operation which can be interpreted as adding a bias term, preventing the model from collapsing [26]. Next, we calculate the cross-entropy (CE) error between  $z_{1}$  and  $zz_{2}$  as  $\mathrm{CE}(z_1,zz_2) = -\mathrm{softmax}(zz_2 / \tau_t)\log (\mathrm{softmax}(z_1 / \tau_s))$ , where  $\tau_{t}$  and  $\tau_{s}$  denote the temperatures used to sharpen the representations of the target and base encoder respectively. Finally, we calculate loss  $\mathcal{L}_{dino} = \mathrm{CE}(z_1 + zz_2) + \mathrm{CE}(z_2,zz_1)$  which is used to update  $f$  and the weights of  $f_{t}$  are updated using EMA similar to  $\pmb{v}$  -BYOL and  $\pmb{v}$  -MoCo.
$\pmb{v}$  MAE. Unlike the methods mentioned above,  $\pmb{v}$  MAE takes a single view and performs masking and reconstruction using an autoencoder architecture [27, 1, 57]. Moreover, different from the other models, the encoder does not contain a projection head, i.e.,  $f = \theta$  , instead uses a decoder  $f_{d}$  for reconstruction. For a given sample  $\pmb{v}$  , we obtain masked tokens  $\pmb{v}^{[m]}$  as  $\{\pmb {v}^i |m^i = 1\}_{i = 1}^P$  and the corrupted inputs as  $\hat{\pmb{v}} = \{\pmb {v}|m^{i} = 0\}_{i = 1}^{P}$  . We calculate the reconstruction loss as  $\mathcal{L}_{mae} = (f_d(\theta (\hat{v})) - \pmb{v}^{[m]})^2$  , which is used to train the  $\theta$  and  $f_{d}$
After pretraining using the aforementioned methods, we discard the auxiliary network components and utilize only  $\theta$  for downstream evaluations.
Datasets. We use 2 popular large-scale datasets Kinetics400 [61] and Kinetics700 [62] for pretraining. Kinetics400 consists of 240K samples including 400 action classes, whereas, Kinetics700 consists of 537K training samples spread over 700 action classes. Following [97, 42], we discard the overlapping classes between Kintics700 and UCF101 [66], HMDB51 [67] from pretraining, which results in approximately 480K samples spread over 663 action classes. We follow such a setup so that the pretrained encoder can be used for zero-shot recognition on full UCF101 and HMDB51.
Pretraining. To ensure a fair comparison between the VSSL methods, we pretrain them in identical setups with necessary adjustments in hyperparameters. Specifically, we keep the encoder, inputs, batch size, and maximum number of pretraining epochs identical for all the methods. However, we adjust the data augmentation strategy and auxiliary network components (e.g., predictor head, projector head, decoder) as required to achieve their best performance on 2 validation sets Kinetics400 and UCF101. For example, the projector head of  $v$ -DINO is based on the configuration proposed in [26], whereas, the projector head for the other Siamese frameworks is made of an MLP head similar to [24, 25]. The details of the frameworks are presented in Table S1. Considering the widespread use of Transformers in vision, VSSL methods are implemented using ViT-B [75] as the encoder with a patch size of  $4 \times 16^2$ . Videos are downsampled to 8 FPS and resized to  $3 \times 112^2$  frames. Except for  $v$ -MAE, all VSSL methods are fed with 16 frames in each view and 32 frames are fed as input to  $v$ -MAE. We use AdamW [98, 99] optimizer with cosine learning rate scheduler and train all methods up to 800 epochs with a batch size of 768. We present the hyperparameters in Table S3. All the methods are pretrained using 8 V100 32 GB GPUs in parallel. We present the computational specifics
in Table S2, which shows that  $v$ -MAE is the most computationally efficient framework amongst all the VSSL methods.
Table S1: Architecture details of the VSSL frameworks.
v-SimCLR	v-MoCo	v-BYOL	v-SimSiam	v-DINO	v-MAE
Encoder	ViT-B
Target Encoder	X	✓	✓	X	✓	X
Predictor (# layers)	X	1	2	1	X	X
Projector (# layers)	4	3	3	4	3	X
Decoder (# depth)	X	X	X	X	X	4
Table S2: Computational details of VSSL pretraining.
v-SimCLR	v-MoCo	v-BYOL	v-SimSiam	v-DINO	v-MAE
Encoder Param	88M
Total Params	106M	110M	114M	114M	96M	96M
Pretraining Flops	36G	72G	72G	36G	72G	10G
Time/100 epochs	23Hrs.	11 Hrs.
Table S3: Hyperparameters of VSSL pretraining.
v-SimCLR	v-MoCo	v-BYOL	v-SimSiam	v-DINO	v-MAE
Batch size	768
Crop scale	[0.08, 1.0]	[0.2, 0.766]	[0.5, 1]
Clip duration	2.0	4.0
Color	[0.6, 0.6, 0.6, 0.15]	-
Grayscale	0.2	-
Gaussian blur	0.5	-
Hrohorizontal flip	0.5
Epochs	800
Warmup epochs	30
Optimizer	AdamW [0.9, 0.95]
Weight decay	0.05	0.05 to 0.5
Learning rate (Base LR)	2e-4	3e-4	3e-4	1e-4	3e-4	3e-4
Pred LR (× Base LR)	-	10×	10×	10×	-	-
EMA (cosine sch.)	-	0.998-1	0.997-1	-	0.997-1	-
Online temp. (fixed)	0.1	0.1	-	-	0.1	-
Target temp. (cosine sch.)	-	-	-	-	0.04-0.07	-
Centering momentum	-	-	-	-	0.9	-
Masking ratio	-	-	-	-	-	85%
B.3 Additional insights
Aggressive cropping. As discussed in the main paper, aggressive cropping leads to viewpoint invariance, therefore we experiment with various multi-crop ratios commonly used in the literature [22, 72] across all six VSSL methods. We present the best setups corresponding to individual methods in Table S3. We observe that aggressive cropping negatively impacts the performance of  $\mathbf{v}$ -MAE. This is primarily because reconstructing a local crop serves as a weak pseudo task due to limited pixel-level variation, resulting in suboptimal representation learning. Additionally, as depicted in Figure S2, the impact of cropping varies between contrastive and non-contrastive methods. Very high aggressive cropping ratios (0.08 - 1.0) prove beneficial for contrastive methods, while slightly less aggressive cropping ratios (0.2 - 0.766) yield better results for non-contrastive methods.
Pretraining epoch vs. accuracy. We track the performance of different videl learning methods on two downstream benchmarks, Kinetics400 and split-1 of UCF101. In particular, Kinetics400 is used for finetuning and UCF101 for linear evaluation. The results are presented in Figure S3 shows that while  $v$ -MAE achieves the lowest linear evaluation accuracy, it shows the best performance when finetuned. Moreover,  $v$ -BYOL,  $v$ -SimCLR, and  $v$ -MoCo exhibit superior performance compared to  $v$ -DINO and  $v$ -SimSiam in both the linear evaluation and finetuning setups. Interestingly, contrastive
(a) Non-contrastive methods
(b) Contrastive methods
Figure S2: The impact of cropping strategies slightly differs between non-contrastive methods ( $v$ -BYOL,  $v$ -DINO) and contrastive methods ( $v$ -SimCLR,  $v$ -MoCo). Aggressive cropping with a crop ratio of [0.08, 1.0], enhances the performance of contrastive methods, while a ratio of [0.2, 0.766] yields the best performance on non-contrastive methods. We show the performance with a crop ratio of [0.2, 0.766] with solid lines and [0.08, 1.0] with dashed lines. Evaluation is carried out on UCF101 using a linear SVM.
methods  $v$ -SimCLR and  $v$ -MoCo show comparable performance among them, however, there is a notable performance gap between the non-contrastive methods. Specifically,  $v$ -BYOL outperforms both  $v$ -SimSiam and  $v$ -DINO by a significant margin in both the linear evaluation and finetuning setups. Moreover, Figure S3 (a) indicates minimal or no improvements between 600 to 800 epochs. Therefore, we refrain from training the models beyond 800 epochs.
(a) UCF101 (linear SVM)
Figure S3: The pretraining epochs vs. top-1 accuracies are presented. VSSL methods are validated in two setups. Kinetics400 is used for finetuning and UCF101 is used for linear evaluation.  $v$ -BYOL,  $v$ -SimCLR, and  $v$ -MoCo show superior performance compared to  $v$ -DINO and  $v$ -SimSiam in both setups. Interestingly,  $v$ -MAE achieves the lowest linear evaluation accuracy, while performing the best when finetuned.
(b) Kinetics400 (finetuned)
Spatial vs. temporal dependency. In addition to the out-distribution tests discussed in the main paper, we also conduct several interesting analyses to understand the dependency of these VSSLs
No Temporal
Partial Appearance
No Appearance
Figure S4: Sample frames from UCF101 depicting different spatio-temporal dependency scenarios.
Original Frames
Table S4: Comparing video models on controlled temporal and appearance information.
Method	No Temporal	No Appe.	Original
Lin. / FT.	Lin. / FT.	Lin. / FT.	Lin. / FT.
v-Supervised	43.7 / 36.6	4.1 / 4.9	1.3 / 1.0	81.7 / 87.4
v-SimCLR	57.1 / 46.5	4.3 / 4.9	0.9 / 1.1	84.1 / 90.3
v-MoCo	56.0 / 48.8	4.7 / 6.7	1.7 / 1.1	84.9 / 90.7
v-BYOL	58.2 / 50.9	3.5 / 6.0	1.1 / 2.6	85.4 / 90.2
v-SimSiam	47.7 / 28.3	5.6 / 4.5	2.0 / 1.5	82.5 / 87.2
v-DINO	41.9 / 38.6	3.2 / 7.7	1.2 / 1.4	80.9 / 88.5
v-MAE	5.6 / 25.7	3.1 / 5.3	1.1 / 2.7	76.2 / 90.5
methods towards spatial vs. temporal representations. To explore the spatial vs. temporal dependency of the VSSL methods, we use UCF101 as the base validation dataset and create 3 synthetic variant of it, (i) No Temporal: we completely remove the temporal information from the input and treat single frames as an input. In particular, we obtain 80 frames per video that represent 10 seconds of video sampled at 8 FPS. (ii) Partial Appearance: Next, to suppress the spatial information, we obtain the optical flow [100] from the original videos. (iii) No Appearance: Following [101], we also obtain the videos of no appearance, i.e., the static frames have no meaning and actions are present in the temporal dimension. We present sample frames of these variants in Figure S4.
The results are presented in Table S4 show several intriguing observations. First, removing the temporal dimension from the videos leads to a significant drop in performance in all the methods. Amongst all the video models,  $v$ -MAE suffers the worst in the absence of temporal information in both linear and finetune setups, which shows its overly dependent nature on temporal information and it is a weak spatial learner (also discussed in the main paper). Second, in the absence of temporal information, the frozen encoder outperforms the finetuned variant even in the case of  $v$ -Supervised. We also notice the performance of the non-contrastive methods significantly varies under such setups, for example, while  $v$ -BYOL achieves the best performance in linear evaluation,  $v$ -DINO performs worse other than  $v$ -MAE. Additionally, our tests under partial or no appearance reveal that none of the video models is capable of understanding actions just from the temporal information. In fact, the performance of video models drops near chance level ( $1\%$ ) in no appearance setup and achieve slightly better ( $3 - 7\%$ ) in partial appearance setup.
C Distribution Shifts
C.1 Overview
We provide a summary of our proposed out-of-distribution test bed in Table S5, and the details of our setup are described in the following subsections. Due to the large number of experiments conducted across various setups, we will release configuration files for individual evaluation on the project website rather than sharing them here.
Table S5: An overview of our out-of-distribution test-bed. #Samples are in following order training samples/InD test samples/OoD test samples; In zero-shot and open-set recognition, #Classes indicates the number of InD/OoD classes and for others the number of InD and OoD classes remains the same.
# Distribution Shift	InD	OoD	#Classes	#Samples
1. Context shift (10 classes)	Kinetics400	Mimetics10	10	5930/494/136
2. Context shift (50 classes)	Kinetics400	Mimetics50	50	34K/2481/713
3. Viewpoint shift (egocentric)	CharadesEgo	CharadesEgo	157	34K/9386/9145
4. Viewpoint shift (surveillance)	MiT-v2	TinyVirat-v2	14	41K/1400/2644
5. Actor shift (animal)	Kinetics400	ActorShift	7	15K/1018/165
6. Viewpoint + Actor shift (top-down+synthetic)	MiTv2	Sims4Action	6	19K/600/950
7. Source shift (UCF/HMDB)	UCF	HMDB	17	1877/746/510
8. Source shift (HMDB/UCF)	HMDB	UCF	17	1190/510/746
9. Zero-shot (K400/UCF)	Kinetics400	UCF	400/31	240K/20K/3965
10. Zero-shot (K400/HMDB)	Kinetics400	HMDB	400/22	240K/20K/3288
11. Zero-shot (K400/RareAct)	Kinetics400	RareAct	400/149	240K/20K/1961
12. Zero-shot (K700/UCF)	Kinetics700	UCF	663/101	480K/ - /13K
13. Zero-shot (K700/HMDB)	Kinetics700	HMDB	663/51	480K/ - /6.7K
14. Zero-shot (K700/RareAct)	Kinetics700	RareAct	663/149	480K/ - /1961
15. Open-set (K400/UCF)	Kinetics400	UCF	400/31	240K/20K/3965
16. Open-set (K400/HMDB)	Kinetics400	HMDB	400/22	240K/20K/3288
17. Open-set (U101/HMDB)	UCF101	HMDB	101/34	9537/3783/4366
C.2 Context shift
We use the two splits of Mimetics (i.e., Mimetics10 [102] and Mimetics50 [28]) as the OoD validation benchmark to study video models under context shift. Mimetics consists of a subset of action classes from Kinetics400, where the context in the videos is either partial or misleading or completely absent. We train the pretrained encoders using videos of corresponding action classes from Kinetics400, and subsequently perform InD and OoD validations using Kinetics400 and Mimetics, respectively. For linear evaluation, we extract the fixed features by taking the mean of all the patches from the last layer of the encoder. We then employ an SVM with a linear kernel for classification. In the finetuning stage, we add a fully-connected layer on top of the encoder and train it end-to-end using the AdamW optimizer and cross-entropy error. The top-1 accuracy is reported for both InD and OoD validation. Please note, the different VSSL methods are individually tuned to achieve their best performance.
C.3 Viewpoint shift
We study three types of viewpoint shifts, (i) egocentric view, (ii) surveillance camera view, and (iii) top-down view. We use CharadesEgo [36], TinyVirat-v2 [63], and Sims4Action [64] as the OoD benchmark for egocentric view, low-resolution surveillance camera view, and top-down view respectively. CharadesEgo is a collection of paired third-person and first-person videos of human activities. Therefore, we use the corresponding third-person videos from CharadesEgo [36] for InD training and validation split while the egocentric view is used as OoD validation. We use MiT-v2 [69] to create the corresponding InD split for TinyVirat-v2 and Sims4Action [64]. MiT-v2 is a large-scale video benchmark of 1M videos spread over 339 different human actions. We use the videos of overlapping classes between MiT-v2 and TinyVirat-v2 to study surveillance camera viewpoint shift.
Similarly, the videos of overlapping classes between MiT-v2 and Sims4Action are used to study top-down viewpoint shifts. The class assignments between InD and OoD benchmarks are mentioned at the end of this subsection.
For training the models on CharadesEgo, we adopt the approach proposed in [103]. In this setup, instead of using one-hot encoded labels, we utilize the text descriptions associated with the videos. We generate tokens from these descriptions and train a video-text encoder using an InfoNCE objective function [55]. In particular, we use the DistilBERT base [104] as the text encoder along with the pretrained ViT-B is used as the video encoder. We redirect the readers to [103], to know more about the training scheme of CharadesEgo. In the case of linear evaluation, the above-mentioned setup remains the same, except we keep the video encoder frozen. Moreover, our training strategy with MiT-v2 to study surveillance camera viewpoint and top-down viewpoint shifts follow a standard finetuning setup using cross-entropy error [72, 2, 4, 6, 78, 79, 7]. However, due to the difference in the nature of the datasets, as TinyVirat contains multiple actions per video, whereas, MiT-v2 consists of single actions per video, we make an adjustment in the prediction layer. Instead of using a Softmax layer, we utilize a Sigmoid layer to predict multiple possible outputs for a given input video. We set the threshold for classification at 0.5. Similarly, the setup for linear evaluation on TinyVirat remains the same, except the video encoder is kept frozen. Next, in the case of Sims4Action, we use SVM for linear evaluation. We report mean average precision and F1-score for CharadesEgo and TinyVirat, respectively, and top-1 accuracy for the rest of them.
TinyVirat-v2 and MiT-v2. opening: opening, pull: pulling, activity_carrying: carrying, entering: entering, exiting: exiting, loading: loading, talking: talking, activity Runsing: running, riding: riding, closing: closing, activity Walking: walking, push: pushing, activity Standing: standing, specialized_talking_phone: telephoning.
Sims4Action and MiT-v2. cook: cooking, drink: drinking, eat: eating, read_book: reading, use_phone: telephoning, walk: walking.
C.4 Actor shift
We study actor shift in two setups, i.e., animal domain and synthetic domain. We use the ActorShift [65] as the OoD validation set to test generalizability on animal actions, while using the video of overlapping classes from Kinetics700 [62] as InD training and validation. We use SVM for linear evaluation and follow standard techniques [72, 2, 4, 6, 78, 79, 7] as discussed earlier for finetuning. We report top-1 accuracy for both InD and OoD validation. We present the overlapping classes between Kinetics700 and ActorShift at the end of this subsection. The setup for synthetic domain shift is already discussed in the previous subsection (Viewpoint shift).
ActorShift and Kinetics700. sleeping, watching tv, eating, drinking, swimming, running, opening door.
C.5 Source shift
To study source shift, we use the videos from the common classes of UCF101 and HMDB51. While using the UCF as the training split, we use the corresponding videos from HMDB as the OoD validation set and vice versa. Following standard protocol [72, 2, 4, 6, 78, 79, 7], we use SVM for linear evaluation and the pretrained encoder along with a fully-connected layer for finetuning. We report the top-1 accuracy for both setups. The class assignments are as follows.
UCF101 and HMDB51. FrisbeeCatch: catch, RockClimbingIndoor: climb, Diving: dive, (BasketballDunk, Basketball): dribble, Fencing: fencing, GolfSwing: golf, (HandstandPushups, Handstand-Walking): handstand, (LongJump, JumpingJack): jump, SoccerPenalty: kick_ball, PullUps: pullup, Punch: punch, PushUps: pushup, Biking: ride_bike, HorseRiding: ride_horse, ThrowDiscus: throw, WalkingWithDog: walk, Archery: shoot_bow,
C.6 Zero-shot
We perform zero-shot in two setups: (1) we train the video-text encoder on Kinetics400, while the non-overlapping classes from UCF101 and HMDB51 are used for OoD validation [105]. (2) We train the models using the videos from Kinetics700 that do not share classes with UCF101 and HMDB51,
while the full UCF101 and HMDB51 are used for OoD validation. Moreover, in both cases, we also use RareAct as the OoD benchmark, which consists of rare or unusual video actions.
Since, the pretraining of VSSL methods does not employ language, first, a video-text encoder is jointly trained to learn the mapping between video and text representations. Following [42], we use Word2Vec [80] to extract the word embeddings from the labels, which are then fed to a text encoder consists of an MLP head. We measure the similarity between the video and text pairs as the dot product of L2 normalized video-text embeddings, referred to as logits. Next, the video text encoders are trained to minimize the Softmax cross-entropy error using the obtained logits and the class labels. We redirect readers to [42] for additional details of the training method. Please note, we also try with a stronger language model (e.g., BERT[106], DistilBERT[104]) than Word2Vec, however, it does not improve the performance, in fact, slightly worsens it. Since the labels in Kinetics are typically composed of only a few words (mostly less than 2), the average pooled word embeddings work well in such cases. This finding is consistent with the observations of [42]. Moreover, we perform zero-shot in both setups when the pretrained video encoder is kept frozen and finetuned in an end-to-end manner.
C.7 Open-set
We perform open-set recognition in two setups: (1) We use the Kinetics400 for closed-set recognition, while the non-overlapping classes from the UCF101 and HMDB51 are used for open-set, similar splits that have been used in zero-shot. (2) We use UCF101 and the non-overlapping classes of HMDB for closed-set and open-set recognition, respectively. We obtain the non-overlapping classes by manual inspection, presented at the end of this subsection.