SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Our baseline concept space (described in Section 3.2) utilizes labels from three standard video datasets (Kinetics-400, UCF-101, HMDB-51). However, we want to ensure scalability with more data and no label leakage to downstream evaluation tasks. With this goal, we propose 2 additional variants of action concept spaces tagged LSS-B and LSS-C. These variants do not use any form of ground truth textual labels from datasets. Moreover, they leverage the world awareness (i.e. knowledge on videos and actions) of LLMs to generate extensive action categories. Our baseline setup is hereafter referred as LSS-A.
For LSS-B, we use GPT-3 [18] to generate a large set of action labels. We first prompt GPT to categorize all common human actions / activities into 20 groups. For each group, we again ask GPT to generate at least 100 visually diverse action categories. These are all collected to create a set of 2000 action labels. We then use projections of these labels in CLIP text-encoder representation space to eliminate labels of high semantic similarity (spectral clustering in feature space from [75] to identify similar features), achieving 1000 diverse action categories. So our 1000 action categories for LSS-B are generic, not tied to any of our training datasets, and scalable with more data.
For LSS-C, we generate a label set using only videos from the training dataset. We use PCA based clustering to identify 2000 representative videos from a randomly sampled subset (50,000) of our training dataset and then use image captioning models (LLaVa [73]) on video center frames to generate a diverse set of 2000 action labels. This is further reduced to 500 eliminating labels that are similar in feature space of the CLIP text encoder. In this case, our generated labels are tied to the training dataset, but uses no textually annotated category labels. We use only the videos (and an image-to-text captioning model) to generate our label set, still resulting in a scalable framework.
Note that each of these alternate strategies relates to construction of our category concept space. Given the selected set of textual category labels of this space, the description concept space is constructed in the same common way (as described in Section 3.2). We also reiterate that LSS-B and LSS-C variants use no category information from train / test datasets.
In this section, we first describe our experimental setup followed by discussion of results for linear probing self-supervised representations and zero-shot analysis.
Datasets: We use three standard action recognition benchmark datasets in our experiments: Kinetics-400 [70], UCF-101 [71], and HMBD-51 [72]. Kinetics-400 is a large-scale dataset containing 240,000 training videos and 20,000 validation videos belonging to 400 different action classes. On average, these videos are of duration around 10 seconds, with 25 frames per second (i.e., around 250 frames per video). UCF-101 and HMBD-51 are small-scale datasets each containing 13k videos (9.5k/3.7k train/test) belonging to 101 classes and 5k (3.5k/1.5k train/test) videos belonging to 51 classes respectively. They also contain similar duration videos.
Self-supervised Training: Our SSL training phase uses the train split of Kinetics-400 dataset [70] without using any per-video labels. We train for 15 epochs using a batch size of 32 across 4 NVIDIA-A5000 GPUs using ADAM-W [76, 77] optimizer on the student model with an initial learning rate of  $1e - 5$  following a cosine decay schedule. The EMA teacher is updated from student weights after each training iteration with a decay ratio of  $2e - 4$ . Unless otherwise specified, this model is used for all downstream task evaluations.
Transductive Training: For selected experiments, we additionally perform self-supervised training directly on the train split of each downstream dataset. For Kinetics-400, we follow the same setup described above. In the case of HMDB-51 and UCF-101, we perform self-supervised training for a longer duration of 100 epochs (smaller train sets) leaving all other hyper-parameters unchanged.
View Generation: Our self-supervised setup requires two views of a single video. We sample two clips from a video following global view generation in [13]. In detail, we select two random intervals from a video, and uniformly sample (equal time gaps between frames) 8 frames of  $224 \times 224$  spatial dimensions from within that interval. Standard video augmentations from [78] are also applied randomly for each view.
Linear Probing: We follow standard linear probing settings on our two downstream datasets to evaluate quality of representations learned by our self-supervised learning phase. We follow the same settings in [13] for fair comparison. Our visual encoder is frozen and a randomly initialized linear layer is trained on the train split of the downstream dataset in a fully-supervised manner. We train for 15 epochs using a batch size of 128 across 4 NVIDIA-A5000 GPUs using ADAM-W [76, 77] optimizer with an initial learning rate of  $1e - 3$  following a cosine decay schedule. During inference, we sample three  $224\times 224$  dimensional spatial crops with 8 uniformly spaced frames from each video following prior work [13, 36].
Zero-Shot Inference: For zero-shot inference, we project class labels of downstream datasets to our text encoder feature space, and construct an alternate text classifier. Using this text classifier, we make zero-shot predictions. This setup is identical to dot-product similarity based inference in CLIP [4] (explanation in Section 3.3). In line with prior work [13, 36], we feed three  $224 \times 224$  dimensional spatial crops with 8 uniformly spaced frames sampled from each video to the visual encoder and average its output feature embedding prior to normalized dot-product calculation in the text encoder.
4.1 Linear-Probing Analysis
We first evaluate LSS under linear probing settings on HMDB-51 & UCF-101 datasets. Our results (top-1 accuracy) are reported in Table 1. Our proposed LSS achieves state-of-the-art results on both datasets, outperforming prior approaches. Note that MoDist [81] and BraVe [89], both of which additionally utilize video-level optical flow (OF) for self-supervision, are not directly comparable. Still, our LSS showcases competitive performance to those, even without such motion information.
4.2 Zero-Shot Analysis
Our LSS provides the additional advantage of zero-shot operation unlike standard video SSL approaches. To this end, we conduct two forms of zero-shot experiments. First, we evaluate LSS on standard zero-shot classification, where our model trained on Kinetics-400 (under SSL settings) is evaluated on the two downstream datasets, HMDB-51 and UCF-101. We report these results (top-1 accuracy) in Table 2. Compared to prior work utilizing per-video labels / captions for training, we
Table 1: Linear Probing on HMDB-51 [72] and UCF-101 [79]: We compare our method against prior work, reporting top-1 (\%) accuracy (following evaluation procedure in [13]). 'ITP' stands for image-text pre-training. Gray shaded methods use additional optical flow (OF) inputs for training. Nevertheless, our performance is comparable to such methods using per-video OF modality information. In contrast, we use generic language modality information and require no one-to-one language relations with individual videos during training.
Method	Backbone	ITP	TFLOPS	Frames	Epochs	HMDB	UCF
MemDPC [34] (ECCV '20)	R2D3D-34	X	-	64	-	30.5	54.1
CoCLR [35] (NeurIPS '20)	S3D	X	0.07	32	100	52.4	77.8
VideoMoCo [80] (CVPR '21)	R(2+1)D	X	17.5	32	200	49.2	78.7
CVRL [36] (CVPR '21)	R3D-50	X	3.19	32	800	57.3	89.2
MoDist [81] (Arxiv '21)	R3D-50	X	3.19	8	100	63.0	91.5
BraVe [38] (ICCV '21)	R3D-50	X	3.19	16	-	68.3	92.5
Vi²CLR [82] (ICCV '21)	S3D	X	0.07	32	300	47.3	75.4
MCN [83] (ICCV '21)	R3D	X	3.19	32	50	42.9	73.1
CORP [84] (ICCV '21)	R3D-50	X	3.19	16	800	58.7	90.2
SVT [13] (CVPR '22)	ViT-B	X	0.59	16	20	57.8	90.8
VideoMAE [14] (NeurIPS '22)	ViT-B	X	0.59	16	800	60.3	84.7
MERLOT [85] (NeurIPS '21)	ViT-B	✓	-	16	-	55.4	80.1
VATT [86] (NeurIPS '21)	ViT-B	✓	-	32	-	66.4	87.6
TVTS [87] (CVPR '23)	ViT-B	✓	0.59	16	20	58.4	83.4
LaViLa [88] (CVPR '23)	ViT-L	✓	-	4	5	61.5	88.1
LSS-A (ours)	ViT-B	✓	0.59	8	20	69.2	91.0
LSS-B (ours)	ViT-B	✓	0.59	8	20	69.4	91.1
LSS-C (ours)	ViT-B	✓	0.59	8	20	69.1	90.8
Table 2: Zero-shot Transfer on HMDB-51 [72] and UCF-101 [79]: We compare LSS against prior work, reporting top-1 accuracy (\%). Mean across three test splits is reported following [55]. 'ITP' stands for image-text pre-training and 'Video Labels' refers to using per-video annotations (or paired captions) for supervision during video-based training. We highlight how among directly comparable unsupervised (at video level) approaches as well as over the CLIP [4] baseline, LSS boosts zero-shot performance.
Method	Backbone	ITP	Video Labels	Frames	HMDB	UCF
TS-GCN [90] (AAAI '19)	GCN	X	✓	16	23.2	34.2
E2E [91] (CVPR '20)	CNN	X	✓	16	32.7	48.0
ER-ZSAR [48] (ICCV '21)	CNN	X	✓	8	35.3	51.8
ActionCLIP [11]	ViT-B	✓	✓	32	40.8	58.3
X-CLIP [12] (ECCV '22)	ViT-B	✓	✓	32	44.6	72.0
VicTR [55]	ViT-B	✓	✓	32	51.0	72.4
ViFi [62] (CVPR '23)	ViT-B	✓	✓	32	51.3	76.8
MTE [92] (ECCV '16)	-	X	X	-	19.7	15.8
ASR [93] (ECML '17)	CNN	X	X	16	21.8	24.4
ZSECOC [94] (CVPR '17)	-	X	X	-	22.6	15.1
UR [49] (CVPR '18)	CNN	X	X	1	24.4	17.5
CLIP [4] (ICML '21)	ViT-B	✓	X	1	46.5	69.8
CLIP [4] (ICML '21)	ViT-B	✓	X	8	47.2	70.3
LaViLa [88] (CVPR '23)	ViT-L	✓	X	4	16.6	18.2
LSS-A (ours)	ViT-B	✓	X	8	49.5	72.0
LSS-B (ours)	ViT-B	✓	X	8	50.2	73.8
LSS-C (ours)	ViT-B	✓	X	8	51.4	74.2
achieve competitive performance. We note that MOV [7] trained under supervised settings with per-video labels and additional audio information is not a direct comparison.
In contrast to most prior approaches, LSS uses no video level labels for its Kinetics-400 training. In particular, LSS has not seen any labelled videos during its training process. Compared to prior
Table 3: Transductive Zero-shot Transfer on HMDB-51 [72], UCF-101 [79], and Kinetics-400 [70]: We report top-1 accuracy  $(\%)$  following the evaluation procedure in [55]. Similar to prior work, we perform dataset specific unsupervised fine-tuning (using our self-supervised objective) on the train-splits of each downstream dataset (no labels used). 'ITP' refers to image-text pretaining, and 'Video Labels' refers to video level supervised training. Note that CLIP [4] is not transductive and is included only for comparison purposes.
Method	Backbone	ITP	Video Labels	Frames	HMDB	UCF	K400
UR [49](CVPR '18)	CNN	X	X	1	28.9	20.1	-
TS-GCN [90] (AAAI '19)	GCN	X	✓	16	23.2	34.2	-
CLIP [4] (ICML '21)	ViT-B	✓	X	8	47.2	70.3	49.7
MUST [59] (ICLR '23)	ViT-B	✓	X	1	48.9	81.1	51.2
LSS (ours)	ViT-B	✓	X	8	55.0	75.6	54.3
Table 4: Ablation on SSL objectives: We ablate our proposed concept distillation (CD) applied on category  $(\mathrm{CD}^{\mathrm{C}})$  and description  $(\mathrm{CD}^{\mathrm{D}})$  concept spaces and concept alignment (CA) using linear probing (LP) & zero-shot transfer (ZS) on HMDB-51 [72] dataset. Since our approach cannot be trained without any objective, we construct two new baselines from CLIP [4] and SVT [13] (details in Section 4.3). Each proposed component obtains clear improvements over the baselines.
CDC	CDD	CA	LP	ZS
CLIP	X	X	X	63.9	46.5
CLIP†	X	X	X	67.3	47.2
SVT§	X	X	X	62.2	-
Ours	✓	X	X	68.5	48.8
Ours	✓	✓	X	69.0	49.2
Ours	✓	✓	✓	69.2	49.5
Table 5: Concept Space Ablation: We report zero-shot accuracy  $(\%)$  on HMDB-51 and UCF-101 datasets. 'K400 only' shows transfer to unseen downstream classes. The labels of K400 overlapping with UCF/HMDB are not used here. A CLIP [4] baseline (modified for video domain without re-training) is reported in row 1 for comparison purposes. For 2000 words & 10,000 sentences, we utilize most common nouns / verbs and example sentences for action verbs from WordNet [95, 96].
Method	Action Labels	HMDB	UCF
CLIP	-	47.2	70.3
Ours	K400 only (w/o U,H)	48.4	71.1
Ours	K400+U+H (A)	49.5	72.0
Ours	(A)+2000 words	48.6	71.8
Ours	(A)+10K sentences	49.6	71.4
work operating under these settings, LSS achieves state-of-the-art performance on both downstream datasets as seen in the bottom half of Table 2.
An alternate setting in prior zero-shot work is transductive training, where self-supervised learning is perfomed directly on train splits of downstream datasets. Under this setting, we evaluate on all three datasets, Kinetics-400, HMDB-51, and UCF-101, reporting results (top-1 accuracy) in Table 3. In the case of HMBD-51 and Kinetics-400, our method achieves state-of-the-art performance. For UCF-101, we achieve competitive results, and clear improvements over a CLIP [4] baseline.
We next study the contribution of each component within our approach. All ablative experiments follow the same SSL phase on the Kinetics-400 train set (as described in Section 4) followed by zero-shot analysis on validation sets of HMDB-51 and UCF-101. In the case of linear probing results, training is conducted following same settings (see Section 4) on the train set of HMDB-51 followed by evaluation on its validation set.
SSL Objectives: First we ablate each proposed component in Eq. (11) and report results in Table 4. In addition to a direct CLIP [4] baseline, we construct two additional baselines building off CLIP [4] and SVT [13] for better comparison.  $\mathrm{CLIP}^{\dagger}$  baseline applies our backbone modifications (for temporal modeling) with no training, which is identical to averaging per-frame visual encoder features.  $\mathrm{SVT}^{\S}$  baseline performs SVT [13] training with CLIP visual encoder initialization (note that language alignment breaks and zero-shot operation is not possible for this baseline). In comparison to the CLIP baselines, each proposed component, concept distillation in category and description concept spaces as well as concept alignment, leads to improvements. The comparison against  $\mathrm{SVT}^{\S}$  highlights how our SSL approach better preserves language aligned information (contained in CLIP) that is useful even in linear probing. In contrast, the lower performance of  $\mathrm{SVT}^{\S}$  compared to CLIP baselines indicates that generic SSL techniques may be losing useful information contained in CLIP.
Table 6: Ablation on Regularization and Significance Weight: The effect of proposed regularization (left) and significance weight (right) is demonstrated. UDP regularization (see Section 3.4) is particularly essential to prevent collapse during the SSL training phase. Significance weight  $(w_{s})$  also improves performance.
Method	HMDB	UCF
LSS	48.4	71.1
LSS w/o UDP	33.4	54.3
LSS w/o wS	47.2	70.3
Concept Spaces: Our next focus is on construction of concept spaces. We explore how separately augmenting each concept space affects downstream task performance measured with zero-shot transfer. These results are reported in Table 5. First, focused on the category concept space, we construct additional category labels using 1000 most common nouns and verbs each (total of 2000) from the WordNet dataset [95, 96]. Next, we augment the description concept space using 10,000 sentences. We select these from example sentences provided for action verbs in the WordNet dataset [95, 96]. In these experiments, only the concept distillation objective is applied on these augmented spaces and concept alignment operates only on the base category set. This is because independently augmenting one of the action spaces eliminates their shared and aligned dimensionality. Results for these two settings are reported in row 2 & 3 respectively in Table 5.
Uniform Distribution Prior: We ablate on proposed uniform distribution prior (UDP) which acts as a regularization to prevent collapse (Section 3.4). Our results in Table 6 (left) indicate clear necessity of such regularization to prevent collapse during SSL training.
Significance Weight in Concept Distillation: In Eq. (6), we utilize a significance weight term,  $w_{s}$  which represents the confidence of the target concept space projection for a given sample. We note how each sample during training is a clip sampled from a video (which covers a temporal crop of video). Our intuition for this weight is to act as a way of prioritizing more important clips over the less important ones. Our ablations in Table 6 (right) indicate usefulness of this weight term.
We introduce a novel language-based self-supervised learning (SSL) approach for videos, termed LSS, capable of adapting strong language-aligned image representations (CLIP [4]) to the video domain. In particular, we propose two self-distillation based SSL objectives, concept distillation and concept alignment. Our approach trains with no video level labels or paired captions similar to prior video SSL works, but retains language alignment from image CLIP enabling direct zero-shot inference. We demonstrate state-of-the-art performance in terms of linear probing with the learned representations on downstream tasks. For zero-shot operation, LSS demonstrates strong performance under both standard and transductive settings, indicating a promising direction for video SSL.
Limitations, Future Work, & Broader Impact: The language alignment of LSS may be limited mostly to per-frame static information since the alignment is derived from image CLIP [4]. LSS cannot distinguish motion based categories like "moving object left to right". Moreover, while containing highly discriminative and generic information at image level, CLIP features lack spatial awareness at an object level [75]. Our proposed model building off these representations in inherently limited in understanding object level motion and interaction within videos. However, recent progress in localization aware CLIP models [75, 97, 98] opens avenues for leveraging their object-centric or pixel-level representations to better model such video motion patterns, opening up interesting future directions. In terms of broader impact, the datasets and pre-trained models we use possibly contain biases, which may be reflected in LSS. However, our reduced reliance on human annotations may lower additional biases.
Reproducibility Statement: We build a codebase derived from source code of SVT [13] & CLIP [4] and use pre-trained CLIP weights from https://github.com/openai. All experiments use publicly available datasets. Our action descriptions will be released publicly along with our codebase.
Acknowledgements: We thank Xiang Li for helpful discussions and server setup. We also thank Kumara Kahatapitiya and Cristina Mata for helpful discussions.
A Prompting details
Our proposed approach utilizes two sets of language based captions: categories and descriptions. While categories are obtained directly from the class labels of datasets (set of unique labels - e.g. 400 classes in Kinetics-400 dataset), the descriptions are generated automatically utilizing GPT-3 [18]. For each category caption, we query GPT-3 to provide a set of descriptions and visual characteristics. In detail, we use the following two prompts to generate descriptions and visual characteristics: prompt1 = "Give 4 different descriptions for the phrase: {category}?" prompt2 = "List visual objects or characteristics usually seen with the action: {category}?" The resulting two sets of captions are converted to text embeddings using our text-encoder, and a single average text embedding is computed. This averaged embedding is used as the description basis vector for that category. Also, the resulting dataset containing these category-description pairs is made available publicly.
Linear Probing Evaluation: We present more results for linear probing in Table 7 (left). Our proposed LSS improves over the baseline achieving competitive performance on Kinetics-400.
Text-to-video retrieval: An important characteristic of CLIP [4] is its retrieval ability across both language and visual modalities. In order to verify if proposed LSS retains these strengths, we run experiments on MSR-VTT text-to-video retrieval benchmark. We demonstrate how LSS improves over our baseline CLIP, reporting these results in Table 7 (top-right).
Charades Evaluation: We explore an alternate task of zero-shot multi-label classification on the Charades video dataset. We report mAP results for this task in Table 7 (bottom-right) as an additional point of comparison.
Table 7: We report top-1 (\%) accuracy on the Kinetics-400 [70] validation set for linear probing evaluation (left). All models are pre-trained on the training set of Kinetics-400 dataset. We also report a CLIP baseline for comparison purposes. Performance of our proposed approach is on-par with prior state-of-the-art and showcases improvements over our baseline method. We also report retrieval scores (top-right) for MSR-VTT and classification mAP (bottom-right) for Charades dataset.
Method	Backbone	Acc (%)
CVRL [36] (CVPR‘21)	R3D-101	67.6
BraVe [38] (ICCV‘21)	R3D-50	66.7
Vi2CLR [82] (ICCV‘21)	S3D	63.4
CORP [84] (ICCV‘21)	R3D-50	66.6
SVT [13] (CVPR‘22)	ViT-B	68.1
VideoMAE [14] (NeurIPS‘22)	ViT-B	61.3
CLIP [4]	ViT-B	66.4
LSS (ours)	ViT-B	67.3
Method	R@1	R@5	R@10
CLIP [4]	30.6	54.4	64.3
LSS (ours)	33.8	58.2	70.3
Method	Classification mAP
CLIP [4]	19.7
LSS (ours)	23.1