Language-based Action Concept Spaces Improve Video Self-Supervised Learning
Authors: Kanchana Ranasinghe
Michael S Ryoo
Venue: NeurIPS 2023 poster (2023)
Keywords: self-supervised learning for videos
zero-shot action recognition
TL;DR: Adopting image CLIP models to video domain with language based video self-supervision

Abstract: Recent contrastive language image pre-training has led to learning highly transferable and robust image representations. However, adapting these models to video domain with minimal supervision remains an open problem. We explore a simple step in that direction, using language tied self-supervised learning to adapt an image CLIP model to the video domain. A backbone modified for temporal modeling is trained under self-distillation settings with train objectives operating in an action concept space. Feature vectors of various action concepts extracted from a language encoder using relevant textual prompts construct this space. A large language model aware of actions and their attributes generates the relevant textual prompts.
We introduce two train objectives, concept distillation and concept alignment, that retain generality of original representations while enforcing relations between actions and their attributes. Our approach improves zero-shot and linear probing performance on three action recognition benchmarks.

Language-based Action Concept Spaces Improve
Video Self-Supervised Learning
Kanchana Ranasinghe
Michael Ryoo Stony Brook University mryooêcs ieroccs.stoo stoneerooootbbbooooooooooo
Stony Brook University kranasinghe@cs. wtw.aiinnnoesssttttooooooooo stonybrook.edu
Abstract
Recent contrastive language image pre-training has led to learning highly trans- ferable and robust image representations. However, adapting these models to video domain with minimal supervision remains open problem. We explore a simple step in that direction, using language tied self-supervised learning to adapt image CLIP model the video domain. A backbone modified for temporal modeling is trained under self-distillation settings with train objectives operating an action concept space. Feature vectors of various action concepts extracted from a language encoder using relevant textual prompts construct this space. A large language model aware of actions and their attributes generates the relevant textual prompts. We introduce two train objectives, concept distillation and con- cept alignment, that retain generality of original representations while enforcing relations between actions and their attributes. Our approach improves zero-shot and linear probing performance on three action recognition benchmarks.
Introduction
Actions in videos are defined by individual objects, their relationships, and interaction [1, 2]. Video self-supervised learning focuses discovering representations aware of such action attributes directly from video content with no human supervision [3]. Particularly in the case of videos, where manual human annotation can be both expensive and noisy, such self-supervised approaches are invaluable.
recent variant self-supervision explores learning with loosely paired image-caption pairs, leading to highly transferable and robust representations such as CLIP [4]. These approaches obtain zero-shot performance often comparable
sully-super methods. However, their counterparts the video domain [5. 6, 7, 8, 9, 10, 11] do not exhibit the same generality. In fact, some approaches training CLIP on videos [11, 12] perform subpar to image-CI.IP under zero-shot settings (see Table 2). Such behaviour can be attributed to lesser availability and more noisy nature of labelled (or paired caption) video datasets [3]. This motivates exploration into self-supervised learning (SSL) techniques that can learn from videos under less supervision while utilizing existing image CLIP [4] like representations. Existing state-of-the-art video SSL approaches [13, 14] learn highly transferable representations from videos, but combining these with image CLIP representations is not straightforward. In fact, despite methods like SVT [13] being able to utilize image SSL representations [15] for weight initialization to achieve better performance, using image CLIP representations instead for weight initialization leads to performance subpar to image CLIP (see Table 4). This raises necessity for alternate video SSL approaches compatible with CL.IP like image representations and is key motivation.
In this work, we explore supervieed learning techniques that adapt image CLIP models [4] to video domain under entirely self-supervised settings, dependent or captions. Under this setting, natural language can still provide strong cues regarding attributes that compose an action category [16, 17]. We leverage this idea propose novel langnage-based self-supervised learning objective. Following a standard self deff-distillation and multi-view based SSL
form of video level labels
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

formulation [15, 13], we introduce language aligned feature spaces, action concept spaces, where our SSL objectives operate. Large-language models (LLMs) [18], given their extensive world knowledge [19, 20], serve as an ideal tool to generate necessary textual concepts for these spaces. We also introduce regularization suitable for our language aligned SSL, objective to prevent collapse during training. Our resulting framework is termed Language-ber based Self-Supervision,, or LSS.
In contrast to existing video self-supervised learning approaches [13, 14], our proposed LSS retains and improves transferability of image CLIP representations much better (see Tables I and 4). Addi- tionally, our language aligned learning framework allows direct zero- zero-shot operation on downstream tasks. Moreover, unlike video CLIP methods with similar zero-shot capabilities [5, 6, 7, 8, 9, 10, 11] that utilize per-video labels/ / captions for learning, our proposed LSS requires only videos for training.
We summarize our key contributions as follows:
Self-supervised learning paradigm capable of retaining and improving strengths of CLIP like image representations for video domain operation
Video specific self-supervised learning objectives, namely concept distillation and concept aligument, that enforee relations between action categories and their visual attributes
Novel language-based video self-supervised learning framework operating zero-shot on down- stream action classification tasks without requiring per-video labels / captions for training
Experiments on action recognition datasets showcase state-of-the performance for our learned representations under thear-probing. standard zero-shot, and transductive zero-shot settings.
2 Related Work
Self-Supervised Learning in Videos was initially dominated by pretext tasks specific to the video
domain [21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32]. Recently shift to contrastive losses led to [33, 34, 35, 36, 38] with some variants focused on video specific view generation [39, 40, 41, 42, 13]. An alternate direction has been masked auto-encoders [14]. To the best of our knowledge, existing video self-supervised learning (SSL) approaches operate purely within the visual domain. By video SSL. we refer methods that utilize only videos with no paired captions (or labels) for each video during training. In contrast, our proposed LSS learns purely from videos in a self- self-super manner, integrating pre-trained language-image models to learn language aligned representations.
Zero-shot Action Recognition began with manual attribute and feature selection [43, 44, 45, 46, 47] with later works utilizing action word embeddings [16, 17]. The idea of connecting action categories with elaborate descriptions of those actions, within language embedding spaces [48, 49] has been a next step and is closely related to our work. This idea is also explored in image domain to boost zero- shot performance [50]. While our work is inspired by such approaches, in contrast, we use relations between such actions and descriptions
self-supervised signals for learning. Recent image CLIIP models [4, 51] are another line of works achieving strong performance on some video classification tasks, with only single frame processing. Multiple approaches build on image CLIP [4] to learn video level representations [11, 52, 53, 54, 12, 55] under fully-supervised settings. While achieving strong performance on the training datasets, their zero-shot improvements over CLIP are minimal or even subpar (see Table 2). Therein, LSS focuses on zero-shot performance under self sele-super settings while retaining (and improving) the generality of the representation space.
Self-training methods leverage pseudo-labels on unlabeled data [56, 57, 58) for supervised-fashion training. Recently they have been combined with CLIIP models for zero-shot operation [59, 60). While inspired by such self-training approaches, our proposed LSS differs in its continuous feature space dif-disiillatio, language-basee relations enforcing, video domain operation, and cross-dataset transfer for zero-shot operation.
Adapting image-CLIP models to video under fully-supervised settings has gathered much interest [5. 6, 7, 8, 9, 10]. Expanding backbones for temporal modeling, multi-mode fusion, secondary training objectives, partial parameter updates, and scaling-up data key ideas explored [55, 10]. In contrast, LSS is a first to operate under self-super settings using no video annotations.
Contemporary work in [61] adapts image CLIP features to video tasks label free similar to our work. ViFi-CLiP [62] introduces zero-shot action recognition benchmarks and similarly adapts CLIP to videos retaining generality. Using LLMs for action recognition is also explored in [63].

---

Visual Encoder
Text $
target category score
target description
Teacher
Classifier
score distribution
distribution
M
augmented samples
Concept Distillation
Concept Alignment
Concept Distillation
EMA
predicted category
predicted
&
Visual Encoder
Text Classifier
description
score distribution
score distribution
brush cartwheel clap
Mirrrr, hairbrush, fingers, hair ties or clips Arms and legs extended outwards.
Text Encoder
Concept Vectors
Text Encoder
Hands coming together... Moving vertically up, lifting themselves.
climb
Figure I: Our overall setup contains three components: visual teacher model (green), visual student model (red), and language model (blue). We utilize the text encoder of CLIP vectors relevant to action labels descriptions those actions. visual encoder (containing space-time backbone) partially initialized with the visual encoder of CLIP and used to obtain sample specific features. The generated concept vectors are used to project these features to concept space where our proposed concept distillation and concept alignment losses are applied.
language model and extract concept
3 Language-based Self-Supervision (LSS)
In this section, present proposal. Language-base -base Sulf--uupr (LSS). The generality and robustness of shared image-language representation spaces such as that of CLIP [4] allow interesting manipulations of visual representations using language. We explore such manipulations under the setting of visual self-supervised learning focusing on video understanding. Self-supervised objectives operate within latent space constructed with language, retaining language alignment of learned visual representations. This allows better interpretability of representations as well as zero-shot inference. We discuss the four key components of our approach: backbone architecture, concept distillation objective, modifications to avoid collapse, and concept alignment objective.
3.1 Backbone Architecture
Our approach introduces text classifier self-distillation based SSL works [15, 13], in place of the projector network. Given data sample let æ1, 22 E (BC,T.H,W) be two augmented views generated using video specific transformations following [13], where C = 3, = 8, H W 224 channel, time, and spatial dimensions respectively.
Visual Encoder: visual encoder, fg. processes z; to produce feature f. IR 708 We utilize the pre-trained image encoder of CLIP [4] expanded for temporal modelling using factorized space-time attention. The vision transformer variant CLIP is selected to allow our factorized space-time attention. In particular, we use ViT-B/16 architecture for the the image encoder, in which for a given augmented view with = W = 224 and T 8, each transformer block processes temporal and 196 spatial tokens separately in sequential order, and the embedding dimension of each token is 768 In addition to the input tokens from the data sample, one classification token [64, 65] serves as the final feature vector output by the network, namely ft. which is common to the CLIP image encoder. This classification token inflated and processed suitably following [66] to accommodate modifications for factorized space- space-time attention. We follow [66] to pero-initialize additional time-attention parameters, achieving outputs identical to the pre- rraitra CLIP image encoder at start of training.
Text Classifier: Inspired by [67], J, a set of n language embeddings extracted from the CLIP text encoder, 9r, used to construct the weight parameter of linear layer (with no bias term), which

---

run swim walk
Text Encoder
visual feature ft
basis vectors b;
toy concept space
text classifier [b, by. b,]"
Figure 2: We illustrate a toy concept space constructed with the three action concepts: run, swim, and walk. In this example, the text classifier projects visual feature fi into the 3-dimensional toy concept space produce fi.
call our text classifier, e.. The role of this text classifier is project visual features fü to a vector space defined by those embeddings, producing J, IR... Next we discuss these vector spaces (referred to as action concept spaces) and the text classifier module in detail.
3.2 Action Concept Spaces
Self-supervised learning approaches following exponential moving average (EMA) based self- distillation [68, 15, 13] utilize a projector network (MLP) to operate in higher dimensional feature space. This is expected to minimize train-test domain gaps, handle noisy positive sample pairs, and better discriminate nuanced feature differences [69]. Focused on these notions, we propose an alternate concept space composed of set of basis vectors defined by language- based action concepts. Our language- based suf-ssupervsion objectives operate within such concept spaces.
Concept Spaces: Building off the assumption that text encoder features capture subtle differences between distinct actions categories, we hypothesize that necessary nuanced distinctions between these actions will be better captured in our proposed concept spaces. The defining parameters of concept spaces are their basis vectors, b; Normalized embeddings (extracted from text encoder, 0,)
of various natural language captions (c)) relevant to action categories are used as these basic vectors.
b; (ct) / (c)((
b = [b1. b2.. b,]l : R(n.d)
(2)
Note that these basis vectors are not necessarily orthogonal. As illustrated in Fig. 2, a single set of basis vectors, b, defines one action concept space. We define two sets of basic vectors: action category vectors and action description vectors. Action category vectors relate to a single action label which is converted to a caption using textual prompting following [4]. Action description vectors are averaged embeddings of multiple descriptions and visual characteristics relevant to individual action categories. These two distinet sets of basic vectors lead two distinct concept spaces which we name category concept space and description concepí space respectively.
Category Concept Space: We explore 3 different strategies construct the category concept space. The base setup uses action labels from Kinetics-400 [70], UCF-101 [71], and HMDB-51 [72] datasets, leading to set of 530 (400 + 101 + 51, ignoring overlaps) basis vectors. Our next goal of connecting LLMs and their action awareness occurs in the second two strategies. We utilize LLMs [18] and visual-LLMs [73] to extract large sets of action category labels. While we explore this idea of expanding the basis vector set with LMM based additional action labels in Section 4, the base setup containing a modest 530 categories was sufficient to improve downstream task performance.
Description Concept Space: This space is constructed conditioned on the previous category concept space. For each action label used in the latter, we extract 4 distinct descriptions and a of visual characteristics relevant to that action label using a large language model (LLM).. The role of the LLM is to inject its world knowledge (i.c. awareness on vidcos, actions, and their attributes) into our learned representations during self-supervised learning. In detail, we prompt GPT-3 [18] to generate such descriptions and characteristies using procedure outlined in Appendix A. We highlight that GPT-3 is used here as an intelligent LLM containing world knowledge on videos and actions, in order to create natural language descriptions for given action category labels. The textual outputs generated for each action label are processed by our text encoder to produce multiple embeddings for a single action label. These embeddings are averaged to produce the corresponding basis vector for the description concept space. Note how this leads to common dimensionality between the two concept spaces as well as one to one correspondences between the basic vectors of the spaces, which we leverage in our self-supervision objectives.

---

3.3 Concept Distillation
We now describe our primary self-supervised learning objective, concept distillation. Standard multi-view based self-supervision enforces a network to encode the common information between two augmented (distorted) views of a data sample [69]. This common information can be considered as the augmentation invariant signal present in the original data sample [69, 74]. In the case of self-distillation based approaches [15, 13], a higher dimensional feature space is utilized to enforce the self-super supervisee objectives. Instead, we propose to use action concept spaces as an alternative.
Proposed concept distillation depends on an action concept space and visual video features aligned to the basis vectors of that space. Given our visual features fi Rdd, we obtain projected ft R" as,
f b (ft/ [1AI1) - [b1 f/ b2 -f. bn f1'
Similarity Calculation: Projecting normalized visual video features to concept space corresponds to calculating the dot-product similarity with each basic vector of the concept space. The projected vector f; can be viewed as similarity score distribution across all basis vectors of the concept space. Inspired by [67], we implement this similarity calculation as a linear layer with weight matrix b and bias terms zero. We refer to this layer as the text classifier. Similar to [67], our text classifier remains frozen (no parameter updates), but in our case, this is to retain the original language distribution.
Concept Distillation Objective: Viewing projected features for two augmented views of a single video as score distributions, we argue that the underlying signal of the original video would relate to unique score distribution which score distributions of each view should be similar. Therein, following our EMA teacher based self celf-disiillation setup (see Section 3.1 for details), we enforce the score distribution to be consistent across views. Given two views 2 22 of a single video, our teacher and student visual encoders process them respectively to produce fi, fz. The text classifier projects these to concept space, producing score distributions f2. We obtain our objective, LCn as:
exp exp(F. (K))))/A))
[
(4)
Σ exp exp(ffll/M)
Ws = max(f1)
(5)
(cD(Fi. w Σ log febi
(6)
The teacher and student score distributions, f1, f2. are softmax normalized in Eq. (4), with temperature terms \ = 0.1, A2 = 1 for sharpening only the teacher score distribution. A significance score w. is calculated for each sample in Eq. (5). In the softmax normalized teacher score distribution (f1). the maximum value is high when peaked action concepts. Considering the noisy nature of multi-peak teacher score distributions, we utilize w. to minimize their overall effect during training. Our overall LCD is thus implemented as in Eq. (6).
single action concept and low when peaked multiple
Given the two distinct action concept spaces defined in Section 3.2, we utilize two parallel text classifiers to implement each, and obtain two score distributions, one for each concept space. Defining score distributions j f.P for category and description concept spaces respectively, we apply our LCD on each pair separately to obtain two losses L for (C,D, as:
Distinct Concept Spaces:
CxD = (
We highlight how our concept spaces implemented as text classifiers are maintained intact by freezing the text classifier during training. This allows approach to perform direct zero-shot inference, making concept distillation additionally advantageous over standard video SSL techniques.
3.4 Uniform Distribution Prior
Avoiding collapse is a key concern in SSL methods [15, 13, 69] and recent self-distillation based approaches utilize feature sharpening and centering operations to avoid collapse [15, 13]. While we similarly perform sharpening operations on the teacher outputs, given the nature of our action concept space, performing learned vector mean subtraction based centering operations can break the meaningful structure of score distributions. Instead, enforce uniform distribution prior on
5

---

the expected score distribution over the entire training dataset. The centering operation proposed in [15] acts similarly pushing representations towards a uniform distribution while the sharpening operation counters its effect. We approximate expectation over the dataset moving average of mean score distributions at each train iteration and the uniform prior is enforced as:
MA T. 2x + +1 7) AM
(8)
C _ Σ log MaL
where the hyper-parameter T = 0.5 is fixed during training. We highlight that LUP is necessary for convergence with concept distillation and is added to the concept distillation objective, LeD.
3.5 Concept Alignment
Aligning action category labels and their descriptions or attributes within some embedding space has been explored in video SSL under multiple settings [48, 49]. Motivated by these promising results, we explore how such alignment can be integrated improve our framework with concept spaces. In Section 3.2, we define two distinct action concept spaces constructed from category labels and detailed category descriptions respectively. We hypothesize that explicit alignment of video features between these two spaces based on their one one relationship can learn additional information. Therein, we introduce our concept alignment objective, LCA, as follows:
Lca = LcD(Jf, (D) + LCD JT)
(10)
Overall SSL Objective: Reusing LCD from Eq. (6), we match score distributions across our two concept spaces instead of within single concept space. LCD f2) aligns student description score distribution / to teacher category score distribution j while LCD(E (f)) aligns student category
score distribution f to teacher description score distribution f.. Combining all terms, we obtain:
L= (CE LGe) + (CDD + +bp) + LCA
(11)
3.6 Concept Space Variants
Our baseline concept space (described in Section 3.2) utilizes labels from three standard video datasets (Kinetics-400, UCF-101, HMDB-51). However, we want to ensure scalability with more data and no label leakage downstream evaluation tasks. With this goal, propose 2 additional variants of action concept spaces tagged LSS-B and LSS-C. These variants do not use any form of ground truth textual labels from datasets. Moreover, they leverage the world awareness (i.e. knowledge on videos and actions) of LLMs to generate extensive action categories. Our baseline setup is hereafter referred LSS-A.
For LSS-B, we use GPT-3 [18] to generate a large set of action labels. We first prompt GPT to categorize all common human actions / activities into 20 groups. For each group, we again ask GPT to generate at least 100 visually diverse action categories. These are all collected to create a set of 2000 action labels. We then projections of these labels in CLIP text-encoder representation space to eliminate labels of high semantic similarity (spectral clustering in feature space from [75] to identify similar features), achieving 1000 diverse action categories. So our 1000 action categories for LSS-B are generic, not tied to any of our training datasets, and scalable with more data.
For LSS-C, we generate label set using only videos from the training dataset. We use PCA based clustering to identify 2000 representative vidcos from a randomly sampled subset (50,000) of our training dataset and then use umage-captioniin generate a diverse set of 2000 action labels. This is further reduced to 500 eliminating labels that are similar in feature space of the CLIP text encoder. In this case, our generated labels are tied to the training dataset, but uses no textually annotated category labels. We use only the videos (and an image-to to-tett captioning model) to generate our label set, still resulting in a scalable framework.
models (LLaVa [73]) vidco center frames to
Note that each these alternate strategies relates to construction of our category concept space. Given the selected set of textual category labels of this space, the description concept space is constructed in the same common way (as described in Section 3.2). We also reiterate that LSSS- and L.SS-C variants no category information from train test datasets.

---

4
Experiments
In this section, we first describe our experimental setup followed by discussion of results for linear probing self-supervised representations and zero-shot analysis.
Datasets: We use three standard action recognition benchmark datasets in our experiments: Kinetics- 400 [70], UCF-101 [71]. and HMBD-51 [72]. Kinetics-400) is a large-scale dataset containing 240,000 training videos and 20,000 validation videos belonging to 400 different action classes. On average, these videos are of duration around 10 seconds, with 25 frames per second (i.e., around 250 frames per videa). UCF-10) 101 and HMBD--5 are small-scale datasets each containing 13k videos (9.5k/3.7) 7k train/test) belonging to 101 classes and 5k (3.5k/1.5k train/test) videos belonging to 51 classes respectively. They also contain similar duration videos.
Self-supervised Training: Our SSL training phase uses the train split of Kinetics-400 dataset [70] without using any per-video labels. We train for 15 epochs using batch size of 32 across 4 NVIDIA- A5000 GPUs using ADAM-W [76, 77] optimizer on the student model with an initial learning rate of le 5 following a cosine decay schedule. The EMA teacher is updated from student weights after each training iteration with a decay ratio of 2e 4. Unless otherwise specified, this model is used for all downstream task evaluations.
For selected experiments, we additionally perform self-supervised training directly on the train split of each downstream dataset. For Kinetics-400, we follow the same setup described above. In the case of HMDB-51 and UCF-101, we perform self-supervised training for a longer duration of 100 epochs (smaller train sets) leaving all other hyper-parameters unchanged.
Transductive Training:
Our self-supervised setup requires two views single video. We sample two clips from a video following global view generation in [13]. In detail, we select two random intervals from a video. and uniformly sample (cqual time gaps between frames) 8 frames of 224x224 spatial dimensions from within that interval. Standard video augmentations from [78] also applied randomly for each view.
View Generation:
We follow standard linear probing settings on our two downstream datasets to evaluate quality representations learned by our self supe-supe learning phase. We follow the same settings in [13] for fair comparison. Our visual encoder is frozen and a randomly initialized linear layer is trained on the train split of the downstream dataset in a fully-supervised manner. We train for 15 epochs using a batch size of 128 across 4 NVIDIA-A5000 GPUs using ADAM-W [76, 77] optimizer with an initial learning rate of le 3 following a cosine decay schedule. During inference, sample three 224x224 dimensional spatial crops with 8 uniformly spaced frames from each vidco following prior work [13, 36].
Linear Probing:
For zero-shot inference, we project class labels of downstream datasets to our text encoder feature space, and construct an alternate text classifier. Using this text classifier, we make zero-shot predictions. This setup is identical to dot-product similarity based inference in CLIP [4] (explanation in Section 3.3). In line with prior work [13, 36], we feed three 224x224 dimensional spatial crops with 8 uniformly spaced frames sampled from each video to the visual encoder and average its output feature embedding prior to normalized dot-product calculation in the text encoder.
Zero-Shot Inference:
4.1 Linear-Probing Analysis
We first evaluate LSS under linear probing settings HMDB-51 & UCF-101 datasets. Our results (top-1 accuracy) are reported in Table 1. Our proposed LSS achieves state-of-the-an results on both datasets, outperforming prior approaches. Note that MoDist [81] and BraVe [89], both of which additionally utilize video-level optical flow (OF) for self-supervision, are not directly comparable. Still, our LSS showcases competitive performance those, even without such motion information.
4.2 Zero-Shot Analysis
Our LSS provides the additional advantage of zero-shot operation unlike standard video SSL ap- proaches. To this end, we conduct two forms of zero-shot experiments. First, we evaluate LSS on standard zero-shot classification, where our model trained on Kinetics-400 (under SSL settings) is evaluated on the two downstream datasets, HMDB-51 and UCF-101. We report these results (top-1 accuracy) in Table 2. Compared to prior work utilizing per-video labels / captions for training,
7

---

Table 1: Linear Probing HMDB-51 [72] and UCF-101 [79]] We compare method against prior work, reporting top-1 (%) accuracy (following evaluation procedure [13]. 'TTP' stands for image-texi pre-training. Gray shaded methods use additional optical flow (OF) inputs for training. Nevertheless, our performance is comparable to such methods using per-video OF modality information. In contrast, we use generic language modality information and require one-to one language relations with individual videos during training.
Method
Backbone
ITP
TFLOPS
Frames
Epochs
HMDB
UCF
Mcm [34] (BCCV (20)
R2D3D-34
X
30.5
54.1
CoCLR [35] (Neatts (20)
S3D
X
0.07
32
100
52.4
77.8
VideoMoCo [80]cvPR
R(2+1)D
X
17.5
32
200
49.2
78.7
CVRL [36][cVPR 20)
R3D-50
X
3.19
32
800
57.3
89.2
MoDist [81] I] (Aedi 22))
R3D-50
x
3.19
8
100
63.0
91.5
BraVe [38] Joce 220
R3D-50
3.19
16
68.3
92.5
CL [82] (CCV
S3D
X
0.07
32
300
47.3
75.4
MCN [83] (OCCV 221)
R3D
X
3.19
73.1
x
CORP [84] accv
R3D-50
3.19
16
800
58.7
90.2
x
SVT [13 JCVPR (22)
ViT-B
0.59
57.8
VideoMAE [N4](NS '27)
ViT-B
X
0.59
16
800
60.3
84.7
MERLOT [NewNPS
ViT-B
16
55.4
80.1
VATT (NeuttS 21)
ViT-B
32
66.4
87.6
TVTS [87]ivVRR (2))
ViT-B
0.59
16
20
58.4
83.4
LaVil a [88]
ViT-I.
4
5
61.5
88.1
LSS (ours)
VIT-B
0.59
69.2
LSS B (ours)
VIT-B
0.59
20
69.4
91.1
LSS-C (ours)
ViT-B
0.59
Table 2: Zero-shot Transfer on HMDB-51 [72] and UCF-101 [79]: We compare LSS against prior work, reporting top-1 accuracy (%). Mean across three test splits is reported following [55]. 'TP' stands for image-texi pre-training and 'Video Labels' refers using per-video annotations (or paired captions) for supervision during based training. highlight how among directly comparable unsupervised (at video level) approaches as
well as over the CLIP [4] baseline, LSS boosts zero-shot performance.
Method
Backhone
ITP
Video Labels
Frames
HMDB
UCF
TS ISN [90] [90]
GCN
23.2
34.2
E2E [91 1cveR 20)
CNN
32.7
48.0
ER-ZSAR [48]ucev
CNN
35.3
51.8
ActionCLIP [1]]
VIT-B
/
32
40.8
58.3
X-CLIP [12] (22)
VIT-B
/
32
44.6
72.0
VicTR [SS]
VIT-B
/
32
51.0
72.4
ViFi [62]s
VIT-B
32
51.3
76.8
MTE 192] kEcCv (16)
X
19.7
15.8
ASR (F3MM, (17)
CNN
X
X
16
21.8
24.4
7SECOC [94]cvrR
X
X
22.6
15.1
UR [49] JeVer
CNN
X
X
24.4
17.5
CLIP [4]aeM. 21)
VIT-B
X
46.5
69.8
CLIP [4]aCMI. -21)
VIT-B
X
47.2
70.3
LaViLa [88].cvPR
ViT-L
X
18.2
LSS-A (ours)
ViT-B
X
49.5
72.0
LSS-B (ours)
VIT-B
X
50.2
73.8
LSS-C (ours)
ViT-B
X
51.4
74.2
achieve competitive performance. We note that MOV [7] trained under supervised settings with per-video labels and additional audio information is not direct comparison.
In contrast to most prior approaches, LSS uses no video level labels for Kinetics-400 training. In particular, LSS has not seen any labelled videos during its training process. Compared to prior

---

Table 3: Transductive Zero-shot Transfer HMDB-S1 [72], UCF-101 [79], and Kinetics-400 [70]: We report top-1 accuracy (%) following the evaluation procedure [55]. Similar to prior work, we perform dataset specific unsupervised fine-tuning (using our self-supervised objective) on the train-splits of each downstream dataset (no labels used). 'TPp' refers image-text pretaining, and 'Video Labels' refers to video level supervised trainin. Note that CLIP [4] not transductive and is included only for comparison purposes.
Method
Backbone
ITP
Vidco Labels
Frames
HMDB
UCF
K400
UR [49
CNN
28.9
20.1
TS-GCN [90](AАT-11))
GCN
/
16
23.2
34.2
CLIP [4]acML
VIT-B
X
47.2
70,3
49.7
MUST [59]ac1 (23)
VIT-B
48.9
81.1
51.2
L.S (ours)
VIT-B
55.0
75.6
Table 4: Ablation on SSL objectives: We ablate proposed concept distillation (CD) applied on category (CDC) and description (CD") concept spaces and con- cept alignment (CA) using linear probing (IPP) & zero- shot transfer (ZS) HMDB-51 [72] dataset. Since approach cannot be trained without any objective, construct two new baselines from CLIP [4] and SVT [13] (details in Section 4.3). Each proposed component obtains clear improvements over the baselines.
Table 5: Concept Space Ablation: We report zero-shot accuracy (%) on HMDB-51 and UCF-101 datasets. 'K400 only' shows transfer unseen down- stream classes. The labels of K400 overlapping with UCF/HMDB are not used here. A CLIP [4] baseline (modified for video domain without re- traininingn is re- ported in row for comparison purposes. For 2000 words & 10,000 sentences, utilize most common nouns verbs and example sentences action verbs from WordNet [95, 96].
CD
CD
CA
LP
ZS
Method
Action Labels
HMDB
UCF
CLIP
X
X
X
63.9
46.5
CLIP
X
X
X
67.3
47.2
CLIP
47.2
70.3
SVT
X
X
X
62.2
Ours
K400 only (wo U.H)
48.4
71.1
Ours
/
X
X
68.5
48.8
Ours
K400=U+H (A)
49.5
72.0
Ours
/
/
X
69.0
49,2
Ours
(A)+2000 words
48.6
71.8
Ours
V
V
69.2
49.5
Ours
(A)+10K sentences
49.6
71.4
work operating under these settings, LSS achieves state-of-the-an performance on both downstream datasets seen in the bottom half of Table 2.
alternate setting in prior zero-shot work transductive training, where self-supervised learning is perfomed directly on train splits of downstream datasets. Under this setting, we evaluate on all three datasets, Kinetics-400, HMDB-51, and UCF-101, reporting results (top-1 accuracy) Table In the case of HMBD-51 and Kinetics-400, our method achieves kate-of-the-ar performance. For UCF-101, we achieve competitive results, and clear improvements over a CLIP [4] baseline.
4.3 Ablations
We next study the contribution of each component within our approach. All ablative experiments follow the same SSL phase on the Kinetics- train set (as described in Section 4) followed by zero-shot analysis on validation sets of HMDB-51 and UCF-101. In the case of linear probing results, training is conducted following same settings (see Section 4) on the train set of HMDB-51 followed by evaluation on its validation set.
First we ablate each proposed component in Eq. (11) and report results Table 4. In addition to direct CLIP [4] baseline, we construct two additional baselines building off CLIP [4] and SVT [13] for better comparison. CLIPi baseline applies our backbone modifications (for temporal modeling) with no training, which is identical averaging fer-frame visual encoder features. SVT$ baseline performs SVT [13] training with CLIP visual encoder initialization (note that language aligmment breaks and zero-shot operation not possible for this baseline). In comparison to the CLIP baselines, each proposed component, concept distillation in category and description concept spaces as well as concept alignment, leads to improvements. The comparison against SVTS highlights how our SSL approach better preserves language aligned information (contained in CLIP) that is useful even in linear probing. In contrast, the lower performance of SVTS compared to CLIP baselines indicates that generic SSL techniques may be losing useful information contained in CLIP.
SSL Objectives:

---

Table 6: Ablation on Regularization and Significance Weight: The effect of proposed regularization (left) significance weight (right) is demonstrated. UDP regularization (see Section 3.4) is particularly essential to prevent collapse during the SSL training phase. Significance weight (1x) also improves performance.
Method
HMDB
UCF
Method
HMDB
UCF
LSS LSS w/o UDP
48.4
71.1
LSS LSS w/o wr,
48.4
71.1
33.4
54.3
47.2
70.3
Our next focus is on construction of concept spaces. We explore how separately augmenting each concept space affects downstream task performance measured with zero-shot transfer. These results are reported in Table 5. First, focused the category concept space, construct additional category labels using 1000 most common nouns and verbs each (total of 2000) from the WordNet dataset [95, 96]. Next, augment the description concept space using 10,000 sentences. We select these from example sentences provided for action verbs in the WordNet dataset [95, 96.. In these experiments, only the concept distillation objective is applied on these augmented spaces and concept alignment operates only on the base category set. This is because independently augmenting one of the action spaces eliminates their shared and aligned dimensionality. Results for these two settings are reported in row 2 & 3 respectively in Table 5.
Concept Spaces:
We ablate on proposed uniform distribution prior (UDP) which acts as a regularization to prevent collapse (Section 3.4). Our results in Table (left) indicate clear necessity of such regularization prevent collapse during SSL training.
Uniform Distribution Prior:
Significance Weight in Concept Distillation: Eq. (6), we utilize significance weight term, W,,- which represents the confidence of the target concept space projection for given sample. We note how each sample during training a clip sampled from a video (which covers a temporal crop of video). Our intuition for this weight is to act as way of prioritizing more important clips over the less important ones. Our ablations Table (right) indicate usefulness this weight term.
Conclusion
We introduce novel language-based self-supervised learning (SSL) approach for videos, termed LSS, capable of adapting strong language-alignee image representations (CLIP [4]) to the video domain. In particular, we propose two self-distillation based SSL objectives, concept distillation and concept alignment. Our approach trains with no video level labels or paired captions similar to prior video SSL works, but retains language alignment from image CLIP enabling direct zero-shot inference. We demonstrate state-of-the art performance in terms of linear probing with the learned representations on downstream tasks. For zero-shot operation, LSS demonstrates strong performance under both standard and transductive settings, indicating a promising direction for video SSL..
The language alignment of LSS may be limited mostly to per-frame static information since the alignment is derived from image CLIP [4]. LSS cannot distinguish motion based categories like "moving object left to right". Moreover, while containing highly discriminative and generic information at image level, CLIP features lack spatial awareness at an object level [75]. Our proposed model building off these representations in inherently limited in understanding object level motion and interaction within videos. However, recent progress in localization aware CLIP models [75, 97, 98] opens avenues for leveraging their object-centric or pixel-level representations to better model such video motion patterns, opening up interesting future directions. In terms of broader impact, the datasets and pre-trained models possibly contain biases, which may be reflected in LSS. However, our reduced reliance on human annotations may lower additional biases.
Limitations, Future Work, & Broader Impact:
We build a codebase derived from source code of SVT [13] & CLIP [4] and use pre-trained CLIP weights from https: ht///gitt/t com/ openai. All experiments use publicly available datasets. Our action descriptions will be released publicly along with our codebase.
Reproducibility Statement:
Acknowledgements:: We thank Xiang Li for helpful discussions and server setup. We also thank Kumara Kahatapitiya and Cristina Mata for helpful discussions.
10

---

References
[1] Michael S. Ryoo and Jake K. Aggarwal. Recognition of composite human activities through context- free grammar based representation. 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR 066),
2006. 1
[2] Jake K. Aggarwal and Michael S. Ryoo. Human activity analysis. ACM Computing Surveys (CSUR), 43:1
43, 2011.
[3] Madeline Chantry Schiappa, Yogesh Singh Rawat, and Muharak Shah. Self-supervised learning for videos:
A survey, ACM Compuring Surveys, 2022. 1
[4] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark. et al. Learning Transferable Visual Models From Natural Language Supervision. In ICML, pages 8748 8763. PMLR, 2021. 1, 2, 3, 4, 7, 8, 9, 10, 15
[5] Hongwei Xue, Yuchong Sun, Bei Liu, Jianlong Fu, Ruihua Song, Houqiang Li, and Jieho Luo. CLIP-ViP: Adapting Pre-trained Image-Text Model to Video-Language Representation Alignment. arXiv preprint arXiv:2209. 06430, 2022. 1,2
[6] Shen Yan, Tao Zhu, Zirui Wang, Yuan Cao, Mi Zhang, Soham Ghosh, Yonghui Wu, and Jiahui Ya. Video- Text Modeling with Zero-Shot Transfer from Contrastive Captioners. arXiv preprint arXiv:2212.04979 2022. 1,2
[7] Rui Qian, Yeqing Li. Zheng Xu, Ming-Hsuar Yang. Serge Belongie, and Yin Cui. Multimodal Vven-Voeauulary Video Classification via Pre-Trained Vision and Language Models. arXiv preprint arXiv:2207.07666 2022. 2,
[8] Chen Ju, Tengda Han, Kunhao Zheng, Ya Zhang, and Weidi Xie. Prompting Visual- Visual-Language Models for Efficient Video Understanding. In ECCV, pages 105-124. Springer, 2022. 1,2
[9] Hanoona Rasheed, Muhammad Uzair Khattak, Muhammad Maaz, Salman Khan, and Fahad Shahbaz Khan. Fine-tumed CLJP Models are Efficient Video Learners. arXiv preprint arXiv:22 2022. 1, 2
[10] Feng Cheng, Xizi Wang. Jie Lei, David Crandall, Mohit Bansal, and Gedas Bertasius. VindLU: A Recipe for Effective Video-and-Lanuugee Pretraining, arXiv preprint arXiv:2212.0505: 2022. 1,2
[11] Mengmeng Wang. Jiazheng Xing, and Yong Liu. ActionCLIP: A New Paradigm for Video Action Recognition. arXiv preprint 10Xiv:2109.08477 2021. 2,
[12] Bolin Ni, Houwen Peng, Minghao Chen, Songyang Zhang, Gaofeng Meng, Jianlong Fu, Shiming Xiang, and Haibin Ling. Expanding language- -maag pretrained models for general video recognition. In European
Conference on Computer Vision, 2022. 1, 2, 8
[13] Kanchana Ranasinghe, Muzammal Naseer, Salman Hameed Khan, Fahad Shahbaz Khan, and Michael S. Ryoo. Self-supervised video transformer. 2022 IEEE/CVF Conference on Computer Vision and Patern Recognition (CVPR), pages 2864 0104-2274,220 2021. 1, 2, 5, 8, 9, 10, 15
[14] Zhan Tong. Yibing Song, Jue Wang, and Limin Wang. Videomae: Masked autoencoders are data-efficient learners for self-supervised video pre-training ArXiv, abs/2203. 12602, 2022. 2, 8, 15
[15] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In ICCV, 2021. 1, 2, 3, 4, 5,
[16] Biagio Brattoli, Joseph Tighe, Fedor Zhdanov, Pictro Perona, and Krzysztof Chalupka. Rethinking zero-shot video classification: End-to-end training for realistic applications. In CVPR, 2020. 1,
[17]
Xun Xu, Timothy Hospedales, and Shaogang Gong. Transductive zero-shot action recognition by word- vector embedding. IJCV, 2017. 1,2
[18] Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger. Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray. Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever. and Dario Amodei. Language models are sho-shot leamers, 2020. 2, 4, 6, 15
[19] Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, Yifan Du, Chen Yang. Yushuo Chen, Z. Chen, Jinhao Jiang, Ruiyang Ren, Yifan Li. Xinyu Tang, Zikang Liu, Peiyu Liu, Jianyun Nie, and rong Wen. survey of large language models. ArXiv, abs/2303. 18223, 2023. 2
[20] Humza Naveed, Asad Ullah Khan, Shi Qiu, Muhammad Saqib, Saced Anwar, Muhammad Usman, Nick Barnes, and Ajmal S. Mian. A comprehensive overview of large language models. ArXiv, abs/2307.06435. 2023. 2
[21] Michael Mathieu, Camille Couprie, and Yann LeCun. Deep multi-scale video prediction beyond mean square error. In ICLR, 2016.
11

---

[22]
Viorica Paträucean, Ankur Handa, and Roberto Cipolla. Spatio-temporal video autoencoder with differen- tiable memory. In ICLR (Workshop). 2016.
[23] Jacob Walker, Carl Doersch, Abhinav Gupta, and Martial Hebert. An uncertain future: Forecasting from static images using variational autoencoders. In ECCV, 2016. 2
[24]
Nitish Srivastava, Elman Mansimov, and Ruslan Salakhudinov. Unsupervised learning of video representa-
tions using lstms. ICMI,, 2015. 2
[25] Carl Vondrick, Hamed Pirsiavash, and Antonio Torralba. Generating videos with scene dynamics. In NeurIPS, 2016.
[26] Carl Vondrick, Abhinav Shrivastava, Alireza Fathi, Sergio Guadarrama, and Kevin Murphy. Tracking emerges by colorizing videos. In ECCV, 2018. 2
[27] Pulkit Agrawal, Joao Carreira, and Jitendra Malik. Learning to see by moving. In ICCV, 2015.
[28] Ross Goroshin, Joan Bruna, Jonathan Tompson, David Eigen, and Yann LeCun. Unsupervised learning of spatiotemporally coherent metrics. In ICCV, 2015. 2
[29] Phillip Isola, Daniel Zoran, Dilip Krishnan, and Edward H. Adelson. Learning visual groups from co-occurrences in space and time. In ICLR, 2016. 2
[30] Ishan Misra, C. Lawrence Zitnick, and Martial Hebert. Shuffle and learn: Unsupervised learning using temporal order verification. In ECCV, 2016. 2
[31] Xiaolong Wang and Abhinav Gupta. Unsupervised learning of visual representations using videos. In ICCV, 2015. 2
[32] Piergiovanni, Anelia Angelova, and Michael S. Ryoo. Evolving losses for unsupervised video represen- tation learning. In CVPR, 2020. 2
[33] Christoph Feichtenhofer, Haoqi Fan, Bo Xiong, Ross Girshick, and Kaiming He. A large-scale study on unsupervised spatiotemporal representation learning. Arxiv, 2021. 2
[34] Tengda Han, Weidi Xie, and Andrew Zisserman. Video representation learning by dense predictive coding, In ICCV, 2019. 2, 8
[35] Tengda Han, Weidi Xie, and Andrew Zisserman. Sclf-supervised Co-training for Video Representation Learning. NeurlPS, 33:5679-566 (696, 2020. 2, 8
[36] Rui Qian, Tianjian Meng, Boqing Gong, Ming-Hsuan Yang. Huisheng Wang, Serge Belongie, and Yin Cui.
Spatiotemporal contrastive video representation learning. CVPR, 2021. 2, 7, 8, 15
[37] R Devon et al. Representation learning with video deep infomax. Arxiv, 2020. 2
[38] Adrià Recasens, Pauline Luc, Jean- nan-BBptiste Alayrac, Luyu Wang. Florian Strub, Corentin Tallee, Mateusz Malinowski, Viorica Patraucean, Florent Altché, Michal Valko, et al. Broaden your views for self- supervised video learning. ICCV, 2021. 2, 8, 15
[39] Deng Huang, Wenhao Wu, Weiwen Hu, Xu Liu, Dongliang He, Zhihua Wa, Xiangmiao Wu, Mingkui Tan, and Errui Ding. Ascnet: Self-supervised vidco representation learning with appearance-speed consistency. In ICCV, 2021. 2
[40] Peihao Chen, Deng Huang, Dongliang He, Xiang Long, Runhao Zeng, Shilei Wen, Mingkui Tan, and Chuang Gan. Rspnet: Relative speed perception for unsupervised video representation learning. In AAAI, volume 1, 2021.
[41] Nadine Behrmann. Mohsen Fayyaz. Juergen Gall, and Mehdi Noroozi. Long short view feature decompo- sition via contrastive video representation learning. ICCV, 2021. 2
[42] Ishan Rajendra Dave, Rohit Gupta, Mamshad Naycem Rizve, and Mubarak Shah. TCLR: Temporal contrastive learning for video representation. Arxiv, 2021. 2
[43] Jingen Liu, Benjamin Kuipers, and Silvio Savarese. Recognizing human actions by attributes. In CVPR, 2011. 2
[44] Rowan Zellers and Yejin Choi. Zero-shot activity recognition with verb attribute induction. arXiv preprint 2017, 2
[45] Mihir Jain, Jan C Van Gemert, Thomas Mensink, and Cees GM Snock. Objects2action: Classifying and localizing actions without any video example. In ICCV, 201.5. 2
[46] Junyu Gao, Tianzhu Zhang, and Changsheng Xu. T know the relationships: Zero-shot action recognition via two-stream graph convolutional networks and knowledge graphs In AAAI, 2019. 2
[47] Junyu Gao, Tianzuu Zhang, and Changsheng Xu. Learning model relationships for zero-shot video classification. TPAMI, 2020.
[48] Shizhe Chen and Dong Huang. Elaborative Rehearsal for Zero-shot Action Recognition. In ICCV, pages 13638 1368-1337,,200 2021. 2, 6,
12

---

[49] Yi Zhu, Yang Long, Yu Guan, Shawn Newsam, and Ling Shao. Towards Universal Representation for Unseen Action Recognition. In CVPR, pages 9436-9445, 2018. 2, 6.9
[50] Sachit Menon and Carl Vondrick. Visual Classification via Description from Large Language Models. arXiv preprint arXiv:2210.0777
2022. 2
[51] Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoe Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling Up Visual and Vision-Languag Representation Learning With Noisy Text Supervision. In ICML, pages 4904-4916. PMLR, 2021. 2
[52]
Huaishao Luo, Lei Ji. Ming Zhong. Yang Chen, Wen Lei, Nan Duan, and Tianrui Li. CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrieval. Neurocomputing, 508:293-304, 2022. 2
[53] Max Bain, Arsha Nagrani, Gül Varol, and Andrew Zisserman. A CLIP-Hiichhierr's Guide to Long Video Retrieval. arXiv preprint arXiv:2205.0855 2022.
[54] Ziyi Lin, Shijie Geng. Renrui Zhang, Peng Gao, Gerard de Melo, Xiaogang Wang, Jifeng Dai. Yu Qiao, and Hongsheng Li. Frozen CLIP Models are Efficient Video Learners. arXiv preprint 22iv:22008.03500 2022. 2
[55] Kumara Kahatapitiya, Anurag Amab, Arsha Nagrani, and Michael S. Ryoo. Victr: Video-conditioned text representations for activity recognition. ArXiv, abs/2304.02560, 2023. 2, 8,
[56] Antti Tarvainen and Harri Valpola. Mcan teachers are hetter role models: Weight-averaged consistency targets improve semi-supervised deep learning results. In NIPS. pages 1194-1204, 2017.
[57] Kihyuk Sohn, David Berthelot, Chun-Liang Li, Zizhao Zhang, Nicholas Carlini, Ekin Cubuk, Alex Kurakin, Han Zhang, and Colin Raffel. Fixmatch: Simplifying semi-supervised learning with consistency and confidence. In NeurIPS, 2020. 2
[58] David Bertheltt, Nicholas Carlini, Ekin D. Cubuk, Alex Kurakin, Kihyuk Sohn, Han Zhang, and Colin Raffel. Remixmatch: Semi-supervised learning with distribution alignment and augmentation anchoring. In ICLR, 2020. 2
[59] Junnan Li. Silvio Savarese, and Steven H. Hoi. Masked unsupervised self-training for zero-sho shot image classification. ArXiv, abs/2206.0290 02967, 2022. 2,9
[60] Jonathan Kahana, Niv Cohen, and Yedid Hoshen. Improving zero-shot models with label distribution priors. ArXiv, abs/2212.007844 2022. 2
[61] Wei Lin, Leonid Karlinsky. Nina Shvetsova, Horst Possegger, Mateusz Koziński, Rameswar Panda, Rogério Schmidt Feris, Hilde Kuehne, and Horst Bischof. Match, expand and improve: Unsupervised finetuning for zero-shot action recognition with language knowledge. ArXiv, abs/2303.0899 088914 2023.
[62] Hanoona Rasheed, Muhammad Uzair Khattak, Muhammad Maaz. Salman Khan, and Fahad Shahbaz Khan. Fine-tuned clip models are efficient video learners. 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 6545-6554, 2022. 2, 8
[63] Laura Hanu, Anita Lilla Vero, and James Thewlis. Language as the medium: Multimodal video classifica- tion through text only. ArXiv, abs/2309, 107883 2023. 2
[64] Jacob Devlin. Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirec-
tional transformers for language understanding. Arxiv, 2018. 3
[65] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words:
Transformers for image recognition at scale. ICLR, 2021. 3
[66] Gedas Bertasius, Heng Wang, and Lorenzo Torresani. Is Space- Time Attention All You Need for Video Understanding? In ICML, page 4, 2021. 3
[67] Wenhao Wu, Zhun Sun, and Wanli Ouyang. Revisiting Classifier: Transferring Vision-Language Models for Video Recognition. AAAI, 2023. 3.5
[68]
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, al. Bootstrap your own latent: A new approach to sulf-supervised learning. In NeurIPS, 2020.
[69] Randall Balestriero, Mark Ibrahim, Vlad Sobal, Ari S. Morcos, Shashank Shekhar, Tom Goldstein, Florian Bordes. Adrien Bardes, Grégoire Mialon, Yuandong Tian, Avi Schwarzschild, Andrew Gordon Wilson, Jonas Geiping, Quentin Garrido, Pierre Fermandez, Amir Bar, Hamed Pirsiavash, Yann LeCun, and Micah Goldblum. cookbook of self-supervised learning. ArXiv, abs/2304. 12210, 2023. 4,5
[70] Joao Carreira and Andrew Zisserman. Quo vadis, action recognition? A new model and the Kineties dataset. In CVPR, 2017. 4.7,9, 15
[71] Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. UCF101: A dataset of 101 human actions classes from videos in the wild. Arxiv, 2012. 4,
13

---

[72]
Hildegard Kuehne. Hueihan Jhuang, Estibaliz Garrote, Tomaso Poggio, and Thomas Serre. HMDB: A large video database for human motion recognition. In ICCV, pages 2556-2563. IEEE, 2011. 4. 8,9
[73]
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning, 2023. 4, 6
[74] Adrien Bardes, Jean Ponce, and Yann LeCun. Vicreg: Varinnee-ivariine--covrranne regularization for self-supervised learning. ArXiv, abs/2 0bs/2105500906, 2021. 5
[75] Kanchana Ranasinghe, Brandon McKinzie, Sachin Ravi, Yinfei Yang. Alexander Toshev, and Jonathon
Shlens. Perceptual grouping contrastive vision-language models. ICCV, 2023. 6, 10
[76] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In JCLR, 2015. 7
[77] Ilya Loshchilov and Frank Hutter. Decoupled Weight Decay Regularization. ICLR, 2019. 7
[78] Rui Qian, Tianjian Meng. Boqing Gong, Ming- -isng Yang. Huisheng Wang, Serge Belongie, and Yin Cui
Spatiotemporal Contrastive Video Representation Learning. In CVPR, pages 6964 6974, 2021. 7
[79] Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. UCFI01: A Dataset of 101 Human Actions
Classes From Videos in The Wild. arXiv preprint arXiv:1212.0400. 2012. 8,9
[80] Tian Pan. Yibing Song, Tianyu Yang, Wenhao Jiang. and Wei Liu. Videomoco: Contrastive video representation learning with temporally adversarial examples. In CVPR, 2021. 8
[81] Fanyi Xiao, Joseph Tighe, and Davide Modolo. Modist: Motion distillation for self-supervised video
representation learning. Arxiv, 2021. 7,8
[82] Ali Diba, Vivek Sharma, Reza Safdari. Dariush Lotfi, Saquib Sarfraz, Rainer Stiefelhagen, and Luc Van Gool. Vi2elr: Video and image for visual contrastive learning of representation. In ICCV, 2021. 8, 15
[83] Yuanze Lin, Xun Guo, and Yan Lu. Self-supervised video representation learning with meta-contrastive network. In ICCV, 2021. 8
[84] Kai Hu, Jie Shao, Yuan Liu, Bhiksha Raj, Marios Savvides, and Zhiqiang Shen. Contrast and order representations for video self-supervised learning. In JCCV, 2021. 8, 15
[85] Rowan Zellers, Ximing Lu, Jack Hessel, Youngjae Yu, Jae Sung Park, Jize Cao. Ali Farhadi, and Yejin
Choi. Merlot: Multimodal neural script knowledge models. ArXiv, 002222666.066,, 2021. 8
[86]
Hassan Akhari, I Linagzhe Yuan, Rui Qian, Wei-Hong Chuang, Shih-Fu Chang, Yin Cui, and Boqing Gong. Vatt: Transformers for multimodal self-supervised learning from raw video, audio and text. ArXiv, abs/2104. 11178, 2021. 8
[87] Ziyun Zeng, Yuying Ge, Xihui Liu, Bin Chen, Ping Luo, Shutao Xia, and Yixiao Ge. Learning transferable spatiotemporal representations from natural script knowledge. ArXiv, (tts/2209.152880 2022. 8
[88] Yue Zhao, Ishan Misra, Philipp Krahenbuhl, and Rohit Girdhar. Learning video representations from large
language models. ArXiv, abs/2211.00455 2022. 8
[89] Adria Recasens, Pauline Luc, Jean- Sean-Baptiste Alayrac, Luyu Wang, Florian Strub, Corentin Tallec, Maleusz Malinowski, Viorica Păträucean, Florent Altché, Michal Valko, et al. Broaden Your Views for Self-
Supervised Video Learning. In ICCV, pages 1255-1265, 2021. 7
[90] Junyu Gao, Tianzhu Zhang, and Changsheng Xu. I Know the Relationships: Zero-Shot Action Recognition via Two-Stream Graph Convolutional Networks and Knowledge Graphs. In AAAI, pages 8303-8311, 2019.
8,
[91] Biagio Brattoli, Joseph Tighe, Fedor Zhdanov, Pietro Perona, and Krzysztof Chalupka. Rethinking Zero- shot Video Classification: End-to-end Training for Realistic Applications. In CVPR, pages 4613-4623, 2020. 8
192] Xun Xu, Timothy M Hospedales, and Shaogang Gong. Multi- Task Zero-Shot Action Recognition with Prioritised Data Augmentation In ECCV, pages 343-359. Springer, 2016. 8
[93] Qian Wang and Ke Chen. Alternative Semantic Representations for Zero-Shot Human Action Recognition.
In ECML-PKDD, pages 87-102. Springer, 2017. 8
[94] Qin, Li Liu, Ling Shao, Fumin Shen, Bingbing Ni, Jiaxin Chen, and Yunhong Wang. Zero-Shot Action Recognition with Error-Correctngg Output Codes. In CVPR, pages 2833-2842, 2017.
[95] George A Miller. Wordnet: lexical database for english. In Communications of the ACM, pages 39-41. ACM, 1995. 9, 10
[96] Christiane Fellbaum. Wordnet and wordnets. In Keith Brown et al., editors, Encyclopedia of Language & Linguistics, pages 665-670. Elsevier, 2nd edition, 2005. 9, 10
[97] Jilan Xu, Junlin Hou, Yuejie Zhang, Rui Feng, Yi Wang, Yu Qiao, and Weidi Xie. Learning open-vocahulary semantic segmentation models from natural language supervision. ArXiv, 2023. 10
[98] Jiarui Xu, Shalini De Mello, Sifci Liu, Wonmin Bycon, Thomas Breuel, Jan Kautz, and Xiaolong Wang. Group Semantic Segmentation Emerges from Text Supervision. In CVPR, pages 18134-18144, 184, 2022. 10
14

---

APPENDIX
A Prompting details
Our proposed approach utilizes two sets of language based captions: categories and descriptions.
While categories obtained directly from the class labels of datasets (set of unique labels e.g. 400 classes Kinetics-400 dataset), the descriptions are generated automatically utilizing GPT-3 [18]. For each category caption, query GPT-3 to provide set of descriptions and visual characteristics. In detail, we use the following two prompts to generate descriptions and visual characteristics: prompt1 "Give 4 different descriptions for the phrase: {category)?" prompt2 "List visual objects or characteristics usually seen with the action: {category]?" The resulting two sets of captions using our text-encoder, and a single average text embedding is computed. This averaged embedding used the description basis vector for that category. Also, the resulting dataset containing these category-descriptee pairs is made available publicly.
converted to text embeddings
B Additional Experiments
Linear Probing Evaluation: We present more results for linear probing in Table (left). Our proposed LSS improves over the baseline achieving competitive performance on Kinetics-400.
Text-to-video retrieval: An important characteristic of CLIP [4] is its retrieval ability across both language and visual modalities. In order to verify proposed LSS retains these strengths, we run experiments on MSR-VTT text-to-video retrieval benchmark. We demonstrate how LSS improves over our baseline CLIP, reporting these results in Table 7 (top-right).
Charades Evaluation: We explore alternate task of zero-shot multi-label classification the Charades video dataset. We report mAP results for this task Table 7 (bottom-right) as an additional point of comparison.
Table 7: We report top-1 (%) accuracy the Kinetics-400 [70] validation for linear probing evaluation (left), All models are pre-trained on training set Kinetics-400 dataset. also report a CLIP baseline for comparison purposes. Performance of our proposed approach is on-par with prior state-of-the- and showcases improvements over our baseline method. We also report retrieval scores (top-right) for MSR-VTT and classification mAP (bottom-right) for Charades dataset.
Method
Backbone
Acc (%)
Method
R@1 I
R@5
R@ 10
CVRL [36] (CVPR221)
R3D-101
67.6
CLIP [4]
30.6
54,4
64.3
BraVe [38] (0Cv121)
R3D-50
66.7
LSS (ours)
33.8
58.2
70.3
vi'CLR [82] (CCV
S3D
63.4
CORP [84] acCV 201)
R3D-50
66.6
SVT [13] (CVPR (2))
ViT-B
68.1
VideoMAR [14] (Nawdes 122)
ViT-B
61.3
Method
Classification mAP
CLIP [4]
VIT-B
66.4
CLIP [4]
19.7
LSS (ours)
ViT-B
67.3
LSS (ours)
23.1
15


Reviews:
[{"review_id": "kQbz74tYhT", "reviewer": "Reviewer_Cb1q", "summary": "The paper proposes a novel language-based self-supervised learning framework (LSS) for video representation learning. It extends the self-distillation based SSL approaches like BYOL and SimSiam by replacing the randomly initialized project network by the text classifier defined by language embeddings extracted from the image CLIP text encoder. With two novel self-supervised learning objectives, the pretrained video model retains and improves transferability and generality of image CLIP representations better in comparison to existing video SSL methods. LSS achieves state-of-the-art results under linear probing settings and competitive zero-shot transfer performances on HMDB-51 and UCF-101.", "questions": "The key idea of the paper, replacing the project network with the text classifier really makes sense and is very interesting. I will lean towards acceptance if the authors address my concerns in the rebuttal.", "limitations": "The authors have adequately addressed the limitations and potential societal impact of the work.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "+ The paper is overall well written and easy to follow.\n+ The idea of replacing the project network with the text classifier defined by image CLIP embeddings is really interesting and makes sense. The video representation learning suffers from relatively expensive and noisy annotations; we can easily distill knowledges obtained from abundant and diverse image-based datasets using LSS.\n+ The experimental results show the high transferability and generality of the proposed method, LSS.", "weaknesses": "- LSS uses action categories of Kinetics-400, UCF-101 and HMDB-51 for defining the action category and description concept spaces. However, if those action concept spaces can be used only for pretraining on one of the three datasets used for concept space construction, this means that we can only use manually annotated action recognition datasets for pretraining. This limits the data scalability of LSS. The authors should provide experimental results with pretraining on more general, non-labeled video data using the same action concept spaces. Otherwise, for using abundant web videos without labeling, the authors should come up with the action space construction method without manually annotated action categories.\n- Please fix the typos; e.g., the l2 norm, not the squared l2 norm, should be used in Eq. (1) and (3) for normalization.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for positive comments and helpful feedback. We address all concerns below.\n\n1. `Pre-training without dataset labels:` As we also described in the main rebuttal, we run 2 additional experiments for variants of our method that use NO textual labels from any datasets. Results (LSS-B & LSS-C in Tables 1, 2 of rebuttal PDF) showcase equally strong performance by these variants. We also report the same linear probing top-1 accuracy below.\n    |    Method    | ITP | HMDB-51 |  UCF-101 |\n    |------------------|:---:|:----:|:----:|\n    | LSS-A (ours-original) | yes | 69.2 | 91.0 |\n    | LSS-B (ours-new) | yes | **69.4** | **91.1** |\n    | LSS-C (ours-new) | yes | 69.1 | 90.8 |\n\n    Zero-shot top-1 accuracy compared to a CLIP baseline are also reported below again for quick reference. \n    | Method | Action Labels    | HMDB | UCF  |\n    |--------|------------------|------|------|\n    | CLIP   | -                | 47.2 | 70.3 |\n    | Ours   | K400+U+H (LSS-A)     | 49.5 | 72.0 |\n    | Ours   | GPT labels (LSS-B)   | 50.2 | 73.8 |\n    | Ours   | I-VLM labels (LSS-C) | **51.4** | **74.2** |\n\n    These results indicate that LSS can be used with pre-training video datasets that contain no action annotations.   \n\n2. `Label-free action space construction:` The LSS-B variant uses GPT-3 to generate a set of 2000 common activity labels. This is reduced to 1000 by eliminating labels that are similar in feature space of the CLIP text encoder and then used to construct the action space. The LSS-C variant generates a label set using only videos from the training dataset. We use PCA based clustering to identify 2000 representative videos from a randomly sampled subset of our training dataset and then use image-captioning models on video center frames to generate a diverse set of 2000 action labels. This is further reduced to 500 eliminating labels that are similar in feature space of the CLIP text encoder and then used to construct the action space. We will elaborate these details further in our revision.   \n\n\n3. `Typos:` Thank you for pointing out these typos - we have fixed them. \n"}, {"review_id": "mytJYC0BWj", "reviewer": "Reviewer_WHFS", "summary": "The paper presents a method to adapt a vision-language model (CLIP) to represent videos. \nThe method extends the image encoder of CLIP to a video encoder via factored space-time attention. \nThe paper introduces a self-distillation-based objective to adapt the video encoder to train on unlabelled videos (no video captions are required).\nThis self-distillation is performed in a so-called \"action concept space\" which results from projecting visual embeddings into the space spanned by 0-shot action classifiers or text embeddings of action descriptions.\nThe approach is evaluated on Kinetics 400, UCF101, and HMBD in 0-shot and linear action classification experiments.", "questions": "I would appreciate it if the authors could address my concerns listed in the weaknesses above. Most importantly, I would like to know:\n- How does the model perform for text-to-video retrieval?\n- How does the model generalize to actions not used during pre-training? For example, what if Tab 5 only consists of K400 classes? \n\nAdditionally, I'm wondering:\n- What is the importance of w_s (Eq 5)?", "limitations": "Some of the limitations I see (see weaknesses) have not been addressed in the paper.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The adaptation approach does not require additional video-level captions (only knowledge of action categories in the training data is required)\n- The method performs well in \"zero-shot\" and linear probing experiments in the considered action datasets\n- The use of fixed projections obtained through 0-shot classifiers for distillation is interesting and seems effective when the classes observed in transfer are known\n- The paper is overall well written and the method well presented", "weaknesses": "- The experiments are limited to 0-shot and linear classification of the same action classes also observed and used in the adaption training. It is unclear if the method can generalize to new actions (ones not used to define the action concept spaces)\n- The experiments also lack any text-to-video retrieval benchmarks. These would be essential to demonstrate the claim that the method \"preserves and improves the strengths of CLIP ... for video operation\" L49\n- The comparisons to most methods in Table 1 are unfair since they are all fully self-supervised (no captions used in pre-training), and most are trained exclusively on videos (much less effective training data). Also, it is unclear why many methods are included without any performance numbers. Instead, it would be better to compare to other CLIP-based methods as in Tab 2 in this benchmark as well\n- The difference in Tab 2 and 3 suggests the method is very sensitive to the set of actions used to define the projection space", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the detailed feedback and address all comments below. \n\n1. `Text-to-video retrieval:` We run experiments on MSR-VTT text-to-video retrieval benchmark to demonstrate how LSS improves over our baseline CLIP. The performance increase is significant and consistent to our prior results. \n\n    | Method | R@1  | R@5  | R@10 |\n    |--------|------|------|------|\n    | CLIP   | 30.6 | 54.4 | 64.3 |\n    | LSS    | 33.8 | 58.2 | 70.3 |\n\n2. `Generalize to actions not used during pre-training:` We run 3 new experiments to demonstrate how LSS generalizes to unseen actions. First, pre-training on K400 labels only (action classes overlapping with UCF/HMDB are removed here) with UCF/HMDB evaluation is reported in Table 2 (rebuttal PDF) and below (zero-shot top-1 accuracy). \n\n\n    | Method | Action Labels | HMDB | UCF  |\n    |--------|---------------|------|------|\n    | CLIP   | -             | 47.2 | 70.3 |\n    | Ours   | K400 only (w/o U, H)     | 48.4 | 71.1 |\n    | Ours   | K400+U+H (original)     | 49.5 | 72.0 |\n\n    Next we show results for experiments we run on 2 variants introduced in main rebuttal, LSS-B and LSS-C, that use **no dataset textual labels** for pre-training. These results are reported in Table 1 & 2 (rebuttal PDF) and surpass our default setting. We also report these below (zero-shot top-1 accuracy). \n\n    | Method | Action Labels    | HMDB | UCF  |\n    |--------|------------------|------|------|\n    | CLIP   | -                | 47.2 | 70.3 |\n    | Ours   | K400+U+H (original)     | 49.5 | 72.0 |\n    | Ours   | GPT labels (LSS-B)   | 50.2 | 73.8 |\n    | Ours   | I-VLM labels (LSS-C) | 51.4 | 74.2 |\n\n\n3. `Importance of w_s:` This term represents confidence of the target concept space projection for a given sample. Since each basis of the concept space corresponds to action categories, if a sample is more aligned to a single basis, we assume higher confidence in that sample, leading to higher w_s value (max element of softmax normalized projected vector). And the reverse when a sample is equally aligned to a number of basis axes. \nSince each sample during training is a clip sampled from a video (which covers a temporal crop of video), our intuition for this weight is to act as a way of prioritizing more important clips over the less important ones. We also include ablation for w_s below. \n\n    | Method      | HMDB | UCF  |\n    |-------------|------|------|\n    | Default LSS | 48.4 | 71.1 |\n    | w/o w_s     | 47.2 | 70.3 |\n\n\n4. `Unfair comparison in Table 1:` We update Table 1 (please see main rebuttal PDF) to include comparisons to CLIP based methods using image-text pre-training (ITP). Results show how our proposed LSS performs significantly better than prior ITP approaches and retains strengths of CLIP. Missing performance for some methods was a latex typo - we have fixed that, thanks for the pointer! We add updated Table 1 below too (linear probing top-1 accuracy). \n\n    |    Method    | ITP | HMDB-51 |  UCF-101 |\n    |------------|:---:|:----:|:----:|\n    |      SVT     |  no | 57.8 | 90.8 |\n    |   VideoMAE   |  no | 60.3 | 84.7 |\n    |    MERLOT    | yes | 55.4 | 80.1 |\n    |     VATT     | yes | 66.4 | 87.6 |\n    |     TVTS     | yes | 58.4 | 83.4 |\n    |    LaViLa    | yes | 61.5 | 88.1 |\n    | LSS-A (ours-original) | yes | 69.2 | 91.0 |\n    | LSS-B (ours-new) | yes | **69.4** | **91.1** |\n    | LSS-C (ours-new) | yes | 69.1 | 90.8 |\n\n\n"}, {"review_id": "C2SWHLTECy", "reviewer": "Reviewer_mCow", "summary": "The paper introduces a language-tied self-supervised learning approach to adapt an image CLIP model to the video domain. The method employs two video-specific self-supervised learning objectives: concept distillation and concept alignment, for training the model. The authors showcase that the proposed method achieves good zero-shot and linear probing performance on three action recognition benchmarks.", "questions": "One of the main issues with this paper is the misleading setting employed by LSS. As mentioned in the weaknesses section, LSS is not a purely self-supervised method as it requires some dataset-level labels and also uses CLIP pre-trained weights. It is important to differentiate this setting from traditional SSL settings and explain the practical utility of the proposed LSS setting in applications.", "limitations": "The authors have addressed the limitations.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The motivation behind using language for video self-supervised learning is good as it addresses the existing challenges associated with video datasets and holds the potential to offer effective solutions.\n2. The paper presents good zero-shot performance\n3. The paper is well written and the overall message is well understood.", "weaknesses": "1. While this paper refers to itself as a self-supervised method, it relies significantly on labeled information. For instance, the text classifier utilizes language embeddings derived from the dataset categories. Additionally, the construction of action concept spaces and category concept spaces takes into account the awareness of dataset categories. This information leak introduces a potential unfairness in comparing the proposed self-supervised learning method with other SSL methods.\n2. Consider adding experiments that involve removing the awareness of dataset-level labels to align with other SSL methods.\n3. It is not fair to directly compare the proposed method, which utilizes pre-trained CLIP weights, with other SSL methods that are trained from scratch. This distinction should be discussed, and the attribute of pre-training should be added to Table 1.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the helpful comments and address all concerns below.\n\n1. `Experiments removing dataset-level label awareness:` We eliminate the need for dataset-level labels in 2 additional variants (GPT based category generation and image VQA based labels) and run experiments for these variants. Results (presented in rebuttal PDF Table 1, 2) demonstrate how LSS can be trained without any textual information from video datasets, removing all dataset-level label awareness. We also include these results for linear probing (top) and zero-shot (bottom) top-1 accuracy below. \n    |    Method    | ITP | HMDB-51 |  UCF-101 |\n    |:------------:|:---:|:----:|:----:|\n    | LSS-A (ours) | yes | 69.2 | 91.0 |\n    | LSS-B (ours) | yes | **69.4** | **91.1** |\n    | LSS-C (ours) | yes | 69.1 | 90.8 |\n\n    | Method | Action Labels    | HMDB | UCF  |\n    |--------|------------------|------|------|\n    | CLIP   | -                | 47.2 | 70.3 |\n    | Ours   | K400+U+H (LSS-A)     | 49.5 | 72.0 |\n    | Ours   | GPT labels (LSS-B)   | 50.2 | 73.8 |\n    | Ours   | I-VLM labels (LSS-C) | **51.4** | **74.2** |\n\n\n2. `Fair comparison:` We update Table 1 (in main rebuttal PDF) to distinguish methods using CLIP pre-training and include more methods using such pre-training.  LSS performs the best among prior work using CLIP pre-training, showcasing our unique strengths. \n\n3. `Differentiate from traditional SSL:` LSS uses image-text pre-training additionally compared to traditional SSL approaches. However, LSS contains zero-shot capabilities unlike prior SSL works. This unique ability creates multiple practical utilities of LSS over prior SSL works.  We will update our final manuscript to discuss this clearer. \n"}, {"review_id": "vUk311XrTu", "reviewer": "Reviewer_aWeV", "summary": "The paper proposes a new self-supervised approach to adapt image-level CLIP features to video. The key idea is to use a teacher-student self-supervised learning framework, and distill the knowledge in the action concept space, derived from text action concepts using the CLIP's text encoder. The resulting framework produces SoTA self-supervised video features.", "questions": "in line 79, the authors probably meant “self-supervised learning” instead of “semi-supervised”? Semi-supervised assumes you have access to some labeled data.", "limitations": "It is not very clear whether knowing the text action concepts is important for the method to work well. This may be a potential limitation.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The work tackles an important problem - adapting the CLIP visual encoder to video. Doing it without supervision is nice a bonus here.\n- The approach is simple and sound.\n- The evaluation is extensive and the results are strong.", "weaknesses": "- Since the set of concept vectors is taken from Kinetics-400  UCF-101 and HMDB-51, where the evaluation is performed, it is a little unfair to call the method completely unsupervised. While it is true that no video labels are used during training, the training process does use the knowledge of the actions vocabulary, which would  aid the evaluation. To avoid this shortcoming, the authors may consider constructing the action concepts in a dataset-agnostic way.\n- I could not find the ablation of the uniform distribution prior regularization. It is important to understand the influence of that prior on the training. Could the authors please include this?\n- The writing could be significantly improved.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the positive feedback and address all suggestions below.\n\n1. `Use dataset-agnostic action concepts:` We take the reviewers advice and develop two alternate strategies to construct action concepts in a dataset-agnostic way: GPT based category generation (LSS-B) and image VQA based labels (LSS-C). These methods have no reliance on dataset textual information and obtain results (rebuttal PDF Table 1, 2) on par with our default setting (LSS-A). We also include these results for linear probing (top) and zero-shot (bottom) top-1 accuracy below for quick reference.\n \n    |    Method    | ITP | HMDB-51 |  UCF-101 |\n    |:------------:|:---:|:----:|:----:|\n    | LSS-A (ours) | yes | 69.2 | 91.0 |\n    | LSS-B (ours) | yes | **69.4** | **91.1** |\n    | LSS-C (ours) | yes | 69.1 | 90.8 |\n\n    | Method | Action Labels    | HMDB | UCF  |\n    |--------|------------------|------|------|\n    | CLIP   | -                | 47.2 | 70.3 |\n    | Ours   | K400+U+H (LSS-A)     | 49.5 | 72.0 |\n    | Ours   | GPT labels (LSS-B)   | 50.2 | 73.8 |\n    | Ours   | I-VLM labels (LSS-C) | **51.4** | **74.2** |\n \n    This highlights how LSS can operate without knowing the textual action concepts of the training or downstream datasets. \n\n2. `Add uniform distribution prior (UDP) ablation:` This UDP regularization is crucial for stable training. Without it the pre-training stage leads to collapse. We include ablations here and in rebuttal PDF (Table 3). \n\n    | Method      | HMDB | UCF  |\n    |-------------|------|------|\n    | Default LSS | 48.4 | 71.1 |\n    | w/o UDP      | 33.4 | 54.3 |\n\n\n3. `Fix typos and writing:`\nThank you for pointing out the typo on L79 - we will fix that to be *self-supervised learning*. We will also revise our final manuscript further to improve our writing. \n"}, {"review_id": "qDKUO6qhVz", "reviewer": "Reviewer_vhEc", "summary": "This paper proposes to transfer CLIP to the video domain for self-supervised learning. Textual features of video categories are used to obtain text classifiers and fixed during pre-training to obtain transferable information. Multiple complementary loss functions are designed for pre-training. Experimental results on three datasets demonstrate the effectiveness of the method.", "questions": "Please refer to Weaknesses for more details.", "limitations": "Limitations have been listed.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "Pros:\t\n1. A new paradigm is proposed to transfer the knowledge of CLIP to the video domain for self-supervised learning. \n2. Impressive performance has been achieved on multiple datasets. \n3. The results of the ablation experiments demonstrate the effectiveness of the method.", "weaknesses": "Cons:\n1. The authors claim that they did not use labeled or captioned videos in the paper. But they used the labels of the Kinetics-400, UCF101, and HMBD51 data sets during training. Does this violate the self-supervised setting?\n2. Unfair comparison. The proposed LSS leverages the pretrained CLIP for weight initialization while existing methods such as SVT，VideoMAE do not. \n3. LSS is pre-trained on the Kinetics dataset. Why did the label of HMBD51 and UCF101 be added to the pre-training? Does this break the downstream transfer setup? Since the target label has been leaked in the pre-training.\n4. The fully-finetuned experiments and SSv2 results are missing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the positive comments and address all concerns below.\n\n1. `Using dataset labels for training:` We understand this shortcoming and run two new experiments that use no textual labels from datasets for our action concept spaces. We report these results in main rebuttal PDF (LSS-B & C in Tables 1, 2) and highlight the on-par performance of these additional experiments. We also include these results for linear probing (top) and zero-shot (bottom) top-1 accuracy below for quick reference.\n \n    |    Method    | ITP | HMDB-51 |  UCF-101 |\n    |:------------:|:---:|:----:|:----:|\n    | LSS-A (ours) | yes | 69.2 | 91.0 |\n    | LSS-B (ours) | yes | **69.4** | **91.1** |\n    | LSS-C (ours) | yes | 69.1 | 90.8 |\n\n    | Method | Action Labels    | HMDB | UCF  |\n    |--------|------------------|------|------|\n    | CLIP   | -                | 47.2 | 70.3 |\n    | Ours   | K400+U+H (LSS-A)     | 49.5 | 72.0 |\n    | Ours   | GPT labels (LSS-B)   | 50.2 | 73.8 |\n    | Ours   | I-VLM labels (LSS-C) | **51.4** | **74.2** |\n \n    This highlights how LSS can operate without using dataset labels for training. \n\n\n2. `Unfair comparison`: We have updated Table 1 to compare against methods using CLIP pre-trained weights for initialization (reported in main rebuttal PDF). We confirm that LSS also outperforms these prior works taking advantage of CLIP pre-training.\nWe also highlight the zero-shot abilities inherent to our LSS different from traditional SSL approaches. \n\n\n3. `Adding UCF & HMDB to pre-training action set:` We thank the reviewer for this suggestion. Our experiments with LSS-B and LSS-C confirms that adding downstream dataset action classes is not necessary for strong performance. We will highlight this more in the final version of our paper. The motivation behind adding these UCF & HMDB labels in our initial experiment was to confirm capabilities of LSS given all possible action categories. Our further experiments as suggested above illustrate that even without such downstream labels, LSS achieves strong performance better than prior work. These results are presented in Table 2 (rebuttal PDF). We repeat these results below for quick reference.\n\n    | Method | Action Labels | HMDB | UCF  |\n    |--------|---------------|------|------|\n    | CLIP   | -             | 47.2 | 70.3 |\n    | Ours   | K400+U+H      | 49.5 | 72.0 |\n    | Ours   | K400 only     | 48.4 | 71.1 |\n    | Ours   | GPT labels (LSS-B)   | 50.2 | 73.8 |\n\n\n4. `Experiment on more datasets:` \nWe report mAP results for *zero-shot multi-label classification* task on Charades video dataset below as an additional point of comparison:  \n\n    | Method |  Charades mAP |\n    |:------:|:----:|\n    |  CLIP  | 19.7 |\n    |  LSS-B | 23.1 |\n\n    In the case of motion heavy datasets like SSv2, we note that LSS poses limitations (since it has no language awareness for motion). We hope to explore this direction further in future work. \n\n"}]

Author rebuttal:
We thank all reviewers for positive comments: results show high transferability and generality of method (R-Cb1q); interesting and seems effective for transfer on known classes (R-WHFS); presents good zero-shot performance, holds the potential to offer effective solutions (R-mCow); tackles an important problem, extensive evaluation and strong results (R-aWeV); impressive performance on multiple datasets, ablations demonstrate effectiveness (R-vhEc). 


We discuss our two main modifications in response to concerns raised by reviewers below. 

1. `Concern: Reliance on textual action categories of datasets`
  * We run experiments on two additional variants of LSS that do NOT rely on any dataset labels. They utilize different concept spaces (variants B & C in rebuttal PDF Table 1) that use no textual class information from training or downstream datasets. These obtain similar performance (superior to prior work) in both linear probing and zero-shot settings, highlighting how **proposed LSS can operate without access to textual action categories of datasets**. 


2. `Concern: Fairer comparison against prior SSL work`
  * Following suggestions from the reviewers, we modify Table 1 (in rebuttal PDF) to explicitly highlight our use of image-text pretraining (ITP). We also include comparisons to related works that use image-text pretraining. Results show how our proposed LSS performs significantly better than these image-text pretraining approaches. 
  * We also reiterate the additional zero-shot capabilities of our method that traditional SSL do not possess. 

Please refer to attached PDF for tables. Further rebuttals are written as responses to each review addressing the concerns raised by reviewers. 

Meta-review: All of the reviewers recommended the paper. They appreciated the strong zero-shot results and well-written justifications in the paper. The authors should be sure to update the paper based on the rebuttals.
Decision: Accept (poster)