SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Kanchana Ranasinghe
kranasinghe@cs.stonybrook.edu
Michael Ryoo
Stony Brook University, yoo@cs.stonybrook.com
Recent contrastive language image pre-training has led to learning highly transferable and robust image representations. However, adapting these models to video domain with minimal supervision remains an open problem. We explore a simple step in that direction, using language tied self-supervised learning to adapt an image CLIP model to the video domain. A backbone modified for temporal modeling is trained under self-distillation settings with train objectives operating in an action concept space. Feature vectors of various action concepts extracted from a language encoder using relevant textual prompts construct this space. A large language model aware of actions and their attributes generates the relevant textual prompts. We introduce two train objectives, concept distillation and concept alignment, that retain generality of original representations while enforcing relations between actions and their attributes. Our approach improves zero-shot and linear probing performance on three action recognition benchmarks.
Actions in videos are defined by individual objects, their relationships, and interaction [1, 2]. Video self-supervised learning focuses on discovering representations aware of such action attributes directly from video content with no human supervision [3]. Particularly in the case of videos, where manual human annotation can be both expensive and noisy, such self-supervised approaches are invaluable.
A recent variant of self-supervision explores learning with loosely paired image-caption pairs, leading to highly transferable and robust representations such as CLIP [4]. These approaches obtain zero-shot performance often comparable to fully-supervised methods. However, their counterparts in the video domain [5, 6, 7, 8, 9, 10, 11] do not exhibit the same generality. In fact, some approaches training CLIP on videos [11, 12] perform subpar to image-CLIP under zero-shot settings (see Table 2). Such behaviour can be attributed to lesser availability and more noisy nature of labelled (or paired caption) video datasets [3]. This motivates exploration into self-supervised learning (SSL) techniques that can learn from videos under less supervision while utilizing existing image CLIP [4] like representations. Existing state-of-the-art video SSL approaches [13, 14] learn highly transferable representations from videos, but combining these with image CLIP representations is not straightforward. In fact, despite methods like SVT [13] being able to utilize image SSL representations [15] for weight initialization to achieve better performance, using image CLIP representations instead for weight initialization leads to performance subpar to image CLIP (see Table 4). This raises necessity for alternate video SSL approaches compatible with CLIP like image representations and is our key motivation.
In this work, we explore self-supervised learning techniques that adapt image CLIP models [4] to video domain under entirely self-supervised settings, dependent on no form of video level labels or captions. Under this setting, natural language can still provide strong cues regarding attributes that compose an action category [16, 17]. We leverage this idea to propose a novel language-based self-supervised learning objective. Following a standard self-distillation and multi-view based SSL
formulation [15, 13], we introduce language aligned feature spaces, action concept spaces, where our SSL objectives operate. Large-language models (LLMs) [18], given their extensive world knowledge [19, 20], serve as an ideal tool to generate necessary textual concepts for these spaces. We also introduce regularization suitable for our language aligned SSL objective to prevent collapse during training. Our resulting framework is termed Language-based Self-Supervision, or LSS.
In contrast to existing video self-supervised learning approaches [13, 14], our proposed LSS retains and improves transferability of image CLIP representations much better (see Tables 1 and 4). Additionally, our language aligned learning framework allows direct zero-shot operation on downstream tasks. Moreover, unlike video CLIP methods with similar zero-shot capabilities [5, 6, 7, 8, 9, 10, 11] that utilize per-video labels / captions for learning, our proposed LSS requires only videos for training.
We summarize our key contributions as follows:
- Self-supervised learning paradigm capable of retaining and improving strengths of CLIP like image representations for video domain operation
- Video specific self-supervised learning objectives, namely concept distillation and concept alignment, that enforce relations between action categories and their visual attributes
- Novel language-based video self-supervised learning framework operating zero-shot on downstream action classification tasks without requiring per-video labels / captions for training
Experiments on action recognition datasets showcase state-of-the-art performance for our learned representations under linear-probing, standard zero-shot, and transductive zero-shot settings.
Self-Supervised Learning in Videos was initially dominated by pretext tasks specific to the video domain [21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32]. Recently a shift to contrastive losses led to [33, 34, 35, 36, 37, 38] with some variants focused on video specific view generation [39, 40, 41, 42, 13]. An alternate direction has been masked auto-encoders [14]. To the best of our knowledge, existing video self-supervised learning (SSL) approaches operate purely within the visual domain. By video SSL, we refer to methods that utilize only videos with no paired captions (or labels) for each video during training. In contrast, our proposed LSS learns purely from videos in a self-supervised manner, integrating pre-trained language-image models to learn language aligned representations.
Zero-shot Action Recognition began with manual attribute and feature selection [43, 44, 45, 46, 47] with later works utilizing action word embeddings [16, 17]. The idea of connecting action categories with elaborate descriptions of those actions, within language embedding spaces [48, 49] has been a next step and is closely related to our work. This idea is also explored in image domain to boost zero-shot performance [50]. While our work is inspired by such approaches, in contrast, we use relations between such actions and descriptions as self-supervised signals for learning. Recent image CLIP models [4, 51] are another line of works achieving strong performance on some video classification tasks, with only single frame processing. Multiple approaches build on image CLIP [4] to learn video level representations [11, 52, 53, 54, 12, 55] under fully-supervised settings. While achieving strong performance on the training datasets, their zero-shot improvements over CLIP are minimal or even subpar (see Table 2). Therein, LSS focuses on zero-shot performance under self-supervised settings while retaining (and improving) the generality of the representation space.
Self-training methods leverage pseudo-labels on unlabeled data [56, 57, 58] for supervised fashion training. Recently they have been combined with CLIP models for zero-shot operation [59, 60]. While inspired by such self-training approaches, our proposed LSS differs in its continuous feature space self-distillation, language-based relations enforcing, video domain operation, and cross-dataset transfer for zero-shot operation.
Adapting image-CLIP models to video under fully-supervised settings has gathered much interest [5, 6, 7, 8, 9, 10]. Expanding backbones for temporal modeling, multi-modal fusion, secondary training objectives, partial parameter updates, and scaling-up data are key ideas explored [55, 10]. In contrast, LSS is a first to operate under self-supervised settings using no video annotations.
Contemporary work in [61] adapts image CLIP features to video tasks label free similar to our work. ViFi-CLIP [62] introduces zero-shot action recognition benchmarks and similarly adapts CLIP to videos retaining generality. Using LLMs for action recognition is also explored in [63].
Figure 1: Our overall setup contains three components: visual teacher model (green), visual student model (red), and language model (blue). We utilize the text encoder of CLIP as our language model and extract concept vectors relevant to action labels and descriptions of those actions. A visual encoder (containing a space-time backbone) is partially initialized with the visual encoder of CLIP and used to obtain sample specific features. The generated concept vectors are used to project these features to a concept space where our proposed concept distillation and concept alignment losses are applied.
3 Language-based Self-Supervision (LSS)
In this section, we present our proposal, Language-based Self-Supervision (LSS). The generality and robustness of shared image-language representation spaces such as that of CLIP [4] allow interesting manipulations of visual representations using language. We explore such manipulations under the setting of visual self-supervised learning focusing on video understanding. Self-supervised objectives can operate within a latent space constructed with language, retaining language alignment of learned visual representations. This allows better interpretability of representations as well as zero-shot inference. We discuss the four key components of our approach: backbone architecture, concept distillation objective, modifications to avoid collapse, and concept alignment objective.
3.1 Backbone Architecture
Our approach introduces a text classifier to self-distillation based SSL works [15, 13], in place of the projector network. Given a data sample  $x$ , let  $x_{1},x_{2}\in \mathbb{R}^{(C,T,H,W)}$  be two augmented views generated using video specific transformations following [13], where  $C = 3,T = 8,H = W = 224$  are channel, time, and spatial dimensions respectively.
Visual Encoder: A visual encoder,  $\theta_v$ , processes  $x_i$  to produce feature  $f_i \in \mathbb{R}^{768}$ . We utilize the pre-trained image encoder of CLIP [4] expanded for temporal modelling using factorized space-time attention. The vision transformer variant of CLIP is selected to allow our factorized space-time attention. In particular, we use ViT-B/16 architecture for the image encoder, in which for a given augmented view with  $H = W = 224$  and  $T = 8$ , each transformer block processes 8 temporal and 196 spatial tokens separately in sequential order, and the embedding dimension of each token is  $\mathbb{R}^{768}$ . In addition to the input tokens from the data sample, one classification token [64, 65] serves as the final feature vector output by the network, namely  $f_i$ , which is common to the CLIP image encoder. This classification token is inflated and processed suitably following [66] to accommodate our modifications for factorized space-time attention. We follow [66] to zero-initialize additional time-attention parameters, achieving outputs identical to the pre-trained CLIP image encoder at start of training.
Text Classifier: Inspired by [67], a set of  $n$  language embeddings extracted from the CLIP text encoder,  $\theta_t$ , are used to construct the weight parameter of a linear layer (with no bias term), which
Figure 2: We illustrate a toy concept space constructed with the three action concepts: run, swim, and walk. In this example, the text classifier projects visual feature  $f_{i}$  into the 3-dimensional toy concept space to produce  $\tilde{f}_{i}$ .
we call our text classifier,  $\theta_c$ . The role of this text classifier is to project visual features  $f_i$  to a vector space defined by those  $n$  embeddings, producing  $\tilde{f}_i \in \mathbb{R}^n$ . Next we discuss these vector spaces (referred to as action concept spaces) and the text classifier module in detail.
3.2 Action Concept Spaces
Self-supervised learning approaches following exponential moving average (EMA) based self-distillation [68, 15, 13] utilize a projector network (MLP) to operate in a higher dimensional feature space. This is expected to minimize train-test domain gaps, handle noisy positive sample pairs, and better discriminate nuanced feature differences [69]. Focused on these notions, we propose an alternate concept space composed of a set of basis vectors defined by language-based action concepts. Our language-based self-supervision objectives operate within such concept spaces.
Concept Spaces: Building off the assumption that text encoder features capture subtle differences between distinct actions categories, we hypothesize that necessary nuanced distinctions between these actions will be better captured in our proposed concept spaces. The defining parameters of concept spaces are their basis vectors,  $b_{i}$ . Normalized embeddings (extracted from text encoder,  $\theta_{t}$ ) of various natural language captions  $(c_{i})$  relevant to action categories are used as these basic vectors.
$$
b _ {i} = \theta_ {t} \left(c _ {i}\right) / \left| \left| \theta_ {t} \left(c _ {i}\right) \right| \right| _ {2} ^ {2} \tag {1}
$$
$$
\mathbf {b} = \left[ b _ {1}, b _ {2}, \dots b _ {n} \right] ^ {T}; \mathbf {b} \in \mathbb {R} ^ {(n, d)} \tag {2}
$$
Note that these basis vectors are not necessarily orthogonal. As illustrated in Fig. 2, a single set of basis vectors,  $\mathbf{b}$ , defines one action concept space. We define two sets of basic vectors: action category vectors and action description vectors. Action category vectors relate to a single action label which is converted to a caption using textual prompting following [4]. Action description vectors are averaged embeddings of multiple descriptions and visual characteristics relevant to individual action categories. These two distinct sets of basic vectors lead to two distinct concept spaces which we name category concept space and description concept space respectively.
Category Concept Space: We explore 3 different strategies to construct the category concept space. The base setup uses action labels from Kinetics-400 [70], UCF-101 [71], and HMDB-51 [72] datasets, leading to a set of 530  $(400 + 101 + 51$ , ignoring overlaps) basis vectors. Our next goal of connecting LLMs and their action awareness occurs in the second two strategies. We utilize LLMs [18] and visual-LLMs [73] to extract large sets of action category labels. While we explore this idea of expanding the basis vector set with LMM based additional action labels in Section 4, the base setup containing a modest 530 categories was sufficient to improve downstream task performance.
Description Concept Space: This space is constructed conditioned on the previous category concept space. For each action label used in the latter, we extract 4 distinct descriptions and a set of visual characteristics relevant to that action label using a large language model (LLM). The role of the LLM is to inject its world knowledge (i.e. awareness on videos, actions, and their attributes) into our learned representations during self-supervised learning. In detail, we prompt GPT-3 [18] to generate such descriptions and characteristics using procedure outlined in Appendix A. We highlight that GPT-3 is used here as an intelligent LLM containing world knowledge on videos and actions, in order to create natural language descriptions for given action category labels. The textual outputs generated for each action label are processed by our text encoder to produce multiple embeddings for a single action label. These embeddings are averaged to produce the corresponding basis vector for the description concept space. Note how this leads to a common dimensionality between the two concept spaces as well as one to one correspondences between the basic vectors of the spaces, which we leverage in our self-supervision objectives.
3.3 Concept Distillation
We now describe our primary self-supervised learning objective, concept distillation. Standard multi-view based self-supervision enforces a network to encode the common information between two augmented (distorted) views of a data sample [69]. This common information can be considered as the augmentation invariant signal present in the original data sample [69, 74]. In the case of self-distillation based approaches [15, 13], a higher dimensional feature space is utilized to enforce the self-supervision objectives. Instead, we propose to use action concept spaces as an alternative.
Proposed concept distillation depends on an action concept space and visual video features aligned to the basis vectors of that space. Given our visual features  $f_{i} \in \mathbb{R}^{d}$ , we obtain projected  $\tilde{f}_i \in \mathbb{R}^n$  as,
$$
\tilde {f} _ {i} = \mathbf {b} \left(f _ {i} / \left\| f _ {i} \right\| _ {2} ^ {2}\right) = \left[ b _ {1} \cdot f _ {i} ^ {\prime}, b _ {2} \cdot f _ {i} ^ {\prime}, \dots b _ {n} \cdot f _ {i} ^ {\prime} \right] ^ {T} \tag {3}
$$
Similarity Calculation: Projecting normalized visual video features to a concept space corresponds to calculating the dot-product similarity with each basic vector of the concept space. The projected vector  $\tilde{f}_i$  can be viewed as a similarity score distribution across all basis vectors of the concept space. Inspired by [67], we implement this similarity calculation as a linear layer with weight matrix  $\mathbf{b}$  and bias terms zero. We refer to this layer as the text classifier. Similar to [67], our text classifier remains frozen (no parameter updates), but in our case, this is to retain the original language distribution.
Concept Distillation Objective: Viewing projected features for two augmented views of a single video as score distributions, we argue that the underlying signal of the original video would relate to a unique score distribution to which score distributions of each view should be similar. Therein, following our EMA teacher based self-distillation setup (see Section 3.1 for details), we enforce the score distribution to be consistent across views. Given two views  $x_{1}, x_{2}$  of a single video, our teacher and student visual encoders process them respectively to produce  $f_{1}, f_{2}$ . The text classifier projects these to concept space, producing score distributions  $\tilde{f}_{1}, \tilde{f}_{2}$ . We obtain our objective,  $\mathcal{L}_{\mathrm{CD}}$  as:
$$
\hat {f} _ {i} [ k ] = \frac {\exp \left(\tilde {f} _ {i} [ k ] / \lambda_ {i}\right)}{\sum_ {j = 1} ^ {n} \exp \left(\tilde {f} _ {i} [ j ] / \lambda_ {i}\right)} \tag {4}
$$
$$
w _ {s} = \max (\hat {f} _ {1}) \tag {5}
$$
$$
\mathcal {L} _ {\mathrm {C D}} \left(\tilde {f} _ {1}, \tilde {f} _ {2}\right) = - w _ {s} \cdot \sum_ {j = 1} ^ {n} \hat {f} _ {1} [ j ] \log \hat {f} _ {2} [ j ] \tag {6}
$$
The teacher and student score distributions,  $\tilde{f}_1$ ,  $\tilde{f}_2$ , are softmax normalized in Eq. (4), with temperature terms  $\lambda_1 = 0.1$ ,  $\lambda_2 = 1$  for sharpening only the teacher score distribution. A significance score  $w_s$  is calculated for each sample in Eq. (5). In the softmax normalized teacher score distribution  $(\hat{f}_1)$ , the maximum value is high when peaked at a single action concept and low when peaked at multiple action concepts. Considering the noisy nature of multi-peak teacher score distributions, we utilize  $w_s$  to minimize their overall effect during training. Our overall  $\mathcal{L}_{\mathrm{CD}}$  is thus implemented as in Eq. (6).
Distinct Concept Spaces: Given the two distinct action concept spaces defined in Section 3.2, we utilize two parallel text classifiers to implement each, and obtain two score distributions, one for each concept space. Defining score distributions  $\tilde{f}_i^C$ ,  $\tilde{f}_i^D$  for category and description concept spaces respectively, we apply our  $\mathcal{L}_{\mathrm{CD}}$  on each pair separately to obtain two losses  $\mathcal{L}_{\mathrm{CD}}^{\mathrm{X}}$  for  $\mathrm{X} \in \{\mathrm{C}, \mathrm{D}\}$  as:
$$
\mathcal {L} _ {\mathrm {C D}} ^ {\mathrm {X}} = \mathcal {L} _ {\mathrm {C D}} \left(\tilde {f} _ {1} ^ {\mathrm {X}}, \tilde {f} _ {2} ^ {\mathrm {X}}\right) \tag {7}
$$
We highlight how our concept spaces implemented as text classifiers are maintained intact by freezing the text classifier during training. This allows our approach to perform direct zero-shot inference, making concept distillation additionally advantageous over standard video SSL techniques.
3.4 Uniform Distribution Prior
Avoiding collapse is a key concern in SSL methods [15, 13, 69] and recent self-distillation based approaches utilize feature sharpening and centering operations to avoid collapse [15, 13]. While we similarly perform sharpening operations on the teacher outputs, given the nature of our action concept space, performing a learned vector mean subtraction based centering operations can break the meaningful structure of score distributions. Instead, we enforce a uniform distribution prior on
the expected score distribution over the entire training dataset. The centering operation proposed in [15] acts similarly pushing representations towards a uniform distribution while the sharpening operation counters its effect. We approximate expectation over the dataset as a moving average of mean score distributions at each train iteration and the uniform prior is enforced as:
$$
\hat {f} _ {\mathrm {M A}} ^ {\mathrm {X}} = \tau \cdot \hat {f} _ {2} ^ {\mathrm {X}} + (1 - \tau) \cdot \hat {f} _ {\mathrm {M A}} ^ {\mathrm {X}} \tag {8}
$$
$$
\mathcal {L} _ {\mathrm {U P}} ^ {\mathrm {X}} = - \frac {1}{n} \sum_ {j} \log \hat {f} _ {\mathrm {M A}} ^ {\mathrm {X}} [ j ] \tag {9}
$$
where the hyper-parameter  $\tau = 0.5$  is fixed during training. We highlight that  $\mathcal{L}_{\mathrm{UP}}$  is necessary for convergence with concept distillation and is added to the concept distillation objective,  $\mathcal{L}_{\mathrm{CD}}^{\mathrm{X}}$ .
3.5 Concept Alignment
Aligning action category labels and their descriptions or attributes within some embedding space has been explored in video SSL under multiple settings [48, 49]. Motivated by these promising results, we explore how such alignment can be integrated to improve our framework with concept spaces. In Section 3.2, we define two distinct action concept spaces constructed from category labels and detailed category descriptions respectively. We hypothesize that explicit alignment of video features between these two spaces based on their one to one relationship can learn additional information. Therein, we introduce our concept alignment objective,  $\mathcal{L}_{\mathrm{CA}}$ , as follows:
$$
\mathcal {L} _ {\mathrm {C A}} = \mathcal {L} _ {\mathrm {C D}} \left(\tilde {f} _ {1} ^ {\mathrm {C}}, \tilde {f} _ {2} ^ {\mathrm {D}}\right) + \mathcal {L} _ {\mathrm {C D}} \left(\tilde {f} _ {1} ^ {\mathrm {D}}, \tilde {f} _ {2} ^ {\mathrm {C}}\right) \tag {10}
$$
Overall SSL Objective: Reusing  $\mathcal{L}_{\mathrm{CD}}$  from Eq. (6), we match score distributions across our two concept spaces instead of within a single concept space.  $\mathcal{L}_{\mathrm{CD}}(\tilde{f}_1^{\mathrm{C}},\tilde{f}_2^{\mathrm{D}})$  aligns student description score distribution  $\tilde{f}_2^{\mathrm{D}}$  to teacher category score distribution  $\tilde{f}_1^{\mathrm{C}}$  while  $\mathcal{L}_{\mathrm{CD}}(\tilde{f}_1^{\mathrm{C}},\tilde{f}_2^{\mathrm{D}})$  aligns student category score distribution  $\tilde{f}_2^{\mathrm{C}}$  to teacher description score distribution  $\tilde{f}_1^{\mathrm{D}}$ . Combining all terms, we obtain:
$$
\mathcal {L} = \left(\mathcal {L} _ {\mathrm {C D}} ^ {\mathrm {C}} + \mathcal {L} _ {\mathrm {U P}} ^ {\mathrm {C}}\right) + \left(\mathcal {L} _ {\mathrm {C D}} ^ {\mathrm {D}} + \mathcal {L} _ {\mathrm {U P}} ^ {\mathrm {D}}\right) + \mathcal {L} _ {\mathrm {C A}} \tag {11}
$$
3.6 Concept Space Variants