Transferring Textual Knowledge for Visual Recognition
Authors: Wenhao Wu
Zhun Sun
Wanli Ouyang
Venue: NeurIPS 2022 Submitted (2022)
Keywords: video understanding
video recognition
vision-language pretraining

Abstract: Transferring knowledge from task-agnostic pre-trained deep models for downstream tasks is an important topic in computer vision research. Along with the growth of computational capacity, we now have open-source Vision-Language pre-trained models in large scales of the model architecture and amount of data. In this study, we focus on transferring knowledge for vision classification tasks. Conventional methods randomly initialize the linear classifier head for vision classification, but they leave the usage of the text encoder for downstream visual recognition tasks undiscovered. In this paper, we revise the role of the linear classifier and replace the classifier with the embedded language representations of the object categories. These language representations are initialized from the text encoder of the vision-language pre-trained model to further utilize its well-pretrained language model parameters. The empirical study shows that our method improves both the performance and the training speed of video classification, with a negligible change in the model. In particular, our paradigm achieves the state-of-the-art accuracy of 87.3% on Kinetics-400.

In this study, we aim to improve the transferability of such vision-language pre-training models for 33 downstream classification tasks, with the help of their textual encoders. Our motivation comes from the semantic similarity among the ground-truth labels. To demonstrate this, employ the kinetics Transferring Textual Knowledge for Visual Recognition
Anonymous Author(s)
Affiliation Address
email
Abstract
Transferring knowledge from task-agnostic pre-trained deep models for down- stream tasks is an important topic in computer vision research. Along with the growth of computational capacity, now have open-source Vision-Language pre- trained models in large scales the model architecture and amount of data. In this study, focus transferring knowledge for vision classification tasks. Conven- tional methods randomly initialize the linear classifier head for vision classification, but they leave the usage of the text encoder for downstream visual recognition tasks undiscovered. In this paper, we revise the role of the linear classifier and replace the classifier with the embedded language representations of the object categories. These language representations are initialized from the text encoder of the vision-language pre-trained model to further utilize its well-pretrained language model parameters. The empirical study shows that our method improves both the performance and the training speed of video classification, with a negligible change in the model. In particular, our paradigm achieves the state-of-the fi-th-errt art accuracy of 87.3% on Kinetics-400.
2
4
10
11
12
13
14
15
Introduction
16
17
Pre-training a task-agnostic model using large-scale general datasets and then transferring its learning feature representations to downstream tasks is paradigm in many computer vision applications 2 While in the last decade, the convolutional-baas (more precisely, ILSVRC-2012) dataset with supervised style dominated this field. Owing to the dramatically increasing computational capacity, now we can train models that have several magnitude more model parameters and FLOPs significantly larger datasets in either supervised [4] 2 5) weakly-supervised [ or self-supervised g style. Recently, contrastive learning-based vision- language pre-training [I manifest their superior capabilities in improving down-streaming tasks performance such as classification [I], captioning
18
19
models that are optimized on the ImageNet
20
21
23
24
25
[9]. image generation [10 [T]. to name a few. These models are powerful for two reasons: i) the employed large-scale weakly-related datasets provide rich semantics and diverse representations of concepts; ii) the representation vectors of images and texts roughly aligned in the semantic embedding space. However, the most common approach to using these models is fine-tuning the visual encoder on specific tasks. Although the rich semantics and diverse representations of concepts benefit the downstream tasks, the usage of the textual encoder is still left undiscovered.
27
29
31
35 video recognition dataset [12] for the analysis. We extract the embedded textual vectors of class labels using the textual encoder released by CLIP M.. We then calculate the correlation between the
Submitted to 36th Conference on Neural Information Processing Systems (NeurIPS 2022). Do not distribute.

---

drinding
1.00
drinking
0.5
drinking
drinking
drinking Shots
crriking phots
artiig car
0.93
driiig car
driving kicking
drumming ringers
0
drumming fingers
drop kicking
dunking dying har
dunking basketoall
heir
coting burger
ewiing burger
exting
caie
waltng cerrols
cating chips
080
cating chips
0.2
cating doughnuts
eating diughrutt
eaking
cating ice cream
cating spaghetti
-0.75
cating ke cream
cating spagnettl
6.1
esting watermeion
egg huning
ego hunling
exerdiing am
uned Exyyaan
2015 donn
pesaaoen aupung
lie, Buio
io furmee
ziio Buppy
inuufinc Bugen
Sopayu Eunaa
esprmase S Suppe
-uu Suntsawe
exerciing am
buro crip
Jotine Eupeo
oges Buges
szruddno Enno Bupsa
bopyou pecooudulle
Bapuru
tue Sup0///v
0.0
tuhoup
berutuni
Bugns
Buruo
bujes
bugm
Surme
Figure I: Inter-class correlation maps of "embeddings of class labels" for 20 categories on Kinetics- 400. Left: The extracted textual vectors of class labels, Right: The "embeddings" from learned classifier. The color thresholds are adjusted for better understandability. Please zoom in for best view.
37
embedded textual vectors. The plot is shown on the left of Figure I Not surprisingly, the extracted textual vectors of class labels exhibit certain inter-class correlations, since part of them include the same verbs in their labels, such as playing <something>. Meanwhile, the labels with different verbs show a negligible inter-class correlation, such as drinking and driving. Next, we examine the final projection head of a vanilla visual recognition framework. We conduct the visual-only fine-tuning progress with the visual encoder that also released by CLIP [. The detailed configurations are provided in Section 4.2 The projection head is a matrix of d x to compute the pre-softmax values (or logits) from the d-dimensional feature vectors for the classes. Non-rigorously, we can consider the d-dimensional row vectors as the embeddings of the class labels, allowing inter-class correlation between these Iearned "embeddings", as shown on the right side of Figure [] Interestingly, these learned "embeddings" also reveal certain correlations after the training progress, despite being initialized randomly and optimized without knowing any textual information
38
39
40
41
42
44
to explore the
46
48
49
Therefore, we suppose that the semantic information contained in the samples (images and videos) does correlate with inter-classes. Following this motivation, we replace the projection matrix with several variants: i) A projection matrix whose row vectors are randomly sampled (trivial correlation);
50
51
52 ii) projection matrix whose row vectors are orthogonal to each other non-correlated). Then we replace the projection matrix with fixed embedded textual vectors that provide the "proper" correlation. In the empirical studies, we find that the textual knowledge significantly improves the transferability of pre-trained models, regarding both the classification accuracy and the convergence speed. Our main contributions are summarized as follows:
54
56
57
We build a new recognition paradigm to improve the transferability using knowledge from the textual encoder of the well-pretrained vision-language model.
58
We conduct extensive experiments on popular video and image datasets (i.e., Kinetics- 400 [12), UCF-101 [13]. HMDB-51 14] and ImageNet (3)) to demonstrate the transferability of our solution in many types of transfer learning, i.e., image/video recognition, zero-shot recognition, few-shot recognition. Our approach democratizes the training on large-scale video/image datasets and achieves state-of-the-art performance on video recognition tasks, e.g., 87.3% top-1 accuracy on Kinetics-400.
60
81
82
63
64
2 Methodology
66 Denotations. In the rest of the paper, we use bold letters to denote Vector, and capital italic letters to denote Tensor Matrix. For instance, employ z E Rd to denote the feature vector extracted 68 from pre-trained model of dimension d, we employ for the c-class linear classifier. Without ambiguity, we also use capital italic letters to denote the
dd to denote the projection matrix
That is, optimized with cross-entropy loss with one-hot hot labels

---

Classifier
Clussifier W
Textual
Images/ Vidcos
Visual Encoder Emb.
b
ead
logits
CLS1, CLS2 .. CLS
Encoder
FT
Emb.
Visual
Images/ / Videos
+
Emb.
logits Bxc
(a) Standard visual recognition paradigm
Encoder
(c) Our recognition paradigm
bad Emb.
Textual Encoder
photo of'a (CLS)
CLS class name
Encode
bad Emb, -
Images/ Videos
Visual Encoder Sv
+
logits bxb
dot product
Offine encoder
Emb
Emb
Visual embedding
Textual embedding
(b) Vision-language pre-training paradigm
Figure 2: Illustration of (a) standard visual recognition paradigm, (b) vision-language pre-training paradigm, and (c) our proposed recognition paradigm.
modality in subscripts, especially employ V and T to denote the Visual modality and Textual modality, respectively. We further employ lowercase italic letters to denote functions or neural networks. For instance, we employ gv (v)) and gr(; Or) denote the visual encoder and textual encoder, respectively. Additionally, we employ calligraphic letters, e.g., D, to denote sets of elements.
71
73
74 2.1 Revisiting of the standard paradigm and the vision-language pre-training
Standard visual feature transferring paradigm. We start with the most ordinary scenario, 76 where visual feature encoder model !v is optimized using a large-scale dataset 'D that con- tains visual samples with without ground-truth labels. On our labeled downstream dataset
D
{x v1), (x2, 322,... ... ... our empirical learning target can be written
sV. W = argmin E B. (ylo(( 9y (z)))]
(1)
Wv.W
79
where (Qlp) stands for the CrossEntropy between the predicted distribution and the ground-truie -rrth distribution ÿ. a denotes the sof tmax operation, W E Rexd denotes the linear projection matrix for classification. The formulation in Eq. i a standard visual feature transferring paradigm, where the visual encoder gv and the projection matrix (classifier) W are learned simultancously.
80
81
82
Vision-language pre-training in CLIP. As shown in Figure ((b), then review the contrastive pre- preitngin paradigm of the vision-language models in [T. Given a weakly related image-text pair dataset D = [œv11 1);1), (Iv,2, )...) With slight abuse of the notations, employ the vv Ir to denote a mini-batch of size b, then we minimize the following target,
84
86
gv. .
argmin E B H ((vo(((e(x))) θv,er
(IT)))].
(2)
where Q is the set that contains b one-hot labels of size with their 1,2..... b -th element being 88 1 (b < c, denoting the positive image-text pairs. Here we clarify that, the definition in Eq. is not the rigorous form of the Noise-Contrastiv Estimation (NCE) loss proposed in [15] [6] Instead, 90 we employ the cross entropy version implementation in ①I I7]. This implementation depicts a connection between the standard feature transferring paradigm and ours. In which, the gr(ær) can 82 be considered as the projection matrix that map the visual feature gv (xt) ) to the given label set Q.
93
2.2
Our proposed paradigm
94
we replace the learnable randomly initialized linear projection matrix W with pre-defined matrix W. Similarly, the training target can be written as
As discussed in Section |
95
g. argmin E Bey.D [Hylo(WW [ (v()))
(3)
96 Note that W is not in the optimization targets, since we freeze from updating during the fine-tuning 37 on the downstream tasks. We do this for two reasons: Firstly, it could preserve the textual knowledge 98 from being disturbed by the randomness brought by the mini-batch. For instance, when some classes missing, their embedded feature vector might be broken by the other classes; Secondly, want
3

---

100
to provide a fair comparison between different initializations of W (The unfrozen results are given in the supplementary materials). Now we consider how to initialize W. To examine how the correlation between the semantic information contained the samples helps, we investigate the following four types of initialization, where the forth is our proposed initialization.
101
102
103
Randomized matrix For the most simple randomized matrix case, we set each row of the W with a random Gaussian vector of zero mean and standard deviation, that is
104
105
W ~ (0, Ia),
(4)
106
where la denotes the identity matrix of dimension d x d. Arithmetically, a trivial "correlation" would appear between the row of the W, since the sampling size is significantly small to be biased. Evidently, the trivial "correlation" cannot indicate the real correspondence between the classes due to its stochasticity. Therefore we expect the model to have inferior performance since it needs to avoid these incorrect correlations when learning the visual feature representation.
107
108
109
110
111
Randomized Orthogonal matrix We follow the approach of the randomized matrix. We then remove the correlation by ensuring the row vectors are orthogonal. This is achieved by QR decomposition. Concretely, since d > c, we first generate a random matrix of size d x d and select the first rows as our projection matrix. Formally, we have,
113
W
1.
U, ~N, 1,2,
(S)
115
where U is the intermediate randomized matrix, QR(U) is the row orthogonal matrix obtained through the QR decomposition. Similar to the randomized matrix, we also expect this initialization to have inferior performance. Given the fact that the one-hot label vectors are also orthogonal to each other, it will not be helpful to project the visual feature vectors with an orthogonal matrix, which increases the difficulty of learning meaningful visual features.
116
118
120
Linear discriminant projection We consider another way of initializing the projection matrix. We employ the multi-class Fisher's linear discriminant analysis (LDA) to learn a linear classifier, then employ the weight matrix of the classifier as our initialization of the projection matrix. The LDA is optimized using the visual embeddings from the pre-trained model of samples in the train split. Then we compute the projection matrix following previous work [18]. Intuitively, the LDA first projects the feature vectors into a lower dimension space that maximizes the inter-class covariance and then estimates the likelihood of a sample to the class distributions. We, therefore, term this as the maximal correlation initialization. As an essential classifier, this type of initialization delivers reasonable performance, but it is largely dependent on the data employed to compute the projection matrix. When the data is limited, the estimated correlation will be biased. On the other hand, in our proposed paradigm, the pre-trained textual encoder provides unbiased correlations for fine-tuning.
122
124
126
128
129
130
Textual embedding vectors We finally describe our proposed feature transferring paradigm. Briefly, the projection weight W is composed of the embedded textual feature vectors of the labels. Given set of tokenized class labels C {41, b2 ..... ....,,,,) Lc), we have
133
W. ~ gr (4), 1,,,
(6)
where W. the i-th row vector in matrix W. And W; is initialized using the textual encoder output of the textual label of the i-th class. In the experimental analysis, we investigate two types of textual feature encoders: i) The encoder that is trained with visual encoder in the contrastive style; ii) The encoder that is trained solely using only textual samples on tasks such as masked language modeling.
135
136
137
3 Related Works
138
139
Visual Recognition. Convolutional networks have long been the standard for backbone architectures
140
in image recognition [19
20
21
22
23 24 and video recognition 25 26 27
28
29
30
31.. In-
141
spired by the Transformer
32 scaling successes in Natural Language Processing, Vision Transformer (ViT) [33] applies a standard Transformer directly to images, which delivers impressive performance
142
143
on image recognition. Since then, ViT 33 has led a new trend image recognition backbone architectures, shifting from CNNs to Transformers. To improve performance, follow-up studies (e.g.,
145
DeiT 34 Swin (35)) have been developed. Also, many works has begun to adopt transformers in video recognition, such as TimeSFormer 36), ViVIT 37]. VideoSwin [38], and MVIT (39].

---

147
Vision-language Pre-training, Recently, CLIP [ provides good practice in learning the coordinated vision- langungg pre-training models using the image-text InfoNCE contrastive loss 40) Based on CLIP, several variants [41
148
149
43
45
have been proposed by combining more types of learning
150
tasks such as image-text matching and masked image/language modeling. These contrastively learned models have two deserved properties for downstream tasks: the abundant visual feature representations and the aligned textual feature representations. Yet another study [46] merged the downstream classification task into the pre-training progress, which demonstrates a decent improvement of accuracy over the standard cross-entropy loss. Moreover, a few recent works [47 48)
151
152
154
transfer the CLIP [1 pre-trained image-text matching model to the downstream video-text matching framework for video recognition with contrastive loss. Specifically, ActionClip [47 extends the
156
CLIP [ to train a downstream video-text matching model and then perform video recognition indirectly using the similarity between learned video and text encoders during inference. [48] focus on efficient prompting and learning the continuous prompt template as text input for video recognition. Instead of these matching- approaches, we aim to propose a new recognition paradigm that directly transfers textual knowledge for visual recognition. Our approach can balance performance and efficiency, and experiments demonstrate that our approach can reduce computational power requirements while democratizing training on large-scale video/image datasets (see Table 6 and 12 for more information).
158
159
160
162
163
184
4
Experiments: Video Recognition
185
166
4.1
Setups
167
To evaluate our method for video recognition, we conduct experiments on three widely used bench-
168
marks, i.e., Kinetics-400 12] UCF-101 [3] and 11 [14] See Supp. for more details.
169
Training& & Inference. We utilize ResNet 20 and ViT B3 as the visual encoders since they are the
170
representative backbones of CNN and vision transformer, respectively. We employ the pre-trained visual and textual encoder released by CLIP [ in most experiments for simplicity. Given a video, we first uniformly sampled T (e.g.. 8, 16, 32) frames over the entire video. Then image patches with the resolution of x2242224 are randomly cropped from the sampled frames to form the input. The model is optimized using Adam W with momentum set to 0.9. We use an initial learning rate of 5e-0, a cosine learning rate schedule with a 5-epoch linear warmup and a batch size of 128 for experiments on all datasets. For fast training, we set the total training epoch to 30 unless specified otherwise.
171
172
173
174
175
177
To trade off accuracy and speed, we consider two evaluation protocols. (1) Single View: We use only
178
1 clip per video and the center x224 crop for efficient evaluation, (e.g., as in Section 4.2) (2)
179
Mulriple Views: This is widely used setting in previous works [4] 27 [5) to sample multiple clips
180
per video (e.g., 10 clips) with several spatial crops (e.g., 3 crops) in order to get higher accuracy. For
181
comparison with SOTAs, we use four clips with three 224 224 crops ("4x3 Views") in Table 7
182
4.2
Ablations Kinetics.
183
In this section, we conduct extensive ablation experiments to demonstrate our method with the instantiation. Models in this section use 8-frame input, ViT-B/16 as the visual backbone, 30 epochs for training and a single view for testing on Kinetics-400, 400, unless specified otherwise.
194
185
186
Comparison with vision-only framework. Figure a) illustrates the standard visual recognition framework. As a comparison with our method, we train the unimodality video model, which consists of the same visual encoder and learnable classifier with random initialization. To produce video embedding, we just apply temporal average pooling (TAP) to frame embeddings. As presented in Figure 3 our method surpasses Vision-Only baselines across multiple label fractions Kinetics-400. Especially when just only 10% labeled data is available for training, demonstrating that the advantage of our paradigm is more profound when the labeled data is limited. Also, when training with full data, our Vision-Text method leads to an additional 5% improvement with the same training recipe. Figure 4 further demonstrates our paradigm significantly improves convergence speed.
187
189
191
192
193
194
195
Different assignments to the offline classifier. We set different initializations described in section 2.2 to the offline classifier W E Rdxe and then train our visual encoder on Kinetics-400. Table 1 lists their comparisons. We show that feeding the offline classifier a random d-by-e matrix with a normal distribution reduces performance significantly. Then assign the orthogonal matrix to the classifier,
196
197
5

---

Kineties Top-1 Accuracy (%)
80
78.47
80.13
74.9
5
75
71.71
75.27
70
72.06
65
Lo88
Vision Only
60
Vision Text
Vision Only
Vision-Teet
50
46.55
45
0%
20%
40%
60%
80%
100%
10
20
30
Label Fraccion
Epochs
Figure 3: Vixion-Text Vision-only framework Figure 4: The training loss of Vision-Text and under different label fractions on Kinetics-400.
Vision-only framework on Kinetics-400.
199 and we can see that having different classes that are orthogonal will result in inferior performance. Also, choose DistilBERT [51] as the textual encoder to pre-extract the text embeddings of categories. The resulting performance the same as that of the CLIP's s textual encoder. Furthermore, the maximal correlation initialization, Section 2.2 To do so, we first sample 60 videos from each class in the training set and utilize the pre-traine trained visual encoder to extract visual embeddings from these 24,000 videos. Finally, we learn
201
term the linear discriminate projection
stated in
203
205
the linear classifier by performing linear discriminant analysis on these visual embeddings and their that the result of the LDA projection is consistent with statement. More visualizations of these classifiers are in supplementary materials.
ground-trun truth labels. We
207
Table Exploration of different generation methods for the frozen classifier.
Table 2: Temporal modeling for video encoders.
Backbone
Modeling
Top-1
Top-5
Offline classifier from Textual encoder of CLIP
Top 1
TAP
71.20
90.37
81.52
ResNet-50
TID
67.18
88.45
Random normal matrix Random orthogonal matrix
59.30
T-Trans
74.26
91.67
59.44
TAP 80.13
94.98
DistilBERT
81.45
VIT-B/16
TokenTID
80.42
95,03
Linear discriminant projection
80.77
T-Trans
81.52
95.49
208
Temporal modeling. Here we explore more temporal modelings for VIT 33 and ResNet [20]
209
(1) TAP: Temporal average pooling is the most straightforward temporal modeling. (2) TID: The channel-wise temporal ID convolutions, is common strategy [50 52 53 to perform efficient
211
temporal interaction the latter stages (i.e., re84-5) of ResNet. (3) T-Trans: The embeddings
of frames are fed to multi-layer (e.g., 6-layer) temporal transformer encoder. (4) TokenTID: We use TID to model temporal relations for [class] token features that are aggregated from local features via attention in the vision transformer. We perform the TokenTID in multiple positions of a vision transformer. Results are shown in Table 2 On both backbones, TAP provides simple baselines and T-Trans exhibits the best top-1 accuracy. Both of them maintain the original frame-level representations and then perform temporal modeling. An interesting thing we observed is that TID does not seem to work in this scenario. The reason lies in that TID may have the potential to break the learned strong representations provided by CLIP. TokenTID is another internal-backbont temporal modeling, and it does not yield performance drop, and even slightly improves the TAP baseline. We believe this is because Token'iD is only imposed on the global [class] token features instead of patches features, resulting in minimal modifications on pre-trained features.
213
215
217
219
221
222
223
Visual encoder with different pre-training, Besides CLIP-pretrained visual encoders, we further explore our paradigm with different pre-trained visual encoders. As shown in Table K equipped with ImaggeNet-pretrainee visual encoder, our method helps to improve the vision-only counterpart by We 0.9%.
225
that the CLIP- pretrained visual encoder achieves more significant performance, which is probably because CLIP provides the coarse initial aligmment between frames and category names, well covers rich visual concepts.
227
229 Text input forms. Intuitively, the name of class appears to the most straightforward text information. We can see that only using the label text can yield good results in Table 4 Then

---

Table 3: Study on different pre-training.
Table 4: Study on various text input forms.
Visual encoder
Paradigm
Top-1
Text input from
Тop 1
Vision- Vision-Only
75.27
class name
81.37
CLIP-pretrained
Vision- Text 80.13
"a video of person" + class name
81.52
Vision- Vision-Only
74.78
multiple fixed templates + class name
80.88
ImageNet--rettained
Vision-Tess 75.63
learnable template class name
81.22
following the prompt engineering in CLIP [T] utilize the prompt template "a video of person [label)." to help specify the text is about the content of the video. This only slightly increases the baseline of using the label text. We further multiple prompt templates as the text augmentation during training. Performance decreases by 0.64% on Kinetics-400. This may be because different prompt templates may introduce extra noise for the training. In addition the hand-crafted prompt, we also adopt an automated prompt [54] to describe prompt's ' context using of learnable vectors. The results suggest that different templates have little impact our model.
232
performance
234
235
236
237
Table 5: Different instantiations of our method on Kinetics-400. "Single View" indicates one temporal clip with one spatial crop, whereas "4x3 Views" indicates 4 temporal clips with 3 spatial crops.
Single View
4x3 Views
Encoder
Resolution
Frames
Top-1
Top-5
Top-1
Top-5
8
74.26
91.67
75.50
92.61
ResNet-50
224x224
16
74.81
92.20
75.94
93.00
8
77.97
93.80
79.57
94.70
VIT-B/32
224x224
16
79.17
94.24
80.37
94.95
8
81.52
95.49
82.65
96.25
VIT-B/16
224x224
16
82.34
95.71
83.15
96.25
8
84.82
96.59
85.83
97.05
VIT-L/14
224x224
16
85.85
96.47
86.36
96.88
32
86.39
96.75
87.09
97.06
8
84.94
96.55
86.23
97.11
VIT-L/I4
336x336
16
86.05
96.92
86.63
97.27
32
86.60
97.00
87.30
97.46
More instantiations. We assess different instantiations of our paradigm, in terms of different visual encoders, more imput frames, and larger spatial resolution. See Supp. for more details on architectures. In Table 5 present the results of our method with two typical evaluation protocols. In general, 241 more frames, larger spatial resolution, and deeper backbones lead to higher accuracy.
239
Table 6: Ours vs. Matching paradigm with VIT-B/16 on Kinetics-400. The number of V100-days the number of V100 GPU used for training multiplied by the training time in days. * indicates the official result [47] via "Data-pamt parallel training" on 3090 GPUs. For efficient training and fair comparison, we implement all experiments with "Distributed Data-parallel training" in the Table.
Method
Batch gather
Textual encoder
Top-1
Top-5
V100-days
online
81.15
95.42
6.7 (10*)
offline
80.73
95.36
6.6
Matching paradigm
47
X
online
77.77
94.79
3.5
X
offline
76.13
94.57
3.3
Our paradigm
X
offline
81.52
95.49
3.3
242
Our recognition paradigm vs. Matching paradigm. Here we make a comparison with the matching- method mentioned in Section 3 The matching paradigm treats the recognition task as video-text matching problem with contrastive loss, thus requiring batch gathering to collect embeddings of all batches across all GPUs and calculate cosine similarity for a given batch across all other batches. See Supp. for details about the batch gathering. In Table 6 we try to compare
243
245
7

---

Table 7: Comparison to SOTAs on Kinetics-400, "Views" indicates # temporal clip x # spatial crop. The magnitudes are Giga (10%) and Mega (10%) for FLOPs and Param. "IN" denotes ImageNet.
Method
Input
Pre-train
Top-1
Top-5
FLOPsx Views
Param
NL 13D-101
27
x222
IN-IK
77.7
93.3
x59x 10x3
61.8
MV 150
x4x2
IN-1K
79.1
93.8
188x10x3
SlowFast NL101
49
16x2242
Scratch
79.8
93.9
234x10x3
59,9
X3D-XXL 55
16x4402
Scratch
80.4
94.6
144x 10x3
20.3
MVîT-B, 64x3
39
x4x2222
Scratch
81.2
95.1
455x3x3
36.6
Methods with large-scale pre-training
TimeSSormer-L [36
96x2242
IN-21K
80.7
94.7
2380
121.4
ViViT-L/16x2 37
32x3202
IN-21K
81.3
94.7
3992x4x3
310.8
Swin-L 38
32x3842
IN-21K
84.9
96.7
2107x10x3
200.0
ip-CSN-152 [56]
32x224
IG-65M
82.5
95.3
109x10x3
32.8
ViViT-L/16x2 37
32x3202
JFT-300M
83.5
95.5
3992x4x3
310.8
ViViT-H/16x2
37
32x224
JFT-300M
84.8
95.8
8316x4x3
647.5
Toklecarne--I/10
32x2242
JFT-300M
85.4
96.3
4076x4x3
450
MTV-H [88
32x2242
JFT-300M
85.8
96.6
3706x4x3
CoVeR [59
16x4482
JFT-300M
86.3
-x1x3
Florence
44
32x3842
FLD-900M
86.5
97.3
-x4x3
647
CoVeR [59
16x4482
JFT-3B
87.2
-x1x3
Ours ViT-L/14
32x2242
WIT-400M
87.1
97.1
1662 x4x
230.7
Ours ViT-L/14
32x3362
WIT-400M
87.3
97.5
3829x4x3
230.7
that the matching paradigm does not work well without batch gather. This due to contrastive learning favors a large batch size. Besides, involving batch gather will multiply the training time. Also, in this case, the pre- pre-trained
with the matching paradigm 47 fairly as
We
248
250
textual encoder still needs to be updated, which requires larger GPU memory. However, our paradigm employs pre-extracted text embeddings our classifier, so the only thing need fine-tune the visual encoder. Results show that our method achieves the best accuracy-cost trade-off. Specifically, our method achieves the performance of 81. 81.52% with VIT-B/16, which takes only 10 hours run the training using 8 GPUs (2x faster than the matching counterpart).
252
253
254
4.3
Main Results.
256
Comparison to state-of-the-art. In Table 7 on Kinetics-400 we compare to state-of-the. tf-thh-artss that
pre-trained large-scale datasets such ImageNet-21K (3] IG-65M [60], JFT-300M [2). FLD-900M [44] and JFT-3B [5] The suffix represents the magnitude of the dataset, e-g., JFT-3B consists of nearly 3 billion annotated images. We include the details of these web-scale datasets in Supp. To the best of our knowledge, up to now, none of the three largest datasets (i.e., JFT-300M, FLD-900M. JFT-3B) are open-sourced and also do provide pre-trained models. Thus, we use the CLIP [I checkpoints, which are publicly available and have been trained on 400 million web image-text pairs (namely WIT-400M). Observe that achieve state-of-the-art results. Specifically, our model outperforms all JFT300M-preerainee methods in terms of Top-1 and Top-5 accuracy. We achieve 87.3%, which improves even further by 0.8% over Florence [4]. although their model and data scale are both 2 lalaggrr. Besides, our model is even better than JFT3B-por JFT3B-pretrained pretrained Co CeVeR (59), and their data scale is 7.5 llarger. See Supp. for more results on JCF-101 and HMDB-51 datasets.
258
260
252
284
285
286
267
268
Few-shot video recognition. Video recognition using only few samples is known as few-shot video recognition. We study a more challenging K-shot C-way situation instead of the conventional 5-shot 5-way configuration. We scale the task up to categorize all categories the dataset with just K samples per category for training. The upper bound of this situation denoted by the term "All-shot". Table [ reports the top-1 accuracy for the three datasets. In this extreme scenario of few data, we use 200 epochs train models with ViT-B/16 for few-shot video recognition. For temporal modeling,
270
272
274
we use TAP. We can observe that our method provides amazing transferability on diverse domain
275 data in these extreme data-poor cireumstances.

---

Table 8: Few-shot video recognition on three popular datasets under K-shot C-way setting.
Table 9: Zero-shot video recognition under intra- dataset and cross-dataset settings. (A)-(B) indi- cates we train the model on dataset A then perform zero-shot recognition on dataset B.
K- shot
K400
UCF101
HMDB51
63.16
88.77
65.17
67.50
92.78
69.99
K300 -K000
K400~>UCF
5
69.89
93.87
71.03
Ours w/o train Ours w/ train
63.35
63.01
All
80.13
95.24
73.18
66.38
74.67
276
Zero-shot video recognition. We conduct experiments on two open-set settings: 1) Intra-dataset:
The Kinetics-400 was divided into two parts: 300 categories (K300) for training and 100 categories
278
(K 100) for zero-shot recognition. 2) Cross-dataset: We train our models on K400 and then evaluate them UCF101. To avoid catastrophic forgetting [6]]. here we train models with few epochs. As
280
shown in Table 9 unlike the traditional recognition paradigm, ours can achieve zero-shot recognition for unseen categories by replacing the offline classifiers. Appropriately tweaking the pre-trained model slightly can boost performance even further.
282
Experiments: Image Recognition
284
We also evaluate our approach to the image recognition task. Here we conduct experiments on ImageNet
3 and share the same training recipe in section 4.1 with ImageNet.
286
Few-shot image recognition. Here also use the challenging K-shot C-way setting on ImageNet. Specifically, the models are trained using K images (shots) from the training set for each image category and then measure performance on the corresponding standard 1000-class testing set. As shown in Table I0 the results reveal that our method has strong transferability under data-poor conditions, whereas the standard unimodality paradigm ineffective in comparison ours.
287
288
289
290
Table 10: Few-shot image recognition on ImageNet. "Zero- shot" and "All-shot" denote the lower and upper bounds of the task respectively. Top-1 1 accuracy is reported here.
Table 11: Zero-shot image recognition.
We train the model on IN600 then per- form evaluation on IN400.
K-shot 0
3
5
All
IN600~1N400
Ours 66.73
71.50
73.64
74.99
82.25
Ours w/o train 70.28
Vision-Only 0
4.71
30.44
41.70
79.70
Ours w/ train
72.62
291
Zero-shot image recognition. Here we split the ImageNet-IK into two parts, with 600 categories (IN600) for training, and the remaining unseen 400 categories (IN400) for evaluation. Table I1 demonstrates the sero-shot image recognition ability of our method.
292
293
294
Efficient training, For readers' reference, we provide the performance of our approach with different
285
train the VIT-B/16 to achieve 82.25% in 90 minutes, while the VIT-L/14 only takes 6 hours to achieve 86.47%.
visual backbones ImageNet in Tabel [2 Notably, using 8 GPUs,
296
Table 12: Study on various backbones. Models are trained with epochs.
Backbone
Resolution
Top-1
Top-5
FLOPs
Params
A100-days
VIT-B/16
224x224
82.25
96.82
11.3G
57.3M
0.5
VIT-L/14
224x224
86.47
98.11
51.9G
202. IM
2.0
VIT-L/14
336x336
87.12
98.33
116.5G
202.1M
5.7
Conclusion
297
We present new paradigm for improving the transferability of visual recognition that is based on the knowledge from the textual encoder of the well-trained vision-language model. The empirical study shows that method improves both the performance and the convergence speed of visual classi- fication. The proposed approach has superior performance on both general and saro-shot/fewsshol recognition and achieves state-of-le the art performance video recognition tasks, and democratizes training on large-scale video/image datasets.
299
301
303

---

References
304
305
[1] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pages 8748-8763. PMLR, 2021.
308
307
308
309
[2] Chen Sun, Abhinav Shrivastava, Saurabh Singh, and Abhinav Gupta. Revisiting unreasonable effectiveness of data in deep learning era. In Proceedings of the IEEE international conference on computer vision, pages 843-852, 2017.
310
312
[3] Jia Deng. Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In Proc. CVPR, 2009.
314
[4] Tal Ridnik, Emanuel Ben-Baruce Baruch, Asaf Noy, and Lihi Zelnik-Mand Imagenet- pretraining for the masses. arXiv preprint 27Xiv:2104.1097, 2021.
[5] Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer. Scaling vision transform- 316
317
ers.
318
[6] Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoe Le, Yun- Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and rision-languenn representation learning with noisy text supervision. In International Conference on Machine Learning, pages 1904-4916, PMLR, 2021.
319
320
321
322
[7] Priya Goyal, Mathilde Caron, Benjamin Lefaudeux, Min Xu, Pengchao Wang, Vivek Pai, Man- nat Singh, Vitaliy Liptchinsky, Ishan Misra, Armand Joulin, et al. Self-supervised pretraining of visual features in the wild. arXiv preprint arXiv:2
323
324
2021.
325
[8] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 9729-9738, 2020.
327
[9] Ron Mokady, Amir Hertz, and Amit H Bermano. Clipcap: Clip prefix for image captioning. arXiv preprint arXiv:2111.0973: 2021.
329
330
[10] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In International Conference on Machine Learning, pages 8821-8831. PMLR, 2021.
331
332
333
[11] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06115, 2022.
334
[12] Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijaya- narasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, et al. The kinetics human action video dataset. arXiv preprint arXiv: 0tt::/700 7795006 2017.
336
338
[13] Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. Ucf101: A dataset of 101 human actions classes from videos in the wild. arXiv preprint arXiv: 2012.
340
[14] Hildegard Kuehne, Hueihan Jhuang, Estíbaliz Garrote, Tomaso Poggio, and Thomas Serre. Hmdb: a large video database for human motion recognition. In Proc. ICCV, 2011.
342
[15] Aaron Van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding, arXiv e-prints, pages arXiv-1807 2018.
343
344
[16] Cheng-I Lai. Contrastive predictive coding based feature for automatic speaker verification. arXiv preprint arXiv: 1904.01500 2019.
345
[17] Xinlei Chen, Saining Xie, and Kaiming He. An empirical study of training self-supervised vision transformers. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 9640-9649, 2021.
347
10

---

349
[18] Tao Li, Shenghuo Zhu, and Mitsunori Ogihara. Using discriminant analysis for multi-class classification: an experimental investigation. Knowledge and information systems, 10(4))453-
350
2006.
352
[19] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems, 25, 2012.
353
354
[00] Kaiming He, Xiangyu Zhang. Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proc. CVPR, 2016.
355
358
[21] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition, arXiv preprint arXiv:1409 1500.1566,, 2014.
357
358
[22] Yemin Shi, Yonghong Tian, Yaowei Wang, Wei Zeng, and Tiejun Huang. Learning long- dependencies for action recognition with a biologically-inspinn deep network. In Proc. ICCV, 2017.
359
360
361
[23] Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang. Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint 1rXiv:1704.04866 2017.
362
363
364
[24] Mingxing Tan and Quoc Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In International conference on machine learning, pages 6105-6114 6114. PMLR, 2019.
365
388
[25] Karen Simonyan and Andrew Zisserman. Two-stream convolutional networks for action recognition in videos. In Neurips, 2014.
367
368
[26] Limin Wang. Yuanjun Xiong, Zhe Wang, Yu Qiao, Dahua Lin, Xiaoou Tang, and Lue Van Gool. Temporal segment networks: Towards good practices for deep action recognition. In Proc. ECCV, 2016.
389
370
371
[27] Joao Carreira and Andrew Zisserman. Quo vadis, action recognition? a new model and the kinetics dataset. In Proc. CVPR, 2017.
372
373
[28]
Du Tran, Lubomir Bourdev, Rob Fergus, Lorenzo Torresani, and Manohar Paluri. Learning spatiotemporal features with 3d convolutional networks. In Proc. ICCV, 2015.
374
375
[29] Zhaofan Qiu, Ting Yao, and Tao Mei. Learning spatio-temporaa representation with pseudo-3d residual networks. In Proc. ICCV, 2017.
377
[30] Saining Xie, Chen Sun, Jonathan Huang, Zhuowen Tu, and Kevin Murphy. Rethinking spa- tiotemporrl feature learning: Speed-accuracy accuracy trade-offs in video classification. In Proc. ECCV, 2018.
378
[31] Du Tran, Heng Wang, Lorenzo Torresani, Jamie Ray, Yann LeCun, and Manohar Paluri. A closer look at spatiotemporal convolutions for action recognition. In Proc. CVPR, 2018
381
382
[32] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez. tukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pages
384
2017.
305
[33] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale, arXiv preprint a0Xiv:2010.1 1929, 2020.
387
[34] Kai Han, An Xiao, Enhua Wu, Jianyuan Guo, Chunjing Xu, and Yunhe Wang. Transformer in transformer. Advances in Neural Information Processing Systems, 34, 2021.
390
391
[35] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10012-10022, 2021.
393
394
[36] Gedas Bertasius, Heng Wang, and I Lorenzo Torresani. Is space-time attention all you need for video understanding? In ICML, pages 813-824. PMLR, 2021.

---

396
[37] Anurag Arna, Mostafa Dehghani, Georg Heigold. Chen Sun, Mario Lučić, and Cordelia Schmid. Vivit: video vision transformer. Proc. ICCV, 2021.
397
398
[38] Ze Liu, Jia Ning, Yue Cao, Yixuan Wei, Zheng Zhang, Stephen Lin, and Han Hu, Video swin transformer. arXiv preprint arXiv:2106. 13230, 2021.
400
[3]
Haoqi Fan, Bo Xiong, Karttikeya Mangalam, Yanghao Li, Zhicheng Yan, Jitendra Malik, and Christoph Feichtenhofer. Multiscale vision transformers. Proc. ICCV, 2021.
401
402
[40] Aaron Van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv e-prints, pages arXiv-1807, 2018
403
404
[41] Chao Jia, Yinfei Yang. Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoe Lc, Yun- Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-languann representation learning with noisy text supervision. In International Conference on Machine Learning, pages 4904 4916. PMLR, 2021.
406
408
[42] Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language- image pre-training for unified vision-language understanding and generation. arXiv preprint arXiv:2201. /0086, 2022.
409
410
411
[43] Yangguang Li, Feng Liang, Lichen Zhao, Yufeng Cui, Wanli Ouyang, Jing Shao, Fengwei Yu, and Junjie Yan. Supervision exists everywhere: A data efficient contrastive language-image pre-training paradigm. arXiv preprint arXiv:2 10.05200 2021.
413
414
[44] Lu Yuan, Dongdong Chen, Yi-Ling Chen, Noel Codella, Xiyang Dai, Jianfeng Gao, Houdong Hu, Xuedong Huang, Boxin Li, Chunyuan Li, al. Florence: A new foundation model for computer vision. arXiv preprint arXiv:2111.11432 2021.
415
418
[45] Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu. Coca: Contrastive captioners are image-text foundation models. arXiv preprint arXiv:2005.01917, 2022.
418
420
[46] Jianwei Yang. Chunyuan Li. Pengchuan Zhang, Bin Xiao, Ce Liu, Lu Yuan, and Jianfeng Gao. Unified contrastive learning in image-text-label space. arXiv preprint arXiv:2204.03610 2022.
421
422
[47] Mengmeng Wang, Jiazheng Xing, and Yong Liu. Actionclip: A new paradigm for video action recognition. arXiv preprint arXiv:2109.08472 2021.
423
424
[48] Chen Ju, Tengda Han, Kunhao Zheng. Ya Zhang, and Weidi Xic. Prompting visual- Languag models for efficient video understanding. arXiv preprint (xXiv2112.004478 2021.
426
[49] Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, and Kaiming He. Slowfast networks for video recognition. Proc. ICCV, 2019.
427
[50] Wenhao Wu, Dongliang He, Tianwei Lin, Fu Li, Chuang Gan, and Errui Ding. Mvfnet: Multi-view fusion network for efficient video recognition. Proc. AAAI, 2021.
429
430
[51] Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint
431
2019.
432
[52] Limin Wang, Zhan Tong, Bin Ji, and Gangshan Wu. Tdn: Temporal difference networks for efficient action recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1895-19900 1904, 2021.
433
434
435
[53] Zhaoyang Liu, Donghao Luo, Yabiao Wang. Limin Wang, Ying Tai, Chengjie Wang, Jilin Li, Feiyue Huang, and Tong Lu. Teinet: Towards an efficient architecture for video recognition. In Proc. AAAI, pages 1 11669-11676, 2020.
437
438
[54] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for lasion-language models. arXiv preprint arXiv:2109.01134 2021.
439
440
[55] Christoph Feichtenhofer. X3d: Expanding architectures for efficient video recognition. In Proc.
CVPR, pages 203-213, 2020.
12

---

442
[56] Du Tran, Heng Wang, Lorenzo Torresani, and Matt Feiszli. Video classification with channel- separated convolutional networks. In Proc. ICCV, pages 5552-5561, 2019.
443
444
[57] Michael S Ryoo, AJ Piergiovanni, Anurag Arnab, Mostafa Dehghani, and Anelia Angelova. Tokenlearner: What can 8 learned tokens do for images and videos? arXiv preprint 1rXiv:2106.11297, 2021.
448
447
[58)
Shen Yan, Xuehan Xiong, Anurag Arnab, Zhichao Lu, Mi Zhang, Chen Sun, and Cordelia Schmid. Multiview transformers for video recognition. arXiv preprint arXiv:2201.04288. 2022.
449
[59]
Bowen Zhang, Jiahui Yu, Christopher Fifty, Wei Han, Andrew M Dai, Ruoming Pang, and Fei Sha. Co-training transformer with videos and images improves action recognition. arXiv preprint arXiv:21 2021.
450
452
660] Deepti Ghadiyaram, Du Tran, and Dhruv Mahajan. Large-scale weakly-supervised pre-training for video action recognition. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 12046-12055, 2019.
454
455
[61]
Michael McCloskey and Neal J Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. In Psychology of learning and motivation, volume 24, pages 109-165, Elsevier, 1989.
457
13

---

Checklist
458
459
The checklist follows the references. Please read the checklist guidelines carefully for information on
460
how to answer these questions. For each question, change the default [TODO] to [Yes], [No], or
461
[]/A]- - You are strongly encouraged to include a justification your answer, either by referencing the appropriate section of your paper or providing a brief inline description. For example:
462
465
Did you include the license to the code and datasets? [N/A] The codes, datasets and tools will be available after publication.
485
Please do not modify the questions and only use the provided macros for your answers. Note that the Checklist section does not count towards the page limit. In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.
487
488
1. For all authors...
469
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
470
(b) Did you describe the limitations of your work? [Yes]
472
(c) Did you discuss any potential negative societal impacts of your work? [Yes] There is no obvious negative societal impacts of this work.
473
474
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
475
478
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
478
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
480
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as URL)? [Yes]
481
482
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
484
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [Yes]
485
486
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes]
487
488
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
480
(b) Did you mention the license of the assets? [Yes]
491
(c) Did you include any new assets either in the supplemental material or as URL? [No]
492
(d) Did you discuss whether and how consent was obtained from people whose data you 're
using/curating? [Yes]
494
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A] There is no personally identifiable information in the used datasets.
485
496
5. If you used crowdsoureing conducted research with human subjects...
498
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
499
500
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
502
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
503
14


Reviews:
[{"review_id": "v2XIz1L4zQn", "reviewer": "Reviewer_yU63", "summary": "The paper tackles the problem of transferring knowledge from pre-trained vision-language models to other downstream tasks. It proposes a new paradigm that makes use of the pre-trained text embedding to initialize the weights of the linear classifier head when testing on downstream tasks. Finally, the authors test their approach on Visual recognition and Image recognition.", "questions": "1. Any insights on why DistilBERT performs the same as CLIP in Table 1?\n\n2. Why there is no comparison with other methods on the Image Recognition task?\n\n", "limitations": "There is no discussion related to the societal impact", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 1, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper tackles an important problem and seems to have promising results.\n\nI find that the paper is quite confusing and hard to read. I would suggest to re-write some parts by specifying clearly what is the input, output of the model and what aims to be learnt or not (Section 2, lines 169-181, but not only limited to this).\n\nAlso, please be more specific about the downstream task and clearly specify what is the goal of the task, the input and output of the model. \n\nSuggestions\nAdd some visual highlights to the tables (bold, italic, etc) so that the viewer knows what to look at.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "ZlhQgurCs1y", "reviewer": "Reviewer_L6WZ", "summary": "This work proposes a method of transferring vision-language pre-training models to downstream visual recognition tasks. To do so, they propose to initialize the final linear classification layer with corresponding embedding values from a pretrained clip model, generating embedding by passing the class label as a prompt through CLIP. The classification layer is frozen after initialization, forcing the vision pipeline to optimize around the pretrained embeddings. Experiments demonstrate improvements in video and few-shot image recognition downstream tasks.", "questions": "- It would be great if the authors could clarify the LDA formulation, as it wasn't very clear to me. Is a vision-only model first trained on the data, after which LDA is fit, then used to initialize W for a newly trained vision model?", "limitations": "limitations not discussed", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "strengths\n- Proposed method is simple yet an effective means of repurposing trained embeddings for a different dowstream task\n- Large improvement in experimental settings over standard pipeline. \n- Notably, they also propose a strong baseline based on linear discriminant analysis\n\nweaknesses\n- There's some prior work transferring text embeddings from vision-language models to visual recognition tasks: [1]\n- The description of the LDA formulation was a bit difficult to follow. It references a pretrained model on a certain training split, but it's not clear which model and which dataset this is referring to.\n- The authors should probably verify whether there's any overlap in training data between the pretrained CLIP model and the dowstream task's data.\n- For correctness' sake, I don't believe it's accurate to state the dimensionality reduction as part of the LDA algorithm, but rather something that can be done within the LDA framework.\n- If dimensionality reduction is performed with LDA, it would be necessary to ablate whether the results would be better or worse without it, as the overall results seem really close to that of the proposed initialization method.\n\n\n\n\n[1] Gupta et al. Aligned Image-Word Representations Improve Inductive Transfer Across Vision-Language Tasks. ICCV2017\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "TWK_fWZTy9i", "reviewer": "Reviewer_KzRg", "summary": "This paper revises the role of widely used classifiers from a novel perspective that has been overlooked. Based on the vision-language pre-trained model, this work introduces text priors into the standard recognition framework in a fixed-classifier fashion without additional training cost. A variety of different fixed classifiers are also discussed, and qualitative and quantitative results are provided. This work presents a simple yet effective paradigm for visual recognition. Prior to this, the traditional recognition paradigm often used pre-trained visual encoders with a randomly initialized classifier to finetune downstream recognition models without considering pre-trained text encoders. This study showed how properly using visual-language pre-trained models can significantly improve recognition performance and speed up model convergence.\n", "questions": "Please refer to my questions in the weakness part for details. ", "limitations": "Yes.", "rating": 7, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\nThis work democratizes training on large-scale video/image datasets (i.e., Kinetics400, ImageNet) to a certain extent, achieving good accuracy with only a few epochs and also performing well in few-shot/zero-shot scenarios. As far as I know, it seems to be the first work to achieve 87+% on Kinetics400 using only publicly available pre-trained models, indicating that it is reproducible and can inspire future works. And it even performs better than works that used large-scale pre-trained models (e.g., JFT-3B/JFT-300M/FLD-900M) that were never open source.\n\nWeakness:\n- The authors may include more discussion about the computational cost and efficiency issues.\n- What are the training details for few-shot video recognition?\n- The authors may include the study of failure cases to help the readers to understand the limitation of this work.\n- What's the benefit of using randomized orthogonal matrix?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Reject