Learning Spatiotemporal Features via Video and Text Pair Discrimination
Authors: Tianhao Li
Limin Wang
Venue:  (2021)
Keywords: Spatiotemporal Feature Learning
Video and Text Pair Discrimination
Self-/Weakly Supervised Learning

Abstract: Current video representations heavily rely on learning from manually annotated video datasets which are time-consuming and expensive to acquire. We observe videos are naturally accompanied by abundant text information such as YouTube titles and Instagram captions. In this paper, we leverage this visual-textual connection to learn spatiotemporal features in an efficient weakly-supervised manner. We present a general cross-modal pair discrimination (CPD) framework to capture this correlation between a video and its associated text. We train our CPD models on both standard video dataset (Kinetics-210k) and uncurated web video dataset (Instagram-300k) to demonstrate its effectiveness. Without further fine-tuning, the learnt models obtain competitive results for action classification on Kinetics under the linear classification protocol. Moreover, our visual model provides an effective initialization to fine-tune on downstream tasks, which yields a remarkable performance gain for action recognition on UCF101 and HMDB51, compared with the existing state-of-the-art self-supervised training methods. In addition, our CPD demonstrates that pre-training on a relatively small dataset is able to yield a comparable performance to those methods of using order magnitude more data, which is meaningful and practicable for the scenarios with limited computational facilities.

Under review as a conference paper at ICLR 2021
LEARNING SPATIOTEMPORAL FEATURES VIA VIDEO AND TEXT PAIR DISCRIMINATION
Anonymous authors Paper under double-blind review
ABSTRACT
Current video representations heavily rely on learning from manually annotated video datasets which are time-consuming and expensive acquire. We observe videos are naturally accompanied by abundant text information such as YouTube titles and Instagram captions. In this paper, we leverage this visual-textual con- nection to learn spatiotemporal features in an efficient weakly-supervin We present general cross-modal pair discrimination (CPD) framework capture this correlation between a video and its associated text. We train our CPD models on both standard video dataset (Kinetics-210k and uncurated web video dataset [Instagram-3000) to demonstrate its effectiveness. Without further fine-tuning, the learnt models obtain competitive results for action classification on Kinetics under the linear classification protocol. Moreover, our visual model provides an effective initialization fine-tune on downstream tasks, which yields a remarkable perfor- mance gain for action recognition on UCF101 and HMDB51, compared with the existing state-of-the-an self-supervised training methods. In addition, our CPD demonstrates that pre-training relatively small dataset is able yield compa- rable performance to those methods of using order magnitude more data, which is meaningful and practicable for the scenarios with limited computational facilities.
manner.
INTRODUCTION
Deep learning has made remarkable progress for visual recogmition in both image and video do- main (Krizhevsky et al. 2012; He al.] 2016 Carreira & Zisserman| 2017: Feichtenhofer et al.] 2018) by training powerful neural networks on large-scale manually annotated datasets (e.g., Ima- geNet (Deng et al. 2009) and Kinetics (Kay et al. 2017). More importantly, it well-established that this supervised pre-training on large-scale datasets would benefit the downstream tasks (e.g., object detection (Ren et al., 2015), pose estimation (He et al.) 2017), and temporal action detec- tion (Zhao et al. 2017)). in particular when the target datasets are relatively small. Yet, annotating a large-scale dataset for training such deep neural networks is costly and cone-consu,ing, and even more challenging for video due to its various temporal structure and complex semanties. As sult, the existing video datasets size is still smaller than ImageNet in terms of training samples and classes. On the other hand, videos typically contain richer structure with abundant side information such as motion (Diba et al. 2019 Ng et al. 2018), audio (Arandjelovic & Zisserman 2017 Korbar al. 2018), and text (Miech et al. 2019 Sun et al. 2019b). So these expected these associated modalities are expected to provide useful cues to learn video representations in more efficient way.
Language text is probably the most natural and easy way describe the semantic information of a video, and the associated textual information could be easily acquired when collecting video dataset (Rohrbach al. 2017 Miech al.) 2019) from Internet or Movie. We argue that this correlation between a clip and its associated text could serve as an alternative supervision to learn video representation from scratch. This is different from some recent works (Sun et al., 2019b) Miech et al.) 2019). in which these abundant textual information has been used to learn high-level visual-text embedding applied to text-to-video retrieval or video captioning. Intuitively, it is more challenging learn general visual representation solely from text information without any human annotation, for reasons such as large numbers of noise in text, lacking careful initialization, and being hard to design an effective objective.
In this paper, we aim to learn effective video representation from noisy and diverse textual infor- mation, which could serves as the hasis for variety of downstream tasks. Basically, learn

---

Under review as a conference paper at ICLR 2021
mapping of text and video into shared embedding space and leverage their correlation supervi- sion signal. The technical difficulty is how to design an effective objective function, that is capable of modeling this complex visual-textual correlation and well easily optimized by training from scratch on noisy datasets. Inspired by unsupervised feature learning in images (Wu et al. 2018 Tian al.] 2019), we present a cross-modal pair discrimination (CPD) framework, which tries to recognize each video and text pair into a class via non por-maaaetrric classifier. To solve the compu- tational issues imposed by the huge numbers of pair classes, we adapt noise-contrastive estimation technique (Gutmann & Hyvärinen 2010) to approximate the original loss function.
Specifically, we learn the CPD framework from web videos with the associated title or caption that could be directly crawled from web platforms such as YouTube (Kay et al.) (217)) and Insta- gram (Duan et al. 2020). We utilize the off-the-shelf language models such as BERT (Devlin et al, 2019) or Word2vec (Mikolov et al. 2013) and devise curriculum learning strategy to progressively train the video models. We first test the generalization ability of learned video representation by CPD the Kinetics dataset (Kay et al. 2017) by using shallow classifiers such k-NN and linear classifier. shows that our learned spatiotemporal features obtain promising results which are com- parable to some supervised learning methods on the Kinetics dataset (Kay al., 2017). Then, investigate the generalization power of learned spatiotemporal features of CPD by fine-tumi fin-tuning on the Kineties (Kay al. 2017). UCF101 (Soomro al. 2012) and HMDB51 (Kuehne et al. 2011) datasets, demonstrating that our method obtain superior performance to previous state-of-the-art self-supervised methods and comparable performance to the very recent methods of using orders of magnitude more videos (70M-100M vs. 0.3M).
2 RELATED WORK
Self/Weakly Supervised Representation Learning. Self supervised representation was popular in both image and video domains by designing various proxy tasks. In image domain, for in- stance, these tasks could be predicting the image context (Doersch et al., 2015)) counting the ob- jects (Noroozi et al.] 2017), converting gray images to color one (Zhang et 2016)) keeping global and local consistency (Hjelm et al. 2019). In video domain, typical examples include frame predic- tion (Diba et al. 2019;; Vondrick et al. 2016)) optical flow estimation (Ng et al.) 2018 Zhou et al.) 2017 7 layaraman Grauman 2017), instance tracking (Wang& & Gupta 2015:
Wang et al. 2019b),
temporal order or structure prediction (Misra et al.] [016]] Fernando et al. 2017 Wei et al. 2018:] X
al. 2019a). These learnt representations may capture some aspects low-level image or video structures, but are generally outperformed by those using cross modal information.
Several cross-modal self-supervised tasks was proposed to enhance -ogglltyy representation power and typical example is audio-visua representation learning (Aytar et al. 2016 Arandjelovic & Zisserman 20177] Korbar et al. 2018). Meanwhile, some weakly-supervised methods were devel- oped by utilizing web supervision obtained in an automatic way, such as query ID (Chen & Gupta Ghadiyaram et al. 2019), and hashtag (Mahajan et al. 2018). Concurrent work (Miech et al,, tried to learn video representations by using narration as supervision with instructional videos (e.g.. HowTo100M (Miech et al. 2019)). However, they are limited by the video type. Our CPD is applicable more general video type and we experiment with much smaller dataset (0.3M vs. 100M) of both PGC and UGC videos, but achieves a similar performance on UCF101 and HMDB51. Concurrent work (Stroud et al. 2020) proposed a similar framework but required more training videos (0.3M vs. 70M) and richer textual information to obtain similar performance to ours.
2015
2020
Motion, Audio, and Text. Multi-modal information in videos provides natural cues for learning deep models. Motion or temporal information has been studied as to design proxy tasks to assist cross-modal leanning such as optical flow or tracking (Ng et al. 2018 Wang Gupta 2013), frame prediction (Diba et al. 2019 Vondrick et al.] 2016). or high-level temporal structure (Wei et al.) Fernando et al.] 2017). As most video contain synchronized audio and visual signals, audio information has served another common modality to supervised visual learning (Aytar ctt al. 2016 Arandjelovic & Zisserman 2017. Korbar et al. 2018). However, both motion and audio information seem to be low-level signals and may lack high-level semantic for crosl learning.
2018 Xu et 20199) 2019a|
Speech or text has been widely studied as another cross- -odall setting in video learning (Sun et al. 2019b, Miech et al. 2019 Dong al. 2019 Miech et al.. 2018 Pan et al. 2016; Plummer et al. 2017]. These works mainly aimed to learn joint video-text embedding where visual and textual
cues are adjacent if they are semantically. However, these works focused on learn high-level visual-
2

---

Under review as a conference paper at ICLR 2021
Space
weists
Pre- procesing
Todt
Merrory
Clip Sampling
2540
3D ENN
Pideo and Joxt Par Plevedersation
Figure 1: The pipeline of our cross-modal pair discrimination (CPD) framework. First, the visual and text are fed into modality-specific networks for feature extraction. Then, the visual and textual features are mapped into common 256-dimensional space. The cross-modal framework is learned via video and text pair discrimination, which tries to make corresponding pairs closer than other inconsistent pairs using softmax criteria. The learnt spatiotemporal features could be deployed directly or fine-tuned for downstream tasks.
textual embedding by using the off-the-shelf models as feature extractors. Instead, our proposed CPD framework addresses different issue of video representation learning from scratch.
CROSS-MODAL PAIR DISCRIMINATION
In this section provide detailed description our proposed cross-modal pair discrimination (CPD) for weakly supervised spatiotemporal feature learning. First, we present the whole framework and analyze its important properties. Then, describe the training strategy of CPD framework. Finally, we introduce text and video feature extraction networks.
3.1 FRAMEWORK AND ANALYSIS
Our goal is to propose a weakly supervised representation learning method by exploiting the corre- lation between each video clip and its associated text information, which could be easily obtained from variety of sources such as YouTube titles, Instagram captions and automatic speech recog- nition (ASR). It is generally assumed that these text information contains semantic information, but also might be noisy and irrelevant. Therefore, from technical perspective, we need to design an effective objective function and training strategy capture this semantic correlation and well also suppress the effect of noisy and irrelevant information. To this end, we devise a video-text pair discrimination objective and curriculum learning strategy follows.
More formally, as shown in Figure [ we aim to learn modality-specific embedding function F, of N video clips and their associated textual
and F, for the visual and textual information from information {(v,, + Let f," and f denote F, (c:) and (t,),, respectively. These embedding functions would map these two modality into a common space (i.e., f" E Rdd and f E Rd), and related visual and text information should be close to each other. The embedding functions could be implemented by neural networks which will be clarified in next section. We first focus on how devise objective function optimize these embedding functions. Inspired by the work of unsupervised learning in images (Wu et al. 20188), we design cross cross-modal pair discrimination objective to learn these two embedding functions.
Self-instance discrimination. In the original instance-level discrimination framework (Wu et al. 2018), each image is treated as a distinct class and would learn a classifier to categorize each image into its own class. This framework could be naturally extended into the setting of video and text pair by directly using feature concatenation, and we call this extension as self-instance discrim- ination. Formally, this video-text level instance discrimination objective could be implemented with the following softmax criterion:
((i((t,))) = exp(w,TF" wTT IN exp(w? + ftt)
(1)
3

---

Under review as a conference paper at ICLR 2021
where the 7t video-text pair define class i, (w.', w!) is a weight for class i, and the class number is equal to training sample number N. This class weight represent a class prototype for each video-text instance and is probably not easy optimize we only have single sample for each class. Thus, the above parametric classifier could be refined with the following non-parametric variant:
exp(f," P Σラ= PO/T
p(il(,, ())
where 7 is a temperature parameter to control the class concentration level and our training objective optimize the likelihood I[;=1 (()) t,)). This straight forward extension shares the advantage of instance-level discrimination by directly modeling in the joint video-text space. Yet, in fact, the semantic information of text modality is higher than video pixels and we aims at learning video features with the supervision of textual information. To meet this requirement, we propose a refined objective function from the perspective of conditional distribution.
Cross-pait discrimination. According to the above analysis, we design the objective function by considering conditional distribution p(is/u) and p(i,|t) rather than implicitly modeling distribution p(u, t). Specifically, we design the following conditional distribution:
exp(f(F("/T)
p(iclu) Σラ exp(C;TP0//)
where ith text define a text class i, and both f and f with unit-norm constraint. The con- ditional distribution p(i,lt) could be defined at the same way. We call this framework as cross-pair discrimination, and during training phase, the objective is to maximize the likelihood 11 ((i II,=1 p(i.)t,). The key difference between Equation (2) and (3) is that we propose to use cross-correlation term p r to replace the self-correlation term (f r + +f (*). This cross cor- relation is more effective to capture the mutual information between visual and textual information, and thereby better at guiding the spatiotemporal feature learning from video with text information as supervision.
Ranking loss. There is common ranking loss for cross modal matching. To well study the ef- fectiveness of proposed cross-modal pair discrimination objective, we also compare with a baseline of ranking loss, which is defined follows:
C(u,, (.) =
Σ ma S(E), - S(F, fii
(4)
where each video v; has a associated text and unrelated text 1 from current batch. S(f), f(") cosine similarity, n is the batch size and 5 is margin. We apply Equation (4) in both ways of video with its associated text and text with its video. In experiment, we empirically compare this ranking loss with our designed cross-pair discrimination objective.
f3') is the
3.2 TRAINING CPD
The training of CPD framework needs to address two technical issues: (1) large number of video-text pair classes; (2) optimization difficulty on noisy video-text datasets by training from scratch.
Noise-contrastive estimation. In training stage, adopt noise-contrastive estimation tech- nique (Gutmann& Hyvärinen, (010)) to approximate Equation (3) to solve the computational issues by the huge numbers of pairs. The basic idea is transform the multi-class classification problem in Equation (3) into set of binary classification problem. In the binary classification task, the task to distinguish between data sample and noise sample. The approximate training objective is to minimize the following loss function:
L = Er(o) (Era(d))) [log h(i,, w)] + mlE,, (1 v))))
(5)
where ((ir,, u) tmp. Pa(telu) is the actual data distribution and P. (i/lt) is the uni- form distribution for noise, and m denotes the noise frequency. To compute p(t,lu) efficiently and avoid large memory consumption, following (Wu al. 2018), maintain memory bank to
p(icl)
4

---

Under review as a conference paper at ICLR 2021
store the visual and textual features for each training pair. The memory bank is updated dynamically during the training procedure.
Curriculum learning. To handle the optimization difficulty of directly training from scratch on noisy video-text dataset, we present a curriculum training strategy by resorting to the existing un- supervised pre-trained language models. To relieve the training difficulty, strategy divides the training procedure into two stages. In the first stage, we fix the pre-trained language model and only update the parameters of visual model and embedding function. The mo- tivation is that the language model is pre-trained well using corpus much larger than ours and the video model is totally trained from scratch. we train both models simultaneously in the beginning, the random noise produced by video model will destroy the parameters of language model. In the second stage, after the good initialization of video model, we start to jointly train the visual-textual model with smaller learning rate.
curriculum learning
3.3 ARCHITECTURE DESIGN
Video architecture. For video representation, tures from video clip. Specifically, we randomly sample 8 frames from each video clip and sam- pling stride is 4. Following the implementation of slow stream in the recent SlowFast (Feichtenhofer et al. 2018), all filters from cont; to res3 degenerate temporal convolutions into 2D convolution kernels and it only reserves 3D convolution kernels in res, and res; without temporal downsam- pling. We try two kinds of network architectures: (1) 3D ResNet34 trained on 112 x 112 x 8 volumes and (2) 3D ResNet50 trained on 224 x 224 x volumes. The first tiny network efficient for abla- tion study and then we transfer its optimal setting to the larger backbone and frame resolution. We also add a mapping layer to transform the visual features into 356-dimensional embedding space f" and this 256-d vector is fz-normalized.
the 3D CNNs extract spatiotemporal fea-
Text architecture. Our textual stream subnetwork is based on the off-the-shelf language models. We choose Word2vec (Mikolov et al., 2013) and DistilBERT (Devlin et al. [2199] Sanh et al., 2019) as our textual encoders. Word2vee is an unsupervised word encoder, pre-trained by reconstructing the surrounding words of the continue sentences. We average word vectors which are 300 dimen- sional as textual encoder. BERT (Devlin et al., 2019) encodes long sentences by predicting the missing words given their bidirectional context, and DistiBBERT achieves comparable performance with a faster and lighter model via knowledge distillation (Hinton et al. (015). We average word embeddings of title generated by DistilBERT and obtain 768 dimensional text feature. Finally, two fully connected layers with ReLU and Batch Normalization (loffe & Szegedy 2015) are added to our textual encoder obtain textual feature ff in the common embedding space, which is also fz-normalized.
EXPERIMENTS
In this section, we present the experimental results of our proposed CPD framework. First, we describe the training and evaluation datasets with implementation details. Then, we conduct ablation study on our proposed CPD framework. Finally, we verify the effectiveness of CPD from two aspects: weakly representation learning and representation transfer.
4.1 DATASETS
In our experiment, we pre-train our CPD framework on two video-text datasets: Kinetics-210s (Kay et al.) 2017) and Instagram-300k (Duan et al. 2020). Then, we fine-tune the video model on three human action datasets: Kinetics400 (Kay et al. 2017). UCF101 (Soomro et al. 2012) and HMDB51 (Kuchne et al. 2011).
Kinetics-210k. Following the recent self-supervised methods (Wang et al. 2019a| Korbar et al.) 2018 Han et al. 2019), we utilize Kinetics (Kay et al.) 2017) dataset for weakly-supervise pre- training of CPD. It often called Kinetics400 since it has 400 action classes, but we count training do not use any class information for weakly supervssp representation learning. Due to invalid urls and data cleaning, the collected dataset contains around 210k video-text pairs, and thus call this dataset Kinetics-210k. To construct video-text pairs, we equip each clip with
video number
5

---

Under review as a conference paper at ICLR 2021
the video title directly crawled from YouTube, termed as Kinetics-title. As the original title may be very noisy, we pre- sre-process the text information in two ways. First, we delete special symbols and characters such as non-English words and emoji, termed as Kinetics-tille-llan. Second, we StanfordNI.P (Qi et al., 2018) to obtain the dependency tree of sentences in titles and only reserve verbs and nouns, named Kinetics-fitle-treee
Instagram- -30k.. To avoid data bias in Kinetics caused by human annotation (i.e., trimmed videos with an action), we further verify the effectiveness our CPD model on an uncurated web video dataset (Duan et al.] 2020). This new dataset is constructed from Instagram by searching action label of Kinetics-400 but without any manual filtering. Due to limited computation resource and also for fair comparison with pretraining on Kinetics-210k, we randomly sample 300k from the original web video dataset, termed as Instagram-300k.
An important difference is that the these videos are with User Generated Content (UGC) and accom- panied by captions uploaded by users. Therefore, its video content distribution much different with those in Profession Generated Content (PGC) in UCF101 and HMDB51, and the text noise is also much higher. So, it is more challenging to train a pre-trained CPD model on Instagram-300k,
UCF101 and HMDB51. We evaluate the generalization of our pre-trained models by fine-tuning on two small human action datasets: UCF101 (Soomro et al. 2012) and HMDB51 (Kuehne et al.) 2011), which contain 13k videos of 101 classes and 7k video of 51 classes respectively. We report ablation study on the first split and report average performance over three splits for fair comparison.
4.2 IMPLEMENTATION DETAILS
Weakly supervised learning of CPD. We train our CPD model video-text datasets and use video-text retrieval on 1k unseen video-text pairs as validation set duration training. Specifically, frames are sampled from each video clip and the sampling stride is 4. We use SGD to optimize our objective and the training parameters include a momentum of 0.9 and le-4 for weight decay. We set temperature parameter т 0.07 and noise frequency m to 4096. In the beginning, we fix the pre-trained language model and the learning rate is set as 0.2. When the retrieval performance on validation set saturates (170 epochs for 31) ResNet34 and 1 110 epochs for 3D ResNet50), we start to update the language model with learning rate of 3e-5 and decrease the rest learning rate to 0.02. The maximize training number is 250 epochs. For input size of 112 x 112 x 8, , the mini-batch size is 64 clips per GPUs and 16 clips per GPUs for input size of 224 x 224 x 8. We use 8 GPUs for training.
Evaluation on representation learning. We first verify our CPD learned representation by em- ploying a shallow classifier frozen features. Specifically, utilize k-Nearest Neighbor (kNN) and linear classifier based on extracted features for classification. For video feature extraction, we sample 10 clips from each video and each clip contains 8 frames with 4 sampling stride. The 256- dimensional embedding feature and the output of global average pooling are extracted as features. averaged
The extracted features over 10 clips in video choose cosine distance as distance metric in kNN and set h = 25. As for linear classifier, fully connected layer after Batch Normalization is added with cross-entropy loss. We adopt Adam with learning rate of 1e-3 and reduce by a factor of 10 every 10 epochs, stopping at 30 epochs.
video-level representation. We
Evaluation on representation transfer. A main goal of representation learning is to transfer them to downstream tasks. We fine-tune the learned spatiotemporal representation on the UCF101, HMDB51 and small fraction of Kinetics400. During fine-tuning, 16 frames with stride 4 are sam- pled as input. We simply replace the embedding layer of video model with new fully-connected layer and multi way softmax for action recognition. The classifier is trained using the SGD opti- mizer with initial learning rate le-2 and weight decay 5e-4. Learning rate is decreased twice by factor of 10 when the validation loss saturates. During testing, for each video, we uniformly sample 10 clips and each clip contains 3 crops, following the common practice (Feichtenhofer et al. 2018).
4.3 ABLATION STUDY
In this study, we pre-train pre-train our CPD models on Kinetics- dataset and choose the task of repre- sentation transfer by fine tuning on UCF101 split 1 for evaluation.

---

Under review as a conference paper at ICLR 2021
Objective function
Accuracy(%)
Training stralegy
Accuracy(%)
Textual enooder
Data
Accuracy(5)
Random init.
50.0
Random init.
50.0
Random Init.
50.0
WordZvec
Tree
83.1
Ranking loss
79.9
Direct line-tuning
81.3
Self instance
learning
82.2
DisilBERT
Tree
82.1
Cross pair Dis.
82.2
Curr. leamnin?
84,2
DistiBERT
Word2vec
Clean
Clean
82.5
84.2
Study loss functions.
(b) Study training strategies.
(c) Study on textual encoders.
Table Ablation study on UCF101 by fine tuning a pre-trained CPD model from Kinetics-210k.
Sup.
Layer (Dim)
NLD-ConNNet
Kay et 2017
Kinetics-100 Label
56.1
ResNet34
Fira ct m
Kinelics- 400 Label
60.1
3D ResNet50
Kinetics 400 T Lahel
73.2
ResNet50
InageNet Label
res5 (2048)
42.8
56.1
ResNet34
Instugram 300% Cuption
emb (256)
34.5
37.3
3D ResNet34
Instagram 300k Caption
res5 (512)
36.1
44.6
3D ResNet50
Instagram-300% Caption
cmb (256)
51.1
51.7
ResNe150
Instagram-300k Caption
3D ResNet34
Kinetics-2T0k TOK Title
emb (256)
49.9
50.8
ResNet34
Kinetics 210k Title
resS (512)
50.1
53.3
3D ResNet50
Kinetics-2108 10k Title
emb (256)
58.0
59.6
3D ResNet50
Kinetics-210k 210k Titee
res5 (2048)
58.2
63.8
Table 2: Evaluation weakly-supervised representation learning without fine-tuning. Top-1 classification accuracy is reported on Kinetics-400 validation set.
Objective function. We compare three objective functions for cross-modal pair discrimination de- scribed in Section We pre-train models by utilizing DistilBERT as textual encoder without fine- tuning and the experimental results are reported in Table la Self-instance discrimination almost has no contribution to learn effective representation as there no cross-modal correlation modeling. Cross-pair discrimination gives better performance than ranking loss as cross-pair discrimination can construct negative video-text pairs from entire dataset while ranking loss is only optimized by negative pairs from current batch. More theoretical analysis be found in Section. A.I of the Appendix.
Curriculum learning. We design different training strategies from noisy video-text datasets. The first strategy is to fine-tune the pre-trained textual encoder directly at the beginning. Then we com- pare with stage I and stage II of curriculum learning proposed in Section 3.2 All these strategies are pre-trained on Kinetics-tille-leann The numerical results are summarized in Table [b] Fixing the pre-trained language model gives better performance than direct fine-tuning at the beginning (+0.9%). We ascribe this to the fact that the random noise produced by video model destroy the well pre-trained textual encoder the beginning. Also, fine-tuning the language model after the video model is well initialized further boost the accuracy by 2.0%.
Different textual information. In this experiment, we choose video-text pairs from Kinetics-title- tree, Sinetics-title-cles datasets and utilize Word2vec and DistilBERT experimental results are reported in Table Ic For textual encoder, abundant and video-specifie text information benefits to train our CPD model with stronger language model such DistilBERT according to the performance difference between Kinetics-title-tres and Kinetics-tille-llann (82.1% 84.2%). As for shallow textual encoder (e.g.. Word2vec), simple text information from Kinetics- title-tree dataset gives better performance than abundant text information (83.1% vs. 82.5%). From above observation, it be concluded that Word2vec is more good at concise and accurate text while DistilBERT can handle more complex and noisy sentences which is close to realistic setting. Also, it is affordable utilize strong language models due to curriculum learning strategy and lightweight DistilBERT model.
textual extractor. The
4.4 EVALUATION ON REPRESENTATION LEARNING
To evaluate our learned representation, report the classification performance on validation set of Kinetics via training shallow classifiers on frozen features as shown in Table 4.3 We perform kNN classifiers and linear classifiers (LC) on the embedding features visual features from global average pooling after res5. this shallow learning setting, we also compare with ImageNet pre- training representation (ResNet50) by using the same classifier. Fürst, the representation learnt from
7

---

Under review as a conference paper at ICLR 2021
Method
Supervision
Backhone
Pro-trained Datnset
UCFTOT
HMDBS
Kandom Init.
Hara ot
2018
3D ResNeti8
42.4
17.1
Kinetics
2018
Action label
ResNet50
Kinetics
89.3
61.0
Supervised SOTA
Xae el m
Action label
S3D
Kinetics
96.8
75,9
Shafll Learn (Misza ct
2016
Order verilication
CatieNet
UCFI01/HNDBSI
50.2
18.1
OPN [Lee al.
201
Sequence order
VGGNet
UCT10L/HMDBSI
59.8
23.8
CMC
Lin of al
20T9
Optical flow
CaffeNet
UCFI01
55.3
Fernando
2017
Odd-one-out
AlexNet
UCF101
60.3
32.5
Wang et (Wang
al 20199
Mation
C3D
Kinetics 400
61.2
33.4
201
Clip order
ResNer
64.9
29.5
DPC
Haa ét
2019
Prediction
3D ResNet34
Kinetics-400
75.7
35.7
CBT
Sun et al
26
Audiot Texty/oomext
S3D
Kinetics-600
79.5
44,6
AVTS
Audio
I3D
Kinctics-600
83.7
53.0
AVTS
Kothar et al
201
Audia
MC3
Audioset- 1.8M
89.0
61.6
XDC
(AWaasse) et
201
Audio
R(2+DD
Kinetics-400
84.2
47.1
XDC (AlWaa5d] 0
2019
Audio
R(2+DD
IG-65M
91,5
63,1
MIL-NCE
(Mrcch el 22120
Miech
Audio(Text)
S3D
HT-1OUM
91.3
61.0
TWS (Strouid
20200
Text (Title, Des, Tag etc.)
S3D-G
WVT 70M
90.3
65.3
CPD (Ours)
Caption
ResNet50)
Instagram3000
89.9
63.8
CPD (Ours)
Tide
3D ResNet50
Kineticx210k
90.5
63.6
Table Evaluation on representation transfer by fine-tuning. We compare our CPD model with other methods trained on different type of supervision.
Kinetics-210k generally outperforms that of Instagram-300k. The reason could be ascribed to the video distribution gap between UGC (Instagram) and PGC (Youtube), and also much noisier textual Second, we compare with ImageNet pretrained features, and our CPD representation better under the same backbone. Finally, we compare with some end-to- end trained representations with action labels, and there is still a performance gap between our representation and supervised end-to-end representation (c.g. 63.8% vs. 73.2%).
information in Instagram-300
4.5 EVALUATION ON REPRESENTATION TRANSFER
Transferring learned representation downstream tasks main goal of representation learning. We transfer them to action recognition task on small datasets, namely UCF101 and HMDB51. We compare our CPD) model pre-trained on Instagram-300k and Kinetics-210K with randomly ini- tialized network, self-supervised methods solely based on visual information, including Shuffle & Learn Miisra al., 2016). CMC (Tian et al., 2019). MASN (Wang et al.) (019a)). COP (Xu et al.) 2019b). DPC (Han et al. 2019) and so and representation learning methods based on multi- modal information (e.g., audio, text), including CBT (Sun al. 2019a), AVTS (Korbar et al. 2018) XDC (Alwassel et al. 2019), MIL-NCE (Stroud al. (020). and TWS (Stroud et al. 2020).
As shown in Table our CPD models generally outperform those self-supervised learning ap- proaches of only using visual information (2 10% UCF101 and > 20% on HMDB51), which indicates that cross-modal information is useful cue for visual representation learning. Meanwhile, CPD representations obtain comparable performance to the concurrent works (i.e., MIL-NCE and TWS) of using text as weak supervision. However, our CPD uses a much smaller pre-training dataset of around 0.3M videos, while the other methods uses 70M-100M videos. Training CPD model on a such large-scale dataset is almost impossible for a university lab with limited computa- tional facilities. Our work demonstrates that pre-tree traatrinn a relatively small video-text dataset is also possible to match the SOTA performance, and this is quite meaningful and practicable for university lab. Finally, we notice that the gap of CPD models learned from Instagram-3000 300k and Kinetics-200k is very small, indicating that our CPD model can effectively handle high noise in text.
CONCLUSION
In this paper, we have presented a general cross-modal pair discrimination (CPD) framework to capture the correlation between video clip and its associated text from real word and adopt noise- contrastive estimation to approximate the objective. Without fine-tuning, the learned models obtain competitive results for action classification Kinetics dataset with shallow classifier. Also, visual models provide an effective initialization to fine-tune on the datasets of downstream task, and matches the state-of-the- the---- performance with much smaller pre-training dataset.
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
Humam Alwassel, Dhruv Mahajan, Lorenzo Torresani, Bernard Ghanem, and Du Tran. Self- supervised learning by modal audio-video clustering. CoRR, abs/1911.12667, 2019, URL
http://arxiz: org. 1911 126667
Relja Arandjelovic and Andrew Zisserman. Look, listen and learn. In IEEE International Confer- ence on Computer Vision, ICCV 2017, Venice, Italy, October 22-29, 2017, pp. 609-617, 2017.
Yusuf Aytar, Carl Vondrick, and Antonio Torralba. Soundnet: Learning sound representations from unlabeled video. In Advances in Neural Information Processing Systems 29: Annual Conference on Neural Information Processing Systems 2016, December 5-10, 2016, Barcelona, Spain, pp. 892-900, 2016.
João Carreira and Andrew Zisserman. Quo vadis, action recognition? new model and the kinetics dataset. In 2017 IEEE Conference Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017, pp.
2017.
Xinlei Chen and Abhinav Gupta. Webly supervised learning of convolutional networks. In 2015 IEEE International Conference on Computer Vision, ICCV 2015, Santiago. Chile, December 7- 13, 2015, pp. 1431-1439, 2015.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Fei-Fei Li. Imagenet: A large-scale hierarchical image database. In 2009 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR 2009), 20-25 June 2009, Miami, Florida, USA, pp. 248-255, 2009.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume I (Long and 2019.
Short Papers), pp.
Ali Diba, Vivek Sharma, Lue Van Gool, and Rainer Stiefelhagen. Dynamonet: Dynamic action and motion network. CoRR, abs/ 1et/1994.111 1407, 2019.
Carl Doersch, Abhinav Gupta, and Alexei Efros. Unsupervised visual representation learning by context prediction. In 20/5 IEEE International Conference on Computer Vision, ICCV 2015, 2015.
Santiago, Chile, December 7-13, 2015, pp.
Jianfeng Dong, Xirong Li, Chaoxi Xu, Shouling Ji, Yuan He, Gang Yang, and Xun Wang. Dual encoding for zero-example video retrieval. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, pp. 9346-9355, 2019.
Haodong Duan, Yue Zhao, Yuanjun Xiong, Wentao Liu, and Dahua Lin. Omni-sourced webly- supervised learning for video recognition. CoRR, abs/2003.13042, 2020.
Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, and Kaiming He. Slowfast networks for video recognition. CoRR, abs/18 wtss/1812.n. 12.03982, 2018
Basura Fernando, Hakan Bilen, Efstratios Gavves, and Stephen Gould. Self-supervised video repre- sentation learning with odd-one-out networks. In 2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017, pp. 5729-5738, 2017.
Deepti Ghadiyaram, Du Tran, and Dhruv Mahajan. Large-scale weakly-supervised pre-training for video action recognition. IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA. June 16-20, 2019, pp. 12046-12055. 2019.
Michael Gutmann and Aapo Hyvärinen. for unnormalized statistical models. In Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, AISTATS 2010, Chia Laguna Resort, Sardinia, Italy, May 13- 15, 2010, pp. 297-304, 2010.
estimation: A new estimation principle

---

Under review as a conference paper at ICLR 2021
Tengda Han, Weidi Xie, and Andrew Zisserman. Video representation learning by dense predictive coding. CoRR, abs/ 19990099996666 2019.
Kensho Hara, Hirokatsu Kataoka, and Yutaka Satoh. Can spatiotemporal 3d cnns retrace the history of 2d cnns and imagenet? In 2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018, pp. 6546-6552 2018.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pp. 770-778, 2016.
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross B. Girshick. Mask R-CNN. In IEEE In- ternational Conference on Computer Vision, ICCV 2017, Venice, Italy, October 22-29, 2017, pp- 2980-2988, 2017.
Geoffrey E. Hinton, Oriol Vinyals, and Jeffrey Dean. Distilling the knowledge in a neural network. CoRR, Abs/1503.002331 2015.
R. Devon Hjelm, Alex Fedorov, Samuel Maroii-Mrr Karan Grewal, Philip Bachman, Adam Trischler, and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019, 2019.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In Proceedings of the 32nd International Conference on Machine Learning, ICML 2015, Lille, France, 6-11 July 2015, pp. 448-456, 2015.
Dinesh Jayaraman and Kristen Grauman. Learning image representations tied to egomotion from unlabeled video. International Journal of Computer Vision, 125(1-3):136-16) 2017.
Will Kay, João Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijaya- narasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, Mustafa Suleyman, and Andrew Zisserman. The kinetics human action video dataset. CoRR, abs/1 705.06950 2017.
Bruno Korbar, Du Tran, and Lorenzo Torresani. Cooperative learning of audio and video models from self supervveed synchronization. In Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, 3-8 December 2018, Montréal, Canada, pp. 7774-7785, 2018
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. Imagenet classification with deep con- volutional neural networks. In Advances in Neural Information Processing Systems 25: 26th Annual Conference Neural Information Processing Systems 2012. Proceedings of meeting held December 3-6, 2012, Lake Tahoe, Nevada, United States., pp- 1106-1114, 2012.
Hildegard Kuehne, Hueihan Jhuang, Estíbaliz Garrote, Tomaso A. Poggio, and Thomas Serre. HMDB: A large video database for human motion recognition. In IEEE International Confer- ence on Computer Vision, ICCV 2011, Barcelona, Spain, November 6-13, 2011, pp. 2556-2563, 2011.
Hsin-Yin Ying Lee, Jia-Bin Huang, Maneesh Singh, and Ming-Hsuan Yang. Unsupervised representa- tion learning by sorting sequences. In IEEE International Conference on Computer Vision, ICCV 2017, Venice, Italy, October 22-29, 2017, pp. 667-676, 2017.
Dhruv Mahajan, Ross B. Girshick, Vignesh Ramanathan, Kaiming Hc, Manohar Paluri, Yixuan Li, Ashwin Bharambe, and Laurens van der Maaten. Exploring the limits of weakly supervised pretraining. In Computer Vision ECCV 2018 15th European Conference, Munich, Germany, September 8-14, 2018, Proceedings, Part I1, pp. 185-201. 2018.
Pascal Mettes and Cees G. M. Snock. Spatial- object embeddings for zero-shot localization and classification of actions. In The IEEE International Conference on Computer Vision (ICCV), Oct 2017.
Antoine Miech, Ivan Laptev, and Josef Sivic. Learning a text-video embedding from incomplete and heterogeneous data. CORR, abs/ 804.02516, 2018.
10

---

Under review as a conference paper at ICLR 2021
Antoine Miech, Dimitri Zhukov, Jean- Sann-Bapistte Alayrac, Makarand Tapaswi, Ivan Laptev, and Josef Sivic. Howto100m: Learning a text-video embedding by watching hundred million narrated video clips. CoRR, abs/1906.03327, 2019.
Antoine Miech, Jean-Baptiste Alayrac, Lucas Smaira, Ivan Laptev, Josef Sivic, and Andrew Zisser- man. End-to-end learning of visual representations from uncurated instructional videos. In 2020 IEEE/CVF Conference on Computer Vision and Puttern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020, pp. 9876-9886, 2020.
Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word represen- tations in vector space. In Yoshua Bengio and Yann LeCun (eds.), 1st International Conference on Learning Representations, ICLR 2013, Scotisdale, Arizona, USA, May 2-4, 2013, Workshop Track Proceedings, 2013.
Ishan Mísra, C. Lawrence Zitnick, and Martial Hebert. Shuffle and learn: Unsupervised learning using temporal order verification. In Computer Vision - ECCV 2016- 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings. Part I, pp. 527-544, 2016.
Joe Yue-Hei Ng, Jonghyun Choi, Jan Neumann, and Larry S. Davis. Actionflownet: Learning motion representation for action recognition. In 2018 IEEE Winter Conference on Applications of Computer Vision, WACV 2018, Lake Tahoe, NV, USA, March 12-15, 2018, pp. 1616-1624, 2018.
Mehdi Noroozi, Hamed Pirsiavash, and Paolo Favaro. Representation learning by learning to count. In IEEE International Conference on Computer Vision, ICCV 2017, Venice Italy, October 22-29, 2017, pp. 5899-5907, 2017.
Yingwei Pan, Tao Mei, Ting Yao, Houqiang Li, and Yong Rui. Jointly modeling embedding and translation bridge video and language. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas. NV, USA, June 27-30, 2016, pp. 4594-4602 2016.
Bryan A. Plummer, Matthew Brown, and Svetlana Lazebnik. Enhancing video summarization via Dision-language embedding. In 2017 IEEE Conference on Computer Vision and Pattern Recog- nition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017, pp. 1052-1060, 2017.
Peng Qi, Timothy Dozat, Yuhao Zhang, and Christopher Manning, Universal dependency parsing from scratch. In Proceedings of the CoNLL 2018 Shared Task: Mulrilingual Parsing from Raw Text to Universal Dependencies, pp. 160-170, Brussels, Belgium, October 2018. Association for Computational Linguistics.
Shaoqing Ren, Kaiming He, Ross B. Girshick, and Jian Sun. Faster R-CNN: towards real-time object detection with region proposal networks. In Advances in Neural Information Processing Systems 28: Annual Conference Neural Information Processing Systems 2015, December 7-12, 2015, Montreal, Quebec, Canada, pp. 91-99, 2015.
Anna Rohrbach, Atousa Torabi, Marcus Rohrbach, Niket Tandon, Christopher Joseph Pal, Hugo Larochelle, Aaron C. Courville, and Bernt Schiele. Movie description. International Journal of Computer Vision, 123(1):94-120, 2017.
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, distilled version of BERT: smaller, faster, cheaper and lighter. CoRR, abs/1910.01 108, 2019.
Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. UCF101: A dataset of 101 human actions classes from videos in the wild. CORR, abs// 2012.
Jonathan C. Stroud, David A. Ross, Chen Sun, Jia Deng, Rahul Sukthankar, and Cordelia Schmid.
Learning video representations from textual web supervision. CoRR, abs/2007.14937,, 2020. URL
14937
Chen Sun, Fabien Baradel, Kevin Murphy, and Cordelia Schmid. Contrastive bidirectional trans- former for temporal representation learning. CORR, abs/1 htts/190.055433. 2019a.
Chen Sun, Austin Myers, Carl Vondrick, Kevin Murphy, and Cordelia Schmid. Videobert: A joint model for video and language representation learning. CoRR, abs/1904.01766. 2019b.

---

Under review as a conference paper at ICLR 2021
Yonglong Tian, Dilip Krishnan, and Phillip Isola. abs/1906.05849, 2019.
Contrastive multiview coding.
CoRR,
Carl Vondrick, Hamed Pirsiavash, and Antonio Torralba. Anticipating visual representations from unlabeled video. In 2016 IEEE Conference on Computer Vision and Puttern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pp. 98-106, 2016.
Jiangliu Wang, Jianbo Jiao, Linchao Bao, Shengfeng He, Yunhui Liu, and Wei Liu. Self-supervised spatio-temporal representation learning for videos by predicting motion and appearance statistics. In The IEEE Conference Computer Vision and Pattern Recognition (CVPR), June 2019a.
Xiaolong Wang and Abhinav Gupta. Unsupervised learning of visual representations using videos. In 2015 IEEE International Conference on Computer Vision, ICCV 2015, Santiago, Chile, De- cember 7-13, 2015, pp. 2794 2802, 2015.
Xiaolong Wang, Allan Jabri, and Alexei A. Efros. Learning correspondence from the cycle- consistency of time. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, pp. 2566-2576 2019b.
Donglai Wei, Joseph J. Lim, Andrew Zisserman, and William T. Freeman. Learning and using the arrow of time. In 2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018, pp. 8052-8060, 2018
Zhirong Wu, Yuanjun Xiong, Stella X. Yu, and Dahua Lin. Unsupervised feature learning via non- parametric instance discrimination. In 2018 IEEE Conference Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT. USA, June 18-22, 2018, pp. 3733-3742, 2018.
Saining Xie, Chen Sun, Jonathan Huang, Zhuowen Tu, and Kevin Murphy. Rethinking spatiotem- poral feature learning: Speed-accuracy trade-offs in video classification. ECCV, pp. 318-335, 2018.
Dejing Xu, Jun Xiao, Zhou Zhao, Jian Shao, Di Xie, and Yueting Zhuang. Self-super spa- tiotemporal learning via video clip order prediction. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, pp. 10334-10343, 2019a
Dejing Xu, Jun Xiao, Zhou Zhao, Jian Shao, Di Xie, and Yueting Zhuang. Self-supervised spa- tiotemporal learning via video clip order prediction. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2019b.
Richard Zhang, Phillip Isola, and Alexei A. Efros. Colorful image colorization. In Computer Vision ECCV 2016 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings. Part III. pp. 649-666, 2016.
Yue Zhao, Yuanjun Xiong, Limin Wang, Zhirong Wu, Xiaoou Tang, and Dahua Lin. Temporal action detection with structured segment networks. In IEEE International Conference on Computer Vision, ICCV 2017, Venice, Italy, October 22-29, 2017, pp. 2933-2942, 2017.
Tinghui Zhou, Matthew Brown, Noah Snavely, and David G. Lowe. Unsupervised learning of depth and ego-motion from video. In 2017 IEEE Conference Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017, pp. 6612-6619, 2017.
12

---

Under review as a conference paper at ICLR 2021
A TRAINING DETAILS OF CPD
we adopt conte-contrestive estimation technique (NCE) to approximate objective function Equa- tion (5) in our main paper. The purpose NCE transform the multi-class classification problem into a set of binary classification problems by comparing data distribution against noise distribution. So ps is noise distribution and we formalize it as a uniform distribution: Pn N where N the number of video-text pairs. h(4,, v) is the posterior probability of feature from the data distribution which means video and text are matched. m the number of negative pairs and we set as 4096. For each video feature f", we take its related text feature f and sample 4096 unrelated text features fÿ which are all from memory bank. The FPS of training videos are 30. The code of CPD will be released.
A.1 ANALYSIS ON DIFFERENT LOSS FUNCTIONS
More insight about why our loss is better than ranking loss could be found from gradient back- propagation. Let f+ and f represent the associated and unrelated text feature. For ranking loss, the negative gradient w.r.t f" is f ft if L > 0 else O. For CPD loss, is [] h(& w)]/rf(+ Σ h(1 w)/Tf'_ We observe our loss assign different weights to different examples based on their posterior probability h, which helps learn from hard examples while the ranking loss treats them equally.
B REPRESENTATION TRANSFER ON KINETICS
The Amount of Labeled Data
Method
1%
10%
20%
From scratch
0.3
10.7
33.3
ImageNet Inflation
12.8
36.8
45.7
Ours (Instagram-300 -30k))
18.7
41.3
47.4
Ours (Kinetics-210%
25.9
43.1
47.8
Table 4: Results of classification with small amount of labeled data on Kinetics-400 validation set (showing top-1 accuracy). We utilize 3D ResNet34 as backbone and pre-train it on Kinetics-2100 and Instagram-210k.
Our supervse pre-trained representation can be an efficient initialization when training the model with only small amount of labeled data. We randomly choose a small fraction of Kinetics- 400 training set as labeled data and fine-tune the pre-trained model on it. We report the performance of top-1 accuracy which is trained on labeled subset of 1%, 10% and 20% of the entire dataset in Table 4 We compare our method with training from scratch and ImageNet inflated model as baselines. Our method significantly surpasses the baselines
all present proportion of labeled subset especially when the amount of labeled data is extremely small. When only 1% of data is labeled, training from scratch can not learn anything yet our model achieves 18.7% and 25.9% top-1 accuracy. Both our CPD pre-trained models on Instagram and Kinetics outperform the ImageNet pre-traine pre-trained models.
C EVALUATION ON ZERO-SHOT CLASSIFICATION
We evaluate our visual-textual embedding of CPD model with zero-shot classification on UCF101 and Kinetics-400 without any fine-tuning in Table We transform class labels and video clips into the same embedding space and recognize the video clip to closest class with cosine distance. We compare our method with Mettes et al. (Mettes & Snoek 2017) which realizes zero-shot localiza- tion and classification of human action in video via spatial-aware object embeddings on UCF101. Following (Mettes & Snock 2017), we select different classes for 10 times and average their accu- racies for testing except the class number is 101. We outperform for every number of testing classes. For Kinetics-400, addition, top-1 accuracy of 20 random classes reaches to 74.4%, which shows the strong capability of our visual-textual embedding.
achieve top-1 I accuracy of 43.7% without fine-tuning and training label. In
13

---

Under review as a conference paper at ICLR 2021
UCF-101
Kineties-400
Methods
Train
Test
Split
Acc.
Train
Test
Split
Acc.
Mettes (Mettes & Snoek 2017)
101
3
32.8
Ours(3D O ResNet34)
101
3
40.6
400
1
38.2
Ours(3D ResNet50)
101
39.9
400
1
43.7
Mettes MMttes & Snoek 2017)
50
10
40.4
Ours(3D Oesse3)RReNet::4)) ResNet34)
50
10
47.2
100
10
55.3
Ours(3D ResNet50)
50
10
44.8
100
10
57.4
Mettes (Mettes Snoek 2017)
20
51.2
20
10
54.4
20
10
73.1
Ours(3D ResNet50)
20
10
58.1
20
10
74.4
Table 5: Top-1 accuracy of zero-shot classification on UCF-101 and Kinetics-400. We outperform other methods without any extra labeled data and training procedure after pre-training Kinetics- 210k.
D ANALYZE TEXT INFORMATION
D.I ANALYSIS ON KINETICS TITLE
Datasets Kinetics-title-tree
Least One(%)
All(%)
Rel(%)
90.5
44.3
46.3
Kinetics-title-clann title-clean
91.6
38.4
26.0
Table 6: Analyze text information of Kinetics-2100 datasets. At Least One: The proportion of text information that contains at least word in action classes of Kinetics-400. All: The proportion of text information that contains the entire action class. Rel: The proportion of word in text information that is relevant action classes.
We provide an analysis of text information we used and the result in Table 6 First, there exists large overlap between action class and text information (more than 90% for least one word and more than 38% % for complete action class). However, the titles also contain many other words and noisier information than action classes. Only 26% of words Kinetics-title-clen are relevant to action classes.
Figure 2: List of top 10 and bottom 10 kinetics classes sorted by the frequency of at least word dataset. Zoom in for more details. We also report the per-class accuracy of top 10 and bottom 10 classes sorted by word overlapping in Figure and see that this accuracy is not positively correlated with word overlapping percentage. Finally, we provide some examples of videos and their titles from Kinetics-22 -20kk in Figure
in label occurring in according title of
D.2 VISUALIZATION OF INSTAGRAM CAPTION
Since videos from Instagram- are not annotated or filtered by human, both of their visual and textual information are very noisy. Figure 4 demonstrates some examples of videos and their associ- ated captions. Figure 4 presents an example of high-quality video and relative accurate caption that both are about folding napkin. Many captions deseribe some useful information but also contain noisy text that is not related to video content (c.g., summerdays and gettingtattooed in Figure 4 and very long sentences in Figure 4d). In addition, there some correct but not totally accurate
14

---

Under review as a conference paper at ICLR 2021
THE MINAION
PREMIAZION
My Quartet Buskers Festival - Ferrara 2013
(a)
77kg weight class, 95kg clean and jerk
bb
Old Fashioned Watermelon eating contest in Constableville, plus ride Canada!!!
(c)
Bad Driver of the Week 8/10/2012 Car Almost Wrecks Honda CBR
(d)
Quinn on the bagpipes playing "Amazing Grace" with the Choir during Communion- SHS Graduation 2013
(e)
Howcze ccm
How to Dribble Faster Basketball Moves
(f)
Figure 3: Examples of video and title pairs from Kinetics-210k.
descriptions. Figure 4 shows that the action in video is shot putting rather than spinning (appears in associated caption). Figure illustrates that a person is climbing but its caption is mainly about high jumpping. Figure 4e shows that video content can also be noisy due to low video quality and shot transformation.
15

---

Under review as a conference paper at ICLR 2021
Bird of Paradise Napkin Fold #napkinfolding linakiinrrt #napkin #yellow #napkinfold #birdofparadise mkidscrafts #dly #tutorial #decoration #tablesetting
(a)
#roadtrip from #britishcolumbia to Halberta. drivingallnight
#mountains
#summerdays #summerlovin
#highway #music #playlist
#carpoolkarakee #notreallyasinger #bbutitry Iggettingtattooed
#buyingatruck #duoting
(b)
Day learning to spin
work progress having fun with it!
(c)
Good morning beautiful people of #god
Agrateful another day he given aprayingfor #guidance and Sprotection in Alesusname Aprry Ramen
Early adinnersettings today #potroastbeef #bakechicken #vegetables
EAT WITH THIS? #pyjamachef #tooblloger
BRICEANDPEAS
Apyjemachef Whealthyesting
#alltypeoffood #lovecooking
(d)
My pets are unruly... Who wants them...?? #computerwork #selfishpets
(e)
High Jump Challenge Very funny @ #bouldee #highjump #ffinness #climbing #n ininjawrrrior minnjaaariorswitterlaad #challenge nuunny
(f)
Figure 4: Examples of videos and their associated captions from Instagram-3006
16


Reviews:
[{"review_id": "bflqN3UBA0A", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Advantage of this paper:\n1.\tThis paper is well motivated and well written and the topic of this paper is valuable.\n2.\tIt’s interesting to utilize text as weak supervision for video representation learning, and the experiment results also indicate effectiveness of the learned video representation.\n\nWeakness of this paper\n1.\tThe novelty of this paper might be limited. Previous works have explored the possibility of utilizing text as weak supervision for video representation learning (MIL-NCE), from the reviewer’s perspective, the main difference is that the different loss function is adopted. \n2.\tCompared with methods that adopt other information (such as audio) as weak supervision, there is an inherent advantage of using text as supervision since pretrained text models such as BERT can be utilize as a guidance. So a meaningful comparison would be the comparison with TWS and MIL-NCE, although the proposed method can achieve comparable performance with other methods with much less data, the author does not give analysis about what design in the proposed method that enables this. \n3.\tThe performance comparison is not convincing enough. From Table 3, we can see that different backbones are used for different methods, the reviewer worries that the superiority of the proposed method might be brought by a stronger backbone.  \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "SHSm2pF5Ij9", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes a weakly supervised method for learning spatiotemporal features by video and text pair discrimination, namely cross-modal pair discrimination (CPD). This can be considered as an extension of (Wu et al. 2018) to video and text. On technical perspective, the original method Wu et al. is applied on images, while CPD is applied on video and text (video's title for Kinetics or hashtag search for Instagram). The most novel technical contribution of this paper is making Wu et al. 2018 cross-modal (between video and text). However, compared with Wu et al. 2018, it requires more supervision (weakly supervised vs. unsupervised). On the experiments, some comparisons are unfair and some experimental setups are biased (detail below).\n\n(+) Pros\n- Making Non-Parametric Instance Discrimination (Wu et al. 2018) cross-modal is interesting.\n\n(-) Cons\n- Novelty is marginal. As mentioned above the main contribution is making Non-Parametric Instance Discrimination cross-modal. But at the same time, the weakly supervision setup make it less significant compared to the original work. Moreover, cross-modal has been used before for cross-model between audio-visual (L3Net, AVTS, XDC), video-speech (Sun et al. 2019b, Meich et al. 2019). (The reviewer is open to this level of novelty if the experiments are solid, see next for comments on experiments).\n\n- Experiments contains some unfair comparisons and some experimental setups are biased.\n1. Table 3 compares CPD with other self-supervised methods. Note that CPD is a weakly supervised method. The right one to compare with CPD may be Ghadiyaram et al. CVPR'19. The author(s) may argue that this work used a lot more data than CPD. That is true, but they can train a baseline network, e.g. same backbone as CPD, with a cross-entropy loss on Kinetics-title-clean and Instagram-300k using title or search query as label (similar to Ghadiyaram et al. CVPR'19). This is a fair and important baseline to understand the proposed method of CPD.\n2. There are potentially biased setup in Kinetics experiments. Recall how Kinetics was collected and annotated [Kay at et. 2017]. Videos are searched and retrieved from Youtube using Kinetics-taxonomies (mostly contain a verb followed by a noun), then further verified by human annotator. This means if this paper uses titles of the Kinetics videos, it is very likely that it has the correct verb+noun combination of the ground-truth taxonomy for that video. Moreover, not sure if the authors use the temporal annotations (10 second segment of video) when they sample clips from Kinetics videos (the paper mention about how long the clips are and temporal striding, but never mention if they ignore or use Kinetics temporal annotation). \n\n*Some minor comments:\n- abstract: pre-training a relatively small dataset -> pre-training on a relatively small dataset\n- introduction:  So these expected these associated -> So these associated\n- missing i, j indices for f^u, f^t in Eq(1)? Similarly for Eq(2)?\n\n\n\n\n\n\n\n\n\nWu et al. Unsupervised Feature Learning via Non-Parametric Instance Discrimination, CVPR 2018.", "extra_scores": {}, "extra_text": {}}, {"review_id": "45RjaN9deCh", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "*Summary:*\nThe paper proposes an approach to learn a video feature backbone in an unsupervised manner through the use of video titles (text modality) associated with user generated content from Youtube or Instagram. The key idea is to use a contrastive loss that increases the similarity score between a positive pair vs. a negative pair. Contrary to previous works in this direction that require millions to hundreds of millions of paired clips, this work shows that good performance can be achieved by using much fewer (on the order of 100k) clips. The learned video model achieves good performance on standard action recognition datasets.\n\n*Strengths:*\n1. Shows that videos and their titles can be used to learn video features from scratch in an unsupervised manner.\n2. Can achieve comparable performance with much less data in the scale of few 100K videos (rather than 100M clips as done in [Miech, et al. CVPR 2020]).\n3. Evaluation on Kinetics-400, UCF-101, and HMDB shows comparable performance. Ablation studies are quite insightful (especially comparison between ranking loss vs. proposed CPD). It is also nice to see the performance difference by using kNN vs. classifier on Kinetics-400.\n\n*Weaknesses:*\n1. Novelty of the method itself (not the task to which it is applied) seems fairly limited. In fact, a majority of the model development resembles that of [Wu, et al. CVPR 2018]. However, to me, the motivation to arrive at cross-pair discrimination from this direction is unclear. Almost every paper on cross-modal learning in the last five+ years leverages some form of text feature, visual feature, and computes a dot-product between them (as indicated in the ranking loss). In such a case, what do we learn by approaching this problem using the self-instance discrimination which also performs very poorly? An additional complexity is with respect to the memory bank, but I believe this is only used for improving the estimation of the denominator in Eq. 3, right?\n\n2. Experimental details:\n(i) Some statistics about the datasets: Kinetics-210K and Instagram-300K would be nice to include. What is the average video duration? How many clips are obtained from each video? (the latter is especially important as HowTo100M also has about 1.x M videos, but 100+ M clips, albeit with separate text labels for each clip).\n(ii) Why are YouTube videos considered PGC while Instagram UGC?\n(iii) Since the temperature parameter is so specific, 0.07, I wonder how sensitive is the method to this parameter. An ablation with tau = [0.05, 0.1] would be nice, perhaps on the Kinetics-400?\n(iv) Based on the answer to point (i), could the audio-based supervision from Audioset-1.8M be considered a comparable sized dataset?\n(v) Since a major emphasis of the paper is on the smaller size of the data that allows learning similar performing representations, it would be nice to include additional details such as time taken to complete entire training on the 8 GPUs mentioned and perhaps an estimated comparison to larger datasets.\n\n3. Minor points:\n(i) Impact of curriculum learning is limited. While reading the article, it felt like this may play a large role - especially with sentences like \"If we train both models simultaneously in the beginning, the random noise produced by video model will destroy the parameters of language model.\" Clearly, 0.9% difference (between freezing language model or not) in performance is not a destruction of language model parameters. Would request authors to please tone this down.\n(ii) Typo in introduction: \"So these expected these associated modalities ...\" some text is repeated.\n(iii) Sec 3.1, it should probably be i.e., f^v_i \\in R^d and \"f^t_i\" \\in R^d\n(iv) Consider rephrasing grammar in the first sentence of the \"Ranking loss.\" paragraph. Also, has \"a\" associated --> has \"an\" associated\n(v) Typo: \"that the these videos\" below Instagram-300k.\n(vi) Table 4.3 --> Table 2?\n(vii) MIL-NCE (Stroud et al. 2020) --> MIL-NCE (Miech et al. 2020)\n\n*Overall rating:*\nI am generally in favor of accepting this paper given the comparable performance while requiring several orders of magnitude less dataset sizes, so that an academic lab may be able to train models. Among the open questions, I am curious about weakness point 1, and the inspiration to write this work from the point of view of self-instance discrimination.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "2xjYrv3qllc", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper concerns the problem of learning video representation from paired video-text pairs. The proposed framework is weakly-supervised as the text associated with videos comes from user-provided YouTube titles or Instagram captions. The proposed method uses standard visual encoder and textual encoder and similarity measurement for the joint embedding space. Overall, the paper is written in good clarity and has shown decent improvements over some of the existing methods. But the contributions are somewhat incremental considering the numerous existing/concurrent work built upon contrastive learning for video. The reasons are as follows.\n\ni) Some of the claims are not well-substantiated. For example, on page one bottom, the paper claims that existing works learn \"high-level visual-text embedding\" as opposed to \"video representation\" as in this work, which is not true considering papers such as Miech et al., 2020.\n\nAnother notable claim in the paper is that \"[...] Our work demonstrates that pre-training a relatively small video-text dataset is also possible to match the STOA performance [...]\", which is somewhat over-stated given that the pre-training data is curated on a quite restricted domain (e.g., human actions) and sometimes containing manually-curated data, and could intuitively benefit downstream tasks about the same domain (all tested datasets fall into this category) vs. instructional or more generic pre-training data. Besides, there are essential questions that remain to be answered, such as whether bringing in more data could further benefit the model.\n\nii) The technical contributions and empirical results are incremental (e.g., Tab. 3). Some experimental results are not comprehensive, for instance, Tab. 2 should involve SOTA methods evaluated under the same setting to demonstrate model effectiveness.\n\nOther minor comments:\n\ni) PGC and UGC on page two are mentioned before defining.\n\nii) It's unclear what f^v and f^t are in Eq. 1. How are they related to f_i^v and f_i^t.\n\niii) What does the Tab. 4.3 on page seven refers to?", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper presents an approach for weakly supervised pre-training for videos using textual information provided with web videos on Youtube and Instagram.

## Strength
* The work shows strong results with relative small dataset and computational resources compared to other work in the area of self/weakly supervised learning for videos.
* Interesting ablations

## Main Concerns
* The authors don't discuss and compare to the weakly supervised work [Ghadiyaram et al. CVPR'19] adequately. Furthermore, the authors characterize the work incorrectly in their author response as detailed by R2. I agree to R2 here and like to highlight the concern is not that the method of [Ghadiyaram et al. CVPR'19] being similar to this work but the level/type of supervision.
* Limited novelty over prior work.

## Further Concerns
* Some unclarities
* The authors did not provide an updated revision of the pdf

Overall the paper received reject and borderline scores after author response and discussion (With the strongest score 6 from R1) due to the concerns concerns listed above apart from the ability to work with small number of data. I think the missing comparison to  [Ghadiyaram et al. CVPR'19] which operates in a similar setting weights strongly and I recommend reject.
Decision: Reject