Learning Spatiotemporal Features via Video and Text Pair Discrimination

1 INTRODUCTION . Deep learning has made a remarkable progress for visual recognition in both image and video domain ( Krizhevsky et al. , 2012 ; He et al. , 2016 ; Carreira & Zisserman , 2017 ; Feichtenhofer et al. , 2018 ) by training powerful neural networks on large-scale manually annotated datasets ( e.g. , ImageNet ( Deng et al. , 2009 ) and Kinetics ( Kay et al. , 2017 ) ) . More importantly , it is well-established that this supervised pre-training on large-scale datasets would benefit the downstream tasks ( e.g. , object detection ( Ren et al. , 2015 ) , pose estimation ( He et al. , 2017 ) , and temporal action detection ( Zhao et al. , 2017 ) ) , in particular when the target datasets are relatively small . Yet , annotating a large-scale dataset for training such deep neural networks is costly and time-consuming , and even more challenging for video due to its various temporal structure and complex semantics . As a result , the existing video datasets size is still smaller than ImageNet in terms of training samples and classes . On the other hand , videos typically contain richer structure with abundant side information such as motion ( Diba et al. , 2019 ; Ng et al. , 2018 ) , audio ( Arandjelovic & Zisserman , 2017 ; Korbar et al. , 2018 ) , and text ( Miech et al. , 2019 ; Sun et al. , 2019b ) . So these expected these associated modalities are expected to provide useful cues to learn video representations in a more efficient way . Language or text is probably the most natural and easy way to describe the semantic information of a video , and the associated textual information could be easily acquired when collecting video dataset ( Rohrbach et al. , 2017 ; Miech et al. , 2019 ) from Internet or Movie . We argue that this correlation between a clip and its associated text could serve as an alternative supervision to learn video representation from scratch . This is different from some recent works ( Sun et al. , 2019b ; Miech et al. , 2019 ) , in which these abundant textual information has been used to learn a high-level visual-text embedding applied to text-to-video retrieval or video captioning . Intuitively , it is more challenging to learn a general visual representation solely from text information without any human annotation , for reasons such as large numbers of noise in text , lacking careful initialization , and being hard to design an effective objective . In this paper , we aim to learn effective video representation from noisy and diverse textual information , which could serves as the basis for a variety of downstream tasks . Basically , we learn a mapping of text and video into a shared embedding space and leverage their correlation as supervision signal . The technical difficulty is how to design an effective objective function , that is capable of modeling this complex visual-textual correlation and as well easily optimized by training from scratch on noisy datasets . Inspired by unsupervised feature learning in images ( Wu et al. , 2018 ; Tian et al. , 2019 ) , we present a cross-modal pair discrimination ( CPD ) framework , which tries to recognize each video and text pair into a class via a non-parametric classifier . To solve the computational issues imposed by the huge numbers of pair classes , we adapt noise-contrastive estimation technique ( Gutmann & Hyvärinen , 2010 ) to approximate the original loss function . Specifically , we learn the CPD framework from web videos with the associated title or caption that could be directly crawled from web platforms such as YouTube ( Kay et al. , 2017 ) and Instagram ( Duan et al. , 2020 ) . We utilize the off-the-shelf language models such as BERT ( Devlin et al. , 2019 ) or Word2vec ( Mikolov et al. , 2013 ) and devise a curriculum learning strategy to progressively train the video models . We first test the generalization ability of learned video representation by CPD on the Kinetics dataset ( Kay et al. , 2017 ) by using shallow classifiers such k-NN and linear classifier . It shows that our learned spatiotemporal features obtain promising results which are comparable to some supervised learning methods on the Kinetics dataset ( Kay et al. , 2017 ) . Then , we investigate the generalization power of learned spatiotemporal features of CPD by fine-tuning on the Kinetics ( Kay et al. , 2017 ) , UCF101 ( Soomro et al. , 2012 ) and HMDB51 ( Kuehne et al. , 2011 ) datasets , demonstrating that our method obtain superior performance to previous state-of-the-art self-supervised methods and comparable performance to the very recent methods of using orders of magnitude more videos ( 70M-100M vs. 0.3M ) . 2 RELATED WORK . Self/Weakly Supervised Representation Learning . Self supervised representation was popular in both image and video domains by designing various proxy tasks . In image domain , for instance , these tasks could be predicting the image context ( Doersch et al. , 2015 ) , counting the objects ( Noroozi et al. , 2017 ) , converting gray images to color one ( Zhang et al. , 2016 ) , keeping global and local consistency ( Hjelm et al. , 2019 ) . In video domain , typical examples include frame prediction ( Diba et al. , 2019 ; Vondrick et al. , 2016 ) , optical flow estimation ( Ng et al. , 2018 ; Zhou et al. , 2017 ; Jayaraman & Grauman , 2017 ) , instance tracking ( Wang & Gupta , 2015 ; Wang et al. , 2019b ) , temporal order or structure prediction ( Misra et al. , 2016 ; Fernando et al. , 2017 ; Wei et al. , 2018 ; Xu et al. , 2019a ) . These learnt representations may capture some aspects of low-level image or video structures , but are generally outperformed by those using cross modal information . Several cross-modal self-supervised tasks was proposed to enhance single-modality representation power and typical example is audio-visual representation learning ( Aytar et al. , 2016 ; Arandjelovic & Zisserman , 2017 ; Korbar et al. , 2018 ) . Meanwhile , some weakly-supervised methods were developed by utilizing web supervision obtained in an automatic way , such as query ID ( Chen & Gupta , 2015 ; Ghadiyaram et al. , 2019 ) , and hashtag ( Mahajan et al. , 2018 ) . Concurrent work ( Miech et al. , 2020 ) tried to learn video representations by using narration as supervision with instructional videos ( e.g. , HowTo100M ( Miech et al. , 2019 ) ) . However , they are limited by the video type . Our CPD is applicable to more general video type and we experiment with a much smaller dataset ( 0.3M vs. 100M ) of both PGC and UGC videos , but achieves a similar performance on UCF101 and HMDB51 . Concurrent work ( Stroud et al. , 2020 ) proposed a similar framework but required more training videos ( 0.3M vs. 70M ) and richer textual information to obtain similar performance to ours . Motion , Audio , and Text . Multi-modal information in videos provides natural cues for learning deep models . Motion or temporal information has been studied as to design proxy tasks to assist cross-modal learning , such as optical flow or tracking ( Ng et al. , 2018 ; Wang & Gupta , 2015 ) , frame prediction ( Diba et al. , 2019 ; Vondrick et al. , 2016 ) , or high-level temporal structure ( Wei et al. , 2018 ; Xu et al. , 2019a ; Fernando et al. , 2017 ) . As most video contain synchronized audio and visual signals , audio information has served another common modality to supervised visual learning ( Aytar et al. , 2016 ; Arandjelovic & Zisserman , 2017 ; Korbar et al. , 2018 ) . However , both motion and audio information seem to be low-level signals and may lack high-level semantic for cross-modal learning . Speech or text has been widely studied as another cross-modal setting in video learning ( Sun et al. , 2019b ; Miech et al. , 2019 ; Dong et al. , 2019 ; Miech et al. , 2018 ; Pan et al. , 2016 ; Plummer et al. , 2017 ) . These works mainly aimed to learn a joint video-text embedding where visual and textual cues are adjacent if they are semantically . However , these works focused on learn high-level visual- textual embedding by using the off-the-shelf models as feature extractors . Instead , our proposed CPD framework addresses a different issue of video representation learning from scratch . 3 CROSS-MODAL PAIR DISCRIMINATION . In this section we provide an detailed description on our proposed cross-modal pair discrimination ( CPD ) for weakly supervised spatiotemporal feature learning . First , we present the whole framework and analyze its important properties . Then , we describe the training strategy of CPD framework . Finally , we introduce text and video feature extraction networks . 3.1 FRAMEWORK AND ANALYSIS . Our goal is to propose a weakly supervised representation learning method by exploiting the correlation between each video clip and its associated text information , which could be easily obtained from a variety of sources such as YouTube titles , Instagram captions and automatic speech recognition ( ASR ) . It is generally assumed that these text information contains semantic information , but also might be noisy and irrelevant . Therefore , from technical perspective , we need to design an effective objective function and training strategy to capture this semantic correlation and as well also suppress the effect of noisy and irrelevant information . To this end , we devise a video-text pair discrimination objective and a curriculum learning strategy as follows . More formally , as shown in Figure 1 , we aim to learn a modality-specific embedding function Fv and Ft for the visual and textual information from a set of N video clips and their associated textual information { ( vi , ti ) i=1 } N . Let fvi and f ti denote Fv ( vi ) and Ft ( ti ) , respectively . These embedding functions would map these two modality into a common space ( i.e. , fvi ∈ Rd and fvi ∈ Rd ) , and related visual and text information should be close to each other . The embedding functions could be implemented by neural networks which will be clarified in next section . We first focus on how to devise objective function to optimize these embedding functions . Inspired by the work of unsupervised learning in images ( Wu et al. , 2018 ) , we design a cross-modal pair discrimination objective to learn these two embedding functions . Self-instance discrimination . In the original instance-level discrimination framework ( Wu et al. , 2018 ) , each image is treated as a distinct class and it would learn a classifier to categorize each image into its own class . This framework could be naturally extended into the setting of video and text pair by directly using feature concatenation , and we call this extension as self-instance discrimination . Formally , this video-text level instance discrimination objective could be implemented with the following softmax criterion : p ( i| ( v , t ) ) = exp ( w vT i f v +wtTi f t ) ∑N j=1 exp ( w vT j f v +wtTj f t ) , ( 1 ) where the ith video-text pair define a class i , ( wvi , w t i ) is a weight for class i , and the class number is equal to training sample numberN . This class weight represent a class prototype for each video-text instance and is probably not easy to optimize as we only have a single sample for each class . Thus , the above parametric classifier could be refined with the following non-parametric variant : p ( i| ( v , t ) ) = exp ( f vT i f v/τ + f tTi f t/τ ) ∑N j=1 exp ( f vT j f v/τ + f tTj f t/τ ) , ( 2 ) where τ is a temperature parameter to control the class concentration level and our training objective is to optimize the likelihood ∏N i=1 p ( i| ( vi , ti ) ) . This straight forward extension shares the advantage of instance-level discrimination by directly modeling in the joint video-text space . Yet , in fact , the semantic information of text modality is higher than video pixels and we aims at learning video features with the supervision of textual information . To meet this requirement , we propose a refined objective function from the perspective of conditional distribution . Cross-pair discrimination . According to the above analysis , we design the objective function by considering conditional distribution p ( it|v ) and p ( iv|t ) rather than implicitly modeling distribution p ( v , t ) . Specifically , we design the following conditional distribution : p ( it|v ) = exp ( f tTi f v/τ ) ∑N j=1 exp ( f tT j f v/τ ) , ( 3 ) where ith text define a text class it , and both f t and fv with unit-norm constraint . The conditional distribution p ( iv|t ) could be defined at the same way . We call this framework as cross-pair discrimination , and during training phase , the objective is to maximize the likelihood∏N i=1 p ( it|vi ) ∏N i=1 p ( iv|ti ) . The key difference between Equation ( 2 ) and ( 3 ) is that we propose to use cross-correlation term f tT fv to replace the self-correlation term ( fvT fv+f tT f t ) . This cross correlation is more effective to capture the mutual information between visual and textual information , and thereby better at guiding the spatiotemporal feature learning from video with text information as supervision . Ranking loss . There is some common ranking loss for cross-modal matching . To well study the effectiveness of proposed cross-modal pair discrimination objective , we also compare with a baseline of ranking loss , which is defined as follows : L ( vi , ti ) = 1 n− 1 ∑ j 6=i max ( 0 , δ + S ( f tj , fvi ) − S ( f ti , fvi ) ) , ( 4 ) where each video vi has a associated text ti and unrelated text tj from current batch . S ( f tj , fvi ) is the cosine similarity , n is the batch size and δ is a margin . We apply Equation ( 4 ) in both ways of video with its associated text and text with its video . In experiment , we empirically compare this ranking loss with our designed cross-pair discrimination objective .

Summary: The paper proposes a weakly supervised method for learning spatiotemporal features by video and text pair discrimination, namely cross-modal pair discrimination (CPD). This can be considered as an extension of (Wu et al. 2018) to video and text. On technical perspective, the original method Wu et al. is applied on images, while CPD is applied on video and text (video's title for Kinetics or hashtag search for Instagram). The most novel technical contribution of this paper is making Wu et al. 2018 cross-modal (between video and text). However, compared with Wu et al. 2018, it requires more supervision (weakly supervised vs. unsupervised). On the experiments, some comparisons are unfair and some experimental setups are biased (detail below).