Multi-modal Self-Supervision from Generalized Data Transformations

1 INTRODUCTION . Recent works such as PIRL ( Misra & van der Maaten , 2020 ) , MoCo ( He et al. , 2019 ) and SimCLR ( Tian et al. , 2019 ) have shown that it is possible to pre-train state-of-the-art image representations without the use of any manually-provided labels . Furthermore , many of these approaches use variants of noise contrastive learning ( Gutmann & Hyvärinen , 2010 ) . Their idea is to learn a representation that is invariant to transformations that leave the meaning of an image unchanged ( e.g . geometric distortion or cropping ) and distinctive to changes that are likely to alter its meaning ( e.g . replacing an image with another chosen at random ) . An analysis of such works shows that a dominant factor for performance is the choice of the transformations applied to the data . So far , authors have explored ad-hoc combinations of several transformations ( e.g . random scale changes , crops , or contrast changes ) . Videos further allow to leverage the time dimension and multiple modalities . For example , Arandjelovic & Zisserman ( 2017 ) ; Owens et al . ( 2016 ) learn representations by matching visual and audio streams , as a proxy for objects that have a coherent appearance and sound . Their formulation is similar to noise contrastive ones , but does not quite follow the pattern of expressing the loss in terms of data transformations . Others ( Chung & Zisserman , 2016 ; Korbar et al. , 2018 ; Owens & Efros , 2018 ) depart further from standard contrastive schemes by learning representations that can tell whether visual and audio streams are in sync or not ; the difference here is that the representation is encouraged to be distinctive rather than invariant to a time shift . Overall , it seems that finding an optimal noise contrastive formulation for videos will require combining several transformations while accounting for time and multiple modalities , and understanding how invariance and distinctiveness should relate to the transformations . However , the ad-hoc nature of these choices in previous contributions make a systematic exploration of this space rather difficult . In this paper , we propose a solution to this problem by introducing the Generalized Data Transformations ( GDT ; fig . 1 ) framework . GDTs reduce most previous methods , contrastive or not , to a noise contrastive formulation that is expressed in terms of data transformations only , making it simpler to systematically explore the space of possible combinations . This is true in particular for multi-modal data , where separating different modalities can also be seen as a transformation of an input video . The formalism also shows which combinations of different transformations are valid and how to enumerate them . It also clarifies how invariance and distinctiveness to different effects can be incorporated in the formulation and when doing so leads to a valid learning objective . These two aspects allows the search space of potentially optimal transformations to be significantly constrained , making it amenable to grid-search or more sophisticated methods such as Bayesian optimisation . By using GDTs , we make several findings . First , we find that using our framework , most previous pretext representation learning tasks can be formulated in a noise-contrastive manner , unifying previously distinct domains . Second , we show that just learning representations that are invariant to more and more transformations is not optimal , at least when it comes to video data ; instead , balancing invariance to certain factors with distinctiveness to others performs best . Third , we find that by investigating what to be variant to can lead to large gains in downstream performances , for both visual and audio tasks . With this , we are able to set the new state of the art in audio-visual representation learning , with both small and large video pretraining datasets on a variety of visual and audio downstream tasks . In particular , we achieve 95.2 % and 72.8 % on the standardized UCF-101 and HMDB-51 action recognition benchmarks . 2 RELATED WORK . Self-supervised learning from images and videos . A variety of pretext tasks have been proposed to learn representations from unlabelled images . Some tasks leverage the spatial context in images ( Doersch et al. , 2015 ; Noroozi & Favaro , 2016 ) to train CNNs , while others create pseudo classification labels via artificial rotations ( Gidaris et al. , 2018 ) , or clustering features ( Asano et al. , 2020b ; Caron et al. , 2018 ; 2019 ; Gidaris et al. , 2020 ; Ji et al. , 2018 ) . Colorization ( Zhang et al. , 2016 ; 2017 ) , inpainting ( Pathak et al. , 2016 ) , solving jigsaw puzzles ( Noroozi et al. , 2017 ) , as well as the contrastive methods detailed below , have been proposed for self-supervised image representation learning . Some of the tasks that use the space dimension of images have been extended to the space-time dimensions of videos by crafting equivalent tasks . These include jigsaw puzzles ( Kim et al. , 2019 ) , and predicting rotations ( Jing & Tian , 2018 ) or future frames ( Han et al. , 2019 ) . Other tasks leverage the temporal dimension of videos to learn representations by predicting shuffled frames ( Misra et al. , 2016 ) , the direction of time ( Wei et al. , 2018 ) , motion ( Wang et al. , 2019 ) , clip and sequence order ( Lee et al. , 2017 ; Xu et al. , 2019 ) , and playback speed ( Benaim et al. , 2020 ; Cho et al. , 2020 ; Fernando et al. , 2017 ) . These pretext-tasks can be framed as GDTs . Multi-modal learning . Videos , unlike images , are a rich source of a variety of modalities such as speech , audio , and optical flow , and their correlation can be used as a supervisory signal . This idea has been present as early as 1993 ( de Sa , 1994 ) . Only recently , however , has multi-modal learning been used to successfully learn effective representations by leveraging the natural correspondence ( Alwassel et al. , 2020 ; Arandjelovic & Zisserman , 2017 ; Asano et al. , 2020a ; Aytar et al. , 2016 ; Morgado et al. , 2020 ; Owens et al. , 2016 ) and synchronization ( Chung & Zisserman , 2016 ; Korbar et al. , 2018 ; Owens & Efros , 2018 ) between the audio and visual streams . A number of recent papers have leveraged speech as a weak supervisory signal to train video representations ( Li & Wang , 2020 ; Miech et al. , 2020 ; Nagrani et al. , 2020 ; Sun et al. , 2019a ; b ) and recently Alayrac et al . ( 2020 ) , which uses speech , audio and video . Other works incorporate optical flow and other modalities ( Han et al. , 2020 ; Liu et al. , 2019 ; Piergiovanni et al. , 2020 ; Zhao et al. , 2019 ) to learn representations . In ( Tian et al. , 2019 ) , representations are learned with different views such as different color channels or modalities ) to induce invariances . In contrast , our work analyses multi-modal transformations and examines their utility when used as an invariant or variant learning signal . Noise Contrastive Loss . Noise contrastive losses ( Gutmann & Hyvärinen , 2010 ; Hadsell et al. , 2006 ) measure the similarity between sample pairs in a representational space and are at the core of several recent works on unsupervised feature learning . It has been shown to yield good performance for learning image ( Chen et al. , 2020b ; He et al. , 2019 ; Hénaff et al. , 2019 ; Hjelm et al. , 2019 ; Li et al. , 2020 ; Misra & van der Maaten , 2020 ; Oord et al. , 2018 ; Tian et al. , 2019 ; 2020 ; Wu et al. , 2018 ) and video ( Han et al. , 2019 ; Li & Wang , 2020 ; Miech et al. , 2020 ; Morgado et al. , 2020 ; Sohn , 2016 ; Sun et al. , 2019a ) representations , and circumvents the need to explicitly specify what information needs to be discarded via a designed task . We leverage the noise contrastive loss as a learning framework to encourage the network to learn desired invariance and distinctiveness to data transformations . The GDT framework can be used to combine and extend many of these cues , contrastive or not , in a single noise contrastive formulation . 3 METHOD . A data representation is a function f : X → RD mapping data points x to vectors f ( x ) . Representations are useful because they help to solve tasks such as image classification . Based on the nature of the data and the task , we often know a priori some of the invariances that the representation should possess ( for example , rotating an image usually does not change its class ) . We can capture those by means of the contrast function1 c ( x1 , x2 ) = δf ( x1 ) =f ( x2 ) , where c ( x1 , x2 ) = 1 means that f is invariant to substituting x2 for x1 , while c ( x1 , x2 ) = 0 means that f is distinctive to this change . Any partial knowledge of the contrast c can be used as a cue to learn f , but c is not arbitrary : in order for c to be valid , the expression c ( x1 , x2 ) = 1 must be an equivalence relation on X , i.e . be reflexive c ( x , x ) = 1 , symmetric c ( x1 , x2 ) = c ( x2 , x1 ) and transitive c ( x1 , x2 ) = c ( x2 , x3 ) = 1⇒ c ( x1 , x3 ) = 1 . This is justified in Appendix A.1 and will be important in establishing which particular learning formulations are valid and which are not . We introduce next our Generalized Data Transformations ( GDTs ) framework by generalizing two typical formulations : the first is analogous to ‘ standard ’ methods such as MoCo ( He et al. , 2019 ) and SimCLR ( Chen et al. , 2020b ) and the second tackles multi-modal data . Standard contrastive formulation . Recall that the goal is to learn a function f that is compatible with a known contrast c , in the sense explained above . In order to learn f , we require positive ( c ( x1 , x2 ) = 1 ) and negative ( c ( x1 , x2 ) = 0 ) example pairs ( x1 , x2 ) . We generate positive pairs by sampling x1 from a data source and then by setting x2 = g ( x1 ) as a random transformation of the first sample , where g ∈ G is called a data augmentation ( e.g . image rotation ) . We also generate negative pairs by sampling x1 and x2 independently . It is convenient to express these concepts via transformations only . To this end , let D = ( x1 , . . . , xN ) ∈ XN be a collection of N i.i.d . training data samples . A Generalized Data Transformation ( GDT ) T : XN → Z is a mapping that acts on the set of training samplesD to produce a new sample z = TD . Note that the GDT is applied to the entire training set , so that sampling itself can be seen as a transformation . In the simplest case , Z = X and a GDT T = ( i , g ) extracts the sample corresponding to a certain index i and applies an augmentation g : X → X to it , i.e . TD = g ( xi ) . 1We use the symbol δ to denote the Kronecker delta . Usually , we want the function f to be distinctive to the choice of sample but invariant to its augmentation . This is captured by setting the contrast c ( T , T ′ ) 2 to c ( ( i , g ) , ( i′ , g′ ) ) = δi=i′ . Given a batch T = { T1 , . . . , TK } of K GDTs , we then optimize a pairwise-weighted version of the noisecontrastive loss ( Chen et al. , 2020b ; Gutmann & Hyvärinen , 2010 ; Oord et al. , 2018 ; Tian et al. , 2019 ; Wu et al. , 2018 ) , the GDT-NCE loss : L ( f ; T ) = − ∑ T , T ′∈T c ( T , T ′ ) w ( T , T ′ ) log ( exp 〈f ( TD ) , f ( T ′D ) 〉/ρ∑ T ′′∈T w ( T , T ′′ ) exp 〈f ( TD ) , f ( T ′′D ) 〉/ρ ) . ( 1 ) Here , the scalar ρ is a temperature parameter and the weights w ( T , T ′ ) are set to δT 6=T ′ in order to discount contrasting identical transformations , which would result in a weak learning signal . Minimizing eq . ( 1 ) pulls together vectors f ( TD ) and f ( T ′D ) if c ( T , T ′ ) = 1 and pushes them apart if c ( T , T ′ ) = 0 , similar to a margin loss , but with a better handling of hard negatives ( Chen et al. , 2020b ; Khosla et al. , 2020 ; Tian et al. , 2019 ) .3 When using a single modality , T = T ′ and positive pairs are computed from two differently augmented versions . Multi-modal contrastive formulation . We now further extend GDTs to handle multi-modal data . In this case , several papers ( Arandjelovic & Zisserman , 2017 ; Aytar et al. , 2016 ; Korbar et al. , 2018 ; Owens et al. , 2016 ; Wei et al. , 2018 ) have suggested to learn from the correlation between modalities , albeit usually not in a noise-contrastive manner . In order to encode this with a GDT , we introduce modality projection transformationsm ∈M . For example , a video x = ( v , a ) has a visual component v and an audio component a and we we have two projectionsM = { ma , mv } extracting respectively the visualmv ( x ) = v and audioma ( x ) = a signals . We can plug this directly in eq . ( 1 ) by considering GDTs T = ( i , m ) and setting TD = m ( xi ) , learning a representation f which is distinctive to the choice of input video , but invariant to the choice of modality.4 General case . Existing noise contrastive formulations learn representations that are invariant to an ad-hoc selection of transformations . We show here how to use GDTs to build systematically new valid combinations of transformations while choosing whether to encode invariance or distinctiveness to each factor . Together with the fact that all components , including data sampling and modality projection , are interpreted as transformations , this results in a powerful approach to explore a vast space of possible formulations systematically , especially for the case of video data with its several dimensions . In order to do so , note that to write the contrastive loss eq . ( 1 ) , we only require : the contrast c ( T , T ′ ) , the weight w ( T , T ′ ) and a way of sampling the transformations T in the batch . Assuming that each generalized transformation T = tM ◦ · · · ◦ t1 is a sequence of M transformations tm , we start by defining the contrast c for individual factors as : c ( tm , t ′ m ) = { 1 , if we hypothesize invariance , δtm=t′m , if we hypothesize distinctiveness . ( 2 ) The overall contrast is then c ( T , T ′ ) = ∏M m=1 c ( tm , t ′ m ) . In this way , each contrast c ( tm , t ′ m ) is an equivalence relation and so is c ( T , T ′ ) ( see Appendix A.1 ) , making it valid in the sense discussed above . We also assume that w ( T , T ′ ) = 1 unless otherwise stated . Next , we require a way of sampling transformations T in the batch . Note that each batch must contain transformations that can be meaningfully contrasted , forming a mix of invariant and distinctive pairs , so they can not be sampled independently at random . Furthermore , based on the definition above , a single ‘ distinctive ’ factor in eq . ( 2 ) such that tm 6= t′m implies that c ( T , T ′ ) = 0 . Thus , the batch must contain several transformations that have equal distinctive factors in order to generate a useful learning signal . A simple way to satisfy these constraints is to use a hierarchical sampling scheme ( fig . 1 ) First , we sample K1 instances of transformation t1 ; then , for each sample t1 , we sample K2 instances 2Note that , differently from the previous section , we have now defined c on transformations T rather than on samples x directly . In Appendix A.1 , we show that this is acceptable provided that c ( T , T ′ ) = 1 also defines an equivalence relation . 3We can think of eq . ( 1 ) as a softmax cross-entropy loss for a classification problem where the classes are the equivalence classes T /c of transformations . 4For this , as f must accept either a visual or audio signal as input , we consider a pair of representations f = ( fv , fa ) , one for each modality . of transformation t2 and so on , obtaining a batch of K = ∏M m=1Km transformations T . In this manner , the batch contains exactly KM × · · · ×Km+1 transformations that share the same first m factors ( t1 = t′1 , . . . , tm = t ′ m ) . While other schemes are possible , in Appendix A.2.1 , we show that this is sufficient to express a large variety of self-supervised learning cues that have been proposed in the literature . In the rest of the manuscript , however , we focus on audio-visual data .

Summary: The paper introduces a general framework dubbed Generalized Data Transformations (GDT) for self supervised learning. The framework is used to perform video-audio self supervised learning and analyze what kind of transformations the representations should be invariant to or on the contrary variant to thanks to a contrastive loss. The author demonstrate the effectiveness of the proposed approach by showing that the resulting learned video representations achieve very good performance on the HMDB51 and UCF101 downstream task.