Contrastive Learning of Global and Local Video Representations

1 Introduction . Recent years have seen a surge of interest in contrastive self-supervised learning ( CSL ) [ 59 , 38 , 35 , 16 ] to obtain representations that generalize to various downstream scenarios . In CSL , the choice of “ contrasting views ” plays a crucial role because the learned representations capture information shared between different views by maximizing mutual information between them [ 8 ] . This makes it critical to design contrastive objectives with the “ right ” contrasting views tailored for the intended downstream scenarios [ 77 ] , which has been the focus of many recent works [ 38 , 10 , 45 , 74 , 84 , 83 ] . The progress made so far provides important insights for understanding how to select optimal contrasting views for a given task [ 77 ] . However , the current paradigm of designing CSL approaches specific to any intended ( global or local ) downstream scenarios could be suboptimal , as in the real-world case the downstream scenarios are generally unknown in advance . This not only limits the generalizability of the learned representations , evaluating the approaches solely on the intended scenarios could produce misleading conclusions . Although existing approaches achieve impressive results in their intended downstream tasks , they often fail to generalize to tasks that they were not originally designed for , e.g. , as we show later in our experiments , global representations do not generalize well to tasks such as lip reading [ 22 , 21 ] which require local spatio-temporal information . Motivated by this , we take an orthogonal direction to the current CSL approaches : We aim to learn representations agnostic to the types of downstream scenarios and generalize to both the scenarios that require global representations ( e.g. , classification ) and scenarios that require local representations ( e.g. , localization ) . We focus on learning video representations using the natural 1https : //github.com/yunyikristy/global_local 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . audio-visual correspondence as the primary self-supervisory signal . In this scenario , the notion of global/local representations is intertwined in space and time ; we can obtain representations that are spatially global or local , and also temporally global or local . However , most existing video CSL approaches optimize for only global spatio-temporal representations and demonstrate them on audio/visual video classification tasks [ 60 , 46 , 55 , 61 ] . Part of the difficulty here is that formulating a contrastive objective for local representations is not straightforward because of the one-to-many relationship in audio-visual correspondences , i.e. , spatially , multiple pixel regions can contribute to the sound in the corresponding audio , and temporally , multiple temporal slices of audio can map to a single video frame due to sampling rate differences . This hinders the development of CSL for local video representations useful for tasks such as sound source separation and lipreading . In this paper , we present an approach for learning global-local video representations in the CSL framework . We design two cross-modal contrastive objectives that collaboratively capture information shared between audio and visual signals . An important aspect of our approach is the factorization of the spatio-temporal feature space into a spatially-local/temporally-global subspace and a spatially-global/temporally-local subspace , where each of the two contrastive objectives are defined in , respectively ; see Fig . 1 . The explicit space-time factorization helps each contrastive objective focus on capturing either spatially-local or temporally-local information and thus facilitates learning complementary features from audio-visual correspondence more effectively than in the original spatio-temporal space . Furthermore , we define both objectives in the multiple instance learning framework [ 24 , 52 ] to handle the one-to-many relationship between audio and visual signals . This helps the model learn representations without knowing fine-grained audio-visual correspondence . We evaluate our approach on various downstream tasks that need local spatio-temporal information , i.e. , lip reading [ 22 , 21 , 3 ] , deep-fake detection [ 25 ] and audio-visual event localization [ 76 ] , and also discriminative tasks that needs global information , i.e. , audio/visual video classification [ 71 , 47 , 63 , 41 ] . We show that the same pretrained model successfully generalizes to all our scenarios without having to re-pretrain it using different objectives and/or datasets . Furthermore , we demonstrate that the two contrastive objectives mutually benefits each other and helps improve the generalizability of both global and local representations . To the best of our knowledge , our work is the first to demonstrate a CSL approach that learns video representations that generalize to both global and local video understanding scenarios . 2 Related Work . Contrastive self-supervised learning . Contrastive learning leverages multiple views of the same data [ 59 ] , e.g. , multiple perspectives within the same modality such as augmentations of the same image , different frames/clips of a video , etc . [ 35 , 37 , 32 ] or perspectives from different modalities such as RGB and depth , images/videos and text [ 75 , 73 , 53 , 4 ] . DIM [ 38 ] and SimCLR [ 16 ] show that leveraging local information in contrastive learning further improves performance on image classification . DIM [ 38 ] has been extended to multi-scale [ 11 ] and to video data [ 37 ] . However , evaluation is still focused on “ discriminative ” tasks , e.g. , image classification and video classification , while there is little evidence that these models will adapt well to tasks that require local information . Several recent advances happened in the image domain , e.g. , MoCo [ 35 , 17 , 19 ] , BYOL [ 31 ] , SwAV [ 13 ] , SimSiam [ 18 ] , BarlowTwins [ 90 ] . Although evaluation is performed in both global and local downstream scenarios such as image classification , object detection , semantic/instance segmentation and depth estimation , this line of work focuses on evaluating generalizability of the learned global representations rather than studying the importance of global and local information in generalization , which is the focus of this work . They also focus on image recognition tasks only . Audio-visual video representation learning . Learning video representations from the natural audiovisual correspondence has been studied extensively . Most existing approaches aim to capture highlevel semantic information useful for sequence-level ( global ) discrimination tasks such as audio/visual video classification [ 9 , 46 , 5 , 55 , 61 , 23 ] . Along this line of work , AVSlowFast [ 81 ] utilizes different temporal scales of the audio and visual data , which encourages the model to capture fine-grained temporal information . However , their learning objective optimizes for only global representations induced by different sampling rates and their evaluation is still focused on classification tasks . Another line of work focuses on capturing fine-grained spatio-temporal local information suitable for “ local ” task such as sound source separation and localization [ 7 , 69 , 93 , 92 , 28 , 88 , 64 , 50 ] . Lin et al . [ 50 ] learn patch-level audio-visual correspondence by drawing positive/negative patches iteratively from sounding/non-sounding regions induced by audio-visual feature correlation . However , their approach learns spatially-local/temporally-global representations only , and their evaluation is focused on localization tasks . In contrast to these previous work , we explicitly optimize for global and local spatio-temporal representations and evaluate on both classification and localization downstream tasks . There are also some work proposed to learn audio-visual representations , however they are specifically designed for a particular task , e.g . speaker recognition [ 56 ] . While our work learns general-purpose video representation and is demonstrated on a variety of downstream tasks . 3 Approach . We factorize the feature space into a spatially-local/temporally-global ( S-local/T-global ) subspace and a spatially-global/temporally-local ( S-global/T-local ) subspace and define two cross-modal contrastive objectives in each of the subspaces . The S-local/T-global objective captures slowly changing patch-level information ( Sec.3.1 ) while the S-global/T-local objective captures fast changing frame-level information ( Sec.3.2 ) . Furthermore , we utilize the learned patch-level information to guide learning frame-level information via a spatially-aware attention pooling mechanism ( Sec.3.3 ) . 3.1 Spatially-Local Temporally-Global Contrastive Objective . The purpose of this objective is to capture slowly changing patch-level information with high audiovisual correlation . We define an audio encoder Ega that computes audio embeddings z g a ∈ RTa×F , where Ta is the audio sequence length and F is the number of frequency bands , and perform temporal average pooling to obtain zga ∈ R1×F . Similarly , we define a visual encoder Egv that computes visual embeddings zgv ∈ RTv×H×W×C , where Tv is the visual sequence length and H , W , C are the height , width and channel dimensions , and perform temporal average pooling to obtain zgv ∈ R1×H×W×C . To help our model capture slowly changing yet spatially-detailed information , we purposely make input frames lack local temporal information by feeding them at a low sampling rate , e.g. , at one eighths of the original sampling rate , which has been shown to be effective at capturing fine-grained spatial details [ 27 ] . For audio , we found that keeping the original sampling rate to be more effective . We consider zga and z g v that come from the same clip as positive pairs , while features coming from different clips are negatives . To capture local spatial information in zgv , we consider each cell of the H ×W spatial grid as an instance and define pairs by coupling zga with zgv [ i ] , ∀i ∈ H ×W . Note that not all zgv [ i ] will have valid audio-visual correspondence to z g a ; it is rather likely that only a few cells ( e.g. , the lip region of a talking person ) will match the information in zga . To resolve this misalignment issue , we define our contrastive loss in the multiple instance learning framework [ 53 ] : Lg = − log ( ∑ zgv [ i ] ∈P F ( z g a , z g v [ i ] ) ∑ zgv [ i ] ∈P F ( z g a , z g v [ i ] ) + ∑ z′v∈N F ( zga , z′ g v ) ) ( 1 ) where F ( za , zv ) = exp ( zTa zv ) measures the compatibility between za and zv , P is a set of spatial grids in zgv , andN is a set of negative visual instances taken from different clips , i.e. , given a batch of B video clips , we consider H ×W × ( B − 1 ) 2 negative pairs . 3.2 Spatially-Global Temporally-Local Contrastive Objective . To capture information sensitive to temporal changes , e.g. , the lip region of a talking head , we take a temporally-granular approach and contrast visual features at each time step to the corresponding audio features in a local temporal neighborhood . We compute audio embeddings zla ∈ RTa×F and visual embeddings zlv ∈ RTv×H×W×C in the same manner as before but without applying temporal average pooling at the end . To help our model capture fine-grained temporal changes , unlike in the previous objective , we feed frames at a higher sampling rate and perform spatial pooling over visual embeddings to obtain zlv ∈ RTv×1×1×C . We construct contrasting views by taking audio and visual features from the same local temporal block as positive pairs ; audio-visual pairs from different temporal blocks of the same video become negatives . We do not use samples from different videos in order to encourage the shared information between modalities to be just how features vary over time . Note that a single visual feature zlv [ t ] , t ∈ Tv maps to multiple audio features zla [ t ] , t ∈ Ta because audio signals are typically captured at a higher sampling rate . Therefore , unlike in the previous objective , all the pairs within the same local temporal block can be considered as valid positives . There exists several ways to define a contrastive objective with multiple positives [ 42 , 12 , 70 ] ; we opt for a simple approach that sums scores over all positive pairs , which leads to a form similar to Eqn . 1 : Ll = − log ( ∑ zla [ tk ] ∈P F ( zla [ tk ] , z l v [ t ] ) ∑ zla [ tk ] ∈P F ( zla [ tk ] , z l v [ t ] ) + ∑ z′a∈N F ( z′a , z l v ) ) ( 2 ) P is a set of audio features within the same temporal block of zlv [ t ] and N is a set of all possible audio-visual slice pairs that come from different parts of the same video . Given a video with Tv frames and Ta = M × Tv audio slices , we consider M × ( Tv − 1 ) 2 negative pairs for a video sequence .

Summary: This paper studies self-supervised learning of audio-visual representations from video data using neural networks and a contrastive audio-visual correspondence objective.  The key novelty of this paper is that it asymmetrically applies pooling/attention to simultaneously learn two different representations: spatially-local/temporally-global, and spatially-global/temporally-local. Prior work has focused on learning only one of these two, or simply learned spatially-global/temporally-global representations.   The model is pre-trained on both Kinetics-400 as well as a mixed dataset, using a 120k video subset of Kinetics-400 with a 120k video subset of the AVSpeech dataset. A single pre-trained model can then be fine-tuned on multiple downstream tasks, including action recognition on UCF101 and HMDB51, sound classification on ESC50, lip reading on LRW and LRS2, and deepfake detection on DFDC. Qualitative analysis of sounding object localization is also performed on Kinetics-Sounds. The quantitative results are strong, in most cases outperforming previous self-supervised approaches trained under similar conditions.