Contrastive Learning of Global and Local Video Representations

1 Introduction . Recent years have seen a surge of interest in contrastive self-supervised learning ( CSL ) [ 59 , 38 , 35 , 16 ] to obtain representations that generalize to various downstream scenarios . In CSL , the choice of “ contrasting views ” plays a crucial role because the learned representations capture information shared between different views by maximizing mutual information between them [ 8 ] . This makes it critical to design contrastive objectives with the “ right ” contrasting views tailored for the intended downstream scenarios [ 77 ] , which has been the focus of many recent works [ 38 , 10 , 45 , 74 , 84 , 83 ] . The progress made so far provides important insights for understanding how to select optimal contrasting views for a given task [ 77 ] . However , the current paradigm of designing CSL approaches specific to any intended ( global or local ) downstream scenarios could be suboptimal , as in the real-world case the downstream scenarios are generally unknown in advance . This not only limits the generalizability of the learned representations , evaluating the approaches solely on the intended scenarios could produce misleading conclusions . Although existing approaches achieve impressive results in their intended downstream tasks , they often fail to generalize to tasks that they were not originally designed for , e.g. , as we show later in our experiments , global representations do not generalize well to tasks such as lip reading [ 22 , 21 ] which require local spatio-temporal information . Motivated by this , we take an orthogonal direction to the current CSL approaches : We aim to learn representations agnostic to the types of downstream scenarios and generalize to both the scenarios that require global representations ( e.g. , classification ) and scenarios that require local representations ( e.g. , localization ) . We focus on learning video representations using the natural 1https : //github.com/yunyikristy/global_local 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . audio-visual correspondence as the primary self-supervisory signal . In this scenario , the notion of global/local representations is intertwined in space and time ; we can obtain representations that are spatially global or local , and also temporally global or local . However , most existing video CSL approaches optimize for only global spatio-temporal representations and demonstrate them on audio/visual video classification tasks [ 60 , 46 , 55 , 61 ] . Part of the difficulty here is that formulating a contrastive objective for local representations is not straightforward because of the one-to-many relationship in audio-visual correspondences , i.e. , spatially , multiple pixel regions can contribute to the sound in the corresponding audio , and temporally , multiple temporal slices of audio can map to a single video frame due to sampling rate differences . This hinders the development of CSL for local video representations useful for tasks such as sound source separation and lipreading . In this paper , we present an approach for learning global-local video representations in the CSL framework . We design two cross-modal contrastive objectives that collaboratively capture information shared between audio and visual signals . An important aspect of our approach is the factorization of the spatio-temporal feature space into a spatially-local/temporally-global subspace and a spatially-global/temporally-local subspace , where each of the two contrastive objectives are defined in , respectively ; see Fig . 1 . The explicit space-time factorization helps each contrastive objective focus on capturing either spatially-local or temporally-local information and thus facilitates learning complementary features from audio-visual correspondence more effectively than in the original spatio-temporal space . Furthermore , we define both objectives in the multiple instance learning framework [ 24 , 52 ] to handle the one-to-many relationship between audio and visual signals . This helps the model learn representations without knowing fine-grained audio-visual correspondence . We evaluate our approach on various downstream tasks that need local spatio-temporal information , i.e. , lip reading [ 22 , 21 , 3 ] , deep-fake detection [ 25 ] and audio-visual event localization [ 76 ] , and also discriminative tasks that needs global information , i.e. , audio/visual video classification [ 71 , 47 , 63 , 41 ] . We show that the same pretrained model successfully generalizes to all our scenarios without having to re-pretrain it using different objectives and/or datasets . Furthermore , we demonstrate that the two contrastive objectives mutually benefits each other and helps improve the generalizability of both global and local representations . To the best of our knowledge , our work is the first to demonstrate a CSL approach that learns video representations that generalize to both global and local video understanding scenarios . 2 Related Work . Contrastive self-supervised learning . Contrastive learning leverages multiple views of the same data [ 59 ] , e.g. , multiple perspectives within the same modality such as augmentations of the same image , different frames/clips of a video , etc . [ 35 , 37 , 32 ] or perspectives from different modalities such as RGB and depth , images/videos and text [ 75 , 73 , 53 , 4 ] . DIM [ 38 ] and SimCLR [ 16 ] show that leveraging local information in contrastive learning further improves performance on image classification . DIM [ 38 ] has been extended to multi-scale [ 11 ] and to video data [ 37 ] . However , evaluation is still focused on “ discriminative ” tasks , e.g. , image classification and video classification , while there is little evidence that these models will adapt well to tasks that require local information . Several recent advances happened in the image domain , e.g. , MoCo [ 35 , 17 , 19 ] , BYOL [ 31 ] , SwAV [ 13 ] , SimSiam [ 18 ] , BarlowTwins [ 90 ] . Although evaluation is performed in both global and local downstream scenarios such as image classification , object detection , semantic/instance segmentation and depth estimation , this line of work focuses on evaluating generalizability of the learned global representations rather than studying the importance of global and local information in generalization , which is the focus of this work . They also focus on image recognition tasks only . Audio-visual video representation learning . Learning video representations from the natural audiovisual correspondence has been studied extensively . Most existing approaches aim to capture highlevel semantic information useful for sequence-level ( global ) discrimination tasks such as audio/visual video classification [ 9 , 46 , 5 , 55 , 61 , 23 ] . Along this line of work , AVSlowFast [ 81 ] utilizes different temporal scales of the audio and visual data , which encourages the model to capture fine-grained temporal information . However , their learning objective optimizes for only global representations induced by different sampling rates and their evaluation is still focused on classification tasks . Another line of work focuses on capturing fine-grained spatio-temporal local information suitable for “ local ” task such as sound source separation and localization [ 7 , 69 , 93 , 92 , 28 , 88 , 64 , 50 ] . Lin et al . [ 50 ] learn patch-level audio-visual correspondence by drawing positive/negative patches iteratively from sounding/non-sounding regions induced by audio-visual feature correlation . However , their approach learns spatially-local/temporally-global representations only , and their evaluation is focused on localization tasks . In contrast to these previous work , we explicitly optimize for global and local spatio-temporal representations and evaluate on both classification and localization downstream tasks . There are also some work proposed to learn audio-visual representations , however they are specifically designed for a particular task , e.g . speaker recognition [ 56 ] . While our work learns general-purpose video representation and is demonstrated on a variety of downstream tasks . 3 Approach . We factorize the feature space into a spatially-local/temporally-global ( S-local/T-global ) subspace and a spatially-global/temporally-local ( S-global/T-local ) subspace and define two cross-modal contrastive objectives in each of the subspaces . The S-local/T-global objective captures slowly changing patch-level information ( Sec.3.1 ) while the S-global/T-local objective captures fast changing frame-level information ( Sec.3.2 ) . Furthermore , we utilize the learned patch-level information to guide learning frame-level information via a spatially-aware attention pooling mechanism ( Sec.3.3 ) . 3.1 Spatially-Local Temporally-Global Contrastive Objective . The purpose of this objective is to capture slowly changing patch-level information with high audiovisual correlation . We define an audio encoder Ega that computes audio embeddings z g a ∈ RTa×F , where Ta is the audio sequence length and F is the number of frequency bands , and perform temporal average pooling to obtain zga ∈ R1×F . Similarly , we define a visual encoder Egv that computes visual embeddings zgv ∈ RTv×H×W×C , where Tv is the visual sequence length and H , W , C are the height , width and channel dimensions , and perform temporal average pooling to obtain zgv ∈ R1×H×W×C . To help our model capture slowly changing yet spatially-detailed information , we purposely make input frames lack local temporal information by feeding them at a low sampling rate , e.g. , at one eighths of the original sampling rate , which has been shown to be effective at capturing fine-grained spatial details [ 27 ] . For audio , we found that keeping the original sampling rate to be more effective . We consider zga and z g v that come from the same clip as positive pairs , while features coming from different clips are negatives . To capture local spatial information in zgv , we consider each cell of the H ×W spatial grid as an instance and define pairs by coupling zga with zgv [ i ] , ∀i ∈ H ×W . Note that not all zgv [ i ] will have valid audio-visual correspondence to z g a ; it is rather likely that only a few cells ( e.g. , the lip region of a talking person ) will match the information in zga . To resolve this misalignment issue , we define our contrastive loss in the multiple instance learning framework [ 53 ] : Lg = − log ( ∑ zgv [ i ] ∈P F ( z g a , z g v [ i ] ) ∑ zgv [ i ] ∈P F ( z g a , z g v [ i ] ) + ∑ z′v∈N F ( zga , z′ g v ) ) ( 1 ) where F ( za , zv ) = exp ( zTa zv ) measures the compatibility between za and zv , P is a set of spatial grids in zgv , andN is a set of negative visual instances taken from different clips , i.e. , given a batch of B video clips , we consider H ×W × ( B − 1 ) 2 negative pairs . 3.2 Spatially-Global Temporally-Local Contrastive Objective . To capture information sensitive to temporal changes , e.g. , the lip region of a talking head , we take a temporally-granular approach and contrast visual features at each time step to the corresponding audio features in a local temporal neighborhood . We compute audio embeddings zla ∈ RTa×F and visual embeddings zlv ∈ RTv×H×W×C in the same manner as before but without applying temporal average pooling at the end . To help our model capture fine-grained temporal changes , unlike in the previous objective , we feed frames at a higher sampling rate and perform spatial pooling over visual embeddings to obtain zlv ∈ RTv×1×1×C . We construct contrasting views by taking audio and visual features from the same local temporal block as positive pairs ; audio-visual pairs from different temporal blocks of the same video become negatives . We do not use samples from different videos in order to encourage the shared information between modalities to be just how features vary over time . Note that a single visual feature zlv [ t ] , t ∈ Tv maps to multiple audio features zla [ t ] , t ∈ Ta because audio signals are typically captured at a higher sampling rate . Therefore , unlike in the previous objective , all the pairs within the same local temporal block can be considered as valid positives . There exists several ways to define a contrastive objective with multiple positives [ 42 , 12 , 70 ] ; we opt for a simple approach that sums scores over all positive pairs , which leads to a form similar to Eqn . 1 : Ll = − log ( ∑ zla [ tk ] ∈P F ( zla [ tk ] , z l v [ t ] ) ∑ zla [ tk ] ∈P F ( zla [ tk ] , z l v [ t ] ) + ∑ z′a∈N F ( z′a , z l v ) ) ( 2 ) P is a set of audio features within the same temporal block of zlv [ t ] and N is a set of all possible audio-visual slice pairs that come from different parts of the same video . Given a video with Tv frames and Ta = M × Tv audio slices , we consider M × ( Tv − 1 ) 2 negative pairs for a video sequence .

Summary: The paper introduces a novel method for self-supervised video from audio and video. The technique uses both the temporal and spatial dimension of the audio-visual input in order to build two contrastive losses: one capturing slow changing elements in the video present in the audio and the other one capturing temporally changing elements. After learning through these two losses, authors propose a detailed downstream evaluation consisting of task where spatio-temporal granularity is relevant (deepfake detection, lip reading) and other task about global information in the video (action classification).