Contrastive Self-Supervised Learning of Global-Local Audio-Visual Representations

1 INTRODUCTION . Self-supervised learning aims to learn representations of data that generalize to a large variety of downstream tasks . Recently , contrastive self-supervised learning ( CSL ) has achieved impressive results on several computer vision tasks ( Oord et al. , 2018 ; Hjelm et al. , 2018 ; He et al. , 2020 ; Chen et al. , 2020 ) . In CSL , the choice of “ views ” determines the types of information that the representation captures ( Bachman et al. , 2019 ) , as the framework learns representations that focus on the shared information between views . It has been demonstrated that the optimal choice of views depends critically on the downstream task ( Tian et al. , 2020 ) . Therefore , existing works mainly focus on finding different views tailored for the intended downstream tasks . For example , when tailoring views for action classification , Hjelm & Bachman ( 2020 ) extends DIM ( Hjelm et al. , 2018 ) to the spatio-temporal setting by assuming that global and local information useful for action classification ( i.e , global semantics ) should be invariant across time and space within a given video . When dealing with multimodal data , several approaches utilize audio-visual correspondence from videos ( Morgado et al. , 2020 ) . Such a CSL approach is based on an assumption that information needed for audio/video classification should be shared between the two modalities . Although they achieve impressive results in their intended downstream tasks , existing approaches often fail to generalize to tasks that they were not originally designed for . For example , in lip reading ( Chung & Zisserman , 2016 ) , the desired information is the fine-grained spatio-temporal representation around the mouth . However , if we directly apply existing CSL approaches , the shared information across views is that a there is a face , while the useful information , the lip movements , will be suppressed as they are changing across views from the sample clip . Motivated by this , we propose a versatile CSL approach to learn representations that can generalize to both scenarios that require global representations ( e.g. , classification ) and scenarios that require local representations ( e.g. , localization ) ( see Fig . 1 ) . Our approach , which we call global-local cross-modal ( GLCM ) contrastive learning , has four key properties that we assume to be important for our learning objective : 1 ) observations from the same time span of a video should reflect the same content regardless of modalities ; 2 ) the same observations captured at different time scales can reflect both global and local information ; 3 ) when learning on a local temporal scale , the contrasting views should only share the time-varying information ( e.g . only the moving lip ) while ignoring globally invariant information ; 4 ) multi-scale ( global-local ) observations can be trained jointly in a collaborative way so that representations learned at either scale can be reused . sidering only the audio-visual features that lie in the same time window as positive pairs ; the others are all negative pairs . Finally , we utilize information captured at the global scale ( e.g . localizing the source of a sound ) to assist efficient learning at the local scale , thus capturing the fourth property . We show that GLCM pretraining learns representations with global and fine-grained spatio-temporal information from audio-visual signals . The learned representations perform effectively on a variety of downstream tasks . We evaluate our proposed approach on tasks that needs local spatio-temporal information ( i.e lip reading , deep-fake detection , and sound-source localization ) and also discriminative tasks that needs global information ( i.e . action classification and audio-event classification ) . 2 RELATED WORK . Contrastive self-supervised learning . CSL has contributed to strong performance on many tasks and in cases produced comparable results to supervised learning ( Chen et al. , 2020 ; Caron et al. , 2020 ) . Contrastive learning leverage multiple views of the same data ( Hjelm & Bachman , 2020 ; Oord et al. , 2018 ) , e.g. , multiple perspectives within the same modality ( e.g. , augmentations of the same image , different frames of a video , etc . ) ( He et al. , 2020 ; Hjelm & Bachman , 2020 ; Han et al. , 2019a ) or perspectives from different modalities ( e.g. , depth and RGB images , visual and textual signals ) ( Tian et al. , 2019 ; Sun et al. , 2019 ; Miech et al. , 2020 ; Alayrac et al. , 2020 ) . Chen et al . ( 2020 ) and Hjelm et al . ( 2018 ) show that leveraging local information to perform contrastive learning further improves the performance on image classification tasks . DIM ( Hjelm et al. , 2018 ) has been extended to multi-scale ( Bachman et al. , 2019 ) and video data Hjelm & Bachman ( 2020 ) . However , evaluation is still focused on “ discriminative ” tasks ( image classification and video event classification ) , while there is little evidence that these models will adapt well to the local information . Audio-visual representation learning . Several approaches have been proposed to leverage the natural correspondence between audio and visual signals to perform CSL ( Asano et al. , 2020 ; Korbar et al. , 2018 ; Alwassel et al. , 2019 ; Morgado et al. , 2020 ; Patrick et al. , 2020 ; Chung et al. , 2019 ) . Most existing approaches aim to capture high-level semantic information from observations . It has been empirically demonstrated that such learned information is very effective for “ discrimination tasks ” ( classification ) . However , in tasks that needs local information the learned representations may not perform well . Xiao et al . ( 2020a ) design their approach by utilizing different temporal scales of the audio and visual data , which encourages the model to capture fine-grained temporal information and hence improves the performance . However , the evaluation was limited to classification tasks . In contrast with previous work , we demonstrate that our approach effectively learns global-local audio-visual representations by evaluating on a variety of downstream tasks . 3 APPROACH We propose using the audio and visual channels as cross-modal views of video data . As we aim to learn both local and global temporal information , we utilize the same visual sequence processed at different sampling rates to reflect the same observation at different temporal scales . Given that we want each signal to capture complementary views of the same data , we use different encoders to extract the representations from the audio sequence ( Ea ) , subsampled visual sequence ( Egv ) and full sampling rate visual sequence ( Elv ) . The question is , then , how to design a contrastive loss to learn representations from these different views . We achieve this goal by jointly training the model using two contrastive losses : global and local . As shown in Fig . 2 , the global loss is computed by contrasting audio signals with subsampled visual sequence ( Sec.3.1 ) ; while the local loss is computed by contrasting audio signals with visual sequence at a full sampling rate ( Sec.3.2 ) . To jointly train the global and local pathways , we propose a spatially-aware attention pooling mechanism to effectively reuse the information that was captured from the global pathway in the local pathway ( Sec.3.3 ) . 3.1 GLOBAL CONTRASTIVE OBJECTIVE . We design the global contrastive objective to capture slowly changing information with high audiovisual correlation . We use video sequences captured at low sampling rates , which will inevitably lack local temporal information . Ea encodes an audio sequence into an audio embedding za ∈ RT×F , where F is the frequency , and T is the sequence length . After temporal global pooling , it becomes za ∈ R1×F . Similarly , we perform global temporal pooling on features encoded by the global visual encoder Egv , which produces the global visual embedding z g v ∈ R1×H×W×C . Note that , for the visual features , we perform global pooling only along the temporal dimension while keeping the spatial dimension intact . The reason is that when learning in a global temporal space , the model has capacity to capture more spatial information . To compute the global contrastive loss , we consider the audio features za and the visual features zgv that come from the same video sample as positive pairs , while features coming from different video samples are negative pairs . In order to encourage the model to also capture spatial information , we adopt MIL-NCE ( Miech et al. , 2020 ) to compute the loss . Specifically , we consider all H × W spatial grids in zgv as the instances , and therefore , instead of just taking a single audio-visual positive pair za ↔ zgv , the new positive pair becomes multiple visual instances zgv [ i ] H×W , i.e . za ↔ zgv [ i ] H×W . The loss is then defined as : Lg = −log ( ∑ zgv∈P exp ( z T a z g v ) ∑ zgv∈P exp ( z T a z g v ) + ∑ z′∈N exp ( z ′ a T z′v g ) ) ( 1 ) where N is a set of negative audio and visual pairs , P is a set of spatial grids in zgv . 3.2 LOCAL CONTRASTIVE OBJECTIVE . We design the local contrastive objective to capture fine-grained spatio-temporal information that is sensitive to temporal changes while being invariant to different modalities . We thus contrast between local audio features za and local spatio-temporal visual features zlv . Specifically , we consider the temporal local audio and visual features that lie in the same time window to be the positive pairs , za [ t ] ↔ zlv [ t ] , where z [ t ] represents features in the time window t. As shown in Figure 2 , the video and audio features shaded in the same color refer to those in the same time window . The features in different time windows ( e.g . green and orange blocks ) are considered as negative pairs even if they are from the same video sample . As such , the shared information between the modalities is principally how the features vary over time . We obtain the local audio features by using the same audio encoder Ea but without global temporal pooling . The local visual features zlocalv are obtained by feeding the visual sequence with a high sampling rate into the local visual encoder Elv , which produces the visual features zlv ∈ RT×H×W×C . We then perform spatial pooling while keeping the temporal scale the same , the visual features become zlv ∈ RT×1×1×C . As the audio channel in a video generally has a higher sampling rate than the visual channel , visual feature at a single time step will be mapped to multiple audio feature slices . As shown in Figure 2 , at time t1 , the visual features zlv [ t1 ] ( green block ) correspond to multiple audio features za [ t1 ] t∈M ( green blocks ) , where M = 5 in Figure 2 . Specifically , we use a sliding window of size M to map each set of visual features at a given time step to a window of audio feature slices . Then the positive pair is considered as a visual feature and the corresponding window of audio feature slices . Once again we use MIL-NCE ( Miech et al. , 2020 ) to compute the contrastive loss . The reasoning for applying MIL-NCE here is different than in the case of the global contrastive loss . In the global contrastive loss , we aim to let the network capture spatial information . While in the local contrastive objective , the goal of using MIL-NCE is to mitigate the missing strict temporal mapping problem . The loss is therefore defined as : Ll = −log ( ∑ za∈Q exp ( z T a z l v ) ∑ za∈Q exp ( z T a z l v ) + ∑ z′∈N exp ( z ′ a T z′v l ) ) ( 2 ) where Q is a set of audio feature slices in the same time window as zlv , and N is a set of negative audio and visual pairs .

Summary: This paper presents a new contrastive audio-visual learning method. Like previous work, they use self-supervision to learn a video feature set by training a network to associate audio and visual "views" taken from the same video. Their main contribution is to jointly learn from both "local" and "global" information. They simultaneously optimize two contrastive objectives. First, there is a global objective, which computes a feature set using a low framerate video, pools over time, and obtains negatives from other different videos. Second, there is a local contrastive objective that uses a higher framerate video, pools over space but not time, and gets negatives from other timesteps of the video. They optimize both losses jointly using a spatially-aware pooling method that provides information from the global pathway to the local pathway. They compute attention by taking dot products between the visual and audio features, and using this attention to pool local visual features (instead of a global pooling).