Contrastive Learning of Global and Local Video Representations
Authors: Shuang Ma
Zhaoyang Zeng
Daniel McDuff
Yale Song
Venue: NeurIPS 2021 Poster (2021)
Keywords: contrastive learning
self-supervised learning
video representation learning
multimodal learning

Abstract: Contrastive learning has delivered impressive results for various tasks in the self-supervised regime. However, existing approaches optimize for learning representations specific to downstream scenarios, i.e., global representations suitable for tasks such as classification or local representations for tasks such as detection and localization. While they produce satisfactory results in the intended downstream scenarios, they often fail to generalize to tasks that they were not originally designed for. In this work, we propose to learn video representations that generalize to both the tasks which require global semantic information (e.g., classification) and the tasks that require local fine-grained spatio-temporal information (e.g., localization). We achieve this by optimizing two contrastive objectives that together encourage our model to learn global-local visual information given audio signals. We show that the two objectives mutually improve the generalizability of the learned global-local representations, significantly outperforming their disjointly learned counterparts. We demonstrate our approach on various tasks including action/sound classification, lipreading, deepfake detection, event and sound localization.

Contrastive Learning of Global-Local Video Representations
Shuang Ma Microsoft Redmond, WA, USA
Zhaoyang Zeng Sun Yat-sen University Guangzhou, China
Daniel McDuff Microsoft Research Redmond, WA, USA
Yale Song Microsoft Research Redmond, WA, USA
Abstract
Contrastive learning has delivered impressive results for various tasks in the self- supervised regime. However, existing approaches optimize for learning represen- tations specific to downstream scenarios, i.e., global representations suitable for tasks such as classification or local representations for tasks such detection and localization. While they produce satisfactory results in the intended downstream scenarios, they often fail to generalize to tasks that they were not originally designed for. this work, we propose to learn video representations that generalize to both the tasks which require global semantic information (e.g., classification) and the tasks that require local fine-grained spatio-temporal information (c.g., localization). We achieve this by optimizing two contrastive objectives that together encour- age our model to learn global-local visual information given audio signals. We show that the objectives mutually improve the generalizability of the learned global-local representations, significantly outperforming their disjointly learned counterparts. We demonstrate approach various tasks including action/sound classification, lip reading, deepfake detection, event and sound localization !
Introduction
Recent years have seen surge of interest in contrastive self-supervised learning (CSL) 59 38 35 16] to obtain representations that generalize to various downstream scenarios. In CSL, the choice of "contrasting views" plays a crucial role because the learned representations capture information shared between different views by maximizing mutual information between them (8). This makes critical design contrastive objectives with the "right" contrasting views tailored for the intended
downstream scenarios 77),, which has been the focus of many recent works [38] 10 45 74 84, 83).
The progress made so far provides important insights for understanding how to select optimal contrasting views for given task (77). However, the current paradigm of designing CSL approaches specific to any intended (global or local) downstream scenarios could be suboptimal, as in the real-world case the downstream scenarios are generally unknown in advance. This not only limits the generalizability of the learned representations, evaluating the approaches solely on the intended scenarios could produce misleading conclusions. Although existing approaches achieve impressive results in their intended downstream tasks, they often fail to generalize to tasks that they were not originally designed for, c.g., as we show later in our experiments, global representations do not generalize well to tasks such as lip reading [22 21 which require local spatio-temporal information.
Motivated by this, we take an orthogonal direction the current CSL approaches: We aim to learn representations agnostic to the types of downstream scenarios and generalize to both the scenarios that require global representations (e.g., classification) and scenarios that require local representations (e.g., localization). We focus on learning video representations using the natural
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

audio-visure correspondence as the primary self-supervisory signal. In this scenario, the notion of global/local representations is intertwined space and time; we can obtain representations that are spatially global or local, and also temporally global or local. However, most existing video CSL. approaches optimize for only global spatio-temporal representations and demonstrate them on audio/visual video classification tasks [60] 46) 55 61]. Part of the difficulty here is that formulating a contrastive objective for local representations is not straightforward because of the one-to-many relationship in audio-visual correspondences, i.e., spatially, multiple pixel regions can contribute to the sound in the corresponding audio, and temporally, multiple temporal slices of audio can map to a single video frame due to sampling rate differences. This hinders the development of CSL for local video representations useful for tasks such as sound source separation and lipreading.
In this paper, we present an approach for learning global-local video representations in the CSL frame- work. We design two cross-modal contrastive objectives that collaboratively capture information shared between audio and visual signals. An important aspect of our approach is the factoriza- tion of the spatio-temporal feature space into a speciaaly-localltemmmol www.l//aamaaaallllyleeoo.. glgobaal subspace and a sitiiilllyggobba/teeprro
subspace, where each of the two contrastive objectives are defined in, respectively; see Fig. [ The explicit space-time factorization helps each contrastive objective focus on capturing either spatially-local or temporally-
information and thus facilitates learning complementary features from audio-visual correspondence more effectively than in the original apatio-tempont space. Furthermore, we define both objectives in the multiple instance learning framework [24] [52] to handle the one-to- oue-to-many relationship between audio and visual signals. This helps the model learn representations without knowing fine- fine-graine grained audio-visual correspondence.
We evaluate our approach on various downstream tasks that need local spatio-temporal information, i.e., lip reading 21 [) deep-fake detection 25 and audio-visual event localization [76). and also discriminative tasks that needs global information, i.e., audio/visual video classification [71 47 (3
41. We show that the same pretrained model successfully generalizes to all our scenarios without having to re-pretrain using different objectives and/or datasets. Furthermore, we demonstrate that the two contrastive objectives mutually benefits each other and helps improve the generalizability of both global and local representations. To the best of our knowledge, our work is the first to demonstrate CSL approach that learns video representations that generalize to both global and local vidco understanding scenarios.
2 Related Work
Contrastive self-supervised learning. Contrastive learning leverages multiple views of the same data [59]. e.g., multiple perspectives within the same modality such as augmentations of the same image, different frames/clips of video, etc. [35
perspectives from different modalities 4.. DIM (38) and SimCLR [66 show that leveraging local information in contrastive learning further improves performance on image classification. DIM (38) has been extended to multi-scale [T]] and to video data (37). However, evaluation is still focused on "discriminative" tasks, e.g., image classification and video classification, while there is little evidence that these models will adapt well to tasks that require local information.
37
32
such as RGB and depth, images/videos and text 75
73 53
Several recent advances happened in the image domain, e.g., MoCo 35 191, BYOL [1],
SwAV 131, SimSiam [18], BarlowTwin 190. Although evaluation is performed in both global and local downstream scenarios such as image classification, object detection, semantic/instance segmentation and depth estimation, this line of work focuses on evaluating generalizability of the learned global representations rather than studying the importance of global and local information in generalization, which is the focus of this work. They also focus on image recognition tasks only.
Audio-visual video representation learning. Learning video representations from the natural audio- visual correspondence has been studied extensively. Most existing approaches aim to capture high- level semantic information useful for sequence-level (global) discrimination tasks such as audio/visual video classification [9] 46 5
55 61 23 Along this line of work, AV Flowwtast [8T utilizes different temporal scales of the audio and visual data, which encourages the model to capture fine- ane-grained temporal information. However, their learning objective optimizes for only global representations induced by different sampling rates and their evaluation is still focused on classification tasks.
Another line of work focuses on capturing fine-grained, spatio-temporal local information suitable for
"local" task such sound source separation and localization 7 69 93
92
28
88
64 50
Lin et

---

S:Local/T-Global
Spotiall- Lacol Temporolyy Glooa!
Audio
Video
neg
Temporal
Temporal Average Pooling
E
Average
Paoling
pos
E
AV Attention [Sec. 3.3)
Socciaity Globel Temponnallyl nool Subspare
Spatal Attention Pooling
pos
Fyy
E1
S-Global/T-Local Contrast (Sec. 3.2)
We factorize the joint spatio- tentool feature space into ppatially-loca//eemorallll -lobbbl subspace and a sitiiallyygooba//mmorlllll contrastive objective that compare audio- visuul signals in the multiple instance learning framework. For notational simplicity we indicate only the temporal scale in the superscripts of all the variables.
Figure 1:
subspace. In each space we define a cross- rrossl
al. [50] learn patch-level audio-visual correspondence by drawing positive/negative patches iteratively from sounding/non- -oounnding regions induced by audio-visual feature correlation. However, their representations only, and their evaluation is focused on localization tasks. In contrast to these previous work, we explicitly optimize for global and local spatio-temporal representations
approach learns
evaluate on both classification and localization downstream tasks. There are also some work proposed to learn audio-visual representations, however they are specifically designed for a particular task, c.g. speaker recognition [56). While our work learns general-
generall-urpose purpose video representation and is demonstrated on variety of downstream tasks.
3 Approach
We factorize the feature space into a hiniialy-leeaahhmmpprrrl--gggggo (S-boallSSSccaaaa (S-local/T-globa) subspace (S (S-global/T-loca)) subspace and define two cross-modal contrastive objectives each the subspaces. The S-local/T-global objective captures slowly changing patch-level information (S1c while the S-global/T-local objective captures fast changing frame-level information (Sec.3.2). Furthermore, we utilize the learned patch-level information to guide learning frame-level information via spatially attention pooling mechanism (Sec
and ppatially-global/temppe
3.1 Spatially-Local Temporally-Glohal Contrastive Objective
The purpose of this objective is to capture slowly changing patch-level information with high audio- visual correlation. We define an audio encoder Eg that computes audio embeddings 2 Tu F where T, is the audio sequence length and F is the number of frequency bands, and perform temporal average pooling to obtain z E RIXF Similarly, we define a visual encoder Ey that computes visual embeddings 28 IRT.xH xWCC xC, where T, is the visual sequence length and H. W. C are the height, width and channel dimensions, and perform temporal average pooling to obtain z E R1xHxx
To help our model capture slowly changing yet apatially-detaile information, we purposely make input frames lack local temporal information by feeding them at a low sampling rate, c.g., at one eighths of the original sampling rate, which has been shown to be effective at capturing fine-grained spatial details (27]. For audio, we found that keeping the original sampling rate to more effective.
We consider 2 and 24 that come from the same clip as positive pairs, while features coming from different clips are negatives. To capture local spatial information zg, we consider each cell H x W spatial grid as instance and define pairs by coupling z with =8 [6) Vi E H x W. Note that not all = will have valid audio-visual correspondence to 2g; it is rather likely that only few cells (c.g.. the lip region of a talking person) will match the information in z9. To resolve this
3

---

misalignment issue, we define our contrastive loss in the multiple instance learning framework 53
EMP F( 20 (i])
C = -log
EMPEE F( 2 (1) ΣnEEN ( 20)
where F(Ea; 2.) = exp(= (2]T= measures the compatibility between 2n and Eo. a set of spatial grids in zg. and N is set of negative visual instances taken from different clips, i.e. given batch of B video clips, we consider H x W x (B - 1)² negative pairs.
3.2 Spatially-Global Temporally-Loon Local Contrastive Objective
To capture information sensitive to temporal changes, c.g., the lip region of a talking head, we take seoporraly-granulle approach and contrast visual features at each time step to the corresponding audio features in a local temporal neighborhood. We compute audio embeddings z E RT.xF XF and visual embeddings z/, E RRT. H WCC in the same manner as before but without applying temporal average pooling at the end. To help our model capture fine-grained temporal changes, unlike in the previous objective, we feed frames at a higher sampling rate and perform spatial pooling over visual embeddings to obtain z! E R RTxxxxxxx.
We construct contrasting views by taking audio and visual features from the same local temporal block as positive pairs; visual pairs from different temporal blocks of the same video become negatives. We do not use samples from different videos in order to encourage the shared information time.
between modalities be just how features vary
Note that a single visual feature f t E To maps to multiple audio features [t], t E Ta because audio signals are typically captured at a higher sampling rate. Therefore, unlike in the previous objective, all the pairs within the same local temporal block can be considered as valid positives. There exists several ways to define contrastive objective with multiple positives [42 12 70] we opt for a simple approach that sums scores over all positive pairs, which leads to form similar to Equ.
EdpWee F( EEJTEEE F(2 =1((1) EnIeN F(=a: =1)
=
P is set of audio features within the same temporal block of 21,[t] and N is set of all possible audio-visual slice pairs that come from different parts of the same video. Given a video with T, frames and Ta M T. audio slices, we consider M (T. 1)2 negative pairs for video sequence.
3.3 Spatial Pooling with Audio-Visual Attention
To define the two subspaces apply global pooling over either spatial temporal dimensions. For the S-local/T-global subspace, we simply perform temporal average pooling because video inputs to model are relatively of short length (3 seconds) and tend to show content recorded in single scene. For the S-global/T-locaa subspace, however, spatial average pooling could be problematic because not all pixel regions will contribute to the sound in the corresponding audio signals.
Therefore, we use the learned S-local/T-global representation (Sec. 3.1 to obtain an visual attention map and use it to perform spatial attention pooling, as seen in Fig. [ Specifically, we compute the dot product between =Ω and each of 2 [1], Vi H x W to obtain an attention map indicating regions of high audio-visual correlation. We use this map to perform spatial pooling at each time step to obtain 2 Vt E T., We demonstrate the effectiveness spatial attention pooling in Table 8 and show qualitative examples in Fig. 2 for the scenario of sound source localization.
4 Experiments
Implementation details. We use 3D-ResNet 34 for our visual encoders (E and E and ID- ResNet (36) for our audio encoders (Eg and E.), both cases using Batch Normalization [39]. We share the weights of the two audio encoders and denote both simply by F.; for the visual encoders we instead keep the weights separate in order to encourage them to capture complementary information
from two different sampling rates; this can be seen an instantiation of the SlowFast network with no

---

lateral connection (27. All models are trained end-to-end using ADAM A4 with an initial learning rate 7 = 10 3 after a warm-up period of 500 iterations. We use 16 NVIDIA Tesla P100 GPUs with a batch size of 32. For the S-local/T-globbr objective we use features at 16 x 16 spatial resolution. To compute the S-global/T-local objective, we adopt a temporal window of size three without overlap.
During pretraining we sample frames at 10 FPS and apply random cropping, horizomtal flipping, gray- scaling, and temporal jittering. We set the clip length to 32 frames (3 seconds) and resize frames to 112 x 112; we feed 8 frames and 32 frames to E and Ey respectively. We extract mel-spectrograms from the raw waveform using LibROSA and get 80 T matrix with 80 frequency bands; T is proportional to the length of an audio clip. We then segment the mel mel-spectrogram according to the corresponding video clips to ensure temporal alignment. We treat the mel-spectrograms as an 80-channel ID signal. For downstream tasks we follow the standard data preprocessing protocols.
Datasets. Many video tasks involve human actions (e.g., action recognition), faces (e.g.. deepfake), and speech (e.g., lip reading). Pretraining our model on different datasets for different downstream tasks could produce misleading conclusions as the model could simply pick up the biases in the data useful for downstream tasks, c.g., Kinetics (14) contains various human actions useful for solving action recognition and AVSpeech [26] mostly contains human speech useful for lip reading, but not the other way around. Therefore, we pretrain our model only once for all downstream tasks using a combination of Kinetics [14 and AVSpeech (26.. The standard choice of video dataset for pretraining is Kinetics-400 [41] that contains 240K videos. We match the size by randomly selecting 120K video samples from each datasets; we term it as K-AV-240K. For the ablation study, we pretrain our model on subset of 15K samples from the K-AV-240K dataset. For fair comparisons with existing work, we also pretrain our model on the same datasets as with state-of-the-art (SOTA) approaches.
We evaluate our pretrained model on action recognition (UCF101 [71]. HMDB51 [47). and Kinet- ics400 [41)), on sound classification (ESC50 [63], on lip reading (LRW [22) and LRS2 (2I)), on deepfake detection (DFDC (25)). and on audio-visual event localization (AVE (76). We also conduct qualitative analysis on sound source separation on Kinetics-Sounds 6
4.1 Downstream Scenarios
Lip Reading. Visually recognizing a speaker's 's utterance is a challenging task, e.g., lip movements for different sounding letters can be visually similar to each other (c.g.. /b and /p/, /d and /t). This requires visual representations to capture fine- fine-grained spatio-temporal information. For a fair comparison with SOTA, we use the standard data processing protocol of [9T]. We detect 68 facial landmarks in each frame using dlib [15] and use the outer eye and nose tip landmarks to align the detected face in each frame using an affine transform. Finally, an image of size 112x 112 cropped from the aligned face with the lip landmarks at the center, so that the lip region occupies one third of the image width. We apply random horizontal flipping as data augmentation. We concatenate the features produced by our pretrained E and E and feed them to 2-layer MLP prediction head. For LRS2, we apply spatial average pooling before concatenation to preserve the temporal dimension and train the model using the CTC loss [30 For LRW we apply spatio-temporal average pooling before concatenation and train the model on the cross-entropy loss. In both cases we train the whole model end-to-end.
Table I compares our approach with SOTA supervised and self-super methods. For LRS2, we report the word error rate (WER; the lower the better); for LRW, we evaluate on a 500-way word classification task and report top-1 accuracy (the higher the better). The results show that our approach with the same ResNer18 backbone outperforms SOTA supervised approaches on LRS and LRW by large margins, i.e. 4.7% WER reduction on LRS2 and 5.1 1% accuracy improvements on LRW. All the bascline sulf-super methods optimize for variants of global contrastive objectives and generally perform poorly on all three datasets. Our approach outperforms all SOTA self-supervised approaches with the same backbone and using the same pretraining dataset. These results demonstrate the importance of capturing fine-grained spatio-temporal information necessary for lip reading.
Deepfake Detection. We observe that "deepfakes" tend be characterized by fine ime-grained audio- visual inconsistencies such as misalignment between lip motions and audio, unnatural facial and lip appearance/movemeents or asymmetry between facial regions such as the left and right eyes. Detecting such artifacts requires local spatio-temporal features. We take our pretrained model and finetune it on I second video clips from the DFDC dataset (25) for 100 epochs with a batch size of 16. We evaluate performance using wise Area Under the Curve (AUC). We follow the same data preprocessing
5

---

Method
Backbone
Pretrained on
LRS2]
LRWT
Method
Backbone
Premmined on
DEDC (ALCY)
WAS 211
Cons.
N/A
70.4
76.2
Capslle
5S8
VGG-19
N7A
53.3
ResNei18
Multi
37
Y-shape
TM-CTC 2
ResNer18
NIA
65.0
HeadPose
NIA
55.9
TM-spp
ResNer18
N/A
49.8
Two steem
5
Inception3
NIA
61.4
Xception-c2
XCeption
72.2
Perfect Match 251
TC:5
NIA
71.6
Moso4
Inceptions
NIA
75.3
Ne
72
ResNer34
NIA
83.0
DSP-FWA
NIA
75.5
84,41
DFTN 33
ResNe118
NIA
$4.1
MDS 20
ResNet18
NIA
91.5
MoCo 35
ResNei18
K-AV-15K
71.5
61.2
MoCa
ReNNat18
K-AV-15K
ResNet18
K-AV-15K
67.9
DPC 59
ResNe118
K-AV-15K
65.1
67.5
DFC 33
ResNet18
K-AV-15K
71.2
AVSlow Fast
31
ResNei18
K-AV-15K
56.1
75.8
Flow
ResNet18
K-AV-15K
80,9
VDIM
RosNet18
ResNei18
K-AV-15K
47.8
83.7
BesNeal8
K-AV-15K
90.1
ResNe:50
K-AV-15K
45.0
85.5
Ours
ResNet50
K-AV-15K
92.7
ResNei18
K-AV-210K
Table I: Comparison with SOTA on lipreading:
Table 2: Comparison with SOTA on deepfake de-
LRS2 21 (word enror rate (WER); lower is better)
tection [25]. f: [54 (0) use audio audio-visual visual signals; all the other methods use visual signals only.
and LRW 22 (top-1 accuracy; higher is better).
Method
Backbone
Pretrained on
UCFI0I1
HMDBSI1
ESC501
Random Forest 63
MLP
N/A
44.3
ConvNet 62
ConvNet-4
N/A
64.5
ConvRBM
68
ConvNct-4
N/A
86.5
Scratch
ResNet18
N/A
46.5
17.1
Supervised
ResNet18
ImageNet
82.8
46.7
MotionPred
C3D
K400-240K
61.2
33.4
RotNer3D
40
ResNet18
K400-240K
62.9
33.7
ST-Puzzle
43
ResNet18
K400-240K
65.8
33.7
ClipOrder
85
R(2+1)D-18
K400-240K
72.4
30.9
DPC 32
ResNet34
K400-240K
75.7
35.7
CBT 75
S3D&BERT
24000-240K
79.5
44.6
SeLaVi
R(2+1)D-18
K400-240K
83.1
47.1
XDC 5
R(2+1)D-18
K400-240K
84.2
47.1
78.0
AVTS 46
MC3
K400-240K
85.8
56.9
76.7
AVID
R(2+1)D-18
K400-240K
87.5
79.1
GDT
R(2+1)D-18
K400-2400
89.3
60.0
ResNet18
K K-AV-240K
90.1
61.3
80.1
Ours
ResNet18
K400-240K
91.1
61.9
79.8
Table 3: Comparison with SOTA on action classification (UCF101 (71) HMDB51 47 and sound We highlight the best results and the second best results.
classification (ESC50 63)
protocol as in SOTA approaches for this task, and use the same training and test sets as 20 We perform face detection to crop the face region in each video frame. We concatenate the features produced by our pretrained Ey and E after applying spatio-temporal average pooling, feed them to 2-layer MLP prediction head, and train the whole network end-to-end using the cross-entropy loss.
Table shows the results. Two of the baselines, all the other methods use only the visual features. We can see that when using only the visual features, our approach outperforms all previous SOTA approaches (AUC=96.7). We also compare our model with SOTA self-superviser approaches. Again, the baseline self-supervised methods perform poorly on this task which require local spatio-temporal information. Our model outperforms the best self-supervised result, highlighted in blue, by large margin (90.1 vs. 85.3).
20 and
use both visual and audio features;
Audio-Visual Event Localization. "audioovisuc" event" is defined as event that is both visible and audible in video segment. Audio-video event localization is usually evaluated in two settings, i.e. fully-supervised and weakly-supervised. The former aims predict which temporal segment of video has an audio-visual event and what category the event belongs to; the latter assumes that only a video-level event category available and there is no temporal event boundary information during training. This is challenging task because an event usually appears only in small portion of
Bluee comparisons of ours with the self self-supervised approaches under the same setting. Underline: best reported results of supervised methods. All the supervised results are from the literature; all the self supervied results ours. N/A: models trained from scratch target datasets. t higher/lower better.

---

frames within a video. Detecting which temporal segment contains an audio-visual event requires fine-ge aie-grained ("local") spatio-temporal representations, especially in there is no clue about temporal event boundaries.
setting, where
In Table 4 show comparisons with the SOTA on audio-visual event localiza- tion. Again, we take the same K-AV- 240K-pretrained model used in the previ- ous experiments and finetune it on the AVE dataset [76]. We concatenate the features produced by our pretrained E", and E and feed them to 2-layer MLP predic- tion head. Especially, we first utilize the F% to perform a spatial attention pooling on E. to highlight the "important" local spatial information. In this way, the final concatenated audio-visual features contain the desired the ppatio-tempont local infor- mation. We evaluate our model on both fully-supervised (Fully-super.) and weakdy- supervised (Weakly-super.) settings. For a fair comparison, we followed the same protocol and evaluation metric as (76). Without bells and whistles, we achieved 82.1% localization accuracy on fully-supervised setting, and 79.8% weakly-superrvseed setting, which all outperforms the SOTA, iee. PSP (94] (77.8%) (Fully-super.) and 73.5% (weakly-super.), by large margins.
Method
Fully-super.
Weakly-super.
AVEL
76
68.6
66.7
AVSIN
49
72.6
67.3
CMAN
87
73.3
70.4
DAM
80
74,5
AVRB
66
74.8
68.9
AVIN
65
75.2
69.4
AVT [
76.8
70.2
CMRA 86
77.4
72.9
PSP 94
77.8
73.5
Ours
82.1
79.8
Table 4: Comparison with SOTA on audio-visual event localization.
Sound Source Localization. To further demonstrate our approach achieving good audio-visual localization, we visualize audio-visual spatial attention maps on Kinetics-Soundt 6 (see Fig. 2 which contains videos deemed to have high audio- correspondence. Such visualization can also be considered as performing sound source localization, i.c. locate objects that making sound. To plot the figure, we use the learned audio-visual attention map, add softmax layer and apply bilinear interpolation of the 16 16 attention map back to the original image size, i.e. 192 192. The figure shows that our learned attention maps successfully localize sounding sources in videos, especially when visual content is highly related to the corresponding audio signal. For example, the first row (video frames from "playing instruments") shows that our model can successfully "running." localize the sounding region. For other activities like "baby talking." "playing "rruninal,""uuuninngg.. our model successfully highlights regions with humans. However, we find that the attention map incorrectly highlights regions on videos that have ambiguous audio-visual relation. We show example failure cases in the last two columns of the third row: There is no visual content that clearly relates with the audio signal, and thus the model fails to find sounding sources.
Action and Sound Classification. To evaluate the effectiveness of the learned global spatio-temporal representations, we evaluate our approach on action and sound classification. For action classification, we concatenate the visual features from E8 and E", after ppapooltemporal average pooling, feed them to 2-layer MLP prediction head, and train the whole network end-to-end using the cross-entropy loss. For audio classification, we apply temporal average pooling to the audio features from E, and feed them to a 2-leyer MLP prediction head, which is trained using the cross-entropy loss.
Table 3 shows the results. For a fair comparison to existing approaches, we report both the results pretrained on K-AV-240K and the results pretrained on Kinetics-400 [4]] that contains 240K videos (K400-240K). We find that, while the K400-2405 pretrained models leads to better performance on UCF101 and HMDB51 due to the similarities between datasets (all focus on human actions), the K-AV-240K pretrained models also achieve competitive results, outperforming all the baselines. Overall, on all three benchmarks, our approach achieves new SOTA results (91.1% on UCF101, 61.9% on HMDB51 and 80.1% on ESC50), demonstrating the effectiveness of learning global-local representations even for tasks that require global information. This suggests that optimizing for local representations also helps improve performance on classification tasks that require global information.
4.2 Ablation and Analysis
Comparison of different contrastive objectives. In Table 5 we compare various contrastive ob- jectives on tasks that require fine-grained local spatio-temporal information. All the methods use
7

---

19
Figure 2: Visualization of the learned audio-visual spatial attention maps on videos from Kinetics- Sounds show that they successfully locate sounding sources, e.g., musical instruments.
Method
Contrastive Objective
X-Modal
Spa.
Temp.
LRS2
LRW1
DFDC1
MoCo 35
Momentum Contrast
G
G
71.5
61.2
60.2
CPC 559
Predictive Coding
G
66.7
65.3
67.9
DPC 33
Dense Predictive Coding
65.1
67.5
71.2
VDIM 37
Global-Local DIM
L
53.2
70.7
85.3
AVTS 46
Vislio-issull Contrast
G
G
72.1
64.9
63.1
AVSlowFast
81
AVC Rotation
G
56.1
75.8
80.9
Ours
Global-Local AVC
G+L
G+L
47.8
83.7
90.1
Table Comparison of different contrastive objectives. All the results are based on our implemen- tation. X-Modal specifies the methods that use a cross-modal contrastive objective; others define contrastive objective on the visual modality only. also indicate whether each contrastive objective optimizes for global and/or local representations in spatial (Spa.) and temporal (Temp.) dimensions.
the same backbone (eD-ResNet18) and the same pretraining dataset (K-AV-15K), and follow the same experimental protocol. The results show that MoCo (35), which is successful for image clas- sification tasks, falls short on lip reading and deepfake detection. This suggests that the "vanilla" global contrastive objective may not be effective at tasks that require local information We find that the contrastive objectives that explicitly optimize for local spatial representations - DPC [33) VDIM (37], and ours generally perform well, suggesting the importance of local spatial contrastive objectives. We also see that AVSlowFass [8]] achieves competitive performance although does not explicitly optimize local contrastive objectives. This be explained by the fact that, unlike all the other methods, AVSlowFast [81] and ours define two visual encoders to capture complementary temporal information ("slow" and "fast" changing temporal features). In addition AVSlowFast explicitly optimize for global and local spatio- temporal representation via space-time factorization; the strong performance suggests the effectiveness of global-local contrastive objectives.
Importance of global-local joint contrastive objective. To demonstrate the importance of jointly learning global-local representations, the visual encoders Ey and E.? 2) do we need both the contrastive objectives C and L 3) do we need features from both the subspaces 24 and 2!, for the downstream tasks? Table we evaluate various combinations of these. We can see that jointly optimizing both the contrastive objectives holistically improves representations both the subspaces 28 and =! What is particularly interesting is that optimizing the objectives not only helps learn the subspace that they are defined in, but it also helps learn the other subspace. For example, compared to optimizing only L optimizing jointly with L helps also improve 2 (lines and b). We also evaluate variant of our approach
ablate three components of our model: 1) do we need both

---

V. Enc.
Obj.
Feat.
LRS21
LRW1
DFDC1
UCFI0IT
HMDB511
(8)
Both
LG (C")
(z
70.9
65.3
67.9
82.3
57.1
(c)
Both
Both
LG (z²)
47.6
86.8
92.6
89.2
59.9
(c)
Both
GI. (C')
GL (2.)
68.6
65.1
70.3
82.1
55.6
(d)
(E))
Both
GL. (=.)
50.8
81.2
89.7
83.6
57.3
(e)
Both
Both
GL (2%)
46.5
88.9
95.9
88.5
58.3
(f)
Both
Both
Both
45.1
89.2
96.7
90.1
61.3
Table 6: The roles of global and local information on different benchmarks. LG means spatially-
loca//emporaallyggobe and GL means
V. Enc.: visual encoder
setup, Obj.: contrastive objective used during pretraining. Feat.: features used in downstream tasks.
Method
LRS2 40.4
LRW1
DFDC1
UCFI0I
HMDB511
Avg Pooling & Single Pos. Pair
79.2
88.9
87.8
56.3
No Pooling & Multiple Pos. Pairs
47.8(1 7.4)
83.7(1 4.5)
90.1(t 1.2)
88.1(1 0.3)
56.8(1 0.5)
Table 7: Comparison of different methods to handle multiple positive audio-visual pairs in Lt.
that uses only the S-global/T-local encoder E (line d). Pretraining this with both the objectives improves over our full model pretrained with only L (line c), again suggesting the effectiveness of our global-loca joint contrastive objectives. Our full model consistently outperforms all the other variants, demonstrating the importance of all three components (encoders, objectives. and features).
Handling multiple positive pairs in L1 (Eq. 2]. In our S-global/T-local contrastive objective, we handle multiple positive audio-visual pairs (due higher audio sampling rate) by summing up the scores of all positive pairs. Here, we validate the effectiveness of this approach by comparing it to an alternative that performs an average temporal pooling over audio features in each local window and uses the vanilla contrastive loss [16] over the synchronized audio and visual features. Table 7 shows that performance of this alternative approach drops significantly on tasks that require local information (LRS2, LRW, DFDC), while for classification tasks both approaches achieve comparable results. This suggests the importance of our formulation specifically on capturing local information.
Audio window size. In our implementa- tion, each video frame covers one-tenth of a second and roughly maps to three audio slices (M 3). We therefore fix M = 3 to use all available audio slices without overlap between windows; here, we vary M E 31, 5, 7} to see how that affects the performance. Table 9 shows that M = (3. 5} is ideal. M = leads to the worst performance due information loss (we drop two audio slices per window), while M > starts degrading performance due to the increased noise in audio-visual correspondence (each window overlaps with others).
M
LRS2
LRW
DFDC1
UCFt
HMDB1
1
49.8
87.1
95.0
87.8
57.9
3
45.1
89.2
96.7
90.1
61.3
5
44.9
89.0
96.5
89.5
61.6
7
46.8
88.1
95.1
88.3
59.0
Table 9: Comparison of different window sizes (M) for audio-visual temporal matching.
AV spatial attention (Sec. 3.3). The learned audio-visual spatial attention map can highlight diserininative face regions useful for lip reading. In Table 8 we demonstrate the quality of our attention maps by replacing lip/face bounding boxes typically used in lip reading and deepfake detection with our attention map. We note that all SOTA approaches extract features from lip/face cropped regions using off- the-shelf detectors (which require substantial supervision on their own). First, as a baseline we evaluate a variant of our approach that is trained directly on full frames without using attention maps or lip/face detectors ("Ours/Full Frame"); the performance drops significantly on all three benchmarks. Next, we extract features from the entire frame (no lip/face cropping) and use our attention map to pool the features spatially. Note that the purpose this experiment is to evaluate the quality of attention maps; we use audio signal just to obtain attention maps and discard for
detection. The results ("Ours/Attentionn)) show that it achieves results comparable to our best setting ("Ours/Crop"), which extracts features from the cropped líp/face region similar to the SOTA approaches. Notably, the attention-hased approach outperforms SOTA on LRW and DFDC even without relying lip/face region detectors, demonstrating the effectiveness.

---

Task
SOTA Results
Ours/Full Frame
Ours/Attention
Ours/Crop 45.17Lip Crop
LRS21 LRW1
2sc
2
49.8
71.9
51.2
DFTN 82
84.1
62.3
85.1
89.2 / Lip Crop
VDIM 37
85.3 (V)
96.7 (V) Face Crop
DFDCt
68.1 (V)
95.9 (V)
MDS
20
91.5 (V+A)
97.1 (V+A) Face Crop
Table 8: Evaluation of the learned audio-visual spatial attention maps. "V" uses only visual sequence and "V+A" uses both visual and audio sequence for finetuning on downstream tasks.
Kinetics<00
UCF101
HMMBB11
ESC50
LRS2
LRW
DFDC
Method
Backbone
LV
FT
LN
LN
FT
LN
FT
FT
FT
FT
AVTS
40
MC3
89.0
61.6
80.6
XDC
R(2+1)D-18
91.2
61.0
84.8
R(2+1ID-50
64.7
GDT
61
R(2+1)D 18
92.5
66.1
88.5
VA
R(2+1)D-18
55.5"
91.5
60.0
70.1
85.6
VA
S3D-G
59.8"
84.7
90.1
60.4
68.2
86.1
Ours
ResNeti8
63.7
71.5
85.1
93.9
61.2
73.7
83.1
89,3
38.1
95.2
98.9
Table 10: Comparison to SOTA approaches that are pretrained on AudioSet (29) "FT"; finetuning, "LN": linear evaluation. *: evaluation on Kinetics600
Pretraining on large-scale dataset. We also investigate how the pretraining scale affects the results on various downstream tasks. To this end, we pretrain our model on AudioSet [29] and evaluate it on all downstream tasks. The results show in Table TO] We report the results on both the linear (LN) evaluation and finetuning (FT) scenarios. We followed the experimental protocol used in our paper and used the same pretrained checkpoint for all downstream scenarios. For fair comparison, we only show the results evaluated by AudioSet pretrained models for all the other comparing approaches. As we can see, among all the SOTA approaches. "Ours" achieves the best performance. When pretraining on a large-scale dataset, i.c. AudioSet with million video clips, the performance can further be improved comparing with that pretrained on medium scale dataset, e.g. Kinetics and K-AV-240k.
5
Conclusion
We presented contrastive approach for learning global and local spatio-temporal video represen- tations from audio-visual correspondence. We showed that the space-time factorization leads to an effective solution for learning global-local representations. Unlike many prior work in the video self- supervised learning literature, we expand the downstream evaluation scenarios to include both "global" and "local" tasks and demonstrate that approach successfully transfers to various tasks including lip reading, deepfake detection, audio-visual event localization, and action/sound classification.
Limitations. We leverage audio-visual correspondence to compute the spatial attention map. However, when applying to downstream tasks which contains videos with no audio, such mechanism can not be used. In addition, the attention mechanism should also be considered along the temporal dimension. For example, certain frames might play more important roles in deepfake detection, e.g. frames with obvious artifacts, or in the case of lip reading, some frames or audio slices can give more clues for the model to recognize the word through visemes and/or phonemes. How to incorporate temporal or spatio-temporal attention is also an important future work.
Boarder Impact. Our paper studies self-supervised pretraining, with applications to tasks involving both audio and visual signals, e.g. deepfake detection, lip reading, audio-visual event localization and audio/video classification. As one of the core machine learning problems, self-supervised pretraining can enable machine learning to work better and more efficiently with less data and/or task task-specific designs. Especially, audio and video signals are two key sensory signals in many real-world scenarios. In this sense, our work have broader applications in computer vision, audio/speech, bioinformatics, and human-man maman-machine intelligence. The datasets our study are all publicly available. But every data- driven method brings the risk of learning biases in the data. Although our approach is promising in creating more safe and real environment, iee. deepfake detection, we encourage the deployment of our method to be done with careful consideration of sensitive applications that have ethical implications.
10

---

References
D. Afchar, V. Nozick. J. Yamagishi. and I. Echizen. Mesonet: compact facial video forgery detection network. In 2018 IEEE International Workshop on Information Forensics and Security (WIFS), pages 1-7. IEEE, 2018.
[2] T. Afouras, J. S. Chung, A. Senior, O. Vinyals, and A. Zisserman. Deep audio-visual speech recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, pages 1-1, 2018.
[3] T. Afouras. J.S. Chung, and A. Zisserman. Lrs3-ted: large-scale dataset for visual speech recognition. In arXiv preprint arXiv:1809.0m 00496. 2018.
[4] J.-B. Alayrac, A. Recasens, R. Schneider, R. Arandjelovié, Ramapuram, J. De Fauw, L. Smaira, S. Dieleman, and A. Zisserman. supervied multimodal versatile networks. Advances in Neural Information Processing Systems, 33, 2020.
[5] H. Alwassel, D. Mahajan, L. Torresani, B. Ghanem, and D. Tran. Self-supervised learning by cross-modal audio-video clustering. arXiv preprint 191. 2019.
[6] R. Arandjelovic and A. Zisserman. Look, listen and learn. In ICCV, 2017.
[7] R. Arandjelovic and A. 7isserman. Objects that sound. In Proceedings of the European conference on computer vision (ECCV). pages 435-451, 2018.
[8] S. Arora, H. Khandeparkar, M. Khodak, O. Plevrakis, and N. Saunshi. A theoretical analysis of contrastive reppereviieeeppeeeentttttiio
unsupervised representation learning. arXiv preprint
2019.
[9] M. Asano, M. Patrick, Rupprecht, and A. Vedaldi. Labelling unlabelled videos from scratch with multi-modal self-supervision. arXiv preprint arXiv:2006, 2020.
[10] P. Bachman, R. D. Hjelm, and W. Buchwalter. Learning representations by maximizing mutual information acTOSs views. arXiv preprint urXiv:1906.000) 2019.
[11] P. Bachman, R. D. Hjelm, and W. Buchwalter. Learning representations by maximizing mutual information across views. In Advances Neural Information Processing Systems, pages 15535-15545, 2019.
[12] Q Cai, Y. Wang, Y. Pan, T. Yao, and T. Mei. Joint contrastive learning with infinite possibilities. arXiv preprint arxiv:2009. 14777 2020.
[13] Caron, Misra, Mairal, P. Goyal, Bojanowski, and A. Joulin. Unsupervised learning visual features by contrasting cluster assignments. arXiv preprint arXiv:2006.09882, 2020.
[14] J. Carreira, F. Noland, C. Hillier, and A. Zisserman. A short note on the sinetics-700 human action dataset. arXiv preprint arxiv: 00077999777777 2019.
[15] D. Castelli and P. Pagano. Opendlib: A digital library service system. In M. Agosti and C. Thanos, editors, Research and Advanced Technology for Digital Libraries, pages 292-308, Berlin, Heidelberg, 2002. Springer Berlin Heidelberg.
[16] T. Chen, Korbllihh, M. Norouzi, and G. Hinton. A simple framework for contrastive learning visual representations. arXiv preprint 2002::200,.05709, 2020.
[17] X. Chen, H. Fan, R. Girshick, and K. He. Improved baselines with momentum contrastive learning, arXiv preprint arXhn:2003.04297 2020.
[18] X. Chen and K. He. Exploring simple siamese representation learning. arXiv preprint 2xXvv20111.1055 2020.
[19] X. Chen, Xic, and K. He. empirical study of training self self-super vision transformers. arXiv preprint arXiv:2104.022 02005, 2021.
[20] K. Chugh, P. Gupta, Dhall, and R. Subramanian. Not made for each anteer-udio-visste dissonance--asee deepfake detection and localization, arXiv preprint arXiv:2005. 14405, 2020.
[21] J. S. Chung, A. Senior, O, Vinyals, and A. Zisserman. Lip reading sentences in the wild. In 2017 IEEE Conference Computer Vision and Pattern Recognition (CVPR), pages 3444-3453 IEEE, 2017.
[22] J. S. Chung and A. Zisserman. Lip reading the wild. In Asian Conference Computer Vision, 2016.

---

[23] S.-W. Chung. J. S. Chung. and H.-G. Kang. Perfect match: Improved cross- modal embeddings for audio- visual synchronisation. In ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 3965-3969. IEEE, 2019.
[24] T. G. Dietterich, R. H. Lathrop, and T. Lozano-Pérez. Solving the multiple instance problem with axis-parallel rectangles. Artificial intelligence, (9(1-))311 1997.
[25] B. Dolhansky, R. Howes, B. Pflaum, N. Baram, and C. C. Ferrer. The deepfake detection challenge (dfde) preview dataset. arXiv preprint arXiv: 190.08844, 2019.
[26] A. Ephrat, I. Mosseri, O. Lang. T. Dekel, K. Wilson, A. Hassidim, W. T. Freeman, and M. Rubinstein. Looking to listen at the cocktail party: speaker-indepennent audio-visual model for speech separation. arXiv preprint 300.vv800.03660
2018.
[27] C. Feichtenhofer, H. Fan, J. Malik, and K. He. Slowfast networks for video recognition. Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 6202 222-6211, 2019. 2019.
[28] C. Gan, D. Huang. H. Zhao, J. B. Tenenbaum, and A. Torralba. Music gesture for visual sound separation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10478- 10487, 2020.
[29] J. Gemmeke, D. Ellis, D. Freedman, A. Jansen, W. Lawrence, R. Moore, M. Plakal, and M. Ritter. Audio set: An ontology and human-labeled dataset for audio events. pages 776-780, 03 2017.
[30] A. Graves, S. Fernnndez, F. Gomez, and J. Schmidhuber. Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks. In Proceedings of the 23rd international conference Machine learning. pages 369-376, 2006.
[31] J.B. Grill, F. Strub, F. Altché, C. Tallec, P. H. Richemond, E. Buchatskaya, C. Doersch, B. A. Pires, Z. D. Guo, M. G. Azar, al. Bootstrap your latent: A approach self-supervised learning. arXiv preprint (XXi:2006600733: 2020.
[32] T. Han, W. Xie, and A. Zisserman. Video representation learning by dense predictive coding. In ICCV, 2019.
[33] T. Han, W. Xie, and A. Zisserman. Video representation learning by dense predictive coding. In Proceedings of the IEEE International Conference on Computer Vision Workshops, pages 0-0, 2019.
[34] K. Hara, H. Kataoka, and Y. Satoh. Can spatiotemporal 3d cnns retrace the history of 2d enns and imagenet? In CVPR, 2018
[35] K. He, H. Fan, Y. Wu, S. Xie, and Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages 9729-9738, 2020.
[36] K. He, X. Zhang, S. Ren, and Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770-778, 2016.
[37] R. D. Hjelm and P. Bachman. Representation learning with video deep infomax. arXiv preprint arXiv:2007. 13278, 2020.
[38] R. D. Hjelm, A. Fedorov, S. Lavoie-Marchildon, K. Grewal, P. Bachman, A. Trischler, and Bengio. Learning deep representations by mutual information estimation and maximization. arXiv preprint arXiv: 1808. 06670, 2018.
[39] S. loffe and C. Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv:1502.03111 2015.
[40] I. Jing and Y. Tian, Self-supervised spatiotemporal feature learning by video geometric transformations. arXiv preprint /811. / 2018
[41] W. Kay, J. Carreira, K. Simonyan, B. Zhang, C. Hillier, S. Vijyanarasimman, F. Viola, T. Green, T. Back,
P. Natsev, et al. The kinetics human action video dataset. arXiv preprint /705.00000009555 2017.
[42] P. Khosla, P. Teterwak, C. Wang, A. Sama,, Y. Tian, P. Isola, A. Maschinot, C. Liu, and D. Krishnan. Supervised contrastive learning. arXiv preprint arXiv:2004.1 100000/11366 1136,, 2020.
[43] D. Kim, D. Cho, and I. S. Kweon. Self-supervised video representation learning with space-time cubic puzzles. In AAAI, 2019.
12

---

[44] D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. arXiv preprini arXiv:1412.6980. 2014.
[45] Q. Kong, W. Wei, Z. Deng. T. Yoshinaga, and T. Murakami. Cycle-contrast for self-supervised video representation learning. arXiv preprint arXiv:2010. 148000 2020.
[46] B. Korbar, D. Tran, and L. Torresani. Cooperative learning of audio and video models from self-supervised synchronization. In Advances in Neural Information Processing Systems, 2018.
[47] H. Kuchne, H. Jhuang. E. Garrote, T. Poggio, and T. Serre. Hmdb: large video database for human motion recognition. In ICCV, 2011.
[48] Y. Li and S. Lyu. Exposing deepfake videos by detecting face warping artifacts. arXiv preprint 1.11:660 2018.
[49] Y.BB. Lin, Y.-J Li, and Y.-C.F. Wang. Dual-modality seq2seq network for audio- event localization. In ICASSP 2019-2019 JEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 2002-2006. IEEE, 2019.
[50] Y.B. Lin, H.-Y. Tseng, H.-Y. Lee, Y.-Y. Lin, and M.-H. Yang. Unsupervised sound localization via iterative contrastive learning. arXiv preprint (xXi::2004000:: 2021.
[51] Y.-B. Lin and Y.-C. Wang. Audiovisual transformer with instance attention for audio-visual event localization. In H. Ishikawa, C.-L. Liu, T. Pajdla, and J. Shi, editors, Computer Vision ACCV 2020, pages 274-290, Cham, 2021. Springer International Publishing.
[52] O. Maron and Lozano-Pérez. framework for multiple-instance learning. Advances neural information processngg systems, pages 570-576, 1998.
[53] A. Miech, J. Alayrac, I.. Smaira, I. Laptev, J. Sivic, and A. Zisserman. End-to-end learning of visual representations from uncurated instructional videos. In 2020 IEFF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 9876-9886, Los Alamitos, CA, USA, jun 2020. TEEE Computer Society.
[54] T. Mittal, U. Bhattacharya, R. Chandra, A. Bera, and D. Manocha. Emotions don't 't lie: An audio-visual deepfake detection method using affective cues. In Proceedings of the 28th ACM International Conference on Multimedia, pages 2823-2832, 2020.
[55] P. Morgado, N. Vasconcelos, and I. Misra. Audio-visual instance discrimination with cross-modal agreement. arXiv preprini arXiv:2004. 12943, 2020.
[56] A. Nagrami, S. Chung, S. Albanie, and A. Zisserman. Disentangled speech embeddings using cross-modal self-supervision. In ICASSP 2020 2020 IEFE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6829-6833 6833, 2020.
[57] H. H. Nguyen, F. Fang, J. Yamagishi, and Echizen, Multi-task learning for detecting and segmenting manipulated facial images and videos. arXiv preprint 190.06790 2019.
[58] H. H. Nguyen, J. Yamagishi, and Echizen. Capsule-forensics: Using capsule networks to detect forged JEEE International Conference on Acoustics, Speech and Signal P Processing (ICASSP), pages 2307-2311. IEBE, 2019.
images and vidcos. In ICASSP 2019-2019
[59] A. v. d. Oord, Y. L.i, and O. Vinyals. Representation learning with contrastive predictive coding. arXiv preprint Xiv: / 807.03748, 2018.
[60] A. Owens and A. A. Efros. Audio-visual scene analysis with self-supervised multisensory features. In Pro Proceedings of the European Conference Computer Vision (ECCV), pages 631-648, 2018.
[61] M. Patrick, Y. M. Asano, R. Fong, J. F. Henriques, G. Zweig, and A. Vedaldi. Multi-modal self-supervision from generalized data transformations. arXiv preprint 2rXiv:2003.04299, 2020.
[62] K. J. Piczak. Environmental sound classification with convolutional neural networks. In International Workshop on Machine Learning for Signal Processing (MLSP), 2015.
[63] K. J. Piczak. ESC: Dataset for environmental sound classification. In Proceedings of the 23rd ACM international conference Multimedia, 2015.
[64] R. Qian, H. D. Di Hu. M. Wu, N. Xu, and W. Lin. Multiple sound sources localization from coarse to line. arXiv preprint 2rXiv:2007.06335, 2020.
13

---

[65] J. Ramaswamy. What makes the sound?: A dual-modality interacting network for audio-visual event localization. In ICASSP 2020 2020 IELE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 4372 4376, 2020.
[66] J. Ramaswamy and S. Das. See the sound. hear the pixels. In 2020 IEEE Winter Conference on Applications of Computer Vision (WACV), pages 2959-2968, 2020.
[67] A. Rossler, D. Cozzolino, L. Verdoliva, C. Riess, J. Thies, and M. Nießner. Faceforensics++: Learning to detect manipulated facial images. In Proceedings of the IEEE International Conference Computer Vision, pages 1-11 1, 2019.
[68] H. B. Sailor, D. M. Agrawal, and H. A. Patil. Unsupervised filterbank learning using convolutional restricted bolızmann machine for environmental sound classification. In INTERSPEECH, 2017.
[69] A. Senocak, T.-H. Oh, J. Kim, M.-H. Yang, and I. So Kweon. Learning to localize sound source in visual scenes. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4358-4366, 2018.
[70] J. Song and S. Ermon. Multi-label contrastive predictive coding. arXiv preprint arXiv:2007.09852 2020.
[71] K. Soomro, A. R. Zamir, and M. Shah, Ucf101: A dataset of 101 human actions classes from videos in the wild. arXiv preprint vrXiv:1212.0402 2012.
[72] T. Stafylakis and G. Tzimiropoulos. Combining residual networks with Istms for lipreading. arXiv preprint 2017.
[73] C. Sun, F. Baradel, K. Murphy, and C. Schmid. Contrastive bidirectional transformer for temporal representation learning. arXiv preprint arXiv: arXiv: 1906.05743, 2019.
[74] J. Thoma, D. P. Paudel, and L. Van Gool. Soft contrastive learning for visual localization. Advances in Neural Information Processing Systems 33, 2020.
[75] Y. Tian, D. Krishnan, and P. Isola. Contrastive multiview coding. arXiv preprint arXiv:1906.05849, 2019.
[76] Y. Tian, J. Shi, B. Li, Z. Duan, and C. Xu. Audio-visual event localization in unconstrained videos. In Proceedings of the European Conference on Computer Vision (ECCV), pages 247 247-263, 2018.
177] Y. Tian, C. Sun, B. Poole, D. Krishnan, C. Schmid, and P. Isola. What makes for good views for contrastive learning. arXiv preprint Xiv::2005 10243, 2020.
[78] J. Wang, J. Jiao, I.. Bao, S. He, Y. Liu, and W. Liu. Self-supervised spatio-temporal representation learning for videos hy predicting motion and appearance statistics. In CVPR, 2019.
[79] X. Weng and K. Kitani. Learning spatio-temporal features with two-stream deep 3d cnns for lipreading. arXiv preprint arXiv: (aee::905000044 2019.
[80] Y. Wu, L. Zhu, Y. Yan, and Y. Yang. Dual attention matching for musual event localization. In 2019 IEEE/CVF International Conference Computer Vision (ICCV), pages (291-6299, 2019.
[81] F. Xiao, J. Lee, K. Grauman, J. Malik, and C. Feichtenhofer. Audiovisual slowfast networks for video recognition. arXiv preprint arXiv:2001. 008740, 2020.
[82] J. Xiao, S. Yang, Y. Zhang, S. Shan, and X. Chen. Deformation flow based two-stream network for lip reading. arXiv preprint XxXv:2003.005009 2020.
[83] E. Xie, J. Ding. W. Wang, X. Zhan, H. Xu, Z. Li, and P. Luo. Detco: Unsupervised contrastive learning for object detection. arXiv preprint arXiv:2102.0444 0400000000.. 2021.
[84] Y. Xiong, M. Ren, and R. Urtasun. Loco: Local contrastive representation learning. arXiv preprint arXiv:2008.0133 2020.
[85] D. Xu, J. Xiao, Z. Zhao, J. Shao, D. Xie, and Y. Zhuang. Self-supervised spatiotemporal learning via vidco clip order prediction. In CVPR, 2019.
[86] H. Xu, R. Zeng, Q. Wu, M. Tan, and C. Gan. Cross-modal relation-aware networks for audio-visun event localization. In ACM International Conference on Multimedia. 2020.
[87] H. Xuan, Z. Zhang. S. Chen, J. Yang. and Y. Yan. Cross-modal attention network for temporal inconsistent audio-visual event localization. Proceedings of the AAAI Conference on Artificial Intelligence, 34:279-288 04 2020.
14

---

[88] K. Yang. B. Russell, and J. Salamon. Telling left from right: Learning spatial correspondence of sight and sound. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 9932-9941, 2020.
[89] X. Yang. Li, Lyu. Exposing deep fakes using inconsistent head poses. ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 8261-8266 IEFE, 2019.
[90] J. Zbontar, L. Jing, I. Misra, Y. LeCum, and S. Deny. Barlow twins: Self-supervised learning via redundancy reduction. arXiv preprint arXiv:2103.032220 2021.
[91] X. Zhang, F. Cheng, and S. Wang- Spatio-temporal fusion based convolutional sequence learning for lip reading. In Proceedings the IEFF/CVF International Conference on Computer Vision (ICCV), Octoher 2019.
[92] H. Zhao, Gan, W.-C. Ma, and A. Torralba. The sound of motions. Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1735-1744, 2019.
[93] H. 7hao, C. Gan, A. Rouditchenko, C. Vondrick, J. McDermott, and A. Torralba. The sound of pixels. In P Proceedings of the European conference on computer vision (ECCV), pages 570-586, 2018.
[94] J. Zhou, L. Zheng, Y. Zhong, S. Hao, and M. Wang, Positive sample propagation along the audio-visual event line. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8436 8444, 2021.
[95] P. Zhou, Han, V. I. Morariu, and Davis. Two-stream neural networks for tampered face detection. In 2017 IEEE Conference on Compuder Vision and Puttern Recognition Workshops (CVPRW), pages 1831-1839. IEEF, 2017.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [Yes]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or URL)? [Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [N/A]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes]
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] We are using public datasets and they are all cited in our paper.
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either in the supplemental material or as URL? [No]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]
15

---

(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [Yes] We discuss potential bias issues in Sec. [
5. If you used crowdsoureing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
16


Reviews:
[{"review_id": "ZvLp4WfnpTc", "reviewer": "Reviewer_zMzE", "summary": "This paper proposes an audio-visual self-supervised learning approach based on two cross-modal contrastive losses that learn audio-visual representations that can generalize to both the tasks which require global semantic information and localized spatio-temporal information. Extensive experiments on 4 downstream tasks demonstrate the usefulness of the learned representations, compared to their disjoint learned counterparts.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\n- The paper is generally nicely written. Existing works tend to learn either global representations or local representations, while this work aims at learning versatile representations that generalize well to both scenarios.\n\n- The motivation/advantage of learning global and local representations jointly is well justified---learning global/local representations jointly improves the generalizability of the learned representations compared to the disjointly learned counterparts, especially when the downstream scenarios are unknown in advance. \n\n- Good observation and analysis to construct triplets from different spatial and temporal spans to capture either global information or local information. A tailored global-local contrastive learning network is designed to realize the proposed idea.\n\n- Extensive experiments are performed on four downstream tasks across a number of datasets to demonstrate the generality of the learned representations. The proposed self-supervised audio-visual representation learning method compares favorably to other self-supervised learning methods and supervised learning methods.\n\n- Very nice ablation studies in Table 5-7 demonstrate the effectiveness of the key components such as 1) impact of both the local and global features, 2) MIT for local contrast, and 3) attention mechanism.\n\nWeakness / Detailed comments:\n\n- The paper should discuss other prior work that learns global and local representations in a similar spirit, and explicitly discuss the main contribution and difference of the proposed method. E.g. [1][2]. Otherwise, the novelty of the proposed method is not salient.\n\n- Justify the choice of downstream tasks to evaluate the learned global-local representations. Why these four tasks, and how do they represent other potential downstream tasks, since it's claimed that the proposed method learns very general representations? Or, more tasks are needed to justify that the learned representations can generalize to various tasks that leverage spatio-temporal representation.\n\n[1] \"Actbert: Learning global-local video-text representations.\" In CVPR, 2020.\n\n [2] \"Contrastive learning of global and local features for medical image segmentation with limited annotations.\" In NeurIPS, 2020.\n\n\n################################AFTER REBUTTAL###########################\nThank the authors for the response. My initial concern was that the idea of learning global and local information isn't entirely new and prior work is not acknowledged and discussed, and their choice of downstream tasks is not well justified. In their response, the authors have provided more discussions and explanations, which generally addressed my concerns. They have also provided interesting new results on action recognition (Kinetics-400) and audio-visual event localization (AVE). Therefore, I would be happy to see this paper getting accepted and encourage the authors to include the new discussions and results in the final version.\n\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors have adequately addressed the limitations and potential negative social impact of the work.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {"ethical_concerns": "The downstream tasks in this paper involves deepfake detection and lip-reading, which could potentially lead to ethical concerns. The Broader Impacts section in the paper has discussed and well addressed the concerns."}}, {"review_id": "YDzWOvQav6", "reviewer": "Reviewer_afVc", "summary": "This paper studies self-supervised learning of audio-visual representations from video data using neural networks and a contrastive audio-visual correspondence objective.\n\nThe key novelty of this paper is that it asymmetrically applies pooling/attention to simultaneously learn two different representations: spatially-local/temporally-global, and spatially-global/temporally-local. Prior work has focused on learning only one of these two, or simply learned spatially-global/temporally-global representations. \n\nThe model is pre-trained on both Kinetics-400 as well as a mixed dataset, using a 120k video subset of Kinetics-400 with a 120k video subset of the AVSpeech dataset. A single pre-trained model can then be fine-tuned on multiple downstream tasks, including action recognition on UCF101 and HMDB51, sound classification on ESC50, lip reading on LRW and LRS2, and deepfake detection on DFDC. Qualitative analysis of sounding object localization is also performed on Kinetics-Sounds. The quantitative results are strong, in most cases outperforming previous self-supervised approaches trained under similar conditions. \n", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\nThe paper presents a novel and well-motivated approach to self-supervised contrastive learning from videos\n\nThe paper contains extensive experimental results on diverse downstream tasks that demonstrate the strong efficacy of the proposed approach\n\nThe paper is well-written and easy to understand\n\nWeaknesses:\n\nIt would have been nice to evaluate on a downstream task that explicitly evaluates segmentation/localization, as opposed to tasks that only indirectly evaluate it\n\nSuggestions:\n\nLine 187: phoneme transcriptions are missing a slash, e.g. /p should be /p/\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2.5"}, "extra_text": {}}, {"review_id": "Dx8de9lZD4", "reviewer": "Reviewer_EYTG", "summary": "The paper introduces a novel method for self-supervised video from audio and video. The technique uses both the temporal and spatial dimension of the audio-visual input in order to build two contrastive losses: one capturing slow changing elements in the video present in the audio and the other one capturing temporally changing elements. After learning through these two losses, authors propose a detailed downstream evaluation consisting of task where spatio-temporal granularity is relevant (deepfake detection, lip reading) and other task about global information in the video (action classification).", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall, I think the new ideas introduced in the paper are interesting and the evaluation section show that the resulting model has good performance for lip reading, detection and classification. I am missing some comparisons with the state-of-the-art, but overall I believe this is a good paper and should be accepted. \n\nStrengths: \n- The paper is clear and well written. The different parts of the model are well motivated and it is easy for follow and understand the reason behind the design decisions.\n\n- The idea of introducing explicitly the notion of fine-grained spatiotemporal information in contrastive learning is interesting and novel. Authors execution of the idea is simple but effective, building the two losses consistently with their goals. In my opinion, new methods for self-supervised learning will start considering more granular information, following the direction of this paper. \n\n- Authors ablate all the relevant decisions, which to me is interesting and shows the importance and effect of the two losses used in the model. I like how looking at Table 5 you can extract the main conclusions in the paper.\n\n- Authors also present qualitative results of the attention between the audio and the video modality. The results are consistent with the goal of the attention layer (highlight the areas in the video with relevant audio-visual correlation). \n\n- The model is evaluated for very different tasks (lip reading, deepfake detection and classification) which is good for the reader to understand better the properties of the model.\n\nWeaknesses: \n- In terms of evaluation regimes, I am missing linear evaluation for some of the video classification tasks. It is hard to evaluate the quality of the features by only looking at the fine-tuning performance where all the network is updated on the downstream task.\n\n- In terms of evaluation datasets, I am missing larger action classification datasets such as Kinetics-400 (where authors only train without labels). I believe having additional evaluation performances in this datasets would help the readers understand a bit better the strengths of the features.\n\n- In terms of training datasets, I would be curious to know how this model would scale using larger scale dataset such as AudioSet. One question to answer would be if this performance improvement is only present for smaller datasets but goes away when using more data. I think it would be good if authors can answer this question.\n\n- Although I understand that some papers cannot be directly comparable, I think it would be worth for the authors to at least report performances for some of the models trained on different architectures or datasets. I think it is important for the reader to put the quantitative performance on perspective to all the works on the topic. Some examples of this would be reporting XDC or GDT when trained in AudioSet or reporting performance of MMV. \n\n- Authors ablate very well all the design decisions. However, I am curious about whether it is necessary to have such a large negative pool for the S-Global / T-local loss. I wonder if authors can evaluate that by just sampling some of the negatives at each iteration. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes, I think the authors have address the limitations of their work as well as their possible societal impact. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "50xF4E9VQ_z", "reviewer": "Reviewer_jwv4", "summary": "The paper proposes a method to train self-supervised cross-modal embeddings that are strong in both global and local representations. This is done by optimizing two objectives -- spatially-local/temporally-global and spatially-global/temporally-local. The first is trained using contrastive learning by taking negative examples from different clips, whereas the second is trained by taking negative examples from different time in the same clip. The authors conduct a range of experiments lip reading, deepfake detection, sound classification, etc.", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is well motivated. The authors recognise an important problem and present an effective way of solving it. \n\nHowever, the key training method (e.g. sampling method for negatives) overlap strongly with Nagrani et al. [1] which is not cited in the paper. The authors should give a clear summary of the different between this paper and theirs. I may change my rating based on this.\n\nThe experiments are very thorough and convincing. The ablation studies are good. The paper is well written.\n\n[1] https://ieeexplore.ieee.org/document/9054057", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The comments on limitations and broader impact are adequate. There is no particular concern beyond generic risks in ML applications.", "review_text": "", "extra_scores": {"ethical_concerns": "The paper appears to be fine in this regard.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2hr"}, "extra_text": {}}]

Author rebuttal:

Meta-review: All the reviewers appreciated the global and local message, the clear exposition, the ablations, and the additional experiments carried out
in response to the reviews.

Decision: Accept (Poster)