Contrastive Self-Supervised Learning of Global-Local Audio-Visual Representations
Authors: Shuang Ma
Zhaoyang Zeng
Daniel McDuff
Yale Song
Venue:  (2021)
Keywords: Contrastive learning
self-supervised learning
video representation learning
audio-visual representation learning
multimodal representation learning

Abstract: Contrastive self-supervised learning has delivered impressive results in many audio-visual recognition tasks. However, existing approaches optimize for learning either global representations useful for high-level understanding tasks such as classification, or local representations useful for tasks such as audio-visual source localization and separation. While they produce satisfactory results in their intended downstream scenarios, they often fail to generalize to tasks that they were not originally designed for. In this work, we propose a versatile self-supervised approach to learn audio-visual representations that can generalize to both the tasks which require global semantic information (e.g., classification) and the tasks that require fine-grained spatio-temporal information (e.g. localization). We achieve this by optimizing two cross-modal contrastive objectives that together encourage our model to learn discriminative global-local visual information given audio signals. To show that our approach learns generalizable video representations, we evaluate it on various downstream scenarios including action/sound classification, lip reading, deepfake detection, and sound source localization. 

Under review as a conference paper at ICLR 2021
CONTRASTIVE SELF-SUPERVISED LEARNING OF -LOBAL-LOCAL AUDIO-VISUAL REPRESENTATIONS
Anonymous authors Paper under double-blind review
ABSTRACT
Contrastive self-supervised learning has delivered impressive results in many audio-visual recognition tasks. However, existing approaches optimize for learn- ing either global representations useful for high-level understanding tasks such as classification, or local representations useful for tasks such as audio-visual source localization and separation. While they produce satisfactory results in their in- tended downstream scenarios, they often fail to generalize to tasks that they were not originally designed for. In this work, we propose versatile self-supervised approach to learn audio-visual representations that can generalize to both the tasks which require global semantic information (e.g., classification) and the tasks that require fine-grained spatio-temporal information (e.g. localization). We achieve this by optimizing
cross-modal contrastive objectives that together encourage our model to learn discriminative global-local visual information given audio sig- nals. To show that our approach learns generalizable video representations, we evaluate it on various downstream scenarios including action/sound classification, lip reading, deepfake detection, and sound source localization.
INTRODUCTION
Self-supervised learning aims learn representations of data that generalize large variety of downstream tasks. Recently, contrastive self- supervisd learning (CSL) has achieved impressive results on several computer vision tasks (Oord et al., 2018: Hjelm et al.) 2018 He al.) 2020] Chen et al., 2020). In CSL, the choice of "views" determines the types of information that the representation captures (Bachman al. 2019), as the framework learns representations that focus on the shared information between views. It has been demonstrated that the optimal choice of views depends critically on the downstream task (Tian et al. 2020). Therefore, existing works mainly focus on finding different views tailored for the intended downstream tasks. For example, when tailoring views for action classification, HHjelm & Bachman (2000) extends DIM (Hjelm et al.) 2018) to the spatio- temporal setting by assuming that global and local information useful for action classification (i.e, global semantics) should be invariant across time and space within a given video. When dealing with multimodal data, several approaches utilize audio-visual correspondence from videos (Morgado et al. 2020). Such a CSL approach is based on an assumption that information needed for audio/video classification should be shared between the two modalities.
Although they achieve impressive results in their intended downstream tasks, existing approaches often fail to generalize tasks that they were not originally designed for. For example, in lip reading (Chung & Zisserman 2016), the desired information is the fine-grained spatio-temporal representation around the mouth. However, we directly apply existing CSL approaches, the shared information across views is that there is face, while the useful information, the lip movements, will be suppressed as they are changing across views from the sample clip.
Motivated by this, we propose a versatile CSL approach to learn representations that can generalize to both scenarios that require global representations (e.g., classification) and scenarios that require local representations (e.g., localization) (see Fig. Our approach, which we call global-local cross-modal (GLCM) contrastive learning, has four key properties that we assume to be important for our learning objective: 1) observations from the same time span of video should reflect the same content regardless of modalities; 2) the same observations captured at different time scales can reflect both global and local information; 3) when learning on local temporal scale, the contrasting

---

Under review as a conference paper at ICLR 2021
views should only share the time- sare-iaryigg information (e.g. only the moving lip) while ignoring globally invariant information; 4) multi-scale (global-local) observations can be trained jointly in a collaborative way that representations learned at either scale can be reused.
We formulate our GLCM objective using two cross-modal contrastive losses computed at multiple tempo- ral scales. Specifically, we gener- ate global and local views of a visual sequence different sampling rates. The audio sequence is used as an an- chor to contrast with global and local visual features, respectively. Consis- tent with the first property (see above) losses are computed at the same tem- poral scale, i.e. 24 + zy and by + < such that, given the same video sequence, the learned 2g and 2 re- flect both global and local informa- tion; the latter portion satisfies the second property. To implement the third property, the local contrastive loss (2'g 2 is computed by con- sidering only the audio-visual features that lie in the same time window as positive pairs; the others are all negative pairs. Finally, we utilize information captured at the global scale (e.g. localizing the source of a sound) to assist efficient learning at the local scale, thus capturing the fourth property.
Global
Representation
Pool
Local
Ours =
Figure I: While many self-supervised approaches optimize for high-level or low-level tasks, we present an approach to learn both global and local representations from video.
We show that GL.M pretraining learns representations with global and fine-grained spatio-temporal information from audio-visual signals. The learned representations perform effectively on a variety of downstream tasks. We evaluate our proposed approach on tasks that needs local spatio-tempont information (i.e lip reading, deep-fake detection, and sound-source localization) and also discrimi- native tasks that needs global information (i.e. action classification and audio-event classification).
RELATED WORK
Contrastive self-supervised learning. CSL has contributed to strong performance on many tasks and in cases produced comparable results to supervised learning (Chen et al. 2020; Caron et al.) 2020 Contrastive learning leverage multiple views of the same data (Hjelm & Bachman [2020] Oord et al., 2018). e.g., multiple perspectives within the same modality (e.g., augmentations of the same image, different frames of a video, etc.) (He et al. 2020 Hjelm & Bachman, 2020 Han perspectives from different modalities (c.g.. depth and RGB images, visual and
et al., 2019a)
textual signals) (Tian et al., 22099; Sun et al. 2019; Miech et al. 2020: Alayrac et al.) 22020). Chen et al. (2020) and Hijelm et al. (2018) show that leveraging local information to perform contrastive learning further improves the performance on image classification tasks. DIM (Hjelm et al. 2018)
has been extended to multi-scale (Bachman et al. 2019) and video data Hjelm & Bachman (020))
However, evaluation is still focused on "discriminative" tasks (image classification and video event classification), while there is little evidence that these models will adapt well to the local information.
Audio-visual representation learning, Several approaches have been proposed to leverage the nat- ural correspondence between audio and visual signals perform CSL (Asano et al. 2020 Korbar et al. 2018: Alwassel et al.) 2019; Morgado et al.) 2020) Patrick et al., 2020 Chung et al. 2019).
Most existing approaches aim to capture high-level semantic information from observations, has been empirically demonstrated that such learned information is very effective for "discrimination tasks" (classification). However, in tasks that needs local information the learned representations may not perform well. Xiao et al.] (2020a) design their approach by utilizing different temporal scales of the audio and visual data, which encourages the model to capture fine- ome-grained temporal information and hence improves the performance. However, the evaluation cation tasks. In contrast with previous work, we demonstrate that our approach effectively learss global-local audio-visua representations by evaluating on variety of downstream tasks.
limited to classifi-
2

---

Under review as a conference paper at ICLR 2021
3 APPROACH
We propose using the audio and vi- sual channels as cross-modal views of video data. As we aim to learn both local and global temporal infor- mation, we utilize the same visual sequence processed at different sam- pling rates to reflect the same ob- servation at different temporal scales. Given that we want each signal to capture complementary views of the same data, we use different encoders to extract the representations from the audio sequence (Ea), subsam- pled visual sequence (Eg) and full sampling rate visual sequence (E.)). The question is, then, how to de- sign contrastive loss to learn repre- sentations from these different views. We achieve this goal by jointly train- ing the model using two contrastive losses: global and local. As shown in Fig. 2 the global loss is computed by contrasting audio signals with sub- sampled visual sequence (31c3.1)); while the local loss is computed by contrasting audio signals with vi- sual sequence at a full sampling rate (See 3.2p. To jointly train the global and Tocal pathways, we pro- attention pool-
Audio
Ex
Global
neg
Temp
Global Contrast
Locol Contrast
neg
f
G 2 0
Ey
Gitzl
F
Low sampling sequence
High sampling rate visual sequence
Video
Figure 2: Our GLCM architecture. For clarity we omit the "channel" dimension and only show the spatial and tempo- ral dimensions. the "global contrast" pathway, the sual features are shown with different shading patterns (e.g.. diagonal-hatch, filled gray) and indicate those that come from different video samples. In the "local contrast" path- way, we use colors to indicate different time windows.
pose a spatially-awe ing mechanism to effectively reuse the information that was captured from the global pathway in the local pathway (Sec 3.3).
3.1 GLOBAL CONTRASTIVE OBJECTIVE
We design the global contrastive objective capture slowly changing information with high audio- visual correlation. We use video sequences captured at low sampling rates, which will inevitably lack local temporal information. Eg encodes an audio sequence into an audio embedding & E R where F' is the frequency, and T' is the sequence length. After temporal global pooling, it becomes za E RIxF Similarly, we perform global temporal pooling on features encoded by the global visual encoder Eg, which produces the global visual embedding 2 E R R x Note that, for the visual features, we perform global pooling only along the temporal dimension while keeping the spatial dimension intact. The reason is that when learning in global temporal space, the model has capacity to capture more spatial information. To compute the global contrastive loss, we consider the audio features 2n and the visual features 2 that come from the same video sample as positive pairs, while features coming from different video samples are negative pairs. In order to encourage the model also capture spatial information, adopt MIL-NCE (Miech al. 2020) to compute the loss. Specifically, we consider all x W spatial grids in 24 as the instances, and therefore, instead of just taking single audio-visual positive pair 2g 24, the new positive pair becomes multiple visual instances 2 [ i.e. in + 2 [[]]...... The loss is then defined as:
Exep =x
Lg -log EgEp exp(2 2 : EveN expl xf T(.3)
(1)
where A is set of negative audio and visual pairs, P is set of spatial grids in =g.
3

---

Under review as a conference paper at ICLR 2021
3.2 LOCAL CONTRASTIVE OBJECTIVE
We design the local contrastive objective to capture fine-grained spatio-temporal information that is sensitive temporal changes while being invariant to different modalities. We thus contrast between local audio features &a and local apatio-temport visual features z Specifically, we consider the temporal local audio and visual features that lie in the same time window to be the positive pairs, ea(t] <> z [t]. where =(t] represents features in the time window t. As shown in Figure the video and audio features shaded the same color refer to those in the same time window. The features in different time windows (e.g. green and orange blocks) are considered as negative pairs even if they are from the same video sample. As such, the shared information between the modalities is principally how the features vary over time. We obtain the local audio features by using the same audio encoder E, but without global temporal pooling. The local visual features g!oeat are obtained by feeding the visual sequence with high sampling rate into the local visual encoder Et, which produces the visual features z E R xXW XWxC We then perform spatial pooling while keeping the temporal scale the same, the visual features become z! E R RTxxxxIXC.
As the audio channel in a video generally has a higher sampling rate than the visual channel, visual feature a single time step will be mapped to multiple audio feature slices, As shown in Figure 2 at time /,, the visual features z [1] (green block) correspond to multiple audio features =n = 1 (green blocks), where M 5 in Figure
Specifically, we use sliding window of size M to map each set of visual features at a given time step to a window of audio feature slices. Then the positive pair considered as a visual feature and the corresponding window of audio feature slices. Once again we use MIL-NCE (Miech et al. 2020) to compute the contrastive loss. The reasoning for applying MIL-NCEE here is different than in the case of the global contrastive loss. In the global contrastive loss, we aim to let the network capture spatial information. While in the local contrastive objective, the goal of using MIL-NCE is to mitigate the missing strict temporal mapping problem. The loss is therefore defined as:
Ex (T =
Li llog( Σx, EQ exp(2 x1) Eapl/
where Q is set of audio feature slices in the same time window z!, and N is a set of negative audio and visual pairs.
3.3 AATTALLY-AWWRR ATTENTION POOLING
As discussed in Sec. 3.1 when computing the global contrastive loss, we focus on the spatial di- mension. Therefore, we can utilize spatial information captured from the global pathway to assist the local contrastive loss. Specifically, we compute the correlation (i.e. dot product between the audio feature and each of the visual features in spatial grid) computed in the global pathway as the attention score; intuitively, this captures the regions of the spatial grid which likely correspond to the source of the sound. For example, in video of someone talking, the lips will have relatively higher score when compared to the background, and in a video of someone playing a guitar, fingers on the guitar will have high scores. We thus use the score an H x W attention map (see Figure 2 We utilize this attention map to perform spatial attention pooling on local visual features at each time step Patten (=t (4). Comparing with regular spatial average pooling, it helps the network give greater weight to parts within a frame with high audio-visual correspondence. This way, the efficiency of the local contrastive loss can be much improved. We empirically demonstrate the effectiveness of spatial attention pooling mechanism in Table I and
4 EXPERIMENTS
Implementation details. We use 3D- (Hara et al.) 2018) for our visual encoders (E. and
and ID-ReNNet18 for our audio encoder, in both cases using Batch Normalization (BN) Ioffe &
Szegedy 2015). All models are trained end-to-end with the ADAM optimizer (Kingma & Ba 2014) with an initill learning rate y = 10-3 after a warmuup period of 500 iterations. We use 16 IVDDIA Tesla P100 GPUs with batch size of 32 for experiments. For pretraining, we preprocess video frames by sampling at 10 FPS and applying random cropping, horizontal flipping, gray-scaling, and temporal jittering. We resize video frames three-channel images of 112 x 1 112; we set the clip
4

---

Under review as a conference paper at ICLR 2021
length to 32 frames for the local visual pathway, and sampling rate for the global path way (8 frames). For the audio channel, we extract mel-speects from the raw waveform using the LibROSA library and get a 80 x I matrix with 80 frequency bands; T proportional to the length of an audio clip. We then segment the mel-spectrogram according to the corresponding video clips to ensure temporal synchrony. We treat the spelsspectoggamss as 80-channel 1D signal. To compute the global MIL-NCE loss, we use features at a 16 x 16 spatial resolution. To compute the local contrastive loss, we adopt a temporal window of size three without overlap.
Datasets. Many downstream tasks of interest involve human faces (e.g., deepfake detection), speech (e.g.. le. reading) and activity (e.g.. audio and video classification). Therefore, we use a combination of Kinetics-706 (Carreira et al.) 2019) and AVSpeech (Ephrat et al. 2018 for pretraining. Specif- ically, we randomly select 120K video samples from each datasets, which gives us a dataset, we term as K-AV, in total 240K samples. For comparison with the other wate-of-the-an approaches, we pretrain our model on K-AV dataset which is the same scale as the Kinetics-700 dataset. For the ablation study. we pretrain term as K-AV-15K. As for downstream tasks, we evaluate our models on action recognition using UCF101 (Soomro al. 2012) and HMDB51 (Kuehne al., 2011), and on sound classification using ESC50 (Piczak 2015b). For lip reading, we evaluate our model on both LRW (Chung & Zisserman 2016) and RRS2 Chung et al. (2017). For deepfake detection, evaluate model a subset of DFDC (Dolhansky et al. 2019
model on subset of 15K samples from the K-AV dataset,
4.1 DOWNSTREAM SCENARIOS
Training
Model
Backbone
Dataset
LRS
LRW
LRW (Chung & Zisserman, 2016
VGG-M
LRW
N/A
61.10
Res. (Stafylakis & Tzimiropoulos
2017
ResNet34
LRW
N/A
83.00
TwoStream (Weng Kitani
2019
I3D
N/A
84.07
DFTN (Xiao et al. 20200)
ResNet18
LRW500, 1000
N/A
84.1
Superv.
TM-sep2seg (Alouras et al.
2018
ResNet
LRS2-BBC
49.8
N/A
TM-CTC (Afouras
2018)
ResNet
LRS2-BBC
65
N/A
STF (Zhang ct al.
2019
ResNet18
LRW, LRS2,3
51.7
83.7
WAS (Chung cf al.
2017
Conv.
70.4
76.2
Perfect Match (Chung ct al.
2019
TC-5
LRW
71.6
N/A
MoCo (He al.
20200)
ResNet18
K-AV-15K
71.5
61.2
CPC (Ord et al.
2018
ResNet18
K-AV-15K
66.7
65.3
SSL
DPC (Oord et al.
2018
ResNet18
K-AV-15K
65.1
67.5
InfoMax (Hjelm & Bachman
2020
ResNet18
K-AV-15K
53.2
70.7
AVSlowFast (X1a0 al. 2020a)
ResNet18
K-AV-15K
56.1
75.8
ResNet18
LRS2,
46.7
86.9
Ours
ResNet18
K-AV-15K
47.8
83.7
ResNet18
K-AV-240K
45.1
89,2
Table 1: Lip reading comparison with SOTA. LRW (top-1 accuracy, the higher the better) LRS (WER, the lower the better). The supervised approaches (Superv.) directly train the models on the dataset listed in the table. The self-supervised approaches (SSL) and Ours pretrain the models on the datasets listed in the table, and then finetune the model on LRW and LRS2 with the same protocol. Underline refers the best results from of the supervised approaches. Blue color highlights the comparison of ours with the self-supervised approaches under the same setting.
Lip Reading, Visually recognizing a speaker's utterance is useful and challenging task. Lip move- ments for different letters can be visually similar to each other (c.g.. and p. and t). This requires the learned visual representation to contain fine-grained spatio-temporal information, rather than global semanties. In evaluating our pretrained model on the lip reading task, we focus on investigat- ing whether our approach successfully learns fine-grained spatio-temporal visual information. For fair comparison with state-of the-art art (SOTA) approaches, we use the same data processing protocol as Zhang et al. (2019). For LRW and LRS2, we detect 68 facial landmarks in each video frame using dlib (Castelli Pagano 2002). We use the outer eye and nose tip landmarks to align the detected face in each frame using an affine transform. Finally, image of size 112x 1 112 is cropped from the aligned face with the lip landmarks at the center. The cropping is such that the lips occupy 1 of the image width. During finetuning, we apply random horizontal flipping. We concatenate the global
5

---

Under review as a conference paper at ICLR 2021
Training
Model
Visual
Audio
Backbone
AUC
Capsule Nguuyen
(b)
VGG-19
53.3
Multi-task [gguyen ct
2019a)
Y-shape
53.6
HeadPose Yang cf al
2019)
N/A
55.9
Two-stream (7hou et
201
Inception3
61.4
VA-MLP (Mätern el
2019)
N/A
61.9
VA-LogReg (Mater
2019
N/A
66.2
Superv.
Meso4 (Afchar et al. [0018
Inception4
75.3
Xception-c40) (Rosster (Rosssi ct al.
2019
XCeption
69.7
Xception-c23 (Rossler
2019
XCeption
72.2
FWA (Li & Lyu 2018)
N/A
72.7
DSP-FWA
Lyu
2018
N/A
75.5
Siamese MMittal el
N/A
84.4
MDS (Chugh et al.
22200)
ResNet18
91.5
MoCo (He
2020)
ResNetT8
60.2
CPC (Oord al.
ResNet18
67.9
SSL
DPC Ham ct al.
2019h
ResNet18
71.2
InfoMax (Fjell & Bachman
2020
ResNet18
85.3
AVSlowFast (Xao
2020a
ResNet18
80.9
(pretrain DFDC)
ResNer18
95.5
Multimodal (DFDC)
ResNet18
95.6
Ours
Ours (K-AV-15K)
ResNet18 90.1
Ours (K-AV-240K)
ResNet18 96.7
Table 2: Comparison with SOTA approaches on deepfake detection. For training on DFDC, we the same training and testing list Chugh al. 2020). Mittal al., the number of randomly sampled training data (180K). All the numbers of the other SOTA approaches are collected from Chugh al. (2020). All the self-supervised approaches (SSL) pretrained on K-AV-15K, and finetuned and tested on the same training and testing set of DFDC.
features and local features produced by our pretrained Ey and E respectively. Both encoders are finetuned with the whole model.
We compare our approach with SOTA supervised lip reading methods. For LRW, evaluate 500-way word classification task and report top-1 accuracy. For LRS, we report the word етrог rate (WER). Table shows that our approach outperforms SOTA approaches on both LRS and LRW by large margin, i.e. 4.7% WER reduction on LRS and 5. 5.1% % accuracy improvements on I.R.. These results show that our proposed approach can capture the fine-grained spatio-temporal information necessary for lip reading. We also compare our model with other self-supervised approaches with the same backbone and using the same pretraining dataset. Our approach outperforms these by a large margin and demonstrates the effectiveness of our proposed approach.
Deepfake Detection. Our hypothesis here that deepfakes tend to characterized by audio-visual inconsistencies such misalignment between lip motions and audio, unnatural facial and lip appearance/movveennt or asymmetry between facial regions such as the left and right eyes. Such artifacts could be detected through local spatio-temporal features. We and finetune it on the DFDC dataset and evaluate performance using video-wise Area Under the Curve (AUC). We follow the same data preprocessing protocol as SOTA approaches for this task. We perform face detection to crop the face region in each video frame. We concatenate the global and local visual features that are produced by our pretrained global and local visual encoders, and finetune the pretrained encoders with the whole model. The results are shown in Table For fair comparison, we use the same training and compared approaches, Chugh et al. (2020) and Mittal et al.) use both visual and audio sequences, while the other approaches use only the visual sequences for detection. As we can see, when using only visual signal, our approach outperforms all previous SOTA approaches (AUC=96.7), We also compare our model with the other SOTA self-supervised approaches (shown blue color). Again, model outperform the best benchmark by large margin (90.1 vs. 85.3).
pretrained model
video as Chugh al. (2020). Among all the
Action and Sound Classification. To evaluate performance in learning discriminative global spatio- temporal representations, we use pretrained model for action and sound classification. For tion classification, we finetune both pretrained global and local visual encoders by concatenating the global and local representations. For audio classification, finetune pretrained audio encoder

---

Under review as a conference paper at ICLR 2021
Eg with the audio classification model. To evaluate action and audio classification, compare both our models that were pretrained on Kinetics- 700 and K-AV-240K with SOTA approaches pre- trained dataset of the same scale (Kinetics). We find that on all three benchmarks, i.e. UCF101, HMDB51 and ESC50, our approach achieves a new SOTA (91. .2% on UCF101, 61.9% on HMDB51 and 80.1 1 % ESC50) see Table
Method
Pretrain Datasct
UCF101
HMDB51
ESC50
Scratch
46.5
17.1
N/A
Scratch Random Forest (Piczak
2015b
N/A
N/A
44.3
Sperv.
Scratch ConvNet [Piczak] 2015a
N/A
N/A
64.5
Scratch ConvRBM (Sailor ct al.
2017)
N/A
N/A
Supervised
ImageNet
82.8
46.7
N/A
RotNet3D Jing & Tian
2018)
K400 240K
62.9
33.7
N/A
AVTS (Korbar
2018
K400 240K
85.8
56.9
76.7
MotionPred Wang ct 2019
K400 240K
61.2
33.4
N/A
ST-Puzzle (Kim el al.
2019
K400 240K
65.8
33.7
N/A
ClipOrder [Xu
2019
K400 240K
72.4
30.9
N/A
SSL
CBT (un el al.
2019)
K400 240K
79.5
44.6
N/A
DPC Han ct
2019a)
K400 240K
75.7
35.7
N/A
XDC Alwssel ci al.
2019
K400 240K
84.2
47.1
78.0
SeLa Vi
Asano et al
2020
K400 240K
83.1
47.1
N/A
AVID MMrgado
2020
K400 240K
87.5
60.8
79.1
GDT
Patrick
2020)
K400
89.3
60.0
N/A
K700 240K
91.2
61.9
79.7
Ours
K-AV 240K
90.1
61.3
80.1
Table 3: Comparison of SOTA approaches on action classification and sound classification. We specify pretraining dataset and the number of samples used they are reported in the original papers (N/A: not available). We highlight the best results and the second best results.
4.2 ABLATION AND ANALYSIS
The importance of global-local contrast for local information needed tasks. Here, we want to investigate specifically how the pretraining pretext task impacts local information needed for down- stream tasks (i.c.. lip reading and deepfake detection) and compare our task with those used in other work. We pretrain our model and the other state-of-the-ae self ceff-supervise pretraining approaches with the same backbone (3DResNet-1l and pretrain dataset (K-AV-15K). After pretraining, we fine- tune each model on the downstream benchmarks follow the same protocol. The results are listed in Table As we can see that, when we only vary the pretext task during pretraining, our models out- perform all the other SSL-based approaches by large margin, which demonstrate the effectiveness of our proposed approach. We also find that InfoMax and AVSlowFast performs better than the oth- ers (MoCo, DPC and CPC). We believe this is because InfoMax draws more attention to the spatial local information and AlSlowFast is capable of learning more fine-grained temporal information, which are critical for the tasks of lip reading and deepfake detection. MoCo, which is successful for visual classification tasks fails both lip reading and deepfake detection. This supports our argument that naīvely using an SSL approach may not achieve good performance for a large variety of different tasks.
The roles of global and local information. To demonstrate the importance of jointly learning global-local representations during pretraining, we evaluate baseline model that was pretrained without the local contrastive objective (Ours w/o local cont.). As we can see from Table when compared with the model which was pretrained using our full objective (Ours), the performance sig- nificantly drops on all the benchmarks. Optimizing only for global representations during pretraining generalizes poorly to the tasks that require local information. Note that, for a fair comparison, we only use the global features (Global Feat.) for each downstream tasks. Furthermore, we test whether using the local, global, or local and global features after pretraining yields better performance. We that, the best performance is achieved by utilizing both the global and local features and this true for all the benchmarks. For comparison, we report the results achieved by the other SOTA approaches when using both the global and local features.

---

Under review as a conference paper at ICLR 2021
Model
Pretext task
LRS
LRW
DFDC
MoCo (He
2020)
Moment. Cont.
71.5
61.2
60.2
CPC OOrdd ct al.
2018
Pred. Cont.
66.7
65.3
67.9
DPC
Han ct al.
2019h
Dense Pred. Cont.
65.1
67.5
71.2
InfoMax (Hjelm & Hachman
2020)
Glo-loc NCE.
53.2
70.7
85.3
AVTS
(Korbar
2013
AVS
72.1
64.9
63.1
AVSlowFast (Xtao al.
2020a
AVS +
56.1
75.8
80.9
Ours
Glo-loe MIL
47.8
83.7
90.1
Table 4: Comparison with existing self sulerviie pretraining approaches lip reading and deep- fake detection. All the results are computed based on the implementation by us. We pretrain each SOTA model with the same pretext task proposed in their paper. "Moment. Cont.": momentum trast; "Pred. Cont.": predictive contrast; "Dense. Pred. Cont.": dense predictive contrast; "Glo-loc NCE": global-local NCE; "AVS": audio-visual syne: "Rot": rotation detection
Model
Feature
LRS
LRW
DFDC
UCF101
HMDB51
w/o loc.
Glob.
70.9
65.3
67.9
82.3
57.1
Ours
Glob.
47.6 (1 23.3)
86,8
(1 21.5)
92.6
24.7)
89.2 (1 6.9)
59.9 (1 2.8)
Ours
Loc.
46.5
88.9
95.9
88.5
58.3
Ours
Glob
45.1
89.2
96.7
61.3
Table 5: The roles of global and local information on different benchmarks. "Ours": our model that pretrained with the whole objective. "Ours w/o loc.": our model that pretrained without the local contrastive objective. "Glob.": features extracted from the global encoder. "Loc.": features extracted from the local encoder. Underline refers to the best results from the other approaches.
Local Contrast
LRS
LRW
DFDC
UCF101
HMDB51
w/o MIL
40.4
79.2
88.9
87,8
56.3
Ours
47.8(1 7.4)
83.7(1 4.5)
90.1(t 1.2)
88.1(1 0.3)
56.8(1 0.5)
Table 6: Comparison of models pretrained with and without using MIL for local contrast.
The role of MIL-NCE in local contrastive objective. When performing the local contrastive objective, we adopt the MIL-NCE as our loss function. Miech et al. (2020) also employed the MIL-NCE as the loss function mitigate the misalignment in narrated videos. Different from their motivation, our goal here is to encourage fine- grained temporal alignment of the audio with video features. To validate its effectiveness, we evaluate an alternative without MIL-NCE as the local contrastive loss function. Specifically, we adopt an average temporal pooling on each window of the audio features and use the vanilla contrastive loss over the synchronized audio and visual features. The results shown in Table As we can see, when perform the local contrast without the MIL-NCE objective, the performance on lip reading and deepfake detection drops considerably. While for activity classification, both loss function achieves comparable results.
The role of attention. Our pipeline allows us to leverage the global contrastive objective to cap- ture local spatial information and use it as attention map assist local representation learning. Intuitively, our attention maps measure the amount of audio-visual correlation; such attention maps can highlight discriminative face regions useful for lip reading. We demonstrate the quality of our attention maps by replacing lip bounding boxes typically used in lip reading with our attention map. Specifically, instead of extracting features from the cropped lip/face region, we extract features from the entire frame (no lip/face cropping) and use our attention map to pool the features spatially. Note that the purpose of this experiment just to obtain attention maps and discard it for word classification/deepfaa ("Ours Attention")) show that it achieves results comparable to our best setting. On LRW and DFDC, it even outperforms SOTA approaches without relying on lip/face region detectors. It indicates that using global and local information in a collaborative way can yield good performance. In the lip reading task, local spatial information makes the local-contrast pathway pay more attention to lip movement, and thus achieves comparable effects of using lip region detector. We also evaluate our model which finetuned directly on full frames. As see, when discarding the localized region
evaluate the quality of attention maps; we use audio signal
detection. The results
8

---

Under review as a conference paper at ICLR 2021
Benchmark
SOTA
Our Best
Ours Full frame
Ours Attention
LRS
49.8 TM -sse2eee
45.1
71.9
51.2 No lip crop
LRW
84.1 DFTN
89.2
62.3
85.1 No lip crop
85,3 InfoMax V
96.7 V
68.1 V
DFDC
95.9 No face crop
91.5 MDS V+A
97.1 V+A
Table 7: Evaluation on the role attention mechanism in our approach. "V" uses only visual sequence and "V+A" uses both visual and audio sequence for finetuning on downstream tasks.
Figure 3: Visualization attention maps showing the audio-visual correlations in learned representa- tions.
achieved either by detectors or attention mechanism, the performance significantly drops on all three benchmarks. It further validates the critical role of the attention mechanism in our approach.
Interpretation of the learned representation. To investigate how well local spatial information is captured through the global audio-visual contrast, we visualize the attention maps that induced by the pretrained audio and global video encoders. Such visualization can also be considered as performing sound source localization, i.e. locate objects that making sound. To achieve this goal, the network should capture the audio-visual correlation in attention map obtained by our pretrained model to visualize the sounding source in each frame. To investigate further, we use the Kinetics-sound (Carreira et al., 2019) dataset as videos in the dataset generally have a high-level of audio-visual correspondence. Specifically, we add another softmax layer on the obtained attention map, and then do bilinear interpolation of the 16 x 16 attention map back the original image size, i.e. 192 X 192.
grid. We thus use the
Fig. shows that our learned attention maps tend to localize sounding sources in videas accurately. Especially, when visual content highly related the corresponding audio signal, our model per- forms particularly well. For example, the first row (frames from "playing instruments" videos) shows that our model can precisely localize the sounding region. For the other activities, like "baby talking." "playing basketball," "running," our model highlights regions with humans. We find that the attention map incorrectly highlights regions on samples that have ambiguous audio-visual rela- tion. We show failure cases in the last two frames of the third row, As can see, there is no visual content that clearly relates with the audio signal, and thus the model fails to find sounding sources.
5 CONCLUSION
We have presented a contrastive self-supervised approach to learning global and local audio-visual representations. Using audio and low-sampled and high-sampled video sequences as separate "views" of the data, we find that the learned representations can generalize well to tasks that in- volve global semantic understanding and fine-grain fine-grained spatio-temporal understanding. We perform experiments on lip reading, deepfake detection, sound source localization, and action/sound classi- fication tasks and in each case achieve strong results.

---

Under review as a conference paper at ICLR 2021
REFERENCES
Darius Afchar, Vincent Nozick, Junichi Yamagishi, and Isao Echizen. Mesonet: compact facial video forgery detection network. In 2018 IEEE International Workshop on Information Forensics and Security (WIFS), pp. 1-7. IEEE, 2018.
T. Afouras, J. S. Chung, A. Senior, O. Vinyals, and A. Zisserman. Deep audio-visual speech recog- nition. IEEE Transactions on Pattern Analysis and Machine Intelligence, pp. 1-1, 2018.
Jean-Bas Maan-Baptstte Alayrac, Adrià Recasens, Rosalia Schneider, Relja Arandjelović, Jason Ramapuram, Jeffrey De Fauw, Lucas Smaira, Sander Dieleman, and Andrew Zisserman. Supf-supe mul- timodal versatile networks. Advances in Neural Information Processing Systems, 33, 2020.
Humam Alwassel, Dhruv Mahajan, Lorenzo Torresani, Bernard Ghanem, and Du Tran. Self- supervised learning by cross-modal audio-video clustering. arXiv preprint 1eXiv:1911.122 2019.
Yuki M Asano, Mandela Patrick, Christian Rupprecht, and Andrea Vedaldi. Labelling unlabelled videos from scratch with multi-modal sufe-uupevvsioo. arXiv preprint orXiv:2006.1366) 2020.
Philip Bachman, R Devon Hjelm, and William Buchwalter. Learning representations by maximizing mutual information across views. In Advances in Neural Information Processing Systems, pp- 15535-15555, 2019.
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. arXiv preprint arXiv:2006.09882, 2020.
Joao Carreira, Eric Noland, Chloe Hillier, and Andrew Zisserman. A short note the kinetics-700 human action dataset. arXiv preprint arxiv:/907 1907.06987, 2019.
Donatella Castelli and Pasquale Pagano. Opendlib: A digital library service system. In Maristella Agosti and Costantino Thanos (eds.). Research and Advanced Technology for Digital Libraries, pp. 292-308, Berlin, Heidelberg. 2002. Springer Berlin Heidelberg.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. arXiv preprint vrXiv:2002.05000 2020.
Komal Chugh, Parul Gupta, Abhinav Dhall, and Ramanathan Subramanian. Not made for each other-audio-visual dissonance-based deepfake detection and localization. arXiv preprint arXiv:2005.14405, 2020.
S. Chung and A. Zisserman. Lip reading in the wild. Asian Conference on Computer Vision, 2016.
Joon Son Chung, Andrew Senior, Oriol Vinyals, and Andrew Zisserman. Lip reading sentences in the wild. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3454-3453. IEEE, 2017.
Soo-Whan Chung, Joon Son Chung, and Hong-Goo Kang. Perfect match: Improved cross-modal embeddings for mudio-visual synchronisation. In ICASSP 2019-2019 IEEE International Confer- ence on Acoustics, Speech and Signal Processing (ICASSP), pp. 3965-396999 3969, IEEE, 2019.
Brian Dolhansky, Russ Howes, Ben Pflaum, Nicole Baram, and Cristian Canton Ferrer. The deep- fake detection challenge (dfde) preview dataset. arXiv preprint arXiv:1910.00 191000854 2019.
A. Ephrat, I. Mosseri, O. Lang. T. Dekel, K Wilson, A. Hassidim, W. T. Freeman, and M. Rubinstein. Looking to listen at the cocktail party: A weeaker-independee audio-visual model for speech separation. arXiv preprint arXiv: 1804::880..03111 2018.
Tengda Han, Weidi Xie, and Andrew Zisserman. Video representation learning by dense predictive coding. In ICCV, 2019a.
10

---

Under review as a conference paper at ICLR 2021
Tengda Han, Weidi Xie, and Andrew Zisserman. Video representation learning by dense predictive coding. Proceedings of the IEEE International Conference on Computer Vision Workshops, pp. 0-0, 2019b.
Kensho Hara, Hirokatsu Kataoka, and Yutaka Satoh. Can spatiotemporal 3d cnns retrace the history of 2d cnns and imagenet? In CVPR, 2018.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9729-97239 2020.
R. Devon Hjelm and Philip Bachman. Representation learning with video deep infomax. arXiv preprint 2xXx:20077.13777 2020.
Devon Hjelm, Alex Fedorov, Samuel Lavoie-Marchildon, Karan Grewal, Phil Bachman, Adam Trischler, and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization. arXiv preprint arXiv:1808.0660 2018.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint 1rXiv:1502.03167, 2015.
Longlong Jing and Yingli Tian. Self-supervised spatoommppra feature learning by video geometric transformations. arXiv preprint arXiv:1811.1133: 2018.
Dahun Kim, Donghyeon Cho, and So Kweon. Self-supervised video representation learning with space-time cubic puzzles. In AAAI, 2019.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization, arXiv preprint 1rXiv:1412.6990 2014.
Bruno Korbar, Du Tran, and Lorenzo Torresani. Cooperative learning of audio and video models from self-supervised synchronization. In Advances in Neural Information Processing Systems, 2018.
Hildegard Kuchne, Hueihan Jhuang, Estíbaliz Garrote, Tomaso Poggio, and Thomas Serre. Hmdb: large video database for human motion recognition. In ICCV, 2011.
Yuezun Li and Siwei Lyu. Exposing deepfake videos by detecting face warping artifacts. arXiv preprint arXiv: I. 00066,, 2018.
Matern, C. Riess, and M. Stamminger. Exploiting visual artifacts expose deepfakes and face manipulations. In 2019 IEEE Winter Applications of Computer Vision Workshops (WACVW), pp- 83-92, 2019.
A. Miech, J. Alayrac, L. Smaira, Laptev, Sivic, and A. Zisserman. End-to-end learning of visual representations from uncurated instructional videos. In 2020 IEEE/CVF Conference Computer Vision and Pattern Recognition ( (CVPR), pp. 9876-9886, Los Alamitos, CA, USA, jun 2020. IEEE Computer Society. doi: 10.1 /009///YPR4220..2220000
URL https://doi.
leeeconputerrociiety.orgg/ 11t.:1..111////22221200
2020.
Trisha Mittal, Uttaran Bhattacharya, Rohan Chandra, Aniket Bera, and Dinesh Manocha. Emotions
don't lie: An audio-visual deepfake detection method using affective cues.
Pedro Morgado, Nuno Vasconcelos. and Ishan Misra. audio-visual instance discrimination with cross-modal agreement. arXiv preprint arXiv:2004. /2943, 2020.
Huy Н Nguyen, Fuming Fang, Junichi Yamagishi, and Isao Echizen. Multi-task learning for de-
tecting and segmenting manipulated facial images and videos. arXiv preprint arXiv: arXiv:1906.0689. 2019a.
Huy H. Nguyen, Junichi Yamagishi, and Isao Echizen. Capsule- forensics: Using capsule networks to detect forged images and videos. In ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 2307-2311. IEFF, 2019b.

---

Under review as a conference paper at ICLR 2021
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding. arXiv preprint arXiv: 2018.
Mandela Patrick, Yuki M Asano, Ruth Fong, João F Henriques, Geoffrey Zweig, and Andrea Vedaldi. Multi-modal self-supervision from generalized data transformations. arXiv preprint arXiv:2003.04298, 2020.
Karol J Piezak. Environmental sound classification with convolutional neural networks. In Interna- tional Workshop on Machine Learning for Signal Processing (MLSP). 2015a.
Karol Piczak. ESC: Dataset for environmental sound classification. In Proceedings of the 23rd
ACM international conference Multimedia, 2015b.
Andreas Rossler, Davide Cozzolino, Luisa Verdoliva, Christian Riess, Justus Thies, and Matthias Nießsner. Faceforensics+*: Learning to detect manipulated facial images. In Proceedings of the
IEEE International Conference on Computer Vision, pp. 1-11, 2019.
Hardik Sailor, Dharmesh Agrawal, and Hemant Patil. Unsupervised filterbank learning using convolutional restricted boltzmann machine for environmental sound classification. In IN- TERSPEECH, 2017.
Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. Ucf101: A dataset 101 human actions classes from videos in the wild. arXiv preprint arXi::1212.04000 2012.
Themos Stafylakis and Georgios Tzimiropoulos. Combining residual networks with Istms for lipreading, arXiv preprint arXiv:1703.04105 2017.
Chen Sun, Fabien Baradel, Kevin Murphy, and Cordelia Schmid. Contrastive bidirectional trans- former for temporal representation learning. arXiv preprint 0rXiv:1906.05744 2019.
Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding. arXiv preprint vrXiv:/906.000 2019.
Yonglong Tian, Chen Sun, Ben Poole, Dilip Krishnan, Cordelia Schmid, and Phillip Isola. What makes for good views for contrastive learning. arXiv preprint 1024: 2020.
Jiangliu Wang, Jianbo Jiao, Linchao Bao, Shengfeng He, Yunhui Liu, and Wei Liu. Self-supervised spatio-temporal representation learning for videos by predicting motion and appearance statistics. In CVPR, 2019.
Xinshuo Weng and Kris Kitani. Learning spatio-temporal features with two-stream deep 3d enns for lipreading. arXiv preprint arXiv: 0000909555500. 2019.
Fanyi Xiao, Yong Jae Lee, Kristen Grauman, Jitendra Malik, and Christoph Feichtenhofer. Audio- visual slowfast networks for video recognition. arXiv preprint arXiv:2001.08740, 2020a.
Jingyun Xiao, Shuang Yang, Yuanhang Zhang, Shiguang Shan, and Xilin Chen. Deformation flow based two-stream network for lip reading, arXiv preprint arXiv:2003.051: 05709 2020b.
Dejing Xu, Jun Xiao, Zhou Zhao, Jian Shao, Di Xie, and Yueting Zhuang. Self-supervised spa- tiotemporal learning via video clip order prediction. In CVPR, 2019.
Xin Yang, Yuezun Li, and Siwei Lyu. Exposing deep fakes using inconsistent head poses. In ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 8261-8265. IEEE, 2019.
Xingxuan Zhang, Feng Cheng, and Shilin Wang. Spatio-temporal fusion based convolutional se- quence learning for lip reading. Proceedings of the IEEE/CVE International Conference on Computer Vision (ICCV), October 2019.
Peng Zhou, Xintong Han, Vlad I Morariu, and Larry S Davis. Two-stream neural networks for tampered face detection. In 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 1831-1839. IEEE, 2017.
12


Reviews:
[{"review_id": "zGR1IKfWVdl", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "**Summary**\nThis paper presents a new method of using self-supervised contrastive learning to learn global-local audio-visual representations, resulting in strong results on a variety of tasks. The proposed method leverages simultaneously captured information in audio and visual modalities at different time scales to generate different \"views\" of the data for contrastive learning.\n\n**Strengths**\n+ Learning general audio-visual representations by jointly optimizing both global-scale and local-scale objectives is well-motivated, and to the best of my knowledge has not been done before. The presented method of generating matching and contrasting \"views\" is conceptually simple, yet seemingly very effective.\n+ Strong empirical results on a variety of downstream tasks indicate that the learned representations do indeed generalize well to both global and local downstream tasks.\n+ The paper reads well, and method is easy to understand.\n\n**Weaknesses**\n- A better explanation about the motivation behind using the non-standard MIL-NCE training objective would be helpful.\n- I'm not convinced of the need for MIL-NCE for the local contrastive objective. The original motivation behind the use of this objective was misalignment in narrated videos, which is a notorious problem in visual-textual mapping. Here, I'm not sure that there exists a \"strict temporal mapping problem\", as the the modalities are captured simultaneously, and should therefore be temporally synchronized. Perhaps performing a fixed/learned pooling of multiple audio features which correspond to the time span of a single visual feature, and optimizing using a more conventional contrastive loss would give better results.\n- The role of spatial attention is evaluated in Sec. 4.1 and 4.2 by using the attention to extract an ROI, instead of standard lip/face cropping. Adding the results of using the full face frame as a baseline to the comparisons would give a more complete picture of the power of the spatial attention.\n- The experiment section is lacking explanations of how to use the network/representations described in Sec. 3 to solve the downstream tasks. For example, how do you get from learned representations to LRW word classification, LRS WER, etc.?\n\nAnother related work which is worth referencing in Sec. 2 is \"Audiovisual SlowFast Networks for Video Recognition\" by Xiao et al, which is based on the original SlowFast visual-only work, and which also learns from a given video at two different sample rates, along with audio.\n\nThis is a good paper, in which a novel method for solving an important problem is proposed, and the resulting performance on tasks of interest is strong. I would like to see it accepted to ICLR, provided the above concerns are addressed.", "extra_scores": {}, "extra_text": {}}, {"review_id": "j4B0Jr6-Lm", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a self-supervised approach for audio-visual representation learning which consists of two losses, a global contrastive loss and a local contrastive loss. Each loss operates on differently sampled visual features. \n\nStrengths: \n\n- The method and experiments are described clearly.  \n- The idea to combine both local and global information in a self-supervised objective while pre-training a model is a sensible one. \n- The performance on some downstream evaluations - eg.  lip reading as well as action classification on HMDB exceed SOTA comfortably.  \n\nWeaknesses \n- The main motivation for the paper appears to be a self-supervised approach that can capture both global and local information, and hence the “views” of the data do not need to be carefully selected for every downstream task. However, it is worthwhile noting that the pre-training dataset is carefully chosen for each task: LRS for lip reading, DFDC for deep-fake detection and Kinetics for classification tasks. These pre-training datasets themselves capture important invariances in the way the datasets have been constructed. \n- Training both contrastive losses jointly doesn’t seem to help performance -  the local contrastive loss is used for lip reading (Table 1: adding the global loss only shows 0.5% improvement for LRS and actually harms for LRW) while the global loss is sufficient for classification (adding in the local loss shows 0.2% in Table 3). Since the authors have already carefully selected the pre-training dataset for each downstream task, isn’t it also possible to just select which loss is needed based on the downstream task? I believe it would be really great if it were possible to train a single model on a large dataset without knowing the downstream task, and apply the same model to both a downstream task that requires “local information” and another downstream task that requires “global information”\n\nQuestions: \n- Table 2 says 95.6% for multimodal but the text says performance drops to 94.6%, is this a typo? What is the performance on the DeepFakes downstream task with only the local or the global contrastive loss? \n- For lip reading, why not pre-train on larger, “in the wild” datasets such as VoxCeleb2? LRS has been carefully curated as a lip reading dataset already. Then the same model could be used for a number of tasks such as deepfake detection, lip reading, speaker verification etc.\n\n\nSuggestion for future work: (not taken into account for the review)\n- Sec 4.3, is there a way to quantitatively assess sound source localisation performance, eg using datasets such as SoundNet-Flickr or AudioSet-instrument, as shown here: https://arxiv.org/abs/2007.06355\n- It would be good to evaluate the audio representations on VGG-Sound (http://www.robots.ox.ac.uk/~vgg/data/vggsound/)  as well. Another interesting direction would be to compare to this work in the speech domain (https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=9054057), since it also trains with two self-supervised contrastive losses, one trying to capture global representations for identity, and one trying to capture local representations (in time) for content. The downstream task here would be speaker verification. \n- Table 1, some works are missing such as https://arxiv.org/pdf/1809.08001.pdf \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "URIYF9QPf7c", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: This paper proposes an audio-visual self-supervised learning approach based on two cross-modal contrastive loss that learns audio-visual representations that can generalize to both the tasks which require global semantic information and localized spatio-temporal information. Extensive experiments on 4 task demonstrate the usefulness of the learned representation.\n\nStrengths:\n+ The paper is nicely written and well motivated. Existing works tend to learn either global representations or local representations, while this work aims at learning versatile representations that generalize well to both scenarios.\n+ Good observation and analysis to construct triplets from different spatial and temporal span to capture either global information or local information. A tailored global-local contrastive learning network is desgined to realize the proposed idea.\n+ Extensive experiments are performed on 4 task across a number of datasets to demonstrate the generality of the learned representations.\n\nWeakness:\n- Among the tasks evaluated on, only Table 3 compares to prior self-supervised learning approaches. This demontrates the proposed method learns better global representations, but no results are shown to prove that the prior ssl methods fail to learn local representations. In Table 1 and Table 2, the method only compares to prior state-of-the-art methods of the corresponding task. It would be more convincing to compare to the representations of some of the methods in Table 3.\n- For sound source localization, only some qualitative results are shown. Why not showing the quantitative results (IOU)? Only a figure of selected qualitative results is not convincing to this reviewer.\n- For Table 1 and Table 2,  the prior methods all use different network settings from the proposed network. How to tell whether the  gain is from a better network architecture or the proposed global-local representation learning method? Without comparing in an apples-to-apples manner with other self-supervised contrastive learning methods (e.g., GDT, AVID, etc.),  it would be unconvincing that the proposed global-local audio-visual contrastive learning method indeed learns better representations that have the suggested properties.\n- The related work is somewhat short and unorganized. It would be much better to re-organize the related work section (contrastive self-supervised representation learning, audio-visual learing,  etc. ) and highlight the differences to each group.\n\nJustification of rating: \nThe paper proposes a decent idea to learin global-local reprentations and evaluate on various tasks/datasets. However, some necessary comparisons are missing in order to demonstrate the effectivenss of the proposed method. This reviewer is happy to raise the score if the rebuttal can clarify the concerns.\n\nPost-rebuttal:\nThanks for the clarifications in the rebuttal. It addresses some of my concerns. However, I am still concerned about the unfair comparisons of baselines using different settings, and the newly added ssl baselines all outform the proposed method on LRS by a large margin. Therefore, I would like to keep my original rating.", "extra_scores": {}, "extra_text": {}}, {"review_id": "7joq93k5CD4", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: This paper presents a new contrastive audio-visual learning method. Like previous work, they use self-supervision to learn a video feature set by training a network to associate audio and visual \"views\" taken from the same video. Their main contribution is to jointly learn from both \"local\" and \"global\" information. They simultaneously optimize two contrastive objectives. First, there is a global objective, which computes a feature set using a low framerate video, pools over time, and obtains negatives from other different videos. Second, there is a local contrastive objective that uses a higher framerate video, pools over space but not time, and gets negatives from other timesteps of the video. They optimize both losses jointly using a spatially-aware pooling method that provides information from the global pathway to the local pathway. They compute attention by taking dot products between the visual and audio features, and using this attention to pool local visual features (instead of a global pooling).\n\nPros:\n- The paper evaluates the model on a diverse set of downstream tasks: lip reading, deepfake detection, action recognition, and sound classification. \n- The model performs well on the downstream tasks (competitive or better than other self-supervised pretraining methods).\n- The idea of designing new network architectures that can take advantage of local and global information is an interesting direction.\n\nCons:\n- There is not very much novelty in the loss functions. The idea of using losses that sample negative examples from two sources (other videos and other timesteps) has been used in other work such as (Korbar et al. 2018). The main novelty, instead, is the network architecture.\n- The evaluation for the architecture is quite minimal. There are a few ablations in each task, but they aren't chosen very systematically. These ablations also seem to simultaneously change the architecture and the loss, which makes it hard to take much away from them.\n- The paper relies exclusively on pretraining + finetuning experiments. While it is encouraging that the method outperforms other self-supervised pretraining methods, these comparisons are not completely apples-to-apples, since they use different network backbones. This is to some extent unavoidable, since this is the standard practice in the video self-supervision area. However, since this is one of the only forms of evaluation (and there are few ablations) it makes it hard to gauge the effectiveness of the method.\n- The accuracy improvements from using the local contrastive loss are confined to the lipreading task. It does not seem to improve the model on the action recognition task. \n- The evaluation of the localization task is quite minimal. They show a few qualitative results without comparing to other methods. The dot-product attention used in the model is quite similar to previous methods, such as (Arandjelović and Zisserman, 2018).  The paper should explain what the novelty of their approach is, and should compare with previous methods.\n- I didn't find the paper to be particularly well written. I found it challenging to understand the motivation for the method and the description of it. \n\nOverall:\nWhile this is a promising direction, the evaluation of the architecture (the main novelty) is not very thorough. ", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper received mixed reviews. While AnonReviewer1 and AnonReviewer2 liked the idea of jointly learning global-local representations, the other reviewers were concerned about the technical novelty. Reviewers also raised various questions about the experiments and ablation studies. AC found that the rebuttal well addressed the reviewers' questions about the experiments, but it failed to elaborate on the "why" of combining global and local self-supervised representations. AC agreed with AnonReviewer3 and AnonReviewer4's concerns on technical novelty. Considering the reviews, we regret that the paper cannot be recommended for acceptance at this time.  The authors are encouraged to consider the reviewers' comments when revising the paper for submission elsewhere.
Decision: Reject