Contrastive Audio-Visual Masked Autoencoder
Authors: Yuan Gong
Andrew Rouditchenko
Alexander H. Liu
David Harwath
Leonid Karlinsky
Hilde Kuehne
James R. Glass
Venue: ICLR 2023 notable top 25% (2023)
Keywords: multi-modal learning
audio-visual learning
self-supervised learning
masked autoencoder
contrastive learning

Abstract: In this paper, we first extend the recent Masked Auto-Encoder (MAE) model from a single modality to audio-visual multi-modalities. Subsequently, we propose the Contrastive Audio-Visual Masked Auto-Encoder (CAV-MAE) by combining contrastive learning and masked data modeling, two major self-supervised learning frameworks, to learn a joint and coordinated audio-visual representation.
Our experiments show that the contrastive audio-visual correspondence learning objective not only enables the model to perform audio-visual retrieval tasks, but also helps the model learn a better joint representation. As a result, our fully self-supervised pretrained CAV-MAE achieves a new SOTA accuracy of 65.9% on VGGSound, and is comparable with the previous best supervised pretrained model on AudioSet in the audio-visual event classification task. Code and pretrained models are at https://github.com/yuangongnd/cav-mae.

Published as a conference paper at ICLR 2023
CONTRASTIVE AUDIO- VISUAL MASKED AUTOENCODER
Yuan Gong' (yuangengemit.een Andrew Rouditchenko
Alexander H. Liu', David Harwath', Leonid Karlinsky* Hilde Kuehne James Glass' MIIT CSAIL; PT Austin; 'IBM Research AI; "MIT-IBM Watson Lab; "Goethe University Frankfurt
ABSTRACT
In this paper, we first extend the recent Masked Auto-Encoder (MAE) model from a single modality to audio-visual multi-modalities Subsequently, we propose the Contrastive Audio- Visual Masked Auto-Encoder (CAV-MAE) by combining con- trastive learning and masked data modeling, two major self-supervised learning frameworks, to learn a joint and coordinated audio-visual representation.
Our experiments show that the contrastive audio-visual correspondence learning objective not only enables the model to perform audio-visual retrieval tasks, but also helps the model learn a better joint representation. As a result, our fully self-supervised pretrained CAV-MAE achieves new SOTA accuracy of 65.9% on VGGSound, and is comparable with the previous best supervised pretrained model on AudioSet in the visual event classification task. Code and pretrained models are at
htgptttttuug...g hwtp:///wo.//aaggggn//goeooooe cawp:///ddaaaaaaaaaooaaoaaam mae
INTRODUCTION
Acoustic and visual modalities have different properties, yet humans able seamlessly connect and integrate them to perceive the world. Developing learning algorithms to replicate these abilities, especially for multi-modal audio-visust fusion and retrieval is of great interest. Since manually annotating audio and video is expensive and difficult to scale, how to utilize web-scale unlabeled video data in self-suuprvised manner has become core research question.
One major line of audio-visual self-supervised learning research is leveraging the natural audio- visual correspondences found in videos. Among numerous ways to such correspondences, Con- trastive Audio Visual Learning has shown to be a simple yet effective approach (Arandjelovic & Zisserman 2018 Morgado al. 2022 Rouditchenko al. 2021). It learns coordinated repre- sentations that are closer for paired audio and visual samples than for mismatched samples. Such coordinated representations are particularly useful for tasks such as cross cross-modal retrieval.
Another vetted commonly used self-supervised learning framework is Masked Data Modeling (MDM), which learns a meaningful representation with the pretext task of recovering the origi- nal inputs or features from the corrupted ones (Devlin et al.) (019). Particularly, based on the Audio Spectrogram Transformer (Gong al., 2021a) and Vision Transformer (Dosovitskiy et al.) (020) backbones, the single-modal Masked Auto-Enat Encoder (MAE) (He et al. 2022) achieved state-of-the- art (SOTA) performance on images and audio tasks (Huang et | (022a) individually. Inspired by
these vances, we propose to extend the monll- MAE to Audio- Visual Masked -ncooer (AV-MAE), aiming to learn a joint representation that fuses the unimodal signals.
Although these two major self-supervised frameworks have been widely used individually, to the best of our knowledge, they have never been combined in audio-visual lcarning. In fact, we find they complementary: Contrastive audio-visual learning explicitly leverages the very useful audio- visual pair information, but it could discard modality-unique information that is useful in down- stream tasks: The reconstruction task of AV-MAE forces its representation encode the majority of the input information in the fusion, but it lacks an explicit audio-visual correspondence objective.
Multi modal representations can be divided into two categories: joint representations that combine the unimodal signals into the same representation space, and coordinated representations that process unimodal signals separately, but enforce certain similarity constraints on them. (Baltruaaitis al. 2018

---

Published as a conference paper at ICLR 2023
This motivates to design the Contrastive Audio-Visual Masked Autoencoder (CAV-MAE) that integrates contrastive learning and masked data modeling which learns a joint and coordinated audio-visude representation with single model. Our experiments support our design: audio- visual event classification, CAV-MAR significantly outperforms baseline models trained with only contrastive masked data modeling objectives, demonstrating that the two objectives are comple- mentary in learning a strong joint audio-visual representation. As a result, CAV-MAE achieves a new SOTA accuracy of 65.9% on VGGSound, and is comparable with the previous best supervised pretrained model on AudioSet. Moreover, when it comes to audio-visual retrieval, CAV-MAE also performs equally well or even better than models trained with only the contrastive objective, which demonstrates that CAV-MAE can learn both joint and coordinated representation well. Finally, CAV-MAE multi-modal pretraining improves single-modis performance, consequently, CAV-MAE achieves a new SOTA for audio-based event classification on AudioSet-20K and VGGSound.
In summary, our contributions are: (1) We extend the single-modal MAE to multi-modal AV-MAE, which fuses audio-visua inputs for self-supervised learning through cross-modal masked data mod- eling: (2) More importantl, we investigate how to best combine contrastive audio-visual learn- ing with masked data modeling and propose CAV-MAE; (3) We demonstrate that contrastive and masked data modeling objectives are complementary. As a result, CAV-MAE matches or outper- forms SOTA models on audio-visual classification.
CONSTRASTIVE AUDIO VISUAL MASKED AUTOENCODER
2.1 PRELIMINARIES
2. AUDIO AND IMAGE PRE-PROCESSING AND TOKENIZATION
As depicted in Figure (A), we follow pre-processing and tokenization in AST (Gong et al. 202 la) and ViT (Dosovitskiy et al. 2020) for audio and image inputs, respectively. Specifically, we use 10-second videos (with parallel audios) in AudioSet (Gemmeke al., 2017) and VGGSound (Chen it 2020)) to pretrain and fine-tune the model. For audio, each 10-second audio waveform is first converted sequence of 18-dimensional log Mel filterbank (Ibank) features computed with 25ms Hanning window every 10ms. This results in a 102e(time) x 128 (2frqqqu))))) spectrogram. We then split the spectrogram into 512 16 16 square patches a = [a ..... a 2l12] as the input of the model. Processing video with Transformer models is expensive and typically requires industrial- level computation resources. To lower the computational overhead and our resources, we use a frame aggregation strategy. Specifically, we uniformly sample 10 RGB frames from each 10-second video (i.e.. FPS). During training, we randomly select one RGB frame as the input: during infer- ence, we average the model prediction of each RGB frame the video prediction. Compare with concatenating multiple RGB frames as the input of the Transformer that has a quadratic complexity (e.g., in Nagrani al. (202T)), frame aggregation is much more efficient with linear complexity in time at a cost of not considering inter-frame correlation. For each RGB frame, we resize and center crop it to 224 x 224, and then split it into 196 16 16 square patches [v] .... 2966].
2.1.2 THE TRANSFORMER ARCHITECTURE
Throughout this paper, we use the standard Transformer (Vaswani al. 2017) our main model component. Each Transformer layer consists of headi- self-attention (MSA), layer normal- ization (LN), and multilayer perceptron (MLP) blocks with residual connections. Specifically, we denote Transformer layer y Transformer(x; MSA, LN1, LN2, MLP) as:
X' MSA(LN, + +x; y = ML)( +x'
(1)
where MSA computes dot-product attention of each element of x and thus has a quadratic complex- ity w.r.t. to the size of x. Please refer to Vaswani et al. (2017) for further details on Transformers.
2.1.3 CONTRASTIVE AUDIO-VISUAL LEARNING (CAV)
The natural pairing of audio and visual information in videos is a useful signal for learning audio- visual representations through self-supervision. A conventional CAV model is shown in Figure B (top); for mini-batch of N audio-visual pair samples, we first pre-process, and tokenize the audios
2

---

Published as a conference paper at ICLR 2023
A2
Audio
Pasling
ContrusTIVe Loke
16
V2
VL
V2
VI96
Combination
V2
Visual Encoder
Y2
Puoling
A) Audlo and Image Tokenization
Y2
Audio-Visual Contrastive Learning
Recomstruction Loss
Audio
A4
A2
Join Errcoter
M
Join Decoder
A3
JOPE Emccdde
AB
A2
Mal
A
Decofer
Contrastive
Loss
V
Visual
V4
M
C) Contrastive Audio-Visual Masked Auto-Encoder
Vanifla Audio-Visual Masked Auto-Encader
Figure 1: An illustration of method. A) We tokenize audio spectrograms and RGB images into 16x 16 square patches and use them as the input to all models. B) Conventional contrastive audio- visual learning model (top) and vanilla audio-visual masked auto-encoder (bottom, also novel and first introduced in this paper). C) Our proposed contrastive audio-visual masked auto-encoder (CAV- MAE) model. CAV-MAE integrates learning and cross-modal masked data modeling, which learns a joint and coordinate representations and performs well on both multi-modal joint classification tasks and cross-modal retrieval tasks.
major self self-superviee frameworks: contrastive audio-visune
and images and get sequence of audio and visual tokens {a, v. for each sample We then input a, and v, to independent audio and visual Transformer encoders E.(-) and (-), respectively, and get the mean pooled audio and visual representation of and c₁, i.e., of = McanPool (Prooj (a;)) and oº McanPool (E,, (v:)) where Proja and Projy are linear projections that maps each audio and visual token to IR 768 We then apply contrastive loss (Equation) 7 cf and cV.
2.1.4 SINGLE MODALITY MASKED AUTOENCODER (MAE)
Another line of major self-supervised frameworks is masked data modeling (MDM). Among numer- ous variants of MDM (e.g., |ao al. (202I): Wei al. (2022) ). the masked auto-encoder (MAE) a simple yet effective approach. For an input sample x that can be tokenized as [x x .... x x"),
MAE masks a portion of the input Xmask and only inputs the unmasked tokens Xmask Trans- former based encoder-decoder model. The model is asked reconstruct the masked tokens with the goal minimizing the mean square error (MSE) loss. During this process, the model learns meaningful representation of the input data. The advantages of MAE multifold. First, MAE directly uses the original input as the prediction target, which greatly simplifies the training pipeline. Second, MAE only inputs unmaksed tokens to the encoder, and combined with high masking ratio, MAE noticeably lowers the computational overhead. Third, MAE demonstrated strong performance in modgle-modal tasks for both audio and visual modalities. Due the space limitation, please refer to He et al. 2022); Huang et al. (2022a) for single-modal MAEs.
2.2 VANILLA AUDIO- VISUAL MASKED AUTOENCODER (AV-MAE)
While MAE has been applied to both audio and visual modality individually, has never been applied to audio-visual multi-modality learning. As the first contribution of this work, we extend MAE from single modality to audio-visus, multi-modus middalto and build a "vanilla" audio-visual au- toencoder (AV-MAE). As shown in Figure B (bottom), for pair of audio and image inputs, first tokenize them to a = [a". 512) and v [v' ...., v 1966] and project them to 768 with two modal- specific linear projection layer well as add modality type embedding E,, and E, and modality
3

---

Published as a conference paper at ICLR 2023
specific 2-D sinusoidal positional embedding, Eg and EF. i.e., a' Proja E + Eg and v' = Projv(v)) + Ey + E.. We concatenate a' and v' and construct joint embedding x = [a v'l. We then mask portion (75%) of and only input ummasked tokens Xunmask x Xmusk to an audio-visual joint encoder F2 (-) and get the output Xunmask- After that, we pad Xunmask with trainable masked tokens their original position as x'. Again, also add modality type embedding E and E', and -peeiity-specificc 2-D sinusoidal positional embedding Eg' and EP' before feeding x' to joint audio-visual decoder D-(-) to reconstruct the input, i.c. à V = Dj(x + EL Eyy] + [ER', E) Finally, we minimize the mean square error (MSE) between å, V and normalized a. V.
Compared with single-modal MAEs, the AV-MAE features a cross-modal masked data modeling objective that allows the model to reconstruct one modality based on the information of another modality, which may help the model learn audio-visual correlation. However, without an explicit objective of encouraging paired audio-visual correspondence, vanilla AV-MAE actually does not effectively leverage the audio-visual pairing information (discussed in Appendix []. Also, using joint encoder for two modalities allows cross-modal attention, but it also means the two very different modalities are processed with the same weights, which could lead to a sub-optimal solution.
2.3 CONSTRASTIVE AUDIO- VISUAL MASKED AUTOENCODER (CAV-MAE)
As discussed in Section 2.1.3 and 2.2 contrastive audio-visual learning and AV-MAE each has its advantages and disadvantages. Can we integrate the complementary advantages of CAV and AV- MAE? With this goal, we design the Contrastive Audio-Visual Masked Autoencoder (CAV-MAE) (shown in Figure C). For mini-batch of N audio-visual pair samples, we first pre- pre-process and tokenize the audios and images and get a sequence of audio and visual tokens {a,, V.} for each sample i and project them to R66 with two modal-specific linear projection layer. We also add modality type embedding E, and E, and modality-specific 2-D sinusoidal positional embedding Eg and EP. After that, we uniformly mask 75% of tokens of each modality, i.c.
a
Mask (Proja (a) + E. Eg) unmask Masko. (Projj (v.) E, E.)
unmask
(2)
(3)
We then input a. uimmakk and v. unmask to independent audio and visual Transformer encoders E, (-) and (-) and get at and v/. respectively. After that, we apply multi-stream forward passes to MSA, LN1, LN2, LN2, MLP). Specifically, we input
input a/. v to joint audio-visual encoder E-;
audio tokens a/, video tokens v/. and concatenated audio-visude tokens la,, v/Il in three independent forward passes to Ez. For each stream, we use different layer normalization layers LN1 (a. (a.vav) and . ). all other weights (i.e., weights of the MSA and MLP) of Fj shared for all three streams. Formally,
I.N2
4 = MeanPool(Eggp (Ea pmimask)); LN1,, LN2,))
(4)
C = MeanPool Fl (F,, (v ummmask) )); IN1, I,22))
(5)
x; FJ (Fa (a.mmack), F., (V, [mmmask) ); LN1 I.22..)
We use the output of the audio and visual single modality stream of and ca for contrastive learning and the output of the audio-visual modal stream x; for the reconstruction task.
For contrastive audio-visual learning, we use the contrastive loss Le:
N
exp(se./T) Ene exp(si, + exp(ss./T)
L. =
log
where 8i.} = ||< and is the temperature.
For the reconstruction task, we pad x; with trainable masked tokens at their original position x We also add modality type embedding E, and E, and modality-specifii 2-D sinusoidal positional embedding Ey' and EP' before feeding x to joint audio-visual decoder D;(-) to reconstruct the input audio and image. Dj(-) processes audio and visual tokens with a same set of weights except the last sodal-specific projection layer, it outputs â and v. We then apply a mean square error reconstruction loss L,:
à v, Dj(x' + EE E.) + [Eg' EP'))
(8)
4

---

Published as a conference paper at ICLR 2023
N
a amask norm a
mask)
Σ (yimask norm V maskk)
))
L.
mask
mask
where N is the mini-batch size; a mask ymakk, å mmaak denote the original and predicted masked patches (we only calculate the loss based on the masked portion of the input); |a" mamkk]] and [ymask] denote the number of masked audio and visual patches, respectively.
Finally, we sum up the contrastive loss Le (multiplied by a weight Xe) and the reconstruction loss L, as the loss for CAV-MAE, i.e,, CAAV-MAB = L, + X Le.
abandon the decoder and only keep the encoders of the model for downstream tasks. We can use the sum of the single-modad stream output and the multi-mode modality stream output, or just the multi-modal stream output for finetuning. They perform similarly in our experiments.
After pretraining,
Discussion: we next discuss the motivation of some key designs of CAV-MAE:
I. Multi-stream forward passes of the joint encoder. We find it important to restrict the represen- tations used for contrastive audio-visual learning, that e only comes from the audio input and c" only comes from the visual input, otherwise the contrastive objective will collapse. In the mean- time, we hope the encoder fuses the audio and visual information for the reconstruction task and downstream tasks. Therefore, we design the multi-stream forward pass strategy for CAV-MAE.
2. -ppelit encoders and LN layers. While there are few recent attempts (Akbari al. 2021 Dai et al. 2022) to process audio and visual modalities with a unified network, due to the very different nature of audio and visual modalities, the general conclusion that modality -specific networks are still optimal in terms of performance. Therefore, we choose to encode audio and visual inputs with cedalitt-sppciffe encoders before the joint encoder. For the same reason, we also use different normalization statistics for each stream of the joint encoder. Efficiency-wise. having two poodality-speeiffe encoders increases the model size, but lowers the computation as the Transformer has quadratic complexity w.r.t the input sequence length.
Masked contrastive audio-visual learning. Unlike single-modality contrastive learning, conven- tional contrastive audio-visual learning does not typically apply augmentation masking. In this work, we propose to use masked contrastive audio-visual learning, i.e., we randomly mask a por- tion of the input before conducting contrastive learning. This design not only allows to combine CAV with AV-MAE, but also helps to avoid overfitting. In practice, when the masking ratio is 75% and the effective contrastive batch size is 27 (108 on 4 GPUs), the audio-visual matching accuracy during pretraining on the evaluation set is about 72%, which shows the task is neither trivial nor impossible. We discuss the impact of masking on contrastive learning in detail in Appendix F
2.3.1 IMPLEMENTATION DETAILS
By default, all encoder Transformer layers are 768-dimensional and have 12 attention heads. The joint encoder of the Vanilla AV-MAE is 12-layer Transformer; The audio and visual encoders of CAV-MAE are 1lllaeer Transformers (each is 768- dimensional) and the joint encoder is a single- layer Transformer. I.e., we control the total number of encoder layers of all models as 12, but CAV and CAV-MAE are larger models due to the modality-specific encoders. The decoder of AV-MAE and CAV-MAE are 8-layer Transformers with an embedding dimension of 512 and 16 attention heads. These settings are identical to the original vision Me et al. (2022). We fix the contrastive loss temperature
0.05. For CAV-MAE, we use Ae- 0.01. Note the relatively small Ae is due to the scale of the gradient of L. being larger than L, it does not mean the contrastive objective is unimportant. The encoder and decoder of the default CAV-MAE model have about 164M and 27M parameters, respectively.
Following the common practice of audio-visual learning, we initialize the weights of all models with ImageNet pretrained weights. Specifically, we use the weights of the original vision MAE He et al. (022)) Nevertheless, unlike previous work that uses supervised pretrained weights (e.g.. Fayek & Kumar (202l) and Nagrani et al. (202I)), we only use the self-supervised pretrained weights (i.e., without finetuning), which does not lead to the best performance but makes our whole training pipeline self-supervised. The impact of initialization strategy is discussed in detail in Appendix B
5

---

Published as a conference paper at ICLR 2023
3 SELF-SUPERVISED MODEL PRETRAINING
We pretrain and compare the performance of the following models:
1. A /iaa -MAE: Single-modal masked auto-encoder models. The model architec- ture is the same with Vani 11a AV-MAE but they are only pretrained with data of single modality.
2. CAV: The contrastive audio-visure learning model that has no reconstruction objective. For fair comparison, we implement CAV using the same encoder architecture (modal-specific encoders joint encoder) with CAV-MAE but remove the reconstruction objective Lr-
Vanilla :V-MAE: The vanilla audio- visua--vull masked auto-encoder with a joint encoder and no contrastive objective as described in Section 2.2
AV-MAE: The audio-visual masked auto-encoder with two modal-specific encoders and a joint encoder. It has the same architecture with CAV-MAE, but Xe is set to 0 (no contrastive loss). We use this model to disentangle the impact of modal-specific encoders (when compared with Vanilla AV-MAE) and contrastive objective (when compared with CAV-MAE).
5. CAV-MAE: Our proposed contrastive masked auto-encoder as described in Section 2.3
6. CAV-MAEscale The same model with CAV-MAE, but trained with larger batch size=108 (effective contrastive batch size=27) and more epochs=25. We train this model on our best GPUs.
For fair comparison, all models (except M are pretrained with the same pipeline with a batch size of 48 for 12 epochs on the full AudioSet-2M. During pretraining, we intentionally do not use class balanced sampling as that implicitly leverages the label information. Our pretraining process (including the ImageNet pretrained weight initialization) is fully self-supervised. Please refer Appendix for all pretraining details.
AUDIO VISUAL EVENT CLASSIFICATION
We evaluate the representation quality on the audio-visual event classification task, a major audio- visual learning benchmark. Specifically, we fine-tune the pretrained models on three datasets: 1) AudioSet-20K (20K samples, same domain as the pretraining data); 2) AudioSet-2M (2 million samples, same with pretraining data); and 3) VGGSound (200K samples, different domain than the pretraining data), covering various downstream data volume and domain situations.
In the fine-tuning stage, we only keep the encoder of the pretrained models and connect it a randomly initialized linear classification head. To avoid overriding too much of the knowledge learned in pretraining, we use smaller learning rate for the pretrained weights and 10x-100x larger learning rate for the new classification head. We use the standard training pipeline used in prior audio-based and audio-visual event classification work Gong et al. (2021a b); Nagrani et al. 202I) with mixup Zhang et al. (2018), balanced sampling, label smoothing, label enhancement (only for AudioSet-20K: and random time shifts. We fine-tune the model using audio-only data (A), video-only data (V), and audio-vin audio-visual data (AV) to evaluate the dingle-modal and multi-modal representation quality. We show the results in Table Key findings are as follows:
1. Contrastive learning and masked data modeling complementary. While both AV-MAE (only with masked data modeling objective) and CAV (only with contrastive objective) perform better than ensembling two single-modal MAEs, the proposed CAV-MAE that combines the two objectives significantly boosts the performance (e.g., 2.0 and 3.1 mAP boost from CAV and RV-MAE on AudioSet-20K, respectively). Note CAV MAE, AV-MAE, and CAV have the same architecture during fine-tuning, the only difference is the objective in the pretraining stage. This demonstrates that the two major supervuper learning frameworks are complementary in the context of audio- visual learning and CAV-MAE is an effective way to combine their advantages.
CAV-MAE multi-modal pretraining improves single-modal performance. We find the CAV-MAE model pretrained with paired audio-visual data, when fine-tuned with only single modality, performs noticeably better than Midio-MAB and Visual MME on single-modal clas- sification tasks (e.g., 34.2-+37.7 mAP for audio, 15.7- mAP for visual on AudioSet-20K). Note for single-modal fine-tume fine-tunin, CAV-MAE only keeps one branch and has the same architecture with Audio-MAE and Visual-MAE. so the performance improvement can only come from the use of multi-modal data during pretraining. We hypothesize this is due to the two modalities serving soft labels for each other, providing richer information than the binary human-annotated labels.

---

Published as a conference paper at ICLR 2023
Table 1: Comparing audio-visual classification performance on AudioSet and VGGSound. IN Si=ImageNet supervised learning; SSL=sseff-upprrisee learning: "Industry-level computation. Nonstandard data split; ens "Ensemble of single-modal models. We bold the best methods without supervised pretraining, and underline the overall best methods.
AudioSet 20K (mAP)
AudioSet 2M (mAP)
VGGSSund (Acc)
Pretrain
V
A
A-V
V
Existing Audio-Basee Based Models
PANNS (Kong al 2020
27.8
43.9
AST Gong et al. 2211
IN SL
34.7
45.9
HTS. CSS.XiCChenn hen al. 1...400
47.1
PaSST Koulin
IN SL.
47.1
SSAST (Gong et al
2022
SSL
31.0
MAE-AST [Badde et
2022
Audio MAE (vailla) (Huang al. 2022a
46.8
Audio-MAR! (Huang al. 2022a
SSI.
37.1
47.3
Chen al. 2020
48.8
AndiSloo Fast (Kazakos et 2021
50.1
Existing Audio- Visual Models
GBlend
Wang al. 2020
29.1
22.1
37.8
32.4
41.8
Perceiver Daegle
2021
38.4
44.2
Atta AV
Favek & Kunnar
IN SL
38.4
25.7
46.2
MBT
Nagrami al.
IN SL
31.3
27.7
43.9
44.3
32.3
52.1
52.3
51.2
64.1
Our Mingle-Modal MAE
Audio-MAE
36,7th
44.9
46,9
63.1
Visual-MAE
SSL
15.7
24.2
45.7
Our Contrastive Audio- Visual Learning
Audio-Visual
CAV
SSL
34.6
18.4
38.5
43.6
24.4
48.1
57.3
45,1
64.1
Our Multi-Modal MAE
Vanilla AV-MAE
SSL
32.7
15.8
36.5
43.7
48.3
45.4
AV-MAR
SSI.
33.4
15.1
37.4
44.8
24.0
49.6
57.2
45.3
64.1
CAV-MAE
SSL
36.8
18.7
40.5
45.8
25.6
50.5
59.2
46.6
65.4
CAV-MAE
SSL
37.7
19.8
42.0
46.6
26.2
51.2
59.5
47.0
65.5
As a result, CAV-MAE achieves a new SOTA performance on audio- bbiie event classification on 200 (37.7 mAP) and VGGSound (59.5% accuracy), without supervised pretraining and industry-level computational resources.
Fully SSL pretrained CAV-MAE matches or outperforms SOTA models with significantly fewer computational resources. There are two major setting differences between this work and previous SOTA works. First, our pretraining completely self-supervised so that our model can leverage web-scale unlabeled videos, while supervised ImageNet pretraining is commonly used in previous audio-visual works, e.g., in MBT (Nagrani et al. 2021}. ImageNet labels are strong supervision signals that directly impact the visual branch performance (see Table [T]. As result, our visual branch is worse than the SOTA models. Second, we pretrain and fine-tune the model with 4 GPUs (which also makes work easy reproduce), while most SOTA models trained with industry-level resources (c.g.. 32 TPUs for Perceiver (Jaegle et al. 2021). 64 GPUs for Audio-MAE (Huang al. (022a)) and MBT), which brings many benefits such as large batch size (particularly useful for contrastive learning), multiple frames input (MBT uses 8 frames as input), and more training epochs (Audio-MAE pretrains the model for 32 epochs).
Even with such setting differences, on the audio-visual event classification task. our CAV-MAB per- forms better than the best existing audio-visual model MBT on VGGSound (even when CAV-MAE only pretrained
VGGSound, see Table [2] and comparable on AudioSet-20K and AudioSet-2M. On the audio-based event classification task, our CAV-MAE performs better than the best existing audio model Audio-MAE on AudioSet-20k and comparable on AudioSet-2M.
Besides, we find modal-specific encoders are helpful as AV MAMAE outperforms Vanilla AV-MAE. Vanilla 1-MAE with only joint encoder does not outperform the ensemble of single-modal Audio-MAE and Visual- -MA.. Scaling up the batch size and training epochs improves the per- formance as CAV-MA
generally performs better than CAV-MAE. The performance margin smaller on larger fine-tuning datasets. Finally, We also evaluate the models on the audio-visual action recognition task (Appendix C). which leads consistent conclusions.

---

Published as a conference paper at ICLR 2023
Table 2: Ablation studies on audio-visual classification. MM-multi-mode modal, SM=single-moode
(a) Prefrain
(b) Pretrain epochs
(c) Pretrain batch
(d) Pretrain target
(e) Pretrain dataset
>e
AS-20K
Epochs AS-20K
Size AS-20K
Norm
AS-20k
Dataset
VGGSound
0.1
39.3
37.3
48
40.5
w/o norm 40.5
AS-2M
65.5
0.01
40.5
39.1
108
40.8
w/norm 40.5
VS
64.2
0.001
38.6
12
40.8
AS-2M+VS
65.9
25
42.0
(f) Finetuning
(g) SM experiment
(h) Inference frame
(i Linear probe
Strategy AS-20K
Setting
AS- 20K
Frame Used
AS- 20K
Model
AS-20K
A V
A-V
MM
42.0
SM Ensemble
24.2
SM
41.3
Missing Modality 36.7 14.4
Middle
17.4 40.9
AV-MAE
24.0
MM+SM
41.7
SM Fine-tune 37.7 19.8
Aggregation 19.8 42.0
CAV-MAE 29.8
Ablation Studies: We conduct a series of ablation studies to show the impact of each design factor. CAV-MAE MMsalee CAV-MAE the base model, change factor at
For each study, time, and report the downstream classification performance the model on AudioSet-20K or VG- GSound. Our findings performance, too large or too small de leads to a noticeable performance drop (Table 2a); Scaling up the pretraining epochs and batch size consistently leads a performance improvement (Table and 2c Normalizing the prediction target only leads to marginal performance improvement (Table 2d); When finetuning on VGGSound, pretraining with the larger nut-of-domain AudioSet-2M1 is better than pretraining with the smaller in- in-domain VGGSound itself, but pretraining first on AudioSet-20 2M and then on VGGSound leads the best result (Table 2 During fine-tuning, using the output of the multi-modal stream of the encoder leads to better performance than using the concatenated single-mode modal stream outputs, and summing the output two streams generally lead to similar result (Table 21:; When only modality is of interest, is better to fine-tune the model with single-modal data than fine-tune the model with audio-visue data and do single modality inference. However, the performance gap is small for audio (Table 2g): The frame aggregation strategy boosts the perfor- mance without the need to input multiple frames simultaneously to the model (Table 2h); In the linear probe setting, CAV-MAE also noticeably outperform the baselines (Table 2i). We also study the impact of model initialization, masking strategy, and frame rate in Appendix
follows: the weight of the contrastive loss A. has large impact on the
respectively.
5 SUDIO-VISUAL RETRIEVAL
Table 3: Retrieval results on AudioSet and VGGSound.
Query
Visual - Audio
AudioSet Eval Subset
VGGSound Eval Subset
R@
R@5
RØ
R@1
R@5
R@10
Top 1 Video
2
Audio- Visual Models with Only MDM Loxs
Video
Video
Vanilla AV-MAE
0.1
0.3
0.8
0.7
1.4
AV-MAE
0.1
0.7
1.2
Query Video
Top Audio
Top 2 Audio
Top Audio
Audio- Visual Models with Only Contrastive Loss
CAV, d = 0.1
17.4
36.1
47.3
14.2
35.2
46.2
CAV. de 0.01
14.6
32.9
42.8
10.9
28.7
39.8
Construstive Audio- Visual Marked
Anto cadtu-Encuder
CAY-MAE, d
16.1
38.6
49,3
14.7
35,3
45.9
CAV-MEE, A 0.01
31.4
41.9
12.5
28.6
39.1
CAV-MAESON Sa 0.01 18.8
39.5
50.1
14.8
34.2
44.0
Figure 2: Sample retrieval results.
In the previous section, we show that CAV-MAE learns a good audio-visual joint representation that effectively fuses the unimodal signals for the audio-visual event classification task. Next, we study CAV-MAE also learns a good coordinated representation that captures audio-visual correspon- dences for audio-visual retrieval. Specifically, we uniformly sample subset of 1,725 and 1.545 audio-visual samples from the AudioSet and VGGSound evaluation set, respectively (about 10%) to make the similarity matrix of reasonable size. We input audio and image each model in two independent forward passes and take the mean- mean-pooled encoder outputs as the audio and visual repre- sentation, respectively. We then calculate the retrieval recall at rank 1, 5, and 10 (R&1, R@5, R@10)
8

---

Published as a conference paper at ICLR 2023
based on the cosine similarity of the audio and visual representation. All models are self-superrvsed pretrained but not fine-tuned. We show the quantitative results and samples of visual re- trieval in Table 3 and Figure 2 respectively. The results of audio visual retrieval, more samples, and additional retrieval experiments on MSR-VIT (Xu et al. 2016) can be found in Appendix [
We find contrastive objective is necessary for the audio-visual retrieval task as the performance of both Manilla-Wor and AV-MAE are close to random guesses. Nevertheless, the cross-modal masked data modeling objective does not hurt, and in many cases, improves the retrieval perfor- mance, e.g. when A = 0.1, CAV-MA MAE generally performs better than CAV. Scaling up the batch size and training epoch also leads a better retrieval performance. When tested on dataset different from the pretraining dataset ( (VGGSound), the retrieval performance is still competitive, indicating the audio-visual correspondence transfers well in addition to the audio and visual representations. These results demonstrate that the contrastive and mask data modeling objectives do not conflict, single pretrained CAV-MAE can be applied to both audio-visual fusion and correspondence tasks.
RELATED WORK
Contrastive Audio-Visual Learning The natural pairing of audio and visual information in videos has been a useful signal for learning audio-visual representations through self supf-suupr Exist- ing methods include knowledge distillation (Aytar et al., 2016 Owens et al. 2016), paired sample
discrimination (Arandjelovic & Zisserman 2017 Korbar et al. 2018 Owens & Efros 2018), and
contrastive learning (Morgado et al., (2021). To improve contrastive learning, some recent meth- ods sought to mine better negative samples (Ma et al.) 2020 Morgado et al. 2021a). while others
proposed additional data augmentation (Patrick et al.] 2021] Wang et al. a 2021) or using global and local video views (Zeng et al., [222]] Recasens et al. (021). Our approach instead combines the
contrastive loss with masked data modeling, which not only leads to an improvement in classifica-
tion performance but also maintains the compelling ability of audio-visual retrieval (Arandjelovic &
Zisserman 2018 Rouditchenko et al. 202T).
Masking data modeling has long history (Vincent et al.) 2008 and has been applicd on visual and audio domains (Bacvski et al. 2020 Hsu et al. 2021; Srivastava et al.] 2022) Given the success of MAE in the vision domain (He et al. 2022 Bachmann et al. 20022 Gird-
Masked Auto-Encoder.
har et al. 2022] Tong et al.; 2022 Feichtenhofer et al., 2022), several efforts adapt MAE for audio
with relatively minor changes to the overall pipeline (Baade et al., [022)] Niizumi al.) [022 Chong
ee al. 2022 Huang et al. (022a)). There are a few recent works investigating multi-modal MAE for
the vision & language multi-modal scenarios (Geng et al.; 2022: Kwon al. (022). which inspired us to design an audio-visual MAE. To the best of our knowledge, our AV-MAE and CAV-MAE are the first audio-visual masked autoencoders. One closely related concurrent work is CMAE (Huang et al. 20226), which also combines MAE and contrastive loss, but only for single-modal images. Our motivation and implementation are very different from CMAE as we aim leverage the unique audio-visume pair information and CAV-MAE features multi-stream joint encoder design. Finally, while we take a modern approach with Transformers, multi-modal autoencoders have been studied more than a decade ago with much simpler models and datasets (Ngiam et al., 2011).
7 CONCLUSION
In this paper, we introduce CAV-MAE, a novel audio-visual learning model. The main idea of this paper is simple: masked data modeling and contrastive learning are pair of complementary frame- works that should be used together for audio-visual self- eeffssuerrvised learning. Effectively combin- ing the two frameworks and avoiding representation collapse requires some careful design such as the multi-stream forward pass strategy. joint-specific encoder architecture, and masked contrastive learning. From the perspective of representation learning. CAV-MAF learns joint and coordinated representation and can be used for both audio-visual joint event classification task well as the audio-visual retrieval task. As a result, on the audio-visual event classification task, CAV-MAE matches outperforms SOTA models with fully self-supervised pretraining and noticeably fewer computational resources; on the retrieval task, CAV-MAE is comparable to models trained with only the contrastive objective. Finally, CAV-MAE modal pretraining also learns strong single- modal representations, which leads to a new SOTA performance on audio-based event classification.
Acknowledgments: This research is supported by the MIT-IBM Watson Al Lab.

---

Published as a conference paper at ICLR 2023
ETHICS STATEMENT
The data used in this paper are publicly available YouTube videos, we do not use videos that have been removed by the user. The proposed audio-visual model can be applied in a wide range of areas including security-related applications. However, it can also be used for malicious purposes such as surveillance. We are committed distributing our code and model carefully.
REPRODUCIBILITY STATEMENT
We document implementation details Section 2.3. and Appendix B Code and pretrained models are available at https:://g thub. com/ com/yyuagongnd// maee
REFERENCES
Hassan Akhari, Liangzhe Yuan, Rui Qian, Wei-Hong Chuang, Shih-Fu Chang, Yin Cui, and Boqing Gong. Vatr: Transformers for multimodal self-supervised learning from raw video, audio and text. Advances in Neural Information Processing Systems, pp. 24206-24221, 2021.
Relja Arandjelovic and Andrew Zisserman. Look, listen and learn. IEEE International Confer- ence on Computer Vision, pp. 609-617, 2017.
Relja Arandjelovic and Andrew Zisserman. Objects that sound. In European Conference on Com- puter Vision, pp. 435-451, 2018.
Yusuf Aytar, Carl Vondrick. and Antonio Torralba. Soundnet: Learning sound representations from unlabeled video. Advances Neural Information Processing Systems, 29, 2016.
Alan Baade, Puyuan Peng, and David Harwath. Mae-ast: Masked autoencoding audio spectrogram transformer. In Interspeech, 2022.
Roman Bachmann, David Mizrahi, Andrei Atanov, and Amir Zamir. MultiMAE: Multi-mode multi-
task masked autoencoders. The European Conference Computer Vision, 2022.
Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli, wav2vec 2.0: frame- work for self-supervised learning of speech representations. Advances in Neural Information Processing Systems, 13:12449-12460, 2020.
Tadas Baltrušaitis, Chaitanya Ahuja, and Louis-Philippe Morency. Multimodal machine learning: A survey and taxonomy. IEEE Transactions on Pattern Analysis and Machine Intelligence, 41(2): 423 443, 2018.
Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei. Beit: Bert pre-training of image transformers. In International Conference on Learning Representations, 2021.
Angie Boggust, Kartik Audhkhasi, Dhiraj Joshi, David Harwath, Samuel Thomas, Rogerio Feris, Dan Gutfreund, Yang Zhang, Antonio Torralba, Michael Picheny, and James Glass. Grounding spoken words in unlabeled video. In CVPR Sight and Sound Workshop, 2019.
Honglie Chen, Weidi Xie, Andrea Vedaldi, and Andrew Zisserman. Vggsound: A large-scale audio- visual dataset. In ICASSP, 721-725, 2020.
Ke Chen, Xingjian Du, Bilei Zhu, Zejun Ma, Taylor Berg-Kirkpatrick, and Shlomo Dubnov. Hts- at: A hicrarchical -eken-semantic audio transformer for sound classification and detection. In ICASSP, pp. 646-650, IEEE, 2022.
Dading Chong, Helin Wang, Peilin Zhou, and Qingeheng Zeng. Masked spectrogram prediction for self-supervised audio pre-training arXiv preprint arXiv:2204.1276. 2022.
Yong Dai, Duyu Tang, Liangxin Liu, Minghuan Tan, Cong Zhou, Jingquan Wang, Zhangyin Feng, Fan Zhang, Xueyu Hu, and Shuming Shi. One model, multiple modalities: sparsely activated approach for text, sound, image, video and code. arXiv preprint arXiv:2205.06 2022.
10

---

Published as a conference paper at ICLR 2023
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Conference of the North American Chapter of the Association for Computational Linguistics, Minneapolis, Minnesota, June 2019.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An im- age is worth 16x16 words: Transformers for image recognition at scale. International Confer- ence on Learning Reprexentations, 2020.
Haytham M Fayek and Anurag Kumar. Large scale audiovisual learning of sounds with weakly labeled data. International Joint Conferences on Artificial Intelligence, 2021.
Christoph Feichtenhofer, Haoqi Fan, Yanghao Li, and Kaiming He. Masked autoencoders spa- tiotemporal learners. Advances in Neural Information Processing Systems, 2022.
Jort F Gemmeke, Daniel PW Ellis, Dylan Freedman, Aren Jansen, Wade Lawrence, R Channing Moore, Manoj Plakal, and Marvin Ritter. Audio set: An ontology and human-labeled dataset for audio events. In ICASSP, pp. 776-780, 2017.
Xinyang Geng, Hao Liu, Lisa Lee, Dale Schuurams, Sergey Levine, and Pieter Abbeel. Multimodal masked autoencoders learn transferable representations. arXiv preprint ar 2r05 2022.
Rohit Girdhar, Alaaeldin El-Nouby, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, and Ishan Misra. Omnimae: Single model masked pretraining on images and videos. arXiv preprint MxXiv:2206.08356, 2022.
Yuan Gong, Yu-An Chung, and James Glass, AST: Audio Spectrogram Transformer. In Interspeech, pp. 571-575, a0
Yuan Gong. Yu-An Chung, and James Glass. Psla: Improving audio tagging with pretraining, sampling. labeling, and aggregation. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2021b.
Yuan Gong, C Lai, Yu-An Chung, and James Glass. Ssast: Self-supervised audio spectrogram transformer. In AAAI Conference on Artificial Intelligence, volume 36, pp. 10699-10709, 2022.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 16000-16009, 2022.
Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed. Hubert: Self-supervised speech representation learning by masked prediction of hidden units. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 29:3451-3460, 2021.
Po- Yao Huang, Hu Xu, Juncheng Li, Alexei Baevski. Michael Auli, Wojeiech Galuba, Florian Metze, and Christoph Feichtenhofer. Masked autoencoders that listen. Advances in Neural Infor- mation Processing Systems, 2022a.
Zhicheng Huang, Xiaojie Jin, Chengze Lu, Qibin Hou, Ming-Ming Cheng, Dongmei Fu, Xiaohui Shen, and Jiashi Feng. Contrastive masked autoencoders stronger vision learners. arXiv preprint arXiv:2207. 13532,, 2022b.
Andrew Jaegle, Felix Gimeno, Andy Brock, Oriol Vinyals, Andrew Zisserman, and Joao Carreira. Perceiver: General perception with iterative attention. In International conference on machine learning, pp. 4651-4664, 2021.
Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijaya- narasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, et al. The kinetics human action video dataset. arXiv preprint 17Xiv 06950, 2017.
Evangelos Kazakos, Arsha Nagrani, Andrew Zisserman, and Dima Damen. Slow-fast auditory streams for audio recognition. In ICASSP, pp. 855-859. IEFE, 2021.
11

---

Published as a conference paper at ICLR 2023
Qiuqiang Kong. Yin Cao, Turab lqbal. Yuxuan Wang. Wenwu Wang, and Mark D Plumbley. Panns: Large-scale pretrained audio neural networks for audio pattern recognition. IEEE/ACM Transac- tions Audio, Speech, and Language Processing, 28:2880- 2894, 2020.
Bruno Korbar, Du Tran, and Lorenzo Torresani. Cooperative learning of audio and video models from self-supervised synchronization. Advances in Neural Information Processing Systems, 31, 2018.
Khaled Koutini, Jan Schlüter, Hamid Eghbal-zadeh, and Gerhard Widmer. Efficient training of audio transformers with patchout. arXiv preprint arXiv:2110005060 2021.
Gukyeong Kwon, Zhaowei Cai, Avinash Ravichandran, Erhan Bas, Rahul Bhotika, and Stefano Soatto. Masked vision and language modeling for multi-modal representation learning. arXiv preprint arXiv:2208.0211 2022.
Juncheng Li, Shuhui Qu, Po-Yao Huang, and Florian Metze. AudioTagging Done Right: 2nd com- parison of deep learning methods for environmental sound classification. In Interspeech, pp- 1521-1525, 2022.
Shuang Ma, Zhaoyang Zeng, Daniel McDuff, and Yale Song. Active contrastive learning of audio- visual video representations. In International Conference on Learning Representations, 2020.
Pedro Morgado, Ishan Misra, and Nuno Vasconcelos. Robust audio-visual instance discrimination. In JEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12934-12945, 2021a.
Pedro Morgado, Nuno Vasconcelos, and Ishan Misra. Audio-visual instance discrimination with cross-modal agreement. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12475-12486, 2021b.
Arsha Nagrani, Shan Yang. Anurag Arnab, Aren Jansen, Cordelia Schmid, and Chen Sun. Attention bottlenecks for multimodal fusion. Advances in Neural Information Processing Systems, 34: 14200-14213, 2021.
Jiquan Ngiam, Aditya Khosla, Mingyu Kim, Juhan Nam, Honglak Lee, and Andrew Y Ng. Multi- modal deep learning. In International Conference on Machine Learning, 2011.
Daisuke Niizumi, Daiki Takeuchi, Yasunori Ohishi, Noboru Harada, and Kunio Kashino. Masked spectrogram modeling using masked autoencoders for learning general-purposs audio represen- tation. arXiv::204. /2260, 2022.
Andrew Owens and Alexei A Efros. Audio-visual scene analysis with self-supervised multisensory features. In European Conference Computer Vision, pp. 631-648, 2018
Andrew Owens, Jiajun Wu, Josh H McDermott, William T Freeman, and Antonio Torralba. Ambient sound provides supervision for visual learning. In European Conference on Computer Vision, pp- 801-816, 2016.
Mandela Patrick, Yuki M Asano, Polina Kuznetsova, Ruth Fong, João F Henriques, Geoffrey Zweig, and Andrea Vedaldi. On compositions of transformations in contrastive self-supervised learning. In /EEE/CVF International Conference on Computer Vision, pp. 9577-9587, 2021.
Adria Recasens, Pauline Luc, Sean-Baptiste Alayrac, Luyu Wang, Florian Strub, Corentin Tallec, Mateusz Malinowski, Viorica Paträucean, Florent Altché, Michal Valko, et al. Broaden your views for self-supervised video learning. In JEEF/CVE International Conference on Computer Vision, pp. 1255-1265, 2021.
Andrew Rouditchenko, Angie Boggust, David Harwath, Brian Chen, Dhiraj Joshi, Samuel Thomas, Kartik Audhkhasi, Hilde Kuchne. Rameswar Panda, Rogerio Feris, al. Avlnet: Learning audio- visual language representations from instructional videos. In Interspeech, 2021.
Sangeeta Srivastava, Yun Wang, Andros Tjandra, Anurag Kumar, Chunxi Liu, Kritika Singh, and Yatharth Saraf. Conformer- baseed self-supervised learning for non-speech audio tasks. In ICASSP, pp. 8862-8866 IEBE, 2022.
12

---

Published as a conference paper at ICLR 2023
Zhan Tong, Yibing Song, Jue Wang, and Limin Wang. Videomae: Masked autoencoders are data- efficient learners for self-supervised video pre-training. In Advances Neural Information Pro- cessing Systems, 2022.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Eukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in Neural Informa- tion Processing Systems, 2017.
Pascal Vincent, Hugo Larochelle, Yoshua Bengio, and Pierre-Antoine Manzagol. Extracting and composing robust features with denoising autoencoders. In International Conference on Machine Learning, pp. 1096-1103, 2008.
Luyu Wang, Pauline Luc, Adria Recasens, Jean-Baptiste Alayrac, and Aaron van den Oord. Multimodal self-supervised learning of general audio representations. arXiv preprint 1XXi:2204412207, 2021.
Weiyao Wang, Du Tran, and Matt Feiszli. What makes training multi-modal classification networks hard? In IEEE/CVE Conference Computer Vision and Pattern Recognition, pp. 12695-12705, 2020.
Chen Wei, Haoqi Fan, Saining Xie, Chao-Yuan Wu, Alan Yuille, and Christoph Feichtenhofer. Masked feature prediction for self-supervised visual pre-training In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14668-14665 2022.
Fanyi Xiao, Yong Jae Lee, Kristen Grauman, Jitendra Malik, and Christoph Feichtenhofer. Audio- visual slowfast networks for video recognition. arXiv preprint arXiv:2001.08744 2020.
Jun Xu, Tao Mei, Ting Yao, and Yong Rui. Msr-vtt: large video description dataset for bridging video and language. In IEEE Conference on Computer Vision and Pattern Recognition, pp. 5288- 5296, 2016.
Zhaoyang Zeng, Daniel McDuff, Yale Song, et al. Contrastive learning of global and local video representations. Advances Neural Information Processing Systems, 34:7025-7040, 2021.
Hongyi Zhang, Moustapha Cisse, Yann N. Dauphin, and David Lopez-Paz. mixup: Beyond empiri- cal risk minimization. In International Conference Learning Representations, 2018.
A DATASET DETAILS
We use two major audio-visual datasets for our experiments: AudioSet Gemmeke et al. (2017) and VGGSound Chen et al. (2020). AudioSet-2M a collection of 2M 10-second YouTube video clips labeled with the sounds that the clip contains from set of 527 labels of audio events, AudioSet- 20K is a subset of AudioSet-2M with a more balanced class distribution. Due to changes in video availability, downloaded 1,772,023 AudioSet-2M training, 18,691 AudioSet-20K training, and 17,249 evaluation samples, respectively. VGGSound Chen et al. (2020) is collection of 200K 10- second YouTube video clips annotated with 309 classes. We download 183,727 training and 15,44 test samples. We only use the labels in the fine-tuning stage to make our pretraining pipeline fully self-supervised. Compared with AudioSet, always visually evident within the video clip, which is done by filtering the videos with a pretrained vision classifier. As discussed Li al. (2022). different versions of dynamic datasets might cause a performance difference, to improve the reproducibility of this work, we release the training and test samples ids at atthtps:: httpt:::::/ggtbb...p com/ www.em//yyanggggn///aaaaeee cav-me
advantage of VGGSound is that the sound source is
B TRAINING DETAILS
are listed in Table Most of our experiments are run on 4xNVIDIA GTX Titan X Pascal GPUs with 12GB memory, only the scaled-up CAV-MAE500 is pretrained on 4XNVIDIA RTX A5000 GPUs with 24GB memory, making our result easier to reproduce with reasonable resources. Pretraining CAV-MAE takes about one week with 4 GPUs.
Our training
13

---

Published as a conference paper at ICLR 2023
Our model has similar size with "base" MAE models, ie.,, the full encoder and decoder model has ~ 190M parameters (due to two modal-specific branches); the encoder used for audio-visuse down- stream task is ~60M parameters; the encoder used for single-modal downstream task is ~85M parameters.
Table Our pre-training and fine-tuning hyperparameters.
Pretraining
Finetuning
CAV-MAE Scalet
All Other Models
All Models
Dataset
AS-2M
AS-2M
AS-20K
AS-2M
VGG
Optimizer
Adam, weight decay=5e-7,
betas=(0.955
0,999)
Backbone learning rate Classification head LR LR decay start epoch
le-4
5e-5
5e-5
le-5
le-4
5e-2
5e-4
le-3
10
10
5
2
2
LR decay rate
0.5
0.5
0.5
0.5
0.5
LR decay step
5
1
1
1
Epochs
25
12
15
10
Batch size
4x27
4x12
36
48
48
GPUs
A5000
4 Titan Pascal
Class Balance Sampling
No
No
No
Yes
Yes
Mixup
No
No
Yes
Yes
Yes
Random Time Shifting
Yes
Yes
Yes
Yes
Yes
Loss Function
BCE
BCE
CE
Weight Averaging
No
No
Yes
Yes
Yes
Ensemble
No
No
No
No
No
Input Norm Mean
-5.011
-5.081
-5.081
-5.011
-5.011
Input Norm STD
4.485
4.485
4.485
4,485
4.485
C VISUAL ACTION RECOGNITION EXPERIMENTS
In addition to the audio-visual event classification task on AudioSet and VGGSound, we also test our models on audio-visual action recognition tasks. One problem with existing audio-visual action recognition datasets is they are usually visual-heavy and dominated by the performance of the visual branch. Therefore, to test our audio-visual model, we choose to conduct experiments on Kinetics-Sounds (Arandjelovic & Zisserman, 2017), subset of Kinetics-400 dataset (Kay et al.) 2017) with 3 human action classes that have been chosen to potentially manifested visually and aurally.
We conduct two experiments on Linetics-Soundss
First, we pretrain and fine-tune CAV, AV-MAE, and CAV-MAE using the Sinetics-Soundee training set and report the Top-1 accuracy on the Sinetics-Sounde validation set (i.e., no AudioSet pretrain- ing). This is to check if CAV-MAE still outperforms its counterparts recognition task. As shown Table 5] the conclusion on Kinetics-Sounie the audio-visual action AudioSet and VGGSound, i.e., CAV-MAE performs better than both CAV and AV-MAE.
consistent with that on
Second, we compare CAV-MAE models with SOTA MBT model (Nagrani et al. 2021 following the protocol of MBT. Specifically, train the model on Kinetics-400) (K400) dataset and report the top-1 accuracy on Kinetics-Sounds. We find the label set used impacts the accuracy and this setting clear in the MBT paper. Therefore, we report the results on both the Kinetics-400 label set (i.c., not restrict predictions in 32 Kinetics-Souna classes) and the Kinetics-Souma label set (i.c., restrict predictions in 32 Kinetics-Souni Sounds classes). As shown in Table 6 our CAV-MAE matches MBT on Kinetics- Sounds. Please note that our CAV-MAE model is trained in a fully self-supervised manner while MBT uses supervised ImageNet pretrained weights. For the difference between ImageNet
The original Kinetics-Sounds dataset consists of 34 classes with early version of Kinetics-400 label set. We contact the authors and the 32-class label defined in (Xiao al. 2020) for our experiments.
14

---

Published as a conference paper at ICLR 2023
supervised learning (SL) model and self- self-supervised learning (SSL) model initialization, please Table 11
Table 5: Comparison CAV, AV-MAE, and CAV-MAE models on Kinetics-Sounds. For each model, pretrain and fine-tune using the Kinetics- training set and report the Top-1 accuracy on the Kinetics-Soundl validation set. The conclusion is consistent with our AudioSet and VGGSound experiments that CAV-MAE outperforms both CAV and AV-MAE.
Kinetics-Sounds Accuracy
CAV AV-MAE CAV-MAE
86.2
88.0
88.9
Table 6: Comparison of CAV-MAE models with SOTA MBT model (Nagrani et al. 2021 on KEnetics-Sounds. Following the protocol of MBT, we train the model on Kinetics -00tics-4000 (K400) dataset and report the top-1 accuracy Kinetics-Sounds We report the results both the Kinetics- 400 label set (i.e., not restrict predictions 32 Kinetics-Sounds classes) and the Senetics-Sounde label set (i.e., restrict predictions in 32 Kinetics-Sounds classes). Our CAV-MAE matches or out- performs MBT on Kinetics-Sounde with a fully sulf-super learning (SSL) setting.
Out-of-Domain Pretrain
In- Domain Training
K400
Kinetics-Sounds Label Set
Label Set
MBT CAV-MAE
ImageNet SL
K400 SL
85.0
No
K400 SSL + SL.
70.6
83.3
CAV-MAE CAV-MAE
ImageNet SSL
K400 SSL + SL
83.2
90.6
ImageNet + AudioSet SSL
K400 SSL + SL
85.0
90.9
D ADDITIONAL AUDIO-VISULA RETRIEVAL RESULTS.
D.1 AUDIO TO VISUAL RETRIEVAL RESULTS ON AUDIOSET AND VGGSOUND
We show audio to visual retrieval results on AudioSet and VGGSound (zero-shot) in Table
Table 7: Audio to visual retrieval results on AudioSet and VGGSound.
Audio-+Vine Visual Retrieval
AudioSet Eval Subset
VGGSound Eval Subset
R@1
R@5
R@
10
R@1
R@5
R@ 10
Audio- Visual Models with Only MDM Loss
Vanilla AV-MAE
0.2
0.4
0.9
0.0
0.4
0.8
AV-MAE
0.2
0.4
0.9
0.0
0.2
0.6
Audio- Visual Models with Only Contrastive Loss
CAV, A, 0.1
15.5
32.7
42.8
12.4
33.2
44.7
CAV, A = 0.01
11.5
27.5
36.5
10.0
25.6
36.9
Constrastive Audio- Visual Masked Auto-Encoders
CAV-MAE, X = 0.1
13.5
32.5
43.2
12.1
31.6
42,4
CAV-MAE, X_ 0.01
9.5
22.6
32.4
8.3
23.8
32.4
CAV-MAE(Scale), C=0.01
15.1
34.0
43.0
12.8
30.4
40.3
D.2 VGGSOUND RETRIEVAL SAMPLES
We show bi-directional zero-shot VGGSound retrieval samples in Figure and Figure
15

---

Published as a conference paper at ICLR 2023
Table 8: Audio-visual bi-directional retrieval results on MSR-VTT dataset. All models, includ- ing the baseline models, are initialized with ImageNet weights and trained with only MSR-V VTT data. Our CAV and CAV-MAE models outperform existing methods both directions. In addition, comparing CAV and CAV-MAE, we again find the MAE training objective does not hurt, or even improve the retrieval performance.
Audio Visual
-Auioo
R@I
R@5
R@
10
R@D
R@5
R@
Random
0.1
0.5
0.1
0.5
Boggust et al. (2019)
1.0
3.8
7.1
1.8
4.5
8.1
Arandjelovic & Zisserman 2018)
1.3
4.3
8.2
0.3
2.5
6.6
AVLnet (Rouditchenko et al. 2021
0.9
5.0
9.0
0.8
4.6
8.1
CAV, A 0.1
0.2
4.8
10.4
1.9
9.6
14.9
CAV-MAE, > = 0.1
1.5
8.0
12.4
2.6
9.2
13.1
Table Zero-shot audio-visual bi-directional retrieval results on MSR-VTT dataset. Existing meth- ods trained with the 100M How T00M dataset, while models only trained with the 2M AudioSet dataset. With less than 2% of pretraining data, our CAV-MAE model achieves similar re- sults for visual-audio retrieval performance with existing methods. Again, CAV-MAE models have similar or better results compared with CAV models when A₂ is the same.
Pretrain Dataset
Audio Visual
Asuall+Audio
R@1
R@5
R@ 10
R@1
R@5
R 10
Boggust et al. (2019)
HowTo110M
7.6
21.1
28.3
9.3
20.7
28.8
Arandjelovic Zisserman 2018
HowTol T0wTo100M
12.6
26.3
33.7
11.9
25.9
34.7
AVLnet (Rouditchenko et a. 2021
How HowTo100M
17.8
35.5
43.6
17.2
26.6
46.6
CAV, A 0.1
AudioSet-2M
6.2
17.9
26.1
10.5
25.2
35.5
CAV-MAE, A = 0.1
AudioSet-2M
7.0
18.7
28.6
10.0
26.5
38.0
CAV, X. 0.01
AudioSet-2M
4.2
14.8
22.7
6.9
22.5
33.1
CAV-MAF, A 0.01
AudioSet-2M
4.9
14.6
21.9
8.3
23.5
35.0
CAV-MAE Ao 0.01
AudioSet-2M
7.6
19.8
30.2
13.3
29.0
40.5
D.3 MSR-VTT VT DATASET RETRIEVAL EXPERIMENTS
We also conduct audio-visual retrieval experiments MSR-VTT (Xu al. (016) and compare models with existing works. Specifically, we conduct two sets of experiments.
First, train CAV and CAV-MAE models the MSR-VTT training and evaluate them the MSR-VTT test set. Note the models are not pretrained on AudioSet. We then compare the retrieval performance with existing works the same training setting. As shown Table 8 our CAV and CAV-MAE models outperform existing methods in both directions. In addition, comparing CAV and CAV-MAE, again find the MAE training objective does not hurt, or even improve the retrieval performance.
Second, we conduct zero-shot retrieval experiment on MSR-VTV Specifically, we take the Au- dioSet pretrained models and directly evaluate them on the MSR-VTT test set. The MSR-VTT training set is not used. We then compare our models with existing models. As shown in Table g our CAV-MAE model achieves similar results for visual-audio retrieval performance with existing methods but worse for the audio-visual direction. However, existing methods are trained with the 100M HowTo100M dataset, while our models are only trained with the 2M AudioSet dataset. With less than 2% of training data, our CAV-MAE model achieves similar results for visual-audio re- trieval performance with existing methods. Again, CAV-MAE models still have similar better results compared with CAV models when X₂ is the same, demonstrating the MAE and contrastive objective do not conflict.
16

---

Published as a conference paper at ICLR 2023
E IMPACT OF MODEL INITIALIZATION
Existing audio-visual models typically use (supervised) ImageNet pretrained weights to initialize the model. Throughout the paper, always initialize models (including CAV, AV-MAE, and CAV-MAE) with self-supervised ImageNet pretrained weights. Specifically, we use the weight from the original vision MAE model (He al. 2022) (Weights from htttt:://ggttiiiii
com/ mae) with only sell-supervised learning (SSL) pretraining for all audio. visual, and joint encoder and the decoder. This is implemented by duplicating the weights of MAE encoder layer 1-11 for the audio and visual encoder. respectively, and the weights of MAE encoder layer for the joint encoder.
facebookresearco
How important is this initialization? We conduct experiments with various model initialization and pretraining settings. As shown in Table [O] we find that ImageNet initialization always leads performance improvement, decreases with a larger in-domain pretraining dataset, e.g., without ImageNet initialization, CAV- MAE performs just 1.0% mAP lower on AudioSet-2M. Therefore, ImageNet initialization is not an indispensable component of the proposed CAV-MAE pretraining framework.
matter in fine-tuning linear probing test, and such improvement
Finally, quantify the difference between initialing the model with ImageNet SSL pretrained weights and ImageNet SL pretrained weights on the downstream task. As shown Table IT] on AudioSet-20Ks using SL weights leads to 3.7% improvement over using SSL weights in the fine- tuning setting (but interestingly. in the linear probing setting, SL weights lead to worse results). Therefore, directly comparing our fully self-supervised model with existing models with super- vised pretraining component is not exactly fair.
Table 10: CAV-MAE model performance with various model initialization and pretraining settings on AudioSet-20K, VGGSound, and AudioSet-2M. We report both end-to-end fine-tuning and linear probing results. Initializing CAV-MAE with ImageNet pretrained weights consistently improves the model performance, but is not indispensable component. Without ImageNet initialization, CAV- MAE performs just 1.0% mAP lower on AudioSet-2M.
Settings
AudioSet-20K
VGGSound (200K)
AudioSet-2M
ImageNet AudioSet Pretraining
Fine Tuning
Linear Probing
Fine Tuning
Linear Probing
Fine Tuning
Initialization
No
No
8.0
2.4
42.4
10.3
33.5
SSL
No
25.6
10.3
62.1
34.3
47.3
No
SSL
37.3
29.1
62.7
53.0
49.5
SSL
SSL
40.6
29.8
65.4
54.2
50.5
Table 11: Most existing audio-visun models initialize their weights with ImageNet supervise pre- trained weights (e.g.. Nagrani et al. (202T); RRouditchenko et al. (202T)) while we intend to build a fully self-supervised model to avoid using any labels. Compare the AudioSet-20K performance of models initialized with ImageNet supervised pretrained (SL) weights and ImageNet self-supervised pretrained (SSL) weights. The SL weights and SSL weights are from the original MAE models with and without supervised ImageNet finetuning (He et al. 2022), respectively. Since the SL weights only contain weights of the MAE encoder part and cannot be used for further SSL pretraining. We directly fine-tum tene//neellenrr probe the two models on AudioSet-20K, (i.e., no in-domain pretraining) and report the results make fair comparison. We observe that initialing the model SL weights leads to a noticeable advantage for fine-tuning, showing the ImageNet labels are still very valuable su- pervision signals. This also indicates that directly comparing our fully supervisd model with existing models with a supervised pretraining component is not exactly fair.
AudioSet-20K
Initialization Weight
Fine Tuning
Linear Probing
ImageNet SSL ImageNet SL
25.6
10.3
29.3
7.2
17

---

Published as a conference paper at ICLR 2023
F IMPACT OF MASKING STRATEGY AND MASKING RATIO
F.1 IMPACT OF TRAINING MASKING RATIO
Throughout the paper, we use 75% masking ratio for both audio and visual input. This mainly due to many previous MAE works reporting masking ratio ~75% is appropriate for both audio and visual input He et al. (2022); Baade et al. (2022): Huang et al. (2022a); Niizumi et al. (2022). However, it is unclear if such a high masking ratio is also appropriate for the contrastive objective. In particular, aggressive augmentation
not commonly used in audio-visual contrastive learning. Therefore, we conduct experiments to check the impact of the training masking ratio on the audio- visual joint event classification task and the audio-visual retrieval task.
For the audio-visual joint event classification task, shown in Table 12. find the CAV model does perform slightly better with smaller masking ratio (50%), but the difference is minor. When the masking ratio is 75%, CAV still performs well. This shows the audio-visie joint classification task not sensitive to the masking ratio.
For the audio-visun visual retrieval task, shown in Table 13 performance decreases with higher masking ratio, particularly when the masking ratio is very high. If audio-visur retrieval is the main task of interest, a lower masking ratio should be used in training, which does not hurt the audio-visual joint event classification task, but requires more computation. In Section and Appendix D we show CAV-MAE already strong audio-visual retrieval model when the masking ratio is 75%, the performance can be further improved by lowering the masking ratio. Note this result does not conflict with the fact that the reconstruction objective does hurt, and in many cases, improves the retrieval performance.
find that the audio- retrieval
Audio-visual joint event classification performance of CAV, AV-MAE, and CAV-MAE as a function of masking ratio on AudioSet-20K and VGGSound. All models are pretrained with uniform unstructured masking. We find the contrastive learning model CAV performs slightly better with a lower masking ratio while the AV-MAE model performs best with ~75% masking ratio. These results show that 65%~75% masking ratio works well for both contrastive learning and masked data modeling frameworks for the downstream audio-visual joint event classification task.
Table 12:
AudioSet-20K
VGGSound
Masking Ratio
CAV
AV-MAE
CAV-MAE
CAV
AV-MAE
CAV-MAE
0.50
38.5
37.2
41.2
64.3
63.7
65.1
0.65
38.6
37.9
41.3
64.0
64.1
64.9
0.75
38.5
37.4
40.5
64.1
64.1
65.4
0.85
38.1
37.3
40.3
63.3
64.2
64.7
Table 13: Zero-shot audio-visual retrieval performance of CAV-MAE (A, = 0.01) as function of masking ratio on VGGSound evaluation subset. All models are pretrained with uniform unstructured masking. The audio-visual retrieval performance decreases with higher masking ratio.
Audio Visual
Aesual-AAdioo
Masking Ratio
R@1
R@5
R@
R@I
R@5
R@ 10
0.50
14.8
34.9
45,4
15.2
36.9
48.3
0.65
11.3
28.9
39.0
14.2
33.5
45.2
0.75
8.3
23.8
32.4
12.5
28.6
39.1
0.85
5,6
16.0
22.7
8.7
22.6
30.6
F.2 IMPACT OF AUDIO TRAINING MASKING STRATEGY
Another key design of masking is the masking strategy. Throughout the paper, uniform, unstructured masking strategy for both audio and visual input. However, unlike visual modalities, the two dimensions of audio spectrograms are heterogeneous. In this section, explore the impact
18

---

Published as a conference paper at ICLR 2023
of masking strategies for audio input. Specifically, we apply time, frequency, and time- fene-freuenccy masking strategies (depicted in Figure 3) and compare them with the uniform unstructured masking strategy (i.c., uniform masking).
For the audio-visual joint event classification task, as shown in Table [4 we find that all four training masking strategies lead to similar performance when the training masking ratio is 75%. However, as we show in Figure [ structured masking strategies make reconstruction more challenging. There- fore, also pretrain CAV-MAE model trained with time-frequency masking at lower masking ratio of 50%, which shows slightly better performance on both AudioSet-20K and VGGSound. In general, the mudio-visual joint classification task is not sensitive the masking strategy.
For the audio-visual retrieval task, as shown in Table 15 with the same 75% masking ratio, different masking strategies lead noticeably different retrieval performance. Frequency and time-freee frequreyy masking leads to the best retrieval performance while unstructured uniform masking actually leads the worst retrieval performance. In Section 5 and Appendix [D] we show CAV-MAE already strong audio-visual retrieval model when uniform masking is used, the performance can be further improved by using a structured masking strategy, which also does not hurt audio-visual joint event classification.
To summarize, we find both the masking ratio and masking strategy have minor impact on the downstream audio-visual joint event classification task, but have noticeable impact the audio- visual retrieval task. Specifically, there exist masking strategies that lead to better retrieval perfor- mance than the default 75% uniform masking strategy. Finally, also notice the training masking strategy impacts the model reconstruction ability, which is discussed Section
Frog
Time
(A) Time Masking
(B) Frequency Masking
Unmasked
Propercc
Festuescy
Masked
(C) Free-Freyyency Masking
(D) Uniform Masking
Figure 3: Illustration of various masking strategies. We use uniform unstructured masking through- out the paper except Section F
Table 14: Audio-visual joint event classification performance of CAV-MAE as a function of training masking strategy and ratio on AudioSet-20K and VGGSound. We find that all four training masking strategies lead to similar performance when the training masking ratio is 75%. However, as we show in Figure structured masking strategies make reconstruction more challenging. Therefore, we also pretrain a CAV-MAE model trained with time-frequeng masking at a lower masking ratio of 50%, which shows slightly better performance on both AudioSet- and VGGSound.
Masking
AudioSet-20K
VGGSound
Strategy
Ratio
Uniform
0.75
40.5
65.4
Time
0.75
40.6
64.6
Frequency
0.75
40.4
65.0
Time-Freque Free-Frrey
0.75
40.7
64.8
Lime-Frequency
0.50
41.2
65.6
19

---

Published as a conference paper at ICLR 2023
Table 15: Zero-shot audio-visual retrieval performance of CAV-MAE (A. 0.01) function of training masking strategy on VGGSound evaluation subset. All models are trained with masking ratio of 75% on AudioSet. The masking strategy has noticeable impact on retrieval performance.
Audio Visual
Visual Audio
Masking Ratio
R@1
R@5
@@
10
R
R@5
R 10
Uniform Time
8.3
23.8
32.4
12.5
28.6
39.1
10.1
26.0
35.5
12.5
30.2
40.3
Frequency
11.7
31.9
42.7
13.7
34.7
45.8
Time-Frrquency
13.1
32.8
42.1
14.7
36.4
47.3
G IMPACT OF THE NUMBER OF FRAMES USED
In the paper, we sample 10 frames for each 10-second video clip (1 FPS). How does the frame rate impact the performance? As shown in Figure on all LineticsSSundee AudioSet-20K,, and VGGSound, higher FPS consistently improves the downstream classification performance, however, improvement saturates with the increasing of frames.
Kinetícs-Soud Sounds
AudioSet-20K
VGGSound
91.1
42.0
Classification Performance
91.0
65.8
41.8
90.8
65.6
90.7
41.4
65.4
90.6
90.5
41.2
65.2
90.4
41.0
90.3
65.0
5
10
15
25
30
10
10
Frame Used
Frame Used
Frame Used
Figure 4: Classification performance function of the number of frames used Kinetics-Soundh (left), AudioSet-20K (middle), and VGGSound (right). Frames are uniformly sampled from each video clip. The performance consistently improves with more frames being used, but the improve- ment saturates with the increase of frames.
H CAV-MAE RECONSTRUCTION RESULTS
H.1 AUDIO-VISUAA RECONSTRUCTION SAMPLES
We show the CAV-MAE reconstruction samples in Figure 10 and 11 All samples are from VGGSound, different dataset from the pretraining set. The CA A-MAEE model is trained with 75% masking ratio without target normalization. As shown in Table 2d it has a similar performance to the default model with target normalization. CAV-MAE has strong reconstruction ability even if the masking ratio goes to 90%, which makes it potentially can be used for in- in-painting and enhancement tasks. All inference masks are sampled uniformly (i.e., unstructured masking).
H.2 AUDIO SPECTROGRAM RECONSTRUCTION UNDER VARIOUS INFERENCE MASKING SETTINGS
Besides uniform masking samples shown in the previous section, also show the audio spectro- gram reconstruction samples under various structured inference masking settings in Figure 12 (75% masking ratio) and Figure (90% masking ratio). We find structured masking is more challenging
20

---

Published as a conference paper at ICLR 2023
Trained w/ Unifarm Masking
w/ Time-Frequency Masking
Masking
Frequency Masking
Frequency Masking
Masking
0.40
0.40
0.40
Reconstructio Mean Square Emrr
0.35
0.35
0.35
0.35
0.30
0.39
0.30
0.30
0.25
0.25
0.25
0.25
0.20
0.29
0.20
0.20
0:15
0.15
0.15
0.15
0.10
0.10
0.10
0.10
0.05
2.05
07
035
Maasinng Ratio (%)
Masking Ratio (%)
Maskingg Rablo (%)
Maskiing Ratio (%6
Figure 5: Audio spectrogram reconstruction mean squared error (MSE) as function of masking ratio under various inference masking settings (from left to right: time masking, frequency mask- ing, time-frequency masking, and uniform unstructured masking). We compare a CAV-MAB model trained with uniform masking (blue) and CAV-MAE model trained with time- freeffreyency masking (red). Both models are trained with a 75% masking ratio. Key findings are as follows: 1) Even for the same masking ratio, the reconstruction hardness is different for each masking strategy. On av- erage, time masking is the most difficult, followed by frequency masking, time-frequency masking, and uniform unstructured masking. This indicates that CAV-MAE models require local information for the reconstruction task. However, for each specific spectrogram, the order of difficulty varies (see Figure 12 and 13 Second, the CAV-MAE model trained with time-frequency masking generally performs better than its counterpart trained with uniform masking in audio spectrogram reconstruc- tion, particularly for the time masking and frequency masking settings, showing it is stronger in leveraging global information. This indicates different training masking strategies do impact the properties the model.
for reconstruction the mean squared errors are generally higher. On average, time masking is the most difficult, followed by frequency masking, time-freyuency masking, and uniform unstructured masking. This also indicates that the model leverages local neighboring unmasked part information to infer the masked part. When an entire time or frequency span is masked, the model is harder to reconstruct (this is quantified in Figure 5).
Finally, in Figure 12 and Figure 13 we also compare the reconstruction ability of a CAV-MAE model trained with uniform, unstructured masking strategy and a CAV-MAE model trained with fimefrrequency masking strategy (both with 75% masking ratio). We quantify the difference in Fig- find the CAV-MAE model trained with time-frequency masking generally
ure [ Interestingly, performs better than its counterpart trained with uniform masking in audio spectrogram reconstruc- tion, particularly for the time masking and frequency masking settings, showing it is stronger in leveraging global information. This indicates different training masking strategies do impact the properties the model. While the training masking strategy only minorly impacts the downstream classification task, it has relatively large impact on reconstruction.
I CAV-MAE VISUAL SOUND SOURCE LOCALIZATION RESULTS
We evaluate the capability of CAV-MAE (uniform masking, masking ratio 75%, A,=0.01) on the visual sound source localization task with basic similarity-basee method. Specifically, for each audio-image pair, we mean pool the representations of all audio tokens as the clip-level audio rep- resentation, and then calculate the cosine similarity between the clip-level audio representation with all patch-level image representations as the visual sound source localization heat map. In general, find the CAV-MAE model is not strong visual sound source localization model though its audio-visual retrieval performance is good. Figure 6 we show a successful sample (left) and a failed sample (right). In some cases, CAV-MAE localizes the sound to the background instead of
21

---

Published as a conference paper at ICLR 2023
the main sound source object. We hypothesize that it is due to the masked contrastive learning ob- jective. During the training process, the model needs to match positive audio-visual pairs even when both modalities are heavily masked, in situations, the main sound source could be completely masked, the model thus learns to leverage the context information for the matching, which may hurt its performance the visual sound source localization task.
Input
Heatmap
Input
Heatmap
iean Crow
Figure 6: A successful sample (left) and a failed sample (right) CAV-MAE on the visual sound source localization task. In some cases, CAV-MAE localizes the sound to the background instead of the main sound source object.
J IMPACT OF THE AUDIO VISUAL PAIRING INFORMATION IN TRAINING DATASET
Even without contrastive objective, AV-MAE allows the model to reconstruct one modality based on the information of another modality, which theoretically allows the model to learn audio-visual correlation. However, without an explicit objective encouraging paired audio-visual correspon- dence, to which extent the AV-MAE leverages the audio-visual pairing information is unknown. In this section, we evaluate this by the following experiment: we break the original audio-visual pairs of the training set and conduct random shuffle (i.e., randomly match audio and visual samples in the dataset), which removes most the audio-visual pairing information in the training data. We train the CAV, AV-MAE, and CAV-MAE models with the shuffled training dataset, and then finetune these models on the audio-visual joint event classification task with original unshuffled downstream datasets. As shown Table 16 we find the CAV model that solely relies on the audio-visual pairing information has significant performance drop when the training dataset is shuffled;; 2) the AV-MAE model is almost not impacted by the training set shuffle, indicating it weak at leveraging audio-visusa pairing information in the training set and mostly relies on singlc-modality information;
3) CAV-MAE performs almost the same with AV-MAE with the shuffled training set, but noticeably better with the original training set. These findings again justify the main point of this paper that contrastive and reconstruction objectives are most effective when they are combined together. When only the contrastive objective is used, the model performs worse and is less robust to the noise in the training set. When only the reconstruction objective is used, the model does not effectively leverage the audio-visual pair information.
Table 16: Comparing the audio-visuue joint event classification performance of models trained with AudioSet with original audio-visual pairs and AudioSet with randomly shuffled audio-visual pairs.
20
VGGSound
Training Data
CAV
AV-MAE
CAV-MAE
CAV
AV-MAE
CAV-MAE
Shuffled 2M
1.25
37.4
37.4
7.28
64.1
64.1
Original AudioSet-2M
38.5
37.4
40.5
64.1
64.1
65,4
22

---

Published as a conference paper at ICLR 2023
Top video
Top 2 Video
Top 3 Video
Top 4 Video
Video
Query Audio
Query Audio
Top 1 Video
Top Video
Top Video
Top 4 Video
Top Video
Top 2 Video
Top Video
Top Video
Query Audio
Query Audio
Top Video
Top Video
Too Video
Top Video
Top 2 Video
Top video
Top Video
Video
Quary
Query
Top Video
Top 2 Vidco
Top 3 Video
Top Video
Too Videoo
Top Video
Top Video
Video
Quary Audio
Audio
Top
2
Top
Top Video
Top Video
Top 2 Video
Top Video
Top Video
Query Audio
Query Audio
Figure 7: Zero-shot audio image retrieval results VGGSound. Since the spectrograms are hard to read, we show their paired images in the dashed boxes for visualization purposes, only audios are used queries.
23

---

Published as a conference paper at ICLR 2023
Query
Quary Video
Top Audio
Top 2 Audio
Top 3 Audio
Top 4 Audio
Top Audio
Top Audio
Top 3 Audio
Tap Audio
Query Video
Query Video
Top Audio
Top 2 Audio
Top Audio
Top Audio
Top Audio
Top Audio
Top Audio
Tap Audio
Query Video
Query Video
Audio
Audio
Audio
Audio
Audio
Query
Query
Top Audio
Top Audio
Top Audio
Top 4 Audio
Top Audio
Top Audio
Top 3 Audio
Top Audio
Query
Query
Top Audio
Top Audio
Top Audio
Top Audio
Audio
Audio
Audio
Audio
Figure 8: Zero-shot image to audio retrieval results on VGGSound. Since the spectrograms are hard to read, we show their paired images in the dashed boxes for visualization purposes, only audios are used as keys.
24

---

Published as a conference paper at ICLR 2023
Original Image
Masked Image
Reconstructed Image
Original Spectrogram
Masked Spectrogram
Rec Reconstructee ed Spectrogratt am
Original Image
Masked Image
Reconstructed Image
Seund SianK)
Original Spectrogram
Spectrogram
Reconstructed Spectrogram am
Original Image
Masked Image
Reconstructed Image
Original Spectrogram
Masked Spectrogram
Reconstructed Spectrogram
Original Image
Masked Image
Reconstructed Image
Original Spectrogram
Masked Spectrogram
Reconstructed Spectrogram
Figure 9: CAV-MAE reconstruction samples when 50% of the input is masked. Samples are from VGGSound, different dataset from the pretraining dataset. The model is pretrained AudioSet with a 75% masking ratio without target normalization.
25

---

Published as a conference paper at ICLR 2023
Original Image
Masked Image
Reconstructed Image
Original Spectrogram
Masked Spectrogram
Reconstructed Spectrogram
Original Image
Masked Image
Reconstructed Image
Original Spectrogram
Spectrogram
Reconstructed Spectrogram
Original Image
Masked Image
Reconstructed Image
Original Spoctrogram
Masked Spectrogram
Reconstructed Spectrogram
Original
Reconstructed
Original Spectrogram
Masked Spectrogram
Reconstructed Spectrogram
Figure 10: CAV-MAE reconstruction samples when 75% of the input is masked. Samples are from VGGSound, a different dataset from the pretraining dataset. The model is pretrained on AudioSet with 75% masking ratio without target normalization.
26

---

Published as a conference paper at ICLR 2023
Original Image
Masked Image
Reconstructed Image
Original Spectrogram
Masked Spectrogramt am
Reconstructed Spectrogram
Original Image
Masked Image
Reconstructed Image
Scund (Dwn
original Spectrogram
Masked Spectrogram
Reconstructed Spectrogram
Original Image
Masked Image
Reconstructed Image
Original Spectrogram
Masked Spectrogram
Reconstructed Spectrogram
Original Image
Masked Image
Reconstructed Image
Original Spectrogram
Masked Spectrogram
Reconstructed Spectrogram
Figure 11: CAV-MAE reconstruction samples when 90% the input is masked. Samples are from VGGSound, a different dataset from the pretraining dataset. The model is pretrained on AudioSet with 75% masking ratio without target normalization.
27

---

Published as a conference paper at ICLR 2023
Masking Ratio 75 %
CAY-MAE Trained w/ Uniform Masking
CAV-MAE Trained Time-Frequency Masking
Orig inal Spectrogram
Sample
0.257 Reconstructed Spectrogram
MSE: 0.222 Reconstructed Spectrogram
Spectregram
Time Masking
MSE: 0.577 Reconstructed Spectrogram
MSE: 0,362 Reconstructed Spectrogram
Masked Spectrogram
Prequency Masking Spectregram
MSE: 0.312 Reconstructed Spectrogram
Reconstructed Spectrogram
Tene-Feeueecy Masking
MSE: 0.103 Reconstructed Spectrogram
MSE: 0.127 Reconstructed Spectrogram
Masked Spectrogram
Unitom Masking
Original Spectrogram
Sample
MSE: 0.252 Reconstructed Spectrogram
MSE: 0.235 Recorstructed Spectrogram
Masked Spectrogram
Time Masking
MSE: 0.191 Reconstructed Spectrogram
MSE: 0.204 Reconstructed Spectrogram
Masked Spectrogram
Frequency Masking
MSE: 0.103 Reconstructed Spectrogram
Masked Spectrogram
Reconstructed Spectrogram
Time Free-Fegyynyyy Masking
MSE: 0.079
Masked Spectrogram
Reconstructed Spectrogram
Reconstructed Spectrogram
Uniomm Masking
Original Spectrogram
Sample
MSE: 0.705
Masked Spectrogram
Reconstructed Spectrogram
Reconstructed Spectrogram
Mosking
0.349 Reconstructed Spectrogram
MSE: 0.480 Reconstructed Spectrogram
Spectrogram
Prequency Masking
MSE: 0.179 Reconstructed Spectrogram
Masked Spectrogram
Reconstructed Spectrogram
Tere-Frquercy Masking Spectrogram
MSE: 0.157 Reconstructed Spectrogram
0.166 Reconstructed Spectrogram
Masking
Original Spectrogram
Sample
MSE: 0.062 Reconstructed Spectrogram
MSE: 0.065 Reconstructed Spectrogram
Spectrogram
Mosking
MSE: 0.289 Reconstructed Spectrogram
MSE: 0.150 Reconstructed Spectrogram
Masked Spectrogram
Frequency Masking
MSE: 0.051 Reconstructed Spectrogram
MSE: 0.052 Reconstructed Spectrogram
Masked Spectrogram
Preguencyy Masking
MSE: 0.045 Reconstructed Spectrogram
MSE: 0.052 Reconstructed Spectrogram
Masked Spectrogram
Unitorm Masking
Figure 12: Reconstructed audio spectrograms in various inference masking settings with 75% masking ratio. We compare the outputs of CAV-MAE model trained with uniform, unstructured masking strategy (second column) and CAV-MAE model trained with time-frequency masking strategy (third column). Both CAV-MAE models are trained with 75% masking ratio. Reconstruc- tion mean squared error (MSE) shown above each reconstructed spectrogram.
28

---

Published as a conference paper at ICLR 2023
Masking Ratio 90 %
CAV-MAE Trained w/ Uniform Masking
CAV-MAE Trained wf Time-Frequency Masking
Original Spectrogram
Sample 1
MSE: Reconstructed Spectrogram
MSE: 0.352 Reconstructed Spectrogram
Spectrogram
Time Masking
MSE: 0.724 Reconstructed Spectrogram
Masked Spectrogram
Reconstructed Spectrogram
Pequency Masking
MSE: 0.648
MSE: 0 631 Reconstructed Spectrogram
Masked Spectrogram
Reconstructed Spectrogram
Time Vine-Frequency Masking
MSE 0.184
MSE: 0 239 Reconstructed Spectrogram
Masked Spectrogram
Reconstructed Spectrogram
Uniform Masking
Original Spectrogram
Sample 2
MSE: 0.242 Reconstructed Spectrogram
Masked Spectrogram
Reconstructed Spectrogram
Masking
MSE: 0,260 Reconstructed Spectrogram
MSE: 0,213 Reconstructed Spectrogram
Masked Spectrogram
Frequeniyy Masking
MSE: 0.327 Reconstructed Spectrogram
MSE: 0.265 Reconstructed Spectrogram
Masked Spectrogram
Tine-Froquency Masking
MSE: 0.105 Reconstructed Spectrogram
MSE: 0.118 Reconstructed Spectrogram
Masked Spectrogram
Uhitemm Marking
Original Spectrogram
Sample 3
MSE: 0.865 Reconstructed Spectrogram
MSE: 0.832 Reconstructed Spectrogram
Masked Spectrogram
Time Masking
MSE: 0.485 Reconstructed Spectrogram
MSE: 0.591 Reconstructed Spectrogram
Masked Spectrogram
Prequency Masking
MSE: 0.364 Reconstructed Spectrogram
Masked Spectrogram
Reconstructed Spectrogram
Tine-Frequency Masking
MSE: Reconstructed Spectrogram
0.268
Masked Spectrogram
Reconstructed Spectrogram
Uniform Mashing
Original Spectrogram
Sample 4
MSE: 0.090
0.083 Reconstructed Spectrogram
Spectrogram
Reconstructed Spectrogram
Time Mesking
MSE: 0.245 Reconstructed Spectrogram
Masked Spectrogram
Reconstructed Spectrogram
Frequency Masking
MSE: 0,266 Reconstructed Spectrogram
MSE: 0.100 Reconstructed Spectrogram
Masked Spectrogram
Thile Frequancy Masking
MSE: 0.055 Reconstructed Spectrogram
MSE: 0.063 Reconstructed Spectrogram
Masked Spectrogram
Uniiorm Masking
Figure 13: Reconstructed audio spectrograms in various inference masking settings with 90% masking ratio. We compare the outputs a CAV-MAE model trained with uniform, unstructured masking strategy (second column) and CAV-MAE model trained with time-frequene masking strategy (third column). Both CAV-MAE, models are trained with 75% masking ratio. Reconstruc- tion mean squared error (MSE) shown above each reconstructed spectrogram.
29


Reviews:
[{"review_id": "g42mPTq-Pt3", "reviewer": "Reviewer_qVEJ", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a multi-modal Masked Auto-Encoder for audio-visual tasks. More specifically, the proposed method combines contrastive learning and masked data modeling for better joint and coordinated audio-visual representation. The model based on self-supervised training achieves a new SOTA on VGGSound and great performance on AudioSet.", "summary_of_the_review": "This paper combines two powerful self-supervised methods, MAE and contrastive learning, and extends the combination to the audio-visual domain.  The overall pipeline is easy to understand, and the paper is clearly written. However, this paper contributes little to the relative areas, including self-supervised learning and multi-modal learning. And this method is not novelty enough.", "strength_and_weaknesses": "### Strengths\n\n- The proposed method leverages multi-modal information well with two hot self-supervised ways, contrastive learning, and mask auto-encoder. The method achieves great performance on audio-visual benchmarks.\n- The designed method combining contrastive learning and multi-modal MAE is promising for multi-modal tasks.\n- Sufficient ablation results are provided.\n- This paper is well-written and easy to follow.\n\n### Weaknesses\n\n- This work is not novel enough. It is well-known that contrastive learning and MAE are powerful self-supervised methods, and applying them to a new domain based on relative works is simple yet effective. This paper extends the combination of the two approaches to the multi-modal learning literature, but the architecture is trivial. Besides, this work focuses on the audio-visual task. However, no domain knowledge is leveraged in this work, and the pipeline can be the same as those focusing on text-image multi-modal learning.\n- Though lots of experimental results are shown in the paper, the work only focuses on audio-visual event classification and audio-visual retrieval. More relative experiments should be conducted.\n- In section 2.3, the authors list some key designs of CAV-MAE. However, these points are all based on the common sense of contrastive learning, MAE, or multi-modal learning. Overall, the framework's design lacks novelty, and the application of the self-supervised methods is so plain. Therefore, the framework's design can be deemed as simple incremental work.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "PTyDCMT_Og", "reviewer": "Reviewer_M9QZ", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors extend the Masked Auto-Encoder (MAE) model from a single modality to audio-visual multi-modalities and combine it with contrastive learning for self-supervised audio-visual learning. Although the two self-supervised learning (SSL) strategies are not new, the proposed Contrastive Audio-Visual Masked Auto-Encoder nicely unifies them in one single framework. Extensive experimental results demonstrate that the two SSL methods are mutually beneficial and the pre-trained models can boost audio-visual classification and cross-modal retrieval performance. ", "summary_of_the_review": "The proposed  CAV-MAE is technically sound and extensive experiments can validate the superiority of the proposed method. I only have a few concerns about the proposed method and I will be happy to upgrade my rating if the authors can address the concerns during the discussion phase. \n\n***Post-rebuttal***\n\nThe new experiments and analysis can well address my concerns. I have increased my rating to 8 from 6.", "strength_and_weaknesses": "Pros:\n\n+ The proposed Contrastive Audio-Visual Masked Auto-Encoder (CAV-MAE) can nicely unify MAE and contrastive learning for self-supervised audio-visual learning. In CAV-MAE, unimodal and joint representations are utilized for contrastive and MAE learning.\n\n+ Extensive experimental comparison and ablation studies on AudioSet and VGGSound can validate the effectiveness of the proposed CAV-MAE.\n\n+ I enjoy reading this paper. It is quite easy to follow. The analysis paragraphs can well validate and explain the proposed method. \n\nCons:\n\n- Not all visual objects make sounds and not all sound sources are visible. The audio-visual mismatch may introduce a lot of false positive audio-visual pairs for audio-visual self-supervised learning. Whether the proposed method can mitigate the data issue? How does the issue affect MAE and contrastive learning respectively? I suggest the authors add discussions on this. \n\n- I was impressed by the reconstructed spectrograms when 75% of the input is masked. Why does audio MAE work? For visual scenes, objects are grouped in certain regions. But spectrograms have quite isolated TF patterns. But, it seems that MAE works for both modalities. Really hope to hear more clarifications and discussions. \n\n \n\n\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "GVOR7ninvpJ", "reviewer": "Reviewer_Ee8Z", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "A contrastive audio-visual masked autoencoder model is introduced in the paper. The training of self-supervised model is done with both contrastive and masked-autoencoder losses.\n\nThe model architecture is transformer and only unmasked inputs are kept as input for the encoder. There is a separate encoder for each modality as well as a joint encoder. The joint encoder can be run for one or both modalities (with its corresponding layer-norms). In the case when the model is fine-tuned for one of the modalities, the joint decoder would only be run with only that modality input.", "summary_of_the_review": "The paper is an interesting read which generalized masked-autoencoders to multiple modalities of audio and video. In addition a contrastive loss is also used. \n\nRelevant ablations are done by comparing with a vanilla VAE which does not have a contrastive loss. The models are used in classification and retrieval problems and the benefits for CAV-MAE model is shown.", "strength_and_weaknesses": "Strengths:\n\n1. The model seems to be a novel audio-visual self-supervisedly trained autoencoding network that uses both masked autoencoder and contrastive losses. Previous work I believe only used contrastive/coincidence losses for audio-visual learning. Masked autoencoding was only used for each modality separately.\n2. The joint encoder is run separately for each modality for the contrastive loss, but it is run jointly for the masked-prediction loss which makes it possible to use for each modality separately too.\n\nWeaknesses:\n1. The video is only represented with 10 frames from 10 seconds (1 Hz) which seems low. So, the model probably relies on images rather than the motion.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "27DzF_UcXu", "reviewer": "Reviewer_TNd7", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper combines  masked auto-encoder based learning and contrastive learning for self-supervised learning in audio-visual settings. The method referred to as Contrastive Audio-Visual Masked Auto-Encoder learning is used to pre-train the models on audiovisual Audioset dataset and then fine-tuned on Audioset and VGGSound. Performance improvements in the downstream tasks of event classification are shown. ", "summary_of_the_review": "The paper combines masked auto-encoder based learning with contrastive learning. It develops and investigates the method for audio-visual representation learning. \n\n\n1. I believe initializing the weights from MAE models leaves a few things unclear. Performance of the model by training from scratch should be reported. Its unclear how critical such initialization is. Especially, in the current context with MAE based initialization - it becomes a two stage training process. First train each modality with MAE and then combine them using the current approach. \n\n\n2. The downstream tasks are limited and other audio-visual tasks should be included. CUrrently the paper just focuses on event classification on two datasets. These two datasets do not provide a comprehensive understanding of audio-visual representation learning. Diverse set of tasks like audio-visual action classification, Epic Kitchen benchmarks, audio-visual segmentation/localization etc. would be able to provide a clearer picture of how good the SSL approach is. In fact, Audioset is not the best dataset for audio-visual learning of sounds. A large portion of visual data in Audioset is just redundant (black frames, audio overlayed over random non-relevant videos and so on). Moreover, Audioset is used for pre-training as well. So finetuning results especially on the full Audioset is not an ideal set of experiments. \n\n\n\n3. Since the tasks are primarily audio tasks, references to some missing audio and speech SSL works are missing (like Wav2Vec and Hubert). [R1] in particular used wav2vec like learning for non-speech audio representation learning and has results on Audioset and several other audio tasks. \n\n\n\n4. In Section 4.3, authors comment on computational efficiency of the proposed method. I am not sure those arguments are convincing. The models are exactly the same and the computational expense of these models will be the same. \n\n\nR1: Conformer-Based Self-Supervised Learning for Non-Speech Audio Tasks in IEEE ICASSP 2022. \n", "strength_and_weaknesses": "Strength\n1. The proposed approach is simple and logical. It tried to combine strengths of masked data modeling and contrastive learning and applied it to audio-visual settings where they have not been explored. \n\n\n2. The paper is very neatly written and easy to read and follow. \n\n\n3. The results on the datasets used show good performance improvements. \n\n\n\nWeakness\n\n\n1. A more exhaustive set of audio-visual tasks would have been better. \n\n\n2. Some other weaknesses are pointed out in the review below. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "00FfjOti4Pt", "reviewer": "Reviewer_oCcH", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper extends MAE from single modality to audio and visual modalities by combining contrastive learning and masked prediction pretext tasks. Experiments show the improvements on VGGSound dataset and better retrieval score. \n", "summary_of_the_review": "In short, the paper presents a valid idea, but the experiments could be augmented and extended to further strengthen the arguments.", "strength_and_weaknesses": "Strength:\nWith the advent of MAE and audioMAE, it is natural to extend such framework to multimodal settings.\nThe motivation for such idea is strong, and the 3 discussion points are valid given the landscape of the current SOTA.  \n\nWeakness:\nFirst, the major logic is to show performance improvement brought by CAV-MAE, and empirically verifies it indeed leads to learning better joint representation. However, CAV-MAE underperforming VS. MBT on multimodal AudioSet is a counter example. Maybe as <Li, et al. 2022 AudioTagging>pointed out, the data quality issue of AudioSet would cause a variation, I would suggest the authors look into the granular errors in AudioSet and report the exact train/test numbers to ensure an apple-to-apple fair comparison.\n\nNormalization of different branches are tuned differently, and this looks to me like an engineering effort. I understand normalization is crucial for transformers tuning, but just wondering if there could be a better way rather than setting a fixed number since this severely limits the generalizability of the model. \n\nI get the point of reporting retrieval score on subsets of AudioSet and VGGSound, however this is not a standard dataset for retrieval benchmark, I would suggest the authors reporting on MSR-VTT in a zero shot way. \n\nIn appendix and throughout the text, I realized the authors adopted the vanilla MAE masking, which is the random visual-intuitive masking, but as <Huang et al. 2022 AudioMAE> pointed out, this is not necessarily the most suitable way of masking for spectrogram, I would encourage the authors to experiment with various masking strategies. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: notable-top-25%