MAViL: Masked Audio-Video Learners
Authors: Po-Yao Huang
Vasu Sharma
Hu Xu
Chaitanya Ryali
Haoqi Fan
Yanghao Li
Shang-Wen Li
Gargi Ghosh
Jitendra Malik
Christoph Feichtenhofer
Venue: NeurIPS 2023 poster (2023)
Keywords: self-supervised learning
audio representation learning
audio classification
TL;DR: SSL via aligning and reconstructing contextualized audio-video representations delivers SOTA performance on 7 audio-visual classification and textless vision-language tasks.

Abstract: We present Masked Audio-Video Learners (MAViL) to learn audio-visual representations with three complementary forms of self-supervision: (1) reconstructing masked raw audio and video inputs, (2) intra-modal and inter-modal contrastive learning with masking, and (3) self-training to predict aligned and contextualized audio-video representations learned from the first two objectives. Empirically, MAViL achieves state-of-the-art audio-video classification performance on AudioSet (53.3 mAP) and VGGSound (67.1\% accuracy), surpassing recent self-supervised models and supervised models that utilize external labeled data. Notably, pre-training with MAViL not only enhances performance in multimodal classification and retrieval tasks, but it also improves the representations of each modality in isolation, without relying on information from the other modality during uni-modal fine-tuning or inference. The code and models are available at https://github.com/facebookresearch/MAViL.

MAViL: Masked Audio Video Learners
Po- Yao Huang'
Vasu Sharma'
Hu Xu'
Chaitanya Ryali' Haoqi Fan' Yanghao Li
Shang-Woe Wen Li Gargi Ghosh"
Jitendra Malik
Christoph Feichtenhofer
FAIR, Meta
University of California, Berkeley
Abstract
We present Masked Audio Video Learners (MAViL) to learn audio-visual repre- sentations with three complementary forms of self-supervision: (1) reconstructing masked raw audio and video inputs, (2) intra-modal and inter-modal contrastive learning with masking, and (3) self-training predict aligned and contextual- ized audio-video representations learned from the first two objectives. Empir- ically, MAViL achieves state-of-the-art audio-video classification performance on AudioSet (53.3 mAP) and VGGSound (67.1% accuracy), surpassing recent self-supervised models and supervised models that utilize external labeled data. Notably, pre-training with MAViL not only enhances performance in multimodal classification and retrieval tasks, but it also improves the representations of each modality in isolation, without relying on information from the other modality during uni-moda fine-tuning or inference. The code and models are available https: / //g thub. wttn//acebookreeearccc//
Introduction
We study self- seffssueervieed learning (SSL. from audio and video, two rich, heterogeneous, yet closely related modalities human perception. There are two primary forms of self-supervision commonly used: reconstruction and contrastive learning. By reconstructing masked text tokens on large-scale corpora, BERT [20] pre-training has achieved groundbreaking results in various NLP tasks. Masked autoencoders (MAEs) have recently emerged as powerful tools for learning uni-modal representations in various modalities such as image [37], video [27], and audio [41]. They employ an asymmetric encoder-decoder architecture with substantial portion of encoder inputs being masked, resulting in efficient uni- uni-modal SSL. Additionally, contrastive learning has been widely used to learn coordinated multimodal representations for image-text [75, 93] and audio-visual [67, 71] tasks.
In this work, we develop a novel approach that integrates MAE and contrastive learning to enhance audio-video representation learning for both audio-video and audio-only tasks. This integration poses unique challenges under the multimodal paradigm, necessitating meticulous model design to effectively manage diverse multimodal inputs in MAE, while also adapting contrastive learning to accommodate masking of the majority of inputs for efficiency. Our method, Masked Audio- Video Learners (MAViL), comprises pair of encoders to encode audio and video, fusion encoder, and distinct decoders tailored for reconstructing raw inputs or contextualized representations in each modality (as illustrated in Fig. 1). We design MAViL with three types of objectives outlined next.
Firstly, we extend uni uni-modal MAE to multimodal and utilize fusion encoder to exchange information from all modalities. MAViL. reconstructs raw inputs that has been removed under a high masking ratio (e.g. 80%). With this, learns complementary audio-video representations by reconstructing single modality input, with the supplementary context from the other. Secondly, we adapt contrastive learning [34, 85], with the majority of uni-modal inputs being masked, learn an aligned audio-vided latent space efficiently. MAViL employs two types contrastive objectives: (i) An inter-modal contrast that brings together paired video and audio clips from the same video and contrasts them with other
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

View
Audio Encoder
Audio Decoder
Intra-modal
Audio Encoder
Masked Spectrogram
Raw Spectrogram (stage Contextualized Representations
View
(stage2)
Fusion Encoder
Inter-modal contrast
Video
Encoder
Video Decoder
NSEI
Intra-modal contrast
Video
Raw Frames (stage1)
Masked RGB Frames
View
Encoder
Contextualized Representations (stage2)
Masked Reconstruction
$3.2 Masked Contrastive Learning
Masked Contextualized A-V Reconstruction
Figure 1: Masked Audio-Video Learners (MAViL) exploit three objectives for learning represen- tations from audio video pairs with masking: (1) Raw audio-video reconstruction. (2) Inter-modal and intra-modal contrastive learning with masking. (3) Reconstructing aligned and contextualized audio-video representations via student-teacher learning. (Please see Fig. 2 for details.)
unpaired clips sampled a mini-batch. (ii) intra-modal contrast that draws closer the two masked views of the same audio video, while pushing away other samples from the same modality,
Thirdly, unlike conventional MAEs that focus reconstructing heterogeneous and raw inputs (audio or video), we propose a novel pre-training task that reconstructs homogeneous and contextualized audio-video representations in a joint suudio-and-videe latent space (illustrated in Fig. 2).
We are motivated by the recent successes in visual SSL that use disparate teacher models to generate contextualized representations
reconstruction targets. For example, BEiT [8] uses features by DALL-E [77] as its target. MaskFeat [91] predicts masked target features such as HOG [ [19] or DINO [10]. Inspired by these uni-modal teachers, take step further to propose new masked contextualized audio-video reconstruction pretext task. Our core idea is: instead of predicting raw inputs in heterogeneous modalities, the model with masked-view inputs jointly predicts contextualized audio-video representations in the homogeneous (aligned) latent space generated by a teacher (i.e. an identical model pre-trained with the masked reconstruction and contrastive objectives above) with complete-vee vieww inputs. This approach ensures the students learn well well-aligned and contextualized audio- video representations and thus improves their performance on downstream tasks. To achieve this, without relying on external teacher models, we employ simple two-stage self-traing training framework (illustrated in Fig. 2). stage-1, we train the teacher MAViL and use raw inputs as its reconstruction targets. In stage-2, the student MAViL (the final model for later fine-tuning) learns to reconstruct the aligned and contextualized audio video representations generated by the MAViL teacher.
Our experimental results confirm MAViL's superiority in audio-video classification (AudioSet-20K: AudioSet-2M, VGGSounnd) and retrieval tasks (VTT and YouCook) where it surpasses the best self-supervised and supervised pre-trained models by large margins. Notably, MAViL not only learns strong joint audio-video representations, but can also improve single modality encoders, without using the other modality it pre-trained with during fine-tuning or inference.
In summary, MAViL makes the following contributions for learning self-supervised audio-video representations: (1) Efficient contrastive learning for both intra-modal and inter-modal context via masking. (2) Introducing new pretext task for multimodal MAE that predicts aligned and contextualized audio-vides representations, surpassing the performance of reconstructing uni- mndllodal raw inputs as used in conventional uni-modall/mulimood. MAE. (3) Achieving new state-of-thed tfrt performance in 7 audio-visual classification, audio-visse vidial retrieval tasks, and (4) audio-only tasks under the SSL setup without using labeled data for pre-training.
Related Work
Supervised Audio- Video Models. The connection between visual signals and co-occurring audio context for language acquisition and world comprehension in infants [80] has motivated the research of audio-visual learning [76, 18]. Previous studies have explored audio-visual ASR [14, 73] and

---

person identification [1] prior to the deep learning era. More recently, there has been significant attention on audio-video representation learning for classification [65, 46, 22, 43, 95, 64]. However, these supervised approaches rely on abundant labeled data, which are costly and time-consuming to obtain. While efforts have been made to create large-scale labeled datasets [35], the dependency on extensive annotation hinders progress in audio-video modeling. In contrast, MAViL focuses on self-supervised learning of robust audio-video representations without the need for labeled data.
Selr-Suuereiee Audio- Video Representation Learning. To exploit abundant unlabeled video and audio content on the Internet and reduce annotation efforts, self-supervised techniques have been explored for learning audio-video representations [4, 2, 3, 48, 82]. Inter-modal contrastive learning is widely used approach that learns to associate paired audio-video clips self-supervision [57, 67, 63, 71]. Techniques such as data augmentation [70, 87] and harder negatives mining [98, 78, 62] have been studied to improve its performance. MAViL unifies masked autoencoding and contrastive learning. We notice concurrent and independent study CAV-MAE [32] and AV-MAE [29] recently explored joint masked autoencoding to reconstruct raw audio and video. Unlike CAV-MAE, MAViL takes a step further to incorporate intra-modal contrastive learning from two masked views. Most importantly, MAViL employs student-teacher learning to reconstruct contextualized and aligned representations instead of raw inputs as in CAV-MAE and AV-MAE. These innovations lead to MAViL's superior performance.
Student-teacher learning or knowledge distillation (KD) [40, 84, 69] was originally developed for model compression [60, 17], aiming to transfer knowledge from larger teacher model to smaller student model. In the context of SSL., KD has recently gained attention for improving the distribution of representations by distilling [93] or by reconstructing contextualized targets from a teacher model [33, 8, 92]. Approaches like MoCo [36], DINO [10], and dBOT [53] utilize self-training to bootstrap targets from previous model snapshots during pre-training. Data2vec [7] performs self-training in disparate modalities, where each one bootstraps complete-view view contextualized targets independently. In contrast, to our best knowledge, MAViL is the first work that employs masked multimodal context for self-training. It uniquely incorporates complete-view multimodal fusion in the teacher model, while the student model receives masked-view inputs.
3 Masked Audio-Video Learners
We introduce Masked Audio-Vidde Learners (MAViL), self-supervised audio-video representation learning framework (see Fig. 1). MAViL consists of two stages: stage-1, MAViL. jointly reconstructs raw spectrograms and RGB frames by exploiting complementary information from each modality ($3.1), and couples this with contrastive learning to encourage alignment between semantically similar instances, both within and across modalities. ($3.2). In stage-2 (see Fig. 2), we employ either the stage-1 model trained with raw targets (in iter. 1) or the last trained stage-2 model (in iter. 2+) the teacher for self-training. We use the teacher's is aligned and contextualized audio-video representations, obtained with complete-viee inputs, to guide the student with masked-vien inputs ($3.3). In the following, provide details of these three types of self suff-supervision, starting with MAViL-stagel trained with raw inputs.
3.1 Masked Raw Audio-Video Reconstruction
Human perception involves processing visual and acoustic context jointly. In line with this, MAViL utilizes multimodal Transformers to fuse and exploit the complementary information from both audio and video. It aims to reconstruct audio and video simultaneously self-supervision, which sets it apart from uni-modal MAE approaches such as MAE [37], Audio-MAE [41], or Video-MAE [27].
Given raw audio-videe video pair (a, v) E D, we begin by patchifying and tokenizing raw audio spectrograms and video frames into audio and video tokens. This involves applying (audio/video) transforms, followed by convoloonolutsonss and flattening. This process embeds raw inputs into
a = [i] ... aN] audio spectrogram tokens and v = [ei tuI...un video tokens, where a;, V/ E R To incorporate positional information, similar to MAE, we employ fixed 2D sinusoidal positional embeddings with the embedded tokens for each modality. We then randomly mask the majority (i.e.
80%) of audio and vidco tokens. Only the remaining 20% unmasked audio (a') and video (v') tokens
3

---

are respectively fed into the audio (fa()) and video (fv()) Transformer encoders. This process
results in uni-modal embedddings, denoted as aum = faa' and Vum = fv()..
Following the uni-modal encoders, we incorporate a multimodal fusion encoder denoted as Jay (-)
to model multimodal context. We explore two variants for this purpose: Vanilla Transformers [86] and Multimodal Bottleneck Transformers (MBT) [64]. For vanilla Transformers, we jointly encode Transformer's (aum |l where "Il" represents
the audio and video tokens by: [| concatenation. Details of MBT implementation is in Appendix. In both variants, we stack L Transformer layers to obtain the jointly encoded top-layer outputs amm and Vmm-
For reconstruction, employ vanilla Transformer blocks the audio f.) and video f,)(() decoders. The fusion encoder's outputs (amm and Vmm) are firstly projected and padded with trainable [MASK] tokens. After restoring the original order (time-frequency for audio and space-time time for video tokens), we add the decoders' (fixed 2-D sinusoidal) positional embeddings and input the restored sequences into the decoders. At the top of the decoders, we incorporate linear heads to reconstruct the raw inputs. Specifically, the decoder outputs for spectrogram reconstruction are denoted as å - (geo(((a)))) (Ja(a))) and for RGB frame reconstruction as = f- (gav (vv)))) For notation clarity, omit the [MASK] tokens and linear projection head. Let a, a E Rf 1 .... denote the audio decoder's ' output and the ground truth reference of the i-th masked spectrogram patch. Similarly, V,. y E IR fFm: j= = 1... m for video patches. masked audio-vides reconstruction, MAViL is supe-su by minimizing the mean squared error (MSE) loss D.wh defined as:
n
m v
(á.
a
raw )
Σ (V)
m
j=1
3.2 Contrastive Audio-Vider Learning with Masking
Contrastive learning is widely used to learn uni-modal [13, 36, 15] and multimodal [75, 71] represen- tations by aligning multiple "views" of the same instance. These views can be either within-modalig observations of the instance itself (e.g., the same audio under different volumes) or semantic observa- tions across modalities (e.g., a video and its corresponding audio). MAViL utilizes InfoNCE [85] loss
for contrastive learning. Let x = [x] ...... ....,,,,yy y lyi...yr yn): x,, y, E R be the instance-level representations of audio/video in batch of size B. The contrastive loss (x(x, is defined as:
1
B
log exp(S(x;,
Lx =
B
j= (x)(S(x;),)/T))
where y(), xx I is the cosine similarity between x, y, and 7 is the softmax temperature. MAViL employs two types of contrastive losses for self-supervision listed below:
Inter-modal contrastive learning facilitates alignments across modalities. We first average the sequence of uni-modal encoder outputs' the instance-level representations, namely, aemb Avg(aum) and Vemb = Avg(Vum). The positive pairs (the numerator in Eq.(2)) consist video and audio clips from the same video, while all the other combinations of sampled audio-vided pairs considered negatives (the denominator). Inter-modal contrast encourages the representations of paired audio and video to be closer to each other, while simultaneously pushing away mismatched ones.
Intra-modal contrastive learning promotes alignment within each modality. aims to bring the representations of different views, such as different augmented versions of an audio (or video), closer to each other. To achieve this, for each modality, we apply random masking and sample a second view that contains 20% tokens for encoding. The idea is to treat masking as a form of augmentation to generate two contrasting views in the same modality, which has been proven effective in visual SSL [36, 13]. The two views of the same instance are considered as positive pair, which are then contrasted against the other (negative) combinations of the instances in the same modality. Formally, let āemb and Venh be the embeddings of the second-view audio and video. MAViL employs the
We do not use the fusion output amm and Vmm for contrastive learning since the fusion layer provides a shortcut/leakage of information exchange for every paired audio-video clip. This results in sub optimal performance for contrastive learning.

---

Ber.
A-Enc
A-Dec
0
A-Enc
A-Dec
A-Enc
AV-
AV-
ENC
V-Enc
V-Dec
V-Dec
Student
Teacher
Figure 2: Masked contextualized audio-video reconstruction in the joint latent space. Stage-1 (left): Training MAViL with raw imputs targets. Stage-2 (right): Self- self-traaning MAViL student by reconstructing MAViL teacher's 's aligned and contextualized audio-video representations generated with complete inputs. Repeat stage-2 for K iterations. In the first iteration of stage-2, the stage- model used as the teacher. In subsequent iterations (iteration 2+), the last trained stage-2 student model serves the new teacher.
inter-modal (Ciiler) and intra-modal (Ci contrastive objectives defined as:
Cinter
Lo (Aemb, Vemb) + La Vemb; aemh))
Cintra
LL (Bemb, aemb) + CooCooVee Ce(Veb Vamb)]
These contrastive losses effectively learn a latent audio-video space where semantically similar instances are close to each other. Note that unlike prior work (e.g. [36, 13D, MAViL performs contrastive learning under masked-view. This leads to improved computation efficiency as only small portion of input tokens encoded for contrast. And different from CAV-MAE [32], MAViL additionally incorporates intra-modal contrast, which yields superior performance. Overall, a and be the weights that balance loss terms, MAViL--tagel is self-supervised by minimizing:
MAVV = = + Oz Cinter + (
3.3 Masked Contextualized Audio-Vidie Reconstruction
To learn robust audio-video representations, we go beyond raw input reconstruction in uni-model MAEs [37, 41, 25], multimodal MAE [6], and CAV-MAE [32]. We propose new pretext task that reconstructs contextualized audio-video representations in the joint latent space. To achieve this, we employ simple two-stage training framework illustrated in Fig. 2. In stage- we train MAViL with E4) to reconstruct raw inputs. In stage-2, we employ a student-teacher learning framework for iterative self-training. In the first iteration, the pre- pre-trained MAViL from stage- is frozen and serves as the teacher model. It generates audio-video representations with complete-view inputs, which used the reconstruction targets to guide the re- re-initialized student MAViL. In the subsequent iterations, the last trained stage-2 student MAViL serves as the new teacher.
Formally, we provide the teacher model's encoders with complete-view audio and video inputs to generate aligned and contextualized targets: a Teucher yy Teacher The student MAViL then Icarns to reconstruct these contextualized targets with the masked-view inputs. Precisely, ā = (g (a))) for audio and v = fr (gav (v(')) for video, where Teacher Tencher
Jav Teacher [
(Teacher (v)),
a. V E IR The contextualized reconstruction objective defined as:
m
pcontext
aa
a Teacher 2
+
Teacher
n
m
In the stage-2 training, we jointly minimize the masked contextualized reconstruction loss and the contrastive loss. The stage-2 (student) MAViL's objective is:
LMvva.
Coontext
Cints.
Note that (6) contains pure latent targets?. After pre-training, we then fine-tune the audio/video encoders in the final 2 MAViL student model in the downstream tasks.
Empirically, we found no gain when multi-tasking with stage- autoencoding loss pixelUseectrogram.
5

---

4 Experiments
We performed comprehensive evaluations, including audio-vides classification tasks on AudioSet [28] (AS-2M and AS-20K), and VGGSound [11]. Also, we conducted audio-to-vided retrieval experiments on MSR-VTT [96] and YouCook [100]. We use AS-20K for model analysis and ablation studies.
4.1 Datasets
AudioSet contains 2 million 10-second YouTube clips for audio event detection. 527 event types are weakly labeled [51, 38, 39] for each clip, and multiple events can occur in one clip. AudioSet's full training set has two subsets: A class-wise balanced (22K clips) and unbalanced (2M clips) set. The eval set has 20K clips. We downloaded 1.97M unbalanced training, 20K balanced training, and 19K evaluation clips. We use the full (unbalanced+balanced) training set for pre-training. In the 2M task, we tne-tune on the full training set. In the AS-20K task, we fine-tune only on the 20K balanced training set. We report the classification mAP on the 19K eval set used by AST [30].
VGGSound comprises approximately 200K 10-second video clips annotated with 309 event types that include human actions, sound-emitring objects, etc. Unlike AudioSet, VGGSound ensures that an audio event is also visually present in its corresponding clips. VGGSound is divided into 183K training and 15K testing samples. We report top-1 testing classification accuracy.
4.2 Implementation Details
MAViL, adopts different temporal footprints for audio and video. For audio, following [64, 30],
it transforms 10-second audio under 16K sampling rate into 128 Mel-frequency bands with
25ms Hanning window shifting every 10ms. The resulting spectrogram of dimension 1024 x 128. MAViL then tokenizes it into non-overlapping 16 16 patches where both time and frequency have a kernel and stride of 16. The flattened audio tokens have a sequence length of 512. For video, MAViL processes 4-second clips under 2 frames per second. Each frame has size of 224 x 224. Tokenization is achieved by a 3D convolution, where the spatial kernel and stride are 16, and the temporal kernel and stride are 2. The flattened video tokens have sequence length M of 784.
Following the design choices in MAE [37], MAViL. employs 12-layer Transformers (ViT-B) with 12 attention heads as the encoders for each modality. The embedding dimension H is set to 768. The audio-video fusion encoder layer consists of a 2-layer Transformer (vanilla or MBT) on top of the uni-modal encoders. Similarly, the audio and video decoders utilize 8-layer Transformers with an embedding dimension of 512 and 16 attention heads. MAViL's 's audio/video encoder and decoder have 86M and 27M parameters, respectively. The floating point operations (FLOPs) for the audio encoder are 48.6G, comparable to the audio encoders in Audio-MAE [41] and CAV-MAE [32].
For pre-training MAViL's 's audio branch, we randomly initialize it from scratch. For the visual branch, we either randomly initialize it or initialize it with the sulf-super MAE [37] pre-trained on ImageNet (compared in Table 6). Notably, MAViL operates under the fully supe-suuee setup.
4.3 Experimental Sctup
We pre-train MAViL on AS-2M without using any of AS-2M labels. We use 80% masking for audio and video. For balancing the losses in (6)( we set = 0.1, ritter = 0.1 and 3 0.01, 12 min = 1.0. These hyperparameters scale the gradients from the three losses into a comparable range to improve training stability. We pre-train with 64 V100 GPUs with a 512 accumulated batch size and a 0.0002 learning rate. We pre-train for 20 epochs in stage-1 and in each iteration of stage-2 (for (K = 3) iterations). Each session takes 20 hours, resulting in a total pre-training time around 80 hours.
We then evaluate audio and video representations by fine-tuning the pre-trained MAViL in audio-video classification tasks on AS-20K, 20M, and VGGSound. We test 3 scenarios: (1) audio-only, (2) video-only, and (3) audio+video. Following MAE [37], we only keep the pre-trained encoders and use the sverage-pooled toy-layer outputs for classification. We adopt the standard fine-tuning pipeline and augmentation in prior audio/audio-vis
classification works [30, 41, 64). Specifically, we employ SpecAug [68], mixup [99]. balanced sampling [51], and fine-tuning masking [41]. For video, we use standard video augmentations used in video classification [90, 27J. To address the discrepancy in convergence rate between audio and video, apply a 50% learning rate reduction for the video

---

encoder during the audio+video (fusion) fine-tuning. Fine-tuning for 100 epochs on AS-2M takes 10 hours. Please refer Appendix for additional implementation and experimental details.
In the following, we first use AS-20K to analyze MAViL's performance in $4.4. We then present the main comparison to prior works in $4.5. The gray entries represent the shared default setup.
4.4 Model Analysis
Masked Raw Audio- Video Reconstruction. Table 1 shows the contribution of MAViL'' audio- video fusion encoder layer, which exploits complementary multimodal information for reconstructing raw inputs in stage-1. The "None" (col. indicates the uni-modal MAE baselines without the fusion layer, namely Audio-MAE and Video-MAE under our implementation. The fusion layer, whether it's vanilla Transformer or MBT, contributes up 0.4 mAP gains (col. 2-5). While the MBT layers show better results, the improvements is not significant compared to using vanilla Transformers. Also, increasing the depth of the fusion encoder only leads to marginal gains, For simplicity, we default to using a 2- 2-layer vanilla Transformer as the fusion encoder.
Fusion
None
Vanilla
Vanilla
MBT
Ratio 40%
60%
70%
80%
90%
Audio
36.4
36.8404)
36. (6.7
36.8, 3,.8
Audio 35.6
36.5
36.7
36.8
36.8
Video
17.4
17.7ua3)
17.6
17.7G0.)
Video
16.8
17.3
17.5
17.7
17.5
Table I: Fusion Encoder: Transformer type layers)
Table 2:
Masking Ratio.
mAPt in
Note that the 'None' are the uni-modal MAE baselines (Audio-MAE and Video-MAE). mAP in AS-20K.
AS-20K for individual fine-tuning as au- dio or video classification.
Subsequently, we investigate the masking ratio, a key hyperparameter in masked autoencoding. We use the same masking ratio for audio and video, and ablate different values in Table 2. The results suggest masking ratio of 80% achieves the best performance. This aligns with the optimal values of 80% in Audio-MAE [41] and 90% in Video-MAE [27]. MAViL encodes only the non-masked tokens significantly reduces the sequence length, resulting in more efficient computation, as the complexity of self-attention layers in Transformers scales quadratically with the sequence length.
Contrastive Video Learning with Masking. Next, we examine the benefits when contrastive learning is used in conjunction with MAE. MAViL performs efficient contrastive learning with only 20% of visible patches in each modality (80% are masked). The results in Table demonstrate the significance of contrastive losses for learning audio-video representations, visible tokens. Unlike CAV-MAE [32] which uses only inter-modal contrast, we observe that both inter-modal and intra-modal contrast essential, and combining them delivers the best performance.
with only 20%
Objs.
None
Inter
Intra
Inter+intra
Audio
36.8
38. (8.40 38.1, 1,43)) 39.0(-2.)
Video
17.7
21.0us.) 19.8,21 22.2
Table 3: Masked Contrastive Video Learning
Masked Contextualized Audio-Video Reconstruction. As illustrated in Fig. 2, MAViL employs two-stage training. Following stage-1 training (Table 3), MAViL's encoders acquire complementary audio-video representations through masked raw audio-video reconstruction, as well as aligned representations through masked contrastive learning. In stage-2, we employ an iterative approach to initialize the student MAViL and train under masked-view> to predict the aligned and contextualized representations generated by the frozen MAViL teacher with full view. The teacher can be either from the stage-1 model (iter. 1) or the last trained stage-2 model (iter. and beyond).
We first compare the reconstruction targets in Table 4a (stage-2, iter. 1). By default MAViL student predicts the last trained MAViL teacher's multimodal fusion encoder outputs (i.e.. Teacher (f.Teacher (ft)achrraa) [a)]],[]]] ((Teache((v) (v)), M-Fusion, col. 4). The results show that this improves over pre- dicting (col. 1) raw spectrogram/RGGB frames (+1.7 audio mAP) and (col. 2) separately pre-trained Aadio//Vide-MAEEs outputs (+1.2 audio mAP). The fused multimodal targets (M-Fusion, col. are better (+0.5 audio mAP) than the uni-modal targets (M-Uni, col. 3) (ie., fRacher (a) and f Teacher (v) before the fusion encoder).
Inv (f
uni-modal
7

---

Target Raw A/V-MAE
M-Uni M-Fusion
Iter.
None
2
3
4
Audio 39.0 (4.))
40.2
40.7(.1m)
Audio
39.0 40. (0.7 (1.5(-2.5) 41. (1-88)))) 41.8(+2.8)
Video 22.2 23.
23.8(-16)
24.1(119)
Video
22.2 24. (41.9) 24.6(-24) 24.8(+26) 24.9(42.)
(a) Reconstruction Targets (Stage-2, iteration-I)
(b) Iterations (Stage-2)
Table 4: Targets and iterations in MAViL's ' stage-2 contextualized audio-video reconstruction.
(a) Target comparison in the 1s iteration. Raw: spectrogram/RGG frames. Contextualized: A-
MAE/V-MAE (uni-modal MAE output); M-Uni (MAViL's 's uni-modal output); M-Fusion (MAViL's multimodal fusion output). (b) Iteration comparison. 'None' is the MtVil-stagel baseline in Table 3.
For the 2nd and beyond iterations of stage-2 training, we leverage the last trained MAViL student as the new teacher. As shown in Table 4b, substantial improvements (asymptotically saturated after the 3rd iteration) are achieved over iterations through masked prediction of aligned and contextualized audio- video representations by the MAViL teacher. The accumulated gains with the proposed contextualized audio-video reconstruction in stage-2 are +2.8 audio and +2.6 video mAP without using additional data or external teacher models.
Additional Ablations. In Table 5 we ablate several more aspects of MAViL: (a) Pre-training with more data useful (100% 50% > 10% of AS-2M) (Tab. 5a). (b) ImageNet (IN) self-supervised pre-training is beneficial for the video encoder (Table 5b). ImageNet supervised pre-training (IN-SL) useful yet we avoid this to keep MAViL fully self-supervised. (c) Longer pre-training is beneficial and 20 epochs are sufficient (Table 5c). (d) Increasing the encoder sizes (ViT-S/B/L) improves mAp
(Table 5d). We use ViT-B for efficiency and fair comparison as default.
|20KK
IM
2M
Init.
Rand. IN-SSL
IN-SL
Method
Audio
Video
A
34.1
40.5
41.8
A
41.6
41.8
41.8
A-MAE/V-MAF (baseline)
36.4
17.4
20.1
23.0
24.8
V
23.7
24.8
26.1
MAVIL stage-/
Joint AV-MAE
36.8..).)
17.7Ga0))
(a) Dataset size
(b)
Visual backbone init.
Inter contrast
38.4
21.0
Intra and Inter contrast
39.0 22.2(
Ep.
15
20
25
Sz.
VIT-S
VIT-B
ViT-L
MAVIL stage-2
A
41.4
41.8
41.8
A
36.2
41.8
42.2
Student-teacher learning
41.8 24.8.-2.)
V
23.5
24.8
24.7
V
20.5
24.8
27.0
(c) Module-wise contributions
(c) Pre-training epoch
(d) Model size
Table 5: Additional ablations and module-wise contributions (mAP on AS-20K)
Summary. The module-wise contribution is summarized in Table Se. MAViL enhances uni-modal MAE performance by learning from audio and video simultancously, as evidenced by significant increases in audio mAP (36.4-44188 +5.4) and video mAP (17 (17.4-224.8, +7.4) on AS-20K. It is noteworthy that while audio and video are jointly used in the pre-training phase, the fine-tuning is performed with distinct uni-modal encoders on each modality. Therefore, the improvements in the uni-modal classification tasks showcase SSL pre-training from both modalities can enhance the performance of individual modalities.
4.5
Main Results
The full model is the 2 MAViL student (ViT-B) trained with (q.(6) for 3 iterations. We quantitatively compare it with other previous models on the 7 benchmark tasks.
Audio-Video Event Classification. Table 6 presents a comparison of MAViL's audio (A), video (V), and audio-video (A+V) fine-tuning performance on AudioSet (AS) and VGGSound, alongside recent baselines. Notably, MAViL achieves new sttte-ff-theaar results in audio-only and audio+video clas- sification tasks across these datasets. On the balanced AS-20K, unbalanced AS-2M, and VGGSound classification tasks, MAViL surpasses CAV-MAE [32] (both models are with VIT-B encoders) in A, and A+V tasks by large margin. This improvement can be attributed to the benefits of reconstructing aligned and contextualized representations over raw inputs and the enhanced contrastive learning through both intra-modal and inter-modal contrast. Furthermore, MAViL outperforms data2vec [7]. highlighting the advantage of utilizing aligned multimodal contexts (Fq.(5) over uni-modal contexts as the reconstruction targets.

---

AS-20K (mAPf)
AS-2M (mAPf)
VGGSound (Acc.t)
Method
PT
A
V
A+V
A
A+V
A
V
A+V
Audio-only Models
Aud-SlowFast [44]
50.1
VGGSound [11]
48.8
PANNs [47]
27.8
43,9
AST [30]
IN-SL
34.7
45.9
HTS-AT [12]
IN-SL
47.1
PaSST [49]
IN-SL
47.1
Data2vec [7]
AS-SSL
34.5
SS-AST [31]
AS-SSL
31.0
MAE-AST [5]
AS-SSL
30.6
Aud-MAE [41]
AS-SSL
37.0
47.3
Audio- Video Models
G-Blend [88]
22.1
37.8
32.4
18.8
41.8
Perceiver [42]
38.4
25.8
44.2
Attn AV [24]
IN-SL
38.4
25.7
44.2
CAV-MAE [32]
IN-SSL, AS-SSL
37.7
19.8
42.0
46.6
26.2
51.2
59.5
47.0
65.5
MBT [6T]
IN2IK-SL
31.3
27.7
43.9
41.5
31.3
49.6
52.3
51.2
64.1
MAViL
AS-SSI
41.6
23.7
44.6
48.7
28.3
51.9
60.6
50.0
66.5
MAViL.
IN-SSI, AS-SSL
41.8
24.8
44.9
48.7
30.3
53.3
60.8
50.9
67.1
Table 6: Comparison to prior work on AudioSet (AS-20K, AS-2M) and VGGSound the audio (A), video (V) and audio+video (A+V) classification tasks. PT: pre-training dataset and type; IN: ImageNet; SL: supervised learning; SSL: self-supervised learning; ":We de-emphasize the model using non-standard dataset splits. We bold the best-performing single model.
The fully self-supervised MAViL also demonstrates superior performance to supervised audio-video models such MBT [64] in and A+V classification tasks on AS-20K, AS-2M, and VGGSound. For video classification on AudioSet, it worth noting that the fully self-supervised trained video backbone still lags behind MBT which pre-trained with the supervision from ImageNet-218 (11 times larger than ImageNet). This disparity is consequence of the presence of noise and irrelevant visual context in AudioSet also discussed in [6]]. Such bias could make the visual pre-training sub-optimal on AudioSet. We consider resolving this dataset limitation as the future work.
Transfer to Audio/Speech-only Tasks. To access the generalizability of the learned audio repre- sentations, we further evaluate the AS-pre-trained MAViL by transferring it to other out-of-domain speech-only audio-only tasks. Specifically, conduct experiments the Environmental Sound Classification (ESC-50) [72] and Speech Commands (SPC-v1) [89], where only the audio branch of MAViL is fine-tuned. The results, presented in Table demonstrate that MAViL outperforms recent supervised and self-supervised models, establishing a new state-of-the performance on these benchmarks. These findings indicate the desirable transferability of MAViL from audio-video
self-s celf-supervised pre-training to audio-only downstream tasks.
Method
PT
ESC-50
SPC-1
Method
PT data
MSR-VTT
YouCook
AST [30]
IN-SL
88.7
95.5
AVLNet [79]
HT100M
20.1
30.7
SS-AST [31]
AS-SSL
88.8
96.0
TVLT [83]
HT100M
22.6
31.8
Aud-MAF [41]
AS-SSL
94.1
96.9
MAViL
AS-2M
22.8
32,2
MAViL
AS-SSL
94.4
97.3
MAViL
HT-100M
23.8
33.1
MAViL
IN-SSL, AS-SSL
94.4
97.4
Table 7: Audio-only tasks (Acc.t)
Table 8: Audio-to-video retrieval (R@11)
Audio-to-Vider Retrieval. MAViL learns aligned audio-vided representations that are suitable for textless cross-modal retrieval. To verify this, we further conduct uudio-to-vides retrieval experiments on MSR- MISR-VTT-B [96] and YouCook [96]. these tasks, the audio track video serves as the query, and the model performs a search over the testing video collection by computing and ranking the similarity between the query embedding and the video embeddings. We fine-tune MAViL. using the audio-video pairs in the training sets with (6). We report recall@) @ on the testing sets.

---

Video clip and spectrogram reconstruction on the AudioSet eval set. We sample 4 paired (video, audio) examples as follows (click the links to play): Top left: a puppy video; Top right: a recording from an ambulance's s dash camera; Bottom left: a person dialing a phone in dark room; Bottom right: a singer dancing. Input masking ratio: 70%. In each 3-row group, we show the original video and its audio spectrogram (top), masked input to MAViL (middle). and MAViL's video and audio spectrogram reconstructions (bottom). The spectrogram shape is 1024 128; patch size is 16x 16. Each spectrogram has 64x8=52 122 patches. After applying 70% masking, there are 154 patches visible to MAViL. The 8-frame (4-second under fps) video clip size is 8 x x 224 x 224; patch size is 16 x 16. Each video has x 14 14 784 patches after patch embedding (temporal kernel/stride=2). After applying 70% masking, there are 235 patches visible to MAViL.
Figure 3:
Following AVeNet [79] and TVLT [83], we explore alternative setup where we pre-train MAViL on HowTo100M,, [59], dataset consisting of 1.3 million instructional videos, instead of using AudioSet for pre-training. It's worth noting that there are notable domain differences between the pre-training datasets (AudioSet or How TowTo100M) and the downstream datasets (MSR-VI) VTT or YouCook). Table 8 demonstrates that MAViL surpasses supervised pre-trained AVLNet and self-supervised pre-trained TVLT, achieving new state-of-the-art performance these tasks.
Raw Audio-Vidie Video Reconstructions In Fig. 3, we employ a stage-1 1 MAViL (ViT-B) to reconstruct raw audio spectrograms and video frames with masked inputs. The model is trained using 80% masking ratio on the AudioSet-2MM full training set with un-normalized raw spectrograms and video frames the reconstruction targets (4), stage- 1). We visualize the reconstruction results by MAViL''s 's audio and video decoders, wherein 70% of the input tokens are masked to its encoders. This visualization is performed on the AudioSet eval set.
The results show that MAViL accurately reconstructs the highly corrupted versions of both audio spectrograms and video frames. The reconstructions for videos exhibit high fidelity and preserve spatial and temporal consistency of visual objects (e.g., the nearby moving cars recorded by the ambulance's s dash camera). For audio reconstructions, MAViL accurately maintains the positions and arrangements of time-freee tine-freeuency components in the spectrogram (e.g., the ambulance's 's siren and the song by the singer). Furthermore, the reconstructed audio and video components are consistent and well-aligned time, enhancing the overall coherence the reconstructed content.
Conclusion
We have presented MAVIL, self-supervised audio-video representation learning framework where masked autoencoding meets contrastive learning. By leveraging an iisodderfusioo-decooe archi- tecture, MAViL effectively utilizes complementary information from all modalities for masked autoencoding, It facilitates efficient contrastive learning in both inter-modal and intra-modal scenar- ios, even with a high 80% masking ratio. Furthermore, MAViL highlights novel pre-training task with self-training to predict homogeneous contextualized audio-vied video representations. This approach outperforms conventional uni-modal and multimodal MAEs that predict heterogeneous raw inputs. As result, MAViL achieves state-of-the-as performance across 7 audio-video classification and retrieval tasks, as well as audio-only tasks under the scalable self-supervised learning setup.
10

---

Acknowledgements. We thank Luke Zettlemoyer, Kaiming He, Juncheng (Billy) Li, and Florian Metze for their feedback and discussions.
References
[1] Petar S Aleksic and Aggelos K Katsaggelos. Audio-visual biometrics. Proceedings of the IEEE, 94(11):0025-2044, 2006.
[2] Relja Arandjelovic and Andrew Zisserman. Look, listen and learn. ICCV, 2017.
[3] Relja Arandjelovic and Andrew Zisserman. Objects that sound. In ECCV, 2018.
[4] Yusuf Aytar, Carl Vondrick, and Antonio Torralba. Soundnet: Learning sound representations from unlabeled video. In NeurIPS, 2016.
[5] Alan Baade, Puyuan Peng, and David Harwath. MAE-AST: Masked autoencoding audio spectrogram transformer. In Interspeech, 2022.
[6] Roman Bachmann, David Mizrahi, Andrei Atanov, and Amir Zamir. MultiMAE: Multi-modal multi-task masked autoencoders. arXiv preprint 2rXi::2004001678, 2022.
[7] Alexei Baevski, Wei-Ning Hsu, Qiantong Xu, Arun Babu, Jiatao Gu, and Michael Auli. Data2vec: A general framework for self-supervised learning in speech, vision and language. In ICML, 2022.
[8] Hangbo Bao, Dong, and Furu Wei. BEÎT: BERT pre-training of image transformers. In ICLR, 2021.
[9] Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agar- wal, Ariel Herbert- Voss, Gretchen Krueger. Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCan- dlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In NeurlPS, 2020. URL https://proceeding .eesiine..pp.... cttpippscccppppp/22/22 hash/ 0tth/1457 04577700011100044444111111111121
Abstract.html
[10] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in sulf-supervised vision transformers. In ICCV, 2021.
[11] Honglie Chen, Weidi Xie, Andrea Vedaldi, and Andrew Zisserman. VGGSound: A large-scale audio-visual dataset. In ICASSP, 2020.
[12] Ke Chen, Xingjian Du, Bilei Zhu, Zejun Ma, Taylor Berg- Kirkpatrick, and Shlomo Dub- nov. HTS-AT: A hierarchical token-semantic audio transformer for sound classification and detection. In ICASSP, 2022.
[13] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. simple framework for contrastive learning of visual representations. In ICML, 2020.
[14] Tsuhan Chen and Ram R Rao. Audio-visual integration in multimodal communication. Proceedings of the IEEE, 86(5):837-832 1998.
[15] Xinlei Chen, Saining Xie, and Kaiming He. An empirical study of training self-supervised vision transformers. In ICCV, 2021.
[16] Akash Chintha, Bao Thai, Saniat Javid Sohrawardi, Kartavya Bhatt, Andrea Hickerson, Matthew Wright, and Raymond Ptucha. Recurrent convolutional structures for audio spoof and video deepfake detection. IEEE Journal of Selected Topics in Signal Processing, 14(5): 1024-1037, 2020.
[17] Jang Hyun Cho and Bharath Hariharan. On the efficacy of knowledge distillation. In ICCV, 2019.
[18] Grzegorz Chrupata. Visually grounded models of spoken language: A survey of datasets, architectures and evaluation techniques. Journal of Artificial Intelligence Research, 73:673- 707, 2022.
11

---

[19] Navneet Dalal and Bill Triggs. Histograms of oriented gradients for human detection. In
2005 IEEE Computer Society Conference on Computer Vision and Pattem Recognition (CVPR 2005), 20-26 June 2005, San Diego, CA, USA, pp. doi: 10.1 109/CVPR 2P0R.2005.111 177. URL https:ttps:.. /dooi. org/10. 10. 1109/CVPR. 2005. 177.
IEEE Computer Society, 2005.
[20] Jacob Devlin, Ming-Woirr Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. In NAACL-HLT, 2019.
[21]
Konstantinos Drossos, Samuel Lipping, and Tuomas Virtanen. Clotho: audio captioning dataset. In ICASSP 2020 2020 IEEE International Conference on Acoustics, Speech and doi: 10. 11009/CAASP44776620000033333
Signal Processing (ICASSP), pp. 736-740, 2020.
[22] Ariel Ephrat, Inbar Mosseri, Oran Lang, Tali Dekel, Kevin Wilson, Avinatan Hassidim, William T Freeman, and Michael Rubinstein. Looking to listen at the cocktail party: a speaker- independent audio-visual model for speech separation. ACM Transactions on Graphics (TOG), 37(4):1-11, 2018.
[23] Haoqi Fan, Bo Xiong, Karttikeya Mangalam, Yanghao Li. Zhicheng Yan, Jitendra Malik, and Christoph Feichtenhofer. Multiscale vision transformers. In ICCV, 2021.
[24] Haytham M Fayek and Anurag Kumar. Large scale audiovisual learning of sounds with weakly labeled data. In IJCAI, 2020.
[25] Christoph Feichtenhofer, Axel Pinz, and Richard P Wildes. Spatiotemporal residual networks for video action recognition. In NIPS, 2016.
[26] Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, and Kaiming He. Slowfast networks for video recognition. In /CCV, 2019.
[27] Christoph Feichtenhofer, Haoqi Fan, Yanghao Li, and Kaiming He. Masked autoencoders as spatiotemporal learners. In NeurlPS, 2022.
[28] Jort F Gemmeke, Daniel PW Ellis, Dylan Freedman, Aren Jansen, Wade Lawrence, R Channing Moore, Manoj Plakal, and Marvin Ritter. Audio set: An ontology and human-labeled dataset for audio events. In ICASSP. 2017.
[29] Mariana-Iuliana Georgescu, Eduardo Fonseca, Radu Tudor Ionescu, Mario Lucic. Cordelia Schmid, and Anurag Amnb., Audiovisual masked autoencoders. In Proceedings of the IEEE/CVF International Conference Computer Vision (ICCV), pp. 16144-16154, October 2023.
[30] Yuan Gong, Yu-An Chung, and James Glass. AST: Audio Spectrogram Transformer. In Interspeech, 2021.
[31] Yuan Gong, Cheng-I Yu-An Chung, and James R. Glass. SSAST: Self-Supervised Audio Spectrogram Transformer. In AAAI, 2022.
[32] Yuan Gong, Andrew Rouditchenko, Alexander H Liu, David Harwath, Leonid Karlinsky, Hilde Kuehne, and James Glass. Contrastive audio-visual masked autoencoder. arXiv preprint
arXiv:2210.07839, 2022.
[33] Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H. Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Ghesh- laghi Azar, Bilal Piot, Koray Kavukcuoglu, Rémi Munos, and Michal Valko. Bootstrap your own latent: new approach to self-supervised lcarning. In NerulPS, 2020.
[34] Michael Gutmann and Aapo Hyvärinen. Noise-contrastive estimation: A new estimation principle for unnormalized statistical models. In Yee Whye Teh and D. Mike Titterington (eds.), Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, AISTATS 2010, Chia Laguna Resort, Sardinia, Italy, May 13-15, 2010, volume of JMLR Proceedings, pp. 297-304. JMLR.org, 2010. URL http: / httpp:////ooceeinng..... mlr. press/v9/ arress/99/gutman11.0 html.
[35] David Harwath, Adria Recasens, Dídac Surís, Galen Chuang, Antonio Torralba, and James Glass. Jointly discovering visual objects and spoken words from raw sensory input. In ECCV, 2018.
[36] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In CVPR, 2020.
12

---

[37]
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In CVPR, 2022.
[38]
Shawn Hershey, Sourish Chaudhuri, Daniel W. Ellis, Jort F. Gemmeke, Aren Jansen, Channing Moore, Manoj Plakal, Devin Platt, Rif A. Saurous, Bryan Seybold, Malcolm Slaney, Ron Weiss, and Kevin Wilson. CNN architectures for large-scale audio classification. In ICASSP, 2017. URL https: //atxii//axxv..... org/ a0516999
11099.6.... 09430.
[39] Shawn Hershey, Daniel PW Ellis, Eduardo Fonseca, Aren Jansen, Caroline Liu, R Channing Moore, and Manoj Plakal. The benefit of temporally-strong labels in audio event classification. In ICASSP, 2021.
[40] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. NeurIPS Deep Learning and Representation Learning Workshop, 2015.
[41] Po- Yao Huang, Hu Xu, Juncheng Li, Alexei Baevski, Michael Auli, Wojciech Galuba, Florian Metze, Christoph Feichtenhofer, et al. Masked autoencoders that listen. NeurIPS, 2022.
[42] Andrew Jaegle, Felix Gimeno, Andy Brock, Oriol Vinyals, Andrew Zisserman, and Joao
Carreira. Perceiver: General perception with iterative attention. In ICML, 2021.
[43] Evangelos Kazakos, Arsha Nagrani, Andrew Zisserman, and Dima Damen. Epic-fusion: Audio-visual temporal binding for egocentric action recognition. In ICCV, 2019.
[44] Evangelos Kazakos, Arsha Nagrani, Andrew Zisserman, and Dima Damen. Slow-fast auditory streams for audio recognition. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 855-859, 2021. doi: 10. N0.1100/AASS333922..0000 20219943376
[45] Chris Dongjoo Kim, Byeongchang Kim, Hyunmin Lee, and Gunhee Kim. Audiocaps: Gener- ating captions for audios in the wild. In NAACL-HLT,, 2019.
[46] Yelin Kim, Honglak Lee, and Emily Mower Provost. Deep learning for robust feature generation in audiovisual emotion recognition. In ICASSP, 2013.
[47] Qiuqiang Kong, Yin Cao, Turab Iqbal, Yuxuan Wang, Wenwu Wang, and Mark D Plumb- ley. PANNs: Large-scale pretrained audio neural networks for audio pattern recognition. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 28:2880-288 22944, 2020.
[48] Bruno Korbar, Du Tran, and Lorenzo Torresani. Cooperative learning of audio and video models from self-supervised synchronization. Advances in Neural Information Processing Systems, 31, 2018
[49] Khaled Koutini, Jan Schlüter, Hamid Eghbal-zadeh, and Gerhard Widmer. Efficient training of audio transformers with patchout. arXiv preprint
2021.
[50] Gustav Larsson, Michael Maire, and Gregory Shakhnarovich. FractalNet: Ultra-deep neural networks without residuals. In ICLR, 2017.
[51] Juncheng B. Li, Shuhui Qu, Po- Yao Huang, and Florian Metze. AudioTagging Done Right: 2nd comparison of deep learning methods for environmental sound classification. CoRR, doi: 10.48550/arxi..2203..333 48550/arXiv 2203.222222222 13448.
abs/2203. 13448, 2022.
URL https://doi.orr http:////oo.or/11.....
[52] Yanghao Li, Chao- Wu, Haoqi Fan, Karttikeya Mangalam, Bo Xiong, Jitendra Malik, and Christoph Feichtenhofer. Improved multiscale vision transformers for classification and detection. arXiv preprint arXiv:21 12.01126...... 2021.
[53] Xingbin Liu, Jinghao Zhou, Tao Kong, Xianming Lin, and Rongrong Ji. Exploring target representations for masked autoencoders. arXiv preprint arXiv:2209.03911 2022.
[54] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer. and Veselin Stoyanov. Roberta: robustly optimized BERT pretraining approach. CoRR, abs/1907. 1692, 2019. URI. htttp://aaxii.oooo/////// org/ab//977. 11692.
[55] Ilya Loshchilov and Frank Hutter. SGDR: Stochastic gradient descent with warm restarts, In ICLR, 2017.
[56]
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In ICLR, 2019.
[57] Shuang Ma, Zhaoyang Zeng, Daniel McDuff, and Yale Song. Active contrastive learning of audio-visual video representations. In ICLR, 2020.
13

---

[58] Xinhao Mei, Xubo Liu, Qiushi Huang, Mark D. Plumbley, and Wenwu Wang. Audio captioning transformer. In Proceedings of the 6th Detection and Classification of Acoustic Scenes and Events 2021 Workshop (DCASEE222)) pp. 21 211-215, Barcelona, Spain, November 2021. ISBN 978-84-09-3607221
[59] Antoine Miech, Dimitri Zhukov, Jean-Baptiste Alayrac, Makarand Tapaswi, Ivan Laptev, and Josef Sivic. HowTo 00M: Learning text-video embedding by watching hundred million narrated video clips. In ICCV, 2019.
[60] Asit Mishra and Debbie Marr. Apprentice: Using knowledge distillation techniques to improve low-precision network accuracy. In ICLR, 2018.
[61] Abdelrahman Mohamed, Hung-yi Lee, Lasse Borgholt, Jakob D. Havtorn, Joakim Edin, Christian Igel, Katrin Kirchhoff, Shang-War Wen Li, Karen Livescu, Lars Maaløe, Tara N. Sainath, and Shinji Watanabe. Self-supervised speech representation learning: A review. IEEE Journal of Selected Topics in Signal Processing, 1790921079-200, 2022.
[62] Pedro Morgado, Ishan Misra, and Nuno Vasconcelos. Robust audio-visual instance discrimina- tion. In CVPR, 2021.
[63] Pedro Morgado, Nuno Vasconcelos, and Ishan Misra. Audio-visual instance discrimination with cross-modal agreement. In CVPR, 2021.
[64] Arsha Nagrani, Shan Yang, Anurag Arnab, Aren Jansen, Cordelia Schmid, and Chen Sun. Attention bottlenecks for multimodal fusion. In NeurlPS, 2021.
[65] Jiquan Ngiam, Aditya Khosla, Mingyu Kim, Juhan Nam, Honglak Lee, and Andrew Y Ng. Multimodal deep learning. In ICML, 2011.
[66] Andreea-Mante Oncescu, A. Sophia Koepke, João F. Henriques, Zeynep Akata, and Samuel Albanie. Audio retrieval with natural language queries. In Hynek Hermansky, Honza Cer- nocký, Lukás Burget, Lori Lamel, Odette Scharenborg, and Petr Motlícek (eds.), Interspeech 2021, 22nd Annual Conference of the International Speech Communication Association, Brno, Czechia, 30 August- 3 September 2021, pp. 2411-2415. ISCA, 2021. doi: 10.21437/ Interspeech.20211 2021-2227. URL https:///oi. org/ 10. 21437 11437/Intten Interspeech.. 2021-2227.
[67] Andrew Owens and Alexei A Efros. Audio-visual scene analysis with self-supervised multi- sensory features. In ECCV, 2018.
[68] Daniel S. Park, William Chan, Yu Zhang. Chung-Char Chiu, Barret Zoph, Ekin D. Cubuk, and Quoe V. Le. Specaugment: A simple data augmentation method for automatic speech recognition. ArXiv, hts//9904.08779, 2019.
[69] Wonpyo Park, Dongju Kim, Yan Lu, and Minsu Cho. Relational knowledge distillation. In CVPR, 2019.
[70] Mandela Patrick, Yuki M Asano, Polina Kuznetsova, Ruth Fong, João F Henriques, Geoffrey Zweig, and Andrea Vedaldi. On compositions of transformations in contrastive sulf-supervised learning. In ICCV, 2021.
[71] Mandela Patrick, Yao Huang, Ishan Misra, Florian Metze, Andrea Vedaldi, Yuki M. Asano, and João F. Henriques. Space-time crop & attend: Improving cross-modal video representation learning. In 202/ IEEE/CVF International Conference on Computer Vision, ICCV 2021, Montreal, QC, Canada, October 10-17, 2021, pp. 10540-10552. IEEE, 2021. doi: 10. 109/ URL https::s:// / //t.i//oiiioo org/ 10. 1 I1w.///C//9992..... 2021 2021.01039.
[72] Karol J. Piczak. ESC: Dataset for Environmental Sound Classification. In Proceedings of the 23rd Annual ACM Conference on Multimedia, pp. 1015-1018. ACM Press, 2015. ISBN
978-1-4503-3459-4, doi: 10. 10.1145273333322886699 URL //d1. acm. wrg/citaiionn.
cfm/dood==2733373. 0tt.///333373226399.0.
2806390.
[73] Gerasimos Potamianos, Chalapathy Neti, Guillaume Gravier, Ashutosh Garg, and Andrew W Senior. Recent advances in the automatic recognition of audiovisual speech. Proceedings of the IEEE, 306-13333) 2003.
[74] Daniel Povey, Arnab Ghoshal, Gilles Boulianne, Lukas Burget, Ondrej Glembek, Nagendra Goel, Mirko Hannemann, Petr Motlicek, Yanmin Qian, Petr Schwarz, et al. The kaldi speech recognition toolkit. IEEE 2 workshop on automatic speech recognition and understanding, number CONF. IEEE Signal Processing Society, 2011.
14

---

[75] Alec Radford, Jong Wook Kim, Chris Hallacy. Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In ICMI,, 2021.
[76] Dhanesh Ramachandram and Graham W Taylor. Deep multimodal learning: A survey recent advances and trends. IEEE signal processing magazine, 34(6);96-108, 2017.
[77] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Iya Sutskever. Zero-shot text-to-image generation. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pp. 8821-8831. PMLR, 2021. URL ramesh21a. html.
mlr press/v13//
[78] Adria Recasens, Pauline Luc, Baptiste Alayrae, Luyu Wang, Florian Strub, Corentin Tallec, Mateusz Malinowski, Viorica Pătrăucean, Florent Altché, Michal Valko, et al. Broaden your views for self-supervised video learning. ICCV, 2021.
[79] Andrew Rouditchenko, Angie Boggust, David Harwath, Brian Chen, Dhiraj Joshi, Samuel Thomas, Kartik Audhkhasi, Hilde Kuehne, Rameswar Panda, Rogerio Feris, et al. AVLnet: Learning audio-visual language representations from instructional videos. In Interspeech, 2021.
[80] Deb Roy. Learning from sights and sounds: computational model. PhD) Thesis, MIT Media aaboratoyy, 1999.
[81] Md. Sahidullah, Tomi Kinnunen, and Cemal Hanilçi. A comparison of features for synthetic speech detection. In INTERSPEECH 2015, 16th Annual Conference of the International Speech Communication Association, Dresden, Germany, September 6-10, 2015, pp. 2087- 2091. ISCA, 2015. URL //ww:://wwwww// http::::/wwww..saa-eeeeee iessssspeeechh...eeeeeeeeeee htg/arrhivee/tnterreeccccc 2015/ 115_2087 html.
[82] Bowen Shi, Wei-Ning Hsu, Kushal Lakhotia, and Abdelrahman Mohamed. Learning audio- visual speech representation by masked multimodal cluster prediction. In ICLR, 2022.
[83] Zineng Tang, Jaemin Cho, Yixin Nie, and Mohit Bansal. TVLT: Textless Vision-Language Transformer. In NeurlPS, 2022.
[84] Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive representation distillation. In ICLR, 2019.
[85] Aäron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. CoRR, 0bs//107.003744 2018. URL http://arxiv.org wtw..//g//////.....
03748.
[86] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. NeurlPS, 2017.
[87] Luyu Wang, Pauline Luc, Adria Recasens, Cean-Baptiste Alayrac, and Aaron van den Oord. Multimodal self-supervised learning of general audio representations. arXiv preprint arXiv:2104. T0V::2104./807,, 2021.
[88] Weiyao Wang, Du Tran, and Matt Feiszli. What makes training multi-modal classification networks hard? In CVPR, 2020.
[89] P. Warden. Speech Commands: A Dataset for Limited- Vocaaullryy Speech Recognition. ArXiv e-prints, April 2018 URL https: : /arxiviivvva org/ abs/ 1804088000000 032009.
[90] Chen Wei, Haoqi Fan, Saining Xie, Chao-Yuae Wu, Alan L. Yuille, and Christoph Feichten- hofer. Masked feature prediction for self-supervised visual pre-training. CoRR, abs/2112.091333,,
2021. URL https: wrg abss//bs/211 09133.
[91] Chen Wei, Haoqi Fan, Saining Xie, Chao-Yuan Wu, Alan Yuille, and Christoph Feichtenhofer. Masked feature prediction for self-supervised visual pre-training. In CVPR, 2022.
[92] Longhui Wei, Lingxi Xie, Wengang Zhou, Houqiang Li, and Qi Tian. MVP: Multimodality- guided visual pre-training. In ECCV, 2022.
[93] Yixuan Wei, Han Hu, Zhenda Xie, Zheng Zhang, Yue Cao, Jianmin Bao, Dong Chen, and Baining Guo. Contrastive learning rivals masked image modeling in fine-tuning via feature distillation. arXiv preprint arXiv:2205.141411 2022.
15

---

[94] Yusong Wu*, Ke Chen*, Tianyu Zhang*. Yuchen Hui*, Taylor Berg- Kirkpatrick, and Shlomo Dubnov. Large-scale contrastive language-audin pretraining with feature fusion and keyword- to-caption augmentation. In IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP, 2023.
[95] Fanyi Xiao, Yong Jae Lee, Kristen Grauman, Jitendra Malik, and Christoph Feichtenhofer. Audiovisual slowfast networks for video recognition. arXiv preprint arXiv:2001.087400 2020.
[96] Jun Xu, Tao Mei, Ting Yao, and Yong Rui. MSR-VE VTT: large video description dataset for bridging video and language. In CVPR, 2016.
[97] Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. In ICCV, 2019.
[98] Zhaoyang Zeng, Daniel McDuff, Yale Song, et al. Contrastive learning of global and local video representations. In NeurIPS, 2021.
[99] Hongyi Zhang, Moustapha Cisse. Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. In ICLR, 2018.
[100] Luowei Zhou, Chenliang Xu, and Jason Corso. Towards automatic learning of procedures from web instructional videos. In AAAI, 2018.
16

---

Appendix
This appendix are organized follows: $A, we offer the comprchensive experimental details and hyperparameter configurations for pre-training and fine-tuning on each dataset. In $B, we perform additional experiments to evaluate and analyze MAViL's 's performance. These experiments include:
1. Modality-wise masking ratio and masking type analysis.
2. Contrastive weights/
analysis.
3. From-scratch and large model analysis.
4. Linear probing analysis.
5. Text-audio retrieval tasks on AudioCaps [45] and Clotho [21].
In $C, we discuss MAViL's societal impact and limitations.
A Experimental Details & Hyper-parameters
In this section, we provide additional experimental details for data preprocessing, implementation. pre- pre-training fine-tuning and inference. The parammerreestes are summarized in Table 9. The codebase and the pre-trained models will be available.
A.1 Data Preprocessing
In study, obtained total of 2.01 million AudioSet videos, including both the video and audio tracks from the balanced and unbalanced training set and the evaluation set. Additionally, we managed collect 198K VGGSound videos. As part of the preprocessing, we resized the video tracks to 360p while maintaining the aspect ratio and adjusting the longer dimension to 360 pixels. We also resampled the audio tracks to a sampling rate of 16K. We employed different temporal footprints for modeling the audio and video in MAViL, specified as the following:
Following the preprocessing in [64, 30, 41], transform raw audio (with mono- noon--hhnneel and under 16K sampling rate) into 128 Mel- del-frequency bands used in Kaldi [74]. This transformation
involves using 25ms Hanning window that shifted every 10ms. We then normalize the spectrogram according to the mean and variance in each dataset. For a 10-second audio, the resulting spectrogram has dimension of 1024 128.
Regarding the video part, we utilize 4-second clips consisting frames per second (fps). Each input frame has size of 224 224. In the pre-training phase, we apply common data augmentations such as random horizontal flip (with a probability 0.5) and multi-scale random cropping (with scale ranging from 0.2 to 1.0). In contrast, we apply only center cropping during the testing or inference phase. When processing a 10-second vidco clip from AudioSet, we randomly sample starting point and extracted the consecutive 4 seconds of the video (cyclically looping back to the beginning if it was shorter than seconds). As a result, the video clip input, consisting of channels, had dimensions of 8 x 3 224 224.
frames captured at a rate
A.2 Implementation
Uni-modal Encoders. We adopt the main design choices from original MAE for images [37] and Audio-MAE [41]. Specifically, we employ separate 12-layer Transformers with 12 attention heads as the encoders for each modality. The patch embedding and positional embeddings layers are also separated for each modality. During investigation, explored alternative designs, including sharing the audio-video encoder weights with separated inputs or concatenating them as done in Multi-MAE [6]. However, these alternative architectures resulted in inferior performance compared to the proposed architecture of using separated encoders with separated inputs. As result, we chose to adhere the original design of separate encoders for each modality.
In Transformer encoders (with ViT-B as the default), the embedding dimension is set to 768 For each input spectrogram of size 1024 x 128 representing a 10-second audio, we tokenize it into non-overlapping 16 x 16 spectrogram patches using an audio patch embedding layer. The kernel and stride sizes for both the time and frequency dimensions are 16, resulting total of 64 x 8
17

---

spectrogram patches or tokens for the audio sequence. The flattened audio token sequence has length N of 512. Each audio token corresponds to a 168-dimensional vector. After appending the [CLS] token, adding positional embeddings, and applying 80% masking, the final input audio token sequence is represented as a' E 102
For each video clip with dimensions 8 x 3 x 224 x 224 (4 seconds in duration), we tokenize it into non-overlapping cells using video patch embedding layer. The spatial kernel and stride sizes are set to 16, while the temporal kernel and stride sizes are set to 2. This process results in a total of 4 x 14 x 14 784 video patches or tokens. The flattened video token sequence has a length M of 784. Each video token corresponds to a 768-dimensional vector. After appending the [CLS] token, adding positional embeddings, and applying 80% masking, the final imput video token sequence is represented as v'é E R 156
Fusion Encoders. Following the ViT-B uni-modal encoders, we incorporate an audio-video fusion encoder. The fusion encoder consists of two-layer (with L=2) Transformer, which can be either a vanilla Transformer or an MBT Transformer [64].
In the vanilla Transformer setup, the fusion encoder, denoted as 9av (.v(.), jointly encodes the audio and video tokens. This is done by concatenating the output of the uni-modal encoders for audio (aum'+1) +1 and video (vum/+1) as input, resulting in (t [v Transformer' (a' (l'mllVumm) '), where "Il" denotes concatenation.
In the MBT setup, we extend the vanilla Transformer by appending an additional 4 trainable MBT tokens for each modality. MBT encourages the model more selectively collate and condense relevant information in each modality by forcing information exchange between modalities to pass through a small number of learnable bottleneck features bd = [6] ... 61], b, E R The use of MBT tokens was originally proposed in the context of supervised audio-video learning. Preciscly, 4+1 + = g'a (a bb) and v |lb/+] = sax (v l|b)). where b' = (b)+1 b/2
Decoders. The audio and video decoders are 8-layer Transformers with an embedding dimension of 512 and 16 attention heads. In the top decoder layer, we applied a linear prediction head to either predict the raw audio spectrogram and video frame patches in stage-1 (i.e., a raw E IR " and viaw R Hi.) ) predict the aliened and contextualized representations in stage-2 (i.e. a Teacher v Teacher ā v E I²... The audio/video encoder and decoder in MAViL have 86M and 27M parameters, respectively. The floating point operations (FLOPs) for the audio encoder are 48.6G, comparable to the audio encoders in Audio-MAE [41] and CAV-MAE [32].
A.3 Training and Inference
Pre-training. MAViL operates under a fully self-supervised learning setup for pre-training. For pre-training MAViL's audio branch, we randomly initialize it from scratch. For the visual branch, either randomly initialize it initialize it with the self-supervised MAE [37] pre-trat pre-trained on ImageNet where we simply repeat and inflate the convolution kernel in its patch-embedding to handle the additional temporal domain. Different visual initialization methods are compared in Table 6 in the main paper and Table 15 in Appendix. Importantly, MAViL operates under the fully self-supervised setup.
MAViL is pre-trained on the combined unbalanced and balanced training sets of AS-2M. The pre- training process is performed using 64 GPUs with 512 accumulated batch size. In stage-1 and each iteration of stage-2 (for K iterations), we pre-train the model for 20 epochs. Each pre-training session takes approximately 20 hours to complete. In total, the pre-training process takes around 80 hours. Note that the effective Icarning rate (Iran) depends on the base learning rate (Iroase) and the batch size. Precisely, Ir.rr = Irbase batch size
256 In our experiments, we also tried using strong data augmentations (e.g., mixup [99], SpecAug [99], and CutMix [97]) to augment audio spectrograms during the pre-training phase. However, we observed that the resulting performance was either similar or worse compared to the baseline. Therefore, by default, we exclude these strong data augmentations for both audio and video during the pre-training phase.
Besides the proposed two-stage training framework, we also experimented with a one-stage variant that exploits the exponential move average (EMA) of MAViL (with raw input as the targets) as the teacher (similar to data2vee [7]) to generate contextualized targets to be reconstructed by a separated decoder head in each modality. This setup results in less stable training behavior in the
18

---

Pre-training AS-2M
Fine-tuning
Configuration
AS-2M
AS-20K
VGGSound
ESC
SPC
Optimizer
AdamW [56]
Optimizer momentum
31
0.9, 82 0.95
Weight decay
0.00001
Base learning
0.0002
0.0001
0.001
0.0002
0.0005
0.001
Learning rate schedule
half-cycle cosine decay [55]
Minimum learning
0.000001
Gradient clipping
None
Warm-up epochs
4
Epochs
20
100
60
60
60
10
Batch size
512
512
64
256
GPUs
64
64
8
32
4
4
Weighted sampling
False
Truc
False
True
False
Falsc
Weighted sampling size
200,000
200,000
Augmentation
R
R
R+N
R
R+N
SpecAug [68] (time/frequency)
192/48
192/48
192/48
96/24
48/48
Drop path [50]
0.0
0.1
0.1
0.1
0.1
0.1
Mixup [99]
0.0
0.5
0.5
0.5
0.0
0.5
Multilabel
n/a
True
True
False
False
False
Loss Function
MSE
BCE
BCE
BCE
CE
BCE
Dataset Mean for Normalization
-4.268
-4.268
-4.268
-5.189
-6,627
-6.702
Dataset Std Normalization 4.569
4.569
4.569
3.260
5.359
5.448
Table 9: Pre-training (PT) and Fine-tuning (FT) hyper-parameters. For augmentation, R:
sampling random starting points with cyclic rolling in time; N: adding random noise (signal-to-noise ratio (SNR): 20dB) spectrograms. For loss functions, BCE: binary
entropy loss (for multi- label datasets or when using mixup); CE: cross-entropy loss, MSE: mean square error loss. " We repeat and balance each class 50% of the size of the unknown class. f; For ViT-S, We learning rate of 0.0005 on AS-2M FT and 0.002 on AS-20K FT for the ViT-$ model. For the ViT-L model, we use 0.0001 and 0.0005 for AS-2M and AS-20K FT experiments.
audio-video scenario. The instability is presumably consequence of different convergence patterns in each modality, which is also observed in [88]. Consequently. this makes the design and tuning of mixture/update scheduler sub-optimal (i.e., that controls the EMA update in Data2vec Sec 3.3). Regrettably, this led to notable performance degradation:
2-stage
I-stage (EMA)
Audio
41.8
40.3
Video
24.8
21.7
Table 10: 2-stage 1-stage (EMA)
By employing 2-stage training process, MAViL not only simplifies the methodology and achieves better performance, but also mitigates the necessity of devising intricate update schedules and carefully tuned EMA hyperparameters, as required in prior work.
Fine-tuning. We fine-tune MAViL in three scenarios: (1) audio-only, (2) video-only, and (3) au- dio+video. We follow the setup in MAE and retain only the pre-trained uni-modal encoders for fine-tuning. In the audio-only and video-only setups, fine-tune the respective encoders in the MAViL (stage-2). In the audio+video fusion setup, we introduce 2-layer vanilla Transformer on top of the audio and video encoder in the MAViL (stage-2) and fine-tune using both audio and video inputs. The hyperparameter configurations specified Table are employed for finetuning on each dataset. Empirically observed discrepancy in convergence rate between audio and video. We circumvent this by applying a 50% learning rate reduction for the weights of the video encoder when performing audio+video fusion fine-tuning.
We adopt the standard fine-tuning pipeline and augmentation in prior audio/audio-video classification works [30, 41, 64]. Specifically, we employ SpecAug [68], mixup [99], balanced sampling [51], and
19

---

fine-tuning masking [41] (a 20% random masking rate for time and frequency in audio spectrograms; 20% for space and time in video clips). For video, we use standard video augmentations used in video classification [90, 27].
To perform importance sampling that balance the fine-tuning scheme on the unbalanced AS-2M (and VGGSound), we apply distributed weighted sampler as prior works [51, 30, 12, 49]. We set the probability of sampling sample proportional the inverse frequency of its labels, where the label frequency is estimated over the training set. Specifically, for instance in a dataset D with label pool C, its sampling weight is proportional to Ee,ec We: where we Σ.eD +:
1000 and
e e= 0.01 is set to avoid underflow in majority classes. During the fine-tuning process on AS-2M, we randomly sample 200K instances (approximately 10% of AS-2M) with replacement each epoch. We fune-tune MAViL for 100 epochs, which corresponds to approximately 10 full epochs of AS-2M. The entire fine-tuning process typically takes around 10 hours to complete.
Inference. After fine-tuning, we select the last checkpoint for inference. For the video and au- dio+video tasks, we adopt the standard approach used in video action recognition [26, 23, 52] by uniformly sampling ten 4-second video clips throughout the time domain of video. Each of these sampled video clips is individually fed forward through the model to generate predictions. Note that for audio+video classification, the audio input remains the same 10-second audio recording throughout the sampling of video clips.
# Clips (AS-2M)
10
Audio
48.7
48.7
Video
29.4
30.3
Audio+ Video
52.6
53.3
Table 11: Number of video clips in the inference time.
We average the ten predictions as the instance-level prediction and report the classification perfor- mance in Table 6 in $4. Note that these results are based on single-modal predictions, without ensembling multiple models. In Table 11, we compare the results obtained from one-clip predictions and ten-clip predictions (mAP on AS-2M). The sampling of ten clips leads to improvements of up to 0.9 mAP for video-only and audio+video tasks, while the audio-only task remains unaffected.
B Additional Experiments and Analysis
In this section, we present additional analysis to extend the study of the module-wise contribution in Table 12. We then expand our study on another important type of audio task: text-audio retrieval.
We organize this section as follows: Firstly, we investigate how different choices of masking ratio and masking type may affect the model performance. Next, we examine the effects of adjusting contrastive weights in the training objective. By exploring different weight settings, we aim to understand the influence of contrastive learning on the model's 's ability capture audio-video relationships. Furthermore, we compare different approaches to visual backbone initialization and evaluate the performance using larger (ViT-L.) audio/video encoders MAViI models. This analysis helps us understand the benefits and trade-offs of using larger backhone models and different initialization strategies. Additionally, besides audio-vides classification tasks and audio-vides video retricval tasks presented in the main paper. We include our study audio-text retrieval tasks in the last.
Method
Audio Video
A-MAE/V MAE (baseline) 36.4
17.4
MAViL. stage-/
+ Joint AV-MAR
36.8.40.) 17.7(40.))
Intra and Inter contrast
39.0 22.2(44.5)
MAVIL stage-2
+ Student-teacher learning 41.8,28) 24.8.v..)
Table 12: Module-wise Contribution in MAViL).
20

---

B.1 Masking Ratio and Type
In addition applying shared masking ratio for each modality, we also investigated the impact of applying different masking ratios for audio and video. The results of this analysis are summarized in Table 13a. Interestingly we did not observe a significant change in performance (mAP on AS-20K) when using different masking ratios for audio and video. Based on these findings, we simplify the approach by defaulting to an 80% masking ratio for both audio and video, as the Joint AV-MAE entry (the second row) in Table 12.
Ratio 70% (A) 80% (A) 90% (A)
Type
70%
80%
90%
70% (V) 36.7/17.5 36.8/17.5 36,4/17.3
Random (A). Random (V)
36.7/17.5 36.8/17.7
36,8/17.5
80% (V) 36.7/17.2 36.8/17.7 36.8/17.4
Time-Freq (A), Random (V)
36,2/17.5 36.3/17.7 36.3/17.8
90% (V) 36.5/17.3 36.6/17.6 36.8/17.5
Random (A), Space-Time (V)
36,7/17.2 36,7/17.3 36,8/17.5
Time-Freq (A), Space-Time (V)
36.0/17.1 36,2/17.1 36,3/17.3
(a) -ode Masking
(b) Masking Type
Table 13: Masking Ratio and Masking Type (mAP on AS-20K).
The default masking strategy in our model random masking, which applies the same Bernoulli trial parameterized by masking ratio (p) to each spectrogram or RGB patch. In Table 13b, we explored more advanced masking strategies and compare their impacts. For audio spectrogram, in addition to random masking (time-and-freque) frequency agnostic with Bernoulli trials), we investigated time- masking (randomly masks multiple periods of time components) and frequency masking (randomly masks multiple frequency bands). We perform Bernoulli trials on time frequency slots instead of individual patches. For video frames, we explored time-wise masking (randomly masking an entire frame) and space-wise masking (randomly masking a spatial patch across time). We set the masking ratio between spatial/frequency and time as 2:1 and adjusted the overall ratio from 70% to 90% for comparison with random masking.
Surprisingly, we do not observe improvements when applying these advanced masking strategies for multimodal pre-training. The simplest random masking approach achieved the best pre-training performance. This observation aligns with the findings in uni-modal MAEs [37, 27, 41J, suggesting that the random masking strategy is effective and sufficient for multimodal pre-training.
B.2 Contrastive Weights
Table 14 showcases the impact of adjusting contrastive weights and /3 in MAViL. The results show that fine-tuning these contrastive weights leads to improved performance. In our experiments, we set 0.1 and 8 0.01 which yield the best performance.
is important to note that the smaller contrastive weights in (4) do not imply that the contrastive objectives less significant. The weights chosen scale and balance the gradients from the reconstruction and the two contrastive objectives to ensure they fall within a comparable range. This adjustment enhances training stability. Furthermore, the softmax temperatures used in NCE (Eq. (2)) are set as - 0.1 (more tolerant) for inter-modal contrastive learning and iinir = 1.0 (stricter) for intra-modal contrastive learning. These temperature values help regulate convergence across modalities in the contrastive learning process.
Q
0.3
0.1
0.05
0.1
0.05
0.01
Audio
41.5
41.8
41.4
Audio
41.3
41.5
41.8
Vidco
24.3
24.8
24.4
Vidco
24.3
24.7
24.8
(a) Inter-modal a
(b) Intra-modal /
Table 14: Contrastive Weights (mAP on AS-20K).
B.3 From-scratch Visual Backbone and Large Models
Under the fully self-supervised setup, MAVIL initializes its audio branch from scratch and initialize its visual branch either from scratch from ImageNet self-supervised pre-trained MAE (IN-SSL). In
21

---

AS-20K
AS-2M
Model
A-init
V-init
A
A+V
V
A+V
MAVil -Base
scratch
IN-SSL.
41.8
24.8
44.9
48.7
30.3
53.3
MAViL-Base
scratch
scratch
41.6
23.7
44.6
48.7
28.3
51.9
MAViL-
scratch
IN-SSL
42.1
27.1
45.3
48.8
32.4
53.3
MAViL-Large
scratch
scratch
42.3
25.3
45.1
30.6
52.5
Table 15: Visual Backbone Initialization and Model Size (mAP).
this part, further explore and compare the visual backbone initialization strategies under different model sizes.
As shown in the top two rows of Table 15, when considering MAViL-Base models, there is small gap (-0.2 mAP on AS- observed the audio stream when discarding visual initialization from the ImageNet self-supervised model. However, larger gap (-0.9 mAP) is observed in the video stream. A similar trend is observed in the AS-2M experiments. This discrepancy in the visual part likely be attributed biases and visual quality issues such as misalignment, title-only content, and low-resolution videos present in AudioSet.
To address this gap in the visual part, incorporating additional uni-modal pre-training steps could potentially improve model performance. For instance, conducting separate audio-only and video-only large-scale pre-training as the first step. In this work, we focus on audio-video pre-training solely on AudioSet for simplicity and for fair comparison with baselines. The possibility of incorporating additional pre-trat pre-training steps is left for future research.
When using large models (ViT-L. rows 3-4), the gap visual mAP (-1.8 mAP) still persists. Interest- ingly, the audio part of large models actually benefits from from-scratch visual initialization, showing an improvement of +0.2-0.3 mAP. Additionally, when comparing rows 1-2 rows 2-3, the visual stream is benefited more by employing a larger (VTT-L) backbone. Across all the configurations (from-scratch scratch or visual initialization with IN-SSL), MAViL consistently outperforms recent baselines (in Table 6 of the main paper) by significant margin.
B.4 Linear Probing Results
Additionally, evaluate MAViL's representations by performing linear probing ESC and AudioSet-20K. As shown in Table 16, MAViL's representations outperform other recent baselines by large margin. There is still is gap between linear-probing and end-to-end fine-tuning.
PT-Data ESC-FT ESC-Linear AS-20K-FT
XDC
A+V.AS
84.8
AVID
A+VAS
89.1
BraVe
A+VASS
90.4
CrissCross
A+VAS
90.5
SS-AST
A.AS
88.8
85.6
31.0
A-MAE
A.AS
94.1
89.5
37.0
MAViL
A+VASS
94.4
90.8
41.8
30.0
Table 16: Linear-probing and fine-tuning on ESC and AS-20K. PT: Pre-training. FT: end-to-end fine-tuning. Linear: Linear-probing under standard protocol. AS: AudioSet. Metrie: Top-1 accuracy for ESC-50 and mAP for AS-20K.
B.5 Text-Audio Tasks
Another important audio-centered multimodal application involves text-to-audio and audio-to toxtet retrieval tasks. In text-to-audio retrieval, the query text description, and the model performs search over the (testing) audio collection by computing and ranking the similarity between the query embedding and the audio embeddings. To evaluate the audio representations learned by MAViL. following CLAP [94], add text encoder initialized from Roberta [54]. We perform fine-tuning
22

---

with inter-mont modal contrast on the same training set used by CLAP. Specifically, AudioCaps [45] and
Clotho [21], and -AION-630K [94]. In Table 17, we report recall @ 5, and 10 the testing sets.
AudioCaps [45]
Clotho [21]
Textooo-Audio
Audio-to-Tens Text
Texi-to-Audio
Audio-to-Tede
Model
R@1
R@S
R@
10
R@1 R@s
R@ 10
R@!
1
R@s
R@ 10
R@l
R@s
R@ 10
MMT" [66]
36.1
72.0
84.5
39.6
76.8
86.7
6.7
21.6
33.2
7.0
22.7
34.6
ML-ACT" [58]
33.9
69.7
82.6
39.4
72.0
83,9
14.4
36.6
49.9
16.2
37.6
50.2
CLAP [94]
32.7
68.0
81.2
43.9
77.7
87.6
15.6
38.6
52.3
23.7
48.9
59.9
MAViL
37.3
72.8
84.5
49.3
81.8
91.5
17.2
41.0
53.5
23.3
49.5
63.6
Table 17: Text-to-Audio retrieval and Audio-to-Text retrieval (R@ (,,,,,100)) AudioCaps and Clotho. models trained without LAION-630K [94].
As shown above, MAViL significantly outperforms CLAP and other recent audio-text models, achieving new state-of-the-arr performance both audio-to-text and text-to-audio retrieval tasks. These results further validate the effectiveness of MAViL's representations not only in audio-video and audio-only tasks, but also in audio-text tasks.
C Limitations and Impacts
Limitations. There are several limitations associated with MAViL. Firstly, the scale of the data poses a limitation. The AudioSet [28] dataset used by MAViL, with two million samples, is approximately two orders of magnitude smaller than the text corpora used in recent language models [20, 54, 9]. It also an order smaller than image corpora like ImageNet-21K used by MBT [64].
Another limitation pertains to the duration of each audio sample. The 10-second recording in AudioSet are relatively short, which can hinder the proper learning of distant temporal dependencies in audio and video. This limitation restricts the potential upplicability of MAViL to tasks that require modeling longer audio sequences, such automatic speech recognition (ASR). Regarding video modeling, due to GPU memory constraints and choice of video footprints, MAViL only models 4-second video segments. This limitation makes it challenging effectively model long video sequences. Additionally, the presence of low-quality videos and misaligned audio-video pairs in AudioSet may adversely affects pre-training.
Potential Societal Impacts. The datasets used in this paper, including AudioSet and other end task datasets, were properly licensed and publicly available at the time of data collection. It is important to note that some of the data may have been removed by YouTube the dataset uploaders. Most of the data in these datasets are licensed under the Creative Commons BY-NC-ND 4.0 license or the Creative Commons 3.0 International License.
To investigate the bias AudioSet, we selected 200 videos containing speech. In these videos, we did not observe any visual bias in the sampled speakers, which encompassed a wide range of ages, races, and genders. However, it is possible that there may biases in the distribution population and ethnicity within AudioSet. It is important exercise caution and be aware of the potential unintended gender, racial, and societal biases present in AudioSet, which serves as the pre-training data for MAViL.
Given that AudioSet consists of a vast collection YouTube videos, there is a potential risk that MAViL could leam to reconstruct sensitive personal information, which could then be exploited for malicious purposes, including the creation audio deepfakes [81, 16). To address this concern, the released MAViL. would be discriminative models, specifically the audio and video encoders, rather than generative models such as decoders. This shift aims mitigate the potential risks associated with generating synthetic content that could be misused.
23


Reviews:
[{"review_id": "lVwRyvWdjb", "reviewer": "Reviewer_kuAy", "summary": "This paper presents a new self-supervised learning framework for audio-visual modalities. It combines three types of SSL techniques, namely MAE, contrastive learning, and student-teacher distillation. The proposed method is a two-stage process by first training a teacher network to reconstruct the raw signal and then using this network to guide a student network to learn in the second stage. Authors show that this method improves the SOTA on AudioSet, VGG-Sounds as well as several other classification and retrieval benchmarks.", "questions": "I'd appreciate the authors' response on how much data augmentation matters and the potential limitation of this work.\n\n\n====post rebuttal===  \nThe authors' response addressed my concerns and I am keeping the original score.", "limitations": "Limitations and societal impact are not discussed in this work.", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "Although the three SSL losses in this paper are not new, this paper did a good job of putting them together and making them work. The performance improvement over the SOTA method is also pretty decent, validating the effectiveness of the approach.\n\nThe authors conducted a very sufficient ablation study of the model architecture, showing that multi-modal fusion, contrastive learning, reconstruction, and distillation all contribute to the model's performance.\n\nThe writing is very clear-written and easy to follow.", "weaknesses": "In the implementation details, authors mentioned many data augmentations are used in training. How much do they matter? I do not see an ablation on this component.\n\nThe limitation of this work is not discussed in this paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for your review of our manuscript and for acknowledging MAViL’s SOTA performance accompanied by a comprehensive analysis. With regard to your comments and questions please see below:\n\n**Q1: Impact of data augmentation?**\n\nA1: Thank you for the great question! Comprehensive details of the augmentation techniques and configurations can be referenced in Table 1 in Supplementary. In the pretraining phase, MAViL exclusively employs audio/video cyclic rolling. In the fine-tuning phase, standard data augmentation methods, including SpecAug, Mixup, magnitude augmentations, and others that have been utilized in prior studies (such as AST, CAV-MAE, Aud-MAE in Table 6), are incorporated.\n\nFor your question, we conducted an experiment to evaluate the impact of data augmentation by removing all audio/video augmentation. The results are outlined below:\n\n|| AS-20K (V) | AS-20K (A)| AS-20K (A+V) | \n| -----  | ------ | ------ | ------ |\n|MAViL w/ aug | 41.8 | 24.8 | 44.9 |\n|MAViL w/o PT aug | 41.6 | 24.7 | 44.7 |\n|MAViL w/o PT and FT aug | 39.1 | 23.3 | 42.5 |\n\nThe results show that data augmentation plays a less significant role in pre-training, and is relatively important for fine-tuning. Please note that previous SOTA models (eg. MBT, CAV-MAE, AudMAE) are all using augmentations.\n\n**Q2: Limitation of this work?**\n\nA2: With the page limits, we have addressed MAViL’s limitations in the Supplementary materials. The main constraint of MAViL lies in its utilization of AudioSet (2M audio-video recordings) for audio-visual pre-training. This choice raises several concerns:\n1. Smaller dataset size: MAViL required audio-video data for pre-training and AudioSet is currently the best dataset available. Compared to datasets like ImgNet-21K (with 14M images) used in MBT, AudioSet's size is notably smaller.\n2. Short recordings: Each recording in AudioSet is around 10 seconds. This poses challenges when applying MAViL to tasks such as automatic speech recognition (ASR), where extended audio sequences are common.\n3. Weak Audio-Video correspondence: The in-the-wild recordings present in AudioSet exhibit limited audio-video correspondence, posing difficulties in achieving robust audio-visual learning.\n\nWe anticipate that MAViL's performance will see further enhancements with the availability of larger-scale audio-video datasets.\n\n\n"}, {"review_id": "4sw3VMLQTh", "reviewer": "Reviewer_RTTA", "summary": "The authors propose to learn audio-visual representations with two stage process with three self-supervision losses including 1. masked reconstruction 2. intra and inter-modal contrastive learning with masking 3. self-training via teacher-student setup to contrast on contextualized representations in 2nd stage. The authors show that proposed approach achieve state-of-the-art audio-visual classification performance on both AudioSet and VGGSound, audio-only tasks, and audio-to-video retrieval.", "questions": "- In section 4.3 it is mentioned that for downstream evaluation fine-tuning is applied. What is the performance on frozen encoders with linear probe if matching the output number of classes is necessary? Both fine-tuning and frozen can provide different perspectives of proposed system and can be used in different applications. It can provide a more complete view if provided.\n- In table 4(b), what is the performance beyond three iterations, does it keep improving or the performance drops? It helps to include iteration 4 in either case.", "limitations": "- This work provides thorough study on proposed approach applied to audio related tasks including classification and crossmodal retrieval. It lacks some aspects on the video modality. It would be beneficial to include more evaluation with video downstream tasks.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "- The authors provide thorough ablation studies for each loss function introduced, masked ratio, choices of dataset size, backbone init, model size. This helps to understand the proposed system with a more complete view.\n- The proposed approach to combine two popular loss function masked reconstruction and contrastive learning, and further introduce the self-training on contextualized representation is original.\n- This paper is well written with clear narrative on the motivation and it provides sequences of experiments which help the reader to understand their thoughts step by step.", "weaknesses": "- This work lacks evaluation on the visual modality, especially video, as this work claims to be audio-video learner. Both AudioSet and VGGSound are mainly used in audio research, it would be beneficial to provide performance on some video benchmarks. For example, MBT also include EpicKitchens-100.\n- Minor suggestion: consider add AS-20K in the caption of table 1, as it is less clear which dataset/task is used. Relevant to this, the use of AS-20K for ablation study is mentioned in the last paragraph of section 4.3, consider move it to the first paragraph of section 4.4 to better highlight this choice.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer RTTA,\n\nWe appreciate your attention to recognizing the originality of MAViL's multimodal self-training approach, which leverages aligned and contextualized representations. Additionally, we thank you for recognizing our thorough model analysis and we will follow your editing suggestions. We address your comments and question as below:\n\n**W1: Video benchmark on Epic-Kitchen?**\n\nWe thank you for this important suggestion. We experimented on Epic-Kitchen and summarize the top-1 acc for verbs, nouns and actions in below:\n\n||Type|Noun|Verb|Action|\n|---|---|---|---|---|\n|A-SlowFast|A|22.8|46.5|15.4|\n|MBT|A|22.4|44.3|13.0|\n|MAViL|A|26.8|53.1|20.2|\n|SlowFast|V|50.0|65.6|38.5|\n|MTV|V|60.5|67.8|46.7|\n|MBT|V|56.4|62.0|40.7|\n|MAViL|V|54.8|69.4|45.1|\n|MBT|A+V|58.0|64.8|43.4|\n|MAViL|A+V|56.6|71.8|46.0|\n\nMAViL (A,A+V), outperforms MBT(A,A+V) on Noun and Action in Epic-Kitchen. Note that both MTV and MBT exploited ImageNet-21K (14M) supervision while MAViL is fully self-supervised on AS-2M. And MAViL uses 8-frame video inputs compared to 32-frame in MTV. MAViL encodes only 8 frames to maintain a comparable sequence lengths between the audio (512) and video (768) branches. Additional UCF and HMDB fine-tuning results can be found in the pdf attached in the general response.\n\n**Q1: Linear probing results and more downstream tasks?**\n\nA1: Following the suggestion from you and other reviewers, we conducted additional audio probing on ESC and AS-20K and visual probing on HMDB51, UCF101:\n\n||PT-Data|ESC-FT|ESC-Linear|AS-20K-FT|AS-20K-Linear|\n|---|---|---|---|---|---|\n|XDC|A+V,AS|-|84.8|-|-|\n|AVID|A+V,AS|-|89.1|-|-|\n|BraVe|A+V,AS|-|90.4|-|-|\n|CrissCross|A+V,AS|-|90.5|-|-|\n|SS-AST|A,AS|88.8|85.6|31.0|-|\n|A-MAE|A,AS|94.1|89.5|37.0|-|\n|MAViL|A+V,AS|94.4|90.8|41.8|30.0|\n\n||PT-Data|#frms|U101-Linear|U101-FT|H51-Linear|H51-FT|\n|---|---|---|---|---|---|---|\n|XDC|A+V,IG65M|8|-|84.9|-|48.8|\n|AVID|A+V,AS|8|-|88.6|-|57.6|\n|CrissCross|A+V,AS|8|87.7|89.4|56.2|58.3|\n|MAViL|A+V,AS|8|89.1|90.5|58.3|60.7|\n|BraVe*|A+V,AS|32*|93.0|95.6|69.4|76.5|\n\nNote that the BraVe is with high temporal resolution (multiple-views of 128/32 frames, while 8 frames for MAViL). MAViL does not claim SOTA on action recognition tasks because of the visual footprints we chose (8 frame under 2 fps, 4 second video) and simplified visual augmentation.\n\nMAViL focused more on the audio and audio-video classification performance and targeted on the hardest task on AudioSet to compared with other audio and audio-video baselines in Table 6. With the page limit constraint, we were unable to include dedicated visual results and analysis. We will include video-related results and analysis in the additional page granted in the final version.\n\n\n**Q2: Performance improvement in multimodal self-training beyond 3rd iteration?**\n\nA2: Thank you for bringing this up. The table below extends Table 4b and summarizes the full AS-20K results of iteration 0-5 in multimodal self-training.\n \n|Iter.|None|1|2|3|4|5|\n|---|---|---|---|---|---|---|\n|A|39.0|40.7|41.5|41.8|41.9|41.9|41.8|   \n|V|22.2|24.1|24.6|24.8|25.0|24.8|24.9| \n\nSelf-training for more iteration provides marginal gain after the 3rd/4th iteration. We choose to train 3 iterations for simplicity. We will adjust the presentation in Table 4(b) to include these results."}, {"review_id": "0mtfX3aQI4", "reviewer": "Reviewer_z1cD", "summary": "The authors present Masked Audio-Video Learners (MAViL), pretrained on 3 pseudo tasks and consisting of a 2-stage approach. i) masked reconstruction (adoption of MAE, AudioMAE, VideoMAE) ii) intra-modal and inter-modal contrastive learning (similar contrastive loss like SIMCLR, MOCO), and iii)  student-teacher learning (performed in stage 2, by minimizing the error between earlier pretrained encoders referred to as teachers and the current encoders referred to as students). The proposed framework is pretrained and evaluated with AudioSet and VGGSound. Additionally, audio-to-video retrieval experiments are done on MSR-VTT and YouCook.\n\n\n", "questions": "1. The method uses 10 seconds of audio and 4 seconds of video, which means the network sees asynchronous audio-video segments. Why is such a setup used? Is there a reason for using different durations of audio-video sequences? How would the results be affected if the same 4 seconds of audio and video are used? Could you please provide such results?\n\n2. What would happen if you drop the fusion encoder and directly reconstruct the encoded representation? Could you please provide the results by removing the fusion encoder while keeping the rest of the setup intact? The loss function would remain the same. To clarify further, replace $\\hat{v}$ and $\\hat{a}$ in Eq 1 with the following values: $\\hat{v} =f^{-1}_v(f_v(v'))$ and $\\hat{a} =f^{-1}_a(f_a(a')) $.\n\n3. What would happen if you initialize the encoders with VideoMAE and AudioMAE checkpoints and only perform stage 2, which involves training with Eq. 6? Could you please provide the results for such a scenario? You can utilize the pretraining weights released by the respective papers. This approach would simplify the process, as currently, the intra-modal and inter-modal contrastive losses are optimized twice (in both Eq. 4 and 6), which appears to be somewhat redundant.\n\n4. The authors have maintained a fixed teacher in their approach. However, I'm curious to know how it would perform if the teachers are updated slowly using Exponential Moving Average (EMA). Additionally, the current loss between the teachers and students is Mean Squared Error (MSE). I am curious to understand why MSE is used and what would happen if the loss function is changed to minimize cross-entropy or cosine distance instead. Could you please provide the findings for such experiments?\n\n5. The authors are requested to include the following information in the SOTA comparison tables (Tab. 6) to ensure a fair and informed comparison with other methods:\n- model names (e.g., ViT, ResNet, etc.)\n- number of parameters during pretraining (backbone+auxiliary network parameters such as heads, decoders, etc.) \n- number of params during evaluation, \n- pretraining input size\n- evaluation input size. \n\n6. The authors are strongly recommended to compare MAViL with prior works in linear evaluation. As a reference for such evaluation, you may see Brave (Tab. 6).\n\n7. The authors are suggested to evaluate their method on the following setups. \n- linear and finetune comparison on (i) large-scale datasets like Kinetics400 and Kinetics600; (ii) small-scale datasets like UCF101 and HMDB51; and (iii) temporal-focused datasets like SSv2. Similar to: VideoMOCO, CVRL, p-BYOL, ELO, VideoMAE, Brave, AVID, Robust-xID, Selavi, GDT, CrissCross, STiCA, and CM-ACC.\n- action retrieval on UCF101 and HMDB51 to report R@1, R@5, R@20. Similar to: Robust-xID, Selavi, GDT, and CrissCross.\n- You may check Brave Tab 6 as a reference for a comprehensive comparison.\n\n8. The authors are also recommended to compare the results of ESC50 with prior  AV-SSL methods, please use similar references for comparison. Refs.: XDC, AVID, Brave, CrissCross, CM-ACC. \n\n9. Typo: in Figure 2: iter 2, V-Enc -> V-Dec for the reconstruction part.\n\n10. Minor suggestions: \n- There are a few works that are very similar to MAViL and have some overlapping ideas such as joint-masked reconstruction, student-teacher etc. They are available on arXiv for 5-6 months, I would suggest discussing them as well, e.g., AVMAE and XKD.\n- I recommend adding a pseudocode, it gives more clarity.\n\n=========\nReferences: \n\nVideoMAE: https://arxiv.org/pdf/2203.12602.pdf;\nBrave: https://arxiv.org/pdf/2103.16559.pdf;\nAVID: https://arxiv.org/pdf/2004.12943.pdf;\nRobust-xID: https://arxiv.org/pdf/2103.15916.pdf;\nSelavi: https://arxiv.org/pdf/2006.13662.pdf;\nGDT: https://arxiv.org/pdf/2003.04298.pdf;\nCrissCross: https://arxiv.org/pdf/2111.05329.pdf;\nCM-ACC: https://arxiv.org/pdf/2009.09805.pdf;\nVideoMOCO: https://arxiv.org/pdf/2103.05905.pdf;\nCVRL: https://arxiv.org/pdf/2008.03800.pdf;\np-BYOL: https://arxiv.org/pdf/2104.14558.pdf;\nELO: https://arxiv.org/pdf/2002.12177.pdf;\nSTiCA: https://arxiv.org/pdf/2103.10211.pdf;\nAVMAE: https://arxiv.org/pdf/2212.05922.pdf;\nXKD: https://arxiv.org/pdf/2211.13929.pdf;\n", "limitations": "In summary, this work lacks novelty, a comprehensive evaluation on different downstream benchmarks, and a thorough comparison with several prior works. These weaknesses outweigh the reasons to accept.\n\nThe proposed method has three objectives and all of them are heavily explored in the literature, and I do not see solid technical novelty. First, the masked reconstruction technique has already been implemented in several published works such as MAE, MAE_ST, VideoMAE, AudioMAE, CAV-MAE, and many more. Secondly, contrastive learning in multimodal/cross-modal setups has been widely studied. Lastly, the use of a frozen pretrained network to supervise an online encoder is not novel, whether in uni-modal or multi-modal setups.\n\n\n", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The proposed method is a nice extension of masked reconstruction in a multimodal setup. \n- Fair amount of experiments are conducted as part of ablation to support the design choices.\n- Good presentation and easy to follow.\n- Thanks to the authors for making the code to be public and sharing the details of the implementations which are helpful for reproducibility.", "weaknesses": "- This work does not provide results on linear evaluation. The whole point of SSL is to learn rich abstract representations that can be used through minimal training with labelled data, i.e., linear SVM or FC-tuning.\n- The comparison setup with prior works is insufficient. For example, the SOTA evaluation on visual tasks is extremely weak and most of the entries in Tab. 6 for the video are blank (-), while there are plenty of methods that can be compared. This work does not compare the results of ESC50 with several AV-SSL prior works.\n- The proposed method lacks technical novelty as all the pieces (masked reconstruction, contrastive learning, student-teacher) of the proposed method are already been studied quite heavily in the literature, be it in a uni-modal or multimodal/cross-modal setups.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer z1cD, thank you for reviewing our paper.  \n\nRegarding the \"limitation\" about lacking comprehensive evaluation, please note that the focus of this work is on audio and audiovisual classification and we have reported extensive ablations and experiments in this field - and other reviewers agree with our assessment (e.g. dLj4 “Experiments and ablation studies are comprehensive”, kuAy - “very sufficient ablation study”). We are thankful for z1cD’s suggestion and included additional evaluations on EpicKitchen, UCF, HMDB, ... etc. While we do our best within the limited rebuttal period, we believe that the experiments reported already are extensive and more comprehensive than many published works.\n\nRegarding the “novelty” comment: We believe that MAViL's pre-training approach is novel and original (which is also recognized by RTTA and dLj4) and the resulting model is a technical contribution in of itself. For example, our multimodal student-teacher paradigm is entirely new; and we make one-step further to perform this in the aligned and contextualized space, leading to new SOTA results. We have tried to provide further ablations and hope these would convince you to re-evaluate MAViL's value for the audio and audiovisual community.  \n\nWe concur that MAVIL leverages contrastive learning and masked autoencoding, both of which are very broad concepts. If any work using these concepts is considered “lack of technical novelty”, then the works listed in the review would all categorize to lack novelty w.r.t. each other (MAE, VideoMAE, AudioMAE, SimCLR, Selavi, AVID, CVRL, BYOL, etc. are all forms of masked autoencoding as in MAE and contrastive learning as in MoCo). \n\nFor your questions:\n\n**Q1: Why using (10s-A, 4s-V)? What’s the results of (4s-A, 4s-V)?**\nA1: Thank you for raising this point. The videos, along with their corresponding audio, have a duration of 10 seconds in AudioSet. As MBT, we default to using 8 frames in the visual branch. The choice ensures that the input token lengths remain comparable between the audio and video Transformers, with lengths of 512 and 768, respectively. In response to your valuable suggestion, we conducted experiments with various video spans (t_v) and audio spans (t_a) during the pre-training phase. The AS-20K fine-tuning results are:\n\n|(t_a, t_v)|(10,2)|(10,4)|(10,8)|(4,4)|(2,2)|\n| ----- | ------ | ------ | ----- | ------ | ------ | \n|Audio| 41.8 | 41.8 | 41.7| 40.9 |39.1|\n|Video| 24.9 | 24.8 | 24.5|24.7|24.8 |\n\nThese findings indicate several key points: 1) It’s desirable to have a longer audio span (t_a=10). 2) A shorter video span (t_v=2) performs better, although the distinction is not substantial. 3) The overall performance remains considerably stable, whether utilizing asynchronous sampling or synchronous sampling (eg t_v=t_a=2).\n\n**Q2: The results w/o fusion encoder?**\n\nA2: Without the fusion encoder, the overall performance of MAViL will experience a notable decline (~1 mAP). We acknowledge your observation that the fusion encoder might, at first glance, appear to have a diminished role in Table 1a. However, it also serves an important role in MAViL stage-2 as it provides contextualized targets. Following your suggestion, we conducted an experiment that removes the encoder while keeping the other objectives constant. The “leave-fusion-out” results are:\n\n||A|V|\n|---|---|---| \n|w/ fusion|41.8|24.8|\n|w/o fusion|40.7 (-1.1)|23.8 (-1.0)|\n\nThe consistent drops validate the importance of fusion that exchanges complementary audio-video info for masked reconstruction.\n\n**Q3. Initializing w/ A-MAE and V-MAE for stage-2 training?**\n\nA3: Thank you for the suggestion and we request clarification. If you mean initializing the teacher model with A/V-MAE that generates prediction targets, then this ablation is already discussed in Table-4a column 2 (predicting A/V-MAE output), which significantly lags compared to using MAViL stage-1 as the teacher. This validates our intuition of using aligned and contextualized multimodal targets instead of heterogeneous uni-modal context (39.5->40.7(A),23.1->24.1(V)). If you mean student model initialization in stage-2, we experimented this and there is no change in the final mAP. Please let us know if this is consistent with your request. \n\n**Q4: Instead of 2-stage, what if using EMA? Why MSE and what if using cross-entropy or cosine distance?**\n\nA4: Thank you for your valuable suggestion. We did consider the utilization of EMA instead of stage-wise training during our initial investigation. However, unlike the uni-modal scenario (e.g, EMA in data2vec), we observed unstable training behavior when adopting EMA for student-teacher learning under the multimodal scenario. \n\nThe instability is presumably a consequence of different convergence patterns in each modality.\nConsequently, this makes the design and tuning of mixture/update scheduler suboptimal (i.e., $\\tau$ that controls the EMA update in  Data2vec Sec 3.3). Regrettably, this led to notable performance degradation:\n\n||Stage-wise|EMA|\n|---|---|---| \n|A|41.8|40.3|\n|V|24.8|21.7| \n\nBy employing a 2-stage training process, MAViL not only simplifies the methodology and achieves better performance, but it also mitigates the necessity of devising intricate update schedules, as was required in prior work. \n\nAdditionally, the loss function comparison (MAViL-stage1) is as below:\n\n||MSE|L1|BCE|cos-dist|\n|---|---|---|---|---| \n|A|39.0|38.5|38.3|36.9|\n|V|22.2|21.5|21.5|19.0|\n\nMSE is more robust. Please check our comment below for more discussion.\n\n**Q5. More information for Table 6?**\n\nPlease check the pdf in the global response.\n\n**Q678. Linear Probing and additional fine-tuning experiments**\n\nThank you for the suggestion. We conducted additional linear probing and fine-tuning experiments on EpicKitchen, UCF, HMDB, ESC, AS-20K. Tl;dr, MAViL achieved SOTA on audio and audiovisual classification tasks and comparable video performance with 8-frame inputs. Please check the pdf."}, {"review_id": "AHRPtO94Sf", "reviewer": "Reviewer_dLj4", "summary": "This paper proposes a self-supervised audiovisual representation learning framework. It combines MAE with contrastive learning in both intra and inter modality settings. The authors also suggest a self-training schema where the student model, which only sees masked input, needs to align with the latent representations of the unmasked input produced by the teacher. This shows to improve the quality of learned representations in just a few iterations without relying on external pre-trained teacher models.", "questions": "1. What is difference between $\\tilde{a}$ (similarly $\\tilde{v}$) in line 183 and $\\hat{a}$ (similarly $\\hat{v}$) in line 135?\n2. Eq 5 is confusing: $\\tilde{a}$ is in the raw space (ref Eq 1) but here it is being compared against contextualized latent embedding of the audio (similarly for video). It made sense if $\\tilde{a}$ did not have the decoder (i.e $f^{-1}$). The current form also contradicts with default setting in Table 4.a. Authors should clarify.\n3. Authors should provide intuition on $\\tau$ values for inter and intra losses. Why do we need to be more tolerant for intra? How are the actual values been chosen?\n4. Linear probe is more favorable for evaluation than fine-tuning. As also noted in Sec 4.3, there are so many bells and whistles (e.g all sorts of data augmentations and learning rate annealing strategies etc) when it comes to fine-tuning. Linear probe on the other hand relies on minimal parameters and evaluates the quality of learned representations not the pre-trained model. It would be more helpful if we were able to see linear probe results.\n5. Table 5.b: Different visual backbone initializations impact the quality of the corresponding modality in a meaningful way (as expected) however the performance  is flat on the audio side. Considering fusion layer is supposed to mix the representations between modalities, one would expect to see a lift for the audio backbone too. It would help if authors provide some insights.\n6. Table 6: Is it fair to compare against audio-only models when MAViL was pre-trained with video and audio? ", "limitations": "Yes, they have.", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The work is well-motivated and for the most part easy to follow.\n2. Experiments and ablation studies are comprehensive with some reservations (ref to Questions section)\n3. Components of the proposed approach (Fig 1) are not novel, however putting them together and studying how they interact with one another is valuable.\n4. While there are other works which have combined MAE with contrastive learning, this paper has done a good job distinguishing itself from those concurrent/earlier efforts.\n5. Self-training results are promising (Table 4.b) and the ablation for reconstruction target (Table 4.a) provides valuable insight.\n", "weaknesses": "1. The effectiveness of the fusion layer is not clear: Table 1 does not show any meaningful gain added by the fusion layer over unimodal MAE. It is worth noting that “None” has fewer parameters compared to other columns in Table 1 due to not having a fusion layer. Similarly in Table 4.a when comparing M-Fusion against M-Uni. \n2. Table 3: Intra is lower than Inter, and their combination provides marginal gain over the inter (0.6 on audio and 1.2 on video). It is worth remembering that the addition of intra doubles the computation cost since we need to process two views from each input-modality. Intra loss is one of the main points that this work is using to distinguish itself from CAV-MAE. \n3. Table 5.e summarizes the effectiveness of the contributions claimed by MAViL. We can see that the addition of joint AV-MAE adds nothing meaningful to the baseline. Similarly is the marginal gain due to the addition of intra loss over the inter one. The student teacher learning is the only novel contribution that shows to be effective. \n4. Minor note (not a weakness): arxiv paper https://arxiv.org/abs/2303.12001  also combines MAE and contrastive losses though the setup and modalities are different from the current paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your attention to recognizing the unique value of MAViL's multimodal self-training approach, using aligned and contextualized representations with MAE and contrastive learning. Additionally, we thank you for recognizing our comprehensive analysis and experiments. Please see our response to your comments and questions:\n\n**W1 Effectiveness of fusion encoder?**\n\nA-W1: Without fusion, MAViL's performance is expected to drop (\\~1 mAP). We acknowledge your observation that audio-video might appear to have a relatively diminished role at first glance, contributing \\~0.3\\-0.4 mAP over vanilla A/V MAEs. However, as you pointed out, fusion encoder is also significant in stage-2 as it provides contextualized targets. It's important to clarify that, in Table 4a, M-Uni and M-Fusion represent distinct outputs generated by the same teacher model trained with the fusion encoder.\n\nTo accurately assess the effectiveness of fusion, we removed fusion layer and kept all other objectives same in stage-1 (Eq 4) and stage-2 (Eq 6). The result of this “leave-fusion-out” experiment is as below:\n\n||A|V|\n|---|---|---| \n|w/ fusion|41.8|24.8|\n|w/o fusion|40.7 (-1.1)|23.8 (-1.0)|\n\nThe consistent drops w/o fusion validate the effectiveness of using the fusion layer for exchanging complementary audio-video context.\n\n\n**W2 Smaller gain of intra-contrast and marginal gain with combination?**\n\nA-W2: We concur with your discovery that inter-modal contrast is more effective (+1.6 (A), +3.3 (V)) than intra-modal contrast (+1.3 (A), +2.1 (V)). It's important to note, however, that these objectives are complementary and mutually reinforcing, as they encourage the alignment of semantically related audio and video representations from distinct angles. By exploiting both inter-contrast and intra-contrast objectives, MAViL attains the best performance (2.2 (A), +4.5 (V)), surpassing the performance of using either objective in isolation.\n\nAlso, we acknowledge your comment about additional computation required for encoding the second view. However, unlike conventional contrastive learning methods (e.g., SimCLR, MoCo, Wav2Vec2), MAViL encodes only 20% of tokens. This substantial reduction significantly mitigates the computational burden.\n\n\n**Q1-2 Difference between $\\hat{a}$ and $\\tilde{a}$? and the space of $\\tilde{a}$ in Eq 5?**\n\nA1-2: Apologies for any confusion. $\\hat{a}$ and $\\hat{v}$ are the prediction of raw spectrograms and raw pixels. In contrast, $\\tilde{a}$ and $\\tilde{v}$ are the prediction of contextualized audio/video embeddings in the shared latent space. \n\nIn stage-2 (Eq 5), MAViL encodes un-masked raw tokens, conducts fusion, and decodes to predict the contextualized embeddings generated by a previously trained teacher’s encoder, encompassing full views. For example, a student's decoder predicts *BxNxH* contextualized audio embeddings (Eq 5) (H: embedding dimension), instead of predicting *BxNx1x16x16* (spec. patches) (Eq 1).\n\n**Q3 Intuition of $\\tau$ in inter/intra contrast and the choice? Should intra-modal be stricter/tolerant?**\n\nA3: Thank you for your valuable input. We set temperature $\\tau = 0.1$ for inter-modal and \\tau=1.0 for intra-modal contrast.  A higher temperature used in the NCE serves to accentuate the differentiation between positive and negative samples, leading to enhanced alignment or distinction. Intuitively, alignment/distinction is more noticeable when comparing samples within the same modality (eg, clips from the same video, versus other videos) compared to samples from different modalities (eg, a scene of a person talking and his speech, vs other possible speeches). With this in mind, we set a stricter temperature $\\tau=1.0$ for intra-modal contrast, as the alignment/distinction within the same modality demands more precision. Conversely, for inter-modal we adopt a more permissive \\tau=0.1. The best values were selected via a grid search over 0.1, 0.2, 0.5, and 1.0.\n\n**Q4 Linear probe results?**\n\nA4: Thank you for the suggestion. We conducted additional audio probing on ESC and visual probing on HMDB, UCF:\n\n||PT-Data|ESC50-Linear|\n|---|---|---|\n|XDC|A+V,AS|84.8|-|\n|AVID|A+V,AS|89.1|-|\n|BraVe|A+V,AS|90.4|-|\n|CrissCross|A+V,AS|90.5|-|\n|SSAST|A,AS|85.6|-|\n|A-MAE|A,AS|89.5|-|\n|MAViL|A+V,AS|90.8|30.0|\n\n||PT-Data|#fms|UCF101-Linear|HMDB51-Linear|\n|---|---|---|---|---|\n|CrissCross|A+V,AS|8|87.7|56.2|\n|MAViL|A+V,AS|8|90.5|58.3|\n|BraVe*|A+V,AS|32*|93.0|69.4|\n\nAs shown, MAViL yields the best audio classification and comparable video classification performance (8-frms).\n\n**Q5 Saturated audio mAP with visual backbone initialization?**\n\nA5: This is an insightful observation! MAViL exploits aligned and complementary info present in audio-video pairs. Our approach assumes meaningful connections exist between audios and videos. However, these associations might not be perfect in real-world audio/video. For instance, audios in AudioSet could arise where the corresponding videos possess low visual quality (eg, text-only footages) or irrelevant context. These noises hinder the audio branch from fully capitalizing on the visual branch, which might consequently leads to saturated benefits even with an better visual initialization.\n\n**Q6 Fair to compare with audio-only?**\n\nA6: We believe it's crucial to highlight the value of audio-video pre-training, as it not only amplifies performance in audio-video classification and retrieval but also refines the representations of each individual modality in downstream tasks. Being consistent with other audio-video studies in this field (eg MBT, CAV-MAE, Perceiver, and more), we consider it appropriate to draw comparisons and demonstrate the advantages of audio-video models over audio-only models. This is justified by the fact that these models leverage distinct viewpoints or modalities inherent within the same dataset, rather than incorporating additional data (eg by using the same AudioSet)."}]

Author rebuttal:
Dear Reviewers and Area Chairs,

We thank you for the time you invested in reviewing our paper and providing invaluable suggestions that have significantly improved the quality of MAViL. In particular, we wish to express our heartfelt thanks to RTTA and dLj4 for acknowledging the originality and substantial contribution of MAViL's multimodal self-training approach that reconstructs aligned and contextualized representations without reliance on external models or data sources. Furthermore, we extend our thanks to kuAy, RTTA, and dLj4 for acknowledging our thorough model analysis, comprehensive study, and state-of-the-art performance in audio and audiovisual classification and retrieval tasks across seven datasets. Finally, we are grateful for the valuable comments and questions raised by z1cD, kuAy, RTTA and dLj4. Please find the individual rebuttal for our detailed response.

Enclosed within the attached PDF file, we present supplementary fine-tuning and linear probing experiments conducted on EpicKitchen, UCF101, HMDB51, ESC, and AudioSet-20K, in response to reviewer’s requests of additional experiments.

We thank you for your review. Please let us know if you have any remaining concerns and we are happy to address them.

Meta-review: This is a borderline submission. Two reviewers argued that the strong empirical results are sufficient for acceptance. The other two reviewers found that the empirical evaluation is not sufficient for such claims, and expected to see improvements to the unimodal baselines and better positioning with respect to substantial amount of related work. In the rebuttal the authors provided additional empirical evidence which suggests which strengthened the main claims and clarifies the impact of individual loss components. I will recommend acceptance of this submission given strong empirical performance.
Decision: Accept (poster)