SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Po-Yao Huang $^{1}$  Vasu Sharma $^{1}$  Hu Xu $^{1}$  Chaitanya Ryali $^{1}$  Haoqi Fan $^{1}$  Yanghao Li $^{1}$  Shang-Wen Li $^{1}$  Gargi Ghosh $^{1}$  Jitendra Malik $^{1,2}$  Christoph Feichtenhofer $^{1}$
$^{1}$ FAIR, Meta  $^{2}$ University of California, Berkeley
We present Masked Audio-Video Learners (MAViL) to learn audio-visual representations with three complementary forms of self-supervision: (1) reconstructing masked raw audio and video inputs, (2) intra-modal and inter-modal contrastive learning with masking, and (3) self-training to predict aligned and contextualized audio-video representations learned from the first two objectives. Empirically, MAViL achieves state-of-the-art audio-video classification performance on AudioSet (53.3 mAP) and VGGSound (67.1% accuracy), surpassing recent self-supervised models and supervised models that utilize external labeled data. Notably, pre-training with MAViL not only enhances performance in multimodal classification and retrieval tasks, but it also improves the representations of each modality in isolation, without relying on information from the other modality during uni-modal fine-tuning or inference. The code and models are available at https://github.com/facebookresearch/MAViL.
We study self-supervised learning (SSL) from audio and video, two rich, heterogeneous, yet closely related modalities of human perception. There are two primary forms of self-supervision commonly used: reconstruction and contrastive learning. By reconstructing masked text tokens on large-scale corpora, BERT [20] pre-training has achieved groundbreaking results in various NLP tasks. Masked autoencoders (MAEs) have recently emerged as powerful tools for learning uni-modal representations in various modalities such as image [37], video [27], and audio [41]. They employ an asymmetric encoder-decoder architecture with a substantial portion of encoder inputs being masked, resulting in efficient uni-modal SSL. Additionally, contrastive learning has been widely used to learn coordinated multimodal representations for image-text [75, 93] and audio-visual [67, 71] tasks.
In this work, we develop a novel approach that integrates MAE and contrastive learning to enhance audio-video representation learning for both audio-video and audio-only tasks. This integration poses unique challenges under the multimodal paradigm, necessitating meticulous model design to effectively manage diverse multimodal inputs in MAE, while also adapting contrastive learning to accommodate masking of the majority of inputs for efficiency. Our method, Masked Audio-Video Learners (MAViL), comprises a pair of encoders to encode audio and video, a fusion encoder, and distinct decoders tailored for reconstructing raw inputs or contextualized representations in each modality (as illustrated in Fig. 1). We design MAViL with three types of objectives outlined next.
Firstly, we extend uni-modal MAE to multimodal and utilize a fusion encoder to exchange information from all modalities. MAViL reconstructs raw inputs that has been removed under a high masking ratio (e.g.  $80\%$ ). With this, it learns complementary audio-video representations by reconstructing a single modality input, with the supplementary context from the other. Secondly, we adapt contrastive learning [34, 85], with the majority of uni-modal inputs being masked, to learn an aligned audio-video latent space efficiently. MAViL employs two types contrastive objectives: (i) An inter-modal contrast that brings together paired video and audio clips from the same video and contrasts them with other
Figure 1: Masked Audio-Video Learners (MAViL) exploit three objectives for learning representations from audio video pairs with masking: (1) Raw audio-video reconstruction. (2) Inter-modal and intra-modal contrastive learning with masking. (3) Reconstructing aligned and contextualized audio-video representations via student-teacher learning. (Please see Fig. 2 for details.)
unpaired clips sampled in a mini-batch. (ii) An intra-modal contrast that draws closer the two masked views of the same audio or video, while pushing away other samples from the same modality.
Thirdly, unlike conventional MAEs that focus on reconstructing heterogeneous and raw inputs (audio or video), we propose a novel pre-training task that reconstructs homogeneous and contextualized audio-video representations in a joint (audio-and-video) latent space (illustrated in Fig. 2).
We are motivated by the recent successes in visual SSL that use disparate teacher models to generate contextualized representations as reconstruction targets. For example, BEiT [8] uses features by DALL-E [77] as its target. MaskFeat [91] predicts masked target features such as HOG [19] or DINO [10]. Inspired by these uni-modal teachers, we take a step further to propose a new masked contextualized audio-video reconstruction pretext task. Our core idea is: instead of predicting raw inputs in heterogeneous modalities, the model with masked-view inputs jointly predicts contextualized audio-video representations in the homogeneous (aligned) latent space generated by a teacher (i.e. an identical model pre-trained with the masked reconstruction and contrastive objectives above) with complete-view inputs. This approach ensures the students learn well-aligned and contextualized audio-video representations and thus improves their performance on downstream tasks. To achieve this, without relying on external teacher models, we employ a simple two-stage self-training framework (illustrated in Fig. 2). In stage-1, we train the teacher MAViL and use raw inputs as its reconstruction targets. In stage-2, the student MAViL (the final model for later fine-tuning) learns to reconstruct the aligned and contextualized audio-video representations generated by the MAViL teacher.
Our experimental results confirm MAViL's superiority in audio-video classification (AudioSet-20K, AudioSet-2M, VGGSound) and retrieval tasks (VTT and YouCook) where it surpasses the best self-supervised and supervised pre-trained models by large margins. Notably, MAViL not only learns strong joint audio-video representations, but can also improve single modality encoders, without using the other modality it pre-trained with during fine-tuning or inference.
In summary, MAViL makes the following contributions for learning self-supervised audio-video representations: (1) Efficient contrastive learning for both intra-modal and inter-modal context via masking. (2) Introducing a new pretext task for multimodal MAE that predicts aligned and contextualized audio-video representations, surpassing the performance of reconstructing uni-modal raw inputs as used in conventional uni-modal/multimodal MAE. (3) Achieving new state-of-the-art performance in 7 audio-visual classification, audio-visual retrieval tasks, and (4) audio-only tasks under the SSL setup without using labeled data for pre-training.
Supervised Audio-Video Models. The connection between visual signals and co-occurring audio context for language acquisition and world comprehension in infants [80] has motivated the research of audio-visual learning [76, 18]. Previous studies have explored audio-visual ASR [14, 73] and
person identification [1] prior to the deep learning era. More recently, there has been significant attention on audio-video representation learning for classification [65, 46, 22, 43, 95, 64]. However, these supervised approaches rely on abundant labeled data, which are costly and time-consuming to obtain. While efforts have been made to create large-scale labeled datasets [35], the dependency on extensive annotation hinders progress in audio-video modeling. In contrast, MAViL focuses on self-supervised learning of robust audio-video representations without the need for labeled data.
Self-Supervised Audio-Video Representation Learning. To exploit abundant unlabeled video and audio content on the Internet and reduce annotation efforts, self-supervised techniques have been explored for learning audio-video representations [4, 2, 3, 48, 82]. Inter-modal contrastive learning is a widely used approach that learns to associate paired audio-video clips as self-supervision [57, 67, 63, 71]. Techniques such as data augmentation [70, 87] and harder negatives mining [98, 78, 62] have been studied to improve its performance. MAViL unifies masked autoencoding and contrastive learning. We notice concurrent and independent study CAV-MAE [32] and AV-MAE [29] recently explored joint masked autoencoding to reconstruct raw audio and video. Unlike CAV-MAE, MAViL takes a step further to incorporate intra-modal contrastive learning from two masked views. Most importantly, MAViL employs student-teacher learning to reconstruct contextualized and aligned representations instead of raw inputs as in CAV-MAE and AV-MAE. These innovations lead to MAViL's superior performance.
Student-teacher learning or knowledge distillation (KD) [40, 84, 69] was originally developed for model compression [60, 17], aiming to transfer knowledge from a larger teacher model to a smaller student model. In the context of SSL, KD has recently gained attention for improving the distribution of representations by distilling [93] or by reconstructing contextualized targets from a teacher model [33, 8, 92]. Approaches like MoCo [36], DINO [10], and dBOT [53] utilize self-training to bootstrap targets from previous model snapshots during pre-training. Data2vec [7] performs self-training in disparate modalities, where each one bootstraps complete-view contextualized targets independently. In contrast, to our best knowledge, MAViL is the first work that employs masked multimodal context for self-training. It uniquely incorporates complete-view multimodal fusion in the teacher model, while the student model receives masked-view inputs.
3 Masked Audio-Video Learners
We introduce Masked Audio-Video Learners (MAViL), a self-supervised audio-video representation learning framework (see Fig. 1). MAViL consists of two stages: In stage-1, MAViL jointly reconstructs raw spectrograms and RGB frames by exploiting complementary information from each modality ( $\S 3.1$ ), and couples this with contrastive learning to encourage alignment between semantically similar instances, both within and across modalities. ( $\S 3.2$ ). In stage-2 (see Fig. 2), we employ either the stage-1 model trained with raw targets (in iter. 1) or the last trained stage-2 model (in iter.  $2+$ ) as the teacher for self-training. We use the teacher's aligned and contextualized audio-video representations, obtained with complete-view inputs, to guide the student with masked-view inputs ( $\S 3.3$ ). In the following, we provide details of these three types of self-supervision, starting with MAViL-stage1 trained with raw inputs.
3.1 Masked Raw Audio-Video Reconstruction
Human perception involves processing visual and acoustic context jointly. In line with this, MAViL utilizes multimodal Transformers to fuse and exploit the complementary information from both audio and video. It aims to reconstruct audio and video simultaneously as self-supervision, which sets it apart from uni-modal MAE approaches such as MAE [37], Audio-MAE [41], or Video-MAE [27].
Given a raw audio-video pair  $(\mathrm{a},\mathrm{v})\in \mathcal{D}$ , we begin by patchifying and tokenizing raw audio spectrograms and video frames into audio and video tokens. This involves applying (audio/video) transforms, followed by 2D/3D-convolutions and flattening. This process embeds raw inputs into  $\mathbf{a} = [a_1\ldots a_N]$  audio spectrogram tokens and  $\mathbf{v} = [v_{1}\dots v_{M}]$  video tokens, where  $a_{i},v_{j}\in \mathbb{R}^{H}$ . To incorporate positional information, similar to MAE, we employ fixed 2D sinusoidal positional embeddings with the embedded tokens for each modality. We then randomly mask the majority (i.e.  $80\%$  ) of audio and video tokens. Only the remaining  $20\%$  unmasked audio  $(\mathbf{a}^{\prime})$  and video  $(\mathbf{v}^{\prime})$  tokens
are respectively fed into the audio  $(f_{\mathrm{a}}(.))$  and video  $(f_{\mathrm{v}}(.))$  Transformer encoders. This process results in uni-modal embeddings, denoted as  $\mathbf{a}_{\mathrm{um}} = f_{\mathrm{a}}(\mathbf{a}^{\prime})$  and  $\mathbf{v}_{\mathrm{um}} = f_{\mathrm{v}}(\mathbf{v}^{\prime})$ .
Following the uni-modal encoders, we incorporate a multimodal fusion encoder denoted as  $g_{\mathrm{av}}(.)$  to model multimodal context. We explore two variants for this purpose: Vanilla Transformers [86] and Multimodal Bottleneck Transformers (MBT) [64]. For vanilla Transformers, we jointly encode the audio and video tokens by:  $(\mathbf{a}_{\mathrm{um}}^{l + 1}||\mathbf{v}_{\mathrm{um}}^{l + 1}) = \mathrm{Transformer}^l (\mathbf{a}_{\mathrm{um}}^l ||\mathbf{v}_{\mathrm{um}}^l)$ , where “||” represents concatenation. Details of MBT implementation is in Appendix. In both variants, we stack  $L$  Transformer layers to obtain the jointly encoded top-layer outputs  $\mathbf{a}_{\mathrm{mm}}$  and  $\mathbf{v}_{\mathrm{mm}}$ .
For reconstruction, we employ vanilla Transformer blocks as the audio  $f_{\mathrm{a}}^{-1}(.)$  and video  $f_{\mathrm{v}}^{-1}(.)$  decoders. The fusion encoder's outputs  $(\mathbf{a}_{\mathrm{mm}}$  and  $\mathbf{v}_{\mathrm{mm}})$  are firstly projected and padded with trainable [MASK] tokens. After restoring the original order (time-frequency for audio and space-time for video tokens), we add the decoders' (fixed 2-D sinusoidal) positional embeddings and input the restored sequences into the decoders. At the top of the decoders, we incorporate linear heads to reconstruct the raw inputs. Specifically, the decoder outputs for spectrogram reconstruction are denoted as  $\hat{\mathbf{a}} = f_{\mathrm{a}}^{-1}(g_{\mathrm{av}}(f_{\mathrm{a}}(\mathbf{a}'))))$  and for RGB frame reconstruction as  $\hat{\mathbf{v}} = f_{\mathrm{v}}^{-1}(g_{\mathrm{av}}(f_{\mathrm{v}}(\mathbf{v}'))))$ . For notation clarity, we omit the [MASK] tokens and linear projection head. Let  $\hat{\mathbf{a}}_i, \mathbf{a}_i^{\mathrm{raw}} \in \mathbb{R}^{H_{\mathrm{raw}}^{\mathrm{a}}}$ ;  $i = 1 \dots n$  denote the audio decoder's output and the ground truth reference of the  $i$ -th masked spectrogram patch. Similarly,  $\hat{\mathbf{v}}_j, \mathbf{v}_j^{\mathrm{raw}} \in \mathbb{R}^{H_{\mathrm{raw}}^{\mathrm{v}}}$ ;  $j = 1 \dots m$  for video patches. In masked audio-video reconstruction, MAViL is self-supervised by minimizing the mean squared error (MSE) loss  $\mathcal{L}_r^{\mathrm{raw}}$  defined as:
$$
\mathcal {L} _ {r} ^ {\text {r a w}} = \frac {1}{n} \sum_ {i = 1} ^ {n} \left(\hat {\mathbf {a}} _ {i} - \mathbf {a} _ {i} ^ {\text {r a w}}\right) ^ {2} + \frac {1}{m} \sum_ {j = 1} ^ {m} \left(\hat {\mathbf {v}} _ {j} - \mathbf {v} _ {j} ^ {\text {r a w}}\right) ^ {2}. \tag {1}
$$
3.2 Contrastive Audio-Video Learning with Masking
Contrastive learning is widely used to learn uni-modal [13, 36, 15] and multimodal [75, 71] representations by aligning multiple "views" of the same instance. These views can be either within-modality observations of the instance itself (e.g., the same audio under different volumes) or semantic observations across modalities (e.g., a video and its corresponding audio). MAViL utilizes InfoNCE [85] loss for contrastive learning. Let  $\mathbf{x} = [\mathbf{x}_1 \ldots \mathbf{x}_B]$ ,  $\mathbf{y} = [\mathbf{y}_1 \ldots \mathbf{y}_B]$ ;  $\mathbf{x}_i, \mathbf{y}_j \in \mathbb{R}^H$  be the instance-level representations of audio/video in a batch of size  $B$ . The contrastive loss  $\mathcal{L}_{\mathrm{c}}(\mathbf{x}, \mathbf{y})$  is defined as:
$$
\mathcal {L} _ {\mathrm {c}} (\mathbf {x}, \mathbf {y}) = - \frac {1}{B} \sum_ {i = 1} ^ {B} \log \frac {\exp \left(\mathbf {S} \left(\mathbf {x} _ {i} , \mathbf {y} _ {i}\right) / \tau\right)}{\sum_ {j = 1} ^ {B} \exp \left(\mathbf {S} \left(\mathbf {x} _ {i} , \mathbf {y} _ {j}\right) / \tau\right)}, \tag {2}
$$
where  $\mathbf{S}(\mathbf{x}_i,\mathbf{y}_j) = \frac{\mathbf{x}_i^T\mathbf{y}_j}{\|\mathbf{x}_i\|\|\mathbf{y}_j\|}$  is the cosine similarity between  $\mathbf{x}_i$ ,  $\mathbf{y}_j$  and  $\tau$  is the softmax temperature. MAViL employs two types of contrastive losses for self-supervision listed below:
Inter-modal contrastive learning facilitates alignments across modalities. We first average the sequence of uni-modal encoder outputs<sup>1</sup> as the instance-level representations, namely,  $\mathbf{a}_{\mathrm{emb}} = \mathrm{Avg}(\mathbf{a}_{\mathrm{um}})$  and  $\mathbf{v}_{\mathrm{emb}} = \mathrm{Avg}(\mathbf{v}_{\mathrm{um}})$ . The positive pairs (the numerator in Eq.(2)) consist of video and audio clips from the same video, while all the other combinations of sampled audio-video pairs are considered negatives (the denominator). Inter-modal contrast encourages the representations of paired audio and video to be closer to each other, while simultaneously pushing away mismatched ones.
Intra-modal contrastive learning promotes alignment within each modality. It aims to bring the representations of different views, such as different augmented versions of an audio (or video), closer to each other. To achieve this, for each modality, we apply random masking and sample a second view that contains  $20\%$  tokens for encoding. The idea is to treat masking as a form of augmentation to generate two contrasting views in the same modality, which has been proven effective in visual SSL [36, 13]. The two views of the same instance are considered as a positive pair, which are then contrasted against the other (negative) combinations of the instances in the same modality. Formally, let  $\bar{\mathbf{a}}_{\mathrm{emb}}$  and  $\bar{\mathbf{v}}_{\mathrm{emb}}$  be the embeddings of the second-view audio and video. MAViL employs the
<sup>1</sup>We do not use the fusion output  $\mathbf{a}_{\mathrm{mm}}$  and  $\mathbf{v}_{\mathrm{mm}}$  for contrastive learning since the fusion layer provides a shortcut/leakage of information exchange for every paired audio-video clip. This results in sub-optimal performance for contrastive learning.
Figure 2: Masked contextualized audio-video reconstruction in the joint latent space. Stage-1 (left): Training MAViL with raw inputs as targets. Stage-2 (right): Self-training MAViL student by reconstructing MAViL teacher's aligned and contextualized audio-video representations generated with complete inputs. Repeat stage-2 for  $K$  iterations. In the first iteration of stage-2, the stage-1 model is used as the teacher. In subsequent iterations (iteration  $2+$ ), the last trained stage-2 student model serves as the new teacher.
inter-modal  $(\mathcal{L}_{\mathrm{c}}^{\mathrm{inter}})$  and intra-modal  $(\mathcal{L}_{\mathrm{c}}^{\mathrm{intra}})$  contrastive objectives defined as:
$$
\mathcal {L} _ {\mathrm {c}} ^ {\text {i n t e r}} = \frac {1}{2} \left[ \mathcal {L} _ {c} \left(\mathbf {a} _ {\text {e m b}}, \mathbf {v} _ {\text {e m b}}\right) + \mathcal {L} _ {c} \left(\mathbf {v} _ {\text {e m b}}, \mathbf {a} _ {\text {e m b}}\right) \right], \quad \mathcal {L} _ {\mathrm {c}} ^ {\text {i n t r a}} = \frac {1}{2} \left[ \mathcal {L} _ {c} \left(\mathbf {a} _ {\text {e m b}}, \bar {\mathbf {a}} _ {\text {e m b}}\right) + \mathcal {L} _ {c} \left(\mathbf {v} _ {\text {e m b}}, \bar {\mathbf {v}} _ {\text {e m b}}\right) \right], \tag {3}
$$
These contrastive losses effectively learn a latent audio-video space where semantically similar instances are close to each other. Note that unlike prior work (e.g. [36, 13]), MAViL performs contrastive learning under masked-view. This leads to improved computation efficiency as only a small portion of input tokens are encoded for contrast. And different from CAV-MAE [32], MAViL additionally incorporates intra-modal contrast, which yields superior performance. Overall, let  $\alpha$  and  $\beta$  be the weights that balance loss terms, MAViL-stage1 is self-supervised by minimizing:
$$
\mathcal {L} _ {\mathrm {M A V i L}} = \mathcal {L} _ {r} ^ {\text {r a w}} + \alpha \mathcal {L} _ {\mathrm {c}} ^ {\text {i n t e r}} + \beta \mathcal {L} _ {\mathrm {c}} ^ {\text {i n t r a}}, \tag {4}
$$
3.3 Masked Contextualized Audio-Video Reconstruction
To learn robust audio-video representations, we go beyond raw input reconstruction in uni-model MAEs [37, 41, 25], multimodal MAE [6], and CAV-MAE [32]. We propose a new pretext task that reconstructs contextualized audio-video representations in the joint latent space. To achieve this, we employ a simple two-stage training framework illustrated in Fig. 2. In stage-1, we train MAViL with Eq.(4) to reconstruct raw inputs. In stage-2, we employ a student-teacher learning framework for iterative self-training. In the first iteration, the pre-trained MAViL from stage-1 is frozen and serves as the teacher model. It generates audio-video representations with complete-view inputs, which are used as the reconstruction targets to guide the re-initialized student MAViL. In the subsequent iterations, the last trained stage-2 student MAViL serves as the new teacher.
Formally, we provide the teacher model's encoders with complete-view audio and video inputs to generate aligned and contextualized targets:  $\mathbf{a}^{\mathrm{Teacher}}\| \mathbf{v}^{\mathrm{Teacher}} = g_{\mathrm{av}}^{\mathrm{Teacher}}(f_{\mathrm{a}}^{\mathrm{Teacher}}(\mathbf{a})\| f_{\mathrm{v}}^{\mathrm{Teacher}}(\mathbf{v}))$ . The student MAViL then learns to reconstruct these contextualized targets with the masked-view inputs. Precisely,  $\tilde{\mathbf{a}} = f_{\mathrm{a}}^{-1}(g_{\mathrm{av}}(f_{\mathrm{a}}(\mathbf{a}')))$  for audio and  $\tilde{\mathbf{v}} = f_{\mathrm{v}}^{-1}(g_{\mathrm{av}}(f_{\mathrm{v}}(\mathbf{v}')))$  for video, where  $\mathbf{a}^{\mathrm{Teacher}}, \mathbf{v}^{\mathrm{Teacher}}, \tilde{\mathbf{a}}, \tilde{\mathbf{v}} \in \mathbb{R}^{H}$ . The contextualized reconstruction objective is defined as:
$$
\mathcal {L} _ {r} ^ {\text {c o n t e x t}} = \frac {1}{n} \sum_ {i = 1} ^ {n} \left(\tilde {\mathbf {a}} _ {i} - \mathbf {a} _ {i} ^ {\text {T e a c h e r}}\right) ^ {2} + \frac {1}{m} \sum_ {j = 1} ^ {m} \left(\tilde {\mathbf {v}} _ {j} - \mathbf {v} _ {j} ^ {\text {T e a c h e r}}\right) ^ {2}. \tag {5}
$$
In the stage-2 training, we jointly minimize the masked contextualized reconstruction loss and the contrastive loss. The stage-2 (student) MAViL's objective is:
$$
\mathcal {L} _ {\mathrm {M A V i L}} = \mathcal {L} _ {r} ^ {\text {c o n t e x t}} + \alpha \mathcal {L} _ {\mathrm {c}} ^ {\text {i n t e r}} + \beta \mathcal {L} _ {\mathrm {c}} ^ {\text {i n t r a}}. \tag {6}
$$
Note that Eq.(6) contains pure latent targets<sup>2</sup>. After pre-training, we then fine-tune the audio/video encoders in the final stage-2 MAViL student model in the downstream tasks.
${}^{2}$  Empirically,we found no gain when multi-tasking with stage-1 autoencoding loss on raw pixel/spectrogram.
We performed comprehensive evaluations, including audio-video classification tasks on AudioSet [28] (AS-2M and AS-20K), and VGGSound [11]. Also, we conducted audio-to-video retrieval experiments on MSR-VTT [96] and YouCook [100]. We use AS-20K for model analysis and ablation studies.
AudioSet contains 2 million 10-second YouTube clips for audio event detection. 527 event types are weakly labeled [51, 38, 39] for each clip, and multiple events can occur in one clip. AudioSet's full training set has two subsets: A class-wise balanced (22K clips) and an unbalanced (2M clips) set. The eval set has 20K clips. We downloaded 1.97M unbalanced training, 20K balanced training, and 19K evaluation clips. We use the full (unbalanced+balanced) training set for pre-training. In the AS-2M task, we fine-tune on the full training set. In the AS-20K task, we fine-tune only on the 20K balanced training set. We report the classification mAP on the 19K eval set used by AST [30].
VGGSound comprises approximately 200K 10-second video clips annotated with 309 event types that include human actions, sound-emitting objects, etc. Unlike AudioSet, VGGSound ensures that an audio event is also visually present in its corresponding clips. VGGSound is divided into 183K training and 15K testing samples. We report top-1 testing classification accuracy.