SlowGuess/NeurIPS_2023_OCR huggingface dataset:
MAViL adopts different temporal footprints for audio and video. For audio, following [64, 30], it transforms a 10-second audio under 16K sampling rate into 128 Mel-frequency bands with a 25ms Hanning window shifting every 10ms. The resulting spectrogram is of dimension  $1024 \times 128$ . MAViL then tokenizes it into non-overlapping  $16 \times 16$  patches where both time and frequency have a kernel and stride of 16. The flattened audio tokens have a sequence length  $N$  of 512. For video, MAViL processes 4-second clips under 2 frames per second. Each frame has a size of  $224 \times 224$ . Tokenization is achieved by a 3D convolution, where the spatial kernel and stride are 16, and the temporal kernel and stride are 2. The flattened video tokens have a sequence length  $M$  of 784.
Following the design choices in MAE [37], MAViL employs 12-layer Transformers (ViT-B) with 12 attention heads as the encoders for each modality. The embedding dimension  $H$  is set to 768. The audio-video fusion encoder layer consists of a 2-layer Transformer (vanilla or MBT) on top of the uni-modal encoders. Similarly, the audio and video decoders utilize 8-layer Transformers with an embedding dimension of 512 and 16 attention heads. MAViL's audio/video encoder and decoder have 86M and 27M parameters, respectively. The floating point operations (FLOPs) for the audio encoder are 48.6G, comparable to the audio encoders in Audio-MAE [41] and CAV-MAE [32].
For pre-training MAViL's audio branch, we randomly initialize it from scratch. For the visual branch, we either randomly initialize it or initialize it with the self-supervised MAE [37] pre-trained on ImageNet (compared in Table 6). Notably, MAViL operates under the fully self-supervised setup.
4.3 Experimental Setup
We pre-train MAViL on AS-2M without using any of AS-2M labels. We use  $80\%$  masking for audio and video. For balancing the losses in Eq.(6), we set  $\alpha = 0.1$ ,  $\tau_{\mathrm{c}}^{\mathrm{inter}} = 0.1$  and  $\beta = 0.01$ ,  $\tau_{\mathrm{c}}^{\mathrm{intra}} = 1.0$ . These hyperparameters scale the gradients from the three losses into a comparable range to improve training stability. We pre-train with 64 V100 GPUs with a 512 accumulated batch size and a 0.0002 learning rate. We pre-train for 20 epochs in stage-1 and in each iteration of stage-2 (for  $(K = 3)$  iterations). Each session takes 20 hours, resulting in a total pre-training time around 80 hours.
We then evaluate audio and video representations by fine-tuning the pre-trained MAViL in audio-video classification tasks on AS-20K, AS-20M, and VGGSound. We test 3 scenarios: (1) audio-only, (2) video-only, and (3) audio+video. Following MAE [37], we only keep the pre-trained encoders and use the average-pooled top-layer outputs for classification. We adopt the standard fine-tuning pipeline and augmentation in prior audio/audio-video classification works [30, 41, 64]. Specifically, we employ SpecAug [68], mixup [99], balanced sampling [51], and fine-tuning masking [41]. For video, we use standard video augmentations used in video classification [90, 27]. To address the discrepancy in convergence rate between audio and video, we apply a  $50\%$  learning rate reduction for the video
encoder during the audio+video (fusion) fine-tuning. Fine-tuning for 100 epochs on AS-2M takes 10 hours. Please refer to Appendix for additional implementation and experimental details.
In the following, we first use AS-20K to analyze MAViL's performance in §4.4. We then present the main comparison to prior works in §4.5. The gray entries represent the shared default setup.
Masked Raw Audio-Video Reconstruction. Table 1 shows the contribution of MAViL's audio-video fusion encoder layer, which exploits complementary multimodal information for reconstructing raw inputs in stage-1. The 'None' (col. 1) indicates the uni-modal MAE baselines without the fusion layer, namely Audio-MAE and Video-MAE under our implementation. The fusion layer, whether it's a vanilla Transformer or MBT, contributes up to  $0.4\mathrm{mAP}$  gains (col. 2-5). While the MBT layers show better results, the improvements is not significant compared to using vanilla Transformers. Also, increasing the depth of the fusion encoder only leads to marginal gains. For simplicity, we default to using a 2-layer vanilla Transformer as the fusion encoder.
Fusion	None	\( Vanilla^{(2)} \)	\( Vanilla^{(4)} \)	\( MBT^{(4)} \)
Audio	36.4	36.8 \( (+0.4) \)	36.7 \( (+0.3) \)	36.8 \( (+0.4) \)
Video	17.4	17.7 \( (+0.3) \)	17.6 \( (+0.2) \)	17.7 \( (+0.3) \)
Table 1: Fusion Encoder: Transformer type(# layers). Note that the 'None' are the uni-modal MAE baselines (Audio-MAE and Video-MAE). mAP↑ in AS-20K.
Ratio	40%	60%	70%	80%	90%
Audio	35.6	36.5	36.7	36.8	36.8
Video	16.8	17.3	17.5	17.7	17.5
Subsequently, we investigate the masking ratio, a key hyperparameter in masked autoencoding. We use the same masking ratio for audio and video, and ablate different values in Table 2. The results suggest a masking ratio of  $80\%$  achieves the best performance. This aligns with the optimal values of  $80\%$  in Audio-MAE [41] and  $90\%$  in Video-MAE [27]. MAViL encodes only the non-masked tokens significantly reduces the sequence length, resulting in more efficient computation, as the complexity of self-attention layers in Transformers scales quadratically with the sequence length.
Contrastive Audio-Video Learning with Masking. Next, we examine the benefits when contrastive learning is used in conjunction with MAE. MAViL performs efficient contrastive learning with only  $20\%$  of visible patches in each modality (80% are masked). The results in Table 3 demonstrate the significance of contrastive losses for learning audio-video representations, even with only  $20\%$  visible tokens. Unlike CAV-MAE [32] which uses only inter-modal contrast, we observe that both inter-modal and intra-modal contrast are essential, and combining them delivers the best performance.
Table 2: Masking Ratio. mAP↑ in AS-20K for individual fine-tuning as audio or video classification.
Objs.	None	Inter	Intra	Inter+intra
Audio	36.8	38.4(+1.6)	38.1(+1.3)	39.0(+2.2)
Video	17.7	21.0(+3.3)	19.8(+2.1)	22.2(+4.5)
Table 3: Masked Contrastive Audio-Video Learning
Masked Contextualized Audio-Video Reconstruction. As illustrated in Fig. 2, MAViL employs a two-stage training. Following stage-1 training (Table 3), MAViL's encoders acquire complementary audio-video representations through masked raw audio-video reconstruction, as well as aligned representations through masked contrastive learning. In stage-2, we employ an iterative approach to initialize the student MAViL and train it under masked-views to predict the aligned and contextualized representations generated by the frozen MAViL teacher with full view. The teacher can be either from the stage-1 model (iter. 1) or the last trained stage-2 model (iter. 2 and beyond).
We first compare the reconstruction targets in Table 4a (stage-2, iter. 1). By default MAViL student predicts the last trained MAViL teacher's multimodal fusion encoder outputs (i.e.,  $g_{\mathrm{av}}^{\mathrm{Teacher}}(f_{\mathrm{a}}^{\mathrm{Teacher}}(\mathbf{a}) \| f_{\mathrm{v}}^{\mathrm{Teacher}}(\mathbf{v}))$ , M-Fusion, col. 4). The results show that this improves over predicting (col. 1) raw spectrogram/RGB frames (+1.7 audio mAP) and (col. 2) separately pre-trained uni-modal Audio/Video-MAE's outputs (+1.2 audio mAP). The fused multimodal targets (M-Fusion, col. 4) are better (+0.5 audio mAP) than the uni-modal targets (M-Uni, col. 3) (i.e.,  $f_{\mathrm{a}}^{\mathrm{Teacher}}(\mathbf{a})$  and  $f_{\mathrm{v}}^{\mathrm{Teacher}}(\mathbf{v})$  before the fusion encoder).
Target	Raw	A/V-MAE	M-Uni	M-Fusion
Audio	39.0	39.5(+0.5)	40.2(+1.2)	40.7(+1.7)
Video	22.2	23.1(+0.9)	23.8(+1.6)	24.1(+1.9)
(a) Reconstruction Targets (Stage-2, iteration-1)
Iter.	None	1	2	3	4
Audio	39.0	40.7(+1.7)	41.5(+2.5)	41.8(+2.8)	41.8(+2.8)
Video	22.2	24.1(+1.9)	24.6(+2.4)	24.8(+2.6)	24.9(+2.7)
For the  $2^{\mathrm{nd}}$  and beyond iterations of stage-2 training, we leverage the last trained MAViL student as the new teacher. As shown in Table 4b, substantial improvements (asymptotically saturated after the  $3^{\mathrm{rd}}$  iteration) are achieved over iterations through masked prediction of aligned and contextualized audio-video representations by the MAViL teacher. The accumulated gains with the proposed contextualized audio-video reconstruction in stage-2 are +2.8 audio and +2.6 video mAP without using additional data or external teacher models.
Additional Ablations. In Table 5 we ablate several more aspects of MAViL: (a) Pre-training with more data is useful  $(100\% > 50\% > 10\%$  of AS-2M) (Tab. 5a). (b) ImageNet (IN) self-supervised pre-training is beneficial for the video encoder (Table 5b). ImageNet supervised pre-training (IN-SL) is useful yet we avoid this to keep MAViL fully self-supervised. (c) Longer pre-training is beneficial and 20 epochs are sufficient (Table 5c). (d) Increasing the encoder sizes (ViT-S/B/L) improves mAP (Table 5d). We use ViT-B for efficiency and fair comparison as default.
(b) Iterations (Stage-2)
Table 4: Targets and iterations in MAViL's stage-2 contextualized audio-video reconstruction. (a) Target comparison in the  $1^{\text{st}}$  iteration. Raw: spectrogram/RGB frames. Contextualized: A-MAE/V-MAE (uni-modal MAE output); M-Uni (MAViL's uni-modal output); M-Fusion (MAViL's multimodal fusion output). (b) Iteration comparison. 'None' is the MAViL-stage1 baseline in Table 3.
#	200K	1M	2M
A	34.1	40.5	41.8
V	20.1	23.0	24.8
(a) Dataset size
Ep.	15	20	25
A	41.4	41.8	41.8
V	23.5	24.8	24.7
(c) Pre-training epoch
Init.	Rand.	IN-SSL	IN-SL
A	41.6	41.8	41.8
V	23.7	24.8	26.1
(b) Visual backbone init.
Sz.	ViT-S	ViT-B	ViT-L
A	36.2	41.8	42.2
V	20.5	24.8	27.0
(d) Model size
Method	Audio	Video
A-MAE/V-MAE (baseline)	36.4	17.4
MAViL stage-1
+ Joint AV-MAE	36.8(+0.4)	17.7(+0.3)
+ Inter contrast	38.4	21.0
+ Intra and Inter contrast	39.0(+2.2)	22.2(+4.5)
MAViL stage-2
+ Student-teacher learning	41.8(+2.8)	24.8(+2.6)
(e) Module-wise contributions
Table 5: Additional ablations and module-wise contributions (mAP on AS-20K)
Summary. The module-wise contribution is summarized in Table 5e. MAViL enhances uni-modal MAE performance by learning from audio and video simultaneously, as evidenced by significant increases in audio mAP (36.4→41.8, +5.4) and video mAP (17.4→24.8, +7.4) on AS-20K. It is noteworthy that while audio and video are jointly used in the pre-training phase, the fine-tuning is performed with distinct uni-modal encoders on each modality. Therefore, the improvements in the uni-modal classification tasks showcase SSL pre-training from both modalities can enhance the performance of individual modalities.
4.5 Main Results
The full model is the stage-2 MAViL student (ViT-B) trained with Eq.(6) for 3 iterations. We quantitatively compare it with other previous models on the 7 benchmark tasks.
Audio-Video Event Classification. Table 6 presents a comparison of MAViL's audio (A), video (V), and audio-video  $(\mathrm{A} + \mathrm{V})$  fine-tuning performance on AudioSet (AS) and VGGSound, alongside recent baselines. Notably, MAViL achieves new state-of-the-art results in audio-only and audio+video classification tasks across these datasets. On the balanced AS-20K, unbalanced AS-2M, and VGGSound classification tasks, MAViL surpasses CAV-MAE [32] (both models are with ViT-B encoders) in A, V, and  $\mathrm{A} + \mathrm{V}$  tasks by a large margin. This improvement can be attributed to the benefits of reconstructing aligned and contextualized representations over raw inputs and the enhanced contrastive learning through both intra-modal and inter-modal contrast. Furthermore, MAViL outperforms data2vec [7], highlighting the advantage of utilizing aligned multimodal contexts (Eq.(5)) over uni-modal contexts as the reconstruction targets.
Method	PT	AS-20K (mAP↑)	AS-2M (mAP↑)	VGGSound (Acc.↑)
A	V	A+V	A	V	A+V	A	V	A+V
Audio-only Models
Aud-SlowFast [44]	-	-	-	-	-	-	-	50.1	-	-
VGGSound [11]	-	-	-	-	-	-	-	48.8	-	-
PANNs [47]	-	27.8	-	-	43.9	-	-	-	-	-
AST [30]	IN-SL	34.7	-	-	45.9	-	-	-	-	-
HTS-AT [12]	IN-SL	-	-	-	47.1	-	-	-	-	-
PaSST [49]	IN-SL	-	-	-	47.1	-	-	-	-	-
Data2vec [7]	AS-SSL	34.5	-	-	-	-	-	-	-	-
SS-AST [31]	AS-SSL	31.0	-	-	-	-	-	-	-	-
MAE-AST [5]	AS-SSL	30.6	-	-	-	-	-	-	-	-
Aud-MAE [41]	AS-SSL	37.0	-	-	47.3	-	-	-	-	-
Audio-Video Models
G-Blend [88]	-	29.1	22.1	37.8	32.4	18.8	41.8	-	-	-
Perceiver [42]	-	-	-	-	38.4	25.8	44.2	-	-	-
Attn AV [24]	IN-SL	-	-	-	38.4	25.7	44.2	-	-	-
CAV-MAE [32]	IN-SSL, AS-SSL	37.7	19.8	42.0	46.6	26.2	51.2	59.5	47.0	65.5
MBT*[64]	IN21K-SL	31.3	27.7	43.9	41.5	31.3	49.6	52.3	51.2	64.1
MAViL	AS-SSL	41.6	23.7	44.6	48.7	28.3	51.9	60.6	50.0	66.5
MAViL	IN-SSL, AS-SSL	41.8	24.8	44.9	48.7	30.3	53.3	60.8	50.9	67.1
The fully self-supervised MAViL also demonstrates superior performance to supervised audio-video models such as MBT [64] in A and  $\mathrm{A + V}$  classification tasks on AS-20K, AS-2M, and VGGSound. For video classification on AudioSet, it is worth noting that the fully self-supervised trained video backbone still lags behind MBT which pre-trained with the supervision from ImageNet-21K (11 times larger than ImageNet). This disparity is a consequence of the presence of noise and irrelevant visual context in AudioSet as also discussed in [61]. Such bias could make the visual pre-training sub-optimal on AudioSet. We consider resolving this dataset limitation as the future work.
Transfer to Audio/Speech-only Tasks. To access the generalizability of the learned audio representations, we further evaluate the AS-pre-trained MAViL by transferring it to other out-of-domain speech-only or audio-only tasks. Specifically, we conduct experiments on the Environmental Sound Classification (ESC-50) [72] and Speech Commands (SPC-v1) [89], where only the audio branch of MAViL is fine-tuned. The results, presented in Table 7, demonstrate that MAViL outperforms recent supervised and self-supervised models, establishing a new state-of-the-art performance on these benchmarks. These findings indicate the desirable transferability of MAViL from audio-video self-supervised pre-training to audio-only downstream tasks.
Table 6: Comparison to prior work on AudioSet (AS-20K, AS-2M) and VGGSound in the audio (A), video (V) and audio+video (A+V) classification tasks. PT: pre-training dataset and type; IN: ImageNet; SL: supervised learning; SSL: self-supervised learning; *: We de-emphasize the model using non-standard dataset splits. We bold the best-performing single model.
Method	PT	ESC-50	SPC-1
AST [30]	IN-SL	88.7	95.5
SS-AST [31]	AS-SSL	88.8	96.0
Aud-MAE [41]	AS-SSL	94.1	96.9
MAViL	AS-SSL	94.4	97.3
MAViL	IN-SSL, AS-SSL	94.4	97.4
Table 7: Audio-only tasks (Acc.↑)
Method	PT data	MSR-VTT	YouCook
AVLNet [79]	HT100M	20.1	30.7
TVLT [83]	HT100M	22.6	31.8
MAViL	AS-2M	22.8	32.2
MAViL	HT-100M	23.8	33.1
Table 8: Audio-to-video retrieval (R@1↑)
Audio-to-Video Retrieval. MAViL learns aligned audio-video representations that are suitable for textless cross-modal retrieval. To verify this, we further conduct audio-to-video retrieval experiments on MSR-VTT [96] and YouCook [96]. In these tasks, the audio track of a video serves as the query, and the model performs a search over the testing video collection by computing and ranking the similarity between the query embedding and the video embeddings. We fine-tune MAViL using the audio-video pairs in the training sets with Eq.(6). We report recall@1 on the testing sets.
Figure 3: Video clip and spectrogram reconstruction on the AudioSet eval set. We sample 4 paired (video, audio) examples as follows (click the links to play): Top left: a puppy video; Top right: a recording from an ambulance's dash camera; Bottom left: a person dialing a phone in a dark room; Bottom right: a singer dancing. Input masking ratio:  $70\%$ . In each 3-row group, we show the original video and its audio spectrogram (top), masked input to MAViL (middle), and MAViL's video and audio spectrogram reconstructions (bottom). The spectrogram shape is  $1024 \times 128$ ; patch size is  $16 \times 16$ . Each spectrogram has  $64 \times 8 = 512$  patches. After applying  $70\%$  masking, there are 154 patches visible to MAViL. The 8-frame (4-second under 2 fps) video clip size is  $8 \times 3 \times 224 \times 224$ ; patch size is  $16 \times 16$ . Each video has  $4 \times 14 \times 14 = 784$  patches after patch embedding (temporal kernel/stride=2). After applying  $70\%$  masking, there are 235 patches visible to MAViL.
Following AVLNet [79] and TVLT [83], we explore an alternative setup where we pre-train MAViL on HowTo100M [59], a dataset consisting of 1.3 million instructional videos, instead of using AudioSet for pre-training. It's worth noting that there are notable domain differences between the pre-training datasets (AudioSet or HowTo100M) and the downstream datasets (MSR-VTT or YouCook). Table 8 demonstrates that MAViL surpasses supervised pre-trained AVLNet and self-supervised pre-trained TVLT, achieving new state-of-the-art performance on these tasks.
Raw Audio-Video Reconstructed In Fig. 3, we employ a stage-1 MAViL (ViT-B) to reconstruct raw audio spectrograms and video frames with masked inputs. The model is trained using an  $80\%$  masking ratio on the AudioSet-2M full training set with un-normalized raw spectrograms and video frames as the reconstruction targets (Eq.(4), stage-1). We visualize the reconstruction results by MAViL's audio and video decoders, wherein  $70\%$  of the input tokens are masked to its encoders. This visualization is performed on the AudioSet eval set.
The results show that MAViL accurately reconstructs the highly corrupted versions of both audio spectrograms and video frames. The reconstructions for videos exhibit high fidelity and preserve spatial and temporal consistency of visual objects (e.g., the nearby moving cars recorded by the ambulance's dash camera). For audio reconstructions, MAViL accurately maintains the positions and arrangements of time-frequency components in the spectrogram (e.g., the ambulance's siren and the song by the singer). Furthermore, the reconstructed audio and video components are consistent and well-aligned in time, enhancing the overall coherence of the reconstructed content.
We have presented MAViL, a self-supervised audio-video representation learning framework where masked autoencoding meets contrastive learning. By leveraging an encoder-fusion-decoder architecture, MAViL effectively utilizes complementary information from all modalities for masked autoencoding. It facilitates efficient contrastive learning in both inter-modal and intra-modal scenarios, even with a high  $80\%$  masking ratio. Furthermore, MAViL highlights a novel pre-training task with self-training to predict homogeneous contextualized audio-video representations. This approach outperforms conventional uni-modal and multimodal MAEs that predict heterogeneous raw inputs. As a result, MAViL achieves state-of-the-art performance across 7 audio-video classification and retrieval tasks, as well as audio-only tasks under the scalable self-supervised learning setup.
Acknowledgements. We thank Luke Zettlemoyer, Kaiming He, Juncheng (Billy) Li, and Florian Metze for their feedback and discussions.
[1] Petar S Aleksic and Aggelos K Katsaggelos. Audio-visual biometrics. Proceedings of the IEEE, 94(11):2025–2044, 2006.
[2] Relja Arandjelovic and Andrew Zisserman. Look, listen and learn. In ICCV, 2017.
[3] Relja Arandjelovic and Andrew Zisserman. Objects that sound. In ECCV, 2018.
[4] Yusuf Aytar, Carl Vondrick, and Antonio Torralba. Soundnet: Learning sound representations from unlabeled video. In NeurIPS, 2016.
[5] Alan Baade, Puyuan Peng, and David Harwath. MAE-AST: Masked autoencoding audio spectrogram transformer. In Interspeech, 2022.
[6] Roman Bachmann, David Mizrahi, Andrei Atanov, and Amir Zamir. MultiMAE: Multi-modal multi-task masked autoencoders. arXiv preprint arXiv:2204.01678, 2022.
[7] Alexei Baevski, Wei-Ning Hsu, Qiantong Xu, Arun Babu, Jiatao Gu, and Michael Auli. Data2vec: A general framework for self-supervised learning in speech, vision and language. In ICML, 2022.
[8] Hangbo Bao, Li Dong, and Furu Wei. BEiT: BERT pre-training of image transformers. In ICLR, 2021.
[9] Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In NeurIPS, 2020. URL https://proceedings.neurips.cc/paper/2020/ hash/1457c0d6bcbd4967418bf8ac142f64a-Abstract.html.
[10] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In ICCV, 2021.
[11] Honglie Chen, Weidi Xie, Andrea Vedaldi, and Andrew Zisserman. VGGSound: A large-scale audio-visual dataset. In ICASSP, 2020.
[12] Ke Chen, Xingjian Du, Bilei Zhu, Zejun Ma, Taylor Berg-Kirkpatrick, and Shlomo Dubnov. HTS-AT: A hierarchical token-semantic audio transformer for sound classification and detection. In ICASSP, 2022.
[13] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In ICML, 2020.
[14] Tsuhan Chen and Ram R Rao. Audio-visual integration in multimodal communication. Proceedings of the IEEE, 86(5):837-852, 1998.
[15] Xinlei Chen, Saining Xie, and Kaiming He. An empirical study of training self-supervised vision transformers. In ICCV, 2021.
[16] Akash Chintha, Bao Thai, Saniat Javid Sohrawardi, Kartavya Bhatt, Andrea Hickerson, Matthew Wright, and Raymond Ptucha. Recurrent convolutional structures for audio spoof and video deepfake detection. IEEE Journal of Selected Topics in Signal Processing, 14(5): 1024-1037, 2020.
[17] Jang Hyun Cho and Bharath Hariharan. On the efficacy of knowledge distillation. In ICCV, 2019.
[18] Grzegorz Chrupała. Visually grounded models of spoken language: A survey of datasets, architectures and evaluation techniques. Journal of Artificial Intelligence Research, 73:673-707, 2022.
[94] Yusong Wu*, Ke Chen*, Tianyu Zhang*, Yuchen Hui*, Taylor Berg-Kirkpatrick, and Shlomo Dubnov. Large-scale contrastive language-audio pretraining with feature fusion and keyword-to-caption augmentation. In IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP, 2023.
[95] Fanyi Xiao, Yong Jae Lee, Kristen Grauman, Jitendra Malik, and Christoph Feichtenhofer. Audiovisual slowfast networks for video recognition. arXiv preprint arXiv:2001.08740, 2020.
[96] Jun Xu, Tao Mei, Ting Yao, and Yong Rui. MSR-VTT: A large video description dataset for bridging video and language. In CVPR, 2016.
[97] Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. In ICCV, 2019.
[98] Zhaoyang Zeng, Daniel McDuff, Yale Song, et al. Contrastive learning of global and local video representations. In NeurIPS, 2021.
[99] Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. In ICLR, 2018.
[100] Luowei Zhou, Chenliang Xu, and Jason J Corso. Towards automatic learning of procedures from web instructional videos. In AAAI, 2018.
This appendix are organized as follows: In §A, we offer the comprehensive experimental details and hyperparameter configurations for pre-training and fine-tuning on each dataset. In §B, we perform additional experiments to evaluate and analyze MAViL's performance. These experiments include:
1. Modality-wise masking ratio and masking type analysis.
2. Contrastive weights/ hyper-parameters analysis.
3. From-scratch and large model analysis.
4. Linear probing analysis.
5. Text-audio retrieval tasks on AudioCaps [45] and Clotho [21].
In §C, we discuss MAViL's societal impact and limitations.
A Experimental Details & Hyper-parameters