SlowGuess/NeurIPS_2023_OCR huggingface dataset:
In this section, we provide additional experimental details for data preprocessing, implementation, pre-training, fine-tuning, and inference. The hyper-parameters are summarized in Table 9. The codebase and the pre-trained models will be available.
A.1 Data Preprocessing
In our study, we obtained a total of 2.01 million AudioSet videos, including both the video and audio tracks from the balanced and unbalanced training set and the evaluation set. Additionally, we managed to collect 198K VGGSound videos. As part of the preprocessing, we resized the video tracks to 360p while maintaining the aspect ratio and adjusting the longer dimension to 360 pixels. We also resampled the audio tracks to a sampling rate of 16K. We employed different temporal footprints for modeling the audio and video in MAViL, specified as the following:
Following the preprocessing in [64, 30, 41], we transform a raw audio (with mono-channel and under 16K sampling rate) into 128 Mel-frequency bands used in Kaldi [74]. This transformation involves using a 25ms Hanning window that shifted every 10ms. We then normalize the spectrogram according to the mean and variance in each dataset. For a 10-second audio, the resulting spectrogram has a dimension of  $1024 \times 128$ .
Regarding the video part, we utilize 4-second clips consisting of 8 frames captured at a rate of 2 frames per second (fps). Each input frame has a size of  $224 \times 224$ . In the pre-training phase, we apply common data augmentations such as random horizontal flip (with a probability of 0.5) and multi-scale random cropping (with a scale ranging from 0.2 to 1.0). In contrast, we apply only center cropping during the testing or inference phase. When processing a 10-second video clip from AudioSet, we randomly sample a starting point and extracted the consecutive 4 seconds of the video (cyclically looping back to the beginning if it was shorter than 4 seconds). As a result, the video clip input, consisting of 3 channels, had dimensions of  $8 \times 3 \times 224 \times 224$ .
A.2 Implementation
Uni-modal Encoders. We adopt the main design choices from original MAE for images [37] and Audio-MAE [41]. Specifically, we employ separate 12-layer Transformers with 12 attention heads as the encoders for each modality. The patch embedding and positional embeddings layers are also separated for each modality. During our investigation, we explored alternative designs, including sharing the audio-video encoder weights with separated inputs or concatenating them as done in Multi-MAE [6]. However, these alternative architectures resulted in inferior performance compared to the proposed architecture of using separated encoders with separated inputs. As a result, we chose to adhere to the original design of separate encoders for each modality.
In all Transformer encoders (with ViT-B as the default), the embedding dimension  $H$  is set to 768. For each input spectrogram of size  $1024 \times 128$  representing a 10-second audio, we tokenize it into non-overlapping  $16 \times 16$  spectrogram patches using an audio patch embedding layer. The kernel and stride sizes for both the time and frequency dimensions are 16, resulting in a total of  $64 \times 8$
spectrogram patches or tokens for the audio sequence. The flattened audio token sequence has a length  $N$  of 512. Each audio token corresponds to a 768-dimensional vector. After appending the [CLS] token, adding positional embeddings, and applying  $80\%$  masking, the final input audio token sequence is represented as  $\mathbf{a}' \in \mathbb{R}^{102 \times 768}$ .
For each video clip with dimensions  $8 \times 3 \times 224 \times 224$  (4 seconds in duration), we tokenize it into non-overlapping cells using a video patch embedding layer. The spatial kernel and stride sizes are set to 16, while the temporal kernel and stride sizes are set to 2. This process results in a total of  $4 \times 14 \times 14 = 784$  video patches or tokens. The flattened video token sequence has a length  $M$  of 784. Each video token corresponds to a 768-dimensional vector. After appending the [CLS] token, adding positional embeddings, and applying  $80\%$  masking, the final input video token sequence is represented as  $\mathbf{v}' \in \mathbb{R}^{156 \times 768}$ .
Fusion Encoders. Following the ViT-B uni-modal encoders, we incorporate an audio-video fusion encoder. The fusion encoder consists of a two-layer (with  $L = 2$ ) Transformer, which can be either a vanilla Transformer or an MBT Transformer [64].
In the vanilla Transformer setup, the fusion encoder, denoted as  $g_{\mathrm{av}}(\cdot)$ , jointly encodes the audio and video tokens. This is done by concatenating the output of the uni-modal encoders for audio ( $\mathbf{aum}^{l+1}$ ) and video ( $\mathbf{vum}^{l+1}$ ) as input, resulting in  $(\mathbf{a}_{\mathrm{um}}^{l+1} \| \mathbf{v}_{\mathrm{um}}^{l+1}) = \text{Transformer}^l (\mathbf{a}_{\mathrm{um}}^l \| \mathbf{v}_{\mathrm{um}}^l)$ , where “ $\|$ ” denotes concatenation.
In the MBT setup, we extend the vanilla Transformer by appending an additional 4 trainable MBT tokens for each modality. MBT encourages the model to more selectively collate and condense relevant information in each modality by forcing information exchange between modalities to pass through a small number of learnable bottleneck features  $\mathbf{b}^0 = [b_1\ldots b_4], b_i \in \mathbb{R}^H$ . The use of MBT tokens was originally proposed in the context of supervised audio-video learning. Precisely,  $\mathbf{a}_{\mathrm{um}}^{l+1}\| \mathbf{b}_{\mathrm{a}}^{l+1} = g_{\mathrm{av}}^l(\mathbf{a}_{\mathrm{um}}^l\| \mathbf{b}^l)$  and  $\mathbf{v}_{\mathrm{um}}^{l+1}\| \mathbf{b}_{\mathrm{v}}^{l+1} = g_{\mathrm{av}}^l(\mathbf{v}_{\mathrm{um}}^l\| \mathbf{b}^l)$ , where  $\mathbf{b}^{l+1} = (\mathbf{b}_{\mathrm{a}}^{l+1} + \mathbf{b}_{\mathrm{v}}^{l+1}) / 2$ .
Decoders. The audio and video decoders are 8-layer Transformers with an embedding dimension of 512 and 16 attention heads. In the top decoder layer, we applied a linear prediction head to either predict the raw audio spectrogram and video frame patches in stage-1 (i.e.,  $\mathbf{a}^{\mathrm{raw}} \in \mathbb{R}^{H_{\mathrm{raw}}^{\mathrm{a}}}$  and  $\mathbf{v}^{\mathrm{raw}} \in \mathbb{R}^{H_{\mathrm{raw}}^{\mathrm{v}}}$ ), or predict the aliened and contextualized representations in stage-2 (i.e.,  $\mathbf{a}^{\mathrm{Teacher}}, \mathbf{v}^{\mathrm{Teacher}}, \tilde{\mathbf{a}}, \tilde{\mathbf{v}} \in \mathbb{R}^{H}$ ). The audio/video encoder and decoder in MAViL have 86M and 27M parameters, respectively. The floating point operations (FLOPs) for the audio encoder are 48.6G, comparable to the audio encoders in Audio-MAE [41] and CAV-MAE [32].
A.3 Training and Inference
Pre-training. MAViL operates under a fully self-supervised learning setup for pre-training. For pre-training MAViL's audio branch, we randomly initialize it from scratch. For the visual branch, we either randomly initialize it or initialize it with the self-supervised MAE [37] pre-trained on ImageNet where we simply repeat and inflate the convolution kernel in its patch-embedding to handle the additional temporal domain. Different visual initialization methods are compared in Table 6 in the main paper and Table 15 in Appendix. Importantly, MAViL operates under the fully self-supervised setup.
MAViL is pre-trained on the combined unbalanced and balanced training sets of AS-2M. The pretraining process is performed using 64 GPUs with a 512 accumulated batch size. In stage-1 and each iteration of stage-2 (for  $K = 3$  iterations), we pre-train the model for 20 epochs. Each pre-training session takes approximately 20 hours to complete. In total, the pre-training process takes around 80 hours. Note that the effective learning rate  $(lr_{\mathrm{eff}})$  depends on the base learning rate  $(lr_{\mathrm{base}})$  and the batch size. Precisely,  $lr_{\mathrm{eff}} = lr_{\mathrm{base}} * \frac{\text{batch size}}{256}$ . In our experiments, we also tried using strong data augmentations (e.g., mixup [99], SpecAug [99], and CutMix [97]) to augment audio spectrograms during the pre-training phase. However, we observed that the resulting performance was either similar or worse compared to the baseline. Therefore, by default, we exclude these strong data augmentations for both audio and video during the pre-training phase.
Besides the proposed two-stage training framework, we also experimented with a one-stage variant that exploits the exponential move average (EMA) of MAViL (with raw input as the targets) as the teacher (similar to data2vec [7]) to generate contextualized targets to be reconstructed by a separated decoder head in each modality. This setup results in less stable training behavior in the
Configuration	Pre-training AS-2M PT	Fine-tuning
AS-2M	AS-20K	VGGSound	ESC	SPC
Optimizer	AdamW [56]
Optimizer momentum	β1=0.9, β2=0.95
Weight decay	0.00001
Base learning rate	0.0002	0.0001†	0.001	0.0002	0.0005	0.001
Learning rate schedule	half-cycle cosine decay [55]
Minimum learning rate	0.000001
Gradient clipping	None
Warm-up epochs	4	20	4	4	4	1
Epochs	20	100	60	60	60	10
Batch size	512	512	64	256	64	256
GPUs	64	64	8	32	4	4
Weighted sampling	False	True	False	True	False	False*
Weighted sampling size	-	200,000	-	200,000	-	-
Augmentation	R	R	R	R+N	R	R+N
SpecAug [68] (time/frequency)	-	192/48	192/48	192/48	96/24	48/48
Drop path [50]	0.0	0.1	0.1	0.1	0.1	0.1
Mixup [99]	0.0	0.5	0.5	0.5	0.0	0.5
Multilabel	n/a	True	True	False	False	False
Loss Function	MSE	BCE	BCE	BCE	CE	BCE
Dataset Mean for Normalization	-4.268	-4.268	-4.268	-5.189	-6.627	-6.702
Dataset Std for Normalization	4.569	4.569	4.569	3.260	5.359	5.448
audio-video scenario. The instability is presumably a consequence of different convergence patterns in each modality, which is also observed in [88]. Consequently, this makes the design and tuning of mixture/update scheduler sub-optimal (i.e., that controls the EMA update in Data2vec Sec 3.3). Regrettably, this led to notable performance degradation:
Table 9: Pre-training (PT) and Fine-tuning (FT) hyper-parameters. For augmentation, R: sampling random starting points with cyclic rolling in time; N: adding random noise (signal-to-noise ratio (SNR): 20dB) to spectrograms. For loss functions, BCE: binary cross entropy loss (for multi-label datasets or when using mixup); CE: cross-entropy loss, MSE: mean square error loss. *: We repeat and balance each class to  $50\%$  of the size of the unknown class. †: For ViT-S, We use a learning rate of 0.0005 on AS-2M FT and 0.002 on AS-20K FT for the ViT-S model. For the ViT-L model, we use 0.0001 and 0.0005 for AS-2M and AS-20K FT experiments.
2-stage	1-stage (EMA)
Audio	41.8	40.3
Video	24.8	21.7
Table 10: 2-stage vs. 1-stage (EMA)
By employing a 2-stage training process, MAViL not only simplifies the methodology and achieves better performance, but it also mitigates the necessity of devising intricate update schedules and carefully tuned EMA hyperparameters, as were required in prior work.
Fine-tuning. We fine-tune MAViL in three scenarios: (1) audio-only, (2) video-only, and (3) audio+video. We follow the setup in MAE and retain only the pre-trained uni-modal encoders for fine-tuning. In the audio-only and video-only setups, we fine-tune the respective encoders in the MAViL (stage-2). In the audio+video fusion setup, we introduce a 2-layer vanilla Transformer on top of the audio and video encoder in the MAViL (stage-2) and fine-tune it using both audio and video inputs. The hyperparameter configurations specified in Table 9 are employed for finetuning on each dataset. Empirically we observed a discrepancy in convergence rate between audio and video. We circumvent this by applying a  $50\%$  learning rate reduction for the weights of the video encoder when performing audio+video fusion fine-tuning.
We adopt the standard fine-tuning pipeline and augmentation in prior audio/audio-video classification works [30, 41, 64]. Specifically, we employ SpecAug [68], mixup [99], balanced sampling [51], and
fine-tuning masking [41] (a  $20\%$  random masking rate for time and frequency in audio spectrograms;  $20\%$  for space and time in video clips). For video, we use standard video augmentations used in video classification [90, 27].
To perform importance sampling that balance the fine-tuning scheme on the unbalanced AS-2M (and VGGSound), we apply a distributed weighted sampler as prior works [51, 30, 12, 49]. We set the probability of sampling a sample proportional to the inverse frequency of its labels, where the label frequency is estimated over the training set. Specifically, for an instance  $i$  in a dataset  $\mathcal{D}$  with a label pool  $\mathbf{C}$ , its sampling weight is proportional to  $\sum_{c_i\in \mathbf{C}}w_c$ , where  $w_{c} = \frac{1000}{\sum_{i\in\mathbf{D}}c_{i} + \epsilon}$  and  $\epsilon = 0.01$  is set to avoid underflow in majority classes. During the fine-tuning process on AS-2M, we randomly sample 200K instances (approximately  $10\%$  of AS-2M) with replacement in each epoch. We fine-tune MAViL for 100 epochs, which corresponds to approximately 10 full epochs of AS-2M. The entire fine-tuning process typically takes around 10 hours to complete.
Inference. After fine-tuning, we select the last checkpoint for inference. For the video and audio+video tasks, we adopt the standard approach used in video action recognition [26, 23, 52] by uniformly sampling ten 4-second video clips throughout the time domain of a video. Each of these sampled video clips is individually fed forward through the model to generate predictions. Note that for audio+video classification, the audio input remains the same 10-second audio recording throughout the sampling of video clips.
# Clips (AS-2M)	1	10
Audio	48.7	48.7
Video	29.4	30.3
Audio+Video	52.6	53.3
We average the ten predictions as the instance-level prediction and report the classification performance in Table 6 in §4. Note that these results are based on single-modal predictions, without assembling multiple models. In Table 11, we compare the results obtained from one-clip predictions and ten-clip predictions (mAP on AS-2M). The sampling of ten clips leads to improvements of up to  $0.9\mathrm{mAP}$  for video-only and audio+video tasks, while the audio-only task remains unaffected.
B Additional Experiments and Analysis
In this section, we present additional analysis to extend the study of the module-wise contribution in Table 12. We then expand our study on another important type of audio task: text-audio retrieval.
We organize this section as follows: Firstly, we investigate how different choices of masking ratio and masking type may affect the model performance. Next, we examine the effects of adjusting contrastive weights in the training objective. By exploring different weight settings, we aim to understand the influence of contrastive learning on the model's ability to capture audio-video relationships. Furthermore, we compare different approaches to visual backbone initialization and evaluate the performance using larger (ViT-L) audio/video encoders in MAViL-Large models. This analysis helps us understand the benefits and trade-offs of using larger backbone models and different initialization strategies. Additionally, besides audio-video classification tasks and audio-video retrieval tasks presented in the main paper. We include our study on audio-text retrieval tasks in the last.
Table 11: Number of video clips in the inference time.
Method	Audio	Video
A-MAE/V-MAE (baseline)	36.4	17.4
MAViL stage-1
+ Joint AV-MAE	36.8(+)0.4	17.7(+)0.3
+ Intra and Inter contrast	39.0(+)2.2	22.2(+)4.5
MAViL stage-2
+ Student-teacher learning	41.8(+)2.8	24.8(+)2.6
Table 12: Module-wise Contribution in MAViL).
B.1 Masking Ratio and Type
In addition to applying a shared masking ratio for each modality, we also investigated the impact of applying different masking ratios for audio and video. The results of this analysis are summarized in Table 13a. Interestingly, we did not observe a significant change in performance (mAP on AS-20K) when using different masking ratios for audio and video. Based on these findings, we simplify the approach by defaulting to an  $80\%$  masking ratio for both audio and video, as the Joint AV-MAE entry (the second row) in Table 12.
Ratio	70% (A)	80% (A)	90% (A)	Type	70%	80%	90%
70% (V)	36.7/17.5	36.8/17.5	36.4/17.3	Random (A), Random (V)	36.7/17.5	36.8/17.7	36.8/17.5
80% (V)	36.7/17.2	36.8/17.7	36.8/17.4	Time-Freq (A), Random (V)	36.2/17.5	36.3/17.7	36.3/17.8
90% (V)	36.5/17.3	36.6/17.6	36.8/17.5	Random (A), Space-Time (V)	36.7/17.2	36.7/17.3	36.8/17.5
Time-Freq (A), Space-Time (V)	36.0/17.1	36.2/17.1	36.3/17.3
The default masking strategy in our model is random masking, which applies the same Bernoulli trial parameterized by a masking ratio  $(p)$  to each spectrogram or RGB patch. In Table 13b, we explored more advanced masking strategies and compare their impacts. For audio spectrogram, in addition to random masking (time-and-frequency agnostic with Bernoulli trials), we investigated timemasking (randomly masks multiple periods of time components) and frequency masking (randomly masks multiple frequency bands). We perform Bernoulli trials on time or frequency slots instead of individual patches. For video frames, we explored time-wise masking (randomly masking an entire frame) and space-wise masking (randomly masking a spatial patch across time). We set the masking ratio between spatial/frequency and time as 2:1 and adjusted the overall ratio from  $70\%$  to  $90\%$  for comparison with random masking.
Surprisingly, we do not observe improvements when applying these advanced masking strategies for multimodal pre-training. The simplest random masking approach achieved the best pre-training performance. This observation aligns with the findings in uni-modal MAEs [37, 27, 41], suggesting that the random masking strategy is effective and sufficient for multimodal pre-training.
B.2 Contrastive Weights
Table 14 showcases the impact of adjusting contrastive weights  $\alpha$  and  $\beta$  in MAViL. The results show that fine-tuning these contrastive weights leads to improved performance. In our experiments, we set  $\alpha = 0.1$  and  $\beta = 0.01$  which yield the best performance.
It is important to note that the smaller contrastive weights in Eq.(4) do not imply that the contrastive objectives are less significant. The weights are chosen to scale and balance the gradients from the reconstruction and the two contrastive objectives to ensure they fall within a comparable range. This adjustment enhances training stability. Furthermore, the softmax temperatures used in NCE (Eq. (2)) are set as  $\tau_{\mathrm{c}}^{\mathrm{inter}} = 0.1$  (more tolerant) for inter-modal contrastive learning and  $\tau_{\mathrm{c}}^{\mathrm{intra}} = 1.0$  (stricter) for intra-modal contrastive learning. These temperature values help regulate convergence across modalities in the contrastive learning process.
(a) Modality-wise Masking
(b) Masking Type
Table 13: Masking Ratio and Masking Type (mAP on AS-20K).
α	0.3	0.1	0.05
Audio	41.5	41.8	41.4
Video	24.3	24.8	24.4
β	0.1	0.05	0.01
Audio	41.3	41.5	41.8
Video	24.3	24.7	24.8
(a) Inter-modal  $\alpha$
(b) Intra-modal  $\beta$
Table 14:Contrastive Weights (mAP on AS-20K).
B.3 From-scratch Visual Backbone and Large Models
Under the fully self-supervised setup, MAViL initializes its audio branch from scratch and initialize its visual branch either from scratch or from a ImageNet self-supervised pre-trained MAE (IN-SSL). In
Model	A-init	V-init	AS-20K	AS-2M
A	V	A+V	A	V	A+V
MAViL-Base	scratch	IN-SSL	41.8	24.8	44.9	48.7	30.3	53.3
MAViL-Base	scratch	scratch	41.6	23.7	44.6	48.7	28.3	51.9
MAViL-Large	scratch	IN-SSL	42.1	27.1	45.3	48.8	32.4	53.3
MAViL-Large	scratch	scratch	42.3	25.3	45.1	49.1	30.6	52.5
this part, we further explore and compare the visual backbone initialization strategies under different model sizes.
As shown in the top two rows of Table 15, when considering MAViL-Base models, there is a small gap  $(-0.2\mathrm{mAP}$  on AS-20K) observed in the audio stream when discarding visual initialization from the ImageNet self-supervised model. However, a larger gap  $(-0.9\mathrm{mAP})$  is observed in the video stream. A similar trend is observed in the AS-2M experiments. This discrepancy in the visual part can likely be attributed to biases and visual quality issues such as misalignment, title-only content, and low-resolution videos present in AudioSet.
To address this gap in the visual part, incorporating additional uni-modal pre-training steps could potentially improve model performance. For instance, conducting separate audio-only and video-only large-scale pre-training as the first step. In this work, we focus on audio-video pre-training solely on AudioSet for simplicity and for fair comparison with baselines. The possibility of incorporating additional pre-training steps is left for future research.
When using large models (ViT-L, rows 3-4), the gap in visual mAP (-1.8 mAP) still persists. Interestingly, the audio part of large models actually benefits from from-scratch visual initialization, showing an improvement of  $+0.2 - 0.3$  mAP. Additionally, when comparing rows 1-2 to rows 2-3, the visual stream is benefited more by employing a larger (ViT-L) backbone. Across all the configurations (from-scratch or visual initialization with IN-SSL), MAViL consistently outperforms recent baselines (in Table 6 of the main paper) by a significant margin.
B.4 Linear Probing Results
Additionally, we evaluate MAViL's representations by performing linear probing on ESC and AudioSet-20K. As shown in Table 16, MAViL's representations outperform other recent baselines by a large margin. There is still a gap between linear-probing and end-to-end fine-tuning.
Table 15: Visual Backbone Initialization and Model Size (mAP).
PT-Data	ESC-FT	ESC-Linear	AS-20K-FT	AS-20K-Linear
XDC	A+V,AS	-	84.8	-	-
AVID	A+V,AS	-	89.1	-	-
BraVe	A+V,AS	-	90.4	-	-
CrissCross	A+V,AS	-	90.5	-	-
SS-AST	A,AS	88.8	85.6	31.0	-
A-MAE	A,AS	94.1	89.5	37.0	-
MAViL	A+V,AS	94.4	90.8	41.8	30.0
Table 16: Linear-probing and fine-tuning on ESC and AS-20K. PT: Pre-training. FT: end-to-end fine-tuning. Linear: Linear-probing under standard protocol. AS: AudioSet. Metric: Top-1 accuracy for ESC-50 and mAP for AS-20K.
B.5 Text-Audio Tasks
Another important audio-centered multimodal application involves text-to-audio and audio-to-text retrieval tasks. In text-to-audio retrieval, the query is a text description, and the model performs a search over the (testing) audio collection by computing and ranking the similarity between the query embedding and the audio embeddings. To evaluate the audio representations learned by MAViL, following CLAP [94], we add a text encoder initialized from Roberta [54]. We perform fine-tuning
with inter-modal contrast on the same training set used by CLAP. Specifically, AudioCaps [45] and Clotho [21], and LAION-630K [94]. In Table 17, we report recall@1, 5, and 10 on the testing sets.
Model	AudioCaps [45]	Clotho [21]
Text-to-Audio	Audio-to-Text	Text-to-Audio	Audio-to-Text
R@1	R@5	R@10	R@1	R@5	R@10	R@1	R@5	R@10	R@1	R@5	R@10
MMT* [66]	36.1	72.0	84.5	39.6	76.8	86.7	6.7	21.6	33.2	7.0	22.7	34.6
ML-ACT* [58]	33.9	69.7	82.6	39.4	72.0	83.9	14.4	36.6	49.9	16.2	37.6	50.2
CLAP [94]	32.7	68.0	81.2	43.9	77.7	87.6	15.6	38.6	52.3	23.7	48.9	59.9
MAViL	37.3	72.8	84.5	49.3	81.8	91.5	17.2	41.0	53.5	23.3	49.5	63.6
Table 17: Text-to-Audio retrieval and Audio-to-Text retrieval (R@1,5,10↑) on AudioCaps and Clotho. *: models trained without LAION-630K [94].
As shown above, MAViL significantly outperforms CLAP and other recent audio-text models, achieving new state-of-the-art performance on both audio-to-text and text-to-audio retrieval tasks. These results further validate the effectiveness of MAViL's representations not only in audio-video and audio-only tasks, but also in audio-text tasks.
C Limitations and Impacts