Masked Autoencoders that Listen
Authors: Po-Yao Huang
Hu Xu
Juncheng B Li
Alexei Baevski
Michael Auli
Wojciech Galuba
Florian Metze
Christoph Feichtenhofer
Venue: NeurIPS 2022 Accept (2022)
Keywords: self-supervised learning
audio representation learning
audio classification

Abstract: This paper studies a simple extension of image-based Masked Autoencoders (MAE) to self-supervised representation learning from audio spectrograms. Following the Transformer encoder-decoder design in MAE, our Audio-MAE first encodes audio spectrogram patches with a high masking ratio, feeding only the non-masked tokens through encoder layers. The decoder then re-orders and decodes the encoded context padded with mask tokens, in order to reconstruct the input spectrogram. We find it beneficial to incorporate local window attention in the decoder, as audio spectrograms are highly correlated in local time and frequency bands. We then fine-tune the encoder with a lower masking ratio on target datasets. Empirically, Audio-MAE sets new state-of-the-art performance on six audio and speech classification tasks, outperforming other recent models that use external supervised pre-training. Our code and models is available at https://github.com/facebookresearch/AudioMAE.

Masked Autoencoders that Listen
Po-Yao Huang'
Hu Xu'
Juncheng Li Florian Metze*
Alexei Baevski'
Michael Auli
Wojciech Galuba'
Christoph Feichtenhofer*
Meta Al
Carnegie Mellon University
Abstract
This paper studies a simple extension of image-based Masked Autoencoders (MAE) [1] self-supervised representation learning from audio spectrograms. Following the Transformer encoder-decoder design in MAE, our Audio-MAE first encodes audio spectrogram patches with high masking ratio, feeding only the non-masked tokens through encoder layers. The decoder then re-orders and decodes the encoded context padded with mask tokens, in order to reconstruct the input spectrogram. We find it beneficial to incorporate local window atten- tion in the decoder, as audio spectrograms are highly correlated in local time and frequency bands. We then fine-tune the encoder with lower masking ratio on target datasets. Empirically, Audio-MAE sets new state-of-the performance on six audio and speech classification tasks, outperforming other recent models that use external supervised pre-training. Our code and models available at https: /ttps://github.coo hww.aab.cccccccccoooeeeaacccc AudiiMAE..
Introduction
Transformers [2] and self-supervised learning [3, 4, 5, 6, 7, 1] are dominating computer vision (CV) and natural language processing (NLP) research. The revolution firstly started in NLP with the invention of the Transformer architecture and self-attention [8]. Masked autoencoding with BERT [3] set state-of-the-art
various NL. tasks by seff-supervised pre-training large-scale language corpus. Similarly in the CV community, Vision Transformers (ViT) [9] have become popular for CV tasks, and, for self-supervised image representation learning, Masked Autoencoders (MAE) [1] have brought the CV community closer to the success of BERT in NLP. In addition to the existing masked autoencoders that can read (BERT) or see (MAB), in this work we study those that can listen.
Transformer- based models have recently refreshed leaderboards for audio understanding tasks. For example, AST [10] and MBT [1 1] improved the audio classification performance the AudioSet [12], Event Sound Classification [13], etc. The key technique behind this is initialization of audio model weights with ImageNet pre-trained supervised models (e.g., DeiT [14]) by deflating patch embeddings and interpolating positional embeddings for encoding audio spectrograms. However, exploiting ImageNet pre-trained models could be sub-optimal. Unlike initializing video models with weights from image models (e.g., the initial weights of I3D [15] or 3D-ResNets [16] are inflated from ImageNet pre-trained image models), there clear and notable discrepancies between spectrograms representing audio content and natural images. It remains unclear why such heterogeneous image-to- audio transfer is useful beyond arguably similar low-level semantics such shapes of spectrograms and shapes of visual objects. Further, any label bias would inevitably be transferred to audio models.
Addressing these concerns, self-supervised audio representation learning has recently attracted much research attention. Based on BEiT [17] that learns to reconstruct image patches or learnt patch tokens, SS-AST [18] extends to the audio domain and exploits spectrograms (akin to 1-channel 2D images) and use both contrastive and reconstruction objective as self-supervision. Without using any labels, the key enabler to effective self- self-supervised representation learning is large-scale pre-training data. In this work we use AudioSet [12] for pre-training, a common dataset containing ~2 million audio recordings. Performing large-scale training with Transformer architectures is challenging as self-attention in Transformers has quadratic complexity w.r.t. the length of input sequence.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Encoder
Decoder
MSE
Input
Target
Figure I: Audio-MAE for audio self-supervised learning. An audio recording is first transformed into a spectrogram and split into patches. We embed patches and mask out large subset (80%). An encoder then operates on the visible (20%) patch embeddings. Finally, decoder processes the order-restored embeddings and mask tokens to reconstruct the input. Audio-MAF is minimizing the mean square emror (MSE) on the masked portion of the reconstruction and the input spectrogram.
This computational burden has been addressed in different ways. A popular approach is reduce the sequence length in self-attention. Various ViT-based architectures have been developed to alleviate such issues for image and video understanding. For example, Swin- Train-Tranffomer [19] only performs local attention within windows that shift across layers. MViT [20] employs pooling attention to construct hierarchy of Transformers where sequence lengths are downsampled. For self-supervised learning, MAE [1] efficiently encodes only a small portion (25%) of visual patches while the majority of patches discarded. The simplicity and scalability MAE make a promising framework for large-scale self-supervised learning.
In this work, we study MAE for sound recognition and the unique challenges of the audio domain. We present Audio-MAE (Fig. 1) as unified and scalable framework for learning self-supervised audio representations. Similar MAE, it is composed of a pair of a Transformer encoder and decoder. Sound is first transformed and embedded into spectrogram patches. Before feeding them into the Transformer encoder, we mask and discard the majority and only feed small number of non-masked embeddings into the encoder for efficient encoding. After padding encoded patches with learnable embeddings to represent masked patches. it then restores the order of these patches in frequeecy and time and propagates them through a Transformer decoder to reconstruct the audio spectrogram.
Different from image patches, spectrogram patches comparably local-correlated. For example, formants, the vocal tract resonances, are typically grouped and continuous locally in the spectrogram. The location in frequency and time embeds essential information that determines the semanties of spectrogram patch and how it sounds like. To this end, we further investigate using localized attention and hybrid architecture in the Transformer decoder properly decode for reconstruction. This simple-yet-effecctie upgrade leads to improved performance for Audio-MAE.
Similar to MAE for images, minimize the patch- normalzedd mean square error. At the fine-tuning stage, we discard the decoder and fine-tune the encoder with MAE sets new state-of-the-an art performance six audio and speech classification tasks. It is the first audio-only self-supervised model that achieves state-of-the-arr mAP on AudioSet-2M,, outperforming other recent models with external supervision. We further provide the visualization and audible examples to qualitatively demonstrate the effectiveness the Audio-MAL decoder.
Empirically, Audio-
Related Work
Visual masked pre-training. Masked/Denoising autoencoders [21, 22, 3] are general represen- tation learning methodology by reconstructing source from masked or corrupted inputs. In CV, visual masked pre-training has made recent progress [23, 24, 1, 20]. Based on ViT [9] that applies Transformers to image patches, BEiT [17] and MAE [1] present masked image modeling frameworks. BEiT [17] learns predict discrete visual tokens generated by VAE [25] masked patches. MAE [1] reduces sequence length by masking large portion of image patches randomly and encoding only non-masked ones for reconstruction pixel color information. MaskFeat [20] studies features for masked pre-training and finds that Histograms of Oriented Gradients (HoG) [26]. which are in turn related spectrogram features, perform strongly for image and video classification models. Our work extends the MAE framework for representation learning with audio spectrograms.

---

Out-of-domain pre-training for audio. Transferring ImageNet supervised pre-trained VIT [9] or ResNet [27] has become a popular practice for audio models [10, 28, 11, 29, 30, 31]. After
pre-training, these models operate over audio spectrograms by deflating from 3-channels (RGB) into 1-channel (spectrogram) the pre-trained patch embedding in ViT and employing the rest of the transformer blocks on top. For example, HTS-AT [29] encodes spectrograms with hierarchical Transformer initialized from the Swin Transformer [19]. MBT [11] uses ImageNet-2IK pre-trained ViT; AST [10] and PaSST [28] employ DeiT [14] as the Transformer backbone. Without using out-of-domain (non-audio) data, the proposed Audio-MAE focuses on audio-only self-supervised pre-training from scratch.
In-domain pre-training for audio. Existing in in-domain (i.e., audio-only) self-supervised methods can be broadly categorized by the input signal type (e.g., raw waveform [32, 33, 34], frame-level features [35, 36, 37], or spectrogram patches [18, 38]); and the objective used for self-supervision (e.g., contrastive [39, 33, 40, 41, 35] or prediction/recoosttuctiion [18, 34, 37, 36]). For example, wav2vec 2.0 [33] takes raw waveform as inputs and exploits contrastive learning to discriminate contextualized representations in different time segments. Mockingjay [42] proposed masked acoustic model pretext task to reconstruct frame-level Mel-features of masked time frames. SS- AST [18] is the closest work to Audio-MAB and is our main benchmark. Inspired by the success of BERT [3], SS-AST proposed a self-super learning method which operates over spectrogram patches and employs joint contrastive and reconstructive objectives on masked patches. These previous methods generate audio representations by encoding full-view of both masked and non- masked time spectrogram segments for self-supervised pre-training. In contrast, Audio-MAE encodes only the monkmd spectrogram patches.
Our work is done independently and concurrently with [38, 43, 44] related methods. We also compare our model to these concurrent works in the experiments and showcase the superiority of Audio-MAE.
3 Audio Masked Autoencoders (Audio-MAE)
Audio-MAE is conceptually simple extension of MAE learn self- supervieee representations from audio spectrograms. Fig. depicts an overview. The details of each component are as follows.
Spectrogram Patch Embeddings. Following [10, 18), we transform audio recordings into Mel- spectrograms and divide them into non-overlapped regular grid patches. These patches are then flattened and embedded by linear projection. Similar to MAE [1], we add fixed sinusoidal positional embeddings to the embedded patches.
(a) Original
(b) Unstructured
(c) Time
(d) Frequeney
(e) Time+frequency
Figure 2: Audio-MAE's masking strategies on Mel-spectrograms.
Masking Strategies. Audio-MAE masks out large subset of spectrogram patches. As spectrogram can be viewed as a 2D representation of time and frequency components of sound, it is reasonable to explore treating time and frequency differently during masking. In this work, we investigate both the unstructured (i.e., random masking without any prior) and structured (i.e., randomly masking a portion of time, frequency, or time frequency of spectrogram) in the pre-training and fine-tuning phase. Illustrative examples are shown in Fig. 2. We show masked regions with dark overlay.
The masking mechanism, as introduced in MAE [1], is the key ingredient for efficient self supervised learning. For input patch sequence, this can be regarded as a Bernoulli process where each patch is masked/dropped with probability (masking ratio). Masking reduces input patch sequence length and encourages learning global, contextualized representations from limited "visible" patches. We observe that akin to images, large masking rate (80% in our experiments for spectrogram patches, which similar to 75% MAE for images) feasible for learning self-supervised audio representations. Unlike BERT [3] that uses 15% masking rate for self-supervised learning in NLP, most of the
3

---

tokens/patches can be discarded for spectrograms as well as images due to high redundancy in these modalities. Beyond self-supervised pre-training we further explore the effectiveness of masking in the supervised fine-tuning stage. Empirically, we found unstructured (random) masking at higher ratio for pre-training and structured (time+ (time+frrquee frequency masking) at a lower ratio for fine-tuning provide best accuracy (ablations are in $4.4).
Encoder. Audio-MAE uses stack of standard Transformers [2] its encoder. The encoder only processes (20%) non-masked patches to reduce computation overhead which is quadratic to the input sequence length. We use the 12-layer ViT-Base (ViT-B) [9] Transformer as our default.
Decoder with Local Attention. The decoder is also composed of standard Transformer blocks. The encoded patches from the encoder are padded with trainable masked tokens. After restoring the original time-freque fiee-freyyeccy order in the audio spectrogram, we add the decoder's (fixed sinusoidal) positional embeddings and feed the restored sequence into the decoder. At the top of the decoder stack, we add a linear head to predict and reconstruct the input spectrogram.
To address the unique characteristics of audio spectrograms, our work investigates an enhancement to the vamilla MAE decoder. Image-based MAE uses global self-attention the Transformer decoder which is appropriate for visual context, because visual objects are typically invariant under translation or scaling, and their exact position may not affect the semantics of an image. In contrast, the position, scale, and translation of spectrogram features however directly affects the sound or semanties of an audio recording. Consequently, global self-attention is sub-optimal for spectrograms if the time- the frequency components is predominantly local. For instance, would have better success harmonics (e.g.. Fig. 2a) in lower bands of a vowel to predict the spectrogram patch vertically in a higher frequency band rather than horizontally in the time domain. Similarly, a frictional sound of consonant likely only correlates to other part of the consonant, and is without dependency to other silence segments in the audio recording. Compared images, the spectrogram patches are more similar to speech or text tokens where its order and position is more relevant.
To address the nature of audio spectrograms, in addition to us- ing Transformers with global self-attention as vanilla MAE, incorporate the local attention mechanism which groups and separates the spectrogram patches in to local windows in self-attention for decoding. We investigate two types of local attention: (1) Shifted window location: Inspired by the shifted-window in Swin Transformers [19], we shift window attention by 50% between consecutive Transformer decoder layers. For padding the margin when shifting, we eyclically shift the spectrogram to the top-left direction. Fig. illustrates the localized decoder attention by shifted windows. (2) Hybrid window attention gglobal+local attention): Inspired by [45]. to add better windww connections, we design simple hybrid (global+local) attention that computes local attention within a window in all but the last few top layers. In this way, the input feature maps for the final reconstruction layer also contain global information. For simplicity, we use no pooling or hierarchical structure. Decoders with different attention types are compared in $4.4.
Layer L
Layer L++1
Figure 3: Decoder's local attention and shifted window (right).
Objective. The Audio-MAE decoder learns to reconstruct the input spectrogram by predicting the values in the spectrogram patches or their per-patch normalized ones. The objective is the mean squared error (MSE) between the prediction and the input spectrogram, averaged over unknown patches. Empirically we found employing the reconstruction loss alone is sufficient while including additional contrastive objectives (e.g., InfoNCE loss [46]) does not improve Audio-MAE.
Fine-tuning for Downstream Tasks. the fine-tuning stage, we only keep and fine-tune the Audio- MAE encoder and discard the decoder. Different from the original MAE, and inspired by [47, 28], we also explore to employ masking in the fine-tuning stage to remove a portion of patches to further regularize learning from limited view of spectrogram inputs, which, as a side effect, also reduces computation during fine-tuning. Compared to SpecAug [48] which takes full-length input with the masked portion set zero as data augmentation, Audio-MAE sees only subset of real-valued input patches without the nullified ones. Audio-MAE then encodes these non-masked patches and applies an average pooling layer followed by a linear layer on top for fine-tuning classification tasks.

---

4
Experiments
We perform an extensive evaluation on six tasks, including audio classification on AudioSet (AS-2M, AS-20K) and Environmental Sound Classification (ESC-50), and speech classification Speech Commands (SPC-1 and SPC-2) and VoxCeleb (SID). We use AudioSet for ablation studies.
4.1 Datasets and Tasks
AudioSet [12] (AS-2M, AS- contains -2 million 10-second YouTube clips for audio classifi- cation. 527 types of audio events are weakly annotated [49, 50, 51J for each clip. There could be multiple events in clip. The full training set has 2 subsets: A class-wise balanced (22, (22,176 clips) and an unbalanced (2,042,985 clips) set. The eval set has 20,383 clips. We downloaded and processed around 1.96M unbalanced training, 21K balanced training, and 19K evaluation clips.
For the AS-2M experiments, we use the union of unbalanced and balanced training audio for pre- training and fine-tuning, For the AS-20K experiments, we use AS-2M for pre- pre-training and the 20K balanced set for fine-tuning. We report the testing mAP on the 19K eval set used by AST [10].
Environmental Sound Classification (ESC-50) [13] an audio classification dataset consists of 2,000 5-second environmental sound recordings. There are 50 classes in ESC. We report accuracy under 5-fold cross- validation with the same split used by [10].
Speech Commands (SPC-2, SPC-1) [52] are two keyword spotting tasks. In SPC-2, there are 35 set has (..43/9,988/11..
1-second recordings, respectively. In SPC-1, there are 10 classes of keywords, I silence class, and I unknown class that includes all the other 20 common speech commands. We use the data and split provided in the SUPERB [53] benchmark to report the testing accuracy.
speech commands. The reaiiing/allidationeestt
VoxCeleb (SID) [54] contains 150K utterances from 1,251 speakers. The speaker identification task (SID) is to classify the utterances to identify its original speaker. We use the V1 standard train (138,361), validation (6,904), testing (8,251) sets and report the testing accuracy.
4.2 Implementation Details
We use vanilla 12-layer ViT-B by default the Transformer encoder. For the decoder, we use 16-layer Transformer with shifted local attention. We investigate the vanilla (global attention) and hybrid (global+ local attention) decoder variants (see Table. 1c).
Following [10, 11], we transform raw waveform (pre-processed as mono channel under 16,000 sampling rate) into 128 Kaldi [5o]-come Mel-frequency bands with 25ms Hanning window that shifts every 10 ms. For a 10-second recording in AudioSet, the resulting spectrogram of 1x x x 128 dimension.
For patch embedding, we use convolutional kernels with (16, 16) size and stride time and frequency (thus, patches are non-overlapping) to avoid short-cuts via overlap in self-supervisin (though, at high masking ratios such short-cuts are less severe). By default, we use a masking ratio of 0.8 with (unstructured) random masking for pre-training. During fine-tuning, we employ lower masking ratio (0.3 in time and 0.3 in frequency). Ablations on these design choices are given in $4.4.
4.3 Pre-training and Fine-tuning
We use AudioSet-2M for pre-training and randomly iterate over all audio recordings. We train for 32 epochs with a batch size of 512 and 0,0002 learning rate. We distribute the training load over 64 V100 GPUs and the total training time ~36 hours. For each audio, we randomly sample the starting time, cyclically extract 10-second audio, and randomly jitter its magnitude by up + 6dB. We use only natural audio spectrograms and apply no augmentations (e.g., [48, 56, 57]) as we do not find these strong augmentations helpful in the pre- pre-training phase.
In the fine-tuning phase, we remove the decoder and only fine-tune the encoder. For the supervised fine-tuning on AudioSet-2M, since the size of training samples are uneven across classes (unbalanced), we follow the common practice of using a weighted sampling to balance the classes during training. In each epoch, we sample 200K instances (~10% of AudioSet-2M) without replacement. We fine-tune
5

---

472
47.2
4.6
46.8
46.7
47.25
48.75
45.5
unstructurod
timm froq 11mo-frog
unstructured
tio-drrn
44.5
46.25
0.9
(a) Pre-training masking
(b) Fine-tuning masking
Figure 4: Masking strategy. For pre-training, higher ratio and unstructured masking (random) is preferred. For fine-tuning, a lower ratio and structured masking (time+ frreuuecy)))) is better. The y-axes mAP on AS-2M and the x-axes masking ratio. This ablation format follows [1].
for 100 epochs, which aggregate to 10 full epochs of AudioSet-2M. The probability of sampling an instance is inversely proportional to the dataset-wise occurrences of its classes. Fine-tuning on 64 GPUs takes 122 hours. For the smaller balanced AudioSet-201 20K, we fine-tune on GPUs for 60 epochs without weighted sampling. Please see Supplementary for the details on other datasets.
4.4 Ablations and Model Properties
Masking Strategies in Pre-training and Fine-tuning. In Fig. 4, we compare different pre-training and fine-tuning masking strategies for Audio-MAE. First, in Fig. 4a we explore the pre-training masking ratio. We observe, similar as in MAE for images [1]. that a high mre-training masking ratio (80% in our case) is optimal for audio spectrograms. This is due the fact that both audio spectrograms and images are continuous signals with significant redundancy. Further, we find the unstructured random masking works the best for self-supervised pre-training more structured masking (e.g., fre-uureeeency)
Unlike MAE for images, there are clear performance differences among masking strategies when pre-training with audio spectrograms. Comparing Audio-MAE reconstructions between Fig. 6a to 6e and 6d to 6h, under the same masking ratio, we observe the unstructured random masking comparably easier than structured masking (i.e., time and/or frequency) as the model can guess the missing component by extrapolating nearby context (e.g., formants in vowels and frictional sounds in consonants around). We also observe that for higher masking ratios, the structured masking alternatives drop in performance, presumably because the task becomes too difficult while random masking improves steadily up to 80%. This result show that designing a pretext task with proper hardness is important for effective supe-supervised learning of audio representations. We therefore use random masking with ratio 80% as our default for pre-training.
Fig. 4b studies the effect of masking during the fine-tuning phase. We see that in this case, it is more beneficial to use structured masking: time-+ frequency performs better than time- or frequency- masking, and these perform better than unstructured masking. Overall, we see that the optimal masking ratios are lower than for pre-training and we use 0.3 as our default in the fine-tuning phase.
In general, we observe that for task-agnostic pre-training, unstructured masking with a higher ratio is preferred, While in task-specific fine-tuning, structured masking with lower ratios performs better.
Impact of Patch Size and Stride. We compare the performance of Audio-MAE trained with different patch sizes and strides in Table la. A non-zero overlap (i.e.. stride patch size) between patches will increase the number of patches and quadratically increase computation in floating point operations (FLOPs), as reported the table. Most prior works follow AST [10] to use overlapped patches (patch = 16 and stride 10) to boost end task performance. As shown in Table 1a, we do not observe a performance improvement using overlapped patches for Audio-MAE (both 47.3 mAP), presumably because due to overlap, the patch embedding can leak information into the masked patches. The non-overlapped 16x 16 patches achieve a good balance between computation and performance. By default, we use this setup in our experiments.
Encoder. We investigate the design choices of encoder and decoder architectures Audio-MALE Table lb shows the trade-off between encoder model size and performance. As expected, larger models achieve better performance, at cost of computation and memory. The accuracy gain of ViT-I. over ViT-B/S is more significant on the smaller and balanced AS-20K. For ViT-S, the performance

---

Patch size, stride
Seq shape
FLOPs
mAP
Backbone #Params AS-20K AS-2M
(16,16), (16,16)
64x8
48.6
47.3
ViT-S
22M
32.1
45.0
(16,16), (10,10)
101 x12
130.5
47.3
VIT-B
86M
37.1
47.3
(32.16), (16,16)
63x8
47.8
46.6
VIT-L
304M
37.6
47.4
(16,32), (16,16)
64
42.1
46.8
(a) Patch size and stride
(b) Model size (encoder)
Attention type
AS-20K
AS-2M
ESC-50
Depth mAP
Width mAP
Global (vanilla)
36.6
46.8
93.6
94.1
2
46.8
256
46.9
Local (shifted)
37.1
47.3
94.1
94.8
8
47.2
512 47.3
Hwin (local x global
36.8
47.3
93.8
95.0
16
47.3
768
47.3
(c) Decoder attention comparison. Attn type
(d) Decoder depth
(c) Decoder width
% of AS-2M mAP
epoch mAP
scenario IN-SSL IN-SL AS SSL AS-20K AS-2M
1% (AS 200)
46.5
(1)
37.1 (-00) 47.3 (00)
I 5 (AS-2M)
39.6
16
46.8
(2)
32.1 (50) 45.4 (-19)
10%
42.6
24
47.2
(2)
32.5 (46) 45.9 (-1.)
50%
47.3
(3)
36.9 (02) 47.1 (02)
100%
47.3
40
47.3
(3)
36.2 (409) 46.9 (o.4
(f) Pre-training size
(g) Pre-training epoch
(h) External ImageNet (IN) pre-training. SSL: w/ self- supervised MAE. SL: w/ supervised (fine-tuned) MAE.
Table 1: Ablation studies on AS-2M. The gray entries are the default Audio-MAE setup (ViT-B encoder, decoder with shifted local attention, pre-trained for 32 epochs). Table format follows [1].
gap to ViT-B can be significantly closed (5.0 + 2.3 mAP) when fine-tuning with more in-domain data (AS-20K + AS-2M).
Decoder: Table Ic compares decoder attention types in Audio-MAE. Note that decoders are discarded after pre- training and only the equal-sized ViT-B encoders are fine-tuned for the end task. Our results show that local attention with shifted window achieves the best per- formance. Combining local and global attention (i.e., hy brid attention, Hwin) also improves vanilla global self-attention. Fig. shows the qualitative reconstrue- tion comparison. In the spectrogram of vowels, the de- coder with local attention reconstructs better harmonics and recovers more context in the spectrogram. Similar phenomena are observed in the frictional sound in the middle consonant.
Ground Truth global attention w/ local attention
Figure 5: Decoder reconstruction comparison.
Table 1d ablates the impact of decoder depth on mAP. A deeper 16-layer decoder achieves better performance against its shallower variants. Note that our decoder uses local window attention by default where only a fraction of tokens (4x4 local windows vs. 64x8 with global attention) are attended. For global attention we find 8-layer decoders to perform better than 16-layer. Table le compares decoder width (embedding dimension). 512-dimension decoder achieves a good trade-off between computation and performance as wider one is not better.
Pre-training Data and Setup. Table summarizes the impact of pre- pre-training dataset size. Overall the model performance is monotonically increasing when using more data for pre-training. Comparing the performance of using 1% well-annotated AS- S-2 balanced data to using randomly sampled 20K unbalanced data for pre-training, the similar mAPs (39.4 vs 39.6) suggest that the distribution of data classes (balanced vs. unbalanced) is less important for pre-training. Meanwhile, as shown in Table 1g, training for longer is beneficial yet the performance saturates after the 24-th epoch.
Oft-of-domann Pre-training on ImageNet. Initializing audio models from ImageNet pre-trained weights has become popular for audio classification. However, as there are significant discrepancies between image and audio modalities, it is questionable if out-of-dommin pre-training benefits audio representation learning. In Table Ih we design 3 scenarios to investigate this for Audio-MAEE (1)
7

---

Model
Backhone
PT-Data AS-20K AS-2M
ESC-50
SPC-2
SPC-1
SID
No pre-training
ERANN [58]
CNN
45.0
89.2
PANN [59]
CNN
27.8
43.1
83.3
61.8
In-domain self-supervised pre-training
wav2vee 2.0 [33]
Transformer LS
96.2'
75.2"
HuBERT [35]
Transformer LS
96.3"
81.4"
Conformer [37]
Conformer AS
41.1
88.0
SS-AST [18]
ViT-B
AS+LS
31.0
88.8
98.0
96.0
64.3
Concurrent MAE-based works
MaskSpec [43]
VIT-B
AS
32.3
47.1
89.6
97.7
MAE-AST [38]
VIT-B
AS+LS
30.6
90.0
97.9
95.8
63.3
Audio-MAE (global)
VIT-B
AS
36.6...1
46.8
06
93.6
9..33..00
97.61.06
:..16.06
Audio-MAE (local)
VIT-B
AS
37.0
47.3
11
94.1 + 10
98.3 1..6
96.94.00
94.8
Out-of-domain supervised pre-training
PSLA [30]
EffNet [60]
IN
31.9
44.4
96.3
AST [10]
DeiT-B
IN
34.7
45.9
88.7
98.1
95.5
41.1
MBT [11]
ViT-B
IN-21K
31.3
44.3
HTS-AT [29]
Swin-B
IN
47.1
97.01
98.0
PaSST [28]
DeïT-B
47.1
96.81
Table 2: Comparison with other state-of-the-art models audio and speech classification tasks. Metrics are mAP for AS and accuracy (%) for ESC/SPC/SID. For pre-training (PT) dataset, AS:AudisSet, LS:LibriSpeech, and Im.mmeeeNet. t; Fine- Fine-tuning results with additional supervised training on AS-2M. We gray-out models pre pre-trained with external non-audio datasets (e.g., ImageNet). Best single models in AS-2M are compared (no ensembles). ": linear evaluation results from [53].
Audio-only pre-training (AS-SSL) from scratch. We consider this the ideal schema for learning audio representations as it is simple and clean setup that prevents uncontrollable bias transfer from other modalities. (2) Directly using self-superviise ImageNet MAE models (IN-SSL) and its fine-tumed variant (IN-SL). (3) Audio-MAE supe-super pre-training on top of these ImageNet weights.
The results show that (I) from-scratch audio-only pre-training is the best. For scenarios (2) and (3), observe that ImageNet pre-training alone (2) is not sufficient (especially when the downstream data is smaller, AS-20KK, and, in self-supervised pre-training on AudioSet, ImageNet initialization (3) does not help but degrades accuracy. Also in (3), supervised ImageNet pre-training (IN-SL) seems harmful. Consequently, the result suggests that out-of-domain pre-training (i.e., ImageNet) is not helpful for Audio-MAE, possibly due to domain shift.
4.5 Comparison with the State-of-the-art
Table 2 compares Audio-MAE (with 3-run error bars) to prior state-of-the-art. We categorize the comparison into groups. For fair comparison, main benchmark is the models in the middle group with self-supervised pre-training on in-domain (audio) datasets (AudioSet and LibriSpeech). For reference also list other models without pre-trainin me-training (the top group) and other models with supervised pre- tre-training on out-of-domain ImageNet (the bottom group), where the latter contains previous best systems the datasets.
Pre-trained on AudioSet, Audio-MAE achieves the best performance across all tasks compared to other models with in-domain self-supervised pre-training. On AudioSet-20K,, its 37.1 mAP significantly outperforms all other approaches including concurrent works and other models with out- of-domain pre-training. On AudioSet-2M and ESC-50, method also outperforms Conformer [37] and SS-AST [18]. Notably, unlike SS-AST and concurrent MAE-AST [38], which trained with additional 1,000 hours of speech in Librispeech, we use only AudioSet for pre-training.
In the bottom group of Table 2, Audio-MAE also outperforms previous state-of -hrt models with ImageNet supervised pre- training. Note that the proposed Audio-MAFE does not rely on any out-of- domain data and labels, nor using knowledge distillation (e.g., DeiT) from additional CNN-based models. Also, compared to HTS-AT [29] and PaSST [28], Audio-MAE is trained with audio under 16K sampling rate. As experimented [59], there could be up 0.4 potential mAP improvement for Audio-MAr if audio with 32K sampling rate are available.

---

(a) Unstructured 2
(b) Unstructured 2
(c) Unstructured 2.3
(d) Unstructured 2
Structured 2
Structured 2
(g) Structured 1 223 (h) Structured 1 2.3
Figure 6: Spectrogram reconstruction visualizations on the AudioSet eval set. Column-wise type: speech, music, event, others. Masking type: (a-d) unstructured (random); (e-h) structured fimquueucy Masking Ratio: 70%. In each group, we show the original spectrogram (1, top), masked input (2, middle), and MAE output (3, bottom). The spectrogram size is 1024 128; patch size is 16x 16. Each sample has 8451- patches with 154 (70% masked) patches being visible to Audio-MAF. Please click 2 3) for audible. wavrs. More audible examples are Supplementary.
For the speech tasks (SPC-1, SPC-2, and SID), Audio-MAE outperforms other models without pre-training (ERANN [58], PANN [59]), supervised (AST) and self-supervised models (SS-AST, MAE-AST), We further list other works (marked with ") to include the latest results introduced in the SUPERB [53] benehmark. But note that these results are not strictly comparable since SUPERB employs linear evaluation where the underlying pre-trained models are not end-to-end fine-tuned.
In summary, with audio-only from-scratch pre-training AudioSet, Audio-MAE, performs well for both the audio and speech classification tasks.
4.6 Visualization and Audible Examples by Audio-MAE Decoder
For better visualization, we follow MAE [1] to use MSE over non-normalized spectrograms as the self- supervised objective. We use ViT-L as the Audio-MAE encoder for visualization. Fig. 6 illustrraes the reconstruction results sampled from the AudioSet-2M eval set. We further reconstruct
using Griffin-Lim [61] algorithm, audible under the anonymous links (accessible respective 3).
As can be seen and heard, for various masking strategies and different sounds, our Audio-MAF generates reasonable reconstruction. works well for noisy event sounds (e.g., the reconstructed siren in Fig. 6c-3), as well as speech and music (e.g., the reconstructed singing in Fig. 6b-3). Notably, unlike visual contents that are typically ocale/translationpposttiio invariant [19], absolute positions and arrangement of spectrogram components are critical for humans to understand sound [62]. For example, shifting a pitch will make an audio sounds completely different. Also, phoneme sequences in time are important cues for speech understanding. Consequently, unstructured masking produces better aligned outputs that are closer to the ground-truth (top row each subfigure) as model can make better predictions based on nearby spectrogram patches; while structured masking is harder (less accurate or with words missing), especially when masking is performed over the time axis. failure example (missing words) is the reconstructed speech in Fig. 6e-3,

---

5 Conclusion
We have explored simple extension of MAE [1] to audio data. Our Audio-MAE learns reconstruct masked spectrogram patches from audio recordings and achieves state-of-the-art performance on six audio and speech classification tasks. We have drawn four interesting observations: First, simple MAE approach works surprisingly well for audio spectrograms. Second, we find that it is possible to learn stronger representations with local self-attention in the decoder. Third, we show that masking can be applied to both pre- pre-training and fine-tuning, improving accuracy and reducing training computation. The optimal strategy depends on the nature of the data (audio, image, etc.) and the learning type (self-/supervised). Fourth. the best performance can be achieved by pre-training and -unitt under the same modality, without reliance on cross- mods-mmd transfer learning. In future work, we aim to explore multimodal self self-supervised learning with a joint audio-visual MAE approach as these domains share natural correspondences in video data.
Acknowledgements. We thank Kaiming He and Luke Zettlemoyer for their feedback and discussions.
References
[1] K. He, Chen, S. Xie, Y. Li, P. Dollár, and R. Girshick, "Masked autoencoders are scalable vision learners," arXiv preprint (tdi::2/11.063777 2021.
[2] A. Vaswani, N. Shazeer, N. Parmar, Uszkoreit, L. Jones, A. N. Gomez. Kaiser, and I. Polosukhin, "Attention is all you need," in Proceedings of the 31st International Conference on Neural Information Processing Systems, ser. NIPS' 17. USA: Curran Associates Inc., 2017,
pp. 6000-6010.
[3] J. Devlin, M. Chang, K. Lee, and K. Toutanova, "BERT: pre-training of deep bidirectional transformers for language understanding." in Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Mimneapolis, MN, USA, June 2-7, 2019, Volume I (Long and Short Papers). Association for Computational Linguistics, 2019, pp. 4171-4186
[4] T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert- G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei, "Language models are few-shot learners," " Advances in Neural Information Processing Systems 33: Annual Conference Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020.
[5] Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V. Stoyanov, "Roberta: A robustly optimized BERT pretraining approach," CoRR, vol. abs/1907. 1 1692, 2019.
[6] K. He, H. Fan, Y. Wu, S. Xie, and R. B. Girshick, "Momentum contrast for unsupervised visual representation learning," " in 2020 IEEF/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020. Computer Vision Foundation / IEEE, 2020, pp. 9726-9735.
[7] X. Chen, S. Xie, and K. He, "An empirical study of training self-supervised vision transformers." in 2021 IEEE/CVF International Conference on Computer Vision, ICCV 2021, Montreal, QC, Canada, October 10-17, 2021. IEEE, 2021, pp. 9620-9629.
[8] R. Paulus, C. Xiong, and R. Socher, "A deep reinforced model for abstractive summarization."" arXiv, vol. abs// 04304, 2017.
[9] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer. G. Hcigold, S. Gelly et al., "An image is worth 16x16 words: Transformers for image recognition at scale," arXiv preprint arXiv:2010.11929, 2020.
[10] Y. Gong, Y. Chung, and J. R. Glass, "AST: audio spectrogram transformer," in Interspeech 2021, 22nd Annual Conference of the International Speech Communication Association, Brno, Czechia, 30 August 3 September 2021. ISCA, 2021, pp. 571-575.
[11] A. Nagrani, S. Yang, A. Arnab, A. Jansen, C. Schmid, and C. Sun, "Attention bottlenecks for multimodal fusion," in Advances in Neural Information Processing Systems 34: Annual
10

---

Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual. 2021. pp. 14200-14213,
[12] Gemmeke, P. Ellis, D. Freedman, A. Jansen, W. Lawrence, R. C. Moore, M. Plakal, and M. Ritter, "Audio set: An ontology and human-labeled dataset for audio events," in 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2017, pp. 776-780.
[13] K. Piczak, "ESC: Dataset for Environmental Sound Classification," in Proceedings of the 23rd Anmual ACM Conference on Multimedia. ACM Press, 2015, pp. 1015-1018.
[14] H. Touvron, M. Cord, M. Douze, F, Massa, A. Sablayrolles, and H. Jégou, "Training data- efficient image transformers & distillation through attention," in International Conference on Machine Learning. PMLR, 2021, pp. 10 347-10 357.
[15] J. Carreira and A. Zisserman, "Quo vadis, action recognition? A new model and the kinetics dataset," in 2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI. USA, July 21-26, 2017. IEEE Computer Society, 2017, pp. 4724-4733.
[16] C. Feichtenhofer, A. Pinz, and R. P. Wildes, "Spatiotemporal residual networks for video action recognition," in NIPS, 2016.
[17] H. Bao, L. Dong, and F. Wei, "Beit: BERT pre-training of image transformers," CoRR, vol. abs/2106.08254 2021.
[18) Y. Gong, C.-I. Lai, Y.-A. Chung, and J. R. Glass, "Ssast: Self-supervised audio spectrogram transformer," ArXiv, vol. abs/2110.09784, 2021.
[19] Z. Liu, Y. Lin, Y. Cao, H. Hu, Y. Wei, Z. Zhang, S. Lin, and B. Guo, "Swin transformer: Hierarchical vision transformer using shifted windows," in 2021 IEEE/CVF International Conference on Computer Vision, ICCV 2021, Montreal, QC, Canada, October 10-17, 2021. IEEE, 2021, pp. 1992-10002.
[20] C. Wei, Fan, Xie, C. Wu, A. L. Yuille, and C. Feichtenhofer, "Masked feature prediction for self-supervised visual pre-training," CoRR, vol. abs/21 12.09133, 2021.
[21] P. Vincent, H. Larochelle, Y. Bengio, and P. Manzagol, "Extracting and composing robust features with denoising autocncoders." in Machine Learning. Proceedings of the Twenty-Fifth International Conference (ICML 2008), Helsinki, Finland, June 5-9, 2008, ser. ACM International Conference Proceeding Series, vol. 307. ACM, 2008, pp. 1096-1103.
[22] P. Vincent, H. Larochelle, Lajoie, Y. Bengio, and P. Manzagol, "Stacked denoising autoencoders: Learning useful representations in deep network with local denoising criterion," J. Mach. Learn. Res., vol. 11, pp. 3371-3408, 2010.
[23] D. Pathak, P. Krähenbühl, J. Donahue, T. Darrell, and A. A. Efros, "Context encoders: Feature learning by inpainting," in 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV. USA, June 27-30, 2016. IEEE Computer Society, 2016, pp. -264..
[24] M. Chen, A. Radford, R. Child, J. Wu, H. Jun, D. Luan, and Sutskever, "Generative pretraining from pixels," in Proceedings of the 37th International Conference on Machine Learning, ICMI. 2020, 13-18 July 2020, Virtual Event, ser. Proceedings of Machine Learning Research, vol. 119. PMLR, 2020, pp. 1691-1703.
[25] A. Ramesh, M. Pavlov, G. Goh, S. Gray, C. Voss, A. Radford, M. Chen, and I. Sutskever, "Zero-shot text-to-image generation," in Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, Proceedings of Machine Learning Research, vol. 139. PMLR, 2021, pp. 8821-8831.
[26] N. Dalal and B. Triggs, "Histograms of oriented gradients for human detection," in 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR 2005), 20-26 June 2005, San Diego, CA, USA. IEEE Computer Society. 2005, pp. 886-893.
[27] K. He, X. Zhang, S. Ren, and J. Sun, "Deep residual learning for image recognition," 2016 IFFE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770-778, 2015.
[28] K. Koutini, Schlüter, H. Eghhal-zadeh, and G. Widmer, "Efficient training of audio transformers with patchout," CoRR, vol. abs/2110.05069, 2021.

---

[29] K. Chen, X. Du, B. Zhu, Z. Ma, T. Berg- Srrg-KKikpatrick, and S. Dubnov, "Hts-at: A hierarchi- cal token-semantic audio transformer for sound classification and detection," arXiv preprint Stt::2202.008774 2022.
[30] Y. Gong, Y. Chung, and J. Glass, "Psla: Improving audio tagging with pretraining, sampling. la- beling, and aggregation," " IEEF/ACM Transactions on Audio, Speech, and Language Processing, 2021.
[31] Y. Gong, S. Khurana, A. Rouditchenko, and J. Glass, "Cmkd: Cmn/transformer-base.
cross-model knowledge distillation for audio classification," 2022.
[32] S. Schneider, A. Baevski, R. Collobert, and M. Auli, "wav2vec: Unsupervised pre-training for speech recognition," in Interspeech 2019, 20th Annual Conference of the International Speech Communication Association, Graz, Austria, 15-19 September 2019. ISCA, 2019, pp. 3465-3469.
[33] A. Baevski, Y. Zhou, A. Mohamed, and M. Auli, "wav2vec 2.0: A framework for self-supervised learning of speech representations," in Advances in Neural Information Processing Systems 33: Anmual Conference Neural Information Processing Systems 2020, NeurlPS 2020, December 6-12, 2020, virtual, 2020.
[34] A. Baevski, W. Hsu, Q. Xu, A. Babu, J. Gu, and M. Auli, "data2vec: A general framework for self-supervised learning in speech, vision and language." CoRR, vol. abs/2202.035555 2022.
[35] W.-N. Hsu, B. Bolte, Y.-H. H. Tsai, K. Lakhotia, R. Salakhutdinov, and A. Mohamed, "Hu- bert: Self-supervised speech representation learning by masked prediction of hidden units," " IEEE/ACM Transactions Audio, Speech, and Language Processing, vol. 29, pp. 3451-3460, 2021.
[36] B. Shi, W. Hsu, and A. Mohamed, "Robust self-supervised audio-visual speech recognition," CoRR, vol. abs/2201 101763,, 2022.
[37] S. Srivastava, Y. Wang. A. Tjandra, A. Kumar, C. Liu, K. Singh, and Y. Saraf, "Conformer- self-supervised learning for non-speech audio tasks," arXiv preprint arXiv:2110.07311, 2021.
[38] A. Baade, P. Peng, and D. Harwath, "Mae-ast: Masked autoencoding audio spectrogram transformer." arXiv preprint arXiv:2203. 16691, 2022.
[39] A. van den Oord, Y. Li, and O. Vinyals, "Representation learning with contrastive predictive coding," CoRR, vol. abs/ abss/1807.05 807.03748, 2018.
[40] R. Arandjelovic and A. Zisserman, "Objects that sound," in Computer Vision ECCV 2018- 15th European Conference, Munich, Germany, September 8-14, 2018, Proceedings, Part I, ser. Lecture Notes in Computer Science, vol. 11205. Springer, 2018, pp. 451-466.
[41] M. Patrick, P. Huang, I. Misra, F. Metze, A. Vedaldi, Y. M. Asano, and J. F. Henriques, "Space-time crop & attend: Improving cross-modal video representation learning," in 2021 IEEE/CVF International Conference on Computer Vision, ICCV 2021, Montreal, QC, Canada, October 10-17, 2021. IEEE, 2021, pp. 10 540--1055..
[42] A. T. Liu, S. Yang, P. Chi, Hsu, and H. Lee, "Mockingjay: Unsupervised speech representation learning with deep bidirectional transformer encoders," in 2020 IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2020, Barcelona, Spain, May 4-8, 2020. IEEE, 2020, pp. 6429-6423.
[43] D. Chong, H. Wang. P. Zhou, and Q. Zeng, "Masked spectrogram prediction for self- audio pre- preitrainnng," 2022.
[44] D. Niizumi, Takeuchi, Y. Ohishi, N. Harada, and K. Kashino, "Masked spectrogram modeling using masked autoencoders for learning general-purpose audio representation," a2Xiv:2204.1 12260, 2022.
[45] Y. Li, C.-Y. Wu, H. Fan, K. Mangalam, B. Xiong, Malik, and C. Feichtenhofer, "Mvitv2: Improved multiscale vision transformers for classification and detection," in CVPR, 2022.
[46] A. van den Oord, Y. Li, and O. Vinyals, "Representation learning with contrastive predictive coding." CoRR, vol. 0tt//807.0336 807.03748, 2018.
[47] A. Baevski, M. Auli, and A. Mohamed, "Effectiveness of self-supervised pre-training for speech recognition," CoRR, vol.
2019.
12

---

[48] D. S. Park, W. Chan, Y. Zhang, C.-C. Chiu, B. Zoph, E. D. Cubuk, and Q. V. Le. "Specaug- ment: A simple data augmentation method for automatic speech recognition." " ArXiv, vol. abs/1904.08779) 2019.
[49] J. B. Li, S. Qu, P. Huang, and F. Metze, "Audiotagging done right: 2nd comparison of deep learning methods for environmental sound classification," " CoRR, vol. abs/2203. 13448, 2022.
[50] S. Hershey, S. Chaudhuri, D. P. W. Ellis, J. F. Gemmeke, A. Jansen, C. Moore, M. Plakal, D. Platt, R. A. Saurous, B. Seybold, M. Slaney, R. Weiss, and K. Wilson, "Cnn architectures for large-scale audio classification." in International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2017.
[51] S. Hershey, D. Ellis, E. Fonseca, A. Jansen, C. Liu, R. C. Moore, and M. Plakal, "The benefit of temporally-strone labels in audio event classification," in ICASSP 2-21-202/ IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2021,
pp. 366-370.
[52] P. Warden, "Speech Commands: A Dataset for Limited- Vooibuuary Speech Recognition," ArXiv e-prints, Apr. 2018
[53] S. wen Yang, P.-H. Chi, Y.-S. Chuang, C.-L. Lai, K. Lakhotia, Y. Y. Lin, A. T. Liu, J. Shi,
X. Chang, G.-T. Lin, T.-H. Huang, W.-C. Tseng, K. tik Lee, D.-R. Liu, Z. Huang, S. Dong, S.-W. Li, S. Watanabe, A. Mohamed, and H. yi Lee, "SUPERB: Speech Processing Universal PERformance Benchmark," " in Proc. Interspeech 2021, 2021, pp. 1194-1198.
[54] A. Nagrani, J. S. Chung. W. Xie, and A. Zisserman, "Voxceleb: Large-scale speaker verification
in the wild," Comput. Speech Lang., vol. 60, 2020.
[55] D. Povey, A. Ghoshal, G. Boulianne, L. Burget, O. Glembek, N. Goel, M. Hannemann, Motlicek, Y. Qian, P. Schwarz et al., "The kaldi speech recognition toolkit," in JEEE 201 workshop on automatic speech recognition and understanding, no. CONF, IEEE Signal Processing Society, 2011.
[56] S. Yun, D. Han, S. Chun, J. Oh, Y. Yoo, and J. Choe, "Cutmix: Regularization strategy to train strong classifiers with localizable features," in 2019 IEEE/CVF International Conference on Computer Vision, ICCV 2019, Seoul, Korea (South), October 27 November 2, 2019. IEEE, 2019, pp. 6022-6031.
[57] H. Zhang, M. Cissé, Y. N. Dauphin, and D. Lopez-Paz, "mixup: Beyond empirical risk minimization," in 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 May 3, 2018, Conference Track Proceedings. OpenReview.net, 2018
[58] S. Verbitskiy, V. Berikov, and V. Vyshegorodtsev, "Eranns: Efficient residual audio neural networks for audio pattern recognition," arXiv preprint arXiv:2106.0162,, 2021.
[59] Q. Kong, Y. Cao, T. Iqbal, Y. Wang, W. Wang, and M. Plumbley, "Panns: Large-scale pretrained audio neural networks for audio pattern recognition," IEEE ACM Trans. Audio Speech Lang. Process., vol. 28, pp. 2880-2894, 2020.
[60] M. Tan and Q. V. Le. "Efficientnet: Rethinking model scaling for convolutional neural networks," in Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, ser. Proceedings of Machine Learning Research, vol. 97. PMLR, 2019, pp. 6105-6114.
[61] D. Griffin and Lim, "Signal estimation from modified short-time fourier transform," IEEE Transactions on Acoustics, Speech, and Signal Processing, vol. 32, no. 2, pp. 236-243, 1984.
[62] Y. Suzuki and H. Takeshima, FEqual-loudness-levd contours for pure tones," The Journal of the Acoustical Society of America, vol. 1 116, no. 2, pp. 918-933, 2004.
13


Reviews:
[{"review_id": "gladqu-Tt-F", "reviewer": "Reviewer_v3sn", "summary": "This paper extends earlier work on masked autoencoders (MAE) with an application on self-supervised learning paradigm for audio processing. The main subject that is subjected to is mel spectrogram. Both encoder and decoder acquires transformer-based architecture and local attention is deployed at decoder side, out-performing solely global attention. Such network takes advantages of the nature of spectrogram, which encodes correlation on local time and frequency bands.", "questions": "1. The dataset covered in the work and related tasks are not that common for speech processing research community. Could you please address why you used AudioSet to train the model?\n\n2. For random masking, is ther any specific technique involved for generating the masks and define the default mask ratios? From earlier works? Or just pretty random thing we cannot tell?", "limitations": "There are two main limitations in this work which are not addressed:\n1. Acquisition of mel spectrogram only. It would be good if we try on various types of spectrograms.\n2. Since VoxCeleb is involved - a scoring framework might be beneficial for verification experiments.", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n1. The improvement from the conventional MAE encoder is clearly explained and simple to grasp and re-produced, with detailed ablation analysis on receiptive field.\n2. (Personal preference) The idea itself is closely related to the nature of spectrogram, instead of from the purely statistical perspective. Although the idea of regarding spectrogram as image patches is not new, this work put more emphasis on its relation with audio nature.\n\n## Weaknesses\n1. There are minor typos and grammatical mistakes in the paper. Please do proof-reading for the camera-ready version.\n2. Mel spectrogram has its own limitations in information sparsity (especially for higher frequency regions). Therefore, raw or linear spectrogram might be better choices for this, at least as an additional ablation study.\n3. VoxCeleb experiments. I am not 100% sure about how general ML community regard VoxCeleb, but officially it was proposed [1] as a verification task. Identification, in that case, can be regarded as an open-set verification.\n4.  Probably some comparison with supervised models can be useful. But this is trivial.\n5. There are some explanation in section I and II lacks justification or pointers to earlier studies, especially on explanining the nature of spectrogram's relation to speech cues - all things there makes sense to me, but needs some references.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "flF80iOB5PD", "reviewer": "Reviewer_THYp", "summary": "The author apply Masked Autoencoders from the vision domain to learn representations from audio spectrograms in a self-supervised fashion. The learned representaions can be fine-tuned to achieve competetive performance on different tasks. The learned representations can be fine-tuned for competitive performance on various tasks. The authors propose using local attention windows.", "questions": "I've listed the questions in the weaknesses section.", "limitations": "- The authors were up front about the similarity of their work to [42] [38]\n- The authors do not justify the claims about ImageNet transfer learning.\n- The large computational requirements compared to ImageNet transfer learning, with little improvments.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 4, "contribution": 1, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- Clear and well-written presentation.\n- Achieve SOTA on Audioset in a semi-supervised setup\n- Doesn't relay on Imagenet and knowledge transfer from the vision domain.\n- Extensive experiments on Audioset.\n\nWeaknesses:\n1. Low originality: the authors use MAE on spectrograms. I propose emphasizing the contributions of this work more clearly in the introduction (for example, adapting the local attention windows). Similar concurrent work  [42] [38] (as the authors point out clearly) exists. In particular, MaskSpec[42] is very close to the proposed work.  In light of this, what are the contibutions of your work(question to the authors)?\n\n2. The authors argue repeatedly against knowledge transfer from Imagenet. However, the claims against are not quite justified, it's not clear what the drawbacks of imagenet are (question to the authors)?. For example, in line 32, what do you mean by label bias in this situation?\n\n3. The huge computational complexity, even for fine tuning, especially compared to models pre-trained on Imagenet.\n    - For finetuning the supervised methods on Audioset: AST[10] can be trained on 4  (titan RTX) GPUS in 1 week. PaST[28]  (also uses masking with Patchout) in 25-50 hours on a single 2080ti.\n    - In SSL: MaskSpec[42] takes 4 days on 8x  V100 for the pretraining phase. I think this can be close in compute requirements to what the authors report?\n    -  How do you justify the huge computation complexity with the limited improvement (within the error bars) on Audioset? (question to the authors)  \n4.  The large computation complexity limits the usability of the models to extract representations (in my opinion).  (I suggest a large scale evaluation of the learned representaions on the tasks of the NeurIPS21 HEAR challange for example, https://hearbenchmark.com/ )", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "FNNiH84ZhJ", "reviewer": "Reviewer_JVUa", "summary": "The paper proposes a method for self-supervised learning for audio classification. It is based on the prior work MAE which does self-supervised learning on images. The paper under review adapted the MAE model to audio by uses the spectrogram as an image. The decoder was changed to use the local self-attention as the audio tens to have more local correlations.\n\nThen, the paper investigates the proposed architecture on several audio classification datasets. The paper conducts ablation studies on several aspects of the model: the masking strategies (finding that the unstructured masking works best for pre-training, and time+frequency masking works best for fine-tuning), the patch size and stride (finding that the overlap is not necessary), the encoder size, the decoder local or global attention, the length of the pre-training, and using the out-of-domain data for pre-training.\n\nFinally, the paper compares the results to the previously published works. The proposed model outperforms all the previous in-domain models. Then, the in-domain proposed model matches or outperforms some of the out-of-domain methods on some datasets, while it is worse on other datasets.", "questions": "The paper argues in 4.4 that the out-of-domain training is not beneficial or even harmful. Nevertheless, the Table 2 shows that other methods benefit from the out-of-domain training. This needs some clarification.", "limitations": "N/A", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strengths\n\nThe proposed model is sound. Applying and testing the CV models on audio is important and poses its own challenges which were overcome in this publication.\n\nThis paper conducts a thorough experimentation on multiple datasets with many ablations. The experimental section contains a lot of information and very interesting to read.\n\nIn general, the paper is very well written and easy to follow. I was able to all the logical steps and transitions. The amount of details is not overwhelming, but provides all the necessary information.\n\n# Weaknesses\n\nWhile I find it logical that pre-training on image data should not help much for audio training, the previous publications find it useful. This paper concludes that the out-of-domain data is not beneficial. This looks like an important area to address (see the question below).\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "EVxEmlFp7x", "reviewer": "Reviewer_NJ9B", "summary": "This paper presents the application of Masked Autoencoders (MAE) to audio spectrograms in a self-supervised framework. The overall idea is to rely on masking to widely corrupt the input data and train a Transformer AE to fill the blanked part of the spectrogram, leading to a form of inpainting self-supervised task. The authors study several different masking strategies in the context of spectrogram, and also analyse the impact of local or global attention in the decoder. The quality of the proposed method is assessed in a downstream speech and classification task.", "questions": "- My major question is on what you call a \"unstructured\" strategy, if we refer to Figure 2, I do not see any conceptual difference between (b) and (d), given that the \"unstructured\" is simply also a selection of time-frequency patches with a given size ? I think a truly unstructured approach would rather look like a Bernoulli dropout over the whole spectrogram ?\n- I quite disagree with the overall premise (l.57) that \"unlike image patches, spectrogram patches are mostly locally correlated\" ... and I even think that it is exactly the opposite ! The stationarity property is way stronger in images and local correlation are largely more present than in spectrograms. You take the example of formants, which is actually the strongest local correlation that you could find, and even then vocal formants still follow an harmonic structure (hence global and cyclical correlation across frequencies). Any harmonic or noisy sound is a counter-example to this local correlation property.\n- You talk about the high redundancy in the spectrogram (l. 118), and I think that analysing this aspect would make this paper stronger. How much it impacts the masked learning depending on the sound nature ? ", "limitations": "-", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 1, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Overall, the use of MAE for audio is an interesting avenue of research, as they perfectly fit to the self-supervised approaches. The experiments appear sound and are numerous, which is one of the major strength of the paper.\n\nHowever, the two main weaknesses of this paper is that the originality and amount of contributions is quite low (as the author state themselves it is a \"conceptually simple extension of MAE to audio\"). Although the authors study several properties of the model and input data, it feels more like an engineering paper and I would recommend to resubmit this paper to a more applied conference. Furthermore, the paper is sometimes not very well written with several long and hard to follow sentences. The same applies to the related work section, which is a laundry list of citations, without clearly delineating the advantages or flaws of past methods.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept