Masked Autoencoders with Multi-Window Local-Global Attention Are Better Audio Learners
Authors: Sarthak Yadav
Sergios Theodoridis
Lars Kai Hansen
Zheng-Hua Tan
Venue: ICLR 2024 poster (2024)
Keywords: self-supervised learning
masked autoencoder
audio representation learning
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: Masked Autoencoders fitted with Multi-Window Multi-Head Attention (MW-MAEs) explicitly model local-global context in all transformer blocks, and outperform standard Masked Autoencoders for learning a general-purpose audio representation.

Abstract: In this work, we propose a Multi-Window Masked Autoencoder (MW-MAE) fitted with a novel Multi-Window Multi-Head Attention (MW-MHA) module that facilitates the modelling of local-global interactions in every decoder transformer block through attention heads of several distinct local and global windows. Empirical results on ten downstream audio tasks show that MW-MAEs consistently outperform standard MAEs in overall performance and learn better general-purpose audio representations, along with demonstrating considerably better scaling characteristics. Investigating attention distances and entropies reveals that MW-MAE encoders learn heads with broader local and global attention. Analyzing attention head feature representations through Projection Weighted Canonical Correlation Analysis (PWCCA) shows that attention heads with the same window sizes across the decoder layers of the MW-MAE learn correlated feature representations which enables each block to independently capture local and global information, leading to a decoupled decoder feature hierarchy.

Published as a conference paper at ICLR 2024
MASKED AUTOENCODERS WITH MULTI-WINDON LOCAL-GLOBAL ATTENTION ARE BETTER AUDIO LEARNERS
Sarthak Yadav12 Sergios Thcodoridis Lars Kai Hansen Zheng-Huu Hua Tan 'Aalborg University PPooner Centre for Artificial Intelligence, Denmark 'Technical University of Denmark *University of Athens (sarthaky, sthe, zt ßes aau. ak, 1kai@dtu.dk
ABSTRACT
In this work, we propose a Multi- Wonddw Masked Autoencoder (MW-MAF) fit- ted with novel Multi-Window Multi-Head Attention (MW-MHA) module that facilitates the modelling of local-global interactions in every decoder transformer block through attention heads of several distinct local and global windows. Em- pirical results on ten downstream audio tasks show that MW-MAEs consistently outperform standard MAEs in overall performance and learn better audio representations, along with demonstrating considerably better scaling char- acteristics. Investigating attention distances and entropies reveals that MW-MAE encoders learn heads with broader local and global attention. Analyzing attention head feature representations through Projection Weighted Canonical Correlation Analysis (PWCCA) shows that attention heads with the same window sizes across the decoder layers of the MW-MAE learn correlated feature representations which enables each block to independently capture local and global information, leading to a decoupled decoder feature hierarchy.
purpose
INTRODUCTION
With rapid advances in hardware capabilities driving models of ever-increasing complexity and appetite for data, leveraging unlabelled data for learning effective deep representations has garnered significant interest. superviedd learning, which solves pretext task that utilizes labels generated from the data itself, has emerged as notable approach for training deep neural representations without labelled data. Several methods for learning self-supervised representations from audio data have been proposed, including autoregressive methods that learn to predict the future based on the past input (Oord et al., 2018) Chung et al. 2019). methods that learn contrastive representations from different views of the input (Saeed al.) 2021 Schneider al. 2019 Baevski et al. 2020 Sarkar cet al. 2019), and masked predictive modelling methods that learn to predict removed portions of the input data (Devlin et al. 2019).
Together with the transformer architecture (Vaswani et al. 2017) and its successors (Dosovitskiy et al.
2021 Liu al. 2021), masked predictive modelling has led to significant advances across natural language processing (NIP) (Devlin et al., 2019 Lewis et al. 2020), computer vision (Xie et al., 2022 2021a). Masked Autoencoders (MAEs)
Bao al. (022) and audio and speech processing (Hsu
by He et al. (222)) are a recent addition to the masked predictive modelling family. Initially proposed for learning visual representations from randomly masked image patches, MAES are experiencing
widespread adoption across several domains (Feichtenhofer et al. 2022 Wei et al. [022 Hou et al. 2022, Pang et al., 2022, Bachmann et al., [202)] Seo al. (023) due to their inherent scalability and simple design. In the audio domain, several recent works have adapted MAEs to learn general- purpose audio representation from spectrogram inputs (Baade et al., 2022] Niizui et al. 2022).
These works address several challenges that are unique to the audio domain and exhaustively study the effect of masking strategies and other hyperparameters, providing vital information for training MA Es on audio data.

---

Published as a conference paper at ICLR 2024
Agengned
Docoder
Froode
with
(a) MW-MHA module
(b) MW-MAE
Figure An overview of the proposed Multi-Window Multi-Head Attention (MW-MHA) mod- ule, and the overall MW-MAE architecture. In MW-MHA, each attention head operates on non- overlapping windows of different sizes (coded by different colours) of the input matrices. As evident from b) MW-MAE uses the proposed MW-MHA block only in the decoder.
Recent works have shown that leveraging local information in the Multi- Attention (MHA)
module of transformer layer through convolutions (Gulati et al. 2020). attention with local windows
(Beltagy et al. 2020: Liu et al., [202T] Dong et al., (022) or pooling attention (Fan et al.] 202T
Li et al. 2022 Zhu & Omar (2023)
lead to improved performance. Within the framework of
et al. (2022) evaluated the impact of local windowed attention (Liu al. 2 for audio representation learning, demonstrating better performance across downstream audio recognition tasks. However, in these methods: (i) all the attention heads within a MHA module operate at the same local context, thus only capturing local information the transformer layer level, and (ii) they require explicit approaches to mitigate the lack of connections across windows and to capture local-global information.
Masked Autoencoders,HHuaan
In this work, we propose Multi-Windoww Masked Autoencoders (MW-MAE) with both local and global attention for learning general-pure purepos- audio representations from spectrogram inputs. Decoders in an MW-MAE are fitted with novel Multi Window Multi-Heas Attention module (MW-MHA) (Fig D Each attention head in the proposed MW-MHA module computes self-attention over non-overlapping windows of different sizes, which facilitates modelling of local-global interactions in every decoder transformer block. The proposed MW-MAEs outperform standard MAEs on 10 downstream audio recognition tasks. At the same time, MW-MAEs adapt better to varying patch sizes and increasing number patches, perform better low-data scenarios, as well as demonstrate better performance and scaling characteristics with respect to changing encoder and decoder complexities. Exhaustive exploratory analysis of attention distances and entropies shows that attention heads in MW-MAE encoders learn broader local-global attention compared to standard MAEs, despite having an identical architecture. Further, analysis of feature representations from the decoder attention heads using Projection Weighted Canonical Correlation Analysis (PWCCA) (Morcos al., 2018) indicates that attention heads with the same window sizes across the decoder layers of the MW-MAE learn correlated feature representations leading to a decoupled feature hierarchy, confirming that MW-MHA modules learn local-global features in each decoder block.
2 BACKGROUND AND RELATED WORKS
Recently, several works have been proposed for learning audio representations in a self-supervised manner. Most of these works can be loosely categorized into one or more of the following groups based on their underlying pretext task: (i) predictive; (ii) contrastive; and (iii) masked predictive modelling. Several methods adopt a predictive coding approach, which aligns well with the sequential nature of audio input. Autoregressive predictive coding (APC) (Chung et al. 2019 2020 Chung Glass 2020) one such method which utilizes Recurrent Neural Networks (RNNs) to predict future elements of a sequential input given the past, whereas non-autoregressive approaches using Convolutional Neural Networks (CNNs) have also been proposed (Liu et all, 2021). Contrastive predictive coding (Oord et al. 2018) optimizes predictive coding objective in the latent space while simultaneously optimizing a contrastive objective function. This brings us to contrastive represen- tation learning, which operates on the premise of learning representation space that maximizes agreement between views from the same input sample while minimizing inter sample agreement. Several contrastive carning-based methods, originally proposed for computer vision (Chen et al.
2

---

Published as a conference paper at ICLR 2024
2020:
Chen & He
2021 Grill al. 2020). have been adapted for learning audio representations
Elbanna et al.] 2022). A widely used contrastive approach for learning speech representations is the Vec family of algorithms (Schneider et al. 2019
(Seeed et al. 2021
Nizumi et al. [021 1
Baevski et al. 2019: Baevski et al. 2020 Hsu et al. 202Tb), which learn contextualized speech
representations by optimizing contrastive objective hetween quantized latent representations and representations generated from masked time steps. Finally, self-supervised learning methods based on masked predictive modelling have simple premise: remove a portion of the input data, and learn to predict the removed content. After being (re-)popularized by the likes of BERT (Devlin et al. 2019) in NLP and fueled by the recognition of the Transformer (Vaswani et al. 2017) ) as a viable cross-domain neural architecture, masked modelling has seen wide adoption in several domains, such as computer vision (Bao et al. 2022 Xie al. 2022), audio and speech (Liu et al. 2020) as well as multi-domain self-supervised learning frameworks (Baevski et al., (022). In the audio domain, several recent methods use masked predictive modelling to learn self self-supervised representations (Baevski et al., 2019)) These include HuBERT (Hsu et al. [2211a), which trains BERT-like model to predict pre-determined cluster assignments from masked speech features, WavLM (Chen et al. 2022b). which learns joint denoising and masked prediction task, and SSaST (Gong al. 2022), which jointl reconstructs and contrasts masked patches. More recently, [Chen et al. (2023) proposed an iterative masked modelling approach using an iterative self-distileed tokenizer that generates refined discrete labels from audio input data for the next stage of pretraining.
Masked Autoencoders (MAEs): Recently, He et al. (2022) proposed Masked Autoencoders for learning self-supervised image representations. In an MAE, the input is split into non-overlapping patches, which are then linearly projected to a fixed dimension by the patch embedding layer. A large subset of these patches is masked out (e.g.. over 75% of the patches), and the unmasked patches are then encoded by a Vision Transformer (ViT) (Dosovitskiy et al.] (02T). With learnable mask tokens filled in the correct positions to restore the original patch order, these encoded patches are fed to transformer based decoder, whose objective is to learn to reconstruct the masked patches. The high masking ratio allows large encoders to be paired with significantly smaller decoders due to the reduced encoding complexity, while simultancously forcing the encoder to learn better contextualized representations by minimizing extrapolation from redundant neighbouring patches. Several recent works based on MAES have been proposed for training seneral-purpose audio representations. [Badde et al. (202)) explored a joint discriminative and generative objective for training audio MAES and evaluated fine-tuning performance on 5 downstream tasks. By training shallow downstream classifiers on 15 downstream tasks in accordance with the HEAR-2021 (Turian al., 2022) protocol, Niizumi et al. (2022) investigated various hyper-parametens such as patch size and the effect of input audio clip duration on model performance. Huuag al. (0222) investigated shifting windows based local self-attention (Liu et al. 2021) of fixed context (4 x 4 windows) in all but the last few layers of audio MAES. In contrast, this work we propose a Masked Autoencoder fitted with novel Multi- Window Multi-Head Attention module that can model attention at several context levels and can capture local-global interactions in every transformer layer.
PROPOSED APPROACH
3.1 MULTI-WINDOW MULTI-HEAD ATTENTION
To better capture local-global attention, we propose a Multi-Window Multi-Head Attention (MW- MHA) module, where each attention head computes self-attention across spectrogram patches in different local windows and then combines the contribution of each attention head, as illustrated in Figure Ia We define MW-MHA with h parallel heads as follows:
(W K, V)- = Concal(winFeedd, winieeead,,,...,,iaaaaa,,, w
(1)
winHead; WinAttenttinnQWW9 KKWK vwY. wwnn))
(2)
Here, Q, K.K. V. E Baxdm represent query, key and value input matrices, and W.O W.K W.Y E Rdlm xd are their corresponding learnable projection matrices. dm is the model's feature dimension, and d4 = dm. As opposed to MHA, each head winHead, computes local self-attention over non- overlapping windows of size win; by partitioning input matrices QW?, KW.K, VW! Rxdd
h
3

---

Published as a conference paper at ICLR 2024
into Qwin; Kwin; Vwing E Rmmxuinnxd given that standard self-attention Xwin, = Attentionn Q Qwin. Kwin, Vusn.) ) (Vaswani et al. 2017) on these partitioned inputs. Finally, Xwiin R wwinn Xdx is reshaped to X E R to get the output.
win, This is followed by computing
In the proposed MW-MHA module, individual attention heads capture information at multiple local contexts, and the final projection matrix W E R learns the contribution of each of these heads, allowing inter-window interaction and connection. This design facilitates learning both local and global time-frequency information several granularities in every transformer block (as supported by exploratory analysis in Section 5). This is in contrast to shifting (Liu et al. [021]] Chen
ct al. 2022a), striped (Dong al.
2022) windowed self-attention, or pooling attention (Fan et al.
2021
Li et al.] [2022] [Zu & Omar 2023 where all attention heads within the same block have the
same window size and thus only perform local self-attention at the block level. Pseudo-code for the
proposed MW-MHA is provided in Appendix A
3.2 MASKED AUTOENCODER WITH MULTI- WINDOW MULTI-HEAD ATTENTION
Patch embeddings, masking strategy and masking ratio: We use mel-spectrograms as inputs, partitioning them into non-overlapping patches, which are then flattened and embedded into linear projections. For encoding positional information, we use fixed sinusoidal positional embeddings, similar to Baade et al.] (2022); Niizumi et al. (2022); Huang et al. (2022). We use high masking ratio (80%) and random unstructured masking, which have been shown to work well for audio Niizumi al.
2022 Huang et al., 2022).
Encoder: In line with previous work (He et al. 2022: Niizumi et al. 2022; Huang et al. 2022). we use a Vision Transformer (ViT) (Dosovitskiy al. 2021) based encoder, which only processes non-masked patches (20% in this work). Due to the random masking strategy, majority of the patches are not processed by the encoder at training time. This minimizes the benefit of using the proposed MW-MHA modules in the encoder transformer blocks (as evidenced by experiments in Section 4.4). Thus, transformer blocks in our encoder use standard Multi-Head Attention.
Decoder with Multi- Window Multi-Head Attention: We add fixed sinusoidal positional embed- dings to the encoded visible patches concatenated with trainable masked tokens after restoring original patch order. The resulting tensor is then fed to the decoder, which is also a stack of transformer layers, followed by a linear head that reconstructs the original input spectrogram. This is consistent with previous works (He et al. 2022 Nizumi et al. 2022 Huang et al. 2022 Given that the decoder processes all the patches, we replace the Multi Head Attention module with the proposed Multi-Windor Multi-Head Attention, thus modelling local-global attention in every decoder block.
We follow a simple rule for determining the window sizes of each constituent winHead;; given the total number of patches np, we simply take all non-unary factors of n, and add two additional global self-attention heads. As an example, our default configuration yields n, 250, and thus the window sizes for each MW-MHA module in all decoder blocks will be [2, 5, 10, 25, 50, 125, 250, 250] for total of 8 attention heads, which is reasonable number of attention heads inline with previous research (He et al., 2022 Huang et al. 2022). Not only is this method simple to follow, but it also scales well with number of patches, effectively covering several possible local context levels.
Selecting window sizes:
Pre-training objective: During pre-training, we optimize loss function that computes mean squared error (MSE) between the predicted masked patches and their corresponding input spectrogram patches. In early experiments, we observed reduced performance when using per-patch normalization, and thus we do not normalize target spectrogram patches.
4 EXPERIMENTS
4.1 DATASETS AND TASKS
Pre-training: We the full AudioSet dataset (Gemmeke et al. 2017) (AS-5k) for pre-training MAES and MW-MAEs. With over 5000 hours of audio data distributed in 2 million 10-second weakly annotated YouTube clips spanning 527 classes, AudioSet is one of the largest publicly available audio corpora.
4

---

Published as a conference paper at ICLR 2024
Downstream tasks: Recently, several standardized benchmarks have been proposed to evaluate audio representations thoroughly across a wide variety of domains, such as SUPERB (Yang et al. 2021a) and HEAR (Turian et al. 2022). While both benchmarks offer avenues for fast, reproducible and accessible comparison of audio representations, the SUPERB benchmark focuses primarily on speech-processing applications. In contrast, the HEAR benchmark consists of 19 tasks spanning diverse audio domains of speech, music and environmental sounds and redistributes standardized and preprocessed datasets. However, some of these tasks are simply smaller subsets of one another, whereas performance on some HEAR tasks has been demonstrated to be correlated (Turian et al. 2022p. For evaluating audio representations, we utilize a subset of the HEAR benchmark which con- sists of ten diverse tasks spanning multiple domains: Beijing Opera, Crema-D, ESC-50, LibriCount, Mridangam Stroke and Tonic, NSynth Pitch 5h, Speech Commands 5h, FSD50K and VoxLingua More information can be found in Appendix B along with the underlying selection criterion. We believe the selected tasks constitute a balanced evaluation protocol that facilitates assessment of audio representations without doing excessive evaluations. For downstream evaluation, we follow the HEAR protocol, where for each task, a shallow downstream classifier is trained on top of fixed features extracted using pretrained model. This practice has become quite prevalent and allows the evaluation of how representations generalize to broad range of tasks without the drawbacks of fine-tuning large, heterogeneous neural networks.
Mcasuring overall performance: Given the wide variety of downstream tasks and feature repre- sentations evaluated, a single metric to quantify the performance would significantly aid analysis. However, given the differing difficulty levels of the tasks as well as outliers arising from the nature of the representations evaluated, simply averaging the scores is not sufficient. To counteract this, we utilize a normalized overall score to track overall performance of a given audio representation. Mathematically, overall score s(m) E [0., 100.] of model m given as:
Te (m(m)) - mine * 100
smm)
T
maz;-
min,
EET
where (m) denotes performance of the model m task t, and men, and max; represent the worst and the best performance across all models on the task. By taking the relative performance of the best and the worst approach task into consideration, this overall score takes how hard the task is to improve on in consideration. It is worth noting that the normalized score is computed across all the evaluated methods in all upcoming sections, including ablations. This is similar to the overall score used by the public leaderboard of the SUPERB Yang et al. (2021a) benchmark, except that we do not set the normalized value of the worst performing method to 0, and the proposed overall score has an upper range of 100.0.
4.2 IMPLEMENTATIO DETAILS
Features: We use log-sc mel spectrograms with a window size of 25 ms, a hop size of 10 ms and F = 80 mel-spaced frequency bins in the 50- 8000 Hz range, extracted using the torchaudio (Yang) et al. 2021b) toolkit. All datasets have a sampling frequency of 16000 Hz. Instead of normalizing by dataset statistics, we adopt a per-instance standardization scheme.
We use the AudioSet dataset for pre-training our Masked Autoencoders. We extract log-scaled mel spectrograms for the entire AudioSet dataset and randomly crop segment 200 timesteps in length from each data sample. Our default configuration consists of ViT-B encoder. All our MAE variants accept 200 x 80-dimensional (T x F, respectively) input corresponding to an audio duration of 2 seconds, which achieves performance on on-par with longer input durations as demonstrated by Niizumi al. (2022). For our default configuration, our patch embedding computes non-overlapping patches with a patch size of (4 16), given it's desirable performance v/s complexity tradeoff as found by Niizumi et al. (2022). A key characteristic of the Masked Autoencoder paradigm is its asymmetric design, which allows pairing small decoders with large encoders while scaling favourably for linear probe performance (He et al. 2022). Thus, in contrast to Huang et al. (2022). we adopt smaller 4-yayer deep transformer-base default configuration. We train Masked Autoencoders with the proposed MW-MHA module, which referred to MW-MAEs, well as their standard MAE counterparts. All MAEs are pre-trained for 100 epochs with batch size of 1024 and a weight decay of 0.05 on a single TPU-v3 VM with 8
Pre-training:
decoder of width 384 and attention heads for our
5

---

Published as a conference paper at ICLR 2024
Table Comparison with various audio representations from the literature. 95% confidence intervals are reported over 10 runs on downstream classifiers. We pre-trained all highlighted audio representa- tions, with different gray levels indicating directly comparable MAE and MW-MAE configurations. We also pre-trained an AudioMAE model MAudioMAE-B-4x16444 16-41") from scratch, which is directly comparable to our base configurations. We only pre-trained For other pre-trained audio representa- tions, publicly available official implementations were used. All downstream models were trained by us using the hear-eval-Ke -itt s s(m) denotes the proposed normalized overall score (Sec 4 *: same configuration MSM-200 16x4 (Nizumi et al. 2022). with 8 attention heads in the decoder instead
of 6. For model parameter counts, refer to Appendix G
Model
Data
BO
CD
ESC-50
L.
Mri
Mri-T
NS-Sh
SC-Sh
F50K
VL
Naive Baselines
HEAR-Naive (Turian al.
2022
52 622.4 30.9.08 5,8.0.2 33.5a1.1 38.0 36.421.9
18.6144
8,5004
7.1
11.2a0.5
5,0
Supervised
PaSST- buse
Koutini et al.
2022
AS-5L
94.900.5 61.0bo03 94,8 60.. 10.12 96.5a0..1 67.66 23.3at19 66.621.4
64.2 25, 25.5 73.5.004
SSL
W2V2-base
Raevski cl
2020
IS-960
74.0=1 46.4e0.3
+1.4 51.2+0.2 77.3 15., 7.4-0.8 90.8e0.3 |8.1 35.5e0 43. 1
W2V2-large
2020
VP-100k
13.7 66.920.4 60.5 62.410.3 93.9t0.1 77.420.2
42.00.222.... 87.620.5 34.2:0.1 53.621.0 74.0s0.1
WavLM-buse
buse
DCen a.
40440
LS-960
89.4g0.7 56.3 46.622.44 63. 23.33 95, 120.1 83.420.2
37.3311.8 57. 29, 99.91 22.62006 600.5.0.2
WavLM large
Chen al
96.4 57.2+0.2 4 6 96.8+0.1 89.5e0.1 53.7+0.5 46.2 29, 0.000001 23.7=00 64.0s02
HuRERT-basc base
HSu of al
J02 m
TSS966
90.6 70.8+0.2 6 56. 56.5+0.3 94,4+0.1 84.9+0.3 19.4+0,7 99,440,73,22 32,3-0.1 61 61.8800 72.5+02
HuBERT-lagge
[Huu al
20
LL-60k
94.1 70.7a0.1 60.3.0.4 59.920.2 95.3 83.5g0.3 19.3 83,220.7 31.5.0.1 66.1s0.9 73.4
SSaST-base
Goog 0t
212
AS+LS
93.42099 56.520.2 68.420.4 60.7.0.3 96.740.1 96.340.1 66.820.7 53.521.3 38.240.1 28.5g00 71.7202
BEATs-iter3 (Chen ot at
2023
AS-5k 94.0s0.8 67.3 83.7 68 0...00.2 94.710.1 95.810.1 69.420.8 85,2 53.6.0.2 38.5a1. 85.7
MAE based
AudioMAE (Huang et
2022
AS-5k
93.7a06
2..2.0....... 60.6.0.4
42.2 89.2.0.2 86 620.2 64.5.0.8 28, 6.1.51 37.990.1 29.7a1 62.99.0..
AudioMAE-B-XXI--!
AS-5k
76.040.52.44403
72.0a0.5 66.940.4 27.240.0 98.240.1 69.8 89.840.3-49) 49..0.90. 38.3g0.8 16.1
MAE-B-xx16-41
AS-5K
96.240.3
72. 240. 80.9204 67.3 97.420.1 98.320.1 68.320.4 89.4200.3 50.420.1 43.1209 88.1
MAB-B-5x5-I1
AS-5k
96.0s0.4
70.9 80.9 67. 620.4 97.620.1 98.420.0 69.3.0.4 88.420.3 49.33.. 37.720.6 86.8
MAB-L-KI6-81
AS-5K
1604 73.8 81.620.3 68.5.0.2 97.62011 98.3a0.0 69.0sa.5 91.20..22 21.22.......0000
46.920.8 90.0s0.2
Proposed
MW--MAE-B--XI--1
AS-5k
96.0g0.5
73.1
81.2 68.880.2 97.44 97.940.1 69.320.6 90.990.2 01.40.2..1.....
51.2a0..
44.4444000 89 89.22002
MW-MAB-B-Sx5-II
AS-5k
96.620.433.88
82.020.3 70.1 99..5a0.1 98.340.1 72.920.5 91.7202 51.320.1 44.22.0.6 90.6200.
MW-MAE-L-4XI--8
AS-5k
95.920.3 76.1.0.3 83.620.3 69.720.3 97.440.0 98.240.1 71.240.7 93.020.1 53.5 $1.90007 92.620.2
TPU cores, with the default configuration taking around 36 hours to train. We warm up for ten epochs to base learning rate of 1e-5, followed by a cosine decay schedule. A masking ratio of 0.8 with unstructured random masking is used, and no other data augmentations are used during pre-training.
Training downstream models: We first extract fixed feature embeddings for all downstream tasks to train downstream models. In the MAE framework, the decoder is discarded after pretraining and feature embeddings are extracted using just the encoder. To generate scene embeddings consistent with the HEAR protocol, we use the exact patch aggregation process as Niizumi et al. (2022): we break audio clips into non-overlapping second chunks, concatenating the features in time and finally taking a mean over the time axis to generate fixed vector representation independent of the input audio duration. The hear-eval-kit, released alongside the HEAR benchmark, was used to extract fixed feature embeddings and to train a shallow MLP classifier with a single hidden layer with 1024 neurons for each task in a reproducible manner. Experiments are repeated with at least ten random seeds for each task, resulting in 100 experiments for every evaluated representation.
4.3 COMPARISON WITH EXISTING WORKS
Table [ shows how MW-MAE fares against recent audio representations. The highlighted model configurations that we pre-trained from scratch on AudioSet have the following naming convention: the first substring shows the type of MAE (vanilla or proposed MW-MAE), followed by a single alphabet denoting ViT Encoder configuration. This followed by the patch size used, and finally, the depth of the decoder. It's worth noting that while embedding sizes of MAE and corresponding MW-MAE configurations are the same, the embedding sizes of other methods can be different. This is inline with the current consensus of evaluating self-supervised representations in the audio domain (Yang et al. 202 Turian et al. 2022). MW-MAE configurations outperform all other

---

Published as a conference paper at ICLR 2024
comparable MAEs, with the largest "MW-MAE-L-4X6--" configuration outperforning all the methods in overall performance (92.6Ha2), MW-MAEs also outperforms AudioMAE with standard shifting window based attention, well the recent BEATs-iterr approach, which is the pre-trained representation obtained after 3 stages of self-distilled learning as proposed by Chen et al. (2023). MW-MAES perform exceptionally well on pitch perception (NS-5h), while achieving performance on- par with speech specific representations such as WavLM, HuBERT and Wav2Vec2 (denoted W2V2) for Keyword spotting (SC-5h). Perhaps more surprisingly, they outperform speech representations trained on much larger training sets on the emotion recognition (CREMA-D) as well as speaker count classification (LibriCount) tasks. While PaSST, which is a recent state-of-the- approach for training supervised transformers on AudioSet, outperforms every model on ESC-50 and FSD50K tasks, the overall performance of the proposed approach is significantly better. Overall, the proposed MRs learn a better general-purpose audio representation than standard MAFs, generalizing well to several audio domains and demonstrating excellent overall performance in comparison to recent audio representations.
4.4 KEY MODEL CHARACTERISTICS
We conduct several experiments to examine key differences between MAE and the proposed MW- MAE. While we have only reported overall score s(m), detailed results for all these experiments. along with additional experiments on attention heads and window sizes can be found in Appendix H
MW-MHA in the encoder: adding MW-MHA to the encoder block does not im- prove downstream performance. Further, we also inves- tigate the impact of linear probing as well as fine-tuning the entire encoder stack for in-domain classification on AudioSet-20k balanced subset. No data augmentations were used. As evident from Table 2 when compared with including MW-MHA blocks in the decoder only, module placement there is no performance benefit to adding MW-MHA blocks the encoder for neither downstream performance, nor for in-domain linear probe on AudioSet-20k. However, when fine-tuning the entire encoder stack, adding MW-MHA blocks to both the encoder and the decoder provides a slight improvement and is worth considering.
As previously mentioned,
Model
Downetream Linear Prohe (mAP)
*(m)
Fine-tuning
MAE MW-MAF. Rase (deooder only)
88.1a012
23.150.3
26.1204
89.2 200.2
24.2+0.1
1+0.7
MW-MAE Base (encoder only)
89. 190.3
23.640..
26. 1=0.3
MW-MAE Base (encedze)
89.1a113
24.0200.1
26.2a010
Table 2: Performance impact of MW-MHA
Performance impact of various patch sizes: In an MAE, the patch embedding layer generates con-overlapping patches from the input. Thus, the size of the patch governs the number of patches as well as the time-freew frequency resolution that the transformer layers work at, making it an important hyperparameter to investigate.
Mumber
Encoder Parameters (Millions
125
230 502
640
3524 535.0
05-
100
MAE 60-
8x15
4b15
RCB
5x5
VEST
VTTM
VI5
VITL
VI51
2
25 50
50
Patch Size
Encoder Configurationn
Decoder depth
reduction In pre training 3303
(a) Patch size
(b) Encoder complexity
(c) Decoder depth
(d) Pre-training data
Figure 2: Ablation experiments comparing standard MAE v/s proposed MW-MAE at different patch
sizes (a), encoder complexity (b), decoder depth (c) as well as amount of pre-training data used (d). s(m) is the proposed overall score (Sec 4p. Detailed results can be found in Appendix H
Figure 2a shows how different patch sizes affect downstream performance. The proposed MW-MAE model, with an overall score of 90.6Ha.1, outperforms standard MAF for every patch size for identical decoder configurations. It's also worth noting that MAE performance degrades as we decrease the patch size beyond 4 x 16, whereas MW-MAE performance continues to improve. These observations show that the proposed MW-MAE adapts better to varying patch sizes and menefiry resolutions, while scaling well with increasing number of patches.
Encoder size: As shown in Figure 26] we investigate how encoders of five different complexities affect overall performance, All the trained models have the same decoder configuration (384 neurons,
7

---

Published as a conference paper at ICLR 2024
C.40
MAC Encoder 15% AuziSSe:
1AAS EnCRRer 50% AudwORt
IAAE Encoder, 107R Autro
1.025
B22
6062
2.2.44 220112
5
100312
VWNAAE Creoder, 22% Audisiet
WONAC Encoder, 50% Aud dde:
VINKAE Creoder: 100% Audidde
Derstrr
220
Ssideet Black
Ercover Slick
Mean actertior Desoncce
Moan Altencien D Ditaacce
Mian Actertion Distance
(a) Attention entropies in enenderr blocks
(b) Mean attention distances for select transformer blocks.
Figure 3: Investigating MAE and MW-MAE encoder attention heads. (a) depicts average entropies of encoder attention heads over the course of pretraining in a every encoder transformer block. (b) Depicts mean attention distance distributions of the first two and the last two transformer blocks at different amounts of pretraining data used.
depth=4, h=8). With overall score of 89,2.oz, MW-MAE with the ViT-Base encoder performs better than MAEs with encoders of any size in this experiment. The most prominent performance gap is observed for the ViT-Large setting, where MAE and MW-MAE attain overall scores of 88. 88.2+aa and 92.3+a2, respectively. The drop in performance for the ViT-Huge encoder for both MAES and MW-MAEs suggests possible overfitting.
Decoder depth: In Zc] we show how increasing decoder complexity by increasing decoder depth affects overall performance. As expected, increasing decoder depth improves performance for both methods. For decoder depth=8, MW-MAE (89 (922 outperforms MAE (88. 2201) by considerable margin in overall performance. We also observed that with an overall score of 88.3202, MW-MAE with depth=2 performs on par with MAEs with up to decoder blocks. This observation complements the inherent asymmetric nature of Masked Autoencoders, and thus the proposed MW-MAE performs favourably in terms of complexity and scalability.
Pre-training data: Finally, Figure 2d depicts how performance varies as we reduce the amount of data used for pre-training. Overall, performance for both the MAE and the proposed MW-MAE methods continues to decrease monotonically as we remove more and more data. However, the performance loss trend for MW-MAE is much more favourable. A 90% reduction in the amount of pre-training data results in a 28.17% reduction in performance for standard MAEs (from 88. La02 to 63.3-aa), whereas MW-MAE only suffers 13.5% drop in performance (from 89.2+aa 77.2+os). Thus, we conclude that the proposed MW-MAES are more adept at handling low-data scenarios in comparison standard MAES.
EXPLORATORY ANALYSIS
5.1 INSPECTING ENCODER ATTENTION HEADS
Analyzing attention entropies: We first analyze individual attention heads in a ViT-Medium encoder (depth=12, 12, h=8). Figure 3a shows scatter plots of average entropies of individual encoder attention heads computed over the entire NSynth Pitch 5h validation set on block-by block-by-block basis at different stages during pre-training. It's worth noting that the higher the entropy, the more global the attention, with lower attention mass spent on closer tokens (Clark et al. 2019), and thus, higher variance in entropies of individual attention heads highlights more spread out local and global attention. In the carly epochs, MAE encoders actually have higher variance in entropy distribution, especially in the latter transformer layers. As pretraining goes on, interestingly, this effect reversed, and the attention heads in the MW-MAE encoder now start converging towards high entropy variance configurations in the early layers.
Analyzing attention distances: We analyze mean attention distances for attention heads in the first two and the last two encoder blocks. Similar to Raghu et al. (202T), we compute attention attention-weighted patch distances between the query patch position and the locations it attends for each attention head,
8

---

Published as a conference paper at ICLR 2024
70-
03
64
855
55
52-
-63
1162443 B1CAN664
342112 62:44 1444 728098966 Attention heads
Accontion hcods
Abseston hoads
Accuncion hoads
Figure 4: Comparing features learned by different attention heads in the encoder and the decoder of standard MAE and the proposed MW-MAE using PWCCA. Each tick separates the attention heads of transformer block from the next.
averaging it for all patches positions. This is repeated for all inputs in the FSD50K validation set. Figure BuuureBbddar dejpetcss the distribution of mean attention distance for MAE and MW-MAE encoders (base configuration) pretrained with different amounts of training data. We can observe that MW-MAE attention heads demonstrate a broader distribution of attention distances, modelling local-global attention better than the MAE encoder especially in the first two transformer blocks. From these observations, we can conclude that in an MW-MAE, the decoder fitted with an MW-MHA can force the encoder to better capture local-global interactions even without explicit windowed attention modules, leading to improved performance.
5.2 COMPARING ATTENTION FEATURE REPRESENTATIONS THROUGH PWCCA
Several recent works have used Canonical Correlation Analysis (CCA) to compare feature repre- sentations and learning dynamics of deep neural networks (Raghu et al. 2017: Pasad et al. 2021). We use Projection Weighted CCA (PWCCA) (Morcos et al., 2018). which computes a weighted mean of the CCA vectors to compare the representations learned by individual attention heads of the encoder and the decoder in identically configured MAE and MW-MAE (VIT-M encoder: depth= 12, h=8; Default decoder: 384 neurons, depth=4, h= MW-MAE decoder uses default attention head window sizes as specified in Sec 3.2 Figure f depicts correlation matrices of measured PWCCA score between attention heads. We can observe remarkable difference correlation between the decoders: feature representations from the MW-MAE decoder attention heads with the same window sizes are strongly correlated across decoder layers, whereas attention heads with global self-attention (7, 8, 15, 16, 23, 24, 32) are the least correlated, consistent with observations made for the MAE decoder. These observations suggest a decoupling of different aspects of the feature hierarchy in the MW-MAE decoder, as attention heads of specific window sizes in each decoder block capture local information at a specific granularity, which line with our original hypothesis. These observations also corroborated by decoder depth ablation experiments from 4.4 where observed that MW-MAE with a single transformer block performs on par with MAEs fitted with up to decoder blocks. Finally, the difference in correlation matrices between the encoders much less stark, which is expected since both use standard MHA blocks.
6 CONCLUSION
This work presents Multi- Window Masked Autoencoder (MW-MAE) for learning general-purpose au- dio representations. Decoders in MW-MAEs are fitted with a novel Multi-Window Multi-Hean Atten- tion (MW-MHA) module, which learns information captured at multiple granularities of local-global context by its constituent attention heads computing eeff-attention over different non-overlapping windows. Empirical experiments on ten downstream tasks show that the proposed MW-MAES consistently outperform standard MAES in overall performance when pre-trained on the AudioSet dataset, demonstrating better scaling characteristics. Exploratory analyses highlight key differences between the attention representations learned by standard MAEs and the proposed MW-MAES. Based on attention entropy and mean attention distance analysis, we discover that encoder attention heads in an MW-MAE better capture local-globa interactions, even without explicit local-global attention modules. We also learn that attention heads of the same window size across the transformer blocks of the MW-MAE decoder are correlated, learning a decoupled feature hierarchy allowing transformers capture relevant information at the block level, supporting our original motivation.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGEMENTS
This project is supported by the Pioneer Centre for Artificial Intelligence, Denmark. We are also grateful to the TPU Research Cloud Program, a Google Research Initiative, for providing TPU v2 and v3 devices used in this project, as well as DeiC for enabling access to the LUMI supercomputer
REFERENCES
Akshay Anantapadmanabtan. Ashwin Bellur, and Hema A Murthy. Modal analysis and transcription of strokes of the mridangam using non-negative matrix factorization. In 2013 IEEE International Conference on Acoustics, Speech and Signal Processing, pp. 181-185, 2013. doi: 10.1109/CCSSS..
2013.6637633.
Alan Baade, Puyuan Peng, and David Harwath. MAE-AST: Masked Autoencoding Audio Spectro- 2022. doi: 10.214377/nterspeech.
gram Transformer. In Proc. Interspeech 2022, pp. 2022-10961.
Roman Bachmann, David Mizrahi, Andrei Atanov, and Amir Zamir. Multimae: Multi-modal multi- task masked autoencoders. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXXVII, pp. 348-367. Springer, 2022.
Alexei Baevski, Steffen Schneider, and Michael Auli. vq-wav2vec: Self-supervised learning of discrete speech representations. In International Conference on Learning Representations, 2019.
Alexei Baevski. Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. wav2vec 2.0: A frame- work for self-supervised learning of speech representations, In Advances in Neural Information Processing Systems, volume 33, pp. 22449-1246/ 2020.
Alexei Baevski, Wei-Ning Hsu, Qiantong Xu, Arun Babu, Jiatao Gu, and Michael Auli. data2vec: A general framework for self-supervised learning in speech, vision and language. In Kama- lika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Conference Machine Learning. volume 162 of Proceedings of Machine Learning Research, pp. 1312. PMLR, 17-23 Jul 2022. URL https: pettps:/pprocees http:///roeeeddeeee/o mw wtt.gssmlr...rss....m www.s/resseee.... v1 62 bttp//aawwaaaaaaaa htm
Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei. BEit: BERT pre-training of image transformers. In International Conference on Learning Representations, 2022. URLAttps:///peeeeiiii
Bh2S25904
Iz Beltagy, Matthew E. Peters, and Arman Cohan. Longformer: The Lang-Doeument Transformer, December 2020.
arXiv:2004.05150 [cs].
Houwei Cao, David G Cooper, Michael K Keutmann, Ruben C Gur, Ani Nenkova, and Ragini Verma. Crema-d: Crowd-sourced emotional multimodal actors dataset. IEEE transactions on affective computing, 5(4):377-390. 2014.
Ke Chen, Xingjian Du, Bilei Zhu, Zejun Ma, Taylor Berg- Kirk-Kiikpatrick, and Shlomo Dubnov. Hts-at: A hierarchical teken-semantic audio transformer for sound classification and detection. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 646-650. IEEE, 2022a.
Sanyuan Chen, Chengyi Wang, Zhengyang Chen, Yu Wu, Shujie Liu, Zhuo Chen, Jinyu Li, Naoyuki Kanda, Takuya Yoshioka, Xiong Xiao, et al. Wavlm: Large-scale self-supervised pre-training for full stack speech processing. IEEE Journal of Selected Topics in Signal Processing, 16(6):
1505-1518, 2022b.
Sanyuan Chen, Yu Wu, Chengyi Wang. Shujie Liu, Daniel Tompkins, Zhuo Chen, Wanxiang Che, Xiangzhan Yu, and Furu Wei. BEATs: Audio pre-training with acoustic tokenizers. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), Proceedings of the 40th International Conference Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 5178 5178-5193. PMLR, 23-29 Jul 2023. URL https: ftttps://proceese ngs. ml 1 www.spsssseeem.... w2 chen2 html
10

---

Published as a conference paper at ICLR 2024
Ting Chen, Simon Korblithh, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In ICML 2020: 37th International Conference on Machine Learning, volume 1, pp. 1597-1607, 2020.
Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15750-15758, 2021.
Yu-An Chung and James Glass. Generative pre-training for speech with autoregressive predictive coding. In 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 3497-3501, 2020.
Yu-An Chung, Wei-Ning Hsu, Hao Tang, and James R. Glass. An unsupervised autoregressive model for speech representation learning. In Interspeech 2019, pp. 146-150, 2019.
Yu-An Chung, Hao Tang, and James R. Glass. Vector-quantized autoregressive predictive coding, In Interspeech 2020, pp. 3760-3764, 2020.
Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D. Manning. What Does BERT Look at? An Analysis of BERT's Attention. In Proceedings of the 2019 ACL Workshop BlackboxNIPP: Analyzing and Interpreting Neural Networks for NI.,, pp. 276-286, Florence, Italy, August 2019, Association for Computational Linguistics. doi: 10. 10011865//vL/V 1/W19-4828.
URL https
//aalannhology..orgg..o /W19-//19---0 4828
Jacob Devlin, Ming-Wer Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volune (Long and Short Papers), pp. 4171-4186 Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10. 10.18663/// 10.1865//1//1911440 1423.
URL https
Xiaoyi Dong, Jianmin Bao, Dongdong Chen, Weiming Zhang, Nenghai Yu, Lu Yuan, Dong Chen, and Baining Guo. CSWin Transformer: A General Vision Transformer Backbone With Cross-Shaped Windows. In Proceedings the IEEE/CVF international conference on computer vision, pp. 12124- 12134, 2022.
thecvf. com
Dong__Ssi_ii Cswin Transformer A General Vision Transformer Backbone..bbbeeeee Wi Sithh_Crosse Whaposs
Windwww ____s_____________ 2wwpa___22_paarrr..._. html
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition scale. In International Conference on Learning Representations, 2021. URL C
feet/forum? wtw.ooormmuuicccooooooo id=YicbFdNTTT
Gasser Elbanna, Neil Scheidwasser- Mikolaj Kegler, Pierre Beckmann, Karl El Hajal, and Milos Cernak. Byol-s: Learning self-supervised speech representations by bootstrapping. In HEAR: Holistic Evaluation of Audio Representations, pp. 25-47, PMLR, 2022.
Jesse Engel, Cinjon Resnick, Adam Roberts, Sander Dieleman, Douglas Eck, Karen Simonyan, and Mohammad Norouzi. Neural audio synthesis of musical notes with wavenet autoencoders. 2017.
Haoqi Fan, Bo Xiong, Karttikeya Mangalam, Yanghao Li, Zhicheng Yan, Jitendra Malik, and Christoph Feichtenhofer. Multiscale vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 6824-6835, 2021.
Christoph Feichtenhofer, Yanghao Li, Kaiming He, et al. Masked autoencoders as spatiotemporal learners. Advances in neural information processing systems, 35:35946-35958, 2022.
Eduardo Fonseca, Xavier Favory, Jordi Pons, Frederic Font, and Xavier Serra. Fsd50k: an open dataset of human-labeled, sound events. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 30:829-855 2021.
11

---

Published as a conference paper at ICLR 2024
Jort F Gemmeke, Daniel PW Ellis, Dylan Freedman, Aren Jansen, Wade Lawrence, R Channing Moore, Manoj Plakal, and Marvin Ritter. Audio set: An ontology and audio events. In 2017 IEEE international conference acoustics, speech and signal processing (ICASSP), pp- 776-780. IEEE, 2017.
dataset for
Yuan Gong, Cheng-I Lai, Yu-An Chung, and James Glass. Ssast: Self-supervised audio spectrogram transformer. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp. 10699-10709, 2022.
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H. Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, Bilal Piot, Koray Kavukcuoglu, Rémi Munos, and Michal Valko. Bootstrap your own latent: new approach to self-supervised learning. In Advances in Neurul Information Processing Systems, volume 33. pp. 21271 21244, 2020.
Anmol Gulati, James Qin, Chung- -heng Chiu, Niki Parmar, Yu Zhang, Jiahui Yu, Wei Han, Shibo Wang, Zhengdong Zhang, Yonghui Wu, and Ruoming Pang. Conformer: Convolution-augmeene Transformer for Speech Recognition. In Interspeech 2020, pp. 5036-5040. ISCA, October 2020. doi: 10.21437//nerrseecchh2200. 200003331 URL. https:t/////.. //ww: www. - swwech.eeeoooooo org/ archivee/nnterspechhhuu
at 120 interspeech....................e htm
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16000-16009, 2022.
Zhenyu Hou, Xiao Liu, Yukuo Cen, Yuxiao Dong, Hongxia Yang, Chunjie Wang, and Jie Tang. Graphmae: Self-supervised masked graph autoencoders. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 594-604, 2022.
Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed. Hubert: Self-supervisee speech representation learning by masked prediction of hidden units. JEEE Transactions on Audio, Speech, and Language Processing, pp- 1-1, 2021a.
Wei-Ning Hsu, Anuroop Sriram, Alexei Baevski, Tatiana Likhomanenko, Qiantong Xu, Vineel Pratap, Jacob Kahn, Ann Lee, Ronan Collobert, Gabriel Synnaeve, and Michael Auli. Robust wav2vec 2.0: Analyzing domain shift in self-supervised pre-training In Interspeech 2021, 2021b.
Po-Yao Yao Huang, Hu Xu, Juncheng Li, Alexei Baevski, Michael Auli, Wojeiech Galuba, Florian Metze, and Christoph Feichtenhofer. Masked autoencoders that listen. Advances in Neural Information Processing Systems, 0--28708-27722 2022.
Bongjun Kim, Madhay Ghei, Bryan Pardo, and Zhiyao Duan. Vocal imitation set: a dataset of vocally imitated sound events using the audioset ontology. In DCASE, pp. 148-152, 2018.
Khaled Koutini, Jan Schlüter, Hamid Eghbal-zadeh, and Gerhard Widmer. Efficient Training of Audio Transformers with Patchout. In Proc. Interspeech 2022, pp. 2753-2757, 2022. doi: 10.2 09.11447/nterppech.2022-2
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 7871-7880, Online, July 2020. Association for Computational Linguistics. doi: 10. 111113/11/2220.aclllll mann...033..33 URL https: http:///laaaatholooooo
org aww.200..cmmmmmmmmm. 703
Yanghao Li, Chao- Yuan Wu, Haoqi Fan, Karttikeya Mangalam, Bo Xiong, Jitendra Malik, and Christoph Feichtenhofer. Mvitv2: Improved multiscale vision transformers for classification and detection. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 4804-4814, 2022.
Alexander H. Liu, Yu-An Chung, and James R. Glass. Non-autoregressive predictive coding for learning speech representations from local dependencies. In Interspeech 2021, 2021.
12

---

Published as a conference paper at ICLR 2024
Andy T Liu, Shu-wen Yang, Po-Han Chi, Po-chun Hsu, and Hung-yi yi Lee. Mockingjay: Unsupervised speech representation learning with deep bidirectional transformer encoders. ICASSP 2020- 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 141996423. IEEE, 2020.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 10012-10022, 2021.
Ari Morcos, Maithra Raghu, and Samy Bengio. Insights on representational similarity in neural networks with canonical correlation. In Advances in Neural Information Processing Systems. volume 31. Curran Associates, Inc., 2018. URL http:/tttsspppccee/eiitttt/e/eee neurips. cc/ Abst .tmll
p httpp//h//a3d/77cdcclla 737 7tt.a//7ccc611777711110011999999
Daisuke Niizumi, Daiki Takeuchi, Yasunori Ohishi, Noboru Harada, and Kunio Kashino. Byol for audio representation. In 202/ International Joint Conference on Neural Networks (IJCNN), pp. 1-8, 2021.
audio: supe-supe learning for
Daisuke Niizumi, Daiki Takeuchi, Yasunori Ohishi, Noboru Harada, and Kunio Kashino. Masked spectrogram modeling using masked autoencoders for learning ceneral-purpost audio representa- tion. In Joseph Turian, Bjöm W. Schuller, Dorien Herremans, Katrin Kirchoff, Paola Garcia Perera, and Philippe Esling (eds.), HEAR: Holistic Evaluation of Audio Representations (NeurIPS 2021 Competition). volume 166 of Proceedings of Machine Learning Research, pp. 1-24, PMLR, 13-14 22a html
Dec 2022. URL https: http::/ss///ccceeiieneee
mlr
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprini arXiv:/807.03745 2018
Yatian Pang. Wenxiao Wang, Francis EH Tay, Wei Liu, Yonghong Tian, and Li Yuan. Masked autoencoders for point cloud self-supervised learning. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part 11, pp. 604 621. Springer, 2022.
Ankita Pasad, Ju-Ch Ju-Chieh Chou, and Karen Livescu. Layer-Wise Analysis of a Self-Supervised Speech Representation Model. In 2021 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pp. 914-921, December 2021.
doi: 10. 10.1 10.100//SSUSS1003.222 202....02.2.....893 .9888993.
Karol J. Piczak. ESC: Dataset for Environmental Sound Classification. In Proceedings of the 23rd Annual ACM Conference on Multimedia, pp. 1015-1018, ACM Press, 2015. ISBN 948-3 3459 doi: 10. 10.114552733373.2866666
URL http:://at acm. wwwpg///ttttii..eoooeoo cfm?
doid=2733373 2022273333732280999
Maithra Raghu, Justin Gilmer, Jason Yosinski, and Jascha Sohl-Dickstein. SVCCA: Singu- lar Vector Canonical Correlation Analysis for Deep Learning Dynamics and Interpretabil- ity. In Advances in Neural Information Processing Systems, volume 30. Curran Asso- ciates, Inc., 2017. URL http://acccceeaasssssaaaa33eeee. 2
wwpppppe.ppppppp.ppppppp.eep files/paper/
Maithra Raghu, Thomas Unterthiner, Simon Korblith, Chiyuan Zhang, and Alexey Dosovitskiy. Do vision transformers see like convolutional neural networks? Advances in Neural Information Processing Systems, 34:12116-12122 2021.
Aaqib Saeed, David Grangier, and Neil Zeghidour. Contrastive learning of general-purpost audio rep- resentations. In 202/ IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp.
2021.
Achintya Kumar Sarkar, Zheng- Hua Tan, Hao Tang, Suwon Shon, and James Glass. Time-contrastive learning based deep bottleneck features for text-dependent speaker verification. IEEE/ACM Transactions Audio, Speech, and Language Processing,
2019. Steffen Schneider, Alexei Baevski, Ronan Collobert, and Michael Auli. wav2vee: Unsupervised Pre- vre-Trinnng for Speech Recognition. In Proc. Interspeech 2019, pp. 3465-3469 2019. doi: 10.21437/ (447//ntersseech.21191191
13

---

Published as a conference paper at ICLR 2024
Younggyo Seo, Danijar Hafner, Hao Liu, Fangchen Liu, Stephen James, Kimin Lee, and Pieter Abbeel. Masked world models for visual control. In Conference on Robot Learning, pp. 1332-1344. PMLR, 2023.
Fabian-Robert Stöter, Soumitro Chakrabarty, Bernd Edler, and Emanuël AP Habets. Classification vs. regression in supervised learning for single channel speaker count estimation. In 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 436 440. IEEE, 2018.
Tabian-Robert Stöter, Soumitro Chakrabarty, Emanuël Habets, and Bernd Edler. Libricount, a dataset for speaker count estimation, April 2018. URL https://doi. wtt.://...rr.......//....2./../.
1216072
Mi Tian, Ajay Srinivasamurthy, Mark Sandler, and Xavier Serra. A study of instrument-wise onset detection in beijing opera percussion ensembles. In 2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 2159-2163, 2014. doi: 10.1109/ICASSP. 2014.6853981.
Joseph Turian, Jordie Shier, Humair Raj Khan, Bhiksha Raj, Björn W. Schuller, Christian J. Stein- metz, Colin Malloy, George Tzanetakis, Gissel Velarde, Kirk McNally, Max Henry, Nicolas Pinto, Camille Noufi, Christian Clough, Dorien Herremans, Eduardo Fonseca, Jesse Engel, Justin Salamon, Philippe Esling, Pranay Manocha, Shinji Watanabe, Zeyu Jin, and Yonatan Bisk. HEAR: Holistic Evaluation of Audio Representations. In Proceedings the NeurIPS 2021 Competitions and Demonstrations Track, pp. 125-145. PMLR, July 2022. ISSN: 2640-3498.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural information processing systems, 30, 2017.
Pete Warden. Speech commands: A dataset for limited-vocabulary speech recognition, 2018
Chen Wei, Haoqi Fan, Saining Xie, Chao- Yuan Wu, Alan Yuille, and Christoph Feichtenhofer. Masked feature prediction for supe-supervised visual pre-training In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition, pp. 11668-11678, 2022.
Zhenda Xie, Zheng Zhang, Yue Cao. Yutong Lin, Jianmin Bao, Zhuliang Yao, Qi Dai, and Han Hu. Simmim: A simple framework for masked image modeling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 9653-9663, 2022.
Shu-wen Yang, Po-Han Chi, Yung-Sung Chuang, Cheng-I Jeff Lai, Kushal Lakhotia, Yist Y. Lin, Andy T. Liu, Jiatong Shi, Xuankai Chang, Guan-Ting Lin, Tzu-Hsien Huang, Wei-Cheng Tseng, Ko-tik Lee, Da-Rong Liu, Zili Huang. Shuyan Dong, Shang- Wen Li, Shinji Watanabe, AbdeIrah- man Mohamed, and Hung-yi yi Lee. SUPERB: Speech Processing Universal PERformance Bench- mark. In Proc. Interspeech 2021, pp. 1 1194-1198 2021a. doi:
Yao- Yuan Yang, Moto Hira, Zhaoheng Ni, Anjali Chourdia, Artyom Astafurov, Caroline Chen, Ching-Feng Yeh, Christian Puhrsch, David Pollack, Dmiuriy Genzel, Donny Greenberg, Edward Z. Yang, Jason Lian, Jay Mahadeokar, Jeff Hwang, Ji Chen, Peter Goldsborough, Prabhat Roy, Sean Narenthiran, Shinji Watanabe, Soumith Chintala, Vincent Quenneville Béllir, and Yangyang Shi. Torchaudio: Building blocks for audio and speech processing. arXiv preprint arXiv:2110.15018, 2021b.
Wentao Zhu and Mohamed Omar. Multiscale audio spectrogram transformer for efficient audio classification. In ICASSP 2023-2023 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pp. 1-5. IEEE, 2023.
14

---

Published as a conference paper at ICLR 2024
APPENDIX
A MULTI-WINDOW MULTI-HEAD ATTENTION
def LLnA:tention K, V, wia_A):
4_k S.shepe[-2:]
# partition inputs along patch dimension
into pot-overlapping windows
Q G.zeshape: 1, win_i, d_x)
R.reshape (-1, win_i, d_x)
V V.reshape 1-1, win_i, dLX)
# compute self stcencion
softmax (8.csonsposee) sgrt
# reshope resuics
rerehhee d_k1
return X
Figure 5: Pseudocode for WinAttention
B MORE ABOUT DOWNSTREAM TASKS
Table 3: Overview of tasks for downstream evaluation. All these tasks are part of the HEAR (Turian
al. 2022) benchmark.
Short Hand Name
Description
Size (ia Homrs Metric
BO
Reijing Ornm
Tian
2014
et al
Clessfying percussion instraments
Accunacy
CD
Crema D Can Cel EUN
Emotion Recognition
10
Accuracy
ESC--0
ESC
Environmenial Sound Classlication
2.77
Accuracy
LibriColtt
Speaker Mdnttiisctiio,, Simulated Cocktail
Accuncy
Mri-S
Mridangam Sorkk
(lamapp waan-ppp--mmiii anbih 2a 20
Stroke classification in pitched percussiun instruments
1.57
Accuracy
Mri-T
Mridangam Tonic Aninleee mimii el
Tunic classiicttion in pitched percussion instruments
1.57
Accuney
NS
NSynth Pitch Sa
Pitch Chassfication, reduced training subset
5.5
Accuracy
Speech Communds
Spotting.
Accuracy
E50K
FSDS0K Fooseca el (071)
Muliiabel, hage wale Autio Tagging
100
mAP
VL
VaxLingunT1/ top
Tunae el 2022 Kim et
2016
Spoken language idemiiácation
Accurac
The following reasoning behind excluding the other tasks from the HEAR benchmark suite:
Nsynth-Pitee 50hr and Speech Commands Full because we already use the smaller subsets.
2. Gunshot Triangulation: Gunshot is an event both AudioSet and FSD50K ontology, and is thus redundant.
3. GTZAN Music Speech: FSD50k already has music and speech labels, and the model performance correlation study the HEAR paper (Turian et al. 2022) shows high correlation with FSD50k.
4. GTZAN Genre: highly correlated results with FSD50K and ESC-50 (surprisingly) as per
Turian al. 2022)
5. Vocal Imitations: high correlation with LibriCount (Turian et al. 2022).
6. Bee Hive state Classification: large runtime costs, niche task.
7. MAESTRO 5hr and DCASE 2016 Task 2: significant complexity (storage, runtime, timestep based evaluation).
C EXPERIMENTAL DETAILS AND HYPERPARAMETERS
In this section, we provide additional experimental details. Apart from AudioSet, all other datasets are obtained directly from the HEAR where they are pre-processed to 16000 Hz and distributed in a standard format.
Similar to Huang et al. 2022) our effective learning rate (Iven) depends the base learning rate (Iroas)) and the batch size as follows: Iraat = Iroase batch sizs In early experiments, we did
256
https::/heaabbeechhaat.. ctm.
15

---

Published as a conference paper at ICLR 2024
not find strong augmentations pre-training time to improve downstream performance, hence augmentations are used. For more details, refer to Table 4 As previously mentioned, hear-eval-Ke -ival-ii] was used for downstream experiments, and along with the details provided here should allow for consistent, reproducible downstream experimentation.
Table 4: Pre-training (PT) and Downstream (FT) hyperparameters. ": For ViT-L, and ViT-H based models, smallest batch size that didn't 't give OOM used.
Configuration
AS-5k Pre-training
Downstream
Optimizer
AdamW
Adam
Optimizer momentum
0.9, 82 0.999
31
0.9, 82 0.95
Weight decay
0,05
N/A
Base learning rate
0.000015
0.0001
Learning rate schedule linear-warmup- cosine decay
fixed
Minimum learning rate
0.0001
Dropout
0.
0.25
Warm-up epochs
N/A
Epochs
100
500
Early Stopping
N/A
20
Batch size
1024"
1024
Accelcrators
8x TPU-v3 cores
Nvidia-A40
The code for feature extraction and running downstream experiments for our default configura- tions as well as the corresponding pre-trained weights can be found at httttt::/ggghhtt com/
SarthakYadav/mwnat http://wwaaaaaaa-fafaiaaal
jax-official
D AUXILIARY MODALITY TESTED: IMAGENET
Given the in-depth ablations and exploratory analysis, as well as resource constraints, couldn't tt dive deeper into full scale testing of an additional modality given. However, as a proof of concept, we pre-trained MAE and MW-MAES with ViT-Medium encoder on ImageNet for 100 epochs, followed by evaluating linear probe performance (training for 50 epochs) on the ImageNet validation set.
Table 5: Proof of concept ImageNet experiments.
Model
Params
Validation Accuracy
MAE Medium Encoder
38 M
29.0.00
MW-MAE Medium Encoder
38 M
29.8.00
E AUXILIARY TASK TESTED: SUPERVISED ASR
We have conducted some preliminary experiments to investigate the applicability of the proposed MW-MHA module beyond MAES. We evaluated Supervised Transformer based ASR models, trained using CTC objective function. We trained on LibriSpeech 100h subset and reported Word Error Rates (WER) on the test sets, using validated recipes from the SpeechBrain toolkit 3) The network architecture used consists of 2-layer Convolutional frontend, followed by 12-layer encoder and 4-layer decoder, each with 4 attention heads. We trained on mel-filterbanks with 80 mel bins.
Table 6: Preliminary experiments on LibriSpeech ASR.
Model
Params
Test-Clean (WER)
Test-Other (WER)
Transformer
8.7 M
7.56 6.73
17.90
MW-MHA Transformer
8.7 M
17.20
sttss///geechbaainn.gttt.bbbb
16

---

Published as a conference paper at ICLR 2024
For the MW-MHA Transformer, only replaced the MHA blocks in the encoder with MW-MHA blocks, and used attention window sizes 1/8, 1/4, 1/2 and global attention corresponding to the number of timeframes obtained from the largest audio segment in the corpus. MW-MHA transformer outperforms baseline transformer by a considerable margin on both test sets, and although we haven't t yet conducted full scale librispeech experiments, this shows potential avenues for further applications.
F ADDITIONAL EVALUATION ON A FRAME LEVEL TASK: MAESTRO-SH
We have compared our base MAE and MW-MAE configurations the MAESTRO-S5 task, which is a part of the HEAR protocol, to include proof that the proposed approach also works for frame level tasks. MAESTRO-51 is evaluated in a 5-fold cross validation setting, with Onset FMS evaluation metric. We conducted the experiments times and report 95% confidence intervals over the cross validation scores of individual runs.
Table 7: Downstream performance evaluation on the MAESTRO-D5 task.
Model
Onset FMS
MAE Base
46.47=0.05
MW-MAE Base
47.31=0.07
MW-MAE does demonstrate improved performance on MAESTRO-Sh, While this might not look like much, at the time of writing, MW-MAE outperforms every method on the HEAR leaderboard +1 for MAESTRO-5h. including large multi and cross-modal ensemble methods, this excellent performance for single method.
As previously mentioned in Appendix B we didn't evaluate all methods on MAESTRO to begin with because it poses serious computation (and space) complexity: it takes longer to evaluate downstream performance on just MAESTRO than it takes to evaluate all the other downstream tasks combined. This was unfeasible given the number of experiments we did in this paper, more so given the fact that we evaluate on FSD5OK, where local information is known to be of importance.
G PARAMETER COUNT, AVERAGES AND OVERALL SCORES
Table Models, number of parameters, plain average scores and overall scores of models omitted from duu space constraints.
Model
Params
Average
s(mr)
HEAR-Naavv
HEAR Naive
24.3+0.5 67.5+0.3
5.0s0.7
73.5+0.4
SSL
Wav2Vec2-Ber
48.7a0.1
43.120.2
Wav2Vec2 large
315.4 M
67.1#0.2
74.0=0.4
WavLM-large
315.4 M
60.1+0.1
64.0=0.2
HuBERT-base
94.4 M
66.340.1
72.520.2
ULBERT large
315.4 M
66.420.2
73.420.3
SSaST-buse
89 M
63.910.1
71.720.2
75.0w0.2
85.7.0.3
MAE based
AudioMAE
86.0
60.140.2
62.920.3
MAE-B-xx16-I
86.0 M
76.420.1
88. 18.120.2
MAE-B-5:5-41
75.6a0.1
MAE.L-416..1
302.4 M
77.5+0.1
90.0+0.2
Proposed
MW-MAE-B--K16-4
86.0 M
77.0+0.1
89 2+0.2
MW-MAAE-B-5L--4
77.840.1
90.620.1
MW MAE. Lx1166-4x16.8
302.4 M
79.1+0.1
92.640.2
17

---

Published as a conference paper at ICLR 2024
H DETAILED ABLATION RESULTS
Tables
9
10
12 present expanded results of our ablation experiments from Sec 4.4
Table Results from Patch size ablation experiments. ViT-B encoder was used for all experiments. denotes total number of patches, and h denotes the number of attention heads in each decoder transformer block.
Model
BO
CD
ESC-50
LC
Mri-S
Mri-T
NS
SC:Sh
F50K
VL
s(m)
Patch Size=18 16), ==125, l=4
MAE
94.920.8
70.2a0..
80.400.5 66.0a0.3
97.440.1
97.720.1
65 65.9
88.940.5
49.420.1
40.620.5
85 85.940.3
MW-MAE 95.9+0.5
72.3+0.2
81.2:0.3
68.4+0.3
97.340.1
97.8=0.1
67.420.8
90,0+0.3
50.8+0.1
41.9#0.5
88.0+0.2
Patch Size=(4x 16), m=250, h=
MAE
96.240.3
72.2++.2
80.9204
67.3+0.3
97,4x0.1
98.3=0.1
68 38.4
89.4+0.3
50.420.1
43.1
88.2
MW-MAE
96.010.5
73.120..
81.220.4
68.840.2 97.420.1
97.920.1
69.320.6
90.920.2
51.2a0.2
44.2a0.9
89.240.2
Putch Size=(8 2500 A=8
MAR
16.1
72.5+0.2 96.340.4
81.3+0.2
66.040.3
97.5+0.1
98. 1=0.0
58..7
89.540.4
50.2+0.1 I
42.3+0.5
87.740.2
MW-MAE
73.0+0..1
82.640.3
69.3+4.. 97,5+0.1
18.1
70.3+0.8
90.5+0.1
51.4+0.1
42.3+0.5
89.4H0.1
Patch Size=(4x88,,. 5=500, m5000
h=12
MAE 96.740.2
71.340.3
79.020.4 67.8403 97.740.0
98.5=0.0
68.720.4
89.040.4
49.840.2
39.2a0.7 87.240.1
MAE 95.640..7 74.1+0.2
81.9=0.3
70.140.3 97.6+0.1
98.2=0.1
72.0=0.7
91.240.3
51.6+0.1
44.0=0.8
90.340.2
Patch Size=(5xx), 5), n=640, h=16
MAE 96.0+0.4 70.9=0.2
80.9204
67.640.. 97.6x0.1
98.420.0 69.320.4
88.410.3 49.3+0.2
37.7=0.6 86.840.2
MW-MAE 46.620.4 73.820.4
82.0a0.3 70.120.4 97.540.1 98.320.1 72.920.5 91.7a0.2 51.320.1 44.2a0.6 90.620.1
Table 10: Effect of encoder size on performance. Patch size of 4 16 was used for all experiments.
Model
BO
CD
ESC-50
LC
Mri-S
Mr-T
NS-5h
SC-5h
F50K
VL
a ((m)
Encoder=ViI-I
MAE
95.640.5
63,24002
70.1
64.640.3
97.140.1
97.420.1
66.420.7
74.340.8
41.650.1
26.420.6
77.6.0.33
MW-MAR 93.3+1.0
64,4+0.2
71.9:0.5 65.540.3
97.140.1
97.640.1
18.4
77.040.6
43.4+0.1
28.6+1.1 79.0+0.3
VieMder=ViT-M
MAE
95.240.7
69.520.2 77.820.3
67.420.3
97.410.0
98.020.1
66.620,7
88.020.4
48. 110.1
38.3
85 310.2
MW-MAE 95.940.3
71.8+0.3 80.3a0.4 69.740.1 97.240.1 97.840.1
68.55
88.840.6
49.6.0.1
39.8
87 520.2.
Eneoder=VII-B
96.240.3
72.2+0.2
80.9204
67.3
97.420.1
98.320.1
62..
89.420.3
50.4s0.1
43.1a0.9
18.120.2
MW-MAE 96.0H0.5 73.1+0.3
81.2=0.4
68 68.820.2
97.4+0.1
97.950.1
69.3
90.9#0.2
51.240.2
44.2=0.9
89 89.220.2
Eneoder=ViTA
MAE 95.8#0.6
72.420.1 79.720.3
66.8+0.4
97.5+0.1
98 98.2+0.1
69.5=0.6
90.940.2
50.7+0.1
43.6=0.4
88 3:0.2
MW-MAE 95.740.5 75.5a0.2
82.520.5 70.120.3 97.420.0
98.1
70.7
93.2a0.1
53.340.1
51,920.8
92.320.2
Encoder=ViT-II
MAF 96.8+0.2 71.1+0.2 78.3=0.4 67.1 97.5+0.0
98.5+0.0
67.6=0.6 89.640.1 49.5+0.2
40.0=0.7 86.9.0.1
MW-MAE 96.8+0.2 74.8+0.1 81.620.4 69.5+0.4 97, 97.4 98.2+0.1 1 70.8=0.5 92.4+0.2 12.11 47.5+0.6 11.12
Table 11: Effect of decoder depth on downstream performance. ViT-B encoder, patch size of 4x 16 were used for each experiment.
Model
BO
CD
ESC-50
LC
Mri-S
Mri-T
NS-5h
SC-Sh
F50K
VL
+(1a)
depthen
MAE
96.420.2
69.820.3
78.920.3 67 42.33
97.440.1 1. 97 17.90011
66.420.8
88.540.2
49.420.2
39.0=1.1
86, 12.2
MW-MAE 96.620.5
72.420.2
79.0.0.4
68.720.3
97.540.1
98.0s0.1
68 8a0.5
90.240.3
50.640.1
39.120.8
87.820.2
depth=2 MAF
96.840.3
71,3+0.3
78.8=0.2 68.840.2 97.446 17.4
98.2+0.0 67.240.6
90.0+0.2 49.6+0.2
39.4=0.7
87.3+0.1
MW-MAE 96.0+0.7
73.120.2 79.4:0.3 69.240.3 97. 1
98.2+0.1
69.0=0.6 90.640.2 50.7+0.2
40.1=0.6
88 340.3
depthed
MAE
96.240.3
72,240.2
80.920.
67.340.3 97. 97.440.1
98.3+0.1
68.3
89.410.3
50.4
43.1
88 120.2
MW-
96.040.5
73.1
81.2=0.4 68.840.2 97.440.
97.9+0.1
69.340.6 90.940.2
51,2+0.2
44.2+0.9
89.240.2
8 MAE
96.3+0.3
71.7+0.3
81.620.4 67.420.3 97.420.0
98, 1
67.820.7
89.920.3
50.8 43.420.6
88 2:0.1
MW-MAE 96.240.5
73,2400. 82.2a0.4 69.740.3 97.34000 18.1 69.4 91.340.2
52.010.2
44.740.8
89.920.2
H.1 ADDITIONAL WINDOW AND ATTENTION HEAD SIZE ABLATIONS
Although the proposed window size strategy in Sec 3.2 is quite straightforward and covers all potential windows for given number of input patches, we have conducted some additional ablations comparing
18

---

Published as a conference paper at ICLR 2024
Table 12: Amount of pre-training dataset used v/s downstream performance.
Model
BO
CD
ESC-50
LC
Mri-S
Mri-T
NS 5h
SC-Sh
F50K
VL
s =(12)
10% of AS-5k
MAE
93.6t0.7
51,3600.2
49.5=0.3
48 48.4 97. 1+0.1 96.4#0.1
1007
70.4
29.7+0.2
17.3+0.5
63 3..40.2
MW-MAE 94.120.3 63.9t0.3
67.120.3
60.520.2
97.340.1
97.620.0
64.420.5
82.020.4
40.950.2
30.1=1.1
77.240.3
25% 96.2+0.6
MAE
57.5+0.3
64.9=0.4
56.940.3 97.4+0.1
97,5+0.1
65.0=0.6
79.3+0.4
39.2+0.1
24.2=0.7
73.620.2
MW-MAE
96.1 68.0+0.2
75.5=0.4
67.2+0.3 97.340.
98.0+0.1
65.9+0.4
86.5+0.2
46.4+0.1 1 35.7+0.6
83.8+0 83.8+0.2
50% of AS-5k
97.2+0.3
65.5+0.3
10.3
64.340.3
97.5+0.1
I
67.0=0.6
85.3+0.6
45.1+0.1
32,4+0.8
MW-MAE 95,9+0.5 70.9+0.2 79.1=0.3 69.1 97,4+0.1
98.1+0.1
68.4+0.7
88.5+0.2
1901
39.5+0.5
87 27.020.2
75% of AS-5k
MAE
95.340.5
70.2+0.2
79.0=0.3
67.420.2
97.4s0).
98. 1=0.1
67.420.6
88.840.3
49.2s0.1
39,5=0.7
86. 240.2
MW-MAF
96.040.5
72.6+0.3
80.5=0.4 69.540.3
97.4+0.1
97.9+0.1
68.340.4
89.9+0.2
50.5+0.1
41.7+0.8
100% of AS-5k
MAE 96.2#0.3
72.2+0.2
80.9204
67.320.3
97.420.1
98.320.1
68.3
89.420.3
50.420.1
43.120.9
88. 18.120.2
MW-MAE 96.020.5 73.120.3 81.2a0.4 68 820.2 97.4x0.1 97.920.1
69.320.6
90.940.2
51.2a0.2
44.2+0.9
89.2a0..
how performance treads with different window and attention head sizes in the decoder, as shown in Table 13 These experiments were done on the base configuration from the paper (ViT-B encoder,
ppp 250, decoder with 4 layers and 384 hidden dimensions).
Table 13: Ablation experiments on different window sizes and number of attention heads h. Irrespec- tive of the window size/number of heads used, MW-MAE outperforms MAE overall performance by considerable margin.
Madel
BO
ESC-50 50
LC
Mri.S
Mri-T
NS 55
SC-5h
F50K
VI
x x(m)
1=4, Windows for MI-MAEE 15,25,125,200
10.1 71,9+0.1 80.2=0.5
97.5+0.1
98.0s0.1
66.7=0.4
89.740.2
49.840.1
42.8=0.9
87.6a0.2
MW-MAE 95.9+0.5 73.5+0.2 81,5=0.4 69.6+0.2
97.4+0.
98.0+0.0
69.5+0.5
90.640.4
51,2+0.1
44.2+1.0 1.0
89 39.5+0.2
h=6, Windows for MLW-MAA.555.00255500222...
MAE
96.420.7 70.920.2 79.820.4
67.120.3
97.510.1
98.320.1
66.3-0.6
88.510.2 49.820.1
41.520.6
86.920,2
MW-
96.240.5 73.3+0.4 81.3=0.3 68.4a0.2 97.2+0.1
97.9s0.1
68.620.6 90.340.3
44.7=0.9
89.120.2
h=8, Windows for MW-MAE:
MAE
96.2+0.3 72.2+0.2 80.9=04 67 67.3.
4+0.
98.3+0.1
68
68.3
89.440.3 50.4+0.1
18.2
MW-MAE
96.020.5 73.120.3 81.2=0.4 68.8 820.2 97.420.1
97.920.1
69.3=0.6
90.940.2
51.2+0.2
44.2+0.9
89.240.2
1=12, Windows for MW-MAE:
MAE
96.640.4 70,7+0.4 79. 10055 67.5
97.5+0.1 98. 38.3+0.1
67.4=0.6 89.3+0.2 49.2+0.1
40.4+1.1
86.840.3
MW-MAE
96.3+0.6 73.6+0.3 81.0=0.4 68.9+0.2 97, 97.540.0 98.0+0.1 70.3=0.9 91.1+ 50.9+0.1
44.2+0.9
5+0.3
h=16, Windows for MW-MAE: (2.22.5..10.11.2225,... 2...2222...05001255.55522222
MAE 96.510.4 70.720.1 77.1=0.2 67.820.3 97.540.1
98. 98.420.0
68. 2:0.6
88.510.3
48.510.2
38.720.9
86.420.2
MAE 95.640.6 73.1 81.7=0.3 69.9a0.2 97.5.0.. 98.0=0.0 69. 8:0.7 91.640.2
51.2+0.1
44.5+0.8
89.740.2
19

---

Published as a conference paper at ICLR 2024
I LIMITATIONS
The direct limitations of our work are:
Pre-training data scale: As opposed to text corpus used NevlD et (2019) as well speech representations [aevski (Bevski et al. (2020); Hsu al. (2021a), AudioSet is several order magnitudes smaller. While MW-MAEs demonstrate good performance characteristics in low-data scenarios, analysis on larger scales of data would be beneficial. However, given the scope this work (general-pure purpose audio representation learning). we firmly believe AudioSet is the best fit, most of the larger audio corpora speech only.
2. Computational demands: transformer based models are computationally expensive to train, and despite their favourable generalization characteristics, MW-MAEs are no different. MW-MAEs and as well as previous works Niizumi et al. (2022); Huang et al. (2022) ) have showed the efficacy MAEs when pretrained with AudioSet, however, training on longer duration audio data is still a challenge.
3. Runtime Overhead: For our base configuration, the training time throughput of MW-MAF (4109 samples/ sec) 14% slower than an MAE (4536 samples/sec) on the same hardware. This is primarily due to kernel call overhead of the MW-MHA module since we have to make individual calls for each attention head in an MW-MAE, whereas optimized MHA implementations do in one kernel call. A better native implementation can alleviate this difference. However, since the decoder is discarded after pretraining, there is no difference in throughput when running downstream experimenns/infeenneefeeeteen since the encoders are identical.
20


Reviews:
[{"review_id": "4vwPnnauNr", "reviewer": "Reviewer_M3PT", "summary": "The authors propose the multi-window multi-head attention module to model the local-global interactions in deocder transformer for masked autoencoders.", "questions": "The overall engineering effort invested in this paper appears to be relatively limited, and there are certain limitations in the analysis of the multi-window multi-head attention. Could more diverse designs and comparisons be explored, specifically regarding local attention windows? For instance, is it possible to make these window sizes trainable? Alternatively, more engaging designs and comparisons could be conducted by examining the relationship between the number of attention heads, network hierarchies, and window sizes.", "limitations": "", "rating": 3, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The multi-window local-global attention is novel for masked autoencoder learning.", "weaknesses": "1.The conceptualization of this article is relatively straightforward. Building upon the previous global-local attention mechanism, previous studies did not incorporate a hierarchical design for the local window perspective. Hence, the author implemented a mechanism with varying window sizes for each attention head within the multi-head attention mechanism. This initial approach is commendable. I am intrigued to know if the author is the originator of this multi-window local-global attention concept for the first time. No pertinent references were found at the end of the introduction, maybe leaving open the possibility of existing similar ideas in the fields of computer vision or natural language processing.\n\n2.In terms of the experimental aspect, \"the default configuration employed by the authors yields np = 250, resulting in window sizes of [2, 5, 10, 25, 50, 125, 250, 250] for each MW-MHA module in all decoder blocks, encompassing a total of eight attention heads.\" Although the author clarifies that this hyperparameter design covers several possible local context levels and adopts a simplistic set of designs, further ablation experiments pertaining to this aspect would be valuable.\n\n3.Corresponding to the points highlighted by the author in Appendix G, the experiments conducted solely focused on the (AS-5k) dataset. However, as a self-supervised model, it is expected that the author would train on larger datasets, given the significance of datasets such as librilight, which have already amassed 60k hours of data. Furthermore, exploring the model's performance on larger datasets would provide insights into its upper limits.\n\n4.A meticulous examination of the author's ablation experiments on the parameters of MAE in the appendix reveals that, under different parameter configurations and tasks, MW-MAE does not exhibit significant improvements compared to MAE. For instance, [BO, Patch Size=(4×8), n=500, h=12, MAE 96.7±0.2, MW-MAE 95.6±0.7], [Mris-s all], [Mri-T all]. Similar observations can be made in the main experiments presented in Table 1. In certain tasks, a comprehensive enhancement is not evident. Consequently, I eagerly anticipate the author's elucidation regarding this aspect.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9hXC4PrSyf", "reviewer": "Reviewer_a4K8", "summary": "This paper proposed a novel Multi-Window Multi-Head Attention module and replaced the Multi-Head Attention module by it within Masked AutoEncoder. The authors proved this new model Multi-Window Mased AutoEncoder (MW-MAE) as a better audio learner than MAE by empirical results on 10 downstream audio tasks. The authors attributed the improvement on that MW-MAE encoder learn heads with broader local and global attention, and then utilized attention entropies and distances analysis to support this argument. The authors utilized CCA to demonstrate that Multi-Window Multi-Head Attention in the decoder can independently capture local and global information.", "questions": "1. I'm wondering whether the results of audioMAE and MW-MAE are comparable, what is the patch size and spectrogram size of audioMAE you are using? Have you tried re-training audioMAE with your current settings?\n2. According to paper [1], their number of audioMAE(local) on ESC-50 was 94.1 and your number of audioMAE on ESC-50 is 60.6, what is the difference?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The paper proposed a novel module MW-MHA that combines local and global attentions on attention heads level. It also utilized all non-unary factors to decide the number of heads, which is interesting.\n2. The paper is well structured and includes detailed ablation study results and inplementation settings.\n3. The paper adopted the proper analysis to support MW-MAE's strengths and adopted proper plot to elaborate analysis results.", "weaknesses": "1. Typos such as \"we use fixed sinusuidal positional embeddings\"\n2. lack of parameter explanations in 3.1\n3. Another work [1] proposed very similar ideas that introducing local attentions in the decoder can help improve the MAE learner. They tried local attentions only and hybrid attentions in which they applied global attentions on top layers. Although This paper compared MW-MAE with audioMAE and MW-MAE outperformed audioMAE by a lot, it is still unclear to me why MW-MAE is much better with such a similar design. I attached my questions in below.\n4. Introduction section and Related Works section overlaped by a lot.\n5. There should be another line \"enc only\" in table 2.\n\n\n[1] Huang et al, “Masked Autoencoders that Listen,” NeurIPS 2022", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "R3EHQXzWx7", "reviewer": "Reviewer_fDLB", "summary": "This paper introduces a Multi-Window Masked Autoencoder (MW-MAE) tailored to effectively capture both local and global time-frequency details in audio. The approach applies self-attention over non-overlapping windows of different sizes for each head within the multi-head attention of transformer blocks. The authors show that the proposed method outperforms the standard Masked Autoencoders (MAEs) in ten subsequent audio downstream tasks and exhibit improved scaling traits with an increasing number of patches or model parameters.", "questions": "* I have concerns regarding the limited applicability and contribution of this research, as mentioned in the above weaknesses.\n* Typo: (6p, Sec. 4.3.) ... with the largest \"MW-MAE-L-16x4-8l\" -> ...... with the largest \"MW-MAE-L-4x16-8l\"", "limitations": "", "rating": 6, "confidence": 2, "soundness": 2, "presentation": 3, "contribution": 1, "strengths": "* The paper is well-structured and easy to follow. \n* The authors have made their implementation publicly available, which not only validates their results but also contributes to the research field.\n* The proposed approach to enhancing multi-head attention is straightforward, yet it demonstrates a performance boost.\n* The authors provide both quantitative and qualitative evidence to show that their proposed method effectively models both global and local information of data.", "weaknesses": "Limited applicability and contribution. The proposed method involves applying self-attention over non-overlapping windows of varying sizes for each head in the multi-head attention. However, experimentally, the method is only applied within the confines of a self-supervised approach in the audio domain, using the masked autoencoder structure, and specifically for the decoder. Compared to the previous work, Audio-MAE, which addressed challenges in initially applying MAE to the audio domain, this study merely demonstrates the effects of improving a specific part of the structure.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NswhPVUZWQ", "reviewer": "Reviewer_4eUM", "summary": "This paper extends the masked auto-encoders by processing the input audio features with multiple windows of different sizes. The model is pre-trained  to reconstruct  the input features using MSE loss. Once the models are trained, the embeddings are passed through an MLP classifier to perform 10 different utterance-level speech tasks. As compared to the vanilla masked auto encoders, the proposed multi-window masked AE (MW-MAE) performs slightly better on those 10 tasks based on the overall score. If we look at each task separately, on some tasks the vanilla MAE performs better and in others the proposed MW-MAE performs better although the differences are usually small.", "questions": "1) Which ten tasks have been used? (We cannot understand this without looking at the Appendix). Please explicitly mention these in the main text. \n\n2) Evaluations are on utterance level tasks, so how can we make sure that the local information is also well-preserved at the frame level?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "The main strengths are \n1. Extensive experiments \n2. Experimental details are provided in the Appendix (reproducibility)\n\n\n- Originality and Significance:\nThis paper is an extension of the vanilla masked AE, and the proposal uses multi window approach to extract intermediate features at various scale (local to global). Multiscale approaches are not particularly new but this might be the first application of it to audio representation learning with masked AE. \n\n- Quality:\nEven though the proposed method is sound and extensive evaluations have been conducted for it, the performance gains due to using this model is limited as compared to the vanilla masked AE. Given other concerns about the run-time \n\n- Clarity:\nThe paper is mostly easy to follow. However, it could be better to mention the tasks used in evaluation in the main text rather than leaving it for the Appendix.", "weaknesses": "1. Marginal gains with the proposed approach: Overall MW-MAE is slightly better than the original MAE. At individual task levels, sometimes MAE is better and sometimes MW-MAE is better by a small margin.   \n\n2. The local vs. global feature aspect of the model is not well-supported. From the multi-window approach, we can see that there are multiple scales but since there is no frame-level task in the evaluation setup, we cannot judge whether the local features/embeddings are also useful in downstream tasks. All the tasks evaluated here are based on global features at utterance level.\n\n3. It could have been better to provide a quantitative run-time comparison of MAE and MW-MAE, rather than mentioning this shortcoming in the Appendix. \n\n4. [Minor comment] It could have been better if the 10 tasks were mentioned in the main text before Table 1 rather than leaving the list to the Appendix.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)