MERT: Acoustic Music Understanding Model with Large-Scale Self-supervised Training
Authors: Yizhi LI
Ruibin Yuan
Ge Zhang
Yinghao Ma
Xingran Chen
Hanzhi Yin
Chenghao Xiao
Chenghua Lin
Anton Ragni
Emmanouil Benetos
Norbert Gyenge
Roger Dannenberg
Ruibo Liu
Wenhu Chen
Gus Xia
Yemin Shi
Wenhao Huang
Zili Wang
Yike Guo
Jie Fu
Venue: ICLR 2024 poster (2024)
Keywords: self-supervised learning
music
audio
language model
Primary area: representation learning for computer vision, audio, language, and other modalities
TL;DR: We propose MERT, a self-supervised acoustic music understading model that generalises well on a wide range of tasks.

Abstract: Self-supervised learning (SSL) has recently emerged as a promising paradigm for training generalisable models on large-scale data in the fields of vision, text, and speech. 
Although SSL has been proven effective in speech and audio, its application to music audio has yet to be thoroughly explored. This is partially due to the distinctive challenges associated with modelling musical knowledge, particularly tonal and pitched characteristics of music.
To address this research gap, we propose an acoustic **M**usic und**ER**standing model with large-scale self-supervised **T**raining (**MERT**), which incorporates teacher models to provide pseudo labels in the masked language modelling (MLM) style acoustic pre-training.
In our exploration, we identified an effective combination of teacher models, which outperforms conventional speech and audio approaches in terms of performance. 
This combination includes an acoustic teacher based on Residual Vector Quantization - Variational AutoEncoder (RVQ-VAE) and a musical teacher based on the Constant-Q Transform (CQT). 
Furthermore, we explore a wide range of settings to overcome the instability in acoustic language model pre-training, which allows our designed paradigm to scale from 95M to 330M parameters.
Experimental results indicate that our model can generalise and perform well on 14 music understanding tasks and attain state-of-the-art (SOTA) overall scores.

Published as a conference paper at ICLR 2024
MERT: ACOUSTIC MUSIC UNDERSTANDING MODEL WITH LARGE-SCALE SELF-SUPERVISED TRAINING
Yizhi Li 121 Ruibin Yuan 3.
Zhang 9-456- Yinghao Ma 77+ Xingran Chen
Hanzhi Yin
Chenghao Xiso
Chenghun Lin 122 Anton Ragai
Emmanouil Benetos"
Norbert Gyenge*
Roger Dannenherg'
Ruibo Liu"
Wenhu Chen 975
m1l1 Yemin 9,,12
Huang" T6
Wang"
Guo"
Jie Fu 97461
2 paa-ppai 'University of Manchester UUiierriity of Sheffield "Careegi Mellon University
'Hong Kong University Seience and Technology University of Waterloo "Beijing Acadomy of Artificial Intelligence
"Quenn Mary University of London DDurham University "Dartmouth College "MBZUAI "New York University Plinksoul,ai
ABSTRACT
Self-supervised learning (SSL) has recently emerged as promising paradigm for training generalisable models on large-scale data in the fields vision, text, and speech. Although SSL has been proven effective in speech and audio, its application to music audio has yet to be thoroughly explored. This is partially due to the distinctive challenges associated with modelling musical knowledge, particu- larly tonal and pitched characteristics of music. To address this research gap, we propose acoustic Music untaRRst model with large-scale self-supervised Training (MERT), which incorporates teacher models to provide pseudo labels in the masked language modelling (MLM) style acoustic pre-training. In our exploration, we identified an effective combination of teacher models, which out- performs conventional speech and audio approaches in terms of performance. This combination includes an acoustic teacher based on Residual Vector Quantisation Variational AutoEncoder (RVQ-VA VAE) and musical teacher based on the Constant- Q Transform (CQT). Furthermore, we explore wide range of settings overcome the instability acoustic language model pre-training, which allows our designed paradigm scale from 95M to 330M parameters. Experimental results indicate that our model can generalise and perform well on 14 music understanding tasks and attain state-of-the-arr (SOTA) overall scores.
INTRODUCTION
Pre-trained language models (PLMs) can learn generalisable representations of data without human annotated labels in self-supervised learning (SSL) style, leading to remarkable performance im- provement in natural language processing and related fields (Brown et al., 2020; Fang et al., 2022; Chen et al., 2021a). Music widely recognised as a special language that can be used communicate across different cultures (Mehr et al., 2019). The internal similarity between music and language as a communication interface lays promising foundation for adapting PLM-based methods to model music sequences. We argue that the benefit is twofold. First, PLMS can potentially pave the way to unify the modelling of a wide range of music understanding, or the so-called Music Information Retrieval (MIR) tasks, including but not limited music tagging. beat tracking, music transcription, source separation, etc., so that different tasks no longer need task-specific models or features. Second, releasing PLM for acoustic music understanding allows the redistribution of the musical knowledge rather than the data itself, which saves the costs of manual annotation and copyright law restrictions.
a general-purpose and cost-effective open-source PLM acoustic music understanding. Most existing studies are designed to solely address music tagging prob- lems (Pons and Serra, 2019: Spijkervet and Burgoyne, 2021: McCallum et al., 2022; Huang et al.. 2022; Zhu et al., 2021; Zhao and Guo, 2021), and many of them do not provide open-source code bases or checkpoints for further evaluation. A promising model is JukeMIR (Castellon et al., 2021), which is based on Jukebox (Dhariwal et al., 2020) and provides a comprehensive evaluation on MIR
Unfortunately, we yet
Thee authors contributed equally to this work.
Corresponding authors.

---

Published as a conference paper at ICLR 2024
Acoustic Teacher
PQTic
Musical Teacher
MLM
MLMM
/'
Audio Waveform 2
Contextual Representation
d5
Pg
Ri
A4
Transformer
Encoder
1D Convolution Feature Extractor
Masked Audio Features
Figure 1: Illustration of the MERT Pre-training Framework.
downstream tasks. However, this foundation model uses cumbersome hierarchical auto-regressive transformer decoders containing billions of parameters to model music audio, leading to significant inefficiency for conducting general music understanding tasks (e.g., it takes weeks to inference datasets like MTG (Bogdanov et al., 2019) with consumer-grade 3090 GPU).
The aforementioned research gap has urged computationally affordable pre-trained acoustic music model. In this paper, we propose an acoustic Music untandinnging model with large-scale self-supervised Training (MERT). MERT inherits speech SSL paradigm, employing teacher models to generate pseudo targets for sequential audio clips. Specifically, to capture the distinctive pitched and tonal characteristics in music, MERT incorporates multi-task paradigm to balance the acoustic and musical representation learning as demonstrated in Fig. 1. In the proposed design, an Residual Vector Quantisation Variational Autoencoder (RVQ- VAE) (Défossez et al., 2022) is used as the acoustic teacher to provide discretised acoustic-level summarisation of the music signal. The Constant-Q Transformation (CQT) (Brown, 1991) model further introduced as the music teacher for capturing pitch and harmonic inductive bias. Regarding the context dependencies and music hierarchies, as indicated in Borsos et al. (2022), we leave the task of modelling high-level and abstract patterns to the stacked layers of self-attentions in the transformer.
design and open-source generalisable and
We also explore a wide range settings for the transformer and ID convolution encoder to overcome the instability in acoustic model pre-training, which permits effective scaling up of MERT from 95M to 330M size when blending acoustic and musical knowledge. By scaling up to 330M size (only 7% the size of JukeBox), MERT achieves overall state-of-the-ard (SOTA) results various MIR tasks, which demonstrates a strong generalisability on music understanding. Last but not least, we analyse multiple pre-trained settings considering the teachers and share our decision routes 5.2 and 5.3, which could potentially guide future acoustic music understanding pre-training research.
To summarise, our contributions
proposing a multi-tast tasti style predictive acoustic self-supervised learning paradigm, which achieves SOTA performance on various MIR tasks, including important yet unexplored tasks for pre-training such as pitch detection, beat tracking and source separation applications; conducting broad range of analysis based on ablation study of the proposed MERT pre-training paradigm;
exploring robust and stable strategies for acoustic music model training to overcome training instability and frequent crashes when scaling up the pre-training model size;
providing an open-source, generalisable and computationally affordable acoustic music pre- trained model, which addresses the needs of both industry and research communities.
2 RELATED WORK
PLMs for Acoustic Music The field of music information retrieval (MIR) has long been facing challenges in data availability due the costs associated with music audio annotation and country- specific copyright laws (Chen et al., 2019; Castellon et al., 2021). To address this challenge, pre- trained language models (PLMs) for acoustic music have been proposed to provide reusable learned
2

---

Published as a conference paper at ICLR 2024
representations, enabling transfer learning for various downstream MIR tasks without the need for extensive data annotation (Castellon et al., 2021). However, current acoustic music pre-trained models still have room for improvement in terms of providing learned representations suitable for both industrial and research applications (McCallum et al., 2022). Existing acoustic music pre-trained models primarily focus on tagging tasks and rely on supervised tagging labels for pre-training (Pons and Serra, 2019; Spijkervet and Burgoyne, 2021; McCallum al., 2022; Huang et al., 2022). While some studies have explored contrastive learning for acoustic music pre-training, they face limitations in training data and model size, hampering the performance improvements (Choi et al., 2017; Li al., 2022). Additionally, several models trained on inaccessible datasets or without publicly available codes and model weights make it difficult to reproduce or extend their approaches (McCallum et al., 2022; Castellon al., 2021; et al., 2022; Zhu et al., 2021; Zhao and Guo, 2021). Although some general general-purpose purpose audio representation models show potential for music audio representation learning, their performance is mostly evaluated on limited MIR downstream tasks (Saeed et al., 2021; Borsos et al., 2022; Wang et al., 2023). This lack of comprehensive evaluation hinders further studies and a thorough understanding of the pros and cons of existing models.
generalisable, and lightweight
Self-Supervised Speech Processing Music and speech processing are closely related (Jasmin et al., 2020) since they usually use the same audio data formats. Additionally, both acoustic music and speech processing models need deal with the cocktail party problem (Brown and Bidelman, 2022; Petermann et al., 2022) since good source separation capabilities help both separating noises and background sounds with speech and processing polyphonic music audio. These common grounds between music and speech processing inspire us to adapt SOTA speech pre-trained models and tailor them specifically for music audio processing tasks. For instance, existing research work targeting general-pune purpose audio representations (Saeed et al., 2021; Borsos al., 2022: Wang et al., 2023) has verified that suf--vueeeeeed speech processing models can be extended beyond speech to downstream entry-level music tasks, including generating mono piano music and music reconstruction.
Audio Representation with Language Modelling Mask strategy-based large-scale language models have been applied to wide range of domains (Lample and Charton, 2019; Chen et al., 2021a;b; Fang al., 2022), but still remain under-explored in acoustic music understanding. For audio, Dhariwal et al. (2020) investigates generating hierarchical tokens which can be further employed reconstruct music, inspiring subsequent research to understand and generate acoustic music based on extracted discrete tokens from continuous features. Baevski and Mohamed (2020) introduce a pre-trained VA-V (Baevski et al., 2019) to provide prediction targets to conduct speech representation learning with MLM. While introducing K- K-means to provide discrete token codebooks and pre-training the model to detect sound units, Hsu et al. (2021) claim that a better teacher model in SSL could lead to better downstream task performance. Additionally, recent speech processing pre-trained models (Borsos et al., 2022; Wang et al., 2023) propose to train or adopt separately trained codecs (Zeghidour et al., 2021; Défossez al., 2022) for discrete token extraction. Based on the conclusion from previous studies, the recently released RVQ-VAES (Zeghidour et al., 2021; Défossez al., 2022), achieving good results in music reconstruction, could be adopted as teacher models for music understanding pre- pre-training and provide acoustic information guidance. Yet some of the uniqueness of music processing such as timbre and harmony remains unexplored. We thus propose to incorporate corresponding musical teacher model in MERT to fill such an important gap.
METHODOLOGG
This section introduces the pre-training paradigm and architecture of our models. It includes prediction acoustic teachers such as k-means deep music features, and reconstruction to music teachers such as CQT spectrum, both based on the well-established masked language modelling (MLM).
3.1 PRE-TRAINING WITH MILM
(t) 1N Supervised Learning requires labelled dataset D, (x 3i Ji=1 Here, N is the number of data samples, x²(') is the 4th data sample in the dataset, and y, is the corresponding label. From D,, we can train machine learning algorithm so (-) parameterised with 0 that makes label predictions
3

---

Published as a conference paper at ICLR 2024
each data sample. Unsupervised learning, in contrast, learns algorithm based unlabelled dataset D = {2, JM with SSL being a specific type of this class. For each data sample T;, SSL derives data z4 with pseudo label v/. The training process is minimise the loss between each pseudo lahel yf and the prediction based on new data y/s fo(2)) as denoted in Eq.
0 arg
Σ L ). ED
MLM is a famous example of pseudo-label generation. Let x; [,,"),;²²
(1) be the ith data sample in sequential dataset with length L, and M C [L] is a subset of indices randomly chosen from 1 to Then, the new data is defined by the following equation
(1) (2)
,
/ - (1)) xx"" (
(2)
where 1 j (ar) denotes the indicator function, that is, A (x) if and only if æ is outside the masked indices set The pseudo-labe that needs to be learned is typically y I; zi. ie., the masked data. However, reconstructing masked data y/ for raw audio tasks as pseudo-label hard to train. HuBERT (Vaswani et al., 2017; Hsu et al., 2021) uses dimension-reducee feature 2' derived from y' with phonetic acoustic information, which forms the design basis our pre-training strategy.
As speech SSL system, HuBERT utilises offline clustering acquire pseudo labels for BERT-like prediction loss. Specifically, it uses Mel-frequency cepstral coefficients (MFCCs), a widely-used traditional feature in speech-related tasks, acoustic features for clustering. The obtained results are then utilised as pseudo labels in the first iteration of pre-training. It then uses the learned representation for clustering to get pseudo label for the second iteration pre-training. Such a pseudo label includes acoustic information in human speech and can be aligned to phonemes. The loss functions of HuBERT are formulated follows:
LH Eu(f:an M, Z)
Σ 1o |x',t) TEM
where log P- x/,t) is the log- likellkoooood function on clustering results given the masked input x' and position derived from f; likelihood function Pf is the Noise Contrastive Estimation (NCE) loss which is defined
exp(sim((((o),.c))/))
pf(c| (2,t)
Ee= exp((si((T(o))))))))
Here, E [C] is codeword of the clustering results and represents its embedding; sim is the cosine similarity; o; is the output of the model at timestep t; and (or) is the linear transformation of or, making it have the same dimension e.. Besides, 7 scales the logit which is set 0.1 in HuBERT. The linear transformation T, the model to generate outputs, and the embedding of all the clustering results are all learnable.
Overall, we use the same model as HuBERT but introduce several notable variations tailored to music. Specifically, we designed better hidden-unit as pseudo tags for pre-training with multiple music acoustic features. In addition, we added a reconstruction loss to music features and employed additional music augmentation tricks.
3.2 MODELLING ACOUSTIC INFORMATION
The MFCC features are only good at modelling acoustic and timbre information for single-pitel signals, and therefore, the clustering results do not provide much timbre information in music recording. We proposed two potential approaches the teacher on acoustic information: one based on traditional features, and the other based on deep learning.
The first method uses k-means on the log-Mel spectrum and Chroma features for timbre and harmonic acoustic information, respectively. In the case of music representation, each frame contains more information compared to speech, necessitating larger number of classes for k-means clustering. The complexity of the k-means algorithm linear with the number of centroids (clustering centres), leading time-consuming k-means for the music feature. To tackle this problem, we employ
4

---

Published as a conference paper at ICLR 2024
300-means for the log-Mel spectrum with dimension 229, and 200-meams for Chroma features with dimension 264, resulting in a total of 60,000 classes (200 centroids for Chroma features multiplied by 300 centroids for the log-Mel spectrum). Despite the increased number of classes, the computational complexity remains comparable to that of HuBERT. The disadvantage of k-means that it is diffieult to scale up larger number of classes and larger datasets, and the results are sensitive to initialisation.
The second choice for our acoustic teacher is EnCodec (Défossez et al.. 2022), a recent learnable Variational AutoEncoder (VQ-VAE) Each acoustic
feature with 8-layer residual Vector Quantized-Variationall feature, denoted as Zenc (C1,, a 2-dimensional auditory code matrix, and is the length of the recording. The row vector of each matrix Zene[t, represents the results of 8 different clusterings for frame 1, and the column vector of each matrix Zena(:,]] represents the results from the jth codebook of the audio sequence, where j (1.....8) 8). EnCodec converts 24kHz input waveforms to 8 different embeddings at 75Hz with 320-fold reduction, and the quantizer has 1024 dimensions. In this setting, for each 5-second waveform, the discrete acoustic feature is matrix with 375 x 8 entries, representing 375 frames (75Hz x 5s) and deep acoustic features. With these embeddings, the decoder of EnCodec reconstruct the waveform 24 kHz with authentic information in timbre.
3.3 MODELLING MUSICAL INFORMATION
Apart from acoustic information, we added new reconstruction loss to the Constant-Q transform (CQT) spectrogram to emphasise pitch-level information. The CQT is type of frequency transform that widely used in various MIR tasks, such as pitch detection, chord recognition, and music transcription. It is similar to the Fourier transform, but bin widths proportional to frequency rather than equal, giving each octave the same number of bins, resulting in a better fame-frequency trade-off' for music audio where multiple pitches occur in multiple octaves. We utilize mean squared error (MSE) loss to reconstruct the CQT spectrum Zegt from the masked input audio x'. That is,
LCQT (Feqt x, M, Zeqe)
-
|R - feqe (z')el2
(5)
MEM
And the final loss function C is linear combination of both the acoustic loss function Lu and the musical-pine pitch loss function LeQr:
= a -CH + Loot
(6)
EXPERIMENTS
4.1 EVALUATION PROTOCOL
Downstream Tasks We evaluate our method and compare it with baseline models on 14 downstream tasks, including frame-level classification regression tasks like music tagging, key detection, genre classification, emotion score regression, instrument classification, pitch classification, vocal technique detection, and singer identification; and sequential tasks like beat tracking and source separation. For instrument classification, we use the Nsynth (Engel et al., 2017) and MTG-instrument datasets, with receiver operating characteristic (ROC), and average precision (AP). The NSynth dataset is also used for pitch classification, with accuracy (ACC) as the evaluation metric. Vocal technique detection and singer identification based on the VocalSet dataset (Wilkins al., 2018), with accuracy as the metric. For music tagging, we utilise the 2009) and MTG-Jamendo (Bogdanov al., 2019) datasets, averaging multiple embeddings for long audio recordings. Key detection is accomplished using the Giantsteps and Giantsteps -MMtstteps-G--keyee. datasets (Knees al., 2015; Korzeniowski and Widmer, 2017), with a refined accuracy (ACCrefined) metric. Genre classification is performed using the GTZAN (Tzanetakis and Cook, 2002) and MTG- Genre datasets, with ROC, and AP metrics. Emotion score regression conducted on the Emomusic dataset (Soleymani et al., 2013), with the coefficient of determination (R2 score) of arousal and valence as evaluation metrics. Beat tracking is conducted on the GTZAN Rhythm dataset (Marchand and Peeters, 2015), using the F-measure (F1). Finally, source separation is accomplished using the MUSDB 18 dataset (Rafii et al., 2017), with the Source-to- Source-to-Distortion Ratio (SDR) as the evaluation metric. The full descriptions of the datasets and tasks can be found in Appendix B.
(MTT) (Law et al.,
5

---

Published as a conference paper at ICLR 2024
Probing Protocol Following Castellon et al. (2021); Yang et al. (2021), restrict the testing protocol with probing rather than finettniig, i... freezing the backbone pre-trained models as deep feature extractor and only train simple downstream structure, typically a multilayer perceptron (MIP) for frame-level tasks. For a fair comparison, we also limit the space for hyper-parameters searching. For full details please refer to Appendix B.2.
4.2 BASELINE METHODS
We select models pre-trained with various paradigms from both music and speech domains as our baselines to provide comprehensive evaluation of the generalisation ability of the designs. Mu- siCNN (Pons and Serra, 2019) is selected as representative supervised method, which is pre-trained with supervision from the Million Song Dataset tags (Bertin-Mahieul et al., 201 CLMR (Spijkervet and Burgoyne, 2021) and MULE (McCallum et al., 2022) are selected representatives of SOTA music representations trained with contrastive learning. Jukebox (Dhariwal et al., 2020) and the corre- sponding transfer learning method, JukeMIR (Castellon al., 2021) is selected as the representative of transfer learning from a large-scale generative pre-trained musical representation. We also select the recently proposed strong speech SSL models, HuBERT (Hsu et al., 2021) and data2vec (Baevski et al., 2022), as our baselines since they share the same MLM pre-training paradigm with MERT. While HuBERT reconstructs the masked discrete tokens provided by the K-means teacher, data2vec uses the student model updated with an exponential moving average gradient to produce continuous representations for MLM prediction. In order to reveal the effectiveness of the pre-training paradigm itself rather than the training data distribution, we re-train the speech models and denote them by HuBERTausi and mata2ve Additionally, we present the current SOTA for each task including results from both supervised and self supervisd methods.
4.3 IMPLEMENTATION DETAILS
Training Settings We deploy the proposed SSL architecture in the training of various model sizes with matched scales of data. We mined 160K hours of music recordings from the Internet to build a large-scale music dataset. Accordingly, the base size models (95M) are trained with a IK hours subset whereas the whole dataset used for the large model (330M). Specifically, we provide a special edition of the base model, SMEER-99MMPPP 1 ic, that is trained on a totally publicly available music dataset, music4all (Santana et al., 2020), with a data size of 910 hours. the context of self-attention, the computational complexity scales quadratically with the sequence length. Therefore, when dealing with limited computational resources, there exists a trade-off between the batch size and the sequence length. In our preliminary experiments, we have observed that increasing the batch size provides greater performance improvements compared to extending the context length. To allow larger batch size under the computational limitation, we adopt strategy of randomly truncating audio clips into 5-second segments following Ma et al. (2023). This duration roughly corresponds to a 2-bar context in music. It is worth noting that our model utilises a convolutional relative positional embedding, similar to the approach introduced by Bacvski et al. (2020) in Wav2Vec, enabling it to operate effectively in longer contexts, if required. The effective batch sizes and learning rates for the base model and large model are set to 1.5 and 5.5 hours, and their learning rates are set to 5e-4, 1.5c-3, respectively. Pre-training is carried out with the fairseq' framework. Models are trained with 64 A100-40GB GPUs with fp16. We also implement a data augmentation of randomly adding short segments to improve the representation robustness, and describe the details in Appendix A.
Training Stability In our empirical findings, we observe that when scaling up acoustic encoder-only models, they tend to exhibit a higher susceptibility to training instability compared to models of similar size in text or image domains. Such instability can result in decreased performance or, in extreme cases, even lead to crashes in model training. During experimentation with scaling up to the MERT-330M model, we encounter notable instability manifested by constant gradient clipping and sporadic spikes in losses. This instability has a detrimental effect on the accuracy of ML. predictions and results in decreased performance on downstream tasks. Our attempts to resume training from previously saved checkpoints and data batches are proved unsuccessful in mitigating the instability issue. Furthermore, we observe that reducing the learning rate in this context not only fails to address the issue but also leads to a decline in performance and hindered the training convergence. We

---

Published as a conference paper at ICLR 2024
further explore the effectiveness of a seemingly-powerte method DeepNorm (Wang al., 2022a) in stabilising acoustic language model pre- pre-training, but find it to be ineffective. Eventually, we discover that incorporating attention relaxation techniques (Chen et al., 2021b) is beneficial in addressing the instability challenges. We also found that transitioning from post-layer normalisation (Post-LN) to pre-layer normalisation (Pre-LN) offers potential solution of allowing training to continue. More information can be found in Appendix B.3.
Table I: Experimental Performances of MERT and Baselines Downstream Tasks (1/2). The baselines are grouped by supervised and unsupervised pre-training paradigms. The superscripts denote the category of the acoustic teacher used by MERT models. "public" refers to the MERT model trained with only open-source dataset. Results with star* are claimed in the references.
MIT Tagging
GTZAN Gloee
GTZAN Rhythm
EMO Fandion
Naynth
Vocallet Toch
Key
Instrment
Pach
Singer
Metrics
ROC
AP
Acc:
FINe
R2"
R2*
Asc
Auc
Auc
MusiCNN [41]
90.6
38.3*
12.5
79.1*
46.6*
70.34
72.6
64.1
70.3
57.0
CLMR
89
56,1
14.9*
68.6"
45.8*
67.8
67.9
47.0
58.
49.9
Juisbon 5D [15:57]
91.5*
41.4
06.7*
79.7*
61.7"
72.1"
70.4
91.6
MULE
91.4"
40.6*
66.79
73.5*
51.5
70.0"
74.0"
59.2
87.5
HUBEKTTNued [25]
37.7
14.7
70.01
42.1
77.4
dataZvec base [2]
90.0
36.2
30.6
74.1
68.2
52.1
71.0
69.4
93.1
71.1
81.4
MERT-15M
38.4
65.0
78.6
$2.0
71.3
92.3
74.6
MERT-95M
38.4
67.3
72.8
59.7
72.5
70.4
92.3
75.6
MERT--SMM
91.0
30.3
63.5
78.6
68.3
60.0
76.4
70.7
92.6
14.2
83.7
MERT-ERIN Wog
91.3
40.2
65.6
79.3
67.9
61.2
74.7
72.6
94.4
76.9
87.1
(Previous)
(154
[403
(36)
80.6 [24)
61.7
[1.1
(53)
[962
65.6
80.3 (39]
Table 2: Experimental Performances of MERT and Baselines on Downstream Tasks (2/2). Average scores across task are calculated on the SOTA results and models applicable all the tasks.
MTG
MTG MondTTemme
Insirument
Gaure
Toy30
Sourre Seperaionn
Asg.
Metrics
ROC
AP
ROC
AP
BOC
AP
ROC
AP
SDR
SDR
SDR
MusiCNN (41)
74,0
17.2
74.0
12.6
95.0
17.5
82.0
27.5
CLMR [48]
73.5
17.0
73.5
12.6
$4.5
16.2
81.3
26.4
SB [15:57]
5,1*
4.9*
4.14
2.74
MULE [36]
76.6
19.2
75.0
15.4
18.0
20.4
83.7
30.6
HUBERT
76.0
13.9
$6.5
18.0
3,7
55,8
Aate2vas-hrad [2]
76.7
14.3
87.1
18.8
5.5
MERT 953
77.2
19.6
75.9
13.7
$7.0
13.6
82.8
29.4
5.5
5,6
4.0
3.0
62.9
MIKEF99MM puuili- feams
16.2
1333
67.2
18.8
55
63.0
76.4
13.4
$7.1
19.8
5.5
MIKT S30M tro
78.1
19.3
76.5
16.0
86.7
18.6
83.4
29.9
5.3
5.5
3.6
3.0
64.7
(Presions) SOTA
78.8
20.2 11]
75.6
16.1 (76]
87.7
20.3 [1]
84.3
52.1 [76]
9.3
10,8
10.4
(44)
64.5
RESULTS ANALYSIS
5.1 PERFORMANCE EFFICIENCY OF MERT MODELS
The results all the downstream tasks provided in Tab. 1 and Tab. 2. As suggested by the average scores in Tab. 2, MERT- 3.3T-33300 RM vAE achieves the same score as the combination of the previous SOTAs (from 10 different models even including supervised methods) and becomes the new SOTA on 4 metrics. It is also noteworthy that the other smaller MERT- 95Ms still have comparable performance. Generally, MERTS perform well on tasks focusing on local-level musical information such beat, pitch and local timbre such as singer information, and remain competitive on the other tasks requiring more global-level information, such as music tagging, key detection, and genre classification. This indicates the blending of acoustic and musical teachers could provide comprehensive guidance for the understanding of music recordings, though pre-trained in only a 5-second context length. Nevertheless, the performances of our models in tasks with more global music information are close to strong baselines, suggesting MERT models are capable of recognising global patterns well, thanks to the relative position embeddings and the contextualisation of the transformers.
In addition, our models demonstrate good results with limited data, even when training with public and MERT-95K are both trained
data that may lack enough diversity. MERT-95M-publ Ik hour dataset and give competitive performance compared with the SOTA and MERT-330M proving that MERT converge effectively and learns generalisable music representations with
7

---

Published as a conference paper at ICLR 2024
limited training data. Moreover, the MERT-95M-pubi is trained with Music4ALL (Santana et al., 2020), a 910-hours public music dataset with mainly pop music and lack of diversity in music style, and shows comparable performance to other settings. In particular, its performance does not have a significant difference besides genre classification on GTZAN compared to MERT- 95M.
We evaluate the performance of the MERTR VAB model with parameter size of 95M and 330M, given the use of the EnCodec feature enables us to scale up the dataset compared to the K-means. The results demonstrate that increasing the model size 330M yields improved performance or maintains similar performance compared to MERT- 95M VAE (less than 0.1 %) on most of the tasks besides beat tracking. More importantly, the lightweight sizes of MERTs open up new possibilities for transferring one general understanding model for large-scale classification or sequence labelling MIR tasks. MERT series models achieve better or comparable performance with only 1.9% (95M) and 6.6% (330M) parameters compared to the self-supervised baseline Jukebox-5B (Dhariwal et al., 2020). Even when our evaluation is in probing setting, most models could not be trained on sequence labelling tasks like beat tracking or source separation with affordable computational costs except for MERT and baseline models with similar architecture (Hsu et al., 2021; Baevski et al., 2022).
Table 3: Evaluation Results from Models Trained with Different Teacher Settings. Models labeled with 62 and suggest that the K-means teachers are trained with the features from &1 and models. All the listed models are sized in (95M) and not augmented with the in- batch noise mixture.
Acoustic Teacher
Acoustic Target Class
Musical Teacher
GS Key
GTZAN Genre
EMO Emotion
Tagging
Avg.
ROC
AP
Acc
R2y
R2a
K-means MPCC
100
89.8
36,3
15.1
66.2
39.6
67
49.4
K-means MrCe
500
90.3
38
17
70
40.6
67.5
$1.3
K-means MICC
2000
90.2
37.6
15.6
44.3
67.6
51.4
K-means Chona
300+
N/A
90.5
37.6
55.1
75.2
40.1
68.2
62.1
K K-means
2000
90.4
37.5
16.1
68.3
43.9
67.7
$1.0
K-means Lognell
500+2
90.4
37,7
49,2
72.8
46.5
66.9
60.7
K-means
K-mcans-Mcckcup MFCC+COTT
300+200
89.4
35.3
53.2
69.0
45.8
66.8
60.2
K-means Cegrm
300 200
CQT
90.6
3R.4
65.0
778.6
33.7
68.7
67.3
1024 X8 codcbook
N/A
90.7
38.7
60,5
72.8
55.3
69.0
65.0
To24 X mittee
90.5
38.4
63.2
77..2
33.2
72.3
66.9
RVQ-VAB
1024 codeboaá
1024 cxbbook?
CQT
88.6
36.7
34.4
63.5
59.4
62.1
67.2
39.7
33.3
53.2
64.5
60.5
57.6
x024x8 cofooui
90.6
38.1
66.8
48.1
68.6
65.8
5.2 THE EFFECTIVENESS OF ACOUSTIC & MUSICAL TEACHER
As demonstrated in Tab. 3, we explore optimal combinations and selections of the teacher models in the MERT paradigm with a subset of downstream tasks following Castellon et al. (2021), including auto-tagging, key detection, genre classification, and emotion recognition.
We reproduce the original HuBERT (Hsu et al., 2021) setting music datasets with the acoustic teacher K-means MPCC2 and the teacher K-means MFOC& 2 trained on features produced by HuBERT model from the first stage, similar to DeepCluster (Caron al., 2018). We observe that such models perform poorly on the key detection and emotion recognition tasks even we increase the dimension of the MFCC features from 100 2000. As the re-clustering K-means does not bring significant improvement in the second stage pre-training, we stick to the ordinary one stage pre- pre-training to study the influence brought by the teachers with less computational cost.
Given that the key information is highly related to the pitch classes of the audio, we then introduce such inductive bias by providing the K-means acoustic teacher with additional Chroma or CQT features, denoted as K-means-ooge Logmss-ChromaAl and K-means MROC+CQT The additional pitch information indirectly brought by the Chroma and CQT features immediately endow the model a certain of level of key detection ability and raise the accuracy from 15.6 to 55.1 and 53.2 while keeping increasing performances on other tasks. This confirms that the potentials of transformer models can be better excavated from more dimensions by introducing extra pseudo prediction targets in the MLM scheme. Following such an intuition, could be further assumed that designing a proper multi-task learning pre-training paradigm can guide the model to produce more general representations for various music
8

---

Published as a conference paper at ICLR 2024
understanding tasks. We thus propose leveraging the CQT explicitly musical teacher introduce harmonic inductive bias during the pre- pre-training. Compared to models trained with only the acoustic teacher MFCCs
K- K-meansl LigmmllClnmaa MERT models trained with the newly proposed CQT musical teacher that are naturally more aligned to music audio can achieve significant performance gains on not only the key detection task but also the tasks requiring the high-level information like genre classification and emotion recognition.
However, given that K-means models are difficult to scale up on large datasets due to memory and computational requirements, we use the RVQ-VAE model EnCodec (Défossez et al., 2022) as the final version of acoustic teacher without searching for the immeasurable could intuitively provide more comprehensive acoustic information since the audio can be greatly recovered from the intermediate discrete codecs from the encoder by a neural decoder. We observe that leveraging only one top (1024codcbook7) or bottom layer (1024codeook)) of the residual codebooks in RVQ- VAE already provide substantial information in pre-training, and the use of all layers in the codebooks allows the student models to learn richer acoustic patterns. Although the strategy of randomly accessing one of the codebooks for each batch can alleviate the use of GPU memory and lead to similar performance compared to using all of them at time, the setting of predicting coodebooks together is adopted for faster convergence in the finalised design. By replacing the acoustic teacher with RVQ-VAE, MERT achieves average score of 66.9, similar that of the K-means-oggel--ne Logmml- version (i.e., 67.3) while largely reducing the cost of scaling up K-means.
apper-parameter K. The EnCodec
5.3 STUDY ON MUSICAL LOss
Table 4: Evaluation Results for Musical Loss Study, The listed models are not augmented with the in-batch noise mixture.
Parameter Size
Acoustic Teacher Model
Acoustic Target Class
Musical Loss W Weight
MTT Tagging
GS Kry
GTZAN Genre
EMO Emotion
Avg.
RUC
AP
Acc
Polked
Ace
K2"
kz*
N/A
37.6
75.2
40.1
68.2
95M
K means Loprct-Chreea
2
90.6
90.6
38.1
38.4
62.7
78.6
66.9
43.5
68.7
67.9
67.3
62.7
90.4
57.3
65.3
70.3
45.7
68.3
64.1
95M
RVQ-VAB
1024 #l
N/A
91.7
38.7
60.5
72.8
69.0
65.0
1024
90.5
38.4
63.2
77.2
53.2
72,3
66.9
We conducted a hyperparameter search to determine the optimal weight for the musical loss applied to masked audios in the k-means setting. In Table 4, we present the results of the varying musical loss weights, which uses the same evaluation setting in $ 5.2. By adjusting the weight, we find that weight of 1 yielded the best overall performance for the base model. We observe that when switching the acoustic teacher to RVA-VAE, the models performs slightly worse on GS than those with K-means. Overall, our study provides valuable insights into the impact of musical loss and different acoustic models on the performance of the acoustic language model. These findings can inform the future development of more effective and efficient models in the domain of acoustic processing.
6 CONCLUSION
In conclusion, our work underscores the potential of SSL for modelling raw music audio and the efficacy of our approach, MERT, in pre-training sizeable models. We present a novel paradigm that integrates RVQ- and CQT teacher models, providing a unique blend of acoustic and musical information necessary for MLM-based pre-training for music understanding. This integration. bolstered by the application of an in-batch noise mixup data augmentation and Pre-LN, enables the learning of robust music representations with further training stability. The performance of the MERT model surpasses previous SSL baselines, achieving SOTA or comparable results across a wide range of MIR tasks while using significantly smaller parameter size. We anticipate that our method and the forthcoming public release of our codes and models will catalyse further research into the application of SSL in music audio, thereby broadening the scope and depth of human understanding of music. Despite being capable of handling longer sequences with relative positional embedding, our models are limited by the short 5-second training context, so our approach could be further improved for tasks requiring understanding extended musical contexts if trained on longer sequences.

---

Published as a conference paper at ICLR 2024
LIMITATION AND FUTURE WORK
Our models are trained using only 5-second audio signals due to constraints in computational resources and the extended length of audio signals. Despite these models being capable of handling longer sequences thanks to relative positional embedding, this approach could potentially limit their performance in tasks requiring comprehensive understanding of extended musical contexts. We plan to continue training our models on longer context once gaining access to more computing resources. Moreover, although we propose several techniques improve the training stability for the acoustic wre-training, we still suffer from the gradient exploding issues with the half-precision training for settings with larger batch sizes and model sizes. addition, we observe liverse-scaling effect in specific tasks while scaling-up to 330M, which indicates that our design could be further improved by stabilising the training.
ACKNOWLEDGEMENT
This paper tribute our talented friend Anqiao Yang, for his friendship and valuable advice to this work. Yizhi Li is Ph.D. student fully funded by the Department of Computer Science, University of Manchester, UK. This work is partially funded by Theme based Research Scheme (T45- 205/21-N), Research Grants Council of Hong Kong- Yinghao Ma is research student at the UKRI Centre for Doctoral Training Artificial Intelligence and Music, supported by UK Research and Innovation Igrant number EP/S022694/1J. Emmanouil Benetos is supported by RAEng/Leveei Leverhhne Trust Research Fellowship [grant number STRRF222319-111 University of Sheffield for the provision of services for High Performance Computing.
We acknowledge IT Services at The
REFERENCES
[1] Alonso-Jiménez, P., Serra, X., and Bogdanov, D. (2022). Music representation learning based on editorial metadata from discogs. International Society for Music Information Retrieval (ISMIR).
[2] Baevski, A., Hsu, W.-N., Xu, Q., Babu, A., Gu, J., and Auli, M. (2022). Data2vec: A general framework for self- supersuue learning in speech, vision and language. In International Conference on Machine Learning, pages 221--131. PMLR.
[3] Baevski, A. and Mohamed, A. (2020). Effectiveness self-supervised pre-training for ast. In ICASSP 2020 2020 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pages 7694-7698.
[4] Baevski, A., Schneider, S., and Auli, M. (2019). wq-wav2vec: Self-supervised learning of discrete speech representations. arXiv preprint arXiv: 190.0545
[5] Baevski, A., Zhou, Y., Mohamed, A., and Auli, M. (2020). wav2vec 2.0: A framework for self-supervised learning of speech representations. Advances neural information processing systems, 33:12449-1244
[6] Bertin- T., Ellis, D. P.. Whitman, B., and Lamere, P. (2011). The million song dataset. Proceedings of the 12th International Conference on Music Information Retrieval (ISMIR).
[7] Böck, S., Korzeniowski, F., Schlüter, J., Krebs, F,, and Widmer, G. (2016a). madmom: a new Python Audio and Music Signal Processing Library. In Proceedings of the 24th ACM International Conference on Multimedia, pages 1174-1178, Amsterdam, The Netherlands.
[8] Böck, S., Krebs, F., and Widmer, G. (2016b). Joint beat and downbeat tracking with recurrent neural networks. In ISMIR, pages 255-261. New York City.
[9] Bogdanov, D., Won. M., Tovstogan, P., Porter, A., and Serra, X. (2019). The mig-jamendo dataset for automatic music tagging. Machine Learning for Music Discovery Workshop, International Conference on Machine Learning (ICML. 2019).
[10] Borsos, Z., Marinier, R., Vincent, D., Kharitonov, E., Pietquin, O., Sharifi, M., Teboul, O., Grangier, D., Tagliasacchi, M., and Zeghidour, N. (2022). Audiolm: a language modeling approach audio generation. arXiv preprint arXiv:2209.031433
10

---

Published as a conference paper at ICLR 2024
[11] Brown, J. A. and Bidelman, G. M. (2022). Familiarity of background music modulates the cortical tracking of target speech at the "cocktail party". Brain Sciences, 12(10):1320.
[12] Brown, C. (1991). Calculation of a constant q spectral transform. The Journal of the Acoustical Society of America, 89(1):425-44 4344..44..
[13] Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020). Language models are few-shot learners. Advances in neural information processing systems, 13:1877-1901.
[14] Caron, M., Bojanowski. P., Joulin, A., and Douze, M. (2018). Deep clustering for unsupervised learning of visual features, In Proceedings of the European conference on computer vision (ECCV), pages 132-149.
[15] Castellon, R., Donahue, C., and Liang, P. (2021). Codified audio language modeling learns useful representations for music information retrieval. In ISMIR.
[16] Chen, L., Lu, K., Rajeswaran, A., Lee, K., Grover, A., Laskin, M., Abbeel, P., Srinivas, A., and Mordatch, I. (2021a). Decision transformer: Reinforcement learning via sequence modeling. Advances neural information processing systems, 14:15584-5
[17] Chen, S., Wang, C.,, Chen, Z., Wu, Y., Liu, S., Chen, Z., L., Kanda, N., Yoshioka, T., Xiao, X., et al. (2021b). Wavlm: Large-scale self-supervised pre-training for full stack speech processing. arXiv preprint 1rXiv:2110.13900
[18] Chen, W., Keast, J., Moody, J., Moriarty, C., Villalobos, F., Winter, V., Zhang, X., Lyu, X., Freeman, E., Wang, J., et al. (2019). Data usage in mir: history future recommendations.
[19] Choi, K., Fazekas, G., Sandler, M., and Cho, K. (2017). Transfer learning for music classification and regression tasks. arXiv preprint arXiv:1703.09173
[20] Défossez, A., Copet, J., Synnaeve, G., and Adi, Y. (2022). High fidelity neural audio compres- sion. arXiv preprint arXiv:2210.133: 13438.
[21] Dhariwal, P., Jun, H., Payne, C., Kim, J. W., Radford, A., and Sutskever, I. (2020). Jukebox: A generative model for music. arXiv preprint arXiv:2005.00344
[22] Engel, J., Resnick, C., Roberts, A., Dieleman, S., Norouzi, M., Eck, D., and Simonyan, K. (2017). Neural audio synthesis of musical notes with wavenet autoencoders. In International Conference on Machine Learming, pages 1068-100 1077. PMLR.
[23] Fang, Y., Wang, W., Xie, B., Sun, Q., Wu, L., Wang, X., Huang, T., Wang, X., and Cao, (2022). Eva: Exploring the limits masked visual representation learning at scale. arXiv preprint arXiv:2211.07636.
[24] Heydari, M., Cwitkowitz, F., and Duan, Z. (2021). Beatnet: Crnn and particle filtering for online joint beat downbeat and meter tracking. arXiv preprint w:Xiv:2108.03566
[25] Hsu, W.N.,, Bolte, B., Tsai, Y.-H. H., Lakhotia, K., Salakhutdinov, R., and Mohamed, A. (2021). Hubert: Self-supervised speech representation learning by masked prediction of hidden units. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 29:3451-3460.
[26] Huang, Q., Jansen, A., Lee, J., Ganti, R., Li, J. Y., and Ellis, D. P. (2022). Mulan: A joint embedding of music audio and natural language. arXiv preprint arXiv:2208. 12415.
[27] Jasmin, K., Dick, F., Holt, L. L., and Tierney, A. (2020). Tailored perception: Individuals' speech and music perception strategies fit their perceptual abilities. Journal of Experimental Psychology: General, 149(5):
[28] Kereliuk, C., Sturm, B. L., and Larsen, J. (2015). Deep learning and music adversaries. 1EEE Transactions on Multimedia, (7(11)2059-2071

---

Published as a conference paper at ICLR 2024
[29] Knees, P., Faraldo Pérez, A., Boyer, H., Vogl, R., Böck, S., Hörschläger, F., Le Goff, M., et al. (2015). Two data sets for tempo estimation and key detection in electronic dance music annotated from user corrections. In Proceedings of the 16th International Society for Music Information Retrieval Conference (ISMIR); 2015 Oct 26-30; Málaga, Spain.| Málaga]] International Society for Music Information Retrieval, 2015. p. 364-70. International Society for Music Information
Retrieval (ISMIR).
[30] Korzeniowski, F. and Widmer, G. (2017). End-to-end musical key estimation using convolu- tional neural network. In 20/7 25th European Signal Processing Conference (EUSIPCO), pages 966-970. IEEE.
[31] Lample, G. and Charton, F. (2019). Deep learning for symbolic mathematics. arXiv preprint arXiv:1912.01412.
[32] Law, E., West, K., Mandel, M. I., Bay, M., and Downie, S. (2009.. Evaluation of algorithms using games: The case of music tagging. In ISMIR, pages 387-392.
[33] Li, Y., Yuan, R., Zhang, G., MA, Y., Lin, C., Chen, X., Ragni, A., Yin, H., Hu, Z., He, H., al. (2022). Map-music2vee: A simple and effective baseline for self-supervised music audio representation learning. In ISMIR 2022 Hybrid Conference.
[34] Ma, Y., Yuan, R., Li, Y., Zhang, G., Chen, X., Yin, H., Lin, C., Benetos, E., Ragni, A., Gyenge, N., et al. (2023). On the effectiveness of speech supe-suue learning for music. ISMIR 2023: Proceedings of the 24th International Society for Music Information Retrieval Conference, Nov 4-9, 2023, Milan, Italy.
[35] Marchand, U. and Peeters, G. (2015). Swing ratio estimation. In Digital Audio Effects 2015 (Dafx15).
[36] McCallum, M. C., Korzeniowski, F., Oramas, S., Gouyon, F., and Ehmann, A. F. (2022). Supervised and unsupervised learning of audio representations for music understanding. arXiv preprint arXiv:2210.03799
[37] Mehr, S. A., Singh, M., Knox, D., Ketter, D. M., Pickens-Jones, D., Atwood, S., Lucas, C., Jacoby, N., Egner, A. A., Hopkins, E. J., al. (2019). Universality and diversity in human song. Science, 36666468))eaax0868.
[38] Mitsufuji, Y., Fabbro, G., Uhlich, S., Stöter, F.R.,, Défossez, A., Kim, M., Choi, W., Yu, C.-Y., and Cheuk, K.-W. (2022). Music demixing challenge 2021. Frontiers in Signal Processing, 1:18. 88.
[39] Modrzejewski, M., Szachewicz, P., and Rokita, P. (2023). Transfer learning with deep neural embeddings for music classification tasks. In Artificial Intelligence and Soft Computing: 2 International Conference, ICAISC 2022, Zakopane, Poland, June 19-23, 2022, Proceedings, Part I, pages 72-81. Springer.
[40] Petermann, D., Wichern, G., Wang, Z.-Q., and Le Roux, J. (2022). The cocktail fork problem: Three-stem audio separation for real-world soundtracks. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 526-530. IEEE.
[41] Pons. J. and Serra, X. (2019). musicnn: Pre-trained convolutional neural networks for music audio tagging. arXiv preprint
[42] Raffel, C., McFee, B., Humphrey, E. J., Salamon, J., Nieto, O., Liang, D., Ellis, D. P., and Raffel, C. C. (2014). Mir eval: A transparent implementation of common mir metrics. In ISMIR, pages 367-372.
[43] Rafii Z., Liutkus, A., Stöter, F.-R., Mimilakis, S. I. and Bittner, R. (2017). The MUSDB18 corpus for music separation.
[44] Rouard, S., Massa, F., and Défossez, A. (2023). Hybrid transformers for music source separation. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1-5. IEEE
12

---

Published as a conference paper at ICLR 2024
[45] Saeed, A., Grangier. D., and Zeghidour, N. (2021). Contrastive learning of general-purpose audio representations. In /CASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 3875-3879. IEEE.
[46] Santana, I. A. P., Pinhelli, F., Donini, J., Catharin, L., Mangolin, R. B., Feltrim, V. D., Domingues, M. A., et al. (2020). Music4all: A new music database and its applications. In 2020 International Conference on Systems, Signals and Image Processing (IWSSIP), pages 399- 404. IEEE
[47] Soleymani, M., Caro, M. N., Schmidt, E. M., Sha, C.-Y., and Yang, Y.-H. (2013). 1000 songs for emotional analysis of music. In Proceedings of the 2nd ACM international workshop on Crowdsourcing for multimedia, pages 1-6.
[48] Spijkervet, J. and Burgoyne, J. A. (2021). Contrastive learning of musical representations. arXiv preprint arXiv: 200::2241009
[49] Tzanetakis, G. and Cook, P. (2002). Musical genre classification of audio signals. IEEE Transactions on speech and audio processing, 10(5):293-300
[50] Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L.., Gomez, A. N., Kaiser, t.,, and Polosukhin, I. (2017). Attention is all you need. Advances in neural information processing systems, 30.
[51] Wang, C., Chen, S., Wu, Y., Zhang, Z., Zhou, L., Liu, S., Chen, Z., Liu, Y., Wang, H., Li, J., al. (2023). Neural codec language models are zero-shot text speech synthesizers. arXiv preprint arXiv:2301.02111.
[52] Wang, H., Ma, S., Dong, L., Huang, S., Zhang, D., and Wei, F. (2022a). Deepnet: transformers 1,000 layers. arXiv preprint arXiv:2203.0055 00555..
Scaling
[53] Wang, L., Luc, P., Wu, Y., Recasens, A., Smaira, L., Brock, A., Jaegle, A., Alayrac, J.-B., Dieleman, S., Carreira, J., al. (2022b). Towards learning universal audio representations. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 4593-4597, IEEE.
[54] Wilkins, J., Seetharaman, P., Wahl, A., and Pardo, B. (2018). Vocalset: singing voice dataset. In ISMIR, pages 468 -774.
[55] Yamamoto, Y., Nam, J., and Terasawa, H. (2022). Deformable cnn and imbalance- feature learning for singing technique classification. arXiv preprint arXiv:2206. 12230.
[56] Yang, S.-w., Chi, P.-H., Chuang, Y.-S., Lai, C.-L J., Lakhotia, K., Lin, Y. Y., Liu, A. T., Shi, J., Chang, X., Lin, G.-T,, et al. (2021). Superb: Speech processing universal performance benchmark. arXiv preprint arXiv:2
[57] Zai El Amri, W., Tautz, O.,, Ritter, H., and Melnik, A. (2022). Transfer learning with jukebox for music source separation. In Artificial Intelligence Applications and Innovations: 18th IFIP WG 12.5 International Conference, AIAI 2022, Hersonissos, Crete, Greece, June 17-20, 2022, Proceedings, Part pages 426 436-433. Springer.
[58] Zeghidour, N., Luebs, A., Omran, A., Skoglund, J., and Tagliasacchi, M. (2021). Soundstream: An end-to-end neural audio codec. IEEE/ACM Transactions Audio, Speech, and Language Processing. 30:495-507.
[59] Zhao, Y. and Guo, J. (2021). Musicoder: A universal music-acoustic encoder based on transformer. In International Conference on Multimedia Modeling, pages 417-429. Springer.
[60] Zhu, H., Niu, Y., Fu, D., and Wang. H. (2021). Musicbert: A sule-supe learning of music representation. In Proceedings of the 29th ACM International Conference on Multiedia, pages 3955-3963.
13

---

Published as a conference paper at ICLR 2024
APPENDIX A METHODOLOGY
A.1
ROBUST REPRESENTATION LEARNING
Algorithm Pseudocode description of the pre- pre-training loss calculation in Python style.
Bel Loss_cal batach, K_acoLacic 1abaza))
2
retrieve wrloeddi nge For wcoust to dlass enbedding (o lablla)
prepaze egr tarcess
yLCOT = compute, COT betch)
6
conduct. inrbetol nixcure
X_noised regresenncatinn
congute
9
NERT binosse)
10
caloulation
Loss_acoustic Creas_Bntropy idd)) (_V0[nask
13 Tess musicall Mean Square Rrror (nlmank, idx), y000T [masic (dx))
relucn acuua io, mos
We introduce "in-batch noise mixup" for music SSL. The mixup augmentation refers to the audio clip being added up with certain ratio of shorter audio excerpts to form augmented single sample during pre-training, instead of using the original audio. We randomly sample the audio segments from the same batch and add them audio at random positions according some probability. Theoretically, sampling from the whole training dataset would provide more randomness and thus be more beneficial to the representation robustness, but we narrow the sampling pool the same audio batch considering the limited computational resources. The mixup could enable the learning of more robust musical representations and force the model to focus on the useful musical source and to ignore the noise. A pseudocode implementation can be found in Algo. 1.
Using the same evaluation setting in $ 5.2, we alter the in-batch mixup probability to evaluate whether it is affecting the performance of the model when compound with musical loss. We found the mixup probability provides worse results in MERT*-means but provides better performance for MVRTRV Therefore, determined probability of 0.5 to be suitable based on the average performance score, and set it as a hyper-pul papee-parrmeter in the final released model. Such phenomenon deserves more attention.
Table 5: Evaluation Results for Pre-training Setting Ablation Study.
Parameter Size
Acoustic Teacher Model
Acoustic Target Class
Musical
Loss Weight
Mixup
MTT Tagging
Key
GIZAN Genze
EMO Emotion
Avg.
Probability
ROC
Acc
Ace
R2*
55.1
75.2
62.1
78.6
95M
K-meuns Eepreeeehroiaa
300 200
N/A
N/A
90.4
90.5
33.1
37.3
65.3
62.7
70.3
66.9
45.7
45.5
68.3
67.9
62.7
64.1
0.25
90.6
37.9
65.5
70.0
49.6
72.5
65.2
0.5
72.8
65.2
95M
x
N/A
90.5
38.4
63.2
77.2
53.2
72.3
66.9
RVQ-VAR
x dluolbook
14

---

Published as a conference paper at ICLR 2024
A.2
THE EPFECTIVENESS OF VANILLA RVQ- VAE RREPRESENTATION
Table 6: Evaluating the EnCodec RVQ-VAE VAE Embeddings.
GTZAN Genre
VocalSet Toch
VocalSet Singer
Task
Tageing
Key
Emotion
AccReloed
Acc
R2
R24
Acc
RVQ. Embedding
10.3
MERT-95M ME
74.6
77.2
MERT-95M TAE
To verify that the benefits brought by the MERT pre-training, we further evaluate the performances of the continuous representation from the encoder the RVQ-VA (A- model, as shown Tab. These supplementary results indicate that the vanilla continuous representations alone are insufficient for robust music understanding baseline.
15

---

Published as a conference paper at ICLR 2024
APPENDIX B EXPERIMENT DETAILS
B.I DOWNSTREAM TASKS
We evaluate the models on 14 downstream tasks provide a comprehensive view of our method and the comparison between baselines. The full descriptions of the datasets and tasks are given as follows.
Music Tagging involves determining which of a set of fixed tags apply to a particular song. Tag categories may include genre. instrumentation, mood, tempo (e.g. fast) or other tags. We used two (MTT) (Law et al., 2009) and MTG-Jamendo (Bogdanov et al., 2019). For both datasets, we limit the tag vocabulary according to official instructions. We use all clips in MTT and MTG-Jamendo. Since many of the audio recordings among 5.5k MTG-Jamendo excerpts are longer than the 30s, we averaged the multiple embeddings computed with a sliding window as the overall embedding. The window length is set to the same default length as in every system. For MERT series, the window length is typically set to 30s. The metrics are the macro-average of ROC-AUCs and the average precision (AP) / PR-AUC among all top-50 tags.
large datasets: Magna'TagATTun
Key detection predicts the tonal scale and dominant pitch level of a song. We use Giantsteps (Knees al., 2015) as test set and commonly-usen subset of Giantsteps-MTGG-Ke and Widmer, 2017) as the training and validation set. The splitting is the same as in (Castellon et al., 2021). The metric is a refined accuracy with error tolerance, giving partial credit reasonable errors (Raffel et al., 2014).
dataset (Korzeniowski
Genre classification estimates the most appropriate genre for each given song. We report the accuracy of the GTZAN (Tzanetakis and Cook, 2002) dataset along with ROC and AP on MTG-Genre, since the former task is a multi-class classification and the latter is multi-label. We used the standard "Fall-fitered"" split (Kereliuk et al., 2015) for GTZAN.
Emotion score regression. The Emomusic dataset (Soleymani et al., 2013) contains 744 music clips of 45 seconds in length, each reported on a two- vo-dimensional valence- plane after listening, where valence indicates positive and negative emotional responses, and arousal indicates emotional intensity. We use the same dataset split as (Castellon et al., 2021). The official evaluation metric is the determination coefficient (1²) between the model regression results and human annotations of arousal (EmoA) and valence (EmoVV (Soleymani et al., 2013). For inference, we split the 45-second clip into a 5-second sliding window and averaged the prediction.
Instrument classification is the process of identifying which instruments are included in given sound. We use the Nsynth (Engel et al., 2017) and MTG-instrrmmen datasets. The former is a monophonic note-level multi-class task with 306k audio samples in 11 instrument classes with accuracy as an indicator. The latter is a subset of MTG-Jamendo, containing 25k polyphonic audio tracks and 41 instrument tags; each track can contain multiple instruments and is evaluated ROC and AP.
Pitch classification estimates which of the 128 pitch categories the given audio segment belongs to. We use the NSynth dataset for this task. Given these segments are short monophonic audio, this task multi-class, and the accuracy is used an evaluation metric.
Vocal technique detection involves identifying what singing techniques are contained in a given audio clip. We use the VocalSet dataset (Wilkins et al., 2018), which is the only publicly available dataset for the study of singing techniques. The dataset contains the vocals of 20 different professional singers (9 female and 11 male) who perform 17 different singing techniques in various contexts for a total of 10.1 hours. As the audio clips are divided into 3 seconds, the task only requires judgement on the type of technique and not on the start and end of the technique. We used the same 10 different singing techniques in Yamamoto et al. (2022) as a subset and used the same 15 singers as the training and validation sets and 5 singers as the test set. Since there is no accepted division between training and validation sets, we selected 9 singers as the training set and 6 singers the validation set. All the 3-second segments that originate from the same recording are allocated to the same part of the split (e.g. all are in the training set). The evaluation metric is accuracy.
16

---

Published as a conference paper at ICLR 2024
Singer identification identifies the vocal performer from given recording. We use the VocalSet dataset for this task. We randomly divided the dataset into a training set, validation set and testing set based on ratio of 12:8:5, all containing the same 20 singers.
Beat tracking is the process of determining whether there is a beat in each frame of a given piece of music. We use offline approach to the binary classification, i.e. the model can use information following each frame to help with inference. The model needs to output frame-by-fil by-frame predictions certain frequency and post-process them using dynamic Bayesian network (DBN) (Böck et al., 2016b) to obtain the final result. The DBN is implemented using ma dimom (Böck et al., 6a). The dataset we use is GTZAN Rhythm (Marchand and Peeters, 2015). We also label the two adjacent frames of each label as beat, which is a common way of label smoothing in beat tracking to improve the performance of the model and to compare the SSL model fairly with the spin model. The model evaluated using the f_measure implemented in mi r_eval (Raffel et al., 2014), and the prediction considered correct if the difference between the predicted event and the ground truth does not exceed 20ms. In this task, some models were trained on other datasets, and the full GTZAN set was used as the test set.
Source separation. Source separation aims to demix the music recording into its constituent parts, e.g., vocals, drums, bass, and others. We adopt MUSDB18 18 (Rafi et al., 2017), a widely used benchmark dataset in music source separation. MUSDB 18 contains 150 full-length music tracks (IO hours), along with multiple isolated stems. We use 86 tracks for training, 14 tracks for validation, and 50 tracks for evaluation following the official setting in MUSDB18. 18. During training, sample 6-second segments and apply random track mixing for augmentation. Due to the difficulty of this task, we adopt the baseline architecture in the Music Demixing Challenge (MDX) 2021 (Mitsufuji et al., 2022), which consists of three linear layers and three bi-directional LSTM layers. We directly compute the 12-loss between predicted and ground-true truth spectrograms for optimisation. The metric for this task is the Source-to- to-Dsttrrio Ratio (SDR) defined by MDX 2021 (Mitsufuji et al., 2022), which is the mean across the SDR scores of all songs.
randomly
B.2 TESTING PROTOCOL DETAILS
Our aim is to explore the generality and standardisation of the framework. We, therefore, freeze the parameters of the pre-trained model to extract pre-trained features fixed depth embeddings that are fed to each downstream task-specific prediction head. This allows for as lightweight a solution as possible for all tasks, thus testing whether the representations easily reusable across different downstream tasks. In the following, we first describe the selected pre-trained baseline model, followed by the downstream model and training strategy.
In order to detect representations with relevant information about the downstream MIR task, we these representations input features train shallow supervised model on each task. For most tasks we use an MLP with one hidden layer, and for source separation, we use the baseline of the demixing data challenge described above, with the 3-layer LSTM used as post-processing. Since some representations may require different hyperparameter configurations to be successfully trained, we performed the following hyperparameter search for each mentioned SSL mainly based on MARBLE benchmark, using the validation set for each downstream task.
Model: fone-layer MLP with 512 hidden units, 3-layer LSTM (source separation only)}
Batch size: {64]
Learning rate: (1e-4, 5e-4, le-3, Se-3, le-2]
Dropout probability: {0.25}
Optimizer: Default Adam optimizer
Early Stopping: Fixed across all models with task-specific patience
LR Scheduler: Reduce LR On Plateau, fixed across all models with task-specific patience
In addition, although we use the same hyperparameter grid for all tasks, the learning objectives vary from task to task. For the same task with uniform dataset, if there are different evaluation metrics,
Phttps::/maaaai aac.uac.uk
17

---

Published as a conference paper at ICLR 2024
will average the two evaluation metries. We keep the best validation set results, and use the test set results as the final results of the benchmark.
18

---

Published as a conference paper at ICLR 2024
B.3 TRAINING INSTABILITY
In the experiments of scaling up to MERT 330M under mix precision training (fpl6), we have explored several settings and plot the gradient norm, scale of loss, the MLM loss acoustic targets, and the MLM loss on musical targets (see Fig. 2).
We first adopt the Pre-LN setting as in the HuBERT (Hsu et al., 2021) x-large model for stable training. However, the training crashed around 50K step under this vamilla solution from the speech model and thus restart the pre-training 40K step with gradient clipping threshold reduced from 10.0 to 1.0. The second run of Pre-LN lasted for 40K steps and crashed due to the same reason of reaching minimum loss scale.
We suspect the instability could be brought by the increased depth of the Transformer module. Following the strategies in DeepNorm (Wang et al., 2022a), we tried alleviate the instability by initialising the Transformer with smaller values and enhancing the residual connection in the Post-I.N. Unfortunately, such modification causes model collapse around 20K steps.
We then turned back to the stable Pre-LN setting and leveraged the attention relaxation trick proposed Chen et al. (b02 The additional scale constant in softmax calculation the attention module alleviates the overflow problem and allows the final version of MERT-330M model to be trained stably over 100K steps.
Pre-Norm Gradientt Cip-110
Pre-Kom Gradiemt Cip-10
Norm Gradient Glp-1 Run
1.75
Post-Norm DeopNorm Gradient Clio1
Grodlen Norm
Pro Norn AAz, Rdlox Graddoott Clp=-
1.50
Prom Rebax| Gladent-Olpe1
2:1.22
0.50
0.25
oK
40K Training Step
B0K
100K
0.00
Tha hing Step
(a) Gradient Norm
(b) Loss Scale
Acouncte MEM Loss on Colebook-0
Pre-Norm adieent Ckp-10 Bun
Fro-Norm] Gradient p1p-10 Run
Pru Pou-Nomm Gradient Cip=1] Run
144
Pre-Norm Grackent pr1 Run
Post Gradient
Post-torm DDeploom Grudien Cipp1
Nor Relax| Gradienn Clp-1
Pre Norm Atn. Relax Gradient Clp-1
1.0-
0.8
0k
20K
40K
60K
80x
100K
8K
20k
40K
60K
80K
L0OK
Training Step
Training Step
(c) Acoustic MLM Loss on Codebook-O
(d) Music MLM Loss
Figure 2: Illustration of the Training Curves of Trials on Large (330M) Models. Only the acoustic MLM loss on codebook 0 in the RAQ is shown as the other seven show similar trends.
19

---

Published as a conference paper at ICLR 2024
APPENDIX C REPRESENTATION VISUALISATION
We select two of our checkpoints, MERT-95M-publ11 means and MERT-33 RV VAE and visualise the GTZAN representations with genre annotation shown in Fig. 3, Fig. 4, Fig. and Fig. 6, The top and top 8 transformer output layers are used in the visualisation for MERT-95M- 1it-999M-public---eeeee tion is achieved by the Uniform Manifold Approximation and Projection (UMAP)', whereas the representations from the training set are used to learn the dimension reduction mapping. We observe that representations from both of the checkpoints present pattern of clustering according to the genre information under different layer settings. Interestingly, the representations from the higher layers do not necessarily show stronger genre-based clustering tendency, which suggests that I) genre may not be the most abstractive labels for these music examples or 2) the top transformer layers focus more the MLM pre-training objectives.
and MERT-330MR00 3MM correspondingly. The dimension reduc-
Valid
blues
classical
country
hiphop
Jazz
metal
pop
reggae
rack
(a) MERT- MERRT-95M-PP- 95M--ub11ccccccc K-means Transformer Layer Representations of GTZAN.
Valid Set
Test Set
classical
country
disco
hiphop
Jazz
metal
pop
reggae
rock
(b) MERI 95M 355--ublllt K K-neans Transformer Layer Representations of GTZAN.
Valld Set
Test Set
blues
clossical
country
disco
hilphop
pop
reggae
(c) MERT 113M-pubbicc mens Transformer Layer 9 Representations of GTZAN.
Figure 3: Illustration of the MERT- SERR-95M-pubbl1e wwS-MM--ublieMeMe X-means Layer 7 to 9 Pre-trained Representations.
hhans//gitthh.ccm//ncieemmmm
20

---

Published as a conference paper at ICLR 2024
Set
Set
blues
classical
country
hiphop
Jazz
pop
reggao
rock
(a) MERT- PPRRT-95M-ubbl1 K-naans Transformer Layer Representations of GTZAN.
Valid
classical
country
000
hippop
jazz
metal
pop
reggee
rock
(b) MERT SEETT-9MMPubble K-
nene Transformer Layer 11 Representations of GTZAN.
Valid Set
Test Set
blues
country
disco
hiphop
metal
pop
reggao
(c) MERTER-999 111---99--puu--e-c K-means Transformer Layer 12 Representations of GTZAN.
Figure 4: Illustration of the PPRR--95M-puke 6 means Layer 10 to 12 Pre-trained Representations.
21

---

Published as a conference paper at ICLR 2024
Valid Set
Test Set
blues
classical
country
disco
hiphop
jazz
metal
reggee
rock
(a) MERT- 3 0M RVO-VAE Transformer Layer 17 Representations of GTZAN.
Valid Set
Test Set
country
disco
hiphop
jazz
metal
reggae
rock
(b) MERT - 330 VAE Transformer Layer 18 Representations of GTZAN.
Vaild Set
Test Set
blues
classical
country
hiphop
Jazz
pop
reggae
rock
(c) MERT-33 EVO-VE Transformer Layer 19 Representations of GTZAN.
Valid Set
Test Set
classical
country
disco
hiphop
jazz
metal
pop
reggae
rock
(d) MERT-3322 MORRT-330MMN-1A- Transformer Layer 20 Representations of GTZAN.
Figure 5: Illustration the MERT-33 0M RVO-AA Layer 17 to 20 Pre-trained Representations.
22

---

Published as a conference paper at ICLR 2024
Valid Set
Test Set
blues
classical
country
disco
jazz
metal
reggae
rack
(a) MERT- 33 330MM00-0 Transformer Layer 21 Representations of GTZAN.
Valid Set
Test Set
classical
country
disco
Jaz2
metal
reggae
rock
(b) MERT- -30RTT-30MM00 VAE Transformer Layer 22 Representations of GTZAN.
Vaild Set
Test Set
blues
classical
country
disco
hiphop
JazZ
pop
reggae
rock
(c) MERT-333 EVO-VE Transformer Layer 23 Representations of GTZAN.
Valid Set
Test Set
classical
country
disco
hiphop
jazz
metal
pop
reggae
rock
(d) MERT- WERT-330M RV- Transformer Layer 24 Representations of GTZAN.
Figure 6: Illustration the MERT-32 0M RV--AA Layer 21 to Pre-trained Representations.
23

---

Published as a conference paper at ICLR 2024
APPENDIX D ETHICS
We have taken great care to ensure that our research adheres to ethical principles and guidelines the codes of conduct. Specifically, we have used inappropriate user information in the experiments. The audios are collected from open-aceess streaming services and open-source datasets, where the quality of the audios vary among 16KHz, 24KHz, and 48KHz. The audios would not be hosted and distributed. We believe that our work has the potential to contribute to positive social and scientific outcomes regarding the research of automatic music understanding.
24


Reviews:
[{"review_id": "6kQfnN2Qev", "reviewer": "Reviewer_7KYT", "summary": "This paper proposed MERT, an SSL  model for music representation. The model is based on MLM training such as Hubert, and utilizes multiple pseudo targets during the pretraining stage such as k-means, constant-Q Transformation, and neural codec codes. By combining different self-supervised targets, experiments on 14 diverse downstream MIR tasks show that MERT is able to extract good representation for MIR tasks and attains SOTA overall scores. The checkpoints and code are open-source.", "questions": "1. How do you calculate \"Previous SOTA average score\" in Table 2? The number did not match any baselines listed in the table, is it referenced from another work?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. The method is simple but effective. The paper demonstrates the importance of choosing appropriate pseudo targets.\n2. Considers a diverse set of MIR tasks for evaluation, providing a good standard that can be followed by future works.\n3. Unlike many previous closed-source works, this paper has made the checkpoints publicly available, which is a significant contribution to the research community.\n\nOverall, the paper is well-written with a clear goal and provides sufficient experiment results to support the claims. Although the conclusions are not surprising, the work is still significant for the related research community from a practical perspective due to its reproducibility and accessibility.", "weaknesses": "There are 2 minor concerns.\n1. Since RVQ-VAE is pre-trained on a larger dataset, comparing MERT with CQT/k-means and MERT with RVQ-VAE is somewhat unfair.\n2. The experiment should verify how RVQ-VAE code performs on downstream tasks to prove that the proposed MLM training phase with RVQ-VAE code as the target is required. Otherwise, one can directly utilize codes and codebook vectors from RVQ-VAE as upstream representations instead of MERT.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Zzx0468Rod", "reviewer": "Reviewer_EaeP", "summary": "This paper proposes a self-supervised model for acoustic music understanding based on similar self-supervised learning paradigms in speech processing. The authors provide extensive comparison on 10 different MIR tasks some of which require understanding the local characteristics (such as pitch and beat), whereas some require a track-level understanding (such as genre, or emotion).\n\nThe authors experiment with two different teacher paradigms. They work with one 95M parameter model that is trained on publicly available music, and another larger model with 330M parameters that they train on 160k hours of music mined from internet. They compare their variants against the state of the art, and show that the model achieves similar or better results compared to the current state-of-the-art.", "questions": "- We mined 160K hours of music recordings from the Internet ... \n\nWhat are the typical sources for mining? Youtube, streaming services, Freesound, or something else? What is the typical audio quality? Are they copyrighted, or not? Do you keep the audio or only the relevant features (MFCC, CQT?)\n\n- Some references are not well formatted and/or they miss key information (in particular the conference). Examples:\n\nAlonso-Jimenez, P., Serra, X., and Bogdanov, D. (2022).\nBogdanov, D., Won, M., Tovstogan, P., Porter, A., and Serra, X. (2019)\nChen, W., Keast, J., Moody, J., Moriarty, C., Villalobos, F., Winter, V., Zhang, X., Lyu, X., Freeman,\nE., Wang, J., et al.\n\n- While Table 1-2 are compact and informative, it's impossible to track the references apart from following the hyperlinks as the Reference formatting do not include the number. \n\n- Although they should be known in general, I would suggest the authors to mention the full name of all the metrics such as R2 or ROC used in the experiments.\n\nIn addition, some of the \"previous SOTA\" (e.g. 26, 36) are still the best. Wouldn't it mean that they are still the state-of-the-art?\n\n- Appendix D - Ethics. I think there should be a mention of music copyrights here, in particular the implications about mining music from the Internet.\n\nBelow are minor suggestions and nitpicks that I'd like to provide for the sake of completeness. They do not contribute to my decision on the paper.\n\n- The writing switches between British and American spelling, e.g. \"masked language modeling\" vs. \"masked language modelling.\"\n- Nitpick: Page 4 \"data sample in a speech or language dataset...\" -> the dataset doesn't have to be speech or language, e.g. it can contain instrumental music. \n- \"Additionally, we present the advanced SOTA for each task including\" -> This phrase could be read as the proposed model advances the state of the art for all tasks, which is not necessarily the case. If I understand correctly, \"the current SOTA\" is a better wording.\n- \"... longer contexts if required\" ->  longer contexts, if required (missing comma)\n- Page 17 is almost fully empty.\n- Figures 2 - 6 are very useful, however, they are not suitable for color-blind readers. I would suggest to change the line/marker styles for each  element in the legends.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- Extensive comparison between different models, conditions and # parameters on various MIR tasks. \n- The results indicate the strengths of the proposed model (e.g. efficacy on tasks that require local-level musical information) as well as the limitations (e.g. 5 second excerpts)\n- Provide a strong baseline for future research on self-supervised learning on acoustic music that is comparable to the current state of the art.\n- Extensive literature review, which facilitates to convey the basis of the work as well as the motivations\n- The authors explain issues they have faced while training the model, and also how they mitigated these issues, which is invaluable for future research. See the Training Stability part in Section 4.3. as an example.\n- Open source code, experiments and dataset (where shareable)\n- The language is appropriate for an engineering work, and the paper is easy to follow.", "weaknesses": "- Works on short excerpts. The authors argue that this limitation could be overcome in future work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VrNvBt5Sbl", "reviewer": "Reviewer_Md8h", "summary": "This paper describes self-supervised learning technique for music audio. The most relevant previous work is HuBERT in speech domain which uses masked prediction learning using codeword of audio features. The authors applied this method to music audio and developed several techniques on top of the reference model. Basically, the most contributions lie on how they build codewords specifically dedicated to music audio. To do that, they mainly compared three ways which are MFCC-based codeword (K-Means), LogMel+Chroma-based codeword (K-Means), and EnCodec. MFCC and Log-Mel-spectrogram mainly captures timbre information from audio, so they utilized Chroma to compensate tonal information of music. Also, there has been a previous work called EnCodec which is a pre-trained codec encoder designed for music audio, so that this model already has some ability to capture both timbre and tonal information. Also, they added CQT loss to further enhance pitch and chord level information. In the end, the authors verified the models on 14 Music Information Retrieval tasks (mostly segment-level tasks, not note-level or frame-level). The results showed that EnCodec-based approach was the best performing model.", "questions": "If the used split of each downstream tasks can be written more in detail in Appendix, it would be better.\nIn Section 4.1, where GTZAN and MTG-Genre downstream task's metric is explained, only ROC and AP is mentioned, I think accuracy can be added.\nIn Section 4.3, \"1.5 and 5.5 hours\" is not a batch.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The strengths of the paper comes from how the authors tailored the previously proposed method to music audio domain. To do that, they tested various music audio specific techniques such as Chroma, CQT, and EnCodec. The results showed that these additional method improved the model performance on several downstream tasks that are more related to pitch, chord, tonal information of music. For the tasks where timbre information is important, the effect of using these tonal features is marginal.", "weaknesses": "The weaknesses of the paper is on novelty. If we see the results in Table 1 and 2, the trends are quite predictable even though the proposed method achieved SOTA performance on 3 tasks out of 14 tasks. For the models that doesn't utilize chord information, still those models achieve good performance on tasks where timbre is important (such as tagging, genre, mood, theme), however, if any methods includes to use this kind of information, then it shows good performance on both timbral and pitch related tasks. Also, it seems many performance boosts are made through the EnCodec, I think the novelty of the approach itself is a bit weak.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "u1xXyCFvo7", "reviewer": "Reviewer_j7HH", "summary": "The paper presents MERT - large scale self-supervised models for music tasks. The approach is built primarily on top of the speech SSL approach (Hubert). Aside from K-means, MERT proposes to use Encodec to get targets for SSL pre-training. Reconstruction loss on Constant-Q transform is also used for training. The pre-training is done on 1K hours of music data and then a 160K hour of music data. In terms of model size MERT offers two flavors: a 95M parameter model and a 330M parameter model. The MERT model is evaluated on a wide range of music tasks - Music tagging, music genre classification, instrument classification and so on.", "questions": "Please respond to the points in the weakness section.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "– The paper presents self-supervised learned models for music audio. Better large scale SSL models for music are definitely desirable and the paper is a fair attempt at building such models. Moreover, the paper aims to make the model open-source which can definitely help in future development of SSL models for music. \n\n– The paper describes the approach clearly. The datasets, downstreams tasks etc. are also well described. \n\n– Discussions on challenges in scalability and training stabilities are also discussed. I think that’s a good topic to touch upon. \n\n– The paper also does downstream evaluation on a variety of music datasets. This creates a really good benchmark for evaluations.\n\n---- \nIncreased score after rebuttal.", "weaknesses": "– The presented approach is primarily minor modifications of existing SSL models and the significance of the MERT training approach itself  is limited. It’s not fully established that the modifications over Hubert are really adding substantive improvements in performance. \n\n–  “Computationally affordable”, “cost-effective”, “lightweight sizes”  etc. are frequently used for the proposed MERT but it is not really clear how all of these are attained for MERT. How is MERT more computationally efficient or lightweight  than say Hubert-Base. Aren’t the models similar ? What efficiencies are we expecting here? Is it just about K-Means vs uses of codebook from Encodec ? This aspect has been highlighted several times in the paper so it would be good properly establish (quantitatively ??) how MERT is better than others.  \n\n— Is it necessary to use all 8 layers of codebooks ? perhaps some additional experiments to better show how results vary with codebooks from different layers would be good. \n\n— Related to the previous point, how about using Encodec itself as SSL representation. Codec is used in MERT as a teacher - can codec itself be used as a SSL model for music. Isn’t that a baseline one can have ? \n\n— The “AVG” column in Table 2. What is it avg of ? of all the other columns ? I am not sure looking at avg of different types of metrics over different datasets is a good way to look at overall results. \n\n– Comparing “HuBERT base” and MERT-95M^{K-Means} it seems that they are pretty similar. \n\n– For all of the downstream experiments, is the full training set for each dataset used in the experiments ? I think some experiments on “limited training data” would be useful. Otherwise, these models are not really outperforming the supervised baselines – which does not full justify the SSL pre-training. \n\n– In Sec 5.1, the paper describes that the model is doing better on local-level music tasks compared to global level tasks. Some more discussion and perhaps illustration of why this is happening would be super helpful.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)