MARBLE: Music Audio Representation Benchmark for Universal Evaluation
Venue: NeurIPS 2023 Datasets and Benchmarks Poster (2023)
Keywords: music
representation learning
self-supervised learning
evaluation
TL;DR: Propose a comprehensive benchmark and toolkit for music understanding

Abstract: In the era of extensive intersection between art and Artificial Intelligence (AI), such as image generation and fiction co-creation, AI for music remains relatively nascent, particularly in music understanding. This is evident in the limited work on deep music representations, the scarcity of large-scale datasets, and the absence of a universal and community-driven benchmark. To address this issue, we introduce the Music Audio Representation Benchmark for universaL Evaluation, termed MARBLE. It aims to provide a benchmark for various Music Information Retrieval (MIR) tasks by defining a comprehensive taxonomy with four hierarchy levels, including acoustic, performance, score, and high-level description. We then establish a unified protocol based on 18 tasks on 12 public-available datasets, providing a fair and standard assessment of representations of all open-sourced pre-trained models developed on music recordings as baselines. Besides, MARBLE offers an easy-to-use, extendable, and reproducible suite for the community, with a clear statement on copyright issues on datasets. Results suggest recently proposed large-scale pre-trained musical language models perform the best in most tasks, with room for further improvement. The leaderboard and toolkit repository are published to promote future music AI research.

MARBLE: Music Audio Representation Benchmark for Universal Evaluation
Ruibin Yuan 9122 9122
Yinghao Ma 23- Yizhi Li Ge Zhang "1
Xingran Chen'
Hanzhi Yin
Le Zhuo Yigi Liu Jiawen Huang" 3
Zeyue Tian" Binyue Deng"
Ningzhi Wang'
Chenghua Lin 451 Emmanouil Benetos' Anton Ragni' Norbert Gyenge"
Roger Dannenberg"
Wenhu Chen Gus Xia Wei Xue" Li Shi Wang Ruibo Liu
Yike Guo' Jie Fu't
Multimodal Art Projection Research Community 'Beijing Academy of Artificial Intelligence
CCannegie Mellon University 'Queen Mary University London "University Manchester
Univeesitt of Sheffield "University of Waterloo 'University of Michigan Ann Arbor
"Beihang University "Hong Kong University of Science and Technology Peking University
New York University 1BMBZUAI IIIT, Chinese Academy Sciences "Dartmouth College
ruibiny@alumni cmu. edu yinghao ma@qmul ac.uk yizhi 1ithh.11@httmil.... com gezhangoumi ch, edu chenghua, lin@manchester.eee
fujie@baai.. an
Abstract
In the era of extensive intersection between art and Artificial Intelligence (AI), such as image generation and fiction co-creation, AI for music remains relatively nascent, particularly music understanding. This is evident in the limited work on deep music representations, the scarcity of large-scale datasets, and the absence of universal and community-driven benchmark. To address this issue, we introduce the Music Audio Representation Benchmark for universaL Evaluation, termed MARBLE. aims to provide a benchmark for various Music Information Retrieval (MIR) tasks by defining comprehensive taxonomy with four hierarchy levels, including acoustic, performance, score, and high-level description. establish a unified protocol based on 18 tasks on 12 public-availatee datasets, providing a fair and standard assessment of representations of open-sourced pre- pre-trained models developed music recordings basclines. MARBLE offers casy-to-use extendable, and reproducible suite for the community, with clear statements on dataset copyright. Results suggest that recently proposed large-scale pre-trained musical language models perform the best most tasks, with room for further improvement. The leaderboard and toolkit repository are published to promote future music AI research.
Introduction
Despite Artificial Intelligence (AI) rapid advancement in the field of art, it has not yet made significant progress in music, particularly in music understanding. To address this, researchers are studying the interdisciplinary field Music Information Retrieval (MIR) to develop a general music understanding model. MIR focuses on automatically extracting information from raw music audio [40], which enables variety of tasks such as music classification, emotion recognition, pitch estimation, and the analysis of musical features such as rhythm, melody, and harmony. Due to issues such as copyright and annotation costs, labelled music datasets are usually small, which limits the performances of
"The authors contributed equally this work.
1 "Coresponding authors. 3 "ntpss//marble-mmmsheef..
37th Conference on Neural Information Processing Systems (NeurIPS 2023) Track on Datasets and Benchmarks.

---

supervised models. Given that self-supervised learning (SSL) is useful for various tasks (e.g., NLP [23, 18. 51] and CV [41]) with limited annotated datasets, there have been works on SSL-based audio representation learning [26, 33, 32, 2, 20, 49, 588 and music pre-trained models [43, 35, 69, 62, 31, 11, 29, 53, 66, 37J. The existing benchmarks, GLUE [57], SuperGI.UF. [56], and ERASER [10] in NLP, along with VTAB [68] and VISSI [16] in CV, all play an active role in promoting the development of SSL-related research topics in the corresponding domains. However, there are only scattered and fragmented evaluations of the existing music models rather than comprehensive benchmarks, making it difficult to objectively compare and draw insights across techniques.
In the current context, the SSL music systems are evaluated with downstream task datasets, including
genre classification [26, 35, 69, 62, 31, 29, 66, 37, 34], emotion classification [35, 31, 8, 29, 37, 34], instrument classification [35, 62, 49, 37, 34], music tagging [69, 31, 8, 53, 43, 29, 66, 37, 34], key detection [31, 8, 29, 37, 34], music detection [49], beat tracking [34] and cover song detection [66]. Existing works usually conduct evaluations with different experimental setups, and few of them explore sequential tasks such as beat tracking and source separation Although in similar domains, SUPERB [65] and HEAR [54] are proposed to facilitate unified analysis of the learned representations of speech and sound events, the distribution of musical audio is significantly different. Thus, there is an urgent need to construct comparable, extensive, and easy-to-use benchmarks to enhance the development of music SSL.
In this paper, we propose a Music Audio Representation Benchmark for universaL Evaluation (MAR- BLE) to address this problem. MARBLE aims to examine the full spectrum of model capabilities. and thus proposes a taxonomy adapted from Dai et al. [9] to categorise MIR tasks, including acous- tie, performance, score, and high-level description. The four-level hierarchy aligned to musician consensus serves as a guideline to further organise the datasets and helps to identify a diversified set of downstream tasks. We select popular tasks in the (now defunct) Music Information Retrieval Evaluation eXchange (MIREX) Challenge", and use the corresponding public datasets with limited annotations. As demonstrated in Tab. 1, the current version of MARBLE contains 18 downstream tasks, spread over 13 task categories on 12 publicly or commercially available datasets. Except for the common classification tasks, we also integrate the missing piece of the puzzle- sequence labelling tasks that require frame-wise prediction, including source separation and beat tracking. The datasets used in MARBLE are ensured dasy-to-accesss all datasets are available for download directly from the official repository or an external website for downloading a specific version.
In addition, we design a unified protocol and build tool-kits to evaluate the generalisation ability of the models. In MARBLE protocol, the models are regarded as backbones to provide universal representations for all tasks, and task-specific prediction heads are concatenated to further trained under unconstrained, semmiccosttrinee, and constrained settings, which is defined by whether the training hyperparameters are restricted and whether the backbone model is frozen (cf. $ 3.2). The evaluation suite provides codes for dataset preprocessing and examples of evaluating existing popular SSL models in the benchmark. We select representative music SSL models as our baselines (cf. $ 3.1) and release the evaluation results at our publicly available leaderboards" as reference.
Our key contributions are listed as follows: (1) providing a diversified music understanding benchmark with well-defined taxonomy of the MIR tasks; (2) incorporating and organising wide range of datasets to facilitate comprehensive music model evaluation; (3) designing a unified assessment protocol and building corresponding evaluation suites for processing, training, and benchmarking.
2 Benchmark Tasks
As demonstrated in Tab. 1, we collect datasets in MARBLE to provide the community with a standard, easy-to-use benchmark for various tasks covering all aspects of music. Generally, music processing involves discriminative and generative tasks. The discriminative tasks either classify or regress musical recordings as a whole or use a seq2seq model to make Trame-by-fraie decisions on entire sequences. The generative tasks include audio synthesis and music composition. For the initial release of MARBLE, we focus on discriminative tasks, and generative tasks are currently outside our scope. The task collection is guided by the principles of (1) receiving high level of interest
$ 'nttps://wws music-ir otg/mirex/eeaeoeeoo wiki /MIIEE//IE..... HOME
'Considering potential legal constraints, MARBLE allows to submit results on the tasks partially (e.g., tasks commercially available datasets) for the future participants.

---

in the MIR community. (2) having publicly available datasets allowing everyone to participate, and (3) limited labelled data effectively measure the universality the model. Four aspects of music are studied through 18 proposed tasks: High-level description tasks including key detection, music tagging. classification gender, and emotion recognition; Score-level tasks including estimating the pitch of a musical note, tracking beats, extracting melody, estimating the chords, and transcribing the lyrics; Performance-level tasks including detecting musical ornaments or techniques; and Acoustic- level tasks including singer identification, instrument classification, and source separation that focus more on raw audio information.
Table 1: The Dataset, Commercial License, and Prediction Head of Each Task Used for the MARBLE Benchmark. SDR refers Source-toddisortionn Ratio.
Prediction Type
Exaluation Metries
Commercially Avuilable
Taxonomy
Task Type
Task & Annotation
Key Dotection
Giantsteps key [25]
Multi-class
Weight Score [45] RÕC-AUČ & PR-AUCIPP
Yes
MagunTagAATnn [28]
Mulii-label
Music Tagging
MTG 5o [7]
label
ROC-AUC & PR-AUCIAP
GTZZAN [53]
Multi-class
Accuracy
High-level Description
Genre Classification
Genre [7]
Malti-lahel
ROC-AUC PR-AUC/AP
Ememusic [52]
Regression
R2 & R2f0in
Emotion Detection
MTG MoodTheme 171
Multi-label
ROC-AUC & PR-AUC/AP
Score-level
Pith Classification
Nsynhh [14]
Multi-class
Accuracy
Yes
Beat Tracking
GTZAN Rhythm [55]
Seg2Seg, Binary-class F-measure (Threshold 20ms)
Melrdy Extraction
MelodyDB
Seq2Ses, Multi-chiss
Accuracy
Chord Esiimation
GuitarSei [63]
Seg25eq, Multi-class
different mir
MIT
Lyrics Transcription
Mullam2.0
Seu2Seq. Multi-class'
CER, WER
Jamendo [12]
Seg2Soq, Multi-class
CER, WER
MIT
Performance-level
Vocal Technique Detection VacalSet [61]
Multi-class
Accuracy
Yes
Singer Identification
VocalSer [61]
Multi-class
Accuracy
Yes
Nsymth [14]
Multi-class
Accuracy
Yes
Acoustic-level
Instrument Classification
MIG Instrument
Multi-label
ROC-AUC & PR-AUC/AP
Source Separation
MUSDBIS [46]
Seq2Soq. Regression
SDR
2.1 High-level Description Tasks
Key detection involves predicting the scale and key pitch levels of a song. MARBLE solves this task using the Giantsteps [25] and a subset of the Giantsteps-MTG--key dataset [27]. Giantsteps dataset contains 604 songs and is taken as our dedicated test set. Additionally, we leverage a subset of the Slantsteps-MTG-keee
dataset, which contains 1077 music pieces with single-key annotations, for training and validation. Since no standardised split is available for Giantsteps-MTG, we adopt the dataset split strategy employed in [8]. Both datasets contain minutes of electronic dance music covering all 12 pitch classes in major and minor, resulting in a 24-class classification task. For performance evaluation, we employ accuracy with an error tolerance metric, a weighted score metric. This metric grants partial credit for reasonable errors, such as predicting relative secondary keys when the primary key is the ground truth [45].
Music Tagging refers to assigning predefined set of tags to a given song. These tags encompass various aspects such as genre, instrumentation, mood, and tempo (e.g., fast), making music tagging somewhat overlap with genre classification, emotion recognition, and instrument classification. To conduct our study, we utilise two extensive datasets: MaagnTaagAool
(MTT) [28] and JaG-nnmendo (MTG) [7]. The MTT dataset comprises 30-second audio clips with manual annotations for tags. consists of 25.9 clips, amounting to a total duration of 170 hours. For MARBLE, we use the Top50 tags, and adopt a conventional (12:1:3) training, validation, and test split, aligning with all baseline approaches' practices. Besides, the MTG dataset contains 55k clips, corresponding nearly 2k hours of music. As the audio clips in this dataset may exceed 30 seconds in length, we compute multiple embeddings using a sliding window of 30 seconds and then average them to obtain an overall embedding representation. While both datasets encompass large number of tags, we follow the customary to limit the vocabulary to the 50 most common tags in each dataset. The evaluation metrics employed for this task are the macro-average of all tag ROC-AUCs (receiver operating characteristic
3

---

area under the curve) and the average precision (AP) / PR-AUC (precision recalll area under the curve). These metrics provide comprehensive insights into the model's performance across all tags.
Genre classification aims to assign each song the most suitable genre label. This study uses two distinct datasets: GTZAN [55] and MTG-Genre. GTZAN consists of 30-second audio clips from 10 genres, making it suitable for a multi-class classification task. To assess the performance of this dataset, we report the accuracy metric. To ensure consistent evaluation, we utilise the "ail-filtered" split as described in [24] for GTZAN. The filtered dataset comprises 930 audio tracks corresponding to approximately hours of music. Besides, MTG- Genre, derived from MTG-Jamenel contains 55k tracks but focuses solely on 95 genre tags, resulting in multi-label classification problem. We employ the ROC and AP metries to evaluate the performance of MTG-Genre.
Emotion Recognition in music aims to determine the emotional content of music pieces. In our study, we utilise two distinct datasets evaluate the performance of emotion recognition: Emomusic [52] and MTG-MoodThenne annotated with valence and arousal scores. The valence represents the positivity of emotional responses, while arousal indicates emotional intensity. The official evaluation metrics for this dataset the determination coefficient (r²) between the model's regression results and human annotations of arousal and valence [52]. During inference, we split the 45-second clips into 5-second sliding windows and computed the average prediction probability as the final prediction. Since no standard dataset split is available for Emomusic, we adopt the same partitioning as [8]. It is important to note that direct comparison of the SoTA model's results with the benchmark may be challenging due to the different dataset splits. Additionally, we utilise MTG-Mood Theme,, subset of MTG-Jamend consisting of 18.5k audio tracks annotated with 59 human emotion labels. This is a multi-label task with ROC and AP as evaluation metrics.
[7]. Emomusic contains 744 pieces of 45-second music clips and
2.2 Score-level Tasks
Pitch Classification in Music (Monophonic) involves determining the appropriate pitch category for given audio sample, ranging from MIDI note numbers 0 to 127 on a semitone scale. In this study, we perform pitch classification using the Nsynth dataset [14] within the music information retrieval benchmark. It comprises 340 hours of music, with each excerpt lasting 4 seconds. Since the audio recordings in this dataset are monophonic, the pitch classification task is formulated as a 128-class classification problem, covering all possible MIDI pitch categories (fundamental frequencies from 8Hz to 12.5kHz). The evaluation metric used for this task is the accuracy achieved across all audio clips.
Beat Tracking determines the presence of a beat and a downbeat in each frame of a given music piece. In this benchmark, we only focus on beat tracking, making it a binary-llasiffccation task?, An offline approach is employed for beat tracking, allowing the model to utilise frame-level information during inference. The model generates frame-by-frane predictions at a specific frequency, which are then post- post-processed using a dynamic Bayesian network (DBN) [6] implemented with madmom [5] to obtain the final result. The GTZAN Rhythm dataset [36] is used in this study. The dataset provides frame-level annotations for each music clip in GTZAN. To enhance model performance and ensure a fair comparison with the SOTA model, adjacent frames of each beat label are also labelled beats using a label smoothing technique commonly employed in beat tracking. The model is evaluated using the f f_measure metric implemented in _ir_eval [45]. A prediction is considered correct if the difference between the predicted event and the ground truth does not exceed 20ms. It is important to note that while some models may have been trained on other datasets, the GTZAN-train subset is used as the training set, and GTZAN-test is used as the test set for all MARBLE submissions.
Chord Estimation is to recognise the temporal music chord of given piece of music. We imple- mented this task as a 421-class classification including 35 types of chords on 12 different root notes, and none. More information on the chord vocabulary can be found in Appendix D. The probing model consists of an MLP with a hidden size of 512 and concludes with a fully connected output layer. There is no post-processing involved from frame-level prediction to event-level. The predictions aligned with the token rate of the pre- trained model. Performances are measured as by 6 measures in SOTA model [22] including root, majmin, mirex, thirds, triads, and sevenths. We have
'Due to the limitation of time and the size of the dataset, tracking the time signature (e.g., 4/4 metre) and downbeat is deferred future versions with other datasets.

---

added two additional evaluation metrics inspired by MIREX: majmin_ing inv and sevenths_ing to assess the performance of chord recognition at the level of inversions. The metrics are implemented by mir mir_eval[45]. We use the GuitarSet [63] dataset for this task. The dataset comprises 360 excerpts, each around 30 seconds, recorded by players performing 30 lead sheets in two versions (comping and soloing) across 5 styles, 3 progressions, and 2 different tempos. Four audio versions are provided, and we selected the "mix" (a monophonic mixture of the original 6-channel file) for our audio collection. The dataset offers two types of chord annotations for selection, and we chose "performed chord" as our primary annotation and used "instructed chord" to substitute specific colour chords. We divided the audio into 5-second segments and allocated 5 singers into the validation and training sets while designating one singer for the test set. Out of the 5 singers, we allocated 30% to the validation set and 70% to the training set. Segments from the same song assigned to the same set, for instance, all 5-second segments from a particular song are grouped into the training set.
Melody Extraction is to recognise the pitch of melody for a given music, typically pop songs. Adhering to the methodology in [59], we divided the frequency spectrum between 0 and 8000 Hz into 360 bins, treating our task as a classification problem. The probing model consists of single-layer bidirectional LSTM with hidden size of 512, followed by a linear layer. The predictions are aligned with the token rate of the pre-trained model, which is then resampled to match the label rate using the nearest interpolation. Performances are measured as the Overall Accuracy metric from the mir _eval [45] library. We use the MedleyDB [4] dataset for this task. It has 108 full tracks, collectively lasting 7.3 hours. All tracks come with three types of melody labels given at intervals of roughly 5.8 ms. For our study, focused the second annotation, which indicates the fundamental frequency of mixed stems. For data splitting, we followed the partitioning strategy of [59] giving 67, 15 and 26 tracks for training, validation and testing sets respectively which was achieved after omitting a redundant track from the test set in the popular split.
Lyries Transcription aims to identify the linguistic content in audio recordings of singing. In MARBLE, we focus on the evaluation of multilingual lyrics transcription, an aspect that has been under-explored in the field of lyries transcription. We perform the task using the MulJam dataset [70], which comprises 6031 songs in 6 languages: English, French, Spanish, Italian, German, and Russian. This dataset offers a rich repository of around 153k lines with lyrics annotations. The training, validation and testing sets contain 147k, 3k and 2k lines, respectively. We set a standard train/valid/test splitting and re-labelled the Mullam dataset as MulJam2.0 with more human annotation. More information can be found in Appendix E. We also use Jamendo [12] as a test set which includes English, French, German and Spanish pop songs. There are 20 songs for each language and the dataset comes with line-level human annotation for lyrics. In line with recent literature [15, 42], the backend adopts hybrid CTC/Attention architecture design [60]. Given the task's complexity and the necessity to capture long-term dependencies, we use a transformer with 3 encoder layers and 3 decoder layers. The output from the encoder is further processed by a fully connected layer to map it to the target dimension for the CTC loss computation [17]. We also calculate a sequence- soquence-to-sqqueee sossequence (S2S) loss between the output from the decoder and the true lyrics text. The final loss is balanced combination of the CTC loss and the S2S loss. For validation and testing, we employ beam search on the transformer decoder to iteratively select the best predictions. Additionally, a transformer language model is trained from the same data split to incorporate language knowledge at test time. Performance evaluation is conducted using Character Error Rate (CER) and Word Error Rate (WER). Different from all the metrics in other tasks, WER and CER values are the less the better.
2.3 Performance-level Tasks
Vocal Technique Detection task involves identifying different singing techniques within an audio clip. For this task, the MARBLE benchmark utilises the VocalSet dataset [61], the sole publicly available dataset specifically designed for studying singing techniques. This dataset comprises recordings of 20 professional singers (9 female and 11 male) performing 17 distinct singing techniques in various contexts, amounting to total duration of 10.1 hours. Given that the audio clips are segmented into 3-second intervals, the task focuses on determining the type of technique (e.g. Vibrato, Straight) rather than the precise start and end times. To evaluate the performance of models, we employ Accuracy as the evaluation metric. We use a subset of 10 different singing techniques used in Yamamoto et al. [64], which contains 15 singers in the training and validation set, and 5 for the test set. Since there no predetermined division between the training and validation sets, assign 9 singers to the training
5

---

set and 6 singers to the validation set. It is important to note that all 3-second segments originate from the same audio recording file within the same part of the split, such as being exclusively part of the training set. Detailed data partitioning can be found in our provided code.
2.4 Acoustic-level Tasks
Instrument Classification refers to the multi-label or multi-class identification of instruments present in given audio recording. In the MARBLE benchmark, we utilise two datasets: Nsynth and MTG- instrument. The Nsynth dataset comprises 306,000 audio tracks, each corresponding to one of 11 different instruments. The evaluation metric for this dataset is accuracy. On the other hand, MTG-instrument is subset of MTGG-Jmmndo, containing 25,000 audio tracks and instrument lags. Each track can have multiple instrument tags and is evaluated based on ROC and AP.
Singer Identification involves recognizing the singer or vocal performer from an audio recording. In previous work on Singer Identification using the VocalSet dataset [61]. different splits are employed. For the MARBLE benchmark, we randomly split the dataset into training, validation, and test sets, maintaining a ratio of 12:8:5, All sets contain the same 20 singers. The specific data divisions can be found in the provided code.
Source Separation aims to separate different components of a music recording, such as vocals, drums, bass, and others. In MARBLE, adopt the widely-used MUSDBI8 dataset [46] for this task. MUSDB 18 consists of 150 full-length music tracks, totalling approximately 10 hours of audio and multiple isolated stems. Our training set consists of 86 tracks, the validation set contains 14 tracks, and the evaluation set comprises 50 tracks, following the official MUSDB 18 setting. During training, we randomly sample 6-second segments and apply random track mixing for data augmentation. Due to the complexity of this task, we utilise the baseline architecture from the Music Demixing Challenge (MDX) 2021 [38]. This architecture consists of three linear layers and three bi-directional LSTM layers. The optimization is performed by directly computing the 12-loss between the predicted and ground-truth spectrograms. The evaluation metric for this task is the Cource-to-Distortinn Ratio (SDR) as defined in [38], which is calculated as the mean across the SDR scores of all songs.
3 Evaluation Framework
We aim to explore the generality and standardisation of the framework. Therefore, we freeze the parameters of the pre-trained model extract pre-trained features as fixed depth embeddings fed to each downstream task-specific prediction head. This allows for as lightweight a solution as possible for all tasks, thus testing whether the representations are easily reusable across different downstream tasks. We describe pre-trained baseline models, downstream models, and protocols in the following sections.
3.1 Pre-trained baseline systems
The audio pre-training models explored in this paper are summarised in Table. 2. Note that we do not cover models designed entirely for speech or not open source models. We also examine all the open-source SSL systems specifically designed from music audio, in total 9 different versions of 7 pre-trained features; see Table. for information on pre- pre-trained models.
MusiCNN [43] is a convolutional model pre-trained on the music audio tagging task using the MSD dataset [3]. We the default configuration of the method, which is to concatenate the mean pooling of the CNN features for a 3-second input with the output of the maximum pool.
Contrastive learning of musical representations (CLMR) [53] leverages a 9-layer 1-D convolu- tional kernel as the feature extractor, employing a number of data augmentation, and is trained on both MSD and MTT. Both are trained with a contrastive learning approach. The model extracts an embedding every 2.69 seconds.
Jukebox [1 1] is music generation model trained using codified audio language modelling (CALM). trained on 1.2 million private songs, and the size of the training set is difficult to estimate the exact number of hours. However, assuming an average song length of 3-6 minutes, the total length could be 60k-120k hours, which is large and diverse allow Jukebox to learn patterns and structures of different musical genres and styles. We use the same mid-layer representation [8] to improve

---

Table 2: Information of Baseline Systems.
MusiCNN
MAP-MERT-10
MAP-MERT-vI
Method
CLMR
Jukebox
MULE
MAP-Music2Vee
MSD-big
base base-public
base
large
3 Cony,
22 22-Conv
7-Conv,
7-Conv,
7-Conv,
7-Conv,
7 7-Comy
Network
CNN
Conv
36-Tra
2- 2-Trans
12-Trans
12-Trans
12-Trans
12-Trans
12-Trans
#Params
SM
2.5M
5B
62.4M
95M
95M
95M
95M
330M
Input
Tog-mel
waveform
waveform
log-mel
waveform
waveform
waveform waveform waveform
Stride
2.69s
23.78s
20ms
20ms
20ms
13.3ms
13.3ms
Context Length
2.69s
5s
Data (hour)
10-20k
1.7k
60-120k
117.5k
Ik
1k
0.9%
17k
160k
Pre-training
Music Tagging
Contrastive Learning
Contrastive Leariing
MLM Boostrapping
MIMM Clustering
MLM Clustering
MLM Clustering
MLM Clustering
CALM
Task
computational efficiency. Unlike other representations that run on short context windows, JUKEBOX is trained on a long window of 8192 sample points (23.78 seconds) of audio. We use the same strategy as [8] to extract the audio features the downstream dataset.
MULE Musicnet-ULargee [37] is a SSL system based SF NFNet-FO [58], SlowFast Normalizer- Free ResNet. It combines a SlowFast (SF) part (including a slower pathway that captures spatial information and a faster pathway that captures temporal information) with a more efficient and scalable variant of the Normalizer-Fire ResNet (NFNet). MULE is contrastively pre-trained on the whole MusicSet dataset [37] and provides promising results on classification tasks. The model extracts an embedding with 3-second window length and 2-second hop length.
MAP-Music2Vec [31] is supe-vupe learning (SSL) model specifically based on bootstrapping mask prediction pre-training strategy. It consists of two main components: the student and teacher models. Both share the same architecture with 12 transformer layers, with the teacher model's parameters being exponential moving averages of the student model's parameters. The student model takes masked input, and during training, aims learn deep features from the teacher model based on the output of the unmasked input. Specifically, it computes the average of the top layers of the Transformer's output in the teacher model. To train the MAP-Music2Vee model, a private dataset comprising approximately 1,000 hours of music data was used. The input length of the embeddings capture essential features of the music data and can be utilised for various downstream tasks, including sequential tasks such as source separation and beat tracking.
model is set to 30 seconds, producing 50 embeddings per second. These
MAP-MERT-v0, also referred to as MERT-95M K-means in the work by Li et al. [29]. is a pre-trained model built upon the speech self-supervised learning (SSL) system HUBERT [20]. It undergoes pre-training for masked prediction, with discrete heeudollabels obtained from K-Means clustering on music features. The pre-training task of MAP-MERT-VO involves two pseudo-labels based on logmel and Chroma, along with a CQT reconstruction task that emphasises pitch information. Two versions of the MAP-MERT-VO model are included: MAP-MERT-VO", trained on a private dataset of 1,000 hours, and MAP--MERT-OO-pbbllE MAP-MERT-VO model is set seconds, generating 50 embeddings per second. This design facilitates fine-tuning for sequential tasks, enabling efficient and effective processing of music data.
trained on Music4ALL [50]. The input length of the
MAP-MERT-vl encompasses two variants: (AAP-)MERT-VI-basse and (MAP-)MERT-V large" These models, also known as MERT-95MMMVVQAAA and RERT-330M VAE in the work by Li et al. [29], employ EnCodec, a pre-trained discrete deep feature, as a replacement for the K-means feature. This modification facilitates the scaling up of the model. Similar to MAP-MERT-VO, the input length of the MAP-MERT-v] models is seconds, but they produce 75 embeddings per second.
hhatpe:/hhuginngfaee.oooo woteeeeeecoom---///----- -/
phtrpe:/hhuginggfaeeccooo/e http://nnnnaaa.ccccaaaa/////E//e - -00-vb1i--
"Mccpee/hhugghhgaaae..c//aa/
"I "actpee//uuggiingtccoooo// com -oMea/ee/-o.ee--aae/-/e--e-e- -1-30M
7

---

This configuration enables effective fine-tuning for sequential tasks, making the models suitable for processing music data in variety of upplications.
3.2 Downstreams and Training Strategies
To evaluate the relevance of representations for downstream MIR tasks, we design evaluation frameworks: the unconstrained track, semi-constrained track and the constrained track. In the unconstrained track, researchers invited to submit their systems with any hyperparameter and structure configuration, including the option to fine-tune pre-trained models. This track encourages flexibility and exploration, enabling researchers to investigate wide range of approaches. On the other hand, the semi-constrained track requires the submissions to use frozen pre-trained backbones. search
Finally, the constrained track employs standardised setting with limited space (cf. Appendix A), where frozen models are used as feature extractors for training a one-layer 512-unit MLP (or 1/3-layer 512-unit LSTM for melody extraction or source separation, or 3-encoder- 3-decoder layers transformer for lyrics transcription) on each task. In addition, we set a computational wall for MARBLE. The systems need to finish each task within week on our machine equipped with single consumer GPU (RTX3090). By offering these three evaluation tracks, we aim to provide researchers with comprehensive platform assess the performance and relevance of representations in MIR tasks, encouraging innovative approaches and fostering ad adaacemenns in the field. For the same task with uniform dataset, if there are different evaluation metrics (e-g., emotion regression, source separation, and tagging), we will average the two evaluation metrics. We select the checkpoints regarding to the best validation results for final testing and submission.
Table 3: Performances of Baselines Evaluated on MARBLE with constrained settings (1/3). We include previous SOTAs for reference. Note that MARBLE imposes strict constraints downstream structures and hyper-parameter search spaces, while previous SOTAs are not subject to such limita- tions. Best scores MARBLE are bold, and best scores among all systems underlined.
Datamtt
MTT
GS
GTZAN
GTZAN
EMO
Noyuth
Noy Mach
VucalSut Tech
ViscalSei
Taok
Genre
Whythm
Eanotion
Instniment
Singer
Friend
MusCNN (43]
CLMR [59]
800.5
3B
MULE
MiP
MAD-MERT.- (30)
MAR-METT-1-9GM
91.0
74.
26.3
MAP-MM3ET-1111000 (19]
91.1
39.5
61.7
77.6
59.0
75.8
72.6
94.4
26.9
87.1
Previcus SOTA
92.0 [31]
41.4
(273
83.5
90.6 (19)
61.7
72.1 [8)
78.2
09.2
06.6 [64]
80.3 (19)
Table 4: Performances of Baselines Evaluated on MARBLE with constrained settings (2/3), Note that denote the scores of Jukebox-5B MTG tasks with asterisks(*), because it hit the computational wall of MARBLE, meaning that the system was unable to complete the corresponding task within week our machine equipped with single consumer GPU (RTX3090).
Datneet Task
MIG
MTG
MTG
MYG Top50
MUSDE
Instrmmci
MoodTheme
Gase
Suunaz Seueation
SDR
SUggeer
MuSCCNN
17.2
17.5
$2.0
27.5
CLMR
17.0
Jukehax
78.5
23.0"
15:34
20.57
19.2
20.4
Music/ves
19.2
83.0
5.5
MAP-MERT-OOP00 95- [30]
76.5
18.7
75.9
13.7
$6.9
18.5
82.5
25.6
3.6
3.0
MAP-MERC MERT oo-biiluulli (30)
77.5
19.6
76.2
13.3
$7.2
18.5
83.0
25.9
5.5
5.5
3.7
3.0
MAP MERT vi 95M (29)
77.5
19.4
76.4
13.4
$7.1
18.5
83.0
29.0
5.5
5.5
3.8
3.1
MAP MERT vl. 3300 (29)
75.1
19.8
76.5
14.0
56.7
18.5
83.4
29.9
5.3
3.0
Presium SOTA
2022
78.6
16.1
20.1
84.3
12.1
9.1
4 Results and Discussion
According to Table 3, 4 and Fig 1, all pre-trained baseline representations on MARBLE have achieved decent results. Despite strict constraints downstream structures and hyper-paramete search spaces,

---

Table 5: Performances of Baselines Evaluated MARBLE with constrained settings (3/3). The
overall average scores are calculated on the systems applicable to all tasks.
MilseyBB Meledy
Maljan Tyriss
GuitarSat
Tyrics
mrijrsie
majoiia
MAP Nas (31)
36.1
56.4
$7.5
55.7
89.6
13.3
11.1
10.4
10.4
10.4
11.1
9.4
044
MAP MERE IAMM 30g
6010
57.6
52.3
51.8
87.6
45.3
38.4
16.7
16.6
16.5
17.5
30.3
20.11
SEEE-EEEE-AAE 1095M puble (30)
36.1
52.5
82.7
52.6
55.2
40.1
38.7
36.6
36.6
36.4
57.5
20.0
23.6
MAR-BETT-
10.8
16.7
65.39%)
20.5
54,844]
44,144]
33.1
23,1221
CLMR Jukebox-based
MULE
Beat Tracking (GTZAN)
Genre (GTZAN, MTG)
MusicZVec
MERT-V1-95M
Previous SOTA
Emotion (EMO, MTG)
Key
(GS)
Instrument (Nsynth, MTG)
Multi-Label Tagging (MTT, MTG)
Pitch (Nsynth)
Source Seperation (MUSDB)
Vocal Tech. Det. (VocalSet)
Singer Id. (VocalSet)
Figure 1: SSL Baselines Compared to previous SOTA. The performances tasks are merged according to the task types demonstrated in Tab. Results not applicable are to O.
Avg. Store
80.0
80.0
78.0
MAX Wi--
MIFD
60.0
Score
Parformance-ievel High-ievel
Acoustio-level
50.0
70.0
60.0
58.0
Training Data/ A-Hour (Logg Scall)
Training Data / A-Hoour (Log Scale)
(a) Scores at Acoustic-level and lcore-level.
(b) Scores at Performance-lose and High-level.
Figure 2: Results Analysis Regarding Training Data Size. Since some models to the sequence labelling tasks, the performances of source separation and beat tracking tasks are excluded acoustic-level and score-level average score calculation correspondingly. The radii of the scatter points are isometrically log scaling with the parameter sizes.
applicable
they are able to approach, not surpass, the previous state-of-the-ars (SOTA) in many tasks. For instance, the best performance NSynth Pitch classification have achieved up 94.4% accuracy. Nonetheless, the majority of tasks are still far from being solved, including music tagging and source separation tasks. Notably, the performance on MUSDBIS 18 is merely half of the previous SOTAs.

---

The MAP family achieves balanced results, successfully performing tasks including sequence la- belling, which other models fail to accomplish (as they do not provide frame-level representations or are too cumbersome to train). This series of models excel at multiple taxonomy levels. On certain tasks, MAP-MERT' achieve results close to or surpass the previous state-of-the tagging tasks are dominated by Jukebox-5B and MULE. Jukebox may benefit from its massive parameter size and generative modelling of detailed information, as well as the introduction of metadata during the pre-training period. Conversely, MULE benefits from its proprietary large-scale, high-quality dataset, MusicSet, and the highly discriminative representations learned by contrastive pre-training.
However, music
Based on Fig. 2a and 2b, excluding sequence labelling tasks (as some baselines do not support them), we observe general trend: as the volume of data and the size of model parameters increase, the performance of tasks across four levels correspondingly improves. The choice of pre-training method and model size significantly influences the performance. For instance, MAP-Music2 Vec-95M, utilizing only 1k hours of data for self-supervised learning. outperforms both supervised pre-trained MusiCNN-SM M and contrastive pre-trained CLMR-2.5M on the same scale of data. More analysis could be referred to Appendix B.
5
Conclusion
In this work, we introduce the Music Audio Representation Benchmark for universal. Evaluation (MARBLE) as comprehensive benchmark for evaluating pre-trained music features. It encompasses hierarchy taxonomy that covers acoustic, performance, score, and high-level description levels, and utilises publicly available datasets for 18 MIR tasks. We establish a standardised preprocessing and data splitting protocol, along with unified evaluation framework, ensure fair and reproducible assessment. We report the results of all 9 open- sournedd pre-trained models developed on music recordings, showcasing their performance across multiple tasks. The results demonstrate that several pre-trained models achieve comparable or even superior performance to the state-of-the-ar models on various tasks within MARBLE. However, there is still ample room for improvement, particularly in music tagging and source separation. With the release of the toolkit, we hope to facilitate future research by providing easy access, reproducibility, and fair comparison of SSL pre-trained models for music understanding. We encourage engagement from researchers in the audio and Al communities to contribute to the advancement of representation learning for music information retrieval.
Discussion and Future Work
Our benchmark has some shortcomings that can be further improved. To begin with, some of the tasks, such as beat tracking and piano transcription, typically use multiple evaluation metrics, but we only include one two for each of the tasks due to the copyright issues preventing many datasets from being publicly available, lack of standard pre-processing or maintenance, and the limitation of time. Although the selected metric is fundamental and good indicator, an average of all the metrics might be a better choice. Besides, some of the datasets are not sufficient for a single task. For example, the GTZAN dataset does not have a commercially-avaiie less than 10 hours of music recordings, making the evaluation more subject to bias. We will include more commercially-availaae MIR tasks that lack a common dataset currently, such as cover song detection and beerymb-- In the future version, we will include more datasets and tasks. Last but not least, MIR on symbolic music is not included in the first version of our benchmark as well.
license, and it only includes
larger datasets on the same tasks. Moreover, we do not include some
Apart from the traditional MIR tasks, some interesting tasks deserve more attention for benchmark development in the computer music and AIGC communities. With the benchmark and pre-trained models in MIR, developing There may not exist a perfect solution on the subject metrics for music generation to build a bench- mark: otherwise, composing musical art will simply search for the waveform with the highest scores. But one can expect such a benchmark can be helpful for the music industry or music education to preclude some bad music generation. Besides, multi-modal approaches that combine music audio with symbolic music and language (e.g., lyrics and music description) also deserve benchmark.
evaluation score on music generation and synthesis might be possible.
10

---

Acknowledgements
We would like express sincere gratitude to our friends Anqiao Yang and Wei Fan for their invaluable support during the writing of this paper.
Ruibin Yuan is funded by Thsee Research Scheme (T45-205/21-N) and Early Career Scheme (ECS-HKUST222013222 322), Research Grants Council of Hong Kong. Yinghao Ma and Jiawen Huang research students the UKRI Centre for Doctoral Training in Artificial Intelligence and Music, supported by UK Research and Innovation (Grant Number: EP/S022694/1), Yizhi Li is fully funded by an industrial PhD studentship (Grant Number: 171362) from the University of Sheffield, UK. Emmanouil Benetos is supported by RAEng/Leverhulme Trust Research Fellowship LTRF2223-19, 106.
We acknowledge IT Services at The University of Sheffield for the provision of services for High- Performance Computing. This project also made use of time Tier 2 HPC facility JADE2, funded by EPSRC (EP/T022205/1)
Besides, we would like to give special thanks to the following researchers or musicians on the lyrics labelling: Léo Nebel LIP6 Sorbonne Université; Nick Magal in School of Music Carnegie Mellon University; Carey Bunk, Yannis Vasilakis, Christopher Mitcheltree, Nelly Victoria Alexandra Garcia- Sicria,Sihuy,... Teresa Pelinski Ramos, Ilaria Manco, David Südholt, Jordan Shier, and Matthew Rice in Centre for Digital Music at Queen Mary University of London; Emilian Postolache in Sapienza University of Rome; as well as Wenqin Yu in the Chinese Music Institute at Peking University.

---

References
[1] Alonso-Jiménez, P., Serra, X., and Bogdanov, D. (2022). Music representation learning based on editorial metadata from discogs.
[2] Bacvski, A., Zhou, Y., Mohamed, A., and Auli, M. (2020). wav2vec 2.0: A framework for self-supervised learning of speech representations, Advances in neural information processing systems, 33:12449-1244 12460.
[3] Bertinn T., Ellis, D. P., Whitman, B., and Lamere, P. (2011). The million song dataset.
[4] Bittner, R., Salamon, J., Tierney, M., Mauch, M., Cannam, C., and Bello, (2014). Medleydb: A multitrack dataset for annotation-intensive mir research.
[5] Böck, S., Korzeniowski, F., Schlüter, Krebs, F., and Widmer, G. (2016a). madmom: new Python Audio and Music Signal Processing Library. In Proceedings of the 24th ACM International Conference on Multimedia, pages 1174-1 1178, Amsterdam, The Netherlands.
[6] Böck, S.. Krebs, F., and Widmer, G. (2016b). Joint beat and downbeat tracking with recurrent neural networks. In ISMIR, pages 255-261. New York City.
[7] Bogdanov, D., Won, M., Tovstogan, P., Porter, A., and Serra, X. (2019). The mtg-jamendo dataset for automatic music tagging. In International Conference Machine Learning. ICMI.
[8] Castellon, R., Donahue, C., and Liang, P. (2021). Codified audio language modeling learns useful representations for music information retrieval. arXiv preprint arXiv:2107.0567.
[9] Dai, S., Zhang, Z., and Xia, G. G. (2018). Music style transfer: A position paper. arXiv preprint 1rXiv:800300684/
[10] De J., Jain, S., Rajani, N. F., Lehman, E., Xiong, C., Socher, R., and Wallace, C. (2020). Eraser: A benchmark to evaluate rationalized nlp models. Trunsactions of the Association for Computational Linguistics.
[11] Dhariwal, P., Jun, H., Payne, C., Kim, J. W., Radford, A., and Sutskever, I. (2020). Jukebox: A generative model for music. arXiv preprint arXiv:2005.00::: 2034.0041...
[12] Durand, S., Stoller, D., and Ewert, S. (2023a). Contrastive learning based audio to lyrics aligmment for multiple languages. In /CASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1-5. IEEE.
[13] Durand, S., Stoller, D., and Ewert, S. (2023b). Contrastive learning-based audio lyrics aligmment for multiple languages. In ICASSP 2023 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1-5.
[14] Engel, J., Resnick, C., Roberts, A., Dieleman, S., Norouzi, M., Eck, D., and Simonyan, K. (2017). Neural audio synthesis of musical notes with wavenet autoencoders. In International Conference on Machine Learning, pages 1068-1077. PMLR.
[15] Gao, X., Gupta, C., and Li. H. (2023). Polyscriber: Integrated fine-tuning of extractor and lyrics transcriber for polyphonic music. IEEE ACM Trans. Audio Speech Lang. Process., 31:1 31:1168-1998
[16] Goyal, P., Mahajan, D., Gupta, A., and Misra, I. (2019). Scaling and benchmarking self- supervised visual representation learning. In Proceedings of the IEEE/CVF International Confer- ence Computer Vision, pages 6391-6400.
[17] Graves, A., Fernández, S., Gomez, J., and Schmidhuber, (2006). Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks. In Proc. ICML, volume 148, pages 369-376. ACM.
[18] Gururangan, S., Marasović, A., Swayamdipta, S., Lo, K., Beltagy, I., Downey, D., and Smith, A. (2020). Don't ' stop pretraining: Adapt language models domains and tasks. arXiv preprint arXiv:2004.100
12

---

[19] Heydari, M., Cwitkowitz, F., and Duan, Z. (2021). Beatnet: Crnn and particle filtering for online joint beat downbeat and meter tracking. arXiv preprint arXiv:2108.03:: 03576.
[20] Hsu, W.-N., Bolte, B., Tsai, Y.HH. H., Lakhotia, K., Salakhutdinov, R., and Mohamed, A. (2021). Hubert: Self-supervised speech representation learning by masked prediction of hidden units. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 29:3451-3460.
[21] Huang, Q., Jansen, A., Lee, J., Ganti, R., Li, J. Y., and Ellis, D. P. (2022). Mulan: A joint embedding of music audio and natural language. arXiv preprint arXiv:2208. 12415.
[22] Jiang, J., Chen, K., Li, W., and Xia, G. (2019). Large-wocabulary chord transcription via chord structure decomposition. In ISMIR, pages 644-651.
[23] Kenton, J. D. M.-W. C. and Toutanova, L. K. (2019). Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of NAACI-HLT,, pages 4171-4188
[24] Kereliuk, C., Sturm, B. L., and Larsen, (2015). Deep learning and music adversaries. IEEE Transactions on Multimedia, (711))2059-2077
[25] Knees, P., Faraldo Pérez, Á., Boyer, H., Vogl, R., Böck, S., Hörschläger, F., Le Goff, M., et al. (2015). Two data sets for tempo estimation and key detection in electronic dance music annotated from user corrections. In Proceedings of the 16th International Society for Music Information Retrieval Conference (ISMIR); 2015 Oct 26-30; Málaga, Jpain.[Mülagaj, International Society for Music Information Retrieval, 2015. p. 364-70. International Society for Music Information Retrieval (ISMIR).
[26] Kong, Q. Cao, Y., Iqbal. T., Wang, Y., Wang. W., and Plumbley, M. D. (2020). Panns: Large- scale pretrained audio neural networks for audio pattern recognition. JEEE/ACM Transactions Audio, Speech, and Language Processing, 28:2880-2894
[27] Korzeniowski, F. and Widmer, G. (2017). End-to-end musical key estimation using a convolu- tional neural network. In 2017 25th European Signal Processing Conference (EUSIPCO), pages 966-970. IEEE
[28] Law, E., West, K., Mandel, M. I., Bay, M., and Downie, J. S. (2009). Evaluation of algorithms using games: The case of music tagging. In ISMIR, pages 387-392. Citeseer.
[29] Li. Y., Yuan, R., Zhang. G., Ma, Y., Chen, X., Yin, H., Lin, C., Ragni, A., Benetos, E., Gyenge, N., Dannenberg, R., Liu, R., Chen, W., Xia, G., Shi. Y., Huang, W., Guo, Y., and Fu, J. (2023). Mert: Acoustic music understanding model with large-scale self-supervised training.
[30] Li, Y., Yuan, R., Zhang, G., Ma, Y., Lin, C., Chen, X., Ragni, A., Yin, H., Hu, Z., He, H., et al. (2022a). Large-scale pretrained model for self-supervised music audio representation learning.
[31] Li, Y., Yuan, R., Zhang, G., MA, Y., Lin, C., Chen, X., Ragni, A., Yin, H., Hu, Z., He, H., et al. (2022b). Map-music2vec: A simple and effective baseline for self-supervised music audio representation learning. In Ismir 2022 Hybrid Conference.
[32] Ling, S., Liu, Y., Salazar, J., and Kirchhoff, K. (2020). Deep contextualized acoustic repre- sentations for semi-supervised speech recognition. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6429 222396633..1 IEEE
[33] Liu, T., Yang, S.-w., Chi, P.-H., Hsu, P.-c., and Lee, H.-y. (2020). Mockingjay: Unsupervised speech representation learning with deep bidirectional transformer encoders. In ICASSP 2020- 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6419-6423, IEEE
[34] Ma, Y., Yuan, R., Li, Y., Zhang, G., Chen, X., Yin, H., Lin, C., Benetos, E., Ragni. A., Gyenge, N., et al. (2023). On the effectiveness of speech self-supervised learning for music. International Society for Music Information Retrieval Conference (ISMIR).
[35] Manco, I., Benetos, E., Quinton, E., and Fazekas, G. (2022). Learning music audio representa- tions via weak language supervision. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 456-460. IEEE
13

---

[36] Marchand, U. and Peeters, G. (2015). Swing ratio estimation. In Digital Audio Effects 2015 (Dafx15).
[37] McCallum, M. C., Korzeniowski, F., Oramas, S., Gouyon, F., and Ehmann, A. F. (2022). Supervised and unsupervised learning of audio representations for music understanding. Ismir 2022 Hybrid Conference.
[38] Mitsufuji, Y.. Fabbro, G., Uhlich, S., Stöter, F.-R., Défossez, A., Kim, M.. Choi, W., Yu, C.-Y.. and Cheuk, K.-W. (2022). Music demixing challenge 2021. Frontiers in Signal Processing, 1:18.
[39] Modrzejewski, M., Szachewicz, P., and Rokita, P. (2023). Transfer learning with deep neural embeddings for music classification tasks. In Artificial Intelligence and Soft Computing: 21st International Conference, ICAISC 2022, Zakopane, Poland, June 19-23, 2022, Proceedings, Part I, pages 72-81. Springer.
[40] Müller, M. (2015). Fundamentals of music processing: Audio, analysis, algorithms, applica- tions, volume 5. Springer.
[41] Newell, A. and Deng, J. (2020). How useful is self-super pretraining for visual tasks? In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7345-7354.
[42] Ou, L., Gu, X., and Wang, Y. (2022). Transfer learning of wav2vec 2.0 for automatic lyric transcription. In Rao, P., Murthy, H. A., Srinivasamurthy, A., Bittner, R. M., Repetto, R. C., Goto, M., Serra, X., and Miron, M., editors, Proceedings of the 23rd International Society for Music Information Retrieval Conference, ISMIR 2022, Bengaluru, India, December 4-8, 2022, pages 891-899.
[43] Pons, J. and Serra, X. (2019). musienn: Pre-trained convolutional neural networks for music audio tagging. arXiv preprint (tt::/909.066555
[44] Radford, A., Kim, J. W., Xu, T., Brockman, G., McLeavey, C., and Sutskever, I. (2023). Robust speech recognition via large-scale weak supervision. In Krause, A., Brunskill, E., Cho, K., Engelhardt, B., Sabato, S., and Scarlett, J., editors, International Conference Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pages 28492-28511. PMLR.
[45] Raffel, C., McFee, B., Humphrey, E. J., Salamon, J., Nieto, O., Liang, D., Ellis. D. P., and Raffel, C. C. (2014). Mir A transparent implementation of common mir metrics, ISMIR, pages 367-372.
[46] Rafii Z., Liutkus, A., Stöter, F.-R., Mimilakis, S. I., and Bittner, R. (2017). The MUSDBI8 corpus for music separation.
[47] Ravanelli, M., Parcollet, T., Plantinga, P., Rouhe, A., Cornell, S., Lugosch, L.., Subakan, C., Dawalatabad, N., Heba. A., Zhong, J., Chou, J.-C., Ych, S.-L., Fu, S.-W., Liao, C.-F., Rastorgueva, E., Grondin, F., Aris, W., Na, H., Gao, Y. Mori, R. D., and Bengio, Y. (2021). SpeechBrain: A general-purpose speech toolkit, 12ix:2200.04624.
[48] Rouard, S., Massa, F., and Défossez, A. (2023). Hybrid transformers for music source separation. ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1-5. IEEE.
[49] Saeed, A., Grangier, D., and Zeghidour, N. (2021). Contrastive learning of general-purpose audio representations. In ICASSP 2021-2021 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pages 3875-3879. IEEE.
[50] Santana, I. A. P., Pinhelli, F., Donini, J., Catharin, L., Mangolin, R. B., Feltrim, V. D., Domingues, M. A., et al. (2020). Music4all: A new music database and its applications. In 2020 International Conference on Systems, Signals and Image Processing ( (/WSSIP), pages 399- 404. IEEE
14

---

[51] Sarzynska- Wawer, J., Wawer, A., Pawlak, A., Szymanowska, J., Stefaniak, I., Jarkiewicz, M., and Okruszek, L. (2021). Detecting formal thought disorder by deep contextualized word representations. Psychiatry Research, 304: 14135.
[52] Soleymani, M., Caro, M. N., Schmidt, E. M., Sha, C.-Y., and Yang, Y.-H. (2013). 1000 songs for emotional analysis of music. In Proceedings of the 2nd ACM international workshop on Crowdsourcing for multimedia, pages 1-6.
[53] Spijkervet, J. and Burgoyne, J. A. (2021). Contrastive learning of musical representations. arXiv preprint arXiv:2103.094:
[54] Turian, J., Shier, J., Khan, H. R., Raj, B., Schuller, B. W., Steinmetz, C. J., Malloy, C., Tzanetakis, G., Velarde, G., McNally, K., et al. (2022). Hear: Holistic evaluation of audio representations. In NeurlPS 2021 Competitions and Demonstrations Track, pages 125-145. PMLR.
[55] Tzanetakis, G. and Cook, P. (2002). Musical genre classification of audio signals. IEEE Transactions on speech and audio processing, 10(5):293-302.
[56] Wang, A., Pruksachatkun, Y.. Nangia, N., Singh, A., Michael, J.. Hill, F., Levy, O., and Bowman,
S. (2019). Superglue: A stickier benchmark for general-purpon purpose language understanding systems. Advances in neural information processing systems, 32.
[57] Wang, A., Singh, A., Michael, J., Hill, F. Levy, O., and Bowman, S. R. (2018). Glue: A multi-task benchmark and analysis platform for natural language understanding. International Conference on Learning Representations.
[58] Wang, L., Luc, P., Wu, Y., Recasens, A., Smaira, L., Brock, A., Jaegle, A., Alayrac, J.-B., Dieleman, S., Carreira, J., et al. (2022a). Towards learning universal audio representations. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages
IEEE.
[59] Wang, X., Liu, L., and Shi, J. (2022b). Dilated Convolutional Model for Melody Extraction.
[60] Watanabe, S., Hori, T., Kim. S., Hershey, J. R., and Hayashi, T. (2017). Hybrid ctc/attention ar- chitecture for end-to-end speech recognition. IEEE Journal of Selected Topics Signal Processing. 1 11(8): 1240:1220-
[61] Wilkins, J., Seetharaman, P., Wahl, A., and Pardo, B. (2018). Vocalset: A singing voice dataset. ISMIR, pages 468 474.
[62] Wu, H.-H., Kao, C.-C., Tang, Q. Sun, M., McFee, B., Bello, J. P., and Wang, C. (2021). Multi-task sulf-supervised pre-training for music classification. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 556-560. IEEE
[63] Xi, Q., Bittner, M., Pauwels, J., Ye, X., and Bello, J. P. (2018). Guitarset: A dataset for guitar transcription. In ISMIR, pages 453-460.
[64] Yamamoto, Y., Nam, J., and Terasawa, H. (2022). Deformable enn and imbalance-aware feature learning for singing technique classification. arXiv preprint arXiv:2206. 12230.
[65] Yang, S.-w., Chi, P.-H., Chuang, Y.-S., Lai, C.-I. J., Lakhotia, K., Lin, Y. Y., Liu, A. T., Shi, J., Chang, X., Lin, G.-T,, et al. (2021). Superb: Speech processing universal performance benchmark. arXiv preprint arXiv:2105.01051.
[66] Yao, D., Zhao, Z., Zhang, S.. Zhu, J., Zhu, Y., Zhang, R., and He, X. (2022). Contrastive learning with positive-negative frame mask for music representation. In Proceedings of the ACM Web Conference 2022, pages 1906-2915.
[67] Zai El Amri, W., Tautz, O., Ritter, H., and Melnik, A. (2022). Transfer learning with jukebox for music source separation. In Artificial Intelligence Applicutions and Innovations: 18th IFIP WG 12.5 International Conference, AIAI 2022, Hersonissos, Crete, Greece, June 17-20, 2022, Proceedings, Part II, pages 426-433. Springer.
15

---

[68] Zhai, X., Puigcerver, J., Kolesnikov, A., Ruyssen. P., Riquelme, C., Lucic, M., Djolonga, J., Pinto, A. S., Neumann, M.. Dosovitskiy, A., et al. (2019). The visual task adaptation benchmark.
[69] Zhao, Y. and Guo, (2021). Musicoder: A universal music-acoustic encoder based on transformer. In International Conference Multimedia Modeling, pages 417-429. Springer.
[70] Zhuo, L., Yuan, R., Pan, J., Ma, Y., Li, Y., Zhang, G., Liu, S., Dannenberg. R., Fu, J., Lin, C., et al. (2023). Lyricwhiz: Robust multilingual lyrics transcription by whispering to chatgpt. International Society for Music Information Retrieval Conference (ISMIR).
16

---

Appendix A Evaluation Protocol Details
The hypeer-paaamene search range of the constrained evaluation track is given as follow:
1. Layer: (every single layer, weighted sum]
2. Model: (one-layer 5 512-units MLP, one-layer 512-umit LSTM (melody extraction only),
3-layer 512-umit LSTM (source separation only). --encoder--ddecodee layers transformer (lyries transcription only)]
3. Batch size: (64)
4. Learning rate: (5e-5, le-4, Se-4, le-3, 5e-3, le-2]
5. Dropout probability: (0.2)
Appendix B Detail Analysis
What have the music audio pre-trained representations learned? We observe that all the rep- resentations have learned multiple levels of knowledge in Fig. Most of the selected baselines are particularly good at high-level music description tasks, such as genre classification and emotion recognition. However, when pre-trained with a full supervision paradigm, the representations may not be able to model pitch and key well, as they could overfit the supervision signal less relevant to pitch-related information. On the contrary, SSL methods usually mitigate this issue by providing more generalisable representations. Some representations do not support frame-level representations, which makes it difficult to evaluate their performance on tasks such as source separatin and beat tracking. Therefore, it is unclear how well these models have learned such information.
How can we design better pre-training strategies for music audio representation learning? As mentioned in the above paragraph, we suggest that good pre-training strategy needs to prevent overfitting the supervision signal, which makes sule--spe learning a more promising approach. Moreover, we argue that optimal method for music pre-training should be able to scale up to larger data and model size. Based on observations from Figure 2, it appears that larger data and model size have a greater impact on performance than the training paradigm (generative, contrastive, prediction) at the current stage of research. Besides, stacked transformer models are good candidates for future pre-training architecture, as they can be easily scaled up, and usually provide frame-level representations in a well-considered design.
mask
Avg. Score
Avg. Soore
100.0-
80.0
PL
80.0
LFFE
SOP Masiche
70.0
Yo
60.0
Score level Acoustio-level
60.0
Performance level High-level
40.0
50.0
zio
zls
30
1s
20
HALSI
70.0
68.0
60.0
Andorrd
50.0
Macresan
B1D
MAR MarDAee
CUM0-C0 CMM
CURLSEN
50.0
40.0
15
Training Context/ Second
Training Context/ / Second
(a) Scores at Acoustic-level and Score-level.
(b) Scores Performance-level and High-level.
Figure 3: Results Analysis Regarding Training Context Length. The performances of source separa- tion and beat tracking tasks are ignored similar to Fig. 2.
How does context length affect performance? According to Fig. 3, the relationship between context length and performance exhibits rather complex and irregular pattern, for which it is currently difficult to draw any conclusive insights. This is due to the limited number of music audio representations available at the moment, coupled with challenges in controlling variables. However, we are able to derive some preliminary observations when considering factors such as data size (D)
17

---

and parameter size (N). We observe that within a context length (L) of approximately 3 to 5 seconds, scaling up and D can be effective, but the performance quickly saturates. Furthermore, according to MAP-Music2 VecM-99M, solely increasing the I. without scaling the N and D may also lead to performance saturation. Interestingly, when scaling up all three aspects, according Jukebox-5B with 23 seconds context and 60- 120kr data, the performance still saturates. The underlying cause of this saturation may be associated with the training paradigm.
Appendix C Website and Leaderboard
To accompany the MARBLE benchmark with leaderboard data and detailed resources presentation, aac..ac..... we build a website, which can be found at https: : /mmamlleeeeeeeear e- bm. shef ac. uu. All the resources and comprehensible introduction of the benchmark and submission guideline are indexed on the homepage as shown in Fig. 4. The participants can easily find the process of submitting their results according to the guidelines. As demonstrated in Fig. 5, provide well-organised leaderboard for MARBLE, where the evaluated results can be re-ranked according to different metrics and filtered by tasks.
MARBLE
MARBLE.. a Music Audio Representation Benchmark for universaL Evaluation
Uicso MAMBL.
masto inderst lendrttort
What MA
Reproe Beechmak atheral. Restanto MARDLE beacresk proport
inqutre pretanne mones accodang g campuetee
Wolcome your rocatiuts Mania
Supported
The NATOLC bulls are calggorae into our wabb twarchy rovu
Aerfommmee
Figure 4: Website for the Proposed MARBLE Benchmark.
18

---

MARBLE
Dode
Scana
Abou
Leaderboard
dbaling
310
IHERT POE
V11
385
017
252
754
124
300
821
Figure 5: Music Understanding Model Leaderboard Hosted on the MARBLE Website.
19

---

Appendix D Details on Chord Estimation
D.1
Chord Vocabulary
Our chord vocabulary includes "none" and 35 different chords on each of the 12 root notes, 421 in total. The root notes are listed as follows: {C Db D Eb E. F Gb G Ab A Bb B). We do not distinguish between equal notes under the twelve equal temperaments. For example, we think that C# and Db are the same note and have the essentially equivalent function in chord prediction. We use sharp in the code implementation for identification but use flat in the following tables.
The following Tables of the 35 types of chords with examples and number of samples in the datasets.
chord name
maj
min
aug
maj6
minó
maj7
min7
dim?
hdim7
example
Cmaa
Camin
Craug
C:maj6
Cimin6
C:7
Cimaj77
Camin?
C:dim?
Clhdim?
chord lones
1.3,5
1.b3,5
1.3.45
1,3,5,6
1.3.5.77
1,3,5,b7
1,3,5,7
1.b3,5.b7
1.b3.55.b67
1.b3.b5.b7
chard numher
368
12
292
204
2
chord
mieti
maj3
maj/5
miws
example
C9
Cinaj9
Camin9
CiI
Caus2
Couna
Cmaa//3
Comapg5
Commnhh
CominnS
C:1/3
0:08
C.977
chard tunts
1,3587,9
1,3,77,9
1,9,56779
1.355099.111
1,2,5
1,4,5
3,5.1
5,133
63,5.1
5.1.8
3,567,1
5367,3.3
67.1335
chord numberr
78
22
48
58
44
82
264
10
82
10
44
46
chand
majis
madis
dincites
destiesy
Iiteriee)
Istmiih
N
enorptte
Crrognit
[
Catintner
Cistentest
Chitwwhs
NN chool
21.33
chand number
5
16
16
90
43.
2
These are some special rare chords in the dataset and we use some Chord Substitutions based on similar chords or the chord annotation for the music score instead of the ground truth chord annotation the musician actually plays.
1. majmin7 was substituted with 7: The "majmin7" chord is equivalent to the "7" chord, so we are making replacement to standardize the notation.
2. minmaj7 was substituted with min7: Both chords share the root, minor third, and perfect fifth. When mapping minmaj7 to min7, the major seventh is altered to minor seventh, ensuring the "minor" character of both chords remains consistent.
3. minll was substituted with 11: Both chords are minor chords composed of the seventh and eleventh tones. Given their infrequent occurrences, we map "minl 1" to the "11" chord.
4. Substitution for out-ol-vocabulary colour chords: The performed chord annotations in the GuitarSet also contain out-of- vocabulary colour chords such (1,5)/1, (1,5,b7)/11 (5,2, 4)/4. For such chords, we identify the corresponding standard chords in the instructed chord annotations and substitute them.
5. Special Transposition Handling for Standard Chords: Map to the standard transposition that is closest to the corresponding transposed note.
D.2 Chord Recognition Metric Definition
1. root:
Evaluating chord recognition algorithms based on the root notes of the identified chords. Only compares the root of the chords.
2. majmin:
Only compares major, minor, and "no chord" labels. Any other chord types or variations, such as 7th chords, augmented, diminished, and on, not considered in this specific evaluation.
3. mirex: Compare chords along MIREX rules. A estimated chord is considered correct if shares at least three pitch classes in common.
4. thirds: Chords are compared at the level of major minor thirds (root and third). For example, both ('A:7", 'A:maj") and ('A:min', 'A:dim") are equivalent, as the third is major and minor in quality, respectively.
20

---

5. traids: Chords are considered at the level of triads (major, minor, augmented, diminished, suspended). In addition to the root, the quality is only considered through #5th scale degree (for augmented chords). For example, ('A:7', 'A:maj') are equivalent, while ('A:min', 'A:dim') and ('A:aug', 'A:maj") are not.
6. sevenths: Compares according to MIREX "sevenths" rules. Only major, major seventh, seventh, minor, minor seventh and no chord labels are compared.
7. majmin inv: Compares major/minor chords, with inversions. The bass note must exist in the triad.
8. sevenths_ing inv: Compares according to MIREX "sevenths" rules, with inversions. The bass note must exist the chord.
During the evaluation process, frame-level predictions are directly merged to event-level by the mir_eval function we do not apply any post-processing to the prediction.
Appendix E
Details on Lyrics Transcription
E.1
MulJam2.0 dataset
MulJam2.0 is derived from MulJam, featuring larger and more refined human annotation on the test set. We select 34 songs from the training set and obtain human lyrics annotation to expand the test set. For each language, 20 songs are randomly selected from the original training set to form the validation set. A few songs are excluded due to poor alignment for obtaining the line-level annotations (For details, please refer to [70]). We also exclude the songs in the training and validation sets that present in Jamendo (3 songs in training and song in validation), ensuring that the songs in the evaluation datasets remain unseen during training. The numbers of songs by language can be found in Tab. 6.
The human annotation is performed at the song level. We applied similar procedures to obtain line-level annotations, as was done for the training set in MulJam. We use the timestamps provided by Whisper [44], and align the lines predicted by Whisper with the human annotation. As in [70] lines with unusually high character rates (exceeding 37.5 Hz) are removed. However, for the test set we choose not to filter by the similarity between the aligned text pairs, to prevent introducing excessive bias in favor of Whisper predictions.
Table 6: Number of songs in MulJam2.0 and Jamendo datasets.
Datawet Split
MulJam2.0 Valid
Jamendo Test
Tiais
Test
Eaglish (cx)
3557
20
28
French (e)
977
19
Spunish
564
19
107
Itllinn
278
20
106
Touil
5609
114
E.2 Language Model and Tokenizer
The language model (LM) is trained using speechbrain [47] language model recipe 12, The model comprises of transformer encoder layers, with an attention dimension of 768, 12 attention heads, and a position-wis wise feed-forwal layer dimension of 3072. The LM is trained using cross-entropy loss for 20 epochs, and the model with the lowest loss is selected.
The target character set is the union of the character sets from 6 languages, resulting in a total of 91 tokens: e, <bos>, <eos>, <unk>, A. B, C, D. E, H. K. L, M, N. O. Q, R. V, W. E, É, X, Y, A, Á, E.E.E.E.EEEEEEEE.E.EEEEEEE.EEEE. O, O, O, Φ X, II, 9, III, III, b, b, 10, Я.
Ü. A.A.T.A..
XK, 3, H, M. K, JI, M.H.H.
21

---

E.3
Training Details
The beam search used for validation and testing incorporates a combination CTC probabilities, LM probabilitiess (applied only at test time), and S2S probabilities, We assign a weight of 0.4 to the CTC probabilities and 0.3 the LM probabilities. During validation, we utilize beam size of 10 and calculate Word Error Rate every 5 epochs to optimize processing efficiency. For thorough evaluation, we scale up the beam size to 40 during the testing phase. The accuracy of the S2S branch output is continually monitored to determine whether early stopping should be triggered and to facilitate model selection.
E.4 Results and Discussion
The results of multilingual lyrics transcription using different pretrained features can be found in Tab. In addition MulJam, we also present WERs on the Multilingual Jamendo evaluation set [13]. This dataset consists of 80 songs in 4 languages: English, French, Spanish, and German. While Italian and Russian songs are not included, Jamendo's aunan-annotated line-level annotation aligns well with our evaluation setting. For comparison, we reference the state-of-the- of-the-rrt model Whisper [44], robust model designed for speech recognition but also performs effectively on singing voice. Whisper has been trained the internet. It is also the foundation the MulJam dataset.
extensive corpus of multilingual and multitask supervised data collected from
Lyrics transcription is challenging task that involves detecting vocal pronunciations in the presence of background music and making the most probable predictions based on linguistic knowledge. The multilingual context makes this task even more demanding. When performing lyrics transcription with SSL features, it is essential that these features capture clear vocal information, and that the backend provides robust inference to generate coherent text from the vocal pronunciations. Achieving this with SSL features indeed significant challenge. The results presented Table 7 indicate that there is room for improvement in this task.
Among the six languages we considered, English, French, and Spanish, which have a larger number of songs than the other three, yield better results. This suggests that there may be an impact from the imbalanced training data. Russian, the other hand, produces the worst result for two main reasons: 1. Russian employs the Cyrillic writing system, which has its own set of characters. 2. The training data for Russian is insufficient for the model establish connection between the pronunciation rules of Cyrillic and Latin alphabets.
The MulJam test set is results to derive line-level annotations. Therefore, it is worth noting that bias is introduced, as the alignment is reliable only when the human annotation closely matches the Whisper's prediction.
at the song level but relies the alignment with Whisper
Table Multilingual lyries transcription results on MulJam and Jamendo.
Language
Einllish CER
French CER
Spanish CER
German
Italian CER
Russian CER
Whole
Metric
WER
WER
WER
CER
WER
WER
WER
CER
WER
MulJam2. 0 test
MAP-MusicVee: [31]
$1.7
79.2
58.3
90.9
43.2
83.7
63.4
99.5
53.0
91.9
101.6
125.6
56.1
87.8
MAP-MERT- [30]
48.7
71.2
55.5
85.4
41.0
80.1
65.9
100.9
49.1
86.3
99.5
124.9
52.6
82.1
MAP-MERT VOEERRTVV--9MMM publi
public [30]
49.0
71.2
55.
85.4
39.0
76.6
63.5
99.9
50.3
90.3
104.7
129.3
52.5
82.7
MAP-MERT-<1-00 [29)
45.5
66.5
52.5
81.9
38.2
73.9
58.8
93.2
44,4
81.6
96.1
117.8
49.4
77.9
MAT-MERT-11-3300 [29)
42.4
100.5
77.0
[44]
33.2
$2.9
29.9
43.8
36.5
38.1
34.7
53.7
$4.8
Jamendo
MAP-MusiZVes: [31]
49.0
73.6
55.3
87.1
50.3
90.7
67.8
108.8
55.7
89.6
MAP-MERT-<0-95M [30]
48.5
71.8
54.0
85.1
49.3
87.6
67.6
108.1
54.8
87.6
MAP-MERT-OO
MOVER--0-95M-paaa -0-95i--ublic [30]
46.9
71.5
52.0
81.5
44.8
82.8
66.3
106.8
52.6
85.2
MAP-MERT.wI [29]
49.4
79.6
80.6
MAP-MERT-<1-330M (29)
50.2
SOTA [44]
29.2
41.7
44.4
22


Reviews:
[{"review_id": "R4leUrYd49", "reviewer": "Reviewer_fqYR", "summary": "", "questions": "", "limitations": "The authors discussed the method limitations in the paper.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "+ MARBLE provides a comprehensive benchmark for Music Information Retrieval (MIR) tasks. It defines a four-level taxonomy and establishes a unified protocol across 14 tasks and 8 public datasets. \n\n+ It serves as an evaluation tool and can be easily used to measure the performance of different music audio pre-training approaches.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "zQ5qmhMJuL", "reviewer": "Reviewer_XGB3", "summary": "", "questions": "", "limitations": "- not many music tasks other than MIR tasks (only source separation in the paper) is introduced in the paper, which somehow limits the interest of readers\n- only the best used metric for each task is included, e.g., global SDR for source separation, which can only highlight one aspect of the model performance", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "- the first paper proposing a benchmarking framework for music representation learning\n- covers a lot of discriminative tasks in music (MIR tasks) as well as dense prediction tasks like source separation (not covered in any prior work)\n- metrics proposed follow the state-of-the-art setting, e.g., global SDR for music separation\n- evaluation done on a large-scale model like Jukebox with CALM setting\n- well written and easy to follow", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "PJzL5Zae7u", "reviewer": "Reviewer_NaXM", "summary": "", "questions": "", "limitations": "In my opinion, the suggested benchmark would be beneficial to the MIR research community. However, it only collects existing tasks and datasets, which limits its originality. It would be better to introduce a new baseline for the proposed benchmark or provide an analysis of that baseline across various tasks and metrics in MARBEL. \nSection 4 discusses benchmarks. If the analysis is expanded upon and a baseline based on the analysis that excels in the entire benchmark is presented, the paper will hold more significance.\n", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "- Incorporating a wide range of MIR tasks and datasets and providing benchmark (MARBLE) to facilitate comprehensive music model evaluation.\n- Designing a unified assessment protocol and building corresponding evaluation suites for processing, training, and benchmarking.\n- These two components could make a valuable contribution to the MIR research community.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "wVEIh0G6cH", "reviewer": "Reviewer_Rutv", "summary": "", "questions": "", "limitations": "1. The benchmark's ability to generalize across all types of music (genres, periods, cultures) isn't clear from the summary. Some music might be underrepresented or not represented at all in the existing datasets. Perhaps a table showcasing the diversity of each dataset could help.\n2. The benchmark lacks solid analysis and only presents performance comparisons. Some key analyses for example effect of noise, data augmentations, etc could be helpful.\n3. While the benchmark could be useful for academic research, its effectiveness and relevance for real-world applications in the industry are not discussed.", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "The paper provides a strong foundation for comparing models with comprehensive task structures and unified protocols. The authors not only provide the toolkit but also establish a leaderboard for the community, promoting transparency, reproducibility, and further advancement in the field.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "Y5z48wMDF8", "reviewer": "Reviewer_Gtoe", "summary": "", "questions": "", "limitations": "None.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "1. learning msuci audio representation for evalutaion of discriminative music tasks is important and interesting. ", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}]

Author rebuttal:

Meta-review: I believe that this work is of interest and sufficient quality for the community to be presented in this track. I don't see any strong reason for rejecting it, especially among the reviews.
Decision: Accept (Poster)