SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Singer Identification involves recognizing the singer or vocal performer from an audio recording. In previous work on Singer Identification using the VocalSet dataset [61], different splits are employed. For the MARBLE benchmark, we randomly split the dataset into training, validation, and test sets, maintaining a ratio of 12:8:5. All sets contain the same 20 singers. The specific data divisions can be found in the provided code.
Source Separation aims to separate different components of a music recording, such as vocals, drums, bass, and others. In MARBLE, we adopt the widely-used MUSDB18 dataset [46] for this task. MUSDB18 consists of 150 full-length music tracks, totalling approximately 10 hours of audio and multiple isolated stems. Our training set consists of 86 tracks, the validation set contains 14 tracks, and the evaluation set comprises 50 tracks, following the official MUSDB18 setting. During training, we randomly sample 6-second segments and apply random track mixing for data augmentation. Due to the complexity of this task, we utilise the baseline architecture from the Music Demixing Challenge (MDX) 2021 [38]. This architecture consists of three linear layers and three bi-directional LSTM layers. The optimization is performed by directly computing the 12-loss between the predicted and ground-truth spectrograms. The evaluation metric for this task is the Source-to-Distortion Ratio (SDR) as defined in [38], which is calculated as the mean across the SDR scores of all songs.
3 Evaluation Framework
We aim to explore the generality and standardisation of the framework. Therefore, we freeze the parameters of the pre-trained model to extract pre-trained features as fixed depth embeddings fed to each downstream task-specific prediction head. This allows for as lightweight a solution as possible for all tasks, thus testing whether the representations are easily reusable across different downstream tasks. We describe pre-trained baseline models, downstream models, and protocols in the following sections.
3.1 Pre-trained baseline systems
The audio pre-training models explored in this paper are summarised in Table. 2. Note that we do not cover models designed entirely for speech or not open source models. We also examine all the open-source SSL systems specifically designed from music audio, in total 9 different versions of 7 pre-trained features; see Table. 2 for information on pre-trained models.
MusiCNN [43] is a convolutional model pre-trained on the music audio tagging task using the MSD dataset [3]. We use the default configuration of the method, which is to concatenate the mean pooling of the CNN features for a 3-second input with the output of the maximum pool.
Contrastive learning of musical representations (CLMR) [53] leverages a 9-layer 1-D convolutional kernel as the feature extractor, employing a number of data augmentation, and is trained on both MSD and MTT. Both are trained with a contrastive learning approach. The model extracts an embedding every 2.69 seconds.
Jukebox [11] is a music generation model trained using codified audio language modelling (CALM). It is trained on 1.2 million private songs, and the size of the training set is difficult to estimate the exact number of hours. However, assuming an average song length of 3-6 minutes, the total length could be  $60\mathrm{k - }120\mathrm{k}$  hours, which is large and diverse to allow Jukebox to learn patterns and structures of different musical genres and styles. We use the same mid-layer representation as [8] to improve
Table 2: Information of Baseline Systems.
Method	MusiCNN MSD-big	CLMR	Jukebox	MULE	MAP-Music2Vec	MAP-MERT-v0	MAP-MERT-v1
base	base-public	base	large
Network	CNN	9-Conv	3-Conv, 36-Trans	22-Conv, 2-Trans	7-Conv, 12-Trans	7-Conv, 12-Trans	7-Conv, 12-Trans	7-Conv, 12-Trans	7-Conv, 12-Trans
#Params	8M	2.5M	5B	62.4M	95M	95M	95M	95M	330M
Input	log-mel	waveform	waveform	log-mel	waveform	waveform	waveform	waveform	waveform
Stride	3s	2.69s	23.78s	2s	20ms	20ms	20ms	13.3ms	13.3ms
Context Length	3s	2.69s	23.78s	3s	30s	5s	5s	5s	5s
Data (hour)	10~20k	1.7k	60~120k	117.5k	1k	1k	0.9k	17k	160k
Pre-training Task	Music Tagging	Contrastive Learning	CALM	Contrastive Learning	MLM Boostrapping	MLM Clustering	MLM Clustering	MLM Clustering	MLM Clustering
computational efficiency. Unlike other representations that run on short context windows, JUKEBOX is trained on a long window of 8192 sample points (23.78 seconds) of audio. We use the same strategy as [8] to extract the audio features on the downstream dataset.
MULE (Musicnet-ULarge) [37] is a SSL system based on SF NFNet-F0 [58], SlowFast Normalizer-Free ResNet. It combines a SlowFast (SF) part (including a slower pathway that captures spatial information and a faster pathway that captures temporal information) with a more efficient and scalable variant of the Normalizer-Free ResNet (NFNet). MULE is contrastively pre-trained on the whole MusicSet dataset [37] and provides promising results on classification tasks. The model extracts an embedding with a 3-second window length and a 2-second hop length.
MAP-Music2Vec [31] is a self-supervised learning (SSL) model specifically based on a bootstrapping mask prediction pre-training strategy. It consists of two main components: the student and teacher models. Both share the same architecture with 12 transformer layers, with the teacher model's parameters being exponential moving averages of the student model's parameters. The student model takes in masked input, and during training, it aims to learn deep features from the teacher model based on the output of the unmasked input. Specifically, it computes the average of the top 8 layers of the Transformer's output in the teacher model. To train the MAP-Music2Vec model, a private dataset comprising approximately 1,000 hours of music data was used. The input length of the MAP-Music2Vec model is set to 30 seconds, producing 50 embeddings per second. These embeddings capture essential features of the music data and can be utilised for various downstream tasks, including sequential tasks such as source separation and beat tracking.
MAP-MERT-v0, also referred to as MERT-95MK-means in the work by Li et al. [29], is a pre-trained model built upon the speech self-supervised learning (SSL) system HUBERT [20]. It undergoes pre-training for masked prediction, with discrete pseudo-labels obtained from K-Means clustering on music features. The pre-training task of MAP-MERT-v0 involves two pseudo-labels based on logmel and Chroma, along with a CQT reconstruction task that emphasises pitch information. Two versions of the MAP-MERT-v0 model are included: MAP-MERT-v0 $^8$ , trained on a private dataset of 1,000 hours, and MAP-MERT-v0-public $^9$ , trained on Music4ALL [50]. The input length of the MAP-MERT-v0 model is set to 5 seconds, generating 50 embeddings per second. This design facilitates fine-tuning for sequential tasks, enabling efficient and effective processing of music data.
MAP-MERT-v1 encompasses two variants: (MAP-)MERT-v1-base $^{10}$  and (MAP-)MERT-v1-large $^{11}$ . These models, also known as MERT-95M $^{\text{RVQ-VAE}}$  and MERT-330M $^{\text{RVQ-VAE}}$  in the work by Li et al. [29], employ EnCodec, a pre-trained discrete deep feature, as a replacement for the K-means feature. This modification facilitates the scaling up of the model. Similar to MAP-MERT-v0, the input length of the MAP-MERT-v1 models is 5 seconds, but they produce 75 embeddings per second.
https://huggingface.co/m-a-p/MERT-v0  
https://huggingface.co/m-a-p/MERT-v0-public  
https://huggingface.co/m-a-p/MERT-v1-95M  
https://huggingface.co/m-a-p/MERT-v1-330M
This configuration enables effective fine-tuning for sequential tasks, making the models suitable for processing music data in a variety of applications.
3.2 Downstreams and Training Strategies
To evaluate the relevance of representations for downstream MIR tasks, we design evaluation frameworks: the unconstrained track, semi-constrained track and the constrained track. In the unconstrained track, researchers are invited to submit their systems with any hyperparameter and structure configuration, including the option to fine-tune pre-trained models. This track encourages flexibility and exploration, enabling researchers to investigate a wide range of approaches. On the other hand, the semi-constrained track requires the submissions to use frozen pre-trained backbones. Finally, the constrained track employs a standardised setting with limited hyper-parameter search space (cf. Appendix A), where frozen models are used as feature extractors for training a one-layer 512-unit MLP (or 1/3-layer 512-unit LSTM for melody extraction or source separation, or 3-encoder-3-decoder layers transformer for lyrics transcription) on each task. In addition, we set a computational wall for MARBLE. The systems need to finish each task within a week on our machine equipped with a single consumer GPU (RTX3090). By offering these three evaluation tracks, we aim to provide researchers with a comprehensive platform to assess the performance and relevance of representations in MIR tasks, encouraging innovative approaches and fostering advancements in the field. For the same task with a uniform dataset, if there are different evaluation metrics (e.g., emotion regression, source separation, and tagging), we will average the two evaluation metrics. We select the checkpoints regarding to the best validation results for final testing and submission.
Table 3: Performances of Baselines Evaluated on MARBLE with constrained settings (1/3). We include previous SOTAs for reference. Note that MARBLE imposes strict constraints on downstream structures and hyper-parameter search spaces, while previous SOTAs are not subject to such limitations. Best scores on MARBLE are bold, and best scores among all systems are underlined.
Dataset
Task	MTT
Tagging	GS
Key	GTZAN
Genre	GTZAN
Rhythm	EMO
Emotion	Nsynth
Instrument	Nsynth
Pitch	VocalSet
Tech	VocalSet
Singer
Metrics	ROC	AP	Acc	Acc	F1	R2	R2	Acc	Acc	Acc	Acc
MusiCNN [43]	90.3	37.8	14.4	73.5	-	44.4	68.8	72.6	64.1	70.3	57.0
CLMR [53]	89.5	36.0	14.8	65.2	-	44.4	70.3	67.9	47.0	58.1	49.9
Jukebox-5B [8, 67]	91.4	40.6	63.8	77.9	-	57.0	73.0	70.4	91.6	76.7	82.6
MULE [37]	91.2	40.1	64.9	75.5	-	60.7	73.1	74.6	88.5	75.5	87.5
MAP-Music2Vec [31]	90.0	36.2	50.6	74.1	68.2	52.1	71.0	69.3	93.1	71.1	81.4
MAP-MERT-v0-95M [30]	90.7	38.2	64.1	74.8	88.3	52.9	69.9	70.4	92.3	73.6	77.0
MAP-MERT-v0-95M-public [30]	90.7	38.4	67.3	72.8	88.1	59.1	72.8	70.4	92.3	75.6	78.0
MAP-MERT-v1-95M [29]	91.0	39.3	63.5	74.8	88.3	55.5	76.3	70.7	92.6	74.2	83.7
MAP-MERT-v1-330M [29]	91.1	39.5	61.7	77.6	87.9	59.0	75.8	72.6	94.4	76.9	87.1
Previous SOTA	92.0 [21]	41.4 [8]	74.3 [27]	83.5 [37]	80.6 [19]	61.7	72.1 [8]	78.2 [58]	89.2 [37]	65.6 [64]	80.3 [39]
Table 4: Performances of Baselines Evaluated on MARBLE with constrained settings (2/3). Note that we denote the scores of Jukebox-5B on MTG tasks with asterisks $(*)$ , because it hit the computational wall of MARBLE, meaning that the system was unable to complete the corresponding task within a week on our machine equipped with a single consumer GPU (RTX3090).
Dataset Task	MTG Instrument	MTG MoodTheme	MTG Genre	MTG Top50	MUSDB Source Separation
Metrics	ROC	AP	ROC	AP	ROC	AP	ROC	AP	SDR\( ^{vocals} \)	SDR\( ^{drums} \)	SDR\( ^{bass} \)	SDR\( ^{other} \)
MusiCNN [43]	74.0	17.2	74.0	12.6	86.0	17.5	82.0	27.5	-	-	-	-
CLMR [53]	73.5	17.0	73.5	12.6	84.6	16.2	81.3	26.4	-	-	-	-
Jukebox-5B [8, 67]	78.5*	22.0*	77.6*	15.3*	88.0*	20.5*	83.4*	30.4*	-	-	-	-
MULE [37]	76.6	19.2	78.0	15.4	88.0	20.4	83.7	30.6	-	-	-	-
MAP-Music2Vec [31]	76.1	19.2	76.7	14.3	87.1	18.8	83.0	29.2	5.5	5.5	4.1	3.0
MAP-MERT-v0-95M [30]	76.6	18.7	75.9	13.7	86.9	18.5	82.8	28.8	5.6	5.6	4.0	3.0
MAP-MERT-v0-95M-public [30]	77.5	19.6	76.2	13.3	87.2	18.8	83.0	28.9	5.5	5.5	3.7	3.0
MAP-MERT-v1-95M [29]	77.5	19.4	76.4	13.4	87.1	18.8	83.0	29.0	5.5	5.5	3.8	3.1
MAP-MERT-v1-330M [29]	78.1	19.8	76.5	14.0	86.7	18.6	83.4	29.9	5.3	5.6	3.6	3.0
Previous SOTA	78.8	20.2 [1]	78.6	16.1 [37]	87.7	20.3 [1]	84.3	32.1 [37]	9.3	10.8	10.4	6.4 [48]
According to Table 3, 4 and Fig 1, all pre-trained baseline representations on MARBLE have achieved decent results. Despite strict constraints on downstream structures and hyper-parameter search spaces,
Table 5: Performances of Baselines Evaluated on MARBLE with constrained settings (3/3). The overall average scores are calculated on the systems applicable to all tasks.
Dataset Task	MelodyDB Melody	Muljam
Lyrics	Jamendo
Lyrics	GuitarSet
Chord Estimation
Metrics	Acc	CER	WER	CER	WER	root	majmin	mirex	thirds	triads	sevenths	majmin_inv	sevenths_inv
MAP-Music2Vec [31]	36.1	56.4	87.8	55.7	89.6	13.7	11.1	10.4	10.4	10.4	11.1	9.4	9.4
MAP-MERT-v0-95M [30]	60.0	52.6	82.3	54.8	87.6	48.7	38.9	36.7	36.6	36.5	37.5	30.3	29.0
MAP-MERT-v0-95M-public [30]	36.1	52.5	82.7	52.6	85.2	49.1	38.7	36.4	36.6	36.4	37.5	29.9	28.6
MAP-MERT-v1-95M [29]	60.8	49.4	77.9	49.6	82.2	50.5	38.8	36.5	36.7	36.4	36.5	31.2	28.9
MAP-MERT-v1-330M [29]	62.5	48.5	77.0	50.3	83.1	56.3	44.8	45.1	44.0	44.0	40.7	27.9	26.5
Previous SOTA	65.3[59]	39.5	54.8[44]	25.4	44.4[44]	34.8	33.3	33.6	33.3	33.2	24.0	33.1	23.9[22]
Figure 1: SSL Baselines Compared to previous SOTA. The performances of the tasks are merged according to the task types demonstrated in Tab. 1. Results not applicable are set to 0.
(a) Scores at Acoustic-level and Score-level.
(b) Scores at Performance-level and High-level.
Figure 2: Results Analysis Regarding to Training Data Size. Since some models are not applicable to the sequence labelling tasks, the performances of source separation and beat tracking tasks are excluded on acoustic-level and score-level average score calculation correspondingly. The radii of the scatter points are isometrically log scaling with the parameter sizes.
they are able to approach, if not surpass, the previous state-of-the-art (SOTA) in many tasks. For instance, the best performance on NSynth Pitch classification have achieved up to  $94.4\%$  accuracy. Nonetheless, the majority of tasks are still far from being solved, including music tagging and source separation tasks. Notably, the performance on MUSDB18 is merely half of the previous SOTAs.
The MAP family achieves balanced results, successfully performing tasks including sequence labelling, which other models fail to accomplish (as they do not provide frame-level representations or are too cumbersome to train). This series of models excel at multiple taxonomy levels. On certain tasks, MAP-MERTs achieve results close to or surpass the previous state-of-the-art. However, music tagging tasks are dominated by Jukebox-5B and MULE. Jukebox may benefit from its massive parameter size and generative modelling of detailed information, as well as the introduction of metadata during the pre-training period. Conversely, MULE benefits from its proprietary large-scale, high-quality dataset, MusicSet, and the highly discriminative representations learned by contrastive pre-training.
Based on Fig. 2a and 2b, excluding sequence labelling tasks (as some baselines do not support them), we observe a general trend: as the volume of data and the size of model parameters increase, the performance of tasks across four levels correspondingly improves. The choice of pre-training method and model size significantly influences the performance. For instance, MAP-Music2Vec-95M, utilizing only 1k hours of data for self-supervised learning, outperforms both supervised pre-trained MusiCNN-8M and contrastive pre-trained CLMR-2.5M on the same scale of data. More analysis could be referred to Appendix B.
In this work, we introduce the Music Audio Representation Benchmark for universal Evaluation (MARBLE) as a comprehensive benchmark for evaluating pre-trained music features. It encompasses a hierarchy taxonomy that covers acoustic, performance, score, and high-level description levels, and utilises publicly available datasets for 18 MIR tasks. We establish a standardised preprocessing and data splitting protocol, along with a unified evaluation framework, to ensure fair and reproducible assessment. We report the results of all 9 open-sourced pre-trained models developed on music recordings, showcasing their performance across multiple tasks. The results demonstrate that several pre-trained models achieve comparable or even superior performance to the state-of-the-art models on various tasks within MARBLE. However, there is still ample room for improvement, particularly in music tagging and source separation. With the release of the toolkit, we hope to facilitate future research by providing easy access, reproducibility, and fair comparison of SSL pre-trained models for music understanding. We encourage engagement from researchers in the audio and AI communities to contribute to the advancement of representation learning for music information retrieval.
Our benchmark has some shortcomings that can be further improved. To begin with, some of the tasks, such as beat tracking and piano transcription, typically use multiple evaluation metrics, but we only include one or two for each of the tasks due to the copyright issues preventing many datasets from being publicly available, lack of standard pre-processing or maintenance, and the limitation of time. Although the selected metric is fundamental and a good indicator, an average of all the metrics might be a better choice. Besides, some of the datasets are not sufficient for a single task. For example, the GTZAN dataset does not have a commercially-available license, and it only includes less than 10 hours of music recordings, making the evaluation more subject to bias. We will include more commercially-available larger datasets on the same tasks. Moreover, we do not include some MIR tasks that lack a common dataset currently, such as cover song detection and query-by-humming. In the future version, we will include more datasets and tasks. Last but not least, MIR on symbolic music is not included in the first version of our benchmark as well.
Apart from the traditional MIR tasks, some interesting tasks deserve more attention for benchmark development in the computer music and AIGC communities. With the benchmark and pre-trained models in MIR, developing an evaluation score on music generation and synthesis might be possible. There may not exist a perfect solution on the subject metrics for music generation to build a benchmark; otherwise, composing musical art will simply search for the waveform with the highest scores. But one can expect such a benchmark can be helpful for the music industry or music education to preclude some bad music generation. Besides, multi-modal approaches that combine music audio with symbolic music and language (e.g., lyrics and music description) also deserve a benchmark.
We would like to express sincere gratitude to our friends Anqiao Yang and Wei Fan for their invaluable support during the writing of this paper.
Ruibin Yuan is funded by Theme-based Research Scheme (T45-205/21-N) and Early Career Scheme (ECS-HKUST22201322), Research Grants Council of Hong Kong. Yinghao Ma and Jiawen Huang are research students at the UKRI Centre for Doctoral Training in Artificial Intelligence and Music, supported by UK Research and Innovation (Grant Number: EP/S022694/1). Yizhi Li is fully funded by an industrial PhD studentship (Grant Number: 171362) from the University of Sheffield, UK. Emmanouil Benetos is supported by RAEng/Leverhulme Trust Research Fellowship LTRF2223-19-106.
We acknowledge IT Services at The University of Sheffield for the provision of services for High-Performance Computing. This project also made use of time on Tier 2 HPC facility JADE2, funded by EPSRC (EP/T022205/1)
Besides, we would like to give special thanks to the following researchers or musicians on the lyrics labelling: Léo Nebel in LIP6 at Sorbonne Université; Nick Magal in School of Music at Carnegie Mellon University; Carey Bunk, Yannis Vasilakis, Christopher Mitcheltree, Nelly Victoria Alexandra Garcia-Sihuay, Teresa Pelinski Ramos, Ilaria Manco, David Sudholt, Jordan Shier, and Matthew Rice in Centre for Digital Music at Queen Mary University of London; Emilian Postolache in Sapienza University of Rome; as well as Wenqin Yu in the Chinese Music Institute at Peking University.
[68] Zhai, X., Puigcerver, J., Kolesnikov, A., Ruyssen, P., Riquelme, C., Lucic, M., Djolonga, J., Pinto, A. S., Neumann, M., Dosovitskiy, A., et al. (2019). The visual task adaptation benchmark.
[69] Zhao, Y. and Guo, J. (2021). Musicoder: A universal music-acoustic encoder based on transformer. In International Conference on Multimedia Modeling, pages 417-429. Springer.
[70] Zhuo, L., Yuan, R., Pan, J., Ma, Y., Li, Y., Zhang, G., Liu, S., Dannenberg, R., Fu, J., Lin, C., et al. (2023). Lyricwhiz: Robust multilingual lyrics transcription by whispering to chatgpt. International Society for Music Information Retrieval Conference (ISMIR).
Appendix A Evaluation Protocol Details
The hyper-parameter search range of the constrained evaluation track is given as follow:
1. Layer: {every single layer, weighted sum}
2. Model: {one-layer 512-units MLP, one-layer 512-unit LSTM (melody extraction only), 3-layer 512-unit LSTM (source separation only), 3-encoder-3-decoder layers transformer (lyrics transcription only)}
3. Batch size: {64}
4. Learning rate:  $\{5\mathrm{e} - 5,1\mathrm{e} - 4,5\mathrm{e} - 4,1\mathrm{e} - 3,5\mathrm{e} - 3,1\mathrm{e} - 2\}$
5. Dropout probability:  $\{0.2\}$
Appendix B Detail Analysis
What have the music audio pre-trained representations learned? We observe that all the representations have learned multiple levels of knowledge in Fig. 1. Most of the selected baselines are particularly good at high-level music description tasks, such as genre classification and emotion recognition. However, when pre-trained with a full supervision paradigm, the representations may not be able to model pitch and key well, as they could overfit the supervision signal less relevant to pitch-related information. On the contrary, SSL methods usually mitigate this issue by providing more generalisable representations. Some representations do not support frame-level representations, which makes it difficult to evaluate their performance on tasks such as source-separation and beat tracking. Therefore, it is unclear how well these models have learned such information.
How can we design better pre-training strategies for music audio representation learning? As mentioned in the above paragraph, we suggest that a good pre-training strategy needs to prevent overfitting the supervision signal, which makes self-supervised learning a more promising approach. Moreover, we argue that an optimal method for music pre-training should be able to scale up to larger data and model size. Based on observations from Figure 2, it appears that larger data and model size have a greater impact on performance than the training paradigm (generative, contrastive, or mask prediction) at the current stage of research. Besides, stacked transformer models are good candidates for future pre-training architecture, as they can be easily scaled up, and usually provide frame-level representations in a well-considered design.
(b) Scores at Performance-level and High-level.
Figure 3: Results Analysis Regarding Training Context Length. The performances of source separation and beat tracking tasks are ignored similar to Fig. 2.
How does context length affect performance? According to Fig. 3, the relationship between context length and performance exhibits a rather complex and irregular pattern, for which it is currently difficult to draw any conclusive insights. This is due to the limited number of music audio representations available at the moment, coupled with challenges in controlling variables. However, we are able to derive some preliminary observations when considering factors such as data size (D)