SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Ruibin Yuan $^{*,1,2*}$  Yinghao Ma $^{*,3*}$  Yizhi Li $^{*,4,5*}$  Ge Zhang $^{*,1,6*}$  Xingran Chen $^{7}$  Hanzhi Yin $^{2}$  Le Zhuo $^{8}$  Yiqi Liu $^{4,5}$  Jiawen Huang $^{3}$  Zeyue Tian $^{9}$  Binyue Deng $^{10}$  Ningzhi Wang $^{3}$  Chenghua Lin $^{4,5\dagger}$  Emmanouil Benetos $^{3}$  Anton Ragni $^{5}$  Norbert Gyenge $^{5}$  Roger Dannenberg $^{2}$  Wenhu Chen $^{6}$  Gus Xia $^{11,12}$  Wei Xue $^{9}$  Si Liu $^{8}$  Shi Wang $^{13}$  Ruibo Liu $^{14}$  Yike Guo $^{9}$  Jie Fu $^{1\dagger}$  Multimodal Art Projection Research Community  $^{1}$ Beijing Academy of Artificial Intelligence  $^{2}$ Carnegie Mellon University  $^{3}$ Queen Mary University of London  $^{4}$ University of Manchester  $^{5}$ University of Sheffield  $^{6}$ University of Waterloo  $^{7}$ University of Michigan Ann Arbor  $^{8}$ Beihang University  $^{9}$ Hong Kong University of Science and Technology  $^{10}$ Peking University  $^{11}$ New York University  $^{12}$ MBZUAI  $^{13}$ ICT, Chinese Academy of Sciences  $^{14}$ Dartmouth College ruibiny@alumni.cmu.edu yinghao.ma@qmul.ac.uk yizhi.li@ hotmail.com gezhang@umich.edu chenghua.lin@manchester.ac.uk fujie@baai.ac.cn
In the era of extensive intersection between art and Artificial Intelligence (AI), such as image generation and fiction co-creation, AI for music remains relatively nascent, particularly in music understanding. This is evident in the limited work on deep music representations, the scarcity of large-scale datasets, and the absence of a universal and community-driven benchmark. To address this issue, we introduce the Music Audio Representation Benchmark for universal L Evaluation, termed MARBLE. It aims to provide a benchmark for various Music Information Retrieval (MIR) tasks by defining a comprehensive taxonomy with four hierarchy levels, including acoustic, performance, score, and high-level description. We establish a unified protocol based on 18 tasks on 12 public-available datasets, providing a fair and standard assessment of representations of all open-sourced pre-trained models developed on music recordings as baselines. MARBLE offers an easy-to-use, extendable, and reproducible suite for the community, with clear statements on dataset copyright. Results suggest that recently proposed large-scale pre-trained musical language models perform the best in most tasks, with room for further improvement. The leaderboard and toolkit repository are published<sup>34</sup> to promote future music AI research.
Despite Artificial Intelligence (AI) rapid advancement in the field of art, it has not yet made significant progress in music, particularly in music understanding. To address this, researchers are studying the interdisciplinary field of Music Information Retrieval (MIR) to develop a general music understanding model. MIR focuses on automatically extracting information from raw music audio [40], which enables a variety of tasks such as music classification, emotion recognition, pitch estimation, and the analysis of musical features such as rhythm, melody, and harmony. Due to issues such as copyright and annotation costs, labelled music datasets are usually small, which limits the performances of
*The authors contributed equally to this work.
<sup>3</sup>https://marble-bm.shef.ac.uk
$^{4}$ https://github.com/a43992899/MARBLE-Benchmark
supervised models. Given that self-supervised learning (SSL) is useful for various tasks (e.g., NLP [23, 18, 51] and CV [41]) with limited annotated datasets, there have been works on SSL-based audio representation learning [26, 33, 32, 2, 20, 49, 58] and music pre-trained models [43, 35, 69, 62, 31, 11, 29, 53, 66, 37]. The existing benchmarks, GLUE [57], SuperGLUE [56], and ERASER [10] in NLP, along with VTAB [68] and VISSL [16] in CV, all play an active role in promoting the development of SSL-related research topics in the corresponding domains. However, there are only scattered and fragmented evaluations of the existing music models rather than comprehensive benchmarks, making it difficult to objectively compare and draw insights across techniques.
In the current context, the SSL music systems are evaluated with downstream task datasets, including genre classification [26, 35, 69, 62, 31, 8, 29, 66, 37, 34], emotion classification [35, 31, 8, 29, 37, 34], instrument classification [35, 62, 49, 37, 34], music tagging [69, 31, 8, 53, 43, 29, 66, 37, 34], key detection [31, 8, 29, 37, 34], music detection [49], beat tracking [34] and cover song detection [66]. Existing works usually conduct evaluations with different experimental setups, and few of them explore sequential tasks such as beat tracking and source separation. Although in similar domains, SUPERB [65] and HEAR [54] are proposed to facilitate unified analysis of the learned representations of speech and sound events, the distribution of musical audio is significantly different. Thus, there is an urgent need to construct comparable, extensive, and easy-to-use benchmarks to enhance the development of music SSL.
In this paper, we propose a Music Audio Representation Benchmark for universal Evaluation (MARBLE) to address this problem. MARBLE aims to examine the full spectrum of model capabilities, and thus proposes a taxonomy adapted from Dai et al. [9] to categorise MIR tasks, including acoustic, performance, score, and high-level description. The four-level hierarchy aligned to musician consensus serves as a guideline to further organise the datasets and helps to identify a diversified set of downstream tasks. We select popular tasks in the (now defunct) Music Information Retrieval Evaluation eXchange (MIREX) Challenge<sup>5</sup>, and use the corresponding public datasets with limited annotations. As demonstrated in Tab. 1, the current version of MARBLE contains 18 downstream tasks, spread over 13 task categories on 12 publicly or commercially available datasets. Except for the common classification tasks, we also integrate the missing piece of the puzzle - sequence labelling tasks that require frame-wise prediction, including source separation and beat tracking. The datasets used in MARBLE are ensured easy-to-access: all datasets are available for download directly from the official repository or an external website for downloading a specific version.
In addition, we design a unified protocol and build tool-kits to evaluate the generalisation ability of the models. In MARBLE protocol, the models are regarded as backbones to provide universal representations for all tasks, and task-specific prediction heads are concatenated to further trained under unconstrained, semi-constrained, and constrained settings, which is defined by whether the training hyperparameters are restricted and whether the backbone model is frozen (cf. § 3.2). The evaluation suite provides codes for dataset preprocessing and examples of evaluating existing popular SSL models in the benchmark. We select 7 representative music SSL models as our baselines (cf. § 3.1) and release the evaluation results at our publicly available leaderboards<sup>6</sup> as a reference.
Our key contributions are listed as follows: (1) providing a diversified music understanding benchmark with well-defined taxonomy of the MIR tasks; (2) incorporating and organising a wide range of datasets to facilitate comprehensive music model evaluation; (3) designing a unified assessment protocol and building corresponding evaluation suites for processing, training, and benchmarking.
2 Benchmark Tasks
As demonstrated in Tab. 1, we collect datasets in MARBLE to provide the community with a standard, general-purpose, easy-to-use benchmark for various tasks covering all aspects of music. Generally, music processing involves discriminative and generative tasks. The discriminative tasks either classify or regress musical recordings as a whole or use a seq2seq model to make frame-by-frame decisions on entire sequences. The generative tasks include audio synthesis and music composition. For the initial release of MARBLE, we focus on discriminative tasks, and generative tasks are currently outside our scope. The task collection is guided by the principles of (1) receiving a high level of interest
5https://www.music-ir.org/mirex/wiki/MIREX_HOME
<sup>6</sup>Considering potential legal constraints, MARBLE allows to submit results on the tasks partially (e.g., tasks on commercially available datasets) for the future participants.
in the MIR community, (2) having publicly available datasets allowing everyone to participate, and (3) limited labelled data to effectively measure the universality of the model. Four aspects of music are studied through 18 proposed tasks: High-level description tasks including key detection, music tagging, classification gender, and emotion recognition; Score-level tasks including estimating the pitch of a musical note, tracking beats, extracting melody, estimating the chords, and transcribing the lyrics; Performance-level tasks including detecting musical ornaments or techniques; and Acoustic-level tasks including singer identification, instrument classification, and source separation that focus more on raw audio information.
Table 1: The Dataset, Commercial License, and Prediction Head of Each Task Used for the MARBLE Benchmark. SDR refers Source-to-distortion Ratio.
Taxonomy	Task Type	Task & Annotation	Prediction Type	Evaluation Metrics	Commercially Available
High-level Description	Key Detection	Giantsteps key [25]	Multi-class	Weight Score [45]	Yes
Music Tagging	MagnaTagATune [28]	Multi-label	ROC-AUC & PR-AUC/AP	-
MTG Top50 [7]	Multi-label	ROC-AUC & PR-AUC/AP	-
Genre Classification	GTZAN [55]	Multi-class	Accuracy	-
MTG Genre [7]	Multi-label	ROC-AUC & PR-AUC/AP	-
Emotion Detection	Emomusic [52]	Regression	R2Valence & R2Arousal	-
MTG MoodTheme [7]	Multi-label	ROC-AUC & PR-AUC/AP	-
Score-level	Pitch Classification	Nsynth [14]	Multi-class	Accuracy	Yes
Beat Tracking	GTZAN Rhythm [55]	Seq2Seq, Binary-class	F-measure (Threshold 20ms)	-
Melody Extraction	MelodyDB [4]	Seq2Seq, Multi-class	Accuracy	-
Chord Estimation	GuitarSet [63]	Seq2Seq, Multi-class	8 different mir_eval score	MIT
Lyrics Transcription	MulJam2.0	Seq2Seq, Multi-class	CER, WER	-
Jamendo [12]	Seq2Seq, Multi-class	CER, WER	MIT
Performance-level	Vocal Technique Detection	VocalSet [61]	Multi-class	Accuracy	Yes
Acoustic-level	Singer Identification	VocalSet [61]	Multi-class	Accuracy	Yes
Instrument Classification	Nsynth [14]	Multi-class	Accuracy	Yes
MTG Instrument [7]	Multi-label	ROC-AUC & PR-AUC/AP	-
Source Separation	MUSDB18 [46]	Seq2Seq, Regression	SDR	-
2.1 High-level Description Tasks
Key detection involves predicting the scale and key pitch levels of a song. MARBLE solves this task using the Giantsteps [25] and a subset of the Giantsteps-MTG-keys dataset [27]. Giantsteps dataset contains 604 songs and is taken as our dedicated test set. Additionally, we leverage a subset of the Giantsteps-MTG-keys dataset, which contains 1077 music pieces with single-key annotations, for training and validation. Since no standardised split is available for Giantsteps-MTG, we adopt the dataset split strategy employed in [8]. Both datasets contain 2 minutes of electronic dance music covering all 12 pitch classes in major and minor, resulting in a 24-class classification task. For performance evaluation, we employ accuracy with an error tolerance metric, a weighted score metric. This metric grants partial credit for reasonable errors, such as predicting relative secondary keys when the primary key is the ground truth [45].
Music Tagging refers to assigning a predefined set of tags to a given song. These tags encompass various aspects such as genre, instrumentation, mood, and tempo (e.g., fast), making music tagging somewhat overlap with genre classification, emotion recognition, and instrument classification. To conduct our study, we utilise two extensive datasets: MagnaTagATune (MTT) [28] and MTG-Jamendo (MTG) [7]. The MTT dataset comprises 30-second audio clips with manual annotations for tags. It consists of  $25.9\mathrm{k}$  clips, amounting to a total duration of 170 hours. For MARBLE, we use the Top50 tags, and adopt a conventional (12:1:3) training, validation, and test split, aligning with all baseline approaches' practices. Besides, the MTG dataset contains  $55\mathrm{k}$  clips, corresponding to nearly 2k hours of music. As the audio clips in this dataset may exceed 30 seconds in length, we compute multiple embeddings using a sliding window of 30 seconds and then average them to obtain an overall embedding representation. While both datasets encompass a large number of tags, we follow the customary to limit the vocabulary to the 50 most common tags in each dataset. The evaluation metrics employed for this task are the macro-average of all tag ROC-AUCs (receiver operating characteristic
- area under the curve) and the average precision (AP) / PR-AUC (precision-recall - area under the curve). These metrics provide comprehensive insights into the model's performance across all tags.
Genre classification aims to assign each song the most suitable genre label. This study uses two distinct datasets: GTZAN [55] and MTG-Genre. GTZAN consists of 30-second audio clips from 10 genres, making it suitable for a multi-class classification task. To assess the performance of this dataset, we report the accuracy metric. To ensure consistent evaluation, we utilise the "fail-filtered" split as described in [24] for GTZAN. The filtered dataset comprises 930 audio tracks corresponding to approximately 8 hours of music. Besides, MTG-Genre, derived from MTG-Jamendo, contains 55k tracks but focuses solely on 95 genre tags, resulting in a multi-label classification problem. We employ the ROC and AP metrics to evaluate the performance of MTG-Genre.
Emotion Recognition in music aims to determine the emotional content of music pieces. In our study, we utilise two distinct datasets to evaluate the performance of emotion recognition: Emomusic [52] and MTG-MoodTheme [7]. Emomusic contains 744 pieces of 45-second music clips and is annotated with valence and arousal scores. The valence represents the positivity of emotional responses, while arousal indicates emotional intensity. The official evaluation metrics for this dataset is the determination coefficient  $(r^2)$  between the model's regression results and human annotations of arousal and valence [52]. During inference, we split the 45-second clips into 5-second sliding windows and computed the average prediction probability as the final prediction. Since no standard dataset split is available for Emomusic, we adopt the same partitioning as [8]. It is important to note that direct comparison of the SoTA model's results with the benchmark may be challenging due to the different dataset splits. Additionally, we utilise MTG-MoodTheme, a subset of MTG-Jamendo consisting of  $18.5\mathrm{k}$  audio tracks annotated with 59 human emotion labels. This is a multi-label task with ROC and AP as evaluation metrics.
2.2 Score-level Tasks
Pitch Classification in Music (Monophonic) involves determining the appropriate pitch category for a given audio sample, ranging from MIDI note numbers 0 to 127 on a semitone scale. In this study, we perform pitch classification using the Nsynth dataset [14] within the music information retrieval benchmark. It comprises 340 hours of music, with each excerpt lasting 4 seconds. Since the audio recordings in this dataset are monophonic, the pitch classification task is formulated as a 128-class classification problem, covering all possible MIDI pitch categories (fundamental frequencies from  $8\mathrm{Hz}$  to  $12.5\mathrm{kHz}$ ). The evaluation metric used for this task is the accuracy achieved across all audio clips.
Beat Tracking determines the presence of a beat and a downbeat in each frame of a given music piece. In this benchmark, we only focus on beat tracking, making it a binary-classification task<sup>7</sup>. An offline approach is employed for beat tracking, allowing the model to utilise frame-level information during inference. The model generates frame-by-frame predictions at a specific frequency, which are then post-processed using a dynamic Bayesian network (DBN) [6] implemented with madmom [5] to obtain the final result. The GTZAN Rhythm dataset [36] is used in this study. The dataset provides frame-level annotations for each music clip in GTZAN. To enhance model performance and ensure a fair comparison with the SOTA model, adjacent frames of each beat label are also labelled as beats using a label smoothing technique commonly employed in beat tracking. The model is evaluated using the f_measure metric implemented in mir_eval [45]. A prediction is considered correct if the difference between the predicted event and the ground truth does not exceed 20ms. It is important to note that while some models may have been trained on other datasets, the GTZAN-train subset is used as the training set, and GTZAN-test is used as the test set for all MARBLE submissions.
Chord Estimation is to recognise the temporal music chord of a given piece of music. We implemented this task as a 421-class classification including 35 types of chords on 12 different root notes, and none. More information on the chord vocabulary can be found in Appendix D. The probing model consists of an MLP with a hidden size of 512 and concludes with a fully connected output layer. There is no post-processing involved from frame-level prediction to event-level. The predictions are aligned with the token rate of the pre-trained model. Performances are measured as by 6 measures in SOTA model [22] including root, majmin, mirex, thirds, triads, and sevenths. We have
<sup>7</sup>Due to the limitation of time and the size of the dataset, tracking the time signature (e.g., 4/4 metre) and downbeat is deferred to future versions with other datasets.
added two additional evaluation metrics inspired by MIREX: majmin_inv and sevenths_inv, to assess the performance of chord recognition at the level of inversions. The metrics are implemented by mir_eval[45]. We use the GuitarSet [63] dataset for this task. The dataset comprises 360 excerpts, each around 30 seconds, recorded by 6 players performing 30 lead sheets in two versions (comping and soloing) across 5 styles, 3 progressions, and 2 different tempos. Four audio versions are provided, and we selected the "mix" (a monophonic mixture of the original 6-channel file) for our audio collection. The dataset offers two types of chord annotations for selection, and we chose "performed chord" as our primary annotation and used "instructed chord" to substitute specific colour chords. We divided the audio into 5-second segments and allocated 5 singers into the validation and training sets while designating one singer for the test set. Out of the 5 singers, we allocated  $30\%$  to the validation set and  $70\%$  to the training set. Segments from the same song are assigned to the same set, for instance, all 5-second segments from a particular song are grouped into the training set.
Melody Extraction is to recognise the pitch of melody for a given music, typically pop songs. Adhering to the methodology in [59], we divided the frequency spectrum between 0 and  $8000\mathrm{Hz}$  into 360 bins, treating our task as a classification problem. The probing model consists of a single-layer bidirectional LSTM with a hidden size of 512, followed by a linear layer. The predictions are aligned with the token rate of the pre-trained model, which is then resampled to match the label rate using the nearest interpolation. Performances are measured as the Overall Accuracy metric from the mir.eval [45] library. We use the MedleyDB [4] dataset for this task. It has 108 full tracks, collectively lasting 7.3 hours. All tracks come with three types of melody labels given at intervals of roughly  $5.8\mathrm{ms}$ . For our study, we focused on the second annotation, which indicates the fundamental frequency of mixed stems. For data splitting, we followed the partitioning strategy of [59] giving 67, 15 and 26 tracks for training, validation and testing sets respectively which was achieved after omitting a redundant track from the test set in the popular split.
**Lyrics Transcription** aims to identify the linguistic content in audio recordings of singing. In MARBLE, we focus on the evaluation of multilingual lyrics transcription, an aspect that has been under-explored in the field of lyrics transcription. We perform the task using the MulJam dataset [70], which comprises 6031 songs in 6 languages: English, French, Spanish, Italian, German, and Russian. This dataset offers a rich repository of around 153k lines with lyrics annotations. The training, validation and testing sets contain 147k, 3k and 2k lines, respectively. We set a standard train-valid/test splitting and re-labelled the MulJam dataset as MulJam2.0 with more human annotation. More information can be found in Appendix E. We also use Jamendo [12] as a test set which includes English, French, German and Spanish pop songs. There are 20 songs for each language and the dataset comes with line-level human annotation for lyrics. In line with recent literature [15, 42], the backend adopts a hybrid CTC/Attention architecture design [60]. Given the task's complexity and the necessity to capture long-term dependencies, we use a transformer with 3 encoder layers and 3 decoder layers. The output from the encoder is further processed by a fully connected layer to map it to the target dimension for the CTC loss computation [17]. We also calculate a sequence-to-sequence (S2S) loss between the output from the decoder and the true lyrics text. The final loss is a balanced combination of the CTC loss and the S2S loss. For validation and testing, we employ beam search on the transformer decoder to iteratively select the best predictions. Additionally, a transformer language model is trained from the same data split to incorporate language knowledge at test time. Performance evaluation is conducted using Character Error Rate (CER) and Word Error Rate (WER). Different from all the metrics in other tasks, WER and CER values are the less the better.
2.3 Performance-level Tasks
Vocal Technique Detection task involves identifying different singing techniques within an audio clip. For this task, the MARBLE benchmark utilises the VocalSet dataset [61], the sole publicly available dataset specifically designed for studying singing techniques. This dataset comprises recordings of 20 professional singers (9 female and 11 male) performing 17 distinct singing techniques in various contexts, amounting to a total duration of 10.1 hours. Given that the audio clips are segmented into 3-second intervals, the task focuses on determining the type of technique (e.g. Vibrato, Straight) rather than the precise start and end times. To evaluate the performance of models, we employ Accuracy as the evaluation metric. We use a subset of 10 different singing techniques used in Yamamoto et al. [64], which contains 15 singers in the training and validation set, and 5 for the test set. Since there is no predetermined division between the training and validation sets, we assign 9 singers to the training
set and 6 singers to the validation set. It is important to note that all 3-second segments originate from the same audio recording file within the same part of the split, such as being exclusively part of the training set. Detailed data partitioning can be found in our provided code.
2.4 Acoustic-level Tasks
Instrument Classification refers to the multi-label or multi-class identification of instruments present in a given audio recording. In the MARBLE benchmark, we utilise two datasets: Nsynth and MTG-instrument. The Nsynth dataset comprises 306,000 audio tracks, each corresponding to one of 11 different instruments. The evaluation metric for this dataset is accuracy. On the other hand, MTG-instrument is a subset of MTG-Jamendo, containing 25,000 audio tracks and 41 instrument tags. Each track can have multiple instrument tags and is evaluated based on ROC and AP.