SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Luca A. Lanzendorfer
lanzendoerfer@ethz.ch
Emil Funke
efunke@ethz.ch
Florian Grötschla
fgroetschla@ethz.ch
Roger Wattenhofer
wattenhofer@ethz.ch
Music datasets play a crucial role in advancing research in machine learning for music. However, existing music datasets suffer from limited size, accessibility, and lack of audio resources. To address these shortcomings, we present DISCO-10M, a novel and extensive music dataset that surpasses the largest previously available music dataset by an order of magnitude. To ensure high-quality data, we implement a multi-stage filtering process. This process incorporates similarities based on textual descriptions and audio embeddings. Moreover, we provide precomputed CLAP embeddings alongside DISCO-10M, facilitating direct application on various downstream tasks. These embeddings enable efficient exploration of machine learning applications on the provided data. With DISCO-10M, we aim to democratize and facilitate new research to help advance the development of novel machine learning models for music.<sup>1</sup>
Music is a universal language that has captivated and inspired humans for millennia. With the rapid advancements in technology, the domain of music has become increasingly interconnected with the field of machine learning, opening up new possibilities for music analysis, music recommendation systems, and the generation of novel compositions. Central to these developments are large and high-quality music datasets that serve as the foundation for training and evaluating modern machine learning models.
The recent breakthroughs achieved by large language models in the textual domain and diffusion models in the visual domain have been largely attributed to the availability of massive datasets. These datasets have played a pivotal role, enabling large models to learn intricate patterns and generate coherent output. However, in the realm of music, the availability of large-scale datasets has remained relatively limited, impeding the same advancements of research in this domain.
In the field of text processing, datasets such as CommonCrawl [1] and the Pile [13] have made substantial amounts of written content available for training large language models. For instance, the Pile contains an extensive dataset of 825 gigabytes of text, while CommonCrawl contains a staggering 3.3 terabytes of text. These datasets capture a significant portion of the writing available on the Internet, providing an abundant resource for training and evaluating language models.
<https://huggingface.co/DISCOX
Table 1: Comparison of public music datasets.
Dataset	# Clips	# Artists	Track duration	Year	Audio
GTZAN	1,000	300	30s	2002	yes
MagnaTagATune	25,863	230	29s	2009	yes
MTG-Jamendo	55,609	3,565	full-length	2019	yes
Free Music Archive	106,574	16,341	full-length	2017	yes
Music4All	109,269	16,269	30s	2020	yes
Million Song Dataset	1,000,000	44,745	full-length	2011	no2
AudioSet1	1,011,305	-	10s	2017	no2
AcousticBrainz	2,524,739	-	-	2015	no
DISCO-10M	15,296,232	400,047	full-length	2023	no2
1 Only 1,011,305 out of 2,084,320 clips are labeled as Music.
2 Audio not directly available, can be downloaded from YouTube.
Similarly, in the visual domain, there exist various large datasets, such as the famous ImageNet [9] dataset, Open Images [19], Laion-400M [26], and Laion-5B [27] dataset, which all contain massive amounts of images. These datasets have been instrumental in the advancements of computer vision research. Especially Laion-5B, with its five billion image-text caption pairs, has been essential for the success of current text-to-image generative models.
Contrary to the textual and visual domains, existing music datasets face several limitations that hinder the progress of research in machine learning for music. One of the primary challenges is the limited size of available datasets, which restricts the diversity and representativeness of the musical content that can be analyzed, as well as the use-cases that can be tackled. Additionally, accessibility to these datasets has been a concern, with many of the state-of-the-art models for audio and music being trained on proprietary datasets that are not accessible to the broader research community [11, 16, 37, 8, 18, 2, 17]. Moreover, the scarcity of available audio recordings poses a significant hurdle in obtaining datasets and, therefore, training novel machine learning models for music.
To address these shortcomings, we introduce DISCO-10M, a novel and extensive music dataset that surpasses the largest previously available music dataset by an order of magnitude. By curating DISCO-10M, our objective is to overcome the limitations of existing datasets and provide researchers with a rich and diverse collection of music data, enabling further advancements in the field of machine learning for music.
In addition to the extensive music dataset, we also provide precomputed audio embeddings alongside DISCO-10M, which we obtain from a pre-trained open-source CLAP model [35]. By including precomputed audio embeddings, we facilitate the direct application of machine learning models on various downstream tasks, eliminating the need for time-consuming audio retrieval and embedding computation, and reducing the barrier to entry for researchers in the field.
The availability of DISCO-10M aims to democratize access to large and high-quality music data for the research community. We envision that this dataset will serve as a catalyst for new research endeavors, inspiring researchers to explore novel machine learning models, techniques, and applications in the domain of music. By advancing our understanding of music through machine learning, we can foster the development of innovative music analysis tools, personalized recommendation systems, and creative music generation models.
Music Datasets. We list some of the larger music datasets which have influenced our work. While GTZAN is small compared to other music datasets, it was one of the first datasets, originally intended for music genre recognition [32]. The dataset contains 1000 files, each with a 30-second music clip. Every song belongs to one of ten categories: Blues, Classical, Country, Disco, Hip Hop, Jazz, Metal, Pop, Reggae, and Rock. The data was gathered from various sources, including personal CDs, radio, and microphone recordings, and therefore varies greatly in quality. Furthermore, GTZAN does not
provide other metadata, such as artist or album names. Even though GTZAN is small compared to more recent datasets, it remains a widely used dataset for music information retrieval tasks [31].
MagnaTagATune is a dataset that contains 25,863 music clips, each 29 seconds long [20]. The music clips are extracted from 5223 unique songs across 230 artists. The dataset provides 188 tags for each music clip, crowd-sourced from TagATune. TagATune is an online two-player game where each player creates tags for a music clip. Afterward, players must decide based on both their created music tags if they were presented with the same music clip. If a music clip receives the same tag from both players, the tag is added to the music clip in the dataset.
Million Song Dataset (MSD) contains metadata and audio features for one million songs but does not contain the audio files [4]. The dataset was initially created to bridge the gap between academia and industry by enabling academia to benchmark music information retrieval algorithms on industry-scale datasets.
AcousticBrainz is an open source and community-driven database of 2.5 million music entries [22]. While it is the largest music database, it does not distribute music or store links to third-party sites where the music can be found. Like MSD, AcousticBrainz stores audio features and metadata such as overall loudness, rhythm, genres, and instrumentation.
Free Music Archive (FMA) dataset was the first publicly available dataset to contain more than 100,000 freely available full-length music files across 16,000 artists [7]. The dataset is sourced from a freeform radio station of the same name, $^{3}$  which provides music released under the Creative Commons license. FMA provides fine-grained hierarchical genre metadata across 161 genres and contains high-quality audio.
AudioSet is a dataset containing two million YouTube video IDs with associated audio classes [14]. AudioSet was created as a dataset for audio event detection and to train and evaluate machine perception. Each dataset entry has a YouTube ID and a start and end timestamp. Furthermore, the dataset contains an audio label and a short audio description. However, since we are specifically interested in music datasets, it should be noted that only one million IDs contain the music label. Similar to MSD, the audio needs to be downloaded from YouTube as it is not provided with the dataset.
MTG-Jamendo was introduced as a dataset for automatic music tagging [5]. The data is a subset of music sourced from the Jamendo website, which contains royalty-free music. Jamendo offers music for commercial use, which means the music is generally of higher quality. The dataset contains music tracks with 692 tag annotations, consisting of genre, instrument, and mood tags.
Music4All is a dataset that contains 109,269 music clips across 16,269 artists [21]. Each music clip is 30 seconds long and is cut from the middle of the original track. The dataset was created by collecting 15,602 users and their listening histories from last.fm. The music from the listening histories was downloaded from YouTube and post-processed by adding various features. The dataset contains 26 different features for each sample, such as the music clips and tags from last.fm, audio features from Spotify, and lyrics from MusiXMatch.
Embedding models. Learning latent representations of data has been an active field of research over the past decade [3, 23, 15, 33, 6]. One of the more recent advances in representation learning comes from language-image pre-training, where a CLIP (Contrastive Language-Image Pre-training) model was trained on 400M pairs of language-image samples and obtained state-of-the-art zero-shot classification results [24]. The resulting model learned a joint embedding space for both images and texts. This idea was extended to audio with CLAP (Contrastive Language-Audio Pre-training) [12]. We use Laion-CLAP [35], an open-source alternative to CLAP. Although we could also embed text with CLAP, we decided to use it only for audio. For text, we use Sentence BERT [25]. Sentence BERT is trained by fine-tuning a BERT model via a siamese and triplet network to obtain semantically meaningful text embeddings [34, 10].
$^{2}$ https://tagatune.org/
<sup>3</sup>https://freemusicarchive.org/
<sup>4</sup>https://www.jamendo.com/
<sup>5</sup>https://www.last.fm/
<https://www.musixmatch.com/>
Figure 1: Overview of pipeline to create DISCO-10M. Starting from a seed list of artists, we crawl the Spotify artist graph and collect related artists and their metadata. For each artist, we collect the metadata of up to ten "top tracks" and use them to search for up to twenty YouTube videos. The YouTube metadata, together with the Spotify metadata, are used to filter out datapoints according to their duration, title, and description similarity. The remaining matches are downloaded from Spotify and YouTube and compared. A datapoint is discarded if the similarity is below the threshold.
Music Providers. We use Spotify, one of the most popular music, podcast, and audiobook streaming platforms, with over 515 million users, 9 million artists, and more than 100 million available songs [28, 30]. Furthermore, we use YouTube, one of the largest on-demand video streaming platforms, with more than two billion monthly users and more than 500 hours of video uploaded every minute [36]. YouTube offers a wide variety of content on its platform and allows anyone to upload their artistic creations to YouTube. This allows us to find audio samples for relatively unknown artists since the barrier to entry on YouTube is considerably lower than on Spotify. We use YouTube to expand our Spotify data with music videos and additional metadata.
3 Data Collection
Figure 1 gives an overview of the data collection process for DISCO-10M. We compute a list of Spotify artists generated with a breadth-first search over related artists, starting from a seed list of artists that covers multiple genres. We then take the most popular songs for every artist, according to Spotify, and search for the songs on YouTube. At this point, we end up with a one-to-many mapping of songs to YouTube search results that we filter to improve the quality of the matches. In the first filtering step, we apply filters based on duration, the similarity between Spotify song title and YouTube title, and the similarity between the song title and the video description. We then download Spotify previews and YouTube videos for these prefiltered datapoints to generate CLAP audio embeddings. The similarity of these embeddings is used in the last filtering step. A full list of all metadata columns that we provide for DISCO-10M can be found in Appendix B.
3.1 Spotify Artist Graph
Our goal is to obtain songs from diverse artists that span multiple genres. To decide which artists to consider, we start with a hand-curated list of seed artists (cf. Appendix A). We chose popular artists that are influential and represent their genre. To explore additional artists, we look for artists that "fans also like", a feature provided by Spotify [29]. Starting from the seed list, we explore these related artists one hop at a time, adding all related artists to the set of artists we already know. In other words, we obtain a directed graph if we represent artists as nodes and add edges between them according to their related artist relationship. We now consider the nodes representing artists in our seed list and run a breadth-first search on this graph, discovering a subset of all Spotify artists.
Figure 2: Distribution of data before each filtering step. We first filter the data according to duration similarity a). The data is then filtered if it is either below the title similarity threshold b) or below the description similarity threshold c). Finally, the data is filtered according to the audio similarity d).
Table 2: Examples for different levels of text similarity between search query and YouTube video title
Search query	YouTube video title	Text similarity
Take On Me	Danelectro Doublenack	0.382
Superton	Guitar&Bass
Lead Me	Eddie Neblett - Reset	0.663
Eddie Neblett	(Official Lyric Video)
Che pyhare mombyry	Che pyhare mombyry	0.964
- Version polac Perfil	(Version polac)
Table 3: Examples for different levels of text similarity between search query and YouTube video description
Search query	YouTube Description Snippets	Text similarity
Higher Ally Brooke	Listen with headphones on!! —CLICK!! — Hello everyone! My name is Dewi, and I'm making Youtube Empty Arena, Slowed...	0.115
I Feel You Toly Braun	#tolybraun#deephouse.	0.657
Nobody Knows When You're Down And Out Scrapper Blackwell	Nobody Knows You When You're Down and Out by Scrapper Blackwell.	0.955
We stop after we find about 400,000 unique artists. While we collect this data, we also accumulate metadata provided by Spotify; this includes the artist name, artist music genres, artist popularity on Spotify, and the total number of Spotify followers for each artist in our list.
We take up to 10 "top tracks" per artist to sample songs for our dataset. The "top tracks" classification is provided by Spotify and is based on the listening behavior of users in the US market. We end up with approximately 2.5M unique songs and store metadata such as track name, album ID, artist ID, explicit flag, track duration, track release date, and a track preview URL. After collecting the song information, we can use it to search for videos on YouTube with a lookup containing the name of the song and the name of the artist.
3.2 Filtering
After the previous steps, we are left with a one-to-many mapping of Spotify songs to YouTube search results. After cleanup of the data, we keep the following fields: the Spotify track ID, its corresponding track metadata, artist metadata from Spotify, and a YouTube search result with its corresponding metadata.
When keeping all search results from YouTube, we have 46.8M matches. To improve the quality of the dataset and remove bad matches, we filter the dataset by duration similarity  $\delta_{d}$ , followed by text similarity  $\delta_{yt}$  and  $\delta_{yd}$ , and finally by audio similarity  $\delta_{a}$ . We empirically find that the following filtering thresholds work well:
$$
\left(\delta_ {d} > 0. 2 5\right) \wedge \left(\delta_ {y t} > 0. 6 5 \vee \delta_ {y d} > 0. 6 5\right) \wedge \left(\delta_ {a} > 0. 4\right)
$$
Applying duration and text filtering stages removes approximately half of all entries from our dataset, which results in the dataset containing 20M entries before audio similarity filtering. We filter the remaining entries based on their similarity between the Spotify preview and the YouTube audio embedding.
We demonstrate the effect of our filtering pipeline on the data distribution in Figure 2. In a), we observe a small peak in the duration similarity at around 0.05. This is because YouTube search results contain relatively long videos compared to Spotify song duration. These long videos tend to be multi-hour versions of songs, full movies, or news broadcast live-streams. Therefore, we set the filtering cut-off in terms of duration such that YouTube videos are at most four times as long as the Spotify song.
The density distributions shown in Figure 2 b), c) are the basis for the title and description filtering. We found relatively strict filtering to work best as it only keeps good matches and reduces the number of datapoints we have to process.
Figure 3: Comparison of audio similarity between Spotify preview audio and YouTube audio.  $\delta_{a}$  denotes the cosine similarity of the audio embedding. We use Log-Mel Spectrograms to visualize audio. We empirically observe that the similarity of our audio embeddings is related to the similarity of the Log-Mel Spectrograms and that the similarity increases when the spectrograms are closer to each other.
Using the OR operator in the comparison, we still allow some titles and descriptions with lower similarities to pass to the next filtering stage, as long as one of the two similarities is high enough. Table 2 provides examples of search queries and corresponding YouTube video titles, where matches with similarity scores above our threshold will continue to the next stage. The same is shown in Table 3 for YouTube video descriptions.
Duration Similarity The duration similarity  $\delta_{d}$  is computed from the Spotify track duration  $t_s$  and the length of the YouTube video  $t_y$  as follows:
$$
\delta_ {d} = 1 - \frac {\left| t _ {s} - t _ {y} \right|}{\max (t _ {s} , t _ {y})},
$$
where  $|\cdot|$  denotes the absolute value. We chose this similarity metric because it intuitively captures the difference in audio duration, expressing the similarity as a percentage difference between longer and shorter tracks. This allows for fine-granular filtering of the data, where track  $t_s$  and  $t_y$  have the same duration when  $\delta_d = 1$ , and very dissimilar duration as  $\delta_d \to 0$ .
Text Embedding Similarity We use the YouTube search query, the YouTube video title, and the YouTube video description to measure the similarity and thus the match of the resulting video compared to our search query. All three texts are embedded into a latent space using Sentence BERT [25]. To measure the similarity of embeddings, we use a cosine similarity between the search query and the video title, denoted as  $\delta_{yt}$ , and the cosine similarity between the search query and the video description, denoted as  $\delta_{yd}$ . This allows us to filter poor matches in our data before downloading Spotify previews and YouTube videos, which are used to compute the audio embeddings. We qualitatively compare text embedding matches for YouTube video titles in Table 2 and for YouTube descriptions in Table 3.
Audio Embedding Similarity After applying text-based filters, we download Spotify audio track previews and YouTube videos to compute audio similarity and further filter the dataset. The pipeline to compute the audio embedding similarity for one datapoint is as follows: We download the Spotify
Figure 4: Left Song duration for Spotify and YouTube after applying our filtering pipeline. The average track length is 3.5 minutes. We observe a strong correlation between Spotify and Youtube track duration, in part due to the duration similarity filtering. Middle Number of tracks and number of explicit tracks released per year. Right Genre distribution of GTZAN genres in our dataset. The genre embeddings were computed using a CLAP encoder on the sentence This audio is a <genre> song. Each song is mapped to a genre according to the largest cosine similarity between the song embedding and the genre embedding.
overview of the song together with the corresponding YouTube video. Both are then fed into the Laion-CLAP audio encoder model to produce an audio embedding for each audio sample. Once we have the embedding of the Spotify preview audio snippet and the YouTube audio, we compute the cosine similarity of the embedding vectors. We denote the audio similarity as  $\delta_{a}$ .
We qualitatively evaluate the audio embedding and similarity scores in Figure 3. To compare the similarity between the two audio tracks, we use the perceptually relevant log-mel spectrogram. Since the Spotify preview is only 30 seconds, and the YouTube audio contains the entire track, we apply a template matching algorithm to find the best overlapping spectrogram section of 30 seconds in the YouTube audio. We observe an increase in spectrogram similarity and, therefore, audio similarity as  $\delta_{a}$  increases.
3.3 Subsets
In addition to the DISCO-10M dataset, we propose several subsets with varying sizes and quality presets to enable fast prototyping as well as development with less powerful hardware.
- DISCO-10K-random contains 10,000 random samples from DISCO-10M. This split is mainly intended for prototyping and tasks that require substantially less data. It also enables training with fewer computational resources and hardware with memory limitations, even when all audio embeddings are loaded into memory.
DISCO-200k-random contains 200,000 random samples and was used for the data analysis in the following section. The dataset is small enough to remain manageable on common hardware while still being a statistically meaningful representation of our dataset.
- DISCO-200k-high-quality contains a selective subset of DISCO-10M with strict filtering based on duration, text, and audio similarity values. The matches of Spotify titles to YouTube videos are better than in DISCO-10M, resulting in a dataset that can be used for tasks where high-quality matches are required.
We analyze the distributions of the duration of the song and the release year. Due to our similarity filtering, duration distributions for Spotify and YouTube match each other, with an average song duration of about 3.5 minutes. As the release dates are taken from Spotify metadata, they represent the date of the audio recording. The earliest date of recordings in our dataset is in the 1960s; while there are some individual songs that have an earlier date in the metadata, we remove them as outliers. We also observe that the number of songs released per year grows rapidly. This coincides with the
Figure 5: t-SNE plots for Spotify preview embeddings and Youtube audio embeddings computed with CLAP. Colors represent GTZAN genres that were computed from the Spotify embeddings with zero-shot genre classification. The t-SNE plots show the relative positions of music samples, where samples with similar embeddings in the CLAP latent space are located closer to each other and dissimilar points farther apart. We observe that genres are well separated for both the YouTube embeddings and Spotify embeddings. Genres like classical music and metal lie far apart, whereas genres such as metal and rock lie close together.
● Classical	● Country	● Pop	● Reggae	● Jazz
● Rock	● Hip hop	● disco	● Metal	● Blues