SlowGuess/NeurIPS_2023_OCR huggingface dataset:
proliferation of digital music. The cutoff date for our dataset is Spring 2023. We plot song duration and release years for songs in DISCO-10M in Figure 4.
To analyze the genre distribution in DISCO-10M, we utilize the precomputed CLAP embeddings of the 30-second Spotify previews for zero-shot genre classification. We do so by first computing text embeddings of genre prompts and then identifying the closest genre embedding using cosine similarity in the shared latent space for every song. We compute the genre text embeddings with the prompt: This audio is a <genre> song. Note that the genre classification is based on YouTube embeddings, of which multiple variations can exist per song, as DISCO-10M contains approximately 2.6M Spotify songs that are mapped to 15.3M music videos on YouTube.
Figure 4 shows the distribution of GTZAN genres on a representative slice of the dataset. Based on this classification, DISCO-10M is mainly composed of pop and disco music, but offers a broad spectrum overall. We further depict the proportion of explicit songs for each genre. We find that Hip-Hop songs have the highest percentage of explicit songs in our dataset, at around  $50\%$ .
To evaluate the quality of our genre classification, we compute t-SNE plots on the Spotify and Youtube embeddings, together with their assigned genre (see Figure 5). We observe a separation of genres, where the overlap of neighboring genres is also reasonable, e.g., for metal and rock. Classical music is the most fragmented genre, whereas other genres build more contiguous areas. This aligns well with our expectations regarding the genre distribution, lending it a certain degree of credibility.
5 Limitations and Ethical Discussion
We collected data from our server located in Zurich, Switzerland, between January and June 2023. As search results are impacted by time and location, our search results are likely different from other locations, adding bias to our dataset and hindering reproducibility. The artist graph we obtained from Spotify is also subject to changes, and our dataset represents a snapshot of the graph during the mentioned time frame.
Furthermore, the dataset we distribute contains links to online resources for songs and videos that we do not control. These resources can change over time – they may be removed or replaced entirely
by YouTube, Spotify, or their creators. However, since DISCO-10M can contain multiple matching YouTube results for a single song, this redundancy helps counteract dataset degradation over time. Due to this redundancy, we may also include search results that might not represent the original song but something related to it. We believe these samples are still valuable, but users must be aware of this limitation. There is no guarantee for the correctness of the match between the Spotify track and the corresponding YouTube video. For tasks where higher match quality is important, we provide the subset DISCO-200k-high-quality.
The metadata for each song in our dataset is either produced by us (i.e., audio and text embeddings) or available online (i.e., song duration, video duration, view count, etc.). We only provide links to publicly available sources and do not own the copyright of any music referenced in the dataset.
Additionally, the list of artists collected by crawling the "fans also like" section on Spotify might be skewed towards certain genres, artists, or cultures, which can result in a biased representation of music. This can perpetuate stereotypes or hinder the inclusivity of diverse musical expressions.
Another concern stems from restrictions during the data collection process: We only access, download, and process YouTube videos that are not age-restricted. This can lead to some bias in the data, although it should be mentioned that not all Spotify songs that are marked as explicit are also restricted on YouTube. The explicit flags provided by Spotify and the exclusion of age-restricted YouTube videos allow for easier filtering of non-harmful content, but we cannot guarantee that the dataset does not contain hateful or offensive content after filtering. Therefore, in its current form, we strongly advise using this dataset for academic purposes only.
We introduce DISCO-10M, an extensive and novel music dataset that surpasses existing music datasets by an order of magnitude. The data collection process involves leveraging popular songs from Spotify and identifying corresponding YouTube videos through careful search result filtering, ensuring the inclusion of high-quality data. To enhance the usability of the dataset, we provide embeddings for both song previews from Spotify and embeddings from YouTube videos, enabling researchers to apply their own thresholds for further filtering and facilitating efficient exploration of downstream tasks and data analysis. Our analysis demonstrates DISCO-10M's broad coverage of songs from various genres.
Moreover, the paper highlights the limitations of the dataset and addresses the ethical impact associated with its creation and usage. Recognizing the importance of ethical considerations, we encourage responsible and mindful utilization of DISCO-10M, acknowledging the potential implications and ensuring that future research and applications derived from the dataset are conducted with care.
Overall, DISCO-10M represents a valuable resource for the research community, providing a large-scale music dataset that can be used to advance various aspects of music analysis, personalized recommendation systems, or creative music generation models. Its comprehensive coverage, combined with the availability of embeddings, contributes to the broader field of music research by democratizing access to large-scale music datasets.
[1] URL https://commoncrawl.org/.
[2] A. Agostinelli, T. I. Denk, Z. Borsos, J. Engel, M. Verzetti, A. Caillon, Q. Huang, A. Jansen, A. Roberts, M. Tagliasacchi, M. Sharifi, N. Zeghidour, and C. Frank. Musiclm: Generating music from text. ArXiv, abs/2301.11325, 2023.
[3] Y. Bengio, A. C. Courville, and P. Vincent. Representation learning: A review and new perspectives. IEEE Transactions on Pattern Analysis and Machine Intelligence, 35:1798-1828, 2012.
[4] T. Bertin-Mahieux, D. P. W. Ellis, B. Whitman, and P. Lamere. The million song dataset. In International Society for Music Information Retrieval Conference, 2011.
[5] D. Bogdanov, M. Won, P. Tovstogan, A. Porter, and X. Serra. The mtg-jamendo dataset for automatic music tagging. In International Conference on Machine Learning, 2019.
[24] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, and I. Sutskever. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, 2021.
[25] N. Reimers and I. Gurevych. Sentence-bert: Sentence embeddings using siamese bert-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 11 2019. URL http://arxiv.org/abs/1908.10084.
[26] C. Schuhmann, R. Vencu, R. Beaumont, R. Kaczmarczyk, C. Mullis, A. Katta, T. Coombes, J. Jitsev, and A. Komatsuzaki. Laion-400m: Open dataset of clip-filtered 400 million image-text pairs. ArXiv, abs/2111.02114, 2021.
[27] C. Schuhmann, R. Beaumont, R. Vencu, C. Gordon, R. Wightman, M. Cherti, T. Coombes, A. Katta, C. Mullis, M. Wortsman, et al. Laion-5b: An open large-scale dataset for training next generation image-text models. arXiv preprint arXiv:2210.08402, 2022.
[28] Spotify. About Spotify, . URL https://investors.spotify.com/about/default.aspx. Spotify Terms of Service https://developer.spotify.com/terms.
[29] Spotify. How "fans also like" works, . URL https://artists.spotify.com/en/blog/how-fans-also-like-works.
[30] Spotify. Number of artists on Spotify, . URL https://loudandclear.byspotify.com.
[31] B. L. Sturm. The state of the art ten years after a state of the art: Future research in music information retrieval. Journal of new music research, 43(2):147-172, 2014.
[32] G. Tzanetakis, G. Essl, and P. Cook. Automatic musical genre classification of audio signals, 2001. URL http://ismir2001.ismir.net/pdf/tzanetakis.pdf.
[33] A. van den Oord, Y. Li, and O. Vinyals. Representation learning with contrastive predictive coding. ArXiv, abs/1807.03748, 2018.
[34] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. Attention is all you need, 2017.
[35] Y. Wu, K. Chen, T. Zhang, Y. Hui, T. Berg-Kirkpatrick, and S. Dubnov. Large-scale contrastive language-audio pretraining with feature fusion and keyword-to-caption augmentation, 2023.
[36] YouTube. Copyright transparency report. URL https://storage.googleapis.com/transparencyreport/report-downloads/pdf-report-22_2022-1-1_2022-6-30_en_v1.pdf. YouTube Terms of Service https://www.youtube.com/t/terms.
[37] N. Zeghidour, A. Luebs, A. Omran, J. Skoglund, and M. Tagliasacchi. Soundstream: An end-to-end neural audio codec. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 30:495-507, 2022.
A Seed artists
We use the following artists as a seed list for the traversal of the artist graph on Spotify.
startingartists = [ '2ye2Wgw4gimLv2eAKyk1NB', # Metallica '3WrFJ7ztbogyGnTHbHJF12', # The Beatles '74ASZWbe4IxAubB36ztrGX', # Bob Dylan '1Xyo4u8uC1ZmMpatF05PJ', # The Weeknd '4y6J8jwRAwO4dssiSmN91R', # Muddy Waters '776Uo845nYHJpNaStv1Ds4', # Jimi Hendrix '6kACVPfCOnqzgfEF5ryl0x', # Johnny Cash '67ea9eGLXYMsO2eYQRui3w', # The Who '7dGJo4pcD2V6oG8kP0tJRR', # Eminem '5pKCCKE2ajJHZ9KAiaK11H', # Rianna '0dmPX6ovclgOy8WWJaFEUU', # Kraftwerk '1ZwdS5xdxEREPySFridCfh', # 2pac '0kbYTNQb4Pb1rPbbaF0pT4', # Miles Davis '6tbjWDEIzxoDsBA1FuhfPW', # Madonna '7guDJrEfX3qb6FEbdPA5qi', # Stevie Wonder '2QsynagSdAqZj3U9HgDzjD', # Bob Marley '5aIqB5nVVvmFsvSdExz408', # Johann Sebastian Bach '0Kekt6CKSo0m5mivKcoH51', # Sergei Rachmaninoff ]
B Columns in DISCO-10M Dataset
dataset.columns = [ 'video_url_youtu', 'video_title_youtu', 'track_name_spotify', 'video_duration_youtu(sec', 'preview_url_spotify', 'video_view_count_youtu', 'video thumbnails_url_youtu', 'search_query_youtu', 'video_description_youtu', 'track_id_spotify', 'album_id_spotify', 'artist_id_spotify', 'track_duration_spotify-ms', 'primary_artist_name_spotify', 'track_release_date_spotify', 'explicit_content_spotify', 'similarity_duration', 'similarity_query(video_title', 'similarity_query_description', 'similarity.audio', 'audio_embedding_spotify', 'audio_embedding_youtu', ]
C Additional Examples: Audio Similarity Comparison
We demonstrate the results of our audio similarity approach on five additional samples (see Figure 6). Similarly to the spectrograms presented in Section 3.2, we also observe an overlap for these samples when the audio similarity is above  $\delta_{a} > 0.4$ . Even when two music snippets have the same frequency characteristics, there might still be small differences. This can be explained in part due to the audio
quality of a YouTube video, which is dependent on the quality selected by the person uploading the video and can, therefore, vary greatly, unlike the audio quality of Spotify. This difference can be seen best in the high-frequency content of the spectrogram, which tends to be weaker and less pronounced in the YouTube audio samples. We notice a strong dissimilarity in the first example between the Spotify preview audio spectrogram and the YouTube audio spectrogram. This is reflected by the low similarity score of  $\delta_{a} = 0.1985$ .
Figure 6: Comparison of audio similarity between Spotify preview audio and YouTube audio.  $\delta_{a}$  denotes the cosine similarity of the audio embedding. We observe that the similarity of our audio embeddings is related to the similarity of the Log-Mel Spectrograms, and that the similarity increases when the spectrograms are closer to each other.
D FMA Genre Analysis
We repeat the zero-shot genre classification from Section 4 for the 16 FMA root genres. Figure 7 shows the genre distribution for the FMA genres, while Figure 5 depicts the same t-SNE plot as shown in Figure 5 for these genres. We can observe that our results between overlapping genres are consistent, and although there are more genres, we can observe meaningful relationships between them.
Figure 7: Genre distribution of FMA root genres in our dataset. The genre embeddings were computed using a CLAP encoder on the sentence This audio is a <genre> song. Each song is mapped to a genre according to the largest cosine similarity between the song embedding and the genre embedding.
Figure 8: t-SNE plots for Spotify preview embeddings and YouTube audio embeddings computed with CLAP. Colors represent FMA root genres that were computed from the Spotify embeddings with zero-shot genre classification. The t-SNE plots show the relative positions of music samples, where samples with similar embeddings in the CLAP latent space are located closer to each other and dissimilar points farther apart. We observe that genres are well separated for both the YouTube embeddings and Spotify embeddings.
E Subset Details
As described in Section 3.3, we provide three different subsets of DISCO-10M.
DISCO-10K-random contains 10,000 random samples from DISCO-10M. We select the samples randomly from unique Spotify track IDs, meaning that the dataset will contain exactly one YouTube video link per Spotify song.
DISCO-200k-high-quality contains 200,000 high-quality samples filtered more strictly to improve the quality of the matches. We created this subset by filtering DISCO-10M with  $(\delta_{a} > 0.7)\wedge (\delta_{yt} > 0.8)$
F Audio Characteristics
The total playtime of YouTube videos in DISCO-10M is approximately 1,062,604 hours or 121 years.
We provide further insights on the audio attributes of the sample rate, MP3 file bitrate, and number of channels based on the DISCO-10K-random subset. When downloading Spotify previews in MP3 format, the audio quality and characteristics remain consistent. All audited samples share common attributes: a sample rate of  $44.1\mathrm{kHz}$ , a bitrate of 96 kbps, and a 2-channel stereo setup.
In the case of audio from YouTube, there is a noticeable resemblance, albeit slightly less uniform.  $99.78\%$  of the examined videos employ a standard stereo 2-channel configuration.  $0.18\%$  of videos are mono channel, while  $0.04\%$  utilize 6-channel surround for audio output.  $99.96\%$  of videos have a sample rate of  $44.1\mathrm{kHz}$ , aligning with the settings of Spotify previews—the remaining  $0.04\%$  deviate by having a sample rate of  $48\mathrm{kHz}$ .
I Ethical Considerations
Copyright and licensing agreements are a complex issue, particularly when it comes to big data collection for training large machine learning models. We acknowledge the concerns of artists regarding the potential negative impact on their artistic work. However, we believe that openly sharing such data helps democratize the research of music understanding and music creation.
To address artists who disagree with our assessment, we offer two options for reconciliation. First, artists can contact us to request the removal of links associated with their art from our dataset.<sup>7</sup> Second, artists may choose to take down their YouTube video or Spotify song from the respective platform, rendering the link contained in DISCO-10M invalid. It is important to note that our guarantee applies solely to our dataset, while other entities who hold private audio datasets may not offer the same level of control.
Creating a dataset of this magnitude is achievable using publicly available tools and a reasonable timeframe. By making our dataset open-source, we also aim to raise awareness on the ease of creating big datasets and uncover the potential existence of similar datasets held by private institutions. Our goal is to provide an opportunity for anyone interested to explore ideas with this dataset, and to enhance our understanding of music creation and safety with large datasets. We emphasize that this dataset serves as a starting point, pushing the boundaries and fostering research of enhanced datasets for various tasks in machine learning for music. Access to such extensive datasets is crucial, not only in the visual domain as demonstrated by Laion-5B, but also in the domain of music.
In summary, our ethical framework emphasizes the importance of respecting artists' concerns, providing options for data exclusion, promoting transparency in dataset creation, and facilitating meaningful exploration of ML-assisted music creation while prioritizing safety considerations.
7Contact via disco.dataset@ethz.ch email
J Datasheet for Datasets
- We want to provide an open-source large-scale music dataset for the research community. Such large datasets do not yet exist in this domain, and we believe they are needed to democratize innovation in music research and ML-assisted music creation. Working with large data also has inherent risks, which are better analyzed openly by a large community rather than by private institutions behind closed doors.
- The authors created the dataset as part of their ongoing research at ETH Zurich.
3. Who funded the creation of the dataset? If there is an associated grant, please provide the name of the grantor and the grant name and number.
- We share 15,296,232 YouTube links to music with metadata and associated Spotify music metadata. In addition, we provided similarity measures between the YouTube video title, the YouTube description, the Song title, and the name of the artist. Additionally, contribution includes providing audio embeddings for the YouTube video and the Spotify song preview computed with Laion-CLAP [35]. The metadata includes an explicit flag to allow users to filter for explicit or non-explicit music.
15,296,232
- No, DISCO-10M does not cover all artists on Spotify and only a selection of popular songs of those that we do consider. The YouTube search results only contain 20 matches we take into consideration. To improve the dataset quality, we filter out matches that do not meet the threshold described in Section 3.2.
- URLs to YouTube videos and Spotify song previews as well as song-specific metadata, such as artist names, artist/song IDs, YouTube video title/description snippet, video views, duration, Spotify song duration, and creation date
6. Is any information missing from individual instances? If so, please provide a description explaining why this information is missing (e.g. because it was unavailable). This does not include intentionally removed information but might include, e.g., redacted text.
- The artist names are not always known since we do not have this information for every artist ID in our dataset. This is the case in  $3.46\%$  of all datapoints. In addition, we have  $7.81\%$  missing YouTube description snippets and  $0.183\%$  missing YouTube view counts.
7. Are relationships between individual instances made explicit (e.g., users' movie ratings, social network links)? If so, please describe how these relationships are made explicit.
8. Are there recommended data splits (e.g., training, development/Validation, testing)? If so, please provide a description of these splits, explaining the rationale behind them.
- We acknowledge the existence of duplicate songs stemming from different YouTube videos corresponding to the same Spotify song. These duplicates can be removed by filtering with stricter thresholds (cf. Section 3.2).
10. Is the dataset self-contained, or does it link to or otherwise rely on external resources (e.g., websites, tweets, other datasets)? If it links to or relies on external resources, a) are there guarantees that they will exist and remain constant over time; b) are there official archival versions of the complete dataset (i.e., including the external resources as they existed at the time the dataset was created); c) are there any restrictions (e.g., licenses, fees) associated with any of the external resources that might apply to a dataset consumer? Please provide descriptions of all external resources and any restrictions associated with them, as well as links or other access points, as appropriate.
DISCO-10M relies on the availability of the songs on YouTube and Spotify since we only link to those resources. The embeddings and other metadata are self-contained.
11. Does the dataset contain data that might be considered confidential (e.g., data that is protected by legal privilege or by doctor- patient confidentiality, data that includes the content of individuals' non-public communications)? If so, please provide a description.
- No, there are no confidential datapoints in DISCO-10M.
- DISCO-10M contains music with an explicit flag. We do not know in what ways the song is explicit (sexual, abusive, or others), but the flag allows users to filter for such songs easily. Additionally, DISCO-10M does not contain any links to age-restricted YouTube videos.
13. Does the dataset identify any subpopulations (e.g., by age, gender)? If so, please describe how these subpopulations are identified and provide a description of their respective distributions within the dataset.
14. Is it possible to identify individuals (i.e., one or more natural persons), either directly or indirectly (i.e., in combination with other data) from the dataset? If so, please describe how.
- Yes, the Spotify artist ID is directly related to one or multiple natural persons. Additionally, the YouTube video URLs we provide in the dataset are uploaded by one or multiple natural persons.
15. Does the dataset contain data that might be considered sensitive in any way (e.g., data that reveals race or ethnic origins, sexual orientations, religious beliefs, political opinions or union member- ships, or locations; financial or health data; biometric or genetic data; forms of government identification, such as social security numbers; criminal history)? If so, please provide a description.
16. Any other comments?
- We emphasize the focus of our dataset on music and not on individuals. Additionally, we reiterate that this dataset is intended for research purposes only, as described in Section 5.
1. How was the data associated with each instance acquired? Was the data directly observable (e.g., raw text, movie ratings), reported by subjects (e.g., survey responses), or indirectly inferred/derived from other data (e.g., part-of-speech tags, model-based guesses for age or language)? If the data was reported by subjects or indirectly inferred/derived from other data, was the data validated/verified? If so, please describe how.
- The YouTube videos and metadata, and the Spotify tracks and metadata are observable and were collected by accessing the Spotify API as well as the YouTube API. The similarity scores and audio embeddings are computed by us.
2. What mechanisms or procedures were used to collect the data (e.g., hardware apparatuses or sensors, manual human curation, software programs, software APIs)? How were these mechanisms or procedures validated?
- The Spotify API and the YouTube API. Our results were validated manually by assessing the quality of the retrieved information on random samples.
- We started the Spotify artist scraping from the artist seed described in Appendix A. Additionally, we filter high-quality datapoints as described in Section 3.2.
4. Who was involved in the data collection process (e.g., students, crowdworkers, contractors), and how were they compensated (e.g., how much were crowdworkers paid)?
- Only the authors of this paper were involved in the data collection process. Author involvement and payment disclosed after acceptance.
5. Over what timeframe was the data collected? Does this timeframe match the creation timeframe of the data associated with the instances (e.g., recent crawl of old news articles)? If not, please describe the time-frame in which the data associated with the instances was created.
- January 2023 to June 2023 was the timeframe of data collection. The creation time of the songs is diverse and can be seen in Figure 4.
6. Were any ethical review processes conducted (e.g., by an institutional review board)? If so, please provide a description of these review processes, including the outcomes, as well as a link or other access point to any supporting documentation.