DOI: 10.5281/zenodo.3612636
title: FSDKaggle2019

description: FSDKaggle2019 is an audio dataset containing 29,266 audio files annotated with 80 labels of the AudioSet Ontology. FSDKaggle2019 has been used for the DCASE Challenge 2019 Task 2, which was run as a Kaggle competition titled Freesound Audio Tagging 2019. <strong>Citation</strong> If you use the FSDKaggle2019 dataset or part of it, please cite our DCASE 2019 paper: Eduardo Fonseca, Manoj Plakal, Frederic Font, Daniel P. W. Ellis, Xavier Serra. "Audio tagging with noisy labels and minimal supervision". <em>Proceedings of the DCASE 2019 Workshop</em>, NYC, US (2019) You can also consider citing our <strong>ISMIR 2017 paper</strong>, which describes how we gathered the manual annotations included in FSDKaggle2019. Eduardo Fonseca, Jordi Pons, Xavier Favory, Frederic Font, Dmitry Bogdanov, Andres Ferraro, Sergio Oramas, Alastair Porter, and Xavier Serra, "Freesound Datasets: A Platform for the Creation of Open Audio Datasets", In <em>Proceedings of the 18th International Society for Music Information Retrieval Conference</em>, Suzhou, China, 2017 <strong>Data curators</strong> Eduardo Fonseca, Manoj Plakal, Xavier Favory, Jordi Pons <strong>Contact</strong> You are welcome to contact Eduardo Fonseca should you have any questions at eduardo.fonseca@upf.edu. <strong>ABOUT FSDKaggle2019</strong> Freesound Dataset Kaggle 2019 (or <em><strong>FSDKaggle2019</strong></em> for short) is an audio dataset containing 29,266 audio files annotated with 80 labels of the AudioSet Ontology [1]. FSDKaggle2019 has been used for the Task 2 of the <em>Detection and Classification of Acoustic Scenes and Events</em> (DCASE) Challenge 2019. Please visit the DCASE2019 Challenge Task 2 website for more information. This Task was hosted on the Kaggle platform as a competition titled Freesound Audio Tagging 2019. It was organized by researchers from the Music Technology Group (MTG) of Universitat Pompeu Fabra (UPF), and from Sound Understanding team at Google AI Perception. The competition intended to provide insight towards the development of broadly-applicable sound event classifiers able to cope with label noise and minimal supervision conditions. FSDKaggle2019 employs audio clips from the following sources: Freesound Dataset (FSD): a dataset being collected at the MTG-UPF based on Freesound content organized with the AudioSet Ontology The soundtracks of a pool of Flickr videos taken from the Yahoo Flickr Creative Commons 100M dataset (YFCC) The audio data is labeled using a vocabulary of 80 labels from Google’s AudioSet Ontology [1], covering diverse topics: Guitar and other Musical Instruments, Percussion, Water, Digestive, Respiratory sounds, Human voice, Human locomotion, Hands, Human group actions, Insect, Domestic animals, Glass, Liquid, Motor vehicle (road), Mechanisms, Doors, and a variety of Domestic sounds. The full list of categories can be inspected in <code>vocabulary.csv</code> (see Files & Download below). The goal of the task was to build a multi-label audio tagging system that can predict appropriate label(s) for each audio clip in a test set. What follows is a summary of some of the most relevant characteristics of FSDKaggle2019. Nevertheless, it is highly <strong>recommended</strong> to read our DCASE 2019 paper for a more in-depth description of the dataset and how it was built. <strong>Ground Truth Labels</strong> The ground truth labels are provided at the clip-level, and express the presence of a sound category in the audio clip, hence can be considered <em>weak</em> labels or tags. Audio clips have variable lengths (roughly from 0.3 to 30s). The audio content from FSD has been manually labeled by humans following a data labeling process using the Freesound Annotator platform. Most labels have inter-annotator agreement but not all of them. More details about the data labeling process and the Freesound Annotator can be found in [2]. The YFCC soundtracks were labeled using automated heuristics applied to the audio content and metadata of the original Flickr clips. Hence, a substantial amount of label noise can be expected. The label noise can vary widely in amount and type depending on the category, including in- and out-of-vocabulary noises. More information about some of the types of label noise that can be encountered is available in [3]. Specifically, FSDKaggle2019 features <strong>three types of label quality</strong>, one for each set in the dataset: <strong>curated train set</strong>: correct (but potentially incomplete) labels <strong>noisy train set</strong>: noisy labels <strong>test set</strong>: correct and complete labels Further details can be found below in the sections for each set. <strong>Format</strong> All audio clips are provided as uncompressed PCM 16 bit, 44.1 kHz, mono audio files. <strong>DATA SPLIT</strong> FSDKaggle2019 consists of <strong>two train sets</strong> and <strong>one test set</strong>. The idea is to limit the supervision provided for training (i.e., the manually-labeled, hence reliable, data), thus promoting approaches to deal with label noise. <strong>Curated train set</strong> The <strong>curated train set</strong> consists of manually-labeled data from FSD. Number of clips/class: 75 except in a few cases (where there are less) Total number of clips: 4970 Avg number of labels/clip: 1.2 Total duration: 10.5 hours The duration of the audio clips ranges from 0.3 to 30s due to the diversity of the sound categories and the preferences of Freesound users when recording/uploading sounds. Labels are correct but potentially incomplete. It can happen that a few of these audio clips present additional acoustic material beyond the provided ground truth label(s). <strong>Noisy train set</strong> The <strong>noisy train set</strong> is a larger set of noisy web audio data from Flickr videos taken from the YFCC dataset [5]. Number of clips/class: 300 Total number of clips: 19,815 Avg number of labels/clip: 1.2 Total duration: ~80 hours The duration of the audio clips ranges from 1s to 15s, with the vast majority lasting 15s. Labels are automatically generated and purposefully noisy. No human validation is involved. The label noise can vary widely in amount and type depending on the category, including in- and out-of-vocabulary noises. Considering the numbers above, the per-class data distribution available for training is, for most of the classes, 300 clips from the noisy train set and 75 clips from the curated train set. This means 80% noisy / 20% curated at the clip level, while at the duration level the proportion is more extreme considering the variable-length clips. <strong>Test set</strong> The <strong>test set</strong> is used for system evaluation and consists of manually-labeled data from FSD. Number of clips/class: between 50 and 150 Total number of clips: 4481 Avg number of labels/clip: 1.4 Total duration: 12.9 hours The acoustic material present in the test set clips is labeled exhaustively using the aforementioned vocabulary of 80 classes. Most labels have inter-annotator agreement but not all of them. Except human error, the label(s) are correct and complete considering the target vocabulary; nonetheless, a few clips could still present additional (unlabeled) acoustic content out of the vocabulary. During the DCASE2019 Challenge Task 2, the test set was split into two subsets, for the <strong>public</strong> and <strong>private</strong> leaderboards, and only the data corresponding to the <em>public</em> leaderboard was provided. <strong>In this current package you will find the full test set with all the test labels</strong>. To allow comparison with previous work, the file <code>test_post_competition.csv</code> includes a flag to determine the corresponding leaderboard (public or private) for each test clip (see more info in Files & Download below). <strong>Acoustic mismatch</strong> As mentioned before, FSDKaggle2019 uses audio clips from two sources: FSD: curated train set and test set, and YFCC: noisy train set. While the sources of audio (Freesound and Flickr) are collaboratively contributed and pretty diverse themselves, a certain acoustic mismatch can be expected between FSD and YFCC. We conjecture this mismatch comes from a variety of reasons.<br> For example, through acoustic inspection of a small sample of both data sources, we find a higher percentage of high quality recordings in FSD. In addition, audio clips in Freesound are typically recorded with the purpose of capturing audio, which is not necessarily the case in YFCC. This mismatch can have an impact in the evaluation, considering that most of the train data come from YFCC, while all test data are drawn from FSD. This constraint (i.e., noisy training data coming from a different web audio source than the test set) is sometimes a real-world condition. <strong>LICENSE</strong> All clips in FSDKaggle2019 are released under Creative Commons (CC) licenses. For attribution purposes and to facilitate attribution of these files to third parties, we include a mapping from the audio clips to their corresponding licenses. <strong>Curated train set and test set</strong>. All clips in <strong>Freesound</strong> are released under different modalities of Creative Commons (CC) licenses, and each audio clip has its own license as defined by the audio clip uploader in Freesound, some of them requiring attribution to their original authors and some forbidding further commercial reuse. The licenses are specified in the files <code>train_curated_post_competition.csv</code> and <code>test_post_competition.csv</code>. These licenses can be CC0, CC-BY, CC-BY-NC and CC Sampling+. <strong>Noisy train set</strong>. Similarly, the licenses of the soundtracks from <strong>Flickr</strong> used in FSDKaggle2019 are specified in the file <code>train_noisy_post_competition.csv</code>. These licenses can be CC-BY and CC BY-SA. In addition, FSDKaggle2019 as a whole is the result of a curation process and it has an additional license. FSDKaggle2019 is released under CC-BY. This license is specified in the <code>LICENSE-DATASET</code> file downloaded with the <code>FSDKaggle2019.doc </code>zip file. <strong>FILES & DOWNLOAD</strong> FSDKaggle2019 can be downloaded as a series of zip files with the following directory structure: <pre>root │ └───FSDKaggle2019.audio_train_curated/ Audio clips in the curated train set │ └───FSDKaggle2019.audio_train_noisy/ Audio clips in the noisy train set │ └───FSDKaggle2019.audio_test/ Audio clips in the full test set │ └───FSDKaggle2019.meta/ Files for evaluation setup │ │ │ └─── train_curated_post_competition.csv Ground truth for the curated train set │ │ │ └─── train_noisy_post_competition.csv Ground truth for the noisy train set │ │ │ └─── test_post_competition.csv Ground truth for the full test set │ │ │ └─── vocabulary.csv List of sound classes in FSDKaggle2019 │ └───FSDKaggle2019.doc/ │ └───README.md The dataset description file that you are reading │ └───LICENSE-DATASET License of the FSDKaggle2019 dataset as an entity </pre> <strong>Important Note:</strong> the original <code>train_curated.csv</code> and <code>train_noisy.csv</code> files provided during the competition have been updated with more metadata (licenses, Freesound/Flickr ids, etc.) into <code>train_curated_post_competition.csv</code> and <code>train_noisy_post_competition.csv</code>. Likewise, the original <code>test.csv</code> that was not public during the competition is now available with ground truth and metadata as <code>test_post_competition.csv</code>. Each row (i.e. audio clip) of the <code>train_curated_post_competition.csv</code> or <code>train_noisy_post_competition.csv </code>files contains the following information: <code>fname</code>: the file name, e.g., <code>0006ae4e.wav</code> <code>labels</code>: the audio classification label(s) (ground truth). Note that the number of labels per clip can be one, eg, <code>Bark</code> or more, eg, <code>"Walk_and_footsteps,Slam"</code>. <code>freesound_id</code> or <code>flickr_video_URL</code>: the Freesound id or Flickr id for the audio clip <code>license</code>: the license for the audio clip Each row (i.e. audio clip) of the <code>test_post_competition.csv</code> file contains the following information: <code>fname</code>: the file name <code>labels</code>: the audio classification label(s) (ground truth). Note that the number of labels per clip can be one, eg, <code>Bark</code> or more, eg, <code>"Walk_and_footsteps,Slam"</code>. <code>usage</code>: string that indicates to which Kaggle leaderboard the clip was associated during the competition: <code>Public</code> or <code>Private</code> <code>freesound_id</code>: the Freesound id for the audio clip <code>license</code>: the license for the audio clip <strong>Detected corrupted files in the curated train set</strong> The following 5 audio files in the <strong>curated train set</strong> have a wrong label, due to a bug in the file renaming process: <code>f76181c4.wav</code>, <code>77b925c2.wav</code>, <code>6a1f682a.wav</code>, <code>c7db12aa.wav</code>, <code>7752cc8a.wav</code>. The audio file <code>1d44b0bd.wav</code> in the <strong>curated train set</strong> was found to be corrupted (contains no signal) due to an error in format conversion. If you find more corrupted files in FSDKaggle2019, please send an email to eduardo.fonseca@upf.edu. <strong>Download</strong> Each of the folders in the directory structure above is compressed into one corresponding zip file that you can download and unzip with your favorite compression tool. There is one exception: due to the large size of <code>FSDKaggle2019.audio_train_noisy/</code>, it is split into 7 files (note the last file is not <code>*.z07</code>, but <code>*.zip</code>): <pre><code class="language-none">FSDKaggle2019.audio_train_noisy.z01 FSDKaggle2019.audio_train_noisy.z02 FSDKaggle2019.audio_train_noisy.z03 FSDKaggle2019.audio_train_noisy.z04 FSDKaggle2019.audio_train_noisy.z05 FSDKaggle2019.audio_train_noisy.z06 FSDKaggle2019.audio_train_noisy.zip</code></pre> In this case, you first have to download the 7 files. Once downloaded, we convert the split archive to a single-file archive. In other words, we merge the 7 files into one zip file called e.g. <code>unsplit.zip</code> in your local machine. <pre><code class="language-none">zip -s 0 FSDKaggle2019.audio_train_noisy.zip --out unsplit.zip</code></pre> Finally, this merged file is unzipped. <pre><code class="language-none">unzip unsplit.zip</code></pre> <strong>Baseline System</strong> A CNN baseline system for FSDKaggle2019 is available at https://github.com/DCASE-REPO/dcase2019<em>task2</em>baseline. <strong>REFERENCES AND LINKS</strong> [1] Jort F Gemmeke, Daniel PW Ellis, Dylan Freedman, Aren Jansen, Wade Lawrence, R Channing Moore, Manoj Plakal, and Marvin Ritter. "Audio set: An ontology and human-labeled dataset for audio events." In Proceedings of the International Conference on Acoustics, Speech and Signal Processing, 2017. [PDF] [2] Eduardo Fonseca, Jordi Pons, Xavier Favory, Frederic Font, Dmitry Bogdanov, Andres Ferraro, Sergio Oramas, Alastair Porter, and Xavier Serra. "Freesound Datasets: A Platform for the Creation of Open Audio Datasets." In Proceedings of the International Conference on Music Information Retrieval, 2017. [PDF] [3] Eduardo Fonseca, Manoj Plakal, Daniel P. W. Ellis, Frederic Font, Xavier Favory, and Xavier Serra. "Learning Sound Event Classifiers from Web Audio with Noisy Labels." In Proceedings of the International Conference on Acoustics, Speech and Signal Processing, 2019. [PDF] [4] Frederic Font, Gerard Roma, and Xavier Serra. "Freesound technical de