DOI: 10.5281/zenodo.2552859
title: FSDKaggle2018

description: FSDKaggle2018 is an audio dataset containing 11,073 audio files annotated with 41 labels of the AudioSet Ontology. FSDKaggle2018 has been used for the DCASE Challenge 2018 Task 2, which was run as a Kaggle competition titled Freesound General-Purpose Audio Tagging Challenge.

<strong>Citation</strong>

If you use the FSDKaggle2018 dataset or part of it, please cite our <strong>DCASE 2018 paper</strong>:


Eduardo Fonseca, Manoj Plakal, Frederic Font, Daniel P. W. Ellis, Xavier Favory, Jordi Pons, Xavier Serra. "General-purpose Tagging of Freesound Audio with AudioSet Labels: Task Description, Dataset, and Baseline". <em>Proceedings of the DCASE 2018 Workshop</em> (2018)


You can also consider citing our <strong>ISMIR 2017 paper</strong>, which describes how we gathered the manual annotations included in FSDKaggle2018.


Eduardo Fonseca, Jordi Pons, Xavier Favory, Frederic Font, Dmitry Bogdanov, Andres Ferraro, Sergio Oramas, Alastair Porter, and Xavier Serra, "Freesound Datasets: A Platform for the Creation of Open Audio Datasets", In <em>Proceedings of the 18th International Society for Music Information Retrieval Conference</em>, Suzhou, China, 2017


<strong>Contact</strong>

You are welcome to contact Eduardo Fonseca should you have any questions at eduardo.fonseca@upf.edu.

<strong>About this dataset</strong>

Freesound Dataset Kaggle 2018 (or <strong>FSDKaggle2018</strong> for short) is an audio dataset containing 11,073 audio files annotated with 41 labels of the AudioSet Ontology [1]. FSDKaggle2018 has been used for the Task 2 of the <em>Detection and Classification of Acoustic Scenes and Events</em> (DCASE) Challenge 2018. Please visit the DCASE2018 Challenge Task 2 website for more information. This Task was hosted on the Kaggle platform as a competition titled Freesound General-Purpose Audio Tagging Challenge. It was organized by researchers from the Music Technology Group of Universitat Pompeu Fabra, and from Google Research’s Machine Perception Team.

The goal of this competition was to build an audio tagging system that can categorize an audio clip as belonging to one of a set of 41 diverse categories drawn from the AudioSet Ontology.

All audio samples in this dataset are gathered from Freesound [2] and are provided here as uncompressed PCM 16 bit, 44.1 kHz, mono audio files. Note that because Freesound content is collaboratively contributed, recording quality and techniques can vary widely.

The ground truth data provided in this dataset has been obtained after a data labeling process which is described below in the <em>Data labeling process</em> section. FSDKaggle2018 clips are unequally distributed in the following <strong>41 categories</strong> of the AudioSet Ontology:

"Acoustic_guitar", "Applause", "Bark", "Bass_drum", "Burping_or_eructation", "Bus", "Cello", "Chime", "Clarinet", "Computer_keyboard", "Cough", "Cowbell", "Double_bass", "Drawer_open_or_close", "Electric_piano", "Fart", "Finger_snapping", "Fireworks", "Flute", "Glockenspiel", "Gong", "Gunshot_or_gunfire", "Harmonica", "Hi-hat", "Keys_jangling", "Knock", "Laughter", "Meow", "Microwave_oven", "Oboe", "Saxophone", "Scissors", "Shatter", "Snare_drum", "Squeak", "Tambourine", "Tearing", "Telephone", "Trumpet", "Violin_or_fiddle", "Writing".

Some other relevant characteristics of FSDKaggle2018:


	
	The dataset is split into a train set and a test set.
	
	
	The <strong>train set</strong> is meant to be for system development and includes <strong>~9.5k samples unequally distributed among 41 categories</strong>. The minimum number of audio samples per category in the train set is 94, and the maximum 300. The duration of the audio samples ranges from 300ms to 30s due to the diversity of the sound categories and the preferences of Freesound users when recording sounds. The total duration of the train set is roughly 18h.
	
	
	Out of the ~9.5k samples from the train set, <strong>~3.7k have manually-verified ground truth annotations</strong> and <strong>~5.8k have non-verified annotations</strong>. The non-verified annotations of the train set have a quality estimate of <strong>at least</strong> 65-70% in each category. Checkout the <em>Data labeling process</em> section below for more information about this aspect.
	
	
	Non-verified annotations in the train set are properly flagged in <code>train.csv</code> so that participants can opt to use this information during the development of their systems.
	
	
	The <strong>test set</strong> is composed of <strong>1.6k samples with manually-verified annotations</strong> and with a similar category distribution than that of the train set. The total duration of the test set is roughly 2h.
	
	
	All audio samples in this dataset have a <strong>single label</strong> (i.e. are only annotated with one label). Checkout the <em>Data labeling process </em>section below for more information about this aspect. A single label should be predicted for each file in the test set.
	


<strong>Data labeling process</strong>

The data labeling process started from a manual mapping between Freesound tags and AudioSet Ontology categories (or <em>labels</em>), which was carried out by researchers at the Music Technology Group, Universitat Pompeu Fabra, Barcelona. Using this mapping, a number of Freesound audio samples were <strong>automatically annotated</strong> with labels from the AudioSet Ontology. These annotations can be understood as weak labels since they express the presence of a sound category in an audio sample. 

Then, a <strong>data validation process</strong> was carried out in which a number of participants did listen to the annotated sounds and manually assessed the presence/absence of an automatically assigned sound category, according to the AudioSet category description.

Audio samples in FSDKaggle2018 are only annotated with a single ground truth label (see <code>train.csv</code>). A total of <strong>3,710 annotations </strong>included in the train set of FSDKaggle2018 are annotations that have been <strong>manually validated</strong> as present and predominant (some with inter-annotator agreement but not all of them). This means that in most cases there is no additional acoustic material other than the labeled category. In few cases there may be some additional sound events, but these additional events won't belong to any of the 41 categories of FSDKaggle2018.

The rest of the annotations have <strong>not</strong> been manually validated and therefore some of them could be inaccurate. Nonetheless, we have <strong>estimated</strong> that <strong>at least</strong> 65-70% of the non-verified annotations per category <strong>in the train set</strong> are indeed correct. It can happen that some of these non-verified audio samples present several sound sources even though only one label is provided as ground truth. These additional sources are typically out of the set of the 41 categories, but in a few cases they could be within.

More details about the data labeling process can be found in [3].

<strong>License</strong>

FSDKaggle2018 has licenses at two different levels, as explained next.

All sounds in Freesound are released under Creative Commons (CC) licenses, and each audio clip has its own license as defined by the audio clip uploader in Freesound. For attribution purposes and to facilitate attribution of these files to third parties, we include a relation of the audio clips included in FSDKaggle2018 and their corresponding license. The licenses are specified in the files <code>train_post_competition.csv</code> and <code>test_post_competition_scoring_clips.csv</code>.

In addition, FSDKaggle2018 as a whole is the result of a curation process and it has an additional license. FSDKaggle2018 is released under CC-BY. This license is specified in the <code>LICENSE-DATASET</code> file downloaded with the <code>FSDKaggle2018.doc</code> zip file.

<strong>Files</strong>

FSDKaggle2018 can be downloaded as a series of zip files with the following directory structure:

<pre>root
│  
└───FSDKaggle2018.audio_train/                       Audio clips in the train set
│   
└───FSDKaggle2018.audio_test/                        Audio clips in the test set
│   
└───FSDKaggle2018.meta/                              Files for evaluation setup
│   │            
│   └───train_post_competition.csv                   Data split and ground truth for the train set
│   │            
│   └───test_post_competition_scoring_clips.csv      Ground truth for the test set         
│   
└───FSDKaggle2018.doc/
    │            
    └───README.md                                    The dataset description file you are reading
    │            
    └───LICENSE-DATASET                              License of FSDKaggle2018 dataset as a whole   
</pre>

<strong>NOTE</strong>: the original <code>train.csv</code> file provided during the competition has been updated with more metadata (licenses, Freesound ids, etc.) into <code>train_post_competition.csv</code>. Likewise, the original <code>test.csv</code> that was not public during the competition is now available with ground truth and metadata as <code>test_post_competition_scoring_clips.csv</code>. The file name <code>test_post_competition_scoring_clips.csv</code> refers to the fact that only the 1600 clips used for systems' ranking are included. During the competition, an additional subset of <em>padding</em> clips was added in order to prevent undesired practices. This <em>padding</em> subset (that was never used for systems' ranking) is no longer included in the dataset (see our DCASE 2018 paper for more details.)

Each row (i.e. audio clip) of the <code>train_post_competition.csv</code> file contains the following information:


	<code>fname</code>: the file name
	<code>label</code>: the audio classification label (ground truth)
	<code>manually_verified</code>: Boolean (1 or 0) flag to indicate whether or not that annotation has been manually verified; see description above for more info
	<code>freesound_id</code>: the Freesound id for the audio clip
	<code>license</code>: the license for the audio clip


Each row (i.e. audio clip) of the <code>test_post_competition_scoring_clips.csv</code> file contains the following information:


	<code>fname</code>: the file name
	<code>label</code>: the audio classification label (ground truth)
	<code>usage</code>: string that indicates to which Kaggle leaderboard the clip was associated during the competition: <code>Public</code> or <code>Private</code>
	<code>freesound_id</code>: the Freesound id for the audio clip
	<code>license</code>: the license for the audio clip


<strong>Baseline System</strong>

A CNN baseline system for FSDKaggle2018 is available at https://github.com/DCASE-REPO/dcase2018_baseline/tree/master/task2.

<strong>References and links</strong>

[1] Jort F Gemmeke, Daniel PW Ellis, Dylan Freedman, Aren Jansen, Wade Lawrence, R Channing Moore, Manoj Plakal, and Marvin Ritter. "Audio set: An ontology and human-labeled dartaset for audio events." Proceedings of the Acoustics, Speech and Signal Processing International Conference, 2017.

[2] Frederic Font, Gerard Roma, and Xavier Serra. "Freesound technical demo." Proceedings of the 21st ACM international conference on Multimedia, 2013. https://freesound.org

[3] Eduardo Fonseca, Jordi Pons, Xavier Favory, Frederic Font, Dmitry Bogdanov, Andres Ferraro, Sergio Oramas, Alastair Porter, and Xavier Serra. "Freesound Datasets: A Platform for the Creation of Open Audio Datasets." Proceedings of the International Conference on Music Information Retrieval, 2017. PDF here

Freesound Annotator: https://annotator.freesound.org/<br>
Freesound: https://freesound.org<br>
Eduardo Fonseca's personal website: http://www.eduardofonseca.net/<br>
More datasets collected by us: http://www.eduardofonseca.net/datasets/

<strong>Acknowledgments</strong>

This work is partially supported by the European Union’s Horizon 2020 research and innovation programme under grant agreement No 688382 AudioCommons. Eduardo Fonseca is also sponsored by a Google Faculty Research Award 2017. We thank everyone who contributed to FSDKaggle2018 with annotations.