Package: aac-datasets
Audio Captioning datasets for PyTorchAudio Captioning unofficial datasets source code forAudioCaps[1],Clotho[2],MACS[3], andWavCaps[4], designed for PyTorch.Installationpipinstallaac-datasetsIf you want to check if the package has been installed and the version, you can use this command:aac-datasets-infoExamplesCreate Clotho datasetfromaac_datasetsimportClothodataset=Clotho(root=".",download=True)item=dataset[0]audio,captions=item["audio"],item["captions"]# audio: Tensor of shape (n_channels=1, audio_max_size)# captions: list of strBuild PyTorch dataloader with Clothofromtorch.utils.data.dataloaderimportDataLoaderfromaac_datasetsimportClothofromaac_datasets.utilsimportBasicCollatedataset=Clotho(root=".",download=True)dataloader=DataLoader(dataset,batch_size=4,collate_fn=BasicCollate())forbatchindataloader:# batch["audio"]: list of 4 tensors of shape (n_channels, audio_size)# batch["captions"]: list of 4 lists of str...Download datasetsTo download a dataset, you can usedownloadargument in dataset construction :dataset=Clotho(root=".",subset="dev",download=True)However, if you want to download datasets from a script, you can also use the following command :aac-datasets-download--root"."clotho--subsets"dev"Datasets informationHere is the statistics for each dataset :DatasetSamplingrate (kHz)Estimatedsize (GB)SourceSubsetsAudioCaps3243AudioSettrainvaltesttrain_v2Clotho44.153Freesounddevvalevaldcase_aac_testdcase_aac_analysisdcase_t2a_audiodcase_t2a_captionsMACS4813TAU Urban Acoustic Scenes 2019fullWavCaps32941AudioSetBBC Sound EffectsFreeSoundSoundBibleasas_noacbbcfsdfsd_noclsbFor Clotho, thedevsubset should be used for training, val for validation and eval for testing.Here is additional statistics on the train subset for AudioCaps, Clotho and MACS:AudioCaps/trainClotho/devMACS/fullWavCaps/fullNb audios49,8383,8403,930403,050Total audio duration (h)136.6124.010.97563.3Audio duration range (s)0.5-1015-30101-67,109Nb captions per audio152-51Nb captions49,83819,19517,275403,050Total nb words2402,482217,362160,0063,161,823Sentence size22-528-205-402-38Vocabulary2472443692721246001This duration is estimated on the total duration of 46230/49838 files of 126.7h.2The sentences are cleaned (lowercase+remove punctuation) and tokenized using the spacy tokenizer to count the words.RequirementsThis package has been developped for Ubuntu 20.04, and it is expected to work on most Linux-based distributions.Python packagesPython requirements are automatically installed when using pip on this repository.torch >= 1.10.1
torchaudio >= 0.10.1
py7zr >= 0.17.2
pyyaml >= 6.0
tqdm >= 4.64.0
huggingface-hub >= 0.15.1
numpy >= 1.21.2External requirements (AudioCaps only)The external requirements needed to downloadAudioCapsareffmpegandyt-dlp.ffmpegcan be install on Ubuntu usingsudo apt install ffmpegandyt-dlpfrom theofficial repo.You can also override their paths for AudioCaps:fromaac_datasetsimportAudioCapsdataset=AudioCaps(download=True,ffmpeg_path="/my/path/to/ffmpeg",ytdl_path="/my/path/to/ytdlp",)Additional informationCompatibility with audiocaps-downloadIf you want to useaudiocaps-download 1.0package to download AudioCaps, you will have to respect the AudioCaps folder tree:fromaudiocaps_downloadimportDownloaderroot="your/path/to/root"downloader=Downloader(root_path=f"{root}/AUDIOCAPS/audio_32000Hz/",n_jobs=16)downloader.download(format="wav")Then disable audio download and set the correct audio format before init AudioCaps :fromaac_datasetsimportAudioCapsdataset=AudioCaps(root=root,subset="train",download=True,audio_format="wav",download_audio=False,# this will only download labels and metadata files)ReferencesAudioCaps[1] C. D. Kim, B. Kim, H. Lee, and G. Kim, “Audiocaps: Generating captions for audios in the wild,” in NAACL-HLT, 2019. Available:https://aclanthology.org/N19-1011/Clotho[2] K. Drossos, S. Lipping, and T. Virtanen, “Clotho: An Audio Captioning Dataset,” arXiv:1910.09387 [cs, eess], Oct. 2019, Available:http://arxiv.org/abs/1910.09387MACS[3] F. Font, A. Mesaros, D. P. W. Ellis, E. Fonseca, M. Fuentes, and B. Elizalde, Proceedings of the 6th Workshop on Detection and Classication of Acoustic Scenes and Events (DCASE 2021). Barcelona, Spain: Music Technology Group - Universitat Pompeu Fabra, Nov. 2021. Available:https://doi.org/10.5281/zenodo.5770113WavCaps[4] X. Mei et al., “WavCaps: A ChatGPT-Assisted Weakly-Labelled Audio Captioning Dataset for Audio-Language Multimodal Research,” arXiv preprint arXiv:2303.17395, 2023, [Online]. Available:https://arxiv.org/pdf/2303.17395.pdfCite the aac-datasets packageIf you use this software, please consider cite it as "Labbe, E. (2013). aac-datasets: Audio Captioning datasets for PyTorch.", or use the following BibTeX citation:@software{
    Labbe_aac_datasets_2024,
    author = {Labbé, Etienne},
    license = {MIT},
    month = {01},
    title = {{aac-datasets}},
    url = {https://github.com/Labbeti/aac-datasets/},
    version = {0.5.0},
    year = {2024}
}ContactMaintainer:Etienne Labbé "Labbeti":labbeti.pub@gmail.com