Package: audio2dataset
audio2datasetEasily turn large sets of audio urls to an audio dataset.Installpip install audio2datasetExamplesExample of datasets to download with example commands are available in thedataset_examplesfolder. In particular:freesound500k audio/text pairsUsageFirst get some audio url list. For example:echo 'https://freesound.org/apiv2/audios/632625/download/' >> myaudiolist.txt
echo 'https://freesound.org/apiv2/audios/632619/download/' >> myaudiolist.txtThen, run the tool:audio2dataset --url_list=myaudiolist.txt --output_folder=output_folder --thread_count=64 --audio_size=256The tool will then automatically download the urls, subsample them, and store them with that format:output_folder00000.tar containing:000000000.flat000000001.flat000000002.flatwith each number being the position in the list. The subfolders avoids having too many files in a single folder.Ifcaptionsare provided, they will be saved as 0.txt, 1.txt, ...This can then easily be fed into machine learning training or any other use case.Also a .parquet file will be saved with the same name as the subfolder/tar files containing these same metadata.
It can be used to analyze the results efficiently..json files will also be saved with the same name suffixed by _stats, they contain stats collected during downloading (download time, number of success, ...)Python examplesCheckout these examples to call this as a lib:simple_example.pyAPIThis module exposes a single functiondownloadwhich takes the same arguments as the command line tool:url_listA file with the list of url of audios to download. It can be a folder of such files. (required)audio_sizeThe size to subsample audio to (default256)output_folderThe path to the output folder. If existing subfolder are present, the tool will continue to the next number. (default"audios")processes_countThe number of processes used for downloading the pictures. This is important to be high for performance. (default1)thread_countThe number of threads used for downloading the pictures. This is important to be high for performance. (default256)output_formatdecides how to save pictures (defaultfiles)filessaves as a set of subfolder containing pictureswebdatasetsaves as tars containing picturesparquetsaves as parquet containing pictures as bytestfrecordsaves as tfrecord containing pictures as bytesdummydoes not save. Useful for benchmarksinput_formatdecides how to load the urls (defaulttxt)txtloads the urls as a text file of url, one per linecsvloads the urls and optional caption as a csvtsvloads the urls and optional caption as a tsvtsv.gzloads the urls and optional caption as a compressed (gzip) tsv.gzjsonloads the urls and optional caption as a jsonparquetloads the urls and optional caption as a parqueturl_colthe name of the url column for parquet and csv (defaulturl)caption_colthe name of the caption column for parquet and csv (defaultNone)number_sample_per_shardthe number of sample that will be downloaded in one shard (default10000)save_additional_columnslist of additional columns to take from the csv/parquet files and save in metadata files (defaultNone)timeoutmaximum time (in seconds) to wait when trying to download an audio (default10)enable_wandbwhether to enable wandb logging (defaultFalse)wandb_projectname of W&B project used (defaultaudio2dataset)oom_shard_countthe order of magnitude of the number of shards, used only to decide what zero padding to use to name the shard files (default5)compute_md5compute md5 of raw audios and store it in metadata (defaultTrue)distributorchoose how to distribute the downloading (defaultmultiprocessing)multiprocessinguse a multiprocessing pool to spawn processespysparkuse a pyspark session to create workers on a spark cluster (see details below)subjob_sizethe number of shards to download in each subjob supporting it, a subjob can be a pyspark job for example (default1000)retriesnumber of time a download should be retried (default0)disable_all_reencodingif set to True, this will keep the audio files in their original state with no resizing and no conversion, will not even check if the audio is valid. Useful for benchmarks. To use only if you plan to post process the audios by another program and you have plenty of storage available. (defaultFalse)Output format choiceaudio2dataset support several formats. There are trade off for which to choose:files: this is the simplest one, audios are simply saved as files. It's good for up to 1M samples on a local file system. Beyond that performance issues appear very fast. Handling more than a million files in standard filesystem does not work well.webdataset: webdataset format saves samples in tar files, thanks towebdatasetlibrary, this makes it possible to load the resulting dataset fast in both pytorch, tensorflow and jax. Choose this for most use cases. It works well for any filesystemparquet: parquet is a columnar format that allows fast filtering. It's particularly easy to read it using pyarrow and pyspark. Choose this if the rest of your data ecosystem is based on pyspark.petastormcan be used to read the data but it's not as easy to use as webdatasettfrecord: tfrecord is a protobuf based format. It's particularly easy to use from tensorflow and usingtf data. Use this if you plan to use the dataset only in the tensorflow ecosystem. The tensorflow writer does not use fsspec and as a consequence supports only a limited amount of filesystem, including local, hdfs, s3 and gcs. It is also less efficient than the webdataset writer when writing to other filesystems than local, losing some 30% performance.File system supportThanks tofsspec, audio2dataset supports reading and writing files inmany file systems.
To use it, simply use the prefix of your filesystem before the path. For examplehdfs://,s3://,http://, orgcs://.
Some of these file systems require installing an additional package (for example s3fs for s3, gcsfs for gcs).
See fsspec doc for all the details.If you need specific configuration for your filesystem, you may handle this problem by using thefsspec configuration systemthat makes it possible to create a file such as.config/fsspec/s3.jsonand have information in it such as:{
  "s3": {
    "client_kwargs": {
            "endpoint_url": "https://some_endpoint",
            "aws_access_key_id": "your_user",
           "aws_secret_access_key": "your_password"
    }
  }
}Which may be necessary if using s3 compatible file systems such asminio. That kind of configuration also work for all other fsspec-supported file systems.Distribution modesaudio2dataset supports several distributors.multiprocessing which spawns a process pool and use these local processes for downloadingpyspark which spawns workers in a spark pool to do the downloadingmultiprocessing is a good option for downloading on one machine, and as such it is the default.
Pyspark lets audio2dataset use many nodes, which makes it as fast as the number of machines.
It can be particularly useful if downloading datasets with more than a billion audio.pyspark configurationIn order to use audio2dataset with pyspark, you will need to do this:pip install pysparkuse the--distributor pysparkoptiontweak the--subjob_size 1000option: this is the number of audios to download in each subjob. Increasing it will mean a longer time of preparation to put the feather files in the temporary dir, a shorter time will mean sending less shards at a time to the pyspark job.By default a local spark session will be created.
You may want to create a custom spark session depending on your specific spark cluster.
To do that checkpyspark_example.py, there you can plug your custom code to create a spark session, then
run audio2dataset which will use it for downloading.To create a spark cluster check thedistributed audio2dataset tutorialIntegration with Weights & BiasesTo enable wandb, use the--enable_wandb=Trueoption.Performance metrics are monitored throughWeights & Biases.In addition, most frequent errors are logged for easier debugging.Other features are available:logging of environment configuration (OS, python version, CPU count, Hostname, etc)monitoring of hardware resources (GPU/CPU, RAM, Disk, Networking, etc)custom graphs and reportscomparison of runs (convenient when optimizing parameters such as number of threads/cpus)When running the script for the first time, you can decide to either associate your metrics to your account or log them anonymously.You can also log in (or create an account) before by runningwandb login.For developmentEither locally, or ingitpod(doexport PIP_USER=falsethere)Setup a virtualenv:python3 -m venv .env
source .env/bin/activate
pip install -e .to run tests:pip install -r requirements-test.txtthenmake lint
make testYou can usemake blackto reformat the codepython -m pytest -x -s -v tests -k "dummy"to run a specific testBenchmarks10000 audio benchmarkcd tests/test_files
bash benchmark.sh