Package: fadtk
Frechet Audio Distance ToolkitA simple and standardized library for Frechet Audio Distance (FAD) calculation. This library is published along with the paperImproving Frechet Audio Distance for Generative Music Evaluation(link coming soon). The datasets associated with this paper and sample code tools used in the paper are also available under this repository.You can listen to audio samples of per-song FAD outliers on the online demo here:https://fadtk.hydev.org/0x00. FeaturesEasily and efficiently compute audio embeddings with various models.Compute FAD∞ scores between two datasets for evaluation.Use pre-computed statistics ("weights") to compute FAD∞ scores from existing baselines.Compute per-song FAD to find outliers in the datasetSupported ModelsModelName in FADtkDescriptionCreatorCLAPclap-2023Learning audio concepts from natural language supervisionMicrosoftCLAPclap-laion-{audio/music}Contrastive Language-Audio PretrainingLAIONEncodecencodec-embState-of-the-art deep learning based audio codecFacebook/Meta ResearchMERTMERT-v1-95M-{layer}Acoustic Music Understanding Model with Large-Scale Self-supervised Trainingm-a-pVGGishvggishAudio feature classification embeddingGoogleDACdac-44kHzHigh-Fidelity Audio Compression with Improved RVQGANDescriptCDPAMcdpam-{acoustic/content}Contrastive learning-based Deep Perceptual Audio MetricPranay Manocha et al.0x01. InstallationTo use the FAD toolkit, you must first install it. This library is created and tested on Python 3.11 on Linux but should work on Python >3.9 and on Windows and macOS as well.Install torchhttps://pytorch.org/pip install fadtkTo ensure that the environment is setup correctly and everything work as intended, it is recommended to run our tests using the commandpython -m fadtk.testafter installing.Optional DependenciesOptionally, you can install dependencies that add additional embedding support. They are:CDPAM:pip install cdpamDAC:pip install descript-audio-codec==1.0.00x02. Command Line Usage# Evaluationfadtk<model_name><baseline><evaluation-set>[--inf/--indiv]# Compute embeddingsfadtk.embeds-m<models...>-d<datasets...>Example 1: Computing FAD_inf scores on FMA_Pop baseline# Compute FAD-inf between the baseline and evaluation datasets on two different modelsfadtkclap-laion-audiofma_pop/path/to/evaluation/audio--inf
fadtkencodec-embfma_pop/path/to/evaluation/audio--infExample 2: Compute individual FAD scores for each songfadtkencodec-embfma_pop/path/to/evaluation/audio--indivscores.csvExample 3: Compute FAD scores with your own baselineFirst, create two directories, one for the baseline and one for the evaluation, and placeonlythe audio files in them. Then, run the following commands:# Compute FAD between the baseline and evaluation datasetsfadtkclap-laion-audio/path/to/baseline/audio/path/to/evaluation/audioExample 4: Just compute embeddingsIf you only want to compute embeddings with a list of specific models for a list of dataset, you can do that using the command line.fadtk.embeds-mModel1Model2-d/dataset1/dataset20x03. Best PracticesWhen using the FAD toolkit to compute FAD scores, it's essential to consider the following best practices to ensure accuracy and relevancy in the reported findings.Choose a Meaningful Reference Set: Do not default to commonly used reference sets like Musiccaps without consideration. A reference set that aligns with the specific goal of the research should be chosen. For generative music, we recommend using the FMA-Pop subset as proposed in our paper.Select an Appropriate Embedding: The choice of embedding can heavily influence the scoring. For instance, VGGish is optimized for classification, and it might not be the most suitable if the research objective is to measure aspects like quality.Provide Comprehensive Reporting: Ensure that all test statistics are included in the report:The chosen reference set.The selected embedding.The number of samples and their duration in both the reference and test set.This level of transparency ensures that the FAD scores' context and potential variability are understood by readers or users.Benchmark Against the State-of-the-Art: When making comparisons, researchers should ideally use the same setup to assess the state-of-the-art models for comparison. Without a consistent setup, the FAD comparison might lose its significance.Interpret FAD Scores Contextually: Per-sample FAD scores should be calculated. Listening to the per-sample outliers will provide a hands-on understanding of what the current setup is capturing, and what "low" and "high" FAD scores signify in the context of the study.By adhering to these best practices, the use of our FAD toolkit can be ensured to be both methodologically sound and contextually relevant.0x04. Programmatic UsageDoing the above in pythonIf you want to know how to do the above command-line processes in python, you can check out how our launchers are implemented (__main__.pyandembeds.py)Adding New EmbeddingsTo add a new embedding, the only file you would need to modify ismodel_loader.py. You must create a new class that inherits the ModelLoader class. You need to implement the constructor, theload_modeland the_get_embeddingfunction. You can start with the below template:classYourModel(ModelLoader):"""Add a short description of your model here."""def__init__(self):# Define your sample rate and number of features here. Audio will automatically be resampled to this sample rate.super().__init__("Model name including variant",num_features=128,sr=16000)# Add any other variables you need heredefload_model(self):# Load your model herepassdef_get_embedding(self,audio:np.ndarray)->np.ndarray:# Calculate the embeddings using your modelreturnnp.zeros((1,self.num_features))defload_wav(self,wav_file:Path):# Optionally, you can override this method to load wav file in a different way. The input wav_file is already in the correct sample rate specified in the constructor.returnsuper().load_wav(wav_file)0x05. Published Data and CodeWe also include some sample code and data from the paper in this repo.Refined Datasetsmusiccaps-public-openai.csv: This file contains the original MusicCaps song IDs and captions along with GPT4 labels for their quality and the GPT4-refined prompts used for music generation.fma_pop_tracks.csv: This file contains the subset of 4839 song IDs and metadata information for the FMA-Pop subset we proposed in our paper. After downloading the Free Music Archive dataset from theoriginal source, you can easily locate the audio files for this FMA-Pop subset using song IDs.Sample CodeThe method we used to create GPT4 one-shot prompts for generating the refined MusicCaps prompts and for classifying quality from the MusicCaps captions can be found inexample/prompts.0x06. Acknowledgments and LicensesThe code in this toolkit is licensed under theMIT License. Please cite our work if this repository helped you in your project.@inproceedings{fadtk,
  title ={Adapting Frechet Audio Distance for Generative Music Evaluation},
  author ={Azalea Gui, Hannes Gamper, Sebastian Braun, Dimitra Emmanouilidou},
  booktitle ={Submitted to IEEE ICASSP 2024},
  year ={2024},
  url ={https://arxiv.org/abs/2311.01616},}Please also cite the FMA (Free Music Archive) dataset if you used FMA-Pop as your FAD scoring baseline.@inproceedings{fma_dataset,
  title ={{FMA}: A Dataset for Music Analysis},
  author ={Defferrard, Micha\"el and Benzi, Kirell and Vandergheynst, Pierre and Bresson, Xavier},
  booktitle ={18th International Society for Music Information Retrieval Conference (ISMIR)},
  year ={2017},
  archiveprefix ={arXiv},
  eprint ={1612.01840},
  url ={https://arxiv.org/abs/1612.01840},}Special ThanksImmense gratitude to the foundational repositorygudgud96/frechet-audio-distance- "A lightweight library for Frechet Audio Distance calculation". Much of our project has been adapted and enhanced from gudgud96's contributions. In honor of this work, we've retained theoriginal MIT license.Encodec from Facebook:facebookresearch/encodecCLAP:microsoft/CLAPCLAP from LAION:LAION-AI/CLAPMERT from M-A-P:m-a-p/MERTVGGish in PyTorch:harritaylor/torchvggishFree Music Archive:mdeff/fmaFrechet Inception Distance implementation:mseitzer/pytorch-fidFrechet Audio Distance paper:arxiv/1812.08466