Package: dpam
Deep Perceptual Audio Metric (DPAM)[Paper][Webpage][Teaser][Full Video]A Differentiable Perceptual Audio Metric Learned from Just Noticeable DifferencesPranay Manocha,Adam Finkelstein,Richard Zhang,Nicholas J. Bryan,Gautham J. Mysore,Zeyu JinAccepted atInterspeech 2020This is a Tensorflow implementation (a pytorch implementation ishere) of our audio perceptual metric. It contains (0) minimal code to run our perceptual metric (DPAM), (1) code to train the perceptual metric on our JND dataset, and (2) an example of using our perceptual metric as a loss function for speech denoising.Also checkout our newest learned metric: CDPAM: Contrastive learning based deep perceptual audio similarity metricPlease seecd cdpamand follow the instructions in the readme. We also provide pretrained models for:We make use of CDPAM for training a waveform synthesis model. We use the codebase fromhere. We train both single and cross-speaker models.Real-time speech enhancement: We make use of CDPAM as a loss function for training an SE model. We use the codebase fromhere. We just augment CDPAM to their loss for training.Things to note:At the moment, this algorithm requires using 16-bit PCM audio files to perform correctly. You can use sox to convert your file.The current pretrained models support sr=22050Hz. Please make sure to resample your files first before using the metric.
For ease, you can load your audio clip using librosa.load(filename,sr=22050) and then rescale to [-32768 to 32768] using np.round(audio_file.astype(np.float)*32768). Seemetric_code/metric_use_simple.pyfor an example.Quick StartInstalling the metric (DPAM - perceptual audio similarity metric)pipinstalldpamUsing the metric is as simple as:importdpamloss_fn=dpam.DPAM()wav_ref=dpam.load_audio('sample_audio/ref.wav')wav_out=dpam.load_audio('sample_audio/2.wav')dist=loss_fn.forward(wav_ref,wav_out)(0) Setup and basic usageRequired python libraries: Tensorflow with GPU support (>=1.14)(uses tensorflow's slim library so doesnt support tf2.0) + Scipy (>=1.1) + Numpy (>=1.14) + Tqdm (>=4.0.0). To install in your python distribution, runpip install -r requirements.txt.Additional notes:Warning: Make sure your libraries (Cuda, Cudnn,...) are compatible with the TensorFlow version you're using or the code will not run.Required software (for resampling):SoX,FFmpeg.Important note: At the moment, this algorithm requires using 32-bit floating-point audio files to perform correctly. You can use sox to convert your file.Tested on Nvidia GeForce RTX 2080 GPU with Cuda (>=9.2) and CuDNN (>=7.3.0). CPU mode should also work with minor changes.The current pretrained models supportsr=22050Hz. Please make sure to resample your files first before using the metric.There are two ways to run the code:Using pipThis version currently supportsevaluatingthe trained metric, as well as using the metric forbackpropogation(as a loss function). This version currently does not support a simple interface totrainthe metric. For training, please clone this repo and follow the instructions below.pipinstalldpamPlease look atexample_pip.pyas an example on how to use the function.Cloning from the repositoryFollow the instructions below to: 1) Evaluate the metric on a few examples, 2) Train a metric on our dataset, 3) Use the metric to optimize a downstream task and 4) Use our pretrained speech enhancement model (trained using our loss function metric).Minimal basic usage as a distance metricRunning the command below takes two audio files as input and gives the perceptual distance between the files. It should return (approx)distance = 0.1929. Some GPU's are non-deterministic, and so the distance could vary in the lsb.cd metric_code
python metric_use_simple.py --e0 ../sample_audio/ref.wav --e1 ../sample_audio/2.wavFor loading large number of files, batch processing is more efficient. Refer to atmetric_code/metric_use.pyfor more information. In short, you need to change the dataloader functionload_full_data_list(). You also need to provide the path of the trained model as an input argument.Navigating this repositoryPercepAudio- main directorymetric_code- Section 1, training our metric on our JND datasetse_code- Section 2, training a speech enhancement model using our metric, trained abovedataset- our JND framework and dataset and text files containing perceptual judgmentspre-model- sample pre-trained models for easy referencesample_audio- sample audio files for comparisoncreate_space- sample code for creating perturbations(1) Train a perceptual metric on our JND datasetPercepAudio(main directory)metric_codemain.py (train the loss function)metric_use.py (use the trained model as a metric)dataloader.py (collect and load the audio files)helper.py (misc helper functions)network_model.py (NN architecture)summariesfolder to store the new trained model with tensorboard filesDownload the JND DatasetGo tolinkto download the dataset (about 23GB). After downloading the dataset, unzip the dataset into the project folder'PerceptualAudio/dataset'. Here are the steps to be followed:git clone https://github.com/pranaymanocha/PerceptualAudio.git
cd PerceptualAudio/dataset
unzip audio_perception.zipMore information on the JND framework can be found in the paperhere. The text files in the subfolderdatasetcontain information about human perceptual judgments. This sets up the dataset for training the loss function.For using a custom dataset, you need to follow the following steps:Follow a similar framework to obtain human perceptual judgments and store them in thedatasetsubdirectory. Also create a text file containing the results of all human perceptual judgments using a conventionreference_audio_path \t noisy_audio_path \t human judgment(same(0)/different(1)).
For an example, please see any text file indatasetsubdirectory.Make changes to the dataloader.py function to reflect the new name/path of the folders/text file.Run the main.py function (after selecting the most appropriate set of parameters).Once you train a model, you can use the trained model to infer the distances between audio recordings.Using the trained metric for evalYou can use one of our trained models as a metric. You can also use your own trained loss function as a metric for evaluation.For using a custom dataset, you need to follow the following steps:Make sure that you have all the right requirements as specified in therequirements.txtfile on the repo.Look atmetric_use.pyfor more information on how to use the trained model to infer distances between audio files. In short, you need to change the dataloader function (namely functionload_full_data_list()). You also need to provide the path of the trained model as an input argument. Please look at metric_use.py for full information.Pretrained Model“Off-the-shelf” deep network embeddings have been used as an effective training objective that have been shown to correlate well with human perceptual judgments in the vision setting, even without being
explicitly trained on perceptual human judgments. We first investigate if similar trends hold in the audio setting. Hence, we first train a model on two audio datasets: Acoustic scene classification and Domestic audio tagging tasks of theDCASE 2016 Challenge. We keep the architecture of the model same to compare between different training regimes. More information on training this pretrained "off-the-shelf" model can be found inthisrepo.Final Summary of the models (more info in the paper)pretrained- pretrained "off-the-shelf" modellinear-  training linear layers over the pretrained "off-the-shelf" modelfinetune- loading the pretrained "off-the-shelf weights" but training both the linear layer and the bulk modelscratch- training the full model from randomly initialized weights.(2) Speech denoising with our perceptual metric as a loss functionAs an application for our loss function, we use the trained loss function to train a Speech Enhancement Model. We use the Edinburgh Datashare publicly available datasethere. We use the same dataset with any alteration except resampling the dataset at 16KHz.
Direct links to download the dataset and resampling can be foundhere. Follow the instructions to download the SE data in the scriptdownload_sedata.shin the above repo. In general, we follow the same directory structure.PercepAudio(main directory)se_code(train a SE model using the above trained loss function)se_train.py (train the SE system)se_infer.py (Infer the SE system)datasettrainset_cleantrainset_noisyvalset_cleanvalset_noisydata_import.py (Dataloading)network_model.py (NN architecture)summariesfolder to store the new trained model with tensorboard filesAfter you download the dataset, follow this directory structure to copy the audio files accordingly.Training the SE ModelWe make use of our metric as a loss function for training an SE model. After you have downloaded the noisy dataset above (and kept the files at the correct locations), you can start training by running the command:python se_train.py --args.....The trained model is stored under thesummariesfolder under the folder name which you specify as an argument. The model is saved asse_model_'+str(seconds)+'.ckpt'where seconds is the time in seconds since epoch so that the training can be easily monitered.Inferring the SE ModelAfter you train a SE model, you can use the same trained model to denoise audio files. Simply runpython se_infer.py --args....with a suitable set of arguements. The denoised files will be stored in the folder name which you specify as an argument in the script. As the SE model is big, it takes a couple of hours to run on a CPU and less than 5 minutes on a GPU.CitationIf you use our code for research, please use the following to cite.@inproceedings{Manocha:2020:ADP,
   author = "Pranay Manocha and Adam Finkelstein and Richard Zhang and Nicholas J.
      Bryan and Gautham J. Mysore and Zeyu Jin",
   title = "A Differentiable Perceptual Audio Metric Learned from Just Noticeable
      Differences",
   booktitle = "Interspeech",
   year = "2020",
   month = oct
}LicenseThe source code is published under theMIT license. See LICENSE for details. In general, you can use the code for any purpose with proper attribution. If you do something interesting with the code, we'll be happy to know. Feel free to contact us. The primary contact is Pranay Manocha.