Package: audiodiffusion
title: Audio Diffusion
emoji: 🎵
colorFrom: pink
colorTo: blue
sdk: gradio
sdk_version: 3.1.4
app_file: app.py
pinned: false
license: gpl-3.0audio-diffusionApply diffusion models to synthesize music instead of images using the new Hugging FacediffuserspackageSample automatically generated loophttps://user-images.githubusercontent.com/44233095/204103172-27f25d63-5e77-40ca-91ab-d04a45d4726f.mp4Go tohttps://soundcloud.com/teticio2/sets/audio-diffusion-loopsfor more examples.Updates25/12/2022. Now it is possible to train models conditional on an encoding (of text or audio, for example). See the section on Conditional Audio Generation below.5/12/2022. 🤗 Exciting news!AudioDiffusionPipelinehas been migrated to the Hugging Facediffuserspackage so that it is even easier for others to use and contribute.2/12/2022. Added Mel to pipeline and updated the pretrained models to save Mel config (they are now no longer compatible with previous versions of this repo). It is relatively straightforward to migrate previously trained models to the new format (seehttps://huggingface.co/teticio/audio-diffusion-256).7/11/2022. Added pre-trained latent audio diffusion modelsteticio/latent-audio-diffusion-256andteticio/latent-audio-diffusion-ddim-256. You can use the pre-trained VAE to train your own latent diffusion models on a different set of audio files.22/10/2022. Added DDIM encoder and ability to interpolate between audios in latent "noise" space. Mel spectrograms no longer have to be square (thanks to Tristan for this one), so you can set the vertical (frequency) and horizontal (time) resolutions independently.15/10/2022. Added latent audio diffusion (see below). Also added the possibility to train a DDIM (De-noising Diffusion Implicit Models). These have the benefit that samples can be generated with much fewer steps (~50) than used in training.4/10/2022. It is now possible to mask parts of the input audio during generation which means you can stitch several samples together (think "out-painting").27/9/2022. You can now generate an audio based on a previous one. You can use this to generate variations of the same audio or even to "remix" a track (via a sort of "style transfer"). You can find examples of how to do this in thetest_model.ipynbnotebook.DDPM (De-noising Diffusion Probabilistic Models)Audio can be represented as images by transforming to amel spectrogram, such as the one shown above. The classMelinmel.pycan convert a slice of audio into a mel spectrogram ofx_resxy_resand vice versa. The higher the resolution, the less audio information will be lost. You can see how this works in thetest_mel.ipynbnotebook.A DDPM is trained on a set of mel spectrograms that have been generated from a directory of audio files. It is then used to synthesize similar mel spectrograms, which are then converted back into audio.You can play around with some pre-trained models onGoogle ColaborHugging Face spaces. Check out some automatically generated loopshere.ModelDatasetDescriptionteticio/audio-diffusion-256teticio/audio-diffusion-256My "liked" Spotify playlistteticio/audio-diffusion-breaks-256teticio/audio-diffusion-breaks-256Samples that have been used in music, sourced fromWhoSampledandYouTubeteticio/audio-diffusion-instrumental-hiphop-256teticio/audio-diffusion-instrumental-hiphop-256Instrumental Hip Hop musicteticio/audio-diffusion-ddim-256teticio/audio-diffusion-256De-noising Diffusion Implicit Modelteticio/latent-audio-diffusion-256teticio/audio-diffusion-256Latent Audio Diffusion modelteticio/latent-audio-diffusion-ddim-256teticio/audio-diffusion-256Latent Audio Diffusion Implicit Modelteticio/conditional-latent-audio-diffusion-512teticio/audio-diffusion-512Conditional Latent Audio Diffusion ModelGenerate Mel spectrogram dataset from directory of audio filesInstall from GitHub (includes training scripts)gitclonehttps://github.com/teticio/audio-diffusion.gitcdaudio-diffusion
pipinstall.Install from PyPIpipinstallaudiodiffusionTraining can be run with Mel spectrograms of resolution 64x64 on a single commercial grade GPU (e.g. RTX 2080 Ti). Thehop_lengthshould be set to 1024 for better resultspythonscripts/audio_to_images.py\--resolution64,64\--hop_length1024\--input_dirpath-to-audio-files\--output_dirpath-to-output-dataGenerate dataset of 256x256 Mel spectrograms and push to hub (you will need to be authenticated withhuggingface-cli login)pythonscripts/audio_to_images.py\--resolution256\--input_dirpath-to-audio-files\--output_dirdata/audio-diffusion-256\--push_to_hubteticio/audio-diffusion-256Note that the defaultsample_rateis 22050 and audios will be resampled if they are at a different rate. If you change this value, you may find that the results in thetest_mel.ipynbnotebook are not good (for example, ifsample_rateis 48000) and that it is necessary to adjustn_fft(for example, to 2000 instead of the default value of 2048; alternatively, you can resample to asample_rateof 44100). Make sure you use the same parameters for training and inference. You should also bear in mind that not all resolutions work with the neural network architecture as currently configured - you should be safe if you stick to powers of 2.Train modelRun training on local machineacceleratelaunch--config_fileconfig/accelerate_local.yaml\scripts/train_unet.py\--dataset_namedata/audio-diffusion-64\--hop_length1024\--output_dirmodels/ddpm-ema-audio-64\--train_batch_size16\--num_epochs100\--gradient_accumulation_steps1\--learning_rate1e-4\--lr_warmup_steps500\--mixed_precisionnoRun training on local machine withbatch_sizeof 2 andgradient_accumulation_steps8 to compensate, so that 256x256 resolution model fits on commercial grade GPU and push to hubacceleratelaunch--config_fileconfig/accelerate_local.yaml\scripts/train_unet.py\--dataset_nameteticio/audio-diffusion-256\--output_dirmodels/audio-diffusion-256\--num_epochs100\--train_batch_size2\--eval_batch_size2\--gradient_accumulation_steps8\--learning_rate1e-4\--lr_warmup_steps500\--mixed_precisionno\--push_to_hubTrue\--hub_model_idaudio-diffusion-256\--hub_token$(cat$HOME/.huggingface/token)Run training on SageMakeracceleratelaunch--config_fileconfig/accelerate_sagemaker.yaml\scripts/train_unet.py\--dataset_nameteticio/audio-diffusion-256\--output_dirmodels/ddpm-ema-audio-256\--train_batch_size16\--num_epochs100\--gradient_accumulation_steps1\--learning_rate1e-4\--lr_warmup_steps500\--mixed_precisionnoDDIM (De-noising Diffusion Implicit Models)A DDIM can be trained by adding the parameter--schedulerddimInference can the be run with far fewer steps than the number used for training (e.g., ~50), allowing for much faster generation. Without retraining, the parameteretacan be used to replicate a DDPM if it is set to 1 or a DDIM if it is set to 0, with all values in between being valid. Whenetais 0 (the default value), the de-noising procedure is deterministic, which means that it can be run in reverse as a kind of encoder that recovers the original noise used in generation. A functionencodehas been added toAudioDiffusionPipelinefor this purpose. It is then possible to interpolate between audios in the latent "noise" space using the functionslerp(Spherical Linear intERPolation).Latent Audio DiffusionRather than de-noising images directly, it is interesting to work in the "latent space" after first encoding images using an autoencoder. This has a number of advantages. Firstly, the information in the images is compressed into a latent space of a much lower dimension, so it is much faster to train de-noising diffusion models and run inference with them. Secondly, similar images tend to be clustered together and interpolating between two images in latent space can produce meaningful combinations.At the time of writing, the Hugging Facediffuserslibrary is geared towards inference and lacking in training functionality (rather like its cousintransformersin the early days of development). In order to train a VAE (Variational AutoEncoder), I use thestable-diffusionrepo from CompVis and convert the checkpoints todiffusersformat. Note that it uses a perceptual loss function for images; it would be nice to try a perceptualaudioloss function.Train latent diffusion model using pre-trained VAEacceleratelaunch...
...
--vaeteticio/latent-audio-diffusion-256Install dependencies to train with Stable Diffusionpipinstallomegaconfpytorch_lightning==1.7.7torchvisioneinops
pipinstall-egit+https://github.com/CompVis/stable-diffusion.git@main#egg=latent-diffusion
pipinstall-egit+https://github.com/CompVis/taming-transformers.git@master#egg=taming-transformersTrain an autoencoderpythonscripts/train_vae.py\--dataset_nameteticio/audio-diffusion-256\--batch_size2\--gradient_accumulation_steps12Train latent diffusion modelacceleratelaunch...
...
--vaemodels/autoencoder-klConditional Audio GenerationWe can generate audio conditional on a text prompt - or indeed anything which can be encoded into a bunch of numbers - much like DALL-E2, Midjourney and Stable Diffusion. It is generally harder to find good quality datasets of audios together with descriptions, although the people behind the dataset used to train Stable Diffusion are making some very interesting progresshere. I have chosen to encode the audio directly instead based on "how it sounds", using amodel which I trained on hundreds of thousands of Spotify playlists. To encode an audio into a 100 dimensional vectorfromaudiodiffusion.audio_encoderimportAudioEncoderaudio_encoder=AudioEncoder.from_pretrained("teticio/audio-encoder")audio_encoder.encode(['/home/teticio/Music/liked/Agua Re - Holy Dance - Large Sound Mix.mp3'])Once you have prepared a dataset, you can encode the audio files with this scriptpythonscripts/encode_audio\--dataset_nameteticio/audio-diffusion-256\--out_filedata/encodings.pThen you can train a model withacceleratelaunch...
...
--encodingsdata/encodings.pWhen generating audios, you will need to pass anencodingsTensor. See theconditional_generation.ipynbnotebook for an example that uses encodings of Spotify track previews to influence the generation.