Package: audioldm2
AudioLDM 2This repo currently support Text-to-Audio (including Music) and Text-to-Speech Generation.Change Log2023-08-27: Add two new checkpoints!🌟48kHz AudioLDM model: Now we support high-fidelity audio generation! Use this checkpoint simply by setting "--model_name audioldm_48k"16kHz improved AudioLDM model: Trained with more data and optimized model architecture.TODOAdd the text-to-speech checkpointOpen-source the AudioLDM training code.Support the generation of longer audio (> 10s)Optimizing the inference speed of the model.Integration with the Diffusers libraryWeb APPPrepare running environmentcondacreate-naudioldmpython=3.8;condaactivateaudioldm
pip3installgit+https://github.com/haoheliu/AudioLDM2.git
gitclonehttps://github.com/haoheliu/AudioLDM2;cdAudioLDM2Start the web application (powered by Gradio)python3app.pyA link will be printed out. Click the link to open the browser and play.Commandline UsageInstallationPrepare running environment# Optionalcondacreate-naudioldmpython=3.8;condaactivateaudioldm# Install AudioLDMpip3installgit+https://github.com/haoheliu/AudioLDM2.gitIf you plan to play around with text-to-speech generation. Please also make sure you have installedespeak. On linux you can do it bysudoapt-getinstallespeakRun the model in commandlineGenerate sound effect or Music based on a text promptaudioldm2-t"Musical constellations twinkling in the night sky, forming a cosmic melody."Generate sound effect or music based on a list of textaudioldm2-tlbatch.lstGenerate speech based on (1) the transcription and (2) the description of the speakeraudioldm2-t"A female reporter is speaking full of emotion"--transciption"Wish you have a good day"audioldm2-t"A female reporter is speaking"--transciption"Wish you have a good day"Text-to-Speech use theaudioldm2-speech-gigaspeechcheckpoint by default. If you like to run TTS with LJSpeech pretrained checkpoint, simply set--model_name audioldm2-speech-ljspeech.Random Seed MattersSometimes model may not perform well (sounds wired or low quality) when changing into a different hardware. In this case, please adjust the random seed and find the optimal one for your hardware.audioldm2--seed1234-t"Musical constellations twinkling in the night sky, forming a cosmic melody."Pretrained ModelsYou can choose model checkpoint by setting up "model_name":# CUDAaudioldm2--model_name"audioldm_48k"--devicecuda-t"Musical constellations twinkling in the night sky, forming a cosmic melody."# MPSaudioldm2--model_name"audioldm_48k"--devicemps-t"Musical constellations twinkling in the night sky, forming a cosmic melody."We have five checkpoints you can choose:audioldm_48k(default): This checkpoint can generate high fidelity sound effect and music.audioldm2-full: Generate both sound effect and music generation with the AudioLDM2 architecture.audioldm_16k_crossattn_t5: The improved version ofAudioLDM 1.0.audioldm2-full-large-1150k: Larger version of audioldm2-full.audioldm2-music-665k: Music generation.audioldm2-speech-gigaspeech(default for TTS): Text-to-Speech, trained on GigaSpeech Dataset.audioldm2-speech-ljspeech: Text-to-Speech, trained on LJSpeech Dataset.We currently support 3 devices:cpucudamps ( Notice that the computation requires about 20GB of RAM. )Other optionsusage:audioldm2[-h][-tTEXT][-tlTEXT_LIST][-sSAVE_PATH][--model_name{audioldm2-full,audioldm2-music-665k,audioldm2-full-large-1150k,audioldm2-speech-ljspeech,audioldm2-speech-gigaspeech}][-dDEVICE][-bBATCHSIZE][--ddim_stepsDDIM_STEPS][-gsGUIDANCE_SCALE][-nN_CANDIDATE_GEN_PER_TEXT][--seedSEED]optionalarguments:-h,--helpshowthishelpmessageandexit-tTEXT,--textTEXTTextprompttothemodelforaudiogeneration--transcriptionTRANSCRIPTIONTranscriptionusedforspeechsynthesis-tlTEXT_LIST,--text_listTEXT_LISTAfilethatcontainstextprompttothemodelforaudiogeneration-sSAVE_PATH,--save_pathSAVE_PATHThepathtosavemodeloutput--model_name{audioldm2-full,audioldm2-music-665k,audioldm2-full-large-1150k,audioldm2-speech-ljspeech,audioldm2-speech-gigaspeech}Thecheckpointyougonnause-dDEVICE,--deviceDEVICEThedeviceforcomputation.Ifnotspecified,thescriptwillautomaticallychoosethedevicebasedonyourenvironment.[cpu,cuda,mps,auto]-bBATCHSIZE,--batchsizeBATCHSIZEGeneratehowmanysamplesatthesametime--ddim_stepsDDIM_STEPSThesamplingstepforDDIM-gsGUIDANCE_SCALE,--guidance_scaleGUIDANCE_SCALEGuidancescale(Large=>betterqualityandrelavancytotext;Small=>betterdiversity)-nN_CANDIDATE_GEN_PER_TEXT,--n_candidate_gen_per_textN_CANDIDATE_GEN_PER_TEXTAutomaticqualitycontrol.Thisnumbercontrolthenumberofcandidates(e.g.,generatethreeaudiosandchoosethebesttoshowyou).ALargervalueusuallyleadtobetterqualitywithheaviercomputation--seedSEEDChangethisvalue(anyintegernumber)willleadtoadifferentgenerationresult.Cite this workIf you found this tool useful, please consider citing@article{liu2023audioldm2,title={{AudioLDM 2}: Learning Holistic Audio Generation with Self-supervised Pretraining},author={Haohe Liu and Qiao Tian and Yi Yuan and Xubo Liu and Xinhao Mei and Qiuqiang Kong and Yuping Wang and Wenwu Wang and Yuxuan Wang and Mark D. Plumbley},journal={arXiv preprint arXiv:2308.05734},year={2023}}@article{liu2023audioldm,title={{AudioLDM}: Text-to-Audio Generation with Latent Diffusion Models},author={Liu, Haohe and Chen, Zehua and Yuan, Yi and Mei, Xinhao and Liu, Xubo and Mandic, Danilo and Wang, Wenwu and Plumbley, Mark D},journal={Proceedings of the International Conference on Machine Learning},year={2023}}