Package: parallel-wavegan
Parallel WaveGAN implementation with PytorchThis repository providesUNOFFICIALpytorch implementations of the following models:Parallel WaveGANMelGANMultiband-MelGANHiFi-GANStyleMelGANYou can combine these state-of-the-art non-autoregressive models to build your own great vocoder!Please check our samples inour demo HP.Source of the figure:https://arxiv.org/pdf/1910.11480.pdfThe goal of this repository is to provide real-time neural vocoder, which is compatible withESPnet-TTS.Also, this repository can be combined withNVIDIA/tacotron2-based implementation (Seethis comment).You can try the real-time end-to-end text-to-speech and singing voice synthesis demonstration in Google Colab!Real-time demonstration with ESPnet2Real-time demonstration with ESPnet1Real-time demonstration with MuskitsWhat's new2023/08/17LibriTTS-R recipeis available!2022/02/27 Support singing voice vocoder [egs/{kiritan, opencpop, oniku_kurumi_utagoe_db, ofuton_p_utagoe_db, csd, kising}/voc1]2021/10/21 Single-speaker Korean recipe [egs/kss/voc1] is available.2021/08/24 Add more pretrained models of StyleMelGAN and HiFi-GAN.2021/08/07 Add initial pretrained models of StyleMelGAN and HiFi-GAN.2021/08/03 SupportStyleMelGANgenerator and discriminator!2021/08/02 SupportHiFi-GANgenerator and discriminator!2020/10/07JSSSrecipe is available!2020/08/19Real-time demo with ESPnet2is available!2020/05/29VCTK, JSUT, and CSMSC multi-band MelGAN pretrained modelis available!2020/05/27New LJSpeech multi-band MelGAN pretrained modelis available!2020/05/24LJSpeech full-band MelGAN pretrained modelis available!2020/05/22LJSpeech multi-band MelGAN pretrained modelis available!2020/05/16Multi-band MelGANis available!2020/03/25LibriTTS pretrained modelsare available!2020/03/17Tensorflow conversion example notebookis available (Thanks,@dathudeptrai)!2020/03/16LibriTTS recipeis available!2020/03/12PWG G + MelGAN D + STFT-loss samplesare available!2020/03/12 Multi-speaker English recipeegs/vctk/voc1is available!2020/02/22MelGAN G + MelGAN D + STFT-loss samplesare available!2020/02/12 SupportMelGAN's discriminator!2020/02/08 SupportMelGAN's generator!RequirementsThis repository is tested on Ubuntu 20.04 with a GPU Titan V.Python 3.8+Cuda 11.0+CuDNN 8+NCCL 2+ (for distributed multi-gpu training)libsndfile (you can install viasudo apt install libsndfile-devin ubuntu)jq (you can install viasudo apt install jqin ubuntu)sox (you can install viasudo apt install soxin ubuntu)Different cuda version should be working but not explicitly tested.All of the codes are tested on Pytorch 1.8.1, 1.9, 1.10.2, 1.11.0, 1.12.1, 1.13.1, 2.0.1 and 2.1.0.SetupYou can select the installation method from two alternatives.A. Use pip$gitclonehttps://github.com/kan-bayashi/ParallelWaveGAN.git
$cdParallelWaveGAN
$pipinstall-e.# If you want to use distributed training, please install# apex manually by following https://github.com/NVIDIA/apex$...Note that your cuda version must be exactly matched with the version used for the pytorch binary to install apex.To install pytorch compiled with different cuda version, seetools/Makefile.B. Make virtualenv$gitclonehttps://github.com/kan-bayashi/ParallelWaveGAN.git
$cdParallelWaveGAN/tools
$make# If you want to use distributed training, please run following# command to install apex.$makeapexNote that we specify cuda version used to compile pytorch wheel.If you want to use different cuda version, please checktools/Makefileto change the pytorch wheel to be installed.RecipeThis repository providesKaldi-style recipes, as the same asESPnet.Currently, the following recipes are supported.LJSpeech: English female speakerJSUT: Japanese female speakerJSSS: Japanese female speakerCSMSC: Mandarin female speakerCMU Arctic: English speakersJNAS: Japanese multi-speakerVCTK: English multi-speakerLibriTTS: English multi-speakerLibriTTS-R: English multi-speaker enhanced by speech restoration.YesNo: English speaker (For debugging)KSS: Single Korean female speakerOniku_kurumi_utagoe_db/: Single Japanese female singer (singing voice)Kiritan: Single Japanese male singer (singing voice)Ofuton_p_utagoe_db: Single Japanese female singer (singing voice)Opencpop: Single Mandarin female singer (singing voice)CSD: Single Korean/English female singer (singing voice)KiSing: Single Mandarin female singer (singing voice)To run the recipe, please follow the below instruction.# Let us move on the recipe directory$cdegs/ljspeech/voc1# Run the recipe from scratch$./run.sh# You can change config via command line$./run.sh--conf<your_customized_yaml_config># You can select the stage to start and stop$./run.sh--stage2--stop_stage2# If you want to specify the gpu$CUDA_VISIBLE_DEVICES=1./run.sh--stage2# If you want to resume training from 10000 steps checkpoint$./run.sh--stage2--resume<path>/<to>/checkpoint-10000steps.pklSee more info about the recipes inthis README.SpeedThe decoding speed is RTF = 0.016 with TITAN V, much faster than the real-time.[decode]:100%|██████████|250/250[00:30<00:00,8.31it/s,RTF=0.0156]2019-11-0309:07:40,480(decode:127)INFO:finishedgenerationof250utterances(RTF=0.016).Even on the CPU (Intel(R) Xeon(R) Gold 6154 CPU @ 3.00GHz 16 threads), it can generate less than the real-time.[decode]:100%|██████████|250/250[22:16<00:00,5.35s/it,RTF=0.841]2019-11-0609:04:56,697(decode:129)INFO:finishedgenerationof250utterances(RTF=0.734).If you use MelGAN's generator, the decoding speed will be further faster.# On CPU (Intel(R) Xeon(R) Gold 6154 CPU @ 3.00GHz 16 threads)[decode]:100%|██████████|250/250[04:00<00:00,1.04it/s,RTF=0.0882]2020-02-0810:45:14,111(decode:142)INFO:Finishedgenerationof250utterances(RTF=0.137).# On GPU (TITAN V)[decode]:100%|██████████|250/250[00:06<00:00,36.38it/s,RTF=0.00189]2020-02-0805:44:42,231(decode:142)INFO:Finishedgenerationof250utterances(RTF=0.002).If you use Multi-band MelGAN's generator, the decoding speed will be much further faster.# On CPU (Intel(R) Xeon(R) Gold 6154 CPU @ 3.00GHz 16 threads)[decode]:100%|██████████|250/250[01:47<00:00,2.95it/s,RTF=0.048]2020-05-2215:37:19,771(decode:151)INFO:Finishedgenerationof250utterances(RTF=0.059).# On GPU (TITAN V)[decode]:100%|██████████|250/250[00:05<00:00,43.67it/s,RTF=0.000928]2020-05-2215:35:13,302(decode:151)INFO:Finishedgenerationof250utterances(RTF=0.001).If you want to accelerate the inference more, it is worthwhile to try the conversion from pytorch to tensorflow.The example of the conversion is available inthe notebook(Provided by@dathudeptrai).ResultsHere the results are summarized in the table.You can listen to the samples and download pretrained models from the link to our google drive.ModelConfLangFs [Hz]Mel range [Hz]FFT / Hop / Win [pt]# itersljspeech_parallel_wavegan.v1linkEN22.05k80-76001024 / 256 / None400kljspeech_parallel_wavegan.v1.longlinkEN22.05k80-76001024 / 256 / None1Mljspeech_parallel_wavegan.v1.no_limitlinkEN22.05kNone1024 / 256 / None400kljspeech_parallel_wavegan.v3linkEN22.05k80-76001024 / 256 / None3Mljspeech_melgan.v1linkEN22.05k80-76001024 / 256 / None400kljspeech_melgan.v1.longlinkEN22.05k80-76001024 / 256 / None1Mljspeech_melgan_large.v1linkEN22.05k80-76001024 / 256 / None400kljspeech_melgan_large.v1.longlinkEN22.05k80-76001024 / 256 / None1Mljspeech_melgan.v3linkEN22.05k80-76001024 / 256 / None2Mljspeech_melgan.v3.longlinkEN22.05k80-76001024 / 256 / None4Mljspeech_full_band_melgan.v1linkEN22.05k80-76001024 / 256 / None1Mljspeech_full_band_melgan.v2linkEN22.05k80-76001024 / 256 / None1Mljspeech_multi_band_melgan.v1linkEN22.05k80-76001024 / 256 / None1Mljspeech_multi_band_melgan.v2linkEN22.05k80-76001024 / 256 / None1Mljspeech_hifigan.v1linkEN22.05k80-76001024 / 256 / None2.5Mljspeech_style_melgan.v1linkEN22.05k80-76001024 / 256 / None1.5Mjsut_parallel_wavegan.v1linkJP24k80-76002048 / 300 / 1200400kjsut_multi_band_melgan.v2linkJP24k80-76002048 / 300 / 12001Mjust_hifigan.v1linkJP24k80-76002048 / 300 / 12002.5Mjust_style_melgan.v1linkJP24k80-76002048 / 300 / 12001.5Mcsmsc_parallel_wavegan.v1linkZH24k80-76002048 / 300 / 1200400kcsmsc_multi_band_melgan.v2linkZH24k80-76002048 / 300 / 12001Mcsmsc_hifigan.v1linkZH24k80-76002048 / 300 / 12002.5Mcsmsc_style_melgan.v1linkZH24k80-76002048 / 300 / 12001.5Marctic_slt_parallel_wavegan.v1linkEN16k80-76001024 / 256 / None400kjnas_parallel_wavegan.v1linkJP16k80-76001024 / 256 / None400kvctk_parallel_wavegan.v1linkEN24k80-76002048 / 300 / 1200400kvctk_parallel_wavegan.v1.longlinkEN24k80-76002048 / 300 / 12001Mvctk_multi_band_melgan.v2linkEN24k80-76002048 / 300 / 12001Mvctk_hifigan.v1linkEN24k80-76002048 / 300 / 12002.5Mvctk_style_melgan.v1linkEN24k80-76002048 / 300 / 12001.5Mlibritts_parallel_wavegan.v1linkEN24k80-76002048 / 300 / 1200400klibritts_parallel_wavegan.v1.longlinkEN24k80-76002048 / 300 / 12001Mlibritts_multi_band_melgan.v2linkEN24k80-76002048 / 300 / 12001Mlibritts_hifigan.v1linkEN24k80-76002048 / 300 / 12002.5Mlibritts_style_melgan.v1linkEN24k80-76002048 / 300 / 12001.5Mkss_parallel_wavegan.v1linkKO24k80-76002048 / 300 / 1200400khui_acg_hokuspokus_parallel_wavegan.v1linkDE24k80-76002048 / 300 / 1200400kruslan_parallel_wavegan.v1linkRU24k80-76002048 / 300 / 1200400koniku_hifigan.v1linkJP24k80-76002048 / 300 / 1200250kkiritan_hifigan.v1linkJP24k80-76002048 / 300 / 1200300kofuton_hifigan.v1linkJP24k80-76002048 / 300 / 1200300kopencpop_hifigan.v1linkZH24k80-76002048 / 300 / 1200250kcsd_english_hifigan.v1linkEN24k80-76002048 / 300 / 1200300kcsd_korean_hifigan.v1linkEN24k80-76002048 / 300 / 1200250kkising_hifigan.v1linkZH24k80-76002048 / 300 / 1200300kPlease access atour google driveto check more results.Please check the license of database (e.g., whether it is proper for commercial usage) before using the pre-trained model.The authors will not be responsible for any loss due to the use of the model and legal disputes regarding the use of the dataset.How-to-use pretrained modelsAnalysis-synthesisHere the minimal code is shown to perform analysis-synthesis using the pretrained model.# Please make sure you installed `parallel_wavegan`# If not, please install via pip$pipinstallparallel_wavegan# You can download the pretrained model from terminal$python<< EOFfrom parallel_wavegan.utils import download_pretrained_modeldownload_pretrained_model("<pretrained_model_tag>", "pretrained_model")EOF# You can get all of available pretrained models as follows:$python<< EOFfrom parallel_wavegan.utils import PRETRAINED_MODEL_LISTprint(PRETRAINED_MODEL_LIST.keys())EOF# Now you can find downloaded pretrained model in `pretrained_model/<pretrain_model_tag>/`$lspretrain_model/<pretrain_model_tag>
checkpoint-400000steps.pklconfig.ymlstats.h5# These files can also be downloaded manually from the above results# Please put an audio file in `sample` directory to perform analysis-synthesis$lssample/
sample.wav# Then perform feature extraction -> feature normalization -> synthesis$parallel-wavegan-preprocess\--configpretrain_model/<pretrain_model_tag>/config.yml\--rootdirsample\--dumpdirdump/sample/raw100%|████████████████████████████████████████|1/1[00:00<00:00,914.19it/s]$parallel-wavegan-normalize\--configpretrain_model/<pretrain_model_tag>/config.yml\--rootdirdump/sample/raw\--dumpdirdump/sample/norm\--statspretrain_model/<pretrain_model_tag>/stats.h52019-11-1313:44:29,574(normalize:87)INFO:thenumberoffiles=1.100%|████████████████████████████████████████|1/1[00:00<00:00,513.13it/s]$parallel-wavegan-decode\--checkpointpretrain_model/<pretrain_model_tag>/checkpoint-400000steps.pkl\--dumpdirdump/sample/norm\--outdirsample2019-11-1313:44:31,229(decode:91)INFO:thenumberoffeaturestobedecoded=1.[decode]:100%|███████████████████|1/1[00:00<00:00,18.33it/s,RTF=0.0146]2019-11-1313:44:37,132(decode:129)INFO:finishedgenerationof1utterances(RTF=0.015).# You can skip normalization step (on-the-fly normalization, feature extraction -> synthesis)$parallel-wavegan-preprocess\--configpretrain_model/<pretrain_model_tag>/config.yml\--rootdirsample\--dumpdirdump/sample/raw100%|████████████████████████████████████████|1/1[00:00<00:00,914.19it/s]$parallel-wavegan-decode\--checkpointpretrain_model/<pretrain_model_tag>/checkpoint-400000steps.pkl\--dumpdirdump/sample/raw\--normalize-before\--outdirsample2019-11-1313:44:31,229(decode:91)INFO:thenumberoffeaturestobedecoded=1.[decode]:100%|███████████████████|1/1[00:00<00:00,18.33it/s,RTF=0.0146]2019-11-1313:44:37,132(decode:129)INFO:finishedgenerationof1utterances(RTF=0.015).# you can find the generated speech in `sample` directory$lssample
sample.wavsample_gen.wavDecoding with ESPnet-TTS model's featuresHere, I show the procedure to generate waveforms with features generated byESPnet-TTSmodels.# Make sure you already finished running the recipe of ESPnet-TTS.# You must use the same feature settings for both Text2Mel and Mel2Wav models.# Let us move on "ESPnet" recipe directory$cd/path/to/espnet/egs/<recipe_name>/tts1
$pwd/path/to/espnet/egs/<recipe_name>/tts1# If you use ESPnet2, move on `egs2/`$cd/path/to/espnet/egs2/<recipe_name>/tts1
$pwd/path/to/espnet/egs2/<recipe_name>/tts1# Please install this repository in ESPnet conda (or virtualenv) environment$../path.sh&&pipinstall-Uparallel_wavegan# You can download the pretrained model from terminal$python<< EOFfrom parallel_wavegan.utils import download_pretrained_modeldownload_pretrained_model("<pretrained_model_tag>", "pretrained_model")EOF# You can get all of available pretrained models as follows:$python<< EOFfrom parallel_wavegan.utils import PRETRAINED_MODEL_LISTprint(PRETRAINED_MODEL_LIST.keys())EOF# You can find downloaded pretrained model in `pretrained_model/<pretrain_model_tag>/`$lspretrain_model/<pretrain_model_tag>
checkpoint-400000steps.pklconfig.ymlstats.h5# These files can also be downloaded manually from the above resultsCase 1: If you use the same dataset for both Text2Mel and Mel2Wav# In this case, you can directly use generated features for decoding.# Please specify `feats.scp` path for `--feats-scp`, which is located in# exp/<your_model_dir>/outputs_*_decode/<set_name>/feats.scp.# Note that do not use outputs_*decode_denorm/<set_name>/feats.scp since# it is de-normalized features (the input for PWG is normalized features).$parallel-wavegan-decode\--checkpointpretrain_model/<pretrain_model_tag>/checkpoint-400000steps.pkl\--feats-scpexp/<your_model_dir>/outputs_*_decode/<set_name>/feats.scp\--outdir<path_to_outdir># In the case of ESPnet2, the generated feature can be found in# exp/<your_model_dir>/decode_*/<set_name>/norm/feats.scp.$parallel-wavegan-decode\--checkpointpretrain_model/<pretrain_model_tag>/checkpoint-400000steps.pkl\--feats-scpexp/<your_model_dir>/decode_*/<set_name>/norm/feats.scp\--outdir<path_to_outdir># You can find the generated waveforms in <path_to_outdir>/.$ls<path_to_outdir>
utt_id_1_gen.wavutt_id_2_gen.wav...utt_id_N_gen.wavCase 2: If you use different datasets for Text2Mel and Mel2Wav models# In this case, you must provide `--normalize-before` option additionally.# And use `feats.scp` of de-normalized generated features.# ESPnet1 case$parallel-wavegan-decode\--checkpointpretrain_model/<pretrain_model_tag>/checkpoint-400000steps.pkl\--feats-scpexp/<your_model_dir>/outputs_*_decode_denorm/<set_name>/feats.scp\--outdir<path_to_outdir>\--normalize-before# ESPnet2 case$parallel-wavegan-decode\--checkpointpretrain_model/<pretrain_model_tag>/checkpoint-400000steps.pkl\--feats-scpexp/<your_model_dir>/decode_*/<set_name>/denorm/feats.scp\--outdir<path_to_outdir>\--normalize-before# You can find the generated waveforms in <path_to_outdir>/.$ls<path_to_outdir>
utt_id_1_gen.wavutt_id_2_gen.wav...utt_id_N_gen.wavIf you want to combine these models in python, you can try the real-time demonstration in Google Colab!Real-time demonstration with ESPnet2Real-time demonstration with ESPnet1Decoding with dumped npy filesSometimes we want to decode with dumped npy files, which are mel-spectrogram generated by TTS models.
Please make sure you used the same feature extraction settings of the pretrained vocoder (fs,fft_size,hop_size,win_length,fmin, andfmax).Only the difference oflog_basecan be changed with some post-processings (we use log 10 instead of natural log as a default).
See detail inthe comment.# Generate dummy npy file of mel-spectrogram$ipython[ins]In[1]:importnumpyasnp[ins]In[2]:x=np.random.randn(512,80)# (#frames, #mels)[ins]In[3]:np.save("dummy_1.npy",x)[ins]In[4]:y=np.random.randn(256,80)# (#frames, #mels)[ins]In[5]:np.save("dummy_2.npy",y)[ins]In[6]:exit# Make scp file (key-path format)$find-name"*.npy"|awk'{print "dummy_" NR " " $1}'>feats.scp# Check (<utt_id> <path>)$catfeats.scp
dummy_1./dummy_1.npy
dummy_2./dummy_2.npy# Decode without feature normalization# This case assumes that the input mel-spectrogram is normalized with the same statistics of the pretrained model.$parallel-wavegan-decode\--checkpoint/path/to/checkpoint-400000steps.pkl\--feats-scp./feats.scp\--outdirwav2021-08-1009:13:07,624(decode:140)INFO:Thenumberoffeaturestobedecoded=2.[decode]:100%|████████████████████████████████████████|2/2[00:00<00:00,13.84it/s,RTF=0.00264]2021-08-1009:13:29,660(decode:174)INFO:Finishedgenerationof2utterances(RTF=0.005).# Decode with feature normalization# This case assumes that the input mel-spectrogram is not normalized.$parallel-wavegan-decode\--checkpoint/path/to/checkpoint-400000steps.pkl\--feats-scp./feats.scp\--normalize-before\--outdirwav2021-08-1009:13:07,624(decode:140)INFO:Thenumberoffeaturestobedecoded=2.[decode]:100%|████████████████████████████████████████|2/2[00:00<00:00,13.84it/s,RTF=0.00264]2021-08-1009:13:29,660(decode:174)INFO:Finishedgenerationof2utterances(RTF=0.005).NotesThe terms of use of the pretrained model follow that of each corpus used for the training. Please carefully check by yourself.Some codes are derived from ESPnet or Kaldi, which are based on Apache-2.0 licenese.ReferencesParallel WaveGANr9y9/wavenet_vocoderLiyuanLucasLiu/RAdamMelGANdescriptinc/melgan-neuripsMulti-band MelGANHiFi-GANjik876/hifi-ganStyleMelGANAcknowledgementThe author would like to thank Ryuichi Yamamoto (@r9y9) for his great repository, paper, and valuable discussions.AuthorTomoki Hayashi (@kan-bayashi)E-mail:hayashi.tomoki<at>g.sp.m.is.nagoya-u.ac.jp