Package: qppwg
Quasi-Periodic Parallel WaveGAN (QPPWG)This is officialQPPWGPyTorch implementation.
QPPWG is a non-autoregressive neural speech generation model developed based onPWGandQPstructure.In this repo, we provide an example to train and test QPPWG as a vocoder forWORLDacoustic features.
More details can be found on ourDemopage.News2020/7/22Release v0.1.22020/6/27Releasemel-specfeature extraction and the pre-trained models ofvcc20corpus.2020/6/26Release the pre-trained models ofvcc18corpus.2020/5/20Release the first version (v0.1.1).RequirementsThis repository is tested on Ubuntu 16.04 with a Titan V GPU.Python 3.6+Cuda 10.0CuDNN 7+PyTorch 1.0.1+Environment setupThe code works with both anaconda and virtualenv.
The following example uses anaconda.$condacreate-nvenvQPPWGpython=3.6
$sourceactivatevenvQPPWG
$gitclonehttps://github.com/bigpon/QPPWG.git
$cdQPPWG
$pipinstall-e.Please refer to thePWGrepo for more details.Folder architectureegs:
The folder for projects.egs/vcc18:
The folder of the VCC2018 project.egs/vcc18/exp:
The folder for trained models.egs/vcc18/conf:
The folder for configs.egs/vcc18/data:
The folder for corpus related files (wav, feature, list ...).qppwg:
The folder of the source codes.RunCorpus and path setupModify the corresponding CUDA paths inegs/vcc18/run.py.Download theVoice Conversion Challenge 2018(VCC2018) corpus to run the QPPWG example$cdegs/vcc18# Download training and validation corpus$wget-otrain.log-Odata/wav/train.ziphttps://datashare.is.ed.ac.uk/bitstream/handle/10283/3061/vcc2018_database_training.zip# Download evaluation corpus$wget-oeval.log-Odata/wav/eval.ziphttps://datashare.is.ed.ac.uk/bitstream/handle/10283/3061/vcc2018_database_evaluation.zip# unzip corpus$unzipdata/wav/train.zip-ddata/wav/
$unzipdata/wav/eval.zip-ddata/wav/Training wav lists:data/scp/vcc18_train_22kHz.scp.Validation wav lists:data/scp/vcc18_valid_22kHz.scp.Testing wav list:data/scp/vcc18_eval_22kHz.scp.Preprocessing# Extract WORLD acoustic features and statistics of training and testing data$bashrun.sh--stage0--configPWG_30WORLD-related settings can be changed inegs/vcc18/conf/vcc18.PWG_30.yaml.If you want to use another corpus, please create a corresponding config and a file including power thresholds and f0 ranges likeegs/vcc18/data/pow_f0_dict.yml.More details about feature extraction can be found in theQPNetrepo.The lists of auxiliary features will be automatically generated.Training aux lists:data/scp/vcc18_train_22kHz.list.Validation aux lists:data/scp/vcc18_valid_22kHz.list.Testing aux list:data/scp/vcc18_eval_22kHz.list.QPPWG/PWG training# Training a QPPWG model with the 'QPPWGaf_20' config and the 'vcc18_train_22kHz' and 'vcc18_valid_22kHz' sets.$bashrun.sh--gpu0--stage1--confQPPWGaf_20\--trainsetvcc18_train_22kHz--validsetvcc18_valid_22kHzThe gpu ID can be set by --gpu GPU_ID (default: 0)The model architecture can be set by --conf CONFIG (default: PWG_30)The trained model resume can be set by --resume NUM (default: None)QPPWG/PWG testing# QPPWG/PWG decoding w/ natural acoustic features$bashrun.sh--gpu0--stage2--confQPPWGaf_20\--iter400000--trainsetvcc18_train_22kHz--evalsetvcc18_eval_22kHz# QPPWG/PWG decoding w/ scaled f0 (ex: halved f0).$bashrun.sh--gpu0--stage3--confQPPWGaf_20--scaled0.50\--iter400000--trainsetvcc18_train_22kHz--evalsetvcc18_eval_22kHzMonitor training progress$tensorboard--logdirexpThe training time of PWG_30 with a TITAN V is around 3 days.The training time of QPPWGaf_20 with a TITAN V is around 5 days.Inference speed (RTF)Vanilla PWG (PWG_30)# On CPU (Intel(R) Xeon(R) Gold 6142 CPU @ 2.60GHz 32 threads)[decode]:100%|███████████|140/140[04:50<00:00,2.08s/it,RTF=0.771]2020-05-2612:30:27,273(decode:156)INFO:Finishedgenerationof140utterances(RTF=0.579).# On GPU (TITAN V)[decode]:100%|███████████|140/140[00:09<00:00,14.89it/s,RTF=0.0155]2020-05-2612:32:26,160(decode:156)INFO:Finishedgenerationof140utterances(RTF=0.016).PWG w/ only 20 blocks (PWG_20)# On CPU (Intel(R) Xeon(R) Gold 6142 CPU @ 2.60GHz 32 threads)[decode]:100%|███████████|140/140[03:57<00:00,1.70s/it,RTF=0.761]2020-05-3013:50:20,438(decode:156)INFO:Finishedgenerationof140utterances(RTF=0.474).# On GPU (TITAN V)[decode]:100%|███████████|140/140[00:08<00:00,16.55it/s,RTF=0.0105]2020-05-3013:43:50,793(decode:156)INFO:Finishedgenerationof140utterances(RTF=0.011).QPPWG (QPPWGaf_20)# On CPU (Intel(R) Xeon(R) Gold 6142 CPU @ 2.60GHz 32 threads)[decode]:100%|███████████|140/140[04:12<00:00,1.81s/it,RTF=0.455]2020-05-2612:38:15,982(decode:156)INFO:Finishedgenerationof140utterances(RTF=0.512).# On GPU (TITAN V)[decode]:100%|███████████|140/140[00:11<00:00,12.57it/s,RTF=0.0218]2020-05-2612:33:32,469(decode:156)INFO:Finishedgenerationof140utterances(RTF=0.020).Models and resultsThe pre-trained models and generated utterances are released.You can download the whole folder of each corpus and then put it inegs/[corpus]to run speech generations with the pre-trained models.You also can only download the[corpus]/datafolder and the desired pre-trained model and then put thedatafolder inegs/[corpus]and the model folder inegs/[corpus]/exp.Both models with 100,000 iterations (trained w/ only STFT loss) and 400,000 iterations (trained w/ STFT and GAN losses) are released.The generated utterances are in thewavfolder of each model’s folder.CorpusLangFs [Hz]FeatureModelConfvcc18EN22050world(uv + f0 + mcep + ap)(shiftms: 5)PWG_20linkPWG_30linkQPPWGaf_20linkvcc20EN, FI, DE, ZH24000melf0h128(uv + f0 + mel-spc)(hop_size: 128)PWG_20linkPWG_30linkQPPWGaf_20linkUsage of pre-trained modelsAnalysis-synthesisThe minimum code for performing analysis and synthesis is presented.# Make sure you have installed `qppwg`# If not, install it via pip$pipinstallqppwg# Take "vcc18" corpus as an example# Download the whole folder of "vcc18"$lsvcc18
dataexp# Change directory to `vcc18` folder$cdvcc18# Put audio files in `data/wav/` directory$lsdata/wav/
sample1.wavsample2.wav# Create a list `data/sample.scp` of the audio files$taildata/scp/sample.scp
data/wav/sample1.wav
data/wav/sample2.wav# Extract acoustic features$qppwg-preprocess\--audiodata/scp/sample.scp\--indirwav\--outdirhdf5\--configexp/qppwg_vcc18_train_22kHz_QPPWGaf_20/config.yml# The extracted features are in `data/hdf5/`# The feature list `data/sample.list` of the feature files will be automatically generated$lsdata/hdf5/
sample1.h5sample2.h5
$lsdata/scp/
sample.scpsample.list# Synthesis$qppwg-decode\--eval_featdata/scp/sample.list\--statsdata/stats/vcc18_train_22kHz.joblib\--indirdata/hdf5/\--outdirexp/qppwg_vcc18_train_22kHz_QPPWGaf_20/wav/400000/\--checkpointexp/qppwg_vcc18_train_22kHz_QPPWGaf_20/checkpoint-400000steps.pkl# Synthesis w/ halved F0$qppwg-decode\--f0_factor0.50\--eval_featdata/scp/sample.list\--statsdata/stats/vcc18_train_22kHz.joblib\--indirdata/hdf5/\--outdirexp/qppwg_vcc18_train_22kHz_QPPWGaf_20/wav/400000/\--checkpointexp/qppwg_vcc18_train_22kHz_QPPWGaf_20/checkpoint-400000steps.pkl# The generated utterances can be found in `exp/[model]/wav/400000/`$lsexp/qppwg_vcc18_train_22kHz_QPPWGaf_20/wav/400000/
sample1.wavsample1_f0.50.wavsample2.wavsample2_f0.50.wavReferencesThe QPPWG repository is developed based on the following repositories and paper.kan-bayashi/ParallelWaveGANbigpon/QPNetk2kobayashi/sprocketr9y9/wavenet_vocoderParallel WaveGANCitationIf you find the code is helpful, please cite the following article.@article{wu2020qppwg,
title={Quasi-Periodic Parallel WaveGAN Vocoder: A Non-autoregressive Pitch-dependent   Dilated Convolution Model for Parametric Speech Generation},
author={Wu, Yi-Chiao and Hayashi, Tomoki and Okamoto, Takuma and Kawai, Hisashi and Toda, Tomoki},
journal={arXiv preprint arXiv:2005.08654},
year={2020}
}AuthorsDevelopment:
Yi-Chiao Wu @ Nagoya University (@bigpon)
E-mail:yichiao.wu@g.sp.m.is.nagoya-u.ac.jpAdvisor:
Tomoki Toda @ Nagoya University
E-mail:tomoki@icts.nagoya-u.ac.jp