Package: deepvoice3_pytorch
Deepvoice3\_pytorch===================|Build Status|PyTorch implementation of convolutional networks-based text-to-speechsynthesis models:1. `arXiv:1710.07654 <https://arxiv.org/abs/1710.07654>`__: Deep Voice3: 2000-Speaker Neural Text-to-Speech.2. `arXiv:1710.08969 <https://arxiv.org/abs/1710.08969>`__: EfficientlyTrainable Text-to-Speech System Based on Deep Convolutional Networkswith Guided Attention.Audio sampels are available athttps://r9y9.github.io/deepvoice3\_pytorch/.Highlights-----------  Convolutional sequence-to-sequence model with attention fortext-to-speech synthesis-  Multi-speaker and single speaker versions of DeepVoice3-  Audio samples and pre-trained models-  Preprocessor for `LJSpeech(en) <https://keithito.com/LJ-Speech-Dataset/>`__, `JSUT(jp) <https://sites.google.com/site/shinnosuketakamichi/publication/jsut>`__and`VCTK <http://homepages.inf.ed.ac.uk/jyamagis/page3/page58/page58.html>`__datasets-  Language-dependent frontend text processor for English and JapanesePretrained models-----------------+-----+----------+---------+----------------------------------+----------------+-------+| URL | Model    | Data    | Hyper paramters                  | Git commit     | Steps |+=====+==========+=========+==================================+================+=======+| `li | DeepVoic | LJSpeec | ``builder=deepvoice3,preset=deep | `4357976 <http | 21k ~ || nk  | e3       | h       | voice3_ljspeech``                | s://github.com |       || <ht |          |         |                                  | /r9y9/deepvoic |       || tps |          |         |                                  | e3_pytorch/tre |       || :// |          |         |                                  | e/43579764f35d |       || www |          |         |                                  | e6b8bac2b18b52 |       || .dr |          |         |                                  | a06e4e11b705b2 |       || opb |          |         |                                  | >`__           |       || ox. |          |         |                                  |                |       || com |          |         |                                  |                |       || /s/ |          |         |                                  |                |       || cs6 |          |         |                                  |                |       || d07 |          |         |                                  |                |       || 0om |          |         |                                  |                |       || my2 |          |         |                                  |                |       || lmh |          |         |                                  |                |       || /20 |          |         |                                  |                |       || 171 |          |         |                                  |                |       || 213 |          |         |                                  |                |       || _de |          |         |                                  |                |       || epv |          |         |                                  |                |       || oic |          |         |                                  |                |       || e3_ |          |         |                                  |                |       || che |          |         |                                  |                |       || ckp |          |         |                                  |                |       || oin |          |         |                                  |                |       || t_s |          |         |                                  |                |       || tep |          |         |                                  |                |       || 000 |          |         |                                  |                |       || 210 |          |         |                                  |                |       || 000 |          |         |                                  |                |       || .pt |          |         |                                  |                |       || h?d |          |         |                                  |                |       || l=0 |          |         |                                  |                |       || >`_ |          |         |                                  |                |       || _   |          |         |                                  |                |       |+-----+----------+---------+----------------------------------+----------------+-------+| `li | Nyanko   | LJSpeec | ``builder=nyanko,preset=nyanko_l | `ba59dc7 <http | 58.5k || nk  |          | h       | jspeech``                        | s://github.com |       || <ht |          |         |                                  | /r9y9/deepvoic |       || tps |          |         |                                  | e3_pytorch/tre |       || :// |          |         |                                  | e/ba59dc75374c |       || www |          |         |                                  | a3189281f60282 |       || .dr |          |         |                                  | 01c15066830116 |       || opb |          |         |                                  | >`__           |       || ox. |          |         |                                  |                |       || com |          |         |                                  |                |       || /s/ |          |         |                                  |                |       || 1y8 |          |         |                                  |                |       || bt6 |          |         |                                  |                |       || bng |          |         |                                  |                |       || gbz |          |         |                                  |                |       || zlp |          |         |                                  |                |       || /20 |          |         |                                  |                |       || 171 |          |         |                                  |                |       || 129 |          |         |                                  |                |       || _ny |          |         |                                  |                |       || ank |          |         |                                  |                |       || o_c |          |         |                                  |                |       || hec |          |         |                                  |                |       || kpo |          |         |                                  |                |       || int |          |         |                                  |                |       || _st |          |         |                                  |                |       || ep0 |          |         |                                  |                |       || 005 |          |         |                                  |                |       || 850 |          |         |                                  |                |       || 00. |          |         |                                  |                |       || pth |          |         |                                  |                |       || ?dl |          |         |                                  |                |       || =0> |          |         |                                  |                |       || `__ |          |         |                                  |                |       |+-----+----------+---------+----------------------------------+----------------+-------+| `li | Multi-sp | VCTK    | ``builder=deepvoice3_vctk,preset | `0421749 <http | 30k + || nk  | eaker    |         | =deepvoice3_vctk``               | s://github.com | 30k   || <ht | DeepVoic |         |                                  | /r9y9/deepvoic |       || tps | e3       |         |                                  | e3_pytorch/tre |       || :// |          |         |                                  | e/0421749af908 |       || www |          |         |                                  | 905d181f089f06 |       || .dr |          |         |                                  | 956fddd0982d47 |       || opb |          |         |                                  | >`__           |       || ox. |          |         |                                  |                |       || com |          |         |                                  |                |       || /s/ |          |         |                                  |                |       || uzm |          |         |                                  |                |       || tzg |          |         |                                  |                |       || ced |          |         |                                  |                |       || yu5 |          |         |                                  |                |       || 31k |          |         |                                  |                |       || /20 |          |         |                                  |                |       || 171 |          |         |                                  |                |       || 222 |          |         |                                  |                |       || _de |          |         |                                  |                |       || epv |          |         |                                  |                |       || oic |          |         |                                  |                |       || e3_ |          |         |                                  |                |       || vct |          |         |                                  |                |       || k10 |          |         |                                  |                |       || 8_c |          |         |                                  |                |       || hec |          |         |                                  |                |       || kpo |          |         |                                  |                |       || int |          |         |                                  |                |       || _st |          |         |                                  |                |       || ep0 |          |         |                                  |                |       || 003 |          |         |                                  |                |       || 000 |          |         |                                  |                |       || 00. |          |         |                                  |                |       || pth |          |         |                                  |                |       || ?dl |          |         |                                  |                |       || =0> |          |         |                                  |                |       || `__ |          |         |                                  |                |       |+-----+----------+---------+----------------------------------+----------------+-------+See "Synthesize from a checkpoint" section in the README for how togenerate speech samples. Please make sure that you are on the specificgit commit noted above.Notes on hyper parameters--------------------------  Default hyper parameters, used duringpreprocessing/training/synthesis stages, are turned for English TTSusing LJSpeech dataset. You will have to change some of parameters ifyou want to try other datasets. See ``hparams.py`` for details.-  ``builder`` specifies which model you want to use. ``deepvoice3``,``deepvoice3_multispeaker`` [1] and ``nyanko`` [2] are surpprted.-  ``presets`` represents hyper parameters known to work well forparticular dataset/model from my experiments. Before you try to findyour best parameters, I would recommend you to try those presets bysetting ``preset=${name}``. e.g., for LJSpeech, you can try either::python train.py --data-root=./data/ljspeech --checkpoint-dir=checkpoints_deepvoice3 \--hparams="builder=deepvoice3,preset=deepvoice3_ljspeech" \--log-event-path=log/deepvoice3_presetor::python train.py --data-root=./data/ljspeech --checkpoint-dir=checkpoints_nyanko \--hparams="builder=nyanko,preset=nyanko_ljspeech" \--log-event-path=log/nyanko_preset-  Hyper parameters described in DeepVoice3 paper for single speakerdidn't work for LJSpeech dataset, so I changed a few things. Adddilated convolution, more channels, more layers and add guidedattention loss, etc. See code for details. The changes are alsoapplied for multi-speaker model.-  Multiple attention layers are hard to learn. Empirically, one or two(first and last) attention layers seems enough.-  With guided attention (see https://arxiv.org/abs/1710.08969),alignments get monotonic more quickly and reliably if we use multipleattention layers. With guided attention, I can confirm five attentionlayers get monotonic, though I cannot get speech qualityimprovements.-  Binary divergence (described in https://arxiv.org/abs/1710.08969)seems stabilizes training particularly for deep (> 10 layers)networks.-  Adam with step lr decay works. However, for deeper networks, I findAdam + noam's lr scheduler is more stable.Requirements-------------  Python 3-  PyTorch >= v0.3-  TensorFlow >= v1.3-  `tensorboard-pytorch <https://github.com/lanpa/tensorboard-pytorch>`__(master)-  `nnmnkwii <https://github.com/r9y9/nnmnkwii>`__ >= v0.0.11-  `MeCab <http://taku910.github.io/mecab/>`__ (Japanese only)Installation------------Please install packages listed above first, and then::git clone https://github.com/r9y9/deepvoice3_pytorchpip install -e ".[train]"If you want Japanese text processing frontend, install additionaldependencies by:::pip install -e ".[jp]"Getting started---------------0. Download dataset~~~~~~~~~~~~~~~~~~~-  LJSpeech (en): https://keithito.com/LJ-Speech-Dataset/-  VCTK (en):http://homepages.inf.ed.ac.uk/jyamagis/page3/page58/page58.html-  JSUT (jp):https://sites.google.com/site/shinnosuketakamichi/publication/jsut1. Preprocessing~~~~~~~~~~~~~~~~Preprocessing can be done by ``preprocess.py``. Usage is:::python preprocess.py ${dataset_name} ${dataset_path} ${out_dir}Supported ``${dataset_name}``\ s for now are-  ``ljspeech`` (en, single speaker)-  ``vctk`` (en, multi-speaker)-  ``jsut`` (jp, single speaker)Suppose you will want to preprocess LJSpeech dataset and have it in``~/data/LJSpeech-1.0``, then you can preprocess data by:::python preprocess.py ljspeech ~/data/LJSpeech-1.0/ ./data/ljspeechWhen this is done, you will see extracted features (mel-spectrograms andlinear spectrograms) in ``./data/ljspeech``.2. Training~~~~~~~~~~~Basic usage of ``train.py`` is:::python train.py --data-root=${data-root} --hparams="parameters you want to override"Suppose you will want to build a DeepVoice3-style model using LJSpeechdataset with default hyper parameters, then you can train your model by:::python train.py --data-root=./data/ljspeech/ --hparams="builder=deepvoice3,preset=deepvoice3_ljspeech"Model checkpoints (.pth) and alignments (.png) are saved in``./checkpoints`` directory per 5000 steps by default.If you are building a Japaneses TTS model, then for example,::python train.py --data-root=./data/jsut --hparams="frontend=jp" --hparams="builder=deepvoice3,preset=deepvoice3_ljspeech"``frontend=jp`` tell the training script to use Japanese text processingfrontend. Default is ``en`` and uses English text processing frontend.Note that there are many hyper parameters and design choices. Some areconfigurable by ``hparams.py`` and some are hardcoded in the source(e.g., dilation factor for each convolution layer). If you find betterhyper parameters, please let me know!4. Moniter with Tensorboard~~~~~~~~~~~~~~~~~~~~~~~~~~~Logs are dumped in ``./log`` directory by default. You can monitor logsby tensorboard:::tensorboard --logdir=log5. Synthesize from a checkpoint~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~Given a list of text, ``synthesis.py`` synthesize audio signals fromtrained model. Usage is:::python synthesis.py ${checkpoint_path} ${text_list.txt} ${output_dir}Example test\_list.txt:::Generative adversarial network or variational auto-encoder.Once upon a time there was a dear little girl who was loved by every one who looked at her, but most of all by her grandmother, and there was nothing that she would not have given to the child.A text-to-speech synthesis system typically consists of multiple stages, such as a text analysis frontend, an acoustic model and an audio synthesis module.Advanced usage--------------Multi-speaker model~~~~~~~~~~~~~~~~~~~Currently VCTK is the only supported dataset for building amulti-speaker model. Since some audio samples in VCTK have long silencesthat affect performance, it's recommended to do phoneme alignment andremove silences according to `vctk\_preprocess <vctk_preprocess/>`__.Once you have phoneme alignment for each utterance, you can extractfeatures by:::python preprocess.py vctk ${your_vctk_root_path} ./data/vctkNow that you have data prepared, then you can train a multi-speakerversion of DeepVoice3 by:::python train.py --data-root=./data/vctk --checkpoint-dir=checkpoints_vctk \--hparams="preset=deepvoice3_vctk,builder=deepvoice3_multispeaker" \--log-event-path=log/deepvoice3_multispeaker_vctk_presetIf you want to reuse learned embedding from other dataset, then you cando this instead by:::python train.py --data-root=./data/vctk --checkpoint-dir=checkpoints_vctk \--hparams="preset=deepvoice3_vctk,builder=deepvoice3_multispeaker" \--log-event-path=log/deepvoice3_multispeaker_vctk_preset \--load-embedding=20171213_deepvoice3_checkpoint_step000210000.pthThis may improve training speed a bit.Speaker adaptation~~~~~~~~~~~~~~~~~~If you have very limited data, then you can consider to try fine-turnpre-trained model. For example, using pre-trained model on LJSpeech, youcan adapt it to data from VCTK speaker ``p225`` (30 mins) by thefollowing command:::python train.py --data-root=./data/vctk --checkpoint-dir=checkpoints_vctk_adaptation \--hparams="builder=deepvoice3,preset=deepvoice3_ljspeech" \--log-event-path=log/deepvoice3_vctk_adaptation \--restore-parts="20171213_deepvoice3_checkpoint_step000210000.pth"--speaker-id=0From my experience, it can get reasonable speech quality very quicklyrather than training the model from scratch.There are two important options used above:-  ``--restore-parts=<N>``: It specifies where to load model parameters.The differences from the option ``--checkpoint=<N>`` are 1)``--restore-parts=<N>`` ignores all invalid parameters, while``--checkpoint=<N>`` doesn't. 2) ``--restore-parts=<N>`` tell trainerto start from 0-step, while ``--checkpoint=<N>`` tell trainer tocontinue from last step. ``--checkpoint=<N>`` should be ok if you areusing exactly same model and continue to train, but it would beuseful if you want to customize your model architecture and takeadvantages of pre-trained model.-  ``--speaker-id=<N>``: It specifies what speaker of data is used fortraining. This should only be specified if you are usingmulti-speaker dataset. As for VCTK, speaker id is automaticallyassigned incrementally (0, 1, ..., 107) according to the``speaker_info.txt`` in the dataset.Acknowledgements----------------Part of code was adapted from the following projects:-  https://github.com/keithito/tacotron-  https://github.com/facebookresearch/fairseq-py.. |Build Status| image:: https://travis-ci.org/r9y9/deepvoice3_pytorch.svg?branch=master:target: https://travis-ci.org/r9y9/deepvoice3_pytorch