Package: voice100
Voice100Voice100 includes neural TTS/ASR models. Inference of Voice100
is low cost as its models are tiny and only depend on CNN
without recursion.ObjectivesDon't depend on non-commercially licensed datasetSmall enough to run on normal PCs, Raspberry Pi and smartphones.Sample synthesisSample synthesis 1beginnings are apt to be determinative and when reinforced by continuous applications of similar influenceSample synthesis 2which had restored the courage of noirtier for ever since he had conversed with the priest his violent
despair had yielded to a calm resignation which surprised all who knew his excessive affectionSample synthesis 1また、東寺のように五大明王と呼ばれる主要な明王の中央に配されることも多い。Sample synthesis 2ニューイングランド風は牛乳をベースとした白いクリームスープでありボストンクラムチャウダーとも呼ばれるArchitectureTTSTTS model is devided into two sub models, align model and audio model.
The align model predicts text alignments given a text. An aligned text
is generated from the text and the text alignments. The audio model predictsWORLDfeatures (F0, spectral envelope, coded aperiodicity) given
the aligned text.Alignment networkgraph TD
    A[Input text] -->|hello| B(Embedding)
    B --> C(1D inverted residual x4)
    C --> D(Convolution)
    D -->|h:0,1 e:0,2 l:1,1 l:1,1 o:1,2| E[Alignment]Audio networkgraph TD
    A[Aligned text] -->|_hee_l_l_oo| B(Embedding)
    B --> C(1D inverted residual x4)
    C --> D(1D transpose convolution)
    D --> E(1D inverted residual x3)
    E --> F(Convolution)
    F --> G[WORLD parameters]TTS align model| Name      | Type       | Params
-----------------------------------------
0 | embedding | Embedding  | 14.8 K
1 | layers    | Sequential | 8.6 M 
-----------------------------------------
8.6 M     Trainable params
0         Non-trainable params
8.6 M     Total params
17.137    Total estimated model params size (MB)TTS audio model| Name      | Type         | Params
-------------------------------------------
0 | embedding | Embedding    | 14.8 K
1 | decoder   | VoiceDecoder | 11.0 M
2 | norm      | WORLDNorm    | 518   
3 | criterion | WORLDLoss    | 0     
-------------------------------------------
11.1 M    Trainable params
518       Non-trainable params
11.1 M    Total params
22.120    Total estimated model params size (MB)Align model pre-processingThe input of the align model is sequence of tokens of the input text.
The input text is lower cased and tokenized
into characters and encoded by the text encoder. The text encoder
has 28 characters in the vocabulary, which includes lower alphabets,
a space and an apostrophy. All characters which are not found in the
vocabulary, are removed.Align model post-processingThe output of the align model is sequence of pairs of timings which
length is the same as the number of input tokens. A pair has two values,
number of frames before the token and number of frames for the token.
One frame is 20ms. An aligned text is generated from the input text and
pairs of timings. The length of the aligned text is the number of total
frames for the audio.Audio model pre-processing.The input of the audio model is the encoded aligned text, which is
encoded in the same way as the align model pre-processing, except it
has one added token in the vocabulary for spacing between tokens for
the original text.Audio model post-processing.The output of the audio model is the sequence of F0, F0 existences,
log spectral envelope, coded aperiodicity.
A F0 existence is a boolean value, which is true when F0 is available
false otherwise. F0 is forced into 0 when F0 existence is false.
One frame is 10ms. The length of the output is twice as the length
of the input.ASRThe ASR model is 9-layer MobileNet-like inverted residual which is
trained to predict onCTC loss.ASR networkgraph TD
    A[Mel spectrogram] --> B(1D inverted residual x 12)
    B --> C(Convolution)
    C --> G[Logits of aligned text]| Name          | Type                          | Params
----------------------------------------------------------------
0 | encoder       | ConvVoiceEncoder              | 11.6 M
1 | decoder       | LinearCharDecoder             | 14.9 K
2 | loss_fn       | CTCLoss                       | 0     
3 | batch_augment | BatchSpectrogramAugumentation | 0     
----------------------------------------------------------------
11.6 M    Trainable params
0         Non-trainable params
11.6 M    Total params
23.243    Total estimated model params size (MB)Align modelThe align model is 2-layer bi-directional LSTM which is trained to predict
aligned texts from MFCC audio features. The align model is used to
prepare aligned texts for dataset to train the TTS models.| Name          | Type                          | Params
----------------------------------------------------------------
0 | conv          | Conv1d                        | 24.7 K
1 | lstm          | LSTM                          | 659 K 
2 | dense         | Linear                        | 7.5 K 
3 | loss_fn       | CTCLoss                       | 0     
4 | batch_augment | BatchSpectrogramAugumentation | 0     
----------------------------------------------------------------
691 K     Trainable params
0         Non-trainable params
691 K     Total params
1.383     Total estimated model params size (MB)TrainingTrain ASR modelvoice100-prepare-dataset\--datasetljspeech\--languageen\--use_phone

voice100-prepare-dataset\--datasetlibrispeech\--languageen\--use_phone

voice100fit\--configconfig/asr_en_phone_base.yaml\--trainer.acceleratorgpu\--trainer.devices1\--trainer.precision16\--trainer.default_root_dir./outputs/asr_en_phone_base\Align text with small ASR modelThis generates the aligned text asdata/${DATASET}-phone-align.txt.voice100-align-text\--batch_size4\--datasetljspeech\--languageen\--use_phone\--checkpointasr_en_phone_small-20230309.ckptTrain TTS align modelvoice100fit--configvoice100/config/align_en_phone_base.yaml\--trainer.acceleratorgpu\--trainer.devices1\--trainer.precision16\--trainer.default_root_dir=./outputs/align_en_phone_baseCompute audio statisticsThis generates the statistics asdata/${DATASET}-stat.pt.voice100-calc-stat\--datasetljspeech\--languageen\--outputdata/audio-stat.pyTrain TTS audio modelvoice100fit--configvoice100/config/tts_en_phone_base.yaml\--trainer.acceleratorgpu\--gpus1\--precision16\--trainer.default_root_dir=./outputs/tts_en_phone_baseExporting to ONNXvoice100-export-onnx\--checkpointmodel/${MODEL}/lightning_logs/version_0/checkpoints/last.ckptCMU modelsCMU models is a model that use the output ofG2p_enas text representation instead of raw text.Training CMU modelsThese commands convert texts in the dataset into./data/[dataset]-phone-[split].txt.
Then runvoice100-train-[model]with--use-phone.voice100-prepare-dataset\--datasetljspeech
voice100-prepare-dataset\--datasetlibrispeech\--splittrain
voice100-prepare-dataset\--datasetlibrispeech\--splitvalCMU multi-task modelCMU multitask model is a variant of TTS audio model which input is an aligned text and outputs are
WORLD vocoder parameters and CMU phonemes. To train CMU multi-task model, we need alignment data
for English and CMU phonemes../data/ljspeech-align-train.txt./data/ljspeech-phone-align-train.txtThen runMODEL=ttsaudio_en_mt_conv_base

voice100-train-ttsaudio-mt\--gpus1\--dataset${DATASET}\--language${LANGUAGE}\--batch_size32\--precision16\--max_epochs150\--default_root_dir./model/${MODEL}InferenceUseVoice100 runtimeand exported ONNX files.Pretrained modelsNameModel ClassDatasetDownloadasr_en_small-20230225AudioToAlignTextLibriSpeech, LJ Speech 1.1downloadasr_en_base-20230319AudioToAlignTextLibriSpeech, LJ Speech 1.1downloadasr_en_phone_small-20230309AudioToAlignTextLibriSpeech, LJ Speech 1.1downloadasr_en_phone_base-20230314AudioToAlignTextLibriSpeech, LJ Speech 1.1downloadasr_ja_phone_small-20230104AudioToAlignTextCommon Voice 12.0 jadownloadasr_ja_phone_base-20230104AudioToAlignTextCommon Voice 12.0 jadownloadalign_en_base-20230401TextToAlignTextLJ Speech 1.1downloadtts_en_base-20230407AlignTextToAudioLJ Speech 1.1downloadalign_en_phone_base-20230407TextToAlignTextLJ Speech 1.1downloadtts_en_phone_base-20230401AlignTextToAudioLJ Speech 1.1downloadalign_ja_phone_base-20230203TextToAlignTextKokoro Speech v1.2 largedownloadtts_ja_phone_base-20230204AlignTextToAudioKokoro Speech v1.2 largedownloadasr_en_base-20210628 (deprecated)AudioAlignCTCLJ Speech 1.1downloadalign_en_lstm_base_ctc-20210628 (deprecated)AudioAlignCTCLJ Speech 1.1downloadalign_en_phone_lstm_base_ctc-20220103 (deprecated)AudioAlignCTCLJ Speech 1.1downloadalign_ja_lstm_base_ctc-20211116  (deprecated)AudioAlignCTCKokoro Speech v1.1 smalldownloadalign_ja_phone_lstm_base_ctc-20221230 (deprecated)AudioAlignCTCKokoro Speech v1.1 smalldownloadttsalign_en_conv_base-20220409 (deprecated)TextToAlignTextModelLJ Speech 1.1downloadttsalign_en_phone_conv_base-20220409 (deprecated)TextToAlignTextModelLJ Speech 1.1downloadttsalign_ja_conv_base-20220411 (deprecated)TextToAlignTextModelKokoro Speech v1.1 smalldownloadttsaudio_en_conv_base-20220107 (deprecated)AlignTextToAudioModelKokoro Speech v1.1 smalldownloadttsaudio_en_phone_conv_base-20220105 (deprecated)AlignTextToAudioModelLJ Speech 1.1downloadttsaudio_ja_conv_base-20220416 (deprecated)AlignTextToAudioModelKokoro Speech v1.1 smalldownloadttsaudio_en_mt_conv_base-20220316 (deprecated)AlignTextToAudioMultiTaskModelLJ Speech 1.1downloadasr_en_conv_base_ctc-20220126 (deprecated)AudioToTextCTCLibriSpeechdownloadasr_en_phone_conv_base_ctc-20220107 (deprecated)AudioToTextCTCLibriSpeechdownloadstt_ja_conv_base_ctc-20211127 (deprecated)AudioToTextCTCCommon Voice 6.1 jadownloadasr_ja_phone_conv_base_ctc-20221225 (deprecated)AudioToTextCTCCommon Voice 6.1 jadownload