15th Dec. 2019

Tacotron-based acoustic model using phoneme alignment
for practical neural text-to-speech systems
Takuma Okamoto1, Tomoki Toda2,1, Yoshinori Shiga1, and Hisashi Kawai1!
1National Institute of Information and Communications Technology, Japan, 2Nagoya University, Japan

1. Introduction

3.Proposed method

Conventional text-to-speech (TTS) systems
Duration and acoustic pipeline models with source-ﬁlter vocoders
Widely used in practical systems but not high quality synthesis

Full-context label(Phoneme-level)

Phoneme duration
Full-context label(Phoneme-level)

Full-context label(Phoneme-level)
Full-context label(Frame-level)

Tacotron encoder

3 bidirectional
LSTM layers

Frame-level features

3 bidirectional
LSTM layers

End-to-end neural TTS systems
Sequence-to-sequence (seq2seq) model with neural vocoders
Jointly optimizing duration and acoustic models and directly converting characteror phoneme sequences to acoustic features (mel-spectrogram)State-of-the art end-to-end TTS models
Tacotron 2 with autoregressive WaveNet vocoder: Human quality synthesis
ClariNet (Deep voice 3 + parallel WaveNet): Entire end-to-end real-time neural TTS
Transformer-based TTS: Faster training than Tacotron 2
Problem of seq2seq models due to attention prediction error
Speech samples sometimes cannot be successfully synthesized
Crucial problem for practical TTS systems
Real-time, high-ﬁdelity, and stable neutral TTS systems with Tacotron structure
Introducing conventional duration models to sophisticated seq2seq acoustic models
HMM-based forced alignment can be relatively easily obtained
Conventional duration model can estimate almost accurately predict phoneme
durations

2. Seq2seq acoustic model with full-context label input
Tacotron 2 with full-context label input for pitch accent languages
Input: Full-context label (130 dims)
Input text
Tacotron encoder
Output: Mel-spectrogram (80 dims)
Real-time neural TTS with WaveGlow vocoder
Real time factor (RTF) with a GPU: 0.16
Ground-truth waveform x

Acoustic feature hSqueeze tovectors

Upsampling layer
fixa

Invertible 1×1
Wkconvolution

WaveNetAffineCoupling layer

log sj , tj

× 12xazxb

Affinetransformx′b

Full-context label(Phoneme-level)

Acoustic features(Frame-level)1 × 1 convlayer

Bidirectional
LSTM

Tacotron decoder
Linear layer

Tacotron decoder

Phoneme duration
Acoustic features(Frame-level)(a) conventional duration model

Acoustic features(Frame-level)

Acoustic features(Frame-level)(b) conventional acoustic model(c) seq2seq model using phoneme duration(Tacotron with forced attention)(d) proposed acoustic modelwith Tacotron decoder and phoneme duration

CAM

FAT

PAM

Tacotron with forced attention (FAT)
Encoded features are duplicated and redundant for decoder
FAT cannot outperform Tacotron (Y. Yasuda et al., ICASSP 2019)
Proposed acoustic model with Tacotron decoder and phoneme duration (PAM)
HMM-based forced alignment and bidirectional LSTM-based duration model
Acoustic model with bidirectional LSTM and decoder of Tacotron 2
Redundancy in FAT can be reduced

4. Experiments with WaveGlow vocoder
Simulation condition
Japanese Female corpus: 18 h
Acoustic features
Mel-spectrogram: 80 dims, 12.5 ms
Vocoder features (fo, vuv, mel-cepstrum):1 + 1 + 35 = 37 dims, 5 ms

: Analysis-synthesis: TTS (seq2seq): TTS (pipeline with predicted duration)+5 conv layer
post-net

Linearprojection

Linearprojection(A)
Neuralvocoder

Speechwaveform

Stop token

T. Okamoto et al., Interspeech 2019

This model is also unstable due to attention-based seq2seq structure(B) (C) (D) (E) (F) (G) (H)
CAM
FAT
PAM
WaveGlow
Mel-spectrogram

Original

Tacorton 2

PAM

Predicted attention

RTF with an NVIDIA Tesla V100
: STRAIGHT (analysis-synthesis): TTS (pipeline with oracle duration)

Tacotron decoder
2 LSTM
layers3 bidirectional
LSTM layers

Linear layer

Location sensitive attention

2 layer
pre-netPhoneme duration

Forced attention

Results
3 convlayers

Full-context label(Phoneme-level)

Full-context label(Frame-level)

Original

Text analyzer

Phoneme duration(I)(J) (K) (L) (M)CAM PAM
PAM
WaveGlow
WaveNetVFVF

MOS results with 15 listening subjects▼ ✁✂✄☎✭❆✞✟✠✡☛▼☞✌✍✎✏☛❆✍✭✓✞✟✠✡☛▼☞✌✍✎✏☛❚❚✍☛✔ ✕✖✔ ✕✭✏✞✟✠✡☛▼☞✌✍✎✏☛❚❚✍☛✏❆▼✭✛✞✟✠✡☛▼☞✌✍✎✏☛❚❚✍☛✏❆▼ ✭✜✛✞✭☞✞✟✠✡☛▼☞✌✍✎✏☛❚❚✍☛✆❆❚✭✆✞✟✠✡☛▼☞✌✍✎✏☛❚❚✍☛✆❆❚ ✭✜✛✞✭✡✞✟✠✡☛▼☞✌✍✎✏☛❚❚✍☛✎❆▼✭✤✞✟✠✡☛▼☞✌✍✎✏☛❚❚✍☛✎❆▼ ✭✜✛✞✭✥✞✟✠✡☛✦✆☛❆✍✭✧✞✟✠✡☛✦✆☛❚❚✍☛✏❆▼✭★✞✟✠✡☛✦✆☛❚❚✍☛✎❆▼✭✌✞✟✠✩☛✦✆☛❆✍✭▼✞✟✠✩☛✦✆☛❚❚✍☛✎❆▼❆▼ ❘❚✆❚✄✁♦✝ ❘❚✆☛✵✑✵✒✒✵✑✵✒✗✵✑✵✘✙✵✑✵✘✙✵✑✵✢✣✵✑✵✢✣✵✑✵✒✘✵✑✵✒✘✵✑✘✗✵✑✵✚✵✑✵✚✵✑✘✖
✵✑✘✖
✵✑✘✗✵✑✘✗☛✵✑✵✒✵✑✵✢✙✵✑✘✗✚✵✑✘✵✵✑✖✵☛✖✵✵✵✑✵✒✖✵✵

Real-time, high-fidelity, and stable neural TTS can be realized by PAM