High-Quality Statistical Parametric
Speech Synthesis Using
Generative Adversarial Networks
48-166605 Yuki Saito
Supervisor: Prof. Hiroshi Saruwatari©Yuki Saito, 2018/01/25

Research Field: Speech SynthesisSpeech Synthesis–Technique for synthesizing speech using computer

HelloText-To-Speech (TTS)
Text[Sagisaka et al., 1988]

SpeechHelloVoice Conversion (VC)
Inputspeech

OutputspeechApplications––

[Stylianou et al., 1988]

Speech communication assistance (e.g., speech translation)
Entertainments (e.g., singing voice conversion)

DNN-based speech synthesis* [Zen et al. 2013]–High flexibility but low speech quality

DNN: Deep Neural Network

1 /23

Speech parameterization unit
Vocoder-derived
STFT*parametersspectra Waveform

Thesis Overview

Goal(natural speech)
Chapter 2DNNs w/ovocoders[Takaki et al., 2017]

DNNs w/vocoders[Zen et al., 2013]

Chapter 4(proposed)

Chapter 3(proposed)

Generative
Adversarial Nets(GANs)

Global Variance (GV)
[Toda et al., 2007]

Overly smoothed
Naturally smoothed
Smoothness of speech parameters

STFT: Short-Term Fourier Transform2 /23

Table of Contents

Chapter 1. IntroductionChapter 2. Speech Synthesis Using DNNs
Chapter 3. Speech Synthesis Using GANs w/ Vocoders
Chapter 4. Speech Synthesis Using GANs w/o Vocoders
Chapter 5. Conclusion

3 /23

Speech Analysis and Parameter Extraction
Speechsignal

STFT amplitude
spectra(e.g., 513 dim.)

STFT
analysis

Vocoder-derived
speech params.
(e.g., 32 dim.)

Vocoder-based
parameterization[Kawahara et al., 1999]

Speech synthesis
w/o vocoders(Sec. 2.3)

Mel-cepstralcoefficients(timbre)Prosodic feats.
(pitch, hoarseness)
Speech synthesis
w/ vocoders(Sec. 2.2)

4 /23

DNN-based Speech Synthesis w/ Vocoders[Zen et al., 2013]

Linguisticfeats. 𝒙ai
Phoneme u1⋯0Speech𝒚 params.

𝒙1

Mel-cepstralcoefficients
Prosodicfeats.

⋯⋯⋯

12 0Accent 3
10𝒚1

⋯

Frame
position
etc.

𝒙𝑇

𝒚𝑇
Acoustic models
(DNN)

5 /23

DNN-based Speech Synthesis w/o Vocoders[Takaki et al., 2017]

Linguisticfeats. + F0 𝒙
ai
Phoneme u

1⋯0

Amplitude
𝒚 spectra

𝒙1

⋯⋯

⋯

12 0Accent 3
10𝒚1

⋯

Frame
position
etc.
Prosodicfeats.

𝒙𝑇

𝒚𝑇+Acoustic models
(DNN)

6 /23

Minimum Generation Error (MGE) Training Algorithm[Wu et al., 2016]

Linguisticfeats.

𝒙𝒚𝒀Generatedspeechparams.

Naturalspeechparams.

𝒚

𝐿MGE 𝒚, 𝒚
MLPG*

Acoustic models

1
𝐿MGE 𝒚, 𝒚 = 𝒚 − 𝒚 ⊤ 𝒚 − 𝒚 → Minimize
𝑇
MLPG: Maximum Likelihood Parameter Generation [Tokuda et al., 2000]

7 /23

Issue of DNN-based Speech Synthesis:
Over-smoothing of Generated Speech Parameters
MGE

23rd mel-cepstralcoefficient

NaturalNarrow
21st mel-cepstral coefficient

These distributions are significantly different...
(GV [Toda et al., 2007] explicitly compensates the 2nd moment.)
8 /23

Table of Contents

Chapter 1. Introduction
Chapter 2. Speech Synthesis Using DNNs
Chapter 3. Speech Synthesis Using GANs w/ Vocoders
Chapter 4. Speech Synthesis Using GANs w/o Vocoders
Chapter 5. Conclusion

9 /23

Generative Adversarial Nets (GANs) [Goodfellow et al., 2014]
Latentvariable 𝒛

True sample 𝒚𝒚 Fake sample

GAN𝐿D

or

Generator
Prior(e.g., 𝑁 𝟎, 𝑰 )Discriminator𝒚, 𝒚
1: true0: fake𝐷 ⋅Loss to recognize
true sample as trueGAN

𝐿DGAN

𝐿D

GAN

𝒚, 𝒚 = 𝐿D,1

Loss to recognize
fake sample as fake
GAN𝒚 + 𝐿D,0𝒚→ Minimize𝒚, 𝒚 is equivalent to the cross-entropy function.
10 /23

Generative Adversarial Nets (GANs) [Goodfellow et al., 2014]
Latentvariable 𝒛𝒚 Fake sample

GAN

𝐿ADV

Generator
Prior(e.g., 𝑁 𝟎, 𝑰 )𝒚

1: true
Discriminator

𝐷 ⋅Loss to recognize
fake sample as true
GAN

𝐿ADV

GAN

𝒚 = 𝐿D,1𝒚→ Minimize

Minimize approx. JS* divergence betw. dists. of 𝒚 and 𝒚.
JS: Jensen-Shannon

11 /23

Proposed Method:
Acoustic Model Training Using GANs
Linguisticfeats. 𝒙𝒚𝒀

GeneratedNatural𝒚

𝐿MGE 𝒚, 𝒚
MLPG
GAN

𝐿ADV

Acoustic models(generator)

𝒚1: naturalDiscriminator

𝐷 ⋅Loss to recognize
generated params. as natural

𝐸𝐿MGE GAN
𝐿G 𝒚, 𝒚 = 𝐿MGE 𝒚, 𝒚 + 𝜔D
𝐿ADV 𝒚 → Minimize
𝐸𝐿ADV𝜔D : weight, 𝐸𝐿∗ : expectation values of 𝐿∗
12 /23

Distributions of Speech Parameters
GANs = minimizing divergence betw. two distributions

23rd mel-cepstralcoefficient

NaturalMGE

ProposedNarrow
21st mel-cepstral coefficient

Wide asnatural speech

The proposed algorithm alleviates the over-smoothing effect!
13 /23

Discussions
Compensating for distribution differences–

Integrating voice anti-spoofing techniques–



Features that are effective for detecting synthetic speech
can be used (Sec. 3.4.8).

Changing a divergence to be minimized–

The proposed method generalizes the conventional
methods such as the GV.

Earth mover’s distance (Wasserstein GAN [Arjovsky et al., 2017])
was the best for improving synthetic speech quality (Sec.
3.4.10).

Applying various speech synthesis–

Not only TTS (next slides), but also VC (Sec. 3.5).

14 /23

Experimental Conditions

Train / evaluate data

450 sentences / 53 sentences (16 kHz sampling)

Linguistic feats.

442-dimensional vector

Speech params.

Mel-cepstral coefficients and prosodic features

Optimizer

AdaGrad [Duchi et al., 2011]

Acoustic models

Feed-Forward 442 – 3x512 (ReLU) – 94 (linear)

DiscriminatorFeed-Forward 26 – 3x256 (ReLU) – 1 (sigmoid)

Weight 𝜔D

1.0 (Secs. 3.4.2 and 3.4.4)

Methods

MGE [Wu et al., 2016], GV [Toda et al., 2007], Proposed15 /23

Subjective Evaluationsin Terms of Speech QualityPreference AB test (select better sounded speech)
MGE(a) MGE vs. Proposed

Proposed
Improved

MGEGV(b) GV vs. Proposed

Proposed

Proposed
Improved

GV
0.0

0.2

0.4


0.6

0.8

1.0

Preference score

Proposed method improves synthetic speech quality!
Error bars denote 95% confidence intervals.

16 /23

Table of Contents










Chapter 1. Introduction
Chapter 2. Speech Synthesis Using DNNs
Chapter 3. Speech Synthesis Using GANs w/ Vocoders
Chapter 4. Speech Synthesis Using GANs w/o Vocoders
Chapter 5. Conclusion

17 /23Issue in Speech Synthesis w/o Vocoders
Over-smoothing of generated STFT amplitude spectra––Formants (spectral peaks) tend to be weakened.
The method proposed in Chap. 3 cannot be applied directly.
•To deal with the issue...
–Dimensionality reduction retaining spectral structures
Natural amplitudes

Frequency(e.g., 513 bins)

Difficulties in modeling highly complex distribution?
Time18 /23

Acoustic Model Training Using
Low-resolution GANs
Linguisticfeats. + F0 𝒙

𝒚 Generated

Acoustic models𝒚L𝐿G𝝓 ⋅𝒚, 𝒚 = 𝐿MGE 𝒚, 𝒚Natural 𝒚

𝐿MGE 𝒚, 𝒚𝒚L

Averagepooling𝝓 ⋅GAN

𝐿ADV𝒚L

1: natural𝒚L

Low-resolution
discriminator

L 𝐸𝐿MGE GAN+ 𝜔D
𝐿ADV𝐸𝐿ADV𝐷L ⋅𝒚L→ Minimize

19 /23

Examples of
Natural and Generated Amplitude Spectra
NaturalProposed (Chap. 3)

MGE

Proposed (Chap. 4)

Low-resolution GANs capture differences in formants!
20 /23

Subjective Evaluationsin Terms of Speech Quality
Preference AB test (select better sounded speech)
MGE(a) MGE vs. Proposed (Chap. 3)

Proposed(Chap. 3)

Degraded
Proposed(Chap. 3)

MGE(b) MGE vs. Proposed (Chap. 4)

Proposed(Chap. 4)
MGE
0.0

Proposed(Chap. 4)

Improved
0.2

0.4


0.6

0.8

1.0

Preference score

Low-resolution GAN improves synthetic speech quality!
Error bars denote 95% confidence intervals.

21 /23

Table of Contents









Chapter 1. Introduction
Chapter 2. Speech Synthesis Using DNNs
Chapter 3. Speech Synthesis Using GANs w/ Vocoders
Chapter 4. Speech Synthesis Using GANs w/o Vocoders
Chapter 5. Conclusion

22 /23

ConclusionPurpose: improving synthetic speech quality of SPSS
Proposed: acoustic model training algorithms using GANs
They compensate for the distribution differences betw.
natural / generated speech parameters.

•Results–The proposed algorithms improved synthetic speech quality
compared to conventional methods.

Future works––

Investigating anti-spoofing techniques
Further improving speech quality using STFT spectra

23 /23