博士論文

Statistical Speech Synthesis Based on
Human’s Speech Information Processing Abilities（人間の音声情報処理能力に基づく統計的音声合成）

Yuki Saito
齋藤 佑樹

c 2021, Yuki Saito.
Copyright ⃝人間の音声情報処理能力に基づく統計的音声合成

概要
音声合成とは，コンピュータを用いて音声を人工的に生成・変換・加工する技術である．特に，テキストから音声を生成する技術をテキスト音声合成といい，入力された音声の言語情報を保持しつつ，非言語・パラ言語情報を変換する技術を音声変換という．音声合成は，人間とロボットの間での音声コミュニケーションを仲介するマンマシンインターフェースのみならず，音声バーチャルリアリティやエンターテインメント応用などを通じて，物理的制約を超えた音声表現を実現する技術である．本論文ではこれらの背景を踏まえ，多様な話者の音声を高品質に合成でき，かつ，合成された音声の話者性を直感的に制御可能な音声合成技術の確立を目指す．
本論文の主題である統計的音声合成は，統計的機械学習の枠組みに基づき，入力特徴量から合成音声パラメータを生成する音響モデルを学習する手法である．特に，音響モデルとして deep neural network (DNN) を用いる DNN 音声合成は，計算機性能の向上や大規模音声
データベースの公開に恩恵を受け，入力特徴量から音声パラメータへの複雑な写像を学習可能な手法として広く研究されている．しかしながら，従来技術では，DNN 学習時の統計処理に起因する合成音声パラメータの過剰な平滑化により，合成音声の品質が著しく劣化する．また，合成可能な話者は DNN 学習時に用いられたものに限定されるため，合成音声の話者性の多様性を高めることは困難である．さらに，合成音声の話者性を制御するために用いる従来の話者表現は，人間の主観的な話者知覚を無視しているため，多様な話者性の音声を合成可能な

DNN 音声合成には不適切である．
本論文では，これらの問題点を解消するために，(1) 音声敵対過程を統合した高品質な音声合成法，(2) 音声認識過程を統合した多様な音声合成法，そして (3) 人間の音声知覚を導入した解釈性の高い話者表現学習法を提案する．これらの提案法は，音声合成を改善するために，人間の音声情報処理能力（敵対過程，認識過程，知覚過程）を活用するという着想に基づく．

(1) 音声敵対過程を統合した高品質な音声合成法では，generative adversarial network(GAN) を音声合成に導入し，高品質な音声を合成可能な DNN 音響モデルを学習する．この提案法は，人間の自然音声と合成音声を識別する能力（即ち，音声合成に対する敵対過程）を活用して DNN 音響モデルを学習するという着想に基づく．この提案法では，DNN 音響モデ
ルと，自然音声と合成音声を識別する discriminator を交互に学習する．GAN の学習の目的
関数は，生成データ分布と実データ分布の擬距離最小化とみなされるため，この提案法は，過ii

剰な平滑化を定量化するパラメトリックな統計的差異（音声パラメータ分布の 2 次モーメン
トなど）の補償により合成音声の品質を改善する従来法の理論拡張と解釈できる．また，この提案法における discriminator は，音声合成技術の悪用による音声なりすましを防ぐための

anti-spooﬁng として解釈できる．したがって，anti-spooﬁng の知見を導入することで，より効率的な DNN 音響モデル学習が実現できる．本論文では，ボコーダ分析合成に基づくテキス
ト音声合成および音声変換のための GAN に基づく音声合成法を提案し，その有効性を評価する．その後，短時間フーリエ変換スペクトルを用いたより先進的なテキスト音声合成に向けてこの提案法を拡張する．実験的評価の結果から，この提案法によって合成音声の品質が有意に改善することを示す．

(2) 音声認識過程を統合した多様な音声合成法では，音声から発話内容と発話者を認識する
DNN を導入し，高品質かつ多様な話者性を持つ音声を合成可能な DNN 音響モデルを学習す
る．この提案法は，人間の音声認識と話者認識（即ち，音声合成の逆過程）の能力を活用してDNN 音響モデルを学習するという着想に基づく．この提案法では，発話内容が明瞭で高品質な音声を合成するために，音声認識を用いて DNN 音響モデルを学習する．さらに，多様な話者性を持つ音声を合成するために，話者認識由来の連続的な話者表現を用いる．本論文では，
変分オートエンコーダを用いた音声変換においてこの提案法の有効性を評価する．実験的評価の結果から，この提案法によって変換音声の有意な品質改善を達成しつつ，DNN 学習に用いていない未知話者の音声も合成可能になることを示す．

(3) 人間の音声知覚を導入した解釈性の高い話者表現学習法では，合成音声の話者性を直感的に制御可能な音声合成を実現するために，人間の主観的な音声知覚を導入して話者表現を学習する．この提案法は，人間の知覚を計算資源とみなし，解釈しやすい話者表現の学習に活用するという着想に基づく．この提案法では，まず，多数の評価者（音声の聞き手）による大規模主観評価により，多数話者間の知覚的な類似度のスコアを収集する．その後，このスコアを利用して，聞き手の話者知覚を高精度に再現する話者表現を学習する．このように学習された
話者表現は，聞き手の話者知覚を強く反映するため，合成音声の話者性をより直感的に制御できるのみならず，未知話者の音声合成時の品質劣化に対する頑健性の向上が期待できる．本論
文では，この提案法の有効性を (2) 音声認識過程を統合した多様な音声合成法において評価す
る．実験的評価の結果から，この提案法によって学習された話者表現の導入により，合成音声の品質と制御性が向上することを示す．

Abstract
Speech synthesis involves using a computer to synthesize, convert, and retouch humanspeech artiﬁcially. Text-to-speech (TTS) is a technique for synthesizing speech from text.
Voice conversion (VC) is a technique for converting non-/para-linguistic information of
input speech while retaining its linguistic information. These techniques provide a manmachine interface for mediating speech communication between humans and robots andenable voice expressions beyond human physical constraints through virtual reality andentertainment applications. This thesis aims to establish high-quality and intuitively
controllable speech synthesis technology that can synthesize diverse speakers’ voices.
Statistical speech synthesis, the subject of this thesis, trains an acoustic model thatgenerates speech parameters from input features. Especially, deep neural network (DNN)based speech synthesis has been widely studied because it can learn the complicated mapping from input features to output speech parameters beneﬁting from the improvementof computer performance and availability of large speech corpora. However, conventional
DNN-based speech synthesis suﬀers from degrading synthetic speech quality caused byover-smoothing of generated speech parameters. It also has diﬃculty increasing speaker
diversity in synthetic speech because the speakers’ voices can be synthesized are limited to the speakers seen during the DNN training. Furthermore, conventional speakerrepresentations for controlling speaker individuality in synthetic speech do not consider
human subjective speaker perception. The use of such representations can worsen thecontrollability of DNN-based multi-speaker speech synthesis.
This thesis proposes three methods to overcome the above-described issues: 1) highquality speech synthesis integrating a speech adversary process, 2) versatile speech synthesis integrating a speech recognition process, and 3) interpretable speaker representation
learning introducing human speech perception. The core idea is to use human’s speech
information processing abilities for improving speech synthesis.
1) High-quality speech synthesis integrating a speech adversary process: This method
introduces generative adversarial networks (GANs) to DNN-based speech synthesis for
improving synthetic speech quality. The core idea is to use the human’s adversarial ability that discriminates natural speech from synthetic speech. Two DNNs are alternatively
trained with this method: an acoustic model and discriminator that distinguishes natural and synthetic speech. This GAN-like training leads to divergence (i.e., statistical
diﬀerence) minimization between natural and generated speech parameters. Therefore,

ii

this method can be interpreted as a theoretical extension of conventional methods forovercoming over-smoothing by reproducing parametric statistics of natural speech (e.g.,second moments of speech parameter distribution). From a diﬀerent perspective, the discriminator with this method can be regarded as anti-spooﬁng that prevents voice spooﬁng
attacks. Accordingly, the anti-spooﬁng techniques can be introduced for eﬃcient GANbased acoustic model training. The eﬀectiveness of this method is ﬁrst evaluated for
traditional TTS and VC using vocoder parameters. This method is then extended toadvanced TTS using short-term Fourier transform spectra. The results indicate that thismethod signiﬁcantly improves synthetic speech quality.
2) Versatile speech synthesis integrating a speech recognition process: This method
involves integrating speech recognition and speaker classiﬁcation into speech synthesis forincreasing speaker diversity in synthetic speech while achieving higher quality. The coreidea is to use the human’s recognition ability that recognizes phonetic contents and speaker
information of speech accurately. This method trains a DNN-based acoustic model using
speech recognition. As a result, synthetic speech’s phonetic content can be clariﬁed, andhigh-quality speech can be synthesized. It also uses speaker-classiﬁcation-derived speakerrepresentations to diversify speaker individuality in synthetic speech. The eﬀectiveness of
this method is evaluated in VC using variational autoencoders. The results indicate that
this method achieves signiﬁcant improvement in converted speech quality and synthesizes
unseen speakers’ voices.
3) Interpretable speaker representation learning introducing human speech perception:
This method involves introducing human subjective speech perception to learn speakerrepresentations that enable intuitively controllable speech synthesis. The core idea is toregard the human’s perception as computational resources for learning the interpretablespeaker representations. The perceptual similarity scores among multiple speakers’ voices
are ﬁrst collected with this method through a large-scale subjective evaluation involving
many listeners. The scores are then used to learn speaker representations that consider thelisteners’ subjective speaker perception. Since the learned speaker representations strongly
reﬂect listeners’ perception, they are expected to enable highly controllable DNN-based
multi-speaker speech synthesis. Such representations can also enhance the robustness
against the quality degradation of unseen speakers’ synthetic speech. The eﬀectivenessof this method is evaluated with the second proposed method in this thesis. The results
indicate that this method improves the quality and controllability of synthetic speech.

iiiContents
Chapter 1

Introduction

1

1.1

Background . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

1

1.2

Thesis Scope . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

4

1.2.1

Issues to Be Addressed . . . . . . . . . . . . . . . . . . . .

4

1.2.2

Methods Proposed in This Thesis . . . . . . . . . . . . . .

6Thesis Outline . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

8Statistical Speech Synthesis Using DNNs

11

2.1

Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

11

2.2

Feature Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

12

1.3
Chapter 2

2.3

2.4

2.5

2.2.1

Speech Analysis . . . . . . . . . . . . . . . . . . . . . . . .

12

2.2.2

Text Analysis . . . . . . . . . . . . . . . . . . . . . . . . . .

15

2.2.3

Temporal Alignment . . . . . . . . . . . . . . . . . . . . . .

15

Acoustic Modeling . . . . . . . . . . . . . . . . . . . . . . . . . . . .

15

2.3.1

General Purpose . . . . . . . . . . . . . . . . . . . . . . . .

15

2.3.2

Static-dynamic Feature Modeling . . . . . . . . . . . . . . .

16

2.3.3

DNN Architectures for Acoustic Modeling . . . . . . . . . .

17

2.3.4

DNN-based TTS . . . . . . . . . . . . . . . . . . . . . . . .

21

2.3.5

DNN-based VC . . . . . . . . . . . . . . . . . . . . . . . . .

23

2.3.6

Multi-speaker Acoustic Modeling Using Speaker Codes

. .

23

2.3.7

Loss Functions for DNN Training . . . . . . . . . . . . . . .

24

Speech Parameter Generation . . . . . . . . . . . . . . . . . . . . . .

25

2.4.1

MLPG algorithm . . . . . . . . . . . . . . . . . . . . . . . .

25

2.4.2

GV Compensation . . . . . . . . . . . . . . . . . . . . . . .

26

Speech Waveform Synthesis . . . . . . . . . . . . . . . . . . . . . . .

26

2.5.1

Vocoder-based Synthesis . . . . . . . . . . . . . . . . . . . .

27

2.5.2

Vocoder-free Synthesis . . . . . . . . . . . . . . . . . . . . .

27

iv

Contents

2.6

Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

29

Vocoder-based Statistical Speech Synthesis Using GANs

30

3.1

Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

30

3.2

GANs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

32

Chapter 3

3.3

3.4

3.5

3.2.1

Objective of GANs . . . . . . . . . . . . . . . . . . . . . . .

32

3.2.2

Discriminator Training. . . . . . . . . . . . . . . . . . . .

33

3.2.3

Generator Training . . . . . . . . . . . . . . . . . . . . . . .

33

Acoustic Model Training Using GANs . . . . . . . . . . . . . . . . .

34

3.3.1

Acoustic Model Training Criteria Incorporating GANs . . .

34

3.3.2

Integrating Anti-spooﬁng Techniques . . . . . . . . . . . . .

35

3.3.3

Duration Model Training Considering Isochrony . . . . . .

36

3.3.4

Various Divergences Minimized by GANs . . . . . . . . . .

38

3.3.5

Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . .

41

Experimental Evaluations for TTS . . . . . . . . . . . . . . . . . . .

45

3.4.1

Conditions for TTS Evaluation . . . . . . . . . . . . . . . .

45

3.4.2

Objective Evaluation of Spectral Parameter Generation . .

46

3.4.3

Investigation of Convergence . . . . . . . . . . . . . . . . .

47

3.4.4

Subjective Evaluation of Spectral Parameter Generation . .

48

3.4.5

Subjective Evaluation of F0 Generation . . . . . . . . . . .

50

3.4.6

Subjective Evaluation of Duration Generation . . . . . . .

51

3.4.7

Comparison to GV Compensation . . . . . . . . . . . . . .

52

3.4.8

Eﬀect of Feature Function . . . . . . . . . . . . . . . . . . .

52

3.4.9

Subjective Evaluation Using Richer DNN Architecture . . .

53

3.4.10 Eﬀect of Divergence to Be Minimized by GANs . . . . . . .

54

Experimental Evaluations for VC . . . . . . . . . . . . . . . . . . . .

55

3.5.1

Conditions for VC Evaluation . . . . . . . . . . . . . . . . .

55

3.5.2

Subjective Evaluation Using Speech Parameter Conversion

56

3.5.3

Subjective Evaluation Using Spectral Diﬀerentials . . . . .

57

3.5.4

Evaluation in Many-to-one VC . . . . . . . . . . . . . . . .

58

Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

58

Vocoder-free Statistical Speech Synthesis Using GANs

60

4.1

Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

60

4.2

GAN-based Acoustic Model Training Using Low-/multi-frequency-

3.6
Chapter 4

resolution Amplitude Spectra . . . . . . . . . . . . . . . . . . . . . .

62

Contents

4.2.1

4.4
Chapter 5

GAN-based Training Algorithm Using Low-frequencyresolution Amplitude Spectra . . . . . . . . . . . . . . . . .

62

4.2.2

Frequency Scale Conversion Using Frequency Warping . . .

64

4.2.3

GAN-based Training Algorithm Using Multi-frequencyresolution Amplitude Spectra . . . . . . . . . . . . . . . . .

65Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . .

66

Experimental Evaluations . . . . . . . . . . . . . . . . . . . . . . . .

67

4.2.4
4.3

v

4.3.1

Experimental Conditions . . . . . . . . . . . . . . . . . . .

67

4.3.2

Objective Evaluations . . . . . . . . . . . . . . . . . . . . .

68

4.3.3

Subjective Evaluations . . . . . . . . . . . . . . . . . . . . .

70

Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

79

VAE-based Multi-speaker Acoustic Modeling Using Speech
Recognition Process

80

5.1

Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

80

5.2

VAE-based Multi-speaker Acoustic Model Using Phonetic and
Speaker Latent Variables . . . . . . . . . . . . . . . . . . . . . . . .
5.2.1

Phonetic Latent Variable to Improve Synthetic Speech Quality 82

5.2.2

Speaker Latent Variable to Increase Speaker Diversity . . .

5.2.3

Non-parallel and Many-to-many VC Using Proposed VAE-

5.4
Chapter 6

84

based Acoustic Model . . . . . . . . . . . . . . . . . . . . .

86Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . .

86

Experimental Evaluation . . . . . . . . . . . . . . . . . . . . . . . .

87

5.2.4
5.3

82

5.3.1

Experimental Conditions . . . . . . . . . . . . . . . . . . .

87

5.3.2

Objective Evaluation. . . . . . . . . . . . . . . . . . . . .

88

5.3.3

Subjective Evaluation . . . . . . . . . . . . . . . . . . . . .

89

5.3.4

Eﬀects of Training Data and d-vector Dimensionality . . .

91

Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

96

Perceptual-similarity-aware Deep Speaker Representation
Learning

98

6.1

Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

98

6.2

Deep Speaker Representation Learning Considering Perceptual
Speaker-pair Similarity . . . . . . . . . . . . . . . . . . . . . . . . .

100

6.2.1

101

Perceptual Speaker Similarity Matrix . . . . . . . . . . . .

viContents

6.2.2

Perceptual-similarity-aware Deep Speaker Representation
Learning Algorithms . . . . . . . . . . . . . . . . . . . . . .

6.2.3

Active Learning of Perceptual-similarity-aware Speaker Representations . . . . . . . . . . . . . . . . . . . . . . . . . . .

104

Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . .

106

Experimental Evaluation . . . . . . . . . . . . . . . . . . . . . . . .

107

6.2.4
6.3

101

6.3.1

Experimental Conditions . . . . . . . . . . . . . . . . . . .

107

6.3.2

Analysis of Perceptual Similarity Scores . . . . . . . . . . .

109

6.3.3

Evaluation in Deep Speaker Representation Learning

. . .

109

6.3.4

Evaluation of Active Learning . . . . . . . . . . . . . . . .

117

6.3.5

Visualization of Speaker Space . . . . . . . . . . . . . . . .

118

Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

119

Conclusion

121

7.1

Summary of This Thesis . . . . . . . . . . . . . . . . . . . . . . . . .

121

7.2

Future Directions . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

123

6.4
Chapter 7

7.2.1

Improving Synthetic Speech Quality Further . . . . . . . .

123

7.2.2

Implementing with Real-time Speech Synthesis . . . . . . .

123

7.2.3

Extending to Multilingual Speech Synthesis . . . . . . . . .

123

7.2.4

Modeling Broader Speech Perception . . . . . . . . . . . . .

123

Publications and Research Activities

125

Bibliography

136

Acknowledgements

151

Appendix A VC Using Input-to-output Highway Networks

153

A.1

Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

153

A.2

Proposed VC Method . . . . . . . . . . . . . . . . . . . . . . . . . .

153

A.2.1 Input-to-Output Highway Networks for VC . . . . . . . . .

153

A.2.2 Discussions . . . . . . . . . . . . . . . . . . . . . . . . . . .

154

Experimental Evaluation . . . . . . . . . . . . . . . . . . . . . . . .

157

A.3.1 Experimental Conditions . . . . . . . . . . . . . . . . . . .

157

A.3.2 Subjective Evaluation . . . . . . . . . . . . . . . . . . . . .

158

A.3

Appendix B Joint Adversarial Training Algorithm for DNN-based Many-

Contentsvii

to-one VC

160

B.1

Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

160

B.2

Conventional Algorithm for DNN-based Many-to-one VC . . . . . .

160

B.2.1 Speech Recognition Model Training . . . . . . . . . . . . .

161

B.2.2 Speech Synthesis Model Training . . . . . . . . . . . . . . .

161

B.2.3 Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . .

162

B.3

Proposed Algorithm for DNN-based Many-to-one VC Using Adversarial Training . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

163

B.3.1 Joint Adversarial Training of Speech Recognition and Syn-

B.4

thesis Models . . . . . . . . . . . . . . . . . . . . . . . . . .

163

B.3.2 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . .

164

Experimental Evaluation . . . . . . . . . . . . . . . . . . . . . . . .

166

B.4.1 Experimental Conditions . . . . . . . . . . . . . . . . . . .

166

B.4.2 Objective Evaluations . . . . . . . . . . . . . . . . . . . . .

168

B.4.3 Subjective Evaluations . . . . . . . . . . . . . . . . . . . . .

170

Appendix C Speech Recognition and Speaker Classiﬁcation Performances 177

ix

List of Figures
1.1

Two speech synthesis techniques covered in this thesis, TTS and VC.
Diﬀerence between TTS and VC is input information of speech synthesis
systems. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

2

1.2

Applications of speech synthesis technology

2

1.3

Human abilities in speech communication (left) and related speech information processing (right)

1.4. . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . .

6

Overview of this thesis. This thesis aims to develop high-quality, diverse,
and interpretable DNN-based speech synthesis technology based on human’s speech information processing abilities: adversary (Chapters 3 and4), recognition (Chapter 5), and perception (Chapter 6). . . . . . . . . .

2.1

10

Flowcharts for DNN-based TTS and VC. DNN-based acoustic model istrained to represent mapping from input features to output speech parameters.

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

12

2.2

Overview of Chapter 2 . . . . . . . . . . . . . . . . . . . . . . . . . . . .

12

2.3

Conceptual diagram of speech analysis. STFT analysis is ﬁrst applied
to speech waveform to obtain time-frequency representations of speech.
Vocoder analysis is then carried out to decompose STFT amplitude spectra into spectral parameters and excitation parameters.

2.4. . . . . . . . .

Example of speech spectrum represented as product of spectral envelope
and excitation parameters in frequency domain . . . . . . . . . . . . . .

2.5

14

Example of continuous F0 modeling. Continuous F0 sequence and U/Vlabels are separately modeled.

2.6

13

. . . . . . . . . . . . . . . . . . . . . . .

14

Temporal alignment of features in TTS. Given phoneme boundary, eachphoneme-wise linguistic feature is duplicated to align its length with corresponding speech parameters.

. . . . . . . . . . . . . . . . . . . . . . .

16

x

List of Figures

2.7

DTW algorithm for feature alignment in VC. Phoneme boundaries of input and reference speech are superimposed for clear visualization.

. . .

17

. . . . .

18

. . . . . . . . .

18

2.8

Matrix computation to obtain static-dynamic feature sequence

2.9

Forward propagation procedure in Feed-Forward DNNs

2.10

Forward propagation procedure in LSTM. “Concat” denotes concatenat⊤ ⊤ing operation of given vectors, i.e., Concat(ht−1 , xt ) = [h⊤t−1 , xt ] .

2.11

. .

20

Forward propagation procedure in VAEs. Based on reparameterization
trick, VAE latent variable z is calculated as z = µϕ + σ ϕ ◦ ϵ, where ϵ issampled from N (ϵ; 0, I). . . . . . . . . . . . . . . . . . . . . . . . . . . .

21

2.12

Basic framework for DNN-based TTS

21

2.13

DNN-based acoustic model in TTS using vocoder parameters. Frame-. . . . . . . . . . . . . . . . . . .

wise static-dynamic features are predicted from corresponding linguisticfeatures. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
2.14

DNN-based multi-speaker acoustic modeling using speaker code as speakerrepresentation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

2.15

22
23

Speech synthesis using MLSA ﬁlter. Excitation signal is ﬁrst generated
from F0 and aperiodic components, then MLSA ﬁlter is applied to it forsynthesizing speech waveform.

2.16. . . . . . . . . . . . . . . . . . . . . . .

VC using spectral diﬀerentials. Input speech waveform is converted using
estimated spectral diﬀerentials ﬁlter. . . . . . . . . . . . . . . . . . . . .

2.17

27
28

Vocoder-free TTS process using STFT spectra. “G & L” indicates “Griﬃnand Lim.” . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

29

3.1

Conceptual diagram of proposed method in Chapter 3 . . . . . . . . . .

31

3.2

Relation between conventional and proposed methods in Chapter 3 . . .

31

3.3

Overview of Chapter 3 . . . . . . . . . . . . . . . . . . . . . . . . . . . .

32

3.4

GAN framework. Discriminator is trained to distinguish y and ŷ, while
generator is trained to fool it. Here, ŷ is generated from x through generator. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

3.5

33

Loss function and gradients for updating discriminator. “Param. Gen.”
indicates speech parameter generation using MLPG algorithm. Acoustic
model’s parameters are not updated in this step.

3.6. . . . . . . . . . . . .

34

Loss functions and gradients for updating acoustic model in proposed
GAN-based algorithm. Model parameters of discriminator are not updated in this step.

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

35

List of Figures

3.7

xi

Architecture to calculate isochrony-level duration from phoneme duration.
In case of Japanese, which has mora isochrony, each mora duration is
calculated from corresponding phoneme durations. For example, mora
duration of /ra/ is calculated as sum of phoneme durations of /r/ and /a/. 37

3.8

Matrix representation to calculate isochrony-level durations. This is example in syllable-timed language case such as Chinese. . . . . . . . . . .

3.9

37

Scatter plots of mel-cepstral coeﬃcients with several pairs of orders. Fromleft, ﬁgures correspond to natural speech, conventional MGE algorithm,
conventional MGE algorithm with GV compensation, and proposed algorithm (ωD = 1.0), respectively. These mel-cepstral coeﬃcients wereextracted from one utterance of evaluation data.

3.10. . . . . . . . . . . . .

42

Averaged GVs of natural and generated mel-cepstral coeﬃcients. Blackdashed line denotes GVs of natural mel-cepstral coeﬃcients. Gray andblue lines correspond to GVs of mel-cepstral coeﬃcients generated byconventional MGE and proposed algorithms, respectively.

3.11. . . . . . . .

43

MICs of natural and generated mel-cepstral coeﬃcients. MIC ranges from
0.0 to 1.0, and two variables with strong correlation have value closer to
1.0. From left, ﬁgures correspond to natural speech parameters, generated
ones by conventional MGE algorithm, and generated ones by proposed
algorithm, respectively. These MICs were calculated from one utteranceof evaluation data. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

3.12Parameter generation loss (above) and spooﬁng rate (below) curves
against various settings of hyperparameter ωD . . . . . . . . . . . . . . .

3.13

47

Parameter generation loss (above) and adversarial loss (below) curves fortraining data (blue-dashed line) and evaluation data (red line)

3.14

43

. . . . .

48

Preference scores of speech quality with 95% conﬁdence intervals (spectral
parameter generation in TTS). From top, numbers of listeners were 22,24, and 22, respectively. . . . . . . . . . . . . . . . . . . . . . . . . . . .

3.15

49

Preference scores of speech quality with 95% conﬁdence intervals (spectral
parameter and F0 generation in TTS). From top, numbers of listeners were
19 and 28, respectively.

3.16. . . . . . . . . . . . . . . . . . . . . . . . . . .

50

Preference scores of speech quality with 95% conﬁdence intervals (duration
generation in TTS). From top, numbers of listeners were 19, 20, and 21,
respectively. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

51

xii

3.17

List of Figures

Classiﬁcation accuracy of anti-spooﬁng.

“sp+F0”, “phoneme”, and“mora” denote using spectral parameters and F0 , phoneme durations,and mora durations to discriminate natural and synthetic speech,
respectively. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
3.18

Preference scores of speech quality with 95% conﬁdence intervals (comparison with GV compensation)

3.19. . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . .

57

Preference scores of speech quality with 95% conﬁdence intervals (DNNbased VC using spectral diﬀerentials)

3.25

57

Preference scores of speaker individuality with 95% conﬁdence intervals(DNN-based VC using speech parameter conversion) . . . . . . . . . . .

3.24

55

Preference scores of speech quality with 95% conﬁdence intervals (DNNbased VC using speech parameter conversion) . . . . . . . . . . . . . . .

3.23

54

MOS evaluation results of speech quality with 95% conﬁdence intervals(comparing various GANs)

3.22

53

Preference scores of speech quality with 95% conﬁdence intervals (eﬀectof LSTM-based acoustic model and discriminator)

3.21

53

Preference scores of speech quality with 95% conﬁdence intervals (eﬀectof feature function) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

3.20

52

. . . . . . . . . . . . . . . . . . .

58

Preference scores of speaker individuality with 95% conﬁdence intervals(DNN-based VC using spectral diﬀerentials) . . . . . . . . . . . . . . . .

58

4.1

Conceptual diagram of proposed method in Chapter 4 . . . . . . . . . .

61

4.2

Relation between conventional and proposed methods in Chapter 4 . . .

61

4.3

Overview of Chapter 4 . . . . . . . . . . . . . . . . . . . . . . . . . . . .

62

4.4

Matrix computation in average-pooling function

63

4.5

Examples of amplitude spectra of natural and synthetic speech after fre-. . . . . . . . . . . . .

quency warping for (a) linear, (b) mel, and (c) inverse mel frequency scales.
These spectra were extracted from one utterance of evaluation data. Synthetic amplitude spectra were generated from acoustic model trained tominimize MSE (Eq. (2.15)). . . . . . . . . . . . . . . . . . . . . . . . . .
4.6

64Frequency warping functions for (a) linear, (b) mel, and (c) inverse melfrequency scales . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

65

List of Figures

4.7

xiii

Loss functions to update acoustic model in proposed GAN-basedalgorithm using multi-frequency-resolution amplitude spectra. Averagepooling function ϕ(·) lowers frequency resolution of amplitude spectra
while keeping their rough structures unchanged.

4.8. . . . . . . . . . . . .

66

Examples of low-frequency-resolution spectra of natural and synthetic
speech. These spectra were extracted from one utterance of evaluation
data. Zero-padding size p and pooling stride s were set to 6 and w/2,
respectively. “(a) w = 1” corresponds to amplitude spectra in original
frequency resolution. These synthetic amplitude spectra were generated
from acoustic model trained to minimize Eq. (2.15).

4.9. . . . . . . . . . .

67

Examples of amplitude spectra of (a) natural speech and synthetic speech
generated by algorithms named as (b) “Baseline,” (c) “Low,” (d) “Original,” and (e) “Multi.” These spectra were extracted from one utteranceof evaluation data. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

4.10

73

Examples of amplitude spectra of (a) natural speech and synthetic speech
generated by proposed algorithms using GANs in low frequency resolution
with (b) linear scale, (c) mel scale, and (d) inverse mel scales. These
spectra were extracted from one utterance of evaluation data. . . . . . .

4.11

77

Conceptual diagram of split-and-pooling procedures. This ﬁgure corresponds to proposed algorithm with average-pooling applied to both low
and high frequency bands, i.e., “(w/, w/)” in Table 4.13.

. . . . . . . .

78

5.1

Conceptual diagram of proposed method in Chapter 5 . . . . . . . . . .

81

5.2

Relation between conventional and proposed methods in Chapter 5 . . .

81

5.3

Overview of Chapter 5 . . . . . . . . . . . . . . . . . . . . . . . . . . . .

82

5.4

Three directed graphical models of VAE-based multi-speaker acoustic
models. From left, VAEs are conditioned by (a) one-hot speaker codec, (b) c and phonetic latent variable z p , and (c) speaker latent variablez s and z p , respectively. Black and red arrows denote inference of latentvariables z ∗ and generation of speech parameter y, respectively.

5.5. . . .

83

Example of PPGs. Horizontal and vertical axes represent temporal axis
and phoneme index, respectively. Brighter values denote high posterior
probabilities.

5.6. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

83

Training procedure for DNN-based speech recognition model. PPG p̂ is
output of DNNs.

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

84

xiv

5.7

List of Figures

Proposed VAE-based multi-speaker acoustic model integrating speech
recognition and speaker classiﬁcation models. This model corresponds todirected graphical model shown in Fig. 5.4(c).

5.8. . . . . . . . . . . . . .

Training procedure for DNN-based speaker classiﬁcation model. d-vectord is extracted as output of squeeze layer immediately before output layer.

5.9. . . . . . . . . . . . . . . .

89

MCDs of converted speech in VAE-based non-parallel and many-to-many
VC . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

5.11

85

MCDs of converted speech in one-to-one VC. Here, only “FFNN” wastrained using fully-parallel speech corpora.

5.10

85

90

Scatter plots of d-vectors compressed with PCA. Each point denotes representation of one speaker, and three male and three female speakers usedin evaluation were marked with texts “M*” and “F*” in this plot, respectively. To place males’ cluster in left side and females’ cluster in rightside, some ﬁgures were rotated 180 degrees. . . . . . . . . . . . . . . . .

5.12

96

Scatter plots of VAE latent variables compressed with PCA. Each point
denotes latent variable averaged over 50 utterances of each speaker, andthree male and three female speakers used in evaluation were marked withtexts “M*” and “F*” in this plot, respectively. “CSM” in this plot meansclass separability measure calculated as trace of between-class scatter matrix divided by that of within-class scatter matrix.

. . . . . . . . . . . .

97

6.1

Conceptual diagram of proposed method in Chapter 6 . . . . . . . . . .

99

6.2

Relation between conventional and proposed methods in Chapter 6 . . .

100

6.3

Overview of Chapter 6 . . . . . . . . . . . . . . . . . . . . . . . . . . . .

100

6.4

Perceptual scoring of speaker-pair similarity. Speaker-pair pool storesspeaker pairs to be scored. Listener is asked to score perceptual similarityof two presented speakers’ voices as integer between −v and v. In thisﬁgure, v = 3.

6.5

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

(a) Perceptual speaker similarity matrix of 153 female Japanese speakers
obtained by large-scale perceptual scoring and (b) its sub-matrix . . . .

6.6

101
102

Calculation of loss functions in proposed algorithms based on (a) similarity
vector embedding, (b) similarity matrix embedding, and (c) similarity
graph embedding . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

103

List of Figures

6.7

xv

Speaker similarity graph deﬁned by similarity matrix shown in Fig. 6.5(b).
Each node represents one speaker, and links connect perceptually similarpairs (i.e., si,j > 0). Wider links indicate more similar speaker pairs. . .

104

6.8

Active learning of perceptual-similarity-aware speaker representations . .

105

6.9

(a) Fully scored (FS) and (b) partially scored (PS) settings . . . . . . . .

109

6.10

Histogram of perceptual similarity scores of 153 female Japanese speakers.
Red line denotes cumulative ratio. . . . . . . . . . . . . . . . . . . . . . .

6.11

Histogram of perceptual similarity scores of 13 speakers (from “F001” to“F013”) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

6.12

110

Scatter plots of similarity scores and predicted similarity with their correlation coeﬃcient r . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

6.13

110

111

ROC curves of similar speaker-pair detection using speaker representations. When curve becomes closer to left corner, speaker representations
can be used to ﬁnd similar speaker pairs more accurately.

6.14. . . . . . . .

112

Results of XAB tests on speaker similarity of the interpolated speech using(a) most dissimilar speaker pair (“F033-F134”) and (b) most similar one(“F017-F149”). When score curves become closer to red lines, listeners
accurately infer two speakers’ mixing ratio from interpolated speech. . .

6.15

Curves of similar speaker-pair detection AUC against active learning iteration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

6.16

116
118

PCA plots of speaker representations learned by (a) d-vec., (b) Prop.
(vec), (c) Prop. (mat), and (d) Prop. (graph). Similar-speaker pairs areconnected by gray edges. Red points denote unseen speakers. PCA wasapplied to 153 speakers’ embedding learned by each method independently.120

A.1

VC using input-to-output highway networks . . . . . . . . . . . . . . . .

A.2

Scatter plots of speech parameters. µT denotes transform gate’s value
averaged over one utterance.

. . . . . . . . . . . . . . . . . . . . . . . .

A.3

Examples of transform gate’s values predicted from mel-ﬁlter banks

A.4

Examples of transform gate’s values predicted from mel-cepstral coeﬃcients

A.5

. .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

154
155
156
157

Preference scores of converted speech’s speech quality with 95% conﬁdence
intervals (DNN-based VC using input-to-output highway networks) . . .

158

xvi

A.6

List of Figures

Preference scores of converted speech’s speaker individuality with 95%
conﬁdence intervals (DNN-based VC using input-to-output highway networks) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

B.1

159

Examples of PPGs predicted by speech parameters of four diﬀerent speakers who uttered same sentences used in subjective evaluation described in
section B.4.3. Horizontal and vertical axes represent temporal axis andphoneme index, respectively. Brighter values denote high posterior probabilities. Ranges of temporal axes in these ﬁgures were modiﬁed for clear
illustration. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

B.2

161

Scatter plots mel-cepstral coeﬃcients of (a) natural speech and (b)–(d)converted speech by three diﬀerent algorithms. Female target speaker’s
one utterance excluded from training data is used for extracting thesemel-cepstral coeﬃcients. . . . . . . . . . . . . . . . . . . . . . . . . . . .

B.3

162

Schematic diagram of proposed joint adversarial training algorithm. Domain classiﬁer C(·) and discriminator of GANs D(·) are ﬁrst updated.
Recognition model R(·) = Rp (Rf (·)) and synthesis model G(·) are thenupdated. These updates are iterated during training, and DNNs for manyto-one VC are constructed by concatenating ﬁnal R(·) and G(·). . . . .

165

xvii

List of Tables
3.1

Statistics of natural (“Natural”) and generated (“MGE” and “Proposed”)
continuous F0 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

3.2

Statistics of natural (“Natural”) and generated (“MSE” and “Proposed(*)”) phoneme duration . . . . . . . . . . . . . . . . . . . . . . . .

3.3

44

Architectures of DNNs used in TTS evaluations. Feed-Forward DNNs
were used for all architectures . . . . . . . . . . . . . . . . . . . . . . . .

4.1

44

Statistics of natural (“Natural”) and generated (“MSE” and “Proposed(*)”) mora duration . . . . . . . . . . . . . . . . . . . . . . . . . .

3.4

44

45

Objective evaluation results with their standard deviations. “RMSE” denotes root mean square error between natural and generated amplitude(L)

spectra. Various hyperparameter settings (ωD , ωD ) for proposed algorithms were used and compared. Pooling parameters were set to w = 30,
s = 15, and p = 6. These evaluation values were calculated using all
evaluation data and averaged . . . . . . . . . . . . . . . . . . . . . . . .
4.2

69

Preference scores of synthetic speech quality (GANs using originalfrequency-resolution amplitude spectra with various hyperparametersettings of ωD ). Bold values indicate that method is more preferred thanother with p-value < 0.05

4.3

. . . . . . . . . . . . . . . . . . . . . . . . . .

70

Preference scores of synthetic speech quality (GANs using low-frequencyresolution amplitude spectra with various settings of w). Bold values
indicate that method is more preferred than other with p-value < 0.05.
Here, number of hidden units in D(L) (·) was changed in accordance withsettings of w

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

71

xviii

4.4

List of Tables

Preference scores of synthetic speech quality (GANs using low-frequencyresolution amplitude spectra with various settings of w). Bold values
indicate that method is more preferred than other with p-value < 0.05.
Here, number of hidden units in D(L) (·) was ﬁxed regardless of settings of
w . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

4.5

71

Preference scores of synthetic speech quality (GANs using low-frequency(L)resolution amplitude spectra with various hyperparameter settings of ωD

and ﬁxed pooling width w = 30). Bold values indicate that method ismore preferred than other with p-value < 0.05
4.6

. . . . . . . . . . . . . .

72

Preference scores of synthetic speech quality (comparison between
GANs using low-frequency-resolution and smoothed original-frequencyresolution amplitude spectra). Bold value indicates that method is more
preferred than other with p-value < 0.05 . . . . . . . . . . . . . . . . . .

4.7

72

Preference scores of speech quality (GANs using original-, low-, andmulti-frequency-resolution amplitude spectra). Bold values indicate thatmethod is more preferred than other with p-value < 0.05 . . . . . . . . .

4.8

73

Preference scores of synthetic speech quality (GANs using low-frequencyresolution amplitude spectra with gated-CNN-based acoustic model).
Bold values indicate that method is more preferred than other withp-value < 0.05

4.9

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

74

Preference scores of synthetic speech quality (comparison between FeedForward DNNs (“FFNN”) and gated CNNs (“CNN”) for acoustic modeling). Bold values indicate that method is more preferred than other withp-value < 0.05

4.10

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

75

DNN architectures for discriminator. Numbers of input units of “FFNNO,” “FFNN-L,” “CNN-1D,” and “CNN-2D,” were 513, 34, 513, and513, respectively. “ReLU” or “GLU” denotes hidden activation function.
Width, stride, and zero-padding size of Conv1D layer in “CNN-1D”
were set to 30, 15, and 6, respectively. Width parameters of Conv2D
layers in “CNN-2D” were set to 9, 7, 5, and 3. Accordingly, stride andzero-padding parameters were set to 4, 3, 2, and 1

4.11. . . . . . . . . . . .

75

Preference scores of synthetic speech quality (GANs using gated CNNs as
acoustic models and various DNN architectures for discriminative models).
Bold values indicate that method is more preferred than other with pvalue < 0.05 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

76

List of Tables

4.12

xix

Preference scores of synthetic speech quality (GANs using low-frequencyresolution amplitude spectra with various frequency scale). Bold values
indicate that method is more preferred than other with p-value < 0.05 .

4.13

77

Preference scores of synthetic speech quality (comparison among the combination of frequency band (low or high) and average-pooling (w/ or w/o)).
Bold values indicate that method is more preferred than other with pvalue < 0.05 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

5.1

78

Subjective evaluation results for naturalness of speech converted with different six DNNs and their 95% conﬁdence intervals. Bold and underlinedscores mean highest and lowest ones among ﬁve DNNs except for “FFNN,”
respectively . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

5.2

90

Subjective evaluation results for speaker similarity of speech convertedwith diﬀerent six DNNs and their 95% conﬁdence intervals. Bold andunderlined scores mean highest and lowest ones among ﬁve DNNs except
for “FFNN,” respectively

5.3. . . . . . . . . . . . . . . . . . . . . . . . . .

91

Subjective evaluation results for naturalness of speech converted by proposed VAE-based VC methods with diﬀerent d-vector dimensionality andtheir 95% conﬁdence intervals. Bold and underlined scores mean highest
and lowest ones among six settings, respectively

5.4. . . . . . . . . . . . .

92

Subjective evaluation results for speaker similarity of speech converted byproposed VAE-based VC methods with diﬀerent d-vector dimensionality
and their 95% conﬁdence intervals. Bold and underlined scores mean
highest and lowest ones among six settings, respectively

5.5. . . . . . . . .

92

Subjective evaluation results for naturalness of converted speech with ﬁxed
numbers of seen speakers and varied d-vector dimensionality. Bold indicates more preferred method with p-value < 0.05 . . . . . . . . . . . . .

5.6

93

Subjective evaluation results for speaker similarity of converted speechwith ﬁxed numbers of seen speakers and varied d-vector dimensionality.
Bold indicates more preferred method with p-value < 0.05

5.7

. . . . . . .

94

Subjective evaluation results for naturalness of converted speech with varied numbers of seen speakers and best d-vector dimensionality for eachsetting. Here, d-vector dimensionality was “32d” for “50spk,” “16d” for“130spk,” and “8d” for “260spk,” respectively. Bold indicates more preferred method with p-value < 0.05 . . . . . . . . . . . . . . . . . . . . . .

94

xx

5.8

List of Tables

Subjective evaluation results for speaker similarity of converted speechwith varied numbers of seen speakers and the best d-vector dimensionality
for each setting. Here, d-vector dimensionality was “32d” for “50spk,”“16d” for “130spk,” and “8d” for “260spk,” respectively. Bold indicatesmore preferred method with p-value < 0.05

5.9

. . . . . . . . . . . . . . . .

95

RMSEs of PPGs predicted from source and target speakers. These results
were averaged over all evaluation data. Bold indicates lowest RMSE ineach row . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

6.1

AUC values of similar speaker-pair detection using speaker representations
learned by four diﬀerent algorithms . . . . . . . . . . . . . . . . . . . . .

6.2

95

112

Preference scores on naturalness of synthetic speech (left: conventional dvector, right: proposed algorithm). Bold indicates more preferred methodwith p-value < 0.05 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

6.3

113

Preference scores on speaker similarity of synthetic speech (left: conventional d-vector, right: proposed algorithm). Bold indicates more preferred
method with p-value < 0.05 . . . . . . . . . . . . . . . . . . . . . . . . .

6.4

114

Results of MOS test on naturalness and DMOS test on speaker similarity
with their 95% conﬁdence intervals (comparison among three proposed
algorithms)

6.5. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

114

Results of MOS test on naturalness of interpolated speech with their 95%
conﬁdence intervals. Bold scores are signiﬁcantly higher than those of
d-vec. (p < 0.05) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

6.6

115

Results of MOS test on naturalness of synthetic speech with their 95%
conﬁdence intervals (proposed algorithms using active learning). Second
column denotes percentages of number of additionally scored speaker pairs
compared with “PS.” Bold scores are comparable to those of “FS” (p >0.05) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

6.7

118

Results of DMOS test on speaker similarity of synthetic speech with their
95% conﬁdence intervals (proposed algorithms using active learning). Second column denotes percentages of number of additionally scored speakerpairs compared with “PS.” Bold scores are comparable to “FS” (p > 0.05) 119

List of Tables

B.1

DNN architectures used in experimental evaluation.

xxi

In this table,“Conv1D(Cin , Cout , k , s)” and “Deconv1D(Cin , Cout , k , s)” denote 1D
convolution and deconvolution layers, respectively. Cin and Cout mean
number of channels of input and output, respectively. k and s denoteconvolution window size and stride width, respectively . . . . . . . . . .
B.2

167

LogGVDs between natural and generated mel-cepstral coeﬃcients with
their standard deviations . . . . . . . . . . . . . . . . . . . . . . . . . . .

169

B.3

Frame-wise phoneme recognition accuracy of speech recognition models [%]169

B.4

MCCs of domain classiﬁers . . . . . . . . . . . . . . . . . . . . . . . . . .

B.5

Results of MOS test on naturalness of converted speech with their 95%

170

conﬁdence intervals. Bold values indicate that method is more naturallysounded than “Baseline” with p-value < 0.05
B.6

. . . . . . . . . . . . . . .

171

Preference scores of converted speech speaker similarity. Bold values indicate that method is more preferred than other with p-value < 0.05. Here,“Prop. (DAT-GAN)” was compared with “Baseline” or “Prop. (GAN)”

B.7

172

Preference scores of converted speech naturalness. Bold values indicatethat method is more preferred than other with p-value < 0.05. Here,“Baseline” was compared with “Prop. (Joint)” or “Prop. (DAT)”

B.8

. . .

173

Preference scores of converted speech speaker similarity. Bold values
indicate that method is more preferred than other with p-value < 0.05.
Here, “Baseline” was compared with “Prop. (Joint)” or “Prop. (DAT)”

B.9

174

Preference scores of converted speech naturalness. Bold values indicatethat method is more preferred than other with p-value < 0.05. Here,“StarGAN-VC” was compared with “Prop. (DAT-GAN)” . . . . . . . .

B.10

175

Preference scores of converted speech speaker similarity. Bold values
indicate that method is more preferred than other with p-value < 0.05.
Here, “StarGAN-VC” was compared with “Prop. (DAT-GAN)”

C.1. . . .

176

Frame-wise PER of speech recognition DNNs and EER of speaker veriﬁcation using d-vectors. Diﬀerent settings of number of seen speakers
and d-vector dimensionality were considered. Note that, EER for onedimensional d-vectors cannot be computed since d-vector in this evaluation always takes positive value and is normalized so that its L2 norm
becomes 1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

178

1

Chapter 1

Introduction
1.1

BackgroundSpeech is the most natural means for humans to communicate with each other. A speakerin speech communication transmits his/her intention, attitude, and emotion through the
air as a speech waveform to a listener. Speech information processing aims to emulate
such speech generation and recognition processes using a computer and assist in speechcommunication among humans and robots.
This thesis focuses on speech synthesis, a technology for artiﬁcially synthesizing, transforming, and retouching human speech by using a computer. This thesis covers two basic
techniques in particular: text-to-speech (TTS) [1] and voice conversion (VC) [2]. Figure 1.1 illustrates the general frameworks of TTS and VC. TTS synthesizes speech fromtext and enables the development of a speech-based man-machine interface for computers, smartphones, and smart speakers. VC transforms non-/para-linguistic information of
input speech while keeping its phonetic content unchanged and actualizes speech communication beyond human physical constraints. Figure 1.2 shows the typical applications of
TTS and VC. These techniques can enrich human speech communication through language
education [3, 4], virtual reality [5], and singing voice synthesis or conversion [6, 7, 8]. They
can also remove barriers in human communication by developing speech translation [9, 10]
and speaking aid [11, 12]. Therefore, the development of speech synthesis technology forsynthesizing high-quality, versatile, and intuitively controllable speech will contribute toan advanced society where humans and computers cooperate.
One of the challenging aspects of speech synthesis research is to accurately synthesize
human speech, which contains various information such as phonetic content and speaker
identity. In general, even if the phonetic content is the same, human speech varies greatly

2Chapter 1 Introduction
TTS

/ŶƉƵƚ
ƚĞǆƚ

VC

KƵƚƉƵƚƐƉĞĞĐŚ

/ŶƉƵƚƐƉĞĞĐŚ

KƵƚƉƵƚƐƉĞĞĐŚ

Fig. 1.1. Two speech synthesis techniques covered in this thesis, TTS and VC. Diﬀerence
between TTS and VC is input information of speech synthesis systems.
Man-machineinterface

Speech translation

Speech communication
assistance

TTS/VCEntertainment
applications

Singing voice
conversion

Fig. 1.2. Applications of speech synthesis technology

due to the speaker diﬀerence and intra-speaker variability (e.g., speaking style and emotion). Therefore, for example, TTS is required not only to read a given text accurately,
but also to reproduce the identity and speaking style of the speaker with high ﬁdelity.
This challenge needs to be addressed for developing a speech synthesis technology thatcan synthesize high-ﬁdelity speech and remove barriers to speech communication. However, a uniﬁed solution has not yet been established. This thesis focuses on monolingual
speech synthesis and aims to establish a technology that can synthesize various speakers’
voices with high quality, which can be the basis for technology to generate multilingualand diverse style voices. It also aims to develop a speech synthesis technology that canconsider not only the speaker’s voice reproduction but also the listener’s voice perception
to facilitate speech communication between humans and robots.
Speech synthesis methods can be classiﬁed into two categories: rule-based and corpusbased methods. Rule-based speech synthesis methods utilize some rules designed byexperts, such as formant frequency, voicing, and noise levels, to synthesize speech artiﬁcially. Formant synthesis [13] is a well-known example of this method, which canChapter 1 Introduction

3

achieve a lightweight speech synthesis system with few parameters. However, the qualityof speech synthesized by this method is far from that of natural speech due to the limited rules and the diﬃculty in synthesizing naturally-sounding consonants. On the other
hand, corpus-based speech synthesis methods utilize a speech corpus containing humanspeech recordings to develop high-quality speech synthesis systems. Concatenative speechsynthesis [14] and statistical one [15] are classiﬁed as this category. The former properly
concatenate segments of speech waveforms or parameters stored in a speech corpus tosynthesize speech. This method can achieve high naturalness of synthetic speech because
natural speech’s voice characteristics are preserved as much as possible. However, it requires a heavy footprint to store various speech recordings and has diﬃculty in controllingthe voice characteristics of synthetic speech. Meanwhile, the latter train a statistical modelthat generates speech parameters from input features. This method’s main advantage isthe high controllability of synthetic speech with a relatively small footprint, thanks to its
theoretically-grounded machine learning framework. However, synthetic speech quality
tends to degrade due to many factors, such as improper speech parameterization and insuﬃcient statistical modeling. Among these various methods, this thesis mainly focuses
on statistical speech synthesis because it can approach the essence of human behavior thatlearns from and adapts to observed data.
A basic framework of statistical speech synthesis consists of two phases: training andsynthesis [15]. In the training phase, input features, i.e., linguistic features in TTS andsource speaker’s speech parameters in VC, and output speech parameters are ﬁrst extracted from a training speech corpus. An acoustic model is then trained to represent therelation between input features and output speech parameters. In the synthesis phase,
input features are ﬁrst extracted from the given input, i.e., arbitrary text in TTS andsource speaker’s arbitrary voice in VC. Speech parameters are then generated from the
input features by using the trained model. Finally, a speech waveform is synthesized fromthe generated parameters. Through the training and synthesis phases, statistical speechsynthesis oﬀers a means to synthesize diverse speech and control voice characteristics inthe synthetic speech better than concatenative TTS [14] and codebook-based VC [16].
Also, it can be extended to multi-speaker statistical speech synthesis [17, 18] that synthesizes multiple speakers’ voices using a single acoustic model. Such multi-speaker acoustic
modeling can widen the range of speech synthesis applications (e.g., speaker anonymization [19] and data augmentation [20]).
Deep neural network (DNN)-based speech synthesis [21, 22] involves DNNs [23] asacoustic models, one of the hottest topics in speech synthesis research. The main ad-

4Chapter 1 Introduction

vantage of using DNNs is to synthesize high-ﬁdelity speech better than traditional hidden Markov model (HMM)-based TTS [24] and Gaussian mixture model (GMM)-based
VC [25]. The reason why is that DNNs can learn the complicated relation between input
and output through a series of linear transformation and nonlinear activation functions.
The improvement of computer performance and the availability of large speech corpora
have further advanced DNN-based speech synthesis. Also, knowledge of other related
DNN-based speech information processing, such as speech recognition [23, 26], speakerclassiﬁcation [27, 28], and anti-spooﬁng [29], can be introduced through the uniﬁed DNN
training framework based on the backpropagation (BP) algorithm [30]. This thesis aims todevelop high-quality, versatile, and intuitively controllable DNN-based speech synthesis,
following these backgrounds.

1.2

Thesis Scope

1.2.1

Issues to Be Addressed

This section describes the issues with conventional DNN-based speech synthesis that are
addressed in this thesis.
Low-quality Synthetic Speech Due to Over-smoothing
Although DNN-based speech synthesis can learn the complicated mapping from input
features to speech parameters, the quality of synthetic speech degrades. A primary causeof quality degradation is over-smoothing [15, 31] of generated speech parameters. It
removes the ﬁne structures of natural speech parameters and makes the synthetic speechsound muﬄed. One can alleviate over-smoothing by training an acoustic model to consider
the statistical diﬀerence between natural and generated speech parameters. For example,
the probability distributions of natural speech parameters’ statistics are ﬁrst modeled ina parametric [25] or non-parametric [32] manner. Speech parameters are then generated
or transformed using the distributions to reproduce the statistical properties of natural
speech. A more eﬀective approach is to use analytically derived features that quantify
the degree of over-smoothing. A global variance (GV) [25] and modulation spectrum(MS) [33] of a speech parameter sequence are well-known examples of such features, andwork as constraints in the acoustic model training [34, 35]. Nose and Ito [36] proposeda method of reducing the GV diﬀerence between natural and synthetic speech, assumingthat their GVs follow the Gaussian distribution. Takamichi et al. [34] extended this idea
to MS-based statistical property reproduction. However, quality degradation remains aChapter 1 Introduction

5

critical problem in both TTS and VC, regardless of the speech parameterization level, i.e.,
vocoder-derived parameters or short-term Fourier transform (STFT) spectra.
Limited Speaker Diversity in Synthetic Speech
A suﬃciently large speech corpus that includes multiple speakers’ speech utterances isrequired to train a DNN-based high-quality multi-speaker acoustic model. This requirement can be problematic in DNN-based parallel VC that trains DNNs using pairs ofsource and target speakers’ speech utterances with the same phonetic content. However,
it is unrealistic to record multiple speakers’ parallel speech utterances. Non-parallel VC
using variational autoencoders (VAEs) [37] has been proposed to solve this problem [38].
Although it can train a VAE-based acoustic model without preparing a parallel speech
corpus, the converted speech quality degrades. One of the reasons for this quality degradation is over-regularization [39] of VAEs’ latent variables expected to represent phonetic
content of input speech. One can alleviate over-regularization by using a more informative prior distribution of the latent variables such as GMMs [40]. However, it is diﬃcult
to determine the number of clusters in a GMM prior because variations in the phonetic
content are typically large. Also, the conventional VAE-based VC method cannot handle
unseen speakers who are not included in training data, in spite of the VAEs’ ability thatcan learn various speakers’ voice characteristics.
Uninterpretable Speaker Representation
Speaker representations are additional input features for controlling speaker individualityof synthetic speech in DNN-based speech synthesis. Therefore, they play an essential rolein developing intuitively controllable speech synthesis technology. Speaker representations
for such technology should be designed considering how humans subjectively perceive thediﬀerences among speakers. The most straightforward speaker representation is a speaker
code [41] that represents speaker individuality as a one-hot vector. However, humanspeaker perception is completely ignored because diﬀerent speakers are orthogonal to eachother in the speaker space constructed by speaker codes. Such speaker space prevents auser from ﬁnding his/her favorite voice characteristics intuitively. Also, a speaker code
cannot deﬁne an unseen speaker’s individuality due to its discrete nature, limiting thenumber of speakers’ voices that can be synthesized.

6

Chapter 1 Introduction(1) Human abilities in speechcommunication(2) Related speech
information processing
1: Natural

Anti-spoofing
TTS/VC

0: Synthetic(a) Detection of synthetic speech: "Hi!"Hi!

(a) Anti-spoofing

Speechrecognition

"Hi!"

Spkr.
classification(b) Recognition of
speech content & spkr. individuality(b) Speech recognition &spkr. classificationൎ
Humanlisteners(c) Similarity-aware
spkr. perception

Similar/
Dissimilar(c) Human computation

Fig. 1.3. Human abilities in speech communication (left) and related speech information
processing (right)

1.2.2

Methods Proposed in This Thesis

This thesis proposes three methods to overcome these issues and further advance DNNbased speech synthesis technology: 1) high-quality speech synthesis integrating a speechadversary process, 2) versatile speech synthesis integrating a speech recognition process,and 3) interpretable speaker representation learning introducing human speech perception.
The core idea of these methods is to utilize human’s abilities related to speech information processing, i.e., adversary, recognition, and perception, to develop better statisticalspeech synthesis. Figure 1.3 contrasts the three abilities with related speech information
processing. Humans can (1a) detect synthetic speech using their knowledge and experi-Chapter 1 Introduction

7

ences, (1b) recognize the phonetic content or speaker individuality of speech accurately,and (1c) remember an unseen speaker’s voice characteristics by associating them withmemories. The ﬁrst two can be substituted by data-driven machines created by humansas (2a) anti-spooﬁng and (2b) speech recognition or speaker classiﬁcation, and the thirdis directly modeled on human responses through (2c) human computation.
High-quality Speech Synthesis Integrating Speech Adversary Process
A generative adversarial network (GAN)-based method is proposed to overcome the quality degradation caused by over-smoothing. The core idea is to use the human’s adversarial
ability that discriminates natural speech from synthetic speech. A discriminator that distinguishes natural speech parameters from generated ones is introduced with this method
to emulate this ability. The DNN-based acoustic model is trained to fool the discriminatorby generating speech parameters indistinguishable from natural ones. Since the objectiveof GANs is the minimization of divergence (i.e., the distribution diﬀerence) between natural and generated speech parameters, this method eﬀectively alleviates over-smoothing.
From a diﬀerent perspective, the discriminator with this method can be interpreted asanti-spooﬁng, i.e., a technique to detect synthetic speech and prevent voice spooﬁng attacks. Accordingly, techniques and ideas concerning anti-spooﬁng can be introduced to
improve synthetic speech quality further.
Versatile Speech Synthesis Integrating Speech Recognition Process
A VAE-based speech synthesis method is proposed to synthesize arbitrary speakers’ voiceswith high quality. The core idea is to use the human’s recognition ability that recognizes
phonetic contents and speaker information of speech accurately. A DNN-based speech
recognition model is introduced to train VAEs for speech synthesis. As a result, thesynthetic speech’s phonetic content is clariﬁed, and high-quality synthetic speech is synthesized. Continuous speaker representations derived from a DNN-based speaker classiﬁcation model are used to overcome the limitations of discrete speaker codes and increase
speaker diversity.
Interpretable Speaker Representation Learning Introducing Human Speech
Perception
A speaker representation learning method is proposed to increase the interpretability of
speaker representations and enable controllable speech synthesis technology. The coreidea is to regard the human’s perception as computational resources for learning theinterpretable speaker representations. This method incorporates human listeners into the8Chapter 1 Introduction

speaker representation learning framework based on human computation. A large-scale
subjective scoring in terms of multiple speakers’ perceptual similarity is ﬁrst conducted
involving a large number of listeners. DNNs are then trained for speaker representation
learning considering the similarity scores. An active learning algorithm is proposed toreduce subjective scoring and DNN training costs.

1.3

Thesis Outline

The rest of this thesis is organized as follows (see also Fig. 1.4).
Chapter 2 brieﬂy reviews the basic framework of DNN-based speech synthesis. The
framework’s four crucial factors are speciﬁcally described: 1) feature analysis, 2) acoustic
modeling, 3) speech parameter generation, and 4) speech waveform synthesis.
Chapter 3 discusses the proposed GAN-based method to improve synthetic speech quality. The basic framework of GANs is ﬁrst described. The introduction of GANs to train
DNNs for speech synthesis is then discussed. From the divergence minimization perspective, the eﬀects of the divergence in improving synthetic speech quality are investigated.
Experimental evaluations are conducted to demonstrate this method’s eﬀectiveness in
TTS and VC using vocoder-derived speech parameters. The results indicate that 1) thismethod can generate natural speech parameters regardless of its hyperparameter settings,
and 2) Wasserstein GAN minimizing the earth-mover’s distance works the best among
other image-processing-related or speech-processing-related GANs in terms of improving
synthetic speech quality.
Chapter 4 extends the proposed GAN-based method described in Chapter 3 to DNNbased speech synthesis using STFT spectra. A simple but eﬀective GAN-based approachis proposed to overcome the diﬃculty in modeling high-dimensional and complicated amplitude spectra. Various frequency scales that are related to human speech perception
are also introduced. The eﬀectiveness of this method is evaluated in TTS using STFT
spectra. The results indicate that 1) GANs using low-frequency-resolution amplitude
spectra improve speech quality and work robustly against the settings of the frequency
resolution and hyperparameters, 2) in comparison of low-, original-, and multi-frequencyresolution amplitude spectra, the use of low-frequency-resolution spectra works best to
improve synthetic speech quality, and 3) the use of the inverse mel frequency scale forobtaining low-frequency-resolution amplitude spectra further improves synthetic speech
quality.
Chapter 5 presents the proposed high-quality and versatile speech synthesis method

Chapter 1 Introduction

9

based on VAEs. The VAE-based speech parameter generation process is mathematicallyformulated to explicitly model the phonetic content and speaker individuality as latentvariables. Non-parallel and many-to-many VC that can reproduce and transform arbitrary speaker’s voice characteristics is established using VAEs. The trade-oﬀ between
the number of seen speakers and dimensionality of continuous speaker representation
with this method is also investigated. The eﬀectiveness of this method is objectively
and subjectively evaluated in VC. The results indicate that 1) the introduction of a
DNN-based speech recognition model contributes to signiﬁcant quality improvement inconverted speech, 2) the use of continuous speaker representations achieves high-quality
VC even if the source and target speakers’ speech utterances are not used for the VAE
training, and 3) high-dimensional speaker representation does not necessarily improve theconverted speech quality, but a large number of seen speakers consistently does improve
this.
Chapter 6 presents the proposed perceptual-similarity-aware speaker representation
learning method for increasing the interpretability of speaker representations. The subjective scoring of perceptual speaker-pair similarity to obtain a perceptual speaker similarity
matrix is ﬁrst described. Methods of training DNNs for speaker representation learning using a loss function deﬁned by the similarity matrix are then presented. An active
learning algorithm to reduce the costs of scoring and training is ﬁnally introduced. Thismethod’s eﬀectiveness is evaluated with the proposed VAE-based speech synthesis method
described in Chapter 5. The results indicate that 1) the proposed speaker representation
learning method learns speaker representations strongly correlated with perceptual similarity scores, 2) the representations improve synthetic speech quality better than conventional representations derived from a DNN-based speaker recognition model, and 3) the
active learning algorithm achieves higher synthetic speech quality while reducing the costsof scoring and training.
Chapter 7 summarizes this thesis and mentions future directions.

10

Chapter 1 Introduction

Goal: High-quality, diverse, & interpretable speech synth.

Quality
improvement

Diversity
improvement

Interpretability
improvement

Adversary

Recognition

Perception

Chapters 3 & 4:
GAN-based
high-quality
speech synth.

Chapter 5:
VAE-based
multi-spkr.
speech synth.

Chapter 6:
Perceptualsimilarity-aware
spkr. reprs.

Chapter 2: DNN-based speech synth.
Fig. 1.4. Overview of this thesis. This thesis aims to develop high-quality, diverse, andinterpretable DNN-based speech synthesis technology based on human’s speech
information processing abilities: adversary (Chapters 3 and 4), recognition(Chapter 5), and perception (Chapter 6).

11

Chapter 2

Statistical Speech Synthesis
Using DNNs
2.1

Introduction

Statistical speech synthesis using DNNs, i.e., DNN-based speech synthesis, consists of twophases: training and synthesis. Figure 2.1 illustrates the ﬂowcharts for DNN-based TTS
and VC. In the training phase, input features (i.e., linguistic information of the given text
in TTS or speech parameters of source speaker in VC) and output speech parameters are
ﬁrst extracted in the feature analysis step. A mapping from the input features to speechparameters is then modeled using DNNs in the acoustic modeling step. In the synthesis
phase, input features are ﬁrst extracted from the given input. Speech parameters arethen generated by the trained DNNs using the extracted input features in the parameter
generation step. A speech waveform is ﬁnally synthesized using the generated speechparameters in the speech waveform synthesis step. This chapter focuses on two primary
techniques: vocoder-based and vocoder-free synthesis. DNN-based multi-speaker speechsynthesis for synthesizing speech with versatile speaker individuality using a single acoustic
model is also described.
This chapter is organized as follows (see also Fig. 2.2). Section 2.2 describes the feature
analysis step, i.e., speech analysis for speech parameter extraction and text analysis forlinguistic feature extraction. Section 2.3 explains the acoustic modeling step, including
the DNN architecture design, speciﬁc TTS or VC modeling, the extension to multi-speakermodeling, and basic training objective functions. Section 2.4 gives an explanation of the
speech parameter generation step. Section 2.5 explains the speech waveform synthesis
step. Section 2.6 summarizes this chapter.

Chapter 2 Statistical Speech Synthesis Using DNNsdƌĂŝŶŝŶŐƉŚĂƐĞ

12

EEͲďĂƐĞĚdd^KƵƚƉƵƚƐƉĞĞĐŚ
SpeechTraining

EEͲďĂƐĞĚsKƵƚƉƵƚƐƉĞĞĐŚ
SpeechTraining

datasetdatasetanalysis/ŶƉƵƚ
ƐƉĞĞĐŚ

/ŶƉƵƚ
ƚĞǆƚ
Textanalysis

Acoustic
modeling

Speechanalysis

Acoustic
modeling

EEͲďĂƐĞĚ
ĂĐŽƵƐƚŝĐ
ŵŽĚĞů

EEͲďĂƐĞĚ
ĂĐŽƵƐƚŝĐ
ŵŽĚĞů/ŶƉƵƚ
ƐƉĞĞĐŚ

/ŶƉƵƚ
ƚĞǆƚ^ǇŶƚŚĞƐŝƐƉŚĂƐĞ

analysisParameter
generationfrom DNN

Text
analysisKƵƚƉƵƚƐƉĞĞĐŚ

Speechwaveform
synthesis

Parameter
generationfrom DNN

SpeechanalysisKƵƚƉƵƚƐƉĞĞĐŚ

Speechwaveform
synthesis

Fig. 2.1. Flowcharts for DNN-based TTS and VC. DNN-based acoustic model is trained
to represent mapping from input features to output speech parameters.

TTS

Text
analysis

VC

SpeechanalysisSect. 2.2

Acoustic
Modeling

Sect. 2.3

Speechparameter
generation

Speechwaveform
synthesis

Sect. 2.4

Sect. 2.5

Fig. 2.2. Overview of Chapter 2

2.2

Feature Analysis

2.2.1

Speech Analysis

Figure 2.3 illustrates the conceptual diagram of speech analysis in statistical speech synthesis. STFT analysis is applied to a waveform for obtaining the time-frequency repre-

Chapter 2 Statistical Speech Synthesis Using DNNs

13

Spectral params.
(vocal tract feats.)
Speechwaveform

STFT
analysis

Vocoderanalysis

STFT
amplitude spectra

Excitation params.
(vocal cord feats.)

Fig. 2.3. Conceptual diagram of speech analysis. STFT analysis is ﬁrst applied to speech
waveform to obtain time-frequency representations of speech. Vocoder analysis is then carried out to decompose STFT amplitude spectra into spectral
parameters and excitation parameters.

sentations of the waveform. Amplitude and phase spectra are given as the STFT analysis
results. One can use these STFT spectra as the speech parameters to be modeled by a
DNN-based acoustic model. However, it is diﬃcult to predict the phase spectra using
a statistical model due to the high randomness. Therefore, only the amplitude spectra
are often considered as the speech parameters [42]. The amplitude spectra are further
decomposed into spectral parameters (i.e., vocal tract features) and excitation parameters (i.e., vocal cord features). This decomposition is based on source-ﬁlter modeling [43]to obtain more controllable speech representations. The former and latter represent the
spectral envelope (shown with a red dashed line in Fig. 2.4) and ﬁne structure of theamplitude spectra, respectively. The excitation parameters are further decomposed into
periodic factors typically represented as the fundamental frequency (F0 ) and aperiodic
factors [44].
As the dimensionality of spectral parameters tends to be high, a dimensionality reduction technique is applied to the parameters before the acoustic modeling. A common
technique uses mel-cepstral coeﬃcients [45] that consider the perceptual eﬀects of humanlistening in lower frequency components for dimensionality reduction.
The diﬀerence between voiced regions (V) and unvoiced regions (U) must be consideredin modeling F0 . Continuous F0 modeling [46] was proposed to represent the F0 parameters
eﬃciently. It uses one-dimensional continuous values to represent the observed log F0 and
one-dimensional discrete values representing U/V (0 for U and 1 for V). The log F0values in unvoiced regions are estimated using SPLINE interpolation. Figure 2.5 shows
an example of a continuous F0 sequence and U/V labels.
Vocoder systems are typically utilized to extract the spectral and excitation parameters. A well-known example is the STRAIGHT [47] vocoder that can achieve high-quality

14

Chapter 2 Statistical Speech Synthesis Using DNNs

Log amplitude

Observed spectrum
Spectral envelope

Frequency
Fig. 2.4. Example of speech spectrum represented as product of spectral envelope andexcitation parameters in frequency domain

Observed log F0Continuous log F0

Log F0

5.5
5.0
4.5
4.0

U/V

3.5
1

0
150

200

250

300
Frame index

350

400

450

Fig. 2.5. Example of continuous F0 modeling. Continuous F0 sequence and U/V labels
are separately modeled.

parameter extraction and speech synthesis. However, the deployment of speech synthesis
systems using the STRAIGHT vocoder is limited due to its patent protection. Anotherexample is the freely available WORLD [48, 49] vocoder.

Chapter 2 Statistical Speech Synthesis Using DNNs

2.2.2

15

Text Analysis

Text analysis is conducted to extract linguistic features from input text in TTS. This thesis
focuses on Japanese TTS systems [50]. Japanese linguistic features consist of phoneme,
accent type, word, part-of-speech, breath group, and sentence length, etc. A text analyzer,
such as MeCab [51], is used to extract these features. The extracted linguistic features
are represented as multi-dimensional vectors, including categorical factors (e.g., phoneme
identity and accent type) and numeric factors (e.g., the total number of phonemes andsentence length). Because Japanese is a mora-timed language (i.e., mora isochrony), themora features are also included in the linguistic features.

2.2.3

Temporal Alignment

The lengths of the input features and output speech parameters are aligned for acoustic
modeling.
In TTS, the lengths of the linguistic features are much shorter than those of the speechparameters. Thus, each linguistic feature is duplicated, considering the phoneme durations
to align its length with the corresponding speech parameters. The Viterbi algorithm using
HMMs is often used for phoneme duration prediction. Figure 2.6 shows an example ofthe feature alignment in TTS.
In VC, the source and target speech parameters’ lengths are typically diﬀerent; therefore, they need to be aligned before acoustic modeling. The dynamic time warping (DTW)
algorithm [25] is used to align the lengths. Figure 2.7 shows a conceptual diagram of the
DTW algorithm.

2.3

Acoustic Modeling

2.3.1

General Purpose

An acoustic model G(·; θG ) parameterized by θG represents the mapping from an input⊤⊤ ⊤feature sequence x = [x⊤to output speech parameter sequence y =1 , · · · , xt , · · · , xT ]
⊤
⊤ ⊤
[y ⊤
1 , · · · , y t , · · · , y T ] , i.e., y = G(x; θG ). The frame index and total frame length are de-noted by t and T , respectively. A Dx -dimensional input feature vector and Dy -dimensional
output speech parameter vector at frame t are given as xt = [xt (1), · · · , xt (Dx )]⊤ andy t = [yt (1), · · · , yt (Dy )]⊤ , respectively. The goal with acoustic modeling is to estimate

16

Chapter 2 Statistical Speech Synthesis Using DNNs/ŶƉƵƚƚĞǆƚ;:ĂƉĂŶĞƐĞƐĞŶƚĞŶĐĞͿ



Text
analysisWŚŽŶĞŵĞͲǁŝƐĞůŝŶŐƵŝƐƚŝĐĨĞĂƚƵƌĞƐ

a

r

a

y

u

r

u

Duplication withphoneme durationůŝŐŶĞĚ
ůŝŶŐƵŝƐƚŝĐĨĞĂƚƵƌĞƐ

aaa r r aaaay yu r uuZĞĨĞƌĞŶĐĞƐƉĞĞĐŚǁŝƚŚƉŚŽŶĞŵĞďŽƵŶĚĂƌǇ
Ă

ƌ

Ă

Ǉ

Ƶ ƌ Ƶ

Fig. 2.6. Temporal alignment of features in TTS. Given phoneme boundary, eachphoneme-wise linguistic feature is duplicated to align its length with corresponding speech parameters.

model parameters θG using a training dataset (i.e., a speech corpus) that includes pairsof input features and output speech parameters.

2.3.2

Static-dynamic Feature Modeling

Static-dynamic features of speech parameters are used for considering temporal continuity.
⊤⊤ ⊤be a static-dynamic feature vector at frame t. Dynamic
Let Y t = [y ⊤t , ∆y t , ∆∆y t ]features ∆y t and ∆∆y t are calculated as1
1
y t+1 − y t−1 ,2
2∆∆y t = y t+1 − 2y t + y t−1 .
∆y t =

(2.1)
(2.2)

17

Ă

ZĞĨĞƌĞŶĐĞƐƉĞĞĐŚĂ
Ǉ Ƶ ƌ Ƶƌ

Chapter 2 Statistical Speech Synthesis Using DNNsĂƌ Ă Ǉ Ƶ ƌ/ŶƉƵƚƐƉĞĞĐŚ

Ƶ

Fig. 2.7. DTW algorithm for feature alignment in VC. Phoneme boundaries of input
and reference speech are superimposed for clear visualization.

⊤ ⊤⊤is calculated as Y =A static-dynamic feature sequence Y = [Y ⊤1 ,··· ,Y t ,··· ,Y T]

M y, where M is a 3Dy T -by-Dy T matrix used to calculate the dynamic features [24].
Figure 2.8 shows the matrix computation to obtain the static-dynamic feature sequence.

2.3.3

DNN Architectures for Acoustic Modeling

A DNN is a class of artiﬁcial neural networks with more than one hidden layer between
its input and output layers [23]. It provides a uniﬁed framework for acoustic modeling inboth TTS and VC. This section focuses on three basic DNN architectures: Feed-Forward
DNNs [21, 22], long-short term memory (LSTM) [52, 53], and VAEs [37].
Feed-Forward DNNs
Feed-Forward DNNs constitute the foundation of every DNN, which transform an input vector into an output vector through stacked nonlinear transformations. The layerwise nonlinear transformations are deﬁned as element-wise activation functions g (l) (·), l ∈{1, · · · , L + 1}, where l and L denote the layer index and number of layers in the DNNs,
respectively. A hidden vector at the lth layer h(l) is calculated using the activation function as h(l) = g (l) (W (l) h(l−1) + b(l) ), where W (l) and b(l) denote a weight matrix of
network connection and bias term at the lth layer, respectively. The input and output
vectors of the DNNs correspond to h(0) and h(L+1) , respectively. Figure 2.9 shows the18

Chapter 2 Statistical Speech Synthesis Using DNNs

ࡹ

ࢅ
૚

૙

૙

૙Ǥ ૞

οο࢟ଵ

െ૛

૚

࢟ଶ

૙

૚

૙

െ૙Ǥ ૞

૙

૙Ǥ ૞

૚

െ૛

૚

࢟ଵ
ο࢟ଵ

ࢅଵ

͵‫ܦ‬௬
οο࢟ଶ

࢟ଵ

૙

͵‫ܦ‬௬

ο࢟ଶ

ࢅଶ

࢟

͵‫ܦ‬௬ ܶ‫ܦ‬௬ ܶ
்࢟͵‫ܦ‬௬ ܶ
ͳ

‫ڰ‬
்࢟

૙

ο࢟ ்

‫ܦ‬௬

‫ڭ‬

ൌ

‫ڭ‬

ࢅ்

࢟ଶ

οο࢟ ்

૙

૚

െ૙Ǥ ૞

૙

૚

െ૛

૙
ͳ

‫ڰ‬
૙

ͳ

‫ܦ‬௬

‫ܦ‬௬

‫ܦ‬௬ ܶ

Fig. 2.8. Matrix computation to obtain static-dynamic feature sequenceࢎ ௟ିଵ݃௟ ‫ڄ‬

‫ڮ‬
ࢃ௟

ࢎ௟
‫ڮ‬

࢈௟

Fig. 2.9. Forward propagation procedure in Feed-Forward DNNs

forward propagation procedure to calculate h(l) from h(l−1) in the Feed-Forward DNNs.
The deﬁnition of the activation function plays a crucial role in the DNN framework.
The following functions are often used for the hidden layers (l = 1, · · · , L):• Sigmoid: σ(h) = 1/(1 + e−h )• Hyperbolic tangent (tanh): tanh(h) = (1 − e−h )/(1 + e−h )• Rectiﬁed linear unit (ReLU) [54]: ReLU(h) = max{0, h}
The activation function for the output layer (l = L + 1) is designed in accordance witha task to be solved. The sigmoid function is used in binary classiﬁcation tasks, e.g.,P
speaker veriﬁcation. The softmax function Softmax(h) = eh / eh is used in multi-classclassiﬁcation tasks, e.g., speech recognition and speaker classiﬁcation. The linear function
Linear(h) = h is used in regression tasks, e.g., TTS and VC.

Chapter 2 Statistical Speech Synthesis Using DNNs

19

The model parameters of Feed-Forward DNNs, i.e., the weight matrices and bias vectorsof each hidden layer, are updated by supervised learning using the BP algorithm [30]. An
output vector ŷ is ﬁrst predicted from x through the DNNs. A deﬁned loss function
L(y, ŷ) is then computed with y and ŷ. The model parameters are ﬁnally updated using
the stochastic gradient descent (SGD) algorithm with the gradient ∇θG L(y, ŷ).
LSTM
LSTM is one of the most popular recurrent neural network (RNN) architectures for sequence modeling. It has an LSTM block consisting of a memory cell C t , input gate it ,output gate ot , and forget gate f t to learn long-short-term dependencies. Using a pasthidden state ht−1 , a past memory cell C t−1 , and an input vector at current time step xt ,the next ht and C t are calculated as follows:h̃t−1 = Concat (ht−1 , xt ) ,
f t = σ W (f ) h̃t−1 + b(f ) ,
it = σ W (i) h̃t−1 + b(i) ,g t = tanh W (g) h̃t−1 + b(g) ,

(2.3)
(2.4)
(2.5)
(2.6)

C t = C t−1 ◦ f t + it ◦ g t ,ot = σ W (o) h̃t−1 + b(o) ,(2.8)

ht = tanh (C t ) ◦ ot ,

(2.9)

(2.7)

where Concat(·) and ◦ denote the concatenating operation of given vectors and the
Hadamard product (i.e., element-wise product), respectively. Information about the previous sequence is stored in C t−1 , and the forget gate f t controls the weight for C t−1 . The
input and output gates, it and ot , control the weights for g t and tanh(C t ), respectively.
Figure 2.10 shows the forward propagation procedure to calculate ht and C t in LSTM.
The model parameters of LSTM, i.e., the weight matrices W (∗) and bias vectors b(∗) ,are updated by supervised learning using the backpropagation through time (BPTT)
algorithm [55]. The inner loops of LSTM are ﬁrst unfolded, then the algorithm is run toestimate the gradients used for the training.
VAEs
VAEs are probabilistic generative models that generate x from a latent variable z. The
model parameters of VAEs, θ, are estimated by maximizing the marginal likelihood of x

20

Chapter 2 Statistical Speech Synthesis Using DNNs

࡯௧ିଵ

࡯௧

‫ڮ‬

‫ڮ‬
ƚĂŶŚ ‫ڄ‬

ࢌ௧
ߪ ‫ڄ‬

ࢎ௧ିଵ
‫ڮ‬

ߪ ‫ڄ‬

ࢍ௧

࢏௧

ߪ ‫ڄ‬

ƚĂŶŚ ‫ڄ‬

࢈௙

࢈௜

࢈௚

࢈௢

ࢃ௙

ࢃ௜

ࢃ௚

ࢃ௢

࢕௧

ࢎ௧

Concat‫ڮ‬࢞௧Fig. 2.10. Forward propagation procedure in LSTM. “Concat” denotes concatenating⊤ ⊤operation of given vectors, i.e., Concat(ht−1 , xt ) = [h⊤t−1 , xt ] .

deﬁned asZpθ (x) =pθ (x|z) pθ (z) dz,(2.10)

where pθ (z) is a prior distribution of z. Since the integral in Eq. (2.10) is intractable,
two DNNs are introduced: an encoder qϕ (z|x) and decoder pθ (x|z). The former andlatter approximate the latent variable’s true posterior pϕ (z|x) and the data’s true posterior pθ (x|z), respectively. Model parameters of the encoder and decoder are ϕ and θ,
respectively. These model parameters are estimated to maximize the variational lower
bound of the log likelihood deﬁned as
L (θ, ϕ; x) = −DKL (qϕ (z|x) || pθ (z)) + Eqϕ (z|x) [log pθ (x|z)] ,(2.11)

where DKL (·||·) denotes the Kullback-Leibler (KL) divergence between two distributions.
Both the encoder and decoder are assumed to represent the diagonal Gaussian distributions. These distributions’ mean and covariance are estimated by the DNNs. The
isotropic Gaussian distribution N (z; 0, I) is typically adopted to pθ (z) for obtaining theclosed form of the KL term in Eq. (2.11). The reparameterization trick [37] is used for the
BP algorithm to work. Figure 2.11 shows the forward propagation procedure to generatex̂ in the VAEs. The encoder ﬁrst predicts the mean and variance of the latent variable’s
posterior probability, i.e., µϕ and σ 2ϕ , from x. The reparameterization trick is then ap-

Chapter 2 Statistical Speech Synthesis Using DNNs

Input
data
࢞

ࣆథ
Encoder‫ݍ‬థ ࢠȁ࢞

Latentvariableࢠ

21

Generateddataෝ࢞
Decoder‫݌‬ఏ ࢞ȁࢠ࣌థࣕࣨ ࣕǢ ૙ǡ ࡵ
Fig. 2.11. Forward propagation procedure in VAEs. Based on reparameterization trick,
VAE latent variable z is calculated as z = µϕ + σ ϕ ◦ ϵ, where ϵ is sampledfrom N (ϵ; 0, I).

Phonemedurations

Phoneme-wise
linguistic feats͘

‫ڮ‬

‫ڮ‬

Duration
model

Frame-wise
linguistic feats͘
Acoustic
model

‫ڮ‬

‫ڮ‬

Duplication
withphonemedurations

Speechparams.

Fig. 2.12. Basic framework for DNN-based TTS

plied to emulate the latent variable sampling as z = µϕ + σ ϕ ◦ ϵ, where ϵ is sampled fromN (ϵ; 0, I). Finally, the decoder generates x̂ from z.

2.3.4

DNN-based TTS

Basic Framework
Figure 2.12 shows the basic framework for DNN-based TTS. Two DNNs are prepared:
one for the duration model and the other for the acoustic model. The former predicts
phoneme durations from phoneme-wise linguistic features. The latter generates speechparameters from the linguistic features duplicated with the phoneme durations.

22

Chapter 2 Statistical Speech Synthesis Using DNNs

Linguisticfeatures

Static-dynamic
features

෡
ࢅ

࢞

෡ଵ
ࢅ

࢞ଵ

‫ڮ‬

a
1
i
Phoneme u 㻜Continuous F0
Aperiodic
components
Voiced/unvoiced

‫ڮ‬

Frame
position

‫ڮ‬

‫ڮ‬
Mora
position

‫ڮ‬

1
2 㻜
Accent 31㻜Mel-cepstralcoefficients

෡்
ࢅ

்࢞
Acoustic modeletc.
Fig. 2.13. DNN-based acoustic model in TTS using vocoder parameters. Frame-wisestatic-dynamic features are predicted from corresponding linguistic features.

DNN-based TTS Using Vocoder Parameters
An acoustic model in DNN-based TTS using vocoder parameters [21] generates a joint
vector of U/V and static-dynamic features of mel-cepstral coeﬃcients, continuous log F0 ,and aperiodic components. Figure 2.13 shows the acoustic model representing the relation
between linguistic features and speech parameters.
DNN-based Vocoder-free TTS Using STFT Spectra
DNN-based TTS using vocoder parameters works reasonably well. However, the use of
vocoder-based parameterization sometimes causes buzziness in synthetic speech. One
method for preventing this is DNN-based vocoder-free TTS using STFT spectra [42]. An
acoustic model predicts a static feature sequence of STFT amplitude spectra from a joint
vector of linguistic features, continuous F0 , and U/V. The use of F0 parameters as input
features is eﬀective in predicting the harmonic information of amplitude spectra [42], andenables controlling the prosody of the synthetic speech intuitively. Accordingly, DNNs
for F0 parameter prediction need to be prepared in addition to the duration and acoustic
models.

Chapter 2 Statistical Speech Synthesis Using DNNs

Seen
spkrs.

23

Input
feats.

100Multi-spkr.
acoustic
model

0
1
0

ࢉ

0
0
1

Spkr.
code

Speechparams.

Fig. 2.14. DNN-based multi-speaker acoustic modeling using speaker code as speakerrepresentation

2.3.5

DNN-based VC

An acoustic model in DNN-based VC predicts static-dynamic features of target speaker’s
mel-cepstral coeﬃcients from those of source speaker’s. F0 is often linearly converted
using the F0 statistics of the source and target speech. The domains of the input andoutput features of acoustic modeling are the same in VC. Therefore, the acoustic model
can be trained to represent the mapping from the input features to the diﬀerence between
the two features. Kobayashi et al. [56, 57] proposed VC using spectral diﬀerentials thattrains an acoustic model to represent y − x = G(x; θG ), rather than y = G(x; θG ).

2.3.6

Multi-speaker Acoustic Modeling Using Speaker Codes

One can achieve multi-speaker acoustic modeling by conditioning DNNs for speech synthesis with speaker representations that control the speaker individuality of synthetic speech.
Hojo et al. [41] proposed DNN-based multi-speaker acoustic modeling using a speakercode. The speaker code c = [c(1), · · · , c(n), · · · , c(Ns )]⊤ represents the identity of one of
the seen (i.e., pre-stored) Ns speakers. The ith speaker’s identity ci is deﬁned as follows:1 if n = i
ci (n) =0 otherwise(1 ≤ n ≤ Ns ).

(2.12)

Figure 2.14 shows a conceptual diagram of DNN-based multi-speaker acoustic modelingusing speaker codes.

24

Chapter 2 Statistical Speech Synthesis Using DNNs

Similarly, Hsu et al. [38] proposed VAE-based multi-speaker acoustic modeling using
speaker codes. They assumed that z is independent of the speaker individuality, anddeﬁned the marginal likelihood of y conditioned by c as follows:Zpθ (y|c) =pθ (y|z, c) pθ (z) dz.

(2.13)The speaker-dependent speech parameter generation process is modeled based on theconditional VAE [58] framework using a speaker-independent encoder qϕ (z|y) and speakerdependent decoder pθ (y|z, c). The variational lower bound of the log likelihood is deﬁnedas
L (θ, ϕ; y, c) = −DKL (qϕ (z|y) || pθ (z)) + Eqϕ (z|y) [log pθ (y|z, c)] .

(2.14)

Since z is independent of the speaker individuality, it can be expected to represent thephonetic content of speech. Accordingly, VC can be conducted using the conditional VAEs
by feeding the target speaker’s code into the decoder. For instance, when converting source
speech parameters into those of the jth speaker, cj is fed into the decoder frame by frame.
Note that parallel speech corpora are not required because the VAEs are trained in thesame manner as in the auto-encoding process.

2.3.7

Loss Functions for DNN Training

The DNN-based acoustic model is trained to minimize a loss function calculated fromthe target speech parameters and output of the DNNs. The standard loss function is
the mean squared error (MSE) between natural and generated speech parameters. The
acoustic model in TTS or VC using vocoder parameters is trained to minimize the MSE
between a natural static-dynamic feature sequence Y and that predicted by DNNs Ŷ :


⊤ 

1
LMSE Y , Ŷ =Ŷ − YŶ − Y .
T(2.15)The MSE is also used for the duration model training in TTS. Because the dynamic
features of the STFT amplitude spectra are not considered to be modeled by the acoustic
model in DNN-based vocoder-free TTS, Y and Ŷ in Eq. (2.15) are replaced with y andŷ, respectively,
Minimum generation error (MGE) training was proposed [59, 60] to consider the staticdynamic constraint of the speech parameters. The loss function in MGE training isChapter 2 Statistical Speech Synthesis Using DNNs

25

deﬁned as the MSE between the natural and generated speech parameters after applying
the maximum likelihood parameter generation (MLPG) algorithm [61]1⊤(ŷ − y) (ŷ − y)
T
⊤ 

1
RŶ − yRŶ − y ,=T

LMGE (y, ŷ) =(2.16)

where R is a Dy T -by-3Dy T matrix deﬁned as
⊤

R= M Σ

−1M

−1M ⊤ Σ−1 .

(2.17)A 3Dy T -by-3Dy T covariance matrix Σ = diag[Σ1 , · · · , Σt , · · · , ΣT ] consists of Σt , i.e.,a 3Dy -by-3Dy covariance matrix at frame t. The covariance matrix Σ is separately estimated using a training dataset. The gradient ∇Ŷ LMGE (y, ŷ) is given as R⊤ (ŷ−y)/T [60].
The loss function in VAE-based acoustic modeling [38] is deﬁned as the negative log
likelihood of the VAEs, i.e., DKL (qϕ (z|y) || pθ (z)) − Eqϕ (z|y) [log pθ (y|z, c)]. The ﬁrst
term DKL (qϕ (z|y) || pθ (z)) can be regarded as the regularization term on z. The KL
divergence can be evaluated in closed form with the assumption of the Gaussian prior: 
DKL (qϕ (z|y) || pθ (z)) = DKL N z; µϕ , diag σ 2ϕ ||N (z; 0, I)
Dz

1X
=
1 + log σϕ2 (d) − µ2ϕ (d) − σϕ2 (d) ,2(2.18)

d=1

where Dz denotes the dimensionality of z. The second term −Eqϕ (z|y) [log pθ (y|z, c)] canbe calculated as the reconstruction error of the VAEs, i.e., LMSE (y, ŷ) [37].

2.4

Speech Parameter Generation

2.4.1

MLPG algorithm

Speech parameters are generated from the trained acoustic model. In DNN-based TTS
using vocoders, the phoneme durations of the given linguistic features are ﬁrst predictedusing the trained duration model. The static-dynamic feature sequence of the speechparameters is then predicted using this model. The MLPG algorithm is ﬁnally applied
to obtain the static features of the speech parameters. Similar steps are taken in DNNbased vocoder-free TTS, except for applying the MLPG algorithm because the acoustic
model predicts static features of the STFT amplitude spectra directly. In DNN-based VC,

26

Chapter 2 Statistical Speech Synthesis Using DNNs

the static-dynamic feature sequence of the target speech parameters is predicted using thesource speech parameters, and the MLPG algorithm is applied to obtain the static featuresof the target speech parameters.

2.4.2

GV Compensation

Although the generated speech parameters after the MLPG algorithm are temporallysmoothed, their ﬁne structures tend to vanish due to over-smoothing, which considerably
degrades synthetic speech quality. One way to prevent the ﬁne structures from vanishing is
to reproduce the statistics of the natural speech. GV compensation [62, 63] is a commonlyused technique for improving synthetic speech quality. A GV is deﬁned as the second
moment of the speech parameter sequence. A Dy -dimensional GV vector of y is calculatedusing⊤

v (y) = [v (1) , · · · , v (d) , · · · , v (Dy )] ,(2.19)1X
2(yt (d) − ⟨y (d)⟩) ,v (d) =
T t=1

(2.20)T1X⟨y (d)⟩ =yt (d) .
T t=1
T(2.21)The GV of a generated speech parameter sequence tends to be smaller than that of anatural one. The synthetic speech quality can be improved by compensating for thediﬀerence between natural and generated GVs. The generated speech parameters after
GV compensation are calculated using
s
(GV)

ŷt

(d) =

µ(GV) (d){ŷt (d) − ⟨ŷ (d)⟩} + ⟨ŷ (d)⟩ ,µ̂(GV) (d)(2.22)

where µ(GV) (d) and µ̂(GV) (d) are the dth components of the GV mean vectors of thenatural and generated speech, respectively. The mean vectors are calculated using training
data.

2.5

Speech Waveform Synthesis

This section describes two types of speech waveform synthesis: vocoder-based synthesis
and vocoder-free synthesis.

Chapter 2 Statistical Speech Synthesis Using DNNs^ƉĞĞĐŚƉĂƌĂŵĞƚĞƌƐ
&Ϭ

27KƵƚƉƵƚ
ƐƉĞĞĐŚ
Excitation
generation

MLSA
filterƉĞƌŝŽĚŝĐĐŽŵƉŽŶĞŶƚƐ
DĞůͲĐĞƉƐƚƌĂůĐŽĞĨĨŝĐŝĞŶƚƐ
Fig. 2.15. Speech synthesis using MLSA ﬁlter. Excitation signal is ﬁrst generated fromF0 and aperiodic components, then MLSA ﬁlter is applied to it for synthesizing speech waveform.

2.5.1

Vocoder-based Synthesis

Mel-log Spectrum Approximation (MLSA) Filter
The synthetic speech waveform in DNN-based speech synthesis using vocoder parameters
is synthesized from the generated speech parameters using a synthesis ﬁlter such as the
MLSA ﬁlter [64]. Figure 2.15 illustrates the speech synthesis process using the MLSA
ﬁlter.

2.5.2

Vocoder-free Synthesis

The analysis of the excitation parameters often incurs errors such as the U/V decision
error. Therefore, several vocoder-free synthesis methods have been developed for avoiding
errors and improving speech quality.
Spectral Diﬀerentials Filter
The converted speech waveform in VC using spectral diﬀerentials [56, 57] is synthesized byapplying a spectral diﬀerential ﬁlter to the input speech waveform. Figure 2.16 illustrates
the VC process using the spectral diﬀerentials.
Phase Reconstruction from Spectral Amplitudes
Only the amplitude spectra are modeled by the acoustic model in vocoder-free TTS using
STFT spectra. Therefore, phase spectra are reconstructed using the Griﬃn and Lim
algorithm [65] with the predicted amplitude spectra. Let y(n) be a speech waveform

28

Chapter 2 Statistical Speech Synthesis Using DNNs/ŶƉƵƚƐƉĞĞĐŚ

KƵƚƉƵƚ
ƐƉĞĞĐŚ
Spectraldifferentials
filterSpectrum
analysis

Spectraldifferential
estimation^ƉĞĐƚƌĂůƉĂƌĂŵĞƚĞƌƐ

^ƉĞĐƚƌĂůĚŝĨĨĞƌĞŶƚŝĂůƐ

Fig. 2.16. VC using spectral diﬀerentials. Input speech waveform is converted using
estimated spectral diﬀerentials ﬁlter.

Algorithm 2.1 Phase reconstruction from spectral amplitudes
1: set initial phase information ϕt (d) to random values
2: set initial STFT spectra Yt (d) to yt (d) exp(jϕt (d))
3: for number of iterations do
4:generate y(n) from Yt (d) using inverse STFT (ISTFT):
y(n) ← ISTFT [Yt (d)] .
5:

reconstruct Yt (d) from y(n) using STFT:
Yt (d) ← STFT [y(n)] .

6:

update Yt (d) with ﬁxed amplitude spectra yt (d):Yt (d) ← yt (d)Yt (d).
|Yt (d)|7: end forsample at time index n. The phase information for the given STFT amplitude spectrayt (d) is reconstructed in accordance with Algorithm 2.1. This algorithm can synthesize aspeech waveform without the vocoding process. Figure 2.17 illustrates the TTS processusing STFT spectra.

Chapter 2 Statistical Speech Synthesis Using DNNs

29KƵƚƉƵƚ
ƐƉĞĞĐŚ

/ŶƉƵƚ
ƚĞǆƚ
Text
analysis

Acoustic
models

G&L
algorithm

DurationmodelsF0modelsFig. 2.17. Vocoder-free TTS process using STFT spectra. “G & L” indicates “Griﬃnand Lim.”

2.6

SummaryThis chapter reviewed the basic framework of statistical speech synthesis using DNNs
as the acoustic models. The framework consists of two phases: training and synthesis.
The framework’s four crucial factors: 1) feature analysis, 2) acoustic modeling, 3) speechparameter generation, and 4) speech waveform synthesis, were described. The DNN-based
acoustic model plays an essential role in representing the complicated relation between
input features and speech parameters. Feed-Forward DNNs, LSTM, and VAEs are often
used for acoustic modeling. The model parameters of DNNs are estimated using the BP(or BPTT) algorithm to minimize the deﬁned loss function.

30

Chapter 3

Vocoder-based Statistical
Speech Synthesis Using GANs
3.1

Introduction

This chapter proposes a novel algorithm using GANs to train a DNN-based acoustic modeland prevent over-smoothing of generated speech parameters. Figure 3.1 shows a conceptual diagram of the proposed algorithm. GANs consist of two DNNs: a discriminator to
distinguish natural and generated samples, and a generator to generate samples that foolthe discriminator. A new training criterion for the acoustic model is deﬁned based on
the GAN framework. The criterion is the weighted sum of the conventional MGE and
GAN-derived adversarial loss. The adversarial loss makes the discriminator recognize thegenerated speech parameters as natural. Since the objective of GANs leads to minimize
divergence (i.e., distribution diﬀerence) between natural and generated speech parameters, this algorithm can alleviate over-smoothing and improve synthetic speech quality.
Moreover, this algorithm can be regarded as a generalization of conventional methods
that model analytically derived features such as GVs and MSs explicitly. The reason
why is that this algorithm eﬀectively minimizes the divergence without explicit statisticalmodeling of GVs or MSs. Figure 3.2 illustrates the comparison of the conventional andproposed methods. Also, the discriminator with this algorithm can be interpreted as antispooﬁng, i.e., a technique to detect synthetic speech and prevent voice spooﬁng attacks.
Accordingly, techniques and ideas concerning anti-spooﬁng can be applied to the acoustic
modeling for speech synthesis. In addition, the eﬀects of the divergences are investigatedfrom the perspective of divergence minimization by GANs. This chapter adopts leastsquares GAN (LS-GAN) and Wasserstein GAN (W-GAN) as image-processing-related

Chapter 3 Vocoder-based Statistical Speech Synthesis Using GANs
NaturalspeechMGE

11th mel-cepstral coefficient

23rd mel-cepstralcoefficient

3rd mel-cepstralcoefficient

NaturalDiscriminator(anti-spoofing)

1st mel-cepstral coefficient
ProposedωD .

Speech ( =1 0)
params.
distrs.

13th mel-cepstralcoefficient

Synthetic
speech

1st mel-cepstral coefficient

13th mel-cepstralcoefficientInputfeats.

Speechsynth.

31

Proposed(ωD =1.0)

MGE

3rd mel-cepstralcoefficient

NaturalNaturalDivergenceapproximation

21st mel-cepstral coefficient

23rd mel-cepstralcoefficient

11th mel-cepstral coefficient

Synthetic
21st mel-cepstral coefficient(a) Discriminator update

MGE

23rd mel-cepstralcoefficient

Proposed(ωD =1.0)

1st mel-cepstral coefficient

11th mel-cepstral coefficient

1st mel-cepstral coefficient

13th mel-cepstralcoefficient
23rd mel-cepstralcoefficient

13th mel-cepstralcoefficient

NaturalDiscriminator(anti-spoofing)

Proposed(ωD =1.0)

MGE

3rd mel-cepstralcoefficient

Natural

3rd mel-cepstralcoefficient

Speechsynth.

Generation error
minimization

Divergence
minimization

21st mel-cepstral coefficient
11th mel-cepstral coefficient

Natural(b) Speech synthesis update

21st mel-cepstral coefficientFig. 3.1. Conceptual diagram of proposed method in Chapter 3
Parametric statistics
compensation

Goal

GV/MS
Over-smoothedspeech params.

[25,33][15, 31]

Chapter 3(proposed)

Natural speechparams.

GAN-based distribution
compensation
Low

High
Speech quality

Fig. 3.2. Relation between conventional and proposed methods in Chapter 3

GANs, and f -divergence GAN (f -GAN) as speech-processing-related GANs, to be investigated. This algorithm’s eﬀectiveness is evaluated in DNN-based TTS or VC using
vocoder parameters.
This chapter is organized as follows (see also Fig. 3.3). Section 3.2 explains a basic
framework of GANs. Section 3.3 describes the proposed algorithm for the DNN-based
acoustic model training incorporating GANs. Sections 3.4 and 3.5 present experimental
evaluations of this algorithm in TTS and VC, respectively. Section 3.6 summarizes thischapter.

32

Chapter 3 Vocoder-based Statistical Speech Synthesis Using GANs
Conventional
methods

Chapter 3: GAN-based high-quality speech synth.
Sect. 3.3: proposed methods

DNN-based
TTS/VC[21, 22]

Integrating
GAN-based
TTS/VCGANs [66]Sect. 3.2
GV/MS

Different
viewpoint

Mathematicalextension

Theoreticalextension[25, 33]

Anti-spoofing
integration

Variousdivergence
minimization

Sects. 3.4 & 3.5: evaluations in TTS & VC
GAN vs. MGE [60]Effect of antispoofing integration

GAN vs. GV [25]Investigation ofeffective divergenceFig. 3.3. Overview of Chapter 3

3.2

GANs

3.2.1

Objective of GANs

GAN [66] is a class of deep generative models that simultaneously trains two DNNs:
a generator and discriminator D(y; θD ). The discriminator’s model parameters θD aregiven as neural networks. The value obtained by taking the sigmoid function from thediscriminator’s output, 1/(1 + exp(−D(y))), represents the posterior probability that y isa natural sample. The discriminator is trained to make the posterior probability 1 for y
and 0 for generated ŷ. Meanwhile, the generator is trained to fool the discriminator, i.e.,
it tries to make the discriminator make the posterior probability 1 for ŷ. The two DNNs
in the GAN training are iteratively updated by minibatch SGD. Figure 3.4 illustrates aconceptual diagram of the GAN framework.

Chapter 3 Vocoder-based Statistical Speech Synthesis Using GANs

33

1

‫ڮ‬

0.5

0

࢟
࢞

Training iterations
ෝDistribution of ࢟Distribution of ࢟

Posterior prob. of ‫࢟ ܦ‬

Fig. 3.4. GAN framework. Discriminator is trained to distinguish y and ŷ, while generator is trained to fool it. Here, ŷ is generated from x through generator.

3.2.2

Discriminator Training

The discriminator is trained to distinguish y from ŷ by minimizing the discriminator loss
(GAN)

LD

(y, ŷ) deﬁned as


T
1
1
1X
log
−
log 1 −.
T t=1
1 + exp (−D (y t )) T t=1
1 + exp (−D (ŷ t ))

1
(GAN)
LD
(y, ŷ) = −

T
X

(3.1)
(GAN)

The stochastic gradient ∇θD LD(y, ŷ) is used for updating θD . Figure 3.5 illustrates

the procedure for computing the discriminator loss and its gradient.

3.2.3

Generator Training

The generator is trained to fool the updated discriminator by minimizing the adversarial(GAN)

loss LADV (ŷ) deﬁned as
1X
1
log
.
T t=1
1 + exp (−D (ŷ t ))
T

(GAN)

LADV (ŷ) = −

(3.2)

(GAN)

The stochastic gradient ∇θG LADV (ŷ) is used for updating the generator’s model parameters θG . Goodfellow et al. [66] showed this adversarial framework minimizes theapproximated Jensen–Shannon (JS) divergence between distributions of y and ŷ.

34

Chapter 3 Vocoder-based Statistical Speech Synthesis Using GANs

Linguisticfeats.
Param.
Gen.
ࡾ

Generated
speechparams.
or

Naturalspeechparams.

ෝ
࢟

࢟

෡
ࢅ

࢞

ሺୋ୅୒ሻ

Acoustic
model




‫ܮ‬ୈ

Discriminator
ෝ
߲ࣘ ࢟
ෝ
߲࢟

or

߲ࣘ ࢟
߲࢟

Feat.
function
ࣘ ‫ڄ‬

ሺୋ୅୒ሻ

߲‫ܮ‬ୈ
ෝ
߲࢟

0 or 1

ሺୋ୅୒ሻ

߲‫ܮ‬ୈ
or
߲࢟

Fig. 3.5. Loss function and gradients for updating discriminator. “Param. Gen.” indicates speech parameter generation using MLPG algorithm. Acoustic model’s
parameters are not updated in this step.

3.3

Acoustic Model Training Using GANs

This section describes a novel training algorithm for a DNN-based acoustic model incorporating GANs.

3.3.1

Acoustic Model Training Criteria Incorporating GANs

The acoustic model with the proposed algorithm is trained to fool the discriminator that
distinguishes natural and generated speech parameters. The loss function of the acoustic
model training is deﬁned as follows:
LG (y, ŷ) = LMGE (y, ŷ) + ωD

ELMGE (GAN)(ŷ) .
L
ELADV ADV

(3.3)

(GAN)

The second term LADV (ŷ) is the GAN-derived adversarial loss to make the discriminator recognize the generated speech parameters as natural and minimize the divergence
between the distributions of natural and generated speech parameters. Therefore, theproposed algorithm not only minimizes the generation error but also makes the distribution of generated speech parameters close to that of natural ones. The ELMGE and ELADV(GAN)

denote the expectation values of LMGE (y, ŷ) and LADV (ŷ), respectively. Their ratioChapter 3 Vocoder-based Statistical Speech Synthesis Using GANs

෡
߲ࢅ
߲࢞

߲‫୑ܮ‬ୋ୉ෝ െ ࢟ሻൌ ࡾୃ ሺ࢟
෡߲ࢅGenerated
speechparams.

Linguisticfeats.
Param.
Gen.
ࡾ
෡
ࢅ

࢞
Acoustic
model

35



‫୑ܮ‬ୋ୉Naturalspeechparams.

ෝ
࢟

࢟
ሺୋ୅୒ሻ

ෝ
߲࢟
ൌ ࡾୃ
෡
߲ࢅ

‫ܮ‬୅ୈ୚

1
Discriminatorሺୋ୅୒ሻ߲‫ܮ‬୅ୈ୚
ෝ߲࢟

Fig. 3.6. Loss functions and gradients for updating acoustic model in proposed GANbased algorithm. Model parameters of discriminator are not updated in thisstep.

ELMGE /ELADV is the scale normalization term between the two loss functions. The hyperparameter ωD controls the weight of the second term. When ωD = 0, the loss function isequivalent to the conventional MGE described in Section 2.3.7, and when ωD = 1, the twoloss functions have equal weights. The acoustic model parameters θG are updated using
the stochastic gradient ∇θG LG (y, ŷ). Figure 3.6 illustrates the procedure for computing
the proposed loss function and its gradient.
Algorithm 3.1 describes the iterative optimization for the acoustic model and discriminator in the proposed algorithm. When one module is being updated, the model parametersof the other are ﬁxed. For example, the discriminator’s model parameters θD are not
updated by the BP algorithm for the acoustic model.

3.3.2

Integrating Anti-spooﬁng Techniques

The discriminator with the proposed algorithm can be regarded as DNN-based antispooﬁng [29, 67] that distinguishes natural speech from synthetic speech. From thisperspective, a feature function ϕ(·) can be applied to natural or generated speech parameters as shown in Fig. 3.5. The function calculates more distinguishable features inanti-spooﬁng than the direct use of speech parameters themselves; i.e., ϕ(y) and ϕ(ŷ)
are used instead of y and ŷ in Eqs. (3.1) and (3.2), respectively. The stochastic gradient∂ϕ(ŷ)/∂ ŷ is used for the BP algorithm in the acoustic model training.

36

Chapter 3 Vocoder-based Statistical Speech Synthesis Using GANs

Algorithm 3.1 Iterative optimization for acoustic model and discriminator
1: η := learning rate
2: for number of training iterations do
3:
for all training data (x, y) do
4:generate ŷ from the acoustic models:ŷ = G(x; θG ).
5:

update θD while ﬁxing θG :(GAN)

θD ← θD − η∇θD LD
6:(y, ŷ).

update θG while ﬁxing θD :
θG ← θG − η∇θG LG (y, ŷ).

7:
end for
8: end for

The dynamic features of spectral parameters can be used as the eﬀective features todetect synthetic speech [68]. The feature function is deﬁned as ϕ(ŷ) = M ŷ, and thegradient M ⊤ is used for the BP algorithm. One can use other features to be incorporated
into the proposed algorithm. Because the vocoder systems are based on a minimumphase vocal tract model, the diﬀerences between phase spectra of natural and synthetic
speech can be used [69]. Features related to F0 statistics are also eﬀective to detect
spooﬁng attacks [70, 71] owing to the diﬃculty in reliable prosody modeling. Temporal
magnitude/phase modulation features can be used [72] to capture long-term dependenciesof speech parameters.

3.3.3

Duration Model Training Considering Isochrony

The proposed algorithm can be applied to the spectral parameters and continuous F0 generation straightforwardly. One can also apply the proposed algorithm to phoneme duration
generation directly. However, naturally-distributed phoneme durations do not guarantee
to have natural isochrony of the target language (e.g., moras in Japanese) [73]. Therefore,
the proposed algorithm for duration generation can be modiﬁed so that the generated durations naturally distribute in the language-dependent isochrony level. Figure 3.7 shows
the architecture to calculate the isochrony-level durations from the phoneme durations. In
the case of Japanese, which has mora isochrony, each mora duration is calculated from thecorresponding phoneme durations.

The discriminator is trained to approximate the di-Chapter 3 Vocoder-based Statistical Speech Synthesis Using GANs

Phonemedur.

Isochrony-level
dur.

sil

sil

sil

a

a

a

r

ra

a
y

yu

r
a
y

…

DNN-based
duration
model…

Linguisticfeats.

u

u

…

…

sil

sil

37

Languagedependent
poolingsilFig. 3.7. Architecture to calculate isochrony-level duration from phoneme duration. In
case of Japanese, which has mora isochrony, each mora duration is calculated
from corresponding phoneme durations. For example, mora duration of /ra/is calculated as sum of phoneme durations of /r/ and /a/.

Syllable Transformation Phonemedur.
matrixdur. Phoneme
ϭ݀ଵϬSyllable (Isochrony)
ϭ ϭ
݀ଶ
Word
…

=

݀ଷ

Ϭ

Sentence
…

ϭ

…
…

…

…

1 denotes phonemes
within one syllable݀௉

Parse tree
Fig. 3.8. Matrix representation to calculate isochrony-level durations. This is example
in syllable-timed language case such as Chinese.

vergence between natural and predicted durations considering the isochrony. Meanwhile,the duration model is trained to minimizes the weighted sum of the MSE between naturaland predicted phoneme durations and the adversarial loss using the isochrony-level durations. Since the calculation of the isochrony-level duration is represented as the matrix
multiplication shown in Fig. 3.8, the BP algorithm can be done using the transpose of thetransformation matrix.

38

Chapter 3 Vocoder-based Statistical Speech Synthesis Using GANs

3.3.4

Various Divergences Minimized by GANs

The GAN framework works as a divergence minimization between natural and generated
speech parameters. As described in Section 3.2.1, the original GAN [66] minimizes the approximated JS divergence. From the perspective of the divergence minimization, this section discusses additional GANs minimizing other divergences: f -GAN [74], W-GAN [75],and LS-GAN [76]. The divergences encompassed by f -GAN are strongly related to speechprocessing techniques such as nonnegative matrix factorization [77, 78]. The eﬀectivenessof W-GAN and LS-GAN in the image processing is well known. The discriminator loss(∗-GAN)LD(∗-GAN)(y, ŷ) and adversarial loss LADV(ŷ) introduced below can be used instead of

Eqs. (3.1) and (3.2), respectively.
f -GANf -GAN [74] is the uniﬁed framework that encompasses the original GAN [66]. The
diﬀerence between distributions of natural and generated data is deﬁned as the f divergence [79], a large class of diﬀerent divergences including the KL and JS divergences.
The f -divergence Df (y∥ŷ) is deﬁned as follows:Z
Df (y∥ŷ) =q (ŷ) fp (y)q (ŷ)


dy,

(3.4)

where p(·) and q(·) are absolutely continuous density functions of y and ŷ, respectively.
The f (·) is a convex function satisfying f (1) = 0. Although various choices of f (·) forrecovering popular divergences are available, this section adopts speech-processing-related
ones.
KL-GAN: Deﬁning f (r) = r log r gives the KL divergence as follows:Z
DKL (y∥ŷ) =(KL-GAN)The discriminator loss LD(KL-GAN)LD

p (y) log

p (y)
dy.
q (ŷ)(3.5)(y, ŷ) is deﬁned as follows:(y, ŷ) = −

T
T
1X
1X
D (y t ) +exp (D (ŷ t ) − 1) ,
T t=1
T t=1

(3.6)

Chapter 3 Vocoder-based Statistical Speech Synthesis Using GANs
(KL-GAN)

while the adversarial loss LADV39(ŷ) is deﬁned as follows:
1X
D (ŷ t ) .
T t=1
T(KL-GAN)LADV(ŷ) = −(3.7)

Reversed KL (RKL)-GAN: Since the KL divergence is asymmetric, the reversed
version, called the RKL divergence DRKL (y∥ŷ) diﬀers from DKL (y∥ŷ) as follows:Z
DRKL (y∥ŷ) =q (ŷ) log

q (ŷ)
dy = DKL (ŷ∥y) .
p (y)(RKL-GAN)

Deﬁning f (r) = − log r gives the discriminator loss LD(RKL-GAN)LD(y, ŷ) =(y, ŷ) as follows:

T
T
1X
1X
exp (−D (y t )) +(−1 + D (ŷ t )) ,
T t=1
T t=1

(RKL-GAN)

while the adversarial loss LADV

(3.8)

(3.9)

(ŷ) is deﬁned as follows:
1X
exp (−D (ŷ t )) .
T t=1
T

(RKL-GAN)
LADV(ŷ) =(3.10)

JS-GAN: The JS divergence without approximation can be formed within the f -GAN
framework. Deﬁning f (r) = −(r + 1) log r+1
2 + r log r gives the JS divergence as follows:
1
DJS (y∥ŷ) =2Z

2p (y)
1p (y) logdy +p (y) + q (ŷ)
2(JS-GAN)The discriminator loss LDZ
q (ŷ) log

2q (ŷ)dy.
p (y) + q (ŷ)(3.11)(y, ŷ) is deﬁned as follows:



T
2
1X
2
log
−
log 2 −,
T t=1
1 + exp (−D (y t )) T t=1
1 + exp (−D (ŷ t ))1
(JS-GAN)LD(y, ŷ) = −TX(3.12)(JS-GAN)

while the adversarial loss LADV(ŷ) is deﬁned as follows:

1(JS-GAN)LADV(ŷ) = −

T
X

T t=1

log

2
.
1 + exp (−D (ŷ t ))(3.13)

40

Chapter 3 Vocoder-based Statistical Speech Synthesis Using GANs

Note that, the approximated JS divergence minimized by the original GAN is 2DJS (y∥ŷ)−log(4) [66].
W-GAN
Arjovsky et al. [75] proposed W-GAN that minimizes the earth-mover’s distance(Wasserstein-1) to stabilize unstable training of the original GAN. The earth-mover’s
distance is deﬁned as follows:
DEM (y, ŷ) = inf E(y,ŷ)∼γ [∥y − ŷ∥] ,(3.14)

γ

where γ(y, ŷ) is the joint distribution whose marginals are respectively the distributionsof y and ŷ. On the basis of the Kantorovich–Rubinstein duality [80], the discriminator(W-GAN)loss LD(y, ŷ) is deﬁned as follows:
1X
D (y t ) +D (ŷ t ) ,
T t=1
T t=1

1
(W-GAN)LD(y, ŷ) = −(W-GAN)

while the adversarial loss LADV

T
X

T

(3.15)(ŷ) is deﬁned as follows:

1(W-GAN)LADV(ŷ) = −

T
X

T t=1

D (ŷ t ) .

(3.16)The discriminator is assumed to be the K-Lipschitz function. Therefore, the discriminator’s weight parameters are clamped to a ﬁxed interval such as [−0.01, 0.01] after updating
the discriminator.
LS-GAN
Mao et al. [76] proposed LS-GAN that formulates the objective function based on the
MSE. The MSE-based objective function can avoid the gradient vanishing problem of the(LS-GAN)original GAN that uses the sigmoid cross entropy. The discriminator loss LD(y, ŷ)is deﬁned as follows:(LS-GAN)LD(y, ŷ) =

T
T
1 X
1 X
2
2
(D (y t ) − b) +(D (ŷ t ) − a) ,2T t=1
2T t=1(3.17)

Chapter 3 Vocoder-based Statistical Speech Synthesis Using GANs
(LS-GAN)(ŷ) is deﬁned as follows:(LS-GAN)(ŷ) =

while the adversarial loss LADV

1 X
2(D (ŷ t ) − c) ,2T t=1

41

T

LADV

(3.18)

where a, b, and c denote the labels that make the discriminator recognize the generated
data as generated, the natural data as natural, and the generated data as natural, respectively. When these labels satisfy the conditions b − c = 1 and b − a = 2, the divergence tobe minimized corresponds to the Pearson X 2 divergence between p(y) + q(ŷ) and 2q(ŷ).
Because results of preliminary experiments showed that these conditions degraded synthetic speech quality, alternative conditions suggested in Eq. (9) of [76], i.e., a = 0, b = 1,
and c = 1, were used in this thesis.

3.3.5

Discussion

The proposed loss function (Eq. (3.3)) is the combination of a multi-task learning algo(GAN)

rithm using a discriminator [81] and GANs. In deﬁning LG (y, ŷ) = LADV (ŷ), the loss
function is equivalent to that for GANs. The proposed algorithm considers a fully supervised setting unlike GANs, i.e., it utilizes the referred input and output parameters [82]without latent variables for the training. Since only the BP algorithm is used for training,a variety of DNN architectures such as LSTM [83] can be used for the acoustic model anddiscriminator.
The use of the feature function ϕ(·) for the proposed algorithm enables choosing not
only analytically derived features (e.g., GVs and MSs) but also automatically derived
features (e.g., auto-encoded features [84]) to be reproduced. However, the features thatare eﬀective in anti-spooﬁng may degrade synthetic speech quality because they do notalways relate to human speech perception.
The proposed algorithm makes the distribution of generated speech parameters close tothat of natural ones. Figure 3.9 plots natural and generated speech parameters with several mel-cepstral coeﬃcient pairs to illustrate this eﬀect clearly. The proposed algorithm
widens the distributions of generated speech parameters as much as those of naturalones, whereas “MGE” does not. This eﬀect is strongly observed in the distribution ofhigher-order of mel-cepstral coeﬃcients. A noteworthy fact is that the proposed algorithm compensates for the distribution diﬀerences between natural and generated speechparameters better than “MGE-GV,” which can be expected to improve synthetic speech
quality.

42

Chapter 3 Vocoder-based Statistical Speech Synthesis Using GANs

MGE

MGE-GV

Proposed(ωD = 1.0)

3rd mel-cepstralcoefficient

Natural11th mel-cepstral coefficient

23rd mel-cepstralcoefficient

13th mel-cepstralcoefficient

1st mel-cepstral coefficient

21st mel-cepstral coefficientFig. 3.9. Scatter plots of mel-cepstral coeﬃcients with several pairs of orders. Fromleft, ﬁgures correspond to natural speech, conventional MGE algorithm, conventional MGE algorithm with GV compensation, and proposed algorithm(ωD = 1.0), respectively. These mel-cepstral coeﬃcients were extracted fromone utterance of evaluation data.

One can explore what components (e.g., analytically derived features and intuitive
reasons [85]) the proposed algorithm changes. Firstly, GVs of natural and generated
mel-cepstral coeﬃcients were calculated. Figure 3.10 shows the calculation results. The
proposed algorithm reduced the GV diﬀerence between natural and generated speechparameters. This is a quite natural result because compensating for the distribution
diﬀerences is related to minimizing the moment diﬀerences [86, 87]. Secondly, a maximal information coeﬃcient (MIC) [88] of speech parameters was calculated to quantify
a nonlinear correlation among them. Figure 3.11 shows the calculation results. Weakcorrelations are observed among the natural speech parameters as reported in [89]. On
the other hand, strong correlations are observed among speech parameters generated
by the conventional MGE algorithm. The proposed algorithm weakens the correlations
among generated speech parameters. These results indicate that the proposed algorithm
reproduces not only natural GVs but also natural correlation among speech parameters.
Finally, the statistics of continuous F0 , phoneme duration, and mora duration were calcu-

Chapter 3 Vocoder-based Statistical Speech Synthesis Using GANs

Global variance

101
10

43

NaturalProposed (ωD =1.0)
MGE

0

10-1
10-2
10-3
10-4

0

5

10
15
Feature index

20

Fig. 3.10. Averaged GVs of natural and generated mel-cepstral coeﬃcients. Blackdashed line denotes GVs of natural mel-cepstral coeﬃcients. Gray and bluelines correspond to GVs of mel-cepstral coeﬃcients generated by conventional
MGE and proposed algorithms, respectively.

Natural24

Proposed(ωD =1.0)

MGE

1.0
0.8

18

0.6

12

0.4

6
0

0.2
0

6

12

18

24 0

6

12

18

24 0

6

12

18

24

0.0

Fig. 3.11. MICs of natural and generated mel-cepstral coeﬃcients. MIC ranges from
0.0 to 1.0, and two variables with strong correlation have value closer to 1.0.
From left, ﬁgures correspond to natural speech parameters, generated ones
by conventional MGE algorithm, and generated ones by proposed algorithm,
respectively. These MICs were calculated from one utterance of evaluation
data.

lated. Tables 3.1, 3.2, and 3.3 list the calculation results regarding continuous F0 , phoneme
duration, and mora duration, respectively. The bold values in these tables are the closest
to natural statistics in the results.

In Tables 3.2 and 3.3, “Proposed (phoneme)” and“Proposed (mora)” are the proposed algorithms using phoneme and mora durations to
calculate the adversarial loss, respectively. These tables indicate that the proposed algorithm also makes the statistics closer to those of natural speech than the conventionalalgorithm. In the results concerning duration generations, “Proposed (mora)” tends to44

Chapter 3 Vocoder-based Statistical Speech Synthesis Using GANs
Table 3.1. Statistics of natural (“Natural”) and generated (“MGE” and “Proposed”)
continuous F0

Mean

Variance

Natural4.87840.076853

MGE

4.83880.032841

Proposed (ωD = 1.0)

4.84100.032968

Table 3.2. Statistics of natural (“Natural”) and generated (“MSE” and “Proposed(*)”)
phoneme duration

Mean

Variance

Natural16.314126.20MSE

14.96747.665

Proposed (phoneme, ωD = 1.0)

14.96375.471

Proposed (mora, ωD = 1.0)

15.07473.207

Table 3.3. Statistics of natural (“Natural”) and generated (“MSE” and “Proposed(*)”)
mora duration

Mean

Variance

Natural25.141131.93MSE

23.49260.891

Proposed (phoneme, ωD = 1.0)

24.79496.828Proposed (mora, ωD = 1.0)

24.97896.682

reduce the mean diﬀerence rather than the variance diﬀerence.
The proposed algorithm for spectrum and F0 generation (Section 3.2.3) learns the joint
distribution of them. Therefore, one can perform the distribution compensation considering correlations [90] between diﬀerent features. Also, the dimensionality diﬀerences [91]can be applied to fool the discriminator. Since the time resolutions in a phoneme duration and mora duration are diﬀerent, the proposed algorithm considering the languagedependent isochrony is related to multi-resolution GAN [92] and hierarchical duration
modeling [93].
Regarding related work, Kaneko et al. [94] proposed a GAN-based post-ﬁlter for TTS.
The post-ﬁltering has high portability because it is independent of original speech synthesis
procedures. However, it comes at a high computational cost and has a heavy disk footprintChapter 3 Vocoder-based Statistical Speech Synthesis Using GANs

45

Table 3.4. Architectures of DNNs used in TTS evaluations. Feed-Forward DNNs wereused for all architectures

Spectralparameter
generation(through Sections
3.4.2 and 3.4.4)

Spectral and F0
parameter
generation(Section 3.4.5)

Duration
generation(Section 3.4.6)

Acoustic model

274–400×3–75

442–512×3–94

442–512×3–94

Discriminator

25–200×2–1

26–256×3–1

1–256×3–1

Duration model

N/A

439–256×3–1

439–256×3–1

in the speech synthesis phase. In contrast, the proposed algorithm does not require
additional resources (i.e., computational cost and other modules) while achieving highquality speech synthesis.

3.4

Experimental Evaluations for TTS

3.4.1

Conditions for TTS Evaluation

A speech corpus of a male speaker was used. The speaker uttered 503 phonetically balanced sentences [95]. The numbers of sentences for the training and evaluation were 450(subsets A to I) and 53 (subset J), respectively. Speech signals were sampled at a rateof 16 kHz. The shift length was set to 5 ms. The 0th-through-24th mel-cepstral coeﬃcients were used as spectral parameters. F0 and 5 band-aperiodicity [44, 96] were used
as excitation parameters. The STRAIGHT vocoder [47] was used for speech parameter
extraction and speech waveform synthesis. Speech parameter trajectory smoothing [97]with a 50 Hz cutoﬀ modulation frequency was applied to the spectral parameters in the
training data for improving training accuracy. Eighty percent of the silent frames were
removed from the training data.
Table 3.4 lists the DNN architectures used for the evaluation. The ReLU [54] was used
for the activation function for the hidden layers. The linear function was used for the output activation function of the acoustic model and duration model. The sigmoid function
was used for the output activation function of the discriminator. In the spectral parameter
generation (Sections 3.4.2 through 3.4.4), the acoustic model predicted a static-dynamic
feature sequence of the mel-cepstral coeﬃcients (75-dim.) from the 274-dimensional linguistic features frame by frame, and the discriminator used frame-wise static mel-cepstral

46

Chapter 3 Vocoder-based Statistical Speech Synthesis Using GANs

coeﬃcients (25-dim.). Since F0 , band-aperiodicity, and duration of natural speech were directly used for the speech waveform synthesis, some of the prosody-related features, suchas the accent type, were included in the linguistic features. In the spectral parameterand F0 generation (Section 3.4.5), the acoustic model predicted a static-dynamic feature
sequence of the mel-cepstral coeﬃcients, continuous log F0 [46], and band-aperiodicity
with U/V (94-dim.) from the 442-dimensional linguistic features frame by frame, andthe discriminator used the joint vector of the frame-wise static mel-cepstral coeﬃcientsand continuous log F0 (26-dim.). In the duration generation (Section 3.4.6), the duration
model was trained to predict phoneme durations from corresponding linguistic features(439-dim).
In the training phase, speech parameters and real-valued linguistic features were normalized to have zero-mean and unit-variance. The training algorithm based on minimizing
the MSE (Eq. (2.15)) [21] was ﬁrst performed with 25 iterations, then the conventional
MGE training algorithm [60] was performed with 25 iterations for the initialization of theacoustic model. Here, “iteration” means using all the training data (450 utterances) once
for the training. The discriminator was initialized using natural speech parameters andgenerated ones after the MGE training. The number of iterations for the discriminatorinitialization was 5. The proposed algorithm was ﬁnally performed with 25 iterations using the initialized acoustic model and discriminator. The expectation values ELMGE andELADV were estimated at each iteration. The optimization algorithm was AdaGrad [98].
The learning rate was set to 0.01.

3.4.2

Objective Evaluation of Spectral Parameter Generation

The parameter generation loss deﬁned in Eq. (2.16) and spooﬁng rate were calculated toevaluate the proposed algorithm objectively. The spooﬁng rate is the number of spooﬁng
synthetic speech parameters divided by the total number of synthetic speech parametersin the evaluation data. Here, “spooﬁng synthetic speech parameter” indicates a generated
speech parameter that fools the discriminator. The discriminator for the spooﬁng rate
calculation was trained using natural speech parameters and generated speech parametersof the conventional MGE training. The generation loss and spooﬁng rate were calculatedwith various settings of the hyperparameter ωD .
Figure 3.12 shows the evaluation results. The generation loss monotonically increases
as ωD increases from 0.0 to 0.4. However, this tendency is not observed when ωD > 0.4.
On the other hand, the spooﬁng rate signiﬁcantly increases as ωD increases from 0.0 to
0.2 and becomes almost nearly 1.0 when ωD > 0.3. These results demonstrate that theChapter 3 Vocoder-based Statistical Speech Synthesis Using GANs

47

0.75
Generation loss

0.70
0.65

0.60
0.55

0.50
0.45
1.0
Spoofing rate

0.8

1.000

0.6

0.995

0.4

0.990

0.2
0.00.0

0.985
0.980

0.2

0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0

0.4

ωD

0.6

0.8

1.0

Fig. 3.12. Parameter generation loss (above) and spooﬁng rate (below) curves against
various settings of hyperparameter ωD

proposed algorithm increases the generation loss, but it can train the acoustic model
to fool the discriminator. In other words, the proposed algorithm tries to reduce thestatistical diﬀerence between natural and generated speech parameters, although it does
not necessarily decrease the generation error.

3.4.3

Investigation of Convergence

The proposed algorithm was performed with 100 iterations to investigate the convergenceproperty. Figure 3.13 plots the generation loss and adversarial loss curves for the training
and evaluation data. This ﬁgure indicates that both loss values are almost monotonically48

Chapter 3 Vocoder-based Statistical Speech Synthesis Using GANs

1.3

Training
Evaluation

Generation loss

1.2
1.1
1.0
0.9
0.8

Adversarial loss

0.7
0.6
0.95
0.90
0.85
0.80
0.75
0.70
0.65
0.60
0.55
1

Training
Evaluation

25

50
Iteration

75

100

Fig. 3.13. Parameter generation loss (above) and adversarial loss (below) curves fortraining data (blue-dashed line) and evaluation data (red line)

decreased in training. Although the loss values for the evaluation data strongly vary aftera few iterations, they can converge after several more iterations.

3.4.4

Subjective Evaluation of Spectral Parameter Generation

Preference AB tests were conducted to evaluate the quality of speech produced by the
proposed algorithm. Speech samples were generated with three algorithms:
MGE: conventional MGE algorithm (= Proposed (ωD = 0.0))
Proposed (ωD = 0.3): proposed algorithm using least hyperparameter setting that
achieves spooﬁng rate > 0.99

Chapter 3 Vocoder-based Statistical Speech Synthesis Using GANs

Proposed(ωD =0.3)
MGE
0.0
Proposed(ωD =1.0)
MGE49(a) MGE vs. Proposed (ωD =0.3)

0.2
0.4
0.6
0.8
(b) MGE vs. Proposed (ωD =1.0)

1.0

0.0
0.2
0.4
0.6
0.8
1.0
(c) Proposed (ωD =0.3) vs. Proposed (ωD =1.0)
Proposed(ωD =1.0)
Proposed(ωD =0.3)
0.0
0.2
0.4
0.6
0.8
1.0
Preference score
Fig. 3.14. Preference scores of speech quality with 95% conﬁdence intervals (spectral
parameter generation in TTS). From top, numbers of listeners were 22, 24,
and 22, respectively.

Proposed (ωD = 1.0): proposed algorithm using standard hyperparameter setting
Pairs of speech samples generated by each algorithm were presented to listeners in random order. Listeners participated in the assessment by using crowdsourced subjective
evaluation systems.
Figure 3.14 shows the evaluation results. In Figs. 3.14(a) and (b), the proposed algorithm signiﬁcantly outperforms the conventional MGE algorithm in both hyperparametersettings. These results demonstrate that the proposed algorithm robustly yields signiﬁcant improvement in terms of synthetic speech quality regardless of its hyperparametersetting. Henceforth, the hyperparameter ωD was set to 1.0 for the following evaluations
because Fig. 3.14(c) shows that the score of “Proposed (ωD = 1.0)” was slightly better
than that of “Proposed (ωD = 0.3).”

50

Chapter 3 Vocoder-based Statistical Speech Synthesis Using GANs

Proposed(sp+F0)(a) MGE vs. Proposed (sp+F0)

MGE
0.0
Proposed(sp+F0)
Proposed
(sp)
0.0

0.2
0.4
0.6
0.8
1.0
(b) Proposed (sp) vs. Proposed (sp+F0)

0.2

0.4
0.6
Preference score

0.8

1.0

Fig. 3.15. Preference scores of speech quality with 95% conﬁdence intervals (spectral
parameter and F0 generation in TTS). From top, numbers of listeners were
19 and 28, respectively.

3.4.5

Subjective Evaluation of F0 Generation

The eﬀectiveness of the proposed algorithm for F0 generation was investigated. Preference
AB tests were conducted using the following three algorithms:
MGE: conventional MGE algorithm
Proposed (sp): proposed algorithm applied to only spectral parameters
Proposed (sp+F0): proposed algorithm applied to both spectral parameters and F0
Pairs of speech samples generated by each algorithm were presented to listeners in random
order. Here, “Proposed (sp)” was not compared with “MGE” since Fig. 3.14 has alreadydemonstrated that the proposed algorithm improved synthetic speech quality in spectral
parameter generation. Listeners participated in the assessment by using crowdsourced
subjective evaluation systems.
Figure 3.15 shows the evaluation results. The score of “Proposed (sp+F0)” is signiﬁcantly higher than those of “Proposed (sp)” and “MGE.” These results demonstrate theeﬀectiveness of the proposed algorithm not only for spectral parameters but also for F0 .

Chapter 3 Vocoder-based Statistical Speech Synthesis Using GANs

Proposed(mora)

51(a) MSE vs. Proposed (mora)

MSE
0.0
Proposed(phoneme)

0.2
0.4
0.6
0.8
(b) MGE vs. Proposed (phoneme)

1.0

MSE
0.0
0.2
0.4
0.6
0.8
1.0
(c) Proposed (mora) vs. Proposed (phoneme)
Proposed(phoneme)
Proposed(mora)
0.0
0.2
0.4
0.6
0.8
1.0
Preference score
Fig. 3.16. Preference scores of speech quality with 95% conﬁdence intervals (duration
generation in TTS). From top, numbers of listeners were 19, 20, and 21,
respectively.

3.4.6

Subjective Evaluation of Duration Generation

The eﬀectiveness of the proposed algorithm in duration generation was investigated. Preference AB tests were conducted using the following three algorithms:
MSE: conventional MSE algorithm
Proposed (phoneme): proposed algorithm applied to phoneme duration
Proposed (mora): proposed algorithm applied to mora duration
Figure 3.16 shows the evaluation results. There are no signiﬁcant diﬀerences in the
resulting scores. The classiﬁcation accuracy of DNN-based anti-spooﬁng was calculated
to investigate the reason. The anti-spooﬁng was trained to distinguish natural speechparameters from generated ones by the conventional MSE or MGE algorithm. The pro-

52

Chapter 3 Vocoder-based Statistical Speech Synthesis Using GANs

sp+F0
phoneme
mora
0.0

0.2

0.4
0.6
Accuracy

0.8

1.0

Fig. 3.17. Classiﬁcation accuracy of anti-spooﬁng. “sp+F0”, “phoneme”, and “mora”
denote using spectral parameters and F0 , phoneme durations, and mora durations to discriminate natural and synthetic speech, respectively.

posed algorithm can be expected to work eﬀectively when the classiﬁcation accuracy ishigher; i.e., the statistical diﬀerences between natural and generated speech parameters
are larger. Figure 3.17 shows the classiﬁcation accuracy. The accuracy of anti-spooﬁng
using phoneme or mora duration is lower than that using spectral parameters and F0 .
These results suggest that the distribution compensation by the proposed algorithm does
not work well in duration generation. Henceforth, the proposed algorithm was not applied
to duration generation.

3.4.7

Comparison to GV Compensation

Figure 3.10 demonstrated that the proposed algorithm reproduced GVs of natural
speech parameters.

Here, this algorithm was compared with GV compensation toinvestigate whether the former improves speech quality more than the latter. The GV
post-ﬁltering [63] was applied to the spectral parameters and F0 generated by the MGE
training. A preference AB test involving 29 listeners was conducted using crowdsourced
subjective evaluation systems.
Figure 3.18 shows the evaluation result. The score of “Proposed” is signiﬁcantly higher
than that of the conventional GV post-ﬁlter (“MGE-GV”). This result shows that the
proposed algorithm produces more gain in speech quality than GV compensation.

3.4.8

Eﬀect of Feature Function

The eﬀectiveness of the feature function for anti-spooﬁng was investigated. The following
two functions were compared:

Chapter 3 Vocoder-based Statistical Speech Synthesis Using GANs

53

MGE-GV
Proposed
0.0

0.2

0.4
0.6
Preference score

0.8

1.0

Fig. 3.18. Preference scores of speech quality with 95% conﬁdence intervals (comparison
with GV compensation)

Static &delta
Identity
0.0

0.2

0.4

0.6

0.8

1.0

Preference score

Fig. 3.19. Preference scores of speech quality with 95% conﬁdence intervals (eﬀect offeature function)

Identity: ϕ(y) = y
Static & delta [68]: ϕ(y) = M y“Identity” is equivalent to not using the feature function. When “Static & delta” wasadopted, joint vectors of the static, delta, and delta-delta mel-cepstral coeﬃcients andcontinuous F0 were used for the discriminator. A preference AB test involving 31 listeners
was conducted using crowdsourced subjective evaluation systems.
Figure 3.19 shows the evaluation result. The score of “Static & delta” is much lower
than that of “Identity.” This result suggests that “Static & delta” eﬀectively distinguishes
natural and synthetic speech, but does not improve speech quality.

3.4.9

Subjective Evaluation Using Richer DNN Architecture

Only simple Feed-Forward DNNs were used in the above-described evaluations. Here, twolayer uni-directional LSTM [83] was used for both the acoustic model and discriminator toinvestigate the eﬀectiveness of using a richer DNN architecture. The numbers of memory

54

Chapter 3 Vocoder-based Statistical Speech Synthesis Using GANs

MGE
Proposed
0.0

0.2

0.4
0.6
Preference score

0.8

1.0

Fig. 3.20. Preference scores of speech quality with 95% conﬁdence intervals (eﬀect of
LSTM-based acoustic model and discriminator)

cells in the acoustic model and discriminator were 256 and 128, respectively. The proposed
algorithm was applied to spectral parameters and F0 . The MGE (“MGE”) and proposed(“Proposed”) algorithms were compared. A preference AB test involving 19 listeners was
conducted using crowdsourced subjective evaluation systems.
Figure 3.20 shows the evaluation result. The score of “Proposed” is higher than thatof “MGE,” demonstrating that the proposed algorithm works for not only simple DNN
architectures but also richer ones.

3.4.10Eﬀect of Divergence to Be Minimized by GANs

As the ﬁnal investigation regarding TTS, various GANs were adopted to the proposed
algorithm. The following GANs were compared:
GAN: Eqs. (3.1) and (3.2)
KL-GAN: Eqs. (3.6) and (3.7)
RKL-GAN: Eqs. (3.9) and (3.10)
JS-GAN: Eqs. (3.12) and (3.13)
W-GAN: Eqs. (3.15) and (3.16)
LS-GAN: Eqs. (3.17) and (3.18)
A ﬁve-scale point mean opinion score (MOS) test on speech quality was conducted. Speechsamples generated with each GAN were presented to listeners in random order. Fiftyﬁve listeners participated in the assessment by using crowdsourced subjective evaluation
systems.
Figure 3.21 shows the evaluation results. The proposed algorithm works in the cases of
all GANs except for “KL-GAN” and “JS-GAN.” In addition, two points are noteworthy:Chapter 3 Vocoder-based Statistical Speech Synthesis Using GANs

55

LS-GANW-GAN
GAN
RKL-GAN
JS-GAN
KL-GAN
1.0

2.0

3.0
MOS score

4.0

5.0

Fig. 3.21. MOS evaluation results of speech quality with 95% conﬁdence intervals (comparing various GANs)

1) minimizing the KL-divergence (“KL-GAN”) does not improve synthetic speech quality,
but the reversed version (“RKL-GAN”) works, and 2) the JS-divergence (“JS-GAN”) does
not work well, but the approximated version (“GAN”) works. One possible reason is the
behavior of each GAN in density ﬁtting: mode-seeking or mode-covering. As reported
in [99], when the target distribution is multi-modal, “RKL-GAN” and “GAN” tend toresult in mode-seeking learning while “KL-GAN” and “JS-GAN” lead to mode-coveringlearning. These insights indicate that the mode-seeking-based learning is suitable for the
acoustic model training in speech synthesis. The best GAN in terms of synthetic speech
quality is W-GAN, whose MOS value is signiﬁcantly higher than those of LS-GAN, JSGAN, and KL-GAN. This result indicates that the stable optimization in GAN-based
acoustic model training is crucial for improving synthetic speech quality.

3.5

Experimental Evaluations for VC

3.5.1

Conditions for VC Evaluation

The experimental conditions such as the dataset used in the evaluation, speech parameters,
pre-processing of data, and training procedure were the same as the previous evaluations,

56

Chapter 3 Vocoder-based Statistical Speech Synthesis Using GANs

except for the dimensionality of spectral parameters and DNN architectures. The effectiveness of the proposed algorithm was investigated in 1) VC using speech parameter
conversion, and 2) VC using spectral diﬀerentials.
In evaluation of VC using speech parameter conversion, two DNNs were trained: onefor male-to-male VC and the other for male-to-female VC. Architectures for the acoustic
model and discriminator were Feed-Forward DNNs. The hidden layers of the acoustic model and discriminator had 512 × 3 units and 256 × 3 units, respectively. The
1st-through-59th mel-cepstral coeﬃcients were converted by DNNs. The input 0th melcepstral coeﬃcients were directly used as those of the converted speech. F0 was linearlytransformed, and band-aperiodicity was not transformed. The DTW algorithm was used
to align frame lengths of the input and output speech parameters.
In evaluation of VC using spectral diﬀerentials, DNNs for male-to-male VC were trained.
Here, input-to-output highway networks (described in Appendix A) were adopted to theacoustic model instead of Feed-Forward DNNs. The transform gate of the highway networks only had a 59-unit input and 59-unit sigmoid output layers.
Speech samples were generated with the conventional MGE and proposed algorithms.
Preference AB tests were conducted to evaluate converted speech quality. Pairs of speechsamples of the two sets were presented to listeners in random order. Listeners selectedspeech samples that sounded better in quality. Similarly, XAB tests on the speakerindividuality were conducted using natural speech of the target speaker as the reference
“X.”

3.5.2

Subjective Evaluation Using Speech Parameter Conversion

Eight listeners participated in assessment of male-to-male VC case, and 27 listeners participated in assessment of male-to-female VC case by using crowdsourced subjective evaluation systems. Figures 3.22 and 3.23 show the evaluation results regarding the speech
quality and speaker individuality, respectively. The results shown in Fig. 3.22 demonstratethat the proposed algorithm signiﬁcantly improves the speech quality. Similar tendency isobserved in the evaluation results regarding the speaker individuality shown in Fig. 3.23.
These improvements are observed in both male-to-male and male-to-female VC settings.
One of the reasons for these improvements are GV reproduction by the proposed algorithm, which aﬀects not only speech quality but also speaker individuality [25].

Chapter 3 Vocoder-based Statistical Speech Synthesis Using GANs

57(a) Male-to-male
MGE
Proposed
0.0

0.2

0.4
0.6
(b) Male-to-female

0.8

1.0

0.2

0.4
0.6
Preference score

0.8

1.0

MGE
Proposed
0.0

Fig. 3.22. Preference scores of speech quality with 95% conﬁdence intervals (DNN-based
VC using speech parameter conversion)(a) Male-to-male
MGE
Proposed
0.0

0.2

0.4
0.6
(b) Male-to-female

0.8

1.0

0.2

0.4
0.6
Preference score

0.8

1.0

MGE
Proposed
0.0

Fig. 3.23. Preference scores of speaker individuality with 95% conﬁdence intervals(DNN-based VC using speech parameter conversion)

3.5.3

Subjective Evaluation Using Spectral Diﬀerentials

The number of listeners that participated in each of the preference AB or XAB tests waseight. Figures 3.24 and 3.25 show the evaluation results regarding the speech quality and58

Chapter 3 Vocoder-based Statistical Speech Synthesis Using GANs

MGE
Proposed
0.0

0.2

0.4
0.6
Preference score

0.8

1.0

Fig. 3.24. Preference scores of speech quality with 95% conﬁdence intervals (DNN-based
VC using spectral diﬀerentials)

MGE
Proposed
0.0

0.2

0.4
0.6
Preference score

0.8

1.0

Fig. 3.25. Preference scores of speaker individuality with 95% conﬁdence intervals(DNN-based VC using spectral diﬀerentials)

speaker individuality, respectively. The results demonstrate that the proposed algorithm
is also eﬀective in VC using spectral diﬀerentials.

3.5.4

Evaluation in Many-to-one VC

Experimental evaluation of the proposed algorithm was conducted in many-to-one VC.
The details of the evaluation are described in Appendix B. In summary, the proposed
algorithm is eﬀective to improve converted speech quality in not only one-to-one VC but
also many-to-one VC.

3.6

SummaryThis chapter proposed a novel training algorithm for DNN-based high-quality speechsynthesis. This algorithm incorporates a framework of GANs that adversarially trains a
generator and discriminator to learn high-ﬁdelity deep generative models. The acoustic
model with this algorithm is trained to fool the discriminator that distinguishes naturaland synthetic speech. Since the GAN framework minimizes the divergence between naturaland generated data distributions, the acoustic model can be trained to not only minimize

Chapter 3 Vocoder-based Statistical Speech Synthesis Using GANs

59

the generation loss but also make the distribution of generated speech parameters close tothat of natural ones. As a result, this algorithm reproduces not only natural GVs but also
natural correlation among speech parameters. Experimental evaluations were conductedin DNN-based TTS or VC using vocoder parameters. The evaluation results indicatedthat this algorithm yielded signiﬁcant improvements in terms of speech quality regardlessof its hyperparameter settings. The results also showed that the use of W-GAN in thisalgorithm improved synthetic speech quality the best in comparison with various GANs.

60

Chapter 4

Vocoder-free Statistical Speech
Synthesis Using GANs
4.1

Introduction

This chapter extends the GAN-based acoustic model training algorithm proposed in Chapter 3 to a vocoder-free speech synthesis method using STFT spectra. The vocoder-freemethod can incorporate other DNN-based signal-processing techniques applied in thetime-frequency domain, such as speech enhancement [100] and speech separation [101],
into the acoustic model training. However, it is diﬃcult to apply the GAN-based algorithm to the vocoder-free method directly. One of the reasons is that the distribution ofhigh-dimensional STFT amplitude spectra (e.g., 1,024-dim.) is more complicated thanthat of the low-dimensional vocoder parameters. This chapter proposes a novel algorithm
based on GANs using low-frequency-resolution amplitude spectra to train a DNN-based
acoustic model. Figure 4.1 shows the conceptual diagram of this algorithm. The frequency resolution of amplitude spectra is ﬁrst lowered using an average-pooling function
along with a frequency axis while keeping rough structures of the spectra (i.e., the spectral envelope) preserved. The GAN-based training algorithm is then performed using the
low-frequency-resolution amplitude spectra to calculate the adversarial loss. Since thespectral envelopes are dominant features in perceiving the quality of synthetic speech,
this algorithm can be expected to improve synthetic speech quality better than using the
GANs in the original frequency resolution. Furthermore, a frequency warping function canbe applied to amplitude spectra for introducing various frequency scales that are related
to human speech perception (e.g., mel and inverse mel frequency scales). This chapter
also proposes a training algorithm based on GANs using multi-frequency-resolution am-

Chapter 4 Vocoder-free Statistical Speech Synthesis Using GANs
High

Synthetic

61

NaturalFrequency resolution(dim. of spectra)

GANs in original frequency resolution:difficult to model complicated distribution

Average pooling alongfrequency axis

Dim. reduction while preserving roughstructure of spectra (i.e., spectral envelop)

GANs in low frequency resolution:easy to reduce distribution difference

Low

Fig. 4.1. Conceptual diagram of proposed method in Chapter 4

Waveform
STFT
spectraChapter 2

Vocoder-freemethod [42]Vocoder-derived
parameters

Speech parameterization unit

Goal

Vocoder-based
methodIntroducing
GANs

High-quality
speech synth.

Chapter 4(proposed)

Using low-/multi-freq.-res.
amplitude spectra[21, 22]

Chapter 3(proposed)
GV/MS[25, 33]

Overly smoothed
Naturally smoothed
Smoothness of speech parameters

Fig. 4.2. Relation between conventional and proposed methods in Chapter 4

plitude spectra that uses both low- and original-frequency-resolution amplitude spectra.
This algorithm can be expected to compensate for not only the diﬀerences in rough structures but also in ﬁne structures of natural and generated amplitude spectra. Figure 4.2
illustrates the relation between conventional and proposed methods.
This chapter is organized as follows (see also Fig. 4.3). Section 4.2 describes the proposed GAN-based algorithms using low- and multi-frequency-resolution amplitude spectra. Section 4.3 presents experimental evaluations of these algorithms in DNN-based
vocoder-free TTS using STFT spectra. Section 4.4 summarizes this chapter.

62

Chapter 4 Vocoder-free Statistical Speech Synthesis Using GANs
Conventional
methods
Vocoder-free
TTS [42]
Integrating

Chapter 4: GAN-based vocoder-free speech synth.
Sect. 4.2: proposed methods
Introducing
GAN-based
GANs inmulti-freq. res.
vocoder-freelow-/multiTTS
freq. res.
Introducing human's
speech perception

GANs [66]GAN-based
TTS/VC

Theoreticalextension

GANs inmel-/inv.-mel
freq. scale

Chapter 3
Sect. 4.3: evaluations in vocoder-free TTS
GAN vs. MSE [42]Effect of freq. scale

Effect of freq. res.

Investigation of
freq. band

Fig. 4.3. Overview of Chapter 4

4.2

GAN-based Acoustic Model Training Using
Low-/multi-frequency-resolution Amplitude
Spectra4.2.1

GAN-based Training Algorithm Using Low-frequencyresolution Amplitude Spectra

Frequency Resolution Lowering Using Average-pooling⊤⊤ ⊤Let P = [O ⊤be a (F + 2p)T -by-F T zero-padding matrix, where p, O p,F ,p,F I F O p,F ]and I F denote the size of zero padding, p-by-F zero matrix, and F -by-F identity matrix,
respectively. A (F + 2p)-dimensional zero-padded amplitude spectra vector at frame t,
⊤ ⊤
⊤
[0⊤
p , y t , 0p ] , is calculated as P y t , where 0p denotes a p-dimensional zero vector. Let

W be a F (L) -by-(F + 2p) pooling matrix. The F (L) is the total number of frequency bins
calculated as
F (L) =

F + 2p − w
+ 1,
s

(4.1)

Chapter 4 Vocoder-free Statistical Speech Synthesis Using GANs୐௧

Zero-padding matrix Pooling matrix 
ͳ
ൌ ൈ
୐



ͳ ‫ͳ ڮ‬ͳ ‫ͳ ڮ‬

‫ݓ‬

‫ݏ‬

૙

ࡻ௣ǡி

૙
‫ڰ‬

ࡵி

௧



୐

ͳ ‫ͳ ڮ‬63  ൅ 2 ൅ 2

ࡻ௣ǡி
Fig. 4.4. Matrix computation in average-pooling function

where w and s denote the width and stride of the pooling operation, respectively.

(L)

A low-frequency-resolution amplitude spectra vector at frame t, y t(L)(L)[yt (1), · · · , yt (F (L) )]⊤ ,is calculated as W P y t .

=An average-pooling function ϕ(·)

transforms amplitude spectra in the original frequency resolution y into those in the low
frequency resolution y (L) as ϕ(y) = [(W P y 1 )⊤ , · · · , (W P y T )⊤ ]⊤ , using the matrices P
and W . Figure 4.4 shows the matrix computation in the average-pooling function. The
above processes are similar to conversion from raw amplitude spectra into ﬁlter-bankparameters that represent spectral envelopes of speech.
GAN-based Training Using Low-frequency-resolution Amplitude Spectra
A low-frequency-resolution discriminator D(L) (·) is trained to distinguish natural and generated amplitude spectra in the low (i.e., rough) frequency resolution. The discriminatorloss for D(L) (·) is deﬁned as the same manner as Eq. (3.1), but y and ŷ in the equation
are replaced with y (L) and ŷ (L) , respectively.
A loss function for training an acoustic model is deﬁned as follows:
(Low)

LG

(L)

(y, ŷ) = LMSE (y, ŷ) + ωDEŷ [LMSE ]LADV ŷ (L) ,Eŷ(L) [LADV ]

(4.2)
(L)

where ŷ (L) = ϕ(ŷ) denotes generated spectra in the low frequency resolution. The ωD

is a hyperparameter to control the eﬀect of the second term. This loss function is formulated as the weighted sum of the MSE in the original frequency resolution and adversarial
loss in the low frequency resolution. Since the distributions of amplitude spectra in the
low frequency resolution become simpler than those in the original frequency resolution,
this algorithm can overcome the diﬃculty in modeling complicated distribution of highdimensional amplitude spectra. Furthermore, it can be expected to improve synthetic

64

Chapter 4 Vocoder-free Statistical Speech Synthesis Using GANs(a) Linear

Frequency [kHz](1) Natural

8.0

Frequency [kHz](c) Inv-mel

6.0
4.0
2.0
0.0
8.0

(2) Synthetic(b) Mel

100

200

300

100

200

300

100

200

300

100

200

300

100

200
Frame

300

100

200

300

6.0
4.0
2.0
0.0

Fig. 4.5. Examples of amplitude spectra of natural and synthetic speech after frequency
warping for (a) linear, (b) mel, and (c) inverse mel frequency scales. These
spectra were extracted from one utterance of evaluation data. Synthetic amplitude spectra were generated from acoustic model trained to minimize MSE(Eq. (2.15)).

speech quality by reducing the diﬀerence between spectral envelopes (i.e., dominant features regarding speech quality) of natural and synthetic speech.

4.2.2

Frequency Scale Conversion Using Frequency Warping

Besides the above-described approximated ﬁlter bank extraction, the proposed algorithm
can use diﬀerent frequency scales that are related to human speech perception and antispooﬁng [29, 67]. For example, the mel frequency scale and its inverted version [68] canbe used instead of an ordinary linear frequency scale. This can be done by applying
a frequency warping function to amplitude spectra before feeding them into the lowfrequency-resolution discriminator. Figure 4.5 shows examples of the amplitude spectraof natural and synthetic speech in various frequency scales after applying the frequency
warping functions [102] shown in Fig. 4.6. In the mel frequency scale (Fig. 4.5(b)), there
are fewer diﬀerences between natural and synthetic amplitude spectra than those in the
inverse mel frequency scale. This observation suggests that the GAN-based distribution
compensation may work well in the inverse mel frequency scale.

Chapter 4 Vocoder-free Statistical Speech Synthesis Using GANs

65

0.8

a
r

)
(b

l

L

in
e

0.6

e
M

(a
)

Warped frequency

1.0

0.4

)
(c

0.2

0.0
0.0

0.2

0.4

-m
v
n
I

0.6

e

l

0.8

1.0

Original frequency

Fig. 4.6. Frequency warping functions for (a) linear, (b) mel, and (c) inverse mel frequency scales

4.2.3

GAN-based Training Algorithm Using Multi-frequencyresolution Amplitude Spectra

This section presents a GAN-based training algorithm using multi-frequency-resolution
amplitude spectra. This algorithm introduces not only the low-frequency-resolution discriminator D(L) (·) but also the original-frequency-resolution one D(·). A loss function for
training an acoustic model is deﬁned as follows:(Multi)LG

Eŷ [LMSE ]LADV (ŷ)
Eŷ [LADV ](L) Eŷ [LMSE ](L)+ ωD
LADV ŷ.
Eŷ(L) [LADV ](y, ŷ) = LMSE (y, ŷ) + ωD

(4.3)

When ωD = 0, this loss function is the same as that in Eq. (4.2). This algorithm canbe expected to compensate for not only the diﬀerences in rough structures (i.e., spectralenvelopes) but also in ﬁne structures of natural and generated amplitude spectra. Figure
4.7 illustrates the computation procedure of the loss function. Note that D(L) (·) and D(·)
are separately trained.

66

Chapter 4 Vocoder-free Statistical Speech Synthesis Using GANs

Linguisticfeats.

Generated
spectraෝ‫୑ܮ‬ୗ୉ ࢟ǡ ࢟

Naturalspectra࢟ෝAcoustic ࢟modelෝ‫ܮ‬୅ୈ୚ ࢟
1
‫ڄ ܦ‬

ෝ
࢟

ࣘ ‫ڄ‬
ෝ୐
‫ܮ‬୅ୈ୚ ࢟

ࣘ ‫ڄ‬

1
ෝ୐
࢟

‫ڄ ୐ܦ‬

Average-poolingFig. 4.7. Loss functions to update acoustic model in proposed GAN-based algorithm
using multi-frequency-resolution amplitude spectra. Average-pooling function
ϕ(·) lowers frequency resolution of amplitude spectra while keeping their roughstructures unchanged.

4.2.4

Discussion

The average-pooling function in the proposed algorithms can be regarded as the ﬁlter
bank parameter extraction. When the pooling width w is set to a larger value, ﬁnestructures of the amplitude spectra get smoother. Figure 4.8 shows examples of the lowfrequency-resolution spectra of natural and synthetic speech with various settings of thepooling width. This ﬁgure illustrates that spectral peaks (i.e., formants) of the synthetic
amplitude spectra tend to be weaker than those of the natural ones. This tendency may
be one of the causes of the speech quality degradation.
Regarding related work, Kaneko et al. [103] proposed a GAN-based post-ﬁlter for STFT
amplitude spectra. This post-ﬁlter-based approach requires additional computation in
the synthesis stage, but the proposed approaches do not. Also, their approach ignores
the overall spectral structures (i.e., spectral envelopes) and their correlation because it
splits amplitude spectra into several sub-frequency bands and applies GANs to each band
independently. On the other hand, the proposed approaches can eﬀectively capture themby reducing the dimensionality of the spectra while preserving the whole spectral structure.

Chapter 4 Vocoder-free Statistical Speech Synthesis Using GANs(a) w = 1

Frequency [kHz](1) Natural

8.0

Frequency [kHz](d) w = 70

6.0
4.0
2.0
0.0
8.0

(2) Synthetic(c) w = 30(b) w = 14

67

100

200

300

100

200

300

100

200

300

100

200

300

100

200

300

100

200

300
100
Frame

200

300

100

200

300

6.0
4.0
2.0
0.0

Fig. 4.8. Examples of low-frequency-resolution spectra of natural and synthetic speech.
These spectra were extracted from one utterance of evaluation data. Zeropadding size p and pooling stride s were set to 6 and w/2, respectively. “(a)w = 1” corresponds to amplitude spectra in original frequency resolution.
These synthetic amplitude spectra were generated from acoustic model trained
to minimize Eq. (2.15).

This chapter extends the GAN-based proposed algorithm for TTS using vocoder parameters in Chapter 3 to TTS using STFT amplitude spectra. Although Juvela et al. [104]proposed a GAN-based method to synthesize a speech waveform from natural mel frequency cepstral coeﬃcients (MFCCs), their method was not conditioned by linguistic
information and cannot be directly applied to TTS. The proposed algorithms can be expected to achieve TTS that directly synthesizes a speech waveform from linguistic features.
The idea using GANs in the low frequency resolution can also be applied to WaveGAN
and SpecGAN [105], which synthesize an audio waveform by using unconditional GANs.

4.3

Experimental Evaluations

4.3.1

Experimental Conditions

A speech corpus of a Japanese female speaker was used. The speaker uttered 4,007
sentences (part of the JSUT corpus [106]). The numbers of sentences used for training
and evaluation were 3,808 and 199, respectively. The sampling rate of the speech signals
was 16 kHz. The frame length, shift length, and FFT length were 400, 80, and 1,024

68

Chapter 4 Vocoder-free Statistical Speech Synthesis Using GANs

samples, respectively. The Hamming window was used for STFT analysis. Accordingly,
513-dimensional STFT spectra were obtained. Ninety percent of the silence frames were
removed from the training data to improve the training accuracy.
Architectures for the acoustic model and discriminators were Feed-Forward DNNs. The
input of the acoustic model was a 444-dimensional vector including 439-dimensional linguistic features, 3-dimensional duration features, continuous log F0 , and U/V. The linguistic features included phonemes, mora position, accent type, frame position in a phoneme,
etc. F0 was extracted from speech data using the STRAIGHT vocoder [47]. Two DNNs
for predicting duration and F0 from linguistic features were trained in advance. The
acoustic model included three 1,024-unit hidden layers with the ReLU [54] activation
function and 513-unit output layer with the linear activation function. The originalfrequency-resolution discriminator included three 512-unit hidden layers with the ReLU
activation function and one unit output layer with the sigmoid activation function. The
low-frequency-resolution discriminator was almost the same as the original-frequencyresolution one; i.e., the activation functions used in the hidden and output layers were
ReLU and sigmoid, respectively, and the number of hidden layers was three. However,the numbers of input and hidden units of the discriminator varied in accordance with theparameters of the pooling function ϕ(·). The pooling width w was set to 14, 30, or 70.
Accordingly, F (L) was set to 74, 34, or 14. The number of the hidden units in D(L) (·) waschanged to 128, 64, or 32 as F (L) decreased. The zero padding size p and stride s were
set to 6 and w/2, respectively.
In the DNN training, real-valued linguistic features and log-amplitude spectra were normalized to have zero-mean and unit-variance. The acoustic model was ﬁrst initialized byminimizing the MSE between natural and generated amplitude spectra with 25 iterations.
The original- and low-frequency-resolution discriminators were then initialized to distinguish amplitude spectra of natural speech and ones generated by the initialized acoustic
model. The discriminator initialization was performed with 5 iterations. The proposed
algorithms were ﬁnally performed with 25 iterations using the initialized acoustic modeland discriminators. The expectation values for scaling the loss functions were estimatedat each iteration. The optimization algorithm was AdaGrad [98]. The learning rate was
set to 0.01.

4.3.2

Objective Evaluations

The root mean square error (RMSE) between natural and generated amplitude spectra andthe spooﬁng rates of D(·) and D(L) (·) were calculated as the objective evaluation criteria.

Chapter 4 Vocoder-free Statistical Speech Synthesis Using GANs

69

Table 4.1. Objective evaluation results with their standard deviations. “RMSE” denotes root mean square error between natural and generated amplitude spec(L)tra. Various hyperparameter settings (ωD , ωD ) for proposed algorithms wereused and compared. Pooling parameters were set to w = 30, s = 15, andp = 6. These evaluation values were calculated using all evaluation data andaveraged(L)(ωD , ωD )RMSE
D(·) spooﬁng rate
D(L) (·) spooﬁng rate

(0.0, 0.0)
1.09±0.09
0.0019 ± 0.0042
0.0273 ± 0.0177(0.0, 1.0)
1.12±0.09
0.5507 ± 0.0696
0.9704 ± 0.0269(1.0, 0.0)
1.25±0.08
0.9999 ± 0.0001
0.9965 ± 0.0055(1.0, 1.0)
1.24±0.08
0.9999 ± 0.0005
0.9955 ± 0.0065

The discriminators D(·) and D(L) (·) for the spooﬁng rate calculation were trained to
distinguish amplitude spectra of natural speech from those of synthetic speech generated
by the conventional algorithm [42]. The proposed algorithms were compared using the(L)combination of the hyperparameters (ωD , ωD ) setting each to 0.0 or 1.0.
Table 4.1 shows the evaluation results. From the results, the algorithm with the hy(L)perparameter setting (ωD = 0.0, ωD = 0.0), i.e., the same as the conventional algorithm,
achieves the lowest RMSE. However, the algorithm’s spooﬁng rates are the lowest among
the four algorithms. This result suggests that the conventional algorithm does not train
the acoustic model to fool the two discriminators. On the other hand, the proposed al(L)gorithm setting to (ωD = 0.0, ωD = 1.0) generates amplitude spectra that fool D(L) (·),
although the RMSE becomes slightly worse than that of the conventional algorithm. Notethat the similar tendency (i.e., increase of speech parameter generation error) was also
reported in Section 3.4.2. Some of amplitude spectra generated by this algorithm alsofool D(·), nevertheless deceiving the model is out of the consideration during the acoustic
model training. These results indicate that the GAN-based proposed algorithm using
low-frequency-resolution amplitude spectra reduces the diﬀerences between natural andgenerated amplitude spectra observed in their rough structures. Meanwhile, the proposed
algorithms using original-frequency-resolution amplitude spectra, i.e., “(1.0, 0.0)” and“(1.0, 1.0)” in Table 4.1, have similar tendencies. Although these algorithms improve
spooﬁng rates of both D(·) and D(L) (·) much higher than the other two algorithms, theyalso considerably degrade the RMSE of the amplitude spectra. Such RMSE degradation
may deteriorate synthetic speech quality signiﬁcantly.

70

Chapter 4 Vocoder-free Statistical Speech Synthesis Using GANs
Table 4.2. Preference scores of synthetic speech quality (GANs using original-frequencyresolution amplitude spectra with various hyperparameter settings of ωD ).
Bold values indicate that method is more preferred than other with p-value< 0.05

Method A
ωD = 0.5
ωD = 1.0
ωD = 0.5

4.3.3

Score (A vs. B)
0.300 vs. 0.700
0.280 vs. 0.720
0.496 vs. 0.504

Method B
Baseline
Baseline
ωD = 1.0

Subjective Evaluations

Preference AB tests in terms of synthetic speech quality were conducted. Twenty-ﬁve
listeners participated in each of the following evaluations by using crowdsourced evaluation
systems. Each listener evaluated 10 speech samples. The total number of listeners was
1,125. In the following evaluations, “Baseline” denotes the conventional training algorithm
that minimizes the MSE loss shown in Eq. (2.15) only [42].
Evaluation of GANs Using Original-frequency-resolution Amplitude Spectra
The eﬀect of the GAN-based algorithm using the original-frequency-resolution amplitude
spectra (i.e., the same algorithm as described in Chapter 3) was investigated by ﬁxing(L)ωD = 0 and by setting ωD = 0.5 or 1.0. Three algorithms were compared: “Baseline”
and proposed algorithm with the setting “ωD = 0.5” or “ωD = 1.0.”
Table 4.2 shows the evaluation results. The GAN-based algorithm signiﬁcantly degrades
synthetic speech quality compared with “Baseline,” regardless of the hyperparameter settings. This result demonstrates that just using the GAN-based training algorithm, whichis eﬀective in TTS using vocoder parameters (Chapter 3), does not improve synthetic
speech quality in TTS using STFT amplitude spectra.
Evaluation of GANs Using Low-frequency-resolution Amplitude SpectraFirstly, the eﬀect of pooling width w was investigated by ﬁxing ωD = 0 and by setting(L)ωD= 1. Speech samples generated by “Baseline” and the proposed algorithm using

low-frequency-resolution amplitude spectra were compared. The pooling width w was set
to 14, 30, or 70. The number of hidden units in D(L) (·) was changed in accordance withthe pooling parameter settings. Table 4.3 shows the evaluation results. The proposed
algorithm always achieves higher scores than “Baseline” regardless of the pooling width
settings. This result demonstrates the algorithm’s eﬀectiveness in improving synthetic

Chapter 4 Vocoder-free Statistical Speech Synthesis Using GANs

71

Table 4.3. Preference scores of synthetic speech quality (GANs using low-frequencyresolution amplitude spectra with various settings of w). Bold values indicate that method is more preferred than other with p-value < 0.05. Here,
number of hidden units in D(L) (·) was changed in accordance with settingsof w(a) Results comparing “Baseline” with each of GANbased algorithms

Method A
w = 14
w = 30w = 70Score (A vs. B)
0.568 vs. 0.432
0.572 vs. 0.428
0.528 vs. 0.472

Method B
Baseline
Baseline
Baseline(b) Results comparing three GAN-based algorithms

Method A
w = 14
w = 30w = 70Score (A vs. B)
0.488 vs. 0.512
0.532 vs. 0.468
0.472 vs. 0.528

Method B
w = 30w = 70w = 14

Table 4.4. Preference scores of synthetic speech quality (GANs using low-frequencyresolution amplitude spectra with various settings of w). Bold values indicate that method is more preferred than other with p-value < 0.05. Here,
number of hidden units in D(L) (·) was ﬁxed regardless of settings of w(a) Results comparing “Baseline” with each of GANbased algorithms

Method A
w = 14
w = 30w = 70Score (A vs. B)
0.548 vs. 0.452
0.600 vs. 0.400
0.560 vs. 0.440

Method B
Baseline
Baseline
Baseline(b) Results comparing three GAN-based algorithms

Method A
w = 14
w = 30w = 70Score (A vs. B)
0.472 vs. 0.528
0.528 vs. 0.472
0.476 vs. 0.524

Method B
w = 30w = 70w = 14speech quality.
Secondly, a subjective evaluation of the proposed algorithm with the ﬁxed numberof hidden units in D(L) (·) was conducted. The number of hidden units was set to 128
regardless of the pooling parameter settings. Table 4.4 shows the evaluation results. The
results shown in this table have tendencies similar to Table 4.3, demonstrating that the

72

Chapter 4 Vocoder-free Statistical Speech Synthesis Using GANs
Table 4.5. Preference scores of synthetic speech quality (GANs using low-frequency(L)resolution amplitude spectra with various hyperparameter settings of ωD
and ﬁxed pooling width w = 30). Bold values indicate that method is more
preferred than other with p-value < 0.05

Method A(L)ωD = 0.5(L)ωD = 1.0(L)ωD = 0.5

Score (A vs. B)
0.544 vs. 0.456
0.588 vs. 0.412
0.504 vs. 0.496

Method B
Baseline
Baseline(L)ωD = 1.0

Table 4.6. Preference scores of synthetic speech quality (comparison between GANs
using low-frequency-resolution and smoothed original-frequency-resolution
amplitude spectra). Bold value indicates that method is more preferredthan other with p-value < 0.05

Method A
Pooling

Score (A vs. B)
0.580 vs. 0.420

Method B
Smoothingalgorithm works robustly against the discriminator’s size. The pooling width w was set to
30 in the following evaluations because the results in Tables 4.3(b) and 4.4(b) indicatedthat “w = 30” was the best among the three settings. The number of hidden units in
D(L) (·) was set to 64 for reducing the number of model parameters.
Finally, the hyperparameter’s eﬀect in the proposed algorithm was investigated by ﬁxing(L)ωD = 0 and by setting ωD = 0.5 or 1.0. “Baseline” and the proposed algorithm with
(L)

(L)

the setting “ωD = 0.5” or “ωD = 1.0” were compared. Table 4.5 shows the evaluation
results. These results demonstrate that the GAN-based proposed algorithm using lowfrequency-resolution amplitude spectra improves synthetic speech quality regardless of its
hyperparameter settings.
Comparison between Pooling and Smoothing of Amplitude Spectra
The dimensionality reduction by the average-pooling function is one of the crucial factors
in the proposed algorithm. Here, the average-pooling function (“Pooling”) was comparedwith a simple moving average ﬁlter (“Smoothing”) to investigate the eﬀectiveness of thedimensionality reduction. The ﬁlter size of “Smoothing” was set to 25. Amplitude spectra
after applying “Smoothing” can be regarded as the spectral envelope parameters without
the dimensionality reduction. Table 4.6 shows the evaluation result. “Pooling” signiﬁcantly outperforms “Smoothing” from this table. This result suggests that the dimensionality reduction is one of the essentials for the synthetic speech quality improvement.

Chapter 4 Vocoder-free Statistical Speech Synthesis Using GANs

73

Table 4.7. Preference scores of speech quality (GANs using original-, low-, and multifrequency-resolution amplitude spectra). Bold values indicate that method
is more preferred than other with p-value < 0.05

Method A
Low
Multi
OriginalFrequency [kHz]8.0(a) Natural

Score (A vs. B)
0.808 vs. 0.192
0.492 vs. 0.508
0.192 vs. 0.808(b) Baseline(c) Low

Method B
Multi
Original
Low(d) Original(e) Multi

6.0
4.0
2.0
0.0

100

200

300

100

200

300

100

200 300
Frame

100

200

300

100

200

300

Fig. 4.9. Examples of amplitude spectra of (a) natural speech and synthetic speech
generated by algorithms named as (b) “Baseline,” (c) “Low,” (d) “Original,”and (e) “Multi.” These spectra were extracted from one utterance of evaluation
data.

Evaluation of GANs Using Multi-frequency-resolution Amplitude Spectra
The eﬀects of the proposed algorithm using multi-frequency-resolution amplitude spectra
were investigated. Speech samples were generated using the following three algorithms:(L)

Original: (ωD , ωD ) = (1.0, 0.0)
(L)

Low: (ωD , ωD ) = (0.0, 1.0)
(L)

Multi: (ωD , ωD ) = (1.0, 1.0)
Table 4.7 shows the experimental results. “Low” in this table signiﬁcantly outperforms
the others. Amplitude spectra of synthetic speech used for this evaluation were plottedin Fig. 4.9 to investigate the reason. The diﬀerence in spectral peaks between naturaland synthetic speech is reduced by the proposed algorithms (Figs. 4.9(c), (d), and (e)).
However, some temporal discontinuities are observed in the amplitude spectra generated
by “Original” and “Multi” (Figs. 4.9(d) and (e)). These discontinuities may be one of thecauses of speech quality degradation.

74

Chapter 4 Vocoder-free Statistical Speech Synthesis Using GANs
Table 4.8. Preference scores of synthetic speech quality (GANs using low-frequencyresolution amplitude spectra with gated-CNN-based acoustic model). Bold
values indicate that method is more preferred than other with p-value < 0.05(a) Results comparing “Baseline” with each of GANbased algorithms

Method A
Original
Low
Multi

Score (A vs. B)
0.180 vs. 0.820
0.608 vs. 0.392
0.252 vs. 0.748

Method B
Baseline
Baseline
Baseline(b) Results comparing three GAN-based algorithms

Method A
Low
Multi
Original

Score (A vs. B)
0.808 vs. 0.192
0.496 vs. 0.504
0.172 vs. 0.828

Method B
Multi
Original
Low

Evaluation of DNN Architectures for Acoustic model
Richer architectures than Feed-Forward DNNs were used for acoustic modeling to deal
with the temporal discontinuities of generated amplitude spectra. Gated convolutional
neural networks (CNNs) [107] were used instead of LSTM [52, 83] because they can beapplied to sequential modeling in speech processing [108, 109] and can be trained fasterthan LSTM. Here, a 1D convolutional (Conv1D) layer along the time axis with the gatedlinear unit (GLU) activation function [107] was prepared after the output layer of theacoustic model. A residual connection [110] was introduced between the output andConv1D layers for better modeling. The width and zero-padding size of the convolution
were set to 15 and 7, respectively.
Speech samples were generated by the four algorithms (“Baseline,” “Original,” “Low,”
and “Multi”) using the gated CNNs as the acoustic model. Table 4.8 shows the evaluation results. Similar tendencies to the previous evaluation results are observed; i.e.,
1) “Low” signiﬁcantly outperforms the others, and 2) the GAN-based algorithms using original-frequency-resolution amplitude spectra signiﬁcantly degrade synthetic speech
quality. These results indicate that the sequential modeling is insuﬃcient to deal with thetemporal discontinuities in the generated amplitude spectra.
The eﬀectiveness of the CNN-based acoustic model in the proposed algorithm using
low-frequency-resolution amplitude spectra was investigated. Two acoustic models, 1)
Feed-Forward DNNs (“FFNN”) and 2) gated CNNs (“CNN”), were compared. TableChapter 4 Vocoder-free Statistical Speech Synthesis Using GANs

75

Table 4.9. Preference scores of synthetic speech quality (comparison between FeedForward DNNs (“FFNN”) and gated CNNs (“CNN”) for acoustic modeling).
Bold values indicate that method is more preferred than other with p-value< 0.05

Method A
CNN

Score (A vs. B)
0.644 vs. 0.356

Method B
FFNN

Table 4.10. DNN architectures for discriminator. Numbers of input units of “FFNNO,” “FFNN-L,” “CNN-1D,” and “CNN-2D,” were 513, 34, 513, and 513,
respectively. “ReLU” or “GLU” denotes hidden activation function. Width,
stride, and zero-padding size of Conv1D layer in “CNN-1D” were set to 30,
15, and 6, respectively. Width parameters of Conv2D layers in “CNN-2D”
were set to 9, 7, 5, and 3. Accordingly, stride and zero-padding parameters
were set to 4, 3, 2, and 1

FFNN-OFFNN-LCNN-1D
CNN-2D

Hidden layers
Linear 512 units × 3 (ReLU)
Linear 64 units × 3 (ReLU)
Conv1D 4 channels (GLU) + Linear 64 units × 3 (ReLU)
Conv2D 32–16–8–4 channels (ReLU) + Fully-connected

OutputFrame-wise
Frame-wise
Frame-wise
Segment-wise

4.9 shows the evaluation results. The results demonstrate that the CNN-based acoustic
model is eﬀective in improving synthetic speech quality better than the Feed-ForwardDNN-based one.
Evaluation of DNN Architectures for Discriminator
The eﬀects of DNN architectures for the discriminator in the proposed algorithms were
investigated. The gated-CNN-based acoustic model was used, and the discriminators
using following four DNN architectures were compared:
FFNN-O: the same as used in “Original”
FFNN-L: the same as used in “Low”
CNN-1D: replacing average-pooling with a Conv1D layer along the frequency axis with
the GLU activation function
CNN-2D: using 2D convolutional (Conv2D) layers for capturing the time-frequency
structures of the spectra
The Conv1D layer in “CNN-1D” can be regarded as a trainable multi-channel averagepooling function to obtain low-frequency-resolution amplitude spectra. Table 4.10 shows
the details of the four architectures.

76

Chapter 4 Vocoder-free Statistical Speech Synthesis Using GANs
Table 4.11. Preference scores of synthetic speech quality (GANs using gated CNNs as
acoustic models and various DNN architectures for discriminative models).
Bold values indicate that method is more preferred than other with p-value< 0.05(a) Results comparing GANs using original-frequencyresolution amplitude spectra

Method A
FFNN-OFFNN-O
CNN-1D

Score (A vs. B)
0.376 vs. 0.624
0.120 vs. 0.880
0.348 vs. 0.652

Method B
CNN-1D
CNN-2D
CNN-2D(b) Results comparing GAN-based algorithms using
original- and low-frequency-resolution amplitude spectra

Method A
FFNN-L
FFNN-L

Score (A vs. B)
0.820 vs. 0.180
0.828 vs. 0.172

Method B
CNN-1D
CNN-2D

Speech samples were generated using the gated-CNN-based acoustic model trained withthe four diﬀerent discriminators. Table 4.11 shows the evaluation results. The results
shown in Table 4.11(a) demonstrate that 1) both “CNN-1D” and “CNN-2D” outperform“FFNN-O,” and 2) “CNN-2D” is superior to “CNN-1D,” regarding synthetic speech quality. However, the results in Table 4.11(b) show that the CNN-based discriminators areinferior to “FFNN-L.” One of the reasons may be the diﬃculty in training GANs withmore complicated DNN architectures for the proposed algorithm.
Evaluation of Frequency Scale of Low-frequency-resolution Amplitude Spectra
The eﬀect of the frequency scale in the GAN-based algorithm using low-frequencyresolution amplitude spectra was investigated. Three types of frequency scales, linear,mel, and inverse mel, were compared. Table 4.12 shows the evaluation results. “Mel”
signiﬁcantly degrades synthetic speech quality, while “Inv-mel” successfully improves it
and even outperforms “Linear.” Amplitude spectra of synthetic speech used for thisevaluation were plotted in Fig. 4.10 to investigate the reason. The diﬀerences among
the three generated amplitude spectra are observed in a higher frequency band inparticular. The generated spectra of “Low w/ inv-mel” become closer to the natural
spectra compared to the others. This eﬀect may by one of the causes of the quality
improvement.

Chapter 4 Vocoder-free Statistical Speech Synthesis Using GANs

77

Table 4.12. Preference scores of synthetic speech quality (GANs using low-frequencyresolution amplitude spectra with various frequency scale). Bold values
indicate that method is more preferred than other with p-value < 0.05(a) Results comparing “Baseline” with GANs using
low-frequency-resolution amplitude spectra with mel orinverse mel frequency scale

Method A
Mel
Inv-melScore (A vs. B)
0.392 vs. 0.608
0.636 vs. 0.364

Method B
Baseline
Baseline(b) Results comparing three GAN-based algorithms
with diﬀerent frequency scale

Method A
LinearMel
Inv-melFrequency [kHz](b) Low w/ linear(a) Natural

8.0

Score (A vs. B)
0.752 vs. 0.248
0.272 vs. 0.728
0.576 vs. 0.424

Method B
Mel
Inv-melLinear(c) Low w/ mel(d) Low w/ inv-mel

6.0
4.0
2.0
0.0

100

200

300

100

200

300

100

200

300

100

200

300

Frame

Fig. 4.10. Examples of amplitude spectra of (a) natural speech and synthetic speech
generated by proposed algorithms using GANs in low frequency resolution
with (b) linear scale, (c) mel scale, and (d) inverse mel scales. These spectra
were extracted from one utterance of evaluation data.

Evaluation of Frequency Band Used for Average-pooling
The average-pooling function in the proposed algorithm can be regarded as the reduction of undesired components in amplitude spectra fed into the discriminator. Here, thespectra were ﬁrst split into two sub-bands: 0–2 kHz (including at least the ﬁrst and second formants) and 2–8 kHz. The average-pooling was then applied to each sub-band
individually. The results of the comparison among the combinations of these two factors:
frequency band (low or high) and average-pooling (with or without), should be meaningfulfor clarifying what component is eﬀective in improving synthetic speech quality. Figure
4.11 illustrates a conceptual diagram of the split-and-pooling procedures.

78

Chapter 4 Vocoder-free Statistical Speech Synthesis Using GANsHigh

Frequency [Hz]Amplitude
spectra8

Averagepooling

Concat.

Split
2
Averagepooling0
Low

Fig. 4.11. Conceptual diagram of split-and-pooling procedures. This ﬁgure corresponds
to proposed algorithm with average-pooling applied to both low and highfrequency bands, i.e., “(w/, w/)” in Table 4.13.

Table 4.13. Preference scores of synthetic speech quality (comparison among the combination of frequency band (low or high) and average-pooling (w/ or w/o)).
Bold values indicate that method is more preferred than other with p-value< 0.05

Method A(low, high)(w/o, w/o)
(w/o, w/o)
(w/o, w/o)
(w/o, w/)(w/, w/o)(w/, w/)

Score (A vs. B)
0.212 vs. 0.788
0.160 vs. 0.840
0.668 vs. 0.332
0.160 vs. 0.840
0.468 vs. 0.532
0.844 vs. 0.156

Method A(low, high)(w/, w/)(w/, w/o)(w/o, w/)(w/, w/o)(w/, w/)(w/o, w/)

Table 4.13 shows the evaluation results. Here, Feed-Forward DNNs were used for bothacoustic model and discriminator. Two noteworthy points are observed in Table 4.13:
1) using average-pooling in the low frequency band always achieves higher scores, and 2)
there is no signiﬁcant diﬀerence between average-pooling in both low and high frequency
bands and that in an only low frequency band. These results suggest that applying the
GAN-based distribution compensation to the low frequency band tends to degrade thesynthetic speech quality. This tendency corresponds to the results shown in Table 4.12(a)that the use of the mel frequency scale signiﬁcantly deteriorates synthetic speech quality.

Chapter 4 Vocoder-free Statistical Speech Synthesis Using GANs

4.4

79

SummaryThis chapter proposed two training algorithms to incorporate GANs into DNN-based
vocoder-free speech synthesis using STFT spectra. The proposed algorithm using lowfrequency-amplitude spectra trains an acoustic model to minimize the MSE between natural and generated amplitude spectra in original frequency resolution and the diﬀerenceof their distributions in low frequency resolution. This algorithm can be extended to one
using multi-frequency-resolution amplitude that also minimizes the distribution diﬀerences in the original frequency resolution. Experimental results showed that GANs using
original-/multi-frequency-resolution amplitude spectra degraded synthetic speech quality,
but using low-frequency-resolution amplitude spectra successfully improved it better thanthe conventional algorithm. Moreover, GANs using low-frequency-resolution amplitude
spectra with the inverse mel frequency scale further improved speech quality.

80

Chapter 5

VAE-based Multi-speaker
Acoustic Modeling Using
Speech Recognition Process
5.1

Introduction

This chapter proposes a VAE-based multi-speaker speech synthesis method that can synthesize arbitrary speakers’ high-quality voices and can transform the voice characteristicsof the synthetic speech using a single model. Figure 5.1 shows a conceptual diagram ofthe proposed method. The proposed method trains a VAE-based acoustic model withthe aid of a DNN-based speech recognition model that predicts phonetic content of input
speech. As a result, the phonetic content of synthetic speech can be clariﬁed and highquality speech can be synthesized. The proposed method also utilizes continuous speakerrepresentations derived from a DNN-based speaker classiﬁcation model to overcome thelimitation in the use of conventional speaker codes (i.e., discrete speaker representations).
Such continuous speaker representations enable establishing VAE-based non-parallel andmany-to-many VC that can convert arbitrary speakers’ voice characteristics into otherarbitrary speakers’ without requiring parallel speech corpora for the training. Figure 5.2
illustrates the relation between conventional and proposed methods.
This chapter is organized as follows (see also Fig. 5.3). Section 5.2 describes the proposed VAE-based multi-speaker speech synthesis method. Non-parallel and many-to-many
VC based on this method is also presented. Section 5.3 presents experimental evaluationsof this method in VC. Section 5.4 summarizes this chapter.

Chapter 5 VAE-based Multi-spkr. Acoust. Modeling Using Speech Recog. Process

Extraction ofphonetic latentvariable
SourcespeechPre-trained
speechrecognition

Arbitrarytarget spkrs.

Pre-trained
spkr.
classification

Synthetic
speech

VAE-based
multi-spkr.
speech synth.

⋯

Extraction of
spkr. latentvariable⋯Fig. 5.1. Conceptual diagram of proposed method in Chapter 5

Goal

Spkr. diversity

High

High-quality &versatile speechsynth.

Low

Chapter 2

VAE-based
multi-spkr.
modeling [38]Quality
improvementSpkr.
latent var.

Diversity
improvement

Phonetic
latent var.

Low

High
Speech quality

Fig. 5.2. Relation between conventional and proposed methods in Chapter 5

81

82

Chapter 5 VAE-based Multi-spkr. Acoust. Modeling Using Speech Recog. Process
Conventional
methods

Chapter 5: VAE-based versatile speech synth.
Sect. 5.2: proposed methods

VAE-based
multi-spkr.
modeling [38]High-quality
multi-spkr.
modeling

PPGs [111]d-vectors [27]Introducing
as phonetic
latent vars.

Introducing
as spkr.
latent vars.

Versatile
multi-spkr.
modeling

Non-paralleland many-tomany VC

Sect. 5.3: evaluations in VC
One-to-one VC

Investigation of
hyperparameters

Many-to-many VC

Visualization oflatent variables

Fig. 5.3. Overview of Chapter 5

5.2

VAE-based Multi-speaker Acoustic Model Using
Phonetic and Speaker Latent Variables

5.2.1

Phonetic Latent Variable to Improve Synthetic Speech
Quality

VAE Training Objective Using Phonetic Latent Variable
A phonetic latent variable, derived from pre-trained DNNs for speech recognition, is introduced for alleviating quality degradation of synthetic speech due to over-regularization
of VAE latent variables. Let z p be a phonetic latent variable extracted from speechparameters y. The objective function in Eq. (2.14) is rewritten as:
L (θ, ϕ; y, c, z p ) = −DKL (qϕ (z|y, z p ) || pθ (z)) + Eqϕ (z|y,zp ) [log pθ (y|z, z p , c)] . (5.1)
Here, z p is fed into both the encoder and decoder networks for keeping the phonetic
content of input speech preserved. Figure 5.4(b) shows the directed graphical model ofthe proposed VAE-based acoustic model using phonetic latent variables and speaker codes.

Chapter 5 VAE-based Multi-spkr. Acoust. Modeling Using Speech Recog. Process

࢟

ࢠ

ࢉ
(a)

࢟

ࢠ

࢟

ࢠ

ࢉ

ࢠ୮

ࢠୱ

ࢠ୮

(b)

83

(c)

Phoneme index

Fig. 5.4. Three directed graphical models of VAE-based multi-speaker acoustic models.
From left, VAEs are conditioned by (a) one-hot speaker code c, (b) c and phonetic latent variable z p , and (c) speaker latent variable z s and z p , respectively.
Black and red arrows denote inference of latent variables z ∗ and generation of
speech parameter y, respectively.

Frame index
Fig. 5.5. Example of PPGs. Horizontal and vertical axes represent temporal axis andphoneme index, respectively. Brighter values denote high posterior probabilities.

Phonetic Posteriorgram (PPG) as Phonetic Latent Variable
A PPG [111], predicted by a DNN-based speech recognition model, is used as the phonetic
latent variable. It represents a posterior probability sequence of phoneme labels given
input speech parameters. Figure 5.5 shows an example of PPGs. The speech recognition
model takes speech parameters as input and predicts a phoneme label that represents
phonetic content of input speech. A loss function for the DNN training is deﬁned as thesoftmax cross-entropy (SCE) of speech recognition:

LSCE (p, p̂) = −NpXp (n) log p̂ (n) ,

(5.2)

n=1

where p = [p(1), · · · , p(n), · · · , p(Np )]⊤ and p̂ = [p̂(1), · · · , p̂(n), · · · , p̂(Np )]⊤ are aphoneme label and output vector of the DNNs (i.e., PPG), respectively.

The Npdenotes the number of phoneme set. Figure 5.6 shows the training procedure for speech

84

Chapter 5 VAE-based Multi-spkr. Acoust. Modeling Using Speech Recog. Process

Speechparams.

Phoneme
label

PPG
ෝ
࢖

࢖
‫ܮ‬ୗେ୉ ‫ڄ‬

a
i
u
e
o

Fig. 5.6. Training procedure for DNN-based speech recognition model. PPG p̂ is outputof DNNs.

recognition DNNs. The use of PPGs can be expected to improve synthetic speech quality
since its eﬀectiveness is well-known in training DNNs for many-to-one VC (Appendix B).

5.2.2

Speaker Latent Variable to Increase Speaker Diversity

VAE Training Objective Using Phonetic and Speaker Latent Variables
A speaker latent variable, derived from pre-trained DNNs for speaker classiﬁcation, isused as a speaker representation to increase the speaker diversity in synthetic speech.
It represents input speaker’s individuality as a continuous vector even if the speaker is
unseen (i.e., not included in training data). Let z s be a speaker latent variable extracted
from speech parameters y. The objective function Eq. (5.1) is rewritten as:
L (θ, ϕ; y, z s , z p ) = −DKL (qϕ (z|y, z p ) || pθ (z)) + Eqϕ (z|y,zp ) [log pθ (y|z, z p , z s )] .
(5.3)
Here, the conventional discrete speaker code c is replaced with the continuous speakerlatent variable z s . The use of continuous representations enables reproducing an unseen
speaker’s voice characteristics from his/her few speech utterances. Figures 5.4(c) and5.7 show the directed graphical model and overview of the proposed VAE-based acoustic
model using phonetic and speaker latent variables, respectively.
d-vector as Speaker Latent Variable
A d-vector [27], a bottleneck feature of a DNN-based speaker classiﬁcation model, is usedas the speaker latent variable. The speaker classiﬁcation model takes speech parameters as
input and predicts a one-hot speaker code c = [c(1), · · · , c(n), · · · , c(Ns )]⊤ that represents
the identity of one of the seen (i.e., pre-stored) Ns speakers. A loss function for the DNN

Chapter 5 VAE-based Multi-spkr. Acoust. Modeling Using Speech Recog. Processࢠ୮Speechrecognition
Input
speechparams.

Phonetic
latent vars.


Encodernets.
࢟Speakerclassification85

ࢠ

Decoder
nets.


ෝ
࢟

Generated
speechparams.

Speakerࢠୱ latent vars.

Fig. 5.7. Proposed VAE-based multi-speaker acoustic model integrating speech recognition and speaker classiﬁcation models. This model corresponds to directed
graphical model shown in Fig. 5.4(c).

Speechparams.

Outputvectorࢉො

Targetspkr. code
ࢉ

Spkr.
codes

‫ܮ‬ୗେ୉ ‫ڄ‬
d-vec.
ࢊ
Fig. 5.8. Training procedure for DNN-based speaker classiﬁcation model. d-vector d isextracted as output of squeeze layer immediately before output layer.

training is deﬁned as the SCE of speaker classiﬁcation:
LSCE (c, ĉ) = −NsXc (n) log ĉ (n) ,

(5.4)

n=1

where ĉ = [ĉ(1), · · · , ĉ(n), · · · , ĉ(Ns )]⊤ is an output vector of the DNNs. Figure 5.8 shows
the training procedure for speaker classiﬁcation DNNs.
After the training, an Nd -dimensional d-vector d = [d(1), · · · , d(Nd )]⊤ is extracted
from a bottleneck layer of the DNNs. One layer before the output is often used as thebottleneck layer. The d-vector dimensionality Nd is typically set to a smaller value than Ns
for using the lower-dimensional speaker representation. One speaker’s individuality can bedeﬁned by averaging all d-vectors extracted from his/her speech parameter sequences invoiced regions. This usage slightly diﬀers from the traditional use of d-vectors for speaker
veriﬁcation that deﬁnes the speaker individuality as a d-vector averaged over those in all(i.e., voiced and unvoiced) regions. The reason why is that speech parameters in unvoiced

86

Chapter 5 VAE-based Multi-spkr. Acoust. Modeling Using Speech Recog. Process

regions are less aﬀected by speaker individuality than those in voiced regions [112, 57].

5.2.3

Non-parallel and Many-to-many VC Using Proposed VAEbased Acoustic Model

A non-parallel and many-to-many VC method can be established using the VAE-based
multi-speaker acoustic model described in Section 5.2.2. The VAEs in this method are
ﬁrstly trained using multiple speakers’ speech utterances with the pre-trained DNNs forspeech recognition and speaker classiﬁcation. The target speaker’s d-vector is secondly
extracted from his/her speech parameters using the speaker classiﬁcation DNNs. PPGs
and VAE latent variables are thirdly predicted from source speaker’s speech parameters
using the speaker recognition DNNs and encoder networks of VAEs, respectively. The
converted speech parameters are ﬁnally generated from the PPGs, VAE latent variables,
and target speaker’s d-vectors using decoder networks of VAEs.

5.2.4

Discussion

In the proposed method, DNNs for speech recognition and speaker classiﬁcation must
be pretrained using large speech corpora including many speakers. Although making
transcriptions to train the speech recognition DNNs requires considerable cost, semisupervised learning of conditional VAEs [58] can be used for reducing the cost. Also,
techniques for end-to-end speech processing [113, 114] and dual learning of speech synthesis/recognition models [115, 116] can be applied to the proposed method. Furthermore,
one can extend the proposed method to multilingual speech synthesis by introducing
multilingual adaptation of DNN-based speech recognition [117].
The number of seen speakers and d-vector dimensionality are hyperparameters of the
proposed method. The former should be large enough to learn better phonetic and speakerlatent variables. The latter also aﬀects the synthetic speech quality since it can be regarded
as the number of basis in the speaker space. Section 5.3.4 empirically investigates the eﬀectof the two hyperparameters.

Chapter 5 VAE-based Multi-spkr. Acoust. Modeling Using Speech Recog. Process

5.3

Experimental Evaluation

5.3.1

Experimental Conditions

87

The eﬀectiveness of the proposed method was evaluated in DNN-based VC. Two speechcorpora were used: a parallel speech corpus including few speakers and non-parallel speech
corpus including many speakers. The ﬁrst included six Japanese speakers (three males andthree females) who uttered 425 fully-parallel sentences. The second included 260 Japanese
speakers (130 males and 130 females) who uttered about 100 non-parallel sentences. The
sampling rate was 22.05 kHz.
The proposed method was evaluated in one-to-one VC to investigate the eﬀectivenessof the phonetic latent variables (i.e., PPGs) in improving synthetic speech quality. DNNbased acoustic models were trained for each pair of source and target speakers taken fromthe ﬁrst corpus. Since the three male and three female speakers were used, there weretotally 12 inter-gender VC settings: six male-to-male (m2m) and six female-to-female(f2f), and 18 inter-gender ones: nine male-to-female (m2f) and nine female-to-male (f2m).
The 1st-through-400th utterances were divided into two subsets for making a non-parallel
VC setting: the half for source speakers and the remainders for target speakers. The
parallel 401st-through-425th utterances were used for the evaluation.
The proposed method was also evaluated in many-to-many VC. Here, DNNs for speech
recognition and speaker classiﬁcation were trained using the second corpus. After the
pretraining, VAEs were trained using the second corpus and evaluated using the ﬁrst
corpus. Note that the six speakers (i.e., the three males and the three females) were
excluded from the training data. The speakers’ some utterances were therefore used for
estimating their speaker representations.
The performances of the following four DNNs were compared:
FFNN: Feed-Forward DNNs
VAE-SC: Conventional VAEs conditioned by speaker codes [38]
VAE-SC-PPG: Proposed VAEs conditioned by speaker codes and PPGs
VAE-DV-PPG: Proposed VAEs conditioned by d-vectors and PPGs
In the one-to-one VC evaluation, “VAE-*” were trained with a completely non-parallel
speech corpus, while “FFNN” was trained with a fully-parallel speech corpus aligned with
the DTW algorithm. Therefore, “FFNN” can be regarded as the ideal baseline of the

88

Chapter 5 VAE-based Multi-spkr. Acoust. Modeling Using Speech Recog. Process

VC. The proposed “VAE-SC-PPG” and “VAE-DV-PPG” were also evaluated in many-tomany VC. In performing many-to-many VC with “VAE-SC-PPG,” each target speaker’s
speaker representation was estimated using the speaker code adaptation method [118].
This adaptation method performs the BP algorithm to ﬁnd a speaker representation thatminimizes the MSE between target and generated speech parameters.
The STRAIGHT vocoder [47] was used for extracting the 0th-through-39th mel-cepstralcoeﬃcients, 10 band-aperiodicities, log F0 , and U/V at 5 ms steps. During the training
phase, the mel-cepstral coeﬃcients were normalized to have zero-mean and unit-variance.
In the VC phase, the 1st-through-39th mel-cepstral coeﬃcients and their dynamic features
were converted by the DNNs. The input speaker’s 0th mel-cepstral coeﬃcients and bandaperiodicity were not converted. Input F0 was linearly transformed using the F0 statisticsof the source and target speakers. The MLPG algorithm [61] was performed to generatestatic mel-cepstral coeﬃcients considering their temporal dependencies.
All architectures for the DNNs and VAEs were Feed-Forward networks. The speech
recognition DNNs used a set of 56 Japanese phonemes and predicted 56-dimensional PPGs
frame by frame. The hidden layers of the DNNs had 4 × 1024-units with sigmoid nonlinearity. The speaker classiﬁcation DNNs predicted posterior probabilities of the speaker
identity. Here, in addition to the 260 seen speakers, one discrete value representing an
unvoiced region was attached to the speaker identity. The hidden layers of the speakerclassiﬁcation DNNs had 4 × 256-units with sigmoid non-linearity. Sixteen-dimensional dvectors were extracted from the bottleneck layer of the DNNs. The optimization algorithm
for pretraining the two DNNs was AdaGrad [98]. The learning rate for pretraining was set
to 0.01. The pretraining was performed with 100 iterations. The encoder networks of the
VAEs had two hidden layers with the ReLU [54] non-linearity. The ﬁrst and second hidden
layers had 256 and 128 units, respectively. The architecture for the decoder networks was
symmetric about that for the encoder. The dimensionality of the VAE latent variables
was 64. Feed-Forward DNNs (“FFNN”) were trained using fully-parallel speech corpora
including only source and target speakers. The hidden layers of the DNNs had 4 × 128
units with the ReLU non-linearity. The optimization algorithm for training the VAEs and“FFNN” was AdaGrad. The learning rate for the training was set to 0.01. All DNN-based
acoustic models for VC were trained with 25 iterations.

5.3.2

Objective Evaluation

Mel-cepstral distortions (MCDs) between target and converted mel-cepstral coeﬃcientswere calculated. The frame lengths of these mel-cepstral coeﬃcients were aligned using

Mel-cepstral distortion [dB]

Chapter 5 VAE-based Multi-spkr. Acoust. Modeling Using Speech Recog. Process

9.5
9.0
8.5
8.0
7.5
7.0
6.5
6.0

FFNN

(a) m2m

5

10

25

50

100

200 5

10

VAE-SC(b) m2f

25

50

100

VAE-SC-PPG

200 5

10

Number of utterences(c) f2m

25

50

VAE-DV-PPG

100

200 5

89

(d) f2f

10

25

50

100

200

Fig. 5.9. MCDs of converted speech in one-to-one VC. Here, only “FFNN” was trainedusing fully-parallel speech corpora.

the DTW algorithm. The number of utterances for training DNNs in one-to-one VC or
estimating speaker representations in many-to-many VC was changed. In one-to-one VC,
the four DNNs were trained using 5, 10, 25, 50, 100, or 200 utterances. In many-tomany VC, speaker representations for the target speakers were estimated using the same
numbers of utterances as used in one-to-one VC. The MCDs were averaged over all of thepossible VC settings in each of the “m2m,” “m2f,” “f2m,” and “f2f” conversion.
Figure 5.9 shows the evaluation results in one-to-one VC. The MCDs of the proposed
non-parallel “VAE-SC-PPG” and “VAE-DV-PPG” signiﬁcantly improve compared withthose of the conventional “VAE-SC,” and become closer to those of “FFNN” trained withparallel speech corpora. Moreover, the proposed methods even outperform “FFNN” when
the number of the training utterances is small. One possible reason is misalignment by
the DTW-based pre-processing; i.e., “FFNN” requires the DTW to align features, but the
proposed methods do not. Also, the MCDs of “VAE-DV-PPG” are slightly lower thanthose of “VAE-SC-PPG,” suggesting that the continuous speaker representations work
better in VAE-based non-parallel VC.
Figure 5.10 shows the evaluation results of the proposed VAE-based non-parallel andmany-to-many VC method. In all of the VC settings, the MCDs of “VAE-DV-PPG” arealways lower than those of “VAE-SC-PPG,” regardless of the number of utterances used
for the speaker adaptation. These results indicate that using d-vectors is more eﬀective
than adapting speaker codes in VAE-based non-parallel and many-to-many VC.

5.3.3

Subjective Evaluation

Subjective evaluations on naturalness and speaker similarity of converted speech wereconducted. The following six DNNs were compared: the two conventional DNNs (“FFNN”
and “VAE-SC”) and the two proposed DNNs (“VAE-SC-PPG” and “VAE-DV-PPG ”) forMel-cepstral distortion [dB]90

Chapter 5 VAE-based Multi-spkr. Acoust. Modeling Using Speech Recog. Process(a) m2m

VAE-SC-PPG(b) m2f

VAE-DV-PPG(c) f2m

25

10(d) f2f

8.4
8.2
8.0
7.8
7.6

5

10

25

50

100

200 5

10

50

100

200 5

Number of utterences

25

50

100

200 5

10

25

50

100

200

Fig. 5.10. MCDs of converted speech in VAE-based non-parallel and many-to-many VC
Table 5.1. Subjective evaluation results for naturalness of speech converted with different six DNNs and their 95% conﬁdence intervals. Bold and underlinedscores mean highest and lowest ones among ﬁve DNNs except for “FFNN,”
respectively
VC setting
One-to-one

Many-to-many

DNNs
FFNN
VAE-SCVAE-SC-PPG
VAE-DV-PPG
VAE-SC-PPG
VAE-DV-PPG

m2m
3.77±0.13
1.09±0.05
3.33±0.13
3.33±0.13
2.94±0.11
2.73±0.11m2f
3.30±0.13
1.12±0.06
3.21±0.13
3.21±0.13
2.88±0.122.75±0.12f2m
3.68±0.14
1.07±0.04
3.29±0.12
3.45±0.15
2.94±0.12
2.85±0.13

f2f
3.53±0.12
1.10±0.05
3.08±0.13
3.04±0.13
2.87±0.13
2.69±0.11

both of the one-to-one and many-to-many VC settings. The fully-parallel 400 utterances ofthe source and target speakers were only used in “FFNN.” The non-parallel 200 utteranceswere used in “VAE-*” for one-to-one VC. The target speakers’ 100 utterances were used forthe speaker representation estimation in VAE-based many-to-many VC. Five-point scale
MOS tests were conducted for evaluating the naturalness. Speech samples generated byusing each acoustic model were presented to listeners in random order. Similarly, ﬁve-point
scale diﬀerential MOS (DMOS) tests were conducted for evaluating the speaker similarity.
The reference samples were re-synthesized speech presented with corresponding converted
speech. Fifty listeners participated in each of the evaluations for “m2m,” “m2f,” “f2m,”and “f2f” conversion, using crowdsourced evaluation systems. Each listener evaluated thenaturalness or speaker similarity of 30 converted speech samples randomly selected fromall possible combinations of the test utterances and target speakers. The total number oflisteners was 400. Similar to the objective evaluation in Section 5.3.2, the results of the
MOS or DMOS tests were averaged over all of the possible VC settings.
Tables 5.1 and 5.2 shows the MOS and DMOS results, respectively. In this evaluation,
only “FFNN” was trained with fully-parallel utterances, and the scores were referredto the ideal baseline. Focusing on the one-to-one VC results, the proposed methods(“VAE-*-PPG”) achieve signiﬁcantly higher scores than the conventional “VAE-SC” inChapter 5 VAE-based Multi-spkr. Acoust. Modeling Using Speech Recog. Process

91

Table 5.2. Subjective evaluation results for speaker similarity of speech converted with
diﬀerent six DNNs and their 95% conﬁdence intervals. Bold and underlinedscores mean highest and lowest ones among ﬁve DNNs except for “FFNN,”
respectively
VC setting
One-to-one

Many-to-many

DNNs
FFNN
VAE-SCVAE-SC-PPG
VAE-DV-PPG
VAE-SC-PPG
VAE-DV-PPG

m2m
3.49±0.16
1.28±0.10
3.23±0.14
3.13±0.14
2.08±0.122.31±0.12m2f
3.20±0.14
1.21±0.08
2.90±0.13
2.87±0.13
1.92±0.12
1.94±0.11

f2m
3.13±0.14
1.24±0.09
3.17±0.14
3.04±0.13
2.14±0.12
2.33±0.11

f2f
2.99±0.15
1.27±0.09
2.91±0.13
2.84±0.14
2.09±0.12
2.10±0.12

terms of both naturalness and speaker similarity. These results demonstrate that the
speech-recognition-derived phonetic latent variables successfully improve converted speech
quality in the VAE-based non-parallel VC. Focusing on the many-to-many VC results, the
MOS and DMOS of the proposed “VAE-*-PPG” are lower than those in one-to-one VC,
although they are still superior to the conventional “VAE-SC.” This is reasonable because
the target speakers are unseen in many-to-many VC but are seen in one-to-one VC. These
results suggest that the conventional VAE-based non-parallel VC can be extended to
many-to-many VC using the proposed multi-speaker acoustic model. Regarding speakerrepresentation estimation in many-to-many VC, the speaker code adaptation is eﬀective
to improve the naturalness, while the use of d-vectors is eﬀective to improve the speakersimilarity of converted speech.

5.3.4

Eﬀects of Training Data and d-vector Dimensionality

The eﬀects of hyperparameters in the proposed VAE-based non-parallel and many-tomany VC, i.e., the number of seen speakers and d-vector dimensionality, were investigated.
The number of seen speakers was changed with three settings: 25 males and 25 females(“50spk”), 65 males and 65 females (“130spk”), and 130 males and 130 females (“260spk”).
The d-vector dimensionality was changed with six settings: 1 (“1d”), 2 (“2d”), 4 (“4d”),
8 (“8d”), 16 (“16d”), and 32 (“32d”). In total, there were 18 hyperparameter settings to
be investigated.
Evaluation of d-vector Dimensionality
The eﬀect of d-vector dimensionality was investigated. Firstly, the number of seen speakers
was ﬁxed to 260, and speech samples converted by the proposed VC method with the six
diﬀerent settings for the d-vector dimensionality (i.e., “1–32d”) were compared. Similar to
Section 5.3.3, MOS and DMOS tests were conducted for the evaluation of naturalness and92

Chapter 5 VAE-based Multi-spkr. Acoust. Modeling Using Speech Recog. Process
Table 5.3. Subjective evaluation results for naturalness of speech converted by proposed
VAE-based VC methods with diﬀerent d-vector dimensionality and their 95%
conﬁdence intervals. Bold and underlined scores mean highest and lowest
ones among six settings, respectively

1d
2d
4d
8d
16d
32d

m2m
2.60±0.12
2.71±0.12
2.80±0.13
2.79±0.13
2.85±0.13
2.94±0.14m2f
2.63±0.13
2.64±0.13
2.58±0.13
2.60±0.13
2.61±0.14
2.64±0.14

f2m
2.67±0.14
2.67±0.14
2.84±0.14
2.75±0.12
2.89±0.13
2.83±0.13

f2f
2.80±0.14
2.80±0.14
2.75±0.14
2.83±0.14
2.86±0.14
2.84±0.14

Table 5.4. Subjective evaluation results for speaker similarity of speech converted byproposed VAE-based VC methods with diﬀerent d-vector dimensionality andtheir 95% conﬁdence intervals. Bold and underlined scores mean highest andlowest ones among six settings, respectively

1d
2d
4d
8d
16d
32d

m2m
2.14±0.13
2.18±0.14
2.39±0.15
2.44±0.14
2.41±0.14
2.49±0.14m2f
2.32±0.14
2.26±0.13
2.19±0.132.42±0.14
2.39±0.13
2.47±0.14

f2m
2.01±0.13
2.01±0.13
2.40±0.14
2.45±0.13
2.36±0.14
2.43±0.15

f2f
2.11±0.14
2.12±0.13
2.07±0.13
2.18±0.13
2.25±0.14
2.22±0.13

speaker similarity of converted speech, respectively. Fifty listeners participated in each of
the evaluation, using crowdsourced evaluation systems. The total number of listeners was
400.
Tables 5.3 and 5.4 show the MOS and DMOS results, respectively. From the results
shown in Table 5.4, the speaker similarity signiﬁcantly degrades when the d-vector dimensionality is set to smaller than eight. These results are consistent with the speaker
veriﬁcation performance described in Appendix C, and suggest that the use of inaccurate
d-vectors to distinguish speaker individuality degrades speaker similarity of converted
speech. Henceforth, “1d,” “2d,” and “4d” were omitted from the following pairwise comparisons.
Secondly, the number of seen speakers was ﬁxed to 50, 130, or 260, and speech samples
converted by the proposed VC method with the three diﬀerent d-vector dimensionality
settings (i.e., “8d,” “16d,” or “32d”) were compared. Preference AB tests for the naturalness and preference XAB tests for the speaker similarity were conducted to ﬁnd optimal
settings of the d-vector dimensionality. Twenty-ﬁve listeners participated in each of theChapter 5 VAE-based Multi-spkr. Acoust. Modeling Using Speech Recog. Process

93

Table 5.5. Subjective evaluation results for naturalness of converted speech with ﬁxed
numbers of seen speakers and varied d-vector dimensionality. Bold indicatesmore preferred method with p-value < 0.05

spk
50

130

260

m2m
m2f
f2m
f2f
m2m
m2f
f2m
f2f
m2m
m2f
f2m
f2f

8d vs. 16d
0.315 - 0.685
0.442 - 0.558
0.337 - 0.663
0.368 - 0.632
0.468 - 0.532
0.492 - 0.508
0.396 - 0.604
0.512 - 0.488
0.512 - 0.488
0.500 - 0.500
0.472 - 0.528
0.524 - 0.476

16d vs. 32d
0.577 - 0.423
0.444 - 0.556
0.524 - 0.476
0.464 - 0.536
0.600 - 0.400
0.664 - 0.336
0.664 - 0.336
0.692 - 0.308
0.468 - 0.532
0.516 - 0.484
0.492 - 0.502
0.556 - 0.444

8d vs. 32d
0.396 - 0.604
0.337 - 0.663
0.384 - 0.616
0.304 - 0.696
0.428 - 0.572
0.538 - 0.462
0.415 - 0.585
0.568 - 0.432
0.532 - 0.468
0.532 - 0.468
0.552 - 0.448
0.504 - 0.496

evaluations, using crowdsourced evaluation systems. The total number of listeners was
1,800. Each listener evaluated 10 samples.
Tables 5.5 and 5.6 show evaluation results for converted speech naturalness and speakersimilarity, respectively. Two noteworthy points are found from the results. When thenumber of seen speaker is small, the use of the higher d-vector dimensionality (i.e., “16d”
and “32d”) signiﬁcantly improves both the naturalness and speaker similarity in mostcases. In contrast, these tendencies are not observed when the largest number of seen
speakers (i.e., “260spk”) is used, and “8d” even achieves higher speaker similarity than
“32d.” These results suggest that a trade-oﬀ between the d-vector dimensionality andnumber of seen speakers exists in the proposed method. In other words, the d-vector
dimensionality should be large enough to deal with unseen speakers in the VC process
when the small number of seen speakers is used.
Evaluation of Number of Seen Speakers
Thirdly, the d-vector dimensionality was ﬁxed to the best settings referring to preference
scores shown in Tables 5.5 and 5.6, and speech samples converted by the proposed VC
method with the three diﬀerent number of seen speakers (i.e., “50spk,” “130spk,” or“260spk”) were compared. Tables 5.7 and 5.8 show the subjective evaluation results
for naturalness and speaker similarity, respectively. The increase of the number of seen
speakers tends to improve both the naturalness and speaker similarity. The RMSEs
between two PPGs predicted from source and target speakers’ utterances were calculated

94

Chapter 5 VAE-based Multi-spkr. Acoust. Modeling Using Speech Recog. Process
Table 5.6. Subjective evaluation results for speaker similarity of converted speech withﬁxed numbers of seen speakers and varied d-vector dimensionality. Bold
indicates more preferred method with p-value < 0.05

spk
50

130

260

m2m
m2f
f2m
f2f
m2m
m2f
f2m
f2f
m2m
m2f
f2m
f2f

8d vs. 16d
0.450 - 0.550
0.472 - 0.528
0.420 - 0.580
0.462 - 0.538
0.428 - 0.572
0.548 - 0.452
0.484 - 0.516
0.588 - 0.412
0.452 - 0.548
0.488 - 0.512
0.464 - 0.536
0.508 - 0.492

16d vs. 32d
0.485 - 0.515
0.472 - 0.528
0.468 - 0.532
0.496 - 0.504
0.508 - 0.492
0.480 - 0.520
0.504 - 0.496
0.476 - 0.524
0.527 - 0.473
0.536 - 0.464
0.540 - 0.460
0.464 - 0.536

8d vs. 32d
0.528 - 0.472
0.476 - 0.524
0.412 - 0.588
0.438 - 0.562
0.454 - 0.546
0.528 - 0.472
0.500 - 0.500
0.560 - 0.440
0.516 - 0.484
0.460 - 0.540
0.432 - 0.568
0.569 - 0.431

Table 5.7. Subjective evaluation results for naturalness of converted speech with varied
numbers of seen speakers and best d-vector dimensionality for each setting.
Here, d-vector dimensionality was “32d” for “50spk,” “16d” for “130spk,”
and “8d” for “260spk,” respectively. Bold indicates more preferred methodwith p-value < 0.05

m2m
m2f
f2m
f2f

50spk vs. 130spk
0.316 - 0.684
0.476 - 0.524
0.364 - 0.636
0.496 - 0.504

130spk vs. 260spk
0.460 - 0.540
0.408 - 0.592
0.438 - 0.562
0.492 - 0.508

50spk vs. 260spk
0.240 - 0.760
0.440 - 0.560
0.324 - 0.676
0.473 - 0.527

to investigate the reason. The DTW algorithm was performed to align PPGs’ frame
lengths. Table 5.9 shows the results. The RMSEs tend to decrease in proportion to thenumber of seen speakers, which may account for the improvement in converted speech
quality.
Visualization of Latent Variables
Finally, scatter plots of the d-vectors and the VAE latent variables were made using
principal component analysis (PCA) to explore the latent spaces learned by DNNs. In
the following visualization, the d-vectors or the VAE latent variables were averaged over
50 utterances of each speaker. Each point in the scatter plots corresponds to each speaker,
and male and female speakers are colored with blue and red, respectively. The six speakers
used in the evaluation are marked with the texts “M*” (male) or “F*” (female).

Chapter 5 VAE-based Multi-spkr. Acoust. Modeling Using Speech Recog. Process

95

Table 5.8. Subjective evaluation results for speaker similarity of converted speech withvaried numbers of seen speakers and the best d-vector dimensionality foreach setting. Here, d-vector dimensionality was “32d” for “50spk,” “16d”
for “130spk,” and “8d” for “260spk,” respectively. Bold indicates more
preferred method with p-value < 0.05

m2m
m2f
f2m
f2f

50spk vs. 130spk
0.436 - 0.564
0.496 - 0.504
0.408 - 0.592
0.462 - 0.538

130spk vs. 260spk
0.476 - 0.524
0.540 - 0.460
0.427 - 0.573
0.532 - 0.468

50spk vs. 260spk
0.285 - 0.715
0.416 - 0.584
0.360 - 0.640
0.448 - 0.552

Table 5.9. RMSEs of PPGs predicted from source and target speakers. These results
were averaged over all evaluation data. Bold indicates lowest RMSE in each
row

m2m
m2f
f2m
f2f

50spk
0.05640.0566
0.0566
0.0563130spk
0.05340.05420.05420.0541260spk
0.05390.0541
0.0541
0.0539Figure 5.11 shows the visualization of d-vectors. The use of larger d-vector dimensionality tends to widen the distribution of speakers, and the six speakers (“M*” and “F*”)shown in Fig. 5.11(c)(3) are clearly diﬀerentiated in the speaker space. However, such a
speaker space does not always contribute to the improvement of converted speech quality
as shown in Table 5.5. These results indicate that the densely distributed speaker space
may be suited to the proposed VAE-based VC, rather that the space constructed for the
better speaker veriﬁcation performance.
Figure 5.12 shows the visualization of VAE latent variables. The “CSM” in this plotmeans the class separability measure calculated as the trace of the between-class scatter
matrix divided by that of the within-class scatter matrix [119]. Since the CSM values
were calculated regarding speakers as classes, the numerator and denominator of theCSM deﬁnition should represent inter-speaker variation and inter-phonetics variation,
respectively. The CSM values shown in Fig. 5.12 are always greater than 1.0, indicatingthat the inter-speaker variation is dominant in the latent variables. In fact, some of thesix test speakers tend to position out of the distribution of the latent variables when thesmaller number of seen speakers is used (e.g., “M1” and “M3” in Fig. 5.12(c)(1)). These
results suggest that the VAE latent variables can quantify how well the VAEs ﬁt the
speaker individuality, rather than phonetic contents suggested in [38].

96

Chapter 5 VAE-based Multi-spkr. Acoust. Modeling Using Speech Recog. Process(2) 130spk(1) 50spk(a) 8d

1
M1
M3
M2

0

F2
F3
F1

M2
M3
M1

(3) 260spk

F2
F1
F3

M3
M1
M2

F2
F1
F3

−1

(b) 16d

1
M1M2
M3

0

M1
M3
M2

F1
F3
F2

−1

M3
M1M2

F3
F1
F2

F2F1
F3

(c) 32d

1
0

F1

M3
M1
M2

F3

M3
M1

M2

F2 F1
F3

M1
M3 M2

F2

−1
−1

0

1

−1
Male

0

1

−1

0

F3
F2
F1

1

FemaleFig. 5.11. Scatter plots of d-vectors compressed with PCA. Each point denotes representation of one speaker, and three male and three female speakers used in
evaluation were marked with texts “M*” and “F*” in this plot, respectively.
To place males’ cluster in left side and females’ cluster in right side, someﬁgures were rotated 180 degrees.

5.4

SummaryThis chapter proposed the VAE-based multi-speaker speech synthesis method integrating
speech recognition and speaker classiﬁcation to synthesize versatile speakers’ high-quality
voices. This method introduces a phonetic latent variable predicted by a DNN-based
speech recognition model to improve synthetic speech quality. It also uses a speakerlatent variable extracted from a DNN-based speaker classiﬁcation model to increase thespeaker diversity of synthetic speech. VAE-based non-parallel and many-to-many VC thatcan reproduce and transform arbitrary speakers’ voice characteristics is established as aresult. Experimental results demonstrated that the phonetic latent variables signiﬁcantly

(a) 8d

Chapter 5 VAE-based Multi-spkr. Acoust. Modeling Using Speech Recog. Process(1) 50spk(2) 130spk(3) 260spk

20

CSM = 1.951624

CSM = 1.910201

CSM = 1.864400

0

M2
F3 M1
F1
F2
M3

M3
M2
F1
F2F3
M1

M3
M1
F1M2
F2
F3

−20

(b) 16d9720

CSM = 1.879926

CSM = 1.905318

CSM = 1.852439

0

M1

F2
F3
F1M2
M3

F2FM2
3
M1 M3
F1

M3F1
M1
F2
F3M2

CSM = 1.881059
M1
M3
F2
F3
F1M2

CSM = 1.843709

CSM = 1.800509

−20

−20

−20

(c) 32d

20
0

M3
F3
F1
F2
M2

M1

0

20

F3F2
F1
M2
M1M3

−20
0

20

Male

−20

0

20

FemaleFig. 5.12. Scatter plots of VAE latent variables compressed with PCA. Each point denotes latent variable averaged over 50 utterances of each speaker, and threemale and three female speakers used in evaluation were marked with texts“M*” and “F*” in this plot, respectively. “CSM” in this plot means class separability measure calculated as trace of between-class scatter matrix divided
by that of within-class scatter matrix.

improved both naturalness and speaker similarity of the converted speech. The results
also indicated that the speaker latent variables were eﬀective speaker representations for
the VAE-based non-parallel and many-to-many VC. The eﬀects of hyperparameters of
the proposed method were investigated. The investigation results indicated that 1) alarge d-vector dimensionality that gives the better speaker veriﬁcation performance did
not necessarily improve converted speech quality, and 2) a large number of seen speakers
tended to improve converted speech quality.

98

Chapter 6

Perceptual-similarity-aware
Deep Speaker Representation
Learning
6.1

Introduction

This chapter proposes novel algorithms for learning speaker representations using DNNs,
i.e., deep speaker representation learning, considering perceptual similarity among speakers. As described in Chapter 5, d-vectors, speaker representations learned by speakerclassiﬁcation DNNs, can be used for controlling speaker individuality in DNN-based multispeaker acoustic modeling. However, they do not consider the perceptual similarity among
speakers, resulting in less interpretability for the speaker individuality control. Also, theycan even worsen synthetic speech quality when the speaker is unseen [120]. The proposed
algorithms incorporate human listeners to the speaker representation learning framework,on the basis of human computation [121]. Figure 6.1 illustrates a conceptual diagram ofthe proposed deep speaker representation learning algorithms. These algorithms ﬁrst conduct large-scale scoring of perceptual speaker-pair similarity to obtain a perceptual speakersimilarity matrix, then train DNNs for speaker representation learning (i.e., speaker encoder) to minimize a loss function deﬁned by the similarity matrix. This chapter presents
three algorithms with diﬀerent representations of the similarity matrix: a set of similarity
vectors, the Gram matrix, and a graph. Similarity vector embedding regards the similarity
matrix as a set of similarity vectors and trains a speaker encoder to predict a similarity
vector from a speaker representation. Similarity matrix embedding directly utilizes theChapter 6 Perceptual-similarity-aware Deep Speaker Representation Learning

Perceptualsim. matrix

Spkr. pairs

3 (similar)

Human
listener(a) Perceptual similarity scoring
Acousticfeats.
Spkr.
encoder

െ3 (dissimilar)

Spkr.-repr.derived sim.
Similarity
prediction

Interpretablespkr. reprs.

99

Perceptual
sim.
‫כ‬

‫ܮ‬ୗ୍୑ ‫ڄ‬Loss to predict sim.
Vector Matrix Graph(b) Speaker encoder training
considering perceptual similarity
Fig. 6.1. Conceptual diagram of proposed method in Chapter 6

whole similarity matrix as the target to be predicted by a speaker encoder and trains theencoder to minimize the Frobenius norm between the Gram matrix of a set of speakerrepresentations (i.e., speaker similarity matrix derived from the representations) and theperceptual speaker similarity matrix. Similarity graph embedding deﬁnes a graph that
represents perceptual speaker similarity and trains a speaker encoder to predict a link of
the graph from a pair of speaker representations. This chapter also proposes an active
learning algorithm to reduce the number of scoring times that quadratically increases
with that of seen speakers. This algorithm iterates the perceptual speaker-pair similarity
scoring and speaker encoder training. Queries in this algorithm are generated from thesequentially-trained speaker encoder for prioritizing unscored speaker-pairs to be scorednext. Figure 6.2 illustrates the relation between conventional and proposed methods. In
experimental evaluations, large-scale scoring is ﬁrst conducted, then the proposed speakerrepresentation learning algorithms and active learning algorithm are performed.
This chapter is organized as follows (see also Fig. 6.3). Section 6.2 describes the threecomponents in the proposed method: scoring perceptual speaker-pair similarity, deep
speaker representation learning considering the similarity, and active learning. Section
6.3 presents experimental evaluations. Section 6.4 summarizes this chapter.

100

Chapter 6 Perceptual-similarity-aware Deep Speaker Representation Learning
Discreterepr.
One-hot
spkr.
code

Continuous
repr.

DNN-based
repr. learning

Introducing
human's spkr.
perception

Goal

Chapter 6(proposed)

Interpretablespkr. repr.

d-vector
[27]

[41]

Low

High
Interpretability

Fig. 6.2. Relation between conventional and proposed methods in Chapter 6
Conventional
methods

Chapter 6: interpretable spkr. repr. learning
Sect. 6.2: proposed methods

Human
computation[121]

Integrating

Activelearning of
spkr. repr.

IteratingSubjective
scoring

d-vectors [27]Activelearning [123]Sim.-aware
spkr. repr.
learning

Introducing
Sect. 6.3: evaluations in multi-spkr. speech synth.
Correlation
analysisSpeaker interpolation

Speaker
adaptation

Effectiveness ofactive learning

Fig. 6.3. Overview of Chapter 6

6.2

Deep Speaker Representation Learning
Considering Perceptual Speaker-pair Similarity

The introduction of human speaker-similarity perception into deep speaker representation
learning can improve controllability of a DNN-based multi-speaker acoustic model andthe adaptability of the model to unseen speakers. In this section, a speaker similarity
matrix obtained by perceptual scoring is ﬁrst introduced. Speaker representation learningalgorithms using the matrix are then described. An active learning algorithm to reduce
the scoring and training costs is ﬁnally presented.

Chapter 6 Perceptual-similarity-aware Deep Speaker Representation Learning

101

Spkr. pair pool
: ??
: ??
: ??

To what degree do these two
speakers' voices soundsimilar?
(−3: dissimilar 䡚 +3: similar)

: ??
: ??

“+2”
Score

Audio
Listener

Fig. 6.4. Perceptual scoring of speaker-pair similarity. Speaker-pair pool stores speakerpairs to be scored. Listener is asked to score perceptual similarity of twopresented speakers’ voices as integer between −v and v. In this ﬁgure, v = 3.

6.2.1

Perceptual Speaker Similarity Matrix

A perceptual speaker similarity matrix that represents the pairwise speaker similarity
perceived by listeners is deﬁned. Let S = [s1 , · · · , si , · · · , sNs ] be an Ns -by-Ns symmetric similarity matrix and si = [si,1 , · · · , si,j , · · · , si,Ns ]⊤ be an Ns -dimensional similarity
vector of the ith speaker. Each element si,j takes a value between −v and v that represents the perceptual similarity of the ith and jth speakers. The element si,j is deﬁned
as the average score of perceptual scoring that asks listeners “To what degree do the ith
speaker’s voice and the jth speaker’s sound similar? Please answer the degree of similarity
as a value between −v and v.” The diagonal elements si,i , i.e., intra-speaker perceptual
similarity, are assumed to take the maximum value v. Figure 6.4 illustrates the perceptual
scoring process. Figures 6.5(a) and (b) show a perceptual speaker similarity matrix of 153
female Japanese speakers and its sub-matrix, respectively. Please see Section 6.3.1 fordetails of the perceptual scoring and Section 6.3.2 for the analysis results of the scores.

6.2.2

Perceptual-similarity-aware Deep Speaker Representation
Learning Algorithms

This section presents three algorithms for learning similarity-aware speaker representations with diﬀerent usage of the perceptual speaker similarity matrix: a set of similarity
vectors, the Gram matrix, and a graph.

102

Chapter 6 Perceptual-similarity-aware Deep Speaker Representation Learning(a) Full-matrix(153 female speakers)(b) Sub-matrix(13 female speakers)
F001
F002
F003
F004
F005
F006
F007
F008
F009
F010
F011
F012
F013

3
2
1
0
−1
−2
−3

Fig. 6.5. (a) Perceptual speaker similarity matrix of 153 female Japanese speakers obtained by large-scale perceptual scoring and (b) its sub-matrix

Similarity Vector Embedding
The ﬁrst algorithm uses a speaker similarity vector as the target to be predicted by aspeaker encoder. A loss function for the training is deﬁned as follows:(vec)

LSIM (s, ŝ) =1⊤(ŝ − s) (ŝ − s) ,
Ns

(6.1)

where s ∈ S and ŝ denote a target similarity vector and output vector of the DNNs,
respectively. This algorithm can be regarded as speaker classiﬁcation based on continuousvalued speaker identity considering perceptual speaker similarity. Figure 6.6(a) shows the(vec)computation procedure of LSIM (·).
Similarity Matrix Embedding
The second algorithm directly uses a perceptual speaker similarity matrix as aconstraint on coordinates of speaker representations in the speaker space.

Let

D = [d1 , · · · , di , · · · , dNs ] be an Nd -by-Ns matrix including all seen speakers’ representations. A loss function for the training is deﬁned as follows:(mat)

LSIM (D, S) =2
2
∥1Ns − INs ∥F

eD −S
e
K

e D = KD − (KD ◦ IN ) ,K
se = S − vIN ,
S
s

2

,

(6.2)

F

(6.3)
(6.4)

where ∥·∥F , 1Ns , and INs denote the Frobenius norm of a given matrix, an Ns -by-Ns
matrix whose components are all 1, and the Ns -by-Ns identity matrix, respectively. The

Chapter 6 Perceptual-similarity-aware Deep Speaker Representation Learning

Speechparams.

Outputvector࢙ො

Targetsim. vector࢙Sim. matrix

‫܁‬

୴ୣୡ

Spkr.
encoder‫ڮ‬‫ܮ‬ୗ୍୑ ‫ڄ‬(a) Similarity vector embedding۲ୃ݇ ࢊଵ ǡ ࢊࡺ౩‫ݏ‬ଵǡே౩

‫܁‬

‫ڮ‬

ܰୱ

1

୫ୟ୲

‫ܮ‬ୗ୍୑

ܰୱ

1
‫ڄ‬
‫ڮ‬

‫ڮ‬

‫ڮ‬

ࢊே‫ܛ‬

Sim. matrix

۲

1

ܰୱ th
spkr.

Gram matrix
۹۲

ܰୱ

Calc.
kernel
݇ ‫ڄ‬

1

‫ڮ‬

ࢊଵ

1st
spkr.

103

ܰୱ

(b) Similarity matrix embedding
ࢊଵ

1st
spkr.
‫ڮ‬

ܰୱ th
spkr.

ࢊே‫ܛ‬

Sim. graphLink
prediction୥୰ୟ୮୦‫݌‬ଵǡே౩

‫ܮ‬ୗ୍୑

‫ڄ‬

ܽଵǡே౩

1:
connected
0: disconnected

1
ܽଵǡே౩

ܰୱ

(c) Similarity graph embedding
Fig. 6.6. Calculation of loss functions in proposed algorithms based on (a) similarity
vector embedding, (b) similarity matrix embedding, and (c) similarity graphembeddingnormalization coeﬃcient 2/∥1Ns − INs ∥2F corresponds to the degrees of freedom of thee D − S.
e The Gram matrix of a set of speaker representations KD is deﬁned as:
matrix K
k (d1 , d1 ) · · ·..
..
KD = 
.
.
k (dNs , d1 ) · · ·k (d1 , dNs )

..
,
.

k (dNs , dNs )(6.5)

where k(di , dj ) is a kernel function of a pair of speaker representations di and dj , i.e.,
speaker similarity derived from the speaker representations. This proposed algorithm
therefore makes the speaker-representation-derived similarity closer to the perceptual sim-

104

Chapter 6 Perceptual-similarity-aware Deep Speaker Representation Learning

Fig. 6.7. Speaker similarity graph deﬁned by similarity matrix shown in Fig. 6.5(b).
Each node represents one speaker, and links connect perceptually similar pairs(i.e., si,j > 0). Wider links indicate more similar speaker pairs.

(mat)ilarity. Figure 6.6(b) shows the computation procedure of LSIM (·).
Similarity Graph Embedding
The third algorithm learns relations among speakers deﬁned by a subjective speaker similarity matrix. Let G be a speaker similarity graph deﬁned by the matrix S. Each nodeof the graph G represents the speaker identity of one speaker, and a link connects a pairof similar speakers, as shown in Fig. 6.7. An Ns -by-Ns adjacency matrix A is deﬁned todetermine the existence of the links based on elements of the similarity matrix S. A loss
function for the training is deﬁned as follows:(graph)LSIM (D, A) = −NsXi,j=1,i̸=jai,j log pi,j −NsX(1 − ai,j ) log (1 − pi,j ) ,(6.6)

i,j=1,i̸=jwhere ai,j and pi,j denote an element of the adjacency matrix and link probability, respectively. The link probability is deﬁned as pi,j = exp(−||di − dj ||22 ) referring to [122].
(graph)Figure 6.6(c) shows the computation procedure of LSIM

6.2.3

(·).

Active Learning of Perceptual-similarity-aware Speaker
Representations

The proposed speaker representation learning algorithms require the perceptual scoringof speaker similarity, and the number of scoring times quadratically increases with thatof seen speakers Ns as well as the training time. This section proposes an active learning

Chapter 6 Perceptual-similarity-aware Deep Speaker Representation Learning

105

Vector Matrix Graph
Spkr. pair pool
Scored
: +3

Score

: -1

Annotation

Spkr. encoder
training
Spkr. encoder

Unscored
: ??

Query
selection

: ??
: ??

Score
prediction

Selected: HSF

Predicted
: +3

: USF

: 0

: LSF

: -2

Fig. 6.8. Active learning of perceptual-similarity-aware speaker representations

algorithm to reduce the scoring and training costs. Active learning [123] is a generalframework to sequentially train a machine learning model with a small labeled dataset
and large unlabeled one. It iterates 1) model training with the labeled dataset and 2)
query selection to increase labeled data.
Figure 6.8 shows the active learning of the proposed deep speaker representation learning. In the active learning, the Ns C2 seen speaker pairs are divided into two subsets: 1)
scored pairs Ds and 2) the remaining unscored ones Du . The similarity scores of speakerpairs in Du are unobserved initially.
Speaker Encoder Training Using Scored Pairs
A speaker encoder is trained using scored pairs Ds to learn the perceptual similarity among
them. The loss function for the training is any of the proposed similarity vector, matrix,
or graph embedding algorithms, i.e., Eqs. (6.1), (6.2), or (6.6).
Query Selection from Unscored Pairs
The trained speaker encoder ﬁrst predicts queries (i.e., tentative similarity scores of unscored pairs Du ) that indicate which of the pairs should be scored preferentially. Then,an oracle (e.g., a human annotator) annotates scores to speaker pairs with higher priority. A query strategy has an important role in the query selection since it determines

106

Chapter 6 Perceptual-similarity-aware Deep Speaker Representation Learning

the priority of scoring. Three query strategies are investigated in this chapter: 1) lowersimilarity ﬁrst (LSF) that selects a speaker pair whose predicted similarity is closer to −v,2) higher-similarity ﬁrst (HSF) that corresponds to the inverse version of the LSF, and
3) uncertain-similarity ﬁrst (USF) that selects a speaker pair whose predicted similarity
is closer to 0.

6.2.4

Discussion

Regarding prior work, Tachibana et al. [124] and Ohta et al. [125] proposed controllable
statistical speech synthesis in the HMM and GMM era. They modeled a single speaker’s
voice characteristics with a pair of subjective impression words, e.g., ”warm – cold” and”clear – hoarse,” as latent variables of the HMMs and GMMs. The proposed speakerrepresentation learning algorithms extend these ideas to make DNNs learn the pairwise
speakers’ perceptual similarity, rather than the conventional pointwise speaker’s voice
impression. Furthermore, one can model the relation between a speaker’s intention andlistener’s perception (e.g., diﬀerence in emotion perception [126]) by using the algorithms.
The similarity matrix embedding in Section 6.2.2 can directly learn relations among
speakers as the Gram matrix. One can choose an arbitrary kernel function to construct
the speaker space. When the inner product is used as the kernel function, Eq. (6.2) isequivalent to deep clustering [127] (except for the diagonal component subtraction). Not
only such a simple kernel but also a more complicated one can be utilized.
The similarity graph embedding in Section 6.2.2 introduces knowledge of graph embedding [128] to deep speaker representation learning. Therefore, one can further incorporategraph signal processing [129] and graph neural networks [130] to this algorithm for bettermodeling.
The proposed active learning algorithm in Section 6.2.3 can be regarded as humanin-the-loop (HITL) learning [131] of speaker representations considering human speech
perception. From this viewpoint, one can extend the HITL learning framework to speechsynthesis that considers a human listener’s speech quality assessment for acoustic modeling(e.g., the GAN training incorporating a human-based discriminator [132]).

Chapter 6 Perceptual-similarity-aware Deep Speaker Representation Learning

6.3

Experimental Evaluation

6.3.1

Experimental Conditions

107

Conditions for Large-scale Perceptual Scoring
Large-scale perceptual scoring was conducted to obtain the similarity matrix S using the
JNAS corpus [133] that includes 153 Japanese female speakers. Each speaker utters atleast 150 reading-style utterances (totaling about 44 hours). Five non-parallel utterances
per speaker were used for scoring text-independent perceptual similarity among the speakers. Each listener scored the perceptual similarity of 34 randomly-selected speaker pairs
extracted from all of the 11,628 possible diﬀerent speaker pairs with an integer between−3 (very dissimilar) and +3 (very similar). Listeners were recruited using Lancers*1 , awell-known crowdsourcing platform in Japan. At least 10 diﬀerent listeners scored thesimilarity of one of the 11,628 speaker pairs. The total numbers of listeners and answers
were 4,060 and 138,040, respectively.
Conditions for Deep Speaker Representation Learning
The JNAS corpus was used for training a DNN-based speaker encoder. The 13 speakers
shown in Fig. 6.5(b) (from “F001” to “F013”) were assumed to be unseen during thetraining. In the training, 90% of the remaining 140 seen speakers’ utterances were used.
The number of utterances per speaker was balanced. In the evaluation, the unseen speakers’ 50 utterances and the seen speakers’ remaining ones were used. The ﬁve utterancesused for the perceptual scoring were omitted from both the training and evaluation data.
During the training, each element in the similarity matrix was normalized to be in [−1,+1] for the similarity vector or matrix embedding and in [0, 1] for the similarity graphembedding. Accordingly, the sigmoid kernel k(di , dj ) = tanh(d⊤i dj ) was used for theGram matrix calculation in Eq. (6.5). In the similarity graph embedding, the adjacency
matrix was deﬁned using the normalized similarity matrix that represents the likelihoodof a link existence as a value between [0, 1].
The DNN architecture for the speaker encoder was a Feed-Forward network that included four hidden layers with the tanh activation function. The numbers of hidden units
at the ﬁrst-through-third layers and the fourth layer for the speaker representation extraction were 256 and 8, respectively. In the d-vector learning (Section 5.2.2) and the*1 https://www.lancers.jp

108

Chapter 6 Perceptual-similarity-aware Deep Speaker Representation Learningsimilarity vector embedding, an output layer with 140 units was prepared. The output
activation function was the softmax for the former and the tanh for the latter. The inputof the speaker encoder was a joint vector of the 1st-through-39th mel-cepstral coeﬃcientsand their dynamic features. The STRAIGHT vocoder [47] was used to extract the melcepstral coeﬃcients. The mel-cepstral coeﬃcients were normalized to have zero-mean andunit-variance during the training. The optimization algorithm was AdaGrad [98]. The
learning rate was set to 0.01. The number of iterations for the training was 100.
Conditions for Multi-speaker Statistical Speech Synthesis
A VAE-based multi-speaker acoustic model (described in Chapter 5) was trained. The
DNN architecture for the speech recognition model was a Feed-Forward network that
included four hidden layers with the tanh activation function. The number of hidden
units was 1,024. The recognition model was trained to predict 43-dimensional Japanese
PPGs [111] from the same input vector as the speaker encoder. At least 50 utterances foreach of the 140 seen speakers were used for the training. The number of iterations for the
training was 100. The DNN architecture for the VAEs was a Feed-Forward network thatconsisted of encoder and decoder networks. The encoder had two hidden layers with the
ReLU [54] activation function and extracted the 64-dimensional latent variables from ajoint vector of the static-dynamic mel-cepstral coeﬃcients and PPGs. The ﬁrst and second
hidden layers had 256 and 128 units, respectively. The decoder reconstructed the input
static-dynamic mel-cepstral coeﬃcients from a joint vector of the VAE latent variables,
PPGs, and 8-dimensional speaker representations. The DNN architecture for the decoder
was symmetric about that for the encoder. The VAEs were trained to maximize thevariational lower bound of the log likelihood [37] with 25 iterations using the same training
data as that used in the speaker encoder training. The optimization algorithm for the
speech recognition model and VAEs was AdaGrad. The learning rate was set to 0.01.
The MLPG algorithm [61] was used to generate static mel-cepstral coeﬃcients considering
their temporal dependencies. The generated mel-cepstral coeﬃcients and original speech’s
excitation parameters (i.e., F0 and ﬁve band-aperiodicity [44, 96]) were used for speech
waveform synthesis using the STRAIGHT vocoder [47].
Conditions for Active Learning
In active learning, the 140 seen speakers were divided into two groups (the ﬁrst 70 speakers
and the remaining), and speaker similarity scores among the diﬀerent groups were assumed
to be unobserved as shown in Fig. 6.9(b). The active learning alternatively performed 1)
the speaker encoder training using observed similarity scores with one iteration and 2) theChapter 6 Perceptual-similarity-aware Deep Speaker Representation Learning(a) Fully scored (FS)109(b) Partially scored (PS)

N/A
N/A
Fig. 6.9. (a) Fully scored (FS) and (b) partially scored (PS) settings

query selection using the trained speaker encoder. The number of queries per one active
learning iteration was set to 43 empirically. The number of active learning iterations was
115. Other conditions were the same as the previously described ones.

6.3.2

Analysis of Perceptual Similarity Scores

The perceptual similarity scores that made the similarity matrix shown in Fig. 6.5(a)were analyzed. Figure 6.10 shows a histogram of all the scores. Approximately 70%of the scores are smaller than zero. Figure 6.11 shows a histogram of speaker-pairwisescores of the 13 unseen speakers. The score distributions of dissimilar speaker pairs (e.g.,“F001-F009”) have a lower variance than those of similar ones (e.g., “F010-F011”). These
results suggest that the listeners easily ﬁnd dissimilar speakers rather than similar ones.
The similarity matrix is available online*2 .

6.3.3

Evaluation in Deep Speaker Representation LearningThis section investigated whether the proposed representation learning algorithms learn
speaker representations that consider perceptual speaker similarity and achieve highquality multi-speaker statistical speech synthesis. The following four algorithms werecompared:• d-vec.: Minimizing Eq. (5.4) [27]• Prop. (vec): Minimizing Eq. (6.1)*2 http://sython.org/demo/JSPS-DC1/index.html

Chapter 6 Perceptual-similarity-aware Deep Speaker Representation Learning

40,0001.0

30,000

0.8

Cumulative ratio

Frequency

110

0.6

20,0000.4

10,000
0

0.2
−3

−2

−1
0
1
Similarity score

2

3

0.0

Fig. 6.10. Histogram of perceptual similarity scores of 153 female Japanese speakers.
Red line denotes cumulative ratio.
Score distribution

F001-F002
F001-F003
F001-F004
F001-F005
F001-F006
F001-F007
F001-F008
F001-F009
F001-F010
F001-F011
F001-F012
F001-F013
F002-F003
F002-F004
F002-F005
F002-F006
F002-F007
F002-F008
F002-F009
F002-F010
F002-F011
F002-F012
F002-F013
F003-F004
F003-F005
F003-F006
F003-F007
F003-F008
F003-F009
F003-F010
F003-F011
F003-F012
F003-F013
F004-F005
F004-F006
F004-F007
F004-F008
F004-F009
F004-F010
F004-F011
F004-F012
F004-F013
F005-F006
F005-F007
F005-F008
F005-F009
F005-F010
F005-F011
F005-F012
F005-F013
F006-F007
F006-F008
F006-F009
F006-F010
F006-F011
F006-F012
F006-F013
F007-F008
F007-F009
F007-F010
F007-F011
F007-F012
F007-F013
F008-F009
F008-F010
F008-F011
F008-F012
F008-F013
F009-F010
F009-F011
F009-F012
F009-F013
F010-F011
F010-F012
F010-F013
F011-F012
F011-F013
F012-F013

3
2
1
0
-1
-2
-3

0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0.0

Fig. 6.11. Histogram of perceptual similarity scores of 13 speakers (from “F001” to“F013”)• Prop. (mat): Minimizing Eq. (6.2)• Prop. (graph): Minimizing Eq. (6.6)
Correlation Analysis of Speaker Representations
The Pearson correlation coeﬃcient between the normalized similarity scores si,j andpredicted similarity, i.e., values of the kernel function k(di , dj ) in “d-vec.,” “Prop. (vec),”and “Prop. (mat)” or the link probability pi,j in “Prop. (graph),” was calculated. Figure
6.12 shows the scatter plots of the similarity scores and predicted similarity with theircorrelation coeﬃcients. “Prop. (*)” learn speaker representations that have a stronger
correlation with the similarity scores than “d-vec.,” demonstrating that the proposed
speaker representations consider perceptual similarity among speakers. Among the fouralgorithms, “Prop. (graph)” achieves the strongest correlation not only in (a) “Seen-Seen”
but also in (b) “Seen-Unseen“ speaker-pair cases. This result indicates that the graphembedding-based learning algorithm works the best to learn pairwise relations among
speakers.

Chapter 6 Perceptual-similarity-aware Deep Speaker Representation Learning

Similarity scores(b) Seen-Unseen(a) Seen-Seen1111(1) d-vec.

(2) Prop. (vec)(3) Prop. (mat)(4) Prop. (graph)r = 0.0970

r = 0.3319

r = 0.6785

r = 0.7894

r = 0.1968

r = 0.2973

r = 0.5371

r = 0.6474

0
1

0

0

1 0

1 0

1 0

1

Predicted similarity
Fig. 6.12. Scatter plots of similarity scores and predicted similarity with their correlation coeﬃcient r

Performance in Similar Speaker-pair Detection
A receiver operating characteristic (ROC) curve [134] of a binary classiﬁer that detects
similar speaker pairs using speaker representations was created. An ROC curve represents
the performance of a binary classiﬁer as a true positive rate against a false positive rate atvarious threshold value settings. The closer the curve follows the upper left corner (i.e., afalse positive rate of zero and a true positive rate of one regardless of the threshold value
settings), the more accurate the classiﬁer is. Figure 6.13 shows ROC curves of similar
speaker-pair detection using speaker representations learned by the four diﬀerent algorithms. Here, “similar speaker-pair” is deﬁned as a pair of two speakers whose perceptual
similarity is greater than 0. From this ﬁgure, the proposed algorithms successfully makethe ROC curves closer to the upper left corner while the conventional algorithm does not.
The area under the ROC curve (AUC) [135] that quantiﬁes the performance of a binaryclassiﬁer as a scalar value between 0.5 (random classiﬁcation) and 1.0 (perfect classiﬁcation) was calculated. Table 6.1 shows the AUC values calculated with the ROC curves
shown in Fig. 6.13. The AUC values of “d-vec.” are the lowest among the four algorithms, suggesting that the conventional speaker-classiﬁcation-based learning algorithm
never considers perceptual similarity among speakers. On the other hand, the three proposed algorithms increase the AUC successfully, and “Prop. (graph)” achieves the AUC

112

Chapter 6 Perceptual-similarity-aware Deep Speaker Representation Learning(a) Seen-Seen

1.0(b) Seen-Unseen

True positive rate

0.8
0.6
0.4
d-vec.
Prop. (vec)
Prop. (mat)Prop. (graph)

0.2
0.0
0.0

0.2

0.4

0.6

0.8

1.0 0.0
0.2
False positive rate

0.4

0.6

0.8

1.0

Fig. 6.13. ROC curves of similar speaker-pair detection using speaker representations.
When curve becomes closer to left corner, speaker representations can beused to ﬁnd similar speaker pairs more accurately.
Table 6.1. AUC values of similar speaker-pair detection using speaker representations
learned by four diﬀerent algorithms

Seen-Seen
Seen-Unseen

d-vec.
0.57110.6347

Prop. (vec)0.69090.6909Prop. (mat)0.88470.7712

Prop. (graph)
0.92040.8157higher than 0.8 even in the “Seen-Unseen” speaker-pair case. These results demonstratethat the proposed algorithms construct the speaker space where similar speaker pairs areaccurately found using their speaker representations.
Subjective Evaluation in VAE-based Multi-speaker Speech Synthesis
The eﬀectiveness of the proposed speaker representations was investigated in speaker
adaptation of the VAE-based multi-speaker acoustic model. The speaker adaptation aimed
to reconstruct the 13 unseen speakers’ voices using their speaker representations and thetrained VAEs. Subjective evaluations on naturalness and speaker similarity of unseen
speakers’ synthetic speech were conducted. Fifty utterances of each unseen speaker wereused for the speaker representation extraction. Speech samples were synthesized using
mel-cepstral coeﬃcients generated by the trained VAEs with the four diﬀerent speaker
encoders. Preference AB tests on the naturalness were conducted to compare the con-

Chapter 6 Perceptual-similarity-aware Deep Speaker Representation Learning

113

Table 6.2. Preference scores on naturalness of synthetic speech (left: conventional dvector, right: proposed algorithm). Bold indicates more preferred methodwith p-value < 0.05

Speaker
F001
F002
F003
F004
F005
F006
F007
F008
F009
F010
F011
F012
F013

Prop. (vec)0.408 - 0.592
0.456 - 0.544
0.416 - 0.584
0.452 - 0.548
0.380 - 0.620
0.400 - 0.600
0.424 - 0.576
0.436 - 0.564
0.428 - 0.572
0.436 - 0.564
0.460 - 0.540
0.436 - 0.564
0.428 - 0.572

Prop. (mat)0.456 - 0.544
0.456 - 0.544
0.444 - 0.556
0.460 - 0.540
0.484 - 0.516
0.452 - 0.548
0.484 - 0.516
0.384 - 0.616
0.492 - 0.508
0.464 - 0.536
0.428 - 0.572
0.460 - 0.540
0.436 - 0.564

Prop. (graph)0.388 - 0.612
0.444 - 0.556
0.444 - 0.556
0.448 - 0.552
0.432 - 0.568
0.448 - 0.552
0.424 - 0.576
0.392 - 0.608
0.364 - 0.636
0.448 - 0.552
0.312 - 0.688
0.440 - 0.560
0.372 - 0.628

ventional algorithm (“d-vec.”) with any of the three proposed algorithms (“Prop. (*)”).
Twenty-ﬁve listeners participated in each of the following evaluations by using crowdsourced evaluation systems. Each listener evaluated 10 speech samples randomly extracted
from the 50 utterances of each unseen speaker. Similarly, a series of preference XAB tests
on the speaker similarity were conducted using the natural speech of the unseen speakers
as the reference speech samples “X.” The total number of task sets was 2 (AB or XAB)
× 3 (“d-vec.” vs. “Prop. (*)”) × 13 (unseen speakers) × 25 (listeners per one task set)
= 1,950.
Tables 6.2 and 6.3 list the preference scores on the naturalness and speaker similarity,
respectively. “Prop. (vec)” and “Prop. (graph)” always achieve higher scores than “dvec.” regarding both the naturalness and speaker similarity. These results indicate thatthe proposed similarity-aware speaker representations improve synthetic speech quality inthe speaker adaptation. “Prop. (mat)” also improves the naturalness; however, it significantly degrades the speaker similarity in a number of cases (e.g., “F005” and “F012”).
One of the reasons may be the algorithm’s trend in overﬁtting to seen speakers since it
directly utilizes the similarity scores to determine the speakers’ positions in the speakerspace.
A ﬁve-point scale MOS test on the naturalness and DMOS test on the speaker similarity
were conducted to compare the three proposed algorithms. Fifty listeners participated ineach of the following evaluations by using crowdsourced evaluation systems. Each listener
evaluated 30 speech samples randomly extracted from the 650 (50 × 13) utterances. This

114

Chapter 6 Perceptual-similarity-aware Deep Speaker Representation Learning
Table 6.3. Preference scores on speaker similarity of synthetic speech (left: conventional
d-vector, right: proposed algorithm). Bold indicates more preferred methodwith p-value < 0.05

Speaker
F001
F002
F003
F004
F005
F006
F007
F008
F009
F010
F011
F012
F013

Prop. (vec)0.436 - 0.564
0.468 - 0.532
0.432 - 0.568
0.380 - 0.620
0.428 - 0.572
0.428 - 0.572
0.492 - 0.508
0.424 - 0.576
0.400 - 0.600
0.432 - 0.568
0.348 - 0.652
0.492 - 0.508
0.372 - 0.628

Prop. (mat)0.488 - 0.512
0.496 - 0.504
0.504 - 0.496
0.404 - 0.596
0.616 - 0.384
0.444 - 0.556
0.568 - 0.432
0.500 - 0.500
0.500 - 0.500
0.404 - 0.596
0.444 - 0.556
0.544 - 0.456
0.564 - 0.436

Prop. (graph)
0.412 - 0.588
0.384 - 0.616
0.440 - 0.560
0.448 - 0.552
0.432 - 0.568
0.376 - 0.624
0.452 - 0.548
0.400 - 0.600
0.428 - 0.572
0.420 - 0.580
0.356 - 0.644
0.484 - 0.516
0.436 - 0.564

Table 6.4. Results of MOS test on naturalness and DMOS test on speaker similarity
with their 95% conﬁdence intervals (comparison among three proposed algorithms)

MOS
DMOS

Prop. (vec)3.21±0.09
2.98±0.10

Prop. (mat)3.16±0.09
2.82±0.09

Prop. (graph)3.23±0.09
2.90±0.10

evaluation enables comparing average performances of the three proposed algorithms. The
total number of task sets was 2 (MOS or DMOS) × 50 (listeners per one task set) = 100.
Table 6.4 shows the MOS and DMOS evaluation results. “Prop. (graph)” achieves thehighest MOS among the three algorithms, although there are no signiﬁcant diﬀerencesamong the scores. “Prop. (vec)” outperforms the others regarding the DMOS, and there
is a signiﬁcant diﬀerence between the scores of “Prop. (vec)” and “Prop. (mat).”Subjective Evaluation in Speaker Interpolation
The eﬀectiveness of the proposed speaker representations was investigated in speaker
interpolation [136], a technique to artiﬁcially produce new voice characteristics by mixing
two (or more) speakers’ voices. Better speaker interpolation should satisfy high naturalness and high controllability of interpolated speech. In other words, it should not
deteriorate the speech quality and should provide a way to control the interpolated voice
characteristics intuitively. The conventional and proposed speaker representations were
evaluated in speaker interpolation using a convex combination of speaker representations

Chapter 6 Perceptual-similarity-aware Deep Speaker Representation Learning

115

Table 6.5. Results of MOS test on naturalness of interpolated speech with their 95%
conﬁdence intervals. Bold scores are signiﬁcantly higher than those of d-vec.
(p < 0.05)

F033-F134
F017-F149

d-vec.
2.88±0.13
3.29±0.15

Prop. (vec)3.07±0.13
3.32±0.14

Prop. (mat)3.10±0.13
3.32±0.14

Prop. (graph)
3.16±0.13
3.38±0.14

to interpolate their voice characteristics [137, 138]. Formally, an interpolated representation is calculated as dAB = (1−α)dA +αdB using two speaker representations dA and dB ,where 0 ≤ α ≤ 1 is an interpolation coeﬃcient. In the speaker interpolation evaluation,
two speaker pairs, 1) the most dissimilar pairs (“F033-F134”) and 2) the most similar
ones (“F017-F149”), were considered to be mixed with α ∈ {0.0, 0.25, 0.5, 0.75, 1.0}.
A ﬁve-point scale MOS test on naturalness of interpolated speech (α = 0.5) was conducted to compare speaker representations learned by the four diﬀerent algorithms. Fifty
listeners participated in the MOS test by using crowdsourced evaluation systems. Each listener evaluated 16 samples of the interpolated speech. The total number of task sets was 2(“F033-F134” or “F017-F149”) × 50 (listeners per one task) = 100. Table 6.5 lists resultsof the MOS test. From the results of the interpolation using the most dissimilar speaker
pair (“F033-F134”), the three proposed algorithms achieve higher MOS values than theconventional d-vector, and the similarity matrix and graph embedding signiﬁcantly outperform the d-vector. “Prop. (*)” also outperform “d-vec.” in the interpolation using
the most similar speaker pair (“F017-F149”), although there are no signiﬁcant diﬀerencesamong the MOS values. These results indicate that the proposed speaker representations
improve the synthetic speech quality not only in speaker adaptation but also in speaker
interpolation.
A variant of the preference XAB test was conducted to evaluate the speaker similarity
of interpolated speech. In the evaluation, listeners ﬁrst played three speech samples interpolated with diﬀerent coeﬃcients: “X” (α ∈ {0.0, 0.25, 0.5, 0.75, 1.0}), “A” (α = 0.0), and“B” (α = 1.0), and then answered which of the two samples “A” or “B” sounded similar
to “X.” Thirty listeners participated in the evaluation by using crowdsourced evaluation
systems. Each listener evaluated 20 speech samples. The total number of task sets was 2(“F033-F134” or “F017-F149”) × 30 (listeners per one task) = 60. Figure 6.14 shows thepreference score curves against the ﬁve diﬀerent interpolation coeﬃcients. The shapes of
the curves signiﬁcantly change between Figs. 6.14(a) and (b). This result suggests thatperceptual similarity of a speaker pair greatly aﬀects the result of speaker interpolation;i.e., the more dissimilar a speaker pair is, the larger the diﬀerence of the interpolated

116

Chapter 6 Perceptual-similarity-aware Deep Speaker Representation Learning

Preference XAB scores

1.0(a) Most dissimilar speaker pair(b) Most similar speaker pair

0.5

0.0
0.0

d-vec.
Prop. (vec)
Prop. (mat)Prop. (graph)
0.5

1.0 0.0

Interpolation coefficient

0.5

1.0

Fig. 6.14. Results of XAB tests on speaker similarity of the interpolated speech using (a)most dissimilar speaker pair (“F033-F134”) and (b) most similar one (“F017F149”). When score curves become closer to red lines, listeners accurately
infer two speakers’ mixing ratio from interpolated speech.

speech becomes. Red lines representing “interpolation coeﬃcients and preference XAB
scores are equal” were added to Fig. 6.14 for illustrating this observation clearly. Whenthe score curves become closer to the red lines, listeners accurately infer two speakers’
mixing ratio from interpolated speech. All the preference scores shown in Fig. 6.14(b) arenear 0.5 regardless of the interpolation coeﬃcient settings, i.e., listeners hardly perceived
the diﬀerences among the interpolated speech samples. This is a natural result becausevery similar speakers’ voices are mixed, and listeners therefore cannot detect the diﬀerence
between the two speech samples “A” (α = 0.0) and “B” (α = 1.0). One can quantify thecontrollability of speaker interpolation using a pair of dissimilar speakers as the distance
between the red line and the score curve in Fig. 6.14(a). The linear least squares error
between the red line and preference scores were calculated. The calculation results of “dvec.,” “Prop. (vec),” “Prop. (mat),” and “Prop. (graph)” were 0.1139, 0.0654, 0.0559,
and 0.0429, respectively. Therefore, the consideration of perceptual similarity among
speakers in deep speaker representation learning improves not only the quality but also
the controllability of synthetic speech in the VAE-based multi-speaker speech synthesis.

Chapter 6 Perceptual-similarity-aware Deep Speaker Representation Learning

6.3.4

117

Evaluation of Active Learning

The active learning’s eﬀectiveness was investigated using each of the three proposed representation learning algorithms independently. In addition to the three query strategies
described in Section 6.2.3 (i.e., “LSF”, “HSF,” and “USF”), “FS” and “PS” were compared. The former and latter trained a speaker encoder using the fully observed scoresand partially observed ones shown in Figs. 6.9(a) and (b), respectively.
Evaluation of AUC Improvement
How the proposed active learning aﬀected the AUC of similar speaker-pair detection wasinvestigated. Figure 6.15 shows the AUC curves against the active learning iterations.
Red and blue lines denote the ﬁnal AUC values of “FS” and “PS,” respectively. Note thatthe ﬁnal AUC values of “LSF,” “HSF,” and “USF,” in Fig. 6.15(a) does not necessarily
correspond to those of “FS” because their speaker encoders sequentially learn perceptual
similarity among the 140 seen speakers using diﬀerently ordered similarity scores. The
query strategies signiﬁcantly aﬀect the AUC improvement by the active learning, and“USF” reasonably works among the three strategies regardless of the training algorithm
diﬀerences. Active learning using “Prop. (vec)” or “Prop. (graph)” successfully improves the AUC through the iterations better than “PS” in both the “Seen-Seen” and“Seen-Unseen” speaker-pair cases. Meanwhile, “Prop. (mat)” clearly shows its trend in
overﬁtting to the seen speaker pairs; i.e., it increased the AUC by the active learning
iterations in the “Seen-Seen” speaker-pair case but results in decreasing the AUC in the“Seen-Unseen” case.
Evaluation of Synthetic Speech Quality
Whether the active learning eﬃciently trained a speaker encoder that improved synthetic
speech quality while reducing the scoring and training costs was investigated. Similar
to Section 6.3.3, ﬁve-point scale MOS and DMOS tests were conducted to compare thequality of synthetic speech made by speaker representations of “FS,” “PS,” and “USF.”
The active learning using “USF” was performed with three diﬀerent iterations to increase
the percentage of additionally scored speaker pairs: 25% (30 iterations), 50% (60 iterations), and 75% (90 iterations). Tables 6.6 and 6.7 list the results of the MOS and DMOS
tests, respectively. “USF” achieves synthetic speech quality comparable to that of “FS”
with the fewer number of additional similarity score observations and active learning iterations. This result demonstrates that active learning of the perceptual-similarity-aware

118

Chapter 6 Perceptual-similarity-aware Deep Speaker Representation Learning(1) Prop. (vec)

AUC of similar speaker pair detection(b) Seen-Unseen(a) Seen-Seen

0.70

LSF

0.65
0.60

HSF

USF
(2) Prop. (mat)

0.90

0.95

0.85

0.90

0.80

0.55

PS

0.85

0.50

0.75

0.70

0.78

0.82

0.76

0.80

0.74

0.78

0.72

0.76

0.65
0.60
1

30

60

90

FS
(3) Prop. (graph)

1

30
60
90
Active learning iteration

1

30

60

90

Fig. 6.15. Curves of similar speaker-pair detection AUC against active learning iteration

Table 6.6. Results of MOS test on naturalness of synthetic speech with their 95% conﬁdence intervals (proposed algorithms using active learning). Second column
denotes percentages of number of additionally scored speaker pairs comparedwith “PS.” Bold scores are comparable to those of “FS” (p > 0.05)

PS
USF

FS

(0%)
(25%)
(50%)
(75%)
(100%)

Prop. (vec)2.91±0.14
2.99±0.12
3.11±0.13
3.18±0.13
3.19±0.13

Prop. (mat)2.98±0.13
2.97±0.13
3.02±0.13
3.04±0.13
3.02±0.12

Prop. (graph)
2.94±0.14
3.11±0.13
3.12±0.14
3.13±0.14
3.18±0.13

speaker representations eﬀectively reduces the number of scoring times while achieving
higher synthetic speech quality with fewer training iterations. Focusing on the results of“Prop. (mat),” there are no signiﬁcant diﬀerences among the ﬁve scores, and “FS” marks
the lowest scores among the three proposed algorithms. One of the causes may be theoverﬁtting trend observed in Fig. 6.15(b)(2).

6.3.5

Visualization of Speaker Space

Speaker spaces constructed by speaker representations of the four algorithms were visualized. The purpose of this visualization is to investigate how the four diﬀerent algorithms
aﬀect the speaker space. Figure 6.16 shows the PCA plots of the speaker representations.

Chapter 6 Perceptual-similarity-aware Deep Speaker Representation Learning

119

Table 6.7. Results of DMOS test on speaker similarity of synthetic speech with their
95% conﬁdence intervals (proposed algorithms using active learning). Second
column denotes percentages of number of additionally scored speaker pairs
compared with “PS.” Bold scores are comparable to “FS” (p > 0.05)

PS
USF

FS

(0%)
(25%)
(50%)
(75%)
(100%)

Prop. (vec)2.85±0.14
2.95±0.14
3.04±0.14
3.05±0.14
3.14±0.14

Prop. (mat)2.90±0.13
2.93±0.13
3.00±0.13
3.03±0.13
2.98±0.13

Prop. (graph)
2.86±0.13
3.03±0.13
3.02±0.13
3.06±0.13
3.08±0.14

From this ﬁgure, the speaker space of “d-vec.” is the widest among the four plots, and it
completely ignores the perceptual similarity among the speakers. On the other hand, thethree proposed algorithms construct the speaker spaces where the speakers are densely
distributed and the perceptual similarity is preserved. However, “Prop. (mat)” tends
to excessively separate disjoint speaker clusters, i.e., “F055–F058,” from the other, which
may be one possible cause of the poor generalization shown in Fig. 6.15 and speech quality
degradation.

6.4

SummaryThis chapter proposed novel algorithms for incorporating perceptual similarity among
speakers into deep speaker representation learning. The proposed speaker representation
learning algorithms utilize a perceptual speaker similarity matrix obtained by large-scale
perceptual scoring as the target for the speaker encoder training. The algorithms learn
speaker representations with three diﬀerent usages of the matrix: a set of vectors, theGram matrix, and a graph. This chapter further proposed an active learning algorithm toreduce costs of scoring and training. The active learning algorithm iterates the perceptual
similarity scoring and speaker encoder training. Queries in this algorithm are generated
from the sequentially-trained speaker encoder for prioritizing unscored speaker-pairs tobe scored next. The experimental evaluation results demonstrated that 1) the proposed
speaker representation learning algorithms learned speaker representations strongly correlated with perceptual similarity scores, 2) the representations improved synthetic speech
quality better than conventional speaker-classiﬁcation-derived d-vectors, and 3) the proposed active learning algorithm achieved higher synthetic speech quality while reducing
costs of scoring and training.

120

Chapter 6 Perceptual-similarity-aware Deep Speaker Representation Learning(a) d-vec.

(b) Prop. (vec)

F069
F070
F072
F116 F085
F074
F110
F130
F146B F031
F106
F032
F043
F021
F104
F050
F123
F060
F033
1.5
F084F068
F049
F124
F143B
0.2
F061
F019
FP02 F023
F145BF015
F107
FP03
F045
F121
F090
F134
F123
F003
F100
F121
F103
F144B
F113
F122
F126
F144A
F145A
F035
1.0
F063
F098
F020
F089
F083
FP05
F086
F112
F062
F105
F143A
F101
0.1 F019
F078
F093
F011
F067
F077
F102
F024
F150
F101
F104
F097
F036
F020
F077
F006
F037
F016
F146A
F013
F076
F082
F022
F001
F134
F004
F124
F092
F003
F149
F117
F063
F030
F143B
F015
F098
F128
F107
F096
F064
F075
F127
F025
0.5
F066
F118
F109
F010
F047
F008
F136
F129
F094
F117
FP04
F088
F074
F025FP01F100
F057
F005
0.0 FP05
F017
F006
F055
F093
F002
F087
F017
F095
F125
F149
F004
F026
F125
F009
F056
F114
F115
F095
F035
F012
F073
F136
F057
F127
F042
F076
F046
F132
F065
F058
F064
F126
F007
F106
F065
F099 F105
F081
F133
F135
F080
F027
F042
F038
F111
F039
F026
F138
F135
F059
F054
F059
0.0
F040
F080
F079
F048
F091
F041
F114
F018
F029
F014 F051
F120
F062
F099
F070
F067
F060
F068
F037
F008
F110
F011
F066
−0.1
F001
F096
F131
F138
F091
F010
F040
F018 F102
F069
F058
F092
F084
F119
F048
F021
F071
F072
F144A
F075
F028
FP02
F002
F039 F094
F079
F097
F122
F044
F120
F078
F143A
F130
F027
F087
F036
F137
F005
F071
F046
F088
F044
F012
F061
F150
F032
−0.5
F054
F133
F144B
F007
F118
F137F108
F047 −0.2 F028
F029
F090
F111
F103
F052
F116
F073
F014
F112
F024
F056
F082
F023
F132
F131
F086
F013
F089
F081
F045
F109
F009
F115
F052
F083
F108
F146A
F113
F041
F038
F053
FP03
F119
F145A
F049
F085
F146B
F053
F055
FP04 F043
−1.0
F033
F016
F030
F034 F145B
F129 F022
−0.3
F128
F031
FP01
F051
F050
−1.5
−1
0
1
2
−0.5
0.0
0.5
1.0
F034

2.0

1.50
1.25
1.00

F104
F106
F101
F100
F105
F023
F107
F020

(c) Prop. (mat)

1.00(d) Prop. (graph)
F069

0.75

0.75

F025
F084
F068
F085
F103
F083
F150
F036
F045
FP02
F033
F102
0.25 F008
F001
F021
F069
F043
F063
F116
F051
F143B
F050
FP01
F049
FP04
F145A
F024
F030
F072
F134
F146B
F035
F130
F110
F016
F015
F128
F018
F013
F086
F061
F032
F011
F012
F145B
0.00 F121
F124
F005
F082
F070
F052
F026
F098
F109
F113
F111
F129
F014
F097
F007
F114
F047
F048
F002
F115
F006
F090
F074
F122
F059
F040
F041
F081
F075
F071
F096
F088
F027
F089
F031
F094
F133
F108
F123
F078
F064
F117
F060
F067
F099
F135
F046
F009
F054
F112
F119
F126
F053
F062
F120
F127
F044
F138
F144A
F095
F022
F125
F003
F037
F039
F065
F019
F144B
F010
FP05
F137
F146A
F038
F042
0.25 F143A
F136
F029
F149
F087
F079
F132
F004
F080
F131
F028
F034
F118
F077
F066
F093
F073
FP03
F076
F017
F092
F091
0.50

0

0.3

1

2

F060F070
F134
F110F045
F043
F121
F123
F086
0.50 F062 F064
F083 F085
F072
F101
F015
F049
F004
F019
F106
F116
F003F011
F130
F104
F143A
F084
F095
F090
F057
F078
F021
F024
F077
F061
F098
0.25 F118
F097
F144AF010
F068
F067
F127F074
F029
F149
F065
F143B
F042
F020
F136
F050
F089
F030
F122
F128
F112
FP02F100 F023
F091
F037
F117
F035
F022
F039
F093
F032
F008
F002
F099
0.00 F076
F036
F063
F058
F005
F047
F129
F009
F087
F113
F105
F150
F040
F006
F001
F124
F034
F046
F012
F073
F088
F138
F017
F111
F082
F126
F041
F027
F075
FP05
F038
F056
F080
F079
F025
F007
F115
F144B
FP03
F096
F054
F114
F094
F120
F107
F125
F131
F092
0.25 F028
F133
F137
F146A
F059
F066
F053
F145A
F016
F132
F102
F026
F135
F044
F055
FP04
F033
F013
F108
F145B
F014
F018 F103
F031
F052
F119
0.50
F048
F081 FP01
F109 F146B
F071
0.75
F057
F058
F056
F055
F051
1.00
0.5
0.0
0.5
1.0

Fig. 6.16. PCA plots of speaker representations learned by (a) d-vec., (b) Prop. (vec),(c) Prop. (mat), and (d) Prop. (graph). Similar-speaker pairs are connectedby gray edges. Red points denote unseen speakers. PCA was applied to 153
speakers’ embedding learned by each method independently.

121

Chapter 7

Conclusion
7.1

Summary of This Thesis

This thesis addressed the issues of conventional DNN-based speech synthesis for synthesizing high-quality, versatile, and intuitively controllable speech. The following three issues
were solved: 1) low-quality synthetic speech due to over-smoothing, 2) limited speaker
diversity in synthetic speech, and 3) uninterpretable speaker representation.
Chapter 2 brieﬂy reviewed the basic framework of DNN-based speech synthesis. The
framework’s four crucial factors were described: 1) feature analysis, 2) acoustic modeling,3) speech parameter generation, and 4) speech waveform synthesis.
Chapter 3 presented the proposed GAN-based method to improve synthetic speech
quality. The basic framework of GANs was ﬁrst described. The introduction of GANs
to train DNNs for speech synthesis was then discussed. From the divergence minimization perspective, the eﬀects of the divergence in improving synthetic speech quality were
investigated. Experimental evaluations were conducted to demonstrate this method’s effectiveness in TTS and VC using vocoder parameters. The results indicated that 1) thismethod generated natural speech parameters regardless of its hyperparameter settings,
and 2) W-GAN minimizing the earth-mover’s distance worked the best among several
GANs in terms of improving the synthetic speech quality.
Chapter 4 extended the proposed GAN-based method described in Chapter 3 to DNNbased speech synthesis using STFT spectra. A simple but eﬀective approach was presented
to overcome the diﬃculty in modeling high-dimensional and complicated amplitude spectra based on GANs. Various frequency scales that are related to human speech perception
were introduced to this GAN-based method. The eﬀectiveness of this method was evaluated in TTS using STFT spectra. The results indicated that 1) GANs using low-frequency-

122

Chapter 7 Conclusion

resolution amplitude spectra improved speech quality and worked robustly against thesettings of the frequency resolution and hyperparameters, 2) comparing low-, original, and multi-frequency-resolution amplitude spectra, the use of low-frequency-resolution
ones worked best to improve synthetic speech quality, and 3) the use of the inverse melfrequency scale for obtaining low-frequency-resolution amplitude spectra further improved
synthetic speech quality.
Chapter 5 presented the proposed high-quality and versatile speech synthesis method
based on VAEs. The VAE-based speech parameter generation process was mathematically formulated to explicitly model the phonetic content and speaker individuality aslatent variables. Non-parallel and many-to-many VC that can reproduce and transformarbitrary speaker’s voice characteristics was established using VAEs. The trade-oﬀ between the number of seen speakers and dimensionality of continuous speaker representation with this method was investigated. The eﬀectiveness of this method was objectively
and subjectively evaluated in VC. The results indicated that 1) the introduction of the
DNN-based speech recognition model contributed to signiﬁcant quality improvement inconverted speech, 2) the use of continuous speaker representations achieved high-quality
VC even if the source and target speakers are unseen during the VAE training, and 3)
high-dimensional speaker representation did not necessarily improve the converted speech
quality but a large number of seen speakers consistently did improve this.
Chapter 6 presented the proposed perceptual-similarity-aware speaker representation
learning method for increasing the interpretability of speaker representations. The subjective scoring of perceptual speaker-pair similarity to obtain a perceptual speaker similarity
matrix was ﬁrst described. Deep speaker representation learning algorithms using a loss
function deﬁned by the similarity matrix were then presented. An active learning algorithm to reduce the costs of scoring and training was ﬁnally introduced. This method’s
eﬀectiveness was evaluated with the proposed VAE-based multi-speaker speech synthesis method described in Chapter 5. The results indicated that 1) the proposed speakerrepresentation learning method learned speaker representations strongly correlated withperceptual similarity scores, 2) the representations improved synthetic speech quality better than the conventional representations derived from a DNN-based speaker recognition
model, and 3) the active learning algorithm achieved higher synthetic speech quality while
reducing the costs of scoring and training.

Chapter 7 Conclusion

7.2

Future Directions

7.2.1

Improving Synthetic Speech Quality Further

123

The proposed methods successfully improved synthetic speech quality compared with theconventional methods. However, several subjective evaluation results (e.g., Fig. 3.21 andTable 5.1) revealed that the MOS values on naturalness of synthetic speech were around
3.0, which indicates that there is a still gap between natural speech (MOS > 4.0) andsynthetic speech. Therefore, I will aim to reduce the gap and to achieve communication
using the speech synthesis technology for synthesizing speech almost indistinguishable
from natural speech. One solution is to use recently developed neural vocoders (e.g.,
WaveNet vocoder [139]), which can synthesize high-ﬁdelity speech waveforms from speechparameters using DNNs.

7.2.2

Implementing with Real-time Speech Synthesis

Daily speech communication in most cases takes place in real time. However, the proposed methods do not consider such real-time situations, which limits their application
ranges. Therefore, I will introduce a DNN-based real-time speech synthesis method, suchas incremental TTS [140] and real-time VC [141, 142], into the proposed methods andevaluate their eﬀectiveness.

7.2.3

Extending to Multilingual Speech Synthesis

All experiments for this thesis were conducted using only Japanese speech corpora. To
enable speech communication beyond language barriers, I will evaluate the language dependency of the proposed methods by conducting experiments using other languages, suchas English and Chinese. I will also extend the proposed methods to multilingual speechsynthesis [143] that can synthesize voices in multiple languages using a single acoustic
model.

7.2.4

Modeling Broader Speech Perception

In Chapter 6, inter-speaker perceptual similarity was modeled by using multi-speaker corpus including reading-style speech to learn speaker representations that strongly correlatewith human’s perception and enable intuitively controllable speech synthesis. I will extend

124

Chapter 7 Conclusion

this idea to model human’s broader speech perception, such as various speaking styles,
emotions, and diﬀerent languages. This will enable developing more controllable speechsynthesis systems that can be easily tuned through human-computer interaction.

125

Publications and Research
Activities
Publications Related to This Thesis
Original Journal Papers
1. Yuki Saito,

Shinnosuke Takamichi,

and Hiroshi Saruwatari,“Perceptual-similarity-aware deep speaker representation learning for multi-speaker generativemodeling,” IEEE/ACM Transactions on Audio, Speech, and Language Processing,vol. xx, no. x, pp. xx-xx, 2021. (ACCEPTED, corresponds to Chapter 6)
2. Yuki Saito, Taiki Nakamura, Yusuke Ijima, Kyosuke Nishida, and Shinnosuke
Takamichi, “Non-parallel and many-to-many voice conversion using variational autoencoders integrating speech recognition and speaker veriﬁcation,” Acoustical Science and Technology, vol. 42, no. 1, pp. 1–11, Jan. 2021. (corresponds to Chapter
5 and Appendix Appendix C)
3. Yuki Saito, Kei Akuzawa, and Kentaro Tachibana, “Joint adversarial training ofspeech recognition and synthesis models for many-to-one voice conversion usingphonetic posteriorgrams,” IEICE Transactions on Information and Systems, vol.
103-D, no. 9, pp. 1978–1987, Sep. 2020. (corresponds to Chapter 3 and Appendix
Appendix B)
4. Yuki Saito, Shinnosuke Takamichi, and Hiroshi Saruwatari, “Vocoder-free textto-speech synthesis incorporating generative adversarial networks using low-/multifrequency STFT amplitude spectra,” Computer Speech and Language, vol. 58, pp.
347–363, Nov. 2019. (corresponds to Chapter 4)
5. Yuki Saito, Shinnosuke Takamichi, and Hiroshi Saruwatari, “Statistical parametric speech synthesis incorporating generative adversarial networks,” IEEE/ACM
Transactions on Audio, Speech, and Language Processing, vol. 26, no. 1, pp. 84–

126

Publications and Research Activities

96, Jan. 2018. (corresponds to Chapter 3)
6. Yuki Saito, Shinnosuke Takamichi, and Hiroshi Saruwatari, “Voice conversion using input-to-output highway networks,” IEICE Transactions on Information and
Systems, vol. E100-D, no. 8, pp. 1925–1928, Aug. 2017. (corresponds to Chapter
3 and Appendix Appendix A)
International Conferences (Peer-Reviewed)
1. Yuki Saito, Shinnosuke Takamichi, and Hiroshi Saruwatari, “DNN-based speakerembedding using subjective inter-speaker similarity for multi-speaker modeling inspeech synthesis,” Proc. SSW, pp. 51–56, Vienna, Austria, Sep. 2019. (corresponds
to Chapter 6)
2. Yuki Saito, Yusuke Ijima, Kyosuke Nishida, and Shinnosuke Takamichi, “Nonparallel voice conversion using variational autoencoders conditioned by phonetic
posteriorgrams and d-vectors,” Proc. ICASSP, pp. 5274–5278, Alberta, Canada,
Apr. 2018. (corresponds to Chapter 5)
3. Yuki Saito, Shinnosuke Takamichi, and Hiroshi Saruwatari, “Text-to-speech synthesis using STFT spectra based on low-/multi-resolution generative adversarial
networks,” Proc. ICASSP, pp. 5299–5303, Alberta, Canada, Apr. 2018. (corresponds to Chapter 4)
4. Yuki Saito, Shinnosuke Takamichi, and Hiroshi Saruwatari, “Training algorithm
to deceive anti-spooﬁng veriﬁcation for DNN-based speech synthesis,” Proc.
ICASSP, pp.

4900–4904, New Orleans, U.S.A., Mar.

2017.

(corresponds to

Chapter 3)
Technical Reports
1. Yuki Saito, Shinnosuke Takamichi, and Hiroshi Saruwatari, “Active learning for
DNN-based speaker embedding considering subjective inter-speaker similarity,”
IPSJ SIG Technical Report, 2021-SLP-xxx, no. x, pp. xx–xx, Mar. 2021. (toappear in Japanese, corresponds to Chapter 6)
2. Yuki Saito, Yusuke Ijima, Kyosuke Nishida, and Shinnosuke Takamichi, “Nonparallel and many-to-many voice conversion using variational autoencoders usingphonetic posteriorgrams and d-vectors,” Proc. IEICE Technical Report, SP2017-88,
Vol. 117, No. 517, pp. 21–26, Mar. 2018. (in Japanese, corresponds to Chapter 5)
3. Yuki Saito, Shinnosuke Takamichi, and Hiroshi Saruwatari, “Training algorithm
to deceive anti-spooﬁng veriﬁcation for DNN-based text-to-speech synthesis,” IPSJ

Publications and Research Activities

127

SIG Technical Report, 2017-SLP-115, no. 1, pp. 1–6, Feb. 2017. (in Japanese,
corresponds to Chapter 3)
4. Yuki Saito, Shinnosuke Takamichi, and Hiroshi Saruwatari, “Evaluation of DNNbased voice conversion deceiving anti-spooﬁng veriﬁcation,” IEICE Technical Report, SP2016-69, vol. 116, no. 414, pp. 29–34, Jan. 2017. (in Japanese, corresponds
to Chapter 3 and Appendix Appendix A)
Domestic Conferences
1. Yuki Saito, Shinnosuke Takamichi, and Hiroshi Saruwatari, “DNN-based speakerembedding using graph embedding of subjective inter-speaker similarity,” Proc.
ASJ, Autumn meeting, 1-2-4, pp. 697–698, Sep. 2020. (in Japanese, corresponds
to Chapter 6)
2. Yuki Saito, Kei Akuzawa, and Kentaro Tachibana, ”Joint adversarial training algorithm of speech recognition and synthesis models for many-to-one voice conversion
using phonetic posteriorgrams,” Proc. ASJ, Autumn meeting, 2-4-2, pp. 963–966,
Sep. 2019. (in Japanese, corresponds to Chapter 3 and Appendix Appendix B)
3. Yuki Saito, Shinnosuke Takamichi, and Hiroshi Saruwatari, “Evaluation of DNNbased multi-speaker speech synthesis using DNN-based speaker embedding considering subjective inter-speaker similarity,” Proc. ASJ, Autumn meeting, 1-P-18, pp.
999–1002, Sep. 2019. (in Japanese, corresponds to Chapter 6)
4. Yuki Saito, Shinnosuke Takamichi, and Hiroshi Saruwatari, “DNN-based speakerembedding considering subjective inter-speaker similarity towards DNN-based
speech synthesis,” Proc. ASJ, Spring meeting, 3-10-7, pp. 1067–1070, Mar. 2019.
(in Japanese, corresponds to Chapter 6)
5. Taiki Nakamura, Yuki Saito, Kyosuke Nishida, Yusuke Ijima, and Shinnosuke
Takamichi, “Evaluation of VAE-based non-parallel and many-to-many voice conversion conditioned by phonetic posteriorgrams and d-vectors in terms of training
data and dimensionality of d-vectors,” Proc. ASJ, Spring meeting, 2-P-30, pp.
1149–1150, Mar. 2019. (in Japanese, corresponds to Chapter 5)
6. Yuki Saito, Shinnosuke Takamichi, and Hiroshi Saruwatari, “Adversarial DNNbased speech synthesis using multi-frequency-resolution STFT spectra,” Proc. ASJ,
Spring meeting, 3-8-14, pp. 259–262, Mar. 2018. (in Japanese, corresponds to
Chapter 4)
7. Yuki Saito, Shinnosuke Takamichi, and Hiroshi Saruwatari, “Experimental investigation of divergences in adversarial DNN-based speech synthesis,” Proc. ASJ,

128

Publications and Research Activities

Autumn meeting, 1-8-7, pp. 189–192, Sep. 2017. (in Japanese, corresponds to
Chapter 3)
8. Yuki Saito, Shinnosuke Takamichi, and Hiroshi Saruwatari, “Adversarial DNNbased voice conversion based on spectral diﬀerentials using highway networks,”
Proc. ASJ, Spring meeting, 1-6-14, pp. 235–236, Mar. 2017. (in Japanese, corresponds to Chapter 3 and Appendix Appendix A)
9. Yuki Saito, Shinnosuke Takamichi, and Hiroshi Saruwatari, “F0 contour and duration generation for adversarial DNN-based speech synthesis,” Proc. ASJ, Spring
meeting, 2-6-6, pp. 257–258, Mar. 2017. (in Japanese, corresponds to Chapter 3)
10. Yuki Saito, Shinnosuke Takamichi, and Hiroshi Saruwatari, “Training algorithm
considering anti-spooﬁng veriﬁcation for DNN-based speech synthesis,” Proc. ASJ,
Autumn meeting, 3-5-1, pp. 149–150, Sep. 2016. (in Japanese, corresponds to
Chapter 3)Awards
1. 2020 IEEE Signal Processing Society Young Author Best Paper Award, Jun. 2021.
2. The 49th Awaya Prize Young Researcher Award of ASJ, Mar. 2021.
3. Outstanding Paper Award for Young C&C Researchers, Jan. 2019.
4. The 12th IEEE Signal Processing Society Japan Student Journal Paper Award,
Nov. 2018.
5. 2017 IEICE ISS Young Researcher’s Award in Speech Field, Aug. 2018.
6. Partial Exemption from Repayment of Scholarship Loan for Students with Outstanding Results, Japan Student Services Organization (JASSO), May 2018.
7. The 33rd TELECOM System Technology Award for Students from TAF, Mar.
2018.
8. The 1st IEEE Signal Processing Society Tokyo Joint Chapter Student Award, Nov.
2017.
9. Spoken Language Processing Student Grant Award of ICASSP, Mar. 2017.
10. The 14th Best Student Presentation Award of Acoustical Society of Japan, Mar.
2017.
11. 2017 IEICE ISS Student Poster Award, Jan. 2017.
Competitive Funds
1. Grant-in-Aid for JSPS Fellows, the Japan Society of the Promotion of Science(JSPS), May 2018.

Publications and Research Activities

129

2. Grants for Researchers Attending International Conferences from NEC C&C, Apr.
2018.
Misc.
1. I was invited to Google Speech Technology Summit 2018, Google London, United
Kingdom. I talked about two research papers that were accepted to ICASSP 2018at the poster session.
2. A ﬁgure taken from our paper appeared on the cover of the IEEE/ACM Transactions on Audio, Speech, and Language Processing (January/February issue).

Other Publications
Original Journal Papers
1. Takaaki Saeki, Yuki Saito, Shinnosuke Takamichi, and Hiroshi Saruwatari, “Realtime full-band voice conversion with sub-band modeling and data-driven phase estimation of spectral diﬀerentials,” IEICE Transactions on Information and Systems,
2021. (under review, conditionally accepted)
2. Shinnosuke Takamichi, Ryosuke Sonobe, Kentaro Mitsui, Yuki Saito, Tomoki Koriyama, Naoko Tanji and Hiroshi Saruwatari, “JSUT and JVS: free Japanese voice
corpora for accelerating speech synthesis research,” Acoustical Science and Technology, vol. 41, No. 5, pp. 761–768, Sep. 2020.
3. Hiroki Tamaru, Yuki Saito, Shinnosuke Takamichi, Tomoki Koriyama, andHiroshi Saruwatari, “Generative moment matching network-based neural doubletracking for synthesized and natural singing voices,” IEICE Transactions on
Information and Systems, vol. E103-D, no. 3, pp. 639–647, Mar. 2020.
4. Shinnosuke Takamichi, Yuki Saito, Norihiro Takamune, Daichi Kitamura, andHiroshi Saruwatari, “Phase reconstruction from amplitude spectrograms basedon directional-statistics deep neural networks,” Signal Processing, vol. 169, pp.
107368, Apr. 2020.
International Conferences (Peer-Reviewed)
1. Yota Ueda, Kazuki Fujii, Yuki Saito, Shinnosuke Takamichi, Yukino Baba, andHiroshi Saruwatari, “HumanACGAN: conditional generative adversarial networkwith human-based auxiliary classiﬁer and its evaluation in phoneme perception,”

130

Publications and Research Activities

Proc. ICASSP, pp. xxxx–xxxx, Toronto, Canada, Jun. 2021. (ACCEPTED)
2. Yuki Yamashita, Tomoki Koriyama, Yuki Saito, Shinnosuke Takamichi, Yusuke
Ijima, Ryo Masumura, and Hiroshi Saruwatari, “Investigating eﬀective additional
contextual factors in DNN-based spontaneous speech synthesis,” Proc. INTERSPEECH, pp. 3201–3205, Shanghai, China, Oct. 2020.
3. Shunsuke Goto, Kotaro Ohnishi Yuki Saito, Kentaro Tachibana, and KoichiroMori, “Face2Speech: towards multi-speaker text-to-speech synthesis using an embedding vector predicted from a face image,” Proc. INTERSPEECH, pp. 2947–
2951, Shanghai, China, Oct. 2020.
4. Detai Xin, Yuki Saito, Shinnosuke Takamichi, Tomoki Koriyama, and Hiroshi
Saruwatari, “Cross-lingual text-to-speech synthesis via domain adaptation and perceptual similarity regression in speaker space,” Proc. INTERSPEECH, pp. 1321–
1325, Shanghai, China, Oct. 2020.
5. Takaaki Saeki, Yuki Saito, Shinnosuke Takamichi, and Hiroshi Saruwatari, “Realtime, full-band, online DNN-based voice conversion system using a single CPU,”
Proc. INTERSPEECH, pp. 1021–1022, Shanghai, China, Oct. 2020.
6. Yuki Saito, Shinnosuke Takamichi, and Hiroshi Saruwatari, “SMASH corpus: aspontaneous speech corpus recording third-person audio commentaries on gameplay,” Proc. LREC, pp. 6573–6579, Marseille, France, May 2020.
7. Yuki Yamashita, Tomoki Koriyama, Yuki Saito, Shinnosuke Takamichi, Yusuke
Ijima, Ryo Masumura, and Hiroshi Saruwatari, “DNN-based speech synthesis using abundant tags of spontaneous speech corpus,” Proc. LREC, pp. 6440–6445,
Marseille, France, May 2020.
8. Kazuki Fujii, Yuki Saito, Shinnosuke Takamichi, Yukino Baba, and Hiroshi
Saruwatari, “HumanGAN: generative adversarial network with human-based
discriminator and its evaluation in speech perception modeling,” Proc. ICASSP,
pp. 6239–6243, Barcelona, Spain, May 2020.
9. Takaaki Saeki, Yuki Saito, Shinnosuke Takamichi, and Hiroshi Saruwatari, “Liftertraining and sub-band modeling for computationally eﬃcient and high-quality voice
conversion using spectral diﬀerentials,” Proc. ICASSP, pp. 7784–7788, Barcelona,
Spain, May 2020.
10. Taiki Nakamura, Yuki Saito, Shinnosuke Takamichi, Yusuke Ijima, and Hiroshi
Saruwatari, “V2S attack: building DNN-based voice conversion from automatic
speaker veriﬁcation,” Proc. SSW, pp. 161–165, Vienna, Austria, Sep. 2019.
11. Hiroki Tamaru, Yuki Saito, Shinnosuke Takamichi, Tomoki Koriyama, and Hi-

Publications and Research Activities

131

roshi Saruwatari, “Generative moment matching network-based random modulation post-ﬁlter for DNN-based singing voice synthesis and neural double-tracking,”
Proc. ICASSP, pp. 7070–7074, Brighton, U.K., May 2019.
12. Masakazu Une, Yuki Saito, Shinnosuke Takamichi, Daichi Kitamura, Ryoichi
Miyazaki, and Hiroshi Saruwatari, “Generative approach using the noise generation models for DNN-based speech synthesis trained from noisy speech,” Proc.
APSIPA-ASC, pp. 99–103, Hawaii, U.S.A., Nov. 2018.
13. Shinnosuke Takamichi, Yuki Saito, Norihiro Takamune, Daichi Kitamura, andHiroshi Saruwatari, “Phase reconstruction from amplitude spectrograms based on
directional-statistics deep neural networks,” Proc. IWAENC, pp. 286–290, Tokyo,
Japan, Sep. 2018.
14. Hiroyuki Miyoshi, Yuki Saito, Shinnosuke Takamichi, and Hiroshi Saruwatari,“Voice conversion using sequence-to-sequence learning of context posterior probabilities,” Proc. INTERSPEECH, pp. 1268–1272, Stockholm, Sweden, Aug. 2017.
Technical Reports
1. Masaki Kurata, Shinnosuke Takamichi, Takaaki Saeki, Riku Arakawa, Yuki Saito,
Keita Higuchi, and Hiroshi Saruwatari, “A method for obtaining speaking characteristics based on real-time DNN-based voice conversion feedback,” IPSJ SIG
Technical Report, 2021-SLP-xxx, No. x, pp. xx–xx, Mar. 2021. (to appear inJapanese)
2. Kazuki Fujii, Yuki Saito, Shinnosuke Takamichi, Yukino Baba, and Hiroshi
Saruwatari, “HumanGAN: generative adversarial networks based on humanperception evaluation and its application in speech naturalness modeling,” IEICE
Technical Report, SP2020-06, Vol. 120, No. 57, pp. 15–20, Jun. 2020. (inJapanese)
3. Satoshi Naitou, Yuki Saito, Shinnosuke Takamichi, Yasuyuki Saito, and Hiroshi
Saruwatari, “Automatic estimation of breath position for singing VOCALOID
song,” IPSJ SIG Technical Report, 2020-MUS-127, No. 33, pp. 1–6, Jun. 2020.
(in Japanese)
4. Yuki Yamashita, Tomoki Koriyama, Yuki Saito, Shinnosuke Takamichi, Yusuke
Ijima, Ryo Masumura, and Hiroshi Saruwatari, “The eﬀectiveness of additional
context in DNN-based spontaneous speech synthesis,” IEICE Technical Report,
SP2019-61, Vol. 119, No. 441, pp. 65–70, Mar. 2020. (in Japanese)
5. Takaaki Saeki, Yuki Saito, Shinnosuke Takamichi, and Hiroshi Saruwatari, “Lifter

132

Publications and Research Activities

training and sub-band modeling for DNN-based voice conversion using spectral
diﬀerentials,” IPSJ SIG Technical Report, 2020-SLP-131, No. 2, pp. 1–6, Feb.
2020. (in Japanese)
6. Kazuki Fujii, Yuki Saito, Shinnosuke Takamichi, Yukino Baba, and Hiroshi
Saruwatari, “HumanGAN: generative adversarial networks trained with humanperception evaluation,” IBIS Workshop 2019, Nov. 2019. (in Japanese)
7. Taiki Nakamura, Yuki Saito, Shinnosuke Takamichi, Yusuke Ijima, and Hiroshi
Saruwatari, “Speaker V2S attack: statistical voice conversion built from speaker
veriﬁcation and its evaluation on speaker spooﬁng attack,” CSS 2019, pp. 697–703,
Oct. 2019. (in Japanese)
8. Shinnosuke Takamichi, Kentaro Mitsui, Yuki Saito, Tomoki Koriyama, Naoko
Tanji, and Hiroshi Saruwatari, “JVS corpus: online available Japanese versatile
speech corpus,” IPSJ SIG Technical Report, 2019-SLP-129, No. 1, pp. 1–6, Oct.
2019. (in Japanese)
9. Hiroki Tamaru, Yuki Saito, Shinnosuke Takamichi, Tomoki Koriyama, and Hiroshi Saruwatari, “Generative moment matching network-based random modulation post-ﬁlter for singing voices synthesized using DNNs and its application toneural double-tracking,” IPSJ SIG Technical Report, 2018-SLP-125, No. 1, pp.
1–6, Dec. 2018. (in Japanese)
10. Satoshi Mizoguchi, Yuki Saito, Shinnosuke Takamichi, and Hiroshi Saruwatari,“Evaluation of DNN-based low-musical-noise speech enhancement using kurtosis
matching,” IEICE Technical Report, EA2018-66, Vol. 118, No. 312, pp. 19–24,,
Nov. 2018. (in Japanese)
11. Shinnosuke Takamichi, Yuki Saito, Norihiro Takamune, Daichi Kitamura, andHiroshi Saruwatari, “Phase reconstruction from amplitude spectra based on vonMises distribution DNN,” IPSJ SIG Technical Report, 2018-SLP-122, No. 1, pp.
1–6, Jun. 2018. (in Japanese)
12. Masakazu Une, Yuki Saito, Shinnosuke Takamichi, Daichi Kitamura, Ryoichi
Miyazaki, and Hiroshi Saruwatari, “Generative adversarial training of the noise
generation model for speech synthesis using speech in noise,” IPSJ SIG Technical
Report, 2017-SLP-118, no. 1, pp. 1-6, Oct. 2017. (in Japanese)
13. Hiroyuki Miyoshi, Yuki Saito, Shinnosuke Takamichi, and Hiroshi Saruwatari,“Voice conversion using sequence-to-sequence learning of context posterior probabilities and evaluation of the dual learning,” IEICE Technical Report, SP2017-16,
vol. 117, No. 160, pp. 9–14, Jul. 2017. (in Japanese)Publications and Research Activities

133

Domestic Conferences
1. Yota Ueda, Kazuki Fujii, Yuki Saito, Shinnosuke Takamichi, Yukino Baba, andHiroshi Saruwatari, “HumanACGAN: conditional generative adversarial network
using human-based auxiliary classiﬁer and its evaluation in representing conditional
distribution of phoneme perception,” Proc. ASJ, Spring meeting, 1-2-14, pp. xxx–
xxx, Mar. 2021. (to appear in Japanese)
2. Takaaki Saeki, Yuki Saito, Shinnosuke Takamichi, and Hiroshi Saruwatari, “Implementation and evaluation of real-time full-band DNN-based Voice Conversion
based on sub-band ﬁltering,” Proc. ASJ, Autumn meeting, 1-2-11, pp. 715–718,
Sep. 2020. (in Japanese)
3. Kazuki Fujii, Yuki Saito, Shinnosuke Takamichi, Yukino Baba, and Hiroshi
Saruwatari, “HumanGAN: generative adversarial network with human-based
discriminator and its evaluation in naturalness modeling of speech,” Proc. ASJ,
Spring meeting, 3-P-40, pp. 1181–1184, Mar. 2020. (in Japanese)
4. Shinnosuke Takamichi, Kai Onuma, Taku Kaneda, Takashi Kaneda, Yuki Saito,
Tomoki Koriyama, and Hiroshi Saruwatari, “Crowdsourcing-based parameter optimization for frequency warping-based speaker anonymization,” Proc. ASJ, Spring
meeting, 3-P-31, pp. 1159–1162, Mar. 2020. (in Japanese)
5. Shunsuke Goto, Kotaro Ohnishi, Yuki Saito, Kentaro Tachibana, and KoichiroMori, “Multi-speaker text-to-speech synthesis using an embedding vector based ona face image,” Proc. ASJ, Spring meeting, 2-Q-49, pp. 1141–1144, Mar. 2020. (inJapanese)
6. Takaaki Saeki, Yuki Saito, Shinnosuke Takamichi, and Hiroshi Saruwatari, “Subband lifter-training method for full-band voice conversion using spectral diﬀerentials,” Proc. ASJ, Spring meeting, 2-2-5, pp. 1085–1088, Mar. 2020. (in Japanese)
7. Shinnosuke Takamichi, Yuki Saito, Tomohiko Nakamura, Tomoki Koriyama, andHiroshi Saruwatari, “manga2voice: speech analysis towards audio synthesis fromcomic image,” Proc. ASJ, Spring meeting, 1-2-15, pp. 1065–1068, Mar. 2020. (inJapanese)
8. Yuki Saito, Shinnosuke Takamichi, and Hiroshi Saruwatari, “SMASH corpus:
spontaneous speech corpus of audio commentaries on gameplay,” Proc.

ASJ,

Spring meeting, 1-2-14, pp. 1061–1064, Mar. 2020. (in Japanese)
9. Satoshi Naitou, Yuki Saito, Shinnosuke Takamichi, Yasuyuki Saito, and Hiroshi
Saruwatari, “Estimation of breath position for VOCALOID song sung by user,”
Proc. ASJ, Spring meeting, 1-2-12, pp. 1057–1058, Mar. 2020. (in Japanese)

134

Publications and Research Activities

10. Takaaki Saeki, Yuki Saito, Shinnosuke Takamichi, and Hiroshi Saruwatari, “Filter
estimation for computational complexity reduction of DNN-based voice conversion
using spectral diﬀerentials,” Proc. ASJ, Autumn meeting, 2-4-1, pp. 961–962, Sep.
2019. (in Japanese)
11. Hiroki Tamaru, Yuki Saito, Shinnosuke Takamichi, Tomoki Koriyama, and Hiroshi Saruwatari, “Neural double-tracking based on generative moment matching
network for users’ singing,” Proc. ASJ, Autumn meeting, 1-4-2, pp. 935–938, Sep.
2019. (in Japanese)
12. Hiroki Tamaru, Yuki Saito, Shinnosuke Takamichi, Tomoki Koriyama, and Hiroshi Saruwatari, “Generative moment matching network-based random modulation post-ﬁlter for singing voices and its application to double-tracking,” Proc.
ASJ, Spring meeting, 2-10-5, pp. 1035–1038, Mar. 2019. (in Japanese)
13. Satoshi Mizoguchi, Yuki Saito, Shinnosuke Takamichi, and Hiroshi Saruwatari,“Low-musical-noise DNN-based speech enhancement applied to noise with various
kurtosis,” Proc. ASJ, Spring meeting, 1-6-6, pp. 185–188, Mar. 2019. (in Japanese)
14. Shinnosuke Takamichi, Yuki Saito, Norihiro Takamune, Daichi Kitamura, andHiroshi Saruwatari, “Phase reconstruction from amplitude spectrograms based on
directional-statistics DNNs,” Proc. ASJ, Autumn meeting, 2-4-2, pp. 1127–1130,
Sep. 2018. (in Japanese)
15. Satoshi Mizoguchi, Yuki Saito, Shinnosuke Takamichi, and Hiroshi Saruwatari,“Low-musical-noise speech enhancement based on DNNs and kurtosis matching,”
Proc. ASJ, Autumn meeting, 2-1-7, pp. 177–180, Sep. 2018. (in Japanese)
16. Masakazu Une, Yuki Saito, Shinnosuke Takamichi, Daichi Kitamura, Ryoichi
Miyazaki, and Hiroshi Saruwatari, “Generative approach using the noise generation
models for DNN-based speech synthesis trained from noisy speech,” Proc. ASJ,
Spring meeting, 3-8-8, pp. 243–244, Mar. 2018. (in Japanese)
17. Shinnosuke Takamichi, Tomoki Koriyama, Yuki Saito, and Hiroshi Saruwatari,“Evaluation of inter-utterance variation in speech synthesis based on momentmatching networks,” Proc. ASJ, Autumn meeting, 1-8-9, pp. 195–196, Sep. 2017.
(in Japanese)
18. Hiroyuki Miyoshi, Yuki Saito, Shinnosuke Takamichi, and Hiroshi Saruwatari,“Voice conversion using sequence-to-sequence learning of context posterior probabilities,” Proc. ASJ, Spring meeting, 1-6-15, pp. 237–238, Mar. 2017. (in Japanese)
Co-authors’ Awards

Publications and Research Activities

135

1. IPSJ SIG-MUS/SLP Student Poster Award, Jun. 2020. (Awardee: Kazuki Fujii)
2. FujiSankei Business i Awards, Jun. 2020. (Awardee: Kazuki Fujii)
3. IPSJ Yamashita SIG Research Award, Mar.

2020.

(Awardee:Shinnosuke

Takamichi)
4. The 3rd IEEE Signal Processing Society Tokyo Joint Chapter Student Award, Dec.
2019. (Awardee: Hiroki Tamaru)
5. The 18th Best Student Presentation Award of ASJ, Mar. 2019. (Awardee: Satoshi
Mizoguchi)
6. IPSJ SIG-MUS/SLP Presentation Award, Jun.

2018.

(Awardee: Shinnosuke

Takamichi)
Misc.
1. Four patent applications
2. One copyrighted work (Japanese Versatile Speech: JVS Corpus)

136

Bibliography[1] Y. Sagisaka, “Speech synthesis by rule using an optimal selection of non-uniform
synthesis units,” in Proc. ICASSP, New York, U.S.A., Apr. 1988, pp. 679–682.
[2] Y. Stylianou, O. Cappé, and E. Moulines, “Continuous probabilistic transform forvoice conversion,” IEEE Transactions on Speech and Audio Processing, vol. 6, no.
2, pp. 131–142, Mar. 1988.
[3] A. W. Black, “Speech synthesis for educational technology,” in Proc. SLaTE Workshop, Farmington, U.S.A., Oct. 2007, pp. 104–107.
[4] Y. Oshima, S. Takamichi, T. Toda, G. Neubig, S. Sakti, and Satoshi Nakamura,“Non-native speech synthesis preserving speaker individuality based on partial correction of prosodic and phonetic characteristics,” in Proc. INTERSPEECH, Dresden, Germany, Sep. 2015, pp. 299–303.
[5] J. A. Liao, N. Jincho, and H. Kikuchi, “Interactive virtual reality speech simulation
system using autonomous audience with natural non-verbal behavior,” International
Journal of Machine Learning and Computing, vol. 8, no. 4, pp. 404–407, Aug. 2018.
[6] M. Macon, L. A. Jensen-Link, J. Oliverio, M. A Clements, and E. George, “Asinging voice synthesis system based on sinusoidal modeling,” in Proc. ICASSP,
Munich, Germany, Apr. 1997, pp. 435–438.
[7] H. Doi, T. Toda, T. Nakano, M. Goto, and S. Nakamura, “Singing voice conversion
method based on many-to-many eigenvoice conversion and training data generation
using a singing-to-singing synthesis system,” in Proc. APSIPA ASC. Hollywood,
U.S.A., Nov. 2012.
[8] K. Kobayashi, T. Toda, T. Nakano, M. Goto, G. Neubig, S. Sakti, and S. Nakamura,“Regression approaches to perceptual age control in singing voice conversion,” inProc. ICASSP, Florence, Italy, May 2014, pp. 7954–7958.
[9] N. Hattori, T. Toda, H. Kawai, H. Saruwatari, and K. Shikano, “Speaker-adaptive
speech synthesis based on Eigenvoice conversion and language-dependent prosodic
conversion in speech-to-speech translation,” in Proc. INTERSPEECH, Florence,
Italy, Aug. 2011, pp. 2769–2772.

Bibliography137[10] S. Sitaram, G. Anumanchipalli, J. Chiu, A. Parlikar, and A. W. Black, “Text tospeech in new languages without a standardized orthography,” in Proc. SSW, pp.
95–100. Barcelona, Spain, Aug. 2013.
[11] A. B. Kain, J.-P. Hosom, X. Niu, J. P. H. van Santen, M. Fried-Oken, and J. Staehely,“Improving the intelligibility of dysarthric speech,” Speech Communication, vol. 49,
no. 9, pp. 743–756, Sep. 2007.
[12] K. Nakamura, T. Toda, H. Saruwatari, and K. Shikano, “Speaking-aid systems using
GMM-based voice conversion for electrolaryngeal speech,” Speech Communication,
vol. 54, no. 1, pp. 134–146, Jan. 2012.
[13] D .H. Klatt, “Software for a cascade/parallel formant synthesizer,” Journal of the
Acoustical Society of America, vol. 67, no. 3, pp. 971–995, 1980.
[14] A. J. Hunt and A.W. Black, “Unit selection in a concatenative speech synthesis
system using a large speech database,” in Proc. ICASSP, Atlanta, U.S.A., May
1996, pp. 373–376.
[15] H. Zen, K. Tokuda, and A. Black, “Statistical parametric speech synthesis,” SpeechCommunication, vol. 51, no. 11, pp. 1039–1064, Nov. 2009.
[16] M. Abe, S. Nakamura, K. Shikano, and H. Kuwabara, “Voice conversion throughvector quantization,” in Proc. ICASSP, New York, U.S.A., Apr. 1988, pp. 655–658.
[17] J. Yamagishi and T. Kobayashi,“Average-voice-based speech synthesis using

HSMM-based speaker adaptation and adaptive training,” IEICE Transactions on
Information and Systems, vol. E90-D, no. 2, pp. 533–543, Feb. 2007.
[18] Y. Ohtani, T. Toda, H. Saruwatari, and K. Shikano, “Many-to-many eigenvoice
conversion with reference voice,” in Proc. INTERSPEECH, Brighton, U.K., Sep.
2009, pp. 1623–1626.
[19] F. Fang, X. Wang, J. Yamagishi, I. Echizen, M. Todisco, N. Evans, and J.-F. Bonastre, “Speaker anonymization using x-vector and neural waveform models,” in Proc.
SSW, Vienna, Austria, Sep. 2019, pp. 155–160.
[20] S. Ueno, M. Mimura, S. Sakai, and T. Kawahara, “Multi-speaker sequence-tosequence speech synthesis for data augmentation in acoustic-to-word speech recognition,” in Proc. ICASSP, Brighton, U.K., May 2019, pp. 6161–6165.
[21] H. Zen, A. Senior, and M. Schuster, “Statistical parametric speech synthesis using
deep neural networks,” in Proc. ICASSP, Vancouver, Canada, May 2013, pp. 7962–
7966.
[22] Z.-H. Ling, S. Y. Kang, H. Zen, A. Senior, M. Schuster, X. J. Qian, H. Meng, and
L. Deng, “Deep learning for acoustic modeling in parametric speech generation: A138

Bibliographysystematic review of existing techniques and future trends,” IEEE Signal Processing
Magazine, vol. 32, no. 3, pp. 35–52, May 2015.
[23] G. E. Hinton, L. Deng, D. Yu, G. Dahl, A. r. Mohamed, N. Jaitly, A. Senior,
V. Vanhoucke, P. Nguyen, T. Sainath, and B. Kingsbury, “Deep neural networks foracoustic modeling in speech recognition: The shared views of four research groups,”
IEEE Signal Processing Magazine, vol. 29, no. 6, pp. 82–97, Oct. 2012.
[24] K. Tokuda, Y Nankaku, T. Toda, H. Zen, J. Yamagishi, and K. Oura, “Speechsynthesis based on hidden Markov models,” Proceedings of the IEEE, vol. 101, no.
5, pp. 1234–1252, Apr. 2013.
[25] T. Toda, A. W. Black, and K. Tokuda, “Voice conversion based on maximum likelihood estimation of spectral parameter trajectory,” IEEE Transactions on Audio,
Speech, and Language Processing, vol. 15, no. 8, pp. 2222–2235, Nov. 2007.
[26] O. Abdel-Hamid, A. r. Mohamed, H. Jiang, L. Deng, G. Penn, and D. Yu, “Convolutional neural networks for speech recognition,” IEEE/ACM Transactions on
Audio, Speech, and Language Processing, vol. 22, no. 10, pp. 1533–1545, Oct. 2014.
[27] E. Variani, X. Lei, E. McDermott, I. L. Moreno, and J. Gonzalez-Dominguez, “Deepneural networks for small footprint text-dependent speaker veriﬁcation,” in Proc.
ICASSP, Florence, Italy, May 2014, pp. 4080–4084.
[28] D. Snyder, D. Garcia-Romero, G. Sell, D. Povey, and S. Khudanpur, “X-vectors: Robust DNN embeddings for speaker recognition,” in Proc. ICASSP, Alberta, Canada,
Apr. 2018, pp. 5329–5333.
[29] Z. Wu, P. L. D. Leon, C. Demiroglu, A. Khodabakhsh, S. King, Z. Ling, D. Saito,
B. Stewart, T. Toda, M. Wester, and J. Yamagishi,“Anti-spooﬁng for text-

independent speaker veriﬁcation: An initial database, comparison of countermeasures, and human performance,” IEEE/ACM Transactions on Audio, Speech, andLanguage Processing, vol. 24, no. 4, pp. 768–783, Apr. 2016.
[30] D. Rumelhart, G. E. Hinton, and R. Williams, “Learning representations by backpropagating errors,” Nature, vol. 323, no. 1, pp. 533–536, Oct. 1986.
[31] T. Toda, L. H. Chen, D. Saito, F. Villavicencio, M. Wester, Z. Wu, and J. Yamagishi, “The Voice Conversion Challenge 2016,” in Proc. INTERSPEECH, California,
U.S.A., Sep. 2016, pp. 1632–1636.
[32] Y. Ohtani, M. Tamura, M. Morita, T. Kagoshima, and M. Akamine, “Histogrambased spectral equalization for HMM-based speech synthesis using mel-LSP,” inProc. INTERSPEECH, Portland, U.S.A., Sep. 2012.
[33] S. Takamichi, T. Toda, A. W. Black, G. Neubig, S. Sakti, and S. Nakamura, “Postﬁl-Bibliography139

ters to modify the modulation spectrum for statistical parametric speech synthesis,”
IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 24, no.
4, pp. 755–767, Apr. 2016.
[34] S. Takamichi, T. Toda, A. W. Black, and S. Nakamura, “Modulation spectrumconstrained trajectory training algorithm for GMM-based voice conversion,” inProc. ICASSP, Brisbane, Australia, Apr. 2015, pp. 4859–4863.
[35] K. Hashimoto, K. Oura, Y. Nankaku, and K. Tokuda, “Trajectory training considering global variance for speech synthesis based on neural networks,” in Proc.
ICASSP, Shanghai, China, Mar. 2016, pp. 5600–5604.
[36] T. Nose and A. Ito, “Analysis of spectral enhancement using global variance in
HMM-based speech synthesis,” in Proc. INTERSPEECH, Singapore, May 2014,
pp. 2917–2921.
[37] D. P. Kingma and M. Welling, “Auto-encoding variational Bayes,” arXiv, vol.
abs/1312.6114, 2013.
[38] C.-C. Hsu, H.-T. Hwang, Y.-C. Wu, Y. Tsao, and H.-M. Wang, “Voice conversion
from non-parallel corpora using variational auto-encoder,” in Proc. APSIPA ASC,
Jeju, South Korea, Dec. 2016.
[39] S. R. Bowman, L. Vilnis, O. Vinyals, A. M. Dai, R. Jozefowicz, and S. Bengio,“Generating sentences from a continuous space,” in Proc. CoNLL, Berlin, Germany,Aug. 2016, pp. 10–21.
[40] N. Dilokthanakul, P. A. M. Mediano, M. Garnelo, M. C. H. Lee, H. Salimbeni,
K. Arulkumaran, and M. Shanahan, “Deep unsupervised clustering with gaussian
mixture variational autoencoders,” arXiv, vol. abs/1611.02648, 2016.
[41] N. Hojo, Y. Ijima, and H. Mizuno, “DNN-based speech synthesis using speaker
codes,” IEICE Transactions on Information and Systems, vol. E101-D, no. 2, pp.
462–472, Feb. 2018.
[42] S. Takaki, H. Kameoka, and J. Yamagishi, “Direct modeling of frequency spectra
and waveform generation based on phase recovery for DNN-based speech synthesis,”in Proc. INTERSPEECH, Stockholm, Sweden, Aug. 2017, pp. 1128–1132.
[43] G. Fant, Acoustic theory of speech production with calculations based on X-ray
studies of Russian articulations, De Gruyter, 1970.
[44] H. Kawahara, J. Estill, and O. Fujimura, “Aperiodicity extraction and control using
mixed mode excitation and group delay manipulation for a high quality speech analysis, modiﬁcation and synthesis system STRAIGHT,” in Proc. MAVEBA, Florence,
Italy, Sep. 2001, pp. 59–64.

140

Bibliography[45] T. Fukada, K. Tokuda, T. Kobayashi, and S. Imai, “An adaptive algorithm for melcepstral analysis of speech,” in Proc. ICASSP, San Francisco, U.S.A., Mar. 1992,
pp. 137–140.
[46] K. Yu and S. Young, “Continuous F0 modeling for HMM based statistical parametricspeech synthesis,” IEEE Transactions on Audio, Speech, and Language Processing,vol. 19, no. 5, pp. 1071–1079, Jul. 2011.
[47] H. Kawahara, I. Masuda-Katsuse, and A. D. Cheveigne, “Restructuring speech representations using a pitch-adaptive time-frequency smoothing and an instantaneousfrequency-based F0 extraction: Possible role of a repetitive structure in sounds,”
Speech Communication, vol. 27, no. 3–4, pp. 187–207, Apr. 1999.
[48] M. Morise, F. Yokomori, and K. Ozawa, “WORLD: A vocoder-based high-quality
speech synthesis system for real-time applications,” IEICE Transactions on Information and Systems, vol. E99-D, no. 7, pp. 1877–1884, Jul. 2016.
[49] M. Morise, “D4C, a band-aperiodicity estimator for high-quality speech synthesis,”
Speech Communication, vol. 84, pp. 57–65, Nov. 2016.
[50] T. Yoshimura, K. Tokuda, T. Masuko, T. Kobayashi, and T. Kitamura, “Simultaneous modeling of spectrum, pitch and duration in HMM-based speech synthesis,”in Proc. EUROSPEECH, Budapest, Hungary, Apr. 1999, pp. 2347–2350.
[51] T. Kudo, K. Yamamoto, and Y. Matsumoto, “Applying conditional random ﬁelds
to Japanese morphological analysis,” in Proc. EMNLP, Barcelona, Spain, Jul. 2004,
pp. 230–237.
[52] S. Hochreiter and J. Schmidhuber, “Long short-term memory,” Neural Computation,
vol. 9, no. 8, pp. 1735–1780, Nov. 1997.
[53] S. Fan, Y. Qian, and F. Soong, “TTS synthesis with bidirectional LSTM based
recurrent neural networks,” in Proc. INTERSPEECH, Singapore, Sep. 2014, pp.
1964–1968.
[54] X. Glorot, A. Bordes, and Y. Bengio, “Deep sparse rectiﬁer neural networks,” inProc. AISTATS, Lauderdale, U.S.A., Apr. 2011, pp. 315–323.
[55] M. C. Mozer, “A focused backpropagation algorithm for temporal pattern recognition,” Complex Systems, vol. 3, no. 4, pp. 349–381, Jan. 1989.
[56] K. Kobayashi, T. Toda, G. Neubig, S. Sakti, and S. Nakamura, “Statistical singing
voice conversion with direct waveform modiﬁcation based on the spectrum diﬀerential,” in Proc. INTERSPEECH, Singapore, Sep. 2014, pp. 2514–2518.
[57] K. Kobayashi, T. Toda, and S. Nakamura, “Intra-gender statistical singing voice
conversion with direct waveform modiﬁcation using log-spectral diﬀerential,” SpeechBibliography

141

Communication, vol. 99, pp. 211–220, May 2018.
[58] D. P. Kingma, S. Mohamed, D. J. Rezende, and M. Welling, “Semi-supervised
learning with deep generative models,” in Proc. NIPS, Montreal, Canada, Dec.
2014, pp. 3581–3589.
[59] Y. J. Wu and R. H. Wang, “Minimum generation error training for HMM-based
speech synthesis,” in Proc. ICASSP, Toulouse, France, May 2006, pp. 89–92.
[60] Z. Wu and S. King, “Improving trajectory modeling for DNN-based speech synthesis by using stacked bottleneck features and minimum trajectory error training,”
IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 24, no.
7, pp. 1255–1265, Jul. 2016.
[61] K. Tokuda, T. Yoshimura, T. Masuko, T. Kobayashi, and T. Kitamura, “Speech parameter generation algorithms for HMM-based speech synthesis,” in Proc. ICASSP,
Istanbul, Turkey, Jun. 2000, pp. 1315–1318.
[62] T. Toda and K. Tokuda, “A speech parameter generation algorithm considering
global variance for HMM-based speech synthesis,” IEICE Transactions on Information and Systems, vol. E90-D, no. 5, pp. 816–824, May 2007.
[63] T. Toda, T. Muramatsu, and H. Banno, “Implementation of computationally eﬃcient real-time voice conversion,” in Proc. INTERSPEECH, Portland, U.S.A., Sep.
2012.
[64] S. Imai, K. Sumita, and C. Furuichi, “Mel log spectrum approximation (MLSA)
ﬁlter for speech synthesis,” Electronics and Communications in Japan, vol. 66, no.
2, pp. 10–18, 1983.
[65] D. Griﬃn and J. Lim, “Signal estimation from modiﬁed short-time fourier transform,” IEEE Transactions on Audio, Speech, and Language Processing, vol. 32, no.
2, pp. 236–243, Apr. 1984.
[66] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair,
A. Courville, and Y. Bengio, “Generative adversarial nets,” in Proc. NIPS, Montreal, Canada, 2014, pp. 2672–2680.
[67] N. Chen, Y. Qian, H. Dinkel, B. Chen, and K. Yu, “Robust deep feature forspooﬁng detection - the SJTU system for ASVspoof 2015 Challenge,” in Proc.
INTERSPEECH, Dresden, Germany, Sep. 2015, pp. 2097–2101.
[68] M. Sahidullah, T. Kinnunen, and C. Hanilci¸ , “A comparison of features for synthetic
speech detection,” in Proc. INTERSPEECH, Dresden, Germany, Sep. 2015, pp.
2087–2091.
[69] P. L. D. Leon, M. Pucher, J. Yamagishi, I. Hernaez, and I. Saratxaga, “Evaluation of

142

Bibliography

speaker veriﬁcation security and detection of HMM-based synthetic speech,” IEEE
Transactions on Audio, Speech, and Language Processing, vol. 20, no. 8, pp. 2280–
2290, Oct. 2012.
[70] A. Ogihara, H. Unno, and A. Shiozaki, “Discrimination method of synthetic speechusing pitch frequency against synthetic speech falsiﬁcation,” IEICE Transactions on
Fundamentals of Electronics, Communications and Computer Sciences, vol. E88-A,
no. 1, pp. 280–286, Jan. 2005.
[71] P. L. D. Leon, B. Stewart, and J. Yamagishi, “Synthetic speech discrimination using
pitch pattern statistics derived from image analysis,” in Proc. INTERSPEECH, pp.
370–373. Portland, U.S.A., Sep. 2012.
[72] Z. Wu, X. Xiao, E. S. Chng, and H. Li, “Synthetic speech detection using temporalmodulation feature,” in Proc. ICASSP, Vancouver, Canada, May 2013, pp. 7234–
7238.
[73] E. Grabe and E. L. Low, “Durational variability in speech and the rhythm classhypothesis,” Papers in laboratory phonology 7, pp. 515–546, Jan. 2002.
[74] S. Nowozin, B. Cseke, and R. Tomioka, “f-GAN: Training generative neural samplers
using variational divergence minimization,” in Proc. NIPS, Barcelona, Spain, Dec.
2016, pp. 271–279.
[75] M. Arjovsky, S. Chintala, and L. Bottou, “Wasserstein generative adversarial networks,” in Proc. ICML, Sydney, Australia, Aug. 2017, pp. 214–223.
[76] X. Mao, Q. Li, H. Xie, R. Y. K. Lau, Z. Wang, and S. P. Smolley, “Least squares
generative adversarial networks,” in Proc. ICCV, Venice, Italy, Oct. 2017, pp. 2794–
2802.
[77] D. D. Lee and H. S. Seung, “Algorithms for non-negative matrix factrization,” inProc. NIPS, Denver, U.S.A., Nov. 2000, pp. 556–562.
[78] R. Kompass, “A generalized divergence measure for nonnegative matrix factorization,” Neural Computation, vol. 19, no. 3, pp. 780–891, Mar. 2007.
[79] I. Csiszár and P. C. Shields, “Information theory and statistics: A tutorial,” Foundation and Trends in Communications and Information Theory, vol. 1, no. 4, pp.
417–518, 2004.
[80] Cédric Vilani, Optimal Transport: Old and New, Springer, 2009.
[81] B. Huang, D. Ke, H. Zheng, B. Xu, Y. Xu, and K. Su, “Multi-task learning deepneural networks for speech feature denoising,” in Proc. INTERSPEECH, Dresden,
Germany, Sep. 2015, pp. 2464–2468.
[82] S. Reed, Z. Akata, X. Yan, L. Logeswaran, B. Schiele, and H. Lee, “GenerativeBibliography143

adversarial text-to-image synthesis,” in Proc. ICML, New York, U.S.A., 2016, pp.
1060–1069.
[83] H. Zen and H. Sak, “Unidirectional long short-term memory recurrent neural network with recurrent output layer for low-latency speech synthesis,” in Proc. ICASSP,
Brisbane, Australia, Apr. 2015, pp. 4470–4474.
[84] G. E. Hinton and R. R. Salakhutdinov, “Reducing the dimensionality of data withneural networks,” Science, vol. 313, no. 5786, pp. 504–507, Jul. 2006.
[85] T. R. Marco, S. Sameer, and G. Carlos, ““Why should I trust you?”: Explaining
the predictions of any classiﬁer,” in Proc. KDD, San Francisco, U.S.A., Aug. 2016,
pp. 1135–1164.
[86] Y. Li, S. Kevin, and Z. Richard, “Generative moment matching networks,” in Proc.
ICML, Lille, France, Jul. 2015, pp. 1718–1727.
[87] I. Goodfellow, “NIPS 2016 tutorial: Generative adversarial networks,” arXiv, vol.
abs/1701.00160, 2017.
[88] D. N. Reshef, Y. A. Reshef, H. K. Finucane, S. R. Grossman, G. McVean, P. J.
Turnbaugh, E. S. Lander, M. Mitzenmacher, and P. C. Sabeti, “Detecting novelassociations in large data sets,” Science, vol. 334, no. 6062, pp. 1518–1524, Dec.
2011.
[89] Y. Ijima, T. Asami, and H. Mizuno, “Objective evaluation using association between
dimensions within spectral features for statistical parametric speech synthesis,” inProc. INTERSPEECH, California, U.S.A., Sep. 2016, pp. 337–341.
[90] K. Tanaka, T. Toda, G. Neubig, S. Sakti, and S. Nakamura, “A hybrid approachto electrolaryngeal speech enhancement based on noise reduction and statisticalexcitation generation,” IEICE Transactions on Information and Systems, vol. E97D, no. 6, pp. 1429–1437, Jun. 2014.
[91] S. Kang and H. Meng, “Statistical parametric speech synthesis using weighted multidistribution deep belief network,” in Proc. INTERSPEECH, Singapore, Sep. 2014,
pp. 1959–1963.
[92] H. Zhang, T. Xu, H. Li, S. Zhang, X. Huang, X. Wang, and D. Metaxas, “StackGAN: Text to photo-realistic image synthesis with stacked generative adversarial
networks,” in Proc. ICCV, Venice, Italy, Oct. 2017, pp. 5907–5915.
[93] X. Yin, M. Lei, Y. Qian, F. K. Soong, L. He, Z.-H. Ling, and L.-R. Dai, “ModelingF0 trajectories in hierarchically structured deep neural networks,” Speech Communication, vol. 76, pp. 82–92, Feb. 2016.
[94] T. Kaneko, H. Kameoka, N. Hojo, Y. Ijima, K. Hiramatsu, and K. Kashino, “Gen-144

Bibliography

erative adversarial network-based postﬁlter for statistical parametric speech synthesis,” in Proc. ICASSP, New Orleans, U.S.A., Mar. 2017, pp. 4910–4914.
[95] Y. Sagisaka, K. Takeda, M. Abe, S. Katagiri, T. Umeda, and H. Kawahara, “Alarge-scale Japanese speech database,” in Proc. ICSLP, Kobe, Japan, Nov. 1990,
pp. 1089–1092.
[96] Y. Ohtani, T. Toda, H. Saruwatari, and K. Shikano, “Maximum likelihood voice
conversion based on GMM with STRAIGHT mixed excitation,” in Proc. INTERSPEECH, Pittsburgh, U.S.A., Sep. 2006, pp. 2266–2269.
[97] S. Takamichi, K. Kobayashi, K. Tanaka, T. Toda, and S. Nakamura, “The NAIST
text-to-speech system for the Blizzard Challenge 2015,” in Proc. Blizzard Challenge
workshop, Berlin, Germany, Sep. 2015.
[98] J. Duchi, E. Hazan, and Y. Singer, “Adaptive subgradient methods for online
learning and stochastic optimization,” Journal of Machine Learning Research, vol.
12, pp. 2121–2159, Jul. 2011.
[99] B. Poole, A. Alemi, J. Sohl-dickstein, and A. Angelova, “Improved generator objectives for GANs,” arXiv, vol. abs/1612.02780, 2016.
[100] Y. Xu, J. Du, L.-R. Dai, and C.-H. Lee, “A regression approach to speech enhancement based on deep neural networks,” IEEE/ACM Transactions on Audio, Speech,
and Language Processing, vol. 23, no. 1, pp. 7–19, Jan. 2015.
[101] Q. Wang, W. Rao, S. Sun, L. Xie, E. S. Chng, and H. Li, “Unsupervised domain adaptation via domain adversarial training for speaker recognition,” in Proc.
ICASSP, Alberta, Canada, Apr. 2018, pp. 4889–4893.
[102] K. Tokuda, T. Kobayashi, T. Fukada, H. Saito, and S. Imai, “Spectral estimation of
speech based on mel-cepstral representation,” Transactions on IEICE, vol. J74-A,
no. 8, pp. 1240–1248, Aug. 1991.
[103] T. Kaneko, S. Takaki, H. Kameoka, and J. Yamagishi, “Generative adversarial
network-based postﬁlter for STFT spectrograms,” in Proc. INTERSPEECH, Stockholm, Sweden, Aug. 2017, pp. 3389–3393.
[104] L. Juvela, B. Bollepalli, X. Wang, H. Kameoka, M. Airaksinen, J. Yamagishi, and
P. Alku, “Speech waveform synthesis from MFCC sequences with generative adversarial networks,” in Proc. ICASSP, Calgary, Canada, Apr. 2018, pp. 5679–5683.
[105] C. Donahue J. McAuley and M. Puckette, “Synthesizing audio with GANs,” inProc. ICLR Workshop, Vancouver, Canada, May 2018.
[106] R. Sonobe, S. Takamichi, and H. Saruwatari,“JSUT corpus:

Free large-

scale Japanese speech corpus for end-to-end speech synthesis,”

arXiv, vol.

Bibliography145

abs/1711.00354, 2017.
[107] Y. N. Dauphin, A. Fan, M. Auli, and D. Grangier, “Language modeling with gatedconvolutional networks,” in Proc. ICML, Sydney, Australia, Aug. 2017, pp. 933–941.
[108] T. Kaneko and H. Kameoka, “CycleGAN-VC: Non-parallel voice conversion using
cycle-consistent adversarial networks,” in Proc. EUSIPCO, Rome, Italy, Sep. 2018,
pp. 2114–2118.
[109] H. Kameoka, T. Kaneko, K. Tanaka, and N. Hojo, “StarGAN-VC: Non-parallel
many-to-many voice conversion using star generative adversarial networks,” in Proc.
SLT, Greece, Athens, Dec. 2018, pp. 266–273.
[110] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,”in Proc. CVPR, Las Vegas, U.S.A., Jun. 2016, pp. 770–778.
[111] L. Sun, K. Li, H. Wang, S. Kang, and H. Meng, “Phonetic posteriorgrams for manyto-one voice conversion without parallel data training,” in Proc. ICME, Seattle,
U.S.A., Jul. 2016.
[112] M. Sambur, “Selection of acoustic features for speaker identiﬁcation,” IEEE Transactions on Audio, Speech, and Language Processing, vol. 23, no. 2, pp. 176–182,
Apr. 1975.
[113] Y. Zhang, M. Pezeshki, P. Brakel, S. Zhang, C. L. Y. Bengio, and A. Courville,“Towards end-to-end speech recognition with deep convolutional neural networks,”
in Proc. INTERSPEECH, California, U.S.A., Sep. 2016, pp. 410–414.
[114] G. Heigold, I. Moreno, S. Bengio, and N. Shazeer, “End-to-end text-dependent
speaker veriﬁcation,” in Proc. ICASSP, Shanghai, China, Mar. 2016, pp. 5115–
5119.
[115] A. Tjandra, S. Sakti, and S. Nakamura, “Listening while speaking: Speech chain
by deep learning,” in Proc. ASRU, Okinawa, Japan, Dec. 2017, pp. 301–308.
[116] H. Miyoshi, Y. Saito, S. Takamichi, and H. Saruwatari, “Voice conversion using
sequence-to-sequence learning of context posterior probabilities,” in Proc. INTERSPEECH, Stockholm, Sweden, Aug. 2017, pp. 1268–1272.
[117] M. Miiller, S. Stiiker, and A. Waibel, “Multilingual adapataion of RNN-based ASR
systems,” in Proc. ICASSP, Alberta, Canada, Apr. 2018, pp. 5219–5223.
[118] H.-T. Luong, S. Takaki, G. E. Henter, and J. Yamagishi, “Adapting and controlling
DNN-based speech synthesis using input codes,” in Proc. ICASSP, New Orleans,
U.S.A., Mar. 2017, pp. 1905–1909.
[119] M. Wolfel, “Channel selection by class separability measures for automatic transcriptions on distant microphones,” in Proc. INTERSPEECH, Antwerp, Belgium,

146

Bibliography

Aug. 2007, pp. 582–585.
[120] E. Cooper, C.-I Lai, Y. Yasuda, F. Fang, X. Wang, N. Chen, and J. Yamagishi,“Zero-shot multi-speaker text-to-speech with state-of-the-art neural speaker embeddings,” in Proc. ICASSP, Barcelona, Spain, May 2020, pp. 6184–6188.
[121] A. J. Quinn and B. B. Bederson, “Human computation: A survey and taxonomy ofa growing ﬁeld,” in Proc. SIGCHI, Vancouver, Canada, May 2011, pp. 1403–1412.
[122] J. Li, Y. Baba, and H. Kashima, “Simultaneous clustering and ranking from pairwise
comparisons,” in Proc. IJCAI, Stockholm, Sweden, Jul. 2018, pp. 1554–1560.
[123] B. Settles, “Active learning literature survey,” Computer Sciences Technical Report
1648, University of Wisconsin–Madison, Jan. 2009.
[124] M. Tachibana, T. Nose, J. Yamagishi, and T. Kobayashi, “A technique for controlling voice quality of synthetic speech using multiple regression HSMM,” in Proc.
ICSLP, Pittsburgh, U.S.A., Sep. 2006, pp. 2438–2441.
[125] K. Ohta, Y. Ohtani, T. Toda, H. Saruwatari, and K. Shikano, “Regression approaches to voice quality control based on one-to-many eigenvoice conversion,” inProc. ICSLP, Bonn, Germany, Aug. 2007, pp. 101–106.
[126] J. Lorenzo-Trueba, G. E. Henter, S. Takaki, J. Yamagishi, Y. Morino, and Y. Ochiai,“Investigating diﬀerent representations for modeling and controlling multiple emotions in DNN-based speech synthesis,” Speech Communication, vol. 99, pp. 135–143,
May 2018.
[127] J. R. Hershey, Z. Chen, J. L. Roux, and S. Watanabe, “Deep clustering: Discriminative embeddings for segmentation and separation,” in Proc. ICASSP, Shanghai,
China, Mar. 2016, pp. 31–35.
[128] P. Goyal and E. Ferrara, “Graph embedding techniques, applications, and performance: A survey,” Knowledge-Based Systems, vol. 151, pp. 78–94, Jul. 2018.
[129] D. I. Shuman, S. K. Narang, P. Frossard, A. Ortega, and P. Vandergheynst, “Theemerging ﬁeld of signal processing on graphs: Extending high-dimensional data analysis to networks and other irregular domains,” IEEE Signal Processing Magazine,
vol. 30, no. 3, pp. 83–98, May 2013.
[130] Z. Wu, S. Pan, F. Chen, G. Long, C. Zhang, and P. S. Yu, “A comprehensive survey
on graph neural networks,” IEEE Transactions on Neural Networks and Learning
System, Mar. 2020, (Early Access).
[131] F. M. Zanzotto, “Viewpoint: Human-in-the-loop artiﬁcial intelligence,” Journal of
Artiﬁcial Intelligence Research, vol. 64, pp. 243–252, Feb. 2019.
[132] K. Fujii, Y. Saito, S. Takamichi, Y. Baba, and H. Saruwatari, “HumanGAN: Gen-

Bibliography147

erative adversarial network with human-based discriminator and its evaluation in
speech perception modeling,” in Proc. ICASSP, Barcelona, Spain, May 2020, pp.
6239–6243.
[133] K. Itou, M. Yamamoto, K.Takeda, T. Takezawa, T. Matsuoka, T. Kobayashi,
K. Shikano, and S. Itahashi, “JNAS: Japanese speech corpus for large vocabulary continuous speech recognition research,” Journal of the Acoustical Society ofJapan (E), vol. 20, no. 3, pp. 199–206, May 1999.
[134] C. D. Brown and H. T. Davis, “Receiver operating characteristics curves and related
decision measures: A tutorial,” Chemometrics and Intelligent Laboratory Systems,
vol. 80, no. 1, pp. 24–38, Jan. 2006.
[135] J. A. Hanley and B. J. McNeil, “The meaning and use of the area under a receiver
operating characteristic (ROC) curve,” Radiology, vol. 143, no. 1, pp. 29–36, Apr.
1982.
[136] N. Iwahashi and Y. Sagisaka, “Speech spectrum conversion based on speaker interpolation and multi-functional representation with weighting by radial basis function
networks,” Speech Communication, vol. 16, no. 2, pp. 139–151, Feb. 1995.
[137] K. Akuzawa, Y. Iwasawa, and Y. Matsuo, “Expressive speech synthesis via modelingexpressions with variational autoencoder,” in Proc. INTERSPEECH, Hyderabad,
India, Sep. 2018, pp. 3067–3071.
[138] S. O. Arik, J. Chen, K. Peng, W. Ping, and Y. Zhou, “Neural voice cloning with afew samples,” in Proc. NeurIPS, Montreal, Canada, Dec. 2018, pp. 10019–10029.
[139] A. Tamamori, T. Hayashi, K. Kobayashi, K. Takeda, and T. Toda, “Speakerdependent WaveNet vocoder,” in Proc. INTERSPEECH, Stockholm, Sweden, Aug.
2017, pp. 1118–1122.
[140] T. Baumann, “Partial representations improve the prosody of incremental speechsynthesis,” in Proc. INTERSPEECH, Singapore, Sep. 2014, pp. 2932–2936.
[141] R. Arakawa, S. Takamichi, and H. Saruwatari, “Implementation of DNN-based
real-time voice conversion and its improvements by audio data augmentation andmask-shaped device,” in Proc. SSW, Vienna, Austria, Sep. 2019, pp. 93–98.
[142] T. Saeki, Y. Saito, S. Takamichi, and H. Saruwatari, “Lifter training and subband modeling for computationally eﬃcient and high-quality voice conversion using
spectral diﬀerentials,” in Proc. ICASSP, Barcelona, Spain, May 2020, pp. 7784–
7788.
[143] B. Li and H. Zen, “Multi-language multi-speaker acoustic modeling for LSTM-RNN
based statistical parametric speech synthesis,” in Proc. INTERSPEECH, California,

148

Bibliography

U.S.A., Sep. 2016, pp. 2468–2472.
[144] R. K. Srivastava, K. Greﬀ, and J. Schmidhuber, “Highway networks,” in Proc.
ICML Deep Learning Workshop, Lille, France, Jul. 2015.
[145] X. Wang, S. Takaki, and J. Yamagishi, “Investigating very deep highway networks
for parametric speech synthesis,” in Proc. SSW, California, U.S.A., Sep. 2016, pp.
166–171.
[146] T. Kitamura and M. Akagi, “Speaker individualities in speech spectral envelopes,”
Journal of the Acoustical Society of Japan (E), vol. 16, no. 5, pp. 283–289, Sep.
1995.
[147] D. Erro, A. Moreno, and A. Bonafonte, “Voice conversion based on weighted frequency warping,” IEEE Transactions on Audio, Speech, and Language Processing,vol. 18, no. 5, pp. 922–931, Jul. 2010.
[148] J. Hout and A. Alwan, “A novel approach to soft-mask estmation and log-spectral
enhancement for robust speech recognition,” in Proc. ICASSP, Kyoto, Japan, Mar.
2012, pp. 4105–4108.
[149] Y. Ganin, E. Ustinova, H. Ajakan, P. Germain, H. Larochelle, F. Laviolette,
M. Marchand, and V. Lempitsky, “Domain-adversarial training of neural networks,”
Journal of Machine Learning Research, vol. 17, no. 59, pp. 1–35, Apr. 2016.
[150] S. Sun, C.-F. Yeh, M.-Y. Hwang, M. Ostendorf, and L. Xie, “Domain adversarial
training for accented speech recognition,” in Proc. ICASSP, Alberta, Canada, Apr.
2018, pp. 4854–4858.
[151] J.-C. Chou, C.-C. Yeh, H.-Y. Lee, and L.-S. Lee, “Multi-target voice conversion
without parallel data by adversarially learning disentangled audio representations,”in Proc. INTERSPEECH, Hyderabad, India, Sep. 2018, pp. 501–505.
[152] T. Kaneko, H. Kameoka, K. Tanaka, and N. Hojo, “CycleGAN-VC2: Improved
CycleGAN-based non-parallel voice conversion,” in Proc. ICASSP, Brighton, U.K.,
May 2019, pp. 6820–6824.
[153] J.-Y. Zhu, T. Park, P. Isola, and A. A. Efros, “Unpaired image-to-image translation
using cycle-consistent adversarial networks,” in Proc. ICCV, Venice, Italy, Oct.
2017, pp. 2223–2232.
[154] T. Kaneko, H. Kameoka, K. Tanaka, and N. Hojo, “StarGAN-VC2: Rethinking
conditional methods for stargan-based voice conversion,” in Proc. INTERSPEECH,
Graz, Austria, Sep. 2019, pp. 679–683.
[155] Y. Choi, M. Choi, M. Kim, J.-W. Ha, S. Kim, and J. Choo, “StarGAN: Uniﬁed
generative adversarial networks for multi-domain image-to-image translation,” inBibliography

149

Prop. CVPR, Salt Lake City, U.S.A., Jun. 2018, pp. 8789–8797.
[156] M. Mirza and S. Osindero, “Conditional generative adversarial networks,” arXiv,
vol. abs/1411.1784, 2014.
[157] Y. Zhou, X. Tian, H. Xu, R. K. Das, and H. Li, “Cross-lingual voice conversion
with bilingual phonetic posteriorgram and average modeling,” in Proc. ICASSP,
Brighton, U.K., May 2019, pp. 6790–6794.
[158] S. H. Mohammadi and T. Kim, “One-shot voice conversion with disentangled representations by leveraging phonetic posteriorgrams,” in Proc. INTERSPEECH, Graz,
Austria, Sep. 2019, pp. 704–708.
[159] A. Oord, S. Dieleman, H. Zen, K. Simonyan, O. Vinyals, A. Graves, N. Kalchbrenner,
A. Senior, and K. Kavukcuoglu, “WaveNet: A generative model for raw audio,”arXiv, vol. abs/1609.03499, 2016.
[160] H. Lu, Z. Wu, R. Li, S. Kang, J. Jia, and Helen Meng, “A compact framework forvoice conversion using wavenet conditioned on phonetic posteriorgrams,” in Proc.
ICASSP, Brighton, U.K., May 2019, pp. 6810–6814.
[161] S. Liu, Y. Cao, X. Wu, L. Sun, X. Liu, and H. Meng,“Jointly trained con-version model and WaveNet vocoder for non-parallel voice conversion using melspectrograms and phonetic posteriorgrams,” in Proc. INTERSPEECH, Graz, Austria, Sep. 2019, pp. 704–708.
[162] K. Sugiura, Y. Shiga, H. Kawai, T. Misu, and C. Hori, “A cloud robotics approach
towards dialogue-oriented robot speech,” Advanced Robotics, vol. 29, no. 7, pp.
449–456, Mar. 2015.
[163] K. Maekawa, H. Koiso, S. Furui, and H. Isahara, “Spontaneous speech corpus ofJapanese,” in Proc. LREC, Athens, Greece, May 2000, pp. 947–952.
[164] A. Camacho, “Swipe: A sawtooth waveform inspired pitch estimator for speech andmusic,” Ph.D. dissertation, University of Florida, 2007.
[165] D. Talkin, “A robust algorithm for pitch tracking (RAPT),” Speech Coding and
Synthesis, pp. 495–518, 1995.
[166] A. L. Maas, A. Y. Hannun, and A. Y. Ng, “Rectiﬁer nonlinearities improve neuralnetwork acoustic models,” in Proc. ICML, Atlanta, U.S.A., Jun. 2013.
[167] N. Srivastava, G. E. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov,“Dropout: A simple way to prevent neural networks from overﬁtting,” The Journalof Machine Learning Research, vol. 15, no. 1, pp. 1929–1958, Apr. 2014.
[168] S. Ioﬀe and C. Szegedy, “Batch normalization: Accelerating deep network training
by reducing internal covariate shift,” in Proc. ICML, Lille, France, Jul. 2015, pp.

150

Bibliography

448–456.
[169] B. W. Matthews, “Comparison of the predicted and observed secondary structureof t4 phage lysozyme,” Biochimica et Biophysica Acta (BBA) - Protein Structure,
vol. 405, no. 2, pp. 442–451, Oct. 1975.
[170] A. Kurematsu, K. Takeda, Y. Sagisaka, S. Katagiri, H. Kuwabara, and K. Shikano,“ATR Japanese speech database as a tool of speech recognition and synthesis,”
Speech Communication, vol. 9, no. 4, pp. 357–363, Aug. 1990.

151

Acknowledgements
I would like to express my deepest appreciation to Professor Hiroshi Saruwatari of the
University of Tokyo, my thesis advisor, for his constant guidance and invaluable comments
through my master’s course and doctoral course. I have learned many valuable aspects ofbeing a researcher from his attitude toward research.
I would also like to express my appreciation to Professor Masahiko Inami, Professor
Tatsuya Harada, Associate Professor Masaaki Kondo, and Lecturer at MI center Tomoki
Koriyama of the University of Tokyo, members of the thesis committee, for their valuablecomments and suggestions on this thesis.
I would especially like to express my gratitude to Research Associate Shinnosuke
Takamichi of the University of Tokyo, for his continuous support and various advice.
This work could not have been accomplished without his support. I would also like tothank past Visiting Professor Kunio Kashino (currently Senior Distinguished Researcher
of NTT Communication Science Laboratories), Lecturer Shoichi Koyama, past Research
Assistant Professor Daichi Kitamura (currently Research Associate of National Instituteof Technology, Kagawa College), Research Assistant Professor Tomohiko Nakamura, andAcademic Support Specialist Norihiro Takamune of the University of Tokyo, for theirtechnical comments and lessons.
I want to thank all members of System #1 Lab., Graduate School of Information Scienceand Technology, the University of Tokyo, for their encouragement. I also wish to express
my deep gratitude to Ms. Motoko Kumai, Ms. Hiromi Ogawa, and Ms. Naoko Tanji,
secretaries of our laboratory, for their kind help and support in all aspects of my research.
I experienced many research internship through my master’s course and doctoral course.
I would like to thank my research mentors during the internship, Senior Distinguished Researcher Hirokazu Kameoka of NTT Communication Science Laboratories, Distinguished
Researchers Yusuke Ijima, Kyosuke Nishida, and Ryo Masumura of NTT Media Intelligence Laboratories, and Research Engineer Kentaro Tachibana of LINE Corp. (pastResearch Engineer of DeNA Corp.), for their consistent supports. I would like to oﬀer myspecial thanks to Professor Tomoki Toda of Nagoya University, Professor Junichi Yamag-

152

Acknowledgements

ishi of National Institute of Informatics, and Senior Staﬀ Research Scientist Heiga Zen of
Google Research, for their insightful comments to my work related to this thesis.
Finally, I would like to give my thanks to my family for all their support.

153

Appendix A

VC Using Input-to-output
Highway Networks
A.1

Introduction

In acoustic modeling for VC, not only techniques to alleviate over-smoothing, but also
input speech information can be used since the input and output parameters are often
in the same domain (e.g., cepstrum). This appendix proposes a DNN-based VC method
using input-to-output highway networks. The proposed method generates spectral parameters of converted speech as the sum of input spectral parameters and weighted spectral
diﬀerentials predicted by DNNs. The use of input speech parameters eﬀectively alleviates
over-smoothing, and the weights of the spectral diﬀerentials control which components of
input speech parameters should be converted by the DNNs.

A.2

Proposed VC Method

A.2.1

Input-to-Output Highway Networks for VC

Highway networks [144, 145] are weighted skip-connections between layers, and they often
connect hidden layers. Given that the input and output are often in the same domain(e.g., cepstrum) in VC, this section proposes a VC method using input-to-output highway
networks that have three components: skip connection, transform gate T (·), and spectral diﬀerentials estimator G(·). The converted speech parameters with this method are
generated as follows:ŷ = x + T (x) ◦ G (x) .

(A.1)

154

Appendix A VC Using Input-to-output Highway Networks

ෝ
࢟

࢞

‫ڮ‬

‫ڮ‬
࢞

ࡳ ‫ڄ‬

‫ڮ‬

Spectraldifferentials
estimator

‫ڮ‬

࢞

ࢀ ‫ڄ‬

ࢀ ࢞

‫ڮ‬

Transform
gate

‫ڮ‬

ࡳ ࢞

Fig. A.1. VC using input-to-output highway networks

The transform gate T (·) is described as Feed-Forward DNNs. Each value of T (x) ranges
from 0.0 to 1.0, representing time- and feature-varying weights of spectral diﬀerentials
predicted as G(x). When T (x) = 0, input speech parameters are directly used as converted speech parameters, and when T (x) = 1, the architecture is equivalent to residual
networks [110]. Therefore, input speech parameters are strongly transformed by G(·)
when the transform gate’s value becomes close to 1.0. Figure A.1 shows the proposed VC
method using input-to-output highway networks. The loss function for training DNNs
is equal to the MGE loss shown in Eq. (2.16). Model parameters of T (·) and G(·) are
simultaneously estimated to minimize the loss function.

A.2.2

Discussions

Since the proposed method utilizes both input speech parameters and spectral diﬀerentials
weighted by the transform gate, it eﬃciently alleviates over-smoothing of the converted
speech parameters. Figure A.2 shows scatter plots of the speech parameters. This ﬁgureplots pairs of mel-cepstral coeﬃcients whose corresponding value of the transform gate islarge (i.e., close to use of residual networks) or small (i.e., close to direct use of input speechparameters). The proposed input-to-output highway networks alleviate over-smoothing
better than conventional Feed-forward DNNs in both cases.
The variation in spectral parameters between speakers strongly depends not only on
speaker pairs, but also on frequency bands and phonetic environments. For example,

53rd mel-cepstralcoefficient(µT =0.25966996)

27th mel-cepstralcoefficient(µT =0.58094335)

Appendix A VC Using Input-to-output Highway Networks

NaturalFeed-Forward

155

Input-to-output
highway

19th mel-cepstral coefficient(µT =0.56472015)

44th mel-cepstral coefficient(µT =0.28273895)

Fig. A.2. Scatter plots of speech parameters. µT denotes transform gate’s value averaged over one utterance.

formant structure diﬀers greatly between the genders, and the inter-speaker variation
in low-frequency band is small within the same gender. On the other hand, the interphoneme variation (i.e., intra-speaker variation) in the low frequency band tends to belarge [146]. Therefore, a golden VC method is desirable to have an acoustic model thatavoids excessive conversion when the diﬀerence between input and output features issmall (e.g., frequency warping [147]) and provides ﬂexible conversion when the diﬀerenceis large. The proposed method can achieve such VC by using the spectral diﬀerential estimation learned with data-driven weights. Figure A.3 shows examples of transform gate’svalues predicted from mel-ﬁlter banks. This ﬁgure shows that G(·) greatly transforms
the spectral parameters in the high frequency band, since they strongly represent voice
characteristics of a speaker. Meanwhile, in male-to-male VC (Fig. A.3(a)), G(·) does nottransform the spectral parameters in the low frequency band unlike that of male-to-female
VC (Fig. A.3(b)).
From another perspective, the transform gate in the proposed method can be regarded
as soft selection of features. The dimensionality of spectral features (e.g., the order of156

Appendix A VC Using Input-to-output Highway Networks

Index of mel-filter bank(a) Male-to-male
55
50
45
40
35
30
25
20
15
10
5
100

200

300

(b) Male-to-female

400 100 200
Frame index

300

400

1.0
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0.0

Fig. A.3. Examples of transform gate’s values predicted from mel-ﬁlter banks

mel-cepstral coeﬃcients) is a hyperparameter for VC. For instance, the use of only lower
order of mel-cepstra makes the acoustic model training robust while it degrades speech
quality due to insuﬃcient conversion of spectral features. On the other hand, the useof higher order improves speech quality but suﬀers from the randomness of the spectralfeatures. The former case corresponds to T (x) = 1 for the lower order and T (x) = 0
for the higher order. The latter case corresponds to T (x) = 1 for all orders. Whereas
such a hard selection is often used, the proposed method can utilize a soft selection by
the transform gate. Figure A.4 shows examples of transform gate’s values predicted frommel-cepstral coeﬃcients. Lower orders of mel-cepstral coeﬃcients, which are dominant
in speaker conversion, tend to be strongly transformed by G(·). On the other hand,higher orders of mel-cepstral coeﬃcients tend to be not completely ignored, but weakly
converted.
The transform gate in the proposed method is similar to adaptive soft-masking [148]in speech enhancement. Hence, it is expected that knowledge can be shared between VC
and speech enhancement.

Index of mel-cepstral coefficients

Appendix A VC Using Input-to-output Highway Networks(a) Male-to-male
55
50
45
40
35
30
25
20
15
10
5
100

200

(b) Male-to-female

300 400 100 200
Frame index

300 400

157

1.0
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0.0

Fig. A.4. Examples of transform gate’s values predicted from mel-cepstral coeﬃcients

A.3

Experimental Evaluation

A.3.1

Experimental Conditions

A speech corpus of two male speakers and one female speaker was used. The speakers
uttered 503 phonetically balanced sentences [95]. The numbers of sentences for the training
and evaluation were 450 (subsets A to I) and 53 (subset J), respectively. The sampling
rate of speech signals was 16 kHz. The shift length was set to 5 ms. The 0th-through59th mel-cepstral coeﬃcients were used as the spectral parameters. F0 and 5 bandaperiodicity [44, 96] were used as excitation parameters. The STRAIGHT vocoder [47]was used for the parameter extraction and speech waveform synthesis. The 0th melcepstral coeﬃcients of input speech were used as those of the converted speech. Speechparameter trajectory smoothing [97] with a 50 Hz cutoﬀ modulation frequency was applied
to the spectral parameters in the training data for improving training accuracy. In the
training phase, the spectral parameters were normalized to have zero-mean and unitvariance. The MGE training [60] was performed with 25 iterations. Two DNNs for maleto-male and male-to-female VC were trained. The DNN architectures were Feed-Forward
networks that included three 512-unit ReLU [54] hidden layers and a 118-unit linear

158

Appendix A VC Using Input-to-output Highway Networks(a) Male-to-male
Feed-Forward
Input-to-output
highway
0.0

0.2

0.4

0.6

0.8

1.0

0.4
0.6
0.8
Preference score

1.0(b) Male-to-female
Feed-Forward
Input-to-output
highway
0.0

0.2

Fig. A.5. Preference scores of converted speech’s speech quality with 95% conﬁdence
intervals (DNN-based VC using input-to-output highway networks)

output layer. The acoustic model predicted static and dynamic mel-cepstral coeﬃcients(118-dim.) frame by frame. The transform gate had a 59-unit input and 59-unit sigmoid
output layers. The optimization algorithm was AdaGrad [98]. The learning rate was set to0.01. In the VC process, linearly transformed F0 and source-speaker’s band-aperiodicity
were used.

A.3.2

Subjective Evaluation

Subjective evaluations on quality and speaker individuality of converted speech were conducted. In the subjective evaluation, the proposed method was compared with a conventional one that uses Feed-Forward DNNs as an acoustic model. A preference AB test on
the speech quality was conducted. Every pair of speech samples converted by the two
methods was presented in random order. Listeners were asked to select speech samples
that sounded like they had better quality. Similarly, a preference XAB test on the speakerindividuality was conducted using target speaker’s natural speech as the reference “X.”Thirty listeners participated in each assessment, using crowdsourced evaluation systems.
Figures A.5 and A.6 show the evaluation results regarding the converted speech quality
and speaker individuality, respectively.

The proposed method outperforms the con-

Appendix A VC Using Input-to-output Highway Networks

159(a) Male-to-male
Feed-Forward
Input-to-output
highway
0.0

0.2

0.4

0.6

0.8

1.0

0.4
0.6
0.8
Preference score

1.0(b) Male-to-female
Feed-Forward
Input-to-output
highway
0.0

0.2

Fig. A.6. Preference scores of converted speech’s speaker individuality with 95% conﬁdence intervals (DNN-based VC using input-to-output highway networks)

ventional one regarding both speech quality and speaker individuality. These results
demonstrate the eﬀectiveness of the proposed method.

160

Appendix B

Joint Adversarial Training
Algorithm for DNN-based
Many-to-one VC
B.1

Introduction

This appendix proposes an adversarial algorithm to train DNNs for many-to-one VC using
speech recognition and synthesis. The proposed algorithm incorporates the GAN-based
speech synthesis method (Chapter 3) and domain-adversarial training (DAT) of a speech
recognition model for improving both converted speech naturalness and speaker similarity.
The algorithm jointly trains the speech synthesis and recognition models to optimize themfor many-to-one VC.

B.2

Conventional Algorithm for DNN-based
Many-to-one VC

The conventional algorithm [111] trains DNN-based speech recognition and synthesis models to represent mapping from any arbitrary input speech to the target speech, using PPGs
as the intermediate representation of the VC process.

Phonetic posteriorgrams(c) Prop.
(b) Prop.
(DAT-GAN)(GAN) (a) Baseline

Appendix B Joint Adversarial Training Algorithm for DNN-based Many-to-one VC

Male-1

Male-2

Female-1

161

Female-2

Frame index
Fig. B.1. Examples of PPGs predicted by speech parameters of four diﬀerent speakers
who uttered same sentences used in subjective evaluation described in section
B.4.3. Horizontal and vertical axes represent temporal axis and phoneme index, respectively. Brighter values denote high posterior probabilities. Rangesof temporal axes in these ﬁgures were modiﬁed for clear illustration.

B.2.1

Speech Recognition Model Training

The recognition model R(·) is trained to predict a phoneme label sequence p from an input
speech parameter sequence x. A pair of the phoneme label and input speech parameter
(M)

(M)

(M)

(M)
denotesis sampled from a multi-speaker corpus D(M) = {(xn , pn )}N
n=1 , where N

the amount of training data for the recognition model. A PPG sequence p̂(M) = R(x(M) )is predicted by the recognition model. The recognition model R(·) is trained to minimize
the phoneme prediction loss deﬁned as the SCE between the phoneme label and PPG,
i.e., LSCE (p(M) , p̂(M) ).

B.2.2

Speech Synthesis Model Training

Assuming that R(·) is the speaker-independent speech recognition model, the targetspeaker-dependent speech synthesis model G(·) is trained to generate a target speechparameter sequence y. A pair of the input and target speech parameters is sampled from
(O)

(O)

(O)

(O)
a target speaker corpus D(O) = {(xn , y n )}N
denotes the amount of
n=1 , where N

training data for the synthesis model. A generated speech parameter sequence ŷ (O) isAppendix B Joint Adversarial Training Algorithm for DNN-based Many-to-one VC

25th mel-cepstralcoefficient

162

0.3

(a) Natural

-0.2
-0.2

0.3

(b) Baseline-0.2
0.5 -0.2

0.3-0.2
0.5 -0.2(c) Prop.
(GAN)

0.3

(d) Prop.
(DAT-GAN)-0.2
0.5 -0.2

0.5

23rd mel-cepstral coefficientFig. B.2. Scatter plots mel-cepstral coeﬃcients of (a) natural speech and (b)–(d) converted speech by three diﬀerent algorithms. Female target speaker’s one utterance excluded from training data is used for extracting these mel-cepstralcoeﬃcients.

obtained through the recognition and synthesis models, i.e., ŷ (O) = G(R(x(O) )). The
synthesis model G(·) is trained to minimize the MSE between the target and generated
speech parameter sequences, i.e., LMSE (y (O) , ŷ (O) ). Note that model parameters of R(·)
are not updated by this training.

B.2.3

Problems

The conventional algorithm can train the DNNs for many-to-one VC without requiring
parallel speech corpora. However, actual PPGs fed into the synthesis model can be diﬀerent among input speakers as shown in Fig. B.2(a). One of the reason is that the speech
recognition model training using the phoneme prediction loss does not guarantee to learn
speaker-invariant PPGs. The PPG diﬀerences can degrade the converted speech quality
because the synthesis model trained on only the target speaker’s PPGs does not necessarily generalize to any source speakers’ PPGs. Moreover, over-smoothing of generated
speech parameters shown in Fig. B.1(b) considerably deteriorates the converted speech
quality.

Appendix B Joint Adversarial Training Algorithm for DNN-based Many-to-one VC

B.3

163

Proposed Algorithm for DNN-based
Many-to-one VC Using Adversarial Training

B.3.1

Joint Adversarial Training of Speech Recognition and Synthesis Models

DAT-based Speech Recognition Model Training for Many-to-one VC
The DAT [149] is a general framework to train a DNN-based recognition model that is morerobust towards variation in input features by learning domain-invariant latent variables.
It has been applied to DNN-based speech information processing such as accented speech
recognition [150] and robust speaker classiﬁcation [101]. Although the DAT was originallyinvented to improve the recognition model performance, Chou et al. [151] demonstrated its
eﬃcacy in autoencoder-based VC that learns speaker-independent latent variables. Notethat their method does not guarantee that the latent variables represent phonetic content
of input speech, unlike many-to-one VC using PPGs. To improve the converted speech
quality of many-to-one VC, it is crucial to obtain PPGs that are invariant to variation
in input speakers. Therefore, the DAT in the proposed algorithm regards the two speechcorpora used for the DNN training, i.e., 1) the multi-speaker corpus D(M) and 2) thetarget speaker corpus D(O) , as the domains. The DAT tries to minimize the diﬀerence
between the two domains and learn the speaker-invariant speech recognition model that
produces better PPGs for many-to-one VC.
For clear formulation, the recognition model R(·) are split into two sub-models as
R(·) = Rp (Rf (·)). The ﬁrst sub-model Rf (·) is a feature extractor that extracts features f̂
representing phonetic content of input speech from input speech parameters as f̂ = Rf (x).
The second sub-model Rp (·) is a phoneme predictor that predicts PPGs from the extracted
features, i.e., p̂ = Rp (f̂ ) = Rp (Rf (x)). To capture the domain diﬀerence, the DAT
introduces a domain classiﬁer C(·) that uses the features f̂ to identify a domain where
input speech parameters belong to. The domain classiﬁer C(·) is trained to minimize thedomain classiﬁcation loss deﬁned as: (M) (O)  (O)  (M) Ldc f̂, f̂
= − log C f̂
− log 1 − C f̂,
where f̂(M)

and f̂

(O)(B.1)

are extracted from x(M) and x(O) , respectively. On the other hand,

164

Appendix B Joint Adversarial Training Algorithm for DNN-based Many-to-one VC

the recognition model R(·) is trained to minimize the loss deﬁned as follows:


(M)

LR p

(M)

, p̂

, f̂

(M)

, f̂

(O)





 (M) (O) (M)(M)= LSCE p , p̂
− ωC Ldc f̂
,
, f̂

(B.2)

where ωC is a hyperparameter that controls the eﬀect of the second term. The loss function
can be regarded as the weighted sum of the phoneme prediction loss and the loss to makeC(·) misclassify the domains by learning the domain-invariant features f̂ . Therefore, theminimization of Eq. (B.2) can be expected to reduce the PPG diﬀerences among input
speakers.
GAN-based Speech Synthesis Model Training for Many-to-one VC
The proposed algorithm also introduces the GAN-based training method for speech synthesis (Chapter 3) to overcome over-smoothing of generated speech parameters. Referringto the subjective evaluation results in Section 3.4.10, W-GAN [75]-based algorithm (Section 3.3.4) was adopted.
Joint Optimization of Speech Recognition and Synthesis Models
To optimize both the recognition and synthesis models for many-to-one VC, the proposed
algorithm jointly train the two models with a uniﬁed framework. The domain classiﬁer
C(·) and discriminator of the GANs D(·) are ﬁrst updated by minimizing Eqs. (B.1)
and (3.15), respectively. The recognition model R(·) and synthesis model G(·) are thenjointly updated by minimizing the sum of Eqs. (B.2) and (3.3). Since the loss for training
G(·) is also used for training R(·), the recognition model can be expected to predictspeaker-invariant PPGs that can accurately generate target speech parameters. Figure
B.3 illustrates a schematic diagram of the proposed joint adversarial training.

B.3.2

Discussion

As shown in Figs. B.2(c) and B.2(d), the GAN-based algorithm alleviates over-smoothingof generated speech parameters. However, only using GANs cannot reduce the PPG
diﬀerences as shown in Fig. B.1(b). On the other hand, the proposed algorithm using
both the DAT and GANs successfully reduces the PPG diﬀerences as shown in Fig. B.1(c).
This result can be expected to improve the converted speech quality.
In the GAN-based algorithm, the statistical diﬀerence between target speech parametersy(O)and ones predicted by other speakers, i.e., ŷ (M) = G(R(x(M) )), can be reduced

directly. This can be done by approximating the diﬀerence in the training of D(·) andAppendix B Joint Adversarial Training Algorithm for DNN-based Many-to-one VC

Input
speech Extracted
params.
feats.
PPGs
୑
ෝ୑
࢞
࢖
ࢌ෠ ୑
ࡾ୤ ‫ڄ‬ࡾ୮ ‫ڄ‬
ࢌ෠ ୓

࢞୓

‫ڄ ܥ‬

ࡾ୤ ‫ڄ‬ෝ୓࢟01(a) Updating ‫ ڄ ܥ‬and ‫ڄ ܦ‬

ࢌ෠ ୑

࢞୑

Targetspeechparams.

ࡳ ‫ڄ‬

ෝ୓
࢖

ࡾ୮ ‫ڄ‬

165

‫ܮ‬ୢୡ

࢟୓
‫ڄ ܦ‬

ෝ୓
‫࢟ ܦ‬
െ‫୓ ࢟ ܦ‬Phonemelabelsෝ ୑‫ܮ‬
࢖
࢖୑
ୗେ୉
ࡳ ‫ڄ‬
‫୑ܮ‬ୗ୉

࢞୓

ࢌ෠ ୓

ෝ୓
࢖

ෝ୓
࢟

‫ڄ ܦ‬
1
0
(b) Updating ࡾ୤ ‫ ڄ‬, ࡾ୮ ‫ ڄ‬, and ࡳ ‫ڄ‬

࢟୓

‫ڄ ܥ‬

െ‫ܮ‬ୢୡෝ୓െ‫࢟ ܦ‬

Fig. B.3. Schematic diagram of proposed joint adversarial training algorithm. Domainclassiﬁer C(·) and discriminator of GANs D(·) are ﬁrst updated. Recognition model R(·) = Rp (Rf (·)) and synthesis model G(·) are then updated.
These updates are iterated during training, and DNNs for many-to-one VC
are constructed by concatenating ﬁnal R(·) and G(·).

minimizing the distance in the training of G(·). However, it was found from preliminary
experiment results that this training considerably degraded the converted speech quality.
One possible reason is that the diﬀerences among the target and other speakers in the
speech parameter domain may be larger than those in the extracted feature domain, andminimizing the former using GANs becomes more diﬃcult.
Regarding related work, several methods that incorporate GANs into the DNN training
for VC have been proposed. CycleGAN-VC [108, 152] achieves non-parallel VC based on
adversarial training considering cyclic-consistency [153]. StarGAN-VC [109, 154] extends
this idea to many-to-many VC by introducing StarGAN [155] into the acoustic modeltraining. Although these techniques can achieve non-parallel VC without using any texttranscriptions, they cannot guarantee the quality of converted speech when the input
speaker is unseen. On the other hand, the proposed algorithm can take any arbitrary
source speakers for VC. Although this algorithm requires a large speech corpus with text

166

Appendix B Joint Adversarial Training Algorithm for DNN-based Many-to-one VC

transcriptions and limits the target speaker to a speciﬁc one, semi-supervised training of
the recognition and synthesis models (e.g., machine speech chain [115]) and conditional
GAN [156] using one-hot speaker codes [41] can be expected to alleviate these limitations.
Also, one can apply the proposed algorithm to more practical VC using PPGs, such as
cross-lingual VC [157], one-shot VC [158], and WaveNet [159]-based VC [160, 161].

B.4

Experimental Evaluation

B.4.1

Experimental Conditions

Two professional speakers, i.e., one voice actress (FT) taken from the NICT Voice Actress
Dialogue Corpus [162] and one voice actor (MT) included in an internal dataset of DeNA,
were used as the target speakers for VC. The Corpus of Spontaneous Japanese (CSJ) [163]was used to train the speech recognition model R(·). The CSJ included 1,417 amateur
speakers (470 females and 947 males) with various speaking styles such as a monologue,
dialogue, and reading aloud. About 99% of the CSJ was used as the multi-speaker corpus
D(M) . The remainders of the CSJ was used for objective evaluations described in Section
B.4.2. The FT’s 5,174 utterances or MT’s 2,211 utterances were used as the target speakercorpus D(O) to train the speech synthesis model G(·). The other 50 utterances of the target
speakers were used for objective evaluations described in Section B.4.2. Note that D(M)and D(O) were signiﬁcantly diﬀerent in many aspects, such as recording environments,
speaking styles, and speaking skills. All speech samples were downsampled at 16 kHz.
The WORLD vocoder [48] (D4C edition [49]) was used to extract log F0, 40-dimensional
mel-cepstral coeﬃcients, and band aperiodicity. In many-to-one VC, DNNs predicted
the 1st-through-39th mel-cepstral coeﬃcients of the target speaker. The F0 values wereextracted by integrating results of multiple F0 extractors [48, 164, 165]. The log F0 was
linearly converted. Source speaker’s band aperiodicity and the 0th mel-cepstral coeﬃcientswere directly used for the speech waveform synthesis.
All DNN architectures were 1D CNNs along time axis [26] with a ﬁxed sequence lengthof 128 frames. The feature extractor Rf (·) extracted 256-dimensional features from 13dimensional MFCCs and their dynamic features. The phoneme predictor Rp (·) predicted43-dimensional Japanese PPGs from the extracted features. The synthesis model G(·)
generated the 1st-through-39th mel-cepstral coeﬃcients of the target speaker from the
PPGs. The MFCCs and mel-cepstral coeﬃcients were normalized to have zero-mean andunit-variance. The domain classiﬁer C(·) distinguished D(O) from D(M) using the ex-Appendix B Joint Adversarial Training Algorithm for DNN-based Many-to-one VC

167

Table B.1. DNN architectures used in experimental evaluation.
In this table,“Conv1D(Cin , Cout , k , s)” and “Deconv1D(Cin , Cout , k , s)” denote1D convolution and deconvolution layers, respectively. Cin and Cout mean
number of channels of input and output, respectively. k and s denote convolution window size and stride width, respectively

Recognition R(·) = Rp (Rf (·))
Feature extractor Rf (·)
Conv1D(26, 256, 15, 1)
Conv1D(256, 512, 5, 2)
Conv1D(512, 1024, 5, 2)
Deconv1D(1024, 512, 5, 2)
Deconv1D(512, 256, 5, 2)
Phoneme predictor Rp (·)
Conv1D(256, 43, 15, 1)
Domain classiﬁer C(·)
Conv1D(256, 512, 1, 1)
Conv1D(512, 512, 5, 1)
Conv1D(512, 512, 5, 1)
Conv1D(512, 1, 1, 1)

Synthesis G(·)
Conv1D(43, 256, 15, 1)
Conv1D(256, 512, 5, 2)
Conv1D(512, 1024, 5, 2)
Deconv1D(1024, 512, 5, 2)
Deconv1D(512, 256, 5, 2)
Conv1D(256, 39, 15, 1)

Discriminator of GANs D(·)
Conv1D(39, 512, 1, 1)
Conv1D(512, 512, 5, 1)
Conv1D(512, 512, 5, 1)
Conv1D(512, 1, 1, 1)

tracted features. The discriminator of the GANs D(·) distinguished natural mel-cepstralcoeﬃcients from generated ones. The activation function for hidden layers was the leaky
ReLU [166]. Dropout [167] was applied to all hidden layers for avoiding overﬁtting. Batch
normalization [168] was applied to some hidden layers in the synthesis model for accelerating the DNN training. Table B.1 shows details of the DNN architectures. In this table,“Conv1D(Cin , Cout , k, s)” and “Deconv1D(Cin , Cout , k, s)” denote 1D convolutional anddeconvolutional layers, respectively. The Cin and Cout mean the number of channels of
input and output, respectively. The convolution window size and stride width are denotedby k and s, respectively.
As an initial setting, the recognition model R(·) = Rp (Rf (·)) was pretrained with themulti-speaker corpus D(M) . The initialization was performed with one iteration using allutterances in D(M) . The optimizer used for the initialization was AdaGrad [98]. The
learning rate for the initialization was set to 0.01. The frame-wise phoneme prediction accuracy of the initialized recognition model was 80.4%, calculated with the evaluation dataof the CSJ. Five DNNs for many-to-one VC were trained using the initialized recognition
model with the following algorithms:
Baseline: Training G(·) with the ﬁxed R(·) [111]Prop. (Joint): Jointly training R(·) and G(·) with ωC = 0 and ωD = 0

168

Appendix B Joint Adversarial Training Algorithm for DNN-based Many-to-one VC

Prop. (DAT): Jointly training R(·) and G(·) with ωC = 0.25 and ωD = 0.0
Prop. (GAN): Jointly training R(·) and G(·) with ωC = 0 and ωD = 0.5
Prop. (DAT-GAN): Jointly training R(·) and G(·) with ωC = 0.25 and ωD = 0.5
Here, the hyperparameters (ωC , ωD ) were empirically chosen. All of the ﬁve algorithms
were performed with ﬁve iterations using all utterances included in D(O) . In the training of“Prop. (*),” pairs of labeled training data (x(M) , l(M) ) were randomly sampled from D(M) .
The optimizers for training all DNNs, i.e., R(·), G(·), C(·), and D(·), were AdaGrad. The
learning rates for the training were set to 0.01.

B.4.2

Objective Evaluations

Logarithmic Global Variance Distance
Since non-parallel many-to-one VC was considered, any objective evaluation metrics that
require parallel speech utterances of source and target speakers could not be calculated.
Instead, GVs [25] of the target speaker’s (MT or FT) natural and generated speech were
calculated. This section calculated a logarithmic GV distance (LogGVD) deﬁned as follows:
LogGVD (v, v̂) =1|| log v̂ − log v||22 ,
M

(B.3)

where v and v̂ denote M -dimensional GV vectors of natural and generated mel-cepstralcoeﬃcients, respectively. This evaluation would quantify the ideal performance of the
many-to-one VC method because the domain mismatch in the input speech parameters
never occurred. Fifty utterances of MT or FT were used to calculate the LogGVDs.
Table B.2 lists the averaged LogGVDs and their standard deviations. From the results,“Prop. (GAN)” signiﬁcantly reduces the LogGVDs better than “Baseline,” suggestingthat the GAN-based algorithm overcomes over-smoothing of generated speech parameters
in many-to-one VC. “Prop. (DAT)” also decreases the LogGVDs, and the combination of
the DAT and GANs, i.e., “Prop. (DAT-GAN),” achieves the lowest value among the ﬁve
algorithms. These results indicate that training the speaker-invariant recognition modelincreases the accuracy in modeling the target speech parameters. On the other hand,“Prop. (Joint)” shows the diﬀerent tendencies in accordance with the diﬀerent target
speakers; i.e., its LogGVDs are almost the same as “Baseline” when the MT is used andsimilar to “Prop. (DAT)” in other cases. One of the reasons may be the data imbalanceof the CSJ corpus that includes more male speakers than female ones.

Appendix B Joint Adversarial Training Algorithm for DNN-based Many-to-one VC

169

Table B.2. LogGVDs between natural and generated mel-cepstral coeﬃcients with theirstandard deviations

Baseline
Prop. (Joint)
Prop. (DAT)Prop. (GAN)
Prop. (DAT-GAN)

Target speaker
MT
FT
1.96 ± 0.34
5.05 ± 0.71
1.98 ± 0.38
3.93 ± 0.78
1.44 ± 0.26
3.80 ± 0.67
0.44 ± 0.18
0.21 ± 0.11
0.23 ± 0.10 0.11 ± 0.06

Table B.3. Frame-wise phoneme recognition accuracy of speech recognition models [%]Baseline
Prop. (Joint)
Prop. (DAT)Prop. (GAN)
Prop. (DAT-GAN)

Target speaker
MT
FT
80.4
80.4
63.5
77.6
62.3
77.5
62.8
77.5
62.4
77.0

Speech Recognition Accuracy
Although improving the speech recognition accuracy is not the ﬁnal goal of the manyto-one VC method, whether the proposed algorithms aﬀect the accuracy or not deserves
to be reported. Here, the frame-wise phoneme recognition accuracy of the recognition
models after the conventional or proposed training was calculated. The evaluation dataof the CSJ corpus was used to calculate the accuracy.
Table B.3 lists the evaluation results. From the results, “Baseline” achieves the highest
recognition accuracy. This is a natural result since the recognition model is trained tominimize the recognition error and ﬁxed during the synthesis model training. Meanwhile,
all the proposed algorithms decrease the accuracy, suggesting that the loss functions fortraining the speech synthesis do not necessarily improve the speech recognition accuracy.
Speaker Invariance of Speech Recognition Model
The Matthews correlation coeﬃcients (MCC) [169] of the domain classiﬁer was calculated
to evaluate the speech recognition model’s robustness against the input speaker variation.

170

Appendix B Joint Adversarial Training Algorithm for DNN-based Many-to-one VC
Table B.4. MCCs of domain classiﬁers

Baseline
Prop. (Joint)
Prop. (DAT)Prop. (GAN)
Prop. (DAT-GAN)

Target speaker
MT
FT
0.36
0.33
0.22
0.16
0.02
0.04
0.18
0.18
0.04
0.04

The MCC quantiﬁes the performance of a binary classiﬁcation model and is deﬁned as:
MCC = pTP × TN − FP × FN
(TP + FP)(TP + FN)(TN + FP)(TN + FN)

,

(B.4)

where TP, TN, FP, and FN denote the numbers of true positives, true negatives, false
positives, and false negatives of the classiﬁcation results, respectively. The MCC takes avalue between −1 (complete misclassiﬁcation) and +1 (perfect classiﬁcation), and their
mid-value (i.e., 0) corresponds to no better than random classiﬁcation. Therefore, thegoal in the many-to-one VC method is to cause the speech recognition model to learn
features that make the MCC of the domain classiﬁer become close to 0. Fifty utterancesof the target speaker MT or FT were used as positive examples and 500 utterances (50utterances × 10 speakers) taken from the evaluation data of the CSJ were used as negative
examples to calculate the MCCs.
Table B.4 lists the evaluation results. From the results, all the proposed algorithms
decrease the MCCs compared with “Baseline.” In particular, the use of DAT makes the
MCCs close to almost zero, while the other methods do not. These results demonstratethat the proposed algorithm using the GAN and DAT not only reduces the statistical
diﬀerences between natural and generated speech parameters, but also makes the speech
recognition model more invariant to the domain mismatch between the target and otherspeakers.

B.4.3

Subjective Evaluations

Subjective evaluations on the naturalness and speaker similarity of the converted speech
were conducted. Since the speech corpora for building the many-to-one VC systems werecompletely non-parallel, the ATR Japanese Speech Database (set C) [170] was used forselecting source speakers. The database included 291 amateur speakers (143 females andAppendix B Joint Adversarial Training Algorithm for DNN-based Many-to-one VC

171

Table B.5. Results of MOS test on naturalness of converted speech with their 95%
conﬁdence intervals. Bold values indicate that method is more naturallysounded than “Baseline” with p-value < 0.05(a) Results of FSs/MSs-to-FT VC

Baseline
Prop. (GAN)
Prop. (DAT-GAN)

FSs-to-FT
2.70 ± 0.12
3.00 ± 0.13
2.95 ± 0.13

MSs-to-FT
2.51 ± 0.11
2.55 ± 0.12
2.75 ± 0.12(b) Results of FSs/MSs-to-MT VC

Baseline
Prop. (GAN)
Prop. (DAT-GAN)

FSs-to-MT
2.63 ± 0.10
2.94 ± 0.11
2.96 ± 0.12

MSs-to-MT
2.55 ± 0.11
3.01 ± 0.12
2.96 ± 0.11

148 males) with a reading-aloud style. One parallel speech utterances (phonetically balanced sentence A01) of randomly selected 10 male speakers (MSs) and 10 female speakers(FSs) were used for investigating the eﬀects of variation in source speakers. Here, the performances of conventional and proposed algorithms were evaluated in four many-to-one
VC settings: FSs-to-FT, MSs-to-FT, FSs-to-MT, and MSs-to-MT.
Evaluation of Naturalness
A ﬁve-point scaled MOS test was conducted to compare “Baseline” with the GAN-based
proposed algorithms (i.e., “Prop. (GAN)” and “Prop. (DAT-GAN)”) in terms of theconverted speech naturalness. The converted speech generated by the three many-toone VC systems was presented to listeners in random order. In evaluation of FSs-to-FT
and MSs-to-FT VC, thirty listeners participated in the assessment by using crowdsourced
subjective evaluation systems. Each listener evaluated 60 converted speech samples (20source speakers × the three algorithms). Similarly, the evaluation of FSs-to-MT andMSs-to-MT VC was conducted with 25 listeners.
Table B.5 shows the evaluation results. “Prop. (DAT-GAN)” outperforms “Baseline”
in the all VC tasks, demonstrating that the algorithm incorporating the DAT and GAN
is eﬀective in improving the naturalness of the converted speech. A noteworthy fact isthat the proposed algorithm using only the GAN does not always yield the signiﬁcant
improvement as shown in Table B.5(a). This result suggests that just using the GANbased training is insuﬃcient to deal with the diﬀerences among speakers observed in
PPGs.

172

Appendix B Joint Adversarial Training Algorithm for DNN-based Many-to-one VC
Table B.6. Preference scores of converted speech speaker similarity. Bold values indicate that method is more preferred than other with p-value < 0.05. Here,“Prop. (DAT-GAN)” was compared with “Baseline” or “Prop. (GAN)”(a) Results of FSs-to-FT VC

Method A
Baseline
Prop. (GAN)

Score (A vs. B)
0.317 vs. 0.683
0.387 vs. 0.613

Method B
Prop. (DAT-GAN)
Prop. (DAT-GAN)(b) Results of MSs-to-FT VC

Method A
Baseline
Prop. (GAN)

Score (A vs. B)
0.283 vs. 0.717
0.373 vs. 0.627

Method B
Prop. (DAT-GAN)
Prop. (DAT-GAN)(c) Results of FSs-to-MT VC

Method A
Baseline
Prop. (GAN)

Score (A vs. B)
0.328 vs. 0.672
0.348 vs. 0.652

Method B
Prop. (DAT-GAN)
Prop. (DAT-GAN)(d) Results of MSs-to-MT VC

Method A
Baseline
Prop. (GAN)

Score (A vs. B)
0.308 vs. 0.692
0.276 vs. 0.724

Method B
Prop. (DAT-GAN)
Prop. (DAT-GAN)

Evaluation of Speaker Similarity“Prop. (DAT-GAN)” was compared with “Baseline” or “Prop. (GAN)” in terms ofthe converted speech speaker similarity. Preference XAB tests on the similarity wereconducted. The target speaker’s three speech utterances excluded from the training data
were used as reference “X” to evaluate the similarity. The converted speech pairs ofthe method “A” (“Baseline” or “Prop. (GAN)”) and the method “B” (“Prop. (DATGAN)”) were presented to listeners in random order. In the evaluation of FSs-to-FT
and MSs-to-FT VC, thirty listeners participated in the assessment by using crowdsourced
subjective evaluation systems. Each listener evaluated 40 converted speech samples (20source speakers × the two comparisons). Similarly, the evaluation of FSs-to-MT andMSs-to-MT VC was conducted with 25 listeners.
Table B.6 shows the evaluation results. “Prop. (DAT-GAN)” achieves signiﬁcantly
higher preference scores than not only “Baseline” but also “Prop. (GAN)” in the all
VC settings. These results demonstrate the algorithm’s eﬀectiveness in improving theconverted speech speaker similarity, as well as in the naturalness.

Appendix B Joint Adversarial Training Algorithm for DNN-based Many-to-one VC

173

Table B.7. Preference scores of converted speech naturalness. Bold values indicate thatmethod is more preferred than other with p-value < 0.05. Here, “Baseline”
was compared with “Prop. (Joint)” or “Prop. (DAT)”(a) Results of FSs-to-FT VC

Method A
Baseline
Baseline

Score (A vs. B)
0.468 vs. 0.532
0.448 vs. 0.552

Method B
Prop. (Joint)
Prop. (DAT)(b) Results of MSs-to-FT VC

Method A
Baseline
Baseline

Score (A vs. B)
0.512 vs. 0.488
0.492 vs. 0.508

Method B
Prop. (Joint)
Prop. (DAT)(c) Results of FSs-to-MT VC

Method A
Baseline
Baseline

Score (A vs. B)
0.420 vs.0.580
0.408 vs 0.592

Method B
Prop. (Joint)
Prop. (DAT)(d) Results of MSs-to-MT VC

Method A
Baseline
Baseline

Score (A vs. B)
0.388 vs. 0.612
0.400 vs. 0.600

Method B
Prop. (Joint)
Prop. (DAT)Eﬀects of Joint Training and DAT without GANs
The eﬀects of the other proposed algorithms, i.e., “Prop. (Joint)” and “Prop. (DAT),”
were investigated. As shown in Table B.2, these algorithms were unable to reduce LogGVDs to the extent that GAN-based proposed algorithms were able to do. However, otherobjective evaluation results revealed that there were clear diﬀerences between “Baseline”
and the two proposed algorithms. Therefore, this section compared “Baseline” with “Prop.
(Joint)” or “Prop. (DAT)” by preference AB tests on the naturalness and preference XAB
tests on the speaker similarity. The comparison should clarify the eﬀects caused by thejoint training or the DAT-based training. The converted speech pairs of the method “A”(“Baseline”) and the method “B” (“Prop. (Joint)” or “Prop. (DAT)”) were presented tolisteners in random order. Twenty-ﬁve listeners participated in the evaluations. Each listener evaluated 40 converted speech samples (20 source speakers × the two comparisons).
Tables B.7 and B.8 show the evaluation results on the naturalness and speaker similarity,
respectively. From the results, the preference scores of the two proposed algorithms arecomparable or superior to those of “Baseline.” In particular, two remarkable points are174

Appendix B Joint Adversarial Training Algorithm for DNN-based Many-to-one VC
Table B.8. Preference scores of converted speech speaker similarity. Bold values indicate that method is more preferred than other with p-value < 0.05. Here,“Baseline” was compared with “Prop. (Joint)” or “Prop. (DAT)”(a) Results of FSs-to-FT VC

Method A
Baseline
Baseline

Score (A vs. B)
0.500 vs. 0.500
0.484 vs. 0.516

Method B
Prop. (Joint)
Prop. (DAT)(b) Results of MSs-to-FT VC

Method A
Baseline
Baseline

Score (A vs. B)
0.456 vs. 0.544
0.432 vs. 0.568

Method B
Prop. (Joint)
Prop. (DAT)(c) Results of FSs-to-MT VC

Method A
Baseline
Baseline

Score (A vs. B)
0.424 vs. 0.576
0.404 vs. 0.596

Method B
Prop. (Joint)
Prop. (DAT)(d) Results of MSs-to-MT VC

Method A
Baseline
Baseline

Score (A vs. B)
0.408 vs. 0.592
0.396 vs. 0.604

Method B
Prop. (Joint)
Prop. (DAT)

observed: 1) “Prop. (DAT)” improves the speaker similarity in the inter-gender VC
settings (i.e., MSs-to-FT and FSs-to-MT) and 2) the two proposed algorithms outperform“Baseline” in FSs-/MSs-to-MT VC regarding both the naturalness and speaker similarity
in spite of decreasing the speech recognition accuracy shown in Table B.3. These results
suggest that 1) without using the GANs, the DAT has the eﬀect of improving the speakersimilarity by learning speaker-invariant features in the recognition model, and 2) the highspeech recognition accuracy does not guarantee the quality of converted speech and the
proposed joint training can optimize the recognition model for the many-to-one VC.
Comparison with Another Non-parallel VC Method
The best proposed algorithm, i.e., “Prop. (DAT-GAN),” was compared with anotherstate-of-the-art non-parallel VC method. Here, StarGAN-VC [109] was used as the competitive VC method that can achieve high-quality many-to-many VC without requiring
parallel speech corpora. The StarGAN-VC model was trained using an open-source imple-

Appendix B Joint Adversarial Training Algorithm for DNN-based Many-to-one VC

175

Table B.9. Preference scores of converted speech naturalness. Bold values indicate thatmethod is more preferred than other with p-value < 0.05. Here, “StarGANVC” was compared with “Prop. (DAT-GAN)”(a) Results of FSs-to-FT VC

Method A
StarGAN-VC

Score (A vs. B)
0.168 vs. 0.832

Method B
Prop. (DAT-GAN)(b) Results of MSs-to-FT VC

Method A
StarGAN-VC

Score (A vs. B)
0.152 vs. 0.848

Method B
Prop. (DAT-GAN)(c) Results of FSs-to-MT VC

Method A
StarGAN-VC

Score (A vs. B)
0.300 vs. 0.700

Method B
Prop. (DAT-GAN)(d) Results of MSs-to-MT VC

Method A
StarGAN-VC

Score (A vs. B)
0.400 vs. 0.600

Method B
Prop. (DAT-GAN)

mentation*1 and 100 utterances of 22 speakers (MSs, FSs, MT, and FT) for the training.
Similar to Section B.4.3, preference (X)AB tests were conducted with 25 listeners.
Tables B.9 and B.10 show the evaluation results on the naturalness and speaker similarity, respectively. The results demonstrate that “Prop. (DAT-GAN)” outperforms not only
the conventional many-to-one VC training algorithm but also state-of-the-art non-parallel
VC using the StarGAN.

*1 https://github.com/hujinsen/pytorch-StarGAN-VC

176

Appendix B Joint Adversarial Training Algorithm for DNN-based Many-to-one VC

Table B.10. Preference scores of converted speech speaker similarity. Bold values indicate that method is more preferred than other with p-value < 0.05. Here,“StarGAN-VC” was compared with “Prop. (DAT-GAN)”(a) Results of FSs-to-FT VC

Method A
StarGAN-VC

Score (A vs. B)
0.140 vs. 0.860

Method B
Prop. (DAT-GAN)(b) Results of MSs-to-FT VC

Method A
StarGAN-VC

Score (A vs. B)
0.096 vs. 0.904

Method B
Prop. (DAT-GAN)(c) Results of FSs-to-MT VC

Method A
StarGAN-VC

Score (A vs. B)
0.132 vs. 0.868

Method B
Prop. (DAT-GAN)(d) Results of MSs-to-MT VC

Method A
StarGAN-VC

Score (A vs. B)
0.176 vs. 0.824

Method B
Prop. (DAT-GAN)

177

Appendix C

Speech Recognition andSpeaker Classiﬁcation
PerformancesThis appendix describes the speech recognition and speaker classiﬁcation performances
in VAE-based multi-speaker statistical speech synthesis (Chapter 5). Several settings of
two hyperparameters, the number of seen speakers and d-vector dimensionality, were considered. Two performances were measured: the frame-wise phoneme error rate (PER) of
speech recognition and equal error rate (EER) of speaker veriﬁcation using d-vectors.
For measuring the EER, 50 utterances of each of the six speakers (three males andthree females) were used for enrollment (i.e., d-vector extraction) and other 25 utterances were used for evaluation (i.e., computing the cosine distance between input andclaimed speaker’s d-vectors). The L2 normalization was applied to the d-vectors [27].
Table C.1 shows the results. The PER signiﬁcantly increases under the setting of thesmallest number of seen speakers (i.e., “50spk”). The results also reveal that the EER
drastically increases under the settings of the smaller d-vector dimensionality (i.e., “1d,”“2d,” and “4d”).

178

Appendix C Speech Recognition and Speaker Classiﬁcation Performances

Table C.1. Frame-wise PER of speech recognition DNNs and EER of speaker veriﬁcation using d-vectors. Diﬀerent settings of number of seen speakers and dvector dimensionality were considered. Note that, EER for one-dimensional
d-vectors cannot be computed since d-vector in this evaluation always takes
positive value and is normalized so that its L2 norm becomes 1

PER [%]
50spk
130spk260spk

54.2
48.6
49.2

1d
N/A
N/A
N/A

2d
29.5
29.1
27.4

EER [%]
4d
8d
10.4 6.96
13.1 7.81
10.3 6.22

16d
2.97
1.60
4.07

32d
2.48
1.33
0.78