DOCTORAL DISSERTATIONSIMULTANEOUS MODELING OF PHONETIC
AND PROSODIC PARAMETERS, AND
CHARACTERISTIC CONVERSION FOR
HMM-BASED TEXT-TO-SPEECH SYSTEMSDOCTOR OF ENGINEERINGJANUARY 2002
TAKAYOSHI YOSHIMURA

Supervisors : Professor Tadashi Kitamura
Associate Professor Keiichi Tokuda

Department of Electrical and Computer Engineering
Nagoya Institute of Technology

Acknowledgement
Firstly, I would like to express my sincere gratitude to Professor Tadashi Kitamura
and Associate Professor Keiichi Tokuda, Nagoya Institute of Technology, my thesis
advisor, for their support, encouragement, and guidance. Also, I would like toexpress my gratitude to Professor Takao Kobayashi and Research Associate Takashi
Masuko, Tokyo Institute of Technology, for their kind suggestion.
I would also like to thank all the members of Kitamura Laboratory and for theirtechnical support, encouragement. If somebody was missed among them, my work
would not be completed.
I would be remiss if I would fail to thank Rie Matae, the secretary to Kitamura
Laboratory and Fumiko Fujii, the secretary to the Department of Computer Science,for their kind assistance.
Finally, I would sincerely like to thank my family for their encouragement.

iContents
AcknowledgementiContents

ii

List of Tablesvi

List of Figuresvii

1 Introduction

4

1.1

Text-to-speech(TTS) system . . . . . . . . . . . . . . . . . . . . . . .

4

1.2

Proposition of this thesis . . . . . . . . . . . . . . . . . . . . . . . . .

5

1.3

Original contributions . . . . . . . . . . . . . . . . . . . . . . . . . .

7

2 Mel-cepstral Analysis and Synthesis Technique8

2.1

Source-ﬁlter model . . . . . . . . . . . . . . . . . . . . . . . . . . . .

8

2.2

Mel-cepstral analysis . . . . . . . . . . . . . . . . . . . . . . . . . . .

9

2.3

Synthesis ﬁlter. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10

3 Speech Parameter Modeling Based on HMM
3.1

3.2

14

Spectral parameter modeling . . . . . . . . . . . . . . . . . . . . . . . 14
3.1.1

Continuous density HMM . . . . . . . . . . . . . . . . . . . . 14

3.1.2

Probability calculation . . . . . . . . . . . . . . . . . . . . . . 16

3.1.3

Parameter estimation of continuous density HMM. . . . . . 18

F0 parameter modeling . . . . . . . . . . . . . . . . . . . . . . . . . . 21
3.2.1

Multi-Space Probability Distribution . . . . . . . . . . . . . . 22ii

3.2.2Multi-space distribution HMM. . . . . . . . . . . . . . . . . 23

3.2.3

Reestimation algorithm for MSD-HMM training . . . . . . . . 25

3.2.4

Application to F0 pattern modeling . . . . . . . . . . . . . . . 27

4 Speech parameter generation from HMM

29

4.1 Speech parameter generation based on maximum likelihood criterion . 29
4.1.1

Case 1: Maximizing P (O|Q, λ) with respect to O . . . . . . . 30

4.1.2

Case 2: Maximizing P (O, Q|λ) with respect to O and Q . . . 31

4.1.3

Case 3: Maximizing P (O|λ) with respect to O . . . . . . . . . 32

4.2 Example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
4.2.1

Eﬀect of dynamic feature . . . . . . . . . . . . . . . . . . . . . 34

4.2.2

Parameter generation using multi-mixuture HMM . . . . . . . 36

5 Construction of HMM-based Text-to-Speech System

39

5.1 Calculation of dynamic feature . . . . . . . . . . . . . . . . . . . . . . 39
5.2 Spectrum and F0 modeling . . . . . . . . . . . . . . . . . . . . . . . . 40
5.3 Duration modeling . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42
5.3.1

Overview. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42

5.3.2

Training of state duration models . . . . . . . . . . . . . . . . 42

5.4 Context dependent model . . . . . . . . . . . . . . . . . . . . . . . . 43
5.4.1

Contextual factors . . . . . . . . . . . . . . . . . . . . . . . . 43

5.4.2

Decision-tree based context clustering . . . . . . . . . . . . . . 44

5.4.3

Context clustering using MDL principle . . . . . . . . . . . . 47

6 HMM-based Text-to-Speech Synthesis49

6.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49
6.2 Text analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49
6.3 Duration determination . . . . . . . . . . . . . . . . . . . . . . . . . . 51
6.4 Speech parameter generation . . . . . . . . . . . . . . . . . . . . . . . 51
6.5 Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52
6.5.1

Eﬀect of dynamic feature . . . . . . . . . . . . . . . . . . . . . 52

6.5.2

Automatically system training . . . . . . . . . . . . . . . . . . 55
iii

6.5.3

Speaking rate . . . . . . . . . . . . . . . . . . . . . . . . . . . 56

7 Improvement of Synthesized Speech Quality
7.1

58

Introduction of Mixed Excitation Model . . . . . . . . . . . . . . . . 58
7.1.1

Mixed excitation . . . . . . . . . . . . . . . . . . . . . . . . . 59

7.1.2

Excitation parameter modeling . . . . . . . . . . . . . . . . . 62

7.1.3

Excitation parameter generation . . . . . . . . . . . . . . . . . 64

7.2

Incorporation of postﬁlter . . . . . . . . . . . . . . . . . . . . . . . . 64

7.3

Experiments. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65

7.3.1

Excitation generation . . . . . . . . . . . . . . . . . . . . . . . 65

7.3.2

Eﬀect of mixed excitation . . . . . . . . . . . . . . . . . . . . 65

7.3.3

Eﬀect of postﬁltering . . . . . . . . . . . . . . . . . . . . . . . 67

8 Voice Conversion Technique: Speaker Interpolation

69

8.1

Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69

8.2

Speaker Interpolation . . . . . . . . . . . . . . . . . . . . . . . . . . . 71

8.3

Simulation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73

8.4

Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76

8.5

8.4.1

Generated Spectra . . . . . . . . . . . . . . . . . . . . . . . . 77

8.4.2

Experiment of Similarity . . . . . . . . . . . . . . . . . . . . . 77

Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 78

9 Conclusions

80

9.1

Original contribution revisited . . . . . . . . . . . . . . . . . . . . . . 80

9.2

Future works . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82

List of Publications

87

Journal Papers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 87
International Conference Proceedings . . . . . . . . . . . . . . . . . . . . . 87
Technical Reports . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89
Domestic Conference Proceedings . . . . . . . . . . . . . . . . . . . . . . . 89
Appendix A

Examples of decision trees constructed by using theiv

MDL principle.

92v

List of Tables
2.1

Examples of α for approximating auditory frequency scales. . . . . . . 10

2.2

Coeﬃcients of R4 (ω). . . . . . . . . . . . . . . . . . . . . . . . . . . . 11

6.1

Number of distribution.

8.1

Setting for simulating Japanese vowel “a” . . . . . . . . . . . . . . . 73

. . . . . . . . . . . . . . . . . . . . . . . . . 56

vi

List of Figures
1.1 The scheme of the HMM-based TTS system. . . . . . . . . . . . . . .

6

2.1 Source-ﬁlter model. . . . . . . . . . . . . . . . . . . . . . . . . . . . .

9

2.2 Implementation of Synthesis ﬁlter D(z). . . . . . . . . . . . . . . . . 13
3.1 Representation of a speech utterance using a ﬁve-state HMM. . . . . 15
3.2 Implementation of the computation using forward-backward algorithm in terms of a trellis of observation t and state i. . . . . . . . . . 18
3.3 Example of F0 pattern. . . . . . . . . . . . . . . . . . . . . . . . . . . 21
3.4 Multi-space probability distribution and observations. . . . . . . . . . 23
3.5 An HMM based on multi-space probability distribution. . . . . . . . . 24
4.1 Spectra generated with dynamic features for a Japanese phrase “chiisanaunagi”. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34
4.2 Relation between probability density function and generated parameter for a Japanese phrase “unagi” (top: static, middle: delta, bottom: delta-delta). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35
4.3 Generated spectra for a sentence fragment “kiNzokuhiroo.” . . . . . . 37
4.4 Spectra obtained from 1-mixture HMMs and 8-mixture HMMs. . . . 37
4.5 The result of the pair comparison test. . . . . . . . . . . . . . . . . . 38
5.1 Calculation of dynamic features for F0. . . . . . . . . . . . . . . . . . 40
5.2 Feature vector. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41
5.3 structure of HMM. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41
vii

5.4

Decision trees. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45

5.5

Examples of decision trees. . . . . . . . . . . . . . . . . . . . . . . . . 46

6.1

The block diagram of the HMM-based TTS. . . . . . . . . . . . . . . 50

6.2

Generated spectra for a phrase “heikiNbairitsu” (top: natural spectra, bottom: generated spectra). . . . . . . . . . . . . . . . . . . . . . 52

6.3

Generated F0 pattern for a sentence “heikiNbairitsuwo sageta keisekigaaru” (top: natural F0 pattern, bottom: generated F0 pattern). . . . . 53

6.4

Generated spectra for a phrase “heikiNbairitsu.” . . . . . . . . . . . . 53

6.5

Generated F0 pattern for a sentence “heikiNbairitsuwo sageta keisekigaaru.” . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54

6.6

Eﬀect of dynamic feature. . . . . . . . . . . . . . . . . . . . . . . . . 55

6.7

Eﬀect of diﬀerence of initial model. . . . . . . . . . . . . . . . . . . . 56

6.8

Generated spectra for an utterance “/t-o-k-a-i-d-e-w-a/” with diﬀerent speaking rates (top : ρ = −0.1, middle : ρ = 0, bottom : ρ = 0.1). 57

7.1

Traditional excitation model. . . . . . . . . . . . . . . . . . . . . . . . 59

7.2

Multi-band mixing model. . . . . . . . . . . . . . . . . . . . . . . . . 60

7.3

Mixed excitation model. . . . . . . . . . . . . . . . . . . . . . . . . . 61

7.4

Structure of a feature vector modeled by HMM. . . . . . . . . . . . . 62

7.5

Structure of a HMM. . . . . . . . . . . . . . . . . . . . . . . . . . . . 63

7.6

Eﬀect of postﬁltering(dots: before postﬁltering, solid: after postﬁltering β = 0.5). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65

7.7

Example of generated excitation for phrase “sukoshizutsu.” (top: traditional excitation，bottom: mixed excitation) . . . . . . . . . . . . . 66

7.8

Comparison of traditional and mixed excitation models. . . . . . . . . 67

7.9

Eﬀect of mixed excitation and postﬁltering. . . . . . . . . . . . . . . 68

8.1

Block diagram of speech synthesis system with speaker interpolaiton.

8.2

A space of speaker individuality modeled by HMMs. . . . . . . . . . . 72
viii

70

8.3 Comparison between method (a), (b) and (c) with regard to interpolation between two multi-dimensional Gaussian distributions. . . . . . 74
8.4 Comparison between method (a), (b) and (c) with regard to interpolation between two Gaussian distributions p1 and p2 with interpolation ratios A: (a1 , a2 ) = (1, 0), B: (a1 , a2 ) = (0.75, 0.25), C:(a1 , a2 ) = (0.5, 0.5), D: (a1 , a2 ) = (0.25, 0.75), E: (a1 , a2 ) = (0, 1). . . 75
8.5 Generated spectra of the sentence “/n-i-m-o-ts-u/”. . . . . . . . . . . 77
8.6 Subjective distance between samples. . . . . . . . . . . . . . . . . . . 78
A.1 Examples of decision trees for mel-cepstrum. . . . . . . . . . . . . . . 93
A.2 Examples of decision trees for F0. . . . . . . . . . . . . . . . . . . . . 94
A.3 Examples of decision trees for bandpass voicing strength. . . . . . . . 95
A.4 Examples of decision trees for Fourier magnitude. . . . . . . . . . . . 96
A.5 Examples of decision trees for duration. . . . . . . . . . . . . . . . . . 97ix

x

Abstract
A text-to-speech(TTS) system is one of the human-machine interfaces using speech.
In recent years, TTS system is developed as an output device of human-machineinterfaces, and it is used in many application such as a car navigation system, information retrieval over the telephone, voice mail, a speech-to-speech translation
system and so on. However, although most text-to-speech systems still cannot synthesize speech with various voice characteristics such as speaker individualities andemotions. To obtain various voice characteristics in text-to-speech systems based on
the selection and concatenation of acoustical units, a large amount of speech data isnecessary. However, it is diﬃcult to collect, segment, and store it. From these pointsof view, in order to construct a speech synthesis system which can generate various
voice characteristics, an HMM-based text-to-speech system has been proposed. Thisdissertation presents the construction of the HMM-based text-to-speech system, inwhich spectrum, fundamental frequency and duration are modeled simultaneously
in a uniﬁed framework of HMM.
In the system, mainly three techniques are used; (1) a mel-cepstral analysis/synthesis
technique, (2) speech parameter modeling using HMM and (3) a speech parameter
generation algorithm from HMM. Since the system uses above three techniques,
the system has several capabilities. First, since the TTS system uses the speechparameter generation algorithm, the generated spectral and pitch paramters fromthe trained HMMs can be similar to those of real speech. Second, by transforming
HMM parameters appropriately, voice characteristics of synthetic speech can bechanged since the system generates speech from the HMMs. Third, this system
is trainable. In this thesis, ﬁrst, the above three techniques are presented, andsimultaneous modeling of phonetic and prosodic parameters in a framework of HMM
is proposed.
Next, to improve of the quality of synthesized speech, the mixed excitation model ofthe speech coder MELP and postﬁlter are incorporated into the system. Experimental results show that the mixed excitation model and postﬁlter signiﬁcantly improve
the quality of synthesized speech.
Finally, for the purpose of synthesizing speech with various voice characteristics
1

such as speaker individualities and emotions, the TTS system based on speaker
interpolation is presented.

2

Abstract in Japanese
テキスト音声合成 (TTS) システムは，音声を使ったマンマシンインターフェースの出力モジュールとして，近年，カーナビゲーションシステム，電話を利用した情報
検索，音声翻訳システムなど様々なアプリケーションで用いられている．これまでに数多くのテキスト音声合成システムが提案され，音質的に向上している．しかし，まだ様々な話者の声質で話したり，嬉しそうに，怒ったように，悲しそうになど様々
な発話スタイルで話すことができるものは少ない．その理由として，多くのテキス
ト音声合成システムが波形の素片を接続して音声を合成する方式を採用しているということがあげられる．波形接続型のシステムでは，音声を合成するために音声データを収集し，素片に分割し，分割した素片を格納しなければならない．波形接続型のシステムで様々な声質，発話スタイルを実現するためには，様々な声質，発話ス
タイルで収録された膨大な量の音声データを処理しなければならず，また合成するには膨大な量の素片を格納する記憶媒体が必要となるため，実現は非常に困難である．このような観点から，本論文では様々な話者，発話スタイルの音声を合成する
ことができる音声合成システムの実現を目的とし，HMM に基づく音声合成システ
ムについて述べる．

HMM に基づく音声合成システムは，(1) メルケプストラム分析合成系，(2)HMM を使った音声パラメータモデリング，(3)HMM からの音声パラメータ生成アルゴリズ
ムの 3 つの技術から成っている．本システムは，これら 3 つの技術を用いているためいくつかの能力を持つ．一つ目に，HMM からの音声パラメータ生成アルゴリズム
により，学習した HMM から実音声に近いスペクトルパラメータ，ピッチパラメー
タを出力することができる．二つ目に，HMM のパラメータを適切に変換すること
により様々な声質の音声を合成することができる．三つ目に，本システムは自動的に学習することができる．本論文では，上記の (1),(2),(3) の技術について述べ，音声の音韻・韻律情報を HMM の枠組で統一的にモデル化する手法を提案する．
次に，本論文では合成音声の品質を向上させるため，音声符号化手法 MELP で用いられている混合励振源モデルとポストフィルタを本システムに導入した．混合励振源モデルとポストフィルタが合成音声の品質を大幅に向上させることを実験結果で示す．
最後に，多様な話者性，発話スタイルで音声を合成するシステムの実現を目的として，話者補間手法を用いた声質変換手法を提案する．

3

Chapter 1
Introduction
1.1

Text-to-speech(TTS) system

Speech is the natural form of human communication, and it can enhance humanmachine communication. A text-to-speech(TTS) system is one of the human-machineinterface using speech. In recent years, TTS system is developed an output deviceof human-machine interface, and it is used in many application such as a car navigation system, information retrieval over the telephone, voice mail, a speech-to-speechtranslation system and so on. The goal of TTS system is to synthesize speech withnatural human voice characteristics and, furthermore, with various speaker individualities and emotions (e.g., anger, sadness, joy).
The increasing availability of large speech databases makes it possible to construct
TTS systems, which are referred to as data-driven or corpus-based approach, byapplying statistical learning algorithms. These systems, which can be automaticallytrained, can generate natural and high quality synthetic speech and can reproduce
voice characteristics of the original speaker.
For constructing such a system, the use of hidden Markov models (HMMs) has arisen
largely. HMMs have successfully been applied to modeling the sequence of speech
spectra in speech recognition systems, and the performance of HMM-based speech
recognition systems have been improved by techniques which utilize the ﬂexibility of HMMs: context-dependent modeling, dynamic feature parameters, mixturesof Gaussian densities, tying mechanism, speaker and environment adaptation techniques. HMM-based approaches to speech synthesis can be categorized as follows:
1. Transcription and segmentation of speech database [1].
2. Construction of inventory of speech segments [2]–[5].
43. Run-time selection of multiple instances of speech segments [4], [6].
4. Speech synthesis from HMMs themselves [7]–[10].
In approaches 1–3, by using a waveform concatenation algorithm, e.g., PSOLA algorithm, a high quality synthetic speech could be produced. However, to obtainvarious voice characteristics, large amounts of speech data are necessary, and it isdiﬃcult to collect, segment, and store the speech data. On the other hand, in approach 4, voice characteristics of synthetic speech can be changed by transforming
HMM parameters appropriately. From this point of view, parameter generation
algorithms [11], [12] for HMM-based speech synthesis have been proposed, and aspeech synthesis system [9], [10] has been constructed using these algorithms. Actually, it has been shown that voice characteristics of synthetic speech can be changed
by applying a speaker adaptation technique [13], [14] or a speaker interpolation
technique [15]. The main feature of the system is the use of dynamic feature: by
inclusion of dynamic coeﬃcients in the feature vector, the dynamic coeﬃcients ofthe speech parameter sequence generated in synthesis are constrained to be realistic,
as deﬁned by the parameters of the HMMs.

1.2

Proposition of this thesis

The proposed TTS system is shown in Fig. 1.1. This ﬁgure shows the training andsynthesis parts of the HMM-based TTS system. In the training phase, ﬁrst, spectral
parameters (e.g., cepstral coeﬃcients) and excitation parameters (e.g., fundamental frequency) are extracted from speech database. The extracted parameters aremodeled by context-dependent HMMs. In the systhesis phase, a context-dependentlabel sequence is obtained from a input text by text analysis. A sentence HMM isconstructed by concatenating context dependent HMMs according to the contextdependent label sequence. By using the parameter generation algorithm, spectral
and excitation parameters are generated from the sentence HMM. Finaly, by using
a synthesis ﬁlter, speech is synthesized from the generated spectral and excitation
parameters.
In this thesis, it is roughly assumed that spectral and excitation parameters include
phonetic and prosodic information, respectively. If these phonetic and prosodic parameters are modeled in a uniﬁed framework of HMMs, it is possible to apply speaker
adaptation / interpolation techniques to phonetic and prosodic information, simultaneously, and synthesize speech with various voice characteristics such as speakerindividualities and emotions. From the point of view, the phonetic and prosodic
parameter modeling technique and the voice conversion technique are proposed inthis thesis.
5

SPEECHDATABASE

Speech signal

Excitation
parameter
extraction

Spectralparameter
extraction

Excitation parameter

Spectral parameter
Training of HMM

Label

Training part

Synthesis partContext dependent
HMMs

TEXT
Text analysis
Label

Parameter generationfrom HMM

Excitation parameter

Spectral parameter

Excitation
generation

Synthesis
filterSYNTHESIZEDSPEECH

Figure 1.1: The scheme of the HMM-based TTS system.
The remainder of this thesis is organized as follows:
In Chapters 2–4, the following fundamental techniques of the HMM-based speechsynthesis are described:• Mel-cepstral analysis and synthesis technique (Chapter 2)• Speech parameter modeling based on HMM (Chapter 3)• Speech parameter generation from HMM (Chapter 4)
Chapter 5 presents construction of the proposed HMM-based TTS system in which
spectrum, fundamental frequency (F0), duration are modeled by HMM simultaneously, and Chapter 6 discribes how to synthesize speech. In Chapter 7, mixed
excitation model is incorporated into the proposed TTS system in order to improve
6

quality of synthesized speech. Chapter 8 presents how to synthesize speech withvarious voice characteristics, appling speaker interpolation technique to HMM-based
TTS. Finally Chapter 9 draws overall conclusions and describes possible future
works.

1.3

Original contributions

This thesis describes new approaches to synthesize speech with natural human voice
characteristics and with various voice characteristics such as speaker individuality
and emotion. The major original contributions are as follows:• Speech parameter generation using multi-mixture HMM.
• Duration modeling for the HMM-based TTS system.
• Simultaneous modeling of spectrum, F0 and duration.
• Training of context dependent HMM using MDL principle.
• F0 parameter generation using dynamic features.
• Automatic training of the HMM-based TTS system.
• Improvement of the quality of the synthesized speech by incorporating themixed excitation model and postﬁlter into the HMM-based TTS system.
• Voice conversion using a speaker interpolation technique.

7Chapter 2
Mel-cepstral Analysis andSynthesis Technique
The proposed TTS system is based on source-ﬁlter model. In order to construct
the system, ﬁrst, it is necessary to extract feature parameters, which describe thevocal tract, from speech database for training. For all the work in this thesis, themel-cepstral analysis [16] is used for spectral estimation. This chapter describes
the mel-cepstral analysis, how feature parameters, i.e., mel-cepstral coeﬃcients, areextracted speech signals and how speech is synthesized from the mel-cepstral coefﬁcients.

2.1

Source-ﬁlter model

To treat a speech waveform mathematically, source-ﬁlter model is generally used
to represent sampled speech signals, as shown in 2.1. The transfer function H(z)models the structure of vocal tract. The excitation source is chosen by a switch
which controls the voiced/unvoiced character of the speech. The excitation signal
is modeled as either a periodic pulse train for voiced speech, or a random noise
sequence for unvoiced speech. To produce speech signal x(n), the parameters of themodel must change with time. The excitation signal e(n) is ﬁltered by a time-varying
linear system H(z) to generate speech signals x(n).
The speech x(n) can be computed from the excitation e(n) and the impulse responseh(n) of the vocal tract using the convolution sum expression
x(n) = h(n) ∗ e(n)(2.1)

where the symbol ∗ stands for discrete convolution. The details of digital signal
processing and speech processing are given in Ref. [17]8Fundamental frequency

Vocal Tract
Parameter

Periodic pulse
train

Synthesis Filtere(n)White noiseh(n)Synthesized
speechx(n) = h(n) ∗ e(n)Figure 2.1: Source-ﬁlter model.

2.2

Mel-cepstral analysis

In mel-cepstral analysis[16], the model spectrum H(ejw ) is represented by the M -thorder mel-cepstral coeﬃcients c̃(m) as follows:
H(z) = exp

Mc̃(m)z̃ −m ,

(2.2)

m=0

where

z −1 − α, |a| < 1.
(2.3)
z̃ =
1 − αz −1
The phase characteristic of the all-pass transfer function z̃−1 = e−j ω̃ is given by−1ω̃ = tan−1(1 − α2 ) sin ω.
(1 + α2 ) cos ω − 2α(2.4)

For example for a sampling frequency of 16kHz, ω̃ is a good approximation to themel scale based on subjective pitch evaluations when α = 0.42(Tab. 2.1).
To obtain an unbiased estimate, we use the following criterion [18] and minimize itwith respect to c̃(m)M
m=0
E=

1
2π

 π
−π

exp R(ω) − R(ω) − 1dω,

(2.5)




(2.6)

where

2


R(ω) = log IN (ω) − log H(ejω ) ,and IN (ω) is the modiﬁed periodogram of a weakly stationary process x(n) with atime window of length N . To take the gain factor K outside from H(z), we rewrite
Eq.(2.2) as:
H(z) = exp

M

b(m)Φm (z) = K · D(z),

m=0

9

(2.7)

Table 2.1: Examples of α for approximating auditory frequency scales.
Sampling frequency
8kHz
10kHz
12kHz
16kHz

Mel scale
0.31
0.35
0.37
0.42

Bark scale
0.42
0.47
0.50
0.55

where
K = exp b(0),D(z) = exp

M


(2.8)

b(m)Φm (z),

(2.9)

m=0

and



b(m) =c(m)m=Mc(m) − αb(m + 1) 0 ≤ m < M 1Φm (z) =




2

−1

(2.10)

m=0(1 − α )zz̃ −(m−1) m ≥ 1−1
1 − αz(2.11)

Since H(z) is a minimum phase system, we can show that the minimization of Ewith respect to c̃(m)M
m=0 is equivalent to that of
ε=

1
2π

 π

IN (ω)dω,−π |D(ejω )|(2.12)

with respect tob = [b(1), b(2), · · · , b(M )]T .

(2.13)∂E
The gain factor K that minimizes E is obtained by setting ∂K
= 0:

K=

√

εmin

(2.14)

where εmin is the minimized value of ε.

2.3

Synthesis ﬁlter

The synthesis ﬁlter D(z) of Eq.(2.9) is not a rational function and, therefore, it cannot be realized directly. Howerver, using Mel Log Spectrum Approximation ﬁlter
10Table 2.2: Coeﬃcients of R4 (ω).
l
1
2
3
4

A4,l
4.999273 × 10−1
1.067005 × 10−1
1.170221 × 10−2
5.656279 × 10−4

(MLSA ﬁlter) [19], the synthesis ﬁlter D(z) can be approximated with suﬃcient accuracy and becomes minimum phase IIR system. The complex exponential function
exp ω is qpproximated by a rational function
RL (F (z))

exp ω

1+

L


AL,l ω l

l=1

=

.

(2.15)

exp(F (z)) = D(z)(2.16)

1+LAL,l (−ω)ll=1Thus D(z) is approximated as follows:
RL (F (z))
where F (z) is deﬁned by
F (z) =

M
b(m)Φm (z).

(2.17)

m=1

The ﬁlter structure of F (z) is shown in Fig. 2.2(a). Figure 2.2(b) shows the block
diagram of the MLSA ﬁlter RL (F (z)) for the case of L = 4.
When we use the coeﬃcients A4,l show in Tab. 2.2, R4 (F (z)) is stable and becomesa minimum phase system under the condition|F (ejω )| ≤ 6.2.

(2.18)

Further more, we can show that the approximation error | log D(ejω )−log R4 (F (ejω ))|does not exceed 0.24dB[20] under the condition|F (ejω )| ≤ 4.5.

(2.19)

F (z) = F1 (z) + F2 (z)(2.20)

When F (z) is expressed as

11the exponential transfer function is approximated in a cascade formD(z) = exp F (z)= exp F1 (z) · exp F2 (z)RL (F1 (z)) · RL (F2 (z))(2.21)as shown in Fig. 2.2(c). If
max |F1 (ejω )|, max |F2 (ejω )| < max |F (ejω )|,ωωω

(2.22)

it is expected that RL (F1 (ejω )) · RL (F2 (ejω )) approximates D(ejω ) more accurately
than RL (F (ejω )).
In the following experiments, we let
F1 (z) = b(1)Φ1 (z)F2 (z) =Mb(m)Φm (z).

(2.23)(2.24)

m=2

Since we empirically found thatmax |F1 (ejω )|, max |F2 (ejω )| < 4.5
ωω

(2.25)

for speech sounds, RL (F1 (z))·RL(F2 (z)) approximates the exponential transfer function D(z) with suﬃcient accuracy and becomes a stable system.

12

α

α

α

1 − α2

Input

Z −1

Z −1

Z −1

b(1)

Z −1

b(2)

b(3)
Output(a) Basic ﬁlter F (z)Output
Input

F (z)

F (z)

F (z)

F (z)

A4,1

A4,2

A4,3

A4,4

(b) RL (F (z))

Input

D(z),

R4 (F1 (z))

L=4R4 (F2 (z))

output(c) Two stage cascade structure
Figure 2.2: Implementation of Synthesis ﬁlter D(z).

13Chapter 3
Speech Parameter Modeling
Based on HMM
The most successful and widely used acoustic models in recent years have been
Hidden Markov Models (HMMs). Practically all major speech recognition systems
are generally implemented using HMM. This chapter describes how to model spectral
and excitation parameters in a framework of HMM.

3.1

Spectral parameter modeling

3.1.1

Continuous density HMM

In this thesis, a continuous density HMM is used for the vocal tract modeling in thesame way as speech recognition systems. The continuous density Markov model is a
ﬁnite state machine which makes one state transition at each time unit (i.e, frame).
First, a decision is made to which state to succeed (including the state itself). Thenan output vector is generated according to the probability density function (pdf) forthe current state. An HMM is a doubly stochastic random process, modeling statetransition probabilities between states and output probabilities at each state.
One way of interpreting HMMs is to view each state as a model of a segment ofspeech. Figure 3.1 shows an example of representation of a speech utterance using a
N -state left-to-right HMM where each state is modeled by a multi-mixture Gaussianmodel. Assume that this utterance(typically parameterized by speech analysis as
the D-dimensional observation vector ot ) is divided into N segments di which are
represented by the states Si . The transition probability aij deﬁnes the probabilityof moving from state i to state j and satisﬁes aii + aij = 1. Then, each state can be
14

a11

a22

a12

S1

S2

b1 (ot )Frequency (kHz)

a33

a23

a44

S3

b2 (ot )

a34

a55

a45

S4

b3 (ot )S5b4 (ot )HMM

b5 (ot )

0
2
4
6
8

Spectrum

d1

d2

d3

d4

d5

Figure 3.1: Representation of a speech utterance using a ﬁve-state HMM.
modeled by a M -mixtures Gaussian density function:bj (ot ) =
=

M

k=1
M

k=1

cjk N (ot , µjk , Σjk )cjk

1
exp − (ot − µjk )T Σ−1jk (ot − µjk ) ,2(2π) |Σjk |
1

1
2

D
2

(3.1)

where cjk , µjk and Σjk are the mixture coeﬃcient, D-dimensional mean vector andD × D covariance matrix(full covariance matrix) for the k-th mixture component
in the j-th state, respectively. This covariance can be restricted to the diagonal
elements (diagonal covariance matrix) when the elements of the feature vector are
assumed to be independent. |Σjk | is the determinant of Σjk , and Σ−1
jk is the inverseof Σjk . The mixture gains cjk satisfy the stochastic constraint
M


1≤j≤N

(3.2)

1 ≤ j ≤ N, 1 ≤ k ≤ M(3.3)

cjk = 1,k=1

cjk ≥ 0,15

so that the pdf is properly normalized, i.e., ∞−∞bj (o)do,

1≤j≤N

(3.4)

Since the pdf of Eq. (3.1) can be used to approximate, arbitrarily closely, any ﬁnite,
continuous density function, it can be applied to a wide range of problems and is
widely used for acoustic modeling.
For convenience, to indicate the complete parameter set of the model, we use thecompact notation
λ = (A, B, π),(3.5)
where A = {aij }, B = {bj (o)} and π = {πi }. πi is the initial state distribution ofstate i, and it have the propertyπ=0, i = 1
1, i = 1(3.6)

in the left-to-right model.

3.1.2

Probability calculation

For calculation of P (O|λ), which is the probability of the observation sequence O =(o1 , o2 , · · · , oT ) given the model λ, forward-backward algorithm is generally used.
If we calculate P (O|λ) directly without this algorithm, it requires on the order of
2T N 2 calculation. On the other hand, calculation using forward-backward algorithm
requires on the order of N 2 T calculations, and it is computationally feasible. In thefollowing part, forward-backward algorithm is described.

The forward algorithm
Consider the forward variable αt (i) deﬁned as
αt (i) = P (o1 , o2 , · · · , ot , qt = i|λ)(3.7)

that is, the probability of the partial observation sequence from 1 to t and state i
at time t, given the model λ. We can solve for αt (i) inductively, as follows:
1. Initialization
α1 (i) = πi bi (o1 ),16

1 ≤ ileqN.

(3.8)

2. Induction
αt+1 (j) =Nαt (i)aij bj (ot+1 ),i=1

3. Termination
P (O|λ) =N

1≤t≤T −11≤j≤NαT (i).

(3.9)(3.10)

i=1

The backward algorithm
In the same way as forward algorithm, consider the backward variable βt (i) deﬁned
asβt (i) = P (ot + 1, ot + 2, · · · , oT |qt = i, λ)(3.11)
that is, the probability of the partial observation sequence from t to T , given statei at time t and the model λ. We can solve for βt (i) inductively, as follows:
1. Initialization
βT (i) = 1,

1 ≤ i ≤ N.

(3.12)

t = T − 1, T − 2, · · · , 11≤i≤N(3.13)

2. Induction
βt (i) =Naij bj (ot+1 )βt+1 (j),i=1

3. Termination
P (O|λ) =N
i=1β1 (i).

(3.14)

The forward-backward probability calculation is based on the trellis structure shownin Fig. 3.2. In this ﬁgure, the x-axis and y-axis represent observation sequence andstates of Markov model, respectively. On the trellis, all the possible state sequence
will remerge into these N nodes no matter how long the observation sequence. In
the case of the forward algorithm, at times t = 1, we need to calculate values of
α1 (i), 1 ≤ i ≤ N . At times t = 2, 3, · · · , T , we need only calculate values of αt (j),1 ≤ j ≤ N , where each calculation involves only the N previous values of αt−1 (i)because each of the N grid points can be reached from only the N grid points at
the previous time slot. As the result, the forward-backward algrorithm can reduceorder of probability calculation.
17

3

State

Si

N

2

1

1

2

3

4

T

Observation sequence ot

Figure 3.2: Implementation of the computation using forward-backward algorithm
in terms of a trellis of observation t and state i.

3.1.3

Parameter estimation of continuous density HMM

It is diﬃcult to determine a method to adjust the model parameters (A, B, π) tosatisfy a certain optimization criterion. There is no known way to analytical solve forthe model parameter set that maximizes the probability of the observation sequencein a closed form. We can, however, choose λ = (A, B, π) such that its likelihood,
P (O|λ), is locally maximized using an iterative procedure such as the Baum-Welch
method(also known as the EM(expectation-maximization method))[21], [22].
To describe the procedure for reestimation of HMM parameters, ﬁrst, the probabilityof being in state i at time t, and state j at time t + 1, given the model and theobservation sequence, is deﬁned as follows:ξt (i, j) = P (qt = i, qt+1 = j|O, λ),(3.15)

From the deﬁnitions of the forward and backward variables, ξt (i, j) is writen in theformξt (i, j) ==P (qt = i, qt+1 = j|O, λ)P (O|λ)αt (i)aij bj (ot+1 βt+1 (j))
N Nαt (i)aij bj (ot+1 βt+1 (j))

i=1 j=118(3.16)

Using ξt (i, j), the probability of being in state i at time t, given the entire observationand the model, is represented by
γt (i) =Nξt (i, j).

(3.17)

j=1

Q-function
The reestimation formulas can be derived directly by maximizing Baum’s auxiliary
funciton
P (O, q|λ) log P (O, q|λ)(3.18)Q(λ , λ) =q

over λ. BecauseQ(λ , λ) ≥ Q(λ , λ) ⇒ P (O, q|λ) ≥ P (O, q|λ)(3.19)

We can maximize the function Q(λ , λ) over λ to improve λ in the sense of increasing
the likelihood P (O, q|λ).
Maximization of Q-function
For given observation sequence O and model λ , we derive parameters of λ which
maximize Q(λ , λ). P (O, q|λ) can be written as
T

P (O, q|λ) = πq0

aqt−1 qt bqt (ot )(3.20)

t=1

log P (O, q|λ) = log πq0 +Tlog aqt−1 qt +

T


t=1

log bqt (ot )(3.21)

t=1

Hence Q-function (3.18) can be written as
Q(λ , λ) = Qπ (λ , pi)
+
+
=
+

T

t=1
T

t=1
N


Qai (λ , ai )Qbi (λ , bi )P (O, q0 = i|λ ) log πi

i=1
N 
T


P (O, qt−1 = i, qt = j|λ ) log aij

j=1 t=1+

T

P (O, qt = i|λ ) log bi (ot )

t=1

19

(3.22)

whereπ = [π1 , π2 , · · · , πN ]ai = [ai1 , ai2 , · · · , aiN ] ,(3.23)(3.24)

and bi is the parameter vector that deﬁnes bi (·). The parameter set λ which maximizes (3.22), subject to the stochastic constraintsN

πj = 1,(3.25)j=1
N

aij = 1,

(3.26)

j=1
M

cjk = 1,

(3.27)bj (o)do = 1,

(3.28)

k=1

 ∞
−∞

can be derived asπi =α0 (i)β0 (i)N= γ0 (i)(3.29)αT (j)

j=1
T


aij =

T


αt−1 (i)aij bj (ot )βt (j)

t=1
T


αt−1 (i)βt−1 (i)ξt−1 (i, j)= t=1T

t=1(3.30)
γt−1 (i)t=1

The reestimation formulas for the coeﬃcients of the mixture density, i.e, cjk , µjk
and Σjk are of the form
T


cij =

γt (j, k)

t=1
M
T 


(3.31)
γt (j, k)

t=1 k=1Tµjk =γt (j, k) · ot
t=1T

(3.32)

γt (j, k)

t=1
T


Σjk =

γt (j, k) · (ot − µjk )(ot − µjk )

t=1
T

t=1

20

(3.33)
γt (j, k)

Frequency (Hz)voiced region (continuous value)
200
150
100
50
0

1

2

3

4

5

time (s)unvoiced region (discrete symbol)

Figure 3.3: Example of F0 pattern.
(3.34)
where γt (j, k) is the probability of being in state j at time t with the kth mixture
component accounting for ot , i.e., α (j)β (j)ttγt (j, k) =  Nαt (j)βt (j)  c N (o , µ , Σ ) 
t
jk
jk 
  jk
 M
.
 




(3.35)N (ot , µjk , Σjk )

m=1

j=1

3.2



F0 parameter modeling

The F0 pattern is composed of continuous values in the “voiced” region and adiscrete symbol in the “unvoiced” region (Fig.3.3). Therefore, it is diﬃcult to apply
the discrete or continuous HMMs to F0 pattern modeling. Several methods havebeen investigated [23] for handling the unvoiced region: (i) replacing each “unvoiced”
symbol by a random vector generated from a probability density function (pdf) witha large variance and then modeling the random vectors explicitly in the continuous
HMMs [24], (ii) modeling the “unvoiced” symbols explicitly in the continuous HMMs
by replacing “unvoiced” symbol by 0 and adding an extra pdf for the “unvoiced”
symbol to each mixture, (iii) assuming that F0 values is always exist but they cannot
observed in the unvoiced region and applying the EM algorithm [25]. In this section,
A kind of HMM for F0 pattern modeling, in which the state output probabilities
are deﬁned by multi-space probability distributions (MSDs), is described.
21

3.2.1

Multi-Space Probability Distribution

We consider a sample space Ω shown in Fig. 3.4, which consists of G spaces:
Ω=

G


Ωg

(3.36)

g=1

where Ωg is an ng -dimensional real space Rng , and speciﬁed by space index g. Each
space Ωg has its probability wg , i.e., P (Ωg ) = wg , where G
g=1 wg = 1. If ng > 0, eachspace has a probability density function Ng (x), x ∈ Rng , where Rng Ng (x)dx = 1.
We assume that Ωg contains only one sample point if ng = 0. Accordingly, letting
P (E) be the probability distribution, we have
P (Ω) =GP (Ωg ) =

g=1

G




wg

g=1

Rng

Ng (x)dx = 1.

(3.37)

It is noted that, although Ng (x) does not exist for ng = 0 since Ωg contains only
one sample point, for simplicity of notation, we deﬁne as Ng (x) ≡ 1 for ng = 0.
Each event E, which will be considered in this thesis, is represented by a randomvariable o which consists of a continuous random variable x ∈ Rn and a set of spaceindices X, that is,o = (x, X)(3.38)

where all spaces speciﬁed by X are n-dimensional. The observation probability ofo is deﬁned byb(o) =

wg Ng (V (o))(3.39)

S(o) = X.

(3.40)g∈S(o)

where
V (o) = x,Some examples of observations are shown in Fig. 3.4. An observation o1 consists ofthree-dimensional vector x1 ∈ R3 and a set of space indices X1 = {1, 2, G}. Thus
the random variable x is drawn from one of three spaces Ω1 , Ω2 , ΩG ∈ R3 , and its
probability density function is given by w1 N1 (x) + w2 N2 (x) + wG NG (x).
The probability distribution deﬁned in the above, which will be referred to as multispace probability distribution (MSD) in this thesis, is the same as the discrete distribution and the continuous distribution when ng ≡ 0 and ng ≡ m > 0, respectively.
Further, if S(o) ≡ {1, 2, . . . , G}, the continuous distribution is represented by aG-mixture probability density function. Thus multi-space probability distribution
is more general than either discrete or continuous distributions.
22

Sample Space Ω
Space
Index
1

Space

pdf of xΩ1 = R3

Observation
o1 = ( x1, {1, 2, G})

w1
N1 ( x )2

x1 ∈ R3Ω 2 = R3w2

o2 = ( x2 , {1, G})

N2 ( x )3

x2 ∈ R3Ω 3 = R5w3
N3 ( x )Go3 = ( x3 , {3})x3 ∈ R 5Ω G = R3wG
NG ( x )Figure 3.4: Multi-space probability distribution and observations.

3.2.2

Multi-space distribution HMM

The output probability in each state of MSD-HMM is given by the multi-spaceprobability distribution deﬁned in the previous section. An N -state MSD-HMM λis speciﬁed by initial state probability distribution π = {πj }N
j=1 , the state transition
probability distribution A = {aij }N
,
and
state
outputprobability
distributioni, j=1N
B = {bi (·)}i=1 , wherebi (o) =

wig Nig (V (o)),

i = 1, 2, . . . , N.

(3.41)g∈S(o)As shown in Fig. 3.5, each state i has G probability density functions Ni1 (·), Ni2 (·),
. . ., NiG (·), and their weights wi1 , wi2 , . . ., wiG .
23

1Ω1 = Rn1Ω 2 = Rn2

Ω G = RnG

2

w11

3

w21
N11 ( x )w31
N21 ( x )w12

N31 ( x )w22
N12 ( x )w32
N22 ( x )w1G

N32 ( x )w2 G

N1G ( x )w3G

N2 G ( x )N3 G ( x )Figure 3.5: An HMM based on multi-space probability distribution.
Observation probability of O = {o1 , o2 , . . . , oT } is written as
P (O|λ) =Taqt−1 qt bqt (ot )all q t=1=

T

all q,lt=1

aqt−1 qt wqt lt Nqt lt (V (ot ))(3.42)

where q = {q1 , q2 , . . . , qT } is a possible state sequence, l = {l1 , l2 , . . . , lT } ∈ {S(o1 )×
S(o2 )×. . .×S(oT )} is a sequence of space indices which is possible for the observation
sequence O, and aq0 j denotes πj .
The forward and backward variables:
αt (i) = P (o1 , o2 , . . . , ot , qt = i|λ)βt (i) = P (ot+1 , ot+2 , . . . , oT |qt = i, λ)(3.43)(3.44)

can be calculated with the forward-backward inductive procedure in a manner similar to the conventional HMMs. According to the deﬁnitions, (3.42) can be calculatedas
P (O|λ) =NαT (i) =

i=1

N

i=1

24

β1 (i).

(3.45)

The forward and backward variables are also used for calculating the reestimation
formulas derived in the the next section

3.2.3

Reestimation algorithm for MSD-HMM training

For a given observation sequence O and a particular choice of MSD-HMM, the objective in maximum likelihood estimation is to maximize the observation likelihood
P (O|λ) given by (3.42), over all parameters in λ. In a manner similar to [21], [22], we
derive reestimation formulas for the maximum likelihood estimation of MSD-HMM.

Q-function
An auxiliary function Q(λ , λ) of current parameters λ and new parameter λ isdeﬁned as follows:Q(λ , λ) =P (O, q, l|λ ) log P (O, q, l|λ)(3.46)

all q,lIn the following, we assume Nig (·) to be the Gaussian density with mean vector µig
and covariance matrix Σig .
Theorem 1Q(λ , λ) ≥ Q(λ , λ ) → P (O, λ) ≥ P (O, λ )Theorem 2 If, for each space Ωg , there are among V (o1 ), V (o2 ), . . ., V (oT ), ng +1
observations g ∈ S(ot ), any ng of which are linearly independent, Q(λ  , λ) has aunique global maximum as a function of λ, and this maximum is the one and only
critical point.

Theorem 3 A parameter set λ is a critical point of the likelihood P (O|λ) if andonly if it is a critical point of the Q-function.
We deﬁne the parameter reestimates to be those which maximize Q(λ , λ) as afunction of λ, λ being the latest estimates. Because of the above theorems, thesequence of resetimates obtained in this way produce a monotonic increase in thelikelihood unless λ is a critical point of the likelihood.
25

Maximization of Q-function
For given observation sequence O and model λ , we derive parameters of λ which
maximize Q(λ , λ). From (3.42), log P (O, q, l|λ) can be written aslog P (O, q, l|λ)
=

T 




log aqt−1 qt + log wqt lt + log Nqt lt (V (ot )) .

(3.47)

t=1

Hence Q-function (3.46) can be written as
Q(λ , λ) =NP (O, q1 = i|λ ) log πi

i=1
N
−1
 T

+

P (O, qt = i, qt+1 = j|λ ) log aij

i,j=1 t=1

+

N 
G




P (O, qt = i, lt = g|λ) log wig

i=1 g=1 t∈T (O,g)

+

N 
G




P (O, qt = i, lt = g|λ) log Nig (V (ot ))i=1 g=1 t∈T (O,g)(3.48)
where
T (O, g) = {t | g ∈ S(ot )}.

(3.49)The parameter set λ = (π, A, B) which maximizes (3.48), subject to the stochastic
NG
constraints N
i=1 πi = 1,
j=1 aij = 1 and
g=1 wg = 1, can be derived asπi =g∈S(o1 )γ1 (i, g)

T
−1

aij =

t=1
T
−1

ξt (i, j)(3.51)



t=1 g∈S(ot )



wig =

(3.50)γt (i, g)γt (i, g)t∈T (O,g)G(3.52)

γt (i, h)h=1 t∈T (O,h)



µig =

γt (i, g) V (ot )t∈T (O,g)γt (i, g)t∈T (O,g)26,

ng > 0(3.53)Σig =γt (i, g)(V (ot ) − µig )(V (ot ) − µig )Tt∈T (O,g),γt (i, g)t∈T (O,g)ng > 0(3.54)

where γt (i, h) and ξt (i, j) can be calculated by using the forward variable αt (i) andbackward variable βt (i) as follows:
γt (i, h) = P (qt = i, lt = h|O, λ)wih Nih (V (ot ))αt (i)βt (i)
· 
= N

wig Nig (V (ot ))αt (j)βt (j) g∈S(ot )(3.55)

ξt (i, j) = P (qt = i, qt+1 = j|O, λ)αt (i)aij bj (ot+1 )βt+1 (j)= N N
αt (h)ahk bk (ot+1 )βt+1 (k)(3.56)

j=1

h=1 k=1

From the condition mentioned in Theorem 2, it can be shown that each Σig is
positive deﬁnite.

3.2.4

Application to F0 pattern modeling

The MSD-HMM includes the discrete HMM and the continuous mixture HMM
as special cases since the multi-space probability distribution includes the discrete
distribution and the continuous distribution. If ng ≡ 0, the MSD-HMM is the sameas the discrete HMM. In the case where S(ot ) speciﬁes one space, i.e., |S(ot )| ≡ 1,
the MSD-HMM is exactly the same as the conventional discrete HMM. If |S(ot )| ≥ 1,
the MSD-HMM is the same as the discrete HMM based on the multi-labeling VQ[26]. If ng ≡ m > 0 and S(o) ≡ {1, 2, . . . , G}, the MSD-HMM is the same as
the continuous G-mixture HMM. These can also be conﬁrmed by the fact that if
ng ≡ 0 and |S(ot )| ≡ 1, the reestimation formulas (3.50)-(3.52) are the same as thosefor discrete HMM of codebook size G, and if ng ≡ m and S(ot ) ≡ {1, 2, . . . , G},
the reestimation formulas (3.50)-(3.54) are the same as those for continuous HMM
with m-dimensional G-mixture densities. Further, the MSD-HMM can model thesequence of observation vectors with variable dimension including zero-dimensional
observations, i.e., discrete symbols.
While the observation of F0 has a continuous value in the voiced region, there exist
no value for the unvoiced region. We can model this kind of observation sequence
assuming that the observed F0 value occurs from one-dimensional spaces and the27

“unvoiced” symbol occurs from the zero-dimensional space deﬁned in Section 3.2.1,
that is, by setting ng = 1 (g = 1, 2, . . . , G − 1), nG = 0 and

S(ot ) ={1, 2, . . . , G − 1},{G},(voiced),(unvoiced)(3.57)

the MSD-HMM can cope with F0 patterns including the unvoiced region without
heuristic assumption. In this case, the observed F0 value is assumed to be drawn
from a continuous (G − 1)-mixture probability density function.

28

Chapter 4
Speech parameter generation fromHMM
The performance of speech recognition based on HMMs has been improved by incorporating the dynamic features of speech. Thus we surmise that, if there is a method
for speech parameter generation from HMMs which include the dynamic features, itwill be useful for speech synthesis by rule. This chapter derives a speech parameter
generation algorithm from HMMs which include the dynamic features.

4.1

Speech parameter generation based on maximum likelihood criterion

For a given continuous mixture HMM λ, we derive an algorithm for determining
speech parameter vector sequenceO = o
1 , o2 , . . . , oT

in such a way that

P (O|λ) =

all Q



P (O, Q|λ)

(4.1)

(4.2)

is maximized with respect to O, whereQ = {(q1 , i1 ), (q2 , i2 ), . . . , (qT , iT )}(4.3)

is the state and mixture sequence, i.e., (q, i) indicates the i-th mixture of stateq. We assume that the speech parameter vector ot consists of the static feature
vector ct = [ct (1), ct (2), . . . , ct (M )] (e.g., cepstral coeﬃcients) and dynamic feature29

vectors ∆ct , ∆2 ct (e.g., delta and delta-delta cepstral coeﬃcients, respectively), that2  is, ot = [ct , ∆ct , ∆ ct ] , where the dynamic feature vectors are calculated by
(1)

L+



∆ct =

w (1) (τ )ct+τ

(4.4)

w (2) (τ )ct+τ .

(4.5)

(1)
τ =−L−(2)L+∆2 ct =(2)
τ =−L−We have derived algorithms [11], [12] for solving the following problems:Case 1. For given λ and Q, maximize P (O|Q, λ) with respect to O under theconditions (4.4), (4.5).
Case 2. For a given λ, maximize P (O, Q|λ) with respect to Q and O under theconditions (4.4), (4.5).
In this section, we will review the above algorithms and derive an algorithm for the
problem:Case 3. For a given λ, maximize P (O|λ) with respect to O under the conditions(4.4), (4.5).

4.1.1

Case 1: Maximizing P (O|Q, λ) with respect to O

First, consider maximizing P (O|Q, λ) with respect to O for a ﬁxed state and mixture
sequence Q. The logarithm of P (O|Q, λ) can be written as
1
log P (O|Q, λ) = − O  U −1 O + O  U −1 M + K
2

(4.6)

where


−1
−1
U −1 = diag U −1
q1 ,i1 , U q2 ,i2 , ..., U qT ,iT

M =





µ
q1 ,i1 , µq2 ,i2 , ..., µqT ,iT





(4.7)
(4.8)

µqt , it and U qt , it are the 3M × 1 mean vector and the 3M × 3M covariance matrix, respectively, associated with it -th mixture of state qt , and the constant K is
independent of O.
30

It is obvious that P (O|Q, λ) is maximized when O = M without the conditions(4.4), (4.5), that is, the speech parameter vector sequence becomes a sequence ofthe mean vectors. Conditions (4.4), (4.5) can be arranged in a matrix form:
O = WC

(4.9)

where
C = [c1 , c2 , . . . , cT ]
W = [w 1 , w2 , . . . , wT ]
wt =
(n)

wt



(0)

(1)

(2)

wt , wt , wt

(4.10)(4.11)

(4.12)(n)= [ 0M ×M , . . . , 0M ×M , w (n) (−L− )I M ×M ,
1st

(n)

(t−L− )-th(n)

. . . , w(n) (0)I M ×M , . . . , w(n) (L+ )I M ×M ,
t-th

(n)

(t+L+ )-th

0M ×M , . . . , 0M ×M ] ,
T -th

(0)

n = 0, 1, 2

(4.13)(0)

L− = L+ = 0, and w (0) (0) = 1. Under the condition (4.9), maximizing P (O|Q, λ)with respect to O is equivalent to that with respect to C. By setting∂ log P (W C|Q, λ)= 0,∂C(4.14)W  U −1 W C = W  U −1 M  .

(4.15)

we obtain a set of equations

For direct solution of (4.15), we need O(T 3M 3 ) operations1 because W  U −1 W is a
T M × T M matrix. By utilizing the special structure of W  U −1 W , (4.15) can besolved by the Cholesky decomposition or the QR decomposition with O(T M3 L2 )operations2 , where L = maxn∈{1,2},s∈{−,+} L(n)s . Equation (4.15) can also be solved
by an algorithm derived in [11], [12], which can operate in a time-recursive manner
[28].

4.1.2

Case 2: Maximizing P (O, Q|λ) with respect to O andQ

This problem can be solved by evaluating maxC P (O, Q|λ) = maxC P (O|Q, λ)P (Q|λ)
for all Q. However, it is impractical because there are too many combinations of1

When U q,i is diagonal, it is reduced to O(T 3 M ) since each of the M -dimensions can becalculated independently.
(1)
(1)
2
When U q,i is diagonal, it is reduced to O(T M L 2 ). Furthermore, when L− = −1, L+ = 0,
and w(2) (i) ≡ 0, it is reduced to O(T M ) as described in [27].

31Q. We have developed a fast algorithm for searching for the optimal or sub-optimal
state sequence keeping C optimal in the sense that P (O|Q, λ) is maximized withrespect to C [11], [12].
To control temporal structure of speech parameter sequence appropriately, HMMs
should incorporate state duration densities. The probability P (O, Q|λ) can be writtenas
P (O, Q|λ)=i={i1 , i2 ,P (O, i|q, λ)P (q|λ),where q=
{q1 , q2 , . . . , qT },. . . , iT }, and the state duration probability P (q|λ) is given bylog P (q|λ) =Nlog pqn (dqn )(4.16)

n=1

where the total number of states which have been visited during T frames is N , andpqn (dqn ) is the probability of dqn consecutive observations in state qn . If we determine
the state sequence q only by P (q|λ) independently of O, maximizing P (O, Q|λ) =P (O, i|q, λ)P (q|λ) with respect to O and Q is equivalent to maximizing P (O, i|q, λ)with respect to O and i. Furthermore, if we assume that state output probabilities
are single-Gaussian, i is unique. Therefore, the solution is obtained by solving (4.15)
in the same way as the Case 1.

4.1.3

Case 3: Maximizing P (O|λ) with respect to O

We derive an algorithm based on an EM algorithm, which ﬁnd a critical point of
the likelihood function P (O|λ). An auxiliary function of current parameter vector
sequence O and new parameter vector sequence O is deﬁned byQ(O, O ) =P (O, Q|λ) log P (O  , Q|λ).

(4.17)all Q

It can be shown that by substituting O which maximizes Q(O, O ) for O, thelikelihood increases unless O is a critical point of the likelihood. Equation (4.17)
can be written as
1
Q(O, O ) = P (O|λ) − O  U −1 O  + O  U −1 M + K(4.18)
2
where


−1
−1
U −1 = diag U −1
1 , U2 , . . . , UT

U −1
=
t
U −1 M =U −1
t µt =


q,i





γt (q, i)U −1q,i(4.20)


−1
−1
U −1
1 µ1 , U 2 µ2 , . . . , U T µT

q,i(4.19)

γt (q, i) U −1
q,i µq,i32

 (4.21)(4.22)

and the constant K is independent of O . The occupancy probability γt (q, i) deﬁned
by(4.23)
γt (q, i) = P (qt = (q, i)|O, λ)can be calculated with the forward-backward inductive procedure. Under the condition O = W C  , C  which maximizes Q(O, O ) is given by the following set of
equations:W  U −1 W C  = W  U −1 M .
(4.24)
The above set of equations has the same form as (4.15). Accordingly, it can besolved by the algorithm for solving (4.15).
The whole procedure is summarized as follows:
Step 0. Choose an initial parameter vector sequence C.
Step 1. Calculate γt (q, i) with the forward-backward algorithm.
Step 2. Calculate U −1 and U −1 M by (4.19)–(4.22), and solve (4.24).
Step 3. Set C = C  . If a certain convergence condition is satisﬁed, stop; otherwise,
goto Step 1.
From the same reason as Case 2, HMMs should incorporate state duration densities. If we determine the state sequence q only by P (q|λ) independently of O in amanner similar to the previous section, only the mixture sequence i is assumed to
be unobservable3 . Further, we can also assume that Q is unobservable but phonemeor syllable durations are given.

4.2

ExampleA simple experiment of speech synthesis was carried out using the parameter generation algorithm. We used phonetically balanced 450 sentences from ATR Japanese
speech database for training. The type of HMM used was a continuous Gaussianmodel. The diagonal covariances were used. All models were 5-state left-to-rightmodels with no skips. The heuristic duration densities were calculated after the
training. Feature vector consists of 25 mel-cepstral coeﬃcients including the zerothcoeﬃcient, their delta and delta-delta coeﬃcients. Mel-cepstral coeﬃcients were obtained by the mel-cepstral analysis. The signal was windowed by a 25ms Black man
window with a 5ms shift.
3

For this problem, an algorithm based on a direct search has also been proposed in [29].

33Frequency (kHz)

0
2
4
6
8

Frequency (kHz)(a) without dynamic features
0
2
4
6
8

Frequency (kHz)(b) with delta parameters
0
2
4
6
8

(c) with delta and delta-delta parameters
Figure 4.1: Spectra generated with dynamic features for a Japanese phrase “chiisanaunagi”.

4.2.1

Eﬀect of dynamic featureFirst, we observed the parameter generation in the case 1, in which parametersequence O maximizes P (O|Q, λ). State sequence Q was estimated from the result of Veterbi alignment of natural speech. Fig. 4.1 shows the spectra calculated
from the mel-cepstral coeﬃcients generated by the HMM, which is composed byconcatenation of phoneme models. Without the dynamic features, the parameter sequence which maximizes P (O|Q, λ) becomes a sequence of the mean vectors(Fig. 4.1(a)). On the other hand, Fig. 4.1(b) and Fig. 4.1(c) show that appropriate
parameter sequences are generated by using the static and dynamic feature. Looking at Fig. 4.1(b) and Fig. 4.1(c) closely, we can see that incorporation of delta-delta
parameter improves smoothness of generated speech spectra.
Fig 4.2 shows probability density functions and generated parameters for the zero-thmel-cepstral coeﬃcients. The x-axis represents frame number. A gray box and its
middle line represent standard deviation and mean of probability density function,
respectively, and a curved line is the generated zero-th mel-cepstral coeﬃcients.
From this ﬁgure, it can be observed that parameters are generated taking accountof constraints of their probability density function and dynamic features.
34

c(0)

7.2

∆2 c(0)

∆c(0)

4.9
0.38
-0.31

0.070

-0.068frame
standard deviation

generated parameter

mean

Figure 4.2: Relation between probability density function and generated parameter
for a Japanese phrase “unagi” (top: static, middle: delta, bottom: delta-delta).

35

4.2.2

Parameter generation using multi-mixuture HMM

In the algorithm of the case 3, we assumed that the state sequence (state and mixture
sequence for the multi-mixture case) or a part of the state sequence is unobservable(i.e., hidden or latent). As a result, the algorithm iterates the forward-backward
algorithm and the parameter generation algorithm for the case where state sequenceis given. Experimental results show that by using the algorithm, we can reproduce
clear formant structure from multi-mixture HMMs as compared with that produced
from single-mixture HMMs.
It has found that a few iterations are suﬃcient for convergence of the proposed
algorithm. Fig. 4.3 shows generated spectra for a Japanese sentence fragment “kiNzokuhiroo” taken from a sentence which is not included in the training data. Fig. 4.4
compares two spectra obtained from single-mixure HMMs and 8-mixture HMMs, respectively, for the same temporal position of the sentence fragment. It is seen from
Fig. 4.3 and Fig. 4.4 that with increasing mixtures, the formant structure of thegenerated spectra get clearer.
We evaluated two synthesized speech obtained from single-mixure HMMs and 8mixture HMMs by listening test, where fundamental frequency and duration is obtained from natural speech. Figure 4.5 shows the result of pair comparison test.
From the listening test, it has been observed that the quality of the synthetic speech
is considerably improved by increasing mixtures.
When we use single-mixture HMMs, the formant structure of spectrum corresponding to each mean vector µq,i might be vague since µq,i is the average of diﬀerent
speech spectra. One can increase the number of decision tree leaf clusters. However,it might result in perceivable discontinuities in synthetic speech since overly large
tree will be overspecialized to training data and generalized poorly. We expect thatthe proposed algorithm can avoid this situation in a simple manner.

36

(kHz)

0
2
4
6
8

(kHz)

0
2
4
6
8

(kHz)

(a) 1-mixture

0
2
4
6
8

(b) 2-mixture

0
2
4
6
8

(d) 8-mixture
Figure 4.3: Generated spectra for a sentence fragment “kiNzokuhiroo.”100
Log magnitude (dB)(kHz)(c) 4-mixture

mix1
mix8

80
60
40
20

0

2
4
6
Frequency (kHz)8

Figure 4.4: Spectra obtained from 1-mixture HMMs and 8-mixture HMMs.
37

40.8

1-mixture

8-mixtures

59.2

0

20

40
60
Preference score (%)

80

Figure 4.5: The result of the pair comparison test.

38

100

Chapter 5
Construction of HMM-based
Text-to-Speech System
Phonetic parameter and prosodic parameter are modeled simultaneously in a uniﬁed
framework of HMM. In the proposed system, mel-cepstrum, fundamental frequency(F0) and state duration are modeled by continuous density HMMs, multi-spaceprobability distribution HMMs and multi-dimensional Gaussian distributions, respectively. The distributions for spectrum, F0, and the state duration are clusteredindependently by using a decision-tree based context clustering technique. This
chapter describes feature vector modeled by HMM, structure of HMM and how to
train context-dependent HMM.

5.1

Calculation of dynamic feature

In this thesis, mel-cepstral coeﬃcient is used as spectral parameter. Mel-cepstralcoeﬃcient vectors c are obtained from speech database using a mel-cepstral analysis
technique [16]. Their dynamic feature ∆c and ∆2 c are calculated as follows:11∆ct = − ct−1 + ct+1 ,
2
2
1
1
1
2
∆ ct =
ct−1 − ct + ct+1 .
4
2
4

(5.1)
(5.2)

In the same way, dynamic features for F0 are calculated by11
δpt = − pt−1 + pt+1 ,
2
2
1
1
1
δ 2 pt =pt−1 − pt + pt+1 .
4
2
4
39

(5.3)
(5.4)

represents a continous value ( voiced ).
represents a discrete symbol ( unvoiced ).
Unvoiced region

Voiced region

Delta-deltaδ 2 ptDeltaδpt

Staticpt−1 pt pt+1
1

Unvoiced regiont

T
Frame number

Figure 5.1: Calculation of dynamic features for F0.

where, in unvoiced region, pt , δpt and δ 2 pt are deﬁned as a discrete symbol. When
dynamic features at the boundary between voiced and unvoiced can not be calculated, they are deﬁned as a discrete symbol. For example, if dynamic features arecalculated by Eq.(5.3)(5.4), δpt and δtp at the boundary between voiced and unvoiced
as shown Fig. 5.1 become discrete symbol.

5.2

Spectrum and F0 modelingIn the chapter 3, it is described that sequence of mel-cepstral coeﬃcient vector andF0 pattern are modeled by a continuous density HMM and multi-space probability
distribution HMM, respectively.
We construct spectrum and F0 models by using embedded training because theembedded training does not need label boundaries when appropriate initial models
are available. However, if spectrum models and F0 models are embedded-trained
separately, speech segmentations may be discrepant between them.
To avoid this problem, context dependent HMMs are trained with feature vector
which consists of spectrum, F0 and their dynamic features (Fig. 5.2). As a result,
HMM has four streams as shown in Fig. 5.3.
40

cStream 1(Spectrum part)∆cContinuous probability distribution

2∆ cStream 2(F0 part)

p
δp
2
δ p

Multi−space probability distribution
Multi−space probability distribution
Multi−space probability distribution

Figure 5.2: Feature vector.

Mel-cepstrum

N-dimensional
Gaussiandistribution

Voiced1-dimensional
Gaussiandistribution

Unvoiced

F0

Voiced0-dimensional
Gaussiandistribution

Unvoiced
VoicedUnvoiced

Figure 5.3: structure of HMM.

41

5.3

Duration modeling

5.3.1

Overview

There have been proposed techniques for training HMMs and their state duration
densities simultaneously (e.g., [30]). However, these techniques require a large storage and computational load. In this thesis, state duration densities are estimatedby using state occupancy probabilities which are obtained in the last iteration ofembedded re-estimation [31].
In the HMM-based speech synthesis system described above, state duration densities
were modeled by single Gaussian distributions estimated from histograms of state
durations which were obtained by the Viterbi segmentation of training data. In thisprocedure, however, it is impossible to obtain variances of distributions for phonemes
which appear only once in the training data.
In this thesis, to overcome this problem, Gaussian distributions of state durations
are calculated on the trellis(Section 3.1.2) which is made in the embedded training
stage. State durations of each phoneme HMM are regarded as a multi-dimensional
observation, and the set of state durations of each phoneme HMM is modeled bya multi-dimensional Gaussian distribution. Dimension of state duration densities isequal to number of state of HMMs, and nth dimension of state duration densities
is corresponding to nth state of HMMs1 . Since state durations are modeled bycontinuous distributions, our approach has the following advantages:• The speaking rate of synthetic speech can be varied easily.
• There is no need for label boundaries when appropriate initial models areavailable since the state duration densities are estimated in the embedded
training stage of phoneme HMMs.
In the following sections, we describe training and clustering of state duration models, and determination of state duration in the synthesis part.

5.3.2

Training of state duration models

There have been proposed techniques for training HMMs and their state duration
densities simultaneously, however, these techniques is ineﬃcient because it requires
huge storage and computational load. From this point of view, we adopt anothertechnique for training state duration models.
1

We assume the left-to-right model with no skip.

42

State duration densities are estimated on the trellis which is obtained in the embedded training stage. The mean ξ(i) and the variance σ2 (i) of duration density ofstate i are determined byTTξ(i) =

χt0 ,t1 (i)(t1 − t0 + 1)

t0 =1 t1 =t0
T


,

T


χt0 ,t1 (i)t0 =1 t1 =t0TT

σ 2 (i) =(5.5)

χt0 ,t1 (i)(t1 − t0 + 1)2

t0 =1 t1 =t0
T− ξ 2 (i),T

t0 =1 t1 =t0(5.6)

χt0 ,t1 (i)respectively, where χt0 ,t1 (i) is the probability of occupying state i from time t0 to t1
and can be written as
χt0 ,t1 (i) = (1 − γt0 −1 (i)) ·

t1
t=t0

γt (i) · (1 − γt1 +1 (i)),(5.7)

where γt (i) is the occupation probability of state i at time t, and we deﬁne γ−1 (i) =γT +1(i) = 0.

5.4

Context dependent model

5.4.1

Contextual factors

There are many contextual factors (e.g., phone identity factors, stress-related factors,
locational factors) that aﬀect spectrum, F0 and duration. In this thesis, following
contextual factors are taken into account:• mora2 count of sentence• position of breath group in sentence• mora count of {preceding, current, succeeding} breath group• position of current accentual phrase in current breath group• mora count and accent type of {preceding, current, succeeding} accentualphrase2

A mora is a syllable-sized unit in Japanese.

43• {preceding, current, succeeding} part-of-speech• position of current phoneme in current accentual phrase• {preceding, current, succeeding} phoneme
Note that a context dependent HMM corresponds to a phoneme.

5.4.2

Decision-tree based context clustering

When we construct context dependent models taking account of many combinations of the above contextual factors, we expect to be able to obtain appropriate
models. However, as contextual factors increase, their combinations also increase
exponentially. Therefore, model parameters with suﬃcient accuracy cannot be estimated with limited training data. Furthermore, it is impossible to prepare speech
database which includes all combinations of contextual factors.
Introduction of context clustering
To overcome the above problem, we apply a decision-tree based context clustering
technique [32] to distributions for spectrum, F0 and state duration.
The decision-tree based context clustering algorithm have been extended for MSDHMMs in [33]. Since each of spectrum, F0 and duration have its own inﬂuential
contextual factors, the distributions for spectral parameter and F0 parameter andthe state duration are clustered independently (Fig. 5.4.2).
Example of decision tree
We used phonetically balanced 450 sentences from ATR Japanese speech databasefor training. Speech signals were sampled at 16 kHz and windowed by a 25-ms
Blackman window with a 5-ms shift, and then mel-cepstral coeﬃcients were obtainedby the mel-cepstral analysis3 . Feature vector consists of spectral and F0 parameter
vectors. Spectral parameter vector consists of 25 mel-cepstral coeﬃcients including
the zeroth coeﬃcient, their delta and delta-delta coeﬃcients. F0 parameter vector
consists of log F0, its delta and delta-delta. We used 3-state left-to-right HMMs
with single diagonal Gaussian output distributions. Decision trees for spectrum, F0
and duration models were constructed as shown in Fig. 5.4.2. The resultant trees
3

The source codes of the mel-cepstral
http://kt−lab.ics.nitech.ac.jp/˜tokuda/SPTK/ .

44

analysis/synthesis

can

be

found

in

State Duration
Model
HMM
for Spectrum
and F0S1

S2

S3

Decision Treefor
Spectrum

Decision TreeforState Duration Model

Decision Treefor
F0

Figure 5.4: Decision trees.

45

C_voiced
yesnoC_voicedyes

C_jaw_wide_vowel
yes

no

no

yes
no

C_u
yes

R_silence

C_silenceyesnoR_silencenoyes

L_mouthno

L_voicedyes

yes

R_silence

no

no

no

L_breath_length <= 5yesaccent_type-mora <= 0noR_accent_length <= 8

yes
no

R_i
yes

no

C_mouthyes

R_breath_length <= 7C_r
"pit_s2_2"

R_back_vowel"dur_s2_187"
L_jaw_half_vowel

C_unvoiced_vowel
nosentence_length <= 31

R_voiced

C_U
yes

C_glottis
yesnoR_silence

C_fricativenoC_breath_position <= 1
yes

C_N

C_N

yes

no

L_breath_length <= 6
C_part-of-speech = 17

C_ono

C_pauseyes"pit_s2_1"

C_stop
yes

C_I

no

C_affricate

C_voiced
R_silence(a) Tree for Spectrum
Model(1st state)(b) Tree for F0
Model(1st state)(c) Tree for State
Duration Model

Figure 5.5: Examples of decision trees.

for spectrum models, F0 models and state duration models had 6,615, 1,877 and201 leaves in total, respectively.
Fig. 5.4.2 shows examples of constructed decision trees for spectrum (a), F0 (b) andstate duration (c). In these ﬁgures, “L ∗”, “C ∗” and “R ∗” represent “preceding”,“current” and “succeeding”, respectively. “Silence” represents silence of head or tailof a sentence, or pause. Questions of breath group and accentual phrase are represented by “∗ breath ∗” and “∗ accent ∗”, respectively. “Pit s2 ∗” and “dur s2 ∗”represent leaf nodes. From these ﬁgures, it is seen that spectrum models are muchaﬀected by phonetic identity, F0 models for “voiced” are much aﬀected by accentualphrase and part-of-speech, and F0 models for “unvoiced” are clustered by a very
simple tree. With regard to state duration models, it can be seen that silence andpause models are much aﬀected by accentual phrase and part-of-speech, and theother models are much aﬀected by phonetic identity.
Through informal listening tests, we have found that the stopping rule (a minimum
frame occupancy at each leaf and a minimum gain in likelihood per splice) should bedetermined appropriately in decision tree construction. An overly large tree will beoverspecialized to training data and generalize poorly. On the other hand, a overly
small tree will model the data badly. Therefore we should introduce some stopping
criterion or cross-validation method (e.g., [34]–[35]).
46

5.4.3

Context clustering using MDL principle

In this section, the MDL principle is incorporated as the stopping rule for the treebased clustering [36], Before explaining the use of the MDL principle for the treebased clustering, this section brieﬂy introduces the principle itself.
Introduction of MDL principle
Minimum Description Length (MDL) principle is an information criterion which
has been proven to be eﬀective in the selection of optimal probabilistic models. It
has been used to obtain a good model for data in various problems. According to
the MDL principle, the model with the minimum description length l is selected tobe the optimal model for data xN = x1 , · · · , xN , from among probabilistic models
i = 1, · · · , I. A description length l( i) of model i is given as,l( i) = − log Pθ̂(i) (xN ) +αilog N + log I
2

(5.8)

where αi is the number of free parameters of the model i, and θ̂(i) is the maximum(i)likelihood estimates for the parameters θ( i) = (θ1 , · · · , θα(i)i ) of model i. The ﬁrst
term is the negative of the log likelihood for the data, and the second term represents
the complexity of the model. The third term is the description length required forchoosing model i. As a model becomes more complex, the value of the ﬁrst term
decreases and that of the second term increases. The description length l has its
minimum at a model of appropriate complexity. Furthermore, as one can see in
Eq. (5.8), the MDL principle does not need any externally given parameters; theoptimal model for the data is automatically obtained once a set of models is given.
MDL principle for tree-based clustering
The MDL principle is incorporated to the tree-based clustering for MSD-HMM which
includes continuous density HMM. It is assumed that cluster set S which is a resultof clustering is deﬁned by
S = {S1 , S2 , · · · , Si , · · · , SM }(5.9)

Then, log likelihood L is calculated as follows:
L = −

G

1
s∈S g=1 2(ng (log(2π) + 1) + log |Σsg |−2 log wsg )t∈T (O,g)47

γt (s, g),(5.10)

where g is a space index, and wsg is a weight of space g in cluster s. T (O, g) is a setof time t which satisﬁes that a set of space index of observation vector mot includes
a space index g. γt (s, g) is the probability of being in space g of cluster s at time t.
In the case of the zero-dimensional space, log |Σsg | in Eq.(5.10) is equal to 0. Using
this likelihood L, a description length (DL) l is represented by
l =

G

1
s∈S g=1 2(ng (log(2π) + 1) + log |Σsg |

−2 log wsg )+ 

γt (s, g)t∈T (O,g)
G1s∈S g=1 2· log(2ng + 1)

G






γt (s, g)

(5.11)s∈S g=1 t∈T (O,g)

If it is assumed that the description length is l when a cluster Si is divided two
clusters Si+ and Si− , The change of description length δl is calculated byδl = l − l

G

1



=

s∈{Si+ ,Si− } g=12(log |Σsg | − 2 log wsg )·γt (s, g)t∈T (O,g)

−



G


1
(log |Σsg | − 2 log wsg )2
s∈{Si } g=1


·


+ 

G

1

g=1 2



γt (s, g)t∈T (O,g)(2ng + 1)

· logG

γt (s, g) .

(5.12)s∈S g=1 t∈T (O,g)

If δl < 0 then the node is devided, and if δl ≥ 0 then the node is not devided.
Fig. 5.4.2 shows examples of constructed decision trees constructed by using the
MDL principle.

48

Chapter 6
HMM-based Text-to-Speech
Synthesis
Synthetic speech is generated by using an speech parameter generation algorithm
from HMM and a mel-cepstrum based vocoding technique. Through informal listening tests, we have conﬁrmed that the proposed system successfully synthesizes
natural-sounding speech which resembles the speaker in the training database.

6.1

Overview

The synthesis part of the HMM-based text-to-speech synthesis system is shown in
Fig. 6.1. In the synthesis part, an arbitrarily given text to be synthesized is converted
to a context-based label sequence. Then, according to the label sequence, a sentence
HMM is constructed by concatenating context dependent HMMs. State durationsof the sentence HMM are determined so as to maximize the likelihood of the state
duration densities [31]. According to the obtained state durations, a sequence of melcepstral coeﬃcients and F0 values including voiced/unvoiced decisions is generated
from the sentence HMM by using the speech parameter generation algorithm [12].
Finally, speech is synthesized directly from the generated mel-cepstral coeﬃcientsand F0 values by the MLSA ﬁlter [16], [19].

6.2

Text analysis

The inputed text is converted a context dependent label sequence by a text analyzer.
For the TTS system, the text analyzer should have ability to extracted contextual
49

Context Dependent
Duration Models
Context Dependent
HMMs

TEXT

Synthesis

Context−basedLabel Sequence

State Duration
Densities
Sentence
HMM
State Duration

d1

d2

F0

p1 p2 p3 p4 p5 p 6

pT

Mel−Cepstrumc1 c2 c3 c 4 c 5 c 6

cT

MLSA Filter
SYNTHETIC SPEECH

Figure 6.1: The block diagram of the HMM-based TTS.

50

informations listed in Section 5.4.1 from the text. Currently, some Japanese text analyzer (e.g., ChaSen1 , MSLR2 , etc.) are freely available. However, no text analyzer
has the ability to extract accentual phrase and to decide accent type of accentualphrase. Thus, in this thesis, the inputed text is manually analyzed and converted
context dependent label sequence.

6.3

Duration determination

For a given speech length T , the goal is to obtain a state sequence q = {q1 , q2 , · · · , qT }which maximize
log P (q|λ, T ) =Klog pk (dk )

(6.1)

k=1

under the constraint

K


T =

dk ,

(6.2)

k=1

where pk (dk ) is the probability of duration dk in state k, and K is the number ofstates in HMM λ.
Since each duration density pk (dk ) is modeled by a single Gaussian distribution,state durations {dk }K
k=1 which maximize (6.1) are given by
dk = ξ(k) + ρ · σ 2 (k)
!

ρ =

T−

K


ξ(k)

k=1

"# K


(6.3)
σ 2 (k) ,

(6.4)

k=1

2

where ξ(k) and σ (k) are the mean and variance of the duration density of state k,respectively.
Since ρ is associated with T in (6.4), the speaking rate can be controlled by ρ insteadof T . From (6.3), it can be seen that to synthesize speech with average speaking
rate, ρ should be set to 0, that is, T = K
k=1 ξ(k), and the speaking rate becomes
faster or slower when we set ρ to negative or positive value, respectively. It can also
be seen that the variance σ2 (k) represents “elasticity” of kth state duration.

6.4

Speech parameter generation

According to the estimated state duration, spectral and excitation parameters are
generated from a sentence HMM constructed by concatenating context dependent
1
2

http://chasen.aist-nara.ac.jp/
http://tanaka-www.cs.titech.ac.jp/pub/mslr/

51

Frequency (kHz) Frequency (kHz)

0
2
4
6
8
0
2
4
6
8

Figure 6.2: Generated spectra for a phrase “heikiNbairitsu” (top: natural spectra,
bottom: generated spectra).

HMMs. Fig. 6.2 shows spectra of natural speech and synthesized speech for a
Japanese phrase “heikiNbairitu” Fig. 6.4 shows F0 pattern of natural speech andsynthesized speech for a Japanese sentence “heikiNbairituwo sageta keisekiga aru”.

6.5

Experiments

6.5.1

Eﬀect of dynamic feature

Examples of generated spectra and F0 pattern are shown in Fig. 6.5.1 and Fig. 6.5.1,
respectively. In each ﬁgure, the parameters generated with and without dynamic
features are shown. From these ﬁgures, it can be seen that spectra and F0 patternwhich approximate those of natural speech are generated by using the parameter
generation algorithm with dynamic features.
The eﬀect of speech parameter generation with the dynamic features was evaluatedby a pair comparison test. The following four samples were compared:• spectrum generated without dynamic features+ F0 without dynamic features• spectrum generated with dynamic features+ F0 without dynamic features• spectrum generated without dynamic features
52

Frequency

200
150
100
50

Frequency

0

1

2time (s)

200
150
100
50
0

1

2
time (s)Frequency (kHz)Frequency (kHz)Frequency (kHz)

Figure 6.3: Generated F0 pattern for a sentence “heikiNbairitsuwo sageta keisekigaaru” (top: natural F0 pattern, bottom: generated F0 pattern).

0
2
4
6
8

(a) generated spectrum without dynamic feature
0
2
4
6
8

(b) generated spectrum with dynamic feature(δ)
0
2
4
6
8

(c) generated spectrum with dynamic feature(δ + δ2 )Figure 6.4: Generated spectra for a phrase “heikiNbairitsu.”

53

Frequency

200
150
100
50
0

1

2

Frequency

time (s)(a) generated F0 without dynamic feature
200
150
100
50
0

1

2

Frequency

time (s)(b) generated F0 with dynamic feature(δ)
200
150
100
50
0

1

2

time (s)(c) generated F0 with dynamic feature(δ + δ2 )Figure 6.5: Generated F0 pattern for a sentence “heikiNbairitsuwo sageta keisekiga
aru.”

54

spectrum(static)+ F0(static)

11.8 %

spectrum(static+dynamic)+ F0(static)

35.8 %

spectrum(static)+ F0(static+dynamic)

37.5 %

spectrum(static+dynamic)+ F0(static+dynamic)

91.3 %
0

20
40
60
80
Preference Score (%)

100

Figure 6.6: Eﬀect of dynamic feature.

+ F0 with dynamic features• spectrum generated with dynamic features+ F0 with dynamic features
Figure 6.5.1 shows preference scores. It can be seen that in the case of using dynamic
features synthesized speech is signiﬁcantly improved.

6.5.2

Automatically system training

In the proposed system, there is no need for label boundaries when appropriate
initial models are available since spectral, F0 and duration models are estimated by
the the embedded training. Therefore, the system can be automatically constructedby the following process:1. As a initial model, speaker independent and gender dependent model is prepared.
2. The target speaker dependent model is estimated by using initial model andspeech data with transcription.
55

Table 6.1: Number of distribution.
Spectrum
F0
Duration
SD
942
2597
1528
GD
952
2539
1552
SI
956
2623
1765

0

SD

48.4%

GD

54.7%

SI

46.9%

20
40
60
80
Preference Score(%)

100

Figure 6.7: Eﬀect of diﬀerence of initial model.

We did automatic system construction tests. As a initial model, we prepared speakerdependent model(SD), gender dependent and speaker independent model(GD), andspeaker independent model(SI). SD was trained using speech data from target malespeaker MHT. For GD, speech data from ﬁve male speakers in which target speaker
MHT did not include was used. For SI, speech data from ﬁve male and four female
speakers in which target speaker MHT did not include was used. Figure 6.1 shows
the size of the resultant model.
The diﬀerence of quality of speech synthesized by using each initial model was
evaluated by a pair comparison test. Preference scores are shown in Fig. 6.5.2.
From these ﬁgure, it is seen that there is no diﬀerence of speech quality between
initial model SD, GD and SI.

6.5.3

Speaking rate

Fig. 6.5.3 shows generated spectra for a Japanese sentence which is not includedin the training data, setting ρ to −0.1, 0, 0.1. Only the part corresponding tothe ﬁrst phrase “/t-o-k-a-i-d-e-w-a/”, which means “in a city” in English, is shownin this ﬁgure. From the ﬁgure, it can be seen that some parts such as stationary
parts of vowels have elastic durations, and other parts such as explosives have ﬁxed
56

Frequency (kHz)0Frequency (kHz)0Frequency (kHz)

0
2
4

2
4
6
8

2
4
6
8

6
8

t

o

k

a

i

d

e

w

a

Figure 6.8: Generated spectra for an utterance “/t-o-k-a-i-d-e-w-a/” with diﬀerent
speaking rates (top : ρ = −0.1, middle : ρ = 0, bottom : ρ = 0.1).

durations. From informal listening tests, we found that synthetic speech had a goodquality with natural timing. Furthermore, we conﬁrmed that synthetic speech could
keep natural timing even if its speaking rate was changed in some degree.

57

Chapter 7
Improvement of Synthesized
Speech Quality
This chapter describes improvements on the excitation model of an HMM-based
text-to-speech system. In our previous work, natural sounding speech can be synthesized from trained HMMs. However, it has a typical quality of “vocoded speech”
since the system uses a traditional excitation model with either a periodic impulse
train or white noise. In this thesis, in order to reduce the synthetic quality, a mixed
excitation model used in MELP is incorporated into the system. Excitation parameters used in mixed excitation are modeled by HMMs, and generated from HMMs
by a parameter generation algorithm in the synthesis phase. The result of a listening test shows that the mixed excitation model signiﬁcantly improves quality ofsynthesized speech as compared with the traditional excitation model.

7.1

Introduction of Mixed Excitation ModelIn the previous work [10], natural sounding speech can be synthesized from trained
HMMs. However, synthesized speech has a typical quality of “vocoded speech”
since the HMM-based TTS system used a traditional excitation model with either aperiodic impulse train or white noise shown in Fig. 7.1. To overcome this problem,the excitation model should be replaced with more precise one.
For low bit rate narrowband speech coding at 2.4kbps, the mixed excitation linear
predictive (MELP) vocoder has been proposed [37]. In order to reduce the synthetic
quality and mimic the characteristics of natural human speech, this vocoder has thefollowing capabilities:• mixed pulse and noise excitation
58

Periodic pulse train

White noise

MLSA filter
Synthetic speech

Figure 7.1: Traditional excitation model.
• periodic or aperiodic pulses• pulse dispersion ﬁlter
The mixed excitation is implemented using a multi-band mixing model(Fig. 7.2),
and can reduce the buzz of synthesized speech. Furthermore, aperiodic pulses andpulse dispersion ﬁlter reduce some of the harsh or tonal sound quality of synthesized
speech. In recent years, the mixed excitation model of MELP has been applied
not only to narrowband speech coding but also to wideband speech coder [38] andspeech synthesis system [39].
In this thesis, mixed excitation model which is similar to the excitation model usedin MELP is incorporated into the TTS system. Excitation parameters, i.e., F0,
bandpass voicing strengths and Fourier magnitudes, are modeled by HMMs, andgenerated from trained HMMs in synthesis phase.

7.1.1

Mixed excitation

Analysis phase
In order to realize the mixed excitation model in the system, the following excitation
parameters are extracted from speech data.
• F0• bandpass voicing strengths• Fourier magnitudes
In bandpass voicing analysis, the speech signal is ﬁltered into ﬁve frequency bands,
with passbands of 0–1000, 1000–2000, 2000–4000, 4000–6000, 6000–8000Hz [38].
Note that the TTS system deals with 16kHz sampling speech. The voicing strengthin each band is estimated using normalized correlation coeﬃcients around the pitch
59

Noise excitation
80

70

70
Log magnitude (dB)

Log magnitude (dB)

Pulse excitation
80

60
50
40
30
20
10
0

60
50
40
30
20
10

0

1

2
3
Frequency (kHz)

0

4

0

1

2
3
Frequency (kHz)

4

1

2
3
Frequency (kHz)

4

80

70

70
Log magnitude (dB)

80

60
50
40
30
20
10
0

60
50
40
30
20
10

0

1

2
3
Frequency (kHz)

0

4

0

mixing80
70
Log magnitude (dB)

Log magnitude (dB)bandpass filtering

60
50
40
30
20
10
0

0

1

2
3
Frequency (kHz)4

Mixed Excitation

Figure 7.2: Multi-band mixing model.

60

Periodic pulse train

White noise

Position jitter

Bandpassvoicing
strengths

Bandpass filter

Bandpass filter

MLSA filter
Pulse dispersion
Synthetic speech

Figure 7.3: Mixed excitation model.
lag. The correlation coeﬃcient at delay t is deﬁned byN−1


sn sn+tn=0

ct = $
%

N
−1
−1

%N
&
s s
s
n n

n=0

,

(7.1)

n+t sn+tn=0where sn and N represent the speech signal at sample n and the size of pitch analysis
window, respectively. The Fourier magnitudes of the ﬁrst ten pitch harmonics aremeasured from a residual signal obtained by inverse ﬁltering.
Synthesis phase
A block diagram of the mixed excitation generation and speech synthesis ﬁltering isshown in Fig. 7.3.
The bandpass ﬁlters for pulse train and white noise are determined from generated
bandpass voicing strength. The bandpass ﬁlter for pulse train is given by the sum of
all the bandpass ﬁlter coeﬃcients for the voiced frequency bands, while the bandpass
ﬁlter for white noise is given by the sum of the bandpass ﬁlter coeﬃcients for theunvoiced bands. The excitation is generated as the sum of the ﬁltered pulse andnoise excitations. The pulse excitation is calculated from Fourier magnitudes using
an inverse DFT of one pitch period in length. The pitch used here is adjusted byvarying 25% of its position according to the periodic/aperiodic ﬂag decided from thebandpass voicing strength. By the aperiodic pulses, the system mimics the erratic
61

c

SPECTRALPARAMETERS

Mel-cepstrum∆cContinuous probability distribution

∆2c

F0

p
δp
δ 2p

Multi-space probability distribution
Multi-space probability distribution
Multi-space probability distribution

Vbp

EXCITATIONPARAMETERS

Bandpass voicing
strength∆ Vbp

Continuous probability distribution

∆2Vbp
M

Fourier magnitude∆M

Continuous probability distribution∆M
2

Figure 7.4: Structure of a feature vector modeled by HMM.
glottal pulses and reduces the tonal noise. The noise excitation is generated by auniform random number generator. The obtained pulse and noise excitations are
ﬁltered and added together.
By exciting the MLSA ﬁlter [19], synthesized speech is generated from the melcepstral coeﬃcients, directly. Finally, the obtained speech is ﬁltered by a pulse
dispersion ﬁlter which is a 130-th order FIR ﬁlter derived from a spectrally-ﬂattened
triangle pulse based on a typical male pitch period. The pulse dispersion ﬁlter can
reduce some of the harsh quality of the synthesized speech.

7.1.2

Excitation parameter modeling

Feature vector
The structure of the feature vector is shown in Fig. 7.4. The feature vector consistsof spectral and excitation parameters.
Mel-cepstral coeﬃcients including zero-th coeﬃcient and their delta and delta-delta62

Mel-cepstrum

VoicedUnvoiced

F0

VoicedUnvoiced
VoicedUnvoiced

Bandpass voicing
strength
Fourier magnitude
Gaussian distribution

Figure 7.5: Structure of a HMM.
coeﬃcients are used as spectral parameters. By using a mel-cepstral analysis technique [16] of an order of 24, mel-cepstral coeﬃcients are obtained from speech signal
windowed by a 25-ms Blackman window with a 5-ms shift.
Excitation parameters include F0 represented by log fundamental frequency (log f0 ),ﬁve bandpass voicing strengths, Fourier magnitudes of the ﬁrst ten pitch harmonics,
and their delta and delta-delta parameters.

Context dependent model
Feature vectors are modeled by 5-state left-to-right HMMs. Each state of an HMM
has four streams for mel-cepstrum, F0, bandpass voicing strengths and Fourier
magnitudes, respectively(Fig. 7.5). In each state, mel-cepstrum, bandpass voicing
strengths and Fourier magnitudes are modeled by single diagonal Gaussian distributions, and F0 is modeled by the multi-space probability distribution [40].
63Feature vectors are modeled by context dependent HMM taking account of contextual factors which aﬀect spectral parameter and excitation parameter such as phone
identity factors, stress-related factors and locational factors. Details of the contextual factors are shown in [10]. The trained context dependent HMMs are clusteredusing a tree-based context clustering technique based on MDL principle [36]. Since
each of mel-cepstrum, F0, bandpass voicing strength, Fourier magnitude and duration has its own inﬂuential contextual factors, the distributions for each speechparameter are clustered independently, where state occupation statistics used forclustering are calculated from only the streams of mel-cepstrum and F0.

7.1.3

Excitation parameter generation

A context dependent label sequence is obtained by text analysis of input text, and asentence HMM is constructed by concatenating context dependent phoneme HMMs
according to the obtained label sequence. By using a speech parameter generation
algorithm [41], mel-cepstrum, F0, bandpass voicing strength and Fourier magnitude
are generated from the sentence HMM taking account of their respective dynamic
feature statistics. Speech is synthesized from the obtained spectral and excitation
parameters.

7.2

Incorporation of postﬁlter

Many speech coders, which include the MELP, attempted to improve synthesized
speech quality by incorporating postﬁlter, and succeeded it. The our TTS system
also incorporates the postﬁlter.
In Chapter 2, the synthesis ﬁlter D(z) was realized using the MLSA ﬁlter. In order torealize the postﬁlter, ﬁrst, a transfer function D̄(z) is deﬁned. The transfer function
D̄(z) is the same as D(z) except that c(1) is forced to be zero to compensate for the
global spectral tilt. By setting c(1) = 0, the transfer function D̄(z) is written byD̄(z) = expb̄(m) =

Mb̄(m)Φm (z)

(7.2)

b(m)
2≤m≤M
−αb(2) m = 1

(7.3)

m=1

We can realize the postﬁlter D̄ β (z) in the same manner as D̄(z), by multiplyingc(m) by β. The tunable parameter β control the amount of postﬁltering.
The eﬀect of the postﬁltering is shown in Fig. 7.6 In Fig. 7.6, β is set to 0.5. From
the ﬁgure, it can be observed that formant is emphasized by the postﬁltering.
64

100

Log magnitude (dB)

80
60
40
20
0

0

0.25
Normalized frequency

0.5

Figure 7.6: Eﬀect of postﬁltering(dots: before postﬁltering, solid: after postﬁltering
β = 0.5).

7.3

Experiments

7.3.1

Excitation generation

Excitation parameters were generated from an HMM set trained using phoneticallybalanced 450 sentences of ATR Japanese speech database. The resulting decision
trees for mel-cepstrum, F0, bandpass voicing strength, Fourier magnitude and state
duration models had 934, 1055, 1651, 3745 and 1016 leaves in total, respectively. A
part of each decision tree is shown in Appendix 9.2.
Examples of traditional excitation and mixed excitation are shown in Fig. 7.7, where
the pulse dispersion ﬁlter was applied to mixed excitation. From the ﬁgure, it can beobserved that the voiced fricative consonant “z” has both the periodic and aperiodic
characteristics in the mixed excitation.

7.3.2

Eﬀect of mixed excitation

The TTS system with mixed excitation model was evaluated. We compared traditional excitation and mixed excitation by a pair comparison test. In addition,eﬀects of the Fourier magnitudes, aperiodic pulses and the pulse dispersion ﬁlter
were evaluated.
The following ﬁve excitation models were compared:65

Amplitude

12

-12

0

1144

2288

3432

4576

5720

6864

8008

9152

10296 sample

0

1144

2288

3432

4576

5720

6864

8008

9152

10296 sample

k

o

sh

I

z

u

ts

u

Amplitude

12

-12

s

U

Figure 7.7: Example of generated excitation for phrase “sukoshizutsu.” (top: traditional excitation，bottom: mixed excitation)

TE
ME
FM
AP
PD

:
:
:
:
:

traditional excitation
mixed excitation
ME + Fourier magnitudes
FM + aperiodic pulses
AP + pulse dispersion ﬁlter

The model TE is the traditional excitation model which generates either periodic
pulse train or white noise. Each of models ME, FM, AP and PD is the mixed
excitation model. In the model ME, pulse train was not calculated from Fourier
magnitude, and the aperiodic pulse and the pulse dispersion ﬁlter were not applied.
In the model FM, pulse excitation was calculated from Fourier magnitude. The
model AP used aperiodic pulses, and the model PD used the pulse dispersion
ﬁlter additionally. Eight subjects tested the ﬁve kinds of synthesized speech. Eight
sentences were selected at random for each subjects from 53 sentences which were
not included in the training data. Figure 7.8 shows preference scores. It can beseen that the mixed excitation model signiﬁcantly improved the quality of synthetic
speech. Although no additional gain was obtained by using Fourier magnitudes
and aperiodic pulses, the additional use of pulse dispersion ﬁlter achieved further
improvement in speech quality.
66Preference score (%)

100
80
66.0

60.5

60

56.8

55.9

FM

AP

40
20
0

10.7
TE

ME

PD

Figure 7.8: Comparison of traditional and mixed excitation models.

7.3.3

Eﬀect of postﬁltering

The eﬀect of the postﬁltering was evaluated by a pair comparison test. The following
four samples were compared:• conventional excitation model without postﬁltering(NORMAL)• conventional excitation model with postﬁltering(NORMAL+POST)• mixed excitation model without postﬁltering(MIXED)• mixed excitation model with postﬁltering(MIXED+POST)
The mixed excitation model indicates PD in Section 7.3.
Figure 7.9 shows preference scores. It can be seen that the mixed excitation modelwith postﬁltering signiﬁcantly improved the quality of synthetic speech. Even theconventional excitation model with postﬁltering also can improve the quality.

67

100
84.9
Preference score (%)

80

60

53.1

51.3

40

20
10.7
0
NORMAL NORMAL
+POST

MIXED

MIXED
+POST

Figure 7.9: Eﬀect of mixed excitation and postﬁltering.

68

Chapter 8
Voice Conversion Technique:
Speaker Interpolation
Although TTS systems which synthesize speech by concatenating speech units can
synthesize speech with acceptable quality, they still cannot synthesize speech withvarious voice quality such as speaker individualities and emotions. To obtain various voice quality in text-to-speech synthesis systems based on the selection andconcatenation of acoustical units, large amounts of speech data is required. However, it is diﬃcult to collect, segment, and store these data. In this chapter, tosynthesize speech with various voice characteristics, a TTS system based on speaker
interpolation is proposed.

8.1

Overview

The proposed system[42] synthesizes speech with untrained speaker’s voice quality
by interpolating HMM parameters among some representative speakers’ HMM sets.
The idea of using speaker interpolation has been applied to voice conversion[43].
The proposed method diﬀers from it in that each speech unit is modeled by an
HMM, accordingly mathematically-well-deﬁned statistical distance measures can beused for interpolating HMMs. As a result, the system can synthesize speech withvarious voice quality without large speech database in synthesis phase.
A block diagram of the TTS system based on speaker interpolation is shown in
Fig. 8.1, which is almost equivalent to the previously proposed system except thatmultiple speaker’s HMM sets are trained and a new speaker’s HMM set is generated
by interpolation among them. The procedure can be summarized as follows:69

Training
Speaker 1

Speaker 2

Speaker N

Phoneme
HMMs Λ1

Phoneme
HMMs Λ2

Phoneme
HMMs ΛNInterpolation
weights

a1

a2

aN

Interpolation
Interpolation

Interpolated phoneme HMMs Λ
Synthesis
Sentence HMM

c1 c2

TEXT

cT
Mel-cepstrum

Interpolated
F0

MLSA filter

SYNTHETIC SPEECH

Figure 8.1: Block diagram of speech synthesis system with speaker interpolaiton.

701. Training representative HMM sets(a) Select several representative speakers S1 , S2 , . . . , SN from speech database.
(b) Obtain mel-cepstral coeﬃcients from speech of the representative speakers by mel-cepstral analysis.
(c) Train phoneme HMM sets Λ1 , Λ2 , . . ., ΛN for S1 , S2 , . . . , SN , respectively, using mel-cepstral coeﬃcients, and their deltas and delta-deltas.
2. Interpolation among representative HMM sets(a) Generate a new phoneme HMM set Λ by interpolating among the representative speakers’ phoneme HMM sets Λ1 , Λ2 , . . ., ΛN with an arbitrary
interpolation ratio a1 , a2 , . . . , aN based on a method described in thenext section.
3. Speech synthesis from interpolated HMM(a) Convert the text to be synthesized into a phoneme sequence, and concatenate the interpolated phoneme HMMs according to the phoneme sequence.
(b) Generate mel-cepstral coeﬃcients from the sentence HMM by using speechparameter generation algorithm.
(c) Synthesize speech from the generated mel-cepstral coeﬃcients by using
the MLSA (Mel Log Spectral Approximation) ﬁlter.

8.2

Speaker Interpolation

Figure 8.2 shows a space which represents speaker individuality. Representative
speakers S1 , S2 , . . ., SN are modeled by HMMs, λ1 , λ2 , . . ., λN , respectively. We
assume that representative speaker’s HMMs have the same topology (distributions
could be tied). Under this assumption, interpolation among HMMs is equivalent
to interpolation among output probability densities of corresponding states whenstate-transition probabilities are ignored. If we assume that each HMM state has asingle Gaussian output probability density, the problem is reduced to interpolation
among N Gaussian pdfs, pk (o) = N (o; µk , U k ), k = 1, 2, . . . , N , where µk andU k denote mean vector and covariance matrix, respectively, and o is the speechparameter vector.
We consider three methods to interpolate among pdfs as follows:(a) When we deﬁne the interpolated pdf p(o) = N (o; µ, U ) as pdf of randomvariableo=Nak ok ,

k=1

71

(8.1)

λ1

λ2
I(λ, λ1 )λI(λ, λ4 )I(λ, λ2 )I(λ, λ3 )λ3
λ4
Figure 8.2: A space of speaker individuality modeled by HMMs.
where

N

k=1 ak = 1, the mean µ and variance U is calculated as follows:

µ =
U =

N

k=1
N


ak µk ,

(8.2)

a2k U k .

(8.3)

k=1

(b) We assume that mean µk and covariance U k are trained by using γk feature
vectors of speaker k. If the interpolated pdf p is trained by using feature
vectors of N representative speakers, this pdf p is determined as
N


µ =

γ
N


U =
=

γk µk

k=1

N


ak µk ,γ− µµ

ak (U k + µk µk ) − µµ

k=1respectively, where γ =

(8.4)

k=1

γk U k

k=1
N


=

N

k=1 γk and ak = γk /γ.

72

(8.5)

(c) We assume that the similarity between the interpolated speaker S and each representative speaker Sk can be measured by Kullback information measure between p and pk . Then, for given pdfs p1 , p2 , . . ., pN and weights a1 , a2 , . . . , aN ,consider a problem to obtain pdf p which minimizes a cost functionε=N

ak I(p, pk ),

(8.6)

k=1

that is, we can determine the interpolated pdf p(o) = N (o; µ, U ) by minimizing ε with respect to µ and U , where the Kullback information measurecan be written as
I(p, pk )
=
=


 ∞
−∞

N (o; µ, U ) log

N (o; µ, U )doN (o; µk , U k )|U k |
1
log
+
2
|U |




tr U −1k {(µk − µ)(µk − µ) + U } + I .

(8.7)

As a result, µ and U are determined by
µ =

!N


ak U −1
k

k=1

U =

!N


k=1

"−1! N


,

(8.8)

k=1

"−1

ak U −1k"ak U −1
k µk

,

(8.9)

respectively. The derivation of (8), (9) is shown in Appendix A.

8.3

Simulation

p1(male)
p2(female)Table 8.1: Setting for simulating Japanese vowel “a”
Formant Center frequency (standard deviation) Bandwidthf1
750 Hz (100 Hz)
50 Hz
f2
1100 Hz (130 Hz)
60 Hz
f11050 Hz (100 Hz)
50 Hz
f2
1500 Hz (130 Hz)
60 Hz

In order to check whether these methods generate interpolated distribution appropriately, we evaluated three methods based on simple simulations to avoid a timeconsuming subjective evaluation.
73

o(2)

p1

A
B

p2

C
D
E

o(1)

(a)
o(2)

p1
A
p2

B
C
D

E
o(1)

(b)
o(2)

p1

A

B

C

D
E

p2

o(1)

(c)
Figure 8.3: Comparison between method (a), (b) and (c) with regard to interpolation
between two multi-dimensional Gaussian distributions.
74

Log magnitude (dB)

100

100

100

100

100

80

80

80

80

80

60

60

60

60

60

40

40

40

40

40

20

20

20

20

20

0

0

1
2
3
Frequency (kHz)4(a1 , a2 )= (1, 0)

0

0

1
2
3
Frequency (kHz)40(a1 , a2 )= (0.75, 0.25)

0

1
2
3
Frequency (kHz)4(a1 , a2 )= (0.5, 0.5)

0

0

1
2
3
Frequency (kHz)

4

0

0

(a1 , a2 )= (0.25, 0.75)123Frequency (kHz)4(a1 , a2 )= (0, 1)

Log magnitude (dB)(a), (b)
100

100

100

100

100

80

80

80

80

80

60

60

60

60

60

40

40

40

40

40

20

20

20

20

20

0

0

0

0

0

1
2
3
Frequency (kHz)(a1 , a2 )= (1, 0)

4

0

1
2
3
Frequency (kHz)(a1 , a2 )= (0.75, 0.25)

4

0

1
2
3
Frequency (kHz)(a1 , a2 )= (0.5, 0.5)

4

0

1
2
3
Frequency (kHz)(a1 , a2 )= (0.25, 0.75)

4

0

0

1
2
3
Frequency (kHz)(a1 , a2 )= (0, 1)(c)Figure 8.4: Comparison between method (a), (b) and (c) with regard to interpolation
between two Gaussian distributions p1 and p2 with interpolation ratios A: (a1 , a2 ) =(1, 0), B: (a1 , a2 ) = (0.75, 0.25), C: (a1 , a2 ) = (0.5, 0.5), D: (a1 , a2 ) = (0.25, 0.75),
E: (a1 , a2 ) = (0, 1).
Fig. 8.3 shows Gaussian distributions generated by interpolating between Gaussian
distributions p1 and p2 with two-dimensional diagonal covariances. In this ﬁgure,each ellipse represents the contour of a Gaussian distribution, and each dot represents the mean vector of the distribution. From the ﬁgure, it can be seen that inmethods (a) and (b) the interpolated mean vector is determined irrespective of covariances of representative distributions p1 and p2 . On the other hand, the method(c) can interpolate between two distributions appropriately in the sense that theinterpolated distribution p reﬂects the statistical information, i.e., covariances of p1
and p2 .
By using three interpolation methods, we interpolated between multi-dimensional
Gaussian distributions p1 and p2 . Each distribution is calculated by using 1024
cepstral coeﬃcient vectors which simulate Japanese vowel “a” uttered by male orfemale. Center frequencies, their standard deviations and bandwidths of the ﬁrst
and second formants were determined as shown in Tab. 8.1. Center frequencies are
determined referring to Tab. 3.1 in [44] which shows ranges of male’s and female’s
formant frequency. Standard deviations of center frequencies are estimated from75

4

Tab. 3-5(a)(b) in [45], which shows changes in formant frequency of Japanese vowels
according to diﬀerence of the preceding consonant, Bandwidths are estimated from
Fig. 5.32(a) and Fig. 5.33(b) in [46], which show formant bandwidths and diﬀerenceof male’s and female’s ﬁrst formant bandwidth, respectively. Fig. 8.4 shows spectra
which correspond to mean vectors of interpolated Gaussian distributions. It can beseen that the formant structure of spectra interpolated by the method (a) and (b)are collapsed. On the other hand, the spectra interpolated by the method (c) keep
the formant structure. These results suggest that the method (c) is most appropriate
in the three methods for interpolating among HMMs which model speech spectra,
and we choose the method (c) for the subjective evaluation in the next section.

8.4

Experiments

By analyzing the result of the subjective evaluation of similarity using Hayashi’s
fourth method of quantiﬁcation [47], we investigated whether the quality of synthesized speech from the interpolated HMM set is in between representative speakers’.
In these experiments, we use two representative speakers, since when we use many
speakers, combinations of stimuli increase exponentially in similarity test.
We used phonetically balanced 503 sentences from ATR Japanese speech databasefor training. Speech signals were sampled at 10 kHz and windowed by a 25.6 ms
Hamming window with a 5 ms shift, and then mel-cepstral coeﬃcients were obtainedby mel-cepstral analysis. The feature vectors consisted of 16 mel-cepstral coeﬃcients
including the 0th coeﬃcient, and their delta and delta-delta coeﬃcients. Note thatthe 0th coeﬃcient corresponds to logarithm of the signal gain. We used 5-stateleft-to-right triphone models with single Gaussian diagonal output distributions.
Decision-tree based model clustering was applied to each set of triphone models,
and the resultant set of tied triphone models had approximately 2,600 distributions.
We trained two HMM sets using speech data from a male speaker MHT and a female
speaker FKN, respectively. By using the speech parameter generation algorithm,
ﬁve diﬀerent types of speech were synthesized from ﬁve HMM sets obtained by
setting the interpolation ratio as (aMHT , aFKN ) = (1, 0), (0.75, 0.25), (0.5, 0.5),(0.25, 0.75), (0, 1) (these sound ﬁles can be found in http://kt−lab.ics.nitech.ac.jp/
˜yossie/demo/speaker inter.html). The MLSA ﬁlter was excited by pulse train or
white noise generated according to F0 contours. As the F0 contour for synthetic
speech, we used a F0 contour obtained by linear interpolation between MHT’s and
FKN’s F0 contours extracted from natural speech at a ratio of 1 : 1, where a oneto-one correspondence between MHT’s and FKN’s F0 contour was given by Viterbialignment using spectral information. In each state, the time axis was linearly
expanded or contracted. F0 contours were represented by logarithm of the frequency.
76

0
1 2 3 45
Frequency (kHz)0
1 2 3 45
Frequency (kHz)0
1 2 3 45
Frequency (kHz)0
1 2 3 45
Frequency (kHz)0
1 2 3 45
Frequency (kHz)(aMHT , aFKN )= (1, 0)(aMHT , aFKN )= (0.75, 0.25)(aMHT , aFKN )= (0.5, 0.5)

(aMHT , aFKN )= (0.25, 0.75)(aMHT , aFKN )= (0, 1)Figure 8.5: Generated spectra of the sentence “/n-i-m-o-ts-u/”.
To observe only a change of spectrum, F0 contour was ﬁxed for each sentence.

8.4.1

Generated Spectra

Fig. 8.5 shows spectra of a Japanese sentence “/n-i-m-o-ts-u/” generated from thetriphone HMM sets. From the ﬁgure, it can be seen that spectra change smoothly
from speaker MHT’s to speaker FKN’s according to the interpolation ratio.

8.4.2

Experiment of Similarity

In this experiment, two sentences (Appendix B), which were not included in the
training data, were synthesized. Subjects were eight males. Stimuli consisted of two
samples in ﬁve utterances which were synthesized with diﬀerent interpolation ratios.
Subjects were asked to rate the similarity of each pair into ﬁve categories ranging
from “similar” to “dissimilar”. From the results, we placed each sample in a spaceaccording to the similarities between the samples by using Hayashi’s fourth method
77

1 II
( aMHT , aFKN) =(1, 0)( 0, 1)

I−1( 0.75, 0.25)

01( 0.25, 0.75 )( 0.5, 0.5)

−1Figure 8.6: Subjective distance between samples.
of quantiﬁcation.
Fig. 8.6 shows the relative similarity distance between stimuli. It is observed thatthe ﬁrst axis roughly corresponds to gender. From this ﬁgure, it is seen that a pair ofsamples, whose interpolation ratios were close to each other, were placed closely on a
two-dimensional space. This indicates that the voice quality of interpolated speakerchanges smoothly between those of two representative speakers if interpolation ratio
is linearly changed.

8.5

Discussion

In this thesis, we described an approach to voice quality conversion for an HMMbased text-to-speech synthesis system by interpolation among HMMs of representative speakers. As a result, the system can synthesize speech with various voice
quality without large speech database in synthesis phase. From the results of experiments, we have seen that the quality of synthesized speech from the interpolated
HMM set change smoothly from one male speaker’s to the other female speaker’s
according to the interpolation ratio.
In this thesis, although we choose the method (c) based on the simulation, the78superiority of the method (c) should also be conﬁrmed by the subjective evaluation.
Although the proposed method can be applied to multiple representative speakers,
the subjective evaluation was performed for two representative speakers. Thus, thesubjective evaluation for the interpolation among multiple speakers is also our future
work. We expect that the emotion (e.g., anger, sadness, joy) interpolation might bepossible by replacing HMMs of representative speakers with those of representative
emotions.

79

Chapter 9
Conclusions
In this chapter, we summarize the contributions of the thesis and suggest somedirection for future work.

9.1

Original contribution revisited

The original contributions are summarized as follows. This summary is corresponded
to the “original contribution” list given at the start of this thesis (Section 1.3)
• Speech parameter generation using multi-mixture HMM.
In the previous works, single-mixture HMMs were used. However, the formant
structure of spectrum corresponding to each mean vector might be vague sincemean vector is the average of diﬀerent speech spectra. In chapter 4, we proposed a parameter generation algorithm using multi-mixture HMMs. From theexperimental result, we conﬁrmed that the formant structure of the generated
spectra get clearer with increasing mixtures.
• Duration modeling for the HMM-based TTS system.
In chapter 5, we proposed duration modeling for HMM-based TTS system.
Duration models are constructed taking account of contextual factors that affect durations. From informal listening tests, we found that synthetic speech
had a good quality with natural timing. Furthermore, we conﬁrmed that synthetic speech could keep natural timing even if its speaking rate was changed
in some degree.
• Simultaneous modeling of spectrum, F0 and duration.
In chapter 5, we described an HMM-based speech synthesis system in which80spectrum, F0 and state duration are modeled simultaneously in a uniﬁed
framework of HMM. As a result, it is possible that statistical voice conversion techniques (e.g., speaker adaptation technique, speaker interpolation technique) are applied to the proposed TTS system.
• Training of context dependent HMM using MDL principle.
In chapter 5, the distributions for spectral parameter, F0 parameter and thestate duration are clustered independently by using a decision-tree based context clustering technique based on the MDL principle. By the MDL principle,
decision tree is set appropriate size taking account of amount of data.
• F0 parameter generation using dynamic features.
We conﬁrmed the eﬀect of dynamic feature in the F0 parameter generation inchapter 6. By taking account of dynamic features, very smooth and natural F0
pattern can be generated. From the listening test, synthesized speech, whose
spectra and F0 pattern are generated with dynamic features, is improved thanspeech synthesized without dynamic features.
• Automatic training of the HMM-based TTS system.
The HMM-based TTS system can be constructed automatically using appropriate initial model and speech data without label boundary information. In
chapter 6, actually we did automatic system construction, and conﬁrmed thatsynthesized speech has the same quality with the case of using speech datawith label boundary information.
• Improvement quality of synthesized speech by incorporating mixed
excitation model and postﬁlter into the HMM-based TTS system.
There are many approaches to the improvement of the speech quality for the
HMM-based TTS system. As one of approaches, we tried to incorporate themixed excitation model and the postﬁlter to the system in chapter 7. As aresult, the quality of the synthesized speech could be signiﬁcantly improved.
• Voice conversion using speaker interpolation technique.
For the purpose of synthesizing speech with various voice characteristics suchas speaker individualities and emotions, we proposed the TTS system basedon speaker interpolation in chapter 8. The proposed system synthesizes speechwith untrained speaker’s voice quality by interpolating HMM parameters amongsome representative speakers’ HMM sets. Listening tests show that the proposed algorithm successfully interpolates between representative speakers in
the case where two representative HMM sets are trained by a male and a female speakers’ speech data, respectively; the quality of synthesized speech isin between the male and female speakers’, and can be gradually changed fromone’s to the other’s according to interpolation ratio.
81

9.2

Future works

There are some works as far the proposed TTS system. For example, The following
works are considered:• Improvement of synthesized speech quality.
In Chapter 7, the mixed excitation and postﬁlter were incorporated to thesystem, and they could improve the speech quality. However, this approachis used in the ﬁeld of speech coding in which it is purpose that high quality
speech are obtained with small amount of bit rate. It is considered that there
is some approaches with a large number of speech parameters to improvementof the speech quality. For example, STRAIGHT[48] is a high-quality analysissynthesis method and oﬀers high ﬂexibility in parameter manipulation withno further degradation. If analysis-synthesis method such as STRAIGHT canbe incorporated to the our TTS system, it is considered that the synthesized
speech quality is improved.
• Emotional speech synthesis In Chapter 8, we could change speaker individualities of the synthesized speech using speaker interpolation technique.
Further, we expect that the emotion (e.g., anger, sadness, joy) interpolation
might be possible by replacing HMMs of representative speakers with those ofrepresentative emotions.

82

Bibliography[1] A. Ljolje, J. Hirschberg and J. P. H. van Santen, “Automatic speech segmentation for concatenative inventory selection,” Progress in Speech Synthesis, ed. J.
P. H. van Santen, R. W. Sproat, J. P. Olive and J. Hirschberg, Springer-Verlag,
New York, 1997.
[2] R. E. Donovan and P. C. Woodland, “Automatic speech synthesiser parameter
estimation using HMMs,” Proc. of ICASSP, pp.640–643, 1995.
[3] R. E. Donovan and P. C. Woodland, “Improvements in an HMM-Based Synthesizer,” Proc. of EUROSPEECH, pp.573–576, 1995.
[4] H. Hon, A. Acero, X. Huang, J. Liu and M. Plumpe, “Automatic generationof synthesis units for trainable text-to-speech synthesis,” Proc. of ICASSP,
pp.293–306, 1998.
[5] X. Huang, A. Acero, H. Hon, Y. Ju, J. Liu, S. Meredith and M. Plumpe, “Recent
improvements on Microsoft’s trainable text-to-speech system -Whistler,” Proc.
of ICASSP, pp.959–962, 1997.
[6] R. E. Donovan and E. M. Eide, “The IBM Trainable Speech Synthesis System,”
Proc. of ICSLP, vol.5, pp.1703–1706, Nov. 1998.
[7] A. Falaschi, M. Giustiniani and M. Verola, “A hidden Markov model approachto speech synthesis,” Proc. of EUROSPEECH, pp.187–190, 1989.
[8] M. Giustiniani and P. Pierucci, “Phonetic ergodic HMM for speech synthesis,”
Proc. of EUROSPEECH, pp.349–352, 1991.
[9] T. Masuko, K. Tokuda, T. Kobayashi and S. Imai, “Speech synthesis from
HMMs using dynamic features,” Proc. of ICASSP, pp.389–392, 1996.
[10] T. Yoshimura, K. Tokuda, T. Masuko, T. Kobayashi and T. Kitamura, “Simultaneous Modeling of Spectrum, Pitch and Duration in HMM-Based Speech
Synthesis,” Proc. of EUROSPEECH, S11.PO2.16, vol.5, pp.2347–2350, Sep.
1999.
83

[11] K. Tokuda, T. Kobayashi and S. Imai, “Speech parameter generation from
HMM using dynamic features,” Proc. of ICASSP, pp.660–663, 1995.
[12] K. Tokuda, T. Masuko, T. Yamada, T. Kobayashi and S.Imai, “An Algorithm
for Speech Parameter Generation from Continuous Mixture HMMs with Dynamic Features,” Proc. of EUROSPEECH, pp.757–760, 1995.
[13] T. Masuko, K. Tokuda, T. Kobayashi and S. Imai, “Voice characteristics conversion for HMM-based speech synthesis system,” Proc. of ICASSP, vol.3, pp.16111614, 1997.
[14] M. Tamura, T. Masuko, K. Tokuda and T. Kobayashi, “Speaker Adaptationfor HMM-based Speech Synthesis System Using MLLR,” Proc. of The Third
ESCA/COCOSDA workshop on Speech Synthesis, pp.273–276, Dec. 1998.
[15] T. Yoshimura, K. Tokuda, T. Masuko, T. Kobayashi and T. Kitamura,“Speaker Interpolation in HMM-Based Speech Synthesis System,” Proc. of EUROSPEECH, Th4C.5, vol.5, pp.2523–2526, 25 Sep. 1997.
[16] T. Fukada, K. Tokuda, T. Kobayashi and S. Imai, “An adaptive algorithm formel-cepstral analysis of speech,” Proc. of ICASSP, vol.1, pp.137–140, 1992.
[17] A.V. Oppenheim and R.W. Schafer, “Discrete-time signal processing,”
Prentice-Hall, Englewood Cliﬀs, N.J., 1989.
[18] S. Imai and C. Furuichi, “Unbiased estimator of log spectrum and its application
to speech signal processing,” Proc. of EURASIP, pp.203-206, Sep. 1988.
[19] S. Imai, “Cepstral analysis synthesis on the mel frequency scale,” Proc. of
ICASSP, pp.93–96, Feb. 1983.
[20] T. Kobayashi and S. Imai, “Complex Chebyshev approximation for IIR digitalﬁlters using an iterative WLS technique,” Proc. of ICASSP, pp.377–380, Apr.
1990.
[21] L.A.Liporace, “Maximum Likelihood Estimation for Multivariate Observationsof Markov Sources,” IEEE Trans. Information Theory, IT-28, 5, pp.729–734,
1982.
[22] B.-H. Juang, “Maximum-likelihood estimation for mixture multivariate stachastic observations of Markov chains,” AT&T Technical Journal, vol.64, no.6,
pp.1235–1249, 1985.
[23] U. Jensen, R. K. Moore, P. Dalsgaard, and B Lindberg, “Modeling intonation
contours at the phrase level using continuous density hidden Markov models,”
Computer Speech and Language, vol.8, no.3, pp.247–260, Aug. 1994.
84

[24] G. J. Freij and F. Fallside, “Lexical stress recognition using hidden Markov
models,” Proc. of ICASSP, pp.135–138, 1988.
[25] K. Ross and M. Ostendorf, “A dynamical system model for generating F0 for
synthesis,” Proc. of ESCA/IEEE Workshop on Speech Synthesis, pp.131–134,
1994.
[26] M. Nishimura and K. Toshioka, “HMM-based speech recognition using multidimensional multi-labeling,” Proc. of ICASSP, pp.1163–1166, 1987.
[27] A. Acero, “Formant analysis and synthesis using hidden Markov models,” Proc.
of EUROSPEECH, pp.1047–1050, 1999.
[28] K. Koishida, K. Tokuda, T. Masuko and T. Kobayashi, “Vector quantization of
speech spectral parameters using statistics of dynamic features,” Proc. of ICSP,
pp.247–252, 1997.
[29] W. Tachiwa and S. Furui, “A study of speech synthesis using HMMs,” Proc.
of Spring Meeting of Acoustical Society of Japan, pp.239–240, Mar. 1999 (inJapanese).
[30] S. E. Levinson, “Continuously Variable Duration Hidden Markov Models for
Speech Analysis,” Proc. of ICASSP, pp.1241–1244, 1986.
[31] T. Yoshimura, K. Tokuda, T. Masuko, T. Kobayashi and T. Kitamura, “Duration Modeling in HMM-based Speech Synthesis System,” Proc. of ICSLP, vol.2,
pp.29–32, Nov. 1998.
[32] J. J. Odell, “The Use of Context in Large Vocabulary Speech Recognition,”
PhD dissertation, Cambridge University, 1995.
[33] N. Miyazaki, K. Tokuda, T. Masuko and T. Kobayashi, “A Study on Pitch Pattern Generation using HMMs Based on Multi-space Probability Distributions,”
Technical Report of IEICE, SP98-12, Apr. 1998(in Japanese).
[34] H. J. Nock, M. J. F. Gales and S. J. Young, “A Comparative Study of Methods
for Phonetic Decision-Tree State Clustering,” Proc. of EUROSPEECH, pp.111–
115, 1997.
[35] W. Chou and W. Reichl, “Decision Tree State Tying Based on Penalized Bayesain Information Criterion,” Proc. of ICASSP, pp.345–348, 1999.
[36] K. Shinoda and T. Watanabe, “Speaker Adaptation with Autonomous Model
Complexity Control by MDL Principle,” Proc. of ICASSP, pp.717–720, May
1996.
85

[37] A. V. McCree and T. P. Barnwell III, “A mixed excitation LPC vocoder model
for low bit rate speech coding,” IEEE Trans. Speech and Audio Processing,vol.3, no.4, pp.242–250, Jul. 1995.
[38] W. Lin, S. N. Koh and X. Lin, “Mixed excitation linear prediction coding of
wideband speech at 8kbps” Proc. of ICASSP, vol.2, pp.1137–1140, Jun. 2000.
[39] N. Aoki, K. Takaya, Y. Aoki and T. Yamamoto, “Development of a rule-based
speech synthesis system for the Japanese language using a MELP vocoder,”IEEE Int. Sympo. on Intelligent Signal Processing and Communication Systems, pp.702–705, Nov. 2000.
[40] K. Tokuda, T. Masuko, N. Miyazaki and T. Kobayashi, “Hidden Markov Models
Based on Multi-Space Probability Distribution for Pitch Pattern Modeling,”
Proc. of ICASSP, pp.229–232, May 1999.
[41] K. Tokuda, T. Yoshimura, T. Masuko, T. Kobayashi and T. Kitamura, “Speechparameter generation algorithms for HMM-based speech synthesis,” Proc.
ICASSP, vol.3, pp.1315–1318, June 2000.
[42] T. Yoshimura, K. Tokuda, T. Masuko, T. Kobayashi, “Speaker Interpolation for
HMM-based Speech Synthesis System,” The Journal of the Acoustical Societyof Japan(E), vol.21, no.4, pp.199–206, Jul. 2000.
[43] N. Iwahashi and Y. Sagisaka, “Speech Spectrum Conversion Based on Speaker
Interpolation and Multi-functional Representation with Weighting by Radial
Basis Function Networks,” Speech Communication, 16, pp.139–151, 1995.
[44] S. Saito and K. Nakata, “Fundamentals of Speech Signal Processing,” Ohmsha,
Ltd. pp.36–38, 1981 (in Japanese).
[45] O. Fujimura, “Speech Science(Onsei Kagaku),” University of Tokyo Press,
pp.213–218, 1972 (in Japanese).
[46] J.L. Flanagan, “Speech Analysis, Synthesis, and Perception,” 2nd Edition,
Springer-Verlag, pp.181–184, 1972.
[47] C. Hayashi, “Recent theoretical and methodological developments in multidimensional scaling and its related method in Japan,” Behaviormetrika, vol.18,
pp.1095, 1985.
[48] H. Kawahara, “Speech representation and transformation using adaptive interpolation of weighted spectrum: Vocoder revisited,” Proc. of ICASSP, vol.2,
pp.1303–1306, 1997.

86

List of Publications
Journal Papers
Toshiaki Fukada, Takayoshi Yoshimura and Yoshinori Sagisaka, “Automatic
generation of multiple pronunciations based on neural networks,” Speech Communication, vol.27, no.1, pp.63–73, Sep. 1999
Takayoshi Yoshimura, Keiichi Tokuda, Takashi Masuko, Takao Kobayashi andTadashi Kitamura, “Speaker interpolation for HMM-based speech synthesis
system,” The Journal of the Acoustical Society of Japan(E), vol.21, no.4,
pp199–206, Jul. 2000
Takayoshi Yoshimura, Keiichi Tokuda, Takashi Masuko, Takao Kobayashi andTadashi Kitamura, “Simultaneous modeling of spectrum, pitch and state duration in HMM-based speech synthesis,” IEICE Trans. of Inf. & Syst., vol.J83D-II,no.11,pp.2099–2107,Nov. 2000(in Japanese)

International Conference Proceedings
Takayoshi Yoshimura, Keiichi Tokuda, Takashi Masuko, Takao Kobayashi andTadashi Kitamura, T. Yoshimura, K. Tokuda, T. Masuko, T. Kobayashi andT. Kitamura, “Speaker interpolation in HMM-based speech synthesis system,”
Proc. of EUROSPEECH, vol.5, pp.2523–2526, Sep. 1997
Toshiaki Fukada, Takayoshi Yoshimura and Yoshinori Sagisaka, “Automatic87

generation of multiple pronunciations based on neural networks and language
statistics,” Proc. of ESCA workshop on modeling pronunciation variation forautomatic speech recognition, pp.41–46, 1998

Takayoshi Yoshimura, Keiichi Tokuda, Takashi Masuko, Takao Kobayashi andTadashi Kitamura, T. Yoshimura, K. Tokuda, T. Masuko, T. Kobayashi andT. Kitamura, “Duration modeling in HMM-based speech synthesis system,”
Proc. of ICSLP, vol.2, pp.29–32, Nov. 1998

Toshiaki Fukada, Takayoshi Yoshimura and Yoshinori Sagisaka, “Neural network
based pronunciation modeling with applications to speech recognition,” Proc.
of ICSLP, vol.3, pp.723–726, Nov. 1998

Takayoshi Yoshimura, Keiichi Tokuda, Takashi Masuko, Takao Kobayashi andTadashi Kitamura, T. Yoshimura, K. Tokuda, T. Masuko, T. Kobayashi and T.
Kitamura, “Simultaneous modeling of spectrum, pitch and duration in HMMbased speech synthesis,” Proc. of EUROSPEECH, vol.5, pp.2347–2350, Sep.
1999

Takayoshi Yoshimura, Keiichi Tokuda, Takashi Masuko, Takao Kobayashi andTadashi Kitamura, K. Tokuda, T. Yoshimura, T. Masuko, T. Kobayashi and T.
Kitamura, “Speech parameter generation algorithms for HMM-based speechsynthesis,” Proc. ICASSP, vol.3, pp.1315–1318, June 2000

Takayoshi Yoshimura, Keiichi Tokuda, Takashi Masuko, Takao Kobayashi andTadashi Kitamura, T. Yoshimura, K. Tokuda, T. Masuko, T. Kobayashi andT. Kitamura, “Mixed excitation for HMM-based speech synthesis,” Proc. of
EUROSPEECH, vol.3, pp.2263–2266, Sep. 2001
88

Technical Reports
Takayoshi Yoshimura, Keiichi Tokuda, Takashi Masuko, Takao Kobayashi andTadashi Kitamura, “State duration modeling for HMM-based speech synthesis,” Technical Report of IEICE, DSP98-85, vol.98, no.262, pp.45–50, Sep.
1998(in Japanese)
Takayoshi Yoshimura, Keiichi Tokuda, Takashi Masuko, Takao Kobayashi andTadashi Kitamura, “Simultaneous modeling of spectrum, pitch and state duration in HMM-based speech synthesis,” Technical Report of IEICE, SP99-59,
vol.99, no.255, pp.33–38, Aug. 1999(in Japanese)
Takayoshi Yoshimura, Keiichi Tokuda, Takashi Masuko, Takao Kobayashi andTadashi Kitamura, “Introduction of mixed excitation model and postﬁlter
to HMM-based speech synthesis,” Technical Report of IEICE, SP2001-63,
vol.101, no.325, pp.17-22, Sep. 2001(in Japanese)
Atsushi Sawabe, Kengo Shichiri, Takayoshi Yoshimura, Keiichi Tokuda, Takashi
Masuko, Takao Kobayashi and Tadashi Kitamura, “Application of eigenvoicetechnique to spectrum and pitch pattern modeling in HMM-based speech synthesis,” Technical Report of IEICE, SP2001-72, vol.101, no.325, pp.65-72, Sep.
2001(in Japanese)

Domestic Conference Proceedings
Takayoshi Yoshimura, Tohru Takahashi, Keiichi Tokuda and Tadashi Kitamura, “Word recognition using two-dimensional mel-cepstrum,” Tokai Sec.
Joint Conf. of Seven Institutes of Electrical and Related Engineers ’97, pp250,
Oct. 1996(in Japanese)
Takayoshi Yoshimura, Keiichi Tokuda, Takashi Masuko, Takao Kobayashi andTadashi Kitamura, “Speaker interpolation in HMM-based speech synthesis sys89

tem,” ASJ Autumn Meeting ’97, 1-P-17, I, pp.337–338, Sep. 1997(in Japanese)
Toshiaki Fukada, Takayoshi Yoshimura and Yoshinori Sagisaka, “Automatic
generation of multiple pronunciations based on neural networks and language
statistics,” ASJ Spring Meeting ’98, 2-Q-30, I, pp.179–180, Mar. 1998(inJapanese)
Takayoshi Yoshimura, Keiichi Tokuda, Takashi Masuko, Takao Kobayashi andTadashi Kitamura, “State duration modeling for HMM-based speech synthesis,” ASJ Autumn Meeting ’98, 1-2-8, I, pp.189–190, Sep. 1998(in Japanese)
Takayoshi Yoshimura, Keiichi Tokuda, Takashi Masuko, Takao Kobayashi andTadashi Kitamura, “Spectrum, pitch and state duration modeling for HMMbased speech synthesis,” ASJ Spring Meeting ’99, 2-3-8, I, pp.241–242, Mar.
1999(in Japanese)
Takayoshi Yoshimura, Keiichi Tokuda, Takashi Masuko, Takao Kobayashi andTadashi Kitamura, “Eﬀects of dynamic feature in HMM-based pitch patterngeneration,” ASJ Autumn Meeting ’99, 1-3-16, I, pp.215–216, Sep. 1999(inJapanese)
Takayoshi Yoshimura, Keiichi Tokuda, Takashi Masuko, Takao Kobayashi andTadashi Kitamura, “Evaluation of speech parameter generation from HMM
based on maximum likelihood criterion,” ASJ Spring Meeting ’2000, 1-7-7, I,
pp.209–210, Mar. 2000(in Japanese)
Takayoshi Yoshimura, Keiichi Tokuda, Takashi Masuko, Takao Kobayashi andTadashi Kitamura, “Automatic construction of HMM-based speech synthesis
system,” ASJ Autumn Meeting ’2000, 1-Q-2, I, pp.233–234, Sep. 2000(inJapanese)
Takayoshi Yoshimura, Keiichi Tokuda, Takashi Masuko, Takao Kobayashi andTadashi Kitamura, “A study on Excitation model for HMM-based speech90synthesis,” ASJ Spring Meeting ’2001, 2-6-8, I, pp.297–298, Mar. 2001(inJapanese)
Atsushi Sawabe, Kengo Shichiri, Takayoshi Yoshimura, Keiichi Tokuda, Takashi
Masuko, Takao Kobayashi and Tadashi Kitamura, “Speech synthesis using
triphone based on eigenvoices,” ASJ Spring Meeting ’2001, 2-6-9, I, pp.299–
300, Mar. 2001(in Japanese)
Atsushi Sawabe, Kengo Shichiri, Takayoshi Yoshimura, Keiichi Tokuda, Takashi
Masuko, Takao Kobayashi and Tadashi Kitamura, “Pitch modeling in eigenvoicesbased speech synthesis,” ASJ Autumn Meeting ’2001, 3-2-6, I, pp.315–316,
Oct. 2001(in Japanese)
Takayoshi Yoshimura, Keiichi Tokuda, Takashi Masuko, Takao Kobayashi andTadashi Kitamura, “A study on improvement of speech quality for HMMbased speech synthesis,” ASJ Autumn Meeting ’2001, 1-P-8, I, pp.371-372,
Oct. 2001(in Japanese)

91Appendix A
Examples of decision trees
constructed by using the MDL
principle.

92

C_voiced (phone)

C_voiced (phone)

C_voiced (phone)

no

no

no

yes

C_fricative (phone)
no

yes

yes

yes

no

mora <= 11 (mora)nomora <= 11 (mora)
C_pause (phone)

yes

no

L_"a" (phone)

yes

yes

C_plosive (phone)no

LEAFNODE1

yes
LEAFNODE2

C_palate (phone)

C_plosive (phone)noC_palate (phone)yes

C_"s" (phone)yes

C_"ts|ch" (phone)
C_"h" (phone)noC_silence (phone)noC_pause (phone)noC_"cl" (phone)

yes

LEAFNODE1

yes

R_"a|u|o" (phone)

C_palate (phone)

C_"i|e" (phone)

C_vowel (phone)

C_vowel (phone)

C_"a" (phone)

no

no

no

yes

C_"N" (phone)noC_"N" (phone)noC_nasal (phone)yes
C_"ky" (phone)

yes

yes

yes

R_pause (phone)

C_"a" (phone)noC_"o" (phone)

C_"a" (phone)

yes

no

yes
C_"i|u" (phone)yes
L_"w|f|p|b|m" (phone)for state 1

for state 2

C_silence (phone)noyes

R_silence (phone)

R_"r|z|s|ts|t|d|n|N" (phone)

C_"i|e" (phone)

no

yes

yes

R_silence (phone)noC_plosive (phone)noyes

C_voiced (phone)noC_"r|z|s|ts|t|d|n|N" (phone)

yes

yes
C_palate (phone)

L_part_of_speech_ID == 2(part of speech)
C_"a|u|o" (phone)

C_"a" (phone)

C_"a" (phone)nonoC_"e|o" (phone)yes

C_"a|o" (phone)yes
R_silence (phone)

L_"w|f|p|b|m" (phone)sentence_length <= 61 (sentence)

C_voiced (phone)nono

LEAFNODE1

yes

yes

C_silence (phone)noC_"r|z|s|ts|t|d|n|N" (phone)yes
R_vowel (phone)
LEAFNODE2

C_palate (phone)noR_vowel (phone)yes
C_"y" (phone)for state 4

for state 5

Figure A.1: Examples of decision trees for mel-cepstrum.

93

R_accentual_phrase_length<= 13 (accent)
C_accent_type == 0 (accent)for state 3

R_voiced (phone)

C_voiced (phone)noC_nasal (phone)yes

L_"i|e" (phone)noC_"o" (phone)noC_pause (phone)

C_voiced (phone)

C_"t" (phone)

no

no

no

yes

C_voiced (phone)
no

yes

yes

yes

no

C_unvoiced_vowel (phone)noC_unvoiced_vowel (phone)
C_"h" (phone)

yes

no

C_"h" (phone)

yes

yes

C_unvoiced_vowel (phone)no

LEAFNODE1

yes
LEAFNODE2

R_unvoiced_plosive (phone)

R_silence (phone)noR_silence (phone)yes

C_vowel (phone)no

R_voiced (phone)

yes
LEAFNODE2LEAFNODE1

C_"h" (phone)yes

R_voiced (phone)
C_"ky" (phone)noC_voiced (phone)noC_"ky" (phone)yes

C_part_of_speech_ID == 12(part of speech)
C_vowel (phone)

R_silence (phone)mora <= 3 (mora)

nono

C_"N" (phone)

yes

no

LEAFNODE1

yes
C_"z" (phone)yes
accent_type - mora <= 0(mora)
R_silence (phone)no

LEAFNODE2

L_accent_pause (accent)yes
L_"r" (phone)for state 1

for state 2

C_pause (phone)
no
yes

no

C_"cl" (phone)
no

yes

yes

R_silence (phone)noC_voiced (phone)nofor state 3

R_voiced (phone)

yes

yes

R_fricative (phone)noC_unvoiced_vowel (phone)

C_"i" (phone)yes
R_silence (phone)mora <= 9 (mora)

LEAFNODE2

L_"ch" (phone)noC_"i" (phone)

yes
LEAFNODE1LEAFNODE1

C_voiced (phone)noyesmora <= 8 (mora)
noR_vowel (phone)yes
C_"sh" (phone)
R_accentual_phrase_length <= 13(accent)
C_"i" (phone)noyesmora <= 2 (phone)for state 4

for state 5

Figure A.2: Examples of decision trees for F0.

94

C_vowel (phone)
C_voiced (phone)
no
yes

yes

C_fricative (phone)noyes

C_voiced (phone)
no

no
yes

C_unvoiced_vowel (phone)noaccent_type - more <= 9 (mora)

yes

C_"ts|ch" (phone)noC_vowel (phone)

no

yes

yes

sentence_length <= 49(sentence)
C_fricative (phone)

C_voiced (phone)noyes

C_plosive (phone)noC_silence (phone)

yes

LEAFNODE1

C_"r|z|s|ts|t|d|n|N" (phone)
C_"N" (phone)

L_silence (phone)nono

C_nasal (phone)
C_nasal (phone)

no

yes

C_"r|z|s|ts|t|d|n|N" (phone)yes

C_"y" (phone)yesC_accent_type == 0 (accent)
LEAFNODE2

C_"N" (phone)
L_voiced (phone)

C_vowel (phone)
no
yes

no

R_silence (phone)noC_"i|u" (phone)yes

C_"N" (phone)noyes

C_nasal (phone)

yes

no

R_accentual_phrase_length<= 13 (accent)R_u (phone)yes

R_voiced (phone)noC_"a" (phone)yes

R_palate (phone)

L_voiced (phone)
R_silence (phone)

L_voiced (phone)nonoC_part_of_speech_ID == 36(part of speech)
C_"o" (phone)

C_"a" (phone)

no

yes

L_silence (phone)

yes

yes

L_"r" (phone)
R_silence (phone)accent_type - mora <= 1
(accent)

for state 1

for state 2

C_voiced (phone)
no
yes

no

accent_type - mora <= 9 (mora)noyes

for state 3

C_voiced (phone)yes
L_breath_group_length <= 35(breath group)noR_voiced (phone)

R_voiced (phone)
no

yes

yes

C_unvoiced_vowel (phone)noyes
R_voiced (phone)
C_plosive (phone)

C_silence (phone)nono

C_unvoiced_vowel (phone)yes
C_"r|z|s|ts|t|d|n|N" (phone)noC_"sh" (phone)yes
R_vowel (phone)

R_pause (phone)yes

L_conjugation_form_ID == 2(part of speech)
R_fricative (phone)

R_vowel (phone)noC_fricative (phone)noyes
C_plosive (phone)yes

R_voiced (phone)noR_fricative (phone)yesmora <= 1 (mora)

R_"N" (phone)

C_"a" (phone)

C_palate (phone)nono

C_"i|e" (phone)yes

C_vowel (phone)yes
L_unvoiced_plosive (phone)

C_semivowel (phone)for state 4

for state 5

Figure A.3: Examples of decision trees for bandpass voicing strength.

95

L_accentual_phrase_length <= 13 (accent)

C_voiced (phone)

L_accentual_phrase_length <= 13 (phone)

no

no

no

yes

mora <= 8 (mora)
yesnoC_breath_group_length <= 18(breath group)
R_"d" (phone)

yes

no

L_accent_type <= 9 (accent)yesnoL_part_of_speech_ID == 17(part of speech)
LEAFNODE1

yes

yes

yes

no
yes

L_voiced (phone)nomora <= 8 (mora)
noC_"sh" (phone)

C_fricative (phone)
yes

no
yes

sentence_length <= 59(sentence)
L_breath_group_length <= 21(breath group)
C_"ky" (phone)

no
yes

yes

sentence_length <= 59(sentence)L_accentual_phrase_length <= 13(accent)nomora <= 8 (mora)

C_voiced (phone)

L_breath_group_length <= 23(breath group)
L_breath_group_length <= 21(breath group)

for state 1

no
yes

yes
R_breath_group_length<= 10 (breath group)
LEAFNODE2

L_breath_group_length <= 24(breath group)
C_breath_group_length<= 8 (breath group)
C_"h" (phone)for state 2

C_silence (phone)
no
yes

no

L_accentual_phrase_length <= 13(accent)
mora <= 6 (mora)

yes

no

C_plosive (phone)noyes

L_part_of_speech_ID == 9(part of speech)
C_fricative (phone)noyesmora <= 6 (mora)

yes

C_breath_group_length <= 22(breath group)
LEAFNODE2nomora <= 8 (mora)
yes
LEAFNODE2

mora <= 8 (mora)

C_voiced (phone)no

LEAFNODE1

C_"ts" (phone)yes
C_breath_group_location<= 3 (breath group)
L_accentual_phrase_length <= 11(accent)
no
LEAFNODE1

C_voiced (phone)
no

yes

yes

C_silence (phone)noC_"cl" (phone)yes
L_breath_group_length<= 22 (breath group)
L_breath_group_length <= 23(breath group)noC_breath_group_lengthyes<= 7 (breath group)
C_breath_group_location<= 3 (breath group)

R_conjugation_type_ID == 1
(part of speech)
R_"i|u" (phone)noyes

L_accent_type <= 3
(accent)

for state 4

for state 5

Figure A.4: Examples of decision trees for Fourier magnitude.

96

L_breath_group_length<= 23 (breath group)
L_breath_group_length<= 18 (breath group)

for state 3

no

yes

C_unvoiced_vowel (phone)yes

L_accentual_phrase_length <= 13 (accent)noR_breath_group_length<= 10 (breath group)
LEAFNODE1

C_voiced (phone)
no

no
yes

C_"w" (phone)mora <= 8 (mora)

C_"ts|ch" (phone)

yes

yes

no
yes

R_breath_group_length <= 10(breath group)
LEAFNODE1no

C_part_of_speech_ID == 15(part of speech)
R_silence (phone)noC_silence (phone)noyes

R_silence (phone)noyes

C_"r" (phone)noyes

C_unvoiced_plosive(phone)R_"a" (phone)

R_pause (phone)noL_vowel (phone)yesC_accent_type == 0
(accent)
R_accentural_phrase_length <= 13(accent)noL_"i|u" (phone)
yes

no
yes

L_accentual_phrase_length<= 5 (accent)
LEAFNODE1

C_breath_group_location <= 6
(breath group)noR_"by" (phone)yes
L_"e|o" (phone)

Figure A.5: Examples of decision trees for duration.

97