論文 / 著書情報
Article / Book Information
題目(和文)

音声認識のためのデータ不足問題に対する頑健性の研究

Title(English)

A study on robustness against data insufficiency for speech recognition

著者(和文)

篠田浩一

Author(English)

Koichi Shinoda

出典(和文)

学位:博士(工学),学位授与機関:東京工業大学,
報告番号:乙第3529号,
授与年月日:2001年3月31日,
学位の種別:論文博士,
審査員:

Citation(English)

Degree:Doctor (Engineering),
Conferring organization: Tokyo Institute of Technology,
Report number:乙第3529号,
Conferred date:2001/3/31,
Degree Type:Thesis doctor,
Examiner:

学位種別(和文)

博士論文

Type(English)

Doctoral Thesis

Powered by T2R2 (Tokyo Institute Research Repository)

A Study on Robustness against Data
Insuciency for Speech Recognition
Koichi Shinoda

A Dissertation Submitted to
Department of Computer Science,
Graduate School of Information Science and Engineeringof Tokyo Institute of Technology
in Partial Ful llment of the Requirements
for the Degree of Doctor of Engineering
March 2001

Preface
In statistical pattern recognition, data insuciency often causes serious degradation of recognition accuracy. This problem can be avoided by selecting models
whose sizes are optimal for the given amount of data. In this thesis, a structural
approach is proposed in which numerous models are prepared in the form of atree structure and the model with the optimal size is selected by using information criteria. This approach was applied to two problems of speech recognition:
acoustic modeling and speaker adaptation.
Context-dependent phone units, such as triphones, have recently come to
be used for acoustic modeling in speech recognition systems. While most such
systems cluster the model parameters (e.g., by subword clustering and state clustering) in order to limit the model size and thus avoid poor recognition accuracy
due to a lack of training data, none of them provide e ective criteria for determining the optimal number of clusters. This thesis therefore describes a method
in which the minimum description length (MDL) criterion is used to optimize thenumber of clusters.
Speaker adaptation based on maximum a posteriori (MAP) estimation has
been studied extensively, and this thesis presents a structural maximum a posteriori (SMAP) approach to improve the MAP estimates obtained when the amountof adaptation data is small. The model parameter space is assumed to be structured hierarchically and each model parameter is estimated as the weighted sumof the parameters in more than one tree layer.
These two methods increase the robustness of speech recognition against datai

ii
insuciency, and their e ectiveness was con rmed in a series of recognition experiments.

Acknowledgments
First and foremost, I would like to express my sincere appreciation to my supervisor, Prof. Sadaoki Furui (TITECH, Japan) for his guidance, support andencouragement throughout the years of my Ph.D studentship. He showed me thebest way to put together all the studies I have done in speech recognition. Hiscomments and suggestions have been crucial to the completion of my thesis. I
would also like to thank other members of my thesis committee, Prof. Hidemitsu
Ogawa, Prof. Hozumi Tanaka, Prof. Takao Kobayashi, and Prof. Takenobu
Tokunaga (TITECH, Japan), who have been good enough to give this work avery serious review.
The research reported in Chapter 3 was conducted at Computer and Communication Media Research, NEC Corporation. I would therefore like to express
my sincere gratitude to Dr. Satoshi Goto, Dr. Takao Nishitani, Mr. Masao
Watari, Dr. Kazunori Muraki, Dr. Takao Watanabe, Mr. Kazunaga Yoshida,
Mr. Kaichiro Hatazaki, and Mr. Ken'ichi Iso, for their continuous encouragement
and support. I especially thank Dr. Watanabe and Mr. Iso for their advice. I
also thank Mr. Ryosuke Isotani, Mr. Keizaburo Takagi, and Mr. Tadashi Emori
who were kind enough to discuss problems and issues with me anytime I wished.
Special thanks are also due to Dr. Jun'ichi Takeuchi and Dr. Li Hang. The ideaof applying the MDL criterion to acoustic models was yielded from discussions
with them.
The research reported in Chapter 4 was conducted at Dialogue System Research, Bell Laboratories, Lucent Technologies. Sincere appreciation goes to Dr.
iii

iv
Chin-Hui Lee for giving me the opportunity to work at Bell Laboratories and forproviding much helpful advice with regard to research procedures. He showed metheoretical ways to solve problems, which have been of great help to me in myresearch activities since then. I am also grateful to Dr. Biing-Hwang Juang, Dr.
Frank K. Soong, and Dr. Arun C. Surendran for giving me insightful comments
to my study.
I owe a great many thanks to many people who were kind enough to help meover the course of this work. I would like to express here my sincere appreciation
to all of them. Thanks also go to all the people who read this thesis and mademany helpful comments, though the responsibility for any aws and errors it may
contain remains entirely mine.
Finally, I also would like to express a deep debt of gratitude to my parents,
who instilled in me a love for learning and thinking, and to my wife, Miyuki, forher constant encouragement and support.

Contents

Prefacei

Acknowledgments

iii

1 Introduction

1

1.1 Data insuciency problem : : : : : : : : : : : : : : : : : : : : : :1

1.2 Structural approach : : : : : : : : : : : : : : : : : : : : : : : : : :4

2 Speech Recognition

7

2.1 Overview : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : :7

2.2 Feature extraction : : : : : : : : : : : : : : : : : : : : : : : : : :92.2.1 AD conversion : : : : : : : : : : : : : : : : : : : : : : : : :92.2.2 Short-term spectrum : : : : : : : : : : : : : : : : : : : : :92.2.3 Mel scale : : : : : : : : : : : : : : : : : : : : : : : : : : : :10

2.2.4 Cepstrum : : : : : : : : : : : : : : : : : : : : : : : : : : :10

2.2.5 LPC cepstrum : : : : : : : : : : : : : : : : : : : : : : : : :11

2.2.6 Dynamic features : : : : : : : : : : : : : : : : : : : : : : :12

2.3 Hidden Markov models : : : : : : : : : : : : : : : : : : : : : : : :13

2.3.1 De nition : : : : : : : : : : : : : : : : : : : : : : : : : : :13

2.3.2 Recognition using HMMs : : : : : : : : : : : : : : : : : : :14: : : : : : : : : : : : : : 16
2.3.4 Continuous-density HMMs : : : : : : : : : : : : : : : : : : 19
2.4 Large-vocabulary continuous-speech recognition : : : : : : : : : : 20
2.3.3 Estimation of HMM parametersv

vi

CONTENTS

2.4.1 Subword speech units : : : : : : : : : : : : : : : : : : : : :20

2.4.2 Statistical language modeling : : : : : : : : : : : : : : : :2.5 Data insuciency problem : : : : : : : : : : : : : : : : : : : : : :22
23

2.6 Structural approach in speech recognition : : : : : : : : : : : : : :26

3 MDL-based Acoustic Modeling29

3.1 Motivation : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : 29
3.2 MDL Criterion : : : : : : : : : : : : : : : : : : : : : : : : : : : : 31
3.3 Tree-based state clustering : : : : : : : : : : : : : : : : : : : : : : 33
3.4 Description length for HMMs : : : : : : : : : : : : : : : : : : : : 36
3.4.1 De nition of a model set : : : : : : : : : : : : : : : : : : : 36
3.4.2 Calculation of description length : : : : : : : : : : : : : : 37
3.5 State splitting using the MDL criterion : : : : : : : : : : : : : : : 41
3.6 Experiments : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : 42
3.7 Discussion and summary : : : : : : : : : : : : : : : : : : : : : : : 48

4 SMAP Speaker Adaptation
4.1 Motivation : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : :51
51

4.2 Tree structure : : : : : : : : : : : : : : : : : : : : : : : : : : : : : 56
4.3 Summarization of Gaussian distributions : : : : : : : : : : : : : : 60
4.4 Hierarchical prior : : : : : : : : : : : : : : : : : : : : : : : : : : : 63
4.5 SMAP adaptation using hierarchical priors : : : : : : : : : : : : : 65
4.6 Experiments : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : 70
4.6.1 Summarization : : : : : : : : : : : : : : : : : : : : : : : : 71
4.6.2 Tree clustering : : : : : : : : : : : : : : : : : : : : : : : : 72
4.6.3 Supervised adaptation experiments : : : : : : : : : : : : : 77
4.6.4 Unsupervised adaptation experiments : : : : : : : : : : : :4.7 Discussion and summary : : : : : : : : : : : : : : : : : : : : : : :5 Conclusion
5.1 Contribution of the thesis : : : : : : : : : : : : : : : : : : : : : :

82
85

89
90

CONTENTS

vii

5.2 Future research directions : : : : : : : : : : : : : : : : : : : : : :90

Bibliography

93

A Derivation of Description Length103

B Maximum A Posteriori Estimation

107

Publication List

111

viii

CONTENTS

List of Tables
3.1
3.2
3.3
3.4
3.5

MDL and ML performance. : : : : : : : : : : : : : : : : : : : : : 44
Distributions of questions asked. : : : : : : : : : : : : : : : : : : : 45
Recognition rates (%) for Data A and Data B. : : : : : : : : : : : 46
Recognition rates (%) as a function of coecient c. : : : : : : : : 47
Recognition rates (%) with mixture-Gaussian output pdfs. : : : : 47

4.1
4.2
4.3
4.4

Recognition rates for MIC data when one utterance was used. : : 72
Phone distribution (%) in the rst layer (K = 2) of the tree. : : : 76
Phone distribution (%) in the second layer (K = 3) of the tree. : : 76
Recognition rates (%) of each speaker obtained when using theSMAP method on MIC data. : : : : : : : : : : : : : : : : : : : : 81
Recognition rates (%) of each speaker obtained when using theSMAP method on TEL data. : : : : : : : : : : : : : : : : : : : : 82
Recognition rates (%) obtained with unsupervised adaptation for
MIC data. : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : 82
Recognition rates (%) obtained with unsupervised adaptation for
TEL data. : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : 83
Recognition rates (%) for combining supervised and unsupervised
adaptation. : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : 84

4.5
4.6
4.7
4.8

ix

x

LIST OF TABLES

List of Figures
1.1 Tree structure. : : : : : : : : : : : : : : : : : : : : : : : : : : : :5

2.1 Speech recognition system. : : : : : : : : : : : : : : : : : : : : : :8

2.2 Discrete hidden Markov model (DHMM). : : : : : : : : : : : : : :13

2.3 Forward algorithm. : : : : : : : : : : : : : : : : : : : : : : : : : :15

2.4 Viterbi algorithm. : : : : : : : : : : : : : : : : : : : : : : : : : : :16

2.5 Continuous density hidden Markov model (CDHMM). : : : : : : :19

2.6 A model for \ame" using monophone HMMs. : : : : : : : : : : :21

2.7 A model for \asahi" using triphone HMMs. : : : : : : : : : : : : :22

3.1 The MDL criterion. : : : : : : : : : : : : : : : : : : : : : : : : : :32

3.2 A phonetic decision tree. : : : : : : : : : : : : : : : : : : : : : : :34

3.3 State splitting using phonetic decision trees. : : : : : : : : : : : :35

3.4 Model (node set) in the decision tree. : : : : : : : : : : : : : : : :36

3.5 Viterbi alignment. : : : : : : : : : : : : : : : : : : : : : : : : : : :38

4.1 Recognition performance of MAP, MLLR, and ML. : : : : : : : :53

4.2 Tree structure for Gaussian pdfs in CDHMMs. : : : : : : : : : : :57

4.3 SMAP adaptation for Gaussian pdfs in CDHMMs. : : : : : : : : :66

4.4 Recognition results obtained using di erent tree structures. : : : :73

4.5 The weight for each layer in the tree. : : : : : : : : : : : : : : : :75

4.6 Speaker adaptation adaptation using autonomous model complexity control (TREE adaptation). : : : : : : : : : : : : : : : : : : :

78

xi

xii

LIST OF FIGURES

4.7 Recognition rates obtained with supervised adaptation when the
MIC data were used. : : : : : : : : : : : : : : : : : : : : : : : : : 79
4.8 Recognition rates obtained with supervised adaptation when the
TEL data were used. : : : : : : : : : : : : : : : : : : : : : : : : : 80
A.1 Quantization of parameter space. : : : : : : : : : : : : : : : : : : 104

Chapter 1
Introduction

1.1 Data insuciency problem
In statistical pattern recognition, a model with a number of parameters is prepared and those model parameters are estimated by using data. This process
is called training and the data used for training is called training data. One
might expect that the larger the number of the parameters is, the more accurate
the recognition, since a model with more parameters can represent more features
necessary for recognition. It is often observed, however, that when the numberof model parameters exceeds a certain value, the recognition accuracy becomesworse. This is because there are too many parameters to be estimated preciselyfrom the limited amount of training data. This is called the data insuciency
problem and is inevitable in statistical pattern recognition.
Let me explain this problem with a simple example. Consider the problem ofEnglish-letter speech recognition, in which the utterance of an English letter isassigned to one of the 26 letters in the English alphabet. The pattern recognition
process is roughly divided into two parts: feature extraction and pattern matching. Suppose that the feature-extraction part processes each utterance into one
feature vector with a xed number of dimensions and that the standard pattern
for each category is represented by a Gaussian-mixture distribution in which each1

2

CHAPTER 1.

INTRODUCTION

Gaussian component is multivariate. Suppose too that the number of mixture
components is the same in all the categories and that the recognition accuracy
is measured using test data di erent from the training data. When the numberof mixtures is too small, the recognition accuracy may be very low because thenumber of mixtures is too small to represent the features of the training data.
The recognition accuracy improves as the number of mixtures increases (i.e., as
the model becomes larger), but after the number of mixtures exceeds a certain
level a further increase results in less accurate recognition. This is because themodel is over-trained: a signi cant portion of its parameters represent training
data features unnecessary for recognition of the test data and this noise degrades
the recognition accuracy.
This data insuciency problem appears in many aspects of statistical patternrecognition. To avoid this, we need to obtain a model large enough to represent
the features necessary for recognition but smaller than an over-trained model. Of
models with the same performance, the simplest model is the most preferable,
mainly because it requires less computational resources. The belief that the bestmodel is the simplest has been stated in various ways:

 Pluralitas non est ponenda sine necessitas (plurality should not be posited
without necessity). William of Ockham

 If I had more time I could write a shorter letter.

Blaise Pascal

 Make everything as simple as possible but not simpler.

Albert Einstein

It is dicult, however, to acquire such a parsimonious model.
There have been three di erent approaches to solving this data insuciency
problem, and they all utilize the model selection scheme: a number of models of
various sizes are prepared beforehand and the optimal model is selected. These
approaches are the following: Tuning to test data

1.1.

DATA INSUFFICIENCY PROBLEM

3

 Cross validation
 Information criterion
In the rst one all the prepared models are trained, each is used to recognize data
and the one with the highest accuracy is selected. This approach is implicitly
utilized in most pattern recognition systems since it is simple and needs no extra
information. One signi cant problem with this approach, however, is that theamount of test data is usually limited, and tuning to this limited test data might
result in poor recognition in actual use. The second approach, cross validation,
was developed in response to this problem. In this approach the total data is
divided into several portions and one portion is selected as test data. Then themodel is trained using the rest of the data, and a recognition experiment is carriedout using the test data. This process is repeated using each of the portions as thetest data (i.e., the number of repetitions is the same as the number of portions).
Since the amount of test data seemingly increases, it is expected that the selectedmodel is more robust. The problem of this approach is that its computational
cost is high and there are no established ways to divide the data. The third
approach utilizes an information criterion of model complexity. A number ofsuch criteria have been proposed and the most widely known are the Akaike
information criterion (AIC) [1], the Bayesian information criterion (BIC) [46],and the minimum description length (MDL) criterion [43]. There are a number
of applications of these criteria and some have been proved to be e ective. The
problem with this approach is that it is dicult to apply those criteria directly.
In most case it is necessary to use a number of approximations which are notfully justi ed.
All these approaches based on the model selection scheme share two serious
problems. The rst is that the selected model may be very di erent from atruly optimal model when the number of models prepared is not large enough.
And the second is that they cannot be used in cases for which the amount oftraining data often changed or in which the computational cost for estimating

4

CHAPTER 1.

INTRODUCTION

the parameters of many models is prohibitively high (examples of these cases
will be shown in the next section). The goal is to provide the optimal modelfor the given amount of training data and to do this with a small computationalcost. It may be necessary to prepare a model set comprising numerous models
with di erent sizes, to estimate the model parameters eciently, and to selectthe optimal model without evaluating their performance on test data.

1.2 Structural approach to the data insuciency
problem
The following structural approach to attack the data insuciency problem is
proposed here. First prepare a tree-structured model set in which each layer
in the tree represents a model: to each node in the tree is attached a probabilistic
distribution function (pdf) for data, a node pdf. The root-layer model which
consists only of the root node represents the smallest model, the leaf-layer model
which consists of all the node of the leaf layers represents the largest model, and alayer closer to the root node represents a simpler model. A node in one layer andnodes in its subsidiary layer are connected by branches. Each training sample isassociated with a node in the leaf layer. The pdf of each node is estimated byusing the training samples associated with all the leaf nodes the node governs.
After preparing this tree-structured set of models, select one layer by using acriterion which describes the relationship between the amount of data and thesize of the model.
Let me explain this approach by using an example. One Gaussian distribution
is attached to each node in the tree. The root layer, which consists only of theroot node, represents the simplest model, a single Gaussian distribution. The
leaf layer, comprising all the leaf nodes, represents the most complex model. The
distribution in each node is estimated by using the data associated with the leaf
nodes it governs. In most case, when a certain amount of training data is given,

1.2.

5

STRUCTURAL APPROACH

cut

Figure 1.1: Tree structure.
the root-layer model is too small and the leaf-layer model is too large; the model
represented by one of the intermediate layers is the optimal model. It is easily
understood that models are not limited to the node sets in which all the nodes
belong to the same layer. Any node sets which divide the tree into an upper partand a lower part can be used as a model. Such a node set is called a cut (seeFigure 1.1).
As will be shown later, the problem of model selection can be resolved into
many subproblems of node selection in sub-structures embedded in the tree. The
computational cost of solving these subproblems will be shown to be much smaller
than the computational cost of the greedy approach in which all the possiblemodels have to be trained and evaluated. This subproblem approach is especially
useful in cases where the training cost is high or the amount of training datachanges frequently.
There are three issues important in this approach. First, the root node andthe leaf layer should be designed appropriately. It can be generally assumed thatthe lower and upper bounds on the amount of data available are given. The parameters of the root-layer model can be precisely estimated from the least amountof data, while those of the leaf-layer model should be estimated from an amountof data large enough to represent all the features necessary for recognition. The

6

CHAPTER 1.

INTRODUCTION

second issue is that the way to construct the tree structure should be speci ed.
The tree structure can be made by clustering the leaf nodes hierarchically. The
pdfs for the nodes nearer the root node should represent more global distributionsin the space of data, while those of the nodes nearer to the leaf layer should represent more local distributions. Such trees cannot be made unless the measure of
distance between the node pdfs is de ned appropriately. Finally, the model selection framework needs to be provided: a framework not only providing a method
to resolve the model selection problem into the node selection problem but also
providing a criterion for selecting one node set.
This thesis is organized as follows. Chapter 2 brie y reviews speech recognition and describes the data insuciency problem in speech recognition. For
acoustic modeling it describes a structural approach using the MDL criterion,
and for speaker adaptation it describes a structural Bayes approach. Chapters
3 and 4 describe these approaches in detail. Finally, Chapter 5 concludes thisthesis.

Chapter 2
Speech Recognition

2.1 Overview
Nowadays there are a lot of commercial products using speech recognition, automatic recognition of human speech by machines. Examples are dictation software,call center systems, car navigation systems, educational software for learning foreign languages, video games, and electrical toys. Although the application areasof speech recognition are still limited, it is already regarded as one of the major
tools for the man-machine interface.
Speech recognition has had a long history of research and development, andalmost three decades passed before it was used in commercial products for generaluse. The recent advances in speech recognition have been sped by the following
two factors.

 A probabilistic model, the hidden Markov model (HMM), was developed
and used for speech recognition.

 A large speech database became available.
Of course, it should be noted that the rapid progress of computational hardware(CPUs, memories, etc.) also helped. But although speech recognition is alreadyuseful, there are still a lot of problems to be solved. Some of them are the7

8

CHAPTER 2.

SPEECH RECOGNITION

Word
dictionary

Featureextraction

Grammar

Patternmatching

Recognizedutterance

Speech input(Signal processing)Speakerindependent
pattern

Speaker
adaptedpattern(Enrollment
session )Speaker
adaptation

Figure 2.1: Speech recognition system.
following: Its performance is still much inferior to that of real people.
 Speaker-independent systems, which require no enrollment process for users,
are less accurate than speaker-dependent systems, for which users have toenroll.

 Its performance is degraded under adverse conditions, noisy conditions suchas those provided by telephone lines.

 Its performance is also degraded by changes in speaking style. Speech thatis read is recognized better than spontaneous speech in lectures, meetings,
and conversations.

2.2.

FEATURE EXTRACTION

9

Figure 2.1 shows a typical speech recognition system. The process of speech
recognition is roughly divided into two parts: feature extraction and patternmatching. The input analog signals are rst transformed into digital signals by
analog-to-digital (AD) conversion. Then the features useful for speech recognition
are extracted from the signals. A time series of extracted features is called an
input pattern. Next, the pattern matching between the input pattern and eachstandard pattern is carried out. (One standard pattern for each word in therecognition vocabulary is prepared ahead of time.) The pattern matching yields
the distance between the input pattern and each standard pattern, and the wordcorresponding to the standard pattern closest to the input pattern is selected as
the recognized word.

2.2 Feature extraction
2.2.1 AD conversion
The input analog signals are rst sampled for digital processing. While the frequency range perceptible to human beings is between 20 and 20 000 Hz, the range
including the sound of human speech is only from 50 to 7000 Hz. A sampling frequency of 14 000 Hz is therefore high enough for speech processing. A frequency
lower than 14 000 Hz, however, is sometimes used because of the limitations of
the transmission channel or to reduce computational costs.

2.2.2 Short-term spectrum
Speech research has proved that pitch and formant are important features forhuman perception. In the simplest speech production model, the vocal code
generates excited signals (which correspond the pitch) and the vocal tract (whose
response corresponds the formant) lters the signals. Although it seems thatspeech recognition may be realized by simulating the human speech production
system, for the following two reasons this is not possible with today's technology.

10

CHAPTER 2.

SPEECH RECOGNITIONFirst, it is dicult to observe the real-time movement of the vocal organization.
Second, it is not always possible to estimate the pitch and formant frequency
from the acoustic signals observed.
Instead, in most cases the short-term spectrum is used for speech recognitionin most case. Here it is assumed that speech signals are stationary during a shortperiod, 10-100 ms. Then the spectrum for this period is computed using the fast
Fourier transform (FFT).

2.2.3 Mel scale
The mel scale is de ned in order to take into consideration the human perceptionof pitch. The frequency of 1000 Hz is de ned as 1000 mel, and the frequency
that people feel to be n times as high as 1000 Hz is de ned as (n 2 1000) mel.
It should be noted that this mel scale is di erent from the octave scale used inmusic.
The mel scale can be well approximated by using the following equation:f (mel) =1000
f (Hz)
log10 (
+ 1):
log10 2
1000

(2:1)

This mel scale transform has been often used in speech recognition, mainly
because it emphasizes the lower frequencies more important for the classi cationof phones.

2.2.4 Cepstrum
The cepstrum of a signal is de ned as a Fourier transform of the logarithm of
the signal's power spectrum. It is especially useful when input signals are thesuperposition of excitation signals and linear lters. Speech signals are one suchsuperposition: the excitation signals generated from the vocal code are lteredby the vocal tract response.
Let y(n) be the speech signal at time n, v (n) be the excited signal from the

2.2.

11

FEATURE EXTRACTION

vocal code, and h(n) be the vocal tract response. Theny (n) = v (n) 3 h(n);(2.2)

Y (ejw ) = V (ejw ) 3 H (ejw );(2.3)

log jY (ejw )j = log jV (ejw )j + log jH (ejw )j;(2.4)

where Y (1), V (1), H (1) are the Fourier transforms of y (n), v (n), and h(n).
Then the cepstrum c(k ) isc(k ) = v (k ) + h(k ):(2:5)

Although the dimension of cepstrum is the same as that of time, the term
quefrency is used as the dimension of cepstrum. In the cepstrum, the componentv (k) of the vocal tract response is dominant in the lower quefrency, while thecomponent h(k ) of the vocal code has a strong peak at high quefrencies, whichcorresponds to the pitch frequency and its harmonics
The in uence of the pitch is eciently removed by liftering (analogous toltering in the spectral domain) the components v (k) in the quefrency domain,
and the components representing the vocal tract response are used as the features
for speech recognition.

2.2.5 LPC cepstrum
The cepstral coecients can also be derived by using the linear predictive coding(LPC) method in which speech is modeled as the output of an all-pole lter. The
estimate of speech sample s(n) at time n is approximated as a linear combinationof the past p samples:pXs^(n) = ai s(n 0 i);
(2:6)
i=1

where the coecients ai ; i = 1; . . . ; p are the prediction coecients and are assumed to be constant over the analysis frame.
The prediction error is given bye(n) = s(n) 0 s^(n) = s(n) 0

p

X

i=1

ai s(n 0 i):

(2:7)

12

CHAPTER 2.

SPEECH RECOGNITION

What needs to be minimized is the mean squared error over a segment ofspeech, which is de ned as follows:

En =

X

=

X

e2 (n + m)mm

[s(n + m) 0

(2.8)
p

X

k=1

ak s(n + m 0 k )]2 :(2.9)

Di erentiating En with respect to the coecients ak and setting it to zero,En= 0;
k = 1; . . . ; p;
(2:10)
ak
we obtainXms(n + m 0 k)s(n + m) =

p

X

i=1

ai

X

m

s(n + m 0 i)s(n + m 0 k);

k = 1; . . . ; p:Ifn(i; k) =thenn(k; 0) =s(n + m 0 i)s(n + m 0 k );(2:12)

ak n(i; k);(2:13)

X

m
p

X

i=1

(2.11)

k = 1; . . . ; p:

The LPC coecients ak ; k = 1; . . . ; p are obtained by solving this set of equations,and from these LPC coecients can be derived the LPC cepstral coecients:

c0 = log 2 ;cm = am +
m
01
X

m
01
X

k
ck am0k ;k=1 m(2.14)
1  m  p;kck am0k ;m  p;
k=1 m
where 2 is the gain term in the LPC model.
cm =(2.15)(2.16)

2.2.6 Dynamic features
Dynamic features of the spectrum play an important role in human speech perception, and the delta cepstrum [16] developed to take these features into consideration is de ned as follows:

2.3.

13

HIDDEN MARKOV MODELS

a ii

state ja ijb i (o)...
v1 v2 ... vM

state iFigure 2.2: Discrete hidden Markov model (DHMM).
PK

kc (t + k ):(2:17)
k2
This delta cepstrum and the second derivative feature, delta-delta cepstrum,
are often used in many speech recognition systems and are e ective.
1cn(t) =

n
k =0K
PK
k =0K

2.3 Hidden Markov models
2.3.1 De nition
A statistical approach using hidden Markov models (HMMs) has recently been
widely used for speech recognition(e.g., [41]). In this approach the speech signals
are characterized as outputs from Markov sources. In the recognition of words,
for example, a HMM is assigned to each word in the vocabulary, and for eachutterance the recognized word selected is the one whose HMM is most likely to
produce that utterance.
HMMs are classi ed into three types according to the form of the output
probability density function (pdf) in each state: the discrete HMMs, in which theoutput pdf is discrete; the continuous density HMMs, in which the output pdf

14

CHAPTER 2.

SPEECH RECOGNITION

is continuous; and the semi-continuous HMMs which is the combination of thediscrete HMMs and the continuous-density HMMs.
For the simplicity of explanation, the focus is on the discrete HMMs (DHMM)
for a while (see Figure 2.2). Let O = (o1 ; . . . ; ot ; . . . ; oT ) be an observation
sequence. Let S be the number of states, A = faij g be a set of transition
probability distributions, in which aij is the probability of transition from statei to state j ; let B = fbi (ot )g be a set of the output probability distributions,where bi (ot ) be the output probability of the feature vector ot at state i; let
V = fv1; v2 ; . . . ; vM g be the set of the symbols, where M is the number ofsymbols; and let qt be the state at time t; let q = fq1 ; . . . ; qT g be a state sequencein which qt is the state at time t. Thenbi (ot) = bi (k) = P (ot = vk; jqt = i);i = 1; . . . ; S;

(2:18)

where S is the number of state in the HMM. The initial probability distribution
 = fi g is also de ned, where i is the probability of being state i at time 1:

i = P (q1 = i);i = 1; . . . ; S:

(2:19)The parameter set  = (A; B;  ) is the complete parameter set for the model.
From now on, the focus is on a left-to-right HMM, in which a number of states
form a sequence and from each state only transitions to itself and to the next
state on the right are allowed.

2.3.2 Recognition using HMMs
This subsection shows how the probability of an observation sequence for a given
HMM is calculated. Let O = (o1; . . . ; oT ) be the observation sequence;  =(A; B;  ) be the parameter set of the HMM; and let P (Oj) be the probabilityof O, given the model . In principle, P (Oj) is obtained by adding up theprobabilities of all the possible state transitions, each of which can be expressed
as a path in a two-dimensional plane (see Figure 2.3).

15

HIDDEN MARKOV MODELSHMM state sequence

2.3.

(1)
(2)
(3)

Input data sequencet

Figure 2.3: Forward algorithm.
There are two algorithms that can be used to calculate the probability: the
Forward algorithm and the Viterbi algorithm. In the Forward algorithm theforward probability t (i) is de ned ast(i) = P (o1 ; . . . ; ot ; qt = ij):(2:20)

The forward probability is calculated as follows:1t+1(i) = i bi (o1);(j ) =P (Oj) =

S
X

t

i=1
S
X
i=1

T

1  i  S;

(2.21)(i)aij bj (ot+1 );1  t  T 0 1; 1  j  S; (2.22)

!

(i):

(2.23)

This recursive process nally yields the probability P (Oj).
In the Viterbi algorithm (see Figure 2.4) a maximization process is used instead of the summing procedure used in the Forward algorithm:

1(i) = i bi (o1 );1  i  S;

(2.24)

16

SPEECH RECOGNITION

HMM state sequence

CHAPTER 2.

Input data sequencet

Figure 2.4: Viterbi algorithm.

t (j ) = 1max
( (i)a )b (o );iS t01 ij j t

P 3(Oj) = 1max
 (i):
iS T2  t  T; 1  j  S;

(2.25)(2.26)

Strictly speaking, the probability P 3(Oj) obtained by the Viterbi algorithm isonly an approximation of the probability obtained by the Forward algorithm. The
Viterbi algorithm is often used, however, and it has been proved that the recognition accuracy obtained with the Viterbi algorithm is not signi cantly di erent
from that obtained with the Forward algorithm. In the following, the Viterbi
algorithm is used in the recognition process.

2.3.3 Estimation of HMM parameters
The ideal model parameter set is the one that maximizes the probability of theobservation sequence. While no straightforward way to obtain the optimal parameter set is known, the locally optimized parameter set can be obtained byusing the Expectation-Maximization (E-M) algorithm [11]. This parameter estimation using E-M algorithm is often referred to as the maximum likelihood (ML)

2.3.

17

HIDDEN MARKOV MODELS

estimation.
Let O = (o1 ; . . . ; ot ; . . . ; oT ) be an observation sequence used for parameter
estimation. As in the previous section, the forward probability t (i) is calculated
for t = 1; . . . ; T , i = 1; . . . ; S . Additionally, the backward probability needs tobe calculated. In a manner similar to that in which the forward probability is
calculated, the backward probability t (i) is calculated as follows:

T(i) = 1;1  i  S;

t

(i) =

aij bj (ot+1 ) t+1 (j );

S
X

j =1

t = T 0 1; T 0 2; . . . 1;

(2.27)
1  i  S;

(2.28)

Then let t(i) be the probability of being in state i at time t, which is calculated as(i) = P (qt = ijO; )P (O; qt = ij)= PS
j =1 P (O; qt = j j)(i) (i):(2.29)= PS t t
j =1 t (j ) t (j )Given the model and the observation sequence, let t (i; j ) be the probabilityof being in state i at time t and state j at time t + 1:tt (i; j ) = P (qt = i; qt+1 = j jO; )P (qt = i; qt+1 = j; Oj)=P (Oj)(i)a b (o ) (j )= PS PtS ij j t+1 t+1:
i=1
j =1 t (i)aij bj (ot+1 ) t+1 (j )One can easily see thatt (i) =

SX

j =1

t (i; j ):(2.30)(2:31)In the E-M algorithm the following auxiliary function Q(0 ; ) is maximizedin an iteration process:

Q(0; ) =XqP (O; qj0 ) log P (O; qj);(2:32)

18

CHAPTER 2.

SPEECH RECOGNITION

where the parameter set 0 is the current estimate for the HMM parameter set,and  is the new estimate to be calculated. Since we can express P as

P (O; qj) = q0

Yaqt0 qt bqt (ot );1(2:33)

we can write Q(0; ) as

Q(0 ; ) = Q (0 ;  ) +

S
X
i=1

Qai (0 ; ai ) +

S
X
i=1

Qbi (0 ; bi );(2:34)

where  = f1 ; . . . ; S g, ai = fai1 ; . . . ; aiS g, bi is the parameter vector thatde nes bi (1), andQ (0 ;  ) =Qai (0; ai ) =Qbi (0 ; bi ) =SXP (O; q0 = ij0 ) log i ;

i=1
S X
T
X
j =1 t=1
TX
t=1

P (O; qt01 = i; qt = j j0 ) log aij ;P (O; qt = ij0 ) log bi (ot ):(2.35)(2.36)(2.37)

There are also the following stochastic constraints:
S
X
j =1
S
X

k=1

(2.38)

aij = 1;8i

(2.39)bi (k ) = 1;8i(2.40)j =1
KX j = 1;

where bi (k ) = bi (ot = vk ).
 B ), where
Then the maximization leads to the model estimate  = (; A;
P (O; q0 = ij)i =;(2.41)P (Oj)PT
P (O; qt01 = i; qt = j j)aij = t=1PT;(2.42)
t=1 P (O; qt01 = ij)PTbi = t=1PPT(O; qt = ij) (ot; vk ) :(2.43)
t=1 P (O; qt = ij)Here(ot ; vk ) = 1

if ot = vk

= 0otherwise:(2.44)

2.3.

19

HIDDEN MARKOV MODELS

a ii

b i (o)state ja ijstate iFigure 2.5: Continuous density hidden Markov model (CDHMM).
These re-estimation formulas can be rewritten as follows:

i =

(i);

(2.45)

t=1
PT(2.46)

0PT

t01 (i; j );t=1 t01 (i)PT;ot =vk i (t)bi = t=1:PT
t=1 i (t)aij =(2.47)

Then the newly obtained  is set to 0 and this process is repeated. This
iteration process is stopped when the probability P (Oj) converges.

2.3.4 Continuous-density HMMs
The output pdf of continuous-density HMMs (CDHMMs) is usually a mixture of
the Gaussian distributions (Figure 2.5). In this case, the output probability bj (o)is

bj (o) =
=

M
X
k=1

cjk N (ojjk ; 6jk ))1(2 )n=2j6jk j1=2

(ot 0 jk )T 60jk1 (ot 0 jk );(2.48)

where jk is the mean vector and 6ik is the covariance for the pdf of mixture k
at state j . The mixture weight cjk for each mixture component has the following

20

CHAPTER 2.

constraint:

M
X
k=1

cjk = 1;SPEECH RECOGNITION

1  j  S:

(2:49)The parameters cjk , jk , and 6jk are also estimated using the E-M algorithm:
cjk =
 jk =
6 jk =

PT(j; k);(j; k )
(j; k )o;(j; k )
(j; k )(o 0 )(ot 0 jk )T;(j; k)

t=1 t
PM
t=1 k=1 t
PT
t
t=1 t
PT
t=1 t
P
T
jk
t
t=1 t
PT
t=1 t
PT

(2.50)(2.51)(2.52)

wherec N (ot jjk ; 6jk ):(2:53)(j; k) = t (j ) PM jk
m=1 cjm N (ot jjm ; 6jm )Since it is generally known that CDHMMs perform better than DHMMs, we focus
on CDHMMs from now on.
t

2.4 Large-vocabulary continuous-speech recognition
The previous section reviewed isolated-word speech recognition, in which a standard pattern is prepared for each word in the vocabulary and users utter only
one word at a time. In this type of recognition, it is dicult to increase thenumber of words in the vocabulary because it is necessary to provide training
data for each word. Furthermore, the recognition systems are not easy to use
because it is not natural to speak only one word at a time. People usually utter
phrases (concatenations of words) called continuous speech. This section therefore describes large-vocabulary continuous-speech recognition (LVCSR), in which
the recognition vocabulary usually consists of more than 10,000 words.

2.4.1 Subword speech units

2.4.

LARGE-VOCABULARY CONTINUOUS-SPEECH RECOGNITION

21

ame

/a/

/m/

/e/

Figure 2.6: A model for \ame" using monophone HMMs.
There are many ways to de ne a set of subword speech units so that every word canbe represented by their concatenation. Once such a set is given, a model for each
unit can be de ned and each word model can be represented by a concatenationof such models. Since it is not necessary to prepare training data for every wordin the recognition vocabulary, large-vocabulary speech recognition becomes mucheasier.
A typical example of such a subword unit set is a set of monophones. A modelis prepared for each phone, and every word model is a concatenation of phonemodels (Figure 2.6).
The features of a phone di er greatly from context to context and it is dicult
to model all these di erences by using a single monophone model. To overcome
this problem, context-dependent phones, in which nearby (preceding or succeeding) phones are taken into consideration as the context, have come to be used as
the subword units. In a context-dependent phone set, two units that correspondto the same phone but have di erent contexts are regarded as di erent units.
Examples are diphones, in which the immediately right or the immediately left

22

CHAPTER 2.

SPEECH RECOGNITION

asahi

#-a+s

a-s+a

s-a+h

a-h+i

h-i+#

Figure 2.7: A model for \asahi" using triphone HMMs. For example, the symbol\s-a+h" stands for phone \a" whose preceding phone is \s" and whose succeedingphone is \h".
phone is taken into consideration, and triphones, in which both left and rightphones are taken into consideration (see Figure 2.7).

2.4.2 Statistical language modeling
Let X be the observation and W be a word sequence. In the statistical recognition
process, the word sequence which has the largest probability for the given data
is chosen from all the possible word sequences,^ = argmaxW P (W jX ):W(2:54)

Using Bayes' theorem, we can write:

P (W jX ) =P (X jW )P (W ):P (X )(2:55)

Since P (X ) is the same for all W , Eq. (2.55) is rewritten as^ = argmaxW P (X jW )P (W ):W(2:56)

So far in this chapter, we have dealt with isolated word recognition, in which
it is assumed that each utterance consists of only one word and therefore the2.5.

DATA INSUFFICIENCY PROBLEM

23

recognition vocabulary fW g is a set of isolated words. Additionally it is implicitly
assumed that all words appear with the same probability; the probability P (W )in Eq. (2.56) is the same for all words in the vocabulary.
In LVCSR, however, these assumptions are not realistic. First, a word sequence consists of more than one word, W = fw1 ; w2 ; . . . ; wQ g, where Q is thenumber of words in W . Second, P (W ) di ers among word sequences; some sequences are more likely to appear than others. The statistical model that gives
P (W ) for each word sequence W is called a language model and its parameters
are estimated from a very large linguistic corpus.
The probability P (W ) can be written as

P (W ) = P (w1 w2 . . . wQ )= P (w1 )P (w2 jw1 ) . . . P (wQ jwQ01; . . . ; w2; w1 ):(2.57)The probabilities for all the possible word sequences, are almost impossible toestimate from a limited amount of data. It is therefore often assumed that theprobability of each word is a ected only by the (n 0 1) preceding words:

P (wi jw1; w2 ; . . . ; wi01 ) ' Pn(wi jwi01 ; wi02 ; . . . ; wi0n+1 ):(2:58)

This model is called an n-gram model. Let F (wi ; wi+1 ; . . . ; wi+n01 ) be the numberof occurrences of the word sequence wi wi01 . . . wi0n+1 in the training corpus. Thenthe probability Pn(wi ) for each word wi is calculated as follows:

F (wi ; wi01 ; . . . ; wi0n+1 ):F (wi01 ; . . . ; wi0n+1 )Usually, a bigram in which n = 2 or a trigram in which n = 3 is used.
Pn(wi jwi01 ; . . . ; wi0n+1 ) =(2:59)

2.5 The data insuciency problem in speech
recognition
In speech recognition using CDHMMs, the recognition unit most widely used is
the triphone, in which the right-hand phone and the left-hand phone are taken

24

CHAPTER 2.

SPEECH RECOGNITION

into consideration as the context. The number of triphones is usually very large.
In English, for example, the number of phones is roughly 50, so the number of
triphones is about 503 (some combinations of phones are prohibited by the natureof the language). The number of states for each triphone HMM is usually three,
and the number of mixture components for each state is about ten. Therefore
the number of mean vectors in all the HMMs is 503 2 3 2 10 ' 4; 000; 000.
On the other hand, the amount of training data available is usually less than
100 hours' worth, which amounts to about forty million data frames when theframe interval for the short-term spectrum is 10 ms. Then the number of data
samples per mean vector is 40; 000; 000 4 4; 000; 000 = 10. This number seems
very small for precise estimation of the mean vectors. Furthermore, the training
data is phonetically unbalanced; the number of occurrences of each triphone in
the training data usually di ers greatly from triphone to triphone, and sometriphones do not appear in the training data. From these two reasons, it is clearthat the data insuciency problem greatly a ects the recognition accuracy.
Many clustering methods have been developed in attempts to avoid this datainsuciency problem [34, 3, 31, 22, 73, 68, 74, 12, 37]. They reduce the numberof parameters by grouping some of them. The parameters to be clustered are
triphone units, states, and mixture components. These methods will be outlined
in Section.3.1.
The trigram is often used for the language modeling in LVCSR systems. The
trigrams are usually estimated from a large language corpus, typically comprising
108 words. Even if it is assumed that only 104 kinds of words are mainly used
and the other kinds of words are ignored, the number of kinds of trigrams is
104 2 104 2 104 = 1012. This number is much larger than the number of trigramsin the language corpus, 108. It is clear that the data insuciency problem is also
inevitable in language modeling.
To tackle this problem, investigators have usually taken the following two
approaches:

2.5.

DATA INSUFFICIENCY PROBLEM

25

 Word clustering Smoothing with unigrams and bigrams
The rst approach, word clustering, corresponds to the clustering approach inacoustic modeling: it groups similar words into one group and assumes that words
in the same group have the same probability. The similarity measures usually used
are mutual information [5] and Kullback-Leibler divergence (relative entropy)[39]. Other similarity measures are based on the structure of the language, suchas those using information about parts of speech.
The second approach, smoothing, utilizes bigrams and unigrams, which aremuch less numerous and thus can be more precisely estimated from the sameamount of data. Two methods used in this smoothing approach are the deletedinterpolation method [23] and the back-o smoothing method [28]. The deletedinterpolation method is a kind of cross validation in which the data are divided
into training data and test data; it uses instead of the trigram probability the linear combination of the probabilities of the trigram, the bigram, and the unigram,and then these n-gram probabilities are estimated from the training data and theweighting factors among these n-gram probabilities are estimated so as to maximize the likelihood for the test data. The back-o smoothing method utilizes
Good-Turing estimates [19], which are based on an empirical distribution of thenumber of occurrences of words in a natural language, for smoothing n-grams.
In both acoustic modeling and language modeling, the training procedurerequires much computational time, since the amount of data available and thenumber of model parameters are both signi cantly large. We therefore need away to obtain a model of optimal size without incurring excessive computationalcosts.
The data insuciency problem also appears in adaptation, which is a technique of improving recognition performance by using a small amount of data andmodifying the model parameters under test conditions. A number of utterancesfrom a new user are used in acoustic model adaptation, and some recent writings

26

CHAPTER 2.

SPEECH RECOGNITION

from a new user are used in language model adaptation. Often taken in bothacoustic and language model adaptation is the Bayesian approach (e.g., [30]) in
which the parameters of the initial model are used as a priori knowledge for theestimation of new parameters. Also often taken in acoustic model adaptation
has been the transformation approach, in which some global transformations are
employed and their parameters are estimated (e.g., [35]). Some of the acoustic
model adaptation methods will be introduced in Section 4.1. An example of language model adaptation is the cache model approach, in which the parametersof the initial language model and those estimated from the sentences used by a
new user are interpolated [29]. In the Bayesian approach, only the parameters
corresponding to the data for adaptation are re-estimated. Therefore, as themodel size increases, the adaptation tends to be less e ective. It should also be
noted that the computational cost of the model selection process in adaptation
has to be very small, since the adaptation process should be completed almost
instantaneously in actual applications.

2.6 Structural approach to data insuciency
problems in speech recognition
As mentioned in the previous section, context-dependent phone units such as
triphones have recently come to be used to model units in speech recognition
systems based on the use of HMMs. While most such systems cluster the HMM
parameters (e.g., by subword clustering and state clustering) to control the HMM
size and thus avoid poor recognition accuracy due to a lack of training data, noneof them provide e ective criteria for determining the optimal number of clusters.
This thesis therefore describes a method in which state clustering is accomplishedby using phonetic decision trees and in which the minimum description length(MDL) criterion is used to optimize the number of clusters. Large-vocabulary
recognition experiments show that this method results in recognition more accu-

2.6.

STRUCTURAL APPROACH IN SPEECH RECOGNITION

27

rate than that obtained when the maximum-likelihood estimation is used.
Maximum a posteriori (MAP) estimation is one Baysian approach and has
been used for speaker adaptation in speech recognition systems using hidden
Markov models [30, 18]. When the amount of data is suciently large, MAP
estimation yields recognition performance as good as that obtained using the
maximum-likelihood (ML) estimation explained in Subsection 2.3.3. This thesis
describes a structural maximum a posteriori (SMAP) approach to improving the
MAP estimates obtained when the amount of adaptation data is extremely small.
The model parameter space is assumed to be structured hierarchically and theprobability density functions for model parameters at one level are used as priors
for those of the parameters at adjacent levels. Results of supervised adaptation
experiments using non-native speakers' utterances showed that SMAP estimation
reduced the error rate by half using only three utterances for adaptation, and thatit yielded the same accuracy as conventional MAP and ML estimation when theamount of data was suciently large.
The acoustic modeling using the MDL criterion is described in the next chapter, and the SMAP adaptation method is described in Chapter 4.

28

CHAPTER 2.

SPEECH RECOGNITION

Chapter 3
MDL-based Acoustic Modeling

3.1 Motivation
Over the past few years, extensive studies have been carried out on speakerindependent speech recognition systems that employ continuous density hidden
Markov models. It is well known that in most such systems the use of contextdependent (CD) phone units (e.g., diphones, triphones) rather than contextindependent (CI) phone units (monophones) provides greater recognition accuracy [1-10].
While the large number of CD models employed in a typical system can help
to capture variations in speech data, the amount of available training data islikely to be insucient to support the use of such a large number. Furthermore,
there is great variation in the frequency with which individual CD phone units
can be expected to appear in training data; in most CD phone unit sets, thefrequencies for some units will be so small that they will be unlikely to appear
in training data even when a very large amount of data is provided. Such lack ofdata can seriously degrade speech recognition performance and most recognition
systems using CD models cluster the model parameters to try to alleviate theproblem.
Various clustering methods have been developed for this purpose. One varia29

30

CHAPTER 3.

MDL-BASED ACOUSTIC MODELING

tion among them is the choice of parameter to be clustered: K.F. Lee et al. [34],for example, use subword clustering, Hwang et al. [22] use state clustering, andDigalakis et al. [12] cluster the mixture components of the HMMs with Gaussianmixture state observation densities. There is also variation in the approach toselecting the acoustically-similar parameters to be clustered. One approach is
to use only the acoustic characteristics of the data [73, 68, 22, 12, 37]. Anotherapproach is to utilize a priori knowledge about acoustic similarities (usually represented in the form of decision trees) between the parameters, in addition to theacoustic characteristics themselves [34, 3, 31, 74].
However clustering is performed, the accuracy with which the acoustic similarities are measured will be extremely important. One of the most successful
approaches in this regard is that based on the maximum-likelihood (ML) criterion(e.g., [74]). In this approach, a calculation is made for each parameter cluster inthe model to determine the degree to which the splitting of that cluster would
increase the likelihood of the model's outputting the training data; the cluster
giving the greatest increase is then split. (Here, for the sake of simplicity, only the\splitting" method (i.e., top-down clustering) is considered, but an explanationof the application of ML to bottom-up clustering would be quite similar.)
The diculty with the ML approach, however, is determining when to halt the
splitting process, which could be carried on until the model simply consisted ofa full set of individual, unclustered parameters. Most methods limit splitting by
imposing a threshold value on the increase in the likelihood or on the number ofparameter clusters, but the process required to optimize such thresholds (a seriesof recognition experiments; cross-validation; etc.) is computationally expensive.
In this chapter a new approach that uses the minimum description length(MDL)
criterion [43] for state splitting [55, 58] is proposed. This MDL approach is e ective for deciding when to stop splitting.
The MDL approach is based on an information
This chapter is organized as follows: Section 3.2 brie y reviews the MDL cri-

3.2.

31

MDL CRITERION

terion; Section 3.3 outlines state splitting using a phonetic decision tree; Sections
3.4 and 3.5 explain in detail how the MDL criterion is applied to state splitting;
Section 3.6 describes the results of an experimental evaluation of the proposed
method of state splitting. Finally, Section 3.7 discusses several issues related tothe proposed method.

3.2 MDL Criterion
The MDL criterion [43] has been proven to be e ective in selecting the optimalmodel from among various probabilistic models. It selects the model with theminimum description length for given data. When a set of models f1; . . . ; i; . . . ; I g
is given, the description length li (xN ) for data fxN = x1 ; . . . ; xN g and an underlying model i is given byli (xN ) = 0 log P^ i (xN ) +( )Ki
log N + log I;
2

(3:1)

where Ki is the dimensionality (the number of free parameters) of model i and^ (i) represents the maximum likelihood estimates for the parameters (i) =(1(i) ; . . . ; K(i)i ) of model i. The rst term on the right-hand side of (3.1) represents
the code length for data xN when model i is used as a probabilistic model. This
term is identical to the negative of the log likelihood used in the ML criterion.
The second term is related to the complexity of model i and the number of data
samples, N . The third term is the code length required for choosing model i andis assumed here to be a constant. As a model becomes more complex, the value ofthe rst term decreases and that of the second term increases. The second termworks as a penalty imposed for employing a large model size (see Figure 3.1).
In a comparison among models, the model with the shortest description length
l may be considered the one having the most appropriate size and complexity.
As may be seen in (3.1), the MDL criterion does not need any externally given
parameters; the optimal model for the data is automatically obtained once a setof models has been speci ed. The derivation of the description length is brie y

32

CHAPTER 3.

MDL-BASED ACOUSTIC MODELING-log P^ (ι)(x N )
θ

(ι)

K
log N
2

^

i
Figure 3.1: The MDL criterion.

i

3.3.

TREE-BASED STATE CLUSTERING

33

summarized in Appendix A.
With complex models of the type used in speech recognition, it is often impractical to calculate the description length for all the possible models because
to do so would involve high computational costs. To avoid this, a number ofreasonable assumptions are introduced, which are explained in Section 4.

3.3 Tree-based state clustering
In this section, the outline of the proposed method is described.
For modeling CD phone units, triphones [46] are used, in which a central
phone has left- and right-hand neighbors. Each triphone model is a left-to-right
HMM in which states are placed in a line from the start state to the end state,
and the transitions with respect to a state consist of that to itself and that to thenext state to the right. The output density function for each state is a Gaussian
probability density function (pdf) for which a diagonal covariance is assumed.
All HMMs of triphones whose central phones are the same are assumed to havethe same number of states.
As a clustering scheme, state splitting based on phonetic decision trees [74]is used. An example of the phonetic decision trees for triphones is shown in
Figure 3.2. In the state splitting, those states at the same position in triphone
HMMs having the same central phone are pooled into one set, and one phonetic
decision tree is constructed for each set. Starting from the root node which
represents the whole set, each node from top to bottom splits o into two othernodes representing, respectively, \yes" or \no" answers to such phonetic-context
related questions as: \Is the previous phone unvoiced?" (L-unvoiced?) and \Is
the next phone a fricative?" (R-fricative?) (see Figure 3.3). The MDL criterion
is used to choose the optimal question to be asked at each node and to decide
when to stop splitting. When all splitting has stopped, the pdf parameters ofeach leaf node are copied to the pdf parameters of the triphone states in thecorresponding subset and used for recognition.

34

CHAPTER 3.

MDL-BASED ACOUSTIC MODELING

R- : right-hand context
L- : left-hand context

o-a+s
s-a+e
r-a+p
m-a+i g-a+o
t-a+u d-a+g p-a+d
R-vowel ?

yes

r-a+p p-a+d
d-a+g o-a+s

m-a+i s-a+e
t-a+u g-a+o

R-unvoiced ?

R-front ?
yes

no

m-a+i
s-a+e

t-a+u
g-a+o

L-coronal ?

L-dorsal ?

yes

s-a+e

no

no

yes

no

m-a+i
g-a+o

t-a+u

yes

no

r-a+p
o-a+s

d-a+g
p-a+d

R-fricative ?

L-stop ?

yes

no

o-a+s

r-a+p

no
yes

p-a+d

d-a+g

Figure 3.2: A phonetic decision tree for triphones. Eight triphones are split byphonetic-context related questions.

3.3.

35

TREE-BASED STATE CLUSTERING

R-vowel?
ynR-unvoiced?

R-front?
L-coronal?
y

y

n
n

y

R-fricative? yL-dorsal?

n

n

y

y

n

R-end?
nFigure 3.3: State splitting using phonetic decision trees.

36

CHAPTER 3.

MDL-BASED ACOUSTIC MODELING

S0

S1

S2

Model

U

M=5

S5
S3

S4

ΣS1

dimension: KµS1
Figure 3.4: Model (node set) in the decision tree.

3.4 Description length for HMMs
3.4.1 De nition of a model set
As explained in Section II, the MDL criterion is used to select an optimal modelfrom among a set of various models. Thus, it is rst necessary to prepare themodel set from which that optimal model is to be selected. For speech recognition
using CDHMMs, it is impossible to prepare all the possible models because of thelarge number of possible structures of CDHMMs; the number of subword units
and/or the number of states in each unit, for example, di er among recognition
systems. In this study, the focus is on the clustering of the states in CDHMMs
and constant values are given to those parameters unrelated to state clustering,such as the number of states in a single unit.

3.4.

DESCRIPTION LENGTH FOR HMMS

37

Here a model is de ned as a node set in a phonetic decision tree in whicha Gaussian pdf is assigned for each node. When the root node S0 , which represents the whole set of the triphone states in the tree, is split into M nodes,
S1 ; . . . ; SM , as shown in Figure 3.4, one model U (S1 ; . . . ; SM ) is de ned for thenode set fS1 ; . . . ; SM g. Di erent node sets correspond to di erent models. The
description length for each node set is calculated and the node set with the minimum description length is selected from among various node sets as being the
optimum model.

3.4.2 Calculation of description length
Before clustering is performed, an estimate of each HMM parameter is calculated
by using the Forward-Backward algorithm [41]. Let speech data for training
consist of E examples and each example e be analyzed and represented by a timeseries of feature vectors, foe1; . . . ; oet; . . . ; oeTe g, where Te is the number of data
frames for example e. ML estimates for the Gaussian distribution at state sl canthen be written as:
PE
PT
e
e
e=1
t=1 l (e; t)ot
l = PE PTe
;
(3.2)
e=1
t=1 l (e; t)
PE
PT
e
e
T
e
e=1
t=1 l (e; t)(ot 0 l )(ot 0 l )
;
(3.3)
6l =
P
P
E
Te
e=1
t=1 l (e; t)where l is the mean vector and 6l is the covariance of the Gaussian distribution
at state sl , (ot 0 l )T is the transpose of (ot 0 l ), and l (e; t) is the a posteriori
probability of the data being in state sl at the t-th frame of example e, which is
calculated as follows:l (e; t) l (e; t);(3:4)
l (e; t) = PL
l0 =1 l0 (e; t) l0 (e; t)where L is the total number of all the triphone states in the HMMs, l (e; t) is
the forward probability, and l (e; t) is the backward probability of state sl at thet-th frame of example e.
The rst term on the right-hand side of Eq. (3.1) is the negative of the loglikelihood of a probabilistic model with respect to given data. It is possible to38

CHAPTER 3.

MDL-BASED ACOUSTIC MODELING

Data

HMM
State

Figure 3.5: Viterbi alignment. Each data sample is assigned to one state.
calculate the log-likelihood of the training data for all the possible node sets, but
to do so involves huge computational costs. To reduce these costs, the following
three assumptions are introduced [74]:1. The transition probabilities of HMMs can be ignored in the calculation of
the log-likelihood for a node set.
2. State splitting does not change the frame/state alignment between the data
and the model.
3. The log-likelihood of generating the data for each state is the sum of the
log-likelihoods of generating each data frame, with each log-likelihood being
weighted by the a posteriori probability of the data being in the state.
The third assumption is fully justi ed when the Viterbi algorithm is used forparameter estimation because in this algorithm the posterior probability is either
one or zero (see Figure 3.5).

3.4.

39

DESCRIPTION LENGTH FOR HMMS

Let us next consider the problem of calculating the description length for thenode set U de ned in the previous subsection. All the triphone states pooled
into the set corresponding the root node S0 are renumbered as fs11; . . . ; s1L , . . . ,
m
M
M
m
m
m
sm
1 ; . . . ; sLm , . . . , s1 ; . . . ; sLM g, where fs1 ; . . . ; sl ; . . . ; sLm g is the subset of statesthat are merged into node Sm, and Lm is the number of states in the subset. Using
Eqs. (3.2) and (3.3), the mean vector and the covariance of state sm
l are written
as:
1

m
l

=

6ml =
0lm =

PE

PT

m
e
t=1 l
m
l
P
P
E
Te
m
e=1
t=1 l
e=1

0

Te
E X
X
e=1 t=1ml(e; t)oet

;

(3.5)

e
m T
(e; t)(oet 0 ml )(ot 0 l )
;
0lm

(3.6)

;

(3.7)

where lm (e; t) is the a posteriori probability of the data being in state sm
l at thet-th frame of example e. Then, under the rst and second assumptions, the ML
estimates for the pdf parameters of node Sm are given by [26]
m =

=

6m =
=
m

(e; t) =(e; t)o(e; t)PLmem PE PTt=1 l
tl=1 e=1
PL PE
PT
m
m
e=1 t=1 llPLm m m
l=1 l
l
PL
m m
l
l
PL
m
e
e
T
m PE PTm
m
l=1 e=1
t=1 l
t
l
t
l
PL
m PE PTm
l=1 e=1
t=1 l
PL
m m
m
T
m
m
l=1 l
l
l
l
m m T
PL
m m
l=1 l
Lm
X
m
l
l=1

0

0

;

(e; t)(o 00 (6 + (
0

)(

) )

)(o 0
(e; t)

0(

)(

(3.8)

)

) ;

(e; t);(3.9)(3.10)

where m is the mean vector and 6m is the covariance of the Gaussian distribution
at node Sm. Then, from the third assumption, the approximated log-likelihood
L of node Sm generating data fo1 ; . . . ; oT g is given by

L(Sm ) 'E X
TeX
e=1 t=1= 0m(e; t) log( q

1
e
t 0 e
e0 (ot 0m) 6m (ot 0m) )(2 )K j6mj
12

1m (e; t)(K log(2 ) + log j6m j
e=1 t=1 2
Te
E X
X

1

40

CHAPTER 3.

MDL-BASED ACOUSTIC MODELING+(oet 0 m)t 60m1 (oet 0 m))1
= 0 0m(K + K log(2 ) + log j6mj);
2

0m =

T
X
t=1

m

(t);

(3.11)(3.12)

where K is the dimensionality of the data vector oet , and 0m is the total state
occupancy count at node Sm, which is the sum of m (e; t) over all data frames of
all the examples. The log-likelihood of the data for all the nodes in set U is:

Lall =

M
X

L(Sm)

m=1
M
X

' 0

1
0m(K + K log(2 ) + log j6mj):
m=1 2(3.13)The second term on the right-hand side of Eq. (3.1) represents the complexityof a model. In the proposed approach, it is assumed that the covariance of each
Gaussian pdf is diagonal. The number of parameters to be estimated for model U
is 2KM (with model U containing M mean vectors and M diagonal covariances).
The total number of data samples is the sum of 0 (Sm ) over m. With this total,
the second term may be approximated as:

R = KM log W;(3.14)

P

where W = M
m=1 0m . As has been previously noted, the third term on theright-hand side of (3.1) is xed at a constant value, C , for all possible models.
Finally, using (3.13) and (3.14), the description length for model U is calculated as follows:l(U ) = 0Lall + R + C
MX
1'
0m(K + K log(2 ) + log j6mj)
m=1 2+KM log W + C:

(3.15)

3.5.

STATE SPLITTING USING THE MDL CRITERION

3.5 State splitting using the MDL criterion

41

In order to get an optimal model, it is needed to calculate description lengths for
all possible models, which would involve prohibitively high computational costs.
Instead, an algorithm that obtains only a suboptimal solution is used.
Let us rst assume that node Sm of model U splits into two nodes Smqy and
Smqn ,in response to question q , and then let 1m(q ) be the di erence between
the description lengths after the splitting and before it (i.e., l(U 0 ) 0 l(U )). The
description length of model U 0 is:l (U 0 ) =10m0 (K + K log(2 ) + log j6m0 j)
m0 =1;m0 6=m 2
1+ 0mqy (K + K log(2 ) + log j6mqy j)2
1+ 0mqn(K + K log(2) + log j6mqnj)
2+K (M + 1) log W + C;
M
X

(3.16)

where the number of nodes for U 0 is M + 1, 0mqy is the state occupancy countfor node Smqy , and 0mqn is that for node Smqn. The di erence 1m(q ) will thenbe given by the following equation:
1m (q) = l (U 0) 0 l(U )1= (0mqy log j 6mqy j +0mqn log j 6mqn j
200m log j 6m j) + K log W:(3.17)In state splitting, the question q 0 which would minimize 10 (q0 ) when used
to split root node S0 is rst determined. If 10(q 0 ) > 0, then no splitting is
conducted. If 10(q 0 ) < 0, then node S0 is split into two nodes, Sq0 y and Sq0 n, andthe same procedure is repeated for each of these two nodes. This node splitting
is carried out until there remain no nodes to be split and is conducted for theroot nodes of all the phonetic decision trees in all the HMMs.
For the purpose of comparison, let us also consider here the ML approach [74].
Letting m(q) be the increase in the log-likelihood when node Sm is split into two

42

CHAPTER 3.

MDL-BASED ACOUSTIC MODELING

in response to using question q ,m(q ) = L(Smqy ) + L(Smqn ) 0 L(S )1= 0 (0mqy log j 6mqy j +0mqn log j 6mqn j
200m log j 6m j ):(3.18)In the ML approach, question q 0 which would maximize 0(q 0) is rst chosenfrom among all the questions, and then it is used to split root node S0 into two
nodes S0q0 y and S0q0 n . This splitting process will continue until stopped by someexternally given parameters used to control the number of clusters, since theincrease  is positive in all the splitting. Most methods apply a threshold value
to the total occupancy count 0m and/or to the log-likelihood increase m(q ).
However, the optimization of these parameters requires a series of recognition
experiments which are computationally expensive and require additional data.
The MDL approach needs no external control parameters; the term K log Win (3.17) corresponds to the threshold for increase  in (3.18), and this term
is estimated automatically on the basis of the training data. Additionally, thethreshold term K log W is speci ed for each phone in the MDL approach, while
the threshold delta is shared among all the phones. This indicates that the MDL
approach is more robust against the data imbalance among phones than the ML
approach.

3.6 Experiments
The proposed method was evaluated in experiments testing the recognition of
5000 Japanese words. Each utterance was digitized at a sampling rate of 16
kHz, and analyzed in 10-ms frame periods. The analysis yielded a vector of
21 components (a power derivative, 10 mel-scaled cepstral coecients, and 10
corresponding mel-scaled cepstral time derivatives). 37 Japanese CI phones wereused, from which 4309 triphones were derived. The number of states in each
HMM was set to four. A Gaussian output pdf with a diagonal covariance was3.6.

EXPERIMENTS

43

assumed for each state. The number of questions used in the node splitting was
106. Two data sets, Data A and Data B, were prepared for training. Data A
consisted of 250 phonetically-balanced words uttered by each of 46 male speakers.
Data B consisted of 2150 phonetically-balanced words uttered by each of 36 malespeakers. Speech data from ve other male speakers, none of whom was involvedin the production of Data A or Data B, was used for the evaluation tests. Each ofthese test speakers uttered 250 words. None of the words in the test vocabulary
were used in the training vocabulary.
Table 3.1 shows recognition results obtained with the proposed MDL method(averaged for the ve test speakers) and those obtained with the ML approach
when Data A was used for training. The various results for the ML approach reect the di erent values used for two thresholds, D and V , for the state occupancy
count and for the increase in likelihood, respectively.
Let us consider a speci c instance in the ML approach. For a node, S , thealgorithm determines the set of questions for which neither of the resulting two
response nodes would have a \total occupation count" less than or equal to D,
and for which  (q ) would be larger than V . It determines the question among
this set for which  is largest and used it to split node S .
Table 3.1 shows the results of the ML approach for 17 combinations of D
and V (ML 1{17). As shown in Table 3.1, the proposed method achieved higher
recognition accuracy than any using the ML approach. In each instance, thecomputational cost required for the proposed method was roughly the same asthat with the ML approach. In order to determine a model of a size optimalfor the amount of training data, however, the ML approach must be performed
repeatedly over a range of parameter values. Therefore, the total computational
cost is much less with the MDL approach proposed here.
Table 3.2 shows the frequency of the questions used, summed over all thephonetic decision trees of all the HMMs, when the MDL approach was employed.
\L-begin" corresponds to the question, \Is the phone located at the beginning of44

CHAPTER 3.

MDL-BASED ACOUSTIC MODELING

Table 3.1: MDL and ML performance.

DV# of Recog.
nodes rate(%)

MDL

{

{

2069

80.4

ML 1

60

0

3739

75.4

ML 2

100

0

3000

76.4

ML 3

200

0

2001

76.7

ML 4

300

0

1943

75.4

ML 5

400

0

1200

73.4

ML 6

500

0

1018

71.9

ML 7

1000

0

591

66.6

ML 8

60

200

2777

76.2

ML 9

60

400

2034

77.0

ML 10

60

600

1488

77.8

ML 11

60

800

1248

77.9

ML 12
ML 13

60 1000
200 200

1071
1782

77.4
77.3

ML 14

200

400

1533

77.2

ML 15

200

600

1326

77.0

ML 16

200

800

1142

77.8

ML 17

200 1000

1049

76.5

3.6.

45

EXPERIMENTS

Table 3.2: Distributions of questions asked.
Vowel
ConsonantL-coronal
L-dorsalL-beginR-coronal
L-hL-backL-sonorantR-dorsal
L-unvoiced
L-n
L-fricative
Total

67
55
40
39
35
34
33
31
27
27
27

L-begin130
L-back69
R-a
63
R-high62
L-high60
L-a53
L-front
45
R-e
34
R-back32
L-e
30
L-consonant 281110 Total

822

46

CHAPTER 3.

MDL-BASED ACOUSTIC MODELING

Table 3.3: Recognition rates (%) for Data A and Data B.
Training set Data A Data B
No. of nodes

2069

6223

Male 1

72.8

84.8

Male 2

76.8

84.4

Male 3

89.2

92.4

Male 4

81.6

83.6

Male 5

81.6

84.8

Average

80.4

86.0

the word?". Questions related to left phones were used more often than thoserelated to right phones. For a consonant, the question most frequently used waswhether or not it was located at the beginning of a word.
Let us next consider how the optimal model size changes as the amount ofdata increases. Table 3.3 shows results for Data A and Data B. Data B, which
is seven times larger than Data A, resulted in roughly a threefold increase in thenumber of nodes.
In order to evaluate the optimality of this size, A weight coecient c wasadded to the second term on the right-hand side of (3.15), which results in:M
1X
00
l (U ) =0m(K + K log(2 ) + log j 6m j)

2 m=1+cKM log W + C:

(3.19)As c increases, so does the penalty for a large model size. Table 3.4 shows results
for a range of c values of from 0.1 to 10.0. While the highest recognition accuracy
was achieved for a c value of 2.0, it was only 0.7% higher than that for c = 1 (i.e.,
for the case in which the penalty for increased mode size is the same as that in(3.15), which expresses the description length in the proposed approach.)
Data B was also used to evaluate the recognition performance when single-

3.6.

47

EXPERIMENTS

Table 3.4: Recognition rates (%) as a function of coecient c.
c0.1
0.5 1.0 2.0 4.0 10.0
No. of nodes 13927 9798 6223 3949 2418 1341
Male 1

84.0

84.4

84.8 83.6 82.4

79.6

Male 2
Male 3

81.6
92.0

83.6
92.0

84.4 84.4 84.8
92.4 92.8 92.4

80.8
91.2

Male 4

84.8

85.2

83.6 85.2 84.8

82.0

Male 584.4

84.4

84.8 87.6 85.2

86.8

Average

85.4

85.9

86.0 86.7 85.9

84.1

Table 3.5: Recognition rates (%) with mixture-Gaussian output pdfs.
1 Gauss 2 Gauss
Male 1

84.8

86.8

Male 2

84.4

87.6

Male 3

92.4

94.4

Male 4

83.6

88.8

Male 5

84.8

87.2

Average

86.0

89.0

48

CHAPTER 3.

MDL-BASED ACOUSTIC MODELING

Gaussian output pdfs were replaced with mixture-Gaussian output pdfs. In thisexperiment, the number of Gaussian pdfs assigned to each state was increased totwo, and the model was re-trained using the same training data. The increase inrecognition rates shown in Table 3.5, indicates that further splitting of some of thenodes in models constructed using the MDL criterion might result in improved
recognition rate if a better set of questions were prepared beforehand. Such a set
might include, for example, questions regarding second-to-left and/or second-toright phones, characteristics of individual speakers, recording conditions, etc.

3.7 Discussion and summary
A signi cantly useful method of optimizing the model size without using any
externally given parameters was proposed. In an evaluation test it resulted inrecognition more accurate than that obtained when a conventional approach wasused and it had a much lower overall computational cost.
In the real-world application of speech recognition, the model size often mustbe small enough to operate in real time using limited hardware(CPU, memory).
It would seem that this MDL approach is not applicable to this case because no
hardware limitations are taken into consideration. However, this approach canbe applied to this case by controlling the weight of each data sample in training
data. For example, when the weight is assumed to be halved, the amount of data
for each state is halved. Although one control parameter (the weight of each data
sample) has to be introduced, the advantage of our approach in that the model
size for each phone is controlled individually still remains valid.
A number of problems remain to be solved, however. First, the degree towhich the assumptions implicit in the proposed method a ect its performancewith regard to the control of model size has to be determined. A second problem
is that the set of models provided beforehand may not include the most optimalmodel (\true model") for the given data. A third problem is that, since it isassumed that the amount of data is suciently large in the MDL criterion, it

3.7.

49

DISCUSSION AND SUMMARY

may not apply to the case where the amount of data available is small. Theselatter two problems are of course true not only for the proposed method but
also for other model selection strategies using the MDL criterion, and further
theoretical research addressing these problems is needed. A fourth problem isthat the minimization of the description length does not necessarily minimize
recognition error. Conventional ML approaches encounter the same problem:
maximization of likelihood does not necessarily minimize recognition error. The
MDL criterion has an advantage over the ML criterion in that it has an e ective
penalty used for model size control, one that has good theoretical support.
Two other widely known information criteria used for controlling model size
are the Bayesian information criterion (BIC) [45] and the Akaike information
criterion (AIC) [1]. The formula for the BIC isKi
log N:(3:20)
2Comparing this criterion with the MDL criterion (Eq.3.1), one can easily seethat the rst and the second terms are identical and that the only di erenceis that the MDL criterion has a third term. Since throughout this thesis thethird term is assumed to be constant, the BIC gives exactly the same results
as the MDL criterion here. After the result of our research was rst published[55], the approach using the BIC to control the model size in speech recognition
was extensively studied [6, 7, 8, 70]. It has been successfully applied to speakerclustering [6], Gaussian mixture modeling [7], modeling of mixture of Gaussian
pdf for HMM [8], and segmentation of speech data [70]. Since the BIC gives
exactly the same results as the MDL criterion does, the results of these studies
strongly support the e ectiveness of our approach. They also proved that ourapproach can be applied to many other data insuciency problems in speech
recognition.
In the AIC, the second term in (3.1) is replaced by Ki and there is no third
term:
liAIC (xN ) = 0 log P^ i (xN ) + Ki :(3:21)
liBIC (xN ) = 0 log P^ i (xN ) +
( )

( )

50

CHAPTER 3.

MDL-BASED ACOUSTIC MODELING

Practically speaking, it is well known that in many applications the results givenby the AIC di er little from those given by the MDL criterion. The MDL criterion and the AIC are therefore not compared in this thesis. In theory thedi erence between the MDL criterion and the AIC is still controversial but it is
not discussed here because it is not an important issue here. One typical claim
supporting the MDL criterion is that the AIC tends to overestimate the numberof parameters needed [47]; while the AIC is likely to select the correct model
when the complexity of the true model grows with sample size [48, 49], such acase is unlikely to happen in actual applications.
Recently a new information criterion was proposed: the subspace information
criterion (SIC) for model selection from the functional analytical viewpoint [64].
Since it was reported to work well even when the number of training examples issmall, it may be promising to apply this criterion to acoustic modeling.

Chapter 4
Structural MAP Approach to
Speaker Adaptation

4.1 Motivation
Automatic speech recognition systems using continuous density hidden Markov
models (HMMs) have been recently used in various applications. Speaker-independent(SI) systems are typically constructed using speech samples collected from many
speakers. It has been reported, however, that the performance of SI HMMs isoften degraded when there is a mismatch between the training and testing environments. For example, when the acoustic characteristics of a new speakerare very di erent from those of the speakers in the training data, the recognition
accuracy for the new speaker might be far below the average accuracy. Other major adverse conditions causing mismatches are those due to di erent microphones,
channels, and noise environments.
Many techniques compensating the degradation caused by mismatches havebeen developed. They are roughly grouped into two categories, namely: (1)feature compensation (e.g., [32]), in which the process of feature extraction ismodi ed; and (2) model adaptation (e.g., [18, 35]), in which the parameters ofrecognition models are adjusted. Although combining these two techniques has
51

52

CHAPTER 4.

SMAP SPEAKER ADAPTATION

been shown e ective (e.g., [44]), the discussion is focused on model adaptation in
the present study . It is desirable that adaptation improves speech recognition
accuracies even when little adaptation data is given and more importantly it yields
performance equal to or better than that obtained using maximum likelihood (ML)estimation when enough data is available. Few methods, however, achieve bothobjectives.
The most popular approach to model adaptation is through Bayesian formulation. For example, maximum a posteriori (MAP) estimation algorithms (e.g.,[30, 18]) have been widely adopted recently and successfully applied to speaker
adaptation. In this method the model parameters are regarded as random variables whose joint prior probability density function (pdf) is assumed. The MAP
estimate of the parameter vector is de ned as the mode of the posterior pdf
given the adaptation data. The improvement obtained with MAP estimation issigni cantly larger than that obtained with ML estimation, especially when theamount of adaptation data is small. It is well known, since MAP estimates are
asymptotically equivalent to ML estimates, that the resulting recognition performance is similar to that of speaker-dependent (SD) HMMs when the amount ofdata becomes large. A quasi-Bayes approach [20] has also been adopted to handle
on-line MAP adaptation. In these conventional MAP estimation methods, HMM
parameters of di erent speech units are often assumed to be independent. Therefore, each model can be adapted only if the corresponding speech unit has been
observed in the current set of adaptation data. The improvement is consequently
rather small when the amount of adaptation data is extremely limited. The MAP
estimation is brie y explained in Appendix B.
Another category of adaptation techniques, which do not use the MAP framework, are often referred to as transformation-based approaches, such as cepstrum
mean normalization (CMN) [2], signal bias removal (SBR) [42], maximum likelihood linear regression (MLLR) [35], spectral interpolation [50, 51, 52], vector
eld smoothing (VFS) [36], stochastic matching (SM) [44], nonlinear stochastic

53

MOTIVATION

Recognition Accuracy

4.1.

speaker-dependent

MAP
MLLR
ML

speaker-independent

Amount of Data for Adaptation
Figure 4.1: Recognition performance of maximum a posteriori (MAP) adaptation, maximum likelihood linear regression (MLLR), and maximum likelihood
estimation (ML).

54

CHAPTER 4.

SMAP SPEAKER ADAPTATION

matching [65] and predictive adaptation [66]. This family of techniques limits thenumber of free parameters by tying the HMM parameters or by applying someconstraints on the parameters in order to improve recognition accuracies with asmall amount of data. When the amount of adaptation data exceeds a certain
value, however, the recognition accuracy often becomes inferior to that obtainedwith ML estimation of the model parameters. This is because a model with asmall number of free parameters could not fully utilize the potential information
embedded in the large amount of data. In Figure 4.1, the di erence between therecognition performance of MAP and that of MLLR is shown.
Because the traditional MAP approaches and the transformation-based methods are not capable of either improving recognition accuracy when little data are
available or exploiting the information in a large amount of data, several algorithms supplementing those techniques have been developed. The extended MAP(EMAP) method [63, 75], and the quasi-Bayes technique with correlated mean
vectors [21] are extensions of the traditional MAP approaches. They increase therecognition rates obtained with a small amount of data by taking into account thea priori knowledge in the correlation between the parameters modeling di erent
speech units. For example, the pair-wise correlation between the mean vectors
could be used to enhance estimation of the mean parameters of some speech units
even if they are not directly observed in the adaptation data and therefore therecognition rates are signi cantly improved [21]. Although these methods are intheory quite general, they need to impose some approximation in practice because
it is dicult to estimate such correlations precisely when the amount of training
data is small. In [63, 75], for example, the model parameter space was divided
into several subspaces, the ideal number often depends on the amount of adaptation data available. It is also possible to extend the known ML techniques, such
as MLLR to incorporate the MAP estimation criterion. The recently proposed
maximum a posteriori linear regression (MAPLR) [60] algorithm improves MLLR
in a way similar to MAP enhancement over ML for HMM parameter estimation.

4.1.

MOTIVATION

55

Combinations of MAP and transformation-based approaches have also beenstudied intensively ([13, 9, 67, 69]). Notable examples were in combining MLLR
and MAP [13] and combining MAP and VFS [67, 69]. Chien et al. [9] reported
that signi cantly better recognition accuracy can be obtained by combining MAP
and SM with an iterative process. The shortcoming of these combined methods isagain the use of xed structures, i.e. xed ways of parameter tying, in the acoustic
space. Therefore they have only been shown useful with adaptation data sizes
within a narrow range. To alleviate this problem, a tree structure has been usedin adjusting the number of layers in a tree and the degree of parameter tying
according to the amount of available data (e.g., [53, 54]).
In this study, the nice asymptotic property of MAP estimation for large size
adaptation is taken advantage of and the exible parameter tying strategy in atree for small sample adaptation and formulate a novel structural Bayes adaptation framework that achieves the two desired objectives mentioned earlier. By
assuming that the prior knowledge in a tree node can be used to construct priordensity needed for MAP estimation of all the parameters in the successive childnodes, a new structural maximum a posteriori (SMAP) algorithm [56, 57, 59] isintroduced for speaker and environment adaptation.
Three key steps are required in formulating the proposed SMAP approach.
They are described in the next three Sections. First, a tree with a uniform
structure is needed to characterize the acoustic space represented by the HMM
parameters. In this study an information-theoretical criterion is used to cluster all the Gaussian mixture component densities typically used to model state
observation densities in HMM. This procedure is discussed in detail in Section
4.2 Next, given all the density clusters used to characterize nodes in a tree, it isneeded to nd a Gaussian density to summarize all the Gaussian components inthe cluster so that the likelihood of a sequence of observation vectors representing
the adaptation data can be evaluated at the node level and therefore the MAP
estimate at any node in the tree can be computed. In Section 4.3, a summarizing

56

CHAPTER 4.

SMAP SPEAKER ADAPTATION

procedure that simpli es the preparation for SMAP estimation is introduced andproves to be e ective for speech recognition. For the third step, the prior density
at each tree node needs to be de ned. In order to use every observation sample toestimate all the HMM parameters, a hierarchical prior evolution approximation
is used by assuming that the hyperparameters charactering the prior density ateach node are evaluated based on the knowledge embedded in the prior densityof its parent node. This process is explained in Section 4.4. Once the three key
steps are established, the SMAP estimation algorithm is then derived in Section
4.5.
The proposed SMAP approach was evaluated on the RM (Resource Management) task [40]. Training/adaptation and testing utterances by non-native
speakers were collected over two di erent acoustic conditions, a desktop microphone and a telephone handset through dial-up lines. The e ectiveness of
the SMAP algorithm was demonstrated in a set of supervised and unsupervised
adaptation experiments. The ways to combine fast supervised adaptation andon-line unsupervised adaptation to achieve a sucient recognition accuracy inreal applications were also investigated. The experimental results with di erent adaptation scenarios in these adverse conditions is reported in Section 4.6.
Finally, the ndings are summarized in Section 4.7.

4.2 Tree structure
The de nition of a structure to aid MAP estimation is a key procedure in the
proposed structural Bayes approach. In this study a tree structure is adopted
because it o ers a natural evolution of prior knowledge embedded in the parentchild relationship between nodes at di erent tree layers (see Figure 4.2). Thereexists many ways to generate such a tree that models a structure of the acoustic
space of interest.
Given the set of all the mixture Gaussian components in the set of HMMs, it isneeded to rst de ne a distance measure, d(m; n), between Gaussian components,

4.2.

57

TREE STRUCTURE

Tree Structureof Gaussian Component

Clustering using
k-means algorithm(KL-Divergence)Continous Density HMM

Figure 4.2: Tree structure for Gaussian pdfs in CDHMMs. For simplicity, thecase when the dimension is one (scalar) is shown.

58

CHAPTER 4.

SMAP SPEAKER ADAPTATION

gm(1) and gn(1), in order to build a tree. Here the distance is de ned as the sumof the Kullback-Leibler divergence from gm(1) to gn (1) and that from gn(1) to gm (1)[71]. When diagonal covariance matrices are assumed, the distance d(m; n) is
evaluated as follows:
Z
gm(x)
g (x)
d(m; n) = gm(x) log
dx + gn (x) log n dx;
gn(x)
gm(x)
X  2 (i) 0  2 (i) + ( (i) 0  (i))2
n
m
n
=
[ m
2
n(i)
i
2
2
 (i) 0 m(i) + (n (i) 0 m(i))2+ n];
m2 (i)Z(4.1)

where m(i) is the i-th element of the mean vector m and m2 (i) is the i-th
diagonal element of the covariance matrix 6m. Next, at each node k in a tree
structure, the collection of Gaussian components belonging to node k , fgm(k) (X ) =N (X jm(k); 6m(k)) : m = 1; . . . ; Mk g, is approximated by a single Gaussian pdf,
where Mk is the number of Gaussian components at node k. This pdf is called anode pdf. When it is assumed that the number of data samples from each mixture
components are equal, the parameters for the node pdf are calculated as follows:
Mk
Mk
1 X
1 X
E (x(mk)(i)) =(k) (i);Mk m=1Mk m=1 m
Mk
1 X
E ((x(mk)(i) 0 k (i))2 )k2 (i) =Mk m=1
2
3
Mk
Mk
X
X
1 4
(k )2
=
m2(k) (i) +m(i) 0 Mk 2k (i)5 ;Mk k=1m=1

k (i) =

(4.2)

(4.3)

where x(mk) is a data vector from Gaussian pdf gm(k) .
The following clustering algorithm is used to construct a tree structure for the
mixture components in G, where the distance is calculated using Eq. (4.1) andthe node pdf is calculated using Eqs. (4.2) and (4.3).
1. The structure of the tree structure is designed; the number of layers and thenumber of branches from a node in each layer are determined. There is no

4.2.

59

TREE STRUCTURE

clear way to design the structure automatically since the optimal structure
may be changed according to the size of models.
2. Set the root node to be node k and the set G to be set Gnow . Calculate thenode pdf for the root node using Eqs. (4.2) and (4.3).
3. If node k has no child nodes, stop clustering. Otherwise, give the initial pdf
for each child node using the minimax method that is described as follows.
Here g(k) (1) is the node pdf for node k , Pk is the number of child nodes ofnode k, and g (cp ) (1) is the node pdf for child node cp , p = 1; . . . ; Pk .
(a) Choose among the set Gnow the mixture component m^ that has thelargest distance to g (k) (1) and set it as node pdf for child c1, i.e.,g(c )(1) = gm^ (1).
1(b) Choose mixture components for cp successively from p = 2 to p = Pk
and set those to the node pdfs for child nodes as follows:m^ = argmaxm 1mind(m; ci );
ip01

(4.4)

g(cp ) (1) = gm^ (1):

(4.5)

In Eq.(4.4), m^ is chosen from the rest of mixture components, which
belong to parent node k and not yet assigned any child node.
(c) The node pdf for each child node cp and the node pdf for k is interpolated and resulting pdf are set to be the node pdf for cp as follows:

0cp (i) = (1 0 )k (i) + cp (i)
0

(4.6)
0

c2p (i) = (1 0 )(k2 (i) + 2k (i)) + (c2p (i) + 2cp (i)) 0 c2p (i);(4.7)

where 0 

 1.

4. Repeat the following k -means procedure until the grand sum of distances
converges.

60

CHAPTER 4.

SMAP SPEAKER ADAPTATION(a) For each mixture component in Gnow , calculate the distance from it
to each child node pdf by using Eq. (4.1), and assign each mixture
component to the nearest child node.
(b) Recalculate the child node pdf by using Eqs. (4.2) and (4.3).
(c) Using Eq. (4.1), calculate the sum of distances from each child node
to each of its mixture components and then obtain the grand sum of
distances by summing up the sum of distances over all the child nodes.
5. Set each child node to be node k and its corresponding subset of mixture
components Gnow . Go to Step 3.
It is mostly expected that the substantial number of training samples assigned
to each mixture component during the training process is largely di erent fromcomponent to component. This is because the phonetic distribution of speech isusually far from uniform. This imbalance is not taken into consideration in thismethod; it is assumed that the amount of the training samples for each component is the same among all the mixture components. Therefore, this method is
expected to be robust against the imbalance of the number of training samples.
Additionally, it is also applied when the phonetic distribution of training data
and and that of test data are largely di erent. It should be noted that when thenumber of training samples is the same for all mixture components this method
gives the same result as the clustering method based on maximum-likelihood
criterion(e.g., [26]).

4.3 Summarization of Gaussian distributions
In this study, the focus is on adaptation of the parameters of the mixture Gaussiancomponents in continuous-density HMMs (CDHMMs). Let gm(1) be a normal
density function for mixture component m, N (xjm; 6m ), where m is a mean
vector, 6m is a covariance matrix, and x is a D-dimensional observation vector.

4.3.

SUMMARIZATION OF GAUSSIAN DISTRIBUTIONS

61

Let G = fgm(1) : m = 1; . . . ; M g be the whole set of the mixture components in
HMMs, where M is the total number of mixture components in all the states of all
the speech units. It is assumed that all the parameters of the general CDHMMs
have already been trained by using a sucient amount of training data frommany speakers. Such models are typically used for speaker independent speech
recognition.
Let X = fx1; . . . ; xT g denote a given set of adaptation data. We are interested
in using X to obtain estimates of the parameter set, m = (m; 6m). Furthermore, we would like to make use of the tree structure established in Section 4.2 toconstruct prior information in each tree node for SMAP adaptation. Since in thecase of CDHMM, we don't know the membership of xt , i.e. we have no precise
knowledge which Gaussian component could have generated the observed vector.
We usually associate with xt a membership function, mt , which is the posterior
probability of observing xt in the Gaussian component gm(1) given the parameter values m. mt is usually computed with the well-known forward-backward
algorithm (e.g., [41, 18]).
One way to make the problem easier is to go through a summarization process.
As the rst step of the process, each sample vector xt is transformed into a vector
ymt for each mixture component m as follows:

ymt = 60m = (xt 0 m );1 2

t = 1; . . . ; T; m = 1; . . . ; M:

(4:8)

When there is no mismatch between the training data and the adaptation data,
the pdf for Y m = fym1; . . . ; ymT g is obviously the standard normal distribution
N (Y j~0; I), where ~0 is a zero vector and I is an identity matrix. When there is amismatch between them, however, the pdf for Y is di erent from N (Y j~0; I) for
the adaptation data.
Here the pdf for Y is assumed to be N (Y j ; 9), where  6= ~0 and 9 6= I
represent the shift and rotation needed to compensate for the distortion. It is
expected that this pdf for Y better represents the di erence between the acoustic characteristics of the training data and those of the adaptation data. This

62

CHAPTER 4.

SMAP SPEAKER ADAPTATION

is called the normalized pdf. It is also assumed that the mismatch can be modeled by models simpler than those used for speech recognition. In other words,
the number of the normalized pdfs required to model the acoustic di erence isassumed smaller than M , the number of the mixture components of the HMMs.
Let us consider the case when the whole set of mixture components, G, is divided into subsets fG1 ; . . . ; GP g, where P is the total number of subsets, which isless than M , the total number of mixture components. One common normalized
pdf, h(p) (1) = N (Y j (p); 9(p)), is shared by all the mixture components in eachsubset Gp . In the following explanation, all the mixture components in subset(p)(p)
Gp are renumbered as g1(p)(1); . . . ; gm(p)(1); . . . ; gM
is the number
p (1), where MPPpof the mixture components in subset Gp . Obviously, Pp=1 M
m=1 1 = M . For the
mixture component, gm(p)(1), the observed vector sequence X = fx1 ; . . . ; xT g is(p)normalized to Y = fym(p1) ; . . . ; ymT
g.
( )

( )

The maximum likelihood (ML) estimates for the parameters of the normalized
pdfs can be calculated using the expectation-maximization (EM) algorithm (e.g.,[41]). If the transition probabilities and the weight coecients are assumed to be
xed, the auxiliary function Q(1j1) for the HMM parameters [41, 18] is given by =Q(2j2)

p

( )

T X
P M
X
X

t=1 p=1 m=1(p)mt

log gm(p)(xt j(mp); 6(mp) );(4:9)

where 2 = fm(p) = ((mp) ; 6(mp)) : m = 1; . . . ; M (p); p = 1; . . . ; P g is the new esti = fm(p) = ( (mp); 6 (mp)) : m = 1; . . . ; M (p); p =mate of the HMM parameters and 2(p)1; . . . ; P g is the current estimate of the HMM parameters. The parameter mt
isthe posterior probability of observing mixture component gm(p)(1) at time t. The
relation between the original pdf and the normalized pdf is as follows:(p)g (xt j(mp); 6m
) =

(p)
h(ymt
j (p) ; 9(p))

jJmp jh(ymtp j p ; 9 p )=;j(6 mp ) = j
( )

( )

( )

( ) 1 2

( )(4.10)

 (mp))1=2 is the Jacobian matrix for the normalization transformation
where J(mp) = (6

4.4.

63

HIERARCHICAL PRIOR

in Eq. (4.8). This relation can be used to modify the auxiliary function as follows:
(p)
h(ymt
j (p) ; 9(p)) :(4:11)j(6 (mp))1=2j
t=1 p=1 m=1~ (p)), are calculated,
The ML estimates of the parameters, ( (p); 9(p)) = (~ (p); 9
by di erentiating this equation, as follows: =Q(2j2)~(p)=

9~ (p) =

p

( )

T X
P M
X
X

(p)

mt logy ;PM (p) (p) (p)t=1 m=1 mt mtPT
PM (p) (p)t=1 m=1 mt
PT
PM (p) (p)
(p)
 (p)
mt
t=1 m=1 mt
PT
PM (p) (p)t=1 m=1 mt
(p)
(p)
mt 
PT

(y

(4.12)

0 ~ )(ymtp 0 ~ p )T ;
( )

( )

(4.13)(p)where (ymt
0 ~ (p) )T is the transpose of (y 0 ~ ). These normalized pdf parameters are used to estimate the corresponding HMM parameters by the following
transformations, (mp))1=2 ~ (p) ;~ (mp) =  (mp) + (6

6~ mp = 6 mp = 9~ p (6 mp = )T ;( )( )1 2( )( )1 2(4.14)(4.15)~ (mp) are the updated ML estimates of the mean and covariance
where ~ (mp) and 6of the m-th component, respectively.
Let us compare this normalization technique with the stochastic matching(SM) algorithm [44] for compensating mismatch during speech recognition. As
 (mp) )1=2 ~ (p) corresponds to the bias incan be seen from Eqs. (4.14) and (4.15), (6 (mp) is the covariance for mixture component gm(p) (1), and 9~ (p) correSM, where 6~ (p).
sponds to the scaling factor in SM when the diagonal covariance is used for 9
In the proposed method, the bias for each mixture component changes according
to the variance. When the variance is large, the bias is also large. Experimental
results to compare the two compensation and normalization approaches will begiven later.

4.4 Hierarchical prior
The tree structure representation of the set of Gaussian mixture components
have been developed in Section 4.2. How to construct a node normalized pdf to64

CHAPTER 4.

SMAP SPEAKER ADAPTATION

approximate a collection of heterogeneous Gaussian pdfs in the node cluster, andhow to obtain maximum likelihood estimate of the mean and the covariance ofthe normalized pdf have been also shown in Section 4.3. Next, the framework ofhierarchical priors is established.
As discussed before we are interested in estimating the parameter set, 2 =fm = (m; 6m ) : m = 1; . . . ; M g, based on a small set of adaptation data,
X = fx1; . . . ; xT g. Let the normalized Gaussian pdf, h(kp) (1), at the p-th node of
the k -th layer of the tree, be assigned as the parent node of the m-th mixture
component gm(1) by clustering the set, G = fgm(1)g. Then, the parameters mcan be estimated with the transformations in Eqs. (4.14) and (4.15) using theML estimates shown in Eqs. (4.12) and (4.13).
Since the normalized pdf is only an approximation to gm(1) under mismatch
conditions, we expect a better estimate of m, which is directly connected to the~ (Kp)01 ),K -th layer (or leaf layer), to be obtained with an estimate, ~ (Kp)01 = (~ (Kp)01 ; 9at the immediate parent node of gm(1). To establish a hierarchical prior framework for MAP estimation, we also extend the argument and assume that any
normalized pdf, hk(p) (1), at the k-th layer of the tree is better characterized bysome prior information from the immediate parent nodes at the (k 0 1)-st layer.
One easy way to accomplish this is to assume the prior density for estimatingk is based on some knowledge about k01. An even stronger constraint is to
assume a prior density of the form p(k j^k01), i.e. the hyperparameters for theprior densities are simply derived from some estimate ^ k01 of k01 .
Now for each k = 1; . . . ; K , with a given estimate ^k01 . the MAP estimate
^k is evaluated as follows:^k = argmaxk p(k j^k01 ; Y )p(Y jk ; ^k01 )p(k j^k01 )= argmaxk:p(Y )(4.16)

Note ^ 0 = 0 = (~0; I) is the known parameter assumed at the root of the tree.
By further assuming that p(Y jk ; ^k01 ) does not depend on ^k01 and since p(Y )4.5.

SMAP ADAPTATION USING HIERARCHICAL PRIORS

65

is not a function of k , we have^ k = argmaxk p(Y jk ) 1 p(k j^ k01);

k = 1; . . . ; K:

(4:17)

It has been already assumed in Section 4.3 that p(Y jk ) is a normal density.
To make MAP estimation in Eq. (4.17) more tractable, it is assumed that the
conjugate prior density for the random vector k , p(k j^ k01), to be a normalWishart density of the form (e.g., [10, 18]),^ k 01 ;  k ;  k ) /g ( k ; 9k j^ k01 ; 9




k
1 ^
0
(k 0D )=2
T 01
0
1
j9k j
exp 0 ( k 0 ^ k01 ) 9k ( k 0 ^ k01) exp 0 tr(9k019k ) ;2
2(4.18)
for k = 1; . . . ; K , with k > 0 and k > D 0 1 being the control parametersspeci ed by external constraints with D being the dimension of the observation
vector. We now arrive at a familiar MAP solution [18]. By performing this MAP
estimation sequentially at each layer k , we obtain the MAP estimate ^ (Kp), at thep-th cluster of the leaf layer K . Then the MAP estimate, ^m, for each Gaussian
mixture component can be solved. This novel estimation formulation is describedin the following.

4.5 SMAP adaptation using hierarchical priors
Consider for the set G of all the Gaussian mixture component, we have available atree structure like the one shown in Figure 4.3, where K is the total number of layers or the depth of the tree. Each node in the K -th layer (leaf node) corresponds
to one Gaussian mixture component in the set of CDHMMs. The root node (therst layer) corresponds the whole set G of the mixture components. Each intermediate node corresponds to a subset of G, and each of its subordinate leaf nodes
corresponds to an element of a subset. At each node in the tree, a normalized pdf,
which is shared among the mixture components in the corresponding subset ofG, is assigned. For each node N , at the p-th cluster of the k -th layer, Eqs. (4.12)

66

CHAPTER 4.

SMAP SPEAKER ADAPTATION

1
2

Ψ1

layer 1(root)

ν112

Ψklayer k

νk
1
2

ΨK

layer K
(leaf)

νK
Σmµm

Figure 4.3: SMAP adaptation for Gaussian pdfs in CDHMMs. For simplicity,
the case when the dimension is one (scalar) is shown.

4.5.

SMAP ADAPTATION USING HIERARCHICAL PRIORS

67

and (4.13) are used to calculate the ML estimates of the pdf parameters, ~ (kp) and9~ (kp) (in this case, Gp is the subset related to node N ). In the tree structure, one
node sequence from the root to a leaf corresponds to all the predecessor nodes
that must be traversed to reach a particular mixture component. From now on
the focus is on estimation of the parameter set, m = (m ; 6m), for a particularm-th mixture component in G. The sux identifying the mixture component
is therefore omitted except when doing so causes confusion. The procedure described below is general and can be used to estimate the parameter sets of all theother mixture components in CDHMMs.
Let the node sequence from the root to the leaf corresponding to the m-th
mixture component be fN1 ; . . . ; Nk ; . . . ; NK g, where N1 is the root node and NK
is the leaf node directly attached to mixture component m. We denote k =( k ; 9k ) as the Gaussian pdf parameters for node Nk . Now consider the problemof estimating, the parameter set K that maximizes the posterior probabilities,
after observing a sequence of feature vectors Y . It should be noted that once K is
obtained, the parameter set 2 can be obtained immediately by using Eqs. (4.14)
and (4.15). In the proposed approach, a set of priors, fp(k j^ k01)g, are used ashierarchical priors for estimating K , where 0 is xed to be ^0 = N (~0; I). Basedon the discussion in Section 4.4 the pdf for node Nk , which has the parameter
set, k , is assumed to have a hyperparameter, ^ k01, directly extended from its
immediate parent node, Nk01.

Since p(k jY ) = p(k jk01; Y )p(k01 jY ) dk01, for k = 1; . . . ; K , the posterior distribution for K is then expressed as follows:Rp(K jY ) =ZZ. . . p(K jK 01 ; Y ) . . . p(k jk01; Y ) . . .
. . . p(1 j0 ; Y )p(0 jY )d0 d1 . . . dK 01 ;

with

Z

p(0 jY )d0 = 1;

(4.19)(4:20)

68

CHAPTER 4.

SMAP SPEAKER ADAPTATION

because 0 is assumed to be known.
Because Eq. (4.19) is dicult to maximize directly, a key step here is to
assumeZZp(1 j0 ; Y )p(0 jY )d0 ' p(1 j^0 ; Y );p(k+1jk ; Y )p(k j^ k01; Y )dk

' p(k j^ k ; Y );k = 1; . . . ; K 0 1 ;(4.21)+1(4.22)

where ^k is the MAP estimate obtained in Eq. (4.17). The posterior distributionfor K is thus approximated as follows:p(K jY ) '

KY
01
k=0

p(k+1 j^ k ; Y ):(4:23)

Under these assumptions the MAP estimates in Eq. (4.17) for each node Nk
are calculated. First, the ML estimate for each node k is calculated. the auxiliary
function Q(1j1) for solving ^k is written as follows:
T

X X
Q(k jk ) =t=1 m2Gkmth(y j )log  mt 1=k2 + C;
j(6m ) j(4:24)

 m is known and not a function
where C is the part that is independent of k , 6of k and Gk is a subset of G corresponding to node Nk . We also de ne 0k =PTP
t=1 m2Gk mt , with the membership function mt being evaluation at the given
value of k . Then by excluding the constant term C , the MAP estimates arecalculated by maximizing the following auxiliary function with respect to k :R(k jk ) = Q(k jk ) + log g(k );

T X 
X
1
1
01 ( y 0  ) T +
=
(
0
log
j
9
j
0
(
y
0

)
9
mt
k
k
mt
k
k
2
2 mt
t=1 m2Gk 0D
log j9k j 0 k ( k 0 ^ k01 )90k 1 ( k 0 ^ k01)T
[0 k
2
2
1 ^
(4.25)
0 2 tr(9k01 90k 1))];
where Q(k j k ) and g (k ) are de ned in Eq. (4.24) and the relation in Eq. (4.18).
By di erentiating Eq. (4.25) and setting its result to zero, the MAP estimates,

4.5.

SMAP ADAPTATION USING HIERARCHICAL PRIORS

69^ k ), are calculated as follows: for k = 1; . . . ; K ,( k ; 9k ) = (^ k ; 90 ~ + k ^ k01
^ k = k k;(4.26)
0k +  k
9^ k01 + 0k 9~ k + kk+00kk (~ k 0 ^ k01 )(~ k 0 ^ k01 )T^
9k =;(4.27)(k 0 D ) + 0k
^ 0 = I at the root node and ~ k and 9~ k are ML estimates shownwith ^ 0 = ~0 and 9^ K for the leaf nodein Eqs. (4.12) and (4.13). The mean ^ K and the variance 9

NK are obtained by applying Eqs. (4.26) and (4.27) successively from the root
node to the leaf node. To obtain the approximate MAP estimate, these ^ K and
9^ K values in Eqs. (4.26) and (4.27) are rst assigned as ^ (p) and 9^ (p) according
to their corresponding cluster membership and then used to replace the ~ (p) and9~ (p) values shown in Eqs. (4.14) and (4.15), i.e., (mp))1=2 ^ (p) ;^ (mp) =  (mp) + (66^ mp = 6 mp = 9^ p (6 mp = )T ;( )( )1 2( )( )1 2(4.28)(4.29)(p)^ (mp) are the updated MAP estimates of the mean and covariance
where ^ mand 6of the m-th component, respectively.
It should be noted that the prior parameters, k and k , for all the nodes in thetree need to be speci ed. Since the SMAP framework provides no speci c ways
to calculate these parameters, optimal values should be determined empirically.
One simple way to do this is to use the same  and  for all the nodes and optimize
their values by using preliminary recognition experiments (see Section 4.5.B).
Equation (4.26) can be rewritten for the leaf node as follows:^ K =

KX
k=1

wk ~ k ;(4:30)

where the weighting factor wk is

wk =

K
Y
0k
i
:
0k + k i=k+1 0i + i

(4:31)The mean vector estimated using the SMAP method can be considered as aweighted sum of the ML estimates at the di erent layers of the tree. Two important characteristics of the weight, wk , are highlighted in the following:

70

CHAPTER 4.

SMAP SPEAKER ADAPTATION

1. The weight wk at node Nk becomes larger as the amount of data at that
node, 0k , becomes larger.
2. The weight wk at node Nk decreases as k becomes smaller.
These properties are desirable for adaptation. When the amount of data is small,the ML-estimated parameters in the upper layers, which represent global transformation, are mainly responsible for the resulting pdf. And when the amountof data is large, the parameters in the lower layers, which represent localized
transformation, predominate.
It should be noted that Furui [17] has already developed an unsupervised
adaptation method that utilizes a hierarchical structure for vector quantization.
It is also important to note that similar hierarchical structures have already beenused in the acoustic modeling ([3, 31, 74, 54]). Although this SMAP approach is
not the rst using tree-based adaptation (e.g., [38]), the method described here isthought to be theoretically well-de ned in terms of taking advantage of both theBayesian framework and the tree construction principle. It demonstrates that
this framework and this principle work well together as will be clear in the next
section.

4.6 Experiments
The proposed method was experimented with the 991-word DARPA resource
management (RM) task [40]. New adaptation and testing data from ve nonnative male speakers (labeled as A,B,C,D, and E) were recorded simultaneously
under two acoustic conditions: (1) a close-talking microphone (MIC); and (2) atelephone handset over a dial-up line (TEL). The data for adaptation consistedof 300 utterances from each speaker in each of the two channels. For testing, 75
utterances from each of the two channels were collected from each speaker. In
the following experiments all the 75 utterances were always used in a particularcondition for testing.

4.6.

EXPERIMENTS71

The speech signal was rst down-sampled from 16 kHz to 8 kHz and theanalysis frames were 30-ms wide with a 20-ms overlap. For each frame a 38dimensional feature vector [31] was extracted by using a tenth-order LPC analysis. The feature vector for each frame consists of a 12-dimensional cepstral vector,
a 12-dimensional delta-cepstrum vector, a 12-dimensional delta-delta-cepstrum
vector, a delta log energy feature and a delta-delta log energy feature. For recognition, a set of 1769 context-dependent phone HMMs [31] was used. All units
except the one for background silence had three states, each with a maximumof 16 mixture components. This gives a total of about 5,000 states and about
80,000 distinct Gaussian densities. Forty seven context-independent phones wereused to create all the context-dependent units. For all the experiments, the RM
word pair grammar was used, which gives a perplexity of about 60. A diagonalcovariance was assumed for each mixture Gaussian component.
Speaker-independent HMMs were trained using the NIST/RM SI-109 training
set consisting of 3990 utterances from 109 native American talkers (31 females
and 78 males), each providing 30 or 40 utterances. These models were thenadapted, using a MAP adaptation algorithm [18], with the data from the 78
male talkers in order to create speaker-independent male models. These speakerindependent male models served as initial seed models for further adaptation. The
tree structures used in the experiments were constructed using the parameters of
the speaker-independent male models. The background noise model was excluded
in tree construction.

4.6.1 Summarization
The e ectiveness of the summarization technique using the normalized pdfs is
examined rst. In this experiment, one single normalized pdf was applied to all
the mixture components and its covariance was assumed as the identity matrix,
i.e. no mismatch in scaling was considered. This normalization technique wascompared with two other methods: stochastic matching (SM) [44] and a method

72

CHAPTER 4.

SMAP SPEAKER ADAPTATION

Table 4.1: Recognition rates (%) for MIC data when one utterance was used fornormalization. SI is the result of a speaker-independent recognition experiment.
SI

Normalized pdf SM Tied-Shift

75.2

81.2

81.7

81.7

that uses a shift in the feature vector space (Tied-Shift) [50, 51, 36, 52, 54]. In
SM the i-th element of the mean vector for the m-th component is adapted asfollows:
PP
x (i)0 (i)Mtl
l=1 ltl2 (i)PTPM
lt
t=1 l=1 l2 (i)T
t=1

~m(i) = m(i) +:(4:32)

And in Tied-Shift the mean vector for the m-th component is adapted as follows:~m (i) = m (i) +

PT

t=1

PM

lt

l=1
PT

t=1

(xt (i) 0 l (i))

PMl=1 lt:(4:33)

In contrast, using a single normalized pdf, the mean vector for the m-th component is adapted as follows:~m (i) = m (i) + m(i) 1PTxt (i)0l (i)l=1 ltl (i)
PTPM
t=1 l=1 lt

t=1

PM

:

(4:34)The recognition experiments for the ve speakers were carried out using thedata collected through the microphone (MIC). Supervised adaptation in which
only one utterance was used for adaptation was carried out. The results of the experiments, averaged over ve speakers, were listed in Table 4.1. It shows that thesimpli ed normalization technique gave similar improvement over the SI recognition accuracy compared with the other two methods. This normalization procedure will be used throughout the rest of this chapter.

4.6.2 Tree clustering
It is dicult to choose the optimal tree structure for adaptation. To determine agood experimental condition, several structures were tested in a set of supervised

Recognition accuracy (%)

4.6.

73

EXPERIMENTS

K=4
K=6
K=5
K=3
90

85

K=2
1

3 5 1025 50 100
300
Number of utterances for adaptation

Figure 4.4: Recognition results obtained using di erent tree structures.

74

CHAPTER 4.

SMAP SPEAKER ADAPTATION

adaptation experiments. In each experiment the number of branches from eachnode was xed to ten, and the depth of the tree was changed from one layer(K = 2 with only one common normalized pdf) to ve layers (K = 6 with
11,111 normalized pdfs). The weight used in tree clustering in Eqs. (4.6) and(4.7) is xed at a value of 0.1 for all the experiments. Only the mean vectors
were modi ed and the variances remained unchanged. Recognition accuracies
obtained in the adaptation recognition experiments (averaged over ve speakers)
are shown in Figure 4.4 as a function of the amount of adaptation data. The
result obtained with only one normalized pdf (K = 2) was obviously the worst.
The result obtained with the two-layer tree (K = 3) was much better than thatobtained with one normalized pdf (K = 2). The result obtained with the threelayer tree (K = 4) was better than that obtained with the two-layer tree (K = 3),
and the results were similar for K = 4; 5, and 6. The three-layer (K = 4) tree
was therefore used in the following experiments. In all experiments the control
parameters k and k were shared by all the nodes in the tree. Using k = 2 andk = D + 1 gave the best results for the three-layer (K = 4 with 111 clusters)
tree in some preliminary experiments. These control parameters were therefore
used in all the following experiments.
Next, how the weights in Eq. (4.31) changed according to the amount of data
was investigated. Those changes for each layer in the four-layer (K = 5) tree
were examined. Figure 4.5 shows the distribution of the weight among the layers
in the tree for four di erent numbers of utterances: 1, 5, 25 and 100. When theamount of data was small, the ML estimates of the upper layers (near the rootnode) were the ones mainly used. As the amount of data became larger, those of
the lower layers (near the leaf nodes) predominated.
Next how the acoustic features of the speech units were distributed in thetree structure was examined . All the units were classi ed into nine phonemicclasses [15] according to the feature of the central phone label in a triphone unit
and counted the number of mixture components in each node. Only the mixture

U=1
layer

root
1st
2nd
3rd
leaf
0.0

layer

75

EXPERIMENTS

root
1st

0.5
weight

0.0

U=25

2nd
3rd
leaf
0.0

root
1st
2nd

U=5

3rd
leaf

layer

layer

4.6.

root
1st
2nd

0.5
weight
U=100

3rd
leaf
0.5
weight0.0

0.5
weightFigure 4.5: The weight for each layer in the tree. In each layer, the weights for
all the leaf nodes are averaged.

76

CHAPTER 4.

SMAP SPEAKER ADAPTATION

Table 4.2: Phone distribution (%) in the rst layer (K = 2) of the tree.
Node No.

1

Front vowels

2

3

5

6

7

8

9

10

72.4

0.1 20.9 0.9

3.7

0.4

0.6 0.6

0.5

Central vowels

49.7

0.6

6.5 4.1 23.2

0.7

6.8 6.5

1.8

Back vowels

92.2

1.5 2.2

0.3

Diphthongs

98.6

0.2 0.2

0.2

Fricatives

4.9

Stops
Nasals

0.1

4

1.3

2.5

0.7
1.2

0.8 82.6

0.5 1.2

3.3

2.8 63.2 10.5

4.0 0.5

1.1 12.6

0.9 0.5

3.8

51.6

2.4 0.1

3.2

0.4 0.5 41.4

A ricates

2.9
0.1

2.7
0.2

7.0

1.0

Glides, Semivowels 59.3

0.2
92.0

3.7 1.8

2.0

21.6 9.2

2.4

Table 4.3: Phone distribution (%) in the second layer (K = 3) of the tree.
Node No.

1

2

3

4

Front vowels

68.6 0.5

7.6 0.1

Central vowels

5

6

7

8

9

10

0.0 0.4

21.7

0.8

0.3

10.4 5.9 44.6 0.1

0.8 5.6

0.1 19.2 10.5

2.9

Back vowels
Diphthongs

14.5 0.5 66.8 0.1
0.5
93.7

7.6 4.2
0.2 0.2

0.9
4.0

5.2

Nasals29.9 6.7

1.2

0.8 1.6

Glides, Semivowels 11.5 0.6 18.2 0.2 41.9 0.6 11.8

0.2
0.2

1.1

59.8

0.4 10.4

4.4

4.6.

EXPERIMENTS

77

components in the middle state of each unit were selected because the other two
may be in uenced by their nearby phones. The distribution in the nodes in thesecond layer (the child nodes of the root-node) is shown by the data listed in Table
4.2. The numbers in each row are the percentage of mixture components thatwere classi ed into the various nodes. For the a ricates, for example, 92% of thecorresponding mixture components were clustered into node No. 7. It can be seenthat fricatives, a ricates, and stops were distinguished from the other features,
whereas all the vowels, nasals, glides, and semi-vowels were mostly in the rst
node (Node No. 1). Therefore, the child nodes of Node No. 1 were examined next.
The results were listed in Table 4.3. This time, front vowels were distinguishedfrom the other vowels (central vowels, back vowels, and diphthongs), and glides,
semi-vowels, and nasals were separated from vowels. These results show thatthe clustering using only the distance in the acoustic space results in a mixture
component grouping that is phonologically meaningful.

4.6.3 Supervised adaptation experiments
To verify the e ectiveness of the SMAP method, it was compared with conventional MAP estimation (MAP) [18] and with simple bias estimation using a tree
structure without MAP estimation (TREE) [53]. In the experiment labeled as
MAP estimation, no structure in the acoustic space was assumed and each parameter of HMMs was estimated separately. In the TREE experiment, one nodein the tree was selected for each mixture component by using a threshold dataamount, and the ML estimates for the parameters at that node were used tomodify the parameter of the corresponding mixture component (Figure 4.6).
The recognition results averaged over all ve non-native speakers, for the two
acoustic conditions, MIC and TEL, are shown in Figures 4.7 and 4.8, respectively.
As can be seen, the baseline performance of SI models is much lower for TEL
than for MIC. This is because the combined microphone and channel distortion
was larger in the TEL data than the situation in the MIC data in which the

78

CHAPTER 4.

SMAP SPEAKER ADAPTATION(A) Total amountof data is small∆∆(B) Total amountof data is large
COARSE
∆

∆

∆
∆

∆∆

∆

∆

…

µΝ

FINE
µ1 µ 2 …

µi…

µ1 µ 2 …

µΝ

Amount ofnode dataµi: large: appropriate: smallFigure 4.6: Speaker adaptation adaptation using autonomous model complexity
control (TREE adaptation).
microphone di erence is the main source of distortion on top of the speakerdistortion due to non-native pronunciations and accents. In each gure two setsof results are shown for the SMAP method: mean-only adaptation (solid curve)
in which the variances of the normalized pdfs were assumed to be the identity
matrix I and only the means were estimated, and the other with adaptation forboth means and variances (dotted curve). These gures clearly show that theSMAP method outperformed the MAP and TREE methods at almost every data
point. The recognition rates for the SMAP method were much higher than thosefor MAP when the amount of data was small. With three adaptation utterances
for TEL data, for example, the error rate reduction from the SI performance was56%. This was much larger than the reduction obtained with the MAP method,which was only 1.7%. The recognition rates for the SMAP method became thesame as those for the MAP method when the amount of data became large. The
SMAP method also showed better recognition accuracy than that obtained with
the TREE method not only when the amount of data was large but also when
the amount of data was small. This is probably because parameter estimation

4.6.

95
Recognition accuracy (%)

79

EXPERIMENTS

MIC

90
SMAP
85

TREE

80
MAP
SI
75
1

3 5 1025 50 100
300
Number of utterances for adaptation

Figure 4.7: Recognition rates obtained with supervised adaptation when the MIC
data were used. Two kinds of experiments were done with the SMAP method:mean-only adaptation (solid curve), and adaptation for both means and variances(dotted curve).

80

CHAPTER 4.

TEL

90
Recognition accuracy (%)

SMAP SPEAKER ADAPTATION

SMAP
80
TREE

70
60

MAP

50
SI
40
1

3 5 1025 50 100
300
Number of utterances for adaptation

Figure 4.8: Recognition rates obtained with supervised adaptation when the TEL
data were used. Two kinds of experiments were done with the SMAP method:mean-only adaptation (solid curve), and adaptation for both means and variances(dotted curve).

4.6.

81

EXPERIMENTS

Table 4.4: Recognition rates (%) of each speaker obtained when using the SMAPmethod on MIC data.
No. of Utter. A
BCD
E Ave.
SI

74.8 51.2 74.9 85.7 89.3

75.2

1

83.0 68.5 88.6 82.9 88.1

82.2

3

82.9 70.1 89.2 86.8 89.6

83.7

5

84.8 69.7 89.3 88.7 90.1

84.5

10

83.8 72.8 90.8 89.2 90.4

85.4

25

86.2 76.3 88.3 90.5 91.3

86.5

50

88.3 78.4 91.6 92.2 91.7

88.4

100
300

90.1 84.5 93.2 94.1 91.7
95.0 92.8 96.4 97.6 94.9

90.7
95.3

was more robust than that in TREE, since a weighted sum of parameters from
more than one layer was used. The mean-only adaptation result was better thanadaptation of both means and variances when the amount of data was extremely
small, but became worse as the amount of data became larger. This indicatesthat when the data amount was insucient it is dicult to estimate both themean and the variance as expected.
The results for each speaker, obtained when both means and variances wereadapted are listed in Tables 4.4 and 4.5. The recognition rates were improved forall sizes of adaptation data and for all the speakers when TEL data were used asshown in Table 4.5. The improvement was especially more pronounced for those
speakers with lower accuracy when SI models were used. With the MIC data,
the recognition rates for speakers D and E were slightly less than the SI result
when only one utterance was used but they soon became higher when more than
three utterances were used.

82

CHAPTER 4.

SMAP SPEAKER ADAPTATION

Table 4.5: Recognition rates (%) of each speaker obtained when using the SMAPmethod on TEL data.
No. of Utter. A
BCD
E Ave.
SI

48.8 18.6 50.8 35.6 52.6

41.3

1

67.9 58.9 75.2 77.8 73.7

70.7

3

76.3 58.4 75.2 81.1 78.8

74.0

5

77.6 61.7 75.2 83.6 76.6

74.9

10

80.3 63.2 78.7 83.2 79.4

77.0

25

81.2 65.8 79.3 82.1 80.8

77.8

50

83.2 72.1 80.5 83.8 84.1

80.7

100
300

84.5 79.7 83.6 87.2 83.6
91.4 90.2 91.7 95.6 90.4

83.7
91.9

Table 4.6: Recognition rates (%) obtained with unsupervised adaptation for MIC
data.
A
B
C
D
E Ave.
SI

74.8 51.2 74.9 85.7 89.3

75.2

SMAP 81.2 67.3 78.7 88.4 89.0

80.9

4.6.4 Unsupervised adaptation experiments
Unsupervised adaptation, in which no supervising information is available, isdesirable in actual system operation. An on-line unsupervised adaptation scenario incorporating SMAP was therefore designed and evaluated [57, 59]. During
unsupervised adaptation and testing, the parameters were estimated on a perutterance basis; only one utterance was used for unsupervised adaptation. Firstthe test utterances were decoded by using the initial HMMs and then the parameters were estimated assuming this decoded word string is correct. In these

4.6.

83

EXPERIMENTS

Table 4.7: Recognition rates (%) obtained with unsupervised adaptation for TEL
data.
A
B
C
D
E Ave.
SI

48.8 18.6 50.8 35.6 52.6

41.3

SMAP 67.3 48.5 57.7 43.1 70.1

57.3

experiments, the variances for the normalized pdfs were xed to be the identity
matrices. The recognition results are listed in Tables 4.6 and 4.7. The error rate
reduction was 23% for MIC and 27% for TEL. It should be noted that the e ectof the SMAP method was larger for the speakers with lower SI recognition rates.
For speaker B, for example, the error reduction rate was 33% for MIC and 37%
for TEL.
The recognition rates in Table 4.6 are still rather low for any actual system
usage. In the next experiment, ways to combine batch supervised adaptation
typically used for fast enrollment of new speakers with on-line unsupervised adaptation was examined in order to raise recognition rates to a level high enough forpractical usage. This combined adaptation process was carried out in two steps:

Step 1. Supervised adaptation using a set of adaptation data to generate seedmodels;Step 2. Unsupervised adaptation using the test data, based on the above models.
The recognition rates (averaged over the ve speakers) are listed in Table 4.8. In
this experiment, the number of utterances used in Step 1 varied from 1 to 300.
The supervised adaptation was carried out using the MIC or TEL adaptation
data (labeled as SUP in Table 4.8) and unsupervised adaptation and recognition
were also done for MIC or TEL test data (labeled as TEST in Table 4.8). In
all cases, only one utterance was used for unsupervised adaptation (Step 2). In
this table, the values listed in the column labeled S1 are the recognition rates

84

CHAPTER 4.

SMAP SPEAKER ADAPTATION

Table 4.8: Recognition rates (%) for combining supervised and unsupervised
adaptation.
SUP
MIC
TEL
TEST
MIC
TEL
MIC
TEL
No. of Utter.

S1

S2

S1

S2

S1

S2

S1

S2

0(SI)

75.2 80.9 41.3 57.3 75.2 80.9 41.3 57.3

1

83.0 83.5 41.4 57.8 74.8 81.6 71.5 71.7

3

83.8 83.9 44.3 62.2 77.5 82.8 74.4 75.2

5

83.9 84.2 47.0 63.0 78.9 83.9 76.0 76.3

1085.7 86.1 51.7 67.2 79.1 84.3 77.3 77.2

25

86.3 86.4 54.0 69.4 80.3 85.1 77.5 78.3

50

87.7 87.8 58.1 73.3 82.2 85.8 80.0 80.0

100

90.4 90.4 62.1 75.5 83.8 87.6 82.6 83.2

300

94.3 94.4 70.8 84.6 87.7 91.0 90.6 90.6

4.7.

DISCUSSION AND SUMMARY

85

obtained with Step 1 only and the values listed in the column labeled S2 are therates obtained after Step 2. Although this adaptation was only slightly e ective
when the acoustic conditions for the SUP data and that for the TEST data weresimilar, its e ectiveness when acoustic conditions were di erent was clearly shown.
For example, when the MIC data set was used for SUP and the TEL data set wasused for TEST, the combined method (Step 1 and Step 2) required only threeutterances for supervised adaptation to achieve 60% recognition accuracy, while
the supervised adaptation (Step 1 only) needed 100 utterances. It was provedthat this combined adaptation is especially e ective when there are mismatches
other than the speaker di erences in the current adaptation scenario. It should
also be noted that there was no degradation in the recognition performance ofthe combined strategy when the acoustic conditions for SUP and TEST wereidentical, i.e., when there was no mismatch between the adaptation data and thetesting data.

4.7 Discussion and summary
The SMAP method for adaptation of HMM parameters enhances the performance of the conventional MAP method when the amount of data is small byutilizing a hierarchical structure in the model parameter space. Its e ectivenesswas con rmed in a set of recognition experiments using the speech data from
non-native speakers collected through two di erent channels (MIC and TEL). In
supervised adaptation, for example, with three utterances for TEL data, the error
rate reduction was 56%. This was much better than the 1.7% reduction obtainedwith the MAP method. The SMAP method was also shown to be e ective in
unsupervised adaptation: with only one utterance for TEL data, the error rate
reduction was 27%. In addition, the combination of supervised adaptation andon-line unsupervised adaptation greatly reduced the amount of data required forsupervised adaptation. To obtain a recognition accuracy of 60% for the TEL
data when using the MIC data for supervised adaptation, the combined method

86

CHAPTER 4.

SMAP SPEAKER ADAPTATION

required only three utterances whereas 100 utterances were needed to obtain 60%
accuracy when using supervised adaptation only. It should also be noted that theSMAP method yields as high a recognition accuracy as the MAP method when
the amount of data is suciently large.
Mismatches between training and testing conditions are caused by many differences, such as those between speakers, microphones, channels and noise levels.
Many methods of compensating such mismatches have been developed but mostof them focus on one or two di erences. In an actual operational environment,
however, more than one di erence often contributes to the mismatch. The signi cance of each di erence is usually unknown. In such cases, it is almost impossible
to distinguish the contribution of one di erence from that of another by using only
the speech data. Furthermore, it is dicult to choose an e ective combinationof techniques, each of which is designed to compensate the e ect of a particular
di erence. The SMAP method, on the other hand, compensates the mismatchas a whole; there is no need to specify any particular di erence responsible forthe mismatch. It is therefore more robust than the other methods when thesedi erences are not known.
Recently, the idea of rapid adaptation was studied intensively at the 1998
Johns Hopkins University Summer Workshop and the results obtained were summarized in [14]. Tree structure dependency for speaker adaptation was one of thetechniques explored during the workshop as a means of reducing the requirement
for a large adaptation set [27]. The ndings also support the proposed SMAPalgorithm, especially in the area of unsupervised speaker adaptation.
This SMAP approach is quite general in its framework and can be easily
applied to other adaptation methods. For example, SMAPLR, in which SMAPis applied to maximum a posteriori linear regression (MAPLR), was recently
proposed and proved to be signi cantly better than MAPLR when the amountof adaptation data is extremely small [61]. The importance and e ectiveness of
this SMAP approach was also emphasized in recent surveys of speaker adaptation

4.7.

DISCUSSION AND SUMMARY

87

studies [72, 33].
The SMAP method described here uses a tree structure in the model parameter space. While many kinds of tree structures can be used for SMAP estimation,
it is important to choose one which represents the similarity of the normalized
pdfs of the mixture components well. Good results were obtained when the
Kullback-Leibler divergence between mixture components was used as a measureof similarity in constructing the tree structure, but many other similarity measures can be used. Other structures re ecting the relationship between acoustic
model parameters are also worth investigating.
-

88

CHAPTER 4.

SMAP SPEAKER ADAPTATION

Chapter 5
Conclusion
This thesis has proposed a structural approach to robustness against data insuciency. In this approach, a tree-structured model set is prepared and one node set(a model) is selected by using information-theoretic criteria. Three key factors
in this approach are the design of the root layer and the leaf layer, the methodof constructing the tree structure, and the node selection framework. This approach was applied to two problems in speech recognition: acoustic modeling andspeaker adaptation.
For acoustic modeling, a phonetic decision tree, in which the root is a monophone and the leaves are triphones, was constructed and the optimal model was
selected by using the MDL criterion. This approach achieved recognition moreaccurate than that obtained when a conventional approach was used and it had
a much lower overall computational cost in an evaluation made through a seriesof recognition experiments.
For speaker adaptation, a tree of Gaussian pdfs was constructed by using
Kullback-Leibler divergence. Its leaves correspond to all the mixture components
in CDHMMs and the root node corresponds to the pdf shared among all of them.
The optimal model was selected by using SMAP estimation. It reduced the error
rate by half using only three utterances for adaptation and it yielded the sameaccuracy as conventional MAP and ML estimation when the amount of data was89

90

CHAPTER 5.

CONCLUSION

suciently large in the evaluational experiments.

5.1 Contribution of the thesis
The contribution of this thesis for speech recognition studies is the following: It has proved the importance of the described structural approach in statistical pattern recognition. This approach can be applied to many problems
incurred due to data insuciency.

 It has proved that the use of information criteria is an e ective approach todealing with acoustic modeling problems in speech recognition. Among theinformation criteria, the MDL criterion was focused on and its e ectivenesswas extensively explored.

 It has shown several examples of tree-structured models, which performed
well in the structural approach. While many other models may possibly be
used, those described in this thesis demonstated particularly good performance.

 It o ers excellent methods for acoustic modeling and speaker adaptation;
MDL acoustic modeling and SMAP adaptation. Both have been extensively
studied by many researchers since they were rst published.

5.2 Future research directions
Although I believe the investigations reported in this thesis represent some signi cant progress, further research is clearly still needed.

 The MDL-based modeling can be applied not only to the clustering method
using a phonetic decision tree but also to other clustering methods for acoustic modeling. It can also be easily applied to the language modeling in5.2.

FUTURE RESEARCH DIRECTIONS

91

speech recognition, and some studies relevant to this point have appeared
recently (e.g., [62]).

 The MDL criterion can be used to compare various classes of probabilistic
models, since it utilizes only the model complexity and the likelihood of theprobabilistic model for the data. For example, it can be used to decide whatclass of models can best be used for the output pdfs of HMMs. Furthermore,it can be used to carry out clustering across more than one parameter
set. For example, it enables the similtanuous clustering of parameters fortranstition probabilities and those for output probabilities.

 The SMAP adaptation is quite general in its framework, and can easily beapplied to language adaptation. The n-gram language model, for example,
can be used to construct a tree structure in which each leaf node represents
an n-gram and an (n-1)-gram is assigned to the parent node. In a trigram
model(e.g., [24]), a four-layer tree is constructed; for each word the root
node corresponds to uniform distribution for all the vocabulary, the parent
node in the rst layer corresponds to the unigram, the parent node in thesecond layer corresponds to the bigram, and the leaf node corresponds tothe trigram.

 While this structural approach is not directly aimed at increasing the recognition accuracy under ideal conditions, it will be helpful for research in thisdirection. This approach keeps computational costs low because it makes
it possible to easily obtain a model whose size is appropriate for the given
amount of training data.

 It should be noted that noisy data are not dealt with in this thesis. It is
well known that environmental noise and channel distortion can seriously
degrade speech recognition. In such cases, the likelihood of data becomes
more important in model selection. While one method described in thisthesis used the MDL criterion, in which the likelihood is taken into account,

92

CHAPTER 5.

CONCLUSION

further evaluation using actual noisy data is still needed.

 This approach utilizes the embedded structure of the parameter space. In
the acoustic modeling, it is assumed that the structure of the parameter
space can be represented by phonetic decision trees. And in the speaker
adaptation, it is assumed that the Kullback-Leibler divergence can be usedas the measure of the distance between the Gaussian pdfs. While theseassumptions were justi ed by the evaluation, it may be possible to obtain
more accurate representations of the embedded structure.

Bibliography[1] H. Akaike, \A new look at the statistical model identi cation," IEEE Trans.
Automatic Control, vol. 19, pp. 716-723, 1974.
[2] B. Atal, \E ectiveness of linear prediction characteristics of the speech wave
for automatic speaker identi cation and veri cation," J. Acoust. Soc. Am.,
vol. 55, pp. 1304-1312, 1974.
[3] L. R. Bahl, P. V. de Souza, P. S. Gopalakrishnan, D. Nahamoo, and
M. A. Picheny, \Decision trees for phonological rules in continuous speech,"in Proc. ICASSP-91, Toronto, pp. 185-188, 1991.
[4] J. O. Berger, Optimal Statistical Decisions, in Springer Series in Statistics,
Springer-Verlag, 1980.
[5] P. F. Brown, V. J. D. Pietra, P. V. deSouza, J. C. Lai, and L. R. Mercer,\Class-based n-gram models of natural language," Computational Linguistics, vol. 18, no. 4, pp.467-479, 1993.
[6] S. S. Chen and P. S. Gopalakrishnan, \Clustering via the Bayesian information criterion with applications in speech recognition," in Proc. ICASSP-98,
Seattle, pp. 645-648, 1998.
[7] S. S. Chen, E. M. Eide, M. J. F. Gales, R. A. Gopinath, D. Kanevsky, and
P. A. Olsen, \Recent improvements to IBM's speech recognition system for
automatic transcription of broadcast news," in Proc. ICASSP-99, Phoenix,
1999.
93

94

BIBLIOGRAPHY[8] S. S. Chen and R. A. Gopinath, \Model selection in acoustic modeling," inProc. EuroSpeech-99, Budapest, pp. 1087-1090, 1999.
[9] J.-T. Chien, C.-H. Lee, and H.-C. Wang, \Improved Bayesian learning of hidden Markov models for speaker adaptation," in Proc. ICASSP-97, Munich,pp. 1027-1039, 1997.
[10] M. H. DeGroot, Statistical Decision Theory and Bayesian Analysis,
McGraw-Hill, 1970.
[11] A. P. Dempster, N. M.Laird, and D. B. Rubin, \Maximum likelihood fromincomplete data via the EM algorithm," J. Roy. Stat. Soc., vol. 39, no. 1,
pp. 1-38, 1977.
[12] V. V. Digalakis, P. Monaco, H. Murveit, \Genons: generalized mixture tying
in continuous hidden Markov model-based speech recognizers," IEEE Trans.
Speech Audio Processing, vol. 4, no. 4, pp. 281-289, 1996.
[13] V. V. Digalakis and L. G. Neumeyer, \Speaker adaptation using combinedtransformation and Bayesian methods," IEEE Trans. Speech Audio Processing, vol. 4, no. 4, pp. 294-300, 1996.
[14] V. V. Digalakis, S. Berkowitz, E. L. Bocchieri, C. Boulis, W. J. Byrne,
H. Collier, A. Corduneanu, A. Kannan, S. P. Khudanpur, and A. Sankar,\Rapid speech recognizer adaptation to new speakers," in Proc. ICASSP-99,
Phoenix, pp. 765-768, 1999.
[15] J. L. Flanagan, Speech Analysis, Synthesis, and Perception, 2nd Edition,
Springer-Verlag, 1972.
[16] S. Furui, \Speaker-independent isolated word recognition using dynamic features of speech spectrum," IEEE Trans. Acoustics, Speech, Signal Processing,vol. 34, pp. 52-59, 1986.

BIBLIOGRAPHY

95[17] S. Furui, \Unsupervised speaker adaptation method based on hierarchicalspectral clustering," in Proc. ICASSP-89, Glasgow, pp. 286-289, 1989.
[18] J. L. Gauvain and C.-H. Lee, \Maximum a posteriori estimation for multivariate Gaussian mixture observations of Markov chains," IEEE Trans.
Speech Audio Processing, vol. 2, no. 2, pp. 291-298, 1994.
[19] I. J. Good, \The population frequencies of species and the estimation of
population parameters," Biometrica, vol. 40, pp. 237-264, 1953.
[20] Q. Huo and C.-H. Lee, \On-line adaptive learning of the continuous density hidden Markov model based on approximate recursive Bayes estimate,"
IEEE Trans. Speech Audio Processing, vol. 5, no. 2, pp. 161-172, 1997.
[21] Q. Huo and C.-H. Lee, \On-line adaptive learning of the correlated
continuous-density hidden Markov model for speech recognition," IEEE
Trans. Speech Audio Processing, vol. 6, no. 4, pp. 386-397, 1998.
[22] M.-Y. Hwang, X. Huang, and F. Alleva, \Predicting unseen triphones withsenones," Proc. ICASSP-93, Minneapolis, pp. II-311-314, 1993.
[23] F. Jelinek, \Self-organized language modeling for speech recognition," inReadings in Speech Recognition, A. Waibel and K. F. Lee, eds., MorganKaufmann, pp. 450-506, 1990.
[24] F. Jelinik, Statistical method for speech recognition, MIT Press, 1998.
[25] H. Jiang, K. Hirose, and Q. Huo, \Robust speech recognition based on
Viterbi Bayesian predictive classi cation," in Proc. ICASSP-97, Munich,pp. 1551-1554, 1997.
[26] A. Kannan, M. Ostendorf and J. Rohlicek, \Maximum likelihood clusteringof Gaussians for speech recognition," IEEE Trans. Speech Audio Processing,vol. 2, no. 3, pp. 453-454, 1994.

96

BIBLIOGRAPHY[27] A. Kannan and S. P. Khudanpur, \Tree-structured models of parameter
dependence for rapid adaptation in large vocabulary conversational speech
recognition," in Proc. ICASSP-99, Phoenix, pp. 769-772, 1999.
[28] S. M. Katz, \Estimation of probabilities from sparse data for the language
model component of a speech recognizer," IEEE Trans. Acoust., Speech,
Signal Processing, vol. 35, no. 3, pp. 400-401, 1987.
[29] R. Kuhn and R. De Mori, \A cache-based natural language model for
speech recognition," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 12, no. 6, pp. 570-583, 1990.
[30] C.-H. Lee, C.-H. Lin, and B.-H. Juang, \A study on speaker adaptation of
the parameters of continuous density hidden Markov models," IEEE Trans.
Acoust., Speech, Signal Processing, vol. 39, no. 4, pp. 806-814, 1991.
[31] C.-H. Lee, E. Giachin, L. R. Rabiner, R. Pieraccini, and A. E. Rosenberg,\Improved acoustic modeling for large vocabulary continuous speech recognition," Computer Speech and Language, vol. 6, no. 2, pp. 103-207, 1992.
[32] C.-H. Lee, \On stochastic feature and model compensation approaches torobust speech recognition," Speech Communication, vol. 25, pp. 29-47, 1998.
[33] C.-H. Lee and Q. Huo, \On adaptive decision rules and decision parameter
adaptation for automatic speech recognition," in Proc. of IEEE, vol. 88,
no. 8, pp. 1241-1269, 2000.
[34] K.-F.Lee, S. Hayamizu, H.-W. Hon, C. Huang, J. Swartz, and R. Weide,\Allophone clustering for continuous speech recognition," in Proc. ICASSP90, Albuquerque, pp. 749-753, 1990.
[35] C. J. Leggetter and P. C. Woodland, \Maximum likelihood linear regression for speaker adaptation of continuous-density hidden Markov models,"
Computer Speech and Language, vol. 9, pp. 171-185, 1995.

BIBLIOGRAPHY

97[36] K. Ohkura, M. Sugiyama, and S. Sagayama, \Speaker adaptation based ontransfer vector eld smoothing with continuous mixture density HMMs," inProc. ICSLP-92, Alberta, pp. 369-372, 1992.
[37] M. Ostendorf and H. Singer, \HMM topology design using maximum likelihood successive state splitting," Computer Speech and Language, vol. 11,
pp. 17-41, 1997.
[38] D.-B. Paul, \Extensions to phone-state decision-tree clustering: single-tree
and tagged clustering," in ICASSP-97, Munich, pp. 1487-1490, 1997.
[39] F. Pereira, N. Tishby, L. Lee, \Distributional clustering of English words,"
in Proc. ACL-93, Columbus, Ohio, pp.183-190, 1990.
[40] P. Price, W. Fisher, J. Bernstein, and D. Pallett, \A database for continuous
speech recognition in a 1000-word domain," in Proc. ICASSP-88, New York
City, pp. 651-654, 1988.
[41] L. R. Rabiner and B.-H. Juang, Fundamentals of Speech Recognition, Prentice Hall, 1993.
[42] M. Rahim and B.-H. Juang, \Signal bias removal by maximum likelihood
estimation for robust telephone speech recognition," IEEE Trans. Speech
Audio Processing, vol. 4, no. 1, pp. 19-30, 1996.
[43] J. Rissanen, "Universal coding, information, prediction, and estimation,"
IEEE Trans. Information Technology, vol. 30, no. 4, pp. 629-636, 1984.
[44] A. Sankar and C-.H. Lee, \A maximum likelihood approach to stochastic
matching for robust speech recognition," IEEE Trans. Speech Audio Processing, vol. 4, no. 3, pp. 190-202, 1996.
[45] G. Schwartz, \Estimating the dimension of a model," Annals of Statistics,
vol. 6, pp. 461-464, 1978.

98

BIBLIOGRAPHY[46] R. M. Schwartz, Y. Chow, S. Roucos, M. Kransner, and J. Makhoul, \Improved hidden Markov modeling of phonemes for continuous speech recognition," in Proc. ICASSP84, San Diego, 35.6, 1984.
[47] R. Shibata, \Selection of the order of an autoregressive model by Akaike's
information criterion," Biometrica, vol. 63, pp. 117-126, 1976.
[48] R. Shibata, \Asymptotically ecient selection of the order of the modelfor estimating parameters of a linear process," Annals of Statistics, vol. 8,
pp. 147-164, 1980.
[49] R. Shibata, \Optimal selection of regression variables, " Biometrica, vol. 68,
pp. 45-54, 1981.
[50] K. Shinoda, K. Iso, and T. Watanabe, \Speaker adaptation for demi-syllable
based speech recognition using continuous HMM," in Proc. of ICSLP-90,
Kobe, pp. 261-264, 1990.
[51] K. Shinoda, K. Iso, and T. Watanabe, \Speaker adaptation for demi-syllablebased continuous-density HMM," in Proc. ICASSP-91, Toronto, pp. 857-860,
1991.
[52] K. Shinoda and T. Watanabe, \Unsupervised speaker adaptation for speech
recognition using demi-syllable HMM," in Proc. ICSLP-94, Yokohama,
pp. 435-438, 1994.
[53] K. Shinoda and T. Watanabe, \Speaker adaptation with autonomous control
using tree structure," in Proc. EuroSpeech-95, Madrid, pp. 1143-1146, 1995.
[54] K. Shinoda and T. Watanabe, \Speaker adaptation with autonomous model
complexity control by MDL principle," in Proc. ICASSP-96, Atlanta, pp.717720, 1996.

BIBLIOGRAPHY

99[55] K. Shinoda and T. Watanabe, \Acoustic modeling based on the MDL criterion for speech recognition," in Proc. EuroSpeech-97, Rhodes, vol. 1, pp. 99102, 1997.
[56] K. Shinoda and C.-H. Lee, \Structural MAP speaker adaptation using hierarchical priors," Proc. IEEE Workshop on Speech Recognition and Understanding, Santa Barbara, pp. 381-387, 1997.
[57] K. Shinoda and C.-H. Lee, \Unsupervised adaptation using structural Bayes
approach," in Proc. ICASSP-98, Seattle, pp. II 793-796, 1998.
[58] K. Shinoda and T. Watanabe, \MDL-based context-dependent subword
modeling for speech recognition," J. Acoust. Soc. Jpn.(E), vol. 21, no. 2,
pp.79-86, 2000.
[59] K. Shinoda and C.-H. Lee, \A structural Bayes approach to speaker adaptation", IEEE Trans. Speech Audio Processing (to appear March 2001).
[60] O. Siohan, C. Chesta, and C.-H. Lee, \Hidden Markov model adaptation
using maximum a posteriori linear regression," Proc. Workshop on Robust
Methods for Speech Recognition in Adverse Conditions, Tampere, Finland,
pp.147-150, 1999.
[61] O. Siohan, T. A. Myrvoll, and C.-H.Lee, \Structural maximum a posteriori
linear regression for fast HMM adaptation," Proc. ISCA ITRW ASR2000
Workshop, Paris, 2000.
[62] M. Siu and M. Ostendorf, \Variable n-grams and extensions for conversational speech language modeling," IEEE Trans. Speech Audio Processing,vol. 8, no. 1, pp. 63-75, 2000.
[63] R. M. Stern and M. J. Lasry, \Dynamic speaker adaptation for feature-based
isolated word recognition," IEEE Trans. Speech Audio Processing, vol. 35,
no. 6, pp. 751-763, 1987.

100

BIBLIOGRAPHY[64] M. Sugiyama and H. Ogawa, \A new information criterion for the selectionof subspace models," in Proc. ESANN, Bruges, pp. 69-74, 2000.
[65] A. C. Surendran, C.-H. Lee, and M. Rahim, \Non-linear compensation forstochastic matching," IEEE Trans. Speech Audio Processing, vol. 7, no. 6,
pp. 643-655, 1999.
[66] A. C. Surendran and C.-H. Lee, "Bayesian predictive approach to adaptationof HMMs," in Proc. Workshop Robust Methods for Speech Recognition inAdverse Conditions, Tampere, Finland, pp.155-158, 1999.
[67] J. Takahashi and S. Sagayama, \Vector eld smoothed Bayesian learning forincremental speaker adaptation," in Proc. ICASSP-95, Detroit, pp. 696-699,
1995.
[68] J. Takami and S. Sagayama, \A successive state splitting algorithm for efcient allophone modeling," in Proc. ICASSP-92, San Francisco, pp. I-573576, 1992.
[69] M. Tonomura, T. Kosaka, and S. Matsunaga, \Speaker adaptation basedon transfer vector eld-smoothing using maximum a posteriori probability
estimation," in Proc. ICASSP-95, Detroit, pp. 688-691, 1995.
[70] A. Tritschler and R. Gopinath, \Improved speaker segmentation and segments clustering using the Bayesian information criterion," in EuroSpeech99, Budapest, pp. 679-682, 1999.
[71] T. Watanabe, K. Shinoda, K. Takagi, and E. Yamada, \Speech recognition using tree-structured probability density function," in Proc. ICSLP-94,
Yokohama, pp. 223-226, 1994.
[72] P. C. Woodland, \Speaker adaptation: techniques and challenges," in Proc.
IEEE Automatic Speech and Understanding Workshop '99, Keystone, vol. 1,
pp. Page 85-90, 1999.

BIBLIOGRAPHY

101[73] S. J. Young, \The general use of tying in phoneme-based HMM speech recognizers," in Proc. ICASSP92, San Francisco, pp. 569-572, 1992.
[74] S. J. Young, J. J. Odell, and P. C. Woodland, \Tree-based state tying forhigh accuracy acoustic modeling," in Proc. Human Language Technology,
pp. 307-312, 1994.
[75] G. Zavaliagkos, R. Schwartz, and J. McDonough, \Maximum a posteriori
adaptation for large-scale HMM recognizers," in Proc. ICASSP-95, Detroit,
pp. 725-728, 1995.

102

BIBLIOGRAPHY

Appendix A
Derivation of Description Length
The description length is de ned as the cord length required for encoding a compound information source [43]. The MDL criterion states that the probabilistic
model which minimizes the description length is the optimal model for given data.
Let us assume a set of models f1; . . . ; i; . . . ; I g and data xN = fx1 ; . . . ; xN g are
given. The description length DL(i) for model i is the sum of three parts: thedata description length DL1(i), the parameter description length DL2(i), andthe model description length DL3(i),DL(i) = DL1(i) + DL2(i) + DL3(i):(A:1)

DL1 is the description length given the data and the model, DL2 is the description length required for coding the parameter set for the model, and DL3 is thedescription length for model selection.
In calculating DL3, it is usually assumed that the probability distribution
over the model set is uniform,
1
i = 1; . . . ; I:
(A:2)
P (i) = ;I
Then, DL3(i) is simply calculated as follows,
DL3(i) = log I:(A:3)

From now on, we focus on DL for each model. The sux i identifying the modelis therefore omitted.
103

104

APPENDIX A. DERIVATION OF DESCRIPTION LENGTH

θ3
k=3

δ1
δ3
δ2

θ2

θ1
Figure A.1: Quantization of parameter space.
Next, let us calculate DL1 for a model with parameter set . DL1 is thenegative of the log-likelihood of the model with respect to given data, and isminimum when the maximum likelihood estimate for the parameter set, ^, isused.

DL1 = 0 log P^(xN ):(A:4)

We next consider the minimization of the sum of DL1 and DL2 by choosing
the parameter set. Assuming that the dimension of the parameter space is K ,then  is a vector with K real-valued components: = (1 ; . . . ; K )T :

(A:5)

105
Since each element of the parameter vector  is a real number, an in nite cord
length is needed to encode the parameter vector. One possible way to deal withthis diculty is to discretize the parameter space. The parameter space can bediscretized by de ning a cell as a k -dimensional rectangular solid having lengthk on the axis k (see Figure A.1). The parameter vector ^ is mapped to arepresentative ~ in the same cell. If the volume of the parameter space is V , thenwe have V=(i 1 1 1 k ) cells. The sum of DL1 and DL2 is then,DL1 + DL2 = min D ():

D( ) = 0 log P~ (xN ) + log

(A.6)

V

1 1 1 1 K

;

(A.7)

where  = f1 ; . . . ; K g.
The minimization process of D ( ) over  is carried out as follows. We rst
make Taylor's expansion of the rst term,@ (0 log P (xN ))10 log P~ (x ) = 0 log P^ (x ) +1
 + N 1  T 1 IN (^ ) 1 
@
2
^
3
+O(N 1  );
(A.8)
1
2
N
@ (0 N log P (x ))IN (^ ) =
:
(A.9)
^
@2

N

N

The second term of (A.8) equals 0 because ^ is the maximum likelihood estimate.
Under certain suitable conditions, IN (^ ) is converged to a K -dimensional matrix
of constants, I (^ ), known as the Fisher information matrix, when N ! 1. Then,

D ( ) ' 0 log P^ (xN ) +

1 T ^
V
1
 1 I ( ) 1  + log:
2
1 1 1 1  K(A:10)

Di erentiating this formula with each k and setting a result equal to 0 for each,
we obtain the following equations,1(N 1 I (^ ) 1  )k 0 = 0;
k = 1; . . . ; K:
(A:11)
kSuppose that the eigenvalues of I (^ ) are 1 ; . . . ; K , and the eigenvectors are(u1; . . . ; uK ). If we only consider the case when the axes of a cell are in parallel

106

APPENDIX A. DERIVATION OF DESCRIPTION LENGTH

with the eigenvectors, then (A.11) becomes,
0

N1

B
B
B
B
B
@

1

0

1

0 C B 1 C B  C
B
CB
.. C
.. C
C
B
CB
C
C = B
CB
:
.
. C
C
B
CB
C
A
@
A@
A
1
K
K
K

(A:12)

1k = p;N 1 k(A:13)

N 1  T 1 I (^ ) 1  = K:

(A:14)

1

1

...

0

Hence, we haveand then,1

10Moreover, since 1 1 1 1 k = jI (^ )j, we have1

1 1 1 1 K

p K

= N

q

1 jI (^ )j:(A:15)

Using (A.10), (A.14), and (A.15), (A.6) becomes,pK
1DL1 + DL2 ' 0 log P (x ) + log(V 1 nK 1 jI (^ )j) + + O(p )
2
^

q

N

K
= 0 log P^ (xN ) + log N + O(1):2N(A.16)

Finally, the total description length is,

DL(i) = DL1(i) + DL2(i) + DL3(i)(i)' 0 log P^ i (xN ) + K2 log N + log I;( )(A.17)

where ^ is the maximum likelihood estimate of the parameter set of model i,and K (i) is the dimension of model i.
(i)Appendix B
Maximum A Posteriori
Estimation
We consider the case where the parametric form of the probabilistic density function (pdf) p(x), where x is a k -component vector-valued random variable, is themultivariate Gaussian pdf,


(B:1)
N (xj; 6) = (2)0 k j6j0 exp 0 12 (x 0 )T 601(x 0 ) ;while neither the mean vector  nor the variance 6 are known. Let X =fx1 ; . . . ; xN g be a set of observed samples, which are assumed to be independent and identically distributed (i.i.d.). Our goal is to estimate the parameterset  = f; 6g by using the observation samples X .
Maximum likelihood (ML) estimation is often used for this purpose. In the ML
estimation, the parameter set which maximizes the following likelihood function
is chosen,
21
2

f (Xj ) =

N
Y

n=1

p(xn j):(B:2)~ g, is calculated as follows,
The resulting maximum likelihood estimate, ~ = f~ ; 6N
1 Xx;
N n=1 nNX
6~ = N1 (xn 0 ~ )(xn 0 ~ )T :
n=1

~ =


107

(B.3)
(B.4)

108

APPENDIX B. MAXIMUM A POSTERIORI ESTIMATIONIn the maximum a posteriori (MAP) estimation [10, 18], it is assumed thatthe parameter set  is a random vector in the parameter space and it has aprior distribution p(). Let p(jX ) be the posterior pdf that is obtained after theobservation of X . Then, using Bayes' rule,p( jX ) =p(X j)p()p(Xj)p( )d

R

= C

N
Y

n=1

p(xn j)p();(B.5)

where C is a scale factor that depends on X but is independent of . The MAP
estimate ^ is de ned as the mode of the posterior pdf,

^ = argmax p( jX )= argmax

N
Y

n=1

p(xn j)p( ):(B.6)

The choice of the prior pdf is a key issue in MAP estimation. Mainly fromthe viewpoint of tractability, the conjugate prior pdf is often used; when using
it, the resulting posterior pdf is in the same family as the one that the prior pdf
belongs to. One such pdf for the multivariate Gaussian pdf N (xj; 6) is thenormal-Wishart density of the form,g (; 6j0; 60; ;  ) /




0k

1
0
T 01
0
1
j6j exp 2 ( 0 0) 6 ( 0 0 ) exp 0 2 tr(60 6 ) ; (B.7)
2where (0 ; 60 ; ;  ) are the prior density parameters such that > k 0 1,  > 0,
0 is a vector of dimension k , and 60 is a k 2 k positive de nite matrix.
^ g is the one that maximizes the following
Then, the MAP estimate ^ = f^ ; 6function,g(; 6jX ) =After simple calculations, we get

N
Y

n=1

p(xn j; 6)g (; 6): 0 + Nn=1 xn

^ =
;
P
+N
N
T
T
6^ = 60 + n=1(xn 0 ^ )((xn 00 k^))+ +N  (0 0 ^ )(0 0 ^ ) :
P

(B:8)

(B.9)
(B.10)

109
It should be noted that as the number of samples, N , increases, the MAP^ g approaches the ML estimate ~ = f~ ; 6~ g.
estimate ^ = f^ ; 6

110

Publication List

Reviewed Journal Papers
1. K. Shinoda, K. Iso, and T. Watanabe, \Speaker adaptation using spectral interpolation for speaker adaptation," IEICE Trans. Fundamentals,
vol. J77-A, no. 2, pp. 120-127, 1994 (in Japanese).
2. K. Shinoda and T. Watanabe, \Speaker adaptation using autonomous model
complexity control for speech recognition," IEICE Trans. Inf., Syst., vol. J79D-II, no. 12, pp. 2054-2061, 1996 (in Japanese).
3. K. Shinoda and T. Watanabe, \MDL-based context-dependent subword
modeling for speech recognition," J. Acoust. Soc. Jpn.(E), vol. 21, no. 2,
pp.79-86, 2000.
4. T. Emori and K. Shinoda, \Vocal tract length normalization using rapid
maximum-likelihood estimation for speech recognition," IEICE Trans. Inf.,
Syst., vol. J83-D-II, no. 11, 2000 (in Japanese).
5. K. Shinoda and C.-H. Lee, \A structural Bayes approach to speaker adaptation", IEEE Trans. Speech Audio Processing (to appear March 2001).
111

112

Reviewed Conference Papers
1. K. Shinoda, K. Iso, and T. Watanabe, \Speaker adaptation for demi-syllable
based speech recognition using continuous HMM," in Proc. of ICSLP-90,
Kobe, pp. 261-264, 1990.
2. K. Shinoda, K. Iso, and T. Watanabe, \Speaker Adaptation for DemiSyllable-Based Continuous-Density HMM," in Proc. ICASSP-91, Toronto,
pp. 857-860, 1991.
3. K. Shinoda and T. Watanabe, \Unsupervised speaker adaptation for speech
recognition using demi-syllable HMM," in Proc. ICSLP-94, Yokohama,
pp. 435-438, 1994.
4. T. Watanabe, K. Shinoda, K. Takagi, and E. Yamada, \Speech recognition
using tree-structured probability density function," in Proc. ICSLP-94,
Yokohama, pp. 223-226, 1994.
5. T. Watanabe, K. Shinoda, K. Takagi, and K. Iso, \High speed speech
recognition using tree-structured probability density function," in Proc.
ICASSP-95, Detroit, pp. 556-559, 1995.
6. K. Shinoda and T. Watanabe, \Speaker adaptation with autonomous control using tree structure," in Proc. EuroSpeech-95, Madrid, pp. 1143-1146,
1995.
7. K. Shinoda and T. Watanabe, \Speaker adaptation with autonomous model
complexity control by MDL principle," in Proc. ICASSP-96, Atlanta,
pp.717-720, 1996.
8. K. Takagi, K. Shinoda, H. Hattori and T. Watanabe, \Unsupervised andincremental speaker adaptation under adverse environmental conditions,"in Proc. ICSLP-96, Atlanta, pp. 2079-2082, 1996.

113
9. K. Shinoda and T. Watanabe, \Acoustic modeling based on the MDL criterion for speech recognition," in Proc. EuroSpeech-97, Rhodes, vol. 1,
pp. 99-102, 1997.
10. K. Shinoda and C.-H. Lee, \Structural MAP speaker adaptation using hierarchical priors," Proc. IEEE Workshop on Speech Recognition and Understanding, Santa Barbara, pp. 381-387, 1997.
11. K. Shinoda and C.-H. Lee, \Unsupervised adaptation using structural Bayes
approach," in Proc. ICASSP-98, Seattle, pp. II 793-796, 1998.

114