INCREASING DISCRIMINATIVE CAPABILITY ON MAP-BASED MAPPING FUNCTION
ESTIMATION FOR ACOUSTIC MODEL ADAPTATION
Yu Tsao, Ryosuke Isotani, Hisashi Kawai, and Satoshi Nakamura
SLC Group, National Institute of Information and Communications Technology, Kyoto, 619-0288, Japan
ABSTRACTIn this study, we propose increasing discriminative power on the
maximum a posteriori (MAP)-based mapping function estimation for
acoustic model adaptation. Based on the effective and stable learningadvantages of MAP-based estimation, we incorporate a discriminative
term and derive a new objective function. By applying the newfunction for online mapping function estimation, we developed
discriminative maximum a posteriori (DMAP) linear regression(DMAPLR) and DMAP-based ensemble speaker and speaking
environment modeling (DMAP-based ESSEM). We evaluate the
DMAPLR and DMAP-based ESSEM on the Aurora-2 task in asupervised adaptation mode. The experimental results show that both
DMAPLR and DMAP-based ESSEM consistently provide
improvements over their ML-based and MAP-based counterpartsirrespective of using one, two, or three adaptation utterances. Fromthe improvements, we confirm the strong effect of increasing
discriminative capability on the MAP-based mapping function
estimation. Moreover, we verify that including multiple knowledge
sources in the objective function can efficiently enhance model
adaptation performance. When compared with the baseline result,
DMAP-ESSEM achieves a 15.96% (9.21% to 7.74%) average word
error rate (WER) reduction using only one adaptation utterance.
Index Terms-Automatic speech recognition, MLLR, ESSEM,MAPLR, MAP-based ESSEM, discriminative training
1. INTRODUCTION
Increasing discriminative power on acoustic models is known as aneffective way to improve speech recognition performance underimperfect testing environments [1-3]. Many discriminative training(DT) methods have recently been proposed. These methods typicallyfirst define a particular objective function that measures theseparation between parameters in the acoustic models. An
optimization procedure is then performed on the objective function
with the available training set to increase the separation between
parameters. Popular examples include minimum classification error(MCE) [1], minimum phone error (MPE) [2], and soft marginestimation (SME) [3] methods. Because these DT methods target
increasing parameter separations, we usually apply them to refine
acoustic models after a maximum likelihood-based training.
In addition to DT, model adaptation is another successful way
to reduce acoustic mismatches between training and testing
conditions. Model adaptation approaches usually adopt sometransformation functions to characterize the environment mismatches.
A particular optimization criterion is used to find the parameters of
the transformation function with a set of adaptation data that containacoustic information of the testing conditions. Maximum likelihood(ML) is a popular criterion for performing the optimization.
Maximum likelihood linear regression (MLLR) [4] and ML-based
ensemble speaker and speaking environment modeling (ESSEM) [5]
are two successful examples for the ML-based model adaptation
approaches. Though these ML-based approaches can provide
satisfactory results when sufficient adaptation data are available,
their performance may become unstable due to an over-fitting issue
when the amount of adaptation data is too limited. To avoid overfittings, a class of approaches adopts prior knowledge and uses a

978-1-4577-0539-7/11/$26.00 ©2011 IEEE

5320

maximum a posteriori (MAP) criterion to estimate transformation
function. Corresponding to the previous two examples, their MAPbased counterparts, maximum a posteriori linear regression (MAPLR)[6, 7] and MAP-based ESSEM [8], have been proposed.
More recently, approaches combining the advantages of DT andacoustic model adaptation have been developed. Minimum
classification error linear regression (MCELR) [9] and soft marginestimation-based linear regression (SMELR) [10] are proposed torefine transformation functions by increasing their discriminative
power. In this paper, we propose increasing the discriminative
capability on the MAP-based model adaptation methods by derivinga new objective function. We apply the new function on the two
MAP-based approaches, MAPLR and MAP-based ESSEM, anddevelop discriminative MAPLR (DMAPLR) and discriminative
MAP-based ESSEM (DMAP-based ESSEM), respectively. We
evaluated DMAPLR and DMAP-based ESSEM on the Aurora-2 task
[11] in a supervised adaptation mode. Experimental results indicatethat both DMAPLR and DMAP-based ESSEM can provide clearly
better performance than their MAP-based counterparts.
In a previous study, a confidence score of a combination of
likelihood and likelihood ratio (LR) has been used to extend theconventional speech recognizer to a hybrid decoder [12]. Because thescore integrates multiple knowledge sources, the hybrid decoder
provides better recognition results than the conventional decoder thatuses either the likelihood or LR score alone. The proposed DMAPLR
and DMAP-based ESSEM share a similar concept that adopts a
combined score in the objective function to estimate transformation
functions. To investigate the effect of using multiple knowledge
sources, we designed another set of experiments. We compared theobjective functions using a combined score, namely, DMAP-based
approaches, versus using likelihood and LR scores alone (all theseobjective functions integrate priors to improve performance stability).
Our experimental results indicated that the DMAP-based approaches
do give better performance than the counterpart methods using
likelihood or LR alone. This set of results confirmed that byincorporating multiple knowledge sources in the objective function,
we can further enhance the model adaptation ability.
2. DISCRIMINATIVE MAP-BASED OBJECTIVE FUNCTION
Typically, model adaptation techniques use a mapping function, G߮ ,to transform parameters in the original acoustic model sets, Ω , (Ωmay include one or multiple sets of acoustic models) to a new set ofഥ  , that matches the testing condition by:
acoustic models,Λഥ  ൌ G߮ ሺΩ ሻǤሺͳሻ
Λ
If the transcription, ܹܿ , corresponding to the adaptation data,  , isavailable, ML-based adaptation defines a objective function by:‫ܮ‬ሺ ǡ ߮ǡ Ω ǡ ܹܿ ሻ ൌ ݈‫݃݋‬ሾܲሺ ȁ߮ǡ Ω ǡ ܹܿ ሻሿǤሺʹሻWe estimate the parameters, ߮ො‫ ܮܯ‬, in the function, G߮ , by:߮ො‫ ܮܯ‬ൌ ܽ‫ܮݔܽ݉݃ݎ‬ሺ ǡ ߮ǡ Ω ǡ ܹܿ ሻǤሺ͵ሻ
߮

The MAP-based model adaptation methods, on the other hand, use thefollowing objective function:

ICASSP 2011‫ܯ‬ሺ ǡ ߮ǡ Ω ǡ ܹܿ ሻ ൌ ݈‫݃݋‬ሾܲሺ ȁ߮ǡ Ω ǡ ܹܿ ሻ‫݌‬ሺ߮ǡ Ω ǡ ܹܿ ሻሿǤሺͶሻ
We can calculate parameters, ߮ො‫ ܲܣܯ‬, in G߮ by the optimization of:‫ݐ‬ൌͳ ‫ܹܿאݏ‬߮ො‫ ܲܣܯ‬ൌ ܽ‫ܯݔܽ݉݃ݎ‬ሺ ǡ ߮ǡ Ω ǡ ܹܿ ሻǤሺͷሻ߮Many previous studies have verified that the likelihood ratio (LR)score can provide crucial discrimination information for acoustic
modeling [1-3]. To calculate LR, we require competing lists to thephone/word units in the correct transcription, ܹܿ . In this study, we use
N-best hypotheses by decoding adaptation utterances to obtain thecompeting units. Then, we define an LR-based objective function by:ܰ෩ͳ ǥ ܹ෩ܰ ሻ ൌ ෍ ɉ݊ ݈‫ ݃݋‬ቈ‫ܦ‬ሺ ǡ ߮ǡ ȳ ǡ ܹܿ ǡ ܹ݊ൌͳܲሺ ȁ߮ǡ ȳ ǡ ܹܿ ሻ෩݊ ሻܲሺ ȁ߮ǡ ȳ ǡ ܹ቉ ǡሺ͸ሻwhere λ݊ is a scaling factor that determines the weights of each of෩
the N-best hypotheses, with σܰ݊ൌͳ λ݊ ൌ ͳ , and ܹ݊ is the n-thcompeting phone/word sequence of the N-best lists.
In this study, we derive a new objective function that combines
the objective function of Eq-(4) and the objective function of Eq-(6):෩ͳ ǥ ܹ෩ܰ ሻ ൌ  ߙͳ ‫ܯ‬ሺ ǡ ߮ǡ Ω ǡ ܹܿ ሻ ൅‫ܭ‬ሺ ǡ ߮ǡ Ω ǡ ܹܿ ǡ ܹ෩ͳ ǥ ܹ෩ܰ ሻǡሺ͹ሻߙʹ ‫ܦ‬ሺ ǡ ߮ǡ Ω ǡ ܹܿ ǡ ܹ
where ߙͳ and ߙʹ are weighting parameters. Accordingly, we estimate
the parameters, ߮ො‫ ܲܣܯܦ‬, in the mapping function, G߮ , by:෩ͳ ǥ ܹ෩ܰ ሻǤሺͺሻ߮ො‫ ܲܣܯܦ‬ൌ ܽ‫ܭݔܽ݉݃ݎ‬ሺ ǡ ߮ǡ ȳ ǡ ܹܿ ǡ ܹ
߮

ܶ

݄ͳ
݄ʹ
‫ ߦ ݋‬൅ߟ ߦ ቉ െ
ȭ‫ݏ‬ሺ݅ሻ ‫ ݐ‬ሺ݅ሻ ‫ݏ ݏ‬ሺ݅ሻ ‫ݏ‬ሺ݅ሻ ‫ݏ‬݄ܰ͵෍ ෍ ෍ ɉ݊ ‫ ݈ݎ‬ሺ‫ݐ‬ሻ ቆ‫ ߦ ݋‬ቇ ǡ ሺͳͶሻȭ݈ሺ݅ሻ ‫ ݐ‬ሺ݅ሻ ݈෩݊‫ݐ‬ൌͳ ݊ൌͳ ݈‫ܹא‬෩݊ represents the l-th Gaussian belonging to a competing
where ݈ ‫ܹ א‬
෩݊ . From Eq-(7) to Eqs-(13) and (14),model in the n-th N-best list,ܹ
we set ݄ͳ ൌ ሺߙͳ ൅ ߙʹ ሻ, ݄ʹ ൌ ሺߙͳ Ȁߨሻ , and ݄͵ ൌ ߙʹ , where ߨ is ascaling factor that controls the weight of priors. ߦ݈ is the augmented
mean vector, and ȭ݈ሺ݅ሻ is the i-th element of the covariance matrix, forthe l-th Gaussian. For the s-th Gaussian (from the correcttranscription), we prepare hyper-parameters,ߟ‫ ݏ‬and ‫ ݏ‬, in the offline.
With the calculated ʒ, we can adapt mean parameters by Eq-(9).
3.2. Discriminative MAP-based ESSEM
In our previous ESSEM study, we introduced several different typesof mapping functions, G߮ , in Eq-(1). Here, we present a linear
combination function as an example. Other types of mapping
function can be derived in a similar manner. For the ESSEM
approach, an environment structure consisting of multiple acoustic
model sets is taken for Ω in Eq-(1). Accordingly, for the m-thʹܲ ሽ,Gaussian, we have an environment structure, Η݉ ൌ ሼߤͳ݉ ǡ ߤ݉ǥ ߤ݉‫݌‬where ߤ݉ is the m-th mean vector for the p-th speaker and speaking
environment. ESSEM calculates the adapted mean vector ߤҧ݉ by:ߤҧ݉ ൌ Ȣ݉ ߱ǤሺͳͷሻWe estimate the parameters of the linear combination function, ߱, by:

3. DISCRIMINATIVE MAPLR AND MAP-BASED ESSEM
In this section, we present DMAPLR and DMAP-based ESSEM that
use the objective function in Eq-(7) for estimating mapping function.
3.1. Discriminative MAPLR߱ ൌ ‫ ܩ‬െͳ ݇Ǥሺͳ͸ሻ
If the ML criterion is used to calculate ߱, we have:ܶWhen using the affine transformation as the mapping function, theenvironment structure, ȳ , in Eq-(1) only takes one set of acoustic
models. For the m-th Gaussian in the acoustic model, we have:ߤҧ݉ ൌ ʒߦ݉ ǡሺͻሻ
ሺͳሻ
ሺ‫ܦ‬ሻ
ሺ݅ሻ
where ߦ݉ is the augmented vector: ሾߤ݉ ǡ ǥ ǡ ߤ݉ ǡ ͳሿ′, ߤ݉ is the i-th
element of the m-th Gaussian component. ߤҧ ݉ is the adapted mean
vector. MLLR calculates the affine transformation, ʒ, by [4]:ʒሺ݅ሻ ൌ ሺ‫ܩ‬ሺ݅ሻ ሻെͳ ݇ሺ݅ሻ ǡሺͳͲሻ
along withܶͳ‫ܩ‬ሺ݅ሻ ൌ ෍ ෍ ‫ ݏݎ‬ሺ‫ݐ‬ሻ ቈ
ߦ ߦ ′ ቉ ǡ ሺͳͳሻ
ȭ‫ݏ‬ሺ݅ሻ ‫ݏ ݏ‬‫ݐ‬ൌͳ ‫ܹܿאݏ‬ܶͳ‫ ߦ ݋‬቉ ǡ ሺͳʹሻ݇ሺ݅ሻ ൌ ෍ ෍ ‫ ݏݎ‬ሺ‫ݐ‬ሻ ቈȭ‫ݏ‬ሺ݅ሻ ‫ ݐ‬ሺ݅ሻ ‫ݏ‬‫ݐ‬ൌͳ ‫ ܹܿאݏ‬where ‫ ݐ݋‬is the t-th observation. ‫ ݏݎ‬ሺ‫ݐ‬ሻ is the posterior probability at
the t-th observation, ߦ‫ ݏ‬is the augmented mean vector, and ȭ‫ݏ‬ሺ݅ሻ is thei-th element of the covariance matrix of the s-th Gaussian,respectively. ‫ ܹܿ א ݏ‬indicates that the s-th Gaussian belongs to atarget model in the correct transcription, ܹܿ .
By using the combined score of Eq-(7) for the objectivefunction, along with the N-best information, DMAPLR calculates theaffine transformation, ʒ, also using Eq-(10), but with:ܶ‫ܩ‬ሺ݅ሻ ൌ ෍ ෍ ‫ ݏݎ‬ሺ‫ݐ‬ሻ ቈ‫ݐ‬ൌͳ ‫ܹܿאݏ‬ܶܶ݇ሺ݅ሻ ൌ ෍ ෍ ‫ ݏݎ‬ሺ‫ݐ‬ሻ ቈ݄ͳ݄ʹߦ ߦ′ ൅
ߦ ߦ′ ቉ െȭ‫ݏ‬ሺ݅ሻ ‫ݏ ݏ ݏ‬ሺ݅ሻ ‫ݏ ݏ‬‫ݐ‬ൌͳ ‫ܹܿאݏ‬ܶ݇ ൌ ෍ ෍ ‫ ݏݎ‬ሺ‫ݐ‬ሻൣȢ‫ݏ‬′ ȭ‫ݏ‬െͳ ‫ ݐ݋‬൧ Ǥሺͳͺሻ‫ݐ‬ൌͳ ‫ܹܿאݏ‬For DMAP-based ESSEM, we also use Eq-(16) to calculate ߱, while:ܶ‫ ܩ‬ൌ ෍ ෍ ‫ ݏݎ‬ሺ‫ݐ‬ሻൣ݄ͳ Ȣ‫ݏ‬′ ȭ‫ݏ‬െͳ Ȣ‫ ݏ‬൅ ݄ʹ Ȣ‫ݏ‬′ ‫ݏ‬െͳ Ȣ‫ ݏ‬൧ െ‫ݐ‬ൌͳ ‫ܹܿאݏ‬ܶܰ෍ ෍ ෍ λ݊ ‫ ݈ݎ‬ሺ‫ݐ‬ሻ൫݄͵ Ȣ݈′ ȭ݈െͳ Ȣ݈ ൯ǡ ሺͳͻሻܶ෩݊‫ݐ‬ൌͳ ݊ൌͳ ݈‫ܹא‬݇ ൌ ෍ ෍ ‫ ݏݎ‬ሺ‫ݐ‬ሻൣ݄ͳ Ȣ‫ݏ‬′ ȭ‫ݏ‬െͳ ‫ ݐ݋‬൅ ݄ʹ Ȣ‫ݏ‬′ ‫ݏ‬െͳ ߟ‫ ݏ‬൧ െ
‫ݐ‬ൌͳ ‫ܹܿאݏ‬ܶܰ෍ ෍ ෍ λ݊ ‫ ݈ݎ‬ሺ‫ݐ‬ሻ൫݄͵ Ȣ݈′ ȭ݈െͳ ‫ ݐ݋‬൯Ǥ ሺʹͲሻ
෩݊‫ݐ‬ൌͳ ݊ൌͳ ݈‫ܹא‬

Similarly,݄ͳ ൌ ሺߙͳ ൅ ߙʹ ሻ, ݄ʹ ൌ ሺߙͳ Ȁߨሻ, and ݄͵ ൌ ߙʹ , ߨ is a scaling
factor. Ȣ‫ ݏ‬and Ȣ݈ are the environment structures, and ȭ‫ ݏ‬and ȭ݈ arethe covariance matrices, respectively, for the s-th Gaussian and thecompeting l-th Gaussian. For the s-th Gaussian, we also prepare
hyper-parameters, ߟ‫ ݏ‬and ‫ ݏ‬, for its prior density.
4. EXPERIMENTS݄ܰ͵ߦ ߦ′ ቇ ǡ෍ ෍ ෍ λ݊ ‫ ݈ݎ‬ሺ‫ݐ‬ሻ ቆȭ݈ ሺ݅ሻ ݈ ݈‫ ܩ‬ൌ ෍ ෍ ‫ ݏݎ‬ሺ‫ݐ‬ሻൣȢ‫ݏ‬′ ȭ‫ݏ‬െͳ Ȣ‫ ݏ‬൧ ǡሺͳ͹ሻ

ሺͳ͵ሻ

෩݊
‫ݐ‬ൌͳ ݊ൌͳ ݈‫ܹא‬

5321

In this section, we first introduce the experimental setup. Then, we
present experimental results of DMAPLR and DMAP-based ESSEM
and their ML- and MAP-based counterparts. Finally, we compare theresults of these approaches and discuss our findings.

4.1. Experimental Setup

4.2. Experimental Results

We evaluated the proposed DMAPLR and DMAP-based ESSEM
approaches on the Aurora-2 task [11]. The multi-condition training
set in Aurora-2 was used to estimate acoustic models. This training
set included 17 different speaking environments from the same fourtypes of noise as in test SetA, at different SNRs: 5dB, 10dB, 15dB,
20dB, and clean condition. By further dividing the training set bygenders, we obtained training data for 34 different speaker andspeaking environments. In this paper, we report performance on 50
different conditions (ten noise types at five SNR levels: 0dB, 5dB,
10dB, 15dB, and 20dB). Each speech frame was characterized by 39
coefficients consisting of 13 MFCC with their first- and second-orderderivatives. A cepstral mean subtraction (CMS) was performed fornormalization. All digits were modeled by 16-state whole word
hidden Markov models (HMMs) with each state characterized bythree Gaussian mixture components. The silence and the short pause
were modeled by three states and one state, respectively, with eachstate characterized by six Gaussian mixture components.
In the experiments, we constructed an environment structurethat consists of environment clustering (EC) and environment
partitioning (EP) hierarchical tree structures [5] by using the training
data. We first clustered the training data into C groups based on the
EC tree. The root node comprised the entire set of training data. The
second layer consisted of two nodes, each including training data for
one gender. Then, each gender-specific cluster in the second layer
was further divided into two clusters based on high/low SNR
conditions. Therefore, the 34 training environments were classified
into seven clusters (C=7). Then, we trained a representative HMM
set for each EC cluster using the training data belonging to thatcluster. Each representative HMM set was then used to build an EP
tree structure to cluster mean parameters in the representative HMM
set. Each EP tree had one root, three intermediate, and six leaf nodes.
The weighted Euclidean distance was used as the distance measure
between each pair of mean vectors. During adaptation, we estimateda mapping function for every EP node. Each mean vector searched
for the EP node containing sufficient adaptation statistics and used
its mapping function for adaptation. Along with the EC and EP
environment structures, we also prepared a hyper-parameter set foreach mean vector: hyper-parameters, ሼߟ݉ ǡ ݉ ሽ, for the m-th Gaussian.
The hyper-parameters were used to calculate mapping functions inEqs-(13) and (14) and in Eqs-(19) and (20). Detailed information
about the construction of the EC and EP structures and hyperparameter estimation can be found in our previous study [7, 8].
Each of the 50 testing conditions in Aurora-2 has 1001 speechutterances recorded from 104 speakers (52 male and 52 female).
Each speaker pronounced nine to ten utterances, and these testing
speakers did not participate in the training phase. We used the first
three utterances as the adaptation set and the remaining six or sevenutterances for the testing set. Accordingly, each testing condition
included 312 (104×3) adaptation utterances and 689 (1001-312)
testing utterances. For all the model adaptation experiments in thispaper, we performed four steps on each testing speaker. First, we
implemented a cluster selection (CS) to locate the best suitable EC
cluster whose representative HMM set gave the highest likelihoodfor the speaker’s adaptation data. Second, with the located EC
cluster, we conducted another searching process through its EP tree
to choose an EP node with sufficient adaptation statistics. Third,with the chosen EP node, each mean vector was adapted using the
mapping function for that EP node. Fourth, we used the adapted
HMMs to decode the testing utterances from that same speaker. For
each testing condition, we performed the above four steps 104 times
and calculated an average WER of the overall results (from the 689
testing utterances pronounced by the 104 testing speakers).

This section presents our experimental results. A baseline result isprovided for comparison and listed as Baseline in the following
discussion. To obtain this Baseline result, a CS process is first
performed to find a representative HMM set. The selected HMM set
is directly used for testing recognition without performing adaptation.

5322

4.2.1. Discriminative MAPLR
Figure 1 reports average WERs of MLLR, DMLLR (discriminative
MLLR), MAPLR, and DMAPLR (discriminative MAPLR). In apreliminary experiment, we tested performance using different
combinations of ݄ͳ , ݄ʹ , and ݄͵ , in Eqs-(13) and (14). Here, we only
present the setup that gives the best performance. Our setup in thisset of experiments is: for MLLR, {݄ͳ =1.0, ݄ʹ =0.0, ݄͵ =0.0}; for
DMLLR, {݄ͳ =1.0, ݄ʹ =0.0, ݄͵ =0.6}; for MAPLR, {݄ͳ =1.0, ݄ʹ = 0.2,
݄͵ =0.0}; for DMAPLR, {݄ͳ =1.0, ݄ʹ =0.2, ݄͵ =0.6}. In this study, weused a diagonal regression matrix forʒ in Eq-(9). Meanwhile, we
set ɉ݊ ൌ ͳȀܰǡ ‫ ݊׊‬, and N=8 in Eqs-(13) and (14). Each result in
Figure 1 shows an average WER over 50 testing conditions.
From Figure 1, we can first see an obvious improvement by
A to ˕
C ), and an additional clear gain isusing the prior density (˕C to ˕
D ) whenachieved by incorporating the discriminative term (˕using one, two, and three adaptation utterances. Next, we observethat though the improvement from MLLR to DMLLR is marginal
A to ˕
B ), DMAPLR gives clear improvements over MAPLR (˕C to(˕D ). This result indicates that by increasing discrimination on an ML˕
based transformation estimation, the improvement may not be clear,especially when there is limited adaptation data (one adaptationutterance). However, by incorporating the discriminative term into
the objective function, MAP-based estimation can gain clear benefits.
9.09
A

B

8.8

8.6
8.6

C

D

WER (%)

8.4

A

B

A

C

B

8.2
8.2

C

D
8

D
7.8
7.8

7.6

7.4
7.4

1
1 utt.

2
2 utt.

A MLLR; B DMLLR; C MAPLR;

3
3 utt.

D DMAPLR

Figure 1. WERs (%) with different numbers of adaptation utterances
In real world applications, we are more interested in rapid
adaptation with a very small amount of adaptation data. Therefore,
we list the results of using one adaptation utterance in Table I. Eachblock in Table I shows an average WER over ten conditions, and thebest result for each SNR condition (each column) is marked withbold font. In addition to the four tests in Figure 1, we design another
setup {݄ͳ =1.0, ݄ʹ =0.2, ݄͵ =1.0} and name it the maximum likelihood
ratio linear regression (MLRLR) in the following discussion.
From Table I, we observe that DMAPLR consistently provides
lower WERs than not only Baseline but also the other four
approaches over 0dB to 20dB conditions (only except SNR=10dB).
Special note is made that DMAPLR uses a combined score of
likelihood plus LR in the objective function, MLRLR uses LR alone,and MAPLR uses likelihood alone, wherein all three methods adopt
prior density. Therefore, the improvements achieved by DMAPLR
over MAPLR and MLRLR suggest that the combined scoreincorporating multiple knowledge sources can further enhance affine
transformation-based model adaptation methods.
TABLE I. AVERAGE WER (%) FOR USING ONE ADAPTATION UTTERANCE
SNR(dB)
Baseline
MLLR
DMLLR
MAPLR
DMAPLR
MLRLR

20
1.56
1.55
1.55
1.53
1.51
1.55

15
2.10
2.09
2.09
2.05
2.03
2.05

10
3.13
3.59
3.60
3.45
3.41
3.21

5
8.21
8.77
8.75
8.37
8.16
8.30

0
31.04
28.15
28.12
27.40
26.94
27.99

Ave.
9.21
8.83
8.82
8.56
8.41
8.62

4.2.2. Discriminative MAP-based ESSEM
Figure 2 reports results of ML-based ESSEM, DML-based ESSEM,
MAP-based ESSEM, and DMAP-based ESSEM. We use the same
setups as that used in the previous experiment: for ML-based
ESSEM, {݄ͳ =1.0, ݄ʹ =0.0, ݄͵ =0.0}; for DML-based ESSEM, {݄ͳ=1.0, ݄ʹ =0.0, ݄͵ =0.6}; for MAPLR, {݄ͳ =1.0, ݄ʹ =0.2, ݄͵ =0.0}; for
DMAPLR, { ݄ͳ =1.0, ݄ʹ =0.2, ݄͵ =0.6}. Similarly in this set ofexperiments, we set ɉ݊ ൌ ͳȀܰǡ ‫݊׊‬, N=8 in Eqs-(19) and (20).
From Figure 2, we observe very similar phenomena to Figure 1
for one, two, or three adaptation utterances: First, an obvious gain is
A to ˕
C ), and an
obtained by incorporating the prior probability, (˕Cadditional improvement is achieved by enhancing discrimination, (˕D ). Second, by increasing discrimination, MAP-based estimationto ˕C to ˕
D ) than the ML-based
can achieve more improvements ( ˕A to ˕
B ). We also reported the results of ESSEM using
estimation, (˕one adaptation utterance in Table II. In addition to the four results in
Figure 2, we tested the maximum likelihood ratio-based ESSEM(indicated as MLR-based ESSEM) with {݄ͳ =1.0, ݄ʹ =0.2, ݄͵ =1.0}.
8.20
8.2

8.00

A

B
C

7.80
7.8

D

A

WER (%)

B
C

7.60

A

D

B

C
D

7.40
7.4

7.20

7.00
7.0

6.80

6.6
6.60

1
1 utt.

2
2 utt.

3
3 utt.

A ML-ESSEM; B DML-ESSEM; C MAP-ESSEM; D DMAP-ESSEM

Figure 2. WERs (%) with different numbers of adaptation utterances
Similar observations are obtained to that from Table I: DMAPESSEM gives better performance than MAP-ESSEM, MLE-ESSEM,
or MLR-based ESSEM consistently over almost all SNR conditions.
The results again confirm that the objective function using a
combined score performs better than those using individual scores.
Comparing to Baseline, DMAP-ESSEM provides a performance
improvement of 15.96% (9.21% to 7.74%) average WER reduction.
TABLE II. AVERAGE WER (%) FOR USING ONE ADAPTATION UTTERANCE
SNR(dB)
Baseline
ML-ESSEM
DML-ESSEM
MAP-ESSEM
DMAP-ESSEM
MLR-ESSEM

20
1.56
1.31
1.30
1.31
1.31
1.32

15
2.10
1.73
1.74
1.77
1.77
1.88

10
3.13
2.96
2.94
2.94
2.92
2.95

5
8.21
7.87
7.91
7.60
7.53
7.53

0
31.04
26.06
26.02
25.35
25.18
25.26

Ave.
9.21
7.99
7.98
7.79
7.74
7.79

5323

5. CONCLUSIONS
We designed an objective function that uses a combined score ofposterior probability and likelihood ratio. We used the proposed
objective function to estimate mapping functions and developed
DMAPLR and DMAP-based ESSEM. Compared to their ML-based,
MAP-based, and MLR-based counterparts, DMAPLR and DMAPbased ESSEM provide notable improvements in a supervised
adaptation mode on Aurora-2. The results indicate that increasing
discriminative capability can effectively enhance MAP-based model
adaptation performance. Moreover, from the improvements, it isconfirmed that by using multiple knowledge sources in the objectivefunction, the adaptation performance can be improved. Comparing toour baseline, DMAP-ESSEM achieves a 15.96% (9.21% to 7.74%)
average WER reduction using only one adaptation utterance.
In the SME algorithm [3], a frame selection procedure is taken
to improve the separations between parameters in acoustic models.
In the future, we will incorporate that procedure into DMAPLR and
DMAP-based ESSEM approaches. Meanwhile, some previous studies
pointed out that by setting different weights for eachλ݊ (n=1…N) in
Eqs-(13) and (14), (19), and (20), the discrimination can be further
increased. We will also explore that research direction in the future.
6. REFERENCES[1] B.-H. Juang, W. Chou, and C.-H. Lee, “Minimum classification
error rate methods for speech recognition,” IEEE Trans. Speech
Audio Processing, vol. 5, pp. 257-265, 1997.
[2] D. Povey and P. C. Woodland, “Minimum phone error and ismoothing for improved discriminative training,” in Proc.
ICASSP, pp. I105-I108, 2002.
[3] J. Li, M. Yuan, and C.-H. Lee, “Approximate test risk boundminimization through soft margin estimation,” IEEE Trans. on
Audio, Speech, and Language Processing, vol. 15, pp. 23932404, 2007.
[4] C. J. Leggetter and P. C. Woodland, “Maximum likelihood
linear regression for speaker adaptation of continuous density
hidden Markov models,” Computer Speech and Language, vol.
9, pp. 171-185, 1995.
[5] Y. Tsao and C.-H. Lee, “An ensemble speaker and speaking
environment modeling approach to robust speech recognition,”
IEEE Trans. on Audio, Speech, and Language Proc., vol. 17, pp.
1025-1037, 2009.
[6] C. Chesta, O. Siohan, and C.-H. Lee, “Maximum a posteriori
linear regression for hidden Markov model adaptation,” in Proc.
Eurospeech, pp. 211-214, 1999.
[7] Y. Tsao, R. Isotani, H. Kawai, and S. Nakamura, “Anenvironment structuring framework to facilitating suitable priordensity estimation for MAPLR on robust speech recognition,”
to appear in Proc. ISCSLP, 2010.
[8] Y. Tsao, S. Matsuda, S. Nakamura, and C.-H. Lee, “MAP
estimation of online mapping parameters in ensemble speakerand speaking environment modeling,” in Proc. ASRU, pp. 271275, 2009.
[9] J. Wu and Q. Huo, “A study of minimum classification error(MCE) linear regression for supervised adaptation of MCEtrained continuous-density hidden Markov models,” IEEE
Trans. Speech Audio Processing, vol. 15, pp. 478-488, 2007.
[10] S. Matsuda, Y. Tsao, J. Li, S. Nakamura, and C.-H. Lee, “A study
on soft margin estimation of linear regression parameters forspeaker adaptation,” in Proc. Interspeech, pp. 1603-1606, 2009.
[11] D. Macho, L. Mauuary, B. Noe, Y. M. Cheng, D. Ealey, D.
Jouver, H. Kelleher, D. Pearce, and F. Saadoun, “Evaluation ofa noise-robust DSR front-end on Aurora databases,” in Proc.
ICSLP, pp. 17-20, 2002.
[12] M.-W. Koo, C.-H. Lee, and B.-H. Juang, “Speech recognition andutterance verification based on a generalized confidence score,”
IEEE Trans. Speech Audio Processing, vol. 9, pp. 821-832, 2001.