#2016 The Acoustical Society of Japan

Acoust. Sci. & Tech. 37, 6 (2016)

PAPER

Performance estimation of noisy speech recognition
using spectral distortion and recognition task complexity
Ling Guo, Takeshi Yamaday, Shigeki Miyabez,
Shoji Makinox and Nobuhiko Kitawaki}
Graduate School of Systems and Information Engineering, University of Tsukuba,
1–1–1 Tennodai, Tsukuba, 305–8573 Japan(Received 2 December 2015, Accepted for publication 1 June 2016)
Abstract: Previously, methods for estimating the performance of noisy speech recognition based ona spectral distortion measure have been proposed. Although they give an estimate of recognition
performance without actually performing speech recognition, no consideration is given to any change
in the components of a speech recognition system. To solve this problem, we propose a novel method
for estimating the performance of noisy speech recognition, a major feature of which is the ability toaccommodate the use of diﬀerent noise reduction algorithms and recognition tasks by using two
cepstral distances (CDs) and the square mean root perplexity (SMR-perplexity). First, we veriﬁed theeﬀectiveness of the proposed distortion measure, i.e., the two CDs. The experimental results showedthat the use of the proposed distortion measure achieves estimation accuracy equivalent to the use of
the conventional distortion measures, the perceptual evaluation of speech quality (PESQ) and thesignal-to-noise ratio (SNR) of noise-reduced speech, and has the advantage of being applicable tonoise reduction algorithms that directly output the mel-frequency cepstral coeﬃcient (MFCC) feature.
We then evaluated the proposed method by performing a closed test and an open test (10-fold crossvalidation test). The results conﬁrmed that the proposed method gives better estimates without being
dependent on the diﬀerences among the noise reduction algorithms or the recognition tasks.
Keywords: Performance estimation, Noisy speech recognition, Noise reduction, Spectral distortion,
Recognition task complexity
PACS number: 43.60.Lq, 43.66.Jh

1.

[doi:10.1250/ast.37.286]

INTRODUCTION

Speech recognition services are becoming more prevalent with the spread of smartphones and tablets. However,current speech recognition systems still have a serious
problem, namely, the recognition performance is degradedin noisy environments [1]. The degree of performancedegradation depends on the nature of ambient noise. To
ensure a satisfactory quality of experience (QoE) and to
determine a system conﬁguration suitable for each individual noise environment before starting a speech recognition service, it is essential to establish a method that canbe used to eﬃciently investigate recognition performancein diﬀerent noisy environments.
e-mail: guoling@mmlab.cs.tsukuba.ac.jp
e-mail: takeshi@cs.tsukuba.ac.jpze-mail: miyabe@tara.tsukuba.ac.jp
xe-mail: maki@tara.tsukuba.ac.jp}
e-mail: kitawaki.nobuhiko.gu@u.tsukuba.ac.jpy286

One typical approach is to prepare noisy speech data inthe target noise environment and then perform a recognition experiment. However, this has high computational
complexity and is time-consuming. An alternative approach to this problem is to estimate recognition performance based on the spectral distortion between noisy speech
and its original clean version [2–4]. The original clean
speech is available since we assume that the noisy speech
is generated by recording the noise in diﬀerent noisy
environments and artiﬁcially adding it to the clean speech.
This assumption is reasonable for both approaches fromthe viewpoint of reducing the recording cost. In the latterapproach, a signiﬁcant reduction of the amount of speech
data to be processed is expected by using artiﬁcial voice
signals with average speaker characteristics [3].
These methods give an estimate of recognition performance without actually performing speech recognition.
Previously, we proposed a performance estimation method
using the perceptual evaluation of speech quality (PESQ)

L. GUO et al.: PERFORMANCE ESTIMATION OF NOISY SPEECH RECOGNITION[5] as a distortion measure [3]. In this method, an estimator,
which is a function of the PESQ score, is trained on thebasis of the relationship between the recognition performance and the PESQ score. Fukumori et al. also proposed amethod using the PESQ and room acoustic parameters toestimate recognition performance in noisy and reverberant
environments [4].
Although these methods can give an accurate estimateof recognition performance, no consideration is given to
any change in the components of a speech recognition
system. For example, a noise reduction algorithm, which isused in the preprocessing stage, may be changed dependingon the nature of the ambient noise in order to achieve better
recognition performance. A recognition task, which decides what a speech recognition system is capable ofrecognizing, is often changed according to the content of aservice. In general, a complex task makes speech recognition diﬃcult. These changes require an estimator specialized for each individual noise reduction algorithm andrecognition task. The training of such a specialized
estimator is however labor intensive and time-consuming.
To solve this problem, we recently proposed a performance estimation method using the signal-to-noise ratio(SNR) of noise-reduced speech in addition to the PESQ
as distortion measures [6]. This method can estimate
recognition performance without being dependent on thediﬀerences among noise reduction algorithms. We also
proposed a method using the PESQ as a distortion measureand square mean root perplexity (SMR-perplexity) [7] as atask complexity measure [8]. It allows recognition performance to be estimated for diﬀerent recognition tasks. In
this paper, we integrate these two methods to handle both
diﬀerent noise reduction algorithms and diﬀerent recognition tasks, and propose a novel performance estimation
method. The proposed method estimates recognition
performance using two cepstral distances (CDs) and the
SMR-perplexity. The estimator is deﬁned as a function ofthese three variables and can be used with diﬀerent noise
reduction algorithms and for diﬀerent recognition tasks
without any additional training.
The rest of this paper is organized as follows. In
Sect. 2, the proposed method is explained in detail. In
Sect. 3, we evaluate the eﬀectiveness of the proposed
method. Section 4 summarizes the work.

2.

PROPOSED METHOD

Figure 1 shows an overview of the proposed method.
First, a distortion value, which represents the spectral
distortion between the noisy/noise-reduced speech and its
original clean version, is calculated. Then recognition
performance is estimated from the distortion value and atask complexity value. In the proposed method, thedistortion measure, the task complexity measure, and theOriginal clean speech

Noisy speech

Noise reduction
Calculation of distortion value
Estimation of recognition performance

Task
complexity

Estimated recognition performanceFig. 1 Overview of the proposed method.

estimator are all issues that need to be addressed and are
therefore described in detail below.
2.1. Distortion Measure
As mentioned above, we previously proposed a
performance estimation method using the PESQ and the
SNR of noise-reduced speech as distortion measures [6].
Although this method can estimate recognition performance without being dependent on the diﬀerences amongnoise reduction algorithms, one problem remains, namely,that the waveform of noise-reduced speech is required forthe calculation of the PESQ score. Therefore, it is noteasily applicable to noise reduction algorithms that directly
output a speech feature for speech recognition.
To cope with this problem, we propose the use of two
CDs instead of the PESQ and the SNR of noise-reduced
speech. One is the CD calculated in the speech frames andthe other is the CD calculated in the non-speech frames.
In this paper, the mel-frequency cepstral coeﬃcient(MFCC) with the 0th-order coeﬃcient is used for calculating the two CDs. The MFCC is widely used as a speechfeature for speech recognition. There are many noise
reduction algorithms that directly output not the waveform
but the MFCC. The CD in the speech frames, CDs , isdeﬁned by
(
)
K
1 X
1 X
jcd ðk; mÞ  cr ðk; mÞj2 ; ð1Þ
CDs ¼
NMs m2Ms K þ 1 k¼0
where m is the frame index, k is the cepstral index, K is
the cepstral analysis order, and cr ðk; mÞ and cd ðk; mÞ are the
MFCC of the reference speech (original clean speech) andthe degraded speech (noisy speech or noise-reduced
speech), respectively. Ms is a set of indexes of speechframes and NMs is the number of elements of Ms . Although
there are several distance measures including theMahalanobis distance, we ﬁrst used the simplest Euclidean
distance. The CD in the non-speech frames, CDn , iscalculated in the same manner. The PESQ is one of theobjective quality measures for coded speech and it287

Acoust. Sci. & Tech. 37, 6 (2016)

evaluates the speech quality on the basis of speech
distortion. CDs also represents the spectral distortion; thus,
we adopted CDs in place of the PESQ. The SNR of noisereduced speech corresponds to the ratio of the power in the
speech frames to the power in the non-speech frames. On
the other hand, CDn represents the spectral distortion of thenoise, but this distortion value is strongly aﬀected by thenoise power since the 0th-order coeﬃcient is used. Because
the SNR of noise-reduced speech changes according to thenoise power when the speech power is constant, we
adopted CDn in place of it. In the proposed method, aconventional power-based speech/non-speech detection
algorithm, which is used in the PESQ [5], is applied tothe original clean speech.
2.2. Task Complexity Measure
As a task complexity measure, the size of vocabulary,
the perplexity, and its derivation, can be taken into account.
Since it was conﬁrmed that the SMR-perplexity isappropriate for estimating recognition performance fordiﬀerent recognition tasks [8], the SMR-perplexity is also
adopted in the proposed method.
The SMR-perplexity is expressed in the following
form:(sﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃ sﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃ
1
1
1
þ
þ
¼
nþ1
Pðw1 jÞPðw1 jw2 Þð2Þ
sﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃ !)2
1
;
 þ
Pðjw1    wn Þ
where Pð j Þ is the word occurrence probability,
w1 ; w2 ;    ; wn are the words, and n is the number ofwords. The symbol ‘‘’’ means the beginning or ending of asentence. This measure overcomes a drawback that theconventional perplexity does not exactly represent the taskcomplexity because the deviations of the sentence lengthand the number of branches are insuﬃciently normalized.
The SMR-perplexity changes according to both thevocabulary size and the language model used. The larger
the SMR-perplexity, the more diﬃcult the recognition task
is.
2.3. Estimator
In the previous method [3], the estimator was expressedin the following form:
y ¼ f ðxÞ ¼a;
1 þ ebxþc

ð3Þ

where y and x represent the estimated word accuracy and
the PESQ score, respectively. The constants a, b, and ccorrespond to the word accuracy for clean speech, the rateof performance degradation, and robustness against spectral distortion, respectively. These constants are determined288

by data ﬁtting using the iterative least-squares method, thatis, by approximating the relationship between the word
accuracy and the PESQ score obtained by using a single
noise reduction algorithm in diﬀerent noisy environments.
The estimator in Eq. (3) was then expanded to deal
with two types of distortion values: the PESQ score and the
SNR of noise-reduced speech [6],y ¼ f ðx1 ; x2 Þ ¼

a
1 þ eb1 x1 b2 x2 þc;ð4Þ

where x1 and x2 indicate the PESQ score and the SNR of
the noise-reduced speech, respectively. These constants are
determined by approximating the relationship between theword accuracy, the PESQ score, and the SNR of the noisereduced speech obtained by using diﬀerent noise reduction
algorithms in diﬀerent noisy environments. The estimatorin Eq. (3) was also modiﬁed as follows to introduce the
SMR-perplexity as a task complexity measure [8]:y ¼ f ðx; Þ ¼

ga ðÞ
;g
1 þ e b ðÞxþgc ðÞð5Þ

where x and  indicate the PESQ score and the SMRperplexity, respectively. These constants are determined byapproximating the relationship between the word accuracy,
the PESQ score, and the SMR-perplexity obtained by using
a single noise reduction algorithm in diﬀerent noisy
environments and for diﬀerent recognition tasks. In
Eq. (5), each constant in Eq. (3) is replaced by a functionof the SMR-perplexity . This was motivated by the factthat the task complexity considerably aﬀects the constantsin Eq. (3) as shown in the experiment below. In this paper,
we newly integrate Eqs. (4) and (5) to handle both diﬀerent
noise reduction algorithms and diﬀerent recognition tasks.
The estimator in the proposed method is expressed by
y ¼ f ðx1 ; x2 ; Þ¼

ga ðÞ;gðÞx
1 þ e b1 1 gb2 ðÞx2 þgc ðÞð6Þ

where x1 and x2 represent the two CDs.
To decide the form of the function of the SMRperplexity  in Eq. (6), we ﬁrst trained the estimatordeﬁned by Eq. (4) for each of diﬀerent recognition tasks
with diﬀerent SMR-perplexity values. Here, note that two
CDs are used, x1 and x2 in Eq. (4). In the training, four
diﬀerent noises, seven values of SNR (including clean
speech), and ﬁve diﬀerent noise reduction algorithms areused. The details of the recognition tasks and the otherconditions are described in Sect. 3. We then investigated
the relationship between each of the constants a, b1 , b2 , andc in Eq. (4) and the SMR-perplexity. Figure 2 shows therelationship between each constant and the SMR-perplexity. In this ﬁgure, each point represents the SMR-perplexity
calculated for one of the recognition tasks and the value of

L. GUO et al.: PERFORMANCE ESTIMATION OF NOISY SPEECH RECOGNITION
100.00
95.00

a

90.00
85.00
80.00
75.00
-0.03

b1

-0.18
-0.33
-0.48
-0.63
-0.78
-0.85

b2

-0.93
-1.01
-1.09
-1.17
-1.25
-10.00-11.00c-12.00-13.00-14.00-15.00
0

10000

20000

30000

40000

50000

SMR-perplexity (α)
Fig. 2 Relationship between each of the constants a, b1 ,b2 , c in Eq. (4) and the SMR-perplexity .

one of the constants in the estimator specialized for thatrecognition task. It can be seen that each constant canbe represented by an exponential function of the SMRperplexity, shown as the solid line in the ﬁgure. The formof the function was then decided as follows.
ga ðÞ ¼ p1  q1 þ r1
gb1 ðÞ ¼ p2  q2 þ r2
gb2 ðÞ ¼ p3  q3 þ r3
gc ðÞ ¼ p4  q4 þ r4

ð7Þ

The constants p , q , and r are determined by approximating the relationships among the word accuracy, the twoCDs, and the SMR-perplexity obtained for diﬀerent noisy
environments, noise reduction algorithms, and recognition
tasks.

3.

EVALUATIONIn this section, we ﬁrst explain the experimental
conditions and then verify the eﬀectiveness of the proposed
distortion measure, i.e., the two CDs. Finally, we evaluate
the proposed method by performing a closed test and an
open test (10-fold cross-validation test).
3.1. Experimental Conditions
We used the following four noise reduction algorithms,

in addition to the reference case where no algorithm wasused.
. noise reduction is not used (NON). minimum mean square error short-time spectral
amplitude estimator (MMSE) [9]. Wiener ﬁltering (WF) [10]. advanced front-end of ETSI ES 202 050 (AFE) [11]. stereo-based piecewise linear compensation for environments (SPLICE) [12]NON, MMSE, and WF output the waveform of noisereduced speech, and AFE and SPLICE output the MFCC
feature of noise-reduced speech. In AFE, Wiener ﬁltering
is applied twice and blind equalization is performed in
the feature domain. These methods are commonly used
as a baseline method when developing a noise reduction
algorithm to improve recognition performance and are thebasis of many derivations.
We prepared the following recognition tasks and cleanspeech data corresponding to each task. The sampling rateof all the speech data described below is 16 kHz. In thispaper, we assume that out of vocabulary (OOV) words do
not exist (or the rate of the OOV words is small).
. Isolated word recognition: We used the Tohoku
University-Matsushita spoken word database [13],consisting of 3,285 isolated words (railway station
names) uttered by 12 male and female speakers. The
dictionary size was set to 50, 100, 200, 400, 800,
1,600, and 3,200. The number of speech data for eachspeaker was the same as the dictionary size and OOVwords did not exist.
. Grammar-based recognition: The speech data usedwere 4,004 connected-digit utterances by 52 male andfemale speakers, which were the same as those in the
AURORA-2J database [14]. The grammar allows
arbitrary repetitions of digits, a short pause, and aterminal silence.
. Large-vocabulary continuous speech recognition: We
used two sets of 100 sentence utterances by 23 malespeakers included in the Japanese Newspaper Article
Sentences (ASJ-JNAS) database [15]. The vocabulary
size was set to 5,000 words and 20,000 words, and therate of OOV words for each set was 0.14% and 0.03%,
respectively. The language models were word 3-grammodels with 5,000 and 20,000 words [16], which weretrained with the Balanced Corpus of Contemporary
Written Japanese [17].
The SMR-perplexity for each recognition task is summarized in Table 1. The recognition experiments and thetraining of each estimator were performed using all the
speech data in each task.
As ambient noise, we used eight forms of noise data,car1, hall1, train2, lift2, factory1, road2, crowd, and lift1,
included in the Denshikyo noise database [18]. The former

289

Acoust. Sci. & Tech. 37, 6 (2016)
100

Table 1 SMR-perplexity  for each task.
Task


80

50 words
100 words
200 words
400 words
800 words
1,600 words
3,200 words

50
100
200
400
800
1,600
3,200

Grammar-based recognition

Connected
digits1120

Large-vocabulary continuous
speech recognition

5,000 words
20,000 words

40,58833,9750True word accuracy (%)

Isolated word recognition

NON

60

MMSE
40

WF

R:
0.98
RMSE: 8.64
0

20

40

60

80

100

Estimated word accuracy (%)(a) PESQ.
100

HI
;
%Acc ¼N

ð8Þ

where H, I, and N indicate the number of correct words,
the number of erroneously inserted words, and the total
number of words, respectively.
3.2. Veriﬁcation of the Proposed Distortion Measure
In this subsection, we verify the eﬀectiveness of the
two CDs proposed. For this purpose, we ﬁrst compare thefollowing distortion measures.
. PESQ: The estimator deﬁned by Eq. (3) is used.
. PESQ and SNR of noise-reduced speech [6]: The
estimator deﬁned by Eq. (4) is used.
. Two CDs: The estimator deﬁned by Eq. (4) is also
used, but the two CDs are used as x1 and x2 in Eq. (4).
In this experiment, only isolated word recognition with 800
words is used as a recognition task. This is so that only thevariation in recognition performance caused by diﬀerent
noise reduction algorithms is observed. The noise reduction
algorithms used for training and testing of each estimator

290

True word accuracy (%)

80

NON

60

MMSE
40

WF

20

R:
0.99
RMSE: 4.94

0
0

20

40

60

80

100

Estimated word accuracy (%)(b) PESQ and SNR of noise-reduced speech.
100

80

True word accuracy (%)

four forms of noise data were used for training and thelatter were used for testing. The noisy speech data weregenerated by artiﬁcially adding the noise data to the
speech data at six diﬀerent values of SNR: 20, 15, 10, 5,
0, 5 dB.
The acoustic models are gender-independent monophone models with 16 Gaussians per state [16], which aretrained with the clean-speech data from the ASJ-JNAS
database. In order to prevent the diﬀerence in the acoustic
models used from aﬀecting the recognition performance,
we used common monophone models. We used the Julius
speech recognizer [19] (rev. 4.3.1) to perform the recognition experiment for all the recognition tasks. The feature
vector in Julius has 25 components consisting of 12
MFCCs, 12 delta MFCCs, and a delta log-power. The
recognition performance is represented by the word
accuracy (%Acc), which is deﬁned by

NON
60
MMSE
40

WF

20

R:
0.99
RMSE: 3.81

0
0

20

40

60

80

100

Estimated word accuracy (%)(c) Two CDs.
Fig. 3 Relationship between the true word accuracy
and the word accuracy estimated using each of the
distortion measures.

are NON, MMSE, and WF, since the PESQ requires the
waveform of the noise-reduced speech.
Figure 3 shows the relationship between the true word
accuracy and the word accuracy estimated using each of
the distortion measures, with the correlation coeﬃcient R
and the root mean square error (RMSE). In these ﬁgures,

L. GUO et al.: PERFORMANCE ESTIMATION OF NOISY SPEECH RECOGNITION
Table 2 RMSE for each of the ten recognition tasks.
SMR-perplexity

11

50

100

200

400

800

1,600

3,200

33,97540,588average

PESQ

8.93

8.85

8.88

8.41

8.59

8.64

9.96

11.27

13.62

13.87

10.10

PESQ & output SNR

7.61

6.69

6.65

5.82

4.67

4.94

5.15

6.25

8.77

9.43

6.59

Two CDs

5.17

5.43

4.16

4.34

3.70

3.81

4.26

5.18

5.64

5.62

4.73

each point represents the result obtained using one of thenoise reduction algorithms for one of the 25 noise
conditions, that is, 4 (noise data)  6 (SNRs) conditions
plus one clean case. From Fig. 3(a), it can be seen that
there is signiﬁcant variation. This is caused by thediﬀerence in the noise reduction algorithms. On the other
hand, the use of the PESQ and the SNR of noise-reduced
speech gives good estimates without being dependent on
the diﬀerence in the algorithms. Figure 3(c) also showsthat the use of the two CDs achieves estimation accuracy
equivalent to the use of the PESQ and the SNR of noisereduced speech. Furthermore, we conducted the same
experiments for each of the remaining recognition tasks.
The results are shown in Table 2. From the table, we canagain conﬁrm the validity of the use of the two CDs.
Next, we verify the eﬀectiveness of the two CDs using
all the noise reduction algorithms. The noise reduction
algorithms used for training and testing are AFE andSPLICE, which directly output the MFCC feature forspeech recognition, in addition to NON, MMSE, and WF.
The recognition task is isolated word recognition with 800
words. The estimator deﬁned by Eq. (4) is used, but the
two CDs are used as x1 and x2 in Eq. (4).
Figure 4 shows the relationship between the true word
accuracy and the word accuracy estimated using the twoCDs. As can be seen from this ﬁgure, the use of the two
CDs again achieves estimation accuracy equivalent to thatin Fig. 3(c), in spite of the increased number of noise
reduction algorithms.
These results show that the use of the two proposed
CDs is eﬀective compared with the use of the PESQ and
the SNR of noise-reduced speech, since they are applicableto the noise reduction algorithms that directly output the
MFCC feature.
3.3. Veriﬁcation of the Proposed Method
In this subsection, we verify the eﬀectiveness of the
proposed method for diﬀerent noise reduction algorithms
and recognition tasks. For this purpose, we ﬁrst compare
the proposed method with the use of the two CDs, using
the method described in Sect. 3.2. The eﬃcacy of theintroduction of the SMR-perplexity is investigated by thiscomparison.

100

80

True word accuracy (%)

Method

NON
60

MMSE
WF

40

AFE
SPLICE20

R:
0.99
RMSE: 4.18

0
0

20

40

60

80

100

Estimated word accuracy (%)

Fig. 4 Relationship between the true word accuracy andthe word accuracy estimated using the two CDs.

. Two CDs: The estimator deﬁned by Eq. (4) is used,
and the two CDs are used as x1 and x2 in Eq. (4).
. Proposed method: The estimator deﬁned by Eqs. (6)and (7) is used.
In this experiment, all the recognition tasks and the noise
reduction algorithms described in Sect. 3.1 are used for the
training and testing of each estimator. The estimator of theuse of the two CDs was decided as follows:
y¼

88:71
1 þ e0:77x1 þ0:71x2 12:12:

ð9Þ

The estimator of the proposed method was also trained asfollows.
ga ðÞ
;y¼
1 þ egb ðÞx1 gb ðÞx2 þgc ðÞ12

ga ðÞ ¼ 7:08  0:13 þ 110:22
gb1 ðÞ ¼ 0:03  0:28  0:83
gb2 ðÞ ¼ 0:48  0:12  1:23
gc ðÞ ¼ 0:62  0:20  16:17

ð10Þ

Figure 5 shows the relationship between the true word
accuracy and the word accuracy estimated by each method.
In these ﬁgures, each point represents the result obtainedusing one of the noise reduction algorithms for one of thenoise conditions and one of the recognition tasks. From

291

Acoust. Sci. & Tech. 37, 6 (2016)
100

True word accuracy (%)

80

SMR-perplexity
11
50
100

60

200
400
800
1600

40

3200
33975
40588

20

80

Ture w
word wccuracy (%)

100

SMR-perplexity
50
60

200
800
3200

40

40588
20

R:
0.96
RMSE: 13.46

0
0

20

40

60

80

Estimated word accuracy (%)

R:
0.99
RMSE: 4.46

0

100

0

20

40

60

80

100

Estimated word accuracy (%)(a) Two CDs.
Fig. 6 Relationship between the true word accuracy andthe word accuracy estimated by the proposed method.

100

True word accuracy (%)

80

SMR-perplexity
11
50
100
200
400
800
1600
3200
33975
40588

60

40

20

R:
0.99
RMSE: 5.18

0
0

20

40

60

80

100

Estimated word accuracy (%)(b) Proposed method.
Fig. 5 Relationship between the true word accuracy andthe word accuracy estimated by each method.

Fig. 5(a), it can be seen that there is signiﬁcant variation.
This means that the estimator in the use of the two CDs
cannot eliminate dependence on the recognition tasks. On
the other hand, we can conﬁrm that the proposed method
gives better estimates without being dependent on thediﬀerences in the recognition tasks. The RMSE of 5.18 forthe proposed method is comparable to the average RMSE
of 4.73 for the use of the two CDs shown in Table 2, which
was obtained under the easier condition that only a single
recognition task is considered.
Finally we evaluate the eﬀectiveness of the proposed
method using an open test (noise reduction algorithm andrecognition task complexity open test). For this purpose,
we perform a 10-fold cross-validation test. A testing set
comprises one of the ﬁve noise reduction algorithms andﬁve of the ten recognition tasks. The training set for thistest set comprises the remaining four noise reduction
algorithms and ﬁve recognition tasks. Note here that therecognition tasks are divided into two groups: one has tasks
where  ¼ 11, 100, 400, 1,600, and 33,975 and the other292

has tasks where  ¼ 50, 200, 800, 3,200, and 40,588. The
number of pairs in the test set and the training set therefore
becomes ten. In this experiment, the estimator deﬁned by
Eqs. (6) and (7) is used.
Figure 6 shows the relationship between the true word
accuracy and the word accuracy estimated by the proposed
method. This ﬁgure is an estimation result for the testing
set with the MMSE and the recognition tasks with  ¼ 50,
200, 800, 3,200, and 40,588. In this ﬁgure, each point
represents the result obtained using the MMSE for one of
the noise conditions and one of the recognition tasks. It canbe seen that the proposed method achieves estimation
accuracy equivalent to that in Fig. 5(b), corresponding toa closed test (noise reduction algorithm and recognition
task complexity closed test). The correlation coeﬃcient R
and the RMSE for each of the ten testing sets are
summarized in Table 3. The left column and the top
row indicate the recognition tasks and the noise reduction
algorithm in each testing set, respectively. From Table 3,
we can see that there is little diﬀerence among the RMSEsof the ten test sets. The correlation coeﬃcient R and the
RMSE averaged over the ten testing sets are 0.98 and 5.71,
respectively. This result is comparable to the RMSE of
5.18 for the closed test.
From these results, it is conﬁrmed that the proposed
method gives better estimates without being dependent on
the diﬀerences among the recognition tasks or the noise
reduction algorithms.

4.

CONCLUSION

We proposed a performance estimation method fornoisy speech recognition, in which the major feature is theability to accommodate the use of diﬀerent noise reduction
algorithms and recognition tasks by using two CDs and the
SMR-perplexity. First, we veriﬁed the eﬀectiveness of theL. GUO et al.: PERFORMANCE ESTIMATION OF NOISY SPEECH RECOGNITION
Table 3 Correlation coeﬃcient R (left) and RMSE (right) for each of the ten testing sets.
Noise reduction algorithm

NON

MMSE

WF

AFE

SPLICE ¼ 11; 100; 400; 1;600; 33;975

0.98/6.20

0.99/5.18

0.98/6.16

0.98/5.51

0.98/6.06

 ¼ 50; 200; 800; 3;200; 40;588

0.99/6.16

0.99/4.46

0.98/7.10

0.99/5.07

0.98/5.18

SMR-perplexity

proposed distortion measure, i.e., the two CDs. The
experimental results showed that the use of the proposed
distortion measure achieves estimation accuracy equivalent
to the use of the conventional distortion measures of the
PESQ and the SNR of noise-reduced speech, and has theadvantage of being applicable to noise reduction algorithms that directly output the MFCC feature. We thenevaluated the proposed method by performing a closed test
and an open test (10-fold cross-validation test). The results
conﬁrm that the proposed method gives better estimates
without being dependent on the diﬀerences among thenoise reduction algorithms or the recognition tasks.
REFERENCES[1] J. Li, L. Deng, Y. Gong and R. Haeb-Umbach, ‘‘An overview
of noise-robust automatic speech recognition,’’ IEEE/ACM
Trans. Audio Speech Lang. Process., 22, 745–777 (2014).
[2] H. Sun, L. Shue and J. Chen, ‘‘Investigations into therelationship between measurable speech quality and speech
recognition rate for telephone speech,’’ Proc. ICASSP 2004,
Vol. 1, pp. 865–868 (2004).
[3] T. Yamada, M. Kumakura and N. Kitawaki, ‘‘Performance
estimation of speech recognition system under noise conditions
using objective quality measures and artiﬁcial voice,’’ IEEE
Trans. Audio Speech Lang. Process., 14, 2006–2013 (2006).
[4] T. Fukumori, M. Nakayama, T. Nishiura and Y. Yamashita,‘‘Estimation of speech recognition performance in noisy andreverberant environments using PESQ score and acoustic
parameters,’’ Proc. APSIPA ASC 2013, Paper ID: 144 (2013).
[5] ITU-T Rec. P.862, ‘‘Perceptual evaluation of speech quality(PESQ): An objective method for end-to-end speech quality
assessment of narrow-band telephone networks and speech
codecs’’ (2001).
[6] L. Guo, T. Yamada, S. Makino and N. Kitawaki, ‘‘Performance
estimation of noisy speech recognition using spectral distortionand SNR of noise-reduced speech,’’ Proc. TENCON 2013,
Paper ID: 540 (2013).
[7] S. Nakagawa and M. Ida, ‘‘A new measure of task complexity
for continuous speech recognition,’’ Trans. IEICE, J81-D-2,
1491–1500 (1998) (in Japanese).
[8] T. Yamada, T. Nakajima, N. Kitawaki and S. Makino,‘‘Performance estimation of noisy speech recognition considering recognition task complexity,’’ Proc. INTERSPEECH
2010, pp. 2042–2045 (2010).
[9] Y. Ephraim and D. Malah, ‘‘Speech enhancement using aminimum mean-square error short-time spectral amplitude
estimator,’’ IEEE Trans. Audio Speech Lang. Process., 32,
1109–1121 (1984).
[10] P. C. Loizou, Speech Enhancement: Theory and Practice(CRC Press, Boca Raton, 2013).
[11] ETSI ES 202 050 v1.1.5, ‘‘Speech processing, transmission andquality aspects (STQ); Distributed speech recognition; Ad-vanced front-end feature extraction algorithm; Compression
algorithms’’ (2007).
[12] L. Deng, A. Acero, M. Plumpe and X. Huang, ‘‘Largevocabulary speech recognition under adverse acoustic environments,’’ Proc. INTERSPEECH 2000, pp. 806–809 (2000).
[13] S. Makino, K. Niyada, Y. Mafune and K. Kido, ‘‘Tohoku
University and Matsushita isolated spoken word database,’’J. Acoust. Soc. Jpn. (J), 48, 899–905 (1992) (in Japanese).
[14] S. Nakamura, K. Takeda, K. Yamamoto, T. Tamada, S.
Kuroiwa, N. Kitaoka, T. Nishiura, A. Sasou, M. Mizumachi, C.
Miyajima and M. Fujimoto, ‘‘AURORA-2J: An evaluation
framework for Japanese noisy speech recognition,’’ IEICE
Trans. Inf. Syst., E88-D, 535–544 (2005).
[15] K. Itou, M. Yamamoto, K. Takeda, T. Takezawa, T. Matsuoka,
T. Kobayashi, K. Shikano and S. Itahashi, ‘‘JNAS: Japanese
speech corpus for large vocabulary continuous speech recognition research,’’ J. Acoust. Soc. Jpn. (E), 20, 199–206 (1999).
[16] T. Kawahara, A. Lee, T. Kobayashi, K. Takeda, N. Minematsu,
S. Sagayama, K. Itou, A. Ito, M. Yamamoto, A. Yamada, T.
Utsuro and K. Shikano, ‘‘Free software tool kit for Japaneselarge vocabulary continuous speech recognition,’’ Proc. ICSLP
2000, pp. 476–479 (2000).
[17] K. Maekawa, M. Yamazaki, T. Ogiso, T. Maruyama, H. Ogura,W. Kashino, H. Koiso, M. Yamaguchi, M. Tanaka and Y. Den,‘‘Balanced corpus of contemporary written Japanese,’’ Lang.
Resour. Eval., 48, 345–371 (2004).
[18] Denshikyo noise database, http://research.nii.ac.jp/src/list/
detail.html#JEIDA-NOISE (accessed 2015-12-01).
[19] Julius, http://julius.osdn.jp/ (accessed 2015-12-01).

Ling Guo received her B.S. degree from
Jiaxing University, China, in 2010, and her M.
Eng. degree from University of Tsukuba, Japan,
in 2014. She is currently working toward her
Ph.D. Her research interests includes the performance estimation of noisy speech recognition. She is a student member of the ASJ.

Takeshi Yamada received his B. Eng. degreefrom Osaka City University, Japan, in 1994, andhis M. Eng. and Dr. Eng. degrees from Nara
Institute of Science and Technology, Japan, in
1996 and 1999, respectively. He is presently an
associate professor with Faculty of Engineering, Information and Systems, University ofTsukuba, Japan. His research interests include
speech recognition, sound scene understanding,multichannel signal processing, media quality assessment, and elearning. He is a member of the IEEE, the IEICE, the IPSJ, the ASJ,
and the JLTA.

293

Acoust. Sci. & Tech. 37, 6 (2016)
Shigeki Miyabe received his B.E. degreefrom Kobe University, Japan, in 2003, and his
M.E. and Ph.D. degrees from Nara Institute of
Science and Technology, Japan, in 2005 and2007, respectively. From 2008 to 2009, he was avisiting scholar with Georgia Institute of Technology, USA. In 2009, he joined the Graduate
School of Information Science and Technology,
University of Tokyo, Japan, as a researcher, andbecame an assistant professor in 2010. He is currently an assistantprofessor of Life Science Center of Tsukuba Advanced Research
Alliance, University of Tsukuba, Japan. He is a member of the IEEEand the ASJ.
Shoji Makino received his B.E., M.E., andPh.D. degrees from Tohoku University, Japan,
in 1979, 1981, and 1993, respectively. He joined
NTT in 1981. He is now a professor at
University of Tsukuba, Japan. His researchinterests include adaptive ﬁltering technologies,
the realization of acoustic echo cancellation,
blind source separation of convolutive mixturesof speech, and acoustic signal processing forspeech and audio applications. He is an IEEE Fellow, an IEICE
Fellow, a council member of the ASJ, and a member of the
EURASIP.

294

Nobuhiko Kitawaki received his B. Eng., M.
Eng. and Dr. Eng. degrees from the Tohoku
University, Japan, in 1969, 1971, and 1981,
respectively. From 1971 to 1997 he wasengaged in research on speech and acoustics
information processing at NTT Laboratories.
From 1997 to 2015, he served as a professor of
the Graduate School of Systems and Information Engineering, University of Tsukuba. Emeritus Prof. Kitawaki is an IEEE Fellow, an IEICE Fellow, and amember of the Acoustical Society of Japan.