Performance Estimation of Noisy Speech
Recognition Using Spectral Distortion and SNR of
Noise-reduced Speech
Guo Ling, Takeshi Yamada, Shoji Makino and Nobuhiko Kitawaki
Graduate School of Systems and Information Engineering, University of Tsukuba
1-1-1 Tennodai, Tsukuba, Ibaraki, 305-8573 Japane-mail: guoling@mmlab.cs.tsukuba.ac.jp
Abstract—To ensure a satisfactory QoE (Quality ofExperience) and facilitate system design in speech recognition
services, it is essential to establish a method that can be used toefficiently investigate recognition performance in different noise
environments. Previously, we proposed a performance
estimation method using the PESQ (Perceptual Evaluation of
Speech Quality) as a spectral distortion measure. However, thereis the problem that the relationship between the recognition
performance and the distortion value differs depending on thenoise reduction algorithm used. To solve this problem, we
propose a novel performance estimation method that uses anestimator defined as a function of the distortion value and the
SNR (Signal to Noise Ratio) of noise-reduced speech. The
estimator is applicable to different noise reduction algorithms
without any modification. We confirmed the effectiveness of the
proposed method by experiments using the AURORA-2J
connected digit recognition task and four different noise
reduction algorithms.
Keywords—performance estimation; noisy speech recognition;
noise reduction; spectral distortion; SNRI.

INTRODUCTION

In recent years, speech recognition technology has been
considerably improved by applying a statistical framework.
However, current speech recognition systems still have theserious problem that their recognition performance is degradedin the presence of ambient noise. The degree of theperformance degradation depends on the nature of ambient
noise and the type of noise reduction used as a preprocessing
stage in speech recognition. To ensure a satisfactory QoE(Quality of Experience) and facilitate system design in speech
recognition services, it is essential to establish a method thatcan be used to efficiently investigate recognition performancein different noise environments.
One typical approach is to collect noisy speech data in atarget noise environment and then perform a recognition
experiment. However, this requires a skilled engineer and isFig. 1 Estimation of the recognition performance from the distortion valuelabor and time-consuming. An alternative approach is toestimate recognition performance from a distortion value,
which represents a spectral distortion between noisy (ornoise-reduced) speech and its original clean version as shownin Fig. 1 [1-3].
Previously, we proposed a performance estimation method
using the PESQ (Perceptual Evaluation of Speech Quality) [4]as a distortion measure [3]. In this method, an estimator, which
is a function of the distortion value, is obtained byapproximating the relationship between the recognition
performance and the distortion value. Although this method
gives accurate estimates of the recognition performance, thereis the problem that the relationship between the recognition
performance and the distortion value differs depending on thenoise reduction algorithm used. This means that eachindividual noise reduction algorithm requires the special
estimator.
To solve this problem, we propose a novel performance
estimation method using the PESQ and the output SNR (Signal
to Noise Ratio) that is the SNR of noise-reduced speech. In the
proposed method, an estimator, which is a function of the
distortion value and the output SNR, is introduced. The
estimator is applicable to different noise reduction algorithms
without any modification. We evaluate the effectiveness of theproposed method by experiments using the AURORA-2J
connected digit recognition task and four different noise
reduction algorithms.
II. PROPOSED METHOD
The conventional method estimates the recognition
performance by using the estimator expressed in the following
form [3].
1,1where y and x represent the estimated word accuracy and thePESQ score, respectively. The PESQ score has a range of 0.5
to 4.5. Note that the PESQ score of 4.5 means no spectral
distortion. The constants a, b, and c correspond to the word
accuracy for clean speech, the slope of the performancedegradation, and the robustness against the spectral distortion,
respectively. These constants are determined by approximating
the relationship between the word accuracy and the PESQ
score obtained in various noise environments.
Fig. 2 shows the relationship between the word accuracy
and the PESQ score. In this figure, each point represents the
PESQ score and the word accuracy obtained by one of fivenoise reduction algorithms in one of 28 noise conditions. The
details of the experimental conditions are given in Section III.
From Fig. 2, we can confirm that the relationship between theword accuracy and the PESQ score differs depending on thenoise reduction algorithm used. This means that eachindividual noise reduction algorithm requires the special
estimator.
To solve this problem, we focus on the output SNR that isequivalent to the SNR of noise-reduced speech. In this paper,the output SNR is approximately calculated by
Output SNR

10log

|

|

|

|

,

2

where x(i) and n(i) are the speech segments and noise
segments, respectively. Fig. 3 represents the relationshipbetween the word accuracy, the PESQ score, and the output
SNR. This figure is the same as Fig. 2 except that the type ofmarker represents not the algorithm but the range of the output
SNR. From Fig. 3, we can see that the word accuracy tends tobe higher as the output SNR becomes large, when comparing
with two points with the same PESQ score. This implies thatthe word accuracy can be estimated by using both the PESQ
score and the output SNR.
Based on this finding, we propose the estimator expressed in
the following form.
1,

3

where y, x1, and x2 indicate the estimated word accuracy, the
PESQ score, and the output SNR, respectively. The constants a,
b1, b2, and c are determined by approximating the relationshipbetween the word accuracy, the PESQ score, and theFig. 2 Relationship between the word accuracy and the PESQ score

Fig. 3 Relationship between the word accuracy, the PESQ score, and theoutput SNR

output SNR obtained using different noise reduction
algorithms in various noise environments.
III. VERIFICATION OF PROPOSED METHOD
A. Experimental conditions
We use four noise reduction algorithms, in addition to thereference case of no such algorithm.
(B) Baseline (noise reduction is NOT implemented)(S) Spectral subtraction with smoothing of time direction [5](T) Temporal domain SVD-based speech enhancement [6](G) GMM-based speech estimation [6](K) KLT-based comb-filtering [7]In these algorithms, the speech component in the noisy speechframes is extracted by using the noise component estimatedfrom the non-speech frames. The algorithm (S) is one of themost popular algorithms. The algorithm (G), and algorithms(T) and (K) are based on the Wiener filter and the subspacemethod, respectively. The characteristics of a speech distortionand residual noise in noise-reduced speech vary depending on
the noise-reduction algorithm used.
Recognition experiments were performed on the
AURORA-2J connected digit recognition task [8]. The training
and test sets for the AURORA-2J task are shown in Table I.

TABLE I. EXPERIMENTAL CONDITIONSIn this paper, the clean training set is used for training HMMs
(Hidden Markov Models) and the test sets A and B for
evaluation. The test sets A and B are based on the same 4,004
clean speech signals. For each set these signals are divided into
four groups of 1,001 speech signals, corresponding to four
different types of noise. Then, for each group, the appropriate
noise signal is artificially added to the speech signals at
different seven values of SNR. As a result, each test set has 28
noise conditions.
The speech signal sampled at 8 kHz is windowed by a 25ms
Hamming window every 10ms. The feature vector has 39
components consisting of 12 MFCCs (Mel-Frequency Cepstral
Coefficients) together with log power, and their first andsecond derivatives. The digit models and the silence model are
the HMMs with 16 states and 3 states, respectively. The
number of Gaussians per state is 20 for the digit models and 36
for the silence model. The HMM of the short pause is the 1state tied with the 2nd state of the silence model. The set of
HMMs is trained for each individual noise reduction algorithm,
where all the training data are processed by the algorithm
before training.
The conditions for determination of the estimator's constants
are described below.
(C1) The estimators of the conventional method and the
proposed method are optimized for the test set A with all
the noise reduction algorithms. They are evaluated using
the test set A (Noise-closed test).
(C2) The estimators are the same as those in the condition (C1),but they are evaluated using the test set B (Noise-opentest).
The determined estimators of the conventional method and the
proposed method were

102.7779
1

.

1

.

4

.

and

100.8289
.

.

,

5

respectively.
B. Results
Fig. 4 shows the relationship between the word accuracy
and the overall distortion calculated by b1x1+b2x2 in the
proposed method. We can see the consistent relationshipFig. 4 Relationship between the word accuracy and the overall distortion
calculated by b1x1+b2x2 in the proposed method

between the word accuracy and the overall distortion for all thenoise reduction algorithms used.
We then estimated the recognition performance by using theestimators expressed by Eqs. (4) and (5). Figs. 5 and 6 show
the relationship between the true word accuracy and theestimated word accuracy in the conditions (C1) and (C2),respectively. It can be seen that the proposed method gives
more accurate estimates than the conventional method. The
coefficient of determination, R2, and the RMSE (Root Mean
Square Error), which are given by
1

∑

%
∑

%
%

%

6

and
∑

%

%

7

are shown in Table II. It can be confirmed that the RMSE of
the proposed method decreased almost by half to 7.0 from 13.2
in the condition (C1) and to 7.37 from 14.0 in the condition(C2). The difference in noise type has little effect on theestimation performance. These results confirmed that the
proposed method give accurate estimates of the recognition
performance without preparing the special estimator for eachindividual noise reduction algorithm.
IV. CONCLUSIONSPreviously, we proposed a performance estimation method
using a spectral distortion measure. However, there is the
problem that the relationship between the recognition
performance and the distortion value differs depending on thenoise reduction algorithm used. To solve this problem, we
proposed a novel performance estimation method that uses anestimator defined as a function of the distortion value and the
SNR (Signal to Noise Ratio) of noise-reduced speech. The
experimental results confirmed that the proposed method giveaccurate estimates of the recognition performance without
preparing the special estimator for each individual noise
reduction algorithm.

100

80

80
Estimated Word Accuracy(%)

Estimated Word Accuracy(%)

100

(B)
(G)

60

(K)
(S)

40

(T)
20

(B)
60

(G)
(K)
(S)

40

(T)

20

0

0
0

20

40
60
Word Accuracy(%)

80

100

0

20

80

100

(a) Proposed method(a) Proposed method

100

100

80

80
Estimated Word Accuracy(%)

Estimated Word Accuracy(%)

40
60
Word Accuracy(%)

(B)
60

(G)
(K)
(S)

40

(T)
20

(B)
60

(G)
(K)
(S)

40

(T)

20

0

0
0

20

40
60
Word Accuracy(%)

80

100

0

20

40
60
Word Accuracy(%)

80

100

(b) Conventional method(b) Conventional method
Fig. 5 Relationship between the true word accuracy and the estimated word

Fig. 6 Relationship between the true word accuracy and the estimated word

accuracy in the condition (C1)

accuracy in the condition (C2)REFERENCES
[1]

[2]

[3]

[4]

[5]

H. Sun, L. Shue, J. Chen, “Investigations into the relationship between
measurable speech quality and speech recognition rate for telephonyspeech,” Proc. IEEE International Conference on Acoustics, Speech,
and Signal Processing, ICASSP2004, Vol. 1, pp. 865-868, May 2004.
M. Kondo, K. Takeda, F. Itakura, “Predicting the degradation of
speech recognition performance from sub-band dynamic ranges,” IPSJ
Journal, Vol. 43, No. 7, pp. 2242-2248, July 2002.
T. Yamada, M. Kumakura, N. Kitawaki, “Performance estimation of
speech recognition system under noise conditions using objective
quality measures and artificial voice,” IEEE Transactions on Audio,
Speech and Language Processing, Vol. 14, No. 6, pp. 2006–2013, Nov.
2006.
ITU-T Rec. P.862, “Perceptual evaluation of speech quality (PESQ):
An objective method for end-to-end speech quality assessment of
narrowband telephone networks and speech codecs,” Feb. 2001.
N. Kitaoka, S. Nakagawa, “Evaluation of spectral subtraction withsmoothing of time direction on the AURORA 2 task,” Proc.
International Conference on Spoken Language Processing, ICSLP2002,
pp. 465–468, 2002.

TABLE II. R2 AND RMSE

[6]

[7]

[8]

M. Fujimoto, Y. Ariki, “Combination of temporal domain SVD based
speech enhancement and GMM based speech estimation for ASR innoise – evaluation on the AURORA2 task –,” Proc. European
Conference on Speech Communication and Technology,
EUROSPEECH2003, pp. 1781–1784, 2003.
S.-J. Park, M. Ikeda, K. Takeda, F. Itakura, “Improvement of the ASR
robustness using combinations of spectral subtraction and KLT based
adaptive comb-filtering,” IPSJ SIGNotes, SLP-44-3, pp. 13–18, 2002.
S. Nakamura, K. Takeda, K. Yamamoto, T. Yamada, S. Kuroiwa, N.
Kitaoka, T. Nishiura, A. Sasou, M. Mizumachi, C. Miyajima, M.
Fujimoto, T. Endo, “AURORA-2J: An evaluation framework forJapanese noisy speech recognition,” IEICE Transactions on
Information and Systems, Vol. E88-D, No. 3, pp. 535–544, Mar. 2005.