JAIST Repository
https://dspace.jaist.ac.jp/

Title

Speech Enhancement based on Noise Eigenspace
Projection

Author(s)Ying, Dongwen; Unoki, Masashi; Lu, Xugang; Dang,
JianwuCitation

IEICE Transactions on Information and Systems,
E92‑D(5): 1137‑1145

Issue Date

2009‑05‑01

Type

Journal Article

Text version

publisher

URL

http://hdl.handle.net/10119/9181

RightsCopyright (C)2009 IEICE. Dongwen Ying, Masashi
Unoki, Xugang Lu, and Jianwu Dang, IEICE
Transactions on Information and Systems, E92‑
D(5), 2009, 1137‑1145.
http://www.ieice.org/jpn/trans̲online/

Description

Japan Advanced Institute of Science and Technology

IEICE TRANS. INF. & SYST., VOL.E92–D, NO.5 MAY 2009

1137

PAPER

Speech Enhancement Based on Noise Eigenspace Projection
Dongwen YING†a) , Nonmember, Masashi UNOKI†b) , Member, Xugang LU†c) , Nonmember,
and Jianwu DANG†d) , Member

SUMMARY
How to reduce noise with less speech distortion is a challenging issue for speech enhancement. We propose a novel approach for
reducing noise with the cost of less speech distortion. A noise signal cangenerally be considered to consist of two components, a “white-like” component with a uniform energy distribution and a “color” component witha concentrated energy distribution in some frequency bands. An approachbased on noise eigenspace projections is proposed to pack the color component into a subspace, named “noise subspace”. This subspace is thenremoved from the eigenspace to reduce the color component. For thewhite-like component, a conventional enhancement algorithm is adopted as
a complementary processor. We tested our algorithm on a speech enhancement task using speech data from the Texas Instruments and Massachusetts
Institute of Technology (TIMIT) dataset and noise data from NOISEX-92.
The experimental results show that the proposed algorithm eﬃciently reduces noise with little speech distortion. Objective and subjective evaluations confirmed that the proposed algorithm outperformed conventional
enhancement algorithms.
key words: speech enhancement, noise eigenspace, dimension reduction(DR), Karhunen-Lóeve transform (KLT)

1.

Introduction

Speech enhancement techniques attempt to improve one ormore perceptual aspects of speech communication systems,
namely, overall quality and intelligibility of speech soundfor human or machine recognizers [1] when a signal is corrupted by noise. The improvement is in a sense of minimizing system degradation due to noise.
Numerous approaches have been proposed for this purpose. Most algorithms were presented in frequency domains
such as spectral subtraction [2], [3], Wiener filtering [3], andminimum mean-square error (MMSE) [4] algorithms. They
have been used widely because of their simplicity and highcomputational eﬃciency. Recently, signal subspace approaches [5]–[9] have been proposed for enhancing speech
signals. The core idea of these signal subspace approaches
is to decompose a noisy speech into uncorrelated components in a signal space. In such a space, each vector contains a clean signal component and a noise component. The
desired signal component is estimated with a linear estimator and synthesized by applying the inverse Karhunen-Lóeve
Manuscript received August 28, 2008.
Manuscript revised December 3, 2008.
†
The authors are with the Information School, Japan Advanced
Institute of Science and Technology, Nomi-shi, 923–1292 Japan.
a) E-mail: yingdongwen@hccl.ioa.ac.cnb) E-mail: unoki@jaist.ac.jpc) E-mail: xugang@jaist.ac.jpd) E-mail: jdang@jaist.ac.jp
DOI: 10.1587/transinf.E92.D.1137

transform (KLT) based on the vectors.
In all these algorithms, the entire noise signal is reduced from noisy speech. The characteristics of noise energy distribution are not taken into account in the noise reduction process. Usually, according to the energy distribution, there are two typical noise signals, a color noise signal
with a concentrated energy distribution in certain frequency
bands and a white noise signal with a uniform energy distribution. In fact, most environmental noise signals exist
between white noise and color noise. In other words, theenvironmental noise signal generally consists of a “whitelike” component and a “color” component. If an appropriate strategy is adopted to deal with each component, better
noise reduction is expected.
Based on such considerations, we propose a novel approach for enhancing speech, where diﬀerent strategies are
adopted for diﬀerent components. A noise eigenspace witha high-energy packing eﬃciency [10] is proposed to reducethe color component, while a conventional algorithm is used
as a complementary processor for processing the white-likecomponent. The aim of this approach is to reduce the noisewith as little speech distortion as possible. To track the noise
variation, the noise eigenspace should be updated. Sincenoise signals are assumed to be more stationary than speech,
the noise eigenspace does not need to be updated as often as
the noisy speech eigenspace [6]. Thus, the computation load
can be controlled at an acceptable level.
The rest of the paper is organized as follows. In Sect. 2,
we introduce the construction of the noise eigenspace andthe noise reduction approach in that space. In Sect. 3, wegive details on the implementation of the proposed model
for speech enhancement. In Sect. 4, we conduct some experiments for evaluating the algorithm. We focus on discussions in Sect. 5 and summarize and state the conclusions of
this study in Sect. 6.
2.

Noise Eigenspace and Noise Reduction

Generally speaking, noise can be eﬃciently reduced if we
can clarify the noise energy distribution. To do so, we mustfind a space for noisy speech representation with two basic
characteristics: (1) the distribution of noise energy is easily
represented in the space and (2) noise energy is packed into a
local area. The noise signal in the local area can be removed
from the space to reduce noise.
The noise eigenspace is a desirable space meeting thesec 2009 The Institute of Electronics, Information and Communication Engineers
Copyright IEICE TRANS. INF. & SYST., VOL.E92–D, NO.5 MAY 20091138

demands since it has a high-energy packing eﬃciency tothe noise signal, and the noise energy distribution can be
described by the eigenvalues. According to the energy
distribution, this eigenspace can be separated into a subspace dominated by noise (referred to as “noise subspace”
hereafter) and the complementary subspace dominated byspeech (referred to as “speech subspace”). The noise subspace is used for noise reduction, and the speech subspacefor recovering speech. The details of the process are described in the following subsections.
2.1 Construction of Noise Eigenspace
The noise eigenspace is constructed using the eigenvalue decomposition of a noise covariance matrix according to
Cns ϕk = λk ϕk ,k = 1, 2, · · · , K,(1)

the dimension signal-to-noise ratio (SNR) is defined as
k = 1, · · · , K.
(5)
Qk = 10log10 (sk dk ),Note that, in the following sections, the SNR in the noise
eigenspace is referred to as dimension SNR if there is no
special specification.
We know that the noise eigenspace depends on thenoise signal, but it is independent of the speech signal. Thatis, the noise signal will be packed into a local area, namely
certain large-eigenvalue dimensions; while the speech signal
is not packed by this projection. Since the speech energy
distribution is usually diﬀerent from that of the noise signal, the large-eigenvalue dimensions are dominated by thenoise signal. Therefore, a subspace containing the packed
noise signal and little speech signal can be extracted fromthe eigenspace.

where ϕk is the eigenvector corresponding to the eigenvalue
λk , K is the number of eigenspace dimensions, and Cns isa K-by-K matrix of noise covariance. In this study, Cns isapproximated using the noise correlation matrix (Toeplitz
matrix), which is derived from the autocorrelation sequenceof the noise signal. We detect speech pauses for calculating
the noise autocorrelation sequence by using a voice activity
detector (VAD, mentioned in Sect. 3).
The projection into the noise eigenspace can be calculated with the KLT. Supposing the noisy speech is a zeromean signal, projecting a noisy speech in a frame, y, (column vector) into the k-th dimension of the noise eigenspacecan be represented as inner product:k = 1, 2, · · · , K(2)y, ϕk = ϕk T y,

2.2

where T denotes the operation of an array transpose. If thenoise is additive, a noisy speech frame vector can be described as y = x + n, where x and n denote the speech andnoise in the K-dimension frame vectors, respectively. For anoisy speech utterance, we can decompose it into a frame
sequence, {y(1) , y(2) , · · · , y(M) }, where the superscripts represent the time index. When projecting this sequence into
the noise eigenspace, the noise and speech energies in eachdimension can be represented as

How to estimate the speech energy distribution in a noise
eigenspace is a significant issue for estimating dimension
SNR. For the sake of simplicity, we use the long-term average speech distribution to approximate the speech energy
distribution. The following procedure is used to estimate thelong-term average speech energy distribution in an arbitrary
noise eigenspace.
First, a long-term average covariance matrix of a clean
speech signal, C sp , is calculated from a large-scale clean
speech dataset (Texas Instruments and Massachusetts Institute of Technology (TIMIT) test corpus including 1680 sentences). From the eigenvalue decomposition of C sp , we obtain an average speech eigenspace:dk =1   (m) 2
n , ϕk
M m=1

(3)

sk =

1   (m) 2
x , ϕk
M m=1

(4)

M

M

Since we use the noise correlation matrix instead of thecovariance matrix here, there is a relationship between the
projected noise energies and eigenvalues as dk ≈ λk , where
noise eigenvalues can approximately represent the noise energy distribution. For noise mainly consisting of a color
component (e.g. car noise), several large-eigenvalue dimensions usually cover most energy.
According to the definition of speech and noise energy,

Noise Reduction in Noise Eigenspace

The criterion of extracting the noise subspace is to make thenoise subspace contain as much noise, and as little speech
as possible. Based on this consideration, the noise subspaceshould consist of low-SNR dimensions. Thus, it is necessary to estimate the dimension SNR to determine the noise
subspace.
For such purpose, the speech and noise energy distribution should be investigated in the noise eigenspace. We
can easily obtain the normalized noise energy, d̃k , from thenoise eigenvalues.
Kd̃k = λk
λ j , k = 1, 2, · · · , K.
(6)
j=1

C sp ψk = γk ψk ,k = 1, 2, · · · , K,

(7)

where the eigenvectors, {ψk |k = 1, · · · , K}, represent
this eigenspace, and the eigenvalues, {γk |k = 1, · · · , K},
represent the average speech energy distribution in thiseigenspace. The average eigenspace is a constant space, andit can be taken as prior knowledge of speech signal.
Second, the speech eigenvalues are normalized asK
γ j , k = 1, 2, · · · , K.
(8)γ̃k = γk
j=1

YING et al.: SPEECH ENHANCEMENT BASED ON NOISE EIGENSPACE PROJECTION

1139

Then, the normalized speech distribution in the speecheigenspace is projected into the noise eigenspace using thefollowing equation (see details in Appendix).
s̃k =

K 

2
ψ j T ϕk γ̃ j ,k = 1, 2, · · · , K,

(9)

j=1

where s̃k represents the long-term average speech energy distribution in the current noise eigenspace, {ϕk |k =1, · · · , K}.
After obtaining the speech and noise distributions, theestimated dimension SNR, Q̂k , is derived as follows:Q̂k = 10log10 ( s̃k /d̃k ) + r,

k = 1, 2, · · · , K,

(10)

where the average SNR, r, denotes the intensity relationshipbetween speech and noise signals. The common definitionof average SNR is the total energy ratio of the speech tonoise.
The dimensions of the noise eigenspace are sorted in
an ascending order based on the estimated dimension SNR,Q̂1 ≤ Q̂2 ≤ · · · ≤ Q̂K . A threshold, δNR , for dimension
SNR is introduced to determine how many of the low-SNR
dimensions belong to the noise subspace. An appropriate
threshold should guarantee that the subspace contains asmuch noise while including as little speech as possible. At
the end, the threshold is determined by a tradeoﬀ between
the noise and speech energy in the noise subspace. Froma preliminary experiment of optimizing the segmental SNR
on a dataset, we found out that setting the threshold at −6 dB
achieves an optimal tradeoﬀ.
If only the dimension SNR is considered in determining the noise subspace, another problem occurs. Under low
average SNR conditions, there are a great number of dimensions with (dimension) SNR less than the threshold, δNR . If
all those dimensions were classified into the noise subspace,
the percentage of speech energy inside the subspace (ratioof speech energy in the noise subspace to the total speechenergy) would increase so that a larger speech distortion
would be introduced in the noise reduction. Therefore, besides the dimension SNR, the speech energy percentage in
the noise subspace should also be considered for determining the noise subspace. A threshold, δSD , for the speech energy percentage is used to prevent large speech loss. The
speech energy percentage in the noise subspace should beless than δSD . It is set as 6% according to our preliminary
experiments.
Let qNR denotes the amount of dimensions with SNRless than δNR , and qSD denotes the amount of first dimensions
with an accumulated speech energy percentage less than δSD
SD(namely qj=1
s̃ j / Kj=1 s̃ j < δSD ). To make the noise subspace satisfy the two conditions simultaneously, the numberof the noise subspace dimensions is defined as:
q = min(qSD , qNR ).

(11)

Thus, the noise subspace can be represented by a group of
eigenvectors, [ϕ1 , ϕ2 , · · · , ϕq ], and the remaining subspace,consisting of the last K − q dimensions, [ϕq+1 , · · · , ϕK ] istreated as a speech subspace. The speech and noise subspace
are orthogonal and implemented to each other. It is worthwhile clarifying that the subspaces defined in this paper are
diﬀerent from those defined in conventional subspace approaches [6]–[8]. The subspaces of conventional approaches
contain theoretically either speech or noise signals, while
the subspaces in this study include both speech and noise
signals. As a result, our proposed approach of noise reduction diﬀers from conventional subspace approaches.
Our approach of noise reduction is based on the two
subspaces. Usually, the color component of the noise signal
is contained in the noise subspace. If we directly remove thenoise subspace from the noise eigenspace, the color component would be reduced with only a little loss of speech. The
lost speech is responsible for “speech distortion”, which isthe cost of noise reduction.
After removing the noise subspace by using the DR algorithm, the projections in the speech subspace are transformed into frames in the time domain by using the following equation:ySS = ϕ q+1 , · · · , ϕ K
× y, ϕ q+1 , · · · , y, ϕ K(12)In the above equation, the first item of right side is a
K × (K − q) matrix, and the second item is a (K − q) × 1 vector. Equation (12) can be considered as a noise-dependent
filter for noise reduction. The noisy speech frame vector,y, can be regarded as the input signal. Hereafter, the noise
reduction algorithm mentioned in this section is referred toas “dimension reduction (DR)” algorithm, which includes
Eqs. (6)–(12).
3.

Proposed Method for Speech Enhancement

The proposed DR algorithm eﬃciently reduces the color
component of a noise but contributes less to reducing thewhite-like component. Therefore, a complementary processor is needed to process the remaining white-like component
after using the DR algorithm. As the MMSE algorithm [4]is reported to be a simple and eﬃcient approach for noise reduction [11], we adopted it as the complementary processor
in our study.
Figure 1 shows a block diagram of the proposed enhancement algorithm, which consists of a subspace determination block, a filter block for reducing the color component, and an MMSE block for reducing the white-like component. A VAD block for tracking noise variation is omitted
in this figure.
The noise correlation matrix is calculated from thenoise signal in the speech pauses. By eigenvalue decomposition of the matrix, the noise eigenspace is determined.
Using Eqs. (6)–(11), the dimension SNR is derived, and then
the noise eigenspace is sorted using the dimension SNR.
According to the thresholds of δSD and δNR , q is obtainedfor determining a speech subspace. Finally, a filter for noiseIEICE TRANS. INF. & SYST., VOL.E92–D, NO.5 MAY 2009

1140

Fig. 1 Diagram of proposed method, which consists of DR block (labeled with a dashed rectangle), and MMSE block.

reduction is constructed.
The input noisy speech signal is segmented into a frame
sequence, and is processed with Eq. (12) to suppress thecolor component of the noise signal. The output signal fromthe filter, ySS , is transformed into frequency domain and processed using the conventional MMSE algorithm to suppress
the white-like component. Finally, the enhanced frames
from the MMSE are transformed into the time domain toreconstruct the speech signal by using the overlap-and-add
approach with the Hamming window [12].
As noise usually varies with time, we must track noise
variation to update the noise eigenspace for eﬃciently reducing the noise. For this reason, a robust VAD is very important for improving system performance. Since the noise
is packed into the noise subspace, the speech subspace isusually associated with a higher SNR. Therefore, we designed a robust VAD using the projections in the speech
subspace. The principle of this VAD is closely related to the
DR algorithm. Since the space of this paper is limited, thedetails of the VAD algorithm were introduced in a previous
work [13].
When updating the noise eigenspace, the computational load has to be considered because the eigenvalue decomposition is a time-consuming operation. Updating thenoise eigenspace more frequently is better for accurately
tracking the variation of the noise signal, but it would makethe computation load heavier. For a compromise, the updating period in this study was set at 1.5 seconds. As a result,the computational load could be controlled at an acceptablelevel.
4.

Evaluation

Some experiments were conducted for a thorough evaluation of the proposed algorithm. The experiment data wasselected from the TIMIT and NOISEX-92 databases. The
speech data consisted of twenty utterances from the TIMIT
database. Half of the utterances were from male speakers
and the other half from female speakers. The noise signals
selected from the NOISEX-92 database include leopard (thename of a car), tank, and babble noises. The babble noise
was used because its energy distribution is similar to speech.
The leopard and tank noises are two typical color noises, theFig. 2 Noise reduction rate and speech distortion rate under various
noisy conditions. Solid lines illustrate reduced noise rate, and dashed lines
for speech distortion rate.

energy distribution of which is quite diﬀerent from speech.
All noises were artificially added to the tested clean speechat SNRs ranging from −10 to 15 dB with an interval of 5 dB.
The sampling rate was 16 kHz and a 20-ms analysis window with a 10-ms shift was used in the processing. As aresult, each frame had 320 sampling-points and the noise
eigenspace had the same number of dimensions as the frame
length. To give a convincible evaluation, the proposed algorithm is evaluated on diﬀerent aspects.
4.1

Verification of the DR Algorithm

To evaluate the noise reduction with the DR algorithm, we
defined the noise reduction rate, RNR (percentage rate ofnoise reduced with the DR algorithm) and the speech distortion rate, RSD (percentage rate of lost speech caused by
the DR algorithm) as follows:

q
K

RNR =
dk
dk × 100 (%)
(13)
k=1

RSD =

q

k=1

k=1

sk


K

sk × 100

(%)

(14)

k=1

where q is the number of the noise subspace dimensions.
We evaluated the DR algorithm by the RNR and RSD averaged
over all test samples.
Figure 2 describes the relationship between RNR and RSD
under various noise conditions. It shows, under most color
noise conditions, that the DR algorithm eﬃciently reduces
most noise with little speech distortion. Generally speaking, the DR algorithm reduces more noise when the color
noise has an energy distribution diﬀerent from the speech
signal. For noise with similar energy distribution to speech,such as babble noise, it is found that the DR algorithm
hardly contributes to noise reduction. From Fig. 2, it wasalso found that the DR algorithm contributes more underlow-SNR conditions than under high average SNR conditions. If the average SNR is high enough, no dimension

YING et al.: SPEECH ENHANCEMENT BASED ON NOISE EIGENSPACE PROJECTION

1141

of the noise eigenspace will be reduced. Therefore, boththe noise reduction and speech distortion are zero. For theleopard noise, RNR and RSD were zero when the average SNR
exceeded 10 dB; and 0 dB for the babble noise.
4.2 Objective Evaluation of the Enhanced Speech
From the above experiment, one can see that the DR algorithm contributes more under low-SNR conditions. Therefore, in the following evaluations, we focused on the performance under low-SNR conditions.
First, we conducted objective evaluations, including
two quantitative measures. The first measure was the segmental SNR, in dB, defined by Quackenbush [14]
M 
N−1

1 
SNRSeg =x2 (k, m)
10 log10
M m=1
k=0

N−1 
2 
x(k, m) − x̂(k, m) ,
(15)
k=0

where M represents the number of frames containing speech
signals, N is the number of samples per frame, x(k, m) represents the clean speech at the k-th sample in the m-th frame,and x̂(k, m) for the enhanced speech. The second quantitative measure was the log-spectral distortion (LSD), in dB,
defined by Quackenbush [14]
N/2 
M 

1
1 
20 log10 X(k, m)
LSD =M m=1 N/2 + 1 k=0
2 1/2,(16)− 20 log10 X̂(k, m)
where X(k, m) and X̂(k, m) are the amplitude spectra of clean
and enhanced speeches at the k-th frequency component ofTable 1
SNR
−5 dB
0 dB
5 dB

KLT
4.51
6.99
9.19

SNR
−5 dB
0 dB
5 dB

Segmental SNR under various noise conditions (dB).

Leopard
MMSE
DR+MMSE
3.65
5.87
7.19
8.52
10.36
10.94

Table 2

Leopard
MMSE
DR+MMSE
3.81
3.67
3.04
2.91
2.37
2.29

KLT
1.86
2.16
2.42

Leopard
MMSE
DR+MMSE
1.90
2.11
2.18
2.34
2.46
2.56

Table 3

−5 dB
0 dB
5 dB

KLT
2.60
5.59
8.70

Tank
MMSE
DR+MMSE
2.06
3.50
5.23
6.02
8.53
8.91

KLT
−0.47
2.47
5.55

BabbleMMSE
DR+MMSE
−1.52
−1.61
1.69
1.69
5.36
5.36

Log spectral distortion under various noise conditions (dB).

KLT
3.56
2.86
2.30

SNR

the m-th frame, respectively. These two objective quality
measures take into account both residual noise and speech
distortion. The higher the segmental SNR or the lower the
log-spectral distortion, the better the perceptual quality.
The performance of the proposed algorithm was evaluated by comparing two well-used algorithms. One is thegeneralized subspace algorithm, which is reported to be thebest of all traditional subspace approaches [8]. The other is
MMSE, which is used to show the improvement resulting
from the proposed DR algorithm. All these algorithms in
the evaluation used the same VAD as in Ying et al. [13].
Tables 1 and 2 show the improvement in segmental
SNR and the LSD, respectively, where “KLT” corresponds
to the conventional subspace approach [8], the proposed
method to “DR+MMSE” method and MMSE to the algorithm in [4]. From the experiments, one can see that the
proposed algorithm performed best out of the three algorithms. Under babble noise conditions, the DR algorithm
contributed less to noise reduction. The reason is that the
DR algorithm eﬃciently reduces most noises with diﬀerent energy distributions from speech, but not so eﬃcient forones with similar energy distributions to speech.
In addition to segmental SNR and LSD, perceptual
evaluation of speech quality (PESQ) was also used for evaluating speech quality. The PESQ is a mean opinion score(MOS)-like objective evaluation, which facilitates the objective evaluation of audio signal quality based upon perceptual
criteria. This provides a quantifiable voice quality measurement that tightly correlates with voice quality as perceived
by humans. The International Telecommunications Union(ITU)-PESQ algorithm converts the disturbance parametersin speech into a PESQ score, which ranges from −0.5 to
4.5. The higher the score, the better is the perceptual quality [15]. The evaluation results are listed in Table 3. The
same conclusion as those from Tables 1 and 2 was obtained.

KLT
5.61
4.41
3.30

Tank
MMSE
DR+MMSE
4.97
4.88
4.24
4.18
3.53
3.47

KLT
5.55
4.63
3.74

BabbleMMSE
DR+MMSE
5.46
5.55
4.68
4.68
3.87
3.87

KLT
1.19
1.56
1.89

BabbleMMSE
DR+MMSE
1.20
1.18
1.56
1.56
1.88
1.88

Perceptual evaluation of speech quality.
KLT
1.49
1.80
2.12

Tank
MMSE
DR+MMSE
1.73
1.81
2.04
2.07
2.31
2.33

IEICE TRANS. INF. & SYST., VOL.E92–D, NO.5 MAY 2009

1142
Table 4
SNR
−5 dB
0 dB
5 dB

KLT
1.90
2.36
2.88

Leopard
MMSE
DR+MMSE
2.18
3.27
2.74
3.68
3.57
4.06

Subjective evaluation (MOS).
KLT
1.11
1.53
2.27

Tank
MMSE
DR+MMSE
2.27
2.76
3.07
3.43
3.82
3.78

KLT
1.08
1.42
2.42

BabbleMMSE
DR+MMSE
1.55
1.53
1.67
1.67
3.00
3.00

4.3 Subjective Evaluation of the Enhanced Speech
To validate the objective evaluation, subjective listening
tests were performed under various SNR conditions. Twelve
TIMIT sentences (subset of the sentences used for the objective evaluation) produced by three male and three female speakers were used in the listening tests. The test
dataset consisted of 108 groups (12 sentences × 3 noises × 3
noise levels) and each group included three sentences processed respectively by MMSE, KLT, and the proposed algorithm. This test was taken by four female and four malesubjects. They are normal Chinese speaking subjects aged
22 to 28. Each volunteer gave each test sentence a MOS
score between one and five, corresponding to the subjective terms ‘bad’ (1), ‘poor’ (2), ‘fair’ (3), ‘good’ (4), and‘excellent’ (5) [12]. This evaluation represented each volunteer’s global appreciation of residual noise, backgroundnoise and speech distortion. The test sentences were presented to volunteers with headphones. For each speaker, thefollowing procedure was applied: 1) clean and noisy speech
signals were played before scoring, and 2) in each group,
the test signals were played in random order. The results
are presented in Table 4. From the subjective listening tests,
we obtained similar conclusions as those from the objective
evaluations.
In the above evaluations, the segmental SNR gives aquantitative description of the distance between the waveforms of enhanced and clean speech. LSD describes theirdistance in spectral amplitude. Both LSD and segmental
SNR are the measurement of geometrical distance between
clean and enhanced speech signal. The PESQ and MOS are
the evaluation of opinion score. The former is produced
based upon the objective perceptual criteria while the latter comes from the subjective perception of humans. Summarizing all the above evaluations, although the proposed
algorithm is not always the best on all measurements, it isoutstanding on most measurements. Especially, the subjective MOS evaluations showed clearly that the proposed algorithm demonstrated the best performance. Accordingly,
we can conclude that the proposed algorithm performs better than conventional algorithms.
5.

Discussion

As described in the above sections, the proposed algorithm
shows advantages under color noise conditions. This section
investigates the reasons why the proposed method performs
well for color noises. Since the “color” can be easily described in the frequency-domain, it is better to investigate

Fig. 3 Energy changes of speech and noise in Fourier space under tanknoise conditions.

what happens in the frequency domain when the DR algorithm is applied in the noise eigenspace.
Here, we use noisy speech utterances to investigate theenergy variation of speech and noise in frequency domain.
Two typical color noises, tank and leopard, were used. They
are respectively added to a clean utterance at average SNRof 0 dB to obtain two noisy speech utterances. By applying
the DR algorithm to the leopard noisy utterance, 89.2% of
the noise was reduced and 2.1% of speech distortion was introduced. For the utterance contaminated by the tank noise,
the noise reduction rate and speech distortion were 74.7%
and 2.7%, respectively. This result shows that the DR algorithm produces better results under leopard noise conditions. The reason is that there was more of a color component in the leopard noise than in the tank noise as shownedin Fig. 3 (a) and Fig. 4 (a). As a result, the DR algorithm
reduces more noise in leopard noise conditions.
After applying the DR algorithm, the signal in the
speech subspace was transformed into the frequency domain using fast Fourier transform (FFT). Then, we compare
the diﬀerence of energy distribution before and after DR algorithm. Figure 3 illustrates the speech and noise energychanges in the frequency domain under tank noise conditions. From Fig. 3 (a), one can see that the noise signal in
the low-frequency domain was significantly reduced. At thesame time, the speech signal was aﬀected a little, as shownin Fig. 3 (b), which means the speech distortion is controlled
at a low level. As a result, the SNR in the low-frequency
domain was greatly improved. Figure 4 shows the energychanges under leopard noise conditions. One can see thatthe noise was generally reduced in almost the whole fre-

YING et al.: SPEECH ENHANCEMENT BASED ON NOISE EIGENSPACE PROJECTION

1143

rithms for processing the color component.
6.

Fig. 4 Energy changes of speech and noise in Fourier space under leopard noise conditions.

quency band, while the speech signal was aﬀected a little,
as seen in Fig. 4 (b). Thus, the SNR improved in most frequency components. These figures compare the energy distribution diﬀerence before and after DR algorithm. Fromthese comparisons, we can see that the speech distortion canbe controlled at a low level in the process of noise reduction.
The same phenomena were also observed with other typesof noises. In contrast, conventional algorithms do not havesuch an intrinsic mechanism to control speech distortion.
From a signal decomposition point of view, the DR algorithm decomposes the input noisy speech signal into two
sub-signals, as shown in the following equations.
y = ySS + yNS 
yNS = ϕ1 , · · · , ϕq × <y, ϕ1 >, · · · , <y, ϕq >

(17)
(18)

where yNS is the sub-signal from the noise subspace and ySS
is the sub-signal from the speech subspace. yNS includes
most of the color component and ySS contains most of thewhite-like component. In the proposed algorithm, yNS is set
to zero. In other words, the DR algorithm actually removes
a sub-signal with extremely low SNR to reduce the color
component of the noise signal.
Conventional algorithms (such as MMSE, spectral subtraction, or KLT) reduce the two sub-signals as a whole.
This means, for details of algorithm used by Ephraim [4],not only ySS but also yNS are processed with MMSE. We
know that yNS contains little speech, and its actual posterior SNR is extremely low. The theoretical output fromyNS should approach zero. However, the MMSE algorithm
is unreliable for processing low-SNR signals. In practicalcases, its output from yNS is not normally zero so that it introduces extra residual noise. The same thing will happen to
MMSE and other conventional algorithms. If yNS is directly
cleaned up, the residual noise from yNS can be avoided andlittle speech distortion introduced. In this sense, we can saythat the DR algorithm eﬃciently suppresses residual noise
as well as prevents significant speech distortion. Therefore,
the DR algorithm is more eﬃcient than conventional algo-Conclusion and Future Perspectives

Our aim was to develop a speech enhancement algorithm
based on noise eigenspace projections. The noise is reduced by using two diﬀerent strategies: one for removing
the white-like components and the other for removing thecolor components. Experimental evaluations showed thatthe proposed DR algorithm reduces noise with less speech
distortion. It eﬃciently suppresses residual noise as wellas prevents significant speech distortion. By combining DR
with MMSE, an eﬃcient speech enhancement can be realized.
There is still room for further improvement in the performance of the proposed algorithm. When removing thenoise subspace, a little speech signal is unavoidable to belost. The speech distortion is expected to be further minimized by recovering the lost speech signal in the noise subspace. This issue will be the aim of a future study.
Acknowledgements
This study is mainly supported by the program for the“Fostering Talent in Emergent Research Fields” in Special
Coordination Funds for Promoting Science and Technology by the Ministry of Education, Culture, Sports, Scienceand Technology. This work was partially supported by aGrant Program of the Yazaki Memorial Foundation for Science. It was also supported by the Strategic Information
and Communications R&D Promotion Program (SCOPE)(071705001) of the Ministry of Internal Aﬀairs and Communications (MIC), Japan.
References[1] Y. Ephraim, H. Lev-Ari, and W.J.J. Roberts, “A brief survey ofspeech enhancement,” The Electronic Handbook, pp.2088–2097,
CRC Press, 2005.
[2] S.F. Boll, “Suppression of acoustic noise in speech using spectral subtraction,” IEEE Trans. Acoust. Speech Signal Process.,
vol.ASSP-27, no.2, pp.113–120, April 1979.
[3] P. Scalart and J.V. Filho, “Speech enhancement based on a priorisignal to noise estimation,” Proc. ICASSP1996, vol.2, pp.629–632,
Atlanta, USA, May 1996.
[4] Y. Ephraim and D. Malah, “Speech enhancement using a minimum
mean square error short time spectral amplitude estimator,” IEEE
Trans. Acoust. Speech Signal Process., vol.ASSP-32, pp.1109–
1121, Dec. 1984.
[5] M. Dendrinos, S. Bakamidis, and G. Carayannis, “Speech enhancement from noise: A regenerative approach,” Speech Commun.,vol.10, no.1, pp.45–57, Feb. 1991.
[6] Y. Ephraim, “A signal subspace approach for speech enhancement,”
IEEE Trans. Speech Audio Process., vol.3, no.4, pp.251–266, July
1995.
[7] A. Rezayee and S. Gazor, “An adaptive KLT approach for speechenhancement,” IEEE Trans. Speech Audio Process., vol.9, no.2,
pp.87–94, Feb. 2001.
[8] Y. Hu and P. Loizou, “A generalized subspace approach for enhancing speech corrupted with colored noise,” IEEE Trans. Speech Audio
Process., vol.11, no.4, pp.334–341, July 2003.

IEICE TRANS. INF. & SYST., VOL.E92–D, NO.5 MAY 20091144

[9] K. Hermus, P. Wambacq, and H.V. Hamme, “A review of signal subspace speech enhancement and its application to noise robust speech
recognition,” EURASIP J. Advances in Signal Processing, vol.2007,
no.1, pp.195–210, 2007.
[10] P. Yip and K.R. Rao, “Energy packing eﬃciency for generalized
discrete transforms,” IEEE Trans. Commun., vol.COM-26, no.6,
pp.1257–1262, Aug. 1978.
[11] Y. Hu and P. Loizou, “Subjective evaluation and comparisonof speech enhancement algorithms,” Speech Commun., vol.49,
pp.588–601, 2007.
[12] J.R. Deller, J. Hansen, and J.G. Proakis, Discrete-Time Processingof Speech Signals, IEEE Press, New York, 2000.
[13] D. Ying, Y. Shi, X. Lu, J. Dang, and F. Soong, “Robust voice activity
detection based on noise eigenspace,” Acoust. Sci. and Tech., vol.28,
no.6, pp.413–423, 2007.
[14] S. Quackenbush, T. Barnwell, and M. Clements, Objective Measuresof Speech Quality, Prentice-Hall, Englewood Cliﬀs, NJ, 1988.
[15] ITU-T Rec. P.862, Perceptual Evaluation of Speech Quality (PESQ),an Objective Method for End-to-End Speech Quality Assessment of
Narrowband Telephone Networks and Speech Codecs, International
Telecommunications Union, Geneva, Switzerland, 2001.
[16] X.D. Zhang, Matrix analysis and applications, Tsinghua & Springer,
Beijing, 2004.

=

K

ϕk T E(xxT )ψ j ϕk T ψ j

j=1

=

K


ϕk T C sp ψ j ϕk T ψ j .

(A· 5)

j=1

Substituting Eq. (7) into the Eq. (A· 5), we obtain
sk =

K


ϕk T ψ j γ j ψTj ϕk

j=1

=

K




ϕk T ψ j 2 γ j ,k = 1, 2, · · · , K.

(A· 6)

j=1

Since the projection is a unit-orthogonal transform, according to the law of energy conservation, we obtain Kj=1 s j =K
j=1 γ j . The left and right side of Eq. (A· 6) are respectively
divided by Kj=1 s j and Kj=1 γ j . Then, Eq. (A· 6) can be written as Eq. (9).

Appendix
In this appendix, we prove that the mapping function of
speech energy distribution from the speech eigenspace into
the arbitrary noise eigenspace can be represented as Eq. (9).
On one hand, according to the projection theorem [16], thespeech signal in a frame, x, can be represented using the
speech eigenvector, ψ j .
x=

K 



x, ψ j ψ j .

(A· 1)

j=1

On the other hand, the projection of the speech frame
vector x can be written as(A· 2)
x, ϕk = ϕk T x.
Substituting Eq. (A· 1) into Eq. (A· 2), we obtainK x, ϕk =
xT ψ j ϕk T ψ j .

(A· 3)

j=1

The expectation of projection energy of the speech signal in
the kth dimension of the noise eigenspace is represented as2sk = E x, ϕk ,where E(·) is the expectation operator.
Eqs. (A· 2) and (A· 3) into Eq. (A· 4), we get
⎞
⎛
K

⎟⎟⎟
⎜⎜⎜
T
T
T
⎜
sk = E ⎜⎜⎝ϕk x
x ψ j ϕk ψ j ⎟⎟⎟⎠
j=1
⎞
⎛ K
⎟⎟⎟
⎜⎜⎜
= E ⎜⎜⎜⎝ ϕk T xxT ψ j ϕk T ψ j ⎟⎟⎟⎠
j=1

(A· 4)
Substituting

Dongwen Ying
received his B.E. degree in1998 and his M.E. degree in 2000, respectively,from Harbin Institute of Technology. He earned
his Ph.D. from the School of Information Science, Japan Advanced Institute of Science andTechnology (JAIST), in 2007. He is currently
working at the Institute of Acoustics, Chinese
Academy of Sciences as an associate researcher.
His major interests are speech enhancement androbust speech recognition.

Masashi Unoki
received his M.S. and Ph.D.
(Information Science) from the Japan Advanced
Institute of Science and Technology (JAIST) in
1996 and 1999, respectively. His main researchinterests are auditory-motivated signal processing and the modeling of auditory systems. He
was a JSPS research fellow from 1998 to 2001.
He was associated with the Advanced Telecommunications Research Institute (ATR) Human
Information Processing Laboratories as a visiting researcher during 1999–2000, and from2000 to 2001 he was a visiting research associate at the Centre for the
Neural Basis of Hearing (CNBH) in the Dept. of Physiology at the University of Cambridge. He has been on the faculty of the School of Information Science at JAIST since 2001, and he is now an associate professor. He is a member of the Research Institute of Signal Processing (RISP),
the Acoustical Society of America (ASA), the Acoustical Society of Japan(ASJ), the Institute of Electrical and Electronic Engineering (IEEE), andthe International Speech Communication Association (ISCA). Dr. Unoki
received the Sato Prize for an Outstanding Paper from the ASJ in 1999 andthe Yamashita Taro Prize for Young Researchers from the Yamashita Taro
Research Foundation in 2005.

YING et al.: SPEECH ENHANCEMENT BASED ON NOISE EIGENSPACE PROJECTION

1145

Xugang Lu
received the B.S. and M.S. from
Harbin Institute of Technology, China, in 1994
and 1996, respectively. He earned his Ph.D.
from the National Laboratory of Pattern Recognition, Chinese Academy of Science in 1999.
He joined Nanyang Technological University,
Singapore, as a research fellow after Oct. 1999.
Since Dec. 2001, he was a Postdoctoral fellow
at McMaster University, Canada. After April
2003, he has been with the faculty of the Schoolof Information Science of Japan Advanced Institute of Science and Technology, and now is an Assistant professor. Dr.
Lu received the President award from the Chinese Academy of Science in
1999. He is a member of the Acoustic Society of Japan. His main researchinterests include speech signal processing and recognition, statistic learningand pattern recognition, and computational auditory model.

Jianwu Dang
received his B.E. and M.S. degrees from Tsinghua University, China in 1982
and 1984, respectively. He worked at Tianjin
University as a lecturer from 1984 to 1988. He
earned his Ph.D. Eng. from Shizuoka Univ.,
Japan in 1992. Dr. Dang worked for ATR Human Information Processing Lab., Japan from1992 to 2001. He joined the University ofWaterloo, Canada, as a visiting scholar for one
year from 1998. He joined Japan Advanced Institute of Science and Technology (JAIST) in
2001 and is currently a professor there. He joined the Institute of Communication Parlee, Center of National Research Scientific (CNRS), France,
as a research scientist the first class for one year beginning in 2002. His
research interests are in all of the fields of speech science, especially inspeech production. He is a member of the Acoustic Societies of America
and Japan, and also a member of the Institute of Image Information andTelevision Engineers.