Semi-Blind Algorithm for Joint Noise Suppression
and Dereverberation Based on Higher-Order
Statistics and Acoustic Model Likelihood
Fine Dwinita Aprilyanti∗ , Hiroshi Saruwatari∗ , Kiyohiro Shikano∗ , Satoshi Nakamura∗ and Tomoya Takatani†∗ Graduate School of Information Science, Nara Institute of Science and Technology, Nara, Japan

E-mail: dwinita-a@is.naist.jp Tel: +81-743-72-5287† Toyota Motor Corporation, Aichi, Japan
E-mail: tomoya takatani@mail.toyota.co.jp Tel: +81-565-98-6462

Abstract—In this paper, we propose an automatic optimization
scheme of FD-BSE-based joint suppression of noise and latereverberation to improve the speech recognition accuracy for
spoken-dialogue system. First, we optimize the parameter ofconventional FD-BSE-based method using the assessment ofmusical noise measured by higher-order statistics and acoustic
model likelihood. Next, to maintain the optimum performance ofthe system, we proposed the switching scheme using the distanceinformation provided by image sensor. The experimental results
show that the proposed approach improves the word recognition
accuracy.

I. I NTRODUCTION
Hands-free robot dialogue system captures the user’s utterances at a distance using a microphone array, resulting
in more natural human-machine interaction. In this system,
however, it is difficult to achieve accurate speech recognition
because of the adverse effect from background noise and room
reverberation. While the early reverberation is consideredharmless to speech intelligibility, the late part can deteriorate
the sound quality, depending on the length and strength of thisreverberation [1]. Therefore, a method that can suppress theseinterferences is required to improve the recognition accuracy.
Many speech enhancement method have been studied tosolve these problems. In [2], it is shown that frequency-domain
independent component analysis (FD-ICA) [3] performs betterin estimation of diffuse background noise than that of target
speech. Hence, combining FD-ICA as noise estimator and anonlinear postprocessing to suppress the estimated noise has
been proved to be effective in improving the target soundquality. Some of the authors have proposed an improved
diffuse noise estimator, namely, frequency-domain blind signal extraction (FD-BSE) [4]. Then, by also considering theeffect of room reverberation, they combine the FD-BSE with
two stages of multichannel Wiener filtering (WF) to jointly
suppress diffuse background noise and late reverberation [5](hereafter this method is referred to as joint method). To
synthesize the late reverberation component, this method uses
a priori knowledge of the room reverberation time (T60 ).
Joint method proposed in [5] has been proved to be effective
to improve the recognition accuracy under noisy and reverberant condition. However, some of the parameters still require

manual setting. Therefore, in this paper, first, we address
the optimization problem of FD-BSE-based joint method.
Motivated by previous works of the authors [6], [7], we haveproposed an optimization scheme based on the assessment ofmusical noise, which is the artificial distortion generated as
an effect of nonlinear signal processing [8]. In this paper,
we integrate the current optimization scheme with automatic
speech recognizer (ASR) under maximum likelihood criterion
to improve the recognition accuracy performance.
Next, to achieve high recognition accuracy regardless of
the level of interferences, we utilize information on the user
distance. This information is obtained under the assumptionthat a robot has it own camera that can immediately detect theposition of the target user. Finally, we confirm the effectivenessof proposed method through experimental evaluations.
II. R ELATED W ORK : FD-BSE- BASED J OINT N OISES UPPRESSION AND D EREVERBERATIONIn this section, we will review the previously proposed
joint method. The architecture of this method is shown in
Fig. 1. Generally, the method can be divided into two stages,
namely, noise suppression stage and dereverberation stage.
Two WFs are utilized separately in each stages, providing
flexibility in adjusting filter strength according to the levelof each interferences. The dynamic characteristic of these
nonlinear filters help to improve the quality of captured speechin real environment due to nonstationary characteristics ofinterferences.
The observed signal x(t) captured at microphone array isgiven byx(t) = (hE (τ ) + hL (τ )) ∗ s(t) + n(t),(1)

where s(t) and n(t) are the clean speech source and noise,respectively, and hE (τ ) and hL (τ ) indicate the early and lateroom impulse response. Most hidden Markov model (HMM)
based speech recognizers are capable to handle the effect ofhE (τ ) up to certain time delay τd , for example by applying
cepstral mean normalization.

Fig. 1. Block diagram of joint blind noise suppression and dereverberation
method.

Fig. 3. Performance comparison of FD-BSE and FD-BSS.
TABLE IS YSTEM SPECIFICATION OF ASR
Frame length
Frame period
Pre-emphasis
Feature vectors
Fig. 2. Room configuration for the experiment.
Acoustic model

A. Noise Suppression Stage
Conventional joint method uses FD-BSE algorithm to estimate both direction of arrival (DOA) θ̂ and the backgroundnoise component. Unlike conventional FD-ICA, FD-BSE exploits the sparseness of the modulus of the target speech
signal [4]. In frequency domain, the extracted output Y (f, t) is
obtained by applying extracting vector to the observed signal,
as given by
Y (f, t) = A(f )X(f, t).

(2)

The vector A(f ) is updated using a gradient descent method
to minimize the cost function J(A(f )) given by,
1
J(A(f )) =E{|Y (f, t)|}2 ,(3)
2
E{|Y (f, t)|2 } = 1.
(4)
In the case of a target speech within background noise, thespeech modulus may be considered sparser than that of the
diffuse background noise components in the sense that mostof its values are close to zero and only a few are significantly
large. Thus the cost function is minimum when the target
speech component is extracted. In this way, it is not required
to confirm the selection of noise components, which means
the permutation problem as in frequency-domain blind signal
separation (FD-BSS) can be avoided.
The estimated noise N̂ (f, t) is obtained by applying projection back to the residual output Y (noise) (f, t) which only
contains noise component, given byN̂ (f, t) = A−1 (f )Y (noise) (f, t).

(5)This noise estimate is then suppressed using a set of multichannel WF as given byXˆS (f, t)= G|X(f, t)|ejarg(X(f,t)) ,|X(f, t)|2,
G =|X(f, t)|2 + βN |N̂ (f, t)|2

(6)
(7)

Language model
Training data

25 ms
10 ms
1 − 0.97z −1
12-order MFCC,
12-order ∆MFCC,
1-order ∆E
HMM phonetic tied mixture (PTM),
2000 states, GMM 64 mixtures
standard word trigram model
Adult JNAS database

where βN is a parameter for controlling the strength of noise
suppression.
B. Dereverberation Stage
Assuming that the noise suppression stage is effective, theestimated XˆS (f, t) contains only early reverberant speech
X E (f, t) and late reverberant speech X L (f, t). In this stage,
first the late reverberation component is synthesized according
to equation (1), given byxˆL (t) = hL (τ ) ∗ ŝ(t),(8)

where hL (τ ) is approximated by generating channel-wisesynthetic tail from decayed Gaussian random variable. The
direct speech s(t) estimation requires more strategy as there
is no clean speech can be used as reference. In this method,s(t) is estimated using by projecting back the ouput of noise
suppression stage X̂ s (f, t) to the truncated FD-BSE filter.
After that, dereverberation process is done in the same manner
as noise suppression stage, using multichannel WF given by,XˆE (f, t)

=

G =

ˆ

G|XˆS (f, t)|ejarg(X S (f,t)) ,|XˆS (f, t)|2
|XˆS (f, t)|2 + βR |XˆL (f, t)|2

(9)
,

(10)

where βR is a parameter for controlling the strength of
dereverberation. Consequently, this stage requires a priori
information of T60 .
III. AUTOMATIC O PTIMIZATION S CHEME OF J OINTM ETHOD
A. Motivation and Strategy
In order to confirm the effectiveness of FD-BSE to copewith diffuse background noise and reverberation, we conduct

of the best parameters, namely, βN and βR , for speech
recognition performance. The common way to evaluate thisperformance by using word accuracy, described as
WA = 100 ×Fig. 4. Effect of parameter settings in joint method to recognition accuracy
result for input signal at 5m distance and SNR of 10 dB.

preliminary experiment to compare the output from FD-BSE
and FD-BSS method. Here we use ICA algorithm in [9] for
FD-BSS. In addition, the modulus-sparseness-based permutation solver is also applied to the FD-BSS method, so theperformance of main algorithm in both FD-BSE and FD-BSS
can be compared.
An 8-channel microphone array was use to record the room
impulse response with the configuration as shown in Fig.
2. The estimated T60 is 500 ms. For the input signals, 10utterances (average length ≈ 5 s) were convoluted with real
recorded impulse response at various distance between speakerand microphone array, and then were mixed with 10 dB SNR
noise.
The performance of each methods is evaluated using noise
reduction rate (NRR), which is defined as the difference of
signal-to-noise ratio (SNR) of signal before and after processing. The SNR of a signal is represented by
SNR = 10 log10E[s(t)]2,
E[n(t)]2(11)

where s(t) and n(t) are the speech and noise component ofsignals, respectively. Since high SNR indicates good signal
quality, high NRR result is preferable.
Figure 3 shows the NRR result of estimated output speechfrom each methods. It is clearly shown that the conventional
FD-BSS cannot perform well in the presence of diffuse
background noise and reverberation. The performance is significantly improved in modified FD-BSS due to the use ofmodulus-sparseness-based permutation solver. However, we
can see that the FD-BSE still outperforms in every condition.
Moreover, the computation time of FD-BSE is greatly reducedin comparison to that of FD-BSS, with the ratio of 0.37 in
average. It is possible due to the fact that the update rulein FD-BSE only involves n × 1 vector, while it is computed
for n × n matrix for FD-BSS. Thus, the use of FD-BSE ispreferable.
By combining FD-BSE and WF filters in joint method as
described in Sect. II, we gain more flexibility in suppressing
the interferences, thus making the method more robust to
various acoustical condition. However, one expected problem
to arise is the complex parameter optimization and prediction

N − (I + S + D)
,
N

(12)

where N is the number of words in the reference, I is thenumber of insertions, S is the number of substitutions, and D
is the number of deletions.
Another preliminary experiment has been done to analyze
the effect of parameter settings in joint method to recognition
accuracy. For the recognition task, we use similar condition as
the previous experiment, and a 20K-word Japanese dictation
task from the JNAS database [10] is used as performancemeasure. Here, the reference signal is clean speech convoluted
with early impulse response. We use JULIUS [11] as speech
recognizer. The specification for speech recognition performance evaluation is shown in Table I. As depicted in Fig. 4, thecorrect setting of both parameters of joint method will result inbetter word accuracy compared to speech estimation from FDBSE alone. On the other hand, bad setting of the parameters
will result in worse performance. In the conventional joint
method, the optimization is still done manually, which makes it
impossible to be implemented in real environment. Therefore,
it is our concern to build an efficient method to control theseparameters automatically.
The previous optimization scheme is based on higher-order
statistics which corresponds to the assessment of generated
musical noise [8]. In this paper, we propose an optimization
scheme that utilizes two control parameters, namely, higherorder statistics and acoustic model likelihood. The proposed
optimization scheme is focused on setting of parameters of
WFs, under the assumption that T60 is known.
B. Parameter Optimization Based on Higher-Order Statistics
The amount of musical noise is highly correlated to thenumber of isolated power spectral components and theirlevel of isolation. These isolated components are called tonalcomponents. Since such tonal components have relatively highpower, they are strongly related to the weight of the tail of theirprobability density function (pdf). Therefore, it is possible toassess the amount of musical noise using statistics measuresof their pdf. Kurtosis has been introduced in our studies toevaluate the tail of the pdf [12], successfully showing theeffectiveness of using the higher-order statistics.
In this paper, first, we calculate the frequency subband-wisekurtosis [7] as given byPP(1/M ) f ∈Fi t∈T (|X(f, t)|2 )4
(i)
P
P
,
(13)
kurt ={(1/M ) f ∈Fi t∈T (|X(f, t)|2 )2 }2where kurt(i) is the i-th subband kurtosis of a signal x. Fi
and T represent the evaluated subband time-frequency gridindexes, while M indicates the total number of grids in eachsubband. Here 250-Hz-width Fi and T of 5 s are used, which
are taken from noise-only time-frequency region preceding a

Fig. 5. Optimization scheme of joint method via higher-order statistics and acoustic model likelihood.

of this solution in optimizing several speech enhancement
methods, for example on beamforming method [13].
In speech recognizer, a series of fixed size acoustic vectors
o(β) = [o1 , ..., oT ] is extracted from the output of speech enhancement with parameter β through some feature extraction
process. During decoding, it attempts to hypothesize the word
sequence W = [w1 , ..., wK ] which is the most probable togenerate the sequence o(β), as stated byŴ = arg max P (W |o(β)).

(15)

W

However, the recognition system cannot compute the posterior probability P (W |o(β)) directly. Instead, the above
expression is transformed into the following form based on
Bayes’ theorem:
Ŵ = arg max
W

Fig. 6. Experimental evaluation of optimization scheme; (a) NRR, and (b)Word accuracy.

speech utterance. Then, the assessment of generated musicalnoise is done by applying the kurtosis ratio, given by
kurtosis ratio = kurtproc /kurtorg ,(14)

where kurtproc is the kurtosis of the processed signal andkurtorg is the kurtosis of the observed signal.

P (o(β)|W )P (W ),P (o(β))(16)

where P (o(β)|W ) is the acoustic likelihood or acoustic
score, representing the probability that feature sequence o
is observed given that word sequence W was spoken, andP (W ) is the language score, i.e., the a priori probability ofa particular word sequence W . The former term is calculated
from acoustic model, while latter term is computed using alanguage model.
Since Eq. (16) is maximized with respect to the word
sequence W for a given observed sequence o that is fixed,
the denominator term P (o(β)) can be ignored. Thus, theparameter β can be optimized by maximizing the likelihoodof acoustic model of speech recognizer, as written byβ̂ = arg max P (o(β)|W ).

(17)

β

C. Parameter Optimization Based on Acoustic Model Likelihood

D. Algorithm and Experimental Evaluation

In spoken-dialogue system, the speech enhancement method
can only be expected to improve recognition performance if
it generates results that improve the likelihood of the correcttranscriptions. Therefore, we also optimize the parameter tomaximize the likelihood of acoustic model of speech recognizer. Previous researches have confirmed the effectivenessThe block diagram of the proposed optimization scheme
is shown in Fig. 5. Provided the a priori knowledge ofT60 , the WF parameters are updated consecutively. First, theresidual noise quality is assessed from silence part of signals
preceeding the utterances, using voice activity detection basedon noise estimation from FD-BSE. Using this assessment, βN

Fig. 7. Block diagram of semi-blind optimized joint method.
Fig. 8. Word recognition accuracy result of proposed scheme.

is updated to achieve optimum NRR under a kurtosis ratio
constraint, as given byβˆN = arg max NRR(βN ),βNkurtproc (βN )≤ KRlim , (18)kurtorg

where KRlim is a constraint value of KR. Next, βR is updated
according to acoustic model likelihood according to Eq. (17),to optimizeβˆR = arg max P (o(βR )|W ).

(19)

βR

This two-step scheme has a great advantage that the complex optimization of two parameters βN and βR can bereasonably decomposed into two simple optimization for eachparameters. By using noise kurtosis ratio constraint in noise
suppression stage, the unwanted background noise can besuppressed effectively, thus the late reverberation synthesis becomes more accurate. Moreover, the acoustic model likelihood
criterion for optimization in dereverberation stage will ensure
optimum recognition accuracy of the output speech.
We conduct experiments to evaluate the performance of the
proposed optimization scheme. For this experiment, we use
200 utterances from JNAS database as input signals, whileother settings are set similar to preliminary experiment. The
performance of the optimized joint method (opt) is comparedwith the estimated speech from FD-BSE (bse). The experimentis also conducted on underestimated T60 of 300 ms, to analyze
the effect of mismatched value to the performance of the
proposed scheme. The experimental results are shown in Fig.
6. For the recognition performance, the reference is indicatedby (ref) and the unprocessed observed signal is indicated by(obs).
From the figure, it is shown that the proposed scheme issuperior to FD-BSE in terms of noise reduction performance.
However, the word recognition accuracy result shows thatthe speech estimate from FD-BSE gives best accuracy atclose distance compare to the proposed method. The effect of
mismatched T60 , as one can expect, is not very clear in case of
NRR, but it is significant at recognition accuracy, particularly
under heavily reverberant condition indicated by far speakerto microphone distance.

IV. S EMI - BLIND J OINT N OISE S UPPRESSION AND
D EREVERBERATION U SING I MAGE I NFORMATION
A. Motivation and Strategy
The experimental result shows that the proposed optimization scheme fails to achieve optimum speech recognition
accuracy at close user distance. This may be happened becauseof the following reasons:• FD-BSE as linear filter results in output signal with lessdistortion when the interferences effect is not so severe,
compare to the proposed scheme that includes nonlinear
processing.
• At closer user distance, the effect of room reverberation
is light to moderate. The late reverberation becomesoverestimated in the proposed scheme.
• Some part of late reverberation may have been treated as
background noise due to similar characteristics.
In order to achieve the optimum performance, we need asystem that can select the signal processing method to be
applied according to the condition of environments. In thispaper, we utilize the user position information, provided byrobot’s camera, and develop a multimodal switching scheme
according to the distance information, under assumption thatthe user distance corresponds to the effect of interferences.
B. Algorithm and Experimental Evaluation
The block diagram of the proposed multimodal scheme isshown in Fig. 7. First, the system is trained to estimate thedistance limit to switch method. Both FD-BSE and optimized
joint method are applied to the signals at various speakerto microphone distance, and the results are compared. Next,
after the switching point is decided, the system will apply two
different schemes according to the user position:• For close user distance, only FD-BSE estimation is applied to the input signal. The extracted speech from FDBSE becomes the output signal.
• For far user distance, the optimized joint method isapplied in addition to the FD-BSE.
The average result of recognition performance is depicted
in Fig. 8. We compare the performance of semi-blind joint
method with estimated speech from FD-BSE (bse) and the
optimized joint method (opt). In average, semi-blind method
achieves better word recognition accuracy compare to theconventional method. This shows that the semi-blind joint

approach can maintain optimum performance regardless theinterference condition, in contrast with the other two methods.
V. C ONCLUSIONSIn this paper, first, we have proposed an optimization
scheme of FD-BSE based joint noise suppression and dereverberation via higher-order statistics and acoustic model likelihood. Next, to maintain optimum word recognition accuracy
performance, we developed a semi-blind method selection
scheme using image information of user position. The experimental result confirms the effectiveness of our proposed
method.
VI. ACKNOWLEDGEMENTS
This work is partly supported by JST Core Research of
Evolutional Science and Technology (CREST), Japan.
R EFERENCES[1] Y. Huang, J. Benesty, and J. Chen, “Dereverberation,” Handbook of
Speech Processing, J. Benesty, et al. [Eds], pp.929-942, Springer, 2008.
[2] Y. Takahashi, T. Takatani, K. Osako, H. Saruwatari, and K.
Shikano,“Blind Spatial Subtraction Array for Speech Enhancement inNoisy Environment,” IEEE Trans. Audio, Speech, Lang. Process., vol.17,
no.4, 2009, pp.650-664.
[3] S. Ikeda and N. Murata, “A method of ICA in time-frequency domain,”in Proc. ICA99, Jan. 1999, pp.365-371.
[4] J. Even, H. Saruwatari, and K. Shikano, “Blind signal extraction based
speech enhancement in presence of diffuse background noise,” in Proc.
IEEE SSP 2009, pp.513-516, 2009.
[5] J. Even, H. Saruwatari, K. Shikano, and T. Takatani, “Blind signal
extraction based joint suppression of diffuse background noise and latereverberation,” in Proc. EUSIPCO, pp.1534-1538, 2010.
[6] T. Inoue, H. Saruwatari, K. Shikano, Y. Takahashi, and K. Kondo,“Theoretical analysis of musical noise in generalized spectral subtraction
based on higher-order statistics,” IEEE Trans. Audio, Speech, Lang.
Process., vol.19, no.6, 2011, pp.1770-1779.
[7] H. Saruwatari, Y. Ishikawa, Y. Takahashi, T. Inoue, K. Shikano, and K.
Kondo, “Musical noise controllable algorithm of channelwise spectral
subtraction and adaptive beamforming based on higher order statistics,”
IEEE Trans. Audio, Speech, Lang. Process., vol.19, no.6, 2011, pp.1457–
1466.
[8] F. D. Aprilyanti, H. Saruwatari, K. Shikano, and T. Takatani, “Optimization Scheme of Joint Noise Suppression and Dereverberation Based on
Higher-Order Statistics,” in Proc. APSIPA ASC 2012, Dec. 2012.
[9] P. Smaragdis, “Blind separation of convolved mixtures in the frequency
domain,” Neurocomputing, vol.22, no.1-3, pp.21-34, 1998.
[10] K. Ito, et al., “JNAS: Japanese speech corpus for large vocabularycontinuous speech recognition research,” Journal of Acoust. Soc. Of
Japan, vol.20, 2006, pp.196–206.
[11] Julius, an open-source large vocabulary csr engine http://julius.sourceforge.jp.
[12] R. Miyazaki, H. Saruwatari, T. Inoue, Y. Takahashi, K. Shikano, K.
Kondo, “Musical-noise-free speech enhancement based on optimized
iterative spectral subtraction,” IEEE Trans. Audio, Speech, Lang. Process.,vol.20, no.7, 2012, pp.2080–2094.
[13] M. L. Seltzer and R. M. Stern, “Subband likelihood-maximizing beamforming for speech recognition in reverberant environments,” IEEE Trans.
Audio, Speech, Lang. Process., vol.14, no.6, 2006, pp.2109–2121.