2-1-4
両耳補聴システムにおける HRTF を利用した画像トラッキング併用型マルチモーダル・ブラインド音声抽出∗☆吉江孝太郎, 猿渡洋, 中村哲 (奈良先端大・情報)

1 Introduction
In the binaural hearing-aid systems, preservation ofsound localization cues is required even after noise reduction. However, the most of noise reduction methods are not designed to preserve sound localization cues.
Therefore, we should conduct preservation of sound localization with noise reduction.
The basic theory of the minimum mean-square error(MMSE) short-time spectral amplitude (STSA) estimator with dynamic noise estimation based on independent component analysis (ICA) [1] has been proposedby the authors [2]. This method can achieve high-quality
noise reduction by estimating the noise signals dynamically using ICA. Then, as an application for hearing-aid
system of this method, the sound-localization-preserved
MMSE STSA estimator with ICA-based noise estimation has been proposed [3]. However, in the conventional method [3], equi-binaural optimal spectral
gain is approximately estimated by using optimal spectral gains independently obtained from single-channel
MMSE STSA estimation for each channel. Thus, theconventional method does not use the binaural cue suchas a head ralated transfer function (HRTF). This fact often causes serious degradation of noise reduction performance, especially in the case that the target speechsource is laterally located in the right- (left-) hand side.
In this paper, we propose a new method combining amultichannel MMSE STSA estimator and an image sensor.

2 Previous Works
2.1 MMSE STSA Estimator with ICA-Based Noise
Estimation
One of the authors has proposed a blind speech extraction method [2] combining ICA-based dynamic noise estimation and target speech. Hereafter, we introduce astraightforward extension of the method in [2] to binaural hearing-aid system consisting of two inputs and twooutput for human ears [3]. The noise component estimated by ICA is used to determine a posteriori SNR andthe optimal spectral gain applied to the L and R channel signal, neglecting phase information; this enables the∗method to realize error-robust noise reduction.
First, in the reference path, we perform signal separation using an unmixing matrix W( f ) obtained by ICA.
The output signal y( f, τ) = [ys ( f, τ), yn ( f, τ)] is given byy( f, τ) = [ys ( f, τ), yn ( f, τ)] = W( f )x( f, τ),(1)
[⟨⟩]
[p]
H[p]
W
( f )=W ( f )+µ I − ψ(y( f, τ))y ( f, τ) W ( f ),
[p+1]

τ(2)
where ys ( f, τ) is the estimated target speech signal,yn ( f, τ) is the estimated noise signal, µ is the step-size
parameter, [p] is used to express the value at the pth stepof the iteration and I is the identity matrix. In addition,⟨·⟩τ denotes a time-averaging operator, MH denotes theconjugate transpose of matrix M, and ψ(·) is an appropriate nonlinear activation function. The estimated target
speech signal is discarded as it is not required because
we want to estimate only the noise component. Instead,
we construct a noise-only vector,y(noise) ( f, τ) = [0, yn ( f, τ)]T ,(3)from the output signal obtained by ICA using (1). Following this, we apply the projection back (PB) operation
to remove the ambiguity of amplitude and construct theestimated noise signal at both ears,[ζL ( f, τ), ζR ( f, τ)]T = W −1 ( f )y(noise) ( f, τ),(4)where ζL ( f, τ) and ζR ( f, τ) are the noise components estimated at the left and right channels, respectively.
Next, using the estimated noise components, we derive the gain function optimized on the basis of theMMSE STSA estimation criterion. Hereinafter, for convenience, the subscript ∗ ∈ {L, R} is used to indicate theindex of channels. The estimated a posteriori signal-tonoise ratio (SNR) γ̂∗ ( f, τ) is obtained from the observedsignal x∗ ( f, τ) and the estimated noise signals (4) as{}−1γ̂∗ ( f, τ) = |x∗ ( f, τ)|2 E{|ζ∗ ( f, τ)|2 }τ(τ−τth ) ,(5)where τth is a smoothing parameter denoting a certain
time frame window and E{·}AB denotes the expectation
operator from A to B. Note that we can momentarily
estimate the instantaneous a posteriori SNR (5) by utilizing the noise signal estimated by ICA (4), in contrast“Multimodal Blind Speech Extraction Using Image Tracking in HRTF-Informed Binaural Hearing-Aid System,” By
Kotaro Yoshie, Hiroshi Saruwatari, and Satoshi Nakamura (Nara Institute of Science and Technology).

日本音響学会講演論文集- 673 -

2014年3月

to the case using the original MMSE STSA estimator[4]. Therefore, we consider that our proposed method
can suppress nonstationary noise more eﬃciently thanthe conventional MMSE STSA estimator.
Next, using (5), the a priori SNR estimate ξ̂∗ ( f, τ) isgiven as

based on (10) is given by
Gopt ( f, τ)[= argmin E {|hL ( f )s( f, τ)| − GL ( f, τ)|xL ( f, τ)|}2G( f,τ)

+ {|hR ( f )s( f, τ)| − GR ( f, τ)|xR ( f, τ)|}2+ {(G( f, τ) − GL ( f, τ))|xL ( f, τ)|}2]+{(G( f, τ) − GR ( f, τ))|xR ( f, τ)|}2 + 2A ,ξ̂∗ ( f, τ) = αγ̂∗ ( f, τ − 1)G2∗ ( f, τ − 1)
+ (1 − α)P[γ̂∗ ( f, τ) − 1],

where α is the weighting factor of the decision-directed
estimation, G∗ ( f, τ) is a spectral gain function, and theoperator P[·] is a flooring function in which the negativeinput is floored to zero. Also, G∗ ( f, τ) is given as√()ν∗ ( f, τ)ν∗ ( f, τ)G∗ ( f, τ) =Γ(1.5)
expγ̂∗ ( f, τ)2[()ν∗ ( f, τ)· {1 + ν∗ ( f, τ)}I02()]ν∗ ( f, τ)+ν∗ ( f, τ)I1,(7)2where Γ(·) denotes the gamma function and I0 (·) and I1 (·)
denote modified Bessel function of the zeroth and firstorder, respectively. Moreover, ν∗ ( f, τ) is defined byν∗ ( f, τ) = ξ̂∗ ( f, τ)γ̂∗ ( f, τ){1 + ξ̂∗ ( f, τ)}−1 .

(11)

(6)
where Gopt ( f, τ) is the equi-binaural optimal spectral
gain to by estimated, GL ( f, τ) and GR ( f, τ) are individual spectral gains for the L and R channels, respectively, which are auxiliary parameters for calculating
an approximate solution of Gopt ( f, τ) because the direct
Bayesian estimation of Gopt ( f, τ) is diﬃcult. In addition,
A is related to the correlation between the estimation error and the observed signal in each channel when we estimate the target speech signals in the L and R channels
using the parameters GL ( f, τ) and GR ( f, τ), and is defined by
A = {G( f, τ) − GL ( f, τ)}{GL ( f, τ)|xL ( f, τ)|−|hL ( f )s( f, τ)|}|xL ( f, τ)|+{G( f, τ) − GR ( f, τ)}{GR ( f, τ)|xR ( f, τ)|−|hR ( f )s( f, τ)|}|xR ( f, τ)|.
(12)

(8)

Finally, noise reduction is carried out as follows:z∗ ( f, τ) = G∗ ( f, τ)x∗ ( f, τ),(9)where z∗ ( f, τ) is the final output of this method for bothears.
2.2 Equi-Binaural Optimal Spectral Gain Estimation
In [3], it is essential to apply equivalent spectral gains
for L and R channels to increase the localization accuracy. Therefore, to obtain the optimal spectral gainthat maintains the localization accuracy, we introduce
spectral gain that minimizes the residual noise power interms of the MMSE under the condition that the spectral
gains are equivalent in both channel. Hereafter, we callthis gain the equi-binaural optimal spectral gain. The
derivation of the equi-binaural optimal spectral gain canbe formulated as the minimization problem of the following error e:[e =E {|hL ( f )s( f, τ)| − G( f, τ)|xL ( f, τ)|}2]+{|hR ( f )s( f, τ)| − G( f, τ)|xR ( f, τ)|}2(10)
where G( f, τ) is the equi-binaural spectral gain, whichis considered as a variable. The optimization problem

日本音響学会講演論文集

We discuss the minimization of (11). First, the 1st and2nd terms of the right-hand side correspond to the problem of target speech estimation in each channel. Theseterms can be minimized if we obtain the optimal valuesof GL ( f, τ) and GR ( f, τ) using the MMSE STSA estimator described in Sect. 2.1. Next, A in the 5th term ofthe right-hand side can be disregarded if the parameters
GL ( f, τ) and GR ( f, τ) closely estimate the target speech
signals by approximately considering A to be negligible.
Hence, the residual 3rd and 4th terms, i.e., {(G( f, τ) −GL ( f, τ))|xL ( f, τ)|}2 + {(G( f, τ) −GR ( f, τ))|xR ( f, τ)|}2 , arethe candidates to be minimized. This problem can be formulated as[Gopt ( f, τ) ≒ argmin E {(G( f, τ) − GLopt ( f.τ))|xL ( f, τ)|}2
G( f,τ)]+ {(G( f, τ) − GRopt ( f.τ))|xR ( f, τ)|}2 ,(13)subject to[]GLopt ( f, τ) = argmin E {|hL ( f )s( f, τ)| − GL ( f, τ)|xL ( f, τ)|}2 ,GL ( f,τ)(14)]
GRopt ( f, τ) = argmin E {|hR ( f )s( f, τ)| − GR ( f, τ)|xR ( f, τ)|}2 ,- 674 -[GR ( f,τ)(15)2014年3月

Rmultichannel MMSE STSA estimator [5], which explicitly uses the HRTF in the estimation of the target speech
instead of single-channel MMSE STSA estimators foreach channel. This method can give the equi-binaural
optimal spectral gain without approximation.
The multichannel MMSE STSA estimator can beviewed as a cascade of minimum variance distortionlessresponse (MVDR) beamforming and the single-channel
MMSE STSA estimator [5]. The output of MVDR
beamforming is given byISTFTSTFTL

Equi-binauralgain calculation

Hearing-aid usera posteriori
SNR estimation

Spectral gaincalculation

TargetFDICA0a priori
SNR estimation

Projection
back(PB)MMSE STSA estimatorY( f, τ) =Fig. 1 Block diagram of sound-localization-preserved
MMSE STSA estimator [3].
where GLopt ( f, τ) and GRopt ( f, τ) are the L- and R- channel optimal spectral gains, respectively.
To solve (13) we first obtain GLopt ( f, τ) and GRopt ( f, τ)from the MMSE STSA estimator in (14) and (15), thenby substituting them into (13), we solve the following
equation in G( f, τ):(∂{(G( f, τ) − GLopt ( f, τ))|xL ( f, τ)|}2∂G( f, τ)

where ΣN ( f, τ) is a noise covariance matrix momentarily
estimated by ICA, which is defined asΣN ( f, τ)= E{[ζL ( f, τ), ζR ( f, τ)]H [ζL ( f, τ), ζR ( f, τ)]}τ(τ−τth ) . (19)1Then, the a posteriori SNR for Y( f, τ) is calculated asγ̂Y ( f, τ) = |Y( f, τ)|2 h( f )H ΣN ( f, τ)−1 h( f ).

(16)

+ (1 − α)P[γ̂Y ( f, τ) − 1],GLopt ( f, τ)|xL ( f, τ)| + GRopt ( f, τ)|xR ( f, τ)||xL ( f, τ)|2 + |xR ( f, τ)|2(21)

2.

(17)
Figure 1 shows the configuration of this method basedon the equi-binaural optimal spectral gain.

3 Proposed Method
We propose multimodal blind speech extraction using
image tracking in hearing-aid system and the multichannel MMSE STSA estimator, which explicitly introduces
a HRTF in estimation of the target speech instead of
single-channel MMSE STSA estimators for each channel. Figure 2 shows a block diagram of this method.
In the conventional method [3], the equi-binaural
optimal spectral gain is approximately estimated using optimal spectral gains independently obtained fromsingle-channel MMSE STSA estimation for each channel. Thus, the conventional method does not use binaural cues such as the HRTF. This fact often causes serious
degradation of the noise reduction performance, particularly in the case that the target speech source is laterally located on the right- (left-) hand side. Therefore, inthis section, we introduce the oracle method using the日本音響学会講演論文集(20)ξ̂Y ( f, τ) =αγ̂Y ( f, τ − 1)GY ( f, τ − 1)2

The solution of (16) is given by
Gopt ( f, τ) =(18)Next, using (20), the a priori SNR estimate ξ̂Y ( f, τ) isgiven as)+{(G( f, τ) − GRopt ( f, τ))|xR ( f, τ)|}2 = 0.

2h( f )H ΣN ( f, τ)−1 x( f, τ),h( f )H ΣN ( f, τ)−1 h( f )where GY ( f, τ) is the spectral gain function of the multichannel MMSE STSA estimator and is defined as√νY ( f, τ)
GY ( f, τ) =Γ(1.5)
exp (νY ( f, τ)/2)γ̂Y ( f, τ)[· {1 + νY ( f, τ)}I0 (νY ( f, τ)/2)]+νY ( f, τ)I1 (νY ( f, τ)/2) ,(22)νY ( f, τ) =ξ̂Y ( f, τ)γ̂Y ( f, τ){1 + ξ̂Y ( f, τ)}−1 .

(23)Finally, the resultant equi-binaural optimal spectral gain
without approximation that can strictly minimize (10) isgiven byGoracle ( f, τ) =|hL ( f )||xL ( f, τ)| + |hR ( f )||xR ( f, τ)||xL ( f, τ)|2 + |xR ( f, τ)|2· GY ( f, τ)Y( f, τ).

(24)

We assume that a camera is mounted on the hearingaid user’s head, and prepare the multiple HRTFs forsome degrees in advance and switch them according to
the camera’s image tracking result. However, it is quite
diﬃcult to estimate the HRTF blindly in practice because
the shape of the hearing-aid users’ heads greatly varies.
Therefore, we use single “typical” HRTF and some roomtransfer function (RTF).

- 675 -

2014年3月

User
θHRTFRL

Conventional Method

Equi-binauralgain calculation

CameraMVDR
beamformer

FDICA

Target Projection

0back(PB)

Noise estimator

Fig. 2

Spectral gain
calculationa posteriori
SNR estimationa priori
SNR estimation

multi-channel MMSE STSA estimator
Noise
Spectral gain
Both ears suited spectral gaina priori SNR estimationa posteriori SNR estimation

1514.51413.51312.51530

4560

Speaker’s Direction [deg]Fig. 365432100

Block diagram of proposed method.

Proposed Method
Cepstral Distortion [dB]

LISTFTSTFT

Noise Reduction Rate [dB]

R015304560

Speaker’s Direction [deg]Results of noise reduction rate and cepstral dis-tortion.

4 Experiment and Results
To confirm the eﬀectiveness of the proposed method,
we conducted an experiment. In this experiment, wecompared the noise reduction performance of the conventional and proposed methods by objective evaluations.
We used 20 utterances (10 males and 10 females fromthe Japanese newspaper dictation database) as the target
speech signals and white Gaussian noise signal. Furthermore, the binaural speech signals from five directions, 0,15, 30, 45 or 60 degrees, were obtained by convolutionof the target speech signals and HRTF of each direction.
We used 10 patterns of RTFs (for 10 persons) to confirm that the proposed method achieves a larger amountof noise reduction than the conventional method.
The test data were obtained by combining the binauralspeech signals and noise signals. The input SNR was set
to 0 dB. The weighting factor of the decision-directed
estimation is 0.97 in MMSE STSA estimation. To compare the amount of noise reduction and sound quality, wecalculated the noise reduction rate (NRR) (output SNR- input SNR in dB) and cepstral distortion (CD) (a measure of the degree of spectral envelope distortion) of theprocessed signals.
Figure 3 shows the results for the average NRR andCD of all the target speakers and all RTFs patterns at
each direction. In Fig. 3, we can confirm that NRR ofthe proposed method is superior to that of the conventional method, especially in the case of laterally locatedsources. Also, CDs of both methods are the same.

channels is eﬀective for improvement of noise reduction
performance. The proposed method is advantageous to
the application of binaural hearing-aid systems.

Reference[1] P. Comon, “Independent component analysis, a newconcept?,” Signal Processing, vol.36, pp.287–314,
1994.
[2] R. Okamoto, Y. Takahashi, H. Saruwatari,
K. Shikano, “MMSE STSA estimator with
nonstationary noise estimation based on ICA
for high-quality speech enhancement,” Proc.
ICASSP2010, pp.4778–4781, 2010.
[3] H. Saruwatari, M. Go, R. Okamoto, K. Shikano,“Binaural hearing aid using sound-localizationpreserved MMSE STSA estimator with ICA-basednoise estimation,” Proc. IWAENC, 2010.
[4] Y. Ephraim, D. Malah. “Speech enhancement usinga minimum mean-square error short-time spectral
amplitude estimator,” IEEE Trans. Acoust. Speech
Signal Process., vol.ASSP-32, no.6 pp.1109–1121,
1984.
[5] R. C. Hnedriks, R. Heusdens, U. Kjems, J. Jensen,“On optimal multichannel mean-square error estimators for speech enhancement,” IEEE Signal Processing Letters, vol.16, no.10, pp.885–888, 2009.

5 Conclusions
We presented the Multimodal blind speech extraction
using image tracking in hearing-aids system with somedegrees HRTF filter bank.
From the evaluation, we confirmed that to apply different speech priors for each signals of the left and right

日本音響学会講演論文集- 676 -

2014年3月