Proceedings of APSIPA Annual Summit and Conference 201918-21 November 2019, Lanzhou, China

Evaluation of Multichannel Hearing Aid System by
Rank-Constrained Spatial Covariance Matrix
Estimation
Masakazu Une∗ , Yuki Kubo† , Norihiro Takamune† , Daichi Kitamura‡ , Hiroshi Saruwatari† and Shoji Makino∗∗ University of Tsukuba, Graduate School of Systems and Information Engineering, Ibaraki, Japan† The University of Tokyo, Graduate School of Information Science and Technology, Tokyo, Japan‡ National Institute of Technology, Kagawa College, Kagawa, Japan

Abstract—In a noisy environment, speech extraction techniques
make hearing aid systems more effective and practical. Blindsource separation (BSS) is suitable for hearing aids because
it can be employed without any a priori spatial information.
Among many BSS methods, independent low-rank matrix analysis (ILRMA) achieves high-quality separation performance. Ina diffuse-noise environment, however, ILRMA cannot suppress
the noise since it is based on the determined situation. On theother hand, rank-constrained spacial covariance matrix (SCM)
estimation overcomes the problem. This method utilizes spatial
parameters accurately estimated by ILRMA and compensates forthe deficiency of the spatial basis of diffuse noise. The applicationof BSS methods to a multichannel binaural hearing aid systemwith a smartphone has never been studied in detail thus far. To
clarify the efficacy of the BSS methods in real environments, werecord real sounds by constructing a hearing aid system with adummy head and a smartphone. In this study, we investigate
the applicability of BSS for a multichannel binaural hearing
aid system with microphones on a smartphone. Furthermore,
we apply ILRMA and the rank-constrained SCM estimation to
the recorded data and evaluate these methods in terms of theirseparation performance.

I. I NTRODUCTION
When we use a binaural hearing aid in a noisy environment,
target-speech extraction is necessary since speech is always
contaminated by noise. In binaural hearing aid systems, blindsource separation (BSS) [1] is suitable because it works
well without spatial information, e.g., microphone positions
around both ears, the head-related acoustic condition, thetarget-speaker location (direction), and room reverberation.
Many BSS methods, such as frequency domain independent
component analysis [2], [3], independent vector analysis [4]–[6], and independent low-rank matrix analysis (ILRMA) [7]–[9], have been proposed . In particular, ILRMA achieves
effective and accurate separation by introducing nonnegative
matrix factorization (NMF) [10] to the source model. However,these methods can be applied to only the determined or overdetermined situations (the number of microphones ≥ the numberof sources), and their applicability is not realized when these
conditions are not satisfied (i.e., underdetermined case). On theother hand, rank-constrained spacial covariance matrix (SCM)
estimation [11] has been proposed as an effective method fora situation in which noise arrives from all directions, i.e.,
diffuse-noise case. Basically, this method estimates a full-978-988-14768-7-6©2019 APSIPA

rank SCM [12], which represents spatial characteristics of the
diffuse noise, just as multichannel NMF (MNMF) [13], [14]does. However, MNMF requires the estimation of an enormous
number of parameters, leading to a high computational cost.
In contrast, the rank-constrained SCM estimation reduces thenumber of parameters by using the highly accurate spatial parameters obtained by ILRMA and restores the lost spatial basis
for diffuse noise. It was reported that the rank-constrained
SCM estimation achieves a more efficient and stable extractionof target speech than MNMF under the simplified computersimulation-based acoustic condition, but the evaluation in realsituations remains a problem.
In this paper, first, we propose a new multichannel hearing
aid system composed of a distributed microphone array including binaural ear-attached microphones and smartphone microphones. We utilize eight microphones that are synchronizedwith the same sampling rate. Although all the microphone
positions are not specified in advance owing to variationsin user head sizes and smartphone location, BSS is fully
applicable to the distributed configuration. To the best of ourknowledge, there has been no study on BSS applied to sucha system. Second, we implement the rank-constrained SCM
estimation as a speech extraction algorithm in this hearing
aid system. We prepare a head-and-torso dummy with asmartphone to imitate a person (hearing aid user) holding asmartphone, and record the real diffuse noise at the multiple
microphones. On the basis of the results, we evaluate thespeech extraction performance of the rank-constrained SCM
estimation. The experimental results show that the proposed
method outperforms the conventional ILRMA in a realistic
situation.
II. F ORMULATION AND BSS A LGORITHMS
A. Formulation
Let us consider separating the observed signals, which
are obtained by M microphones capturing the signals arriving from N sources. The source, observed, and separated
signals in each time-frequency slot are denoted as sij =(sij,1 , . . . , sij,N )> ∈ CN , xij = (xij,1 , . . . , xij,M )> ∈ CM ,and y ij = (yij,1 , . . . , yij,N )> ∈ CN , where i = 1, . . . , I,
j = 1, . . . , J, and n = 1, . . . , N indicate the indexes ofthe frequency bins, time frames, and sources, respectively.

1874

APSIPA ASC 2019

Proceedings of APSIPA Annual Summit and Conference 201918-21 November 2019, Lanzhou, China

The operator ·> indicates transpose. When each source is adirectional target source and the window length of short-time
Fourier transform (STFT) is sufficiently larger than that of
the impulse response of the object space, the observed signal
and the mixing matrix Ai = (ai,1 · · · ai,N ) ∈ CM ×N in eachfrequency bin have the relationnoise SCM lacks a rank corresponding to the target source
direction, which results in the rank-(M − 1) SCM, rankconstrained SCM estimation is used to estimate the parameters
to compensate for the lack of the rank. Finally, multichannel
Wiener filtering is applied to suppress the noise diffusing
toward the target source. The overview of the algorithm is
described below.
The rank-constrained SCM estimation assumes the observed
signal xij as the sum of the source image vector hij =(hij,1 , . . . , hij,M )> and the diffuse noise image vector uij =(uij,1 , . . . , uij,M )> ; i.e.,

xij = Ai sij ,(1)

where ai,n is the steering vector for each source. If the numberof microphones is equal to that of sources (M = N ) and
Ai is not a singular matrix, the separated signal y ij can beobtained by estimating the demixing matrix W i = A−1=i(wi,1 · · · wi,N )H ∈ CN ×M as
y ij = W i xij ,(2)

where the operator ·H denotes the Hermitian transpose.
B. ILRMA [7]
In ILRMA, the component of the nth source in each timefrequency slot is assumed to be generated from a statisticalmodel that follows the univariate complex Gaussian distribution as!
X
sij,n ∼ Nc 0,
til,n vlj,n ,(3)lwhere til,n ≥ 0 and vlj,n ≥ 0 are NMF variables, l = 1, . . . , L
is an index of the NMF basis, and L is the number ofbases. Simultaneously, the observed signal xij follows themultivariate complex Gaussian distribution because of the
reproductive property, i.e.,!
XHxij ∼ Nc 0,
rij,n ai,n ai,n ,(4)nrij,n =

X

til,n vlj,n ,(5)lwhere rij,n corresponds to the nth source model that approximates the power spectrogram using nonnegative values til,nand vlj,n . The steering vector ai,n corresponds to the spatialmodel as the rank-1 SCM constructed by the spatial basis
for the nth source. These NMF variables til,n , vlj,n and the
demixing matrix W i are obtained by maximum-likelihood estimation based on the maximization of statistical independence
between the sources.

xij = hij + uij .

The source image vector hij is expressed using a vector(h)corresponding to the target source, ai =: ai,nh out of thespatial bases ai,1 , . . . , ai,N obtained by ILRMA, and the target(h)source image sij as follows:(h) (h)hij = ai sij ,


(h)
(h)
sij ∼ Nc 0, rij ,

(7)
(8)

where nh indicates the index corresponding to the target(h)source and rij is the variance of the target source (power
spectrogram).
(h)
The variance of the target source rij is assumed to havesparsity and the prior distribution follows an inverse gamma
distribution given by!
ββ α  (h) −α−1(h)r
exp − (h) , (9)p(rij ; α, β) =Γ(α) ijr
ijwhere α > 0, β > 0, and Γ(·) are the shape parameter, scaleparameter, and gamma function, respectively. On thie other
hand, the diffuse noise uij follows the following multivariate
complex Gaussian distribution and is statistically independentof the target source hij :(u) (u)uij ∼ Nc 0, rij Ri
,
(10)
(u)

(u)

where rij and Ri are the variance and the full-rank
SCM of the diffuse noise, respectively. Here, N separated
signals ŷ ij,1 , . . . , ŷ ij,N are obtained by ILRMA, and the SCM(u)of the diffuse noise Ri is represented as
(u)

(u)

= R0 i + λi bi bHi,
X (u)  (u) H1(u)R0 i =ŷŷ ij,
J j ijX(u)ŷ ij =ŷ ij,n ,Ri

C. Rank-Constrained SCM Estimation [11]The rank-constrained SCM estimation focuses on a situation
where one directional target source and diffuse noise are
mixed. As the strategy of this method, the full-rank SCM
is estimated using the spatial basis of the one directionaltarget source and the noise SCM is estimated by ILRMA.
First, we apply ILRMA to xij and obtain one a ”noisecontaminated target speech” component and M − 1 ”noiseonly” components (see [15] for the physical mechanism of
this phenomenon). Second, we calculate the noise SCM using
the above-mentioned components. Since the ILRMA-estimated

(6)

(11)
(12)
(13)

n6=nh
(u)

where R0 i is the noise SCM estimated by ILRMA whose
rank is M − 1, bi is a unit eigenvector corresponding to zero(u)eigenvalue of R0 i and λi is the weight variable. Note that,(u)(u)
in Ri , only λi is the variable to be optimized because R0 i
and bi are given by ILRMA as fixed values in advance. By

1875

Proceedings of APSIPA Annual Summit and Conference 201918-21 November 2019, Lanzhou, China
TABLE IR ECORDING CONDITIONS

modeling the prior distribution of the target source variance in(9), we express the negative log likelihood function L of therank-constrained SCM estimation as"X(h) (u)(x) −1(x)
L(rij , rij , λi ) =xH
xij + log det Rijij (Rij )Recording location
Reverberation time (T60 )Microphone
LoudspeakerMicrophone preamplifier
Audio interface
TSP length
Recording sampling freq.

i,j(h)+ (α + 1) log rij +#β+ const.,

(h)

rij

Studio300 msC417 PP (AKG)
ADIVA11 (Anthony Gallo)
Octamic II (RME)
828x (MOTU)
65536 samples
48 kHz(14)
where const. does not depend on the objective variables.
The parameters of this negative log likelihood function L are
optimized by a maximum a posteriori estimation based on the
expectation–maximization (EM) algorithm [11].
The Q function is defined by the expected completedata log-likelihood values regarding the a posteriori proba(h)bility p(sij , uij | xij ; Θ̃) as
"
X
(h)
(u)
(u)
Q(Θ; Θ̃) =−(α + 2) log rij − M log rij − log det Ri
i,j

tr

(h)

r̂ij + β

−

(h)



(u)
Ri

−

−1

(u)
R̂ij


#

(u)

rij

+ const.,

rij

(15)
(h)

(u)

where Θ = {rij , rij , λi } is set of the set of parameters to(h) (u)be updated, Θ̃ = {r̃ij , r̃ij , λ̃i } is the up-to-date parameters,
(u)

(h)

and r̂ij and R̂i are the sufficient statistics obtained by theE-step. The update rules in the E-step are expressed as
(u)

R̃i

(u)

=R0 i

+ λ̃i bi bH
i ,

(16)

(x)
(h) (h)(h)(u) (u)Rij =r̃ij ai (ai )H + r̃ij R̃i ,

2 
H 
−1
(h)
(h)
(h)
(h)
(x)
(h)
r̂ij =r̃ij − r̃ij
ai
Rij
ai

(17)

2

−1
(h)
(x)
(h)
,
(18)
+ r̃ij xHai
ij Rij−1 (u)2 (u) 
(u)
(x)
(u)
(u) (u)R̃i
R̃i
Rij
R̂ij =r̃ij R̃i − r̃ij
2 (u) −1
−1 (u)(u)(x)(x)+ r̃ijR̃i
Rij
xij xH
R
R̃i .
ij
ij
(19)

In the M-step, a coordinate ascent algorithm is applied to the
Q function, i.e.,(h)r̂ij + β
(h)
rij ←
,

Ki =

α+2
1X 1

J

j

(u)
r̃ij

(20)
(u)

R̂ij ,

λ i = bHi Ki bi ,(u)(u)
Ri ← R0 i + λi bi bHi ,−1 (u) 
1
(u)
(u)
rij ←
tr Ri
R̂ij .

M

Fig. 1. (a) Overall view of head-and-torso dummy, (b) right-ear microphonearray, (c) smartphone microphones, and (d) left-ear microphone array.

(21)
(22)
(23)
(24)

After the convergence of the EM algorithm, we can construct(h)the multichannel Wiener filter [11] using the estimated rij ,(u)(u)
rij , and Ri with λi , and output the extracted target speechcomponent.

1876

III. M ULTICHANNEL H EARING A ID S YSTEM
A. Specification of Apparatus
In this experimental scheme we assume the situation wherea person wearing a multichannel hearing aid talks with someone facing him/her. We constructed the recording system andrecorded impulse responses and diffuse noise using eightmicrophones that are synchronized with the same samplingrate. Figure 1(a) shows the head-and-torso dummy, which
imitates a person wearing a binaural hearing aid and holding asmartphone. Three microphones are attached to each ear [seeFigs. 1(b) and (d)]. The smartphone is attached 20 cm apartfrom the chest and two microphones with the interelement
spacing of 4 cm are set on the smartphone [see Fig. 1(c)].
For convenience, we number each microphone as shown in
Figs. 1(b)–(d). The height of the dummy head is set to 170 cm
and the loudspeaker is set in front of the dummy head tomimic the situation of conversation. Accordingly, the heightof the loudspeaker is set at 152 cm, corresponding to the mouth
position of the conversation partner whose head is of the sameheight as the dummy head.

150 cm100 cm75 cm- 20

IV. E XPERIMENTAL E VALUATION
A. Experimental Conditions
The purpose of this experiment is to evaluate the applicability of ILRMA and the rank-constrained SCM estimation for a
multichannel hearing aid system in a real environment. Femaleutterance was convolved with the impulse response to produce
the target speech. We used the utterances from the JNAS
database [18] and the recorded impulse response describedin Sec. III-B. Since the sampling rate of the corpus is 16 kHz,the impulse response and diffuse noise were down-sampledfrom 48 to 16 kHz. The observed signal was generated bymixing the recorded diffuse noise and the target signal at
the input SNRs of −10, −5, and 0 dB. In ILRMA, theobserved signal was preprocessed by sphering transformationby principal component analysis. In the rank-constrained SCM+ 20

Dummy headFig. 2. Position of loudspeaker (mouth of conversation partner) for ninerecordings.
TABLE IIE XPERIMENTAL CONDITIONS FOR BSS
Sampling freq.
FFT length

B. Recording of Impulse Response and Diffuse Noise
The time stretched pulse (TSP) signal is adopted in themeasurement. The conditions are shown in Table I. Eightmicrophones are synchronized by the audio interface in thisresearch. When the proposed hearing aid system is used in an
actual situation, it is necessary to apply the following methods
to synchronize these microphones [16], [17]. The distancefrom the dummy head to the loudspeaker is varied by 75,
100, and 150 cm, and the angle is varied by −20, 0, and 20◦ ,where 0◦ means the normal to the dummy head. The overview
of the positions in nine recordings is shown in Fig. 2.
The diffuse noise is supposed to be a sound in a crowded
place, where many people talk and walk freely. We instructed
the volunteer subjects to walk around the dummy head andread the designated sentences aloud. Assuming that the noise
sources surround the target source, we instructed them to walkoutside the 150 cm radius of the front half circle of the dummyhead.

150 cmWindow
Number of bases
in low-rank model
Number of iterations
in ILRMA
Initialization of W i in ILRMA
Number of iterations
in rank-constrained
SCM estimation

16 kHz1024 sample(50% overlap)
Hamming window
1050
Identity matrix
10

estimation, the shape parameter α was experimentally chosen
and set to 0.5, 1.1, 10, and 20, and the scale parameter β
was set to 10−16 . Initialization trials were conducted tentimes using different random values. The other conditions areshown in Table II. We used source-to-distortion ratio (SDR)
improvement [19] as the objective measurement citation.
B. SDR Improvement Behavior
Figure 3 shows that the average SDR tended to improvefor every iteration under the −10 dB input SNR condition at
microphone 1 (nearest right external auditory canal). From
the figure, we can confirm that the rank-constrained SCM
estimation outperforms ILRMA in all recordings. The valueof the hyperparameter α greatly affects the SDR improvement
and it is better to set α large. Moreover, the rank-constrained
SCM estimation can achieve the highest SDR improvementby the second or third iteration, showing the advantage of thefast convergence.
C. SDR Improvement at Various Input SNRs
We concentrate our attention on the target speech angle of0◦ and investigate SDR improvements by ILRMA and rankconstrained SCM estimation in each input SNR. Note thatthe second iteration score of the the rank-constrained SCM
estimation was adopted on the basis of the results in Sec. IV-B.

1877

Proceedings of APSIPA Annual Summit and Conference 201918-21 November 2019, Lanzhou, China

Fig. 3. Average SDR improvements for each iteration at microphone 1 under−10 dB input SNR condition.

Figure 4 shows the average SDR improvement at microphone 1 in each input SNR. From Fig. 4, ILRMA achieves acceptable separation performance compared with the observed
signal. The SDR improvements by the rank-constrained SCM
estimation are higher than those by ILRMA especially at
lower input SNRs (i.e., −10 and −5 dB). Furthermore, we
can confirm that the sparse prior corresponding to a larger α
improves the separation performance in all the recordings.
V. C ONCLUSIONSIn this study, we investigate the applicability of ILRMA
and the rank-constrained SCM estimation with a binauralhearing aid system including microphones on a smartphone
in a real environment. We construct the experimental system
to record the impulse response and diffuse noise. Using therecorded data, we evaluate the separation performance of
ILRMA and the rank-constrained SCM estimation . The experimental results show that ILRMA and the rank-constrained
SCM estimation work well for a the binaural hearing aid
scheme. Furthermore, the separation performance of the rankconstrained SCM estimation is higher than that of ILRMA
especially under low-SNR conditions.
ACKNOWLEDGMENT
This work was partly supported by SECOM Science andTechnology Foundation and JSPS KAKENHI Grant Numbers
JP19H01116 and 19K20306.

Fig. 4. Average SDR improvements of ILRMA and rank-constrained SCM
estimation after two iterations at microphone 1 when target source is locatedat 0◦ .

R EFERENCES[1] H. Sawada, N. Ono, H. Kameoka, D. Kitamura, and H. Saruwatari, “A
review of blind source separation methods: two converging routes to
ILRMA originating from ICA and NMF,” APSIPA Trans. Signal andInformation Processing, vol. 8, no. e12, pp. 1–14, 2019.
[2] P. Smaragdis, “Blind separation of convolved mixtures in the frequency
domain,” Neurocomputing, vol. 22, no. 1, pp. 21–34, 1998.
[3] H. Saruwatari, T. Kawamura, T. Nishikawa, A. Lee, and K. Shikano,“Blind source separation based on a fast-convergence algorithm combining ICA and beamforming,” IEEE Trans. ASLP, vol. 14, no. 2, pp.
666–678, 2006.
[4] A. Hiroe, “Solution of permutation problem in frequency domain ICA
using multivariate probability density functions,” in Proc. ICA, 2006,
pp. 601–608.
[5] T. Kim, H. T.Attias, S. Y. Lee, and T. W. Lee, “Blind source separation
exploiting higher order frequency dependencies,” IEEE Trans. ASLP,
vol. 15, no. 1, pp. 70–79, 2007.
[6] N. Ono, “Stable and fast update rules for independent vector analysis
based on auxiliary function technique,” Proc. IEEE Workshop on
Applications of Signal Processing to Audio and Acoustics (WASPAA),
pp. 189–192, 2011.
[7] D. Kitamura, N. Ono, H. Sawada, H. Kameoka, and H. Saruwatari, “Determined blind source separation unifying independent vector analysis
and nonnegative matrix factorization,” IEEE/ACM Trans. ASLP, vol. 24,
no. 9, pp. 1626–1641, 2016.
[8] D. Kitamura, N. Ono, H. Sawada, H. Kameoka, and H. Saruwatari,“Determined blind source separation with independent low-rank matrix
analysis and nonnegative matrix factorization,” in Audio Source Separation, S. Makino, Ed., Springer, pp. 125–155, 2018.
[9] D. Kitamura, S. Mogami, Y. Mitsui, N. Takamune, H. Saruwatari,
N. Ono, Y. Takahashi, and K. Kondo, “Generalized independent lowrank matrix analysis using heavy-tailed distributions for blind source

1878

Proceedings of APSIPA Annual Summit and Conference 201918-21 November 2019, Lanzhou, China

separation,” EURASIP Journal on Advances in Signal Processing, vol.
2018, no. 28, pp. 1–25, 2018.
[10] D. D. Lee and H. S. Seung, “Learning the parts of objects by nonnegative matrix factorization,” Nature, vol. 401, no. 6755, pp. 788–791,
1999.
[11] Y. Kubo, N. Takamune, D. Kitamura, and H. Saruwatari, “Efficient fullrank spatial covariance estimation using independent low-rank matrix
analysis for blind source separation,” in Proc. EUSIPCO, 2019.
[12] N. Q. K. Duong, E. Vincent, and R. Gribonval, “Underdetermined
reverberant audio source separation using a full-rank spatial covariancemodel,” IEEE Trans. ASLP, vol. 18, no. 7, pp. 1830–1840, 2010.
[13] A. Ozerov and C. Févotte, “Multichannel nonnegative matrix factorization in convolutive mixtures for audio source separation,” IEEE Trans.
ASLP, vol. 18, no. 3, pp. 550–563, 2010.
[14] H. Sawada, H. Kameoka, S. Araki, and N. Ueda, “Multichannel
extensions of non-negative matrix factorization with complex valued
data,” IEEE Trans. ASLP, vol. 21, no. 5, pp. 971–982, 2013.
[15] Y. Takahashi, T. Takatani, K. Osako, H. Saruwatari, and K. Shikano,“Blind spatial subtraction array for speech enhancement in noisy environment,” IEEE Trans. ASLP, vol. 17, no. 4, pp. 650–664, 2009.
[16] S. Miyabe, N. Ono, and S. Makino, “Blind compensation of interchannel
sampling frequency mismatch for ad hoc microphone array based on
maximum likelihood estimation,” Elsevier Signal Processing, vol. 107,
no. 2015, pp. 185–196, 2015.
[17] R. Sakanashi, N. Ono, S. Miyabe, T. Yamada, and S Makino, “Speechenhancement with ad-hoc microphone array using single source activity,”in Proc. APSIPA2013, 2013, pp. 1–6.
[18] K. Itou, M. Yamamoto, K. Takeda, T. Takezawa, T. Matsuoka,
T. Kobayashi, K. Shikano, and S. Itahashi, “JNAS: Japanese speech
corpus for large vocabulary continuous speech recognition research,”
The Journal of Acoustical Society of Japan (E), vol. 20, no. 3, pp. 199–
206, 1999.
[19] E. Vincent, R. Gribonval, and C. Févotte, “Performance measurementin blind audio source separation,” IEEE Trans. ASLP, vol. 14, no. 4,
pp. 1462–1469, 2006.

1879