2020 RISP International Workshop on Nonlinear Circuits,
Communications and Signal Processing (NCSP 2020)
Honolulu, Hawaii, USA, February 28 - March 2, 2020

Blind Source Separation with Low Latency for In-Car Communication
Tetsuya Ueda, Shota Inoue, Shoji Makino, Mitsuo Matsumoto, Takeshi Yamada
University of Tsukuba
1–1–1 Tennodai, Tsukuba, Ibaraki, Japan
E-mail: {t.ueda@mmlab.cs, s.inoue@mmlab.cs, maki@tara, makilab-research@tara, takeshi@cs}.tsukuba.ac.jp

Abstract
We examined blind source separation (BSS) with low latency
in cars. In frequency-domain BSS, a buffer delay associated with a short-time Fourier transform (STFT) is inevitable.
Thus, we need to shorten the STFT frame length. However,shortening of the STFT frame degrades the source separation
performance because an assumption of instantaneous mixture
model does not hold. On a similar research, the method of
formulating a system for separating highly reverberant mixture signals using a frequency-domain convolutive mixture
model has recently proposed. We paid attention to the method
and considered an application of reducing the delay dependent on the STFT frame. We evaluated the effectiveness ofthe method in cars under a short reverberant condition. Experimental results conﬁrmed that the method not only maintained its performance but also reduced the delay dependent
on the STFT frame.
1. Introduction
Information and communication technology (ICT) is developing rapidly. Recently, several studies of the application of ICT in in-car communication (ICC) have been carried out [1, 2]. Owing to the background noise and seat arrangement, communication within a vehicle is difﬁcult. In
particular, backseat passengers feel uncomfortable listening
to the sound from the speaker located near the driver seat.
The situation can be improved by enhancing particular audio signals and transmitting them without delay. One of thecommon approaches to enhancing particular audio signals is
blind source separation (BSS). BSS is a technique for separating individual source signals from recorded microphone array
inputs without any prior information about source signals.
The most commonly used approach for overdetermined
BSS (where the number of microphones is larger than that ofsources) is independent component analysis (ICA) [3], which
achieves source separation by assuming the statistical independence between the sources. Recently, a number of methods based on frequency-domain ICA have been developed[4, 5, 6], which provide ﬂexibility in utilizing various models
for the time–frequency representations of source signals andarray responses. For example, independent low-rank matrix
analysis (ILRMA) [6] adopts the non-negative matrix factorization (NMF) concept [7, 8] for source spectrogram modeling, which approximates each source power spectrogram asa linear combination of a limited set of spectral templates
scaled by magnitudes varying with time.
In methods based on frequency-domain, there is a delay dependent on the STFT frame. This delay is the time for waiting
for new signals so that the STFT frame buffer is ﬁlled. A highprocessing CPU and an efﬁcient algorithm can not reduce thedelay dependent on the STFT frame. Thus, in low latency
methods based on frequency-domain, we need to shorten the
STFT frame length. However, frequency-domain instantaneous mixture model such as ILRMA has an assumption thatsource separation performance degrades when the reverberation time is longer than the STFT frame length. Generally, thereverberation time in cars is short, but we can not ignore thereverberation problem if we shorten the STFT frame length.
Although we can solve this problem by increasing the STFT
frame length, it is difﬁcult to separate sources with low latency owing to the delay dependent on the STFT frame.
On a similar research, the idea of formulating a system
for separating highly reverberant mixture signals using afrequency-domain convolutive mixture model has recently
been adopted in ILRMA [9]. We refer to this method as
convolutive ILRMA, which has been shown to be effective
for separating sound signals when the reverberation time islonger than the STFT frame length. In this paper, we applied
convolutive ILRMA to low latency BSS in cars, where thereverberation time is shorter than the STFT frame length.
2. Frequency-domain convolutive ILRMA
2.1 Formulation based on frequency-domain instantaneous mixture model
We consider a determined situation where J source signals
are observed by I microphones (J = I). Let xi (f, n) andsj (f, n) denote the STFT coefﬁcients of the signal observedat the ith microphone and the jth source signal, where f andn are the frequency and time indices, respectively.

167

With the frequency-domain instantaneous mixture model,
the relationship between the observed signals x(f, n) =[x1 (f, n), . . . , xI (f, n)]T ∈ CI and source signals s(f, n) =[s1 (f, n), . . . , sI (f, n)]T ∈ CI is written ass(f, n) = W H (f )x(f, n),frame length is not longer than the reverberation time. The relationship between the observed signals x(f, n) and sourcess(f, n) is written as′s(f, n) =(1)

W (f ) = [w1 (f ), . . . , wI (f )] ∈ CI×I ,N
X

W H (f, n′ )x(f, n − n′ ),

(7)

n′ =0

(2)

where W (f, n′ ), 0 ≤ n′ ≤ N ′ are the coefﬁcientwhere W (f ) is the separation matrix and (·) denotes the matrices of size I × I. N ′ is the length of the ﬁlter
Hermitian transpose.
{W H (f, n′ )}f,n′ . W H (f, 0) is equivalent to a separation
Let us assume that sj (f, n) independently follows a matrix of the instantaneous mixture model. When W H (f, 0)
zero-mean complex Gaussian distribution with variance is invertible, the dereverberated mixture signal y(f, n) =vj (f, n) = E[|sj (f, n)|2 ],[yi (f, n), . . . , yI (f, n)]T ∈ CI and the source signal s(f, n)
can be written assj (f, n) ∼ NC (sj (f, n)|0, vj (f, n)).
(3)N′Xy(f, n) = x(f, n) −D H (f, n′ )x(f, n − n′ ), (8)We also assume sj (f, n) to be independent from one source′
n =1
to the others. s(f, n) thus followss(f, n) = W H (f, 0)y(f, n),(9)s(f, n) ∼ NC (s(f, n)|0, V (f, n)),(4)where D H (f, n′ ) = −(W H (f, 0))−1 W H (f, n′ ), 1 ≤where V (f, n) is a diagonal matrix with diagonal entries n′ ≤ N ′ . Equation (8) can be seen as a dereverberav1 (f, n), . . . , vI (f, n). We further assume vj (f, n) as
tion process of the observed mixture signal x(f, n). D ={D H (f, n′ )}f,n′ represents a dereverberation ﬁlter whoseK
X
length is N ′ . Therefore, the negative log-likelihood of invj (f, n) =hj,k (f )uj,k (n),(5) terest is a function of the dereverberation ﬁlter D, separation
k=1
matrices W, and spectral parameters V :H

Hwhere hj,k (f ) ≧ 0 is the (j, k) element of the basis matrix
and uj,k (n) ≧ 0 is the (j, k) element of the activation matrix for the jth source. k = 1, ..., K are the numbers of thebasis. In these assumptions, the negative log-likelihood of
the parameters V = {H, U } with H = {hj,k (f )}j,k,f and
U = {uj,k (n)}j,k,n , and W = {W H (f )}f given the observed mixture signals X = {xi (f, n)}i,f,n is given asc

I(W, V|X ) = −2N log det W (f )X|wjH (f )y(f, n)|2+
log vj (f, n) +, (6)vj (f, n)
H

f,n,j

c

I(D, W, V|X ) = −2N log det W H (f )X|wjH (f )y(f, n)|2+
log vj (f, n) +vj (f, n)
f,n,j

(10)
2.3 Optimization process
We describe the optimization algorithm in this subsection.
The objective function (10) is iteratively decreased using acoordinate descent method in which each iteration comprises
the following three minimization steps:cwhere = denotes equality up to constant terms.
This instantaneous formulation can not shorten the STFT
frame length because source separation performance degrades when the reverberation time is longer than the STFT
frame length.
2.2 Formulation based on frequency-domain convolutive
mixture model
The idea of formulating a system for separating highly reverberant mixture signals using a frequency-domain convolutive mixture model has been proposed [9, 10] and shown to beeffective for separating under the condition where the STFT

168

V̂ ← argmin I(D, W, V|X ),(11)

Ŵ ← argmin I(D, W, V|X ),(12)

D̂ ← argmin I(D, W, V|X ).

(13)

V

W
D

The update rules of minimizing V are written as [8] ,vPuu n |sj (f, n)|2 uj,k (n)vj−2 (f, n)
hj,k (f ) = hj,k (f )t,P−1n uj,k (n)vj (f, n)vPuu f |sj (f, n)|2 hj,k (f )vj−2 (f, n).
uj,k (n) = uj,k (n)tP
−1
f hj,k (f )vj (f, n)

(14)

(15)

The update rules of W can easily be derived on the basisof the IP method [4],separation performance because the assumption of instantaneous mixture model does not hold.
On the similar research in [9], convolutive ILRMA has
wj (f ) ← (WH (f )Σj (f ))−1 ej ,(16) been shown to be effective for separating sound signals under the condition where the reverberation time is longer thanwj (f )
wj (f ) ← q,(17) the STFT frame length. We paid attention to this advantagewH
j (f )Σj (f )w j (f )and considered an application of reducing the delay dependent on the STFT frame. In this paper, we applied convoluP
where Σy/vj (f ) = (1/N ) n y(f, n)y H (f, n)/vj (f, n) tive ILRMA to low latency BSS in cars and investigated theand ej denotes the jth column of the I × I identity matrix.
performance in relation to the delay dependent on the STFT
To obtain the update rules for D, we vectorize D(f, n′ ) as frame and the dereverberation ﬁlter length.
d(f ) = vec({D(f, n′ )})3. Experiments= [dT1 (f, 1), . . . , dTI (f, 1), dT1 (f, 2), . . . , dTI (f, 2),′. . . , dT1 (f, N ′ ), . . . ., dTI (f, N ′ )]T ∈ CI N ,2(18)

where di (f, n′ ) is the ith column of D(f, n′ ). The update
rules for d∗ (f ) are given as∗d (f ) ←X!−1
H

X (f, n)Σw/v(f,n) X(f, n)

n

×

X

!
X H (f, n)Σw/v(f,n) x(f, n) ,To evaluate the effectiveness of convolutive ILRMA applied in cars, we conducted a source separation experiment.
To investigate the improvement in the performance under thecondition where the reverberation time is longer than the
STFT frame length, we compared the performance in relation to the STFT frame length L and the dereverberation ﬁlter
length N ′ . We used speech signals convolved with impulse
responses recorded in a car. We took the average of the signalto-distortion ratios (SDR) as the evaluation criteria [11].

(19)

n

3.1 Experimental conditionsP

wj (f )wjH (f )and (·)∗ represents thejvj (f,n)In this experiment, we used clean data from a total of
10 speakers (6 male speakers and 4 female) and all 503
phoneme-balanced sentences contained in set B of the ATR
T
T
digital speech database. By selecting two different speakers
X(f, n) = [I ⊗ x (f, n − 1), I ⊗ x (f, n − 2), . . . ,from the data set randomly, we obtained 10 patterns of source2′(20) signals. We generated observed signals from the source sigI ⊗ xT (f, n − N ′ )] ∈ CI×I N .
nals by convoluting the impulse response recorded in a car.
Here, I stands for the identity matrix of size I × I and ⊗We resampled the source signals and the observed signals at
stands for the Kronecker product.
8 kHz. We measured the impulse responses using a timeTherefore, the proposed algorithm is summarized as folstretched pulse in a car. The recording environment is shownlows:
in Fig. 1. We set one speaker at the driver seat, anotherspeaker
at a passenger seat, and microphones at a map lamp
1. Initialize V, W, and D.
in the car. The reverberation time T60 in the car was 58 ms. In
2. Repeat the following updates for each j, f , n until con- the experiment, we set L to {2, 4, 8, 16, 32} ms and evaluatedvergence.
each SDR. We set N ′ in the range 0 ≤ N ′ ≤ 10 because ofa trade-off between N ′ and computational time of updating(a) Update hj,k (f ) and uj,k (n) using (14), (15).
(19). N ′ = 0 is equivalent to separation in the instantaneous(b) Update wj (f ) using (16), (17).
method. Moreover, we evaluated SDR at L = 128 ms andN ′ = 0 as baseline performance that satisfy the assumption of(c) Update d∗ (f ) using (19).
instantaneous mixture model. We set the STFT shift length
to one-quarter of the STFT frame length. In [6], it was con2.4 Application of convolutive ILRMA to ICC
ﬁrmed that ILRMA cannot achieve a good performance in
ICC systems must fulﬁll strict delay constraints to ensure the separation of speech signals when the number of bases isthat the ampliﬁed speech is not perceived as a distinguishable large. Thus, we set the number of bases to 1. We ran convoecho. To realize a comfortable situation, mouth-to-ear delays lutive ILRMA for 50 iterations.
should be maintained below 12 ms [2]. Thus, it is necessary
for practical applications to shorten the STFT frame length.
However, shortening of the STFT frame degrades the source
where Σw/v(f,n) =complex conjugate of (·). We write X(f, n) as

169

SDR [dB]

L = 2 ms
L = 4 ms
L = 8 ms
L = 16 ms
L = 32 ms
L = 128 ms (instantaneous)(a) Plan view

dereverberation filter length N' [frame]

Figure 2: average SDR according to STFT frame length L
and dereverberation ﬁlter length N ′(b) Side view

Figure 1: Sound source and microphone layout in experiment
3.2 Results
Figure 2 shows the separation performance at each L. As
the results show, convolutive ILRMA improved SDR at any L
by choosing appropriate N ′ . Moreover, the results at L ≥ 4
ms outperformed the baseline method at L = 128 ms and N ′= 0. Therefore, we conﬁrmed that convolutive ILRMA not
only maintained its performance but also reduced the delay
dependent on the STFT frame.
4. Conclusion
In this paper, we evaluated convolutive ILRMA (updating
the separation matrices, the spectral parameters and the dereverberation ﬁlter iteratively) in cars. We conducted experiments in cars. The result of the experiments conﬁrmed thatthe dereverberation ﬁlter is effective under a short reverberant condition. Therefore, convolutive ILRMA not only maintained its performance but also reduced the delay dependent
on the STFT frame in cars.
Acknowledgment
This work was supported by JSPS KAKENHI Grant Number 19H04131, SECOM Science and Technology Foundation, and Strategic Core Technology Advancement Program(Supporting Industry Program).
References

cation system,” in Proc. Speech Prosody, pp. 479–483,
2016.
[2] A. Theiss et al., “Instrumental evaluation of in-car communication systems,” in Proc. ITG, pp. 1–4, 2014.
[3] A. Hyvärinen et al., “Independent component analysis:
algorithms and applications”, Neural networks, Elsevier, vol. 13, no. 4, pp. 411–430, 2000.
[4] N. Ono, “Stable and fast update rules for independentvector analysis based on auxiliary function technique”,
in Proc. WASPAA, pp. 189–192, 2011.
[5] H. Kameoka et al., “Statistical model of speech signals
based on composite autoregressive system with application to blind source separation”, in Proc. LVA/ICA,
pp. 245–253, 2010.
[6] D. Kitamura et al., “Determined blind source separation with independent low-rank matrix analysis”, Audio
Source Separation, Springer, pp. 125–155, 2018.
[7] D. D. Lee et al., “Algorithms for non-negative matrix
factorization”, in Proc. NIPS, pp. 556–562, 2001.
[8] M. Nakano et al., “Convergence-guaranteed multiplicative algorithms for nonnegative matrix factorization
with β-divergence”, in Proc. MLSP, pp. 283–288, 2010.
[9] H. Kagami et al., “Joint separation and dereverberationof reverberant mixtures with determined multichannel
non-negative matrix factorization”, in Proc. ICASSP,
pp. 31–35, 2018.
[10] T. Yoshioka et al., “Blind separation and dereverberation of speech mixtures by joint optimization”, in IEEE
Transactions on Audio, Speech, and Language Processing, vol. 19, no. 1, pp. 69–84, 2010.
[11] E. Vincent et al., “Performance measurement in blindaudio source separation,” IEEE Trans. ASLP, vol. 14,
no. 4, pp. 1462–1469, 2006.

[1] R. Landgraf et al., “Can you hear me now? reducing thelombard effect in a driving car using an in-car communi-

170