#2021 The Acoustical Society of Japan

Acoust. Sci. & Tech. 42, 4 (2021)

Interactive speech source separation based on independent low-rank matrix analysis
Fuga Oshima1 , Masaki Nakano2 and Daichi Kitamura1; 
12National Institute of Technology, Kagawa College, Chokushi 355, Takamatsu, 761–8058 Japan
University of Tsukuba, 1–1–1 Tennodai, Tsukuba, 305–8577 Japan(Received 25 February 2021, Accepted for publication 16 March 2021)
Keywords: Blind source separation, Independent component analysis, Nonnegative matrix factorization, User annotation
1.

Introduction
Blind source separation (BSS) [1] is a technique toestimate source signals in an observed mixture signal without
knowing any prior information. When the number of sources
is equal to or greater than the number of sensors (microphones
in audio cases), which is called the (over-)determinedsituation, independent component analysis (ICA) [2] and its
extensions have been studied. Since audio sources areconvolutively mixed with room reverberation, frequencydomain ICA (FDICA) [3] is a basic algorithm for audio
BSS.
In FDICA, scales and permutation of frequency-wise
estimated signals cannot be determined, and post-processes
are required to ﬁx these ambiguities. The scale ambiguity caneasily be recovered by the back projection [4], whereas the
permutation alignment of the estimated signals for all thefrequencies (so-called the permutation problem [5]) has beentackled for many years. In particular, independent vector
analysis (IVA) [6,7] and independent low-rank matrix analysis (ILRMA) [8,9] are the most successful approaches forsolving the permutation problem. These methods assume
speciﬁc time-frequency structures of sources (source models)to avoid encountering the permutation problem during the
optimization.
Although ILRMA accurately separates mixtures of musicsources, the performance of IVA and ILRMA often degrades
for speech mixtures. The main reason of the degradation in
IVA and ILRMA is a block permutation problem [10,11],
namely, the permutation misalignment of the estimatedsignals occurs with subband blocks (see Fig. 1). This iscaused by a mismatch between the source model assumed in
IVA or ILRMA and actual time-frequency structure of speechsources.
In this letter, we propose to solve the block permutation
problem based on user annotations and develop a newinteractive speech source separation system. In the proposedsystem, ILRMA-based source separation is performed while
the user annotation is utilized for escaping from bad local
minima in ILRMA optimization and searching for moreaccurate separation results.

respectively. The time-frequency components of source,
mixture, and estimated signals are respectively deﬁned assi; j ¼ ½si; j;1 ;    ; si; j;n ;    ; si; j;N T 2 CN ;ð1Þxi; j ¼ ½xi; j;1 ;    ; xi; j;m ;    ; xi; j;M T 2 CM ;ð2Þyi; j ¼ ½yi; j;1 ;    ; yi; j;n ;    ; yi; j;N  2 C ;ð3ÞT

where i ¼ 1; 2;    ; I, j ¼ 1; 2;    ; J, n ¼ 1; 2;    ; N, andm ¼ 1; 2;    ; M are the indices of frequency, time, source,
and microphone, respectively, and T denotes the transpose.
We assume that the observed signal is represented byxi; j ¼ Ai si; j ;BSS based on ILRMA
Let N and M be the numbers of sources and microphones,

ð4Þ

where Ai 2 CMN is a frequency-wise mixing matrix. Hereafter, we consider the determined situation (M ¼ N). If W i ¼
A1i exists for all the frequencies, the estimated signal can beobtained byyi; j ¼ W i xi; j ;ð5Þwhere W i ¼ ½wi;1 wi;2    wi;N H is called the demixing matrixand H denotes the Hermitian transpose.
ILRMA is an extension of IVA: the low-rank source
model based on nonnegative matrix factorization (NMF) [12]is introduced. ILRMA optimizes both the demixing matrix W iand the NMF source model T n V n , where T n 2 RIL
0 and V n 2
RLJ0 are the basis and activation matrices in NMF.
In ILRMA, the following cost function is minimized:
X
J ¼ 2J
log jdet W i j
i

"
#
2
X jwH
X
i;n xi; j j
P
þ
þ log
ti;l;n vl; j;n ;l ti;l;n vl; j;ni; j;nlð6Þwhere ti;l;n and vl; j;n are the elements of T n and V n ,respectively, and l ¼ 1; 2;    ; L is the index of NMF bases.
The minimization of (6) w.r.t. W i is performed via iterative
projection [7] as
1X
1PU i;n ¼xi; j xi;Hj ;ð7ÞJ jl ti;l;n vl; j;n
wi;n

2.

N

wi;n

ðW i U i;n Þ1 en ;

ð8Þ
1
i;n 2

wi;n ðwH
i;n U i;n w Þ ;ð9Þ

where en 2 RNf0;1g is a unit vector whose nth element is unity.
e-mail: kitamura-d@t.kagawa-nct.ac.jp[doi:10.1250/ast.42.222]

222

Also, the minimization of (6) w.r.t. T n and V n is performed by
iterating the following update rules:

F. OSHIMA et al.: INTERACTIVE SPEECH SEPARATION BASED ON ILRMA

Fig. 1 Example of estimated signals with block permutation problem. Components over 4 kHz are swapped
as subband block because of permutation misalignment.

ti;l;nvl; j;nvﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃ
P

uP H
u j jwi;n xi; j j2 l0 ti;l0 ;n vl0 ; j;n 2 vl; j;nt; ð10Þ
ti;l;nP
1P
j vl; j;nl0 ti;l0 ;n vl0 ; j;nvﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃ
P

uP H
u i jwi;n xi; j j2 l0 ti;l0 ;n vl0 ; j;n 2 ti;l;nt: ð11Þ
vl; j;nP
1Pi ti;l;nl0 ti;l0 ;n vl0 ; j;nThus, all the variables can be optimized by iterating (7)–(11)with appropriate initialization, and the estimated signal is
obtained by (5). However, since the cost function (6) is nonconvex, there exist many local minima in this optimization:some of them provide poor separation performance because of
the block permutation problem.
3. Interactive speech source separation
3.1. Motivation and system overview
BSS of IVA and ILRMA often encounters the block
permutation problem [10,11]. This is because the parameter
optimization gets stuck in a bad local minimum, which
confuses the permutation-misaligned estimated signals withthe source signals. However, as shown in Fig. 1, the boundary
frequency of the block permutation problem is often visually
recognizable by users. On the basis of this idea, we propose tosolve the block permutation problem using user annotations
and develop a new interactive speech source separation
system.
Figure 2 shows the user interface of the proposed system,
where the system is implemented to work on web browsers.
The system returns spectrograms of the estimated signals
obtained by ILRMA to the user as a temporary result. The
user can listen these estimated signals by clicking play
buttons. When the user noticed that the estimated signals
contain a permutation-misaligned subband or the separation
performance is not satisfactory, a region-based annotation(see the red rectangle in Fig. 2) can be fed back to the system.
The detailed explanations of this annotation and processing
after the feedback are presented in the following subsections.
Then, ILRMA re-estimates the demixing matrix using the user
annotation to output permutation-ﬁxed and more accurate

Fig. 2 User interface of proposed interactive speechsource separation system in two-source case. Spectrograms are temporarily estimated signals obtained by
ILRMA, where block permutation problem is occurredin annotated frequency subband.

estimated signals. This interaction between ILRMA and theuser can be iterated until the satisfactory separation isachieved.
3.2. Frequency annotation with block permutation problem
When the user noticed that the estimated signals containa permutation-misaligned subband, frequency annotation
should be fed back to the system. Let i ¼ is and i ¼ ie
(1  is < ie  I) be the frequency indices of the lowest
and the highest frequencies in the permutation-misaligned
subband, respectively. Also, let n ¼ ns and n ¼ nt be thesource and target indices of the signals, respectively, i.e.,
yis ; j;ns ; yis þ1; j;ns ;    ; yie ; j;ns and yis ; j;nt ; yis þ1; j;nt ;    ; yie ; j;nt are
mistakenly swapped. These indices are obtained from thefrequency annotation, which is a frequency range of theselected region, by the user. In this case, since we need toswap the demixing ﬁlters wi;n and the basis components ti;k;n ,the following process is performed:
wis ;ns ; wis þ1;ns ;    ; wie ;ns, wis ;nt ; wis þ1;nt ;    ; wie ;nt ;ð12Þtis ;k;ns ; tis þ1;k;ns ;    ; tie ;k;ns, tis ;k;nt ; tis þ1;k;nt ;    ; tie ;k;nt 8k;

ð13Þ

where , denotes the swapping process between the eachcomponent of left-hand and right-hand sides. In addition, theactivation matrices of the corresponding sources are reset as223

Acoust. Sci. & Tech. 42, 4 (2021)
;  8k; j;

ð14Þ

where
denotes the substitution of each component and 
is a random value that obeys the uniform distribution in therange ð0; 1Þ. The other parameters are inherited from thepreviously performed ILRMA. Thus, we expect that the
optimization parameters can escape from the bad local
minimum with block permutation problem by continuing the
ILRMA algorithm after applying (12)–(14).
3.3. Time annotation with silent segment of the othersources (I)When the user does not satisﬁed with the quality ofseparation, a time annotation is useful to improve theperformance. In this case, the user annotates a silent time
segment of the other sources, namely, the silent segment
must contain only one source components. This annotation isreasonable because most of time of a typical speech mixture(conversation) contains only one speaker, which can easily
be annotated by listening the observed or the temporarily
estimated signal. Similar approach was proposed with IVA[13], but, to the best of our knowledge, ILRMA-based BSS
using the time annotation has not been proposed yet.
Let j ¼ js and j ¼ je (1  js < je  J) be the time indicesof the beginning and the ending time frames in the silent
segment, respectively. Also, let n ¼ nt be the active source inthe silent segment, i.e., the sources n 6¼ nt are inactive. These
indices are obtained from the time annotation, which is a time
range of the selected region, by the user. In this case, since
we need to suppress the time-frequency variance T n V n of thesilent sources, the following process is performed:
vk; js ;n ; vk; js þ1;n ;    ; vk; je ;nwi;n"; ";    ; " 8k; n 6¼ nt ;ð15Þ

½; ;    ; T 8i; n;ð16Þ

where " is a machine epsilon. Similarly to Sect. 3.2, the otherparameters are inherited from the previously performed
ILRMA.
3.4. Time annotation with silent segment of the othersources (II)The process (15) in Sect. 3.3 substitutes " to thecorresponding elements in V n (n 6¼ nt ). In addition to (15),we also propose to reset the other components in V n as
follows:
vk;1;n ; vk;2;n ;    ; vk; js 1;n
; ;    ;  8k; n 6¼ nt ;ð17Þvk; je þ1;n ; vk; je þ2;n ;    ; vk;J;nvk; j;nt

; ;    ;  8k; n 6¼ nt ;ð18Þ 8k; j;ð19Þ

where  is a random value that obeys the uniform distribution
in the range ½1:0  105 ; 1:1  105 , which is suﬃciently larger
than ". Compared with the processing (15) and (16) (hereafterreferred to as time annotation (I)), the processing (15)–(19) (hereafter referred to as time annotation (II))
provides more drastic change to the ILRMA optimization,
resulting in more eﬀective induction to escape from the bad
local minimum.

224

Table 1 Sources obtained from SiSEC2011 dataset[14].
MixtureSource signals
dev1 female3 synthconv 130ms 5cm sim 1

No. 1

dev1 female3 synthconv 130ms 5cm sim 2
dev1 male3 synthconv 130ms 5cm sim 1

No. 2

dev1 male3 synthconv 130ms 5cm sim 2
dev1 male3 synthconv 130ms 5cm sim 1

No. 3

dev1 female3 synthconv 130ms 5cm sim 2䣅䣱䣰䣸䣧䣰䣶䣫䣱䣰䣣䣮䢢䣵䣫䣯䣲䣮䣧䢢䣋䣎䣔䣏䣃䣋䣎䣔䣏䣃䢢䣹䣫䣶䣪䢢䣶䣫䣯䣧䢢䣣䣰䣰䣱䣶䣣䣶䣫䣱䣰䢢䢪䣋䢫䢳䢴䣕䣆䣔䢢䣫䣯䣲䣴䣱䣸䣧䣯䣧䣰䣶䢢䣝䣦䣄䣟vk; j;ns ; vk; j;nt䢪䣣䢫䣋䣎䣔䣏䣃䢢䣹䣫䣶䣪䢢䣨䣴䣧䣳䣷䣧䣰䣥䣻䢢䣣䣰䣰䣱䣶䣣䣶䣫䣱䣰䣋䣎䣔䣏䣃䢢䣹䣫䣶䣪䢢䣶䣫䣯䣧䢢䣣䣰䣰䣱䣶䣣䣶䣫䣱䣰䢢䢪䣋䣋䢫

䢪䣤䢫

䢪䣥䢫

䢳䢲
䢺
䢸
䢶
䢴
䢲
䢯䢴
䢯䢶

䢲

䢶䢲

䢺䢲

䢳䢴䢲

䣋䣶䣧䣴䣣䣶䣫䣱䣰

䢳䢸䢲 䢲

䢶䢲

䢺䢲

䢳䢴䢲

䣋䣶䣧䣴䣣䣶䣫䣱䣰

䢳䢸䢲 䢲

䢶䢲

䢺䢲

䢳䢴䢲

䣋䣶䣧䣴䣣䣶䣫䣱䣰䢳䢸䢲

Fig. 3 SDR improvements of the proposed system withfrequency or time annotation for mixtures (a) no. 1,(b) no. 2, and (c) no. 3.

4. Experiments
4.1. Conditions
We evaluated the performance of conventional simple
ILRMA and the proposed interactive speech source separation
system. In this experiment, we produced three mixture signals(nos. 1–3) as listed in Table 1, where the speech sources wereobtained from SiSEC2011 dataset (see [14] for the detailed
conditions). In short-time Fourier transform, we used 128ms-long hamming window with half-overlap shifting. The
number of bases in ILRMA was set to L ¼ 3, and theimprovement of source-to-distortion ratio (SDR) [15] wascalculated.
In this experiment, the conventional ILRMA estimates yi; jwith 160 iterations (parameter updates). The proposed system
returns temporarily estimated results to the user after 80
iterations of ILRMA and requires the frequency or time
annotation at that time. After the annotation feedback and theprocessing described in Sect. 3, the proposed system continues ILRMA optimization with 80 iterations. The annotation is
subjectively provided.
4.2. Results
Figure 3 shows behaviors of SDR improvement for the
mixtures nos. 1–3 when the frequency or time annotation is
fed back by the user at the 80th iteration. For all the cases,
we can conﬁrm that the proposed system outperforms theconventional ILRMA. In particular, the conventional ILRMA
for the mixture no. 2 (Fig. 3(b)) cannot separate the sources
because of encountering the block permutation problem

F. OSHIMA et al.: INTERACTIVE SPEECH SEPARATION BASED ON ILRMA䣈䣴䣧䣳䣷䣧䣰䣥䣻䢢䣝䣭䣊䣼䣟䣕䣫䣩䣰䣣䣮䣵䢢䣧䣵䣶䣫䣯䣣䣶䣧䣦䢢䣤䣻䣋䣎䣔䣏䣃䢢䣹䣫䣶䣪䢢䣶䣫䣯䣧䢢䣣䣰䣰䣱䢰䢢䢪䣋䢫

䢴

䢴

䢴

䢳

䢳

䢳

References䢲䢲䢵䣈䣴䣧䣳䣷䣧䣰䣥䣻䢢䣝䣭䣊䣼䣟䣕䣫䣩䣰䣣䣮䣵䢢䣧䣵䣶䣫䣯䣣䣶䣧䣦䢢䣤䣻䣵䣫䣯䣲䣮䣧䢢䣋䣎䣔䣏䣃Acknowledgement
This work was partly supported by JSPS KAKENHI
Grant Numbers 19K20306 and 19H01116.

䢵䣑䣴䣣䣥䣮䣧䢢䣵䣱䣷䣴䣥䣧䢢䣵䣫䣩䣰䣣䣮䣵

䢳

䢴

䢵

䢶

䢵

䢷

䢲
䢲
䢵

䢳

䢴

䢵

䢶

䢵

䢷

䢲
䢲
䢵

䢴

䢴

䢴

䢳

䢳

䢳

䢲
䢲

䢳

䢴
䢵
䣖䣫䣯䣧䢢䣝䣵䣟

䢶

䢷

䢲
䢲

䢳

䢴
䢵
䣖䣫䣯䣧䢢䣝䣵䣟

䢶

䢷

䢲
䢲

䢳

䢴

䢵

䢶

䢷

䢳

䢴
䢵
䣖䣫䣯䣧䢢䣝䣵䣟

䢶䢷

Fig. 4 Spectrograms of ﬁrst (top) and second (bottom)
speech sources for mixture no. 2.

during the parameter optimization. However, in the proposedsystem, the user annotation assists to escape from the bad
local minimum and provides signiﬁcant improvement for thecorrect estimation of the demixing matrix W i .
Regarding the type of annotations, we can see thatboth the frequency and time annotations eﬀectively improve
the separation accuracy in the proposed system. Since thedemixing ﬁlters wi;n are completely reset using random values
 in the time annotations (I) and (II), the value of SDR
improvement drops after the annotation feedback. In otherwords, this performance drop is necessary for escaping fromthe bad local minimum or solving the block permutation
problem. Thus, it is important to suﬃciently iterate ILRMA
after the time annotation is fed back to the system (at least
40 iterations).
Figure 4 shows the spectrograms of oracle sources andestimated signals obtained by simple ILRMA and ILRMA
with time annotation (I) for the mixture no. 2. The estimatedsignals obtained by simple ILRMA (the center column in
Fig. 4) include the block permutation problem: the components between 0.4–1.7 kHz are swapped, whereas the proposed ILRMA (the right column in Fig. 4) correctly solves
this misalignment.
This experiment only compares the performance at the160th iteration in conventional ILRMA and the proposedsystem. Note that the interaction between ILRMA and theuser can be iterated until the satisfactory separation isachieved in an actual application. The judgment can be
subjectively taken by listening the estimated signals yi; j .
5.

Conclusion
In this letter, we proposed a new ILRMA-based interactive speech source separation system, where frequency andtime annotations were utilized. The experimental results showthat these annotations signiﬁcantly improve the separation
performance by escaping from the bad local minima in
ILRMA optimization.

[1] H. Sawada, N. Ono, H. Kameoka, D. Kitamura and H.
Saruwatari, ‘‘A review of blind source separation methods:
Two converging routes to ILRMA originating from ICA and
NMF,’’ APSIPA Trans. Signal Inf. Process., 8(e12), 1–14
(2019).
[2] P. Comon, ‘‘Independent component analysis, a new concept?,’’ Signal Process., 36, 287–314 (1994).
[3] P. Smaragdis, ‘‘Blind separation of convolved mixtures in thefrequency domain,’’ Neurocomputing, 22, 21–34 (1998).
[4] N. Murata, S. Ikeda and A. Ziehe, ‘‘An approach to blindsource separation based on temporal structure of speech
signals,’’ Neurocomputing, 41, 1–24 (2001).
[5] H. Sawada, R. Mukai, S. Araki and S. Makino, ‘‘A robust
and precise method for solving the permutation problem offrequency-domain blind source separation,’’ IEEE Trans.
Speech Audio Process., 12, 530–538 (2004).
[6] T. Kim, H. T. Attias, S.-Y. Lee and T.-W. Lee, ‘‘Blind source
separation exploiting higher-order frequency dependencies,’’IEEE Trans. Audio Speech Lang. Process., 15, 70–79 (2007).
[7] N. Ono, ‘‘Stable and fast update rules for independent vector
analysis based on auxiliary function technique,’’ Proc. IEEE
Workshop Appl. Signal Process. Audio Acoust., pp. 189–192(2011).
[8] D. Kitamura, N. Ono, H. Sawada, H. Kameoka and H.
Saruwatari, ‘‘Determined blind source separation unifying
independent vector analysis and nonnegative matrix factorization,’’ IEEE/ACM Trans. Audio Speech Lang. Process., 24,
1626–1641 (2016).
[9] D. Kitamura, N. Ono, H. Sawada, H. Kameoka and H.
Saruwatari, ‘‘Determined blind source separation with independent low-rank matrix analysis,’’ in Audio Source Separation, S. Makino, Ed. (Springer, Cham, 2018), Chap. 6, pp. 125–155.
[10] Y. Liang, S. M. Naqvi and J. A. Chambers, ‘‘Overcoming
block permutation problem in frequency domain blind source
separation when using AuxIVA algorithm,’’ Electron. Lett., 48,
460–462 (2012).
[11] Y. Mitsui, D. Kitamura, N. Takamune, H. Saruwatari, Y.
Takahashi and K. Kondo, ‘‘Independent low-rank matrix
analysis based on parametric majorizaion-equalization algorithm,’’ Proc. Comput. Adv. Multi-Sensor Adapt. Process.,
pp. 98–102 (2017).
[12] C. Févotte and J. Idier, ‘‘Nonnegative matrix factorization
with the Itakura-Saito divergence: With application to music
analysis,’’ Neural Comput., 21, 793–830 (2011).
[13] T. Ono, N. Ono and S. Sagayama, ‘‘User-guided independentvector analysis with source activity tuning,’’ Proc. IEEE Int.
Conf. Acoust. Speech Signal Process., pp. 2417–2420 (2012).
[14] S. Araki, F. Nesta, E. Vincent, Z. Koldovsky, G. Nolte, A.
Ziehe and A. Benichoux, ‘‘The 2011 signal separation evaluation campaign (SiSEC2011): Audio source separation,’’ Proc.
Latent Variable Anal. Signal Separation, pp. 414–422 (2012).
[15] E. Vincent, R. Gribonval and C. Févotte, ‘‘Performancemeasurement in blind audio source separation,’’ IEEE Trans.
Audio Speech Lang. Process., 14, 1462–1469 (2006).

225