Kitamura and Yatabe EURASIP Journal on Advances in Signal(2020) 2020:46
Processing
https://doi.org/10.1186/s13634-020-00704-4

RESEARCHEURASIP Journal on Advancesin Signal Processing

Open AccessConsistent independent low-rankmatrix analysis for determined blind source
separation
Daichi Kitamura1†*

and Kohei Yatabe2†

Abstract
Independent low-rank matrix analysis (ILRMA) is the state-of-the-art algorithm for blind source separation (BSS) in thedetermined situation (the number of microphones is greater than or equal to that of source signals). ILRMA achieves
a great separation performance by modeling the power spectrograms of the source signals via the nonnegative
matrix factorization (NMF). Such a highly developed source model can solve the permutation problem of thefrequency-domain BSS to a large extent, which is the reason for the excellence of ILRMA. In this paper, we further
improve the separation performance of ILRMA by additionally considering the general structure of spectrograms,
which is called consistency, and hence, we call the proposed method Consistent ILRMA. Since a spectrogram is
calculated by an overlapping window (and a window function induces spectral smearing called main- and side-lobes),
the time-frequency bins depend on each other. In other words, the time-frequency components are related to eachother via the uncertainty principle. Such co-occurrence among the spectral components can function as an assistantfor solving the permutation problem, which has been demonstrated by a recent study. On the basis of these facts, we
propose an algorithm for realizing Consistent ILRMA by slightly modifying the original algorithm. Its performance was
extensively evaluated through experiments performed with various window lengths and shift lengths. The results
indicated several tendencies of the original and proposed ILRMA that include some topics not fully discussed in theliterature. For example, the proposed Consistent ILRMA tends to outperform the original ILRMA when the window
length is sufficiently long compared to the reverberation time of the mixing system.
Keywords: Audio source separation, Convolutive mixture, Demixing filter estimation, Phase-aware signal processing,Spectrogram consistency

1 Introduction
Blind source separation (BSS) is a technique for separating individual sources from an observed mixture without
knowing how they were mixed. BSS for multichannel
audio signals observed by multiple microphones has been
particularly studied [1–13]. The BSS problem can bedivided into two situations: underdetermined (the number of microphones is less than the number of sources)and (over-)determined (the number of microphones is*Correspondence: d-kitamura@ieee.org† Daichi Kitamura and Kohei Yatabe contributed equally to this work.
1
National Institute of Technology, Kagawa College, 355 Chokushi, Takamatsu,
Kagawa, 761-8058, Japan
Full list of author information is available at the end of the article

greater than or equal to the number of sources) cases.
This paper focuses on the determined BSS problem, as
high-quality separation can be achieved compared withthe underdetermined BSS methods.
Independent component analysis (ICA) is the most popular and successful algorithm for solving the determined
BSS problem [1]. It estimates a demixing matrix (theinverse system of the mixing process) by assuming statistical independence between the sources. For a mixtureof audio signals, ICA is usually applied in the timefrequency domain via the short-time Fourier transform(STFT) because the sources are mixed up by convolution.
This strategy is called frequency-domain ICA (FDICA)© The Author(s). 2020 Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which
permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit
to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The
images or other third party material in this article are included in the article’s Creative Commons licence, unless indicatedotherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended
use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the
copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.

Kitamura and Yatabe EURASIP Journal on Advances in Signal Processing(2020) 2020:46[2] and independently applies ICA to the complex-valued
signals in each frequency. Then, the estimated frequencywise demixing matrices must be aligned over all frequencies so that the frequency components of the same source
are grouped together. Such alignment of the frequency
components is called a permutation problem [3–6], and acomplete solution to it has not been established. Therefore, a great deal of research has tackled this problem.
To avoid the permutation misalignment as much as
possible, various sophisticated source models have been
proposed. Independent vector analysis (IVA) [7–10] is
one of the most successful methods in the early stageof the development. It assumes higher-order dependencies (co-occurrence among the frequency components) ofeach source by utilizing a spherical generative model ofthe source frequency vector. This assumption enables IVA
to simultaneously estimate the frequency-wise demixing matrices and solve the permutation problem to alarge extent using only one objective function. It has
been further developed by improving its source model.
One natural and powerful extension of IVA is independent low-rank matrix analysis (ILRMA) [11, 12], which
integrates the source model of nonnegative matrix factorization (NMF) [14, 15] based on the Itakura–Saito
divergence (IS-NMF) [16] into IVA. This extension has
greatly improved the performance of separation by taking the low-rank time-frequency structure (co-occurrence
among the time-frequency bins) of the source signals into
account. ILRMA has achieved the state-of-the-art performance and been further developed by several researchers[17–29]. In this respect, ILRMA can be considered the
new standard of the determined BSS algorithms. However, the separation performance of IVA and ILRMA
is still inferior compared to the ideal performance of
ICA-based frequency-domain BSS. In [30], the performances of IVA and ILRMA were compared with that of
FDICA with perfect permutation alignment using reference sources (ideal permutation solver), and it was confirmed that there is still a noticeable room for improvement of ILRMA-based BSS. In fact, IVA and ILRMA often
encounter the block permutation problem, that is, groupwise permutation misalignment of components between
sources [31].
The consistency of a spectrogram is another promising approach for solving the permutation problem. A
recent study has shown that STFT can provide some effective information related to the co-occurrence among thetime-frequency bins [32]. Since an overlapping window isutilized in STFT, the time-frequency bins are related toeach other based on the overlapping segments. The frequency components within a segment are also related toeach other because of the spectral smearing called mainand side-lobes of the window. In other words, the timefrequency components are not independent but related toeach other via the uncertainty principle of time-frequency
representation. Such relations have been well-studied inphase-aware signal processing [33–43] by the name ofspectrogram consistency [44–47]. In the previous study[32], the spectrogram consistency was imposed on BSS
to help the algorithm solve the permutation problem.
This is an approach very different from the conventional
studies of determined BSS because it utilizes the generalproperty of STFT independent of the source model (incontrast to the abovementioned methods that focused onmodeling of the source signals without considering theproperty of STFT). As the spectrogram consistency canbe incorporated with any source model, its combination
with the state-of-the-art algorithm should achieve a highseparation performance.
However, the paper that proposed the combination of
consistency and determined BSS [32] only showed thepotential of consistency in an experiment using FDICA
and IVA. The paper claimed that it was a first step ofincorporating the spectrogram consistency with determined BSS, and no advanced method was tested. In particular, ILRMA was not considered because its algorithm
is far more complicated than that derived in [32], andthus, it is not clear whether (and how much) the spectrogram consistency might improve the state-of-the-art BSS
algorithm.
In this paper, we propose a new variant of ILRMA
called Consistent ILRMA that considers the spectrogram
consistency within the algorithm of ILRMA. The combination of IS-NMF and spectral smoothing of the inverse
STFT (see Figs. 1 and 2 in Section 2.3) achieves thesource modeling for a complex spectrogram. In particular, the spectral smearing in the frequency direction ties
the adjacent frequency bins together, and this effect ofspectrogram consistency helps ILRMA to solve the permutation problem. Since consistency is a concept depending on the parameters related to a window function, we
extensively tested the separation performance of Consistent ILRMA through experiments with various window
lengths and shift lengths. The results clarified several
tendencies of the conventional and proposed methods,
including that the proposed method outperforms theoriginal ILRMA when the window length is sufficiently
long compared to the reverberation time of the mixing
system.

Page 2 of 35

2 Permutation problem of frequency-domain BSS
and spectrogram consistency
2.1 Formulation of frequency-domain BSS

Let the lth sample of a time-domain signal be denoted
as x[ l], and N source signals be observed by M microphones. Then, the lth samples of the multichannel source,
observed, and separated signals are respectively denotedas:Kitamura and Yatabe EURASIP Journal on Advances in Signal Processing(2020) 2020:46

Page 3 of 35

Fig. 1 Inconsistent power spectrograms |Sart |2 (left column) and their consistent version (right column) obtained by applying inverse STFT and
STFT. The top-left spectrogram is artificially produced with random phase. The middle-left and the bottom-left spectrograms are music and speech
signals with random dropout. Enforcing spectrogram consistency can be viewed as a smoothing process of the inconsistent spectrogram along
both time and frequency axess[ l] = [ s1 [ l] , s2 [ l] , · · · , sn [ l] , · · · sN [ l] ]T ∈ RN ,TM

x[ l] = [ x1 [ l] , x2 [ l] , · · · , xm [ l] , · · · xM [ l] ] ∈ R ,T
∈ RN ,
y[ l] = y1 [ l] , y2 [ l] , · · · , yn [ l] , · · · yN [ l]

(1)
(2)
(3)

where n = 1, · · · , N, m = 1, · · · , M, and l = 1, · · · , L are
the indexes of sources, microphones (channels), and discrete time, respectively, and ·T denotes the transpose. BSS
aims at recovering the source signal s from the observed
signal x, i.e., making y as close to s as possible.
In the frequency-domain BSS, those signals are handled
in the time-frequency domain via STFT. Let the window
length and shifting step of STFT be denoted as Q andτ , respectively. Then, the jth segment of a signal z[ l] isdefined as:
 
 


T
z[j] = z (j−1)τ +1 , z (j−1)τ +2 , · · · , z (j−1)τ +Q ,T
∈ RQ ,= z[j] [ 1] , z[j] [ 2] , · · · , z[j] [ q] , · · · , z[j] [ Q](4)where j = 1, · · · , J and q = 1, · · · , Q are the indexes ofthe segments and in-segment samples, respectively, andthe number of segments is given by J = L/τ with somezero-padding for adjusting the signal length L if necessary.
STFT of a signal z = [ z[ 1] , z[ 2] , · · · , z[ L] ]T ∈ RL isdenoted by:
Z = STFTω (z) ∈ CI×J ,

(5)

 
where the i, j th bin of the spectrogram Z is given as:
zij =

Q


ω[ q] z[j] [ q] e−ı2π(q−1)(i−1)/F ,

(6)

q=1

i = 1, · · · , I is the index of frequency bins, F is an integer satisfying F/2 + 1 = I, · is the floor function,
ı denotes the imaginary unit, and ω is an analysis window. The inverse STFT with a synthesis window ω is also
defined in the usual way and denoted as ISTFTω (·). In
this paper, we assume that the window pair satisfies thefollowing perfect reconstruction condition:z = ISTFTω (STFTω (z))∀z ∈ RL .

(7)

Kitamura and Yatabe EURASIP Journal on Advances in Signal Processing(2020) 2020:46

Page 4 of 35

Fig. 2 Smoothing effect of spectrogram consistency applied to permutation misaligned signals: a music and b speech. The left column shows theoriginal source signals |Sn |2 , and the center column shows their randomly permuted versions, which simulates the permutation problem and is(perm)(perm)
denoted as Sn. The right column shows the consistent versions of Sn. The smoothing effect mixes up the signals

 
By applying STFT, the i, j th bin of the spectrograms ofsource, observed, and separated signals can be written as:
T∈ CN ,(8)
sij = sij1 , sij2 , · · · , sijn , · · · sijN
T

M
(9)
xij = xij1 , xij2 , · · · , xijm , · · · xijM ∈ C ,T∈ CN .
(10)
yij = yij1 , yij2 , · · · , yijn , · · · yijN
We also denote the spectrograms corresponding to thenth or mth signals in (8)–(10) as Sn ∈ CI×J , X m ∈ CI×J ,and Y n ∈ CI×J , whose elements are sijn , xijm , and yijn ,respectively. In the ordinary frequency-domain BSS, an
instantaneous mixing process for each frequency bin isassumed:xij = Ai sij ,(11)

where Ai ∈ CM×N is a frequency-wise mixing matrix. The
mixture model (11) is approximately valid when the reverberation time is sufficiently shorter than the length of theanalysis window used in STFT [48].
Hereafter, we consider the determined case, i.e., M = N.
In this case, BSS can be achieved by estimating the inverseof Ai for all frequency bins. By denoting an approximate
inverse as W i ≈ A−1
i , the separation process can be
written as:
yij = W i xij ,(12)

where W i = [wi1 , wi2 , · · · , wiN ]H ∈ CN×M is a frequencywise demixing matrix and ·H denotes the Hermitian transpose. The aim of a determined BSS algorithm is to find

Kitamura and Yatabe EURASIP Journal on Advances in Signal Processing(2020) 2020:46

the demixing matrices for all frequency bins so that the
separated signals approximate the source signals.

Figure 1 demonstrates the effect of spectrogram consistency, where Sart ∈ CI×J is an artificially produced
complex-valued spectrogram and |Sart |2 is its power spectrogram. The notation | · |2 for a matrix input represents the element-wise squared absolute value. By applying STFTω (ISTFTω (·)), the inconsistent spectrogram Sart
shown in the left column of Fig. 1 is converted into the corresponding consistent spectrogram, which is a smoothedversion of Sart , as shown in the right column. This smoothing process occurs because the main- and side-lobes of thewindow function (and the overlap-add process) spread theenergy of a time-frequency bin.
Since the inverse STFT is a process of recovering theconsistency (the inter-time-frequency relation), it has thecapability of aligning the frequency components. This isalso demonstrated in Fig. 2. As a simulation of the permutation problem, the frequency bins in S1 and S2 wererandomly shuffled to obtain the spectrogram with per(perm)
mutation misalignment, Sn(the center column in the
figure), which is a typical output signal of FDICA. Notethat these misaligned spectrograms are perfectly separated for each frequency because each time-frequency bin
contains only one of the two sources. By enforcing spectrogram consistency, the smoothing process spreads thetime-frequency components as shown in the right column of Fig. 2. In other words, the inverse STFT mixes up
the separated signals if the frequency-wise permutation
is not aligned correctly. Therefore, enforcing consistency
within a BSS algorithm by applying STFTω (ISTFTω (·))
can improve the separation performance to some extent[32].

2.2 Permutation problem in determined BSS

In practice, the scale and permutation of the separated signals are unknown because the information of the mixing
process is missing. That is, when the separation is correctly performed by some demixing matrix W i as in (12),the following signal is also a solution to the BSS problem:ŷij = Ŵ i xij

Ŵ i = Di Pi W i ,(13)where Di ∈ CN×N and Pi ∈ {0, 1}N×N are arbitrary diagonal and permutation matrices, respectively. While thesignal scale can easily be recovered by applying the backprojection [49], the permutation of the estimated signals
ŷij must be aligned for all frequency bins, i.e., P i must be
the same for all i. This alignment of the permutation of
estimated signals is the permutation problem, which is themain obstacle of the frequency-domain determined BSS.
In FDICA, a permutation solver (realignment process of Pi ) is utilized as a post-processing applied to
the frequency-wise separated signals ŷij [4–6]. In recent
frequency-domain BSS methods, an additional assumption on sources (or source model) is introduced to circumvent the permutation problem. For example, IVA assumes
simultaneous co-occurrence of all frequency components
in the same source, and ILRMA assumes a low-rank structure of the power spectrogram Y n . Other source models
have also been proposed for improving the separation performance [50–52]. These source models can avoid the
permutation problem to some extent during the estimation of Ŵ i . Recent developments of determined BSS havebeen achieved via the quest to find a better source modelthat represents the source signals more precisely.
2.3 Solving permutation problem by spectrogram
consistency

A recent paper reported another approach for solving the
permutation problem based on the general property of
STFT called spectrogram consistency [32]. The consistency is a fundamental property of a spectrogram. Since
any time-frequency representation has a theoretical limitation called the uncertainty principle, the time-frequency
bins of a spectrogram are not independent but related toeach other. The inverse STFT always modifies the spectrogram Z n that violates this kind of inter-time-frequency
relation so that the relation is recovered. That is, aspectrogram Z n properly retains the inter-time-frequency
relation if and only if

E (Z n ) = Z n − STFTω (ISTFTω (Z n ))(14)

is zero, i.e., E (Z n ) = 0 for a norm ·. Such spectrogramZ n satisfying E (Z n ) = 0 is said to be consistent.

Page 5 of 35

3 Proposed method
By incorporating spectrogram consistency into ILRMA,
we propose a novel BSS method named Consistent
ILRMA. In this section, after stating our motivation andcontributions, we first review the standard ILRMA introduced in [11, 12] and then propose the consistent version
of ILRMA with an algorithm that achieves Consistent
ILRMA and is openly available on the web.
3.1 Motivations and contributions

The previous paper [32] only reported that the performances of traditional BSS algorithms, FDICA and IVA,
were improved by enforcing consistency during the estimation of the demixing matrix W i . In addition, no
detailed experimental analysis related to STFT parameters was provided, even though the parameters of windowfunctions in the STFT and inverse STFT directly affect thesmoothing effect of spectrogram consistency.
The spectrogram consistency is a general property of
STFT, and therefore, it can be combined with any source
model for determined BSS. Its combination with stateof-the-art models, including ILRMA, is of great interest

Kitamura and Yatabe EURASIP Journal on Advances in Signal Processing(2020) 2020:46

because the current mainstream algorithm for determinedaudio source separation is centered on ILRMA, which
is based on an NMF-based richer time-frequency source
model. Indeed, many recent papers are based on theframework of ILRMA [17–29]. Even though combining
ILRMA with the spectrogram consistency should be able
to exceed the limit of existing BSS algorithms, no suchmethod has been investigated in the literature.
In this paper, we propose a new BSS algorithm that combines ILRMA and spectrogram consistency. Our first contribution is an algorithm that achieves Consistent ILRMA
by inserting STFTω (ISTFTω (·)) into the iterative optimization algorithm of ILRMA. The second contribution
is to apply a scale-aligning process called iterative backprojection within the iterative algorithm. This processenhances the separation performance when it is combinedwith spectrogram consistency. The third contribution isan experimental finding that spectrogram consistency
can work properly with the iterative back projection. We
found that both Consistent IVA and Consistent ILRMA
require iterative back projection to achieve a good performance. Our fourth contribution is to provide the massive
experimental results for several window functions, window lengths, shift lengths, reverberation times, and source
types. We also provide discussions for clarifying the tendency of ILRMA with spectrogram consistency.

represent spectral patterns of the nth source signal, areindexed by k = 1, · · · , K. As in FDICA, statistical independence between the source signals is also assumed in
ILRMA:p(Y 1 , Y 2 , · · · , Y N ) =

i,j

(15)
where Nc (μ, r) is the circularly symmetric complex Gaussian distribution with mean μ and variance r. In thismodel, the source component yijn is assumed to obey azero-mean and isotropic distribution, i.e., the phase ofyijn is generated from the uniform distribution in therange [ 0, 2π) and the real and imaginary parts of yijn
are mutually independent. The validity of this assumption is shown in the Appendix. The variance rijn can beviewed as an expectation value of |yijn |2 . This variancerijn as a two-dimensional array indexed by i, j is denoted
as Rn ∈ RI×J
>0 , which is called the variance spectrogram
corresponding to the nth source. In ILRMA, the variancematrix Rn is modeled using the rank-K NMF, as:
Rn = T n V n ,

(16)

K×J
where T n ∈ RI×K
>0 and V n ∈ R>0 are the basis and activation matrices in NMF. The basis vectors in T n , which(17)

ILRMA estimates the demixing matrix W i so that thepower spectrograms of the separated signals |Y n |2 havea low-rank structure that can be well-approximated by
T n V n with small K. This BSS principle of ILRMA isillustrated in Fig. 3. When the low-rank source model
can appropriately fit to the power spectrograms of theoriginal source signals |Sn |2 , ILRMA provides an excellent separation performance without explicitly solving the
permutation problem afterward.
The demixing matrix W i and the nonnegative matrices
T n and V n can be obtained through maximum likelihood
estimation. The negative log-likelihood to be minimized,
denoted by L, is given as [12]:L = − log p(X 1 , X 2 , · · · , X M ),
=−log |det W i |2 − log p(Y 1 , Y 2 , · · · , Y N ),
i,j
c

= −2J



| det W i |+i

The original ILRMA [12] was derived from the following generative model of the spectrograms of the separated
signals:|yijn |2
1,
Nc 0, rijn =exp −Y n ∼ p(Y n ) =
πrijn
rijn

p(Y n ).
n

3.2 Standard ILRMA [12]i,jPage 6 of 35


i,j,n


 H 2
w xij 

in

+log
tikn vkjn ,k tikn vkjn
k

(18)
c

where = denotes equality up to constant factors, andtikn > 0 and vkjn > 0 are the elements of T n and V n ,respectively. The minimization of (18) can be performedby iterating the following update rules for the spatialmodel parameters,
U in ←1
1

xij xHij ,Jtv
k ikn kjn

(19)jwin ← (W i U in )−1 en ,

− 12
,
win ← win wH
in U in win(21)

yijn ← wH
in xij ,(22)

and for the source model parameters,    −2
 j yijn 2
tik n vk jn
vkjn
ktikn ← tikn   
,
−1
vkjn
j
k tik n vk jn−2   2 yijn 
tik n vk jn
tikn
vkjn ← vkjn  i  k
,
−1
t
v
t
ik
n
k
jn
ikn
i
k

(20)

(23)

(24)

where en ∈ {0, 1}N is the unit vector with the nth element
equal to unity. Update rules (19)–(24) ensure the monotonic non-increase of the negative log-likelihood function

Kitamura and Yatabe EURASIP Journal on Advances in Signal Processing(2020) 2020:46

Page 7 of 35

Fig. 3 BSS principle of standard ILRMA

L. After iterative calculations of updates (19)–(24), theseparated signal can be obtained by (12).
Equation 22 is equivalent to beamforming [53] to xij
with the beamformer coefficients win . Thus, FDICA, IVA,
and ILRMA can be interpreted as an adaptive estimation process of beamforming coefficients without having
to know the geometry of microphones and sources [54].
For this reason, the estimated signal Y n obtained by (22)is a complex-valued spectrogram, and we do not need torecover its phase components using, for example, Griffin–
Lim algorithm-based techniques [37–40, 43, 55–59]. Boththe amplitude and phase components of each source arerecovered by the complex-valued linear separation filter
win .
3.3 Proposed Consistent ILRMA

To further improve the separation performance of thestandard ILRMA, we introduce the spectrogram consistency into the parameter update procedure. In the proposed Consistent ILRMA, the following combination of
forward and inverse STFT is performed at the beginningof each iteration of parameter updates:
Y n ← STFTω (ISTFTω (Y n )).

(25)This procedure is the projection of the spectrogram ofa separated signal Y n onto the set of consistent spectrograms [32]. That is, STFTω (ISTFTω (Y n )) performs
nothing if Y n is consistent, but otherwise, it smooths
the complex spectrogram Y n , by going through the time
domain, so that the uncertainty principle is satisfied.
In Consistent ILRMA, the calculation of (25) is performed in each iteration of parameter updates based on(19)–(24). Enforcing the spectrogram consistency for the
temporary separated signal Y n in each iteration guides
the parameters W i , T n , and V n to better solutions, which
results in higher separation performance compared to thatof conventional ILRMA.
Note that this simple update (25) may increase the valueof the negative log-likelihood function (18), and therefore,
the monotonicity of the algorithm is no longer guaranteed. However, we will see later in the experiments thatthe value of the negative log-likelihood function stably
decreases as in the standard ILRMA. The amount of theinconsistent component (14) also settles down to somespecific value after several iterations.
3.4 Iterative back projection

Since frequency-domain BSS cannot determine the scalesof estimated signals (represented by Di in (13)), the spectrogram of a separated signal Y n after an iteration isinconsistent due to the scale irregularity. To take fulladvantage of the projection enforcing spectrogram consistency in (25), we also propose applying the following
back projection at the end of each iteration so that thefrequency-wise scales are aligned.
In determined BSS, the back projection is a standard
procedure for recovering the frequency-wise scales. It canbe written as [49]:en ◦ yij = yijn λin ,ỹijn = W −1
i

(26)

T

 
where ỹijn = ỹijn1 , ỹijn2 , · · · , ỹijnM ∈ CM is the i, j thbin of the scale-fitted spectrogram of the nth separated
signal, λin = [ λin1 , λin2 , · · · , λinM ]T ∈ CM is a coefficient vector of back projection for the nth signal at the ith
frequency, and ◦ denotes the element-wise multiplication.
In the proposed method, this update (26) is performedat the end of each iteration so that the projection (25) atthe beginning of the next iteration properly smooths the
spectrograms without the effect of scale indeterminacy.
One side effect of this back projection is that the valueof the negative log-likelihood function (18) is also changed
due to the scale modification. In IVA, this problem cannot be avoided because the only parameter in IVA is the
demixing matrix W i . However, in ILRMA, since both
the demixing matrix W i and the source model parameter T n V n can determine the scale of estimated signal Y n ,the likelihood variation can be avoided by appropriately
adjusting win and T n after the back projection. To prevent the likelihood variation, the following updates are
required after performing (26):Kitamura and Yatabe EURASIP Journal on Advances in Signal Processing(2020) 2020:46

Algorithm 1 Consistent ILRMA I,J
Input: xij i=1,j=1 , maxIter I,J
Output: yij

4.1 BSS of synthesized mixtures
4.1.1 Conditions

i=1,j=1NN
I
1: Initialize {T n }n=1 , {V n }n=1 , {W i }i=1
2: for iter = 1, 2, · · · , maxIter do

Ensure consistency by calculating (25) ∀nUpdate source model by calculating (23) and (24)∀i, j, k, n5:Update spatial model by calculating (19)–(22)∀i, j, n6:
Apply back projection by calculating (26) ∀i, j, n7:Update parameters by calculating (27)–(29)∀i, j, k, n
8: end for3:4:

win ← win λinmref ,(27)

yijn ← wH
in xij ,2tikn ← tikn λinmref  ,(28)Page 8 of 35

We conducted determined BSS experiments using synthesized music and speech mixtures with two sources andtwo microphones (N = M = 2). The dry sources of music
and speech signals, listed in Table 1, were respectively
obtained from professionally produced music
and underdetermined separation tasks provided as a part of SiSEC2011 [60]. They were convoluted
with the impulse response E2A (T60 = 300 ms) or JR2(T60 = 470 ms), obtained from the RWCP database[61], to simulate the multichannel observation signals.
The recording conditions of these impulse responses areshown in Fig. 4.
In this experiment, we compared the performance ofsix methods: three conventional and three proposed. The
conventional methods were the standard IVA [10], Consistent IVA [32], and standard ILRMA [11]. The proposed
methods were Consistent IVA with iterative back projection (Consistent IVA+BP), Consistent ILRMA, and Consistent ILRMA with iterative back projection (Consistent
ILRMA+BP). For all methods, the initial demixing matrix
was set to an identity matrix. For the ILRMA-based(29)

where mref is the index of the reference channel utilized in
the back projection.
The overall algorithm of the proposed Consistent
ILRMA is summarized in Algorithm 1. The iterative
loop for the parameter optimization appears in the second to eighth lines. The spectrogram consistency of
the temporary separated signal Y n is ensured in thethird line, and the iterative back projection is appliedin the sixth and seventh lines. Note that an algorithm for the conventional ILRMA can be obtainedby performing only the fourth and fifth lines (i.e.,
ignoring the third, sixth, and seventh lines). A Python
code of the conventional ILRMA is openly available online (https://pyroomacoustics.readthedocs.io/en/
pypi-release/pyroomacoustics.bss.ilrma.html), and therefore, the proposed Consistent ILRMA with Python
can be easily implemented by slightly modifying thecodes. A MATLAB code of Consistent ILRMA is alsoavailable online (https://github.com/d-kitamura/ILRMA/
blob/master/consistentILRMA.m).

4 Experiments
In this section, we conducted two experiments usingsynthesized and real-recorded mixtures. The synthesized
mixtures were produced by convoluting the impulse
responses to dry audio sources, while the real-recorded
mixtures were actually recorded by using a microphonearray in an ordinary room with ambient noise.

Table 1 Music and speech dry sources obtained from SiSEC2011
SignalData nameSource (1/2)

Music 1bearlin-roads

acoustic_guit_
main/vocals

Music 2bearlin-roads

piano/acoustic_
guit_mainMusic 3bearlin-roads

piano/vocals

Music 4

another_dreamer-the_ones_we_love

guitar/vocals

Music 5another_dreamer-the_ones_we_love

drums/guitar

Music 6

fort_minor-remember_the_nameviolins_synth/
vocalsMusic 7

fort_minor-remember_the_namevocals/drums

Music 8tamy-que_pena_tanto_faz

guitar/vocals

Music 9ultimate_nz_tour

guitar/synth

Music 10

ultimate_nz_tour

drums/vocals

Speech 1

dev1_female4

src_1/src_2

Speech 2

dev1_female4

src_1/src_4

Speech 3

dev1_female4

src_2/src_3

Speech 4

dev1_female4

src_2/src_4

Speech 5

dev1_female4

src_3/src_4

Speech 6

dev1_male4

src_1/src_2

Speech 7

dev1_male4

src_1/src_4

Speech 8

dev1_male4

src_2/src_3

Speech 9

dev1_male4

src_2/src_4

Speech 10

dev1_male4

src_3/src_4

Kitamura and Yatabe EURASIP Journal on Advances in Signal Processing(2020) 2020:46

Page 9 of 35

Fig. 4 Recording conditions of impulse responses: a E2A and b JR2

methods, the nonnegative matrices T n and V n were initialized using uniformly distributed random values in therange (0, 1). Five trials were performed for each condition
using different pseudorandom seeds. The number of basesfor each source, K, was set to 10 for music mixtures and
2 for speech mixtures, where it was experimentally confirmed that these conditions provide the best performancefor the conventional ILRMA [11]. To satisfy the perfectreconstruction condition (7), the inverse STFT was implemented by the canonical dual of the analysis window. For
both Consistent IVA+BP and Consistent ILRMA+BP, theiterative back projection was applied, where the reference
channel was set to mref = 1. Since the property of spectrogram consistency depends on the window length, shift
length, and type of window function, various combinations of them were tested. The experimental conditions
are summarized in Table 2.
For quantitative evaluation of the separation performance, we measured the source-to-distortion ratio (SDR),
source-to-interference ratio (SIR), and source-to-artifact
ratio (SAR). In a noiseless situation, SDR, SIR, and SAR
are defined as follows [62]:SDR = 10 log10 2
l |st [ l] |,2
l |ei [ l] +ea [ l] ||st [ l] |2,
SIR = 10 log10 l2
l |ei [ l] ||st [ l] +ei [ l] |2,
SAR = 10 log10 l 
2
l |ea [ l] |and SAR are used to quantify the amount of interference rejection and the absence of artificial distortion of
the estimated signal, respectively. SDR is used to quantify the overall separation performance, as SDR is in good
agreement with both SIR and SAR for determined BSS.
In this experiment, the energy of sources was not
adjusted, i.e., the energy ratio of sources (source-to-source
ratio) was automatically determined by the initial volume of the dry sources and the level of the impulse
responses. That is, the source-to-source ratio of each
mixture signal is different from the others. To equally evaluate the performances of different mixtures, we calculated
SDR improvement (SDR) and SIR improvement (SIR)
defined as:
SDR = SDRsep − SDRinput ,(33)

SIR = SIRsep − SIRinput ,

(34)

(30)

where SDRsep and SIRsep are the SDR and SIR of the separated signal, and SDRinput and SIRinput are the SDR andSIR of the initial mixture signal input to the BSS methods.
Note that SAR improvement cannot be defined because
its value of the signal without artificial processing cannot
be defined (SARinput = ∞).

(31)

Table 2 Experimental conditions
Window function(32)

where st [ l], ei [ l], and ea [ l] are the lth sample of target signal, interference, and artificial components of theestimated signal, respectively, in the time domain. SIR

Hann/Hamming/Blackman window

Window length64, 128, 256, 512, 768, 1024 ms

Window shift length

1/16, 1/8, 1/4, 1/2 of window length

Number of bases K for each sourcein ILRMA

10 for music signals and 2 forspeech signals

Number of iterations

100

Kitamura and Yatabe EURASIP Journal on Advances in Signal Processing(2020) 2020:46

Page 10 of 35

Fig. 5 Values of negative log-likelihood function (18) of Consistent ILRMA+BP (window length, 256 ms; shift length, 32 ms)

4.1.2 Results and discussions

Figure 5 shows examples of the value of the negative
log-likelihood function (18) of Consistent ILRMA+BP.
Although the algorithmic convergence of the proposed
method has not been theoretically justified because of theadditional projection (25), we experimentally confirmed asmooth decrease of the cost function. We also confirmedthat such behavior was common for the other experimental conditions and mixtures. This result indicates thatthe additional procedure in the proposed method does
not have a harmful effect on the behavior of the overall
algorithm.
Figures 6 and 7 show examples of the energy of theinconsistent components (14) of standard ILRMA andConsistent ILRMA+BP. The energy was normalized bythat of the initial spectrograms in order to align thevertical axis. Note that the energy of inconsistency components is not directly related to the degree of permutation misalignment or the separation performance.
These figures are shown to confirm whether the proposed
algorithm can properly reduce the degree of inconsistency. These values are completely zero when the separated spectrograms are consistent, and hence, those at
the 0th iteration (the leftmost values) are zero because
no processing is performed at that point. By iterating
the algorithms, this energy rapidly increased because thedemixing matrix for each frequency independently tried
to process and separate the signals. However, the normalized energy tended toward some specific values after
several iterations. We confirmed that the converged valuesof Consistent ILRMA+BP were always lower than thoseof standard ILRMA. This result indicates that Consistent ILRMA+BP reduces the amount of the inconsistentcomponents and tries to make the separated spectrogram
more consistent. In addition, similar to Fig. 5, the algorithmic stability of Consistent ILRMA+BP can be confirmed
from Figs. 6 and 7.

Figures 8 and 9 summarize the SDR improvements forthe music mixtures and speech mixtures, respectively. The
window function was the Hann window. Each box contains 50 results (i.e., 5 pseudorandom seeds × 10 mixturesin Table 1), where SDRs of the two separated sources ineach mixture were averaged. The central lines of the box
plots indicate the median, and the bottom and top edgesof the box indicate the 25th and 75th percentiles, respectively. Each row corresponds to the same window length,
while each column corresponds to the same shift length.
As we conducted the experiment for six window lengths,
four shift lengths, and two impulse responses, each figure
consists of 6 × 4 × 2 subfigures. In each subfigure, sixboxes are shown to illustrate the results of (1) IVA, (2)Consistent IVA, (3) Consistent IVA+BP, (4) ILRMA, (5)Consistent ILRMA, and (6) Consistent ILRMA+BP. Since
the tendency of the results was the same as Figs. 8 and
9, we provide the SDR improvements for the other windows (Hamming and Blackman) in the Appendix. The SIR
improvement and SAR are also given in the Appendix.
Since IVA and ILRMA assume the instantaneous mixing model (11) for each frequency in the time-frequency
domain, the window length should be long relative to thereverberation time to achieve accurate separation. At thesame time, too long a window degrades the separation
performance of IVA and ILRMA, as discussed in [30]. Thisis because capturing the source activity and spectral patterns becomes difficult for IVA and ILRMA as the time
resolution of the spectrograms becomes low due to a long
window. The robustness of IVA and ILRMA is also deteriorated by a long window because the effective number oftime segments is decreased. This trade-off of the separation performance caused by window length in STFT canbe easily confirmed from the results for both music (Fig. 8)
and speech (Fig. 9) mixtures, which is consistent withthe results in [30]. As shown in the figures, the performance was poor for the shorter windows (≤ 128 ms), andKitamura and Yatabe EURASIP Journal on Advances in Signal Processing(2020) 2020:46

Page 11 of 35


Fig. 6 Examples of normalized energy of inconsistent components E (Y)22 /X22 of ILRMA and Consistent ILRMA+BP for music 1: a 256-ms-longwindow and 32-ms shifting and b 1024-ms-long window and 512-ms shifting, where X = [X 1 , X 2 ] , Y = [ Y 1 , Y 2 ] , and E (·) is in (14)the performance for the longer windows (≥ 768 ms) was
more varied than that of the shorter ones. The window
length best suited for these conditions (combinations ofsource signals and impulse responses) seems to be around
256 ms or 512 ms. While the maximum achievable performance becomes higher as the window length becomes
longer due to the mixing model (11), these results indicate that the source modeling becomes difficult for both
IVA and ILRMA when the window length is too long. This
trade-off should be important for discussing the results
further.
By comparing the performances of the conventional(IVA, Consistent IVA, and ILRMA) and proposed (Consistent IVA+BP, Consistent ILRMA, and Consistent
ILRMA+BP) methods, we can see that the proposed
methods tend to outperform the conventional ones. Somecomparisons are made as follows:• Conventional and proposed IVAs. The proposed
Consistent IVA+BP performed better than the conventional IVAs (IVA and Consistent IVA) in Figs. 8band 9b when the window length was sufficiently long(≥ 256 ms). In those cases, the conventional Consistent IVA resulted in a worse performance than IVA,
which indicates that just using spectrogram consistency cannot improve the performance of IVA.
This demonstrates the importance of the iterative
back projection when spectrogram consistency is
considered within determined BSS.
• Conventional and proposed ILRMAs. The
proposed Consistent ILRMA without BP performed
comparably to the conventional ILRMA. In Figs. 8aand 9a, Consistent ILRMA performed better
than ILRMA when the window length was long(≥ 768 ms). In contrast, in Figs. 8b and 9b, Consis-

Kitamura and Yatabe EURASIP Journal on Advances in Signal Processing(2020) 2020:46

Page 12 of 35

Fig. 7 Examples of normalized energy of inconsistent components (E (Y)22 /X22 ) of ILRMA and Consistent ILRMA+BP for speech 1: a
256-ms-long window and 32-ms shifting and b 1024-ms-long window and 512-ms shifting, where X = [ X 1 , X 2 ] , Y = [ Y 1 , Y 2 ] , and E (·) is in (14)tent ILRMA performed worse than ILRMA. Thisis presumably because the scale ambiguity prevented the spectrogram consistency from working
properly. By incorporating iterative back projection
into Consistent ILRMA, the proposed Consistent
ILRMA+BP performed better than the conventional
ILRMA. In the best situation (the top-left subfigureof Fig. 8), Consistent ILRMA+BP performed 8 dB
better than ILRMA by bringing out the potential ofspectrogram consistency in determined BSS.
To further explain the experimental results, somenotable tendencies are summarized as follows:• Short window. When the window length was short(64 ms), all methods performed similarly in termsof SDR. This is because the achievable performance was already limited by the window length thatwas shorter than the reverberation time. This result
contradicted our expectation before performing theexperiment. Since enforcing the consistency spreads
the frequency components based on the main-lobeof the window function, we expected that the ability
to solve the permutation problem would be higher
when the window length was shorter because of thewider main-lobe. In reality, we found that the spectrogram consistency could assist IVA and ILRMA
except for the cases where the window length was
short (≤ 128 ms in this experiment) compared to thereverberation time.
• Large window shift. When the shift length was 1/2of the window length, the performance of ILRMA
significantly dropped compared to smaller shift
lengths (1/4, 1/8, and 1/16), especially when thewindow length was long (e.g., 1024 ms). This is pre-

Kitamura and Yatabe EURASIP Journal on Advances in Signal Processing(2020) 2020:46

Page 13 of 35

Fig. 8 Average SDR improvements for synthesized music mixtures (music 1–10) with a E2A and b JR2, where Hann window is used in STFT

sumably because the number of time segments was
small, i.e., NMF in ILRMA failed to model the source
signals from the given amount of data. In addition,
for a larger window, distinguishing spectral patternsof the sources became difficult for ILRMA due to thetime-directional blurring effect caused by a longer
window. Such performance degradation was alleviated for Consistent ILRMA+BP. This might bebecause the smoothing process of the inverse STFT
provides some additional information for the source
modeling from the adjacent bins.

• Length of boxes. When the length of the box of
ILRMA was long, as in Figs. 8a and 9a, Consistent ILRMA+BP was able to improve the performance. Conversely, when the length of the box of
ILRMA was short, as in Figs. 8b and 9b, Consistent
ILRMA+BP was only able to slightly improve theperformance. Note that the vertical axes are different. This result indicates that the achievable performance decided by the mixing model (11) limits the
improvement obtained by spectrogram consistency.
Since consistency is the characteristic of a spectro-

Kitamura and Yatabe EURASIP Journal on Advances in Signal Processing(2020) 2020:46

Page 14 of 35

Fig. 9 Average SDR improvements for synthesized speech mixtures (speech 1–10) with a E2A and b JR2, where Hann window is used in STFT

gram, it cannot manage the mixing process. The
demixing-filter update of ILRMA, which is the samefor the conventional and proposed methods, manages the mixing process. Hence, when the mixing
model has a mismatch with the observed condition,
there is less room for spectrogram consistency toimprove the performance.
• Improvement by consistency. The proposed methodtended to achieve a good performance when theconventional ILRMA also worked well, e.g., Figs. 8aand 9a. This tendency indicates that the spectro-

gram consistency effectively promotes the separation when the estimated source Y n accurately
approaches the original source Sn during the optimization, as Sn is naturally a consistent spectrogram.
This is the reason we feel that the consistency can be
an assistant of the frequency-domain BSS. An important aspect is that the source model (e.g., NMF in
ILRMA) actually informs the separation cue, and thespectrogram consistency enhances the separation
performance when the source modeling functions
correctly.

Kitamura and Yatabe EURASIP Journal on Advances in Signal Processing(2020) 2020:46

Page 15 of 35

4.2 BSS of real-recorded mixtures
4.2.1 Conditions

Next, we evaluated the conventional and proposed
methods using live-recorded music and speech mixtures
obtained from underdetermined separation
tasks in SiSEC2011 [60], where only two sourceswere mixed to make the BSS problem determined(M = N = 2). The signals used in this experiment arelisted in Table 3. The reverberation time of these signals
was 250 ms, and the microphone spacing was 1 m (see[60]). Since these source signals were actually recorded
using a microphone array in an ordinary room with
ambient noise, the observed signals are more realisticcompared to those in Section 4.1.
For simplicity, in this experiment, we used STFT witha fixed condition, the 512-ms-long Hann window with
1/4 shifting. The experimental conditions other than thewindow were the same as those in Section 4.1.1.
4.2.2 Results and discussion

Figure 10 shows the results of live-recorded music
and speech mixtures. The absolute scores were lower
than those for the synthesized mixtures discussed in
Section 4.1.2 due to the existence of ambient noise. Still,
we can confirm the improvements of the proposed Consistent IVA+BP and Consistent ILRMA+BP compared tothe conventional IVA and ILRMA, respectively, for boththe music (upper row) and speech (lower row) mixtures.
In particular, Consistent IVA+BP improved more than
4 dB over IVA in terms of the median of the SDR
of speech mixtures. Consistent ILRMA+BP achieved thehighest performance in terms of the median of the SDR
improvement for both music and speech mixtures. These
results confirm that the combination of spectrogram consistency and iterative back projection can assist the separation of determined BSS for a more realistic situation.

5 Conclusion
In this paper, we have proposed a new variant of thestate-of-the-art determined BSS algorithm called Consistent ILRMA. It utilizes the smoothing effect of theinverse STFT in order to assist the separation and enhance
the performance. Experimental results showed that the
proposed method can improve the separation performance when the window length is sufficiently large (≥256 ms in the experimental condition of this paper). These
results demonstrate the potential of considering spectrogram consistency within the state-of-the-art determined
BSS algorithm. In addition, we experimentally confirmed
the importance of iterative back projection for considering spectrogram consistency within determined BSS.
It should be possible to construct a new source modelin consideration of the spectrogram consistency, which
can pave the way for the next direction of research on
determined BSS.

Table 3 Live-recorded music and speech signals obtained fromSiSEC2011
SignalSource (1/2)

Music 1

dev1_nodrums_liverec_250ms_1m_sim_1/dev1_
nodrums_liverec_250ms_1m_sim_2

Music 2

dev1_nodrums_liverec_250ms_1m_sim_1/dev1_
nodrums_liverec_250ms_1m_sim_3

Music 3

dev1_nodrums_liverec_250ms_1m_sim_2/dev1_
nodrums_liverec_250ms_1m_sim_3

Music 4

dev1_wdrums_liverec_250ms_1m_sim_1/dev1_
nodrums_liverec_250ms_1m_sim_2

Music 5

dev1_wdrums_liverec_250ms_1m_sim_1/dev1_
nodrums_liverec_250ms_1m_sim_3

Music 6

dev1_wdrums_liverec_250ms_1m_sim_2/dev1_
nodrums_liverec_250ms_1m_sim_3

Music 7

dev2_nodrums_liverec_250ms_1m_sim_1/dev1_
nodrums_liverec_250ms_1m_sim_2

Music 8dev2_nodrums_liverec_250ms_1m_sim_1/dev1_
nodrums_liverec_250ms_1m_sim_3

Music 9dev2_nodrums_liverec_250ms_1m_sim_2/dev1_
nodrums_liverec_250ms_1m_sim_3

Music 10dev2_wdrums_liverec_250ms_1m_sim_1/dev1_
nodrums_liverec_250ms_1m_sim_2

Music 11

dev2_wdrums_liverec_250ms_1m_sim_1/dev1_
nodrums_liverec_250ms_1m_sim_3

Music 12dev2_wdrums_liverec_250ms_1m_sim_2/dev1_
nodrums_liverec_250ms_1m_sim_3

Speech 1

dev1_female4_liverec_250ms_1m_sim_1/dev1_
female4_liverec_250ms_1m_sim_2

Speech 2

dev1_female4_liverec_250ms_1m_sim_3/dev1_
female4_liverec_250ms_1m_sim_4

Speech 3

dev1_male4_liverec_250ms_1m_sim_1/dev1_
male4_liverec_250ms_1m_sim_2

Speech 4

dev1_male4_liverec_250ms_1m_sim_3/dev1_
male4_liverec_250ms_1m_sim_4

Speech 5

dev1_female4_liverec_250ms_1m_sim_1/dev1_
male4_liverec_250ms_1m_sim_2

Speech 6

dev2_female4_liverec_250ms_1m_sim_3/dev1_
male4_liverec_250ms_1m_sim_4

Speech 7

dev2_female4_liverec_250ms_1m_sim_1/dev1_
female4_liverec_250ms_1m_sim_2

Speech 8

dev2_female4_liverec_250ms_1m_sim_3/dev1_
female4_liverec_250ms_1m_sim_4

Speech 9

dev2_male4_liverec_250ms_1m_sim_1/dev1_
male4_liverec_250ms_1m_sim_2

Speech 10

dev2_male4_liverec_250ms_1m_sim_3/dev1_
male4_liverec_250ms_1m_sim_4

Speech 11

dev2_male4_liverec_250ms_1m_sim_1/dev1_
female4_liverec_250ms_1m_sim_2

Speech 12

dev2_male4_liverec_250ms_1m_sim_3/dev1_
female4_liverec_250ms_1m_sim_4

Kitamura and Yatabe EURASIP Journal on Advances in Signal Processing(2020) 2020:46

Page 16 of 35

Fig. 10 SDR improvements (left column), SIR improvements (center column), and SAR (right column) for live-recorded music and speech mixtures,
where STFT is performed using the 512-ms-long Hann window with 1/4 shifting. Top row shows the performances for music mixtures, and bottom
row shows the performances for speech mixtures

Appendix
Independence between real and imaginary parts ofspectrogram

The source generative model (15) assumes that the realand imaginary parts of a source in the time-frequency
domain are mutually independent because the generativemodel has a zero-mean and circularly symmetric shape in
the complex plane. The independence between real andimaginary parts or amplitude and phase has been investigated, but its validity may depend on the parametersof STFT. Independence can be measured by a symmetric
uncertainty coefficient [63–65]:
C(q1 , q2 ) = 2

H(q1 ) + H(q2 ) − H(q1 , q2 ),H(q1 ) + H(q2 )(35)

where q1 and q2 are random variables, H (q1 ) and H (q2 )are their entropy, and H(q1 , q2 ) is the joint entropy of q1
and q2 . Since the numerator of (35) corresponds to themutual information of q1 and q2 , the symmetric uncertainty coefficient can be interpreted as normalized mutual
information. When q1 and q2 are mutually independent,(35) becomes zero. In contrast, when q1 and q2 are completely dependent, (35) becomes one.
We calculated the symmetric uncertainty coefficient(35) between the real and imaginary parts of a timefrequency bin obtained by applying STFT to music or
speech sources. Let s be a complex-valued time-frequency
bin of a source (the indexes of frequency and time areomitted here). The independence between the real andimaginary parts can be measured by C(Re(s), Im(s)),

Kitamura and Yatabe EURASIP Journal on Advances in Signal Processing(2020) 2020:46

Page 17 of 35

Fig. 11 Symmetric uncertainty coefficient between real and imaginary parts for music and speech sources, where a Hann, b Hamming, or c
Blackman window is used in STFT. Left and right columns correspond to the music sources and speech sources, respectively

where Re(·) and Im(·) return the real and imaginaryparts of an input complex value, respectively. Here,
H(Re(s)), H(Im(s)), and H(Re(s), Im(s)) were approximately obtained by calculating the histograms of Re(s) andIm(s). The number of bins in the histograms was set to
10,000. We used the dry sources listed in Table 1: 15 music(instrumental) and eight speech sources. The parametersof STFT were the same as those in Section 4.
Figure 11 shows the symmetric uncertainty coefficients averaged over all bins and sources. Their values
C(Re(s), Im(s)) were almost zero for all STFT conditions
and source types (music or speech), and thus, the assumption of independence between real and imaginary parts
is valid for music and speech sources. This fact leads

to the generative model assumed in ILRMA. Note thatthose symmetric uncertainty coefficients validated the
independence of real and imaginary parts at each timefrequency bin. That is, the inter-bin relation is not considered here. The proposed method captures such interbin relations imposed by the spectrogram consistency,
which is not apparent in these bin-wise assessments of
independence.
Additional experimental results for synthesized mixtures

Figures 12–15, 16–21, and 22–27 show the SDR improvements, SIR improvements, and SAR, respectively, forsynthesized music and speech mixtures. These figurescorrespond to the results and discussions in Section 4.1.2.

Kitamura and Yatabe EURASIP Journal on Advances in Signal Processing(2020) 2020:46

Page 18 of 35

Fig. 12 Average SDR improvements for synthesized music mixtures (music 1–10) with a E2A and b JR2, where Hamming window is used in STFT

Kitamura and Yatabe EURASIP Journal on Advances in Signal Processing(2020) 2020:46

Page 19 of 35

Fig. 13 Average SDR improvements for synthesized music mixtures (music 1–10) with a E2A and b JR2, where Blackman window is used in STFT

Kitamura and Yatabe EURASIP Journal on Advances in Signal Processing(2020) 2020:46

Page 20 of 35

Fig. 14 Average SDR improvements for synthesized speech mixtures (speech 1–10) with a E2A and b JR2, where Hamming window is used in STFT

Kitamura and Yatabe EURASIP Journal on Advances in Signal Processing(2020) 2020:46

Page 21 of 35

Fig. 15 Average SDR improvements for synthesized speech mixtures (speech 1–10) with a E2A and b JR2, where Blackman window is used in STFT

Kitamura and Yatabe EURASIP Journal on Advances in Signal Processing(2020) 2020:46

Page 22 of 35

Fig. 16 Average SIR improvements for synthesized music mixtures (music 1–10) with a E2A and b JR2, where Hann window is used in STFT

Kitamura and Yatabe EURASIP Journal on Advances in Signal Processing(2020) 2020:46

Page 23 of 35

Fig. 17 Average SIR improvements for synthesized music mixtures (music 1–10) with a E2A and b JR2, where Hamming window is used in STFT

Kitamura and Yatabe EURASIP Journal on Advances in Signal Processing(2020) 2020:46

Page 24 of 35

Fig. 18 Average SIR improvements for synthesized music mixtures (music 1–10) with a E2A and b JR2, where Blackman window is used in STFT

Kitamura and Yatabe EURASIP Journal on Advances in Signal Processing(2020) 2020:46

Page 25 of 35

Fig. 19 Average SIR improvements for synthesized speech mixtures (speech 1–10) with a E2A and b JR2, where Hann window is used in STFT

Kitamura and Yatabe EURASIP Journal on Advances in Signal Processing(2020) 2020:46

Page 26 of 35

Fig. 20 Average SIR improvements for synthesized speech mixtures (speech 1–10) with a E2A and b JR2, where Hamming window is used in STFT

Kitamura and Yatabe EURASIP Journal on Advances in Signal Processing(2020) 2020:46

Page 27 of 35

Fig. 21 Average SIR improvements for synthesized speech mixtures (speech 1–10) with a E2A and b JR2, where Blackman window is used in STFT

Kitamura and Yatabe EURASIP Journal on Advances in Signal Processing(2020) 2020:46

Fig. 22 Average SAR for synthesized music mixtures (music 1–10) with a E2A and b JR2, where Hann window is used in STFT

Page 28 of 35

Kitamura and Yatabe EURASIP Journal on Advances in Signal Processing(2020) 2020:46

Fig. 23 Average SAR for synthesized music mixtures (music 1–10) with a E2A and b JR2, where Hamming window is used in STFT

Page 29 of 35

Kitamura and Yatabe EURASIP Journal on Advances in Signal Processing(2020) 2020:46

Fig. 24 Average SAR for synthesized music mixtures (music 1–10) with a E2A and b JR2, where Blackman window is used in STFT

Page 30 of 35

Kitamura and Yatabe EURASIP Journal on Advances in Signal Processing(2020) 2020:46

Fig. 25 Average SAR for synthesized speech mixtures (speech 1–10) with a E2A and b JR2, where Hann window is used in STFT

Page 31 of 35

Kitamura and Yatabe EURASIP Journal on Advances in Signal Processing(2020) 2020:46

Fig. 26 Average SAR for synthesized speech mixtures (speech 1–10) with a E2A and b JR2, where Hamming window is used in STFT

Page 32 of 35

Kitamura and Yatabe EURASIP Journal on Advances in Signal Processing(2020) 2020:46

Fig. 27 Average SAR for synthesized speech mixtures (speech 1–10) with a E2A and b JR2, where Blackman window is used in STFT

Page 33 of 35

Kitamura and Yatabe EURASIP Journal on Advances in Signal Processing(2020) 2020:46

Abbreviations
BSS: Blind source separation; ICA: Independent component analysis; STFT:
Short-time Fourier transform; FDICA: Frequency-domain independent
component analysis; IVA: Independent vector analysis; ILRMA: Independent
low-rank matrix analysis; NMF: Nonnegative matrix factorization; IS-NMF:
Nonnegative matrix factorization based on the Itakura–Saito divergence; SDR:
Source-to-distortion ratio; SIR: Source-to-interference ratio; SAR:
Source-to-artifact ratio

10. N. Ono, in Proc. WASPAA. Stable and fast update rules for independentvector analysis based on auxiliary function technique (IEEE, 2011),
pp. 189–192
11. D. Kitamura, N. Ono, H. Sawada, H. Kameoka, H. Saruwatari, Determinedblind source separation unifying independent vector analysis andnonnegative matrix factorization. IEEE/ACM Trans. ASLP. 24(9), 1626–1641(2016)
12. D. Kitamura, N. Ono, H. Sawada, H. Kameoka, H. Saruwatari, in Audio Source
Separation, ed. by S. Makino. Determined blind source separation withindependent low-rank matrix analysis (Springer, Cham, 2018), pp. 125–155
13. T. Tachikawa, K. Yatabe, Y. Oikawa, in Proc. IWAENC. Underdeterminedsource separation with simultaneous DOA estimation without initialvalue dependency (IEEE, 2018), pp. 161–165
14. D.D. Lee, H.S. Seung, Learning the parts of objects by non-negative matrix
factorization. Nature. 401(6755), 788–791 (1999)
15. D.D. Lee, H.S. Seung, in Proc. NIPS. Algorithms for non-negative matrix
factorization, (2000), pp. 556–562
16. C. Févotte, N. Bertin, J.-L. Durrieu, Nonnegative matrix factorization with
the Itakura-Saito divergence. With application to music analysis. NeuralComput. 21(3), 793–830 (2009)
17. Y. Mitsui, D. Kitamura, S. Takamichi, N. Ono, H. Saruwatari, in Proc. ICASSP.
Blind source separation based on independent low-rank matrix analysis
with sparse regularization for time-series activity (IEEE, 2017), pp. 21–25
18. H. Kagami, H. Kameoka, M. Yukawa, in Proc. ICASSP. Joint separation anddereverberation of reverberant mixtures with determined multichannel
non-negative matrix factorization (IEEE, 2018), pp. 31–35
19. R. Ikeshita, Y. Kawaguchi, in Proc. ICASSP. Independent low-rank matrix
analysis based on multivariate complex exponential power distribution(IEEE, 2018), pp. 741–745
20. D. Kitamura, S. Mogami, Y. Mitsui, N. Takamune, H. Saruwatari, N. Ono, Y.
Takahashi, K. Kondo, Generalized independent low-rank matrix analysis
using heavy-tailed distributions for blind source separation. EURASIP J.
Adv. Signal Process. 2018, 28 (2018)
21. K. Yoshii, K. Kitamura, Y. Bando, E. Nakamura, T. Kawahara, in EUSIPCO.
Independent low-rank tensor analysis for audio source separation (IEEE,
2018), pp. 1657–1661
22. R. Ikeshita, in EUSIPCO. Independent positive semidefinite tensor analysis
in blind source separation (IEEE, 2018), pp. 1652–1656
23. R. Ikeshita, N. Ito, T. Nakatani, H. Sawada, in WASPAA. Independent low-rankmatrix analysis with decorrelation learning (IEEE, 2019), pp. 288–292
24. N. Makishima, S. Mogami, N. Takamune, D. Kitamura, H. Sumino, S.
Takamichi, H. Saruwatari, N. Ono, Independent deeply learned matrix
analysis for determined audio source separation. IEEE/ACM Trans. ASLP.
27(10), 1601–1615 (2019)
25. K. Sekiguchi, Y. Bando, A.A. Nugraha, K. Yoshii, T. Kawahara,Semi-supervised multichannel speech enhancement with a deep speechprior. IEEE/ACM Trans. ASLP. 27(12), 2197–2212 (2019)26. S. Mogami, N. Takamune, D. Kitamura, H. Saruwatari, Y. Takahashi, K.
Kondo, N. Ono, Independent low-rank matrix analysis based ontime-variant sub-Gaussian source model for determined blind source
separation. IEEE/ACM Trans. ASLP. 28, 503–518 (2019)27. Y. Takahashi, D. Kitahara, K. Matsuura, A. Hirabayashi, in Proc. ICASSP.
Determined source separation using the sparsity of impulse responses(IEEE, 2020), pp. 686–690
28. M. Togami, in Proc. ICASSP. Multi-channel speech source separation anddereverberation with sequential integration of determined andunderdetermined models (IEEE, 2020), pp. 231–235
29. S. Kanoga, T. Hoshino, H. Asoh, Independent low-rank matrix
analysis-based automatic artifact reduction technique applied to three
BCI paradigms. Front. Hum. Neurosci. 14, 17 (2020)
30. D. Kitamura, N. Ono, H. Saruwatari, in Proc. EUSIPCO. Experimental analysis
of optimal window length for independent low-rank matrix analysis,(2017), pp. 1210–1214
31. Y. Liang, S.M. Naqvi, J. Chambers, Overcoming block permutation
problem in frequency domain blind source separation when using
AuxIVA algorithm. Electron. Lett. 48(8), 460–462 (2012)
32. K. Yatabe, Consistent ICA: determined BSS meets spectrogram
consistency. IEEE Signal Process. Lett. 27, 870–874 (2020)
33. T. Gerkmann, M. Krawczyk-Becker, J. Le Roux, Phase processing forsingle-channel speech enhancement: history and recent advances. IEEE
Signal Process. Mag. 32(2), 55–66 (2015)

Acknowledgements
The authors would like to thank Nao Toshima for his support on theexperiment. Also, the authors would like to thank the anonymous reviewers
for their valuable comments and suggestions that helped improve the qualityof this manuscript.
Authors’ contributions
DK derived the algorithm, performed the experiment, drafted the manuscriptfor initial submission, and revised the manuscript. KY proposed the main idea,
gave advice, mainly wrote the manuscript for initial submission, and corrected
the draft of revised manuscript. The authors read and approved the final
manuscript.
Funding
This work was partially supported by JSPS Grants-in-Aid for Scientific Research
19K20306 and 19H01116.
Availability of data and materials
The datasets used for the experiments in this paper are openly available: SiSEC
2011 (http://sisec2011.wiki.irisa.fr/) and RWCP-SSD (http://research.nii.ac.jp/
src/en/RWCP-SSD.html). Our MATLAB implementation of the proposed
method is also openly available at the following site: https://github.com/dkitamura/ILRMA/blob/master/consistentILRMA.m
Competing interests
The authors declare that they have no competing interests.
Author details
1 National Institute of Technology, Kagawa College, 355 Chokushi, Takamatsu,
Kagawa, 761-8058, Japan. 2 Waseda University, 3-4-1 Okubo, Shinjuku-ku,
Tokyo, 169-8555, Japan.
Received: 2 July 2020 Accepted: 29 October 2020

References
1. P. Comon, Independent component analysis, a new concept? Signal
Process. 36(3), 287–314 (1994)
2. P. Smaragdis, Blind separation of convolved mixtures in the frequency
domain. Neurocomputing. 22, 21–34 (1998)
3. S. Kurita, H. Saruwatari, S. Kajita, K. Takeda, F. Itakura, in Proc. ICASSP.
Evaluation of blind signal separation method using directivity patternunder reverberant conditions, vol. 5 (IEEE, 2000), pp. 3140–3143
4. N. Murata, S. Ikeda, A. Ziehe, An approach to blind source separation
based on temporal structure of speech signals. Neurocomputing.
41(1–4), 1–24 (2001)
5. H. Saruwatari, T. Kawamura, T. Nishikawa, A. Lee, K. Shikano, Blind source
separation based on a fast-convergence algorithm combining ICA andbeamforming. IEEE Trans. ASLP. 14(2), 666–678 (2006)
6. H. Sawada, R. Mukai, S. Araki, S. Makino, A robust and precise method forsolving the permutation problem of frequency-domain blind source
separation. IEEE Trans. SAP. 12(5), 530–538 (2004)
7. A. Hiroe, in Proc. ICA. Solution of permutation problem in frequency
domain ICA using multivariate probability density functions (Springer,
Berlin, Heidelberg, 2006), pp. 601–608
8. T. Kim, T. Eltoft, T.-W. Lee, in Proc. ICA. Independent vector analysis: anextension of ICA to multivariate components (Springer, Berlin,
Heidelberg, 2006), pp. 165–172
9. T. Kim, H.T. Attias, S.-Y. Lee, T.-W. Lee, Blind source separation exploiting
higher-order frequency dependencies. IEEE Trans. ASLP. 15(1), 70–79(2007)

Page 34 of 35

Kitamura and Yatabe EURASIP Journal on Advances in Signal Processing(2020) 2020:46

34. P. Mowlaee, R. Saeidi, Y. Stylianou, Advances in phase-aware signal
processing in speech communication. Speech Commun. 81, 1–29 (2016)35. P. Mowlaee, J. Kulmer, J. Stahl, F. Mayer, Single channel phase-aware signal
processing in speech communication: theory and practice. (Wiley, 2016)
36. K. Yatabe, Y. Oikawa, in Proc. ICASSP. Phase corrected total variation foraudio signals (IEEE, 2018), pp. 656–660
37. K. Yatabe, Y. Masuyama, Y. Oikawa, in Proc. IWAENC. Rectified linear unit
can assist Griffin–Lim phase recovery (IEEE, 2018), pp. 555–559
38. Y. Masuyama, K. Yatabe, Y. Oikawa, in Proc. IWAENC. Model-based phase
recovery of spectrograms via optimization on Riemannian manifolds(IEEE, 2018), pp. 126–130
39. Y. Masuyama, K. Yatabe, Y. Oikawa, Griffin–Lim like phase recovery via
alternating direction method of multipliers. IEEE Signal Process. Lett.
26(1), 184–188 (2019)40. Y. Masuyama, K. Yatabe, Y. Koizumi, Y. Oikawa, N. Harada, in Proc. ICASSP.
Deep Griffin–Lim iteration (IEEE, 2019), pp. 61–65
41. Y. Masuyama, K. Yatabe, Y. Oikawa, in Proc. ICASSP. Phase-aware
harmonic/percussive source separation via convex optimization (IEEE,
2019), pp. 985–989
42. Y. Masuyama, K. Yatabe, Y. Oikawa, in Proc. ICASSP. Low-rankness ofcomplex-valued spectrogram and its application to phase-aware audio
processing (IEEE, 2019), pp. 855–859
43. Y. Masuyama, K. Yatabe, Y. Koizumi, Y. Oikawa, N. Harada, in Proc. ICASSP.
Phase reconstruction based on recurrent phase unwrapping with deepneural networks (IEEE, 2020), pp. 826–830
44. J.L. Roux, H. Kameoka, N. Ono, S. Sagayama, in Proc. DAFx. Fast signal
reconstruction from magnitude STFT spectrogram based on spectrogram
consistency, (2010)45. J. Le Roux, E. Vincent, Consistent Wiener filtering for audio source
separation. IEEE Signal Process. Lett. 20(3), 217–220 (2013)
46. N. Perraudin, P. Balazs, P.L. Søndergaard, in Proc. WASPAA. A fast
Griffin–Lim algorithm (IEEE, 2013), pp. 1–4
47. K. Yatabe, Y. Masuyama, T. Kusano, Y. Oikawa, Representation of complex
spectrogram via phase conversion. Acoust. Sci. Tech. 40(3), 170–177(2019)
48. M. Kowalski, E. Vincent, R. Gribonval, Beyond the narrowband
approximation: wideband convex methods for under-determined
reverberant audio source separation. IEEE Trans. ASLP. 18(7), 1818–1829(2010)
49. K. Matsuoka, S. Nakashima, in Proc. ICA. Minimal distortion principle forblind source separation, (2001), pp. 722–727
50. K. Yatabe, D. Kitamura, in Proc. ICASSP. Determined blind source
separation via proximal splitting algorithm (IEEE, 2018), pp. 776–780
51. K. Yatabe, D. Kitamura, in Proc. ICASSP. Time-frequency-masking-based
determined BSS with application to sparse IVA (IEEE, 2019), pp. 715–719
52. K. Yatabe, D. Kitamura, Determined BSS based on time-frequency masking
and its application to harmonic vector analysis. arXiv:2004.14091 (2020)
53. M. Brandstein, D. Ward, Microphone arrays: signal processing techniques
and applications. (Springer Science & Business Media, 2013)
54. S. Araki, S. Makino, Y. Hinamoto, R. Mukai, T. Nishikawa, H. Saruwatari,
Equivalence between frequency-domain blind source separation andfrequency-domain adaptive beamforming for convolutive mixtures.
EURASIP J. Adv. Signal Process. 2003(11), 1157–1166 (2003)
55. D. Griffin, J. Lim, Signal estimation from modified short-time Fouriertransform. IEEE Trans. Acoust. Speech Signal Process. 32(2), 236–243(1984)
56. D. Gunawan, D. Sen, Iterative phase estimation for the synthesis ofseparated sources from single-channel mixtures. IEEE Signal Process. Lett.
17(5), 421–424 (2010)
57. N. Sturmel, L. Daudet, L. Girin, in Proc. DAFx. Phase-based informed source
separation of music, (2012)
58. M. Watanabe, P. Mowlaee, in Proc. INTERSPEECH. Iterative sinusoidal-based
partial phase reconstruction in single-channel source separation, (2013)
59. F. Mayer, D. Williamson, P. Mowlaee, D.L. Wang, Impact of phaseestimation on single-channel speech separation based ontime-frequency masking. J. Acoust. Soc. Am. 141, 4668–4679 (2017)60. S. Araki, F. Nesta, E. Vincent, Z. Koldovsky, G. Nolte, A. Ziehe, A. Benichoux,
in Proc. LVA/ICA. The 2011 signal separation evaluation campaign(SiSEC2011): -Audio source separation, (2012), pp. 414–422
61. S. Nakamura, K. Hiyane, F. Asano, T. Nishiura, T. Yamada, in Proc. LREC.
Acoustical sound database in real environments for sound scene
understanding and hands-free speech recognition, (2000), pp. 965–968

62. E. Vincent, R. Gribonval, C. Févotte, Performance measurement in blindaudio source separation. IEEE Trans. ASLP. 14(4), 1462–1469 (2006)
63. W.H. Press, S.A. Teukolsky, W.T. Vetterling, B.P. Flannery, Numerical Recipes
in C: The Art of Scientific Computing. (Cambridge University Press, New
York, 1992)
64. I. Andrianakis, P. White, Speech spectral amplitude estimators using
optimally shaped gamma and chi priors. Speech Comm. 51(1), 1–14
(2009)
65. P. Mowlaee, J. Stahl, Single-channel speech enhancement with correlated
spectral components: limits-potential. Speech Comm. 121, 58–69 (2020)

Page 35 of 35

Publisher’s NoteSpringer Nature remains neutral with regard to jurisdictional claims inpublished maps and institutional affiliations.