Column-wise update algorithm for independent deeply learned matrix
analysis
Naoki MAKISHIMA(1) , Norihiro TAKAMUNE(1) , Daichi KITAMURA(2) , Hiroshi SARUWATARI(1) , Yu TAKAHASHI(3) ,Kazunobu KONDO(3)(1) The University of Tokyo, Japan(2) National Institute of Technology, Kagawa College, Japan(3) Yamaha Corporation, Japan

Abstract
In this paper, we propose a robust demixing filter update algorithm for audio source separation. Audio source
separation is a task to recover source signals from multichannel mixtures observed in a microphone array, which
can be applied to, e.g., speech recognition and music signal analysis. Recently, independent deeply learned
matrix analysis (IDLMA) has been proposed as a state-of-the-art separation method. IDLMA utilizes deepneural network (DNN) inference of source models and blind estimation of demixing filters based on sources’
independence. In conventional IDLMA, iterative projection (IP) is exploited to estimate the demixing filters.
Although IP is a fast algorithm, when the specific source model is not accurate owing to the bad SNR condition,
the successive update of filters will fail hereafter. This is because IP updates the demixing filters in a source-wisemanner where only one source model is used for each update. In this paper, we derive a new microphone-wise
update rule which exploits all information of the source models simultaneously for each update. Moreover, we
propose a method to select the appropriate source- or microphone-wise update rule depending on the source
signal’s pseudo SNR estimated via the DNNs. Experimental results show the efficacy of the proposed method.
Keywords: Audio source separation, Deep neural network

1

INTRODUCTION

Audio source separation aims to recover source signals from the multichannel mixtures observed by a microphone array. Many types of algorithm have been proposed, e.g., unsupervised (blind) methods [1–17] andsupervised (informed) methods [18–22]. Independent deeply learned matrix analysis (IDLMA) [23, 24] is a stateof-the-art separation method combining blind estimation of the demixing matrix and deep neural network (DNN)
inference of source models. In the conventional source-separation methods including IDLMA, a fast and stable
coordinate descent algorithm called iterative projection (IP) [5, 25] is exploited to estimate the demixing matrix.
IP updates the demixing matrix in a source-wise (row-wise) manner where only one source model is used foreach update. Although IP is a fast algorithm, our preliminary experiments show that it sometimes fails to find agood solution. This is because when the specific source model is not accurate owing to the bad SNR condition,
the successive update of filters will fail hereafter. In this paper, we propose a new microphone-wise (columnwise) update rule of the demixing matrix, which exploits all information of the source models simultaneously
for each update. Since IP and the proposed column-wise update rule are based on the coordinate-descent-type
iterative algorithm, the update order exists and it is arbitrary. In general, different update rule or update orderleads to a different solution because of local minima in the cost function, and it is difficult to choose the mostappropriate update strategy (here the “strategy” includes a type of row/column-wise update rule, update order,and how to switch them in the iterative optimization). In order to find the appropriate update strategy, we
propose an automatic selection method for the update strategy depending on the source signal’s pseudo SNR
estimated via the DNNs.

2805

2

CONVENTIONAL METHODS

2.1 Formulation
Let M and N be the numbers of microphones and sound sources, respectively. We assume a determined casewhere M = N. The short-time Fourier transform (STFT) of the observed mixtures, estimated signals, source
signals are defined as
xi j = (xi j1 , . . . , xi jN )T ,(1)

Tyi j = (yi j1 , . . . , yi jN ) ,

(2)

T

(3)

si j = (si j1 , . . . , si jN ) ,where i and j denote the indexes of frequency bins and time frames, respectively, and T denotes the transpose.
In the determined case, the estimated signals yi j can be represented as
yi j = Wi xi j ,(4)

H is the demixing filter for the nth source, and H
where Wi = (wi1 , . . . , wiN )H is the demixing matrix and win
denotes the Hermitian transpose.

2.2 Generative Model and Cost Function
In IDLMA, the following univariate complex Gaussian distribution is assumed as a source generative model:()|yi jn |2
1
p(y
)
=
(5)
exp
−
,
∏ i jn ∏ π ri jn 2
ri jn 2i, ji, jwhere ri jn is the scale parameter of a Gaussian distribution. Under the model (5), the cost function to beminimized w.r.t. Wi and ri jn (negative log-likelihood of xi j = Wi−1 yi j ) is obtained as
(
)
1
c
H
(6)
L = ∑ ∑ win
Qin win − log |detWi |2 + ∑ log ri2jn ,J i, j,n
n
i
Qin =

H
1 xi j xi j.
2J∑j ri jn(7)

We denote the scale parameter matrix as Rn ∈ RI×J
≥0 , whose elements are ri jn . The aim of IDLMA is to blindly
estimate Wi only from the observed mixtures with the assistance of a DNN. The overview of separation processof IDLMA is shown in Fig. 1.
2.3 Update Rule of Demixing Matrix by IP
In [5, 25], a fast and convergence-guaranteed algorithm called IP was proposed, which can be applied to thesum of a negative log-determinant and a quadratic form. By applying IP to the first and second terms of theright side in (6), the update rule of demixing matrix Wi is obtained as follows:
win ← (Wi Qin )−1 en ,win
win ← √
,
HQ w
win
in in

(8)
(9)

where en denotes the unit vector with the nth element equal to unity. Note that since the update order w.r.t. n
is arbitrary, we have to select the appropriate update order from N! options.

2806

Frequency
Frequency

Demixing
matrixFrequency
Frequency

Estimated Scale parameter estimation by DNN source model
signalObserved
signalSTFT

Updateso that estimated signals
are mutually independent and obey
DNN source model.

Time

Time

Figure 1. Overview of separation process in IDLMA.
2.4 Update Rule of Scale Parameter Matrix by DNN
DNNn is pre-trained so that the scale parameter |Rn |.1 is predicted from an input mixture spectrogram |X̃|.1 ,where X̃ ∈ CI×J is a mixture spectrogram in the training data and | · |.p for matrices denotes the element-wiseabsolute and pth-power operations. In inference for open data, the scale parameter matrix is updated by the
pre-trained DNNn as|Rn |.1 ← DNNn (|Yn |.1 ),(10)

ri jn ← max(ri jn , ε ),(11)

where Yn ∈ CI×J is the spectrogram of the estimated signal whose elements are yi jn , temporally obtained throughthe update of Wi and ε is a small value to increase the numerical stability in IP.
When we define the output scale parameter matrix as Dn = DNNn (|X̃|.1 ) ≈ Rn and source spectrogram asS̃n ∈ CI×J , the loss function of DNNn in (10) can be defined as|s̃i jn |2 + δ1|s̃i jn |2 + δ1
− log
− 1,
2
di jn 2 + δ1
i, j di jn + δ1

L(Dn ) = ∑(12)

where s̃i jn and di jn are the elements of S̃n and Dn , respectively, and δ1 is a small value to avoid division by
zero [20]. Since minimizing (12) corresponds to a simulation for the ML estimation of ri jn in (6) (only limited
to the training data), DNNn can be approximately interpreted as an appropriate source model based on (5). In
making the mixture spectrogram |X̃|.1 , source spectrograms are mixed with various SNR as described in [23].

3

PROPOSED METHODS

3.1 Proposed Update Rule of Demixing Matrix
In [23], demixing matrix Wi is updated by (8) and (9). IP updates win based on given source model Rn
and win′ (n′ ̸= n) for each n, which corresponds to a source-wise update of Wi . Therefore, updating win withaccurate Rn first leads to a good solution and the update order w.r.t. n is important.
In this paper, we propose a column-wise update rule, which corresponds to a microphone-wise update of Wi .
Figure 2 illustrates a difference between the conventional IP and the proposed column-wise update algorithm.
We denote the column vector w̃im of Wi as Wi = (w̃i1 , . . . , w̃iN ), where w̃im is a microphone-wise vector
although win is a source-wise (row) vector. We pick out the partial cost function J w.r.t. demixing matrix Wifrom (6), which is rewritten as follows w.r.t. w̃im :)(H
Qin win − log |detWi |2
J = ∑ ∑ win
i

n

( H
)
H
H
2
= ∑ w̃im
Q̃im w̃im + w̃imh̃im + h̃Him w̃im − log |bim w̃im | ,
c

(13)

i

2807

(a)

(b)
update

fix

…

…

…

…

fix

updateupdate

fix

…

…
fix

updateProposed column- wise update

Conventional row-wise update

Figure 2. (a) Conventional row-wise update of Wi . (b) Proposed column-wise update of Wi .
where
Q̃im = diag(Qi1mm , . . . , QiNmm ),(h̃im =w∗i1m′ Qi1m′ m , . . . , ∑ w∗iNm′ QiNm′ m
∑
′
′

m ̸=m

(14)

)T
.

(15)

m ̸=m

Here Qinm′ m is the (m′ , m)th element of Qin , winm′ is the m′ th element of win , diag(Qi1mm , . . . , QiNmm ) is the
diagonal matrix whose elements are Qi1mm , . . . , QiNmm , bH
im is the mth row vector of the adjugate matrix of Wi ,and ∗ denotes the complex conjugate. Since IP can not minimize the cost function that includes the linear form,
we derive a new coordinate descent algorithm for (13). The derivative of (13) is obtained as

∂J
bim
.
= Q̃im w̃im + h̃im − H∗∂ w̃imw̃im bim(16)∗ = 0, the stationary point is given in [26] asFrom ∂ J /∂ w̃imuim ← (Q̃im WiH )−1 em ,(17)ûim ← Q̃−1
im h̃im ,H
aim ← uim Q̃im uim ,âim ← uH
im Q̃im ûim ,

(18)
(19)


 √uim − ûima]√w̃im ← âimim [ 2a 1 − 1 + 4aim2 uim − ûim
|â |
im
im

(20)
(âim = 0),(âim ̸= 0).

(21)

Similarly to IP, we have to select the appropriate update order from N! options.
Although we obtain two update rules (row-wise or column-wise), each of which has N! update order, it isdifficult to determine the most appropriate update strategy in advance. In order to select the appropriate updatestrategy that derives high separation performance, we propose the following pseudo-SNR-related criterion ζ :ζ=.1∑i, j {|DNNn (|Yn | )|.2 }i j
1
,
.1 .2
N∑
n ∑l ∑i, j {|DNNl (|Yn | )| }i j

(22)where {}i j denotes the (i, j)th element of the matrix. Since DNNn is trained to output the scale parameter ofthe nth source, the numerator of ζ represents the nth source’s power in Yn and the denominator of ζ represents
the sum of all the sources’ power, which means that ζ indicates the separation accuracy of Yn . We propose thefollowing procedure to select the update strategy when the demixing matrix is updated.
Step 1: Make 2N! copies of Wi .
Step 2: Each Wi is updated by each update rule ((8)–(9) or (17)–(21)) and their update order (N! options for
n).

2808

Vo.

Ba.
2m

T60 = 300 ms

40

40

5.66 cm

Figure 3. Recording conditions of impulse responses obtained from RWCP database.
Step 3: ζ is calculated via (4) and (22) for each Wi .
Step 4: Wi with the highest ζ is taken over to the next iteration.

4

EXPERIMENTAL EVALUATION

We confirmed the validity of the proposed update strategy selection by conducting a music source separation
task. We compared six methods: multichannel nonnegative matrix factorization (MNMF) [7], independent lowrank matrix analysis (ILRMA) [9], DNN+WF [27], Duong+DNN [20], conventional row-wise update IDLMA(Row IDLMA), and proposed IDLMA exploiting both row-wise and column-wise updates based on DNN selection (Row+Column IDLMA). MNMF and Duong+DNN estimate the mixing system by the EM algorithm,
where MNMF estimates the source model blindly and Duong+DNN estimates it using DNN. ILRMA is a blind
method that estimates the demixing system by IP. DNN+WF applies a Wiener filter constructed using all the
outputs of the DNN source models to the reference channel signal. Note that MNMF and ILRMA are “blind(unsupervised)” techniques, but we show their performances just for reference to understand to what extent thesupervised methods (DNN+WF, Duong+DNN, Row IDLMA, and Row+Column IDLMA) can improve the performance by using pretraining data. For all methods except DNN+WF, we updated the demixing matrix 200
times. In Duong+DNN, Row IDLMA, and Row+Column IDLMA, the scale parameter matrix Rn was updated
by DNNn after every 10 iterations of the spatial parameter optimization. We treated 10 iterations of the spatial
parameter optimization as one update strategy and DNN selection of update strategy is done every time after
Rn is updated. In Row IDLMA, we averaged the separation performance of all the row-wise update order.
We used the DSD100 dataset of SiSEC2016 [28] as the dry sources and the training dataset of DNN. The 50
songs in the dev data were used to train DNNn and the top 25 songs in alphabetical order in the test data
were used for performance evaluation. The test songs were trimmed only in the interval of 30 to 60 s. To
simulate reverberant mixtures, we produced two-channel observed signals by convoluting the impulse response
E2A (T60 = 300 ms) obtained from the RWCP database [29] with each source, and mixtures of bass and vocal
were created. The recording conditions of E2A is shown in Fig. 3. All the signals were downsampled to
8 kHz. An STFT was performed using a 512-ms-long Hamming window with a 256-ms-long shift. We used thesignal-to-distortion ratio (SDR) [30] to evaluate the total separation performance.
In this paper, the number of hidden layers in the constructed fully connected DNN was set to four. Eachlayer had 1024 units, and a rectified linear unit was used for the output of each layer. To optimize the DNN,
we added the term (λ /2) ∑q gq 2 to (12) for regularization, where gq is the weight coefficient in DNN, and
ADADELTA [31] with a 128-size minibatch was performed for 2000 epochs. The parameter ε was experimentally optimized and set to 0.1 × (IJ)−1 ∑i, j σi jn . The other parameters were set to δ1 = δ2 = 10−5 and λ = 10−5 .
Figure 4 shows the average SDR improvement for bass/vocal separation. The proposed Row+Column IDLMA
achieves the best SDR improvement among all the state-of-the-art blind and supervised methods. In particular,
the proposed Row+Column IDLMA increases the SDR improvement from Row IDLMA by 1.5 dB, which

2809

Blind (Ref.)

Supervised

Figure 4. Average SDR improvement of 25 bass/vocal songs.

Row (ascend)

Row (descend)

Col (ascend)

Col (descend)

Figure 5. Histogram of selected update rule and update order in Row+Col IDLMA
indicates that DNN selects the appropriate update strategy. Figure 5 shows the histogram of the update rule andorder selected in Row+Column IDLMA. In Fig. 5, “Row” or “Col” is the type of update rule and “ascend” or“descend” is the update order. Here “ascend” means that the demixing filter corresponding to Vo. was updated
first in “Row” and the vector corresponding to the right microphone in Fig. 3 was updated first in “Col”. It
is confirmed that the column-wise update is chosen 2.8 times as much as the row-wise update, which indicatesthat the column-wise update often leads to a better solution than IP in IDLMA.

5

CONCLUSIONSIn this paper, we proposed a new microphone-wise update algorithm for demixing matrix. Moreover we proposed a method to select the appropriate row- or column-wise update rule depending on the source signal’s
pseudo SNR estimated via the DNNs. Experimental results showed that the update strategy selection improves
the separation performance, which exploits the proposed column-wise update more frequently than the conventional row-wise update.

2810

ACKNOWLEDGEMENTS
This work was partly supported by SECOM Science and Technology Foundation and JSPS KAKENHI Grant
Numbers JP17H06101, JP19H01116, and JP19K20306.

REFERENCES[1] S. Araki, R. Mukai, S. Makino, T. Nishikawa, and H. Saruwatari. The fundamental limitation of frequency
domain blind source separation for convolutive mixtures of speech. IEEE Trans. SAP, 11(2):109–116, 2003.
[2] H. Saruwatari, T. Kawamura, T. Nishikawa, A. Lee, and K. Shikano. Blind source separation based on afast-convergence algorithm combining ICA and beamforming. IEEE Trans. ASLP, 14(2):666–678, 2006.
[3] A. Hiroe. Solution of permutation problem in frequency domain ica, using multivariate probability density
functions. In Independent Component Analysis and Blind Signal Separation, pages 601–608, 2006.
[4] T. Kim, H. T. Attias, S.-Y. Lee, and T.-W. Lee. Blind source separation exploiting higher-order frequency
dependencies. IEEE Trans. ASLP, 15(1):70–79, 2007.
[5] N. Ono. Stable and fast update rules for independent vector analysis based on auxiliary function technique.
In Proc. WASPAA, pages 189–192, 2011.
[6] A. Ozerov and C. Févotte. Multichannel nonnegative matrix factorization in convolutive mixtures for audio
source separation. IEEE Trans. ASLP, 18(3):550–563, 2010.
[7] H. Sawada, H. Kameoka, S. Araki, and N. Ueda. Multichannel extensions of non-negative matrix factorization with complex-valued data. IEEE Trans. ASLP, 21(5):971–982, 2013.
[8] H. Kameoka, T. Yoshioka, M. Hamamura, J. L. Roux, and K. Kashino. Statistical model of speech signals
based on composite autoregressive system with application to blind source separation. In Proc. LVA/ICA,
pages 245–253, 2010.
[9] D. Kitamura, N. Ono, H. Sawada, H. Kameoka, and H. Saruwatari. Determined blind source separation unifying independent vector analysis and nonnegative matrix factorization. IEEE/ACM Trans. ASLP,
14(9):1626–1641, 2016.
[10] Y. Mitsui, D. Kitamura, S. Takamichi, N. Ono, and H. Saruwatari. Blind source separation based on
independent low-rank matrix analysis with sparse regularization for time-series activity. In Proc. ICASSP,
pages 21–25, 2017.
[11] Y. Mitsui, D. Kitamura, N. Takamune, H. Saruwatari, Y. Takahashi, and K. Kondo. Independent low-rankmatrix analysis based on parametric majorization-equalization algorithm. In Proc. CAMSAP, pages 1–5,
2017.
[12] D. Kitamura, N. Ono, H. Sawada, H. Kameoka, and H. Saruwatari. Determined blind source separation
with independent low-rank matrix analysis. In Shoji Makino, editor, Audio Source Separation, chapter 6,
pages 125–155. Springer, Cham, 2018.
[13] D. Kitamura, S. Mogami, Y. Mitsui, N. Takamune, H. Saruwatari, N. Ono, Y. Takahashi, and K. Kondo.
Generalized independent low-rank matrix analysis using heavy-tailed distributions for blind source separation. EURASIP J. Adv. Signal Process., 2018(28):1–25, 2018.
[14] R. Ikeshita and Y. Kawaguchi. Independent low-rank matrix analysis based on multivariate complex exponential power distribution. In Proc. ICASSP, pages 741–745, 2018.

2811

[15] S. Mogami, N. Takamune, D. Kitamura, H. Saruwatari, Y. Takahashi, K. Kondo, H. Nakajima, and N. Ono.
Independent low-rank matrix analysis based on time-variant sub-Gaussian source model. In Proc. APSIPA,
pages 1684–1691, 2018.
[16] H. Sawada, N. Ono, H. Kameoka, D. Kitamura, and H. Saruwatari. A review of blind source separation
methods: two converging routes to ILRMA originating from ICA and NMF. APSIPA Trans. Signal. Info.
Process, 8:e12, 2019.
[17] S. Mogami, Y. Mitsui, N. Takamune, D. Kitamura, H. Saruwatari, Y. Takahashi, K. Kondo, H. Nakajima,and H. Kameoka. Independent low-rank matrix analysis based on generalized Kullback-Leibler divergence.
IEICE Trans. FECCS, E102-A(2):458–463, 2019.
[18] D. Kitamura, H. Saruwatari, H. Kameoka, Y. Takahashi, K. Kondo, and S. Nakamura. Multichannel signal
separation combining directional clustering and nonnegative matrix factorization with spectrogram restoration. IEEE/ACM Trans. ASLP, 23(4):654–669, 2015.
[19] S. Araki, T. Hayashi, M. Delcroix, M. Fujimoto, K. Takeda, and T. Nakatani. Exploring multi-channel
features for denoising-autoencoder-based speech enhancement. In Proc. ICASSP, pages 116–120, 2015.
[20] A. A. Nugraha, A. Liutkus, and E. Vincent. Multichannel audio source separation with deep neural networks. IEEE/ACM Trans. ASLP, 24(9):1652–1664, 2016.
[21] Y.-H. Tu, J. Du, L. Sun, and C.-H. Lee. LSTM-based iterative mask estimation and post-processing for
multi-channel speech enhancement. In Proc. APSIPA, 2017.
[22] K. Qian, Y. Zhang, S. Chang, X. Yang, D. Florencio, and M. Hasegawa-Johnson. Deep learning based
speech beamforming. In Proc. ICASSP, pages 5389–5393, 2018.
[23] S. Mogami, H. Sumino, D. Kitamura, N. Takamune, S. Takamichi, H. Saruwatari, and N. Ono. Independent
deeply learned matrix analysis for multichannel audio source separation. In Proc. EUSIPCO, pages 1571–
1575, 2018.
[24] N. Makishima, N. Takamune, D. Kitamura, H. Saruwatari, Y. Takahashi, K. Kondo, and H. Nakajima.
Generalized-Gaussian-distribution-based independent deeply learned matrix analysis for multichannel audio
source separation. In Proc. INTERNOISE, number 1260, 2019 in press.
[25] N. Ono and S. Miyabe. Auxiliary-function-based independent component analysis for super-Gaussian
sources. In Proc. Int. Conf. Latent Variable Anal. Signal Separation, 2010.
[26] Y. Mitsui, N. Takamune, D. Kitamura, H. Saruwatari, Y. Takahashi, and K. Kondo. Vectorwise coordinate
descent algorithm for spatially regularized independent low-rank matrix analysis. In Proc. ICASSP, pages
746–750, 2018.
[27] S. Uhlich, F. Giron, and Y. Mitsufuji. Deep neural network based instrument extraction from music. In
Proc. ICASSP, pages 2135–2139, 2015.
[28] A. Liutkus, F.-R. Stöter, Z. Rafii, D. Kitamura, B. Rivet, N. Ito, N. Ono, and J. Fontecave. The 2016
signal separation evaluation campaign. In Proc. LVA/ICA, pages 323–332, 2012.
[29] S. Nakamura, K. Hiyane, F. Asano, T. Nishimura, and T. Yamada. Acoustical sound database in realenvironments for sound scene understanding and hands-free speech recognition. In Proc. LREC, pages
965–968, 2000.
[30] E. Vincent, R. Gribonval, and C. Févotte. Performance measurement in blind audio source separation.
IEEE Trans. ASLP, 14(4):1462–1469, 2006.
[31] M. D. Zeiler. Adadelta: An adaptive learning rate method. CoRR, abs/1212.5701, 2012.

2812