Proceedings of 2022 APSIPA Annual Summit and Conference

7-10 November 2022, Chiang Mai, Thailand

DNN-Based Frequency-Domain Permutation Solver
for Multichannel Audio Source Separation
Fumiya Hasuike∗ , Daichi Kitamura∗ , and Rui Watanabe†∗ National Institute of Technology, Kagawa College, Kagawa, Japan

I. I NTRODUCTION
Multichannel audio source separation is a technique for
estimating source signals from an observed multichannel mixture signal. The methods that do not require a priori spatial
information, e.g., locations of sources and microphones, arecalled blind source separation (BSS) [1]. For a determinedsituation (in which the numbers of microphones and sources
are the same), independent component analysis (ICA) [2] andits variants have widely been utilized in BSS tasks.
A typical approach for audio BSS is frequency-domain ICA(FDICA) [3]. In FDICA, the observed multichannel signal
is converted to the time-frequency domain by a short-time
Fourier transform (STFT). Then ICA is independently applied
to each frequency to estimate the separated source components.
However, using FDICA leads to the permutation problem
which will be addressed in this paper. Because ICA has permutation indeterminacy of estimated signals, the source orderof ICA outputs depends on the initial values of parameters.
For this reason, the estimated source components of FDICA
are not aligned along the frequencies, as shown in Fig. 1.
This is known as the permutation problem, which often arises
even in underdetermined BSS, e.g., full-rank spatial covarianceanalysis [4].
Various permutation solvers with hand-crafted criteria havebeen studied in the context of FDICA [5], [6], [7], [8]. Subsequently, the mainstream approach for BSS shifted to developing a simultaneous solution of frequency-wise separation andthe permutation alignment. In these methods, to avoid the permutation problem, the time-frequency structure of each source
is assumed as a source model, which is then combined with
FDICA. For example, independent vector analysis (IVA) [9],[10], [11], [12] assumes group sparsity in the spectrogram of
978-616-590-477-3 ©2022 APSIPA

Separated signal 1…

Estimated signal 1

Fr

Sourcesignal 1

Observed
signal 1

ICA
Sourcesignal 2

Observed
signal 2

Estimated signal 2 Permutation
Separated signal 2Freq.

Abstract—This paper focuses on frequency-domain blindsource separation (BSS) for audio signals. This technique estimates frequency-wise source components from an observed spectrogram. Full-rank spatial covariance analysis and frequencydomain independent component analysis are algorithms commonly used for this task. Using these methods, however, results
in an alignment problem of frequency-wise permutations of theestimated source components. This is known as the permutation
problem, which has been addressed for decades and requires
a robust and precise permutation solver post-processing. We
introduce a permutation solver that uses a deep neural networkand predicts the correct source permutations in each frequency.
The experimental results demonstrate the validity of the proposed
approach and its robustness against the domain of the dataset.

equb i n ency

† Japan Advanced Institute of Science and Technology, Ishikawa, Japan

SolverTime

All frequency
components

Fig. 1. Permutation problem in FDICA (N = M = 2).

each source as the source model. Independent low-rank matrix
analysis (ILRMA) [13], [14] assumes that sources inherently
have a low-rank time-frequency structure and models themby nonnegative matrix factorization [15]. Independent deeply
learned matrix analysis [16] utilizes a deep neural network(DNN) that is trained by using specific audio datasets. The
source model based on time-frequency masks has also beenutilized [17], [18]. These methods provide more accurate BSS
when the source model fits well with each of the sources in
the observed signal. However, in [19], we found that FDICA
with an ideal permutation solver (using oracle source signals)
significantly outperforms IVA and ILRMA. This suggeststhat FDICA has the potential to achieve high-quality BSS,
and the remaining task is to solve the permutation problem.
The challenge is building a universal source model that fitsvarious sources (speech signals, vocals, musical instruments,
background noise, etc.).
To directly obtain such a versatile model, DNN-based supervised approaches with a large dataset and extensive training
have shown potential for speech enhancement and audio source
separation. Various methods have been proposed, such as thosein Ref. [20], [21], [22]. In these techniques, the size andversatility of the training data are crucial for obtaining theuniversal source model and improving separation performance,
but collecting and producing such training datasets is costly.
Thus, BSS (without model training) or a separation techniquewith few-shot learning is still important.
We previously proposed an approach that utilizes a DNN tosolve the permutation problem rather than building versatilesource models [23]. This approach only requires a few-shotaudio dataset because the permutation problem can easily besimulated by randomly shuffling the frequency components of
the audio spectrogram. In the experimental section, we will

872

APSIPA ASC 2022

Proceedings of 2022 APSIPA Annual Summit and Conference

7-10 November 2022, Chiang Mai, Thailand

show that the permutation problem can be mostly solved byusing training data that are produced from only two 11-s-longaudio signals. This is an advantage of the permutation solver
over various DNN-based supervised approaches.
In our previous method, the DNN was trained to predictwhether the source components of the reference frequency andthe other frequencies belong to the same source as a binaryclassification, where the source components are estimated by
FDICA. Because predictions are performed for each subband,
implementing the permutation solver based on the predicted
results becomes a complicated process, particularly for threeor more sources. In addition, the performance of the method
depends on the type of sources used in the training dataset.
In this paper, to solve the above-mentioned problems, we
propose a simple algorithm for the DNN-based permutation
solver, which can easily extend to three or more sources. We
demonstrate its performance and robustness against the dataset
domain to evaluate its versatility.

estimates the frequency-wise demixing matrix Wi over all thefrequencies.

II. FDICA AND P ERMUTATION P ROBLEM
A. Definitions of Signals
Let the source and observed signals in the time-frequency
domain be
sij = [sij1 , sij2 , · · · , sijn , · · · , sijN ]T∈ CN ,
T

xij = [xij1 , xij2 , · · · , xijm , · · · , xijM ]

M

∈C ,

(1)
(2)

respectively, where i = 1, 2, · · · , I, j = 1, 2, · · · , J, n =1, 2, · · · , N , and m = 1, 2, · · · , M are the indices of frequency, time, source, and channel, respectively. Also, let theestimated signal obtained by FDICA be

C. Permutation Problem
Because ICA has indeterminacy of scale and permutation of
the estimated signals, the demixing matrix obtained by FDICA,
Ŵi ∈ CN ×M , is represented as
Ŵi = Di Pi Wi ,(6)

where Di ∈ RN ×N is a diagonal matrix that may change thescale of (zij )Jj=1 and Pi ∈ {0, 1}N ×N is a permutation matrix
that may replace the source order of (zij )Jj=1 . For example,




1 0
0 1
Pi =
or
(for N = 2),
(7)
0 1
1 0

 
 
 

1 0 0
1 0 0
0 1 0
0 1 0
Pi = 0 1 0 , 0 0 1 , 1 0 0 , 0 0 1 ,
0 0 1
0 1 0
0 0 1
1 0 0




0 0 1
0 0 1
1 0 0 , or 0 1 0 (for N = 3).
(8)
0 1 0
1 0 0
The permutation matrix is a doubly stochastic matrix(DSM) [25], [26] because the summations of rows and columns
are all unity.
As a result, the estimated signal of FDICA,T
yij = Ŵi xij = yij1 , yij2 , · · · , yijn′i · · · , yijN
∈ CN ,(9)

FDICA [3] assumes the frequency-wise mixing system [19],[24]:

includes inconsistent scales and permutations along the frequencies, where n′i = 1, 2, · · · , N is a new source index torepresent the permutation ambiguity in each frequency. The
scale ambiguity caused by Di can be easily recovered byapplying a projection-back technique [27]. However, it is noteasy to align the source permutations along the frequencies,
which is the permutation problem. Fig. 1 shows a schematicof the problem, where Yn′ ∈ CI×J is the n′ th spectrogram ofthe estimated signal yij (including the permutation problem).
The ideal permutation solver is defined as zij =Pi−1 Di−1 yij . Thus, solving this problem is interpreted as
an estimation of Pi−1 over all frequencies. Strictly speaking,however, even if the source permutations are aligned correctly,
the indeterminacy of permutation of entire frequency components (i.e., the sources) remains. Thus, the separated signal isrepresented as

xij = Ai sij ,zij = Pall Pi−1 Di−1 yij ,zij = [zij1 , zij2 , · · · , zijn′ , · · · , zijN ]T∈ CN ,(3)

where the indices n and n′ are used properly to represent the
ambiguity of a source permutation, e.g., the estimated signalof sij1 could be zij2 . As we only focus on the determinedsituation, N = M is assumed throughout the paper. In
addition, we also define time-frequency matrices (complexvalued spectrograms) of the nth source, the mth observed,
and the n′ th estimated signals as Sn ∈ CI×J , Xm ∈ CI×J ,and Zn′ ∈ CI×J , respectively.
B. BSS Formulation and FDICA

(4)

M ×N

where Ai ∈ C
is a time-invariant frequency-wise mixing
matrix. When the mixing matrix Ai is nonsingular, there exists
a frequency-wise demixing matrix Wi = A−1∈ CN ×M .
iWith this ideal demixing matrix, the estimated signal can beobtained aszij = Wi xij .

(5)Thus, FDICA independently applies ICA to the complexvalued time-series signals of each frequency, (xij )Jj=1 , and

(10)

N ×N

where Pall ∈ {0, 1}
is a frequency-independent permuta−1
tion matrix. The estimation of Pallis out of the scope of thispaper.
III. P ROPOSED METHOD
A. Motivation
As described in Sect. I and [23], the DNN-based permutation
solver directly assists FDICA-based BSS rather than the source
model tailored to the specific source types. In addition, the873

B. Input and Output of DNN
FDICA outputs spectrograms of the estimated signal,(Yn′ )Nn′ =1 , including the permutation problem. As a preprocess, normalized power spectrograms are calculated as|Yn′ |.2

Y n′ = PN

n′ =1 |Yn |′.2∈ [0, 1]

I×J

,

(11)

where | · |.2 for matrices denotes an element-wise absoluteand square operation. This normalization stabilizes the trainingof the DNN and enhances correlations between the same
source components [6]. Then, we extract a temporally local
spectrogram from (Y n′ )N
n′ =1 centered at time j asY̌jn′ = [y (j−β)n′ y (j−β+1)n′ · · · y (j+β)n′ ] ∈ [0, 1]I×(2β+1) ,(12)
where y jn′ ∈ [0, 1]I is the jth column vector of Y n′ .
β > 0 is a hyperparameter that determines the duration of
the local spectrogram Y̌jn′ . The input of the proposed DNN
is a flattened vector of (Y̌jn′ )N
n′ =1 :
T
dj = vec(Y̌j1 )T , · · · , vec(Y̌jN )T ∈ [0, 1]N I(2β+1) , (13)where vec(·) denotes the vectorization of an input matrix.
As shown in Fig. 2, the DNN’s prediction can be definedasL̂j = DNN(dj ) ∈ [0, 1]I×N ! ,(14)

where L̂j is a matrix that consists of the probabilities ˆliqj forthe permutation matrices, and q = 1, 2, · · · , N ! is the index
of the permutations. For example, when N = 2, the predicted

DNN

Frequency

Frequency

Frequency

OutputInput

Convert topermutation
matrix
0.1

0.1
0.9

0.9
1.0

0.9
0.1
0.0
0.9

0.9 0.1
0.7
0.9 0.9
0.1

0.1

・
・
・

0.9
0.5
0.0

0.1
0.5

Time

0.0

1.0

1.0 0.0
1.0

TimeFig. 2. Calculation of predicted permutation matrix for N = 2 case.
Input vector
Output layer
output layers

Output layer

ReLU

3rd dense layer

ReLU

2nd dense layer

ReLU

Three hidden layers

1st dense layer

training data (signals with the permutation problem) can easily
be produced by randomly shuffling the frequency components
of source signals, which enables few-shot learning. For thesereasons, this paper only focuses on developing a simple, robust,and precise DNN-based permutation solver.
The conventional DNN-based permutation solver [23] trainsa DNN that solves the permutation problem within a specific subband and applies the DNN to all subbands. In eachsubband, the center frequency is defined as the reference
frequency. The DNN predicts whether the components in thereference frequency and the other frequencies (within thesubband) belong to the same source. When N = 2, thisbinary classification coincides with estimating a specific source
permutation. However, when N ≥ 3, we cannot determine
the source permutation when the DNN outputs that the twocomponents are not the same source. Therefore, we need toapply the DNN to all of the pair combinations from N sources.
Furthermore, the post-process for solving the permutation
problem among subbands, which is a stitching technique along
subbands, becomes more complicated as the number of sources
increases. To solve this problem, we propose a simple DNNbased permutation solver that directly predicts the permutation
matrix Pi−1 for all frequencies simultaneously.

7-10 November 2022, Chiang Mai, Thailand

Frequency

Proceedings of 2022 APSIPA Annual Summit and Conference

Output vector
Output matrix
Softmax
Softmax

Softmax

Fig. 3. DNN architecture for N = 2 case.

permutation matrix can be constructed by using ˆli1j and ˆli2j
as




1 0
0 1
−1
ˆ
ˆ
P̂ij = li1j
+ li2j∈ [0, 1]N ×N .
(15)
0 1
1 0
When N = 3, the probabilities (ˆliqj )3!
q=1 are predicted, and−1P̂ij is constructed as a linear combination of the matrices
P(8). Note that q ˆliqj = 1 is constrained in the DNN.
Thus, the proposed DNN directly predicts the correct permutation matrix for all frequencies simultaneously. From adifferent perspective, because (15) is a DSM, the DNN predicts
coefficients of a convex combination of the true permutation
matrices, which is known as the Birkhoff–von Neumann theorem [28].
C. DNN Architecture
Fig. 3 shows the architecture of the DNN used in the
proposed method. All of the layers consist of fully connected
layers. Rectified linear unit (ReLU) functions [29] are usedfrom the first to the third hidden layers. The output layer
explicitly branches into N ! (I-dimensional) vectors with fully
connected (non-shared) weights, and the frequency-wise softmax
P ˆ function is applied to each of the elements to ensureq liqj = 1.
D. Loss Function
The process after obtaining P̂ij−1 is shown in Fig. 4, whereN(Ýjn′ )N
n′ =1 is the temporally local spectrograms of (Yn′ )n′ =1(before the normalization (11)) extracted in the same manneras (12). First, (Ýjn′ )N
n′ =1 is softly aligned by multiplying
the predicted permutation matrices (P̂ij−1 )Ii=1 in the samemanner as (10), where the obtained spectrograms are defined874

Proceedings of 2022 APSIPA Annual Summit and Conference

7-10 November 2022, Chiang Mai, Thailand

・・・

0.5 0.5
0.5 0.0
0.5 1.0
0.0 1.0

Frequency

Frequency

Calculate
MSEwith PIT

Frequency

・
・
・

・
・
・

Frequency

0.9 0.1
0.7
0.9 0.9
0.1Frequency

0.1
0.9

Frequency

Frequency-wisematrix multiplication

Time

Time

Time

Time

Time

Time

Fig. 4. Calculation of loss function using MSE with PIT for N = 2 case.
TABLE IS PEECH AND MUSIC SOURCES OBTAINED FROM S I SEC2011 [31]Signal type
SpeechMusic

SourceMale speech
Female speech
Drums
Bass

Data namedev2_male4_inst_src_2.wav
dev3_female4_inst_src_2.wav
dev1_wdrums_src_3.wav
dev1_wdrums_src_2.wav

Length
10.0 s
10.0 s
11.0 s
11.0 sb̌ ′ )N . Then, we prepare the correctly aligned local
as (Zn n′ =1
N
spectrograms (Žn′ )N
n′ =1 for (Yn′ )n′ =1 as the label signals.
b̌ ′ )N
Finally, the mean squared error (MSE) between (Zn n′ =1Nand (Žn′ )n′ =1 is calculated as a loss function value of the
DNN. As mentioned in Sect. II-C, we do not estimate the−1correct order of the sources, i.e., Pall
. To permit this sourceorder ambiguity while training the DNN, permutation invariant
training (PIT) [30] is introduced. The loss function L is definedas
L = min(C1 , C2 , · · · , CN ! ),
Cq =

N
X

b̌ ′ − Ž2||Z
jnjP(q,n′ ) ||2 ,

(16)
(17)

n′

where min(·) is a minimum value of inputs and P(q, n′ )returns the n′ th scalar in the qth permutation of all possible
permutations.
E. Application of Pretrained DNN to Test Data
When we apply the trained DNN to test data (Yn′ )N
n′ =1 , thetemporally local spectrograms (Y̌jn′ )Nwithvarioustime jn′ =1
can be input to the DNN. Because the permutation matrix
Pi is time-invariant, the effect of prediction errors can be
mitigated by taking the majority decision among the predicted
permutation matrices (P̂ij−1 )Jj=1 asJX1P̂ −1  ∈ {0, 1}N ×N ,P̂i−1 = round 
J j=1 ij(18)

where round(·) denotes the element-wise rounding off. The
separated signal is obtained byẑij = P̂i−1 yij .

(19)

IV. E XPERIMENTS
A. Conditions
We conducted experiments to compare performances of
the conventional [23] and proposed methods. Furthermore,
we show the robustness of the proposed method against thedomain of the dataset, which demonstrates the validity of the
DNN-based permutation solvers.
In this experiment, two pairs of dry source signals shown in
Table I were used as (Sn )2n=1 . These sources were obtained
from SiSEC2011 [31]. The sampling frequency of these signals
was 16 kHz. STFT was performed using the 128-ms-long Hann
window with 64-ms-long shifting, resulting in I = 1025 andJ = 158 (speech signal) or J = 173 (music signal). To simulate the block permutation problem [32], which often arisesin IVA and ILRMA, the entire frequencies of (Sn )2n=1 weredivided into 64 blocks (each block containing 16 frequencybins), and these blocks were randomly swapped between S1and S2 to produce the input signal of the permutation solvers,
i.e., (Yn′ )2n′ =1 . The training dataset for the conventional andproposed DNNs was generated from 300 random swapping
patterns. Then, ten new patterns of randomly swapped signals
were used as the test dataset for evaluation.
For the conventional method, the conditions of the DNN
model, hyperparameters, and optimization were set to the sameas those in [23]. For the proposed method, all of the hidden
layers had 4096 units, and the number of local time frames was
set to β = 13. We used the Adam optimizer with the minibatch
size set to eight, and the number of training epochs was 1000.
For both the conventional and proposed methods, we trained
two DNN models, the speech and the music models, using only
the speech and music signals, respectively. Then, we evaluatedthese models using the test dataset of both speech and music
mixtures, i.e., in-domain (using the same sources in the training
and testing) and out-domain (using different sources in the
training and testing) evaluations. We used source-to-distortion
ratio (SDR) [33] as the evaluation score, which represents
accuracy and quality of BSS.
B. Results
Figs. 5 and 6 show examples of permutation-simulated andaligned signals estimated by the conventional and proposed
methods for the in-domain test dataset. Both the conventional875

Proceedings of 2022 APSIPA Annual Summit and Conference

7-10 November 2022, Chiang Mai, Thailand

Fig. 5. Input (left) and estimated signals of conventional (center) and proposed
methods (right): male (top) and female (bottom) speech.

Fig. 6. Input (left) and estimated signals of conventional (center) and proposed
methods (right): drums (top) and bass (bottom).

and proposed methods solve the permutation problem withhigh accuracy. However, the conventional method often fails
to align the sources in a low-frequency band, which is crucial
for SDR and subjective listening. Figs. 7 and 8 show the SDR
values for the in-domain and out-domain test datasets, respectively. The values of the observed and estimated signals shown
are averaged over the two sources. For both the speech andmusic results of the in-domain evaluation, the proposed method
improved SDR by over 20 dB on average. In contrast, theconventional method often failed to improve SDR, particularly
for the music signals. Furthermore, the efficacy of the proposed
method can be verified even in the out-domain evaluation. This
result shows the robustness against the domain of the dataset
used to train the proposed DNN.

R EFERENCES

V. C ONCLUSION
We proposed a DNN-based permutation solver that directlypredicts the permutation matrices for all frequencies simultaneously. The experimental results show significant improvementfrom the conventional method. We also verified the robustness
against the domain of the dataset. The combination of a
BSS technique and the proposed permutation solver for threeor more sources will be compared with other conventional
approaches in future work. In addition, explicit learning ofdependencies along frequencies should be introduced.
ACKNOWLEDGMENT
This work was partly supported by JSPS KAKENHI Grant
Numbers 22H03652 and 19H01116. Also, the authors would
like to thank Shuhei Yamaji for his support on the experiment.

[1] H. Sawada, N. Ono, H. Kameoka, D. Kitamura, and H. Saruwatari, “A
review of blind source separation methods: Two converging routes to
ILRMA originating from ICA and NMF,” APSIPA Trans. Signal andInfo. Process., vol. 8, no. e12, pp. 1–14, 2019.
[2] P. Comon, “Independent component analysis, a new concept?,” Signal
Process., vol. 36, no. 3, pp. 287–314, 1994.
[3] P. Smaragdis, “Blind separation of convolved mixtures in the frequency
domain,” Neurocomputing, vol. 22, pp. 21–34, 1998.
[4] N. Q. K. Duong, E. Vincent, and R. Gribonval, “Under-determined
reverberant audio source separation using a full-rank spatial covariancemodel,” IEEE Trans. Audio, Speech, Lang. Process., vol. 18, no. 7, pp.
1830–1840, 2010.
[5] N. Murata, S. Ikeda, and A. Ziehe, “An approach to blind source separation based on temporal structure of speech signals,” Neurocomputing,vol. 41, no. 1–4, pp. 1–24, 2001.
[6] H. Sawada, R. Mukai, S. Araki, and S. Makino, “A robust and precise
method for solving the permutation problem of frequency-domain blindsource separation,” IEEE Trans. Speech and Audio Process., vol. 12,
no. 5, pp. 530–538, 2004.
[7] H. Saruwatari, T. Kawamura, T. Nishikawa, A. Lee, and K. Shikano,“Blind source separation based on a fast-convergence algorithm combining ICA and beamforming,” IEEE Trans. Audio, Speech, Lang. Process.,
vol. 14, no. 2, pp. 666–678, 2006.
[8] H. Sawada, S. Araki, and S. Makino, “Measuring dependence of binwise separated signals for permutation alignment in frequency-domain
BSS,” in Proc. ISCAS, pp. 3247–3250, 2007.
[9] A. Hiroe, “Solution of permutation problem in frequency domain ICA
using multivariate probability density functions,” in Proc. ICA, pp. 601–
608, 2006.
[10] T. Kim, T. Eltoft, and T.-W. Lee, “Independent vector analysis: anextension of ICA to multivariate components,” in Proc. ICA, pp. 165–
172, 2006.
[11] T. Kim, H. T. Attias, S.-Y. Lee, and T.-W. Lee, “Blind source separation
exploiting higher-order frequency dependencies,” IEEE Trans. Audio,
Speech, Lang. Process., vol. 15, no. 1, pp. 70–79, 2007.
[12] N. Ono, “Stable and fast update rules for independent vector analysis
based on auxiliary function technique,” in Proc. WASPAA, pp. 189–192,
2011.
[13] D. Kitamura, N. Ono, H. Sawada, H. Kameoka, and H. Saruwatari, “Determined blind source separation unifying independent vector analysis

876

Proceedings of 2022 APSIPA Annual Summit and Conference

















    






7-10 November 2022, Chiang Mai, Thailand










    
Fig. 7. Average SDR for in-domain test data: (a) speech mixtures with speechmodels and (b) music mixtures with music models.

and nonnegative matrix factorization,” IEEE/ACM Trans. Audio, Speech,
Lang. Process., vol. 24, no. 9, pp. 1626–1641, 2016.
[14] D. Kitamura, N. Ono, H. Sawada, H. Kameoka, and H. Saruwatari,“Determined blind source separation with independent low-rank matrix
analysis,” in Audio Source Separation, S. Makino, Ed., pp. 125–155.
Springer, Cham, 2018.
[15] D. D. Lee and H. S. Seung, “Learning the parts of objects by nonnegative matrix factorization,” Nature, vol. 401, pp. 788–791, 1999.
[16] N. Makishima, S. Mogami, N. Takamune, D. Kitamura, H. Sumino,
S. Takamichi, H. Saruwatari, and N. Ono, “Independent deeply learned
matrix analysis for determined audio source separation,” IEEE/ACM
Trans. Audio, Speech, Lang. Process., vol. 27, no. 10, pp. 1601–1615,2019.
[17] S. Oyabu, D. Kitamura, and K. Yatabe, “Linear multichannel blindsource separation based on time-frequency mask obtained by harmonic/percussive sound separation,” in Proc. ICASSP, pp. 201–205,
2021.
[18] K. Yatabe and D. Kitamura, “Determined BSS based on time-frequency
masking and its application to harmonic vector analysis,” IEEE/ACM
Trans. Audio, Speech, Lang. Process., vol. 29, pp. 1609–1625, 2021.
[19] D. Kitamura, N. Ono, and H. Saruwatari, “Experimental analysis of
optimal window length for independent low-rank matrix analysis,” inProc. EUSIPCO, pp. 1210–1214, 2017.
[20] Y. Luo and N. Mesgarani, “Conv-TasNet: Surpassing ideal
time–frequency magnitude masking for speech separation,” IEEE/ACM
Trans. Audio, Speech, Lang. Process., vol. 27, no. 8, pp. 1256–1266,
2019.
[21] E. Tzinis, Z. Wang, and P. Smaragdis, “Sudo rm -rf: Efficient networks
for universal audio source separation,” MLSP, pp. 1–6, 2020,[22] A. Défossez, “Hybrid spectrogram and waveform source separation,”
Proc. ISMIR, 2021.
[23] S. Yamaji and D. Kitamura, “DNN-based permutation solver for
frequency-domain independent component analysis in two-source mixture case,” Proc. APSIPA ASC, pp. 781–787, 2020.
[24] M. Kowalski, E. Vincent, and R. Gribonval, “Beyond the narrowband
approximation: wideband convex methods for under-determined reverberant audio source separation,” IEEE Trans. Audio, Speech, Lang.
Process., vol. 18, no. 7, pp. 1818–1829, 2010.

















    















    
Fig. 8. Average SDR for out-domain test data: (a) music mixtures with speechmodels and (b) speech mixtures with music models.

[25] A. Horn, “Doubly stochastic matrices and the diagonal of a rotation
matrix,” Am. J. Math., vol. 76, no. 3, pp. 620–630, 1954.
[26] R. Sinkhorn and P. Knopp. “Concerning nonnegative matrices and doubly
stochastic matrices,” Pacific J. Math., vol. 21, no. 2, pp. 343–348, 1967.
[27] K. Matsuoka and S. Nakashima, “Minimal distortion principle for blindsource separation,” in Proc. ICA, pp. 722–727, 2001.
[28] G. Birkhoff, “Three observations on linear algebra,” Univ. Nac. Tucuman,
Rev. Ser. A., vol. 5, pp. 147–151, 1946.
[29] V. Nair and G. E. Hinton, “Rectified linear units improve restricted
Boltzmann machines,” in Proc. ICML, 2010.
[30] D. Yu, M. Kolbak, Z.-H. Tan, and J. Jensen, “Permutation invariant
training of deep models for speaker-independent multi-talker speech
separation,” in Proc. ICASSP, pp. 241–245, 2017.
[31] S. Araki, F. Nesta, E. Vincent, Z. Koldovský, G. Nolte, A. Ziehe,and A. Benichoux, “The 2011 signal separation evaluation campaign(SiSEC2011):-audio source separation,” in Proc. LVA/ICA, pp. 414–422,
2012.
[32] Y. Liang, S. M. Naqvi, and J. A. Chambers, “Overcoming block
permutation problem in frequency domain blind source separation whenusing AuxIVA algorithm,” Electron. Lett., vol. 48, no. 8, pp. 460–462,
2012.
[33] E. Vincent, R. Gribonval, and C. Févotte, “Performance measurementin blind audio source separation,” IEEE Trans. Audio, Speech, Lang.
Process., vol. 14, no. 4, pp. 1462–1469, 2006.

877