Proceedings, APSIPA Annual Summit and Conference 2020

7-10 December 2020, Auckland, New Zealand

DNN-Based Permutation Solver for
Frequency-Domain Independent Component
Analysis in Two-Source Mixture Case
Shuhei Yamaji∗ and Daichi Kitamura∗∗ National Institute of Technology Kagawa College, Kagawa, Japan

Abstract—Frequency-domain independent component analysis(FDICA) is a popular algorithm for multichannel audio source
separation. The source components in each frequencies estimatedby FDICA must be aligned over all frequencies so that thecomponents of the same source are grouped. This postprocessingof FDICA is the so-called permutation problem. Although various
permutation solvers have been proposed, their performances arestill limited particularly in a multispeaker separation task ina reverberant environment. To improve the performance of the
permutation solver, in this paper, a new data-driven permutation
solver based on deep neural networks (DNNs) is presented.
In the proposed method, the DNN that predicts whether the
input local time-frequency components belong to the same sourceis trained, and the permutation problem is solved by taking
majority decisions of the predicted results.

I. I NTRODUCTION
Blind source separation (BSS) is a method of estimating
original sources from an observed multichannel mixture signal
without knowing a priori information, e.g., the locations of
microphones and sources. When the number of microphones isequal to or greater than the number of sources (overdeterminedcase), independent component analysis (ICA) [1] is a typicalapproach to solving the BSS problem, which estimates a
demixing matrix for the separation.
In general, audio signals are mixed with room reverberation
as a convolutive mixture, and simple ICA cannot separate
the audio sources. To solve this problem, frequency-domain
ICA (FDICA) [2] was proposed. In FDICA, ICA is independently applied in each frequency bin, and the frequency-wisedemixing matrices are estimated. However, since ICA cannot
determine the order (permutation) of the estimated signals,
the frequency-wise components separated by FDICA must bealigned over all frequency bins so that the components of thesame source are grouped. This is the so-called permutation
problem.
A major approach to solving the permutation problem isbased on a correlation between time series components ofthe separated signals in adjacent or local frequency bins [3].
When the positions of microphones are known, the directionsof arrivals (DOAs) of the sources can be utilized for taking
an alignment of the separated components [4]. The uniﬁed
permutation solver combining frequency correlation and DOAs
was also proposed [5].
This work was partly supported by NVIDIA GPU Grant and JSPS KAKENHI Grant Numbers 19K20306 and 19H01116.

978-988-14768-8-3/20/$31.00 ©2020 APSIPA

In recent years, BSS algorithms without encountering the
permutation problem have been proposed. For example, independent vector analysis (IVA) assumes the co-occurrence of all
frequency components of the same source and estimates thefrequency-wise demixing matrices avoiding the permutation
problem [6], [7]. Independent low-rank matrix analysis (ILRMA) [8], [9] assumes a low-rank time-frequency structureof each source, resulting in a more precise estimation of
the permutation-aligned demixing matrices. However, theirperformances are still limited particularly in a multispeakerseparation task. This is because the dominant frequenciesof speech signals are substantially overlapped. Moreover, thesource models assumed in the correlation-based permutation
solver [3], IVA, and ILRMA are not suitable for representing
speech time-frequency structures.
In this paper, a new data-driven permutation solver basedon deep neural networks (DNNs) is presented. The proposed
DNN is trained to predict whether the two input narrowband
frequency components (time-varying powers) belong to thesame source, where the training data can be prepared bymanually shufﬂing the narrowband frequency components ofclean or separated speech spectrograms for all frequency bins.
In the test stage, the trained DNN is applied to all frequency
bins, and majority decisions of the predicted results along thetime frames and the frequency bins are taken to obtain theaccurate permutation predictions. The validity of the proposed
method is conﬁrmed by experiments of multispeaker audio
source separation, where the proposed permutation solver isused after the estimation by simple FDICA.
In the underdetermined case (the number of microphones isless than the number of sources), multichannel audio source
separation based on a spatial covariance matrix [10] is themost popular and reliable algorithm. However, this method is
also faced with the permutation problem. Since the proposed
permutation solver only requires monaural (nonaligned) source
estimates, we can apply the proposed method in the underdetermined case without loss of generality.
II. FDICA AND P ERMUTATION P ROBLEM
A. Formulation and FDICA
We deﬁne the numbers of audio sources and observed
channels (microphones) as N and M , respectively. The source,
mixture, nonaligned separated, and aligned separated signals

781

APSIPA-ASC 2020

Proceedings, APSIPA Annual Summit and Conference 2020

7-10 December 2020, Auckland, New Zealand

Estimated signal 1

obtained by short-time Fourier transform (STFT) are respectively represented asTsi,j = (si,j,1 , · · · , si,j,n , · · · , si,j,N )Txi,j = (xi,j,1 , · · · , xi,j,m , · · · , xi,j,M )Source 1∈ CN ,(1)∈ CM ,

(2)

T

N

∈C ,

(3)

T

∈C ,

(4)

yi,j = (yi,j,1 , · · · , yi,j,n , · · · , yi,j,N )zi,j = (zi,j,1 , · · · , zi,j,n , · · · , zi,j,N )N

where i = 1, 2, · · · , I, j = 1, 2, · · · , J, n = 1, 2, · · · , N ,and m = 1, 2, · · · , M are the indices of frequency bins, timeframes, sources, and channels, respectively, and ·T denotes
the vector transpose. Also, si,j,n , xi,j,m , yi,j,n , and zi,j,n
are the complex-valued elements of spectrogram matrices
Sn ∈ CI×J , Xm ∈ CI×J , Yn ∈ CI×J , and Zn ∈CI×J , respectively. Note that the estimated spectrogram Yn
is problematic because the source permutation of narrow-band
frequency components (yi,1,n , · · · , yi,J,n ) over all frequencies
is not aligned, which is called the permutation problem. The
permutation-aligned spectrograms Zn can be obtained byapplying a permutation solver to Yn . Thus, the aim of thispaper is to estimate Zn from Yn .
In FDICA, we assume that the mixing system is linear timeinvariant in each frequency bin, which can be expressed by
the frequency-wise mixing matrix Ai = (ai,1 · · · ai,N ) ∈CM ×N . The mixture signal is deﬁned asxi,j = Ai si,j ,(5)where ai,n ∈ CM is the steering vector of the nth source.
This mixing model is valid only when the window length in
STFT is longer than the length of room reverberation.
When the number of channels is equal to the number ofsources (M = N ) and Ai is a nonsingular matrix, thedemixing matrix Wi = A−1= (wi,1 · · · wi,N )H ∈ CN ×M
iexists, and the separated signal is deﬁned aszi,j = Wi xi,j ,(6)where wi,n ∈ C is the demixing ﬁlter of the nth source and·H denotes the Hermitian transpose. Therefore, the demixing
matrix Wi for all frequency bins must be estimated by FDICA
to achieve source separation.
M

B. Permutation Problem
FDICA cannot estimate the scales (volumes) and orders(permutations) of the separated signals because ICA is basedon the statistical independence between sources. Therefore,
the uncertainty remains even when the separation is ideally
performed as follows:Ŵi = Di Pi Wi= Di Pi A−1
i ,

(7)
(8)

where Pi is a permutation matrix that may replace ordersof the row vectors wi,n in Wi and Di is a diagonal matrix
that may vary the scale of wi,n in Wi . For this reason, theseparated signals estimated by FDICA, Y1 , · · · , YN , have thepermutation and scale ambiguities in each frequency bin. The

Observed 1

FDICA
Time

Source 2

Permutation
Estimated signal 2
SolverObserved 2

All frequency
components

Fig. 1. Permutation problem in FDICA, where N = 2.

scale ambiguity can easily be recovered by applying the backprojection technique [3]. However, estimating the correctlyaligned permutation is difﬁcult because this problem includes a
combinatorial explosion. The permutation problem is depicted
in Fig. 1, where the permutation solver takes an alignment of
the separated sources over all frequency bins.
The permutation-aligned separated signal zi,j is obtained aszi,j = Pi−1 Di−1 yi,j
= Pi−1 Di−1 Ŵi xi,j(9)

In this paper, we aim to estimate Pi−1 in (9) over all frequency
bins using a new data-driven permutation solver.
III. P ROPOSED M ETHOD
A. Motivation and Strategy
In Ref. [11], the optimal window length in STFT for BSS
was experimentally investigated. Fig. 6(b) in Ref. [11] showsthat FDICA with the DOA-based permutation solver, IVA, andILRMA cannot accurately separate speech sources under areverberant condition (T60 = 470 ms). However, FDICA with
the ideal permutation solver achieves over 10 dB improvementin the signal-to-distortion ratio (SDR) [12], where the ideal
permutation solver uses completely separated (oracle) source
signals si,j for estimating the permutation alignment matrix
Pi−1 . This fact implies that the demixing matrix Ŵi itself
can be accurately estimated by FDICA even for a reverberant
speech mixture, but only the permutation solver fails toestimate Pi−1 . IVA and ILRMA may also fail to solve only
the permutation problem even though it may successfully
separate frequency-wise sources. This may be because thesource model assumed in IVA or ILRMA is not suitable
for speech sources. In fact, the source model in IVA, i.e.,the co-occurrence of all frequency components of the same
source, is too simplistic to avoid the permutation problem.
Also, the source model in ILRMA, i.e., the low-rank timefrequency structure of the same source, does not ﬁt to thepower spectrogram of speech signals, which dynamically andcontinuously changes its spectra.
On the basis of the experimental result in Ref. [11], we canassume that FDICA achieves satisfactory separation in eachfrequency bin. Thus, in this paper, we only focus on accurately
solving the permutation problem and develop a new DNNbased supervised (data-driven) permutation solver. Hereafter,782

7-10 December 2020, Auckland, New Zealand

(10)

×1
r̃i,γ = (ri,γ,1 T , ri,γ,2 T )T ∈ R2τ
≥0 ,
2

×1
· · · , |yi,(γ−1)η+τ,n |2 )T ∈ Rτ≥0
,

(12)

×1
g̃i,ω,γ = (gi,ω,γ,1 T , gi,ω,γ,2 T )T ∈ R2τ
≥0 ,

(13)

gi,ω,γ,n = (|yi+ω,(γ−1)η+1,n | , |yi+ω,(γ−1)η+2,n | ,2

×1· · · , |yi+ω,(γ−1)η+τ,n |2 )T ∈ Rτ≥0
,

2

(14)

where | · | for matrices returns a matrix with elementwise absolute and squared operations, ω = −Ω, −Ω +1, · · · , −1, 0, 1, · · · , Ω is the index that deﬁnes the differencein the number of frequency bins between ri,γ,n and gi,ω,γ,n ,η is the stride length of the short-time activation along thetime-frame axis, and γ = 1, 2, · · · , Γ is the index of shorttime activations. Note that Γ is calculated using the settingsof the length of short-time activations τ and the stride length
η. The vector ri,γ,n corresponds to the short-time activation
.2

DNN
DNN

0 : Same

1 : Diff.

DNN

Frequency

1 : Diff.

0 : Same

Fig. 3. DNN predictions in subband frequency bins, where f1 , f2 , · · · , f5
are frequency bins in subband frequency, and index of short-time activations,γ, is omitted for simplicity. Reference frequency bin is i = f3 , and adjacentor local frequency bins are i + ω = f1 , f2 , · · · , f5 , namely, Ω = 2. Whensource permutation of r̃i and g̃i,ω is correct, DNN ideally outputs zero as“same.” In contrast, when source permutation of r̃i and g̃i,ω is incorrect,
DNN ideally outputs one as “different.”

in the reference frequency bin i, and the vector gi,ω,γ,n is theshort-time activation in the adjacent or local frequency bin
i + ω as shown in Fig. 2.
The input vector for DNN is deﬁned as the normalized
vector of (10) obtained asd˜i,ω,γ =(11)ri,γ,n = (|yi,(γ−1)η+1,n | , |yi,(γ−1)η+2,n | ,2Time

1 : Diff.

…

×1
T
T
di,ω,γ = (r̃i,γ, g̃i,ω,γ)T ∈ R4τ
≥0 ,DNN
outputs

Input vector…

The input vector for the proposed DNN model is shownin Fig. 2. After applying FDICA to the observed mixture,
we obtain the separated spectrograms Yn , which have apermutation problem. From their power spectrograms, |Yn |.2 ,two-source (n = 1 and 2) and two-frequency-binwise (i and
i + ω) short-time activations with length τ are gathered asfollows:

Fig. 2. Input vector of DNN. Matrices |Y1 |.2 and |Y2 |.2 are separated power
spectrograms with permutation problem, and red and blue binwise activations(rows of |Y1 |.2 and |Y2 |.2 ) depict sourcewise components, e.g., red and blue
slots respectively correspond to ﬁrst and second source components.

…

B. DNN Input and Output

Time

…

we only treat a two-source mixture case (N = 2). This isbecause in this study, we focus on only the investigation of
the availability of DNN for solving the permutation problem.
The algorithm for mixtures with more than two sources is ourimportant future work.
The proposed DNN-based permutation solver is summarized
as follows:• Two frequency-binwise activations (time-varying powers)
in the two source spectrograms Y1 and Y2 are input tothe DNN.
• The DNN predicts whether the two input binwise activations of each source are the correct permutation as abinary scalar.
• The DNN is applied to scan all frequency bins and timeframes of the separated sources Y1 and Y2 estimated by
FDICA.
−1• The conclusive estimate of permutation Pi
is decidedusing a majority decision of the predicted results alongfrequency bins and time frames.
The proposed permutation solver can be interpreted as a supervised and nonlinear extension of the conventional binwisecorrelation-based permutation solver [3] because the proposed
DNN exploits the relationship between two activations ofadjacent or local frequency bins. The training data for the
proposed DNN can easily be produced by manually shufﬂing
the clean or separated speech spectrograms to simulate thepermutation problem, where the labels of this input data are
given by the shufﬂing result.

Frequency

Proceedings, APSIPA Annual Summit and Conference 2020

di,ω,γ
×1
∈ R4τ
≥0 ,
∥di,ω,γ ∥2(15)where ∥ · ∥2 denotes the L2 norm.
The proposed DNN model is a binary classiﬁer that outputs zero or one. The “zero” output indicates that the permutations of two input short-time activations (ri,γ,1 , ri,γ,2 )and (gi,ω,γ,1 , gi,ω,γ,2 ) are correct, namely, ri,γ,1 and gi,ω,γ,1
are the same source components and ri,γ,2 and gi,ω,γ,2 arealso the same source components. In contrast, the “one”
output indicates that the permutations of (ri,γ,1 , ri,γ,2 ) and(gi,ω,γ,1 , gi,ω,γ,2 ) are incorrect, namely, ri,γ,1 and gi,ω,γ,1 are
the different source components and ri,γ,2 and gi,ω,γ,2 are also
the different source components. These predicted results areillustrated in Fig. 3. In practice, the result of DNN prediction
is not a binary but a soft scalar deﬁned as()
qi,ω,γ = DNN d˜i,ω,γ ∈ [0, 1],
(16)

783

Proceedings, APSIPA Annual Summit and Conference 2020

Subbandpermutation
vectorDNN
outputs

0 or 1Minimum MSE

StrideFrequency

Output Layer (1 unit)
Sigmoid
Hidden Layer 6 (1 unit)
ReLU
Hidden Layer 5 (64 units)

1

1

1

1

1

1

0

0

1

1

1

0

0

0

…

0

1

Majority
decisionTarget label (1 unit)

7-10 December 2020, Auckland, New Zealand

1
0
1
0

Time

Fig. 5. DNN predictions for all short-time subbands and their majority
decisions. Since permutation does not depend on time frames, this majority
decision along time-frame axis effectively reduces adverse effect of DNN
prediction errors.

ReLU
Hidden Layer 4 (64 units)
ReLU
Hidden Layer 3 (128 units)
ReLU
Hidden Layer 2 (128 units)
ReLU
Hidden Layer 1 (128 units)(r̃f3 , g̃f3 ,2 ), the DNN outputs one. As a result, the correctness
values of the permutation based on the reference frequency bin
i can be estimated as shown on the right side of Fig. 3. This
vector is called (subband) permutation vector. In practice, theDNN output qi,ω,γ is a value in the range [0, 1]. To produce
the subband permutation vector, the following thresholding is
performed:

Input Layer
Fig. 4. DNN architecture.

q̃i,ω,γ = round(qi,ω,γ ) ∈ {0, 1},

which can be interpreted as the reliability of the permutation
correctness.

(17)

where round(·) is a rounding operator, and the subband
permutation vector can be obtained asq̃i,γ = (q̃i,−Ω,γ , q̃i,−Ω+1,γ , · · · , q̃i,−1,γ , q̃i,0,γ ,C. DNN Architecture and Training Loss Function
Fig. 4 depicts an architecture of DNN used in the proposed
permutation solver. This DNN model has full-connected eightlayers (one input layer, six hidden layers, and one output
layer). A rectiﬁed linear unit (ReLU) is applied after theintermediate hidden layers, and a sigmoid function is applied
after the ﬁnal hidden layer. The loss function is deﬁned as amean squared error (MSE) between the predicted result qi,ω,γand its target label.
D. DNN Predictions in Subband Frequency Bins
The examples of DNN prediction are illustrated in Fig. 3,
where f1 , f2 , · · · , f5 are the subband frequency bins, and theindex of short-time activations, γ, is omitted for simplicity.
In this ﬁgure, the reference frequency bin is set to i = f3 ,and its adjacent or local frequency bins are deﬁned as i + ω =f1 , f2 , · · · , f5 , namely, Ω = 2. Note that the component in thereference frequency bin f3 of Y1 corresponds to a red source,
but the components in f2 , f4 , and f5 of Y1 correspond to a
blue source. All the combinations of (r̃i , g̃i,ω ) are input to the
DNN, which are (r̃f3 , g̃f3 ,−2 ), (r̃f3 , g̃f3 ,−1 ), · · · , (r̃f3 , g̃f3 ,2 )in the case of Fig. 3. When the permutations of the two input
frequency bins (i and i + ω) are correct, the DNN ideally
outputs zero (the “same” label). In contrast, when the permutations of the two input frequency bins are incorrect, the DNN
ideally outputs one (the “different” label). In Fig. 3, for the
combinations (r̃f3 , g̃f3 ,−2 ) and (r̃f3 , g̃f3 ,0 ), the DNN outputs
zero, and for the combinations (r̃f3 , g̃f3 ,−1 ), (r̃f3 , g̃f3 ,1 ), andT

q̃i,1,γ , · · · , q̃i,Ω,γ ) ∈ {0, 1}2Ω+1 .

(18)

The lengths of the short-time activations r̃i,γ and g̃i,ω,γ
are deﬁned as τ . Since audio signals have a sparse property inthe time-Frequency-domain, there are so many time-frequency
slots that have almost zero powers. In particular, speech signals
have silent intervals due to breath. If the reference activationr̃i,γ is a silent interval of the sources, the prediction of DNN
becomes unstable.
To cope with this problem, in the proposed permutation
solver, the DNN prediction is multiply applied to all the timeframes by shifting the τ -length input vector d˜i,ω,γ with a ηlength stride as shown in Fig. 5. Therefore, we can collectthe DNN outputs along the time frames. By taking a majority
decision of these DNN outputs, we obtain a more reliable
subband permutation vector ṽi . This process can be described
as
1∑
vi =
q̃i,γ ∈ {0, 1}2Ω+1 ,(19)
Γ γṽi = round(vi ) ∈ {0, 1}2Ω+1 ,(20)

where round(·) for vectors denotes the element-wise rounding
operation. This majority decision is reasonable because thepermutation problem does not depend on the time frames,
and the adverse effect of the prediction errors is effectively
reduced.

784

Proceedings, APSIPA Annual Summit and Conference 2020

7-10 December 2020, Auckland, New Zealand

Fullbandpermutation
vector

Stride

1
1

0

1

1

0

1

0

0

1

1

0

0

1

0

1

1

1

0

Frequency

DNN andmajority decision

Frequency

Subbandpermutation
vectors0

1
1

1. Set

2. Set

1

1

1

1

0

0

0

1

1

1

0

0

0

Time

(a)

Time

Fig. 6. Estimation of subband permutation vectors in all frequency bins.
Subband is shifted with stride length and several overlaps. Similarly to timeframe axis, majority decision is performed for each frequency bin.

3. Majority decision

Fullbandpermutation
vectoru = (u1 , u2 , · · · , uI )T ∈ {0, 1}I .

(21)

The reconstruction process of u is depicted in Fig. 7.
As described in Sect. III-E1, the signiﬁcance of subband
permutation vectors ṽ1 , ṽ2 , · · · , ṽI is not identical. Therefore,
it is necessary to unify the subband permutation vectors
over all frequency bins so that the values “zero” and “one”
respectively indicate the ﬁrst and second sources (red and blue
sources in Fig. 7).

0

1

0

0

1

1

1

0

1

1

1

1

0

0

0

0

1

1

1

0
Time

0
1
1
0

1
0

1. Similarity comparison

(b)

1

0

0

1

0

1

1

0

1

0

1

0

0

0

0

0

1

1

1

1

1

1

1

1

0

0

0

1

1

0
Time

3. Majority decision

Fullbandpermutation
vector

2. SetFrequency

The proposed DNN-based permutation solver consists ofthe following steps: (a) prediction of the subband permutation vectors in all the frequency bins with subband striding(Sect. III-E1, Fig. 6) and (b) calculation of the fullband
permutation vector based on a similarity comparison and amajority decision (Sect. III-E2, Fig. 7).
1) Estimation of Subband Permutation Vectors in All Frequency Bins: The subband permutation vector ṽi is estimated in all the frequency bins by shifting the reference
frequency bin i (striding subband frequency bins in the range[i − Ω, i + Ω]) as shown in Fig. 6. Thus, we obtain I subband
permutation vectors ṽ1 , ṽ2 , · · · , ṽI .
Note that the signiﬁcance of frequency-wise binary scalarsin ṽi is not identical within ṽ1 , ṽ2 , · · · , ṽI . This is because the
DNN output indicates that the frequency-wise components in
Yn correspond to the “same (zero)” source as the component
in the reference frequency bin i or “not (one)” and the source
permutation of the reference frequency bin varies depending
on i. For example, in Fig. 6, zeros and ones in ṽf3 respectively
indicate “red” and “blue” source components, but zeros andones in ṽf4 respectively indicate “blue” and “red” sourcecomponents (because the reference frequency bins f3 and f4
contain red and blue source components, respectively). The
alignment of these subband permutation vectors is processedin Sect. III-E2.
2) Reconstruction of Fullband Permutation Vector: Fromthe estimated subband permutation vectors ṽ1 , ṽ2 , · · · , ṽI , wereconstruct the fullband permutation vector deﬁned as

Frequency

2. SetE. Estimation of Fullband Permutation Vector

1

0
0
1
1
0

1
0

1. Similarity comparison(c)Fig. 7. Reconstruction of fullband permutation label: (a) initialization step, (b)second step, and (c) last step, where similarity comparison is based on MSE.
This process is performed to associate DNN predictions (zeros and ones) witheach source component.

Fig. 7(a) shows the initial step in the reconstruction of
the fullband permutation vector u. The subband permutation
vector of the lowest frequency subband, ṽis , [ṽf3 in Fig. 7(a)]is simply set to the corresponding frequency bins in the
fullband permutation vector u as shown in Fig. 7(a), where isis the index of the lowest reference frequency bin. In Fig. 7(a),since is = f3 and Ω = 2, u1 , u2 , · · · , u5 are determined byṽf3 .
Fig. 7(b) shows the following step of Fig. 7(a). In thisstep, the subband permutation vector adjacent to the lowest
frequency subband, ṽis +1 , and its binary complement vectorṽis +1 [ṽf4 and ṽf4 in Fig. 7(b)] are prepared. When we deﬁne785Proceedings, APSIPA Annual Summit and Conference 2020

7-10 December 2020, Auckland, New Zealand

Fullbandpermutation
vector
1

1

1

1

1

1

1

1

1

0

0

0

1

1

0

2m

0
0
1

1
0

Replace0Frequency

00

Source 2

1

Majority decision00

Source 160

160
5.66 cm

0

Fig. 9. Recording condition of JR2 impulse response.
Time

Audio files

F. Replacing Components Based on Fullband Permutation
Vector
The fullband permutation vector u is equivalent to the estimate of the permutation matrix Pi−1 . Thus, the frequency-wise
source components can be replaced to solve the permutation
problem. This process is illustrated in Fig. 8.
IV. E XPERIMENTS
A. Conditions
To evaluate the performance of the proposed permutation solver, we conducted BSS experiments in which speechsources were separated. We compared three BSS methods,
namely, FDICA with the ideal permutation solver (IPS), ILRMA [9], and FDICA with the proposed permutation solver.
The IPS utilizes the oracle (completely separated) source
signals for solving the permutation problem, which provides
upper-limit performance for FDICA-based BSS. The detailsof FDICA with IPS can be found in Ref. [11].
We used the “nonpara30” dataset obtained from theJapanese versatile speech (JVS) corpus [13] as training data
for the proposed permutation solver. We used 190 speech ﬁles
(95 ﬁles for 46 males and 95 ﬁles for 48 females) from thedataset, and 10-s-long two speech ﬁles were used as the dry

Permutation-aligned
separated
signal

IPS

the
two
similarities
MSE(ṽis +1 , ǔis +1 )andMSE(ṽis +1 , ǔis +1 ) are compared, where MSE(·, ·) returns
the MSE value between two input vectors. Then, the vectorṽis +1 or ṽis +1 that minimizes MSE is selected and stored in
the memory. The fullband permutation vector u is updated
by taking a majority decision using the vectors stored in
the memory as shown in Fig. 7(b). Finally, by iterating theabove-mentioned step, the complete fullband permutation
vector u can be obtained as shown in Fig. 7(c).
It is worth mentioning that the iterative majority decisionused in the reconstruction process of u effectively reduces theadverse effect of the DNN prediction errors, similarly to themajority decision in Fig. 5.

Frequency-wiserandom shuffling(22)

Nonaligned
separated
signalMixturesignal

45 male & 46 female
Speecha part of frequency bins in u asǔi = (ui−Ω , ui−Ω+1 , · · · , ui+Ω−1 )T ∈ {0, 1}2Ω ,Dry sources

FDICA

Fig. 8. Solving permutation problem by replacing frequency-wise components
on the basis of the fullband permutation vector.

Nonaligned
separated
signalInput vectorof DNN

Fig. 10. Process ﬂow of producing input vectors for DNN.

sources. To simulate the multichannel recording, these speech
dry sources, S1 and S2 , were convoluted with the JR2 impulse
response obtained from the RWCP database [14], where thereverberation time is T60 = 470 ms. The recording conditionof the JR2 impulse response is shown in Fig. 9.
The input vectors for DNN were produced by simulating
FDICA separation as depicted in Fig. 10, where the label
vectors are obtained from the result of frequency-wise random
shufﬂing. STFT was applied to the observed signals usinga 512-ms-long Hamming window with 128-ms-long shifting
to produce the mixture spectrograms X1 and X2 , and theseparated signals Y1 and Y2 were estimated by FDICA. The
permutation-aligned spectrograms Z1 and Z2 can be obtainedby applying IPS to Y1 and Y2 . To simulate the FDICAbased separation with the permutation problem, the frequency
components were randomly shufﬂed, and the simulated nonaligned spectrograms Y1′ and Y2′ were produced. The DNN
input vectors and their label vectors were calculated from
Y1′ and Y2′ , and we prepared 400,000 pairs of input andlabel vectors as the training data, where the training data
was split into 200,000 training and 200,000 validation pairs.
The dry sources of test data (four males and four females)
were obtained from underdetermined separation tasks (dev1
data) in the SiSEC2011 dataset [15]. These signals were also
convoluted with the JR2 impulse response, and 28 observed
signals were produced. In DNN training, the batch size was
set to 128, and we used the Adam optimizer to train the
network. We did not use the dropout technique. Also, we set
τ = 40, Ω = 15, and η = 4, where Γ = 37. The number

786

Proceedings, APSIPA Annual Summit and Conference 2020

7-10 December 2020, Auckland, New Zealand

1

V. C ONCLUSION

Accuracy

0.8

In this paper, we proposed a new DNN-based permutation
solver for determined audio source separation using FDICA,
where only the two-source mixture case is treated. The DNN
model in the proposed permutation solver is trained so thatthe DNN predicts whether the two input binwise activationsof each source are the correct permutation as a binary scalar.
In addition, to cope with the prediction errors in the DNN
model, the majority decisions along time frames and frequency
bins are performed. From the speech separation experiments,
FDICA with the proposed permutation solver outperforms theconventional ILRMA. The proposed permutation solver canbe applied to only the two-source mixture case. Developing amore general framework of the DNN-based permutation solver
for an arbitrary number of sources is our important future
work.

0.6
Training

0.4

Validation
0.2
0
0

10

20

30

40

50

Epoch
Fig. 11. Accuracy curves of DNN for training and validation datasets.

SDR improvement

12
10
8

R EFERENCES
6
4
2
0

FDICA
with IPS

ILRMA(2 bases)

ILRMA(3 bases)

ILRMA(4 bases)

ProposedmethodFig. 12. Average SDR improvements of methods.

of dimensions of the input vector was 40 × 4 = 160. We
evaluated the improvement of SDR [12], which shows the total
separation accuracy including both the degree of separationand the absence of artiﬁcial distortion.
B. Results
Fig. 11 shows the accuracy curves of DNN for the training
and validation datasets. From the results, we can conﬁrm thatthe DNN solves the frequency-wise permutation problem withmore than 85% accuracy. That is, there is a 15% chance thatthe DNN fails to estimate the correct permutation. However,
the majority decisions along time frames and frequency bins
can effectively reduce the adverse effect of these prediction
errors.
The SDR improvements of all the methods are shown in
Fig. 12, where all the results for 28 observed signals wereaveraged. As already reported in Ref. [11], the separation
performance of ILRMA is less than 4 dB, whereas FDICA
with IPS achieves over 10 dB improvement. These results
show that ILRMA fails to precisely solve the permutation
problem. The proposed method, FDICA with the DNN-based
permutation solver, outperforms ILRMA and achieves over
8 dB improvement, which is relatively close to the upper-limit
performance of FDICA-based separation. This is because thesupervised permutation solver provides better performance for
estimating the correct source permutation in each frequencybin.

[1] P. Comon, “Independent component analysis, a new concept?,” Signal
Processing, vol. 36, no. 3, pp. 287–314, 1994.
[2] P. Smaragdis, “Blind separation of convolved mixtures in the Frequencydomain,” Neurocomputing, vol. 22, pp. 21–34, 1998.
[3] N. Murata, S. Ikeda, and A. Ziehe, “An approach to blind source separation based on temporal structure of speech signals,” Neurocomputing,vol. 41, no. 1–4, pp. 1–24, 2001.
[4] H. Saruwatari, T. Kawamura, T. Nishikawa, A. Lee, and K. Shikano,“Blind source separation based on a fast-convergence algorithm combining ICA and beamforming,” IEEE Trans. ASLP, vol. 14, no. 2, pp.
666–678, 2006.
[5] H. Sawada, R. Mukai, S. Araki, and S. Makino, “A robust and precise
method for solving the permutation problem of Frequency-domain blindsource separation,” IEEE Trans. SAP, vol. 12, no. 5, pp. 530–538, Sep.
2004.
[6] T. Kim, H. T. Attias, S.-Y. Lee, and T.-W. Lee, “Blind source separation
exploiting higher-order frequency dependencies,” IEEE Trans. ASLP,
vol. 15, no. 1, pp. 70–79, 2007.
[7] N. Ono, “Stable and fast update rules for independent vector analysis
based on auxiliary function technique,” Proc. WASPAA, pp. 189–192,
2011.
[8] D. Kitamura, N. Ono, H. Sawada, H. Kameoka, and H. Saruwatari, “Determined blind source separation unifying independent vector analysis
and nonnegative matrix factorization,” IEEE/ACM Trans. ASLP, vol. 24,
no. 9, pp. 1626–1641, 2016.
[9] D. Kitamura, N. Ono, H. Sawada, H. Kameoka, and H. Saruwatari,“Determined blind source separation with independent low-rank matrix
analysis,” in Audio Source Separation, S. Makino, Ed., pp. 125–155.
Springer, Cham, 2018.
[10] N. Q. K. Duong, E. Vincent, and R. Gribonval, “Under-determined
reverberant audio source separation using a full-rank spatial covariancemodel,” IEEE Trans. ASLP, vol. 18, no. 7, pp. 1830–1840, 2010.
[11] D. Kitamura, N. Ono, and H. Saruwatari, “Experimental analysis of
optimal window length for independent low-rank matrix analysis,” Proc.
EUSIPCO, pp. 1210–1214, 2017.
[12] E. Vincent, R. Gribonval, and C. Févotte, “Performance measurementin blind audio source separation,” IEEE Trans. ASLP, vol. 14, no. 4,
pp. 1462–1469, 2006.
[13] S. Takamichi, K. Mitsui, Y. Saito, T. Koriyama, N. Tanji, and
H. Saruwatari, “JVS corpus: free Japanese multi-speaker voice corpus,”
arXiv preprint, 1908.06248, 2019.
[14] S. Nakamura, K. Hiyane, F. Asano, T. Nishiura, and T. Yamada, “Acoustical sound database in real environments for sound scene understanding
and hands-free speech recognition,” Proc. LREC, pp. 965–968, 2000.
[15] S. Araki, F. Nesta, E. Vincent, Z. Koldovsky, G. Nolte, A. Zieheand A. Benichoux, “The 2011 signal separation evaluation campaign(SiSEC2011): -Audio source separation,” Proc. LVA/ICA, pp. 414-422,
2012.

787