ICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) | 978-1-6654-0540-9/22/$31.00 ©2022 IEEE | DOI: 10.1109/ICASSP43922.2022.9746980

INVESTIGATION AND COMPARISON OF OPTIMIZATION METHODS FOR VARIATIONAL
AUTOENCODER-BASED UNDERDETERMINED MULTICHANNEL SOURCE SEPARATION
Shogo Seki1 , Hirokazu Kameoka1 , Li Li1,2
1

NTT Communication Science Laboratories, Nippon Telegraph and Telephone Corporation, Japan
2
Information Technology Center, Nagoya University, Japan
ABSTRACT

To address the permutation indeterminacy, several methods attempting to have part of model parameters shared among frequencies
have been developed [4, 5], and non-negative matrix factorization(NMF) [6] is regarded as the generalization. NMF was originallyapplied for a music transcription task [7], where the power spectrumof a mixture signal observed at each time frame is approximated by
the sum of a fixed number of basis spectra scaled by time-varying
magnitudes. Multichannel NMF (MNMF) is an FCA variant thatincorporates the NMF concept into the power spectrogram modeling of each source [8–10], and independent low-rank matrix analysis(ILRMA) is the determined version that introduces the NMF source
model on ICA [11, 12]. Using spectral templates as acoustical clues,
MNMF and ILRMA jointly perform frequency-wise source separation and permutation alignment. However, they can fail to workwhen encountering sound sources with spectrograms that do not follow the NMF source model, resulting in performance limitations.
Recently, generative approaches using deep neural networks(DNNs) have been proposed to model source spectrograms more
flexibly than the NMF source model [13–20], where a variational
autoencoder (VAE) [21] plays a central role. For determined multichannel source separation, a method called the multichannel VAE(MVAE) method using a conditional VAE (CVAE) [22] as the source
model has been proposed [18]. The MVAE method demonstrated
that the CVAE source model is better than the NMF source model at
expressing the spectrogram of each source and correctly discriminating the spectrogram of one source from that of another. Motivatedby the great success of the MVAE method, underdetermined counterpart, the generalized MVAE (GMVAE) method, was subsequently
proposed [19].
Similar to the MVAE method, a convergence-guaranteed parameter estimation algorithm of the GMVAE method was developed, using an majorization-minimization/minorization-maximization (MM)algorithm [23]. On the other hand, an expectation-maximization(EM) algorithm [24] is also known as an iterative algorithm thatkeeps increasing a log-likelihood function. Hence, an EM algorithm
allows us to design another convergence-guaranteed parameter estimation algorithm. A comparison of different algorithms for a same
objective typically help in choosing the suitable approach and in considering further developments. Although comparisons of MM-basedand EM-based algorithms for FCA and MNMF have been reported
in [10] and in [25], such comparison for the GMVAE method has not
yet been revealed.
To elucidate this, we study the MM-based and EM-based
convergence-guaranteed parameter estimation algorithms for the
GMVAE method. These algorithms are investigated from the perspectives of computational cost and source separation performance.
Through the investigation, we employs an improved CVAE variant called auxiliary classifier VAE (ACVAE), which has achieved
success in several tasks [20, 26].

In this paper, we investigate two algorithms for variational autoencoder (VAE)-based underdetermined multichannel source separation. We previously extended the multichannel VAE (MVAE)
method for determined multichannel source separation and proposed
the generalized MVAE (GMVAE) method for underdetermined
multichannel source separation. The GMVAE method employsa conditional VAE (CVAE) as the source model representing thepower spectrograms of the underlying sources present in a mixture.
While we developed a convergence-guaranteed parameter estimation algorithm using a majorization-minimization/minorizationmaximization (MM) algorithm, an expectation-maximization (EM)algorithm also allows us to design another algorithm with the same
property. However, a comparison of the MM-based and EM-basedalgorithms has not yet been revealed. To elucidate this, we investigate the MM-based and EM-based algorithms for the GMVAE
method, using an improved CVAE variant called auxiliary classifier
VAE (ACVAE). The experimental results suggest that the EM-basedalgorithm takes less computational cost, achieving comparable separation performance with the MM-based algorithm.
Index Terms— Underdetermined multichannel source separation, variational autoencoder, convergence-guaranteed algorithm
1. INTRODUCTION
Source separation refers to the problem of separating underlying
source signals present in a mixture signal observed by a microphonearray. Source separation can contribute to helping other tasks, e.g.,
automatic speech recognition under a multi-speaker conversation situation and speaker diarization of an over-lapped speech.
Source separation is typically tackled using a frequency-domain
approach with the various source signal and/or array responses
models. As one of the popular approaches for source separation
problems, independent component analysis (ICA) [1] is a wellestablished method where the separation system is assumed to exist.
While ICA employs an assumption that a mixing process generating a multichannel mixture signal from source signals is invertible,
full-rank spatial covariance analysis (FCA) [2] is known as one of
the more flexible methods. Unlike ICA, FCA assumes the mixing
system instead of the separation system and can deal with an underdetermined case where the sources outnumber the microphones.
Since both ICA and FCA perform frequency-wise source separation
and all the model parameters are independent among frequencies,
it is necessary to solve permutation indeterminacy that occurred infrequency-domain approaches [3].
This work was partly supported by JST, CREST Grant Number JPMJCR19A3, Japan.

978-1-6654-0540-9/22/$31.00 ©2022 IEEE

511

ICASSP 2022

Authorized licensed use limited to: NTT. Downloaded on November 07,2023 at 09:27:32 UTC from IEEE Xplore. Restrictions apply.

3. THE GMVAE METHOD

2. PROBLEM FORMULATION

3.1. CVAE and ACVAE Source Model

Suppose that there are J source signals and that I microphones
receive a mixture signal. Let sj (f, n) and xi (f, n) be the shorttime Fourier transform (STFT) coefficients of the j-th source signal
and the mixture signal at i-th microphone, where f and n are thefrequency and time indices, respectively. We denote the vectors
composed of the STFT coefficients of all the sources and the microphones as s(f, n) = [s1 (f, n), s2 (f, n), . . . , sJ (f, n)]T ∈ CJ andx(f, n) = [x1 (f, n), x2 (f, n), . . . , xI (f, n)]T ∈ CI , where (·)Trepresents the transpose and C denotes the set of complex numbers.
We begin by employing the local Gaussian modeling (LGM) [27],which assumes that sj (f, n) independently follows a zero-mean
complex Gaussian distribution with variance, i.e., power spectral
density (PSD), vj (f, n)(= E[|sj (f, n)|2 ]):sj (f, n) ∼ NC (0, vj (f, n)).

The original GMVAE method represents the source power spectrograms as the decoder outputs of the CVAE that is trained in advanceusing labeled training examples. Given a normalized source spectrogram S̃ and the one-hot encoded label c, the encoder and decoder
distributions are assumed to follow a Gaussian distribution and azero-mean complex Gaussian distribution:qφ (Z|S̃, c) = N (µφ (S̃, c), diagσ 2φ (S̃, c)),(8)pθ (S̃|Z, c) = NC (0, diagσ 2θ (Z, c)),(9)where µφ (S̃, c), σ 2φ (S̃, c), and σ 2θ (S̃, c) denote the encoder anddecoder outputs. For CVAE, both encoder and decoder network parameters φ and θ are trained by maximizing the following training
criterion:(1)

When sj (f, n) and sj 0 (f, n) are mutually independent for j 6= j 0 ,s(f, n) follows a complex Gaussian distribution:I(φ, θ) = E(S̃,c)∼pD (S̃,c) [EZ∼qφ (Z|S̃,c) [log pθ (S̃|Z, c)]− DKL [qφ (Z|S̃, c)||p(Z)],s(f, n) ∼ NC (0, V(f, n)),where E(S̃,c)∼pD (S̃,c) [·] denotes the sample mean over a dataset,
DKL [·||·] is the Kullback-Leivler (KL) divergence, and p(Z) is astandard Gaussian distribution N (0, I).
ACVAE is a CVAE variant that incorporates the expectation of
the mutual information I(c; S̃|Z) into the training criterion with theaim of making the decoder output S̃ ∼ pθ (S̃|Z, c) as correlated as
possible with the class label c. Since it is difficult to use the mutual
information directly, ACVAE uses the following variational lower
bound instead:where V(f, n) is a diagonal matrix whose diagonal entries arev1 (f, n), v2 (f, n), . . . , vJ (f, n). In a general situation where J can
outnumber I, a mixing system is given by:x(f, n) = A(f )s(f, n),(3)where A(f ) = [a1 (f ), a2 (f ), . . . , aJ (f )] ∈ CI×J is referred to asa mixing matrix. From Eqs. (2) and (3), x(f, n) is shown to follow
the complex Gaussian distribution with a zero-mean vector 0 and acovariance A(f )V(f, n)AH (f, n), where (·)H represents the conjugate transpose. We further employ a full-rank spatial covariancematrix (SCM) [2] on the outer product of a steering vector aj (f ):Rj (f ) = aj (f )aH
j (f ).

J (φ, θ, ψ) =E(S̃,c)∼pD (S̃,c),Z∼qφ (Z|S̃,c) [Ec0 ∼pD (c),S̃∼pθ (S̃|Z,c0 ) [log rψ (c0 |S̃)]],X
j

K(ψ) = E(S̃,c)∼pD (S̃,c) [log rψ (c|S̃)].
vj (f, n)Rj (f )).

(11)

where rψ (c|S̃) is an auxiliary classifier distribution with the network
parameter ψ. ACVAE also incorporates the cross-entropy:(4)x(f, n) can be rewritten as follows:x(f, n) ∼ NC (0,

(10)

(2)

(12)

Thus, the whole training criterion of ACVAE is given by:(5)

I(φ, θ) + λJ J (φ, θ, ψ) + λK K(ψ),(13)

Thus, the log-likelihood function L is given by:
c

L=−

X
f,n

where λJ ≥ 0 and λK ≥ 0 are weight parameters.
Since the decoder distribution is given in the same form as theLGM, using the trained decoder, Eq. (1) is reformulated as follows:h itr X(f, n)X̂−1 (f, n) + log detX̂(f, n) , (6)vj (f, n) = gj σ 2θ (f, n; Zj , cj ),cwhere = represents the equality up to constant, and X(f, n) =P
x(f, n)xH (f, n) and X̂(f, n) = j vj (f, n)Rj (f ).
Once PSDs {vj (f, n)}j,f,n and SCMs {Rj (f )}j,f are estimated, the j-th separated signal is obtained by applying a multichannel Wiener filter Mj (f, n):ŝj (f, n) = vj (f, n)Rj (f )|Xjvj (f, n)Rj (f ){z

Mj (f,n)

−1

(14)

where σ 2θ (f, n; Z, c) represents the (f, n)-th element of the decoder
output σ 2θ (Z, c). gj is the global scale of j-th source signal, which
compensates for the energy gap between the training and test time.
3.2. MM-based Parameter Estimation Algorithmx(f, n), (7)An MM algorithm refers to an iterative algorithm that searches fora stationary point of an objective function by iteratively maximizing
an auxiliary function called “minorizer” that is guaranteed to never
become above the objective function. Applying an MM algorithm}

followed by applying the inverse STFT.

512
Authorized licensed use limited to: NTT. Downloaded on November 07,2023 at 09:27:32 UTC from IEEE Xplore. Restrictions apply.

{Rj (f )}j,f . Similar to the MM-based algorithm, we can update thedecoder inputs {Zj }j and {cj }j by backpropagation:

to the log-likelihood function in Eq. (6), the following minorizer
LMM (≤ L) can be constructed [19]:cLMM = −



X
f,n,j

{Zj , cj }j ← {Zj , cj }j − η∇{Zj ,cj }j LEM .

tr(X(f, n)Pj (f, n)R−1
j (f )Pj (f, n))vj (f, n)+ vj (f, n)tr(K−1 (f, n)Rj (f )) , (15)(23)

The optimal updates of {gj }j and {Rj (f )}j,f are obtained as:
gj ←

1 X
1
tr(R−1j (f )Λj (f, n)), (24)f,n σ 2 (f, n; Zj , cj )F NIθwhere Pj (f, n) and K(f, n) are auxiliary variables defined by:
Pj (f, n) ← Mj (f, n),(16)K(f, n) ← X̂(f, n).

(17)

Rj (f ) ←1
1 X
Λj (f, n).
n gj σ 2 (f, n; Zj , cj )
N
θ

(25)

3.4. Discussion
An MM-based parameter estimation algorithm consists of updating auxiliary variables {Pj (f, n)}j,f,n and {K(f, n)}f,n , andmaximizing the minorizer with respect to {gj }j , {Zj }j , {cj }j , and{Rj (f )}j,f . The decoder inputs {Zj }j and {cj }j can be updated
by backpropagation:{Zj , cj }j ← {Zj , cj }j − η∇{Zj ,cj }j LMM ,We analyze the MM-based and EM-based algorithms in terms of matrix inversion and multiplication, ignoring the common backpropagation part. At each iteration, the MM-based algorithm is required
to repeat updating auxiliary variables Pj (f, n) and Kj (f, n) for updating each parameter, and multiple matrix inversions and multiplications are also required for updating {Rj (f )}j,f , which amounts(3N + 2J)F matrix inversions and (5N + 2)JF matrix multiplications. On the other hand, the EM-based algorithm updates an auxiliary variable Λj (f, n) once at each iteration, and {Rj (f )}j,f issimply updated by weighted sum, resulting in (N + J)F matrix inversions and 2N JF matrix multiplications at each iteration. Thus,
the EM-based parameter estimation algorithm takes less computational cost and it is expected to reduce computational time.

(18)

where η represents a learning rate. Note that, to take a sum-to-one
constraint on cj into account, we design a softmax layer that output
cj , and the layer input is treated as the parameter to be estimatedinstead. The optimal updates of {gj }j {Rj (f )}j,f are obtained as:
gj ← gjvPu1
u f,n σ2 (f,n;Ztr(X(f, n)Pj (f, n)R−1
j (f )Pj (f, n))j ,cj )
t
θ
P
,
×
2
−1
(f, n)Rj (f ))f,n σθ (f, n; Zj , cj )tr(K

4. EXPERIMENTAL EVALUATION
4.1. Experimental Settings(19)

The proposed method was experimentally evaluated under an underdetermined multichannel speech separation scenario where the task
Rj (f ) ← Ψ−1(20)
j (f )#Ωj (f ),is to separate out three sources from their mixtures captured by two
microphones.
where # denotes the geometricsemidefinite
We used audio samples from the Voice Conversion Challenge
P mean2 of two positive−1
matrices [28], and Ψj (f ) = n gj σθ (f, n; Zj , cj )K (f, n) and(VCC) 2018 dataset [29], which contains recordings of six female
P Pj (f,n)X(f,n)Pj (f,n).
Ωj (f ) = n
and six male U.S. English speakers and includes 116 utterances of
2 (f,n;Z ,c )g j σθ
j j
individual speakers. We used 100 utterances of two female and two
male speakers for training and another 10 utterances of different two
3.3. EM-based Parameter Estimation Algorithm
female and two male speakers for the test. Similar to [19], we generated 40 mixtures of three speakers by randomly choosing the utterAn EM algorithm maximizes a log-likelihood function by iteratively
ances from the test dataset, where a reverberation time T60 was set
maximizing the conditional expectation of the log-likelihood functo 78 ms. All the speech signals were sampled at 16 kHz, and STFT
tion for complete data called “Q-function” through iterative updates
analysis was conducted with a 128-ms window length with a 64-ms
called E- and M-steps. Regarding the mixture and source spectroshift length.
grams {x(f, n)}f,n and {s(f, n)}f,n as observed and unobserved
We used MNMF methods with the EM-based and MM-based
data, the following Q-function LEM (≤ L) can be obtained:
optimizations (EM-MNMF, MM-MNMF) [9, 10], and the ConvTas"#
Net [30] as the baseline methods, which are also used for initialX
tr(R−1cj (f )Λj (f, n))
LEM = −+ log detvj (f, n)Rj (f ) . izing the GMVAE method. The number of NMF bases was set tof,n,jvj (f, n)
10 for each speaker, and the spectral dictionaries of each speaker(21)
were obtained using Itakura-Saito NMF (IS-NMF) [31] with 1000
iterations. At the test time, both MNMFs were run for 300 iteraΛj (f, n) is the conditional expectation of the outer product oftions, where the intermideate separated signals at 200-th iterations(f, n) defined by:
were used for initializing the GMVAE method. We used an asteroid [32] recipe and trained a ConvTasNet model, using a singleΛj (f, n) ←Mj (f, n)x(f, n)xH (f, n)MHj (f, n)channel three-speaker source separation dataset generated from LibriSpeech [33], i.e., Libri3Mix [34]. After the training, we indepen+ (I − Mj (f, n))vj (f, n)Rj (f ),(22)dently fed the individual channels of a multichannel mixture signal to the ConvTasNet, and the separated signals at each channel
which amounts to conducting the E-step. The M-step consists ofwere then concatenated to construct multichannel separated signals.
maximizing the Q-function with respect to {gj }j , {Zj }j , {cj }j , and513
Authorized licensed use limited to: NTT. Downloaded on November 07,2023 at 09:27:32 UTC from IEEE Xplore. Restrictions apply.

10

8

Baseline
EM (1)EM (3)EM (9)EM (27)18

MM (1)
MM (3)MM (9)
MM (27)

16
14

Baseline
EM (1)EM (3)EM (9)EM (27)18

MM (1)
MM (3)MM (9)
MM (27)

16
14

4

2

0

12
10

10
8

Baseline
EM (1)EM (3)EM (9)EM (27)

MM-MNMF
Initialization method

8
6

4

4

2

2

ConvTasNet

0
EM-MNMF

2.0

MM (1)
MM (3)MM (9)
MM (27)10

6

0
EM-MNMF

1.8

MM (1)
MM (3)MM (9)
MM (27)

12
SIR [dB]6

ISR [dB]

SDR [dB]

12

Baseline
EM (1)EM (3)EM (9)EM (27)

Baseline
EM (1)EM (3)EM (9)EM (27)

MM-MNMF
Initialization method

ConvTasNet

EM-MNMF

MM-MNMF
Initialization method

ConvTasNet

1.0

MM (1)
MM (3)MM (9)
MM (27)

0.8

6

1.6

STOI [-]PESQ [-]SAR [dB]

8

1.4

0.6

0.4

4
1.2

2
0

0.2

1.0
EM-MNMF

MM-MNMF
Initialization method

ConvTasNet

0.0
EM-MNMF

MM-MNMF
Initialization method

ConvTasNet

Baseline
EM (1)EM (3)EM (9)EM (27)

MM (1)
MM (3)MM (9)
MM (27)

EM-MNMF

MM-MNMF
Initialization method

ConvTasNet

Fig. 1: Source separation performances under a speaker-open condition, where the error bars and the numbers in parentheses show the 95 %confidence intervals and the number of backpropagation optimizations, respectively.
Table 1: Computational times per iteration for a 6.89-second speech
mixture [s], where the numbers in parentheses show the number ofbackpropagation optimizations.
Optimization
approach
MM (CPU)
EM (CPU)
MM (GPU)
EM (GPU)

MNMF
0.95
3.34
−
−

or as fast as the MM-based MNMF. Furthermore, even when usinga CPU, the EM-based GMVAE method with one backpropagation is
as fast as the MM-based MNMF.
Fig. 1 shows a comparison of the separation performance of eachmethod with different initialization. We can see that, when increasing the number of backpropagations, the GMVAE method consistently achieves better performance and that the EM-based GMVAE
method achieves comparable performances with the MM-based GMVAE method. Comparing the initialization methods, we can see that,the better initialization method is used, i.e., ConvTasNet, the larger
performance improvements the GMVAE method can gain.
These results demonstrate that 1) the EM-based GMVAE
method is as fast as the conventional MM-based MNMF whileachieving better performance, 2) the EM-based GMVAE method isfaster than the MM-based GMVAE method while achieving comparable performance, and 3) better initialization methods can help the
GMVAE method achieve higher separation performances.

The GMVAE method
(1)
(3)
(9)
(27)
1.41 2.07 4.16 10.15
0.89 1.55 3.64
9.46
1.05 1.15 1.47
2.43
0.59 0.71 1.00
1.92

These separated signals were also used for initializing the GMVAE
method.
We used three-layer convolutional, three-layer deconvolutional,
and four-layer convolutional neural networks with gated linear units(GLUs) for the encoder, decoder, and auxiliary classifier, respectively. All the weight parameters λJ and λK were set to 1. The
number of training epochs was set to 1000. We used the Adam algorithm [35] with a learning rate of 0.0001 and 0.01 for the training
and test, respectively.
As the evaluation metrics, we used the signal-to-distortion ratio(SDR), the source image-to-spatial distortion ratio (ISR), the signalto-inference ratio (SIR), the signal-to-artifact ratio (SAR) [36], theperceptual evaluation of speech quality (PESQ) [37], and the shorttime objective intelligibility (STOI) [38].

5. CONCLUSION
This paper developed the GMVAE method for underdetermined multichannel source separation and investigated the MM-based and EMbased convergence-guaranteed parameter estimation algorithms. We
analyzed and compared the MM-based and EM-based algorithms
in terms of computational cost and separation performance, using
an ACVAE instead of a CVAE as the source model of the GMVAE
method. The experimental results demonstrated that the EM-based
GMVAE method consistently outperformed a conventional MNMF
and that the EM-based GMVAE was faster than the MM-based GMVAE method achieving comparable performance.
Future work includes use of inferences from encoder and auxiliary classifier [20] and jointly-diagonalizability constraint on a fullrank SCM [39, 40] for speeding up and improving the algorithm.

4.2. Experimental Results
Table 1 shows a comparison of the computational time of eachmethod, where faster performances are denoted in bold fonts. We
used an Intel(R) Xeon(R) CPU E5-2680 v4 @ 2.40GHz and a
NVIDIA Tesla K80 for the comparison. Unlike MNMF, in the GMVAE method, the EM-based algorithm is consistently faster than the
MM-based one. We can see that, when using a GPU, the EM-based
GMVAE methods with less than 10 backpropagations are faster than

514
Authorized licensed use limited to: NTT. Downloaded on November 07,2023 at 09:27:32 UTC from IEEE Xplore. Restrictions apply.

6. REFERENCES[20] L. Li, H. Kameoka, S. Inoue, and S. Makino, “FastMVAE:
A Fast Optimization Algorithm for the Multichannel Variational Autoencoder Method,” IEEE Access, vol. 8, pp. 228740–
228753, 2020.
[21] D. P. Kingma and M. Welling, “Auto-Encoding Variational
Bayes,” in Proc. ICLR, 2014.
[22] D. P. Kingma, S. Mohamed, D. J. Rezende, and M. Welling,“Semi-Supervised Learning with Deep Generative Models,” inProc. NIPS, pp. 3581–3589, 2014.
[23] D. R. Hunter and K. Lange, “A Tutorial on MM Algorithms,”
The American Statistician, vol. 58, no. 1, pp. 30–37, 2004.
[24] A. P. Dempster, N. M. Laird, and D. B. Rubin, “Maximum
Likelihood from Incomplete Data via the EM Algorithm,”
Journal of the Royal Statistical Society: Series B (Methodological), vol. 39, no. 1, pp. 1–22, 1977.
[25] H. Sawada, R. Ikeshita, and T. Nakatani, “Experimental Analysis of EM and MU Algorithms for Optimizing Full-rank Spatial
Covariance Model,” in Proc. EUSIPCO, pp. 885–889, 2021.
[26] H. Kameoka, T. Kaneko, K. Tanaka, and N. Hojo, “ACVAEVC: Non-Parallel Voice Conversion With Auxiliary Classifier
Variational Autoencoder,” IEEE/ACM Trans. ASLP, vol. 27,
no. 9, pp. 1432–1443, 2019.
[27] C. Févotte and J.-F. Cardoso, “Maximum likelihood approachfor blind audio source separation using time-frequency Gaussian source models,” in Proc. WASPAA, pp. 78–81, 2005.
[28] K. Yoshii, “Correlated Tensor Factorization for Audio Source
Separation,” in Proc. ICASSP, pp. 731–735, 2018.
[29] J. Lorenzo-Trueba, J. Yamagishi, T. Toda, D. Saito, F. Villavicencio, T. Kinnunen, and Z. Ling, “The Voice Conversion
Challenge 2018: Promoting Development of Parallel and Nonparallel Methods,” in Proc. Odyssey, pp. 195–202, 2018.
[30] Y. Luo and N. Mesgarani, “Conv-TasNet: Surpassing Ideal
Time–Frequency Magnitude Masking for Speech Separation,”
IEEE/ACM Trans. ASLP, vol. 27, no. 8, pp. 1256–1266, 2019.
[31] C. Févotte, N. Bertin, and J.-L. Durrieu, “Nonnegative Matrix
Factorization with the Itakura-Saito Divergence: With Application to Music Analysis,” Neural computation, vol. 21, no. 3,
pp. 793–830, 2009.
[32] M. Pariente, S. Cornell, J. Cosentino, S. Sivasankaran, E. Tzinis, J. Heitkaemper, M. Olvera, F.-R. Stöter, M. Hu, J. M.
Martı́n-Doñas, D. Ditter, A. Frank, A. Deleforge, and E. Vincent, “Asteroid: The PyTorch-Based Audio Source Separation
Toolkit for Researchers,” in Proc. Interspeech, pp. 2637–2641,
2020.
[33] V. Panayotov, G. Chen, D. Povey, and S. Khudanpur, “Librispeech: An ASR corpus based on public domain audio
books,” in Proc. ICASSP, pp. 5206–5210, 2015.
[34] J. Cosentino, M. Pariente, S. Cornell, A. Deleforge, and E. Vincent, “LibriMix: An Open-Source Dataset for Generalizable
Speech Separation,” 2020.
[35] D. P. Kingma and J. L. Ba, “Adam: A Method for Stochastic
Optimization,” in Proc. ICLR, 2015.
[36] E. Vincent, R. Gribonval, and C. Févotte, “Performance measurement in blind audio source separation,” IEEE Trans. ASLP,
vol. 14, no. 4, pp. 1462–1469, 2006.
[37] A. W. Rix, J. G. Beerends, M. P. Hollier, and A. P. Hekstra, “Perceptual evaluation of speech quality (PESQ)-a new
method for speech quality assessment of telephone networks
and codecs,” in Proc. ICASSP, pp. 749–752, 2001.
[38] C. H. Taal, R. C. Hendriks, R. Heusdens, and J. Jensen, “AnAlgorithm for Intelligibility Prediction of Time–Frequency
Weighted Noisy Speech,” IEEE Trans. ASLP, vol. 19, no. 7,
pp. 2125–2136, 2011.
[39] K. Sekiguchi, Y. Bando, A. A. Nugraha, K. Yoshii, and
T. Kawahara, “Fast Multichannel Nonnegative Matrix Factorization with Directivity-Aware Jointly-Diagonalizable Spatial
Covariance Matrices for Blind Source Separation,” IEEE/ACM
Trans. ASLP, vol. 28, pp. 2610–2625, 2020.
[40] N. Ito, R. Ikeshita, H. Sawada, and T. Nakatani, “A Joint Diagonalization Based Efficient Approach to Underdetermined
Blind Audio Source Separation Using the Multichannel Wiener
Filter,” IEEE/ACM Trans. ASLP, vol. 29, pp. 1950–1965, 2021.

[1] A. Hyvärinen and E. Oja, “Independent Component Analysis:
Algorithms and Applications,” Neural networks, vol. 13, no. 45, pp. 411–430, 2000.
[2] N. Q. Duong, E. Vincent, and R. Gribonval,“UnderDetermined Reverberant Audio Source Separation Using a
Full-Rank Spatial Covariance Model,” IEEE Trans. ASLP,
vol. 18, no. 7, pp. 1830–1840, 2010.
[3] H. Sawada, S. Araki, and S. Makino, “Underdetermined
Convolutive Blind Source Separation via Frequency Bin-Wise
Clustering and Permutation Alignment,” IEEE Trans. ASLP,
vol. 19, no. 3, pp. 516–527, 2010.
[4] T. Kim, T. Eltoft, and T.-W. Lee, “Independent Vector Analysis: An Extension of ICA to Multivariate Components,” inProc. ICA, pp. 165–172, 2006.
[5] A. Hiroe, “Solution of Permutation Problem in Frequency Domain ICA, Using Multivariate Probability Density Functions,”in Proc. ICA, pp. 601–608, 2006.
[6] D. D. Lee and H. S. Seung, “Learning the parts of objects by
non-negative matrix factorization,” Nature, vol. 401, no. 6755,
pp. 788–791, 1999.
[7] P. Smaragdis and J. C. Brown, “Non-negative matrix factorization for polyphonic music transcription,” in Proc. WASPAA,
pp. 177–180, 2003.
[8] A. Ozerov and C. Févotte, “Multichannel Nonnegative Matrix
Factorization in Convolutive Mixtures for Audio Source Separation,” IEEE Trans. ASLP, vol. 18, no. 3, pp. 550–563, 2009.
[9] S. Arberet, A. Ozerov, N. Q. Duong, E. Vincent, R. Gribonval,
F. Bimbot, and P. Vandergheynst, “Nonnegative matrix factorization and spatial covariance model for under-determined
reverberant audio source separation,” in Proc. ISSPA, pp. 1–4,
2010.
[10] H. Sawada, H. Kameoka, S. Araki, and N. Ueda, “Multichannel Extensions of Non-Negative Matrix Factorization With
Complex-Valued Data,” IEEE Trans. ASLP, vol. 21, no. 5,
pp. 971–982, 2013.
[11] H. Kameoka, T. Yoshioka, M. Hamamura, J. Le Roux, and
K. Kashino, “Statistical Model of Speech Signals Based on
Composite Autoregressive System with Application to Blind
Source Separation,” in Proc. LVA/ICA, pp. 245–253, 2010.
[12] D. Kitamura, N. Ono, H. Sawada, H. Kameoka, and
H. Saruwatari, “Determined Blind Source Separation Unifying
Independent Vector Analysis and Nonnegative Matrix Factorization,” IEEE/ACM Trans. ASLP, vol. 24, no. 9, pp. 1626–
1641, 2016.
[13] A. A. Nugraha, A. Liutkus, and E. Vincent, “Multichannel Audio Source Separation With Deep Neural Networks,”
IEEE/ACM Trans. ASLP, vol. 24, no. 9, pp. 1652–1664, 2016.
[14] S. Mogami, H. Sumino, D. Kitamura, N. Takamune,
S. Takamichi, H. Saruwatari, and N. Ono, “Independent
Deeply Learned Matrix Analysis for Multichannel Audio
Source Separation,” in Proc. EUSIPCO, pp. 1557–1561, 2018.
[15] Y. Bando, M. Mimura, K. Itoyama, K. Yoshii, and T. Kawahara, “Statistical Speech Enhancement Based on Probabilistic
Integration of Variational Autoencoder and Non-Negative Matrix Factorization,” in Proc. ICASSP, pp. 716–720, 2018.
[16] K. Sekiguchi, Y. Bando, A. A. Nugraha, K. Yoshii, and
T. Kawahara, “Semi-Supervised Multichannel Speech Enhancement With a Deep Speech Prior,” IEEE/ACM Trans.
ASLP, vol. 27, no. 12, pp. 2197–2212, 2019.
[17] S. Leglaive, L. Girin, and R. Horaud, “Semi-Supervised Multichannel Speech Enhancement with Variational Autoencoders
and Non-negative Matrix Factorization,” in Proc. ICASSP,
pp. 101–105, 2019.
[18] H. Kameoka, L. Li, S. Inoue, and S. Makino, “Supervised
Determined Source Separation with Multichannel Variational
Autoencoder,” Neural computation, vol. 31, no. 9, pp. 1891–
1914, 2019.
[19] S. Seki, H. Kameoka, L. Li, T. Toda, and K. Takeda, “Underdetermined Source Separation Based on Generalized Multichannel Variational Autoencoder,” IEEE Access, vol. 7,
pp. 168104–168115, 2019.

515
Authorized licensed use limited to: NTT. Downloaded on November 07,2023 at 09:27:32 UTC from IEEE Xplore. Restrictions apply.