Received November 19, 2020, accepted December 4, 2020, date of publication December 18, 2020,
date of current version December 31, 2020.
Digital Object Identifier 10.1109/ACCESS.2020.3045704

FastMVAE: A Fast Optimization Algorithm for theMultichannel Variational Autoencoder Method
LI LI 1 , (Student Member, IEEE), HIROKAZU KAMEOKASHOTA INOUE1 , AND SHOJI MAKINO1 , (Fellow, IEEE)

2 , (Senior Member, IEEE),

1 Graduate School of Systems and Information Engineering, University of Tsukuba, Ibaraki 3050821, Japan
2 NTT Communication Science Laboratories, Kanagawa 2430198, Japan

Corresponding author: Li Li (lili@mmlab.cs.tsukuba.ac.jp)
This work was supported in part by the JSPS KAKENHI under Grant 18J20059, and in part by the JST CREST under Grant JPMJCR19A3.

ABSTRACT This paper proposes a fast optimization algorithm for the multichannel variational autoencoder (MVAE) method, a recently proposed powerful multichannel source separation technique. The MVAE
method can achieve good source separation performance thanks to a convergence-guaranteed optimization
algorithm and the idea of jointly performing multi-speaker separation and speaker identification. However,
one drawback is the high computational cost of the optimization algorithm. To overcome this drawback,this paper proposes using an auxiliary classifier VAE, an information-theoretic extension of the conditional
VAE (CVAE), to train the generative model of the source spectrograms and using it to efficiently update
the parameters of the source spectrogram models at each iteration of the source separation algorithm. We
call the proposed algorithm ‘‘FastMVAE’’ (or fMVAE for short). Experimental evaluations revealed thatthe proposed fast algorithm can achieve high source separation performance in both speaker-dependent andspeaker-independent scenarios while significantly reducing the computational time compared to the original
MVAE method by more than 90% on both GPU and CPU. However, there is still room for improvement of
about 3 dB compared to the original MVAE method.
INDEX TERMS Multichannel source separation, multichannel variational autoencoder (MVAE) method,
FastMVAE algorithm, auxiliary classifier VAE.

I. INTRODUCTION

Blind source separation (BSS), a technique for separating out
individual source signals from microphone array inputs without any information about the sources or array geometry, has awide range of applications, including hearing aids, automatic
speech recognition, music editing, and music information
retrieval.
In BSS, the frequency-domain approach is usually preferred since it enables a fast implementation comparedwith the time-domain approach. It is also notable in thatit provides the flexibility of allowing us to utilize various
models for the time-frequency (TF) representations of source
signals. One example of this approach involves independentvector analysis (IVA) [1], [2], which achieves frequency-wise
source separation and permutation alignment simultaneously
by assuming the magnitudes of the frequency components
originating from the same source vary coherently over time.
The associate editor coordinating the review of this manuscript andapproving it for publication was Lin Wang
228740.

Multichannel extensions of non-negative matrix factorization(NMF), e.g., multichannel NMF (MNMF) [3], [4] and independent low-rank matrix analysis (ILRMA) [5]–[7], adopt the
NMF concept for source spectrogram modeling in order tomake use of the spectro-temporal structure underlying eachsource as a clue to separation. Specifically, the power spectrogram of each source signal is approximated as the linear sumof a limited number of basis spectra scaled by time-varying
amplitudes. Owing to the fact that ILRMA reduces to IVA
when it has only one flat basis spectrum, it can be seenthat ILRMA has more flexibility than IVA in capturing thespectro-temporal structure in each source [6]. Each of thesemethods is designed to solve an inverse problem of estimating
source signals based on a generative model of mixture signals.
In this sense, these methods are categorized as a generative
approach.
Meanwhile, given the recent advances achieved by deepneural network (DNN)-based speaker separation methods,
including deep clustering (DC) [8], [9] and permutation
invariant training (PIT) [10], [11], a discriminative approachThis work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/

VOLUME 8, 2020

L. Li et al.: FastMVAE: A Fast Optimization Algorithm for the MVAE Method

has recently proved powerful in monaural source separation
tasks, including both speaker-dependent and -independent
scenarios [12]–[15]. The general idea is to train a DNN thatpredicts TF masks or TF embeddings from a given mixture
signal based on spectro-temporal features. When multiple
microphones are available, spatial information can be utilized to improve separation performance [16]. Although thesemethods can achieve reasonably good separation, the TF
masking process can cause unwanted distortion or musicalnoise in the separated speech. To avoid distortion and artificial noise and fully exploit the benefits of multichannel
inputs, some efforts have been made to integrate DNNs into
traditional microphone array processing frameworks such as
beamforming [17], [18].
The success of these single-channel DNN-based methods attests to the excellent ability of DNNs to capture andlearn the structure of spectrograms. Recently, some attempts
have also been made to incorporate DNNs into the generative approach mentioned earlier [19]–[23]. As an example,
a multichannel source separation method using a conditionalvariational autoencoder (multichannel VAE or MVAE forshort) [21] has been proposed with notable success in supervised determined source separation tasks. With the MVAE
method, a conditional VAE (CVAE) [24] is trained using
the spectrograms of clean speech samples along with thecorresponding speaker ID as a conditioning class variable.
This is done so that the trained decoder distribution can beused as a generative model of signals produced by all thesources included in a given training set, where the latent
space variables and the class variables are the parameters tobe estimated from an input mixture signal. This generativemodel is called the CVAE source model. At the separation
phase, the MVAE algorithm iteratively updates the demixing
matrix using the iterative projection (IP) method [25] and theunderlying parameters of the CVAE source model using a gradient descent method, where the gradients of the parameters
are computed by backpropagation. The separated signals canthen be obtained by applying the estimated demixing matrix
to the observed mixture signals. One important feature of thismethod is that it is designed to jointly perform multi-speakerseparation and speaker identification.1 This is particularly
reasonable since these two tasks are interdependent in thesense that the solution to one task can help find the solution
to the other task. Another advantage worth noting is that byusing a carefully chosen step size or applying a backtracking
line search, the model parameters can be updated so as not todecrease the log-likelihood at each iteration of the algorithm.
However, one downside is the high computational cost of the
backpropagation process involved in each iteration.
To address this drawback, this paper proposes anaccelerated version of the MVAE algorithm called the ‘‘FastMVAE (or fMVAE)’’ algorithm. The idea is to use an

auxiliary classifier VAE (ACVAE) [26], an informationtheoretic extension of a CVAE, to pretrain the generative
distribution of source spectrograms. An ACVAE consists of
decoder, encoder, and classifier networks. These three networks are trained simultaneously so that the decoder can beused as a generative model of spectrograms conditioned ona speaker ID, and the encoder and classifier can be used toinfer the latent variables characterizing the generative modeland the speaker ID from a spectrogram input. Since the backpropagation process involved in the original MVAE algorithm
can be replaced with the forward propagation of the trained
encoder and classifier networks, the entire algorithm can bemade extremely efficient. It should be noted that this paper isan extended full-paper version of our conference paper [27].
The additional contributions in comparison to [27] are as
follows:• To stabilize the parameter inference process of the
fMVAE algorithm, especially in speaker-independentconditions, we propose an improved version of the fastalgorithm based on a Product-of-Experts (PoE) framework [28] and evaluate the impact of different hyperparameter settings.
• We demonstrate the capability of the MVAE and fMVAE
algorithms to handle speaker-independent scenarios bysufficiently increasing the variety of speakers and thenumber of samples in the training dataset.
The rest of this paper is structured as follows. After
describing the formulation of the determined multichannel
BSS problem and the MVAE method in Section II, we review
related work in Section III. In Section IV, we present thecore idea of the fMVAE algorithm along with the ACVAE
concept and describe the details of the proposed algorithm.
We demonstrate the effectiveness of the proposed method
in both speaker-dependent and speaker-independent source
separation tasks in Section V. Finally, we conclude the paper
in Section VI.

1 When applied to other tasks such as music source separation and speechenhancement, it should be rephrased as musical instrument identification andspeech/noise classification, respectively.

s(f , n) = WH (f )x(f , n),W(f ) = [w1 (f ), . . . , wI (f )] ∈ CI ×I ,VOLUME 8, 2020

II. MVAE METHOD
A. PROBLEM FORMULATIONLet us consider a determined situation where I source signals
are captured by I microphones. Let xi (f , n) and sj (f , n) denotethe short-time Fourier transform (STFT) coefficients of thesignal observed at the ith microphone and the jth source signal, where f and n are the frequency and time indices, respectively. We denote the vectors containing x1 (f , n), . . . , xI (f , n)and s1 (f , n), . . . , sI (f , n) by
x(f , n) = [x1 (f , n), . . . , xI (f , n)]T ∈ CI ,s(f , n) = [s1 (f , n), . . . , sI (f , n)]T ∈ CI ,

(1)
(2)

where (·)T denotes transpose. In a determined situation,
the relationship between observed signals and source signals
can be described as(3)(4)
228741

L. Li et al.: FastMVAE: A Fast Optimization Algorithm for the MVAE Method

where WH (f ) is called the demixing matrix and (·)H denotes
the Hermitian transpose. The aim of BSS methods is toestimate W = {W(f )}f solely from the observation
X = {x(f , n)}f ,n .
In the following, we assume each source signal follows
the local Gaussian model (LGM) [29], [30]. Namely, sj (f , n)
is assumed to independently follow a zero-mean complex
proper Gaussian distribution with variance (power spectral
density) vj (f , n):p(sj (f , n)|vj (f , n)) = NC (sj (f , n)|0, vj (f , n)),(5)where vj (f , n) = E[|sj (f , n)|2 ]. If sj (f , n) and sj0 (f , n) (j 6= j0 )are independent, the density of s(f , n) becomesYp(sj (f , n)|vj (f , n))p(s(f , n)|V(f , n)) =j= NC (s(f , n)|0, V(f , n)),(6)where V(f , n) = diag[v1 (f , n), . . . , vI (f , n)]. From (3) and(6), the density of x(f , n) is obtained asp(x(f , n)|W(f ), V(f , n))= |WH (f )|2 p(s(f , n) = WH (f )x(f , n)|V(f , n)),(7)where |WH (f )|2 is the Jacobian of the mapping x(f , n) 7→s(f , n). Hence, the log-likelihood of the separation matrices W = {W(f )}f and source model parameters V ={vj (f , n)}f ,n,j given the observed mixture signals X ={x(f , n)}f ,n is given bylog p(X |W, V)
X
X= 2N
log | det WH (f )| +log p(Sj |V j )
f

=c 2N

X

j
H

log | det W (f )|

f

−

X
f ,n,jlog vj (f , n) +2
|wH
j (f )x(f , n)|vj (f , n)


,

(8)

where we have used =c to denote equality up to constantterms, and a bold italic font to indicate a set consisting of TF
elements, namely, Sj = {sj (f , n)}f ,n , and V j = {vj (f , n)}f ,n .
Note that (8) will be split into frequency-wise source separation problems if there is no additional constraint imposedon vj (f , n). This indicates that there is a permutation ambiguity in the separated components for each frequency. Thus,
we usually need to group together the separated components
of different frequency bins that originate from the same
source after or during source separation. This process is called
permutation alignment.
B. CVAE MODEL

One efficient way to eliminate the permutation ambiguity is
to incorporate a constraint into vj (f , n) so that the spectral
structures of sources can be utilized as a clue to the estimationof W. The idea of the MVAE method is to use a CVAE [24]conditioned on a class variable c to model the complex spectrograms S = {s(f , n)}f ,n of source signals. Here, c is a228742

FIGURE 1. Illustration of CVAE used in MVAE.

one-hot vector consisting of C elements, indicating to whichclass the spectrogram S belongs. For example, if we consider
speaker IDs as the class category, each element of c will be
associated with a different speaker, and c will be filled with 1
at the index of a certain speaker and with 0 everywhere else.
A VAE is a stochastic neural network model consistingof an encoder and decoder, and a CVAE is an extended
version that allows the encoder and decoder to include a
conditioning class variable. In a CVAE, the decoder is modeled as a neural network (decoder network) that produces
a set of parameters for a conditional distribution pθ (S|z, c)of data S given a latent space variable z and a class variable c, where θ denotes the network parameters. Figure 1
shows an illustration of CVAE. Computing the exact posteriorpθ (z|S, c) = pθ (S|z, c)p(z)/pθ (S|c) of z given S and c isusually very difficult since pθ (S|c) involves an intractable
integral over z. The idea of CVAEs is to sidestep thedirect computation of this posterior by introducing anotherneural network (encoder network) for approximating theexact posterior pθ (z|S, c). As with the decoder network,
the encoder network generates a set of parameters for theconditional distribution qφ (z|S, c), where φ denotes the network parameters. The goal is to learn the parameters of
the encoder and decoder networks so that the encoder distribution qφ (z|S, c) becomes consistent with the posteriorpθ (z|S, c) ∝ pθ (S|z, c)p(z). Specifically, we train the encoderand decoder networks so that KL[qφ (z|S, c)||pθ (z|S, c)]is minimized given M class-labeled training samples{Sm , cm }M
m=1 . Since KL[qφ (z|S, c)||pθ (z|S, c)] = log p(S) +KL[qφ (z|S, c||p(z))] − Ez∼qφ (z|S,c) log pθ (S|z, c), this process
amounts to minimizingJ (φ, θ) = E(S,c)∼pD (S,c) KL[qφ (z|S, c)||p(z)]−Ez∼qφ (z|S,c) log pθ (S|z, c) , (9)where E(S,c)∼pD (S,c) [·] can be approximated as the sample mean over {Sm , cm }M
m=1 , and KL[·||·] denotes the
Kullback-Leibler divergence. While pD (S, c) can be approximated as the empirical distribution of {Sm , cm }M
m=1 ,qφ (z|S, c), pθ (S|z, c) and p(z) are distributions that need tobe modeled.
In the MVAE method, a CVAE is used to model the entire
complex spectrogram S of an utterance, conditioned on a
speaker ID vector c. p(z) and qφ (z|S, c) are described as
Gaussian distributions as with a regular CVAE:
p(z) = N (z|0, I),(10)
VOLUME 8, 2020

L. Li et al.: FastMVAE: A Fast Optimization Algorithm for the MVAE Method

Algorithm 1 MVAE Algorithm
Require: Network parameter θ trained using (9), observed
mixture signal x(f , n), iteration number L
1: randomly initialize W, 9
2: optional: update W using a BSS method
3: for ` = 1 to L do
4:for each source j of J do
5:yj (f , n) = wH
j (f )x(f , n)6:
(updating source model parameters)
7:
initialize gj using (19)8:normalization: S̄j = {yj (f , n)/gj }f ,n9:
for k = 1 to 100 do
10:
update zj and cj using backpropagation
11:
while keeping θ fixed12:
end for13:
calculate σj2 (f , n; zj , cj , gj = 1, θ)
14:
update gj using (19)15:
compute vj (f , n) = gj · σj2 (f , n; zj , cj , gj = 1, θ)
16:(updating demixing matrices)17:
update wj (f ) using the IP method (17), (18)18:
end for
19: end forqφ (z|S, c) = N (z|µφ (S, c), diag(σ 2φ (S, c))),Y=
N (z(k)|µφ (k; S, c), σφ2 (k; S, c)), (11)kwhere z(k), µφ (k; S, c), and σφ2 (k; S, c) denote the kth ele-ment of the latent space variable z and the encoder outputsµφ (S, c) and σ 2φ (S, c), respectively. pθ (S|z, c) is defined asa zero-mean complex proper Gaussian distribution with thesame form as the LGM:
Qpθ (Sj |zj , cj ) = f ,n NC (sj (f , n)|0, σθ2 (f , n; zj , cj )), (12)where σθ2 (f , n; zj , cj ) denotes the (f , n)th element of thedecoder output. Once the parameters θ and φ of the encoderand decoder are trained using speaker-labeled training utterances, the decoder with fixed θ can be used as a generativemodel of spectrograms for each speaker at test time.
Normalizing the mean and variance of each training sample
is one of the common practices in neural network training.
Similarly, in the CVAE training in the MVAE method, thetotal energy of each training utterance is normalized to 1.
However, of course, the total energy of the spectrogram ofeach source in a test mixture can vary from source to sourceand does not necessarily equal 1. So that the generative model
can flexibly bridge this gap, a scale parameter g is additionally incorporated into (12) and treated as a free parameter tobe estimated at test time. Namely, the generative model ofthe complex spectrograms Sj of utterances of speaker j canbe expressed as
Qpθ (Sj |zj , cj , gj ) = f ,n pθ (sj (f , n)|zj , cj , gj ),
(13)
where
pθ (sj (f , n)|zj , cj , gj ) = NC (sj (f , n)|0, gj σθ2 (f , n; zj , cj )),(14)
VOLUME 8, 2020and zj , cj , and gj are the unknown parameters to be estimated.
(13) is called the CVAE source model. We can immediately
confirm that the decoder distribution in (12) corresponds toa particular case of (13) where gj = 1. Since the CVAE
source model is given in the same form as the LGM in (5),where vj (f , n) is given by gj σθ2 (f , n; zj , cj ), using it as thegenerative model of each source leads to the same form of
the log-likelihood as (8):log p(X |W, 9, G)
X
X= 2N
log | det WH (f )| +log pθ (Sj |zj , cj , gj )
f

=c 2N

j

X

H

log | det W (f )|

f

−

X
f ,n,jlog gj σθ2 (f , n; zj , cj ) +2|wH
j (f )x(f , n)|gj σθ2 (f , n; zj , cj )


,
(15)

where G = {gj }j and 9 = {zj , cj }j .
Since z is assumed to follow N (z|0, I) when θ and φ aretrained, it would be reasonable to assume it as a prior distribution for z also at test time. The prior p(c) is the empirical
distribution of the training examples {cm }m , expressed as amultinomial distribution. Thus, the log-posterior
log p(X |W, 9, G; θ) + log p(z) + log p(c)(16)

is the objective function to be maximized with respect to W,
9, and G. A stationary point of (16) can be found by iteratively updating these parameters so that (16) is guaranteed to
be non-decreasing. To update W, the following update rules,
called the iterative projection (IP) [25], can be used:wj ← (WH (f )6 j (f ))−1 ej ,(17)
wj (f )
wj ← H,(18)
wj (f )6 j (f )wj (f )P
where 6 j (f ) = N1 n x(f , n)xH (f , n)/vj (f , n) and ej denotes
the jth column of an I × I identity matrix. To update G,
the following update rule can be used:
1 X |wj (f )x(f , n)|.
FNσθ2 (f , n; zj , cj )
H

gj ←

2

(19)

f ,n

Note that (19) maximizes (16) with respect to gj when W
and 9 are fixed. While keeping W and G fixed, a gradientdescent method can be used to search for the optimal zj andcj that maximize (16), or equivalently log pθ (Sj |zj , cj , gj ) +log p(zj ) + log p(cj ) for each j in parallel, where each elementof Sj is given by sj (f , n) = wH
j (f )x(f , n). Note that estimating cj from a test mixture corresponds to identifying which
speaker is present in the mixture signal. When updating cj ,the sum-to-one constraint must be taken into account. Thisis easily implemented by inserting an appropriately designed
softmax layer that outputs cj ,cj = softmax(uj ),(20)228743

L. Li et al.: FastMVAE: A Fast Optimization Algorithm for the MVAE Method

FIGURE 2. Illustration of ACVAE used in fMVAE.

and treating uj as the parameter to be estimated instead.
The source separation algorithm of the MVAE method issummarized in Algorithm 1. This algorithm is noteworthy
that, if it is implemented appropriately, the log-likelihood of
the model parameters is guaranteed to be non-decreasing at
each iteration.
III. RELATED WORK
A. ILRMA

Another reasonable way of constraining power spectrograms
involves employing the NMF model [31]. The NMF model
expresses vj (f , n) as a linear sum of spectral templates
bj,1 (f ), . . . , bj,t (f ), . . . , bj,Tj (f ) ≥ 0 scaled by time-varying
magnitudes hj,1 (n), . . . , hj,t (n), . . . , hj,Tj (n) ≥ 0:
vj (f , n) =Tj
X

bj,t (f )hj,t (n).

(21)

t

Note that a particular case where Tj = 1 and bj,t
) = 1 for all
q(fP2
j is equivalent to assuming the norm rj (n) =f |sj (f , n)|follows a complex Gaussian distribution with time-varying
variance hj (n). This is analogous to the assumption in IVA
that the magnitudes of the STFT coefficients in all frequency
bands originating from the same source tend to vary coherently over time [32].
The optimization algorithm of ILRMA consists of iteratively updating the demixing matrices W using the IP
method, the basis templates B = {bj,t (f )}f ,j,t , and the activation matrix H = {hj,t (n)}n,j,t so that (8) is guaranteed to be
non-decreasing at each iteration. To update B and H, we canuse the majorization-minimization (MM) algorithm [33]. The
MM-based update rules can be derived assP
22n |yj (f , n)| hj,t (n)/vj (f , n)P,
bj,t (f ) ← bj,t (f )n hj,t (n)/vj (f , n)v
uPu f |yj (f , n)|2 bj,t (f )/vj (f , n)2P
hj,t (n) ← hj,t (n)t.
f bj,t (f )/vj (f , n)
B. DNN-BASED METHODS

Some attempts have recently been made to incorporate DNNs
into the LGM-based multichannel source separation framework [19], [20]. With these methods, vj (f , n) is updated ateach iteration as the output of pretrained DNNs. Independent
228744

deeply low-rank matrix analysis (IDLMA) [20] is a method
designed to train a DNN for each source so that the jth
DNN produces spectra related to source j when noisy spectraof the jth source are given as the input. Thus, each DNN
can be seen as a source-dependent noise reduction system.
One drawback of IDLMA is that updating vj (f , n) in thisway does not guarantee an increase in the log-likelihood.
Another drawback would be that it can perform poorly in
speaker-independent scenarios.
C. VAE-BASED METHODS

Recently, deep generative models such as VAEs and generative adversarial networks (GANs) have proved powerfulin source separation tasks [22], [23], [27], [34]–[40]. The
idea of using a VAE to model the spectrum within each
short-term frame was first proposed for single-channel speechenhancement [22]. This method, called VAE-NMF, enables
speech enhancement in a semi-supervised manner by using
a VAE to model the spectrogram of a target speaker and an
NMF model to express unseen noise spectrograms. In thismethod, the Metropolis algorithm is used to iteratively update
the latent space variable z. An extension of this model was
subsequently developed, which incorporates a loudness gain
for robust speech modeling and adopts a noise model basedon alpha-stable distributions [23], [36]. The Monte Carlo
expectation-maximization algorithms were used for estimating the model parameters.
To the best of our knowledge, the idea of incorporating
the VAE concept into the multichannel framework was first
introduced in a preprint article [41] and later published asa journal paper [21]. Unlike the above VAE-NMF methods,
this method, namely the MVAE method, uses a CVAE witha fully convolutional architecture to model the entire spectrogram of an utterance of each source. While the original
MVAE method was designed to deal with determined anechoic mixtures only, its modified versions have subsequently
been proposed to handle underdetermined scenarios [34] andhighly reverberant conditions [39]. Like the original version,these two versions use gradient descent (backpropagation)
to update the source model parameters. Extensions of the
VAE-NMF methods to multichannel inputs were later developed [35], [37], [42] for application to multichannel speechenhancement tasks. In these methods, the Markov chain
Monte Carlo (MCMC) methods [43], such as Gibbs sampling
VOLUME 8, 2020

L. Li et al.: FastMVAE: A Fast Optimization Algorithm for the MVAE Method

and the Metropolis algorithm, are used to iteratively update
the latent space variable as with the original VAE-NMF
methods.
Although these methods have been shown to perform
impressively compared with conventional NMF-based methods, the use of sampling and backpropagation to update latent
space variables can be computationally expensive. To reducethe computational cost, we previously proposed to exploit
the pretrained encoder of a CVAE as an approximate posterior estimator to infer the latent space variable z in [27].
With the same motivation, a fast algorithm for estimating theparameters of the VAE-NMF model was later derived basedon the Bayesian inference in [38] for single-channel speechenhancement.
IV. FMVAE ALGORITHM
A. IDEAIn this section, we describe the idea of the proposed fast
optimization algorithm for the MVAE method. Since theprocess of updating the parameters of the CVAE source model
is more computationally costly than that of updating theother parameters, our main focus is on how to accelerate thisprocess. When W is fixed, each element of Sj will be fixedat sj (f , n) = wH
j (f ) x(f , n). Now, since the terms that dependon zj and cj in (16) are given aslog pθ (Sj |zj , cj , gj ) + log p(zj ) + log p(cj )c= log pθ (zj , cj |Sj , gj ),(22)

we would like to find zj and cj that maximize the posteriorp(zj , cj |Sj , gj ) after updating W. This posterior can be factorized as p(zj , cj |Sj , gj ) = p(zj |Sj , cj , gj )p(cj |Sj , gj ). Here,
we notice that the first factor, p(zj |Sj , cj , gj ), resembles theencoder (or inference) distribution in the CVAE in (11), withthe difference being that it is also conditioned on the scaleparameter gj . Since the total energy of each training utterance
is assumed to be normalized to 1 in the CVAE training as
mentioned earlier, gj can be thought of as a parameter thatplays the role of normalizing the total energy of an unnormalized input Sj to 1 at test time so that the scale of theencoder input is ensured to be consistent with the training
utterances. Specifically, the encoder distribution that allows
for unnormalized inputs is implicitly assumed to be given as
the following expression:qφ (z|S, c, g) = N (z|µφ (S/g, c), diag(σ 2φ (S/g, c))),Y=
N (z(k)|µφ (k; S/g, c), σφ2 (k; S/g, c)),k

(23)
which reduces to (11) when g = 1. Thus, we can use
the trained encoder qφ (zj |Sj , cj , gj ) as an approximation of
the first factor of the posterior p(zj , cj |Sj , gj ). This meansthat if we could obtain the true distribution p(cj |Sj , gj ) or
its approximate distribution r(cj |Sj , gj ), we would be able tofind an approximation of the maximum point of the posterior
VOLUME 8, 2020

p(zj , cj |Sj , gj ) by finding the maximum point of the corresponding approximate distribution.
In this section, we review the concept of an auxiliary
classifier VAE (ACVAE), present how this concept can beused to obtain r(cj |Sj , gj ), and introduce the details of the
proposed optimization algorithm.
B. AUXILIARY CLASSIFIER VAE

An auxiliary classifier VAE (ACVAE) [26] is a CVAE variant,which incorporates an information-theoretic regularization[44] that assists in making the decoder outputs as correlated as
possible with the class variable c by maximizing the mutual
information between c and an output S ∼ pθ (S|z, c) fromthe decoder, conditioned on z. The mutual information isexpressed as
I (c, S|z) = Ec∼pD (c),S∼pθ (S|z,c),c0 ∼p(c|S) [log p(c0 |S)]+H (c),(24)
where pD (c) is the empirical distribution of c in the training
dataset, and H (c) represents the entropy of c, which canbe considered as a constant term. Although it is difficult
to optimize I (c, S|z) directly since it requires access to theposterior p(c|S), we can derive a variational lower bound of
the first term of I (c, S|z) by using a variational distributionr(c|S) to approximate p(c|S):
Ec∼pD (c),S∼pθ (S|z,c),c0 ∼p(c|S) [log p(c0 |S)]r(c0 |S)p(c0 |S) = Ec∼pD (c),S∼pθ (S|z,c),c0 ∼p(c|S) log
r(c0 |S)00= Ec∼pD (c),S∼pθ (S|z,c) KL[p(c |S)||r(c |S)]
+ Ec0 ∼p(c|S) [log r(c0 |S)]
≥ Ec∼pD (c),S∼pθ (S|z,c),c0 ∼p(c| VecS) [log r(c0 |S)]
= Ec∼pD (c),S∼pθ (S|z,c) [log r(c|S)],(25)
where the equality holds if and only if r(c|S) = p(c|S).
This technique of lower bounding mutual information is
known as variational information maximization [45]. The lastline of (25) follows the lemma presented in [44]. Therefore,
we can indirectly maximize I (c, S|z) by increasing the lower
bound with respect to pθ (S|z, c) and r(c|S). One way toachieve this involves expressing the variational distributionr(c|S) as a neural network and training it along with qφ (z|S, c)and pθ (S|z, c). Specifically, r(c|S) can be expressed as amultinomial distributionrψ (c|S) = Mult(c|ρ ψ (S)).
(26)
Q
Here, Mult(c|ρ) ∝ i ρici denotes a multinomial distribution,
where c = [c1 , . . . , cI ]T and ρ = [ρ1 , . . . , ρI ]T . ρ ψ (S)denotes a neural network that takes S as an input and produces
a probability vector consisting of C elements. (26) is called
an auxiliary classifier.
Therefore, the regularization term that we would like tomaximize over the training samples with respect to φ, θ, and228745

L. Li et al.: FastMVAE: A Fast Optimization Algorithm for the MVAE Method

FIGURE 3. Flowchart of fMVAE for I = 2 case.

ψ becomesL(φ, θ, ψ) = E(S,c)∼pD (S,c),qφ (z|S,c) [Ec∼pD (c),S∼pθ (S|z,c) [log rψ (c|S)]],(27)

where rψ (c|S) must satisfy the sum-to-one constraint. With
the regularization term (27), the auxiliary classifier is trainedusing only the reconstructed spectrograms. Since we can also
use the spectrograms of real speech to train the auxiliary
classifier, we can further use the cross-entropy
I(ψ) = E(S,c)∼pD (S,c) [log rψ (c|S)](28)

as the training criterion. The entire training criterion is thus
given byJ (φ, θ) − λL L(φ, θ, ψ) − λI I(ψ),(29)

where λL ≥ 0 and λI ≥ 0 are the parameters weighing
the importance of the regularization terms. Figure 2 shows
an illustration of ACVAE.
C. FAST ALGORITHM

As mentioned above, the auxiliary classifier distributionrψ (c|S) trained using {Sm , cm }M
m=1 is expected to be a good
approximation of the conditional distribution p(c|S). Now,
in the same way that we considered the encoder that flexibly
allows for an unnormalized input, here we also consider
an auxiliary classifier rψ (c|S, g) that incorporates the global
scale parameter g such thatrψ (c|S, g) = Mult(c|ρ ψ (S/g)).

(30)

Using the trained auxiliary classifier and encoder, we canobtain an approximation p(zj , cj |Sj , gj ) ≈ rψ (cj |Sj , gj )qφ (zj |Sj , cj , gj ). Since the maximum points of rψ (cj |Sj , gj )and qφ (zj |Sj , cj , gj ) can be found immediately, we can use
these approximate distributions to find an approximate solution to (zj , cj ) = argmaxzj ,cj p(zj , cj |Sj , gj ) instead of thegradient descent update for increasing log pθ (Sj |zj , cj , gj ) +log p(zj ) + log p(cj ). Figure 3 shows the flowchart of the
proposed algorithm for the I = 2 case. The algorithm is228746summarized in Algorithm 2. The main difference between the
new algorithm from the original version is that the optimal zj
and cj are estimated using the forward propagations of thetwo pretrained networks instead of using gradient descent
updates. Specifically, zj is given as the mean of the encoder
distribution µφ (Sj /gj , cj ). There are two possible ways to
update the class variable cj . One is to directly use the probability vector produced by the auxiliary classifier network
cj ← ρ ψ (Sj /gj ).

(31)

We hereafter refer to the proposed algorithm using this update
rule as fMVAE_c. The other is to use the one-hot vector closest
to the output of the auxiliary classifier(
1, (k = k̂),[cj ]k ←(32)
0, (k 6= k̂),k̂ = argmax [ρ ψ (Sj /gj )]k ,

(33)

k

where [·]k is used to denote the kth element of a vector. We hereafter refer to the algorithm using this update
rule as fMVAE_o. Here, the subscripts are the first lettersof ‘‘continuous’’ and ‘‘one-hot’’, respectively. rψ (cj |Sj , gj )can be seen as a speaker recognizer trained with explicit
supervision. Hence, the proposed algorithm is expected toperform better than the original version in terms of speaker
identification accuracy. However, one downside would bethat it does not guarantee a non-decrease in the objectivefunction because of the approximation p(zj , cj |Sj , gj ) ≈rψ (cj |Sj , gj )qφ (zj |Sj , cj , gj ). How this actually affects source
separation performance will be discussed later.
D. PRIOR-WEIGHTED INFERENCE

The encoder network is trained so that qφ (z|S, c) becomesas close as possible to p(z) = N (z|0, I). However, throughpreliminary experiments, we found that at test time thetrained encoder occasionally produced outliers that significantly deviated from the assumed distribution N (z|0, I). Thismay be because the encoder did not generalize very well
VOLUME 8, 2020

L. Li et al.: FastMVAE: A Fast Optimization Algorithm for the MVAE Method

Algorithm 2 fMVAE Algorithm
Require: Network parameter θ, φ, ψ trained using (29),observed mixture signal x(f , n), iteration number L ,weight parameter α
1: randomly initialize W, 9
2: optional: update W using a BSS method
3: for ` = 1 to L do
4:for each source j of J do
5:yj (f , n) = wH
j (f )x(f , n)6:
(updating source model paremeters)
7:
initialize gj using (19)8:normalization: S̄j = {yj (f , n)/gj }f ,n9:update cj using (31) or (32)10:
update zj using (36)11:
compute σj2 (f , n; zj , cj , gj = 1, θ)
12:update gj using (19)13:
compute vj (f , n) = gj · σj2 (f , n; zj , cj , gj = 1, θ)
14:(updating demixing matrices)15:
update wj (f ) by IP method with (17), (18)16:
end for17: end fordue to the limited amount of training data or the mismatch
between the training and test conditions. Since the decoder
network was trained under the assumption that its input follows N (z|0, I), these outliers tended to negatively affect theresulting decoder outputs and eventually the estimate of W.
One heuristic way to address this problem would be to reapply
the prior distribution p(z) during inference. In the following,
we omit the source index j in this subsection for simplicity ofnotation.
As a way of reapplying the prior, we adopt the concept of
product-of-experts (PoE) [28] and define ẑ asẑ = argmax p(z|S, c, g)p(z)αz≈ argmax qφ (z|S, c, g)p(z)αz= argmax log qφ (z|S, c, g) + α log p(z),

(34)

z

where α weighs the importance of the prior in the inference.
Since both qφ (z|S, c, g) and p(z) are multivariate Gaussian
distributions, (34) can be expressed aslog qφ (z|S, c, g) + α log p(z)
T
 α T
1
=c − z − µφ (S/g, c) 6 −1
φ z − µφ (S/g, c) − z z
2
2
6 −1
+ αI
φ
=c −
(z − µ)T (z − µ),
(35)
2
−1
where 6 φ = diag(σ 2φ (S/g, c)) and µ = 6 −1
φ (6 φ +−1αI) µφ (S/g, c). Therefore, the update rule for z can beeasily derived as−1−1z ← 6 −1
φ (6 φ + αI) µφ (S/g, c).

(36)

Note that (36) reduces to the mean of the encoder distribution
when α = 0.
VOLUME 8, 2020

E. POTENTIAL ADVANTAGE OF CVAE OVER REGULAR VAE
IN TERMS OF SOURCE MODELING

Although the MVAE method uses a CVAE for source spectral
modeling, one can also think of using a regular (unconditional) VAE, as in the VAE-NMF framework. In this case,
all the factors of variations in speech spectra, including
the speaker identity factor, will be encoded into the latentvariables. However, this can lead to an overparametrized
representation since even though the speaker identity factor should be considered time-invariant (unlike phonemeand F0 -related factors), the latent variables are allowed tovary over time. Hence, when estimating the latent variablesequence of each source in a given mixture, we would want
to separate out only the speaker identity factor from the latentvariable sequence and force it to be time-invariant so as not toallow it to change during the utterance. This is the motivation
behind the idea of using a CVAE instead of a regular VAE. A
quantitative comparison between these choices is provided in
Subsection V-E.
V. EXPERIMENTAL EVALUATIONS

To evaluate the effectiveness of the proposed method,
we conducted several multi-speaker source separation
experiments in which we considered speaker-dependentand speaker-independent separation tasks. Specifically,
the speaker-dependent and speaker-independent conditions
indicate whether the test speaker is seen in the training
dataset. It should be noted that even in the speaker-dependentcondition, the training and test sets are disjoint at the sentence level. In this section, we first provide the details of
the baseline algorithms in Subsection V-A and the networkarchitectures used in the baseline and proposed methods
in Subsection V-B. We then show how the dataset was
created and present the experimental results obtained under
the speaker-dependent condition in Subsections V-C – V-F.
In Subsection V-G, we describe the large-scale dataset
designed for the speaker-independent task and show theexperimental results.
A. BASELINE METHODS FOR COMPARISON

We chose ILRMA [6], IDLMA [46], and the original MVAE
method 2 [21] as the baseline methods for comparison.
We tested several different versions of the proposed and baseline methods. We use the terms ‘‘supervised/unsupervised’’and ‘‘informed/uninformed’’ to properly categorize each version of the methods. The terms ‘‘supervised’’ and ‘‘unsupervised’’ indicate whether a method requires training examples
of source signals prior to source separation, while the terms‘‘informed’’ and ‘‘uninformed’’ indicate whether a method
is informed about which sources are present in a test mixture signal. Categorization of each version is summarized inTable 1.
We set the basis number Tj = 10 for u.u.ILRMA andrandomly initialized the basis spectra and activation matrix.
2 Code: https://github.com/lili-0805/MVAE

228747

L. Li et al.: FastMVAE: A Fast Optimization Algorithm for the MVAE Method

TABLE 1. Methods for comparison.

FIGURE 5. Learning curves of CVAE and ACVAE source models.

can be written as

Ol = (Ol−1 ∗ Wlf + bfl ) ⊗ σ (Ol−1 ∗ Wlg + Bgl ), (37)
W Wg B

FIGURE 4. Network architectures of the encoder and decoder used for
MVAE and fMVAE and the classifier used for fMVAE. The inputs andoutputs are one-dimensional data, where the frequency dimension of thespectrograms is regarded as the channel dimension. The ‘w’, ‘c’, and ‘k’denote the width, channel number, and kernel size, respectively. Convand Deconv denote one-dimensional convolution and deconvolution; BN
and GLU stand for batch normalization and gated linear unit.

For supervised ILRMA, basis spectra with T = 10 were pretrained for each speaker in the training dataset using the NMF
algorithm. They were then concatenated and used as a unified
model to represent all the sources in s.u.ILRMA, whereas thebasis spectra corresponding to the specific speakers present ina mixture signal were provided to the method in s.i.ILRMA.
B. NETWORK ARCHITECTURES

Figure 4 depicts the details of the network architectures
employed in the MVAE and fMVAE methods. We used
the same network architectures to train the CVAE and
ACVAE. All the networks were designed to be fully convolutional to handle input spectrograms of signals with arbitrary lengths. We used one-dimensional gated convolutional
neural networks (CNNs) [47] to model spectrograms, whichallows the networks to capture time dependencies in spectral
sequences. Gated CNNs were initially introduced to model
word sequences for language modeling and shown to outperform long short-term memory (LSTM) language models
trained in a similar setting. The effectiveness of employing
a gated CNN to model a spectrogram has already been confirmed [48], [49]. By using l−1 to denote the output of the(l − 1)th layer, the output of the lth layer l of a gated CNN

O228748OBg

ff , and
where
l,
l,
l
l are weight and bias parameters of the lth layer, ⊗ denotes element-wise multiplication, and σ is the sigmoid function. The main difference
between a gated CNN and a regular CNN layer is that agated linear unit (GLU), namely the second term of (37),is used as a nonlinear activation function. Like LSTMs,
GLUs have data-driven gates, which control the information
passed on in the hierarchy. At each gated CNN layer in theencoder and decoder, a broadcast version of c is appended
along the channel dimension to the output of the previous
layer. Adam [50] was used to train the networks. Note that
Algorithm 1 and Algorithm 2 correspond to s.u.MVAE ands.u.fMVAE_o/s.u.fMVAE_c, respectively. For s.i.MVAE ands.i.fMVAE, the correct class label cj is given and fixed during
the update. Figure 5 shows the learning curves of the CVAE
and ACVAE training processes. The curves demonstrate thatthe networks were trained stably with fast convergence.
For s.i.IDLMA, we used a fully connected neural networkwith four hidden layers. Each layer had 1024 units, and a rectified linear unit was used for the output of each layer, which
was the same as the network architecture used in [46]. We
implemented the training settings described in [46], namely
using the Gaussian-IDLMA loss function and concatenationof the current, preceding, and succeeding frames to capture
the temporal dependency, data augmentation, and regularization. The only difference was the optimization algorithm,
where we used Adam to train the network for 700 epochs
instead of Adadelta [51] for 200 epochs. More training details
are available in [46].

C. DATASET FOR SPEAKER-DEPENDENT SEPARATIONAs in the original MVAE paper [21], we used speechutterances of two male speakers (SM1, SM2) and two
female speakers (SF1, SF2) excerpted from the Voice
Conversion Challenge (VCC) 2018 dataset [52] for the
speaker-dependent source separation experiment. The audio
files for each speaker were about seve minutes long andmanually segmented into 116 short sentences, where 81 andVOLUME 8, 2020

L. Li et al.: FastMVAE: A Fast Optimization Algorithm for the MVAE Method

TABLE 2. Average SDR [dB] obtained with various STFT settings. The bold
font shows the best scores.

FIGURE 6. Configuration of room, where ◦ and × represent the positionsof microphones and sources, respectively.

35 sentences (about five and two minutes long, respectively)
served as training and test sets, respectively.
We used two-channel mixture signals of two sources as
the test data, which were synthesized using simulated room
impulse responses (RIRs) generated using the image method[53] and real RIRs measured in an anechoic room (ANE)
and an echo room (E2A). Figure 6 shows the configurationof the room used for simulating RIRs. To meet the instantaneous mixing model assumption, the reverberation times(RT60 ) [54] of the simulated RIRs were set at 78 and 351 ms,
which were controlled by setting the reflection coefficient of
the walls at 0.20 and 0.80, respectively. For the measured
RIRs, we used the data included in the RWCP Sound Scene
Database in Real Acoustic Environments [55]. The RT60 of
ANE and E2A were 173 and 225 ms, respectively. The test
data included 4 pairs of speakers, i.e., SF1+SF2, SF1+SM1,
SM1+SM2, and SF2+SM2. For each speaker pair, we generated ten mixture signals. Hence, there were a total of 40 test
signals for each reverberation condition, each of which was
about four to seven seconds long. All the speech signals were
resampled at 16 kHz.
D. EXPERIMENTAL ANALYSIS OF WINDOW LENGTH,
INITIALIZATION, AND WEIGHT PARAMETER α

In this subsection, we compare the separation performanceacross different STFT window lengths, different initialization
methods for the MVAE and fMVAE algorithms, and different
α settings.
Since all the methods are based on the instantaneous linear
mixture model, the STFT window length may affect the
separation performance of each of them, especially underreverberant conditions. We computed the STFT using a Hamming window with a length of {32, 64, 128, 256} ms, and byshifting half of the length for each frame. In this experiment,
all the MVAE and fMVAE methods were initialized by running u.u.ILRMA for 30 iterations. The MVAE or fMVAE
algorithm was then run for 30 iterations, where Adam wasused to update zj and cj in the MVAE methods with a step size
set of 0.01. We used α = 0 for fMVAE in this experiment.
Table 2 shows the SDR scores obtained with each method.
From these results, the optimal window length that gave thebest overall performance was 128 ms for the current dataset.
VOLUME 8, 2020

TABLE 3. Average SDR [dB] obtained by MVAE and fMVAE methods
adopting different initialization approaches. The bold font shows the bestscores.

Therefore, we conducted all the following experiments using
a window length of 128 ms.
To confirm the impact of the initialization for the MVAE
and fMVAE methods on the source separation performance,
we compared the algorithms using the following three initialization methods: 1) random initialization with the demixing matrices initialized at identity matrices; 2) IVA; and3) u.u.ILRMA. To keep the number of updates of the demixing matrices constant, each algorithm was run for 60 iterations for the random initialization case and 30 iterations
after an initialization algorithm was run for 30 iterations
for the other cases. Table 3 shows the SDR scores overthe 160 test samples. From these results, we found that themethods adopting ILRMA for initialization achieved better
performance than those using IVA for initialization. One
possible reason could be that block permutation had occurredin IVA. It is worth noting that the MVAE methods withrandom initialization obtained more than 3 dB higher SDR
improvements than when using IVA and ILRMA for initialization. Meanwhile, though random initialization slightly
outperformed ILRMA in s.u.fMVAE_c and s.i.fMVAE, there
were no noticeable differences. Therefore, we adopted random initialization in the following experiments.
Finally, we investigated how much the performance
depends on the weight parameter α in the prior-weighted
inference. We set α at {0, 1, 10, 50, 100, 200, 300, mean},
where ‘‘mean’’ indicates the data-dependent setting
K

α=

1 X 2
σφ (k; S, c).
K

(38)

k

Figure 7 shows the average SDR scores over 160 test signals.
We found that the effectiveness of the prior distribution p(z)in improving the source separation performance was modest
in the speaker-dependent case and that the SDRs started todecrease at α > 10, which indicates that a smaller value
228749

L. Li et al.: FastMVAE: A Fast Optimization Algorithm for the MVAE Method

TABLE 5. Average SDR, SIR, SAR, PESQ, and STOI scores achieved by eachmethod with the optimal parameter setting. The bold font indicates thebest scores.

FIGURE 7. Average SDR achieved with various α in a speaker-dependentcondition.

TABLE 6. Computational times of MVAE and fMVAE methods withrandom initialization.

TABLE 4. Average SDR, SIR, SAR, PESQ, and STOI scores achieved by
MVAE with CVAE and VAE for source modeling. The bold font indicates thebest scores.

leads to better performance for the speaker-dependent case.
Moreover, the curve of fMVAE_o was entirely above thecurve of fMVAE_c without regard for the choice of the initialization methods, which indicates that fMVAE_o is more
effective in speaker-dependent scenarios.
E. SOURCE SEPARATION PERFORMANCEIn addition to SDRs, we used signal-to-interference
ratios (SIRs) and signal-to-artifact ratios (SARs) [56] to evaluate the source separation performance. Perceptual evaluations of speech quality (PESQ)3 [57] and short-time objective
intelligibility (STOI)4 [58] were also conducted to ascertain
the speech quality and intelligibility. All the criteria were
calculated using a dry source as the reference signal.
We first confirmed the effectiveness of conditional modeling by comparing the performance obtained with the CVAE
source model and its unconditional counterpart under the
MVAE framework. Table 4 shows SDR, SIR, SAR, PESQ,
and STOI scores. As can be seen from the results, the CVAE
source model obtained a 1.7-dB higher SDR than a source
model based on a regular VAE.
Table 5 shows scores obtained by each method with the
optimal parameter setting. By comparing supervised methods
to the blind method (u.u.ILRMA), we confirmed that an
appropriately pretrained source model could lead to considerably improved source separation performance. The MVAE
methods achieved the best scores in both the uninformed andinformed categories, which significantly outperformed theother methods. The fMVAE method yielded an average SDR
score that was 2.8 dB lower than the original MVAE method,
but about 0.75 dB higher than the other baseline methods.
3 Code: https://github.com/vBaiCai/python-pesq
4 Code: https://github.com/mpariente/pystoi

228750F. COMPUTATIONAL TIME

The average computational times of the MVAE and fMVAE
methods with random initialization are summarized inTable 6. All the programs were run using an Intel (R) Corei7-7800X CPU@3.50 GHz and a TITAN V GPU with 12-GB
memory. Here, ‘‘runtime/iteration’’ means the computationaltime required to update the parameters once using the MVAE
or fMVAE algorithm. The ‘‘total’’ time indicates the time
taken by the entire process, including the time for constructing the system (e.g., loading the pretrained networks to aGPU), updating parameters, and performing the separation.
Through the comparison of the runtime at each iteration,
we found that the fMVAE algorithm was about 70 times
faster than the MVAE algorithm. Moreover, fMVAE wasfound to reduce the computational time by more than 90%
even when using a CPU. These results indicate a tradeoff
between the source separation performance and computational time: the MVAE method provides better separation performance with high computational cost, whereas fMVAE significantly reduces computational cost but with performancedegradation.
G. SPEAKER-INDEPENDENT SEPARATIONIn practical applications, the speakers in a given mixture
signal are not always included in the training dataset. In
this subsection, we show the performance of the MVAE andfMVAE methods in speaker-independent tasks and comparethem with u.u.ILRMA, which requires no prior information
about the speakers.
We created datasets using utterances from the Wall Street
Journal (WSJ0) corpus [59]. All the utterances in WSJ0 folder
si_tr_s (around 25 hours) were used as the training set,
which consists of 101 speakers in total. If there is a large
number of utterances of a sufficiently wide variety of speakers
VOLUME 8, 2020

L. Li et al.: FastMVAE: A Fast Optimization Algorithm for the MVAE Method

superior to u.u.ILRMA, where s.u.MVAE achieved an
improvement of more than 3.5 dB over u.u.ILRMA. As withthe speaker-dependent case, the fMVAE methods provided
less improvement than the MVAE method.
VI. CONCLUSION

FIGURE 8. Average SDR over 200 test signals achieved with various α.

TABLE 7. Average SDR, SIR, SAR, PESQ, and STOI scores obtained with
uninformed methods. The bold font shows the best scores.

in the training dataset, the trained model is expected to havean ability to express spectrograms of unseen speakers. Whena test mixture contains unseen speakers, (31) can be interpreted as how similar speaker j is to the speakers in the training set, whereas (32) indicates the speaker in the training set
most similar to speaker j. A test set was created by randomly
mixing two different speakers selected from the WSJ0 folders
si_dt_05 and si_et_05, where the number of speakers
was 18. We generated test data using simulated RIRs with
RT60 = 78 ms and RT60 = 351 ms, where 100 mixture
signals were generated under each reverberation condition.
The average SDRs of the datasets were about 0.60 dB and-0.78 dB, respectively. Other experimental conditions andnetwork architectures were the same as those described in
Subsection V-C.
As in the speaker-dependent case, we first investigatedthe dependence of the separation performance on the αsetting. Figure 8 shows the average SDR scores over theentire test dataset achieved with various α settings. Since
the scores obtained with α = 200 and α = 300 increased
continuously, we additionally evaluated the performanceobtained when α = {500, 700, 1000, 1500, 2000}. The optimal α settings were 500 for s.u.fMVAE_o and 2000 for
s.u.fMVAE_c, respectively. This was considerably different
from the speaker-dependent case, where a smaller α performed better. From these results, we can assume that the proposed prior-weighted update rule was more effective underopen-set conditions than under closed-set conditions.
Table 7 summarizes the average SDR, SIR, SAR, PESQ,
and STOI scores obtained with each method with random
initialization. The results demonstrate the ability of the
MVAE and fMVAE methods to handle speaker-independent
scenarios with an increasing variety and amount of training data. Both the MVAE and fMVAE methods were
VOLUME 8, 2020

This paper proposed a novel optimization algorithm for the
MVAE method, which is called FastMVAE (or fMVAE). The
proposed method exploits an auxiliary classifier VAE insteadof a regular CVAE to learn the generative distribution ofsource signals and employs the trained auxiliary classifier andencoder for inference. We newly introduced a prior-weighted
update rule for the latent variables of each CVAE source
model and different update rules for the class label of eachsource. We conducted experiments to investigate the optimal
window length, initialization, and weight parameter and performed speaker-dependent and speaker-independent source
separation experiments to confirm the effectiveness of the
proposed method. Experimental results revealed that fMVAE
can significantly reduce computational time by more than
90% compared with the original MVAE method; the MVAE
and fMVAE methods outperformed conventional methods
under speaker-dependent conditions; and the MVAE andfMVAE methods can handle a speaker-independent scenario
by using a large set of training data.
ACKNOWLEDGMENT

This article was presented in part at the ICASSP 2019 as aconference paper [27].
REFERENCES[1] T. Kim, T. Eltoft, and T.-W. Lee, ‘‘Independent vector analysis: An extension of ICA to multivariate components,’’ in Proc. Integr. Comput.-Aided
Eng., 2006, pp. 165–172.
[2] A. Hiroe, ‘‘Solution of permutation problem in frequency domain ICA
using multivariate probability density functions,’’ in Proc. Integr. Comput.Aided Eng., 2006, pp. 601–608.
[3] A. Ozerov and C. Fevotte, ‘‘Multichannel nonnegative matrix factorizationin convolutive mixtures for audio source separation,’’ IEEE Trans. Audio,
Speech, Lang. Process., vol. 18, no. 3, pp. 550–563, Mar. 2010.
[4] H. Sawada, H. Kameoka, S. Araki, and N. Ueda, ‘‘Multichannel extensions of non-negative matrix factorization with complex-valued data,’’IEEE Trans. Audio, Speech, Lang. Process., vol. 21, no. 5, pp. 971–982,
May 2013.
[5] H. Kameoka, T. Yoshioka, M. Hamamura, J. Le Roux, and K. Kashino,‘‘Statistical model of speech signals based on composite autoregressive
system with application to blind source separation,’’ in Proc. LVA/ICA,
2010, pp. 245–253.
[6] D. Kitamura, N. Ono, H. Sawada, H. Kameoka, and H. Saruwatari,‘‘Determined blind source separation unifying independent vector analysis
and nonnegative matrix factorization,’’ IEEE/ACM Trans. Audio, Speech,
Language Process., vol. 24, no. 9, pp. 1622–1637, Sep. 2016.
[7] D. Kitamura, N. Ono, H. Sawada, H. Kameoka, and H. Saruwatari, ‘‘Determined blind source separation with independent low-rank matrix analysis,’’in Audio Source Separation, S. Makino, Ed. Cham, Switzerland: Springer,
2018, pp. 125–155.
[8] J. R. Hershey, Z. Chen, J. Le Roux, and S. Watanabe, ‘‘Deep clustering:
Discriminative embeddings for segmentation and separation,’’ in Proc.
IEEE Int. Conf. Acoust., Speech Signal Process. (ICASSP), Mar. 2016,
pp. 31–35.
[9] Y. Isik, J. L. Roux, Z. Chen, S. Watanabe, and J. R. Hershey, ‘‘Singlechannel multi-speaker separation using deep clustering,’’ in Proc. Interspeech, Sep. 2016, pp. 545–549.

228751

L. Li et al.: FastMVAE: A Fast Optimization Algorithm for the MVAE Method[10] D. Yu, M. Kolbaek, Z.-H. Tan, and J. Jensen, ‘‘Permutation invariant training of deep models for speaker-independent multi-talker speech separation,’’ in Proc. IEEE Int. Conf. Acoust., Speech Signal Process. (ICASSP),
Mar. 2017, pp. 241–245.
[11] M. Kolbaek, D. Yu, Z.-H. Tan, and J. Jensen, ‘‘Multitalker speech separation with utterance-level permutation invariant training of deep recurrent neural networks,’’ IEEE/ACM Trans. Audio, Speech, Lang. Process.,
vol. 25, no. 10, pp. 1901–1913, Oct. 2017.
[12] D. Wang and J. Chen, ‘‘Supervised speech separation based on deeplearning: An overview,’’ IEEE/ACM Trans. Audio, Speech, Lang. Process.,
vol. 26, no. 10, pp. 1702–1726, Oct. 2018.
[13] Y. Liu and D. Wang, ‘‘Divide and conquer: A deep CASA approachto talker-independent monaural speaker separation,’’ IEEE/ACM Trans.
Audio, Speech, Lang. Process., vol. 27, no. 12, pp. 2092–2102, Dec. 2019.
[14] J. Le Roux, G. Wichern, S. Watanabe, S. Sarroff, and J. R. Hershey, ‘‘Phasebook and friends: Leveraging discrete representations forsource separation,’’ IEEE J. Sel. Topics Signal Process., vol. 13, no. 2,
pp. 370–382, May 2019.
[15] M. Delfarah and D. Wang, ‘‘Deep learning for talker-dependent reverberant
speaker separation: An empirical study,’’ IEEE/ACM Trans. Audio, Speech,
Lang. Process., vol. 27, no. 11, pp. 1839–1848, Nov. 2019.
[16] Z.-Q. Wang, J. Le Roux, and J. R. Hershey, ‘‘Multi-channel deepclustering: Discriminative spectral and spatial embeddings for speakerindependent speech separation,’’ in Proc. IEEE Int. Conf. Acoust., Speech
Signal Process. (ICASSP), Apr. 2018, pp. 1–5.
[17] T. Higuchi, K. Kinoshita, M. Delcroix, K. Žmolíková, and T. Nakatani,‘‘Deep clustering-based beamforming for separation with unknown number of sources,’’ in Proc. Interspeech, Aug. 2017, pp. 1183–1187.
[18] L. Drude and R. Haeb-Umbach, ‘‘Tight integration of spatial and spectralfeatures for BSS with deep clustering embeddings,’’ in Proc. Interspeech,
Aug. 2017, pp. 2650–2654.
[19] A. A. Nugraha, A. Liutkus, and E. Vincent, ‘‘Multichannel audio source
separation with deep neural networks,’’ IEEE/ACM Trans. Audio, Speech,
Lang. Process., vol. 24, no. 9, pp. 1652–1664, Sep. 2016.
[20] N. Makishima, S. Mogami, N. Takamune, D. Kitamura, H. Sumino,
S. Takamichi, H. Saruwatari, and N. Ono, ‘‘Independent deeply learned
matrix analysis for determined audio source separation,’’ IEEE/ACM
Trans. Audio, Speech, Lang. Process., vol. 27, no. 10, pp. 1601–1615,
Oct. 2019.
[21] H. Kameoka, L. Li, S. Inoue, and S. Makino, ‘‘Supervised determinedsource separation with multichannel variational autoencoder,’’ NeuralComput., vol. 31, no. 9, pp. 1891–1914, Sep. 2019.
[22] Y. Bando, M. Mimura, K. Itoyama, K. Yoshii, and T. Kawahara, ‘‘Statistical speech enhancement based on probabilistic integration of variational
autoencoder and non-negative matrix factorization,’’ in Proc. IEEE Int.
Conf. Acoust., Speech Signal Process. (ICASSP), Apr. 2018, pp. 716–720.
[23] S. Leglaive, L. Girin, and R. Horaud, ‘‘A variance modeling framework
based on variational autoencoders for speech enhancement,’’ in Proc. IEEE
28th Int. Workshop Mach. Learn. for Signal Process. (MLSP), Sep. 2018,
pp. 1–6.
[24] D. P. Kingma, S. Mohamed, D. J. Rezende, and M. Welling, ‘‘Semisupervised learning with deep generative models,’’ in Proc. NIPS, 2014,
pp. 3581–3589.
[25] N. Ono, ‘‘Stable and fast update rules for independent vector analysis basedon auxiliary function technique,’’ in Proc. IEEE Workshop Appl. Signal
Process. Audio Acoust. (WASPAA), Oct. 2011, pp. 189–192.
[26] H. Kameoka, T. Kaneko, K. Tanaka, and N. Hojo, ‘‘ACVAE-VC: nonparallel voice conversion with auxiliary classifier variational autoencoder,’’ IEEE/ACM Trans. Audio, Speech, Lang. Process., vol. 27, no. 9,
pp. 1432–1443, Sep. 2019.
[27] L. Li, H. Kameoka, and S. Makino, ‘‘Fast MVAE: Joint separation andclassification of mixed sources based on multichannel variational autoencoder with auxiliary classifier,’’ in Proc. ICASSP - IEEE Int. Conf. Acoust.,
Speech Signal Process. (ICASSP), May 2019, pp. 546–550.
[28] G. E. Hinton, ‘‘Training products of experts by minimizing contrastive
divergence,’’ Neural Comput., vol. 14, no. 8, pp. 1771–1800, Aug. 2002.
[29] C. Févotte and J. F. Cardoso, ‘‘Maximum likelihood approach for blindaudio source separation using time-frequency Gaussian source models,’’in Proc. IEEE Workshop Appl. Signal Process. Audio Acoust., Oct. 2005,
pp. 78–81.
[30] E. Vincent, S. Arberet, and R. Gribonval, ‘‘Underdetermined instantaneousaudio source separation via local Gaussian modeling,’’ in Proc. Integr.
Comput.-Aided Eng., 2009, pp. 775–782.
228752[31] D. D. Lee and H. S. Seung, ‘‘Algorithms for non-negative matrix factorization,’’ in Proc. NIPS, 2001, pp. 556–562.
[32] A. Ozerov and H. Kameoka, ‘‘Gaussian model based multichannel separation,’’ in Audio Source Separation and Speech Enhancement, E. Vincent,T. Virtanen, S. Gannot, Eds. Springer, pp. 289–315, 2018.
[33] D. R. Hunter and K. Lange, ‘‘A tutorial on MM algorithms,’’ Amer.
Statistician, vol. 58, no. 1, pp. 30–37, Feb. 2004.
[34] S. Seki, H. Kameoka, L. Li, T. Toda, and K. Takeda, ‘‘Underdeterminedsource separation based on generalized multichannel variational autoencoder,’’ IEEE Access, vol. 7, pp. 168104–168115, 2019.
[35] K. Sekiguchi, Y. Bando, A. A. Nugraha, K. Yoshii, and T. Kawahara,‘‘Semi-supervised multichannel speech enhancement with a deep speechprior,’’ IEEE/ACM Trans. Audio, Speech, Lang. Process., vol. 27, no. 12,
pp. 2197–2212, Dec. 2019.
[36] S. Leglaive, U. Simsekli, A. Liutkus, L. Girin, and R. Horaud, ‘‘Speechenhancement with variational autoencoders and alpha-stable distributions,’’ in Proc. IEEE Int. Conf. Acoust., Speech Signal Process. (ICASSP),May 2019, pp. 541–545.
[37] S. Leglaive, L. Girin, and R. Horaud, ‘‘Semi-supervised multichannel speech enhancement with variational autoencoders and non-negative
matrix factorization,’’ in Proc. IEEE Int. Conf. Acoust., Speech Signal
Process. (ICASSP), May 2019, pp. 101–105.
[38] M. Pariente, A. Deleforge, and E. Vincent, ‘‘A statistically principled andcomputationally efficient approach to speech enhancement using variational autoencoders,’’ in Proc. Interspeech, Sep. 2019, pp. 3158–3162.
[39] S. Inoue, H. Kameoka, L. Li, S. Seki, and S. Makino, ‘‘Joint separationand dereverberation of reverberant mixtures with multichannel variational
autoencoder,’’ in Proc. IEEE Int. Conf. Acoust., Speech Signal Process.
(ICASSP), May 2019, pp. 96–100.
[40] S. Pascual, A. Bonafonte, and J. Serra, ‘‘SEGAN: Speech enhancement generative adversarial network,’’ in Proc. Interspeech, Aug. 2017,
pp. 3642–3646.
[41] H. Kameoka, L. Li, S. Inoue, and S. Makino, ‘‘Semi-blind source separation with multichannel variational autoencoder,’’ 2018, arXiv:1808.00892.
[Online]. Available: http://arxiv.org/abs/1808.00892[42] K. Sekiguchi, Y. Bando, K. Yoshii, and T. Kawahara, ‘‘Bayesian multichannel speech enhancement with a deep speech prior,’’ in Proc. Asia–PacificSignal Inf. Process. Assoc. Annu. Summit Conf. (APSIPA ASC), Nov. 2018,
pp. 1233–1239.
[43] W. R. Gilks, S. Richardson, and D. Spiegelhalter, Markov Chain Monte
Carlo in Practice. Boca Raton, FL, USA: CRC Press, 1995.
[44] X. Chen, Y. Duan, R. Houthooft, J. Schulman, I. Sutskever, and P. Abbeel,‘‘InfoGAN: Interpretable representation learning by information maximizing generative adversarial nets,’’ in Proc. NIPS, 2016, pp. 2172–2180.
[45] D. Barber and F. V. Agakov, ‘‘The IM algorithm: A variational approach
to information maximization,’’ in Proc. NIPS, 2003, pp. 1–8.
[46] S. Mogami, H. Sumino, D. Kitamura, N. Takamune, S. Takamichi,
H. Saruwatari, and N. Ono, ‘‘Independent deeply learned matrix analysis for multichannel audio source separation,’’ in Proc. 26th Eur. Signal
Process. Conf. (EUSIPCO), Sep. 2018, pp. 1571–1575.
[47] Y. N. Dauphin, A. Fan, M. Auli, and D. Grangier, ‘‘Language modelingwith gated convolutional networks,’’ in Proc. ICML, 2017, pp. 933–941.
[48] L. Li and H. Kameoka, ‘‘Deep clustering with gated convolutional
networks,’’ in Proc. IEEE Int. Conf. Acoust., Speech Signal Process.
(ICASSP), Apr. 2018, pp. 16–20.
[49] T. Kaneko, H. Kameoka, K. Hiramatsu, and K. Kashino, ‘‘Sequence-toSequence voice conversion with similarity metric learned using generative
adversarial networks,’’ in Proc. Interspeech, Aug. 2017, pp. 1283–1287.
[50] D. P. Kingma and J. Ba, ‘‘Adam: A method for stochastic optimization,’’in Proc. ICLR, 2015.
[51] M. D. Zeiler, ‘‘ADADELTA: An adaptive learning rate method,’’ 2012,arXiv:1212.5701. [Online]. Available: https://arxiv.org/abs/1212.5701[52] J. Lorenzo-Trueba, J. Yamagishi, T. Toda, D. Saito, F. Villavicencio,
T. Kinnunen, and Z. Ling, ‘‘The voice conversion challenge 2018:
Promoting development of parallel and nonparallel methods,’’ 2018,arXiv:1804.04262. [Online]. Available: http://arxiv.org/abs/1804.04262[53] J. B. Allen and D. A. Berkley, ‘‘Image method for efficiently simulatingsmall-room acoustics,’’ J. Acoust. Soc. Amer., vol. 65, no. 4, pp. 943–950,
1979.
[54] M. R. Schroeder, ‘‘New method of measuring reverberation time,’’J. Acoust. Soc. Amer., vol. 37, no. 3, pp. 409–412, Mar. 1965.
[55] S. Nakamura, K. Hiyane, F. Asano, and T. Endo, ‘‘Sound scene data collection in real acoustical environments,’’ J. Acoust. Soc. Jpn. (E), vol. 20,
no. 3, pp. 225–231, 1999.
VOLUME 8, 2020

L. Li et al.: FastMVAE: A Fast Optimization Algorithm for the MVAE Method[56] E. Vincent, R. Gribonval, and C. Fevotte, ‘‘Performance measurement inblind audio source separation,’’ IEEE Trans. Audio, Speech Lang. Process.,
vol. 14, no. 4, pp. 1462–1469, Jul. 2006.
[57] A. W. Rix, J. G. Beerends, M. P. Hollier, and A. P. Hekstra, ‘‘Perceptual
evaluation of speech quality (PESQ)–A new method for speech quality
assessment of telephone networks and codecs,’’ in Proc. IEEE Int. Conf.
Acoust., Speech, Signal Processing. Proc. (Cat. No.01CH1), vol. 2, 2001,
pp. 749–752.
[58] C. H. Taal, R. C. Hendriks, R. Heusdens, and J. Jensen, ‘‘A shorttime objective intelligibility measure for time-frequency weighted noisy
speech,’’ in Proc. IEEE Int. Conf. Acoust., Speech Signal Process., 2010,
pp. 4214–4217.
[59] J. S. Garofolo, CSR-I (WSJ0) Complete LDC93S6A. Philadelphia,
PA, USA: Linguistic Data Consortium, 1993. [Online]. Available:
https://catalog.ldc.upenn.edu/LDC93S6A

LI LI (Student Member, IEEE) received the B.E.
degree from the Shanghai University of Financeand Economics, China, in 2014, and the M.S.
degree from the University of Tsukuba, Japan,
in 2018, where she is currently pursuing the Ph.D.
degree with the Graduate School. Since 2018, she
has been a Research Fellow of the Japan Societyof Promotion of Science. Her research interests
include audio and speech signal processing, source
separation, and machine learning. She received the13th Student Presentation Award from the Acoustical Society of Japan andthe second IEEE Signal Processing Society Tokyo Joint Chapter Student
Award.

HIROKAZU KAMEOKA (Senior Member, IEEE)
received the B.E., M.S., and Ph.D. degrees fromThe University of Tokyo, Japan, in 2002, 2004, and2007, respectively. From 2011 to 2016, he was an
Adjunct Associate Professor with The Universityof Tokyo. He is currently a Senior Distinguished
Researcher with NTT Communication Science
Laboratories, Nippon Telegraph and Telephone
Corporation. He is also an Adjunct Associate Professor with the National Institute of Informatics.
He is the author or coauthor of about 150 articles in journal articles andpeer-reviewed conference proceedings. His research interests include audio,
speech, and music signal processing, and machine learning. He has beena member of the IEEE Audio and Acoustic Signal Processing TechnicalCommittee, since 2017, and a member of the IEEE Machine Learning
for Signal Processing Technical Committee, since 2019. He has received
17 awards, including the IEEE Signal Processing Society 2008 SPS Young
Author Best Paper Award. Since 2015, he has been an Associate Editor of
the IEEE/ACM TRANSACTIONS ON AUDIO, SPEECH, AND LANGUAGE PROCESSING.

VOLUME 8, 2020

SHOTA INOUE received the B.S. degree in
information technology from the University ofTsukuba, Japan, where he is currently pursuing
the master’s degree. His research interests include
audio and speech signal processing, source separation, and machine learning.

SHOJI MAKINO (Fellow, IEEE) received the
B.E., M.E., and Ph.D. degrees from Tohoku
University, Japan, in 1979, 1981, and 1993,
respectively.
He joined NTT, in 1981. He is currently a Professor with the University of Tsukuba. He is theauthor or coauthor of more than 200 papers in journals and conference proceedings. He is responsible
for more than 150 patents. His research interests include adaptive filtering technologies, realization of acoustic echo cancellation, blind source separation of convolutive
mixtures of speech, and acoustic signal processing for speech and audio
applications.
Dr. Makino is an IEICE Fellow, a Board Member of the ASJ, and a
Member of EURASIP. He received the ICA Unsupervised Learning Pioneer
Award, in 2006, the IEEE MLSP Competition Award, in 2007, the IEEESPS Best Paper Award, in 2014, the Achievement Award for Science andTechnology by the Minister of Education, Culture, Sports, Science andTechnology, in 2015, the Hoko Award of the Hattori Hokokai Foundation,in 2018, the Outstanding Contribution Award of the IEICE, in 2018, the Technical Achievement Award of the IEICE, in 2017 and 1997, the Outstanding
Technological Development Award of the ASJ, in 1995, and eight Best Paper
Awards. He was the Chair of SPS Audio and Acoustic Signal Processing
Technical Committee, from 2013 to 2014 and the Blind Signal Processing Technical Committee of the IEEE Circuits and Systems Society, from2009 to 2010. He was the General Chair of IWAENC 2018, WASPAA2007,
IWAENC2003, and the Organizing Chair of ICA2003. He is the designatedPlenary Chair of ICASSP2012. He has served on IEEE SPS Board ofGovernors, from 2018 to 2020, Technical Directions Board, from 2013 to2014, Awards Board, from 2006 to 2008, Conference Board, from 2002 to
2004, and Fellow Evaluation Committee, from 2018 to 2020. He was amember of the IEEE Jack S. Kilby Signal Processing Medal Committee,
from 2015 to 2018, and the James L. Flanagan Speech & Audio Processing
Award Committee, from 2008 to 2011. He was an Associate Editor of theIEEE TRANSACTIONS ON SPEECH AND AUDIO PROCESSING, from 2002 to 2005, andthe EURASIP Journal on Advances in Signal Processing, from 2005 to 2012.
He was a Guest Editor of the Special Issue of the IEEE Signal Processing
Magazine, from 2013 to 2014. He was a Keynote Speaker at ICA2007,a Tutorial speaker at EMBC2013, Interspeech2011, and ICASSP2007. From2009 to 2010, he was an IEEE SPS Distinguished Lecturer.

228753