FAST MVAE: JOINT SEPARATION AND CLASSIFICATION OF MIXED SOURCES BASEDON MULTICHANNEL VARIATIONAL AUTOENCODER WITH AUXILIARY CLASSIFIER
Li Li1 , Hirokazu Kameoka2 , Shoji Makino1
1

2

University of Tsukuba, JapanNTT Communication Science Laboratories, NTT Corporation, Japan
ABSTRACT

can be interpreted as a generalized IVA method that incorporates a source model with stronger representation power,
which has been shown to significantly improve source separation performance [6].
Motived by this fact and the high capability of deep neural networks (DNNs) as regards spectrogram modeling, someattempts have recently been made to use DNNs as source
models instead of the NMF model [7]–[12]. The multichannel variational autoencoder (MVAE) [11] is one such method,and it has achieved great success in multi-speaker separation
tasks. MVAE trains a conditional VAE (CVAE) [13, 14] using power spectrograms of clean speech samples and the corresponding speaker ID as auxiliary label inputs so that thetrained decoder distribution can be used as a universal generative model of source signals, where the latent space variables and the class labels are the unknown parameters. At
the separation phase, MVAE iteratively updates the separation matrix using the iterative projection (IP) method [15]and the unknown parameters of the source generative modelwith backpropagation. The separated signals are obtainedby applying the estimated separation matrix to the observed
mixture signals. This optimization algorithm is notable inthat convergence to a stationary point is guaranteed and it allows the source-class labels to be estimated simultaneouslywith source separation. However, there are two major limitations. Firstly, the backpropagation needed for each iteration causes the optimization algorithm to be highly timeconsuming, which can be troublesome in practical applications. Secondly, the encoder and decoder in a regular CVAE
are free to ignore the class labels by finding networks that canreconstruct any data without using additional information. In
such a situation, the additional class labels will have a limited effect on spectrogram generation, which therefore leads
to an unsatisfactory source classification result as we show in
Section 4.
To address these limitations, this paper proposes “fast
MVAE (fMVAE)”, which employs an auxiliary classifier
VAE (ACVAE) [16] to learn the generative distribution ofsource spectrograms and adopts a trained auxiliary classifier
for optimization at the separation phase.

This paper proposes an alternative algorithm for the multichannel variational autoencoder (MVAE), a recently proposed
multichannel source separation approach. While MVAE isnotable for its impressive source separation performance, itsconvergence-guaranteed optimization algorithm and the factthat it allows us to estimate source-class labels simultaneously with source separation, there are still two major drawbacks, namely, the high computational complexity and theunsatisfactory source classification accuracy. To overcome
these drawbacks, the proposed method employs an auxiliary
classifier VAE, which is an information-theoretic extension of
the conditional VAE, for learning the generative model of thesource spectrograms. Furthermore, with the trained auxiliary
classifier, we introduce a novel algorithm for the optimization that can both reduce the computational time and improve
the source classification performance. We call the proposed
method “fast MVAE (fMVAE)”. Experimental evaluations revealed that fMVAE achieved source separation performancecomparable to that of MVAE and a source classification accuracy rate of about 80% while reducing computational time byabout 93%.
Index Terms— Multichannel source separation, multichannel variational autoencoder, auxiliary classifier, sourceclassification
1. INTRODUCTION
Blind source separation (BSS) is a technique for separatingout individual source signals from microphone array inputs
when both the sources and the mixing methodology are unknown. The frequency-domain BSS approach allows us toperform instantaneous mixture separation and provides theflexibility of utilizing various models for the time-frequency
representations of source signals. For example, independentvector analysis (IVA) [1, 2] solves frequency-wise source separation and permutation alignment simultaneously by assuming that the magnitudes of the frequency components originating from the same source tend to vary coherently over time.
Multichannel extensions of non-negative matrix factorization(NMF), e.g., multichannel NMF (MNMF) [3, 4] and independent low-rank matrix analysis (ILRMA) [5, 6], provide analternative solution to jointly solving these two problems byadopting the NMF concept for the source spectrogram modeling. Specifically, the power spectrograms of the underlying
source signals are approximated as the linear sum of a limited
number of basis spectra scaled by time-varying amplitudes. It
is noteworthy that IVA is equivalent to ILRMA in a particularcase where only a single basis spectrum consisting of ones isused for each source signal. From this standpoint, ILRMA

2. MVAE FOR DETERMINED MULTICHANNEL
SOURCE SEPARATION
2.1. Problem formulation
Let us consider a determined situation where I source signals
are captured by I microphones. Let xi (f, n) and sj (f, n)denote the short-time Fourier transform (STFT) coefficientsof the signal observed at the i-th microphone and the jth source signal, where f and n are the frequency andtime indices respectively. We denote the vectors contain-

This work was supported by JSPS KAKENHI Grant Number 17H01763
and 18J20059, and SECOM Science and Technology Foundation.

978-1-5386-4658-8/18/$31.00 ©2019 IEEE

546

ICASSP 2019

ing x1 (f, n), . . . , xI (f, n) and s1 (f, n), . . . , sI (f, n) byx(f, n) = [x1 (f, n), . . . , xI (f, n)]T ∈ CI ,T

I

s(f, n) = [s1 (f, n), . . . , sI (f, n)] ∈ C ,trained by maximizing the following variational lower bound(1)

J (φ, θ) = E(S,c)∼pD (S,c) [Ez∼qφ (z|S,c) [log pθ (S|z, c)](2)− KL[qφ (z|S, c)||p(z)]],where E(S,c)∼pD (S,c) [·] denotes the sample mean over the
training examples {S m , cm }M
m=1 and KL[·||·] denotes Kullback–Leibler divergence. Here, the encoder distribution
qφ (z|S, c) and the prior distribution of the latent space variable p(z) are expressed as Gaussian distributions

where (·)T denotes the transpose. In a determined situation,
the relationship between observed signals and source signals
can be described ass(f, n) = W H (f )x(f, n),
W (f ) = [w1 (f ), . . . , wI (f )] ∈ C

(3)
I×I

H

,

(4)

qφ (z|S, c) = N (z|µφ (S, c), diag(σ 2φ (S, c))),H

where W (f ) is called the separation matrix. (·) denotes
the Hermitian transpose.
Let us assume that source signals follow the local Gaussian model (LGM), i.e., sj (f, n) independently follows azero-mean complex proper Gaussian distribution with variance vj (f, n) = E[|sj (f, n)|2 ]sj (f, n) ∼ NC (sj (f, n)|0, vj (f, n)).

p(z) = N (z|0, I),(5)

Hence, the log-likelihood of the separation matrices W ={W (f )}f and source model parameters V = {vj (f, n)}j,f,ngiven the observed mixture signals X = {x(f, n)}f,n isgiven byXclog p(X |W, V) = 2N
log | det W H (f )|flog vj (f, n) +

f,n,j

2
|wH
j (f )x(f, n)|,vj (f, n)(13)where σθ2 (f, n; z, c) denotes the (f, n)-th element of the decoder output. It is worth mentioning that the decoder distribution (12) is given in the same form as the LGM (5) sothat the trained decoder distribution can be used as a universal
generative model with the ability to generate complex spectrograms belonging to all the source classes involved in the
training examples. If we use pθ (S j |z j , cj , gj ) to express thegenerative model of the complex spectrogram of the sourcej, a stationary point of the log-likelihood (8) that we want tomaximize can be searched by iteratively updating (A) the separation matrices W using the iterative projection (IP) method[15], (B) the CVAE source model parameters Ψ = {z j , cj }jwith backpropagation, and (C) the global scale parameter G ={gj }j with the following update rulex(f, n) ∼ NC (x(f, n)|0, (W H (f ))−1 V (f, n)W (f )−1 ).
(7)

−

(11)

f,n

v(f, n) = g · σθ2 (f, n; z, c),(6)where V (f, n) = diag[v1 (f, n), . . . , vI (f, n)]. From (3) and(5), we can show that x(f, n) follows

X

(10)

where µφ (S, c), σ 2φ (S, c) are the encoder network outputs.
The decoder distribution pθ (S|z, c) is defined as a zero-mean
complex proper Gaussian distribution and a scale parameterg is incorporated to eliminate the energy difference between
the normalized training data and test data. Hence, the decoder
distribution is expressed as
Ypθ (S|z, c, g) =NC (s(f, n)|0, v(f, n)),(12)

When sj (f, n) and sj 0 (f, n)(j 6= j 0 ) are independent, s(f, n)followss(f, n) ∼ NC (s(f, n)|0, V (f, n)),

(9)

(8)

gj ←

1 X |yj (f, n)|2,FN
σθ2 (f, n; z j , cj )

(14)

f,n

where =c denotes equality up to constant terms. (8) will be
split into frequency-wise source separation problems if there
is no additional constraint imposed on vj (f, n). This indicates that there is a permutation ambiguity in the separated
components for each frequency.

where yj (f, n) = wH
j (f )x(f, n). Note that since the class labels are the model parameters estimated during the optimization, MVAE is also able to perform source classification.
3. PROPOSED METHOD: FAST MVAE

2.2. Multichannel VAE

While MVAE is noteworthy in that it works reasonably wellfor source separation and has the capability to perform sourceclassification simultaneously, there is still huge room for improvement in the source classification performance. With aregular CVAE, which imposes no restrictions on the way inwhich the encoder and decoder may use the class labels, theencoder and decoder are free to ignore c by finding a distribution that satisfies qφ (z|S, c) = qφ (z|S) and pθ (S|z, c) =pθ (S|z). As a result, c will have little effect on the generation of source spectrograms and that will limit source classification performance. To avoid such situations, this paper
proposes using an auxiliary classifier VAE [16] for learning
the generative distribution pθ (S|z, c).

To eliminate the permutation ambiguity during the estimationof W, MVAE trains a conditional VAE (CVAE) to model thecomplex spectrograms S = {s(f, n)}f,n of source signals
so that the spectral structures can be captured. CVAE consists of an encoder network qφ (z|S, c) and a decoder networkpθ (S|z, c), where the network parameters φ and θ are trained
jointly using a set of labeled training samples {S m , cm }M
m=1 .
Here, c = {1, 2, . . . , C} denotes the corresponding class label indicating to which class the spectrogram S belongs. For
example, if we consider speaker identities as the class category, c will be associated with a different speaker. CVAE is547

3.1. Auxiliary classifier VAE
Auxiliary classifier VAE (ACVAE) [16] is a variant of CVAE
that incorporates information-theoretic regularization [17] toassist the decoder outputs to be correlated as far as possible
with the class labels c by maximizing the mutual information
between c and S ∼ pθ (S|z, c) conditioned on z. The mutual
information is expressed asI(c, S|z)= Ec∼p(c),S∼pθ (S|z,c),c0 ∼p(c|S) [log p(c0 |S)] + H(c), (15)Fig. 1. Configuration of the room where ◦ and × represent
the position of microphones and sources respectively.

where H(c) represents the entropy of c that can be considered as a constant term. However, it is difficult to optimize
I(c, S|z) directly since it requires access to the posteriorp(c|S). Fortunately, we can obtain a variational lower boundof the first term of I(c, S|z) by using a variational distribution r(c|S) to approximate p(c|S):

distribution and the encoder distribution p(z j , cj |S j ) ≈pθ (z|S j , cj )rψ (cj |S j ), we can replace the step (B) with theforward calculation of the encoder and the auxiliary classifier.
The proposed fast algorithm is summaried as follows:Ec∼p(c),S∼pθ (S|z,c),c0 ∼p(S|c) [log p(c0 |S)]r(c0 |S)p(c0 |S)]r(c0 |S)≥Ec∼p(c),S∼pθ (S|z,c),c0 ∼p(S|c) [log r(c0 |S)]

1. Train φ, θ and ψ using (19).
2. Initialize W.
3. Iterate the following steps for each j:=Ec∼p(c),S∼pθ (S|z,c),c0 ∼p(S|c) [log=Ec∼p(c),S∼pθ (S|z,c) [log r(c|S)],(16)(a) Calculate the temporarily signals
S j = {w(f )Hj x(f, n)}f .

the equality of which holds if r(c|S) = p(c|S). Therefore, we can indirectly maximize I(c, S|z) by increasing thelower bound with respect to pθ (S|z, c) and r(c|S). One way
to realize this involves expressing the variational distribution as a neural network rψ (c|S) and training it along withqφ (z|S, c) and pθ (S|z, c). rψ (c|S) is called an auxiliary
classifier. Therefore, the regularization term that we would
like to maximize with respect to φ, θ, ψ becomes(b) Update cj ← argmaxcj ∈{1,2,...,C} rψ (cj |S j ).
(c) Update z j ← µφ (S j , cj ).
(d) Update gj using (14).
(e) Update wj (0), . . . , wj (F ) using IP.

L(φ, θ, ψ)(17)= E(S,c)∼pD (S,c),qφ (z|S,c) [Ec∼p(c),S∼pθ (S|z,c) [log rψ (c|S)]],
PC
where k=1 rψ (c = k|S) = 1. In the regularization term(17), the auxiliary classifier is trained only using reconstructed spectrograms, which is undesirable since it may limit
the capability of the trained classifier of classifying the original spectrograms. To remedy this, ACVAE also includes the
cross-entropy
I(ψ) = E(S,c)∼pD (S,c) [log rψ (c|S)](18)

To evaluate the effect of incorporating an auxiliary classifier into both the source model training and the optimization process, we conducted experiments designed to compare
the multi-speaker separation performance, source classification accuracies and computational times of fMVAE and theconventional methods, i.e., ILRMA [5, 6] and MVAE [11].
4.1. Experimental conditions
We excerpted speech utterances from two male speakers(‘SM1’, ‘SM2’) and two female speakers (‘SF1’, ‘SF2’) fromthe Voice Conversion Challenge (VCC) 2018 dataset [18].
The audio files for each speaker were about 7 minutes longand manually segmented into 116 short sentences, where 81
and 35 sentences (about 5 and 2 minutes long, respectively)
were used as training and test sets, respectively. The mixture
signals were created by using simulated two-channel recordings of two sources where the room impulse responses weresynthesized using the image method. We tested two different
reverberant conditions where the reverberation times (RT60 )were set at 78 and 351 ms, respectively. Fig. 1 shows theconfiguration of the room. We generated test data involving
4 speaker pairs and 10 sentences for each pair, namely there
were a total of 40 test signals, each of which was about 4 to
7 seconds long. All the speech signals were resampled at 16
kHz. The STFT was computed using a Hamming windowthat was 256 ms long and the window shift was128 ms.
ILRMA was run for 100 iterations and both the proposed
method and MVAE were run for 40 iterations. To initialize W

in the training criterion. The entire training criterion is thus
given byJ (φ, θ) + λL L(φ, θ, ψ) + λI I(ψ),

4. EXPERIMENTS(19)

where λL ≥ 0 and λI ≥ 0 are weight parameters.
3.2. Fast algorithm
Note that the auxiliary classifier rψ (c|S) both assists theencoder and decoder to learn a more disentangled representation, and provides an alternative to the backpropagation process in the original MVAE optimization, which significantly
reduces the computational time. Specifically, since the maximum of the distribution p(z j , cj |S j ) = p(z j |S j , cj )p(cj |S j )searched with backpropagation in the step (B) can be approximately obtained with the trained auxiliary classifier

548

w 32
c 4

w 64
c 4

OutputProduct

Softmax

w 32 w 16
w 64
c 512 c 128 c 4
k 1x4 k 1x4 k 1x4

Conv

BN
GLU

w 128
w 128
c 2048 c 1024
k 1x5

OutputDeconv

Deconv
BN
GLU

Input
One-hot
encoding

Class label

w 128 w 64
c 4
c 4

w 128 w 128
c 1024 c 2048
k 1x4 k 1x5

Conv

w 64
w 32
c 512
c 256
k 1x4

Output

Conv
BN
GLU

Input
One-hot
encoding

Class label

w 128
c 4

Classifier

Decoder
w 32
c 256
k 1x4

Conv

w 64
c 512
k 1x4

w 128 w 128
c 2048 c 1024
k 1x5

Input

Encoderw 128
c 4

Fig. 2. Network architectures of the encoder and decoder used for MVAE and fMVAE and the classifier used for fMVAE.
The inputs and outputs are 1-dimensional data, where the frequency dimension of spectrograms is regarded as the channel
dimension. “w”, “c” and “k” denote the width, channel number and kernel size, respectively. “Conv”, “Deconv”, “BN” and“GLU” denote 1-dimensional convolution and deconvolution, batch normalization, gated linear unit, respectively.

Table 1. Average SDR, SIR and SAR scores of ILRMA,MVAE and fMVAE. The bold font shows the highest scores.
RT60 = 78 ms
methodSDR [dB] SIR [dB] SAR [dB]
ILRMA
14.899721.327718.0584
MVAE
21.591227.2663
25.1616
fMVAE
22.5976
29.8476
24.8967
RT60 = 351 ms
methodSDR [dB] SIR [dB] SAR [dB]
ILRMA
4.684011.6284
7.2364MVAE
8.315718.08349.2206fMVAE
6.781415.7728
7.7883

Table 3. Accuracy rates of source classification obtained withMVAE and fMVAE.
all iterations final estimation
MVAE
27.91%
37.50%
fMVAE
78.63%
80.00%(SAR) [21] over the 40 test signals to evaluate the source separation performance and measured the computational times.
Table 1 and Table 2 show the source separation results obtained under different RT60 conditions and the computationaltimes of ILRMA, MVAE and fMVAE, respectively. fMVAE
was about 15 times faster than the original MVAE and evenlightly faster than ILRMA in GPU machines. Furthermore,
it is noteworthy that fMVAE achieved source separation performance comparable to that of the original MVAE. As theresults show, MVAE and fMVAE significantly outperformed
ILRMA in terms of all the source separation performance criteria, which confirmed the effect of the incorporation of the
CVAE source model. fMVAE obtained a higher SDR and SIR
than MVAE in a slightly reverberant environment, but the performance deteriorated slightly when RT60 became long. It isinteresting to further compare fMVAE with MVAE in highly
reverberant environments, which is one direction for our future work.
To evaluate the performance of source classification, we
computed the classification accuracy rates over the results estimated in each iteration and in the final estimation alone. Table 3 provides results showing that fMVAE significantly improved source classification accuracy achieving an 80% accuracy rate. Our future work will also include further improving
the source classification accuracy.

Table 2. Computational times of MVAE, fMVAE and ILRMA. MVAE and fMVAE were initialized with run ILRMA
algorithm for 30 iterations in CPU and run 40 iterations of the
optimization algorithms in CPU or GPU. ILRMA runs 100 iterations in CPU.
rumtime/iteration[sec] total [sec]
MVAE (GPU)6.071632
260.5953
fMVAE (CPU)0.389762
21.54129
fMVAE (GPU)0.097272
17.56694
ILRMA (CPU)
0.113571
18.38221

for MVAE and fMVAE, we used ILRMA, which we ran for 30
iterations. Adam [19] was used for training CVAE and ACVAE, and estimating the latent variables in MVAE. The network architectures used for CVAE and ACVAE are shown in
Fig. 2. Note that we used the same network architectures for
CVAE and ACVAE as we used for the encoder and decoder.
All the networks were designed to be fully convolutional withgated linear units [20] so that the inputs were allowed to havearbitrary lengths. The programs were run using an Intel (R)Core i7-6800K CPU@3.40 GHz and a GeForce GTX 1080Ti
GPU.

5. CONCLUSIONSThis paper proposed a fMVAE method that (i) uses an auxiliary classifier VAE instead a regular CVAE to learn the generative distribution of source signals; (ii) employs a trained
auxiliary classifier and encoder for optimization. fMVAE allows us to significantly reduce the computational time andimprove the source classification performance. The results
revealed that fMVAE was about 15 times faster than the original MVAE and achieved a source classification accuracy rateof about 80% with notable source separation performance.

4.2. Results
We calculated the average signal-to-distortion ratios (SDR),
signal-to-interference ratios (SIR) and signal-to-artifact ratios549

6. REFERENCES[15] N. Ono, “Stable and fast update rules for independentvector analysis based on auxiliary function technique,”in Proc. WASPAA, pp. 189–192, 2011.

[1] T. Kim, T. Eltoft and T.-W. Lee, “Independent vector
analysis: An extension of ICA to multivariate components,” in Proc. ICA, pp. 165–172, 2006.

[16] H. Kameoka, T. Kaneko, K. Tanaka and N. Hojo,“ACVAE-VC: Non-parallel many-to-many voice conversion with auxiliary classifier variational autoencoder,” eprint arXiv: 1808.05092, Aug. 2018.

[2] A. Hiroe, “Solution of permutation problem in frequency domain ICA using multivariate probability density functions,” in Proc. ICA, pp. 601-608, 2006.

[17] X. Chen, Y. Duan, R. Houthooft, J. Schulman, I.
Sutskever and P. Abbeel “Infogan: Interpretable representation learning by information maximizing generative adversarial nets,” in Adv. Neural Information Processing Systems (NIPS), pp. 2172–2180, 2016.

[3] A. Ozerov and C. Févotte, “Multichannel nonnegative
matrix factorization in convolutive mixtures for audio
source separation,” IEEE Trans. ASLP, vol. 18, no. 3,
pp. 550–563, 2010.

[18] J. Lorenzo-Trueba, J. Yamagishi, T. Toda, D. Saito, F,
Villavicencio, T. Kinnunen and Z. Ling, “The voice
conversion challenge 2018: Promoting developmentof parallel and nonparallel methods,” eprint arXiv:
1804.04262, Apr. 2018.

[4] H. Sawada, H. Kameoka, S. Araki and N. Ueda, “Multichannel extensions of non-negative matrix factorization
with complex-valued data,” IEEE Trans. ASLP, vol. 21,
no. 5, pp. 971–982, 2013.
[5] H. Kameoka, T. Yoshioka, M. Hamamura, J. Le. Rouxand K. Kashino, “Statistical model of speech signals
based on composite autoregressive system with application to blind source separation,” in Proc. LVA/ICA,
pp. 245–253, 2010.

[19] D. Kingma, J. Ba, “Adam: A method for stochastic optimization,” in Proc. ICLR, 2015.
[20] Y. N. Dauphin, A. Fan, M. Auli and D. Grangier, “Language modeling with gated convolutional networks,” inProc. ICML, pp. 933–941, 2017.

[6] D. Kitamura, N. Ono, H. Sawada, H. Kameoka and H.
Saruwatari, “Determined blind source separation unifying independent vector analysis and nonnegative matrix
factorization,” IEEE/ACM Trans. ASLP, vol. 24, no. 9,
pp. 1622–1637, 2016.

[21] E. Vincent, R. Gribonval and C. Févotte, “Performancemeasurement in blind audio source separation,” IEEE
Trans. ASLP, vol. 14, no. 4, pp. 1462–1469, 2006.

[7] A. A. Nugraha, A. Liutkus and E. Vincent, “Multichannel Audio Source Separation With Deep Neural
Networks,” IEEE/ACM Trans. ASLP, vol. 24, no. 9,
pp. 1652–1664, 2016.
[8] Y. Bando, M. Mimura, K. Itoyama, K. Yoshii and T.
Kawahara, “Statistical speech enhancement based on
probabilistic integration of variational autoencoder andnon-negative matrix factorization,” in Proc. ICASSP,
pp. 716–720, 2018.
[9] S. Mogami, H. Sumino, D. Kitamura, N. Takamune,
S. Takamichi, H. Saruwatari and N. Ono, “Independent
deeply learned matrix analysis for multichannel audio
source separation,” in Proc. EUSIPCO, pp. 1571–1575,
2018.
[10] S. Leglaive, L. Girin and R. Horaud, “A variance modeling framework based on variational autoencoders forspeech enhancement,” in Proc. MLSP, 2018.
[11] H. Kameoka, L. Li, S. Inoue and S. Makino, “Semiblind source separation with multichannel variational
autoencoder,” eprint arXiv: 1808.00892, Aug. 2018.
[12] S. Seki, H. Kameoka, L. Li, T. Toda and K.
Takeda, “Generalized multichannel variational autoencoder for underdetermined source separation,” eprint
arXiv: 1810.00223, Oct. 2018.
[13] D. P. Kingma, S. Mohamed, D. J. Rezende and M.
Welling, “Semi-supervised learning with deep generative models,” in Adv. Neural Information Processing
Systems (NIPS), pp. 3581–3589, 2014.
[14] K. Sohn, H. Lee and X. Yan, “Learning structured
output representation using deep conditional generativemodels,” in Adv. Neural Information Processing Systems(NIPS), pp. 3483–3491, 2015.

550