FAST MVAE: JOINT SEPARATION AND CLASSIFICATION OF MIXED SOURCES BASEDON MULTICHANNEL VARIATIONAL AUTOENCODER WITH AUXILIARY CLASSIFIER
Li Li1 , Hirokazu Kameoka2 , Shoji Makino1
1

arXiv:1812.06391v2 [cs.LG] 13 Feb 20192

University of Tsukuba, JapanNTT Communication Science Laboratories, NTT Corporation, Japan

ABSTRACTThis paper proposes an alternative algorithm for multichannel variational autoencoder (MVAE), a recently proposed
multichannel source separation approach. While MVAE isnotable in its impressive source separation performance, theconvergence-guaranteed optimization algorithm and that itallows us to estimate source-class labels simultaneously withsource separation, there are still two major drawbacks, i.e.,the high computational complexity and unsatisfactory sourceclassification accuracy. To overcome these drawbacks, theproposed method employs an auxiliary classifier VAE, aninformation-theoretic extension of the conditional VAE, forlearning the generative model of the source spectrograms.
Furthermore, with the trained auxiliary classifier, we introduce a novel algorithm for the optimization that is able tonot only reduce the computational time but also improve
the source classification performance. We call the proposed
method fast MVAE (fMVAE). Experimental evaluations revealed that fMVAE achieved comparative source separation
performance to MVAE and about 80% source classification
accuracy rate while it reduced about 93% computational time.
Index Terms— Multichannel source separation, multichannel variational autoencoder, auxiliary classifier, sourceclassification
1. INTRODUCTION
Blind source separation (BSS) is a technique for separatingout individual source signals from microphone array inputs
when both the sources and the mixing methodology are unknown. The frequency-domain BSS approach allows us toperform instantaneous mixture separation and provides theflexibility of utilizing various models for the time-frequency
representations of source signals. For example, independentvector analysis (IVA) [1, 2] solves frequency-wise source separation and permutation alignment simultaneously by assuming that the magnitudes of the frequency components originating from the same source tend to vary coherently over time.
Multichannel extensions of non-negative matrix factorization(NMF), e.g., multichannel NMF (MNMF) [3, 4] and independent low-rank matrix analysis (ILRMA) [5, 6], provide analternative solution to jointly solving these two problems byadopting NMF concept for the source spectrogram modeling.
Specifically, the power spectrograms of the underlying source
signals are approximated as the linear sum of a limited number of basis spectra scaled by time-varying amplitudes. It isnoteworthy that IVA is equivalent to ILRMA in a particularThis work was supported by JSPS KAKENHI Grant Number 17H01763
and 18J20059, and SECOM Science and Technology Foundation.

case where only a single basis spectrum consisting of ones isused for each source signal. In this point of view, ILRMA
can be interpreted as a generalized method of IVA that incorporates a source model with stronger representation power,
which has shown to significantly improve the source separation performance [6].
Motived by this fact and the high capability of deep neuralnetworks (DNNs) to spectrogram modeling, some attempts
have recently been made to use DNNs for source models
instead of the NMF model [7]–[12]. Multichannel variational autoencoder (MVAE) [11] is one of these methods
that achieved great success in multi-speaker separation tasks.
MVAE trains a conditional VAE (CVAE) [13, 14] using power
spectrograms of clean speech samples and the corresponding speaker ID as auxiliary label inputs so that the trained
decoder distribution can be used as a universal generativemodel of source signals, where the latent space variables andthe class labels are the unknown parameters. At separation
phase, MVAE iteratively updates the separation matrix using
iterative projection (IP) method [15] and the unknown parameters of source generative model using backpropagation.
The separated signals are obtained by applying the estimatedseparation matrix to the observed mixture signals. This optimization algorithm is notable in that the convergence to astationary point is guaranteed and it allows estimating thesource-class labels simultaneously with source separation.
However, there are two major limitations. Firstly, the backpropagation needed for each iteration causes the optimization
algorithm highly time-consuming, which can be troublesome
in practical applications. Secondly, the encoder and decoder
in a regular CVAE are free to ignore the class labels by finding networks that can reconstruct any data without using theadditional information. In such a situation, the additionalclass labels will have limited effect on the spectrogram generation, which therefore leads to an unsatisfactory sourceclassification result as we will show it in Section 4.
To address these limitations, this paper proposes “fast
MVAE (fMVAE)” that employs an auxiliary classifier VAE(ACVAE) [16] for learning the generative distribution ofsource spectrograms and adopts the trained auxiliary classifier to the optimization at separation phase.
2. MVAE FOR DETERMINED MULTICHANNEL
SOURCE SEPARATION
2.1. Problem formulation
Let us consider a determined situation where I source signals
are captured by I microphones. Let xi (f, n) and sj (f, n)denote the short-time Fourier transform (STFT) coefficients

of the signal observed at the i-th microphone and the jth source signal, where f and n are the frequency andtime indices respectively. We denote the vectors containing x1 (f, n), . . . , xI (f, n) and s1 (f, n), . . . , sI (f, n) byx(f, n) = [x1 (f, n), . . . , xI (f, n)]T ∈ CI ,T

I

s(f, n) = [s1 (f, n), . . . , sI (f, n)] ∈ C ,

(1)
(2)

a one-hot vector indicating to which class the spectrogram S
belongs. For example, if we consider speaker identities as theclass category, each element of c will be associated with a different speaker. CVAE is trained by maximizing the following
variational lower boundJ (φ, θ) = E(S,c)∼pD (S,c) [Ez∼qφ (z|S,c) [log pθ (S|z, c)]− KL[qφ (z|S, c)||p(z)]],(9)Twhere (·) denotes transpose. Under determined situation, therelationship between observed signals and source signals canbe described ass(f, n) = W H (f )x(f, n),
W (f ) = [w1 (f ), . . . , wI (f )] ∈ C

(3)
I×I

,

(4)

where W H is called the separation matrix. (·)H denotes Hermitian transpose.
Let us assume that source signals follow the local Gaussian model (LGM), i.e., sj (f, n) independently follows
a zero-mean complex Gaussian distribution with variancevj (f, n) = E[|sj (f, n)|2 ]sj (f, n) ∼ NC (sj (f, n)|0, vj (f, n)).

(5)

When sj (f, n) and sj 0 (f, n)(j 6= j 0 ) are independent, s(f, n)followss(f, n) ∼ NC (s(f, n)|0, V (f, n)),(6)where V (f, n) = diag[v1 (f, n), . . . , vI (f, n)]. From (3) and(5), we can show that x(f, n) follows
x(f, n) ∼ NC (x(f, n)|0, (W H (f ))−1 V (f, n)W (f )−1 ).
(7)Hence, the log-likelihood of the separation matrices W ={W (f )}f and source model parameters V = {vj (f, n)}j,f,ngiven the observed mixture signals X = {x(f, n)}f,n isgiven byXclog p(X |W, V) = 2N
log | det W H (f )|
f

−

X

log vj (f, n) +

f,n,j

2
|wH
j (f )x(f, n)|,vj (f, n)(8)where =c denotes equality up to constant terms. (8) will be
split into frequency-wise source separation problems if there
is no additional constraint imposed on vj (f, n). This indicates that there is a permutation ambiguity in the separated
components for each frequency.

where E(S,c)∼pD (S,c) [·] denotes the sample mean over the
training examples {S m , cm }M
m=1 and KL[·||·] denotes KullbackLeibler divergence. Here, the encoder distribution
qφ (z|S, c) and the prior distribution of the latent space variable p(z) are expressed as Gaussian distributions
qφ (z|S, c) = N (z|µφ (S, c), diag(σ 2φ (S, c))),p(z) = N (z|0, I),

(10)
(11)

where µφ (S, c), σ 2φ (S, c) are the encoder network outputs.
The decoder distribution pθ (S|z, c) is defined as a zero-mean
complex Gaussian distribution and a scale parameter g is incorporated to eliminate the energy difference between the normalized training data and test data. Hence, the decoder distribution is expressed as
Ypθ (S|z, c, g) =NC (s(f, n)|0, v(f, n)),
(12)
f,n

v(f, n) = g · σθ2 (f, n; z, c),(13)where σθ2 (f, n; z, c) denotes the (f, n)-th element of the decoder output. It is noteworthy to mention that the decoder
distribution (12) is given in the same form as the LGM (5) sothat the trained decoder distribution can be used as a universal generative model with ability to generate complex spectrograms belonging to all the source classes involved in the
training examples. If we use pθ (S j |z j , cj , gj ) to express thegenerative model of the complex spectrogram of the source j,a convergence-guaranteed optimization algorithm can be applied to search for a stationary point of the log-likelihood byiteratively updating (i) the separation matrices W using iterative projection (IP) method [15], (ii) the CVAE source model
parameters Ψ = {z j , cj }j using backpropagation, and (iii)the global scale parameter G = {gj }j with the following update rule
1 X |yj (f, n)|2
gj ←
,
(14)
FN
σθ2 (f, n; z j , cj )f,nwhere yj (f, n) = wH
j (f )x(f, n). Note that since the class labels are the model parameters estimated during the optimization, MVAE is able to perform source classification as well.

2.2. Multichannel VAE
To eliminate the permutation ambiguity during the estimationof W, MVAE trains a conditional VAE (CVAE) to model thecomplex spectrograms S = {s(f, n)}f,n of source signals
so that the spectral structures can be captured. CVAE consists of an encoder network qφ (z|S, c) and a decoder networkpθ (S|z, c), where the network parameters φ and θ are trained
jointly using a set of labeled training samples {S m , cm }M
m=1 .
Here, c denotes the corresponding class label represented as

3. PROPOSED METHOD: FAST MVAE
While MVAE is notable in that it works reasonably well forsource separation and has capability to perform source classification simultaneously, there is still a huge room for improvement in the source classification performance. With aregular CVAE imposing no restrictions on the manner how
the encoder and decoder may use the class labels, the encoderand decoder are free to ignore c by finding distribution satisfying qφ (z|S, c) = qφ (z|S) and pθ (S|z, c) = pθ (S|z). As
a result, c will have little effect on generating source spectrograms that leads to a limited source classification performance. To avoid such situations, this paper proposes using
an auxiliary classifier VAE [16] for learning the generative
distribution pθ (S|z, c).
3.1. Auxiliary classifier VAE
Auxiliary classifier VAE (ACVAE) [16] is a variant of CVAE
that incorporates an information-theoretic regularization [17]to assisting the decoder outputs to be correlated as far as possible with the class labels c by maximizing the mutual information between c and S ∼ pθ (S|z, c) conditioned on z. The
mutual information is expressed asI(c, S|z)0= Ec∼p(c),S∼pθ (S|z,c),c0 ∼p(c|S) [log p(c |S)] + H(c), (15)where H(c) represents the entropy of c that can be considered as a constant term. However, it is difficult to optimize
I(c, S|z) directly since it requires access to the posteriorp(c|S). Fortunately, we can obtain a variational lower boundof the first term of I(c, S|z) by using a variational distribution r(c|S) to approximate p(c|S):Ec∼p(c),S∼pθ (S|z,c),c0 ∼p(S|c) [log p(c0 |S)]

tion, but also provides an alternative to the backpropagation
process in the original MVAE optimization, which is able tosignificantly reduce the computational time. We summarize
the proposed fast algorithm as follows:
1. Train φ, θ and ψ using (19).
2. Initialize W.
3. Iterate the following steps for each j:(a) Update cj using rψ (cj |S j ).
(b) Update z j using qφ (z j |S j , cj ).
(c) Update gj using (14).

r(c0 |S)p(c0 |S)]=Ec∼p(c),S∼pθ (S|z,c),c0 ∼p(S|c) [logr(c0 |S)≥Ec∼p(c),S∼pθ (S|z,c),c0 ∼p(S|c) [log r(c0 |S)]
=Ec∼p(c),S∼pθ (S|z,c) [log r(c|S)],

Fig. 1. Configuration of the room where ◦ and × represent
the position of microphones and sources respectively.

(d) Update wj (0), . . . , wj (F ) using IP.
(16)

4. EXPERIMENTS

the equality of which holds if r(c|S) = p(c|S). We therefore
can indirectly maximizing I(c, S|z) by increasing the lower
bound with respect to pθ (S|z, c) and r(c|S). One way to do
this involves expressing the variational distribution as a neural network rψ (c|S) and training it along with qφ (z|S, c) andpθ (S|z, c). rψ (c|S) is called the auxiliary classifier. Therefore, the regularization term that we would like to maximize
with respect to φ, θ, ψ becomesTo evaluate the effect of incorporating an auxiliary classifier
into both the source model training and the optimization process, we conducted experiments to compare the multi-speakerseparation performances, source classification accuracies andcomputational times of fMVAE and the conventional methods, i.e., ILRMA [5, 6] and MVAE [11].

L(φ, θ, ψ)(17)= E(S,c)∼pD (S,c),qφ (z|S,c) [Ec∼p(c),S∼pθ (S|z,c) [log rψ (c|S)]].

We excerpted speech utterances from two male speakers(‘SM1’, ‘SM2’) and two female speakers (‘SF1’, ‘SF2’) fromthe Voice Conversion Challenge (VCC) 2018 dataset [18].
The audio files for each speaker were about 7 minutes longand manually segmented into 116 short sentences, where 81
and 35 sentences (about 5 and 2 minutes respectively) wereused as training and test sets, respectively. The mixture signals were created by simulated two-channel recordings of twosources where the room impulse responses were synthesizedusing the image method. We tested two different reverberant
conditions where the reverberation time (RT60 ) was set at 78
ms and 351 ms, respectively. Fig. 1 shows the configurationof the room. We generated test data involving 4 speaker pairs
and 10 sentences for each pair, namely there were totally 40
test signals, each of which was about 4 to 7 seconds long.
All the speech signals were resampled at 16 kHz. The STFT
was computed using Hamming window with 256 ms long andwindow shift was128 ms.
ILRMA was run for 100 iterations and both the proposed
method and MVAE were run 40 iterations. To initialize W for

Since the labeled training samples can also be used to train theauxiliary classifier rψ (c|S), ACVAE also includes the crossentropy
I(ψ) = E(S,c)∼pD (S,c) [log rψ (c|S)](18)

in the training criterion. The entire training criterion is thus
given byJ (φ, θ) + λL L(φ, θ, ψ) + λI I(ψ),(19)

where λL ≥ 0 and λI ≥ 0 are weight parameters.
3.2. Fast algorithm
Note that the auxiliary classifier rψ (c|S) not only assists theencoder and decoder to learn a more disentangled representa-

4.1. Experimental conditions

w 32
c 4

w 64
c 4

OutputProduct

Conv

Softmax

Conv
BN
GLU

Input

OutputDeconv

w 64 w 32 w 16
c 512 c 128 c 4
k 1x4 k 1x4 k 1x4

w 128 w 128
c 2048 c 1024
k 1x5

w 128 w 128
c 1024 c 2048
k 1x4 k 1x5

Deconv
BN
GLU

Input
Class label

w 128 w 64
c 4
c 4

Output

Conv
BN
GLU

Input
Class label

w 64
w 32
c 512
c 256
k 1x4

w 32
c 256
k 1x4

Conv

w 64
c 512
k 1x4

w 128 w 128
c 2048 c 1024
k 1x5

w 128
c 4

Classifier

Decoder

Encoderw 128
c 4

Fig. 2. Network architectures of the encoder and decoder used for MVAE and fMVAE and the classifier used for fMVAE.
The inputs and outputs are 1-dimensional data, where the frequency dimension of spectrograms is regarded as the channel
dimension. “w”, “c” and “k” denote the width, channel number and kernel size, respectively. “Conv”, “Deconv”, “BN” and“GLU” denote 1-dimensional convolution and deconvolution, batch normalization, gated linear unit, respectively.

Table 1. Average SDR, SIR and SAR scores of ILRMA,MVAE and fMVAE. The bold font shows the highest scores.
RT60 = 78 ms
methodSDR [dB] SIR [dB] SAR [dB]
ILRMA
14.899721.327718.0584
MVAE
21.591227.2663
25.1616
fMVAE
22.5976
29.8476
24.8967
RT60 = 351 ms
methodSDR [dB] SIR [dB] SAR [dB]
ILRMA
4.684011.6284
7.2364MVAE
8.315718.08349.2206fMVAE
6.781415.7728
7.7883
Table 2. Computational times of MVAE, fMVAE and ILRMA. MVAE and fMVAE were initialized with run ILRMA
algorithm for 30 iterations in CPU and run 40 iterations of the
optimization algorithms in CPU or GPU. ILRMA runs 100 iterations in CPU.
rumtime/iteration[sec] total [sec]
MVAE (GPU)6.071632
260.5953
fMVAE (CPU)0.389762
21.54129
fMVAE (GPU)0.097272
17.56694
ILRMA (CPU)
0.113571
18.38221

MVAE and fMVAE, we used ILRMA run for 30 iterations.
Adam [19] was used for training CVAE and ACVAE, and estimating the latent variables in MVAE. The network architectures used for CVAE and ACVAE is shown in Fig. 2. Notethat we used the same network architectures of the encoderand decoder for CVAE and ACVAE. All the networks are designed as fully convolutional with gated linear units [20] sothat the inputs are allowed to have arbitrary lengths. The programs were run using Intel (R) Core i7-6800K CPU@3.40
GHz and GeForce GTX 1080Ti GPU.
4.2. Results
We calculated the average of the signal-to-distortion ratios(SDR), signal-to-interference ratios (SIR) and signal-to-

Table 3. Accuracy rates of source classification obtained withMVAE and fMVAE.
all iterations final estimation
MVAE
27.91%
37.50%
fMVAE
78.63%
80.00%

artifact ratios (SAR) [21] over the 40 test signals to evaluate the source separation performance and measured thecomputational times. Table 1 and Table 2 show the source
separation results obtained under different RT60 conditions
and the computational times of ILRMA, MVAE and fMVAE, respectively. fMVAE was about 15 times faster thanthe original MVAE and even a little faster than ILRMA in
GPU machines. Furthermore, it is noteworthy that fMVAE
achieved comparative source separation performance withthe original MVAE. As the results show, MVAE and fMVAE
significantly outperformed ILRMA in terms of all the criteriaof source separation performance, which confirmed the effectof the incorporation of the CVAE source model. fMVAE
obtained higher SDR and SIR than MVAE under a low reverberant environment, but the performance slightly decreased
when RT60 became long. It is interesting to further compare
fMVAE with MVAE in high reverberant environments, which
is one direction of our future work.
To evaluation the performance of source classification, we
computed the classification accuracy rates over the results estimated in each iteration and only in the final estimation. Table 3 shows the results that fMVAE significantly improved
source classification accuracy with achieving 80% accuracy
rate. Our future work also includes further improving thesource classification accuracy.
5. CONCLUSIONSThis paper proposed fMVAE that (i) uses an auxiliary classifier VAE instead a regular CVAE for learning the generative
distribution of source signals; (ii) employs the trained auxiliary classifier and encoder for the optimization. fMVAE allows us to significantly reduce the computational time andimprove the source classification performance. The results
revealed that fMVAE achieved about 15 times faster than theoriginal MVAE and about 80% source classification accuracy
rate with notable source separation performance.

6. REFERENCES[1] T. Kim, T. Eltoft and T.-W. Lee, “Independent vector
analysis: An extension of ICA to multivariate components,” in Proc. ICA, pp. 165–172, 2006.
[2] A. Hiroe, “Solution of permutation problem in frequency domain ICA using multivariate probability density functions,” in Proc. ICA, pp. 601-608, 2006.
[3] A. Ozerov and C. Févotte, “Multichannel nonnegative
matrix factorization in convolutive mixtures for audio
source separation,” IEEE Trans. ASLP, vol. 18, no. 3,
pp. 550–563, 2010.

[14] K. Sohn, H. Lee and X. Yan, “Learning structured
output representation using deep conditional generativemodels,” in Adv. Neural Information Processing Systems(NIPS), pp. 3483–3491, 2015.
[15] N. Ono, “Stable and fast update rules for independentvector analysis based on auxiliary function technique,”in Proc. WASPAA, pp. 189–192, 2011.
[16] H. Kameoka, T. Kaneko, K. Tanaka and N. Hojo,“ACVAE-VC: Non-parallel many-to-many voice conversion with auxiliary classifier variational autoencoder,” eprint arXiv: 1808.05092, Aug. 2018.

[4] H. Sawada, H. Kameoka, S. Araki and N. Ueda, “Multichannel extensions of non-negative matrix factorization
with complex-valued data,” IEEE Trans. ASLP, vol. 21,
no. 5, pp. 971–982, 2013.

[17] X. Chen, Y. Duan, R. Houthooft, J. Schulman, I.
Sutskever and P. Abbeel “Infogan: Interpretable representation learning by information maximizing generative adversarial nets,” in Adv. Neural Information Processing Systems (NIPS), pp. 2172–2180, 2016.

[5] H. Kameoka, T. Yoshioka, M. Hamamura, J. Le. Rouxand K. Kashino, “Statistical model of speech signals
based on composite autoregressive system with application to blind source separation,” in Proc. LVA/ICA,
pp. 245–253, 2010.

[18] J. Lorenzo-Trueba, J. Yamagishi, T. Toda, D. Saito, F,
Villavicencio, T. Kinnunen and Z. Ling, “The voice
conversion challenge 2018: Promoting developmentof parallel and nonparallel methods,” eprint arXiv:
1804.04262, Apr. 2018.

[6] D. Kitamura, N. Ono, H. Sawada, H. Kameoka and H.
Saruwatari, “Determined blind source separation unifying independent vector analysis and nonnegative matrix
factorization,” IEEE/ACM Trans. ASLP, vol. 24, no. 9,
pp. 1622–1637, 2016.
[7] A. A. Nugraha, A. Liutkus and E. Vincent, “Multichannel Audio Source Separation With Deep Neural
Networks,” IEEE/ACM Trans. ASLP, vol. 24, no. 9,
pp. 1652–1664, 2016.
[8] Y. Bando, M. Mimura, K. Itoyama, K. Yoshii and T.
Kawahara, “Statistical speech enhancement based on
probabilistic integration of variational autoencoder andnon-negative matrix factorization,” in Proc. ICASSP,
pp. 716–720, 2018.
[9] S. Mogami, H. Sumino, D. Kitamura, N. Takamune,
S. Takamichi, H. Saruwatari and N. Ono, “Independent
deeply learned matrix analysis for multichannel audio
source separation,” in Proc. EUSIPCO, pp. 1571–1575,
2018.
[10] S. Leglaive, L. Girin and R. Horaud, “A variance modeling framework based on variational autoencoders forspeech enhancement,” in Proc. MLSP, 2018.
[11] H. Kameoka, L. Li, S. Inoue and S. Makino, “Semiblind source separation with multichannel variational
autoencoder,” eprint arXiv: 1808.00892, Aug. 2018.
[12] S. Seki, H. Kameoka, L. Li, T. Toda and K.
Takeda, “Generalized multichannel variational autoencoder for underdetermined source separation,” eprint
arXiv: 1810.00223, Oct. 2018.
[13] D. P. Kingma, S. Mohamed, D. J. Rezende and M.
Welling, “Semi-supervised learning with deep generative models,” in Adv. Neural Information Processing
Systems (NIPS), pp. 3581–3589, 2014.

[19] D. Kingma, J. Ba, “Adam: A method for stochastic optimization,” in Proc. ICLR, 2015.
[20] Y. N. Dauphin, A. Fan, M. Auli and D. Grangier, “Language modeling with gated convolutional networks,” inProc. ICML, pp. 933–941, 2017.
[21] E. Vincent, R. Gribonval and C. Févotte, “Performancemeasurement in blind audio source separation,” IEEE
Trans. ASLP, vol. 14, no. 4, pp. 1462–1469, 2006.