2020 IEEE INTERNATIONAL WORKSHOP ON MACHINE LEARNING FOR SIGNAL PROCESSING, SEPT. 21–24, 2020, ESPOO, FINLANDDETERMINED AUDIO SOURCE SEPARATION WITH
MULTICHANNEL STAR GENERATIVE ADVERSARIAL NETWORK
Li Li1 , Hirokazu Kameoka2 , Shoji Makino1
1

2

University of Tsukuba, JapanNTT Communication Science Laboratories, NTT Corporation, Japan
ABSTRACTThis paper proposes a multichannel source separation approach, which uses a star generative adversarial network(StarGAN) to model power spectrograms of sources. Various
studies have shown the significant contributions of a precise source model to the performance improvement in audio
source separation, which indicates the importance of developing a better source model. In this paper, we explore thepotential of StarGAN for modeling source spectrograms andinvestigate the effectiveness of the StarGAN source model indetermined multichannel source separation by incorporating
it into a frequency-domain independent component analysis(ICA) framework. The experimental results reveal that the
proposed StarGAN-based method outperformed conventional
methods that use non-negative matrix factorization (NMF) or
a variational autoencoder (VAE) for source spectrogram modeling.
Index Terms— Multichannel audio signal processing,determined source separation, star generative adversarial network (StarGAN), spectrogram modeling, deep generativemodel
1. INTRODUCTION
The aim of blind source separation (BSS) [1] is to separate individual source signals from microphone array inputs without
prior information about the sources and the mixing methodology. The frequency-domain BSS approach is usually preferred since it allows a fast implementation based on the instantaneous mixture assumption and provides the flexibilityof utilizing various models for the time-frequency representations of source signals, e.g., spectrograms.
For example, independent vector analysis (IVA) [2, 3]
models power spectrograms of sources as a single flat-shaped
spectral basis scaled by time-varying amplitudes based on theassumption that the magnitudes of the frequency components
originating from the same source tend to vary coherently
over time. Multichannel extensions of non-negative matrix
factorization (NMF), e.g., multichannel NMF (MNMF) [4, 5]This work was supported by JSPS KAKENHI 18J20059 and 19H04131,
and JST CREST JPMJCR19A3.

978-1-7281-6662-9/20/$31.00 c 2020 IEEEand independent low-rank matrix analysis (ILRMA) [6, 7],
incorporate the NMF concept into the source model to capture spectral structures of sources. Although these methods
work reasonably well in most cases, they can fail to separate sources that do not satisfy the low-rank assumption.
Motivated by the strong power of deep generative models,
including variational autoencoders (VAEs) [8] and generative
adversarial networks (GANs) [9], to learn data distributions,some attempts have recently been made to apply these models
to speech enhancement and source separation tasks [10–15].
The multichannel variational autoencoder (MVAE) [12] is
one such method, where a conditional VAE (CVAE) [8] istrained using the spectrograms of clean speech samples alongwith the corresponding speaker identity as an auxiliary label
input so that the trained decoder distribution can be used
as a universal generative model of source signals. At the
separation phase, the trained decoder is applied to estimate
the spectrograms of sources in a mixture. MVAE and its
generalized version, GMVAE [16], have been demonstrated
to significantly outperform ILRMA and MNMF, which confirms the effectiveness of incorporating a more precise source
model in improving the source separation performance.
Compared to VAE, which explicitly assumes the prior distribution about the data, e.g., a Gaussian distribution, andlearns data distribution by forcing an approximate posterior
distributions to become consistent with the true one, GAN
trains a generator network to deceive a real/fake discriminator network so that the generator distribution is optimized to
fit the target distribution without explicit density estimation.
This allows us to avoid the mismatch between the assumed
and real distributions and the approximation error occurringin the posterior estimation. Thanks to this learning strategy, it
is expected that GAN can learn a data distribution more accurately than VAE.
To take advantage of GAN, this paper proposes a determined multichannel source separation method that employs
StarGAN [17] to learn the generative distribution of power
spectrograms of sources. StarGAN is a GAN variant consisting of a generator, discriminator, and domain classifier.
StarGAN, which was originally proposed for multi-domaintranslation, has recently been adapted for use in many-tomany voice conversion [18] and shown to perform remarkably. In addition, the following three benefits motivate us toadopt StarGAN in the proposed method. First, by using thedomain classifier to measure how likely the generated spectrogram is to belong to the corresponding speaker, the model
learns to avoid ignoring the class index input when generating spectrograms. This is in contrast to a regular conditional
GAN, which is free to ignore the class index input when the
networks have sufficient capacity. Second, it is expected thattraining a model in a conversion manner can promote the disentanglement between the latent representation and the classindex, which makes the latent representation more meaningful. Third, the network composition involving a domain classifier makes it possible to apply the fast algorithm implemented in FastMVAE [19]. We evaluate the effectiveness of
StarGAN in modeling source spectrograms by comparing theperformance of the proposed StarGAN-based method, which
we refer to as multichannel StarGAN (MSGAN), with ILRMA and MVAE in determined source separation.

(5), we can show that x(f, n) follows
x(f, n) ∼ NC (x(f, n)|0, (W H (f ))−1 V (f, n)W (f )−1 ).
(7)Hence, the log-likelihood of the separation matrices W ={W (f )}f and source model parameters V = {vj (f, n)}j,f,ngiven the observed mixture signals X = {x(f, n)}f,n is expressed asXclog p(X |W, V) = 2N
log | det W H (f )|
f

X

−

log vj (f, n) +

f,n,j

2
|wH
j (f )x(f, n)|,vj (f, n)(8)where =c denotes equality up to constant terms.
2.2. Conventional methods with different source models

2. DETERMINED MULTICHANNEL SOURCESEPARATION

2.2.1. ILRMA

2.1. Problem formulation
Let us consider a determined situation where I source signals
are captured by I microphones. Let xi (f, n) and sj (f, n)denote the short-time Fourier transform (STFT) coefficientsof the signal observed at the ith microphone and the jthsource signal, where f and n are the frequency and time
indices, respectively. We denote the vectors containing
x1 (f, n), . . . , xI (f, n) and s1 (f, n), . . . , sI (f, n) byx(f, n) = [x1 (f, n), . . . , xI (f, n)]T ∈ CI ,T

I

s(f, n) = [s1 (f, n), . . . , sI (f, n)] ∈ C ,

(1)
(2)

where (·)T denotes transpose. In a determined situation, therelationship between observed signals and source signals canbe described ass(f, n) = W H (f )x(f, n),
W (f ) = [w1 (f ), . . . , wI (f )] ∈ C

(3)
I×I

,

(4)

where W H (f ) is called the separation matrix. (·)H denotes
the Hermitian transpose.
We assume source signals follow the local Gaussianmodel (LGM) [20]. Namely, sj (f, n) independently follows a zero-mean complex proper Gaussian distribution withpower spectral density vj (f, n) = E[|sj (f, n)|2 ]:sj (f, n) ∼ NC (sj (f, n)|0, vj (f, n)).

(5)

When sj (f, n) and sj 0 (f, n)(j 6= j 0 ) are independent, s(f, n)followss(f, n) ∼ NC (s(f, n)|0, V (f, n)),Constraints on vj (f, n) are usually imposed to eliminate thepermutation ambiguity during the estimation of W. The
NMF model incorporated in ILRMA [7] is an example,
which approximates vj (f, n) as a linear sum of spectral
templates bj,k (f ) ≥ 0, k = 1, . . . , Kj scaled by timevarying magnitudes hj,k (n) ≥ 0, k = 1, . . . , Kj , namely,
PKj
vj (f, n) =k bj,k (f )hj,k (n), to capture spectral structures of sources. The parameter estimation algorithm ofILRMA consists of iteratively updating W using the iterative
projection (IP) method [21], and updating B = {bj,k (f )}j,k,fand H = {hj,k (n)}j,k,n using majorization-minimization(MM) algorithm-based update rules.
2.2.2. MVAE
In MVAE [12], a decoder distribution pθ (S|z, c, g) is trained
jointly with an encoder distribution qφ (z|S, c) so that theencoder distribtuion qφ (z|S, c) becomes consistent with thetrue posterior pθ (z|S, c, g) ∝ pθ (S|z, c, g)p(z)p(c), where
φ and θ are parameters of the encoder and decoder, respectively. Here, S and c respectively denote complex spectrograms and class labels indicating to which class the spectrogram S belongs. For example, if speaker identity is considered as the class category, c will be associated with a different
speaker, which is represented as a one-hot vector consistingof M elements, which is filled with 1 at the index of a certain speaker and with 0 everywhere else. The trained decoder
distribution, specifically defined as
Ypθ (S|z, c, g) =NC (s(f, n)|0, v(f, n)),(9)f,nv(f, n) = g · σθ2 (f, n; z, c),

(10)

(6)

where V (f, n) = diag[v1 (f, n), . . . , vI (f, n)]. From (3) andcan then be applied as a spectrogram generator, which takes
latent variable z, speaker ID c, and global scale parameter g

as inputs, and outputs the distribution parameter v(f, n). Thisis called the CVAE source model. At the separation phase,pθ (S j |z j , cj , gj ) is then used as the generative model of thecomplex spectrogram of the source j in a mixture. A stationary point of the log-likelihood (8) that we want to maximize
is searched by iteratively updating (A) the separation matrices
W using the IP method [21]:wj ← (W H (f )Σj (f ))−1 ej ,wj (f ),
wj ← Hwj (f )Σj (f )wj (f )

(11)
(12)

P
where Σj (f ) = N1 n x(f, n)xH (f, n)/vj (f, n) and ej denotes the j-th column of an I × I identity matrix; (B) the
CVAE source model parameters Ψ = {z j , cj }j with gredient descent (backpropagation); (C) the global scale parameter
G = {gj }j with the following update rule:gj ←2
1 X |wHj (f )x(f, n)|.
FNσθ2 (f, n; z j , cj )

(13)

f,n

Fig. 1. Concept of StarGAN training.
ployed to produce a probability D(Ŝ) to measure how likely
the generated Ŝ is a real spectrogram, whereas C is employed
to produce class probabilities pC (c|Ŝ) of Ŝ.
First, we define an adversarial loss using the Wasserstein
GAN with gradient penalty (WGAN-GP) [22], which can stabilize the training procedure of GAN:
LD
adv (D) = Ec∼p(c), S∼p(S) [D(G(S, c))] − ES∼p(S) [D(S)]+ λgrad ES̃∼p(S̃) [(||∇S̃ D(S̃)||2 − 1)2 ],3. MSGAN WITH STARGAN SOURCE MODEL
Compared to the linear NMF model, the nonlinear CVAE
source model not only increases the representation power but
also makes it possible to capture the temporal structures ofsources thanks to its well-designed network architectures forsequential modeling. MVAE has been shown to exceed ILRMA in multi-speaker separation tasks [12], which implies
the effectiveness of the source model with stronger representation power in the LGM-based BSS framework. One promising approach to achieve further improvement includes GAN,
where the generator distribution is optimized to fit the real
data distribution by playing a minimax game between a generator and a discriminator. Thanks to the training strategy,
GAN is expected to learn data distributions much closer tothe real one than those that VAE can learn without suffering
the mismatch between the assumed and real data distributions
and the learning error coming from the approximation. This
motivates us to exploit GAN to model power spectrograms. In
this section, we first introduce a source model that is trainedwith StarGAN in a voice conversion fashion in Subsec. 3.1.
We then describe network architectures and summarize thealgorithm of MSGAN in Subsec. 3.2 and Subsec. 3.3, respectively.
3.1. Objective functions of StarGAN
Let G be a generator that takes a power spectrogram S anda target speaker ID c as the inputs and generates a power
spectrogram Ŝ = G(S, c). Note that the generated Ŝ corresponds to σθ (f, n; z, c) in (10). One of the goals of StarGAN
is to make Ŝ as realistic as real spectrograms belonging to the
speaker c. To realize this, a real/fake discriminator D as withregular GAN and a domain classifier C are used. D is em-LGadv (G) = −ES∼p(S), c∼p(c) [D(G(S, c))].

(14)
(15)

Here, E[·] denotes sample mean, || · ||2 denotes L2 norm, andλgrad is a non-negative weight parameter. S̃ denotes data sampled uniformly along straight lines between pairs of points
sampled from the real spectrogram distribution p(S) and thegenerator distribution pG (Ŝ). LD
adv (D) takes a small value
when D correctly classifies G(S, c) and S as fake and real
spectrograms, whereas LG
adv (G) takes a small value when G
successfully deceives D so that G(S, c) is misclassified as areal spectrogram by D. Next, we consider domain classification losses for classifier C and generator G, which are definedasLCcls (C) = −Ec∼p(c), S∼p(S|c) [log pC (c|S)],LG
cls (G) = −Ec∼p(c), S∼p(S) [log pC (c|G(S, c))].

(16)

G
Both LCcls (C) and Lcls (G) take small values when C correctlyclassifies S ∼ p(S|c) and G(S, c) as belonging to speakerc. Training G, D, and C using only the above losses does
not guarantee that G will preserve the linguistic informationof the input spectrogram. To encourage G(S, c) to be a bijection, a cycle consistency loss is also employed for training,which is expressed as

Lcyc (G)(17)= Ec0 ∼p(c), S∼p(S|c0 ), c∼p(c) [||G(G(S, c), c0 ) − S||1 ],where || · ||1 denotes L1 norm. We also consider an identity
mapping loss
Lid (G) = Ec∼p(c), S∼p(S|c) [||G(S, c) − S||1 ](18)

to ensure that an input spectrogram into G will remain un-

Fig. 2. Network architectures of the generator, discriminator, and domain classifier. “s”, “c”, and “k” denote data size, channel
number, and kernel size, respectively. “Conv”, “Deconv”, “IN”, and “LReLU” denote 2-dimensional convolution and deconvolution, instance normalization, and Lecky ReLU, respectively. Class index is concatenated along channel dimension.
changed when the input already belongs to the target speaker.
To summarize, the full objectives of StarGAN to be minimized with respect to G, D, and C are given asG
IG (G) =LG
adv (G) + λcls Lcls (G)+ λcyc Lcyc (G) + λid Lid (G),ID (D) =LD
adv (D),
IC (C) =LC
cls (C),

(19)
(20)
(21)

respectively, where λcls ≥ 0, λcyc ≥ 0, λid ≥ 0 are regularization parameters weighing the importance of the domainclassification loss, the cycle consistency loss, and the identity
mapping loss relative to the adversarial losses. Fig. 1 shows
the concept of a StarGAN.
3.2. Network architectures
For network architectures, we consider networks constructedusing fully convolutional layers to cope with signals having
arbitrary lengths, and capture time dependencies. Specifically, we use 2-dimensional convolutional neural networks(CNNs) to design the architectures for all the networks. The
generator consists of two parts, similar to an encoder-decoder
structure. The first part aims to extract the low-dimensional
latent representation z of the input spectrogram, whereas thesecond part takes class index c as an auxiliary input and performs spectrogram conversion. We used the second part as
the source model at the separation phase, where the latentvariable z and the auxiliary input c are treated as the model
parameters to be estimated. We leverage the idea of PatchGANs [23] to devise a real/fake discriminator D, the output of
which is a sequence of probabilities that measures how likely
each segment of the input is to be real. This forces the generator to generate more local details. Otherwise, it will failto deceive the discriminator. The domain classifier C is designed to share the low-level features with the discriminator.
More architecture details are shown in Fig. 2.
3.3. Algorithm of MSGAN for source separation
The proposed MSGAN method for determined multichannel
source separation is summarized as follows.

1. Train G, D, and C using (19), (20), and (21).
2. Initialize W using a BSS method, e.g., ILRMA.
3. Iterate the following steps for each j:(a) Compute S j = wH
j (f )x(f, n).
(b) Update Ψj = {z j , cj } using backpropagation to
maximize (8).
(c) Update gj using (13).
(d) Update wj (0), . . . , wj (F ) using (11), (12).
Note that the global scaling parameter gj has to be considered
as with MVAE to eliminate the scale mismatch between thenormalized training data and test data.
4. EXPERIMENTAL EVALUATIONS
To evaluate the effectiveness of the proposed StarGAN source
model, we conducted experiments designed to compare themulti-speaker separation performance of MSGAN with ILRMA [7] and MVAE1 [12].
4.1. Experimental conditions
We excerpted speech utterances from two male speakers(‘SM1’, ‘SM2’) and two female speakers (‘SF1’, ‘SF2’) fromthe Voice Conversion Challenge (VCC) 2018 dataset [24].
The audio files for each speaker were about 7 minutes longand manually segmented into 116 short sentences, where 81
and 35 sentences (about 5 and 2 minutes long, respectively)
were used as training and test sets, respectively. We used
two-channel mixture signals of two sources as the test data,
which were synthesized using the simulated room impulse
responses (RIRs) generated using the image method and real
RIRs (ANE and E2A) excerpted from the RWCP Sound
Scene Database in Real Acoustic Environments [25]. The
configuration of the room was the same as the one in [12].
The reverberation times (RT60 ) of the simulated RIRs were
78 and 351 ms, and those of real RIRs were 173 and 225
ms. We generated test data comprising 4 speaker pairs and
10 sentences for each pair, each of which was about 4 to 7
seconds long. All the speech signals were resampled at 16
kHz. To decrease memory usage during the network training,
1 Code: https://github.com/lili-0805/MVAE

Fig. 3. Example of different source models obtained under “ANE” condition.
we computed STFT using a Hamming window with a lengthof 64 ms and a shift of 32 ms, whereas the spectrograms werecomputed with a window length of 128 ms and window shiftof 64 ms at the separation phase.
ILRMA was run for 60 iterations. Both MVAE and MSGAN were run for 30 iterations after the initialization. To
initialize W for MVAE and MSGAN, we used ILRMA run
for 30 iterations. The basis number of ILRMA was set at
1. Adam [26] was used to train networks and estimate theparameter Ψ in the algorithms. The source-to-distortion ratio (SDR), source-to-interferences ratio (SIR), and sourcesto-artifacts ratio (SAR) [27] were calculated for source separation performance. Perceptual evaluation of speech quality(PESQ) 2 [28] and short-time objective intelligibility (STOI)
3[29] were also conducted to evaluate the speech quality andintelligibility.
4.2. Results
Table 1 shows SDR, SIR, SAR, PESQ, and STOI scores obtained by ILRMA, MVAE, and the proposed MSGAN. All
the results were averaged over the 40 test signals under eachreverberant condition. The results reveal that MSGAN outperformed ILRMA in terms of all the criteria. Meanwhile,
it achieved slight improvement in MVAE in terms of SDR
and comparative results in terms of other criteria. Comparing the results under each reverberant condition, we find that
MSGAN performed better in low reverberant situations andthe performance degraded with relatively heavy reverberation. Fig. 3 depicts an example of the power spectrograms
estimated by different methods. Compared to the spectrogram estimated by ILRMA where the basis number was 1,
the CVAE and StarGAN source models used in the MVAE
and MSGAN captured spectro-temporal structures of sources
more precisely. Moreover, we found that the StarGAN could
represent more details of harmonics than CVAE, while it
2 Code: https://github.com/vBaiCai/python-pesq
3 Code: https://github.com/mpariente/pystoi

Table 1. SDR, SIR, SAR [dB], PESQ, and STOI achieved
by ILRMA, MVAE, and MSGAN under various reverberant
conditions.
Reverberant conditions ILRMA MVAE MSGAN
RT60 = 78 ms
21.22
22.69
24.08
RT60 = 351 ms
5.93
7.63
6.09
ANE (RT60 = 173 ms)
19.61
19.44
20.92
E2A (RT60 = 225 ms)
6.05
6.76
6.36
Average SDR
13.20
14.13
14.36
RT60 = 78 ms
27.32
27.38
28.85
RT60 = 351 ms
12.45
14.95
12.34
ANE (RT60 = 173 ms)
25.16
23.73
25.69
E2A (RT60 = 225 ms)
13.43
15.28
13.98
Average SIR
19.59
20.33
20.21
RT60 = 78 ms
23.25
26.31
27.21
RT60 = 351 ms
7.81
8.97
8.11
ANE (RT60 = 173 ms)
21.97
23.41
24.08
E2A (RT60 = 225 ms)
7.61
7.94
7.95
Average SAR
15.16
16.66
16.84
RT60 = 78 ms
3.38
3.40
3.50
RT60 = 351 ms
1.96
2.05
1.96
ANE (RT60 = 173 ms)
3.11
3.18
3.19
E2A (RT60 = 225 ms)
2.32
2.36
2.31
Average PESQ
2.69
2.75
2.74
RT60 = 78 ms
0.9472 0.9375
0.9480
RT60 = 351 ms
0.8059 0.8221
0.8074
ANE (RT60 = 173 ms)
0.9065 0.9047
0.9047
E2A (RT60 = 225 ms)
0.7635 0.7666
0.7585
Average STOI
0.8558 0.8577
0.8547might lead to more distortions locally.
5. CONCLUSIONSThis paper proposed a determined multichannel source separation method, which incorporates a source model trainedusing StarGAN into the LGM-based BSS framework. We
investigated the effectiveness of the StarGAN source model
in source separation and compared it with the NMF model
adopted in ILRMA and the CVAE source model employed in
the MVAE. The results showed that the proposed method outperformed ILRMA in terms of all the criteria and exceeded
MVAE in terms of SDR, which confirmed the effectivenessof StarGAN in modeling spectrograms.
6. REFERENCES[1] S. Makino, “Blind source separation of convolutive mixturesof speech,” in Adaptive signal processing, pp. 195–225, 2003.
[2] T. Kim, T. Eltoft and T.-W. Lee, “Independent vector analysis: An extension of ICA to multivariate components,” in Proc.
ICA, pp. 165–172, 2006.
[3] A. Hiroe, “Solution of permutation problem in frequency domain ICA using multivariate probability density functions,” inProc. ICA, pp. 601-608, 2006.

[14] X. Hao, X. Su, Z. Wang, H. Zhang, and Batushiren, “UNetGAN: A Robust Speech Enhancement Approach in Time Domain for Extremely Low Signal-to-Noise Ratio Condition,” inProc. Interspeech, pp. 1786–1790, 2019.
[15] J. Lin, S. Niu, Z. Wei, X. Lan, A. Jvan Wijngaarden, M. C
Smith, and K. Wang, “Speech enhancement using forked generative adversarial networks with spectral subtraction,” in Proc.
Interspeech, pp. 3163–3167, 2019.
[16] S. Seki, H. Kameoka, L. Li, T. Toda and K. Takeda, “Generalized multichannel variational autoencoder for underdeterminedsource separation,” in Proc. EUSIPCO, pp. 1973–1977. 2019.
[17] Y. Choi, M. Choi, M. Kim, J.-W. Ha, S. Kim, and J. Choo,“StarGAN: Unified generative adversarial networks for multidomain image-to-image translation,” in Proc. CVPR, pp. 8789–
8797, 2018.
[18] H. Kameoka, T. Kaneko, K. Tanaka, and N. Hojo, “StarGANVC: Non-parallel many-tomany voice conversion using star
generative adversarial networks,” in Proc. SLT, pp. 266–273,
2018.

[4] A. Ozerov and C. Févotte, “Multichannel nonnegative matrix
factorization in convolutive mixtures for audio source separation,” IEEE Trans. ASLP, vol. 18, no. 3, pp. 550–563, 2010.

[19] L. Li, H. Kameoka, and S. Makino, “Fast MVAE: Joint separation and classification of mixed sources based on multichannel variational autoencoder with auxiliary classifier,” in Proc.
ICASSP, pp. 546–550, 2019.

[5] H. Sawada, H. Kameoka, S. Araki and N. Ueda, “Multichannel
extensions of non-negative matrix factorization with complexvalued data,” IEEE Trans. ASLP, vol. 21, no. 5, pp. 971–982,
2013.

[20] C. Févotte and J-F Cardoso, “Maximum likelihood approachfor blind audio source separation using time-frequency Gaussian source models,” in Proc. WASPAA, pp. 78–81, 2005.

[6] H. Kameoka, T. Yoshioka, M. Hamamura, J. Le. Roux and K.
Kashino, “Statistical model of speech signals based on composite autoregressive system with application to blind source
separation,” in Proc. LVA/ICA, pp. 245–253, 2010.
[7] D. Kitamura, N. Ono, H. Sawada, H. Kameoka and H.
Saruwatari, “Determined blind source separation unifying independent vector analysis and nonnegative matrix factorization,” IEEE/ACM Trans. ASLP, vol. 24, no. 9, pp. 1622–1637,
2016.

[21] N. Ono, “Stable and fast update rules for independent vector
analysis based on auxiliary function technique,” in Proc. WASPAA, pp. 189–192, 2011.
[22] I. Gulrajani, F. Ahmed, M. Arjovsky, V. Dumoulin, and A. C
Courville, “Improved training of Wasserstein GANs,” in Proc.
NIPS, pp. 5767–5777, 2017.
[23] P. Isola, J.-Y. Zhu, T. Zhou, and A. A Efros, “Image-to-imagetranslation with conditional adversarial networks,” in Proc.
CVPR, pp. 1125–1134, 2017.

[8] D. P. Kingma, S. Mohamed, D. J. Rezende and M. Welling,“Semi-supervised learning with deep generative models,” inAdv. Neural Information Processing Systems (NIPS), pp. 3581–
3589, 2014.

[24] J. Lorenzo-Trueba, J. Yamagishi, T. Toda, D. Saito, F, Villavicencio, T. Kinnunen and Z. Ling, “The voice conversion challenge 2018: Promoting development of parallel and nonparallel methods,” eprint arXiv: 1804.04262, Apr. 2018.

[9] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. WardeFarley, S. Ozair, A. Courville, and Y. Bengio, “Generative adversarial nets,” in Proc. NIPS, pp. 2672–2680, 2014.

[25] S. Nakamura, K. Hiyane, F. Asano and T. Endo, “Sound scene
data collection in real acoustical environments,” J. Acoust. Soc.
Jpn. (E), vol. 20, no. 3, pp. 225–231, 1999.

[10] Y. Bando, M. Mimura, K. Itoyama, K. Yoshii and T. Kawahara, “Statistical speech enhancement based on probabilistic
integration of variational autoencoder and non-negative matrix
factorization,” in Proc. ICASSP, pp. 716–720, 2018.

[26] D. Kingma, J. Ba, “Adam: A method for stochastic optimization,” in Proc. ICLR, 2015.

[11] S. Leglaive, L. Girin and R. Horaud, “A variance modelingframework based on variational autoencoders for speech enhancement,” in Proc. MLSP, 2018.
[12] H. Kameoka, L. Li, S. Inoue, and S. Makino, “Supervised determined source separation with multichannel variational autoencoder,” Neural computation, vol. 31, no. 9, pp. 1891–1914, 2019.
[13] S. Pascual, A. Bonafonte, and J. Serra, “SEGAN: Speechenhancement generative adversarial network,” eprint arXiv:
1703.09452, March, 2017.

[27] E. Vincent, R. Gribonval and C. Févotte, “Performance measurement in blind audio source separation,” IEEE Trans. ASLP,
vol. 14, no. 4, pp. 1462–1469, 2006.
[28] A. W. Rix, J. G. Beerends, M. P. Hollier and A. P. Hekstra, “Perceptual evaluation of speech quality (PESQ)-a new
method for speech quality assessment of telephone networks
and codecs,” in Proc. ICASSP, Cat. No. 01CH37221, vol. 2,
pp. 749–752, 2001.
[29] C. H. Taal, R. C. Hendriks, R. Heusdens and J. Jensen, “A
short-time objective intelligibility measure for time-frequency
weighted noisy speech,” in Proc. ICASSP, pp. 4214–4217,
2010.