INTERSPEECH 2019
September 15–19, 2019, Graz, Austria

StarGAN-VC2:
Rethinking Conditional Methods for StarGAN-Based Voice Conversion
Takuhiro Kaneko, Hirokazu Kameoka, Kou Tanaka, Nobukatsu Hojo
NTT Communication Science Laboratories, NTT Corporation, Japantakuhiro.kaneko.tb@hco.ntt.co.jp

Abstract

fails and requires other painstaking processes, i.e., careful prescreening or manual correction.
As a solution, non-parallel VC has begun to be studied.
Non-parallel VC, which is comparable to parallel VC, is generally quite challenging to achieve owing to its disadvantageous
training conditions. To mitigate this difficulty, several studies
have used additional data (e.g., parallel utterance pairs amongreference speakers [10, 11, 12, 13]) or extra modules (e.g., automatic speech recognition (ASR) modules [14, 15]). These additional data and extra modules are useful for simplifying training but require other costs for preparation. To avoid such additional costs, recent studies have introduced probabilistic deep
generative models, such as an RBM [16], variational autoencoders (VAEs) [17, 18]), and GANs [17, 19]. Among them,
CycleGAN-VC [19] (published [20] and further improved [21])shows promising results by configuring CycleGAN [22, 23, 24]with a gated CNN [25] and identity-mapping loss [26]. This
makes it possible to learn a sequence-based mapping function without relying on parallel data. With this improvement,
CycleGAN-VC performs comparably to parallel VC [2].
Along with non-parallel VC, another practically important
issue is non-parallel multi-domain VC, i.e., learning mappings
among multiple domains (e.g., multiple speakers) without relying on parallel data. This problem is challenging in terms ofscalability because typical VC methods (including CycleGANVC) are designed to learn a one-to-one mapping; therefore, theyrequire the learning of multiple generators to achieve multidomain VC. For this problem, StarGAN-VC [27] provides a
promising solution by extending CycleGAN-VC to a conditional setting and incorporating domain codes. Through this extension, StarGAN-VC makes it possible to achieve non-parallel
multi-domain VC by only using a single generator while maintaining the advantage of CycleGAN-VC. The subjective evaluation [27] demonstrates that StarGAN-VC outperforms anotherstate-of-the-art method, i.e., VAE/GAN-VC [17].
However, even using StarGAN-VC, there is still an insurmountable gap between real and converted speech. To
bridge this gap, we rethink conditional methods of StarGANVC, which are key components for solving non-parallel multidomain VC using a single generator, and propose an improved
variant called StarGAN-VC2. In particular, we rethink conditional methods in two aspects: training objectives and networkarchitectures. For the former, we propose a source-and-target
conditional adversarial loss, which encourages all source domain data to be converted into the target data. For the latter, we introduce a modulation-based conditional method that
can transform the modulation of acoustic features in a domaindependent manner. We examined the performance of the proposed methods on the multi-speaker VC task using the Voice
Conversion Challenge 2018 (VCC 2018) dataset [28]. An objective evaluation demonstrates that the proposed methods effectively bring the converted acoustic feature sequence close to

Non-parallel multi-domain voice conversion (VC) is a technique for learning mappings among multiple domains without
relying on parallel data. This is important but challenging owing
to the requirement of learning multiple mappings and the nonavailability of explicit supervision. Recently, StarGAN-VC has
garnered attention owing to its ability to solve this problem only
using a single generator. However, there is still a gap between
real and converted speech. To bridge this gap, we rethink conditional methods of StarGAN-VC, which are key components
for achieving non-parallel multi-domain VC in a single model,
and propose an improved variant called StarGAN-VC2. Particularly, we rethink conditional methods in two aspects: training
objectives and network architectures. For the former, we propose a source-and-target conditional adversarial loss that allows
all source domain data to be convertible to the target domain
data. For the latter, we introduce a modulation-based conditional method that can transform the modulation of the acoustic
feature in a domain-specific manner. We evaluated our methods on non-parallel multi-speaker VC. An objective evaluation
demonstrates that our proposed methods improve speech quality in terms of both global and local structure measures. Furthermore, a subjective evaluation shows that StarGAN-VC2 outperforms StarGAN-VC in terms of naturalness and speaker similarity.1
Index Terms: voice conversion (VC), non-parallel VC,
multi-domain VC, generative adversarial networks (GANs),StarGAN-VC

1. Introduction
Voice conversion (VC) is a technique for converting thenon/para-linguistic information between source and target
speech while preserving the linguistic information. VC has
been studied intensively owing to its high potential for various
applications, such as speaking aids and style/pronunciation conversion.
One well-established approach to VC involves statisticalmethods based on Gaussian mixture models (GMMs) [1, 2]
and neural networks (NNs) (including restricted Boltzmann
machines (RBMs) [3], feed forward NNs (FNNs) [4], recurrent NNs (RNNs) [5, 6], convolutional NNs (CNNs) [7], attention networks [8, 9], and generative adversarial networks(GANs) [7]). Many VC methods (including the abovementioned) are categorized as parallel VC, which learns a mapping using the training data of parallel utterance pairs. However, obtaining such data is often time-consuming or impractical. Moreover, even if such data are obtained, most VC methods rely on a time alignment procedure, which occasionally
1 The converted speech samples are provided at http://www.kecl.ntt.co.jp/people/kaneko.takuhiro/
projects/stargan-vc2/index.html.

Copyright © 2019 ISCA

679

http://dx.doi.org/10.21437/Interspeech.2019-2236

[Previous]

RealA[Previous]
Real
B

Classifiable
Fake
A→A
Fake
B→A

Real
A

Fake
A→B
Real
Real
B
A

(a) Classification loss

[Proposed]
Fake
A

Fake
B→A

Fake
B→A

[Previous]
Fake
A→A

Conv

Domain
code

Real
A

Hard negative(b) Target conditional
adversarial loss

RealA

Modulate

γc'

Easy negative

Fake
A→A

[Proposed]

βc'

SelectFake
C→AExpandConcatDomain code(c) Source-and-target
conditional adversarial loss(a) Channel-wise(b) Modulation-based

Figure 2: Comparison of conditional methods in generator networks. We consider the case when convolutional networks areused. Such networks are commonly used in state-of-the-art VC
models (e.g., CycleGAN-VC [19]and StarGAN-VC [27]).

Figure 1: Comparison of conditional methods in training objectives. “A” and “B” denote the domain codes and “A → B”
represents the data converted from “A” to “B.” Circle and triangle markers denote real and fake data, respectively. (a) In
the classification loss, G prefers to generate classifiable (i.e.,
far from the decision boundary) data. (b) In the target conditional adversarial loss, D needs to simultaneously handle hard
negative samples (e.g., conversion between the same speaker
A → A) and easy negative samples (e.g., conversion between
completely different speakers B → A). (c) The proposedsource-and-target conditional adversarial loss can bring all theconverted data close to the target data in both source-wise andtarget-wise manners.

Classification loss: The aim of StarGAN-VC is to synthesize the acoustic feature that belongs to the target domain. To
achieve this, the classification loss is used. First, the classifier
C is trained for real acoustic features:
Lrcls = E(x,c)∼P (x,c) [− log C(c|x)],(2)

where C attempts to classify a real acoustic feature x to thecorresponding domain c by minimizing this loss. Subsequently,
G is optimized for C:

the target one in terms of both global and local structure measures. A subjective evaluation shows that StarGAN-VC2 outperforms StarGAN-VC in terms of both naturalness and speakersimilarity.
In Section 2, we review the conventional StarGAN-VC. In
Section 3, we describe the proposed StarGAN-VC2. In Section 4, we report the experimental results. We conclude in Section 5 with a brief summary and mention of future work.

Lfcls = Ex∼P (x),c0 ∼P (c0 ) [− log C(c0 |G(x, c0 ))],

(3)

where G attempts to generate an acoustic feature that is classified to the target domain c0 by minimizing this loss.
Cycle-consistency loss: Although the adversarial loss andclassification loss encourage a converted acoustic feature to become realistic and classifiable, respectively, they do not guarantee that the converted feature will preserve the input composition. To mitigate this problem, the cycle-consistency loss isused:

2. Conventional StarGAN-VC
2.1. Training objectives

Lcyc = E(x,c)∼P (x,c),c0 ∼P (c0 ) [kx − G(G(x, c0 ), c)k1 ]. (4)The aim of StarGAN-VC is to obtain a single generator G thatlearns mappings among multiple domains (e.g., multiple speakers). To achieve this, StarGAN-VC extends CycleGAN-VC toa conditional setting with a domain code (e.g., a speaker identifier). More precisely, StarGAN-VC learns a generator G thatconverts an input acoustic feature x into an output feature x0
conditioned on the target domain code c0 , i.e., G(x, c0 ) → x0 .
Here, let x ∈ RQ×T be an acoustic feature sequence where Q
is the feature dimension and T is the sequence length, and letc ∈ {1, . . . , N } be the corresponding domain code where N is
the number of domains. Inspired by StarGAN [29], which was
originally proposed in computer vision for multi-domain imageto-image translation, StarGAN-VC solves this problem by using an adversarial loss [30], classification loss [31], and cycleconsistency loss [32]. Additionally, inspired by CycleGANVC [19], StarGAN-VC also uses an identity-mapping loss [26]to preserve linguistic composition.
Adversarial loss: The adversarial loss is used to render theconverted feature indistinguishable from the real target feature:

This cyclic constraint encourages G to find out an optimal
source and target pair that does not compromise the composition.
Identity-mapping loss: To impose a further constraint on
the input preservation, the identity-mapping loss is used:
Lid = E(x,c)∼P (x,c) [kG(x, c) − xk].

(5)

Full objective: The full objective is written as
LD = − Lt-adv ,LC = λcls Lrcls ,

(6)
(7)

LG = Lt-adv + λcls Lfcls + λcyc Lcyc + λid Lid ,(8)

where D, C, and G are optimized by minimizing LD , LC , andLG , respectively.
2.2. Network architectures

Lt-adv = E(x,c)∼P (x,c) [log D(x, c)]+ Ex∼P (x),c0 ∼P (c0 ) [log(1 − D(G(x, c0 ), c0 ))],

Regarding the network architectures, this study focuses on theconditional method in the generator. Hence, here we review
the StarGAN-VC generator network architecture. As shown in
Figure 2(a), StarGAN-VC incorporates conditional information
into the generator in a channel-wise manner, i.e., first creates
the one-hot vector indicating the domain code, subsequently
expands the one-hot vector to the feature map size, and finallyconcatenates it to the feature map. Concatenated features areconvoluted together and propagated to the next layer.

(1)where D is a target conditional discriminator [33]. By maximizing this loss, D attempts to learn the best decision boundary
between the converted and real acoustic features conditioned on
the target domain codes (c and c0 ). In contrast, G attempts torender the converted feature indistinguishable from real acoustic features conditioned on c0 by minimizing this loss.

680

3. StarGAN-VC2

problem, we introduce a modulation-based conditional method,which can directly modulate features in a domain-dependent
manner. In particular, we introduce a conditional instance normalization (CIN) [37], which was originally proposed in computer vision for style transfer. As shown in Figure 2(b), given
the feature f , CIN conducts the following procedure:f − µ(f )CIN(f ; c0 ) = γc0+ βc0 ,
(10)σ(f )3.1. Rethinking conditional method in training objectives
We first rethink a conditional method in training objectives. As
described in Section 2.1, StarGAN-VC uses two conditional
methods to make the converted feature belonging to the target
domain: the classification loss (Equations 2 and 3) and the target conditional adversarial loss (Equation 1). We illustrate theirtraining strategies in Figure 1(a) and (b), respectively.
In the classification loss (Figure 1(a)), via Equation 2, thedecision boundary (black line) is learned among real-data domains (e.g., between “Real A” and “Real B” in Figure 1(a)).
For this decision boundary, G attempts to generate easily “classifiable” data via Equation 3. This means that G prefers to generate data that are far from the decision boundary even when thereal data exist around the decision boundary. As discussed elsewhere [31, 34, 35], this prevents G from covering the whole real
data distribution. In VC, this may result in a partial conversion.
Meanwhile, the target conditional adversarial loss (Figure 1(b)) encourages the generated data close to the real data
conditioned on the target domain code. As discussed in the previous study [34], this objective prevents G from leaning towards
generating only classifiable data. However, a possible difficultyis that this loss needs to simultaneously handle diverse data,
including hard negative samples (e.g., conversion between thesame speaker A → A in Figure 1(b)) and easy negative samples(e.g, conversion between completely different speakers B → A
in Figure 1(b)). This unfair condition makes it difficult to bringall the converted data close to real target data.
To solve this problem, we develop a source-and-target conditional adversarial loss defined aswhere µ(f ) and σ(f ) are the average and standard deviationof f that are calculated over for each instance. γc0 and βc0 are
domain-specific scale and bias parameters that allow the modulation to be transformed in a domain-specific manner. These
parameters are learnable and optimized through training.
In the above, we explain the case when the generator is conditioned on the target domain code c0 . When using a sourceand-target conditional generator introduced in Equation 9, we
replace γc0 and βc0 with γc,c0 and βc,c0 , respectively, which are
selected depending on both the source c and target c0 .

4. Experiments
4.1. Experimental conditions
Dataset: We evaluated our method on the multi-speaker VC
task using VCC 2018 [28], which contains recordings of professional US English speakers. Following the StarGAN-VC study[27], we selected a subset of speakers as covering all inter- andintra-gender conversions: VCC2SF1, VCC2SF2, VCC2SM1,
and VCC2SM2, where F and M indicate female and malespeakers, respectively. Thus, the number of domains N is set
to 4. Our goal is to learn 4 × 3 = 12 different source-andtarget mappings in a single model. Each speaker has sets of 81
and 35 sentences for training and evaluation, respectively. The
recordings were downsampled to 22.05 kHz for this challenge.
We extracted 34 Mel-cepstral coefficients (MCEPs), logarithmic fundamental frequency (log F0 ), and aperiodicities (APs)every 5 ms by using the WORLD analyzer [41].
Conversion process: In these experiments, we focused on
analyzing the performance in MCEP conversion. Hence, we applied the proposed method only to MCEP conversion, and forthe other parts, we used typical methods, i.e., converted log F0using logarithm Gaussian normalized transformation [42], directly used APs, and synthesized speech using the WORLD
vocoder [41]. To examine the pure effect of the proposed methods, we did not use any postfilter [43, 44, 45] or powerful
vocoder such as the WaveNet vocoder [46, 47]. Incorporatingthem remains possible future work.
Implementation details: We designed the network architectures on the basis of CycleGAN-VC2 [21], i.e., we used a2-1-2D CNN in G and a 2D CNN in D. We formulate D using the projection discriminator [34]. In the pre-experiment, wefound that skip connections in residual blocks [48] result in partial conversion. Thus, we removed them in G. The details of the
network architectures are given in Figure 3. For a GAN objective, we used a least squares GAN [49]. We conducted speakerwise normalization for a pre-process. We trained the networks
using the Adam optimizer [50] with a batch size of 8, in which
we used a randomly cropped segment (128 frames) as one instance. The number of iterations was set to 3 × 105 , learning
rates for G and D were set to 0.0002 and 0.0001, respectively,
and the momentum term was set to 0.5. We set λcyc = 10,
λid = 5, and λcls = 1. We used Lid only for the first 104
iterations to stabilize the training at the beginning.

Lst-adv = E(x,c)∼P (x,c),c0 ∼P (c0 ) [log D(x, c0 , c)]+ E(x,c)∼P (x,c),c0 ∼P (c0 ) [log D(G(x, c, c0 ), c, c0 )], (9)where c0 ∼ P (c0 ) is randomly sampled independently of realdata. Differently from Equation 1, both G and D are conditioned on the source code c0 in addition to the target code c.
We call such G and D a source-and-target conditional generator and discriminator, respectively. As shown in Figure 1(c), by
using both source and target domain codes as conditional information, this loss encourages all the converted data to be close
to real data in both source-wise and target-wise manners. This
resolves the unfair training condition in the target conditional
adversarial loss (Figure 1(b)) and allows all the source domain
data to be converted into the target domain data.
One possible disadvantage of the source-and-target conditional generator is that this requires the availability of the sourcecode in inference, which is not required in the conventional
StarGAN-VC. However, note that speaker recognition has been
actively studied, and this problem can be alleviated by using it
as a pre-process.
3.2. Rethinking conditional method in networks
As indicated by previous studies on VC postfilters (e.g., global
variance [2] and modulation spectrum [36] postfilters), accurate modulation translation is important to achieve high-quality
VC. Particularly, to achieve multi-domain VC only using a single generator, a framework must be incorporated that can conduct diverse domain-specific modulations effectively. For thischallenge, a channel-wise conditional method (Figure 2(a)) is
not effective because the concatenated conditional information
can be additively used in a convolutional procedure but cannot
be multicatively used to modulate features. To alleviate this681

1D→2D Upsample (2D)

k1x5
c1024
s1x1

c, c'

Inner
product

Real/Fake

k3x3
c1024
s2x2

GSP FC

IN

k3x3
c512
s2x2

GLU

GLU

k3x3
c256
s2x2

Conv

Input

h35 k3x3
w128 c128
c1
s1x1

Conv

Conv

OutputPS

k5x15 h35
c35 wTs1x1 c1

GLU

Conv

1x1Conv

9 Blocks (1D)

k5x5
c512
s1x1

Discriminator(Projection)

Downsample (2D) 2D→1D Codes c, c'

ReshapeCIN

k1x1 h9k5x5c2304 wT/4 c1024
s1x1 c256 s1x1

GLU

IN

k1x5
c512
s1x1

Conv

1x1Conv

h1 k1x1
wT/4 c256
c2304 s1x1

Reshape

Conv

IN

Input

k5x5
c512
s2x2

GLU

k5x5
c256
s2x2

GLU

k5x15
c128
s1x1

Conv

Generator(2-1-2D CNN)

h35
wT
c1

Embed

Downsample (2D) Codes Projection

Figure 3: Generator and discriminator architectures. In each layer, h, w, c, k, and s denote height, width, number of channels, kernel
size, and stride, respectively. IN, GLU, PS, and GSP indicate instance normalization [38], gated linear unit [25], pixel shuffler [39],and global sum pooling, respectively. G is fully convolutional [40]. This allows an arbitrary length T to be input in inference.
All

Objective
Lcls
Lt-adv
Lt-adv + Lcls (StarGAN-VC)
Lst-adv (StarGAN-VC2)

MCD [dB]
7.73 ± .07
7.21 ± .16
7.11 ± .10
6.90 ± .07

MSD [dB]
1.96 ± .03
2.87 ± .51
2.41 ± .13
1.89 ± .03

MCD [dB]
6.90 ± .08
6.90 ± .07

Inter-gender
StarGAN-VC2

StarGAN-VC

Figure 4: MOS for naturalness with 95% confidence intervals
All

Table 2: Comparison of MCD and MSD among models using
different conditional methods in G networks. We fix the conditional method in the training objective as Lst-adv .
G network
Channel-wise (StarGAN-VC)
Modulation-based (StarGAN-VC2)

Intra-gender

MOS

Table 1: Comparison of MCD and MSD among models using
different conditional methods in training objectives. We fix theconditional method in G network as modulation-based.

StarGAN-VC2

Intra-gender

Fair

Inter-gender

StarGAN-VC

Figure 5: Average preference scores (%) on speaker similarity
sentences (4 × 3 source-target combinations × 2 sentences). To
eliminate bias in the order of stimuli, we presented all pairs inboth orders (AB and BA). For each sentence pair, the listeners
were asked to select their preferred one (“A” or “B”) or “Fair.”
12 well-educated English speakers participated in the tests.
Figures 4 and 5 show the MOS for naturalness and the preference scores for speaker similarity, respectively. We summarized the results on the basis of three categories: all conversion,inter-gender conversion, and intra-gender conversion. These results empirically demonstrate that StarGAN-VC2 outperforms
StarGAN-VC in terms of both naturalness and speaker similarity for every category.

MSD [dB]
2.55 ± .20
1.89 ± .03

4.2. Objective evaluation
We conducted an objective evaluation to validate the advantages of the proposed conditional methods over other conditional methods. The same as the previous study [21], we used
two evaluation metrics for comprehensive analysis: the Melcepstral distortion (MCD), which measures the global structuraldifferences by calculating the distance between the target andconverted MCEPs, and the modulation spectra distance (MSD),
which measures the local structural differences by computing
the distance between the target and converted logarithmic modulation spectra of MCEPs. For both metrics, a smaller value
indicates that the target and converted features are more similar.
We conducted comparative studies in two aspects: training
objectives and network architectures, which are listed in Tables
1 and 2, respectively. We have calculated the scores averaged
over three models trained with different initializations to eliminate the effect of initialization. In Table 1, the proposed sourceand-target conditional loss Lst-adv outperforms the other losses
in terms of both the MCD and MSD. This indicates that the proposed loss is useful for improving the feature quality in terms ofboth the global and local structure measures. In Table 2, the proposed modulation-based conditional method outperforms theconventional channel-wise conditional method in terms of the
MSD. This indicates that the proposed architecture is particularly useful for improving the local structure. Through theseexperiments, we empirically confirm that the proposed conditional methods in objectives and networks effectively bring theconverted acoustic feature sequence close to the target one.

5. Conclusions
To advance the research on multi-domain non-parallel VC, we
have rethought conditional methods in StarGAN-VC in two aspects: training objectives and network architectures. We developed a source-and-target conditional adversarial loss for theformer and a modulation-based conditional method for the latterand have proposed StarGAN-VC2 incorporating them. The empirical studies on non-parallel multi-speaker VC demonstratethat StarGAN-VC2 outperforms StarGAN-VC in both objective and subjective measures. StarGAN-VC2 is a general model
for multi-domain VC and is not limited to multi-speaker VC.
Adapting it to other tasks (e.g., multi-emotion VC and multipronunciation VC) remains a promising future direction.

6. Acknowledgements
This work was supported by JSPS KAKENHI 17H01763.

7. References[1] Y. Stylianou, O. Cappé, and E. Moulines, “Continuous probabilistic transform for voice conversion,” IEEE Trans. Speech and Audio Process., vol. 6, no. 2, pp. 131–142, 1998.
[2] T. Toda, A. W. Black, and K. Tokuda, “Voice conversion basedon maximum-likelihood estimation of spectral parameter trajectory,” IEEE Trans. Audio Speech Lang. Process., vol. 15, no. 8,
pp. 2222–2235, 2007.
[3] L.-H. Chen, Z.-H. Ling, L.-J. Liu, and L.-R. Dai, “Voice conversion using deep neural networks with layer-wise generative training,” IEEE/ACM Trans. Audio Speech Lang. Process., vol. 22,
no. 12, pp. 1859–1872, 2014.
[4] S. Desai, A. W. Black, B. Yegnanarayana, and K. Prahallad,“Spectral mapping using artificial neural networks for voice conversion,” IEEE Trans. Audio Speech Lang. Process., vol. 18, no. 5,
pp. 954–964, 2010.

4.3. Subjective evaluation
We conducted listening tests to analyze the performance compared with StarGAN-VC [27], which is a state-of-the-art multidomain non-parallel VC. To measure naturalness, we conducteda mean opinion score (MOS) test (5: excellent to 1: bad), in
which we included the analysis-synthesized speech (which is
the upper limit of the converted speech) as a reference (MOS:4.2). For each model, we generated 36 sentences (4 × 3 sourcetarget combinations × 3 sentences). We conducted an XAB
test to measure speaker similarity. Here, “X” was target speechand “A” and “B” were speech converted by StarGAN-VC andStarGAN-VC2, respectively. For each model, we generated 24682

[5] T. Nakashika, T. Takiguchi, and Y. Ariki, “High-order sequencemodeling using speaker-dependent recurrent temporal restricted
Boltzmann machines for voice conversion,” in Proc. Interspeech,
2014, pp. 2278–2282.
[6] L. Sun, S. Kang, K. Li, and H. Meng, “Voice conversion using
deep bidirectional long short-term memory based recurrent neuralnetworks,” in Proc. ICASSP, 2015, pp. 4869–4873.
[7] T. Kaneko, H. Kameoka, K. Hiramatsu, and K. Kashino,“Sequence-to-sequence voice conversion with similarity metriclearned using generative adversarial networks,” in Proc. Interspeech, 2017, pp. 1283–1287.
[8] K. Tanaka, H. Kameoka, T. Kaneko, and N. Hojo, “AttS2S-VC:
Sequence-to-sequence voice conversion with attention and context preservation mechanisms,” in Proc. ICASSP, 2019, pp. 6805–
6809.
[9] H. Kameoka, K. Tanaka, T. Kaneko, and N. Hojo, “ConvS2S-VC:
Fully convolutional sequence-to-sequence voice conversion,” inarXiv preprint arXiv:1811.01609, Nov. 2018.
[10] A. Mouchtaris, J. Van der Spiegel, and P. Mueller, “Nonparallel
training for voice conversion based on a parameter adaptation approach,” IEEE Trans. Audio Speech Lang. Process., vol. 14, no. 3,
pp. 952–963, 2006.
[11] C.-H. Lee and C.-H. Wu, “MAP-based adaptation for speech conversion using adaptation data selection and non-parallel training,”
in Proc. ICSLP, 2006, pp. 2254–2257.
[12] T. Toda, Y. Ohtani, and K. Shikano, “Eigenvoice conversion basedon Gaussian mixture model,” in Proc. Interspeech, 2006, pp.
2446–2449.
[13] D. Saito, K. Yamamoto, N. Minematsu, and K. Hirose, “One-tomany voice conversion based on tensor representation of speakerspace,” in Proc. Interspeech, 2011, pp. 653–656.
[14] F.-L. Xie, F. K. Soong, and H. Li, “A KL divergence and DNNbased approach to voice conversion without parallel training sentences,” in Proc. Interspeech, 2016, pp. 287–291.
[15] Y. Saito, Y. Ijima, K. Nishida, and S. Takamichi, “Non-parallelvoice conversion using variational autoencoders conditioned byphonetic posteriorgrams and d-vectors,” in Proc. ICASSP, 2018,
pp. 5274–5278.
[16] T. Nakashika, T. Takiguchi, and Y. Minami, “Non-parallel training
in voice conversion using an adaptive restricted Boltzmann machine,” IEEE/ACM Trans. Audio Speech Lang. Process., vol. 24,
no. 11, pp. 2032–2045, 2016.
[17] C.-C. Hsu, H.-T. Hwang, Y.-C. Wu, Y. Tsao, and H.-M. Wang,“Voice conversion from unaligned corpora using variational autoencoding Wasserstein generative adversarial networks,” in Proc.
Interspeech, 2017, pp. 3364–3368.
[18] H. Kameoka, T. Kaneko, K. Tanaka, and N. Hojo, “ACVAE-VC:
Non-parallel voice conversion with auxiliary classifier variational
autoencoder,” IEEE/ACM Trans. Audio Speech Lang. Process.,
vol. 27, no. 9, pp. 1432–1443, 2019.
[19] T. Kaneko and H. Kameoka, “Parallel-data-free voice conversion
using cycle-consistent adversarial networks,” in arXiv preprint
arXiv:1711.11293, Nov. 2017.
[20] ——, “CycleGAN-VC: Non-parallel voice conversion using
cycle-consistent adversarial networks,” in Proc. EUSIPCO, 2018,
pp. 2114–2118.
[21] T. Kaneko, H. Kameoka, K. Tanaka, and N. Hojo, “CycleGANVC2: Improved cyclegan-based non-parallel voice conversion,” inProc. ICASSP, 2019, pp. 6820–6824.
[22] J.-Y. Zhu, T. Park, P. Isola, and A. A. Efros, “Unpaired image-toimage translation using cycle-consistent adversarial networks,” inProc. ICCV, 2017, pp. 2223–2232.
[23] Z. Yi, H. Zhang, P. Tan, and M. Gong, “DualGAN: Unsupervised
dual learning for image-to-image translation,” in Proc. ICCV,2017, pp. 2849–2857.
[24] T. Kim, M. Cha, H. Kim, J. K. Lee, and J. Kim, “Learning todiscover cross-domain relations with generative adversarial networks,” in Proc. ICML, 2017, pp. 1857–1865.
[25] Y. N. Dauphin, A. Fan, M. Auli, and D. Grangier, “Language
modeling with gated convolutional networks,” in Proc. ICML,
2017, pp. 933–941.
[26] Y. Taigman, A. Polyak, and L. Wolf, “Unsupervised cross-domain
image generation,” in Proc. ICLR, 2017.

[27] H. Kameoka, T. Kaneko, K. Tanaka, and N. Hojo, “StarGAN-VC:
Non-parallel many-to-many voice conversion using star generative adversarial networks,” in Proc. SLT, 2018, pp. 266–273.
[28] J. Lorenzo-Trueba, J. Yamagishi, T. Toda, D. Saito, F. Villavicencio, T. Kinnunen, and Z. Ling, “The voice conversion challenge
2018: Promoting development of parallel and nonparallel methods,” in Proc. Speaker Odyssey, 2018, pp. 195–202.
[29] Y. Choi, M. Choi, M. Kim, J.-W. Ha, S. Kim, and J. Choo, “StarGAN: Unified generative adversarial networks for multi-domain
image-to-image translation,” in Proc. CVPR, 2018, pp. 8789–
8797.
[30] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. WardeFarley, S. Ozair, A. Courville, and Y. Bengio, “Generative adversarial nets,” in Proc. NIPS, 2014, pp. 2672–2680.
[31] A. Odena, C. Olah, and J. Shlens, “Conditional image synthesis
with auxiliary classifier GANs,” in Proc. ICML, 2017, pp. 2642–
2651.
[32] T. Zhou, P. Krähenbühl, M. Aubry, Q. Huang, and A. A. Efros,“Learning dense correspondence via 3D-guided cycle consistency,” in Proc. CVPR, 2016, pp. 117–126.
[33] M. Mirza and S. Osindero, “Conditional generative adversarial
nets,” arXiv preprint arXiv:1411.1784, Nov. 2014.
[34] T. Miyato and M. Koyama, “cGANs with projection discriminator,” in Proc. ICLR, 2018.
[35] T. Kaneko, Y. Ushiku, and T. Harada, “Class-distinct and classmutual image generation with GANs,” in Proc. BMVC, 2019.
[36] S. Takamichi, T. Toda, G. Neubig, S. Sakti, and S. Nakamura,“A postfilter to modify the modulation spectrum in HMM-based
speech synthesis,” in Proc. ICASSP, 2014, pp. 290–294.
[37] V. Dumoulin, J. Shlens, and M. Kudlur, “A learned representation
for artistic style,” in Proc. ICLR, 2017.
[38] D. Ulyanov, A. Vedaldi, and V. Lempitsky, “Instance normalization: The missing ingredient for fast stylization,” in arXiv preprint
arXiv:1607.08022, July 2016.
[39] W. Shi, J. Caballero, F. Huszár, J. Totz, A. P. Aitken, R. Bishop,
D. Rueckert, and Z. Wang, “Real-time single image and video
super-resolution using an efficient sub-pixel convolutional neuralnetwork,” in Proc. CVPR, 2016, pp. 1874–1883.
[40] J. Long, E. Shelhamer, and T. Darrell, “Fully convolutional networks for semantic segmentation,” in Proc. CVPR, 2015, pp.
3431–3440.
[41] M. Morise, F. Yokomori, and K. Ozawa, “WORLD: A vocoderbased high-quality speech synthesis system for real-time applications,” IEICE Trans. Inf. Syst., vol. 99, no. 7, pp. 1877–1884,
2016.
[42] K. Liu, J. Zhang, and Y. Yan, “High quality voice conversion through phoneme-based linear mapping functions withSTRAIGHT for Mandarin,” in Proc. FSKD, 2007, pp. 410–414.
[43] T. Kaneko, H. Kameoka, N. Hojo, Y. Ijima, K. Hiramatsu, and
K. Kashino, “Generative adversarial network-based postfilter for
statistical parametric speech synthesis,” in Proc. ICASSP, 2017,
pp. 4910–4914.
[44] T. Kaneko, S. Takaki, H. Kameoka, and J. Yamagishi, “Generative
adversarial network-based postfilter for STFT spectrograms,” inProc. Interspeech, 2017, pp. 3389–3393.
[45] K. Tanaka, T. Kaneko, N. Hojo, and H. Kameoka, “Synthetic-tonatural speech waveform conversion using cycle-consistent adversarial networks,” in Proc. SLT, 2018, pp. 632–639.
[46] A. van den Oord, S. Dieleman, H. Zen, K. Simonyan, O. Vinyals,
A. Graves, N. Kalchbrenner, A. Senior, and K. Kavukcuoglu,“WaveNet: A generative model for raw audio,” in arXiv preprint
arXiv:1609.03499, Sept. 2016.
[47] A. Tamamori, T. Hayashi, K. Kobayashi, K. Takeda, and T. Toda,“Speaker-dependent WaveNet vocoder,” in Proc. Interspeech,
2017, pp. 1118–1122.
[48] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning forimage recognition,” in Proc. CVPR, 2016, pp. 770–778.
[49] X. Mao, Q. Li, H. Xie, R. Y. Lau, Z. Wang, and S. P. Smolley,“Least squares generative adversarial networks,” in Proc. ICCV,2017, pp. 2794–2802.
[50] D. Kingma and J. Ba, “Adam: A method for stochastic optimization,” in Proc. ICLR, 2015.

683