INTERSPEECH 2021
30 August – 3 September, 2021, Brno, Czechia

StarGAN-VC+ASR: StarGAN-based Non-Parallel Voice Conversion
Regularized by Automatic Speech Recognition
Shoki Sakamoto1 , Akira Taniguchi1 , Tadahiro Taniguchi1 , Hirokazu Kameoka2
1 College of Information Science and Engineering, Ritsumeikan University, Japan
2 NTT Communication Science Laboratories, NTT Corporation, Japan{sakamoto.shoki, a.taniguchi, taniguchi}@em.ci.ritsumei.ac.jp,
kameoka.hirokazu@lab.ntt.co.jp

Abstract

VC systems may beneﬁt from being used in combination
with automatic speech recognition (ASR) systems owing to therecent advances in ASR performance. One successful example involves the use of phonetic posteriorgrams (PPGs) [14–16].
Similar to the PPG-based approach, the method proposed in thispaper improves the ability of StarGAN-VC to preserve the linguistic content of input speech by using ASR to assist modeltraining. Speciﬁcally, we propose using the ASR results applied
to each training utterance to construct phoneme-dependent regularization terms for the latent vectors generated from the encoder in StarGAN-VC. These penalty terms are derived basedon a Gaussian mixture model (GMM); therefore, the latentvectors become more phonetically distinct, resulting in more
intelligible speech. We call the proposed method StarGANVC+ASR.
The main contributions of this study are as follows. We
propose a non-parallel many-to-many VC method (StarGANVC+ASR) that combines StarGAN-VC and ASR. We experimentally demonstrate that StarGAN-VC+ASR can produce
more intelligible speech than the original StarGAN-VC.
The remainder of this paper is organized as follows. Section 2 brieﬂy introduces StarGAN-VC, which is the basis of the
proposed method. Section 3 describes the proposed method,
StarGAN-VC+ASR. Section 4 describes an experiment that
demonstrates the performance of StarGAN-VC+ASR. Finally,
Section 5 concludes the paper.

Preserving the linguistic content of input speech is essential
during voice conversion (VC). The star generative adversarial
network-based VC method (StarGAN-VC) is a recently developed method that allows non-parallel many-to-many VC. Although this method is powerful, it can fail to preserve the linguistic content of input speech when the number of available
training samples is extremely small. To overcome this problem, we propose the use of automatic speech recognition toassist model training, to improve StarGAN-VC, especially in
low-resource scenarios. Experimental results show that using
our proposed method, StarGAN-VC can retain more linguistic
information than vanilla StarGAN-VC.
Index Terms: voice conversion, speech recognition, linguistic
information

1. Introduction
Non-parallel many-to-many voice conversion (VC) is a powerful and useful framework for building VC systems [1–3]. VC
is the task of converting the para-linguistic characteristics contained in a given speech signal without changing the linguistic
content (transcription) [4]. Conventionally, most VC systems
have been developed based on a parallel dataset, consisting ofpairs of utterances in the source and target domains reading
the same sentences [4–6]. However, preparing a large-scaleparallel dataset is very expensive, and this poses a potential
limitation to both the ﬂexibility and performance of VC systems. Recently, non-parallel VC methods have been proposed
and have attracted much attention [7–9]. StarGAN-VC [2] isa non-parallel many-to-many VC method based on a variantof the generative adversarial network (GAN) [10] called StarGAN [11]. This method is particularly attractive in that it cangenerate converted speech signals fast enough to allow real-time
implementation and can generate reasonably realistic sounding
speech even with limited training samples.
In StarGAN-VC, the use of cycle consistency and identity
mapping losses along with adversarial and domain classiﬁcation losses encourages the generator to preserve the linguistic
content of input speech. However, this function can still failwhen the number of available training utterances is extremely
small. In such cases, the linguistic content of converted speech
can often be corrupted, resulting in inaudible speech. Althoughnon-parallel data are easier to prepare than parallel data, they
can be expensive, as in applications such as emotional voice
conversion [12, 13]. Hence, there is a need for a method thatcan overcome this problem that can arise under limited data resources. A possible solution would be to design StarGAN-VC
such that it can make explicit use of linguistic information formodel training.

Copyright © 2021 ISCA

2. Preliminaries: StarGAN-VC
StarGAN-VC is based on a model consisting of a generator,
G, real/fake discriminator, D, and domain classiﬁer, C. Leto ∈ RQ×T be an acoustic feature sequence, where Q is thefeature dimension, and T is the length of the sequence. Letc ∈ {1, . . . , N} be a target domain code, where N is the number of domains. G is a neural network (NN) that takes anacoustic feature sequence, o, in an arbitrary domain and thetarget domain code, c, as inputs and generates an acoustic feature sequence, ô′ = G(o, c). The network is trained such that
G(o, c) becomes indistinguishable from the real speech feature
sequence in the domain c. This is made possible by using D
and C to train G, whose roles are to discriminate fake samples,G(o, c), from real samples, o′ , and identify the class to whichG(o, c) is likely to belong. Figure 1 shows the overview of
StarGAN-VC training. D is designed to produce a sequenceof probabilities D(o′ , c), each of which indicates the likelihoodthat a different segment of input o′ is real, whereas C is designed to produce a sequence of class probabilities pC (c|o′ ),each of which indicates the likelihood that a different segmentof o′ belongs to a particular class.
For training, StarGAN-VC uses adversarial, cycle consis-1359

http://dx.doi.org/10.21437/Interspeech.2021-492

Reallatent feature

𝐺𝐷

𝑦

𝐺𝐸

𝐶

Fake

tency, identity mapping, and domain classiﬁcation losses.
Adversarial Loss: The adversarial losses are deﬁned for discriminator D and generator G as

k 3×5
c 128
s 1×1

k 4×8
c 32
s 2×2

k 4×8
c 64
s 2×2

h1
w 128
c4

(1)

h9
w 128
c4

h9
w 128
c4

h 18
w 128
c4

(3)

G
Lcls
(G) = −Eo∼p(o),c∼p(c) [log pC (c|G(o, c))] .

(4)

h 36
wT
c1

h 36
w 128
c4

DecoderD
ID (D) =Ladv(D),C
IC (C) =Lcls (C),

(2)

[
]
C
Lcls
(C) = −Ec∼p(c),o′ ∼p(o′ |c) log pC (c|o′ ) ,

k 3×9
c1
s 1×1

Output: 𝑜′

k 9×5
c 64
s 9×1

Deconv

8
T
1

DeconvBatch norm
GLU

h
w
c

Latentfeature 𝑦

Conv
Batch norm
GLU

k 9×5
c 5
s 9×1

G
G
(G)
IG (G) =Ladv(G) + λcls Lcls
+ λcyc Lcyc (G) + λid Lid (G),

where o′ ∼ p(o′ |c) denotes a training sample of acoustic feature
sequences of real speech belonging to domain c, and o ∼ p(o)denotes an arbitrary domain. Equation 1 takes a small value
when D correctly identiﬁes G(o, c) as fake and o′ as real. By
contrast, Equation 2 takes a small value when D misclassiﬁesG(o, c) as real. Thus, the subgoals of D and G are to minimize
Equations 1 and 2, respectively.
Domain Classiﬁcation Loss: The domain classiﬁcation losses
for classiﬁer C and generator G are deﬁned as

(7)
(8)
(9)

λcls ≥ 0, λcyc ≥ 0, and λid ≥ 0 are regularization parametersthat weigh the importance of the domain classiﬁcation, cycleconsistency, and identity-mapping losses.
Regarding network architectures, we assume an encoderdecoder type architecture for G, as shown in Figure 2. The encoder is responsible for extracting a speaker-independent latentfeature sequence y, which is expected to correspond to the linguistic content of the input utterance. The decoder, on the other
hand, is responsible for reconstructing an acoustic feature sequence using both target domain codes c and y.

3. StarGAN-VC+ASR
Here, we present the idea of StarGAN-VC+ASR. Figure 3 shows
the overview of the StarGAN-VC+ASR training．

Equations 3 and 4 have small values when o′ ∼ p(o′ |c) andG(o, c) are correctly classiﬁed as belonging to domain c by C.
Thus, the subgoals of C and G are to minimize Equations 3 and4, respectively.
Cycle-Consistency Loss: The adversarial and classiﬁcation
losses encourage G(o, c) to become realistic and classiﬁable,
respectively. However, using these losses alone does not guarantee that G preserves the linguistic content of the input speech.
To promote content-preserving conversion, the cycle consistency loss

3.1. GMM-based phoneme model in latent space
G has an encoder-decoder structure similar to variational autoencoders (VAEs) [17]. The encoder part in G is responsible
for extracting a speaker-independent latent feature sequence,y, corresponding to the linguistic content of the input. Now,
to associate each element of y with the phoneme in the corresponding frame of the input speech, we consider constructing
a prior distribution over y based on the result of applying ASR
to that speech in advance. Let P = {1, 2, . . . , K} denote a setof phoneme indices, where K denotes the number of phonemes.
In the ASR step, the input is the acoustic feature sequence ofddth utterance, denoted as xd = x1:T
= (x1d , x2d , . . . , xTdd ), whered
Td and t denote the frame length and index, respectively. The
ASR output is assumed to be a sequence of phoneme indicesd ) of the same length. The encoder part inẑd = ẑd1:Td = ASR(x1:Td
G takes the acoustic feature sequence od of the same utterance
as the input and produces a latent feature sequence yd = yd1:Td =(yd1 , yd2 , . . . , ydTd ). To ensure that the latent features corresponding to the same phoneme are close to each other, we assign asingle Gaussian distribution to each phoneme index and model

Lcyc (G)(5)

is used, where o ∼ p(o|c′ ) denotes a training sample of acoustic
feature sequences of real speech belonging to the source domainc′ , and ρ is a positive constant.
Identity-Mapping Loss: To ensure that G keeps its input, o,unchanged if o already belongs to the target domain, the identity
mapping loss is deﬁned as[]
Lid (G) = Ec′ ∼p(c),o∼p(o|c′ ) ||G(o, c′ ) − o||ρ .

k 3×5
c 64
s 1×1

The full loss function is given as

[
]
D
Ladv
(D) = − Ec∼p(c),o′ ∼p(o′ |c) log D(o′ , c)[]
= Ec′ ∼p(c),o∼p(o|c′ ),c∼p(c) ||G(G(o, c), c′ ) − o||ρ ,

k 4×8
c 128
s 2×2

Figure 2: The network architecture of generator G. In input
and latent features, “h,” “w,” and “c” represent height, width,
and number of channels, respectively. In addition, “k,” “c, ”
and “s” denote kernel size, number of channels, and stride size,
respectively. “Conv,” “Batch norm,” “GLU,” “Deconv,” “Sigmoid,” “Softmax,” and “Product” denote convolution, batchnormalization, gated linear unit, transposed convolution, sigmoid, softmax, and product pooling layers, respectively.

Figure 1: Concept of StarGAN-VC training. GE and GD represent the encoder and decoder in G, respectively.

G
Ladv(G) = − Eo∼p(o),c∼p(c) [log D(G(o, c), c)] ,

k 4×8
c 64
s 2×2

Encoder

DomainClassifier− Eo∼p(o),c∼p(c) [log(1 − D(G(o, c), c))] ,

k 3×9
c 32
s 1×1

Input: 𝑜attribute 𝑐

𝐺(𝑜, 𝑐)

𝐷

h 36
w T
c 1

Attribute 𝑐

Real

𝐺𝐷

attribute 𝑐′

𝑜

𝐺𝐸

𝑦

𝑜′

1, … , 𝑁𝑜

Generator

Real/Fake

Training example
of attribute 𝑐′

attribute 𝑐

Training example
of attribute 𝑐

Real/Fakediscriminator

(6)

1360

Phonemetranscription

Training part
Featureextraction

𝑥

a

r

a

y

u

r

u

ASR module

Acoustic feature
(ASR)

𝑧

One-hot vector

GMM
Parameter estimation(supervised)

Regularization

Inference part
Featureextraction𝑜Latent feature order

Encoder in 𝐺

𝐺𝐸

Acoustic feature(StarGAN-VC)

Input speech

Decoder in 𝐺

Latent feature 𝑦𝐺𝐷

𝑮(𝒐, 𝒄)Acoustic feature(Generated)

Frame length

Figure 3: Overview of StarGAN-VC+ASR training

4. Experimentsa prior distribution of yd according to ẑd . Given yd and ẑd , themaximum likelihood estimates of the mean and covariance ofeach Gaussian can be obtained as

4.1. Experimental setup

D

µp =

1
ytd ,
D #(T (ẑd ))∑d=1p
d=1 t∈Tp (ẑd )

Σp =

)(
)T
(
D
1
d
d
µ
y
−
µ
,
y
−
p
p
∑
∑
t
t
d
∑D
d=1 #(Td (ẑ )) d=1 t∈Tp (ẑd )∑ ∑Dataset: We evaluated our method on a multi-speaker VC taskusing an ATR digital sound database [18], which consists ofrecordings of ﬁve female and ﬁve male Japanese speakers. The
spoken sentences were recorded as waveforms and were sampled at 20 kHz. We used a subset of speakers for training andevaluation. We selected two female speakers, “FKN” and “FTK,”
and two male speakers, “MMY” and “MTK”, resulting in twelve
different combinations of source and target speakers. We selected 24 and 20 sentences for training and evaluation, respectively. Therefore, there were 12 × 20 = 240 test signals in total.

(10)

(11)
where p and D denote the phoneme index in P and the numberof training utterances, respectively. Tp (ẑd ) denotes the set consisting of t that satisﬁes ẑtd = p, and #(·) denotes the number of
elements in a set. In the following, we use µP and ΣP to denotethe sets µP = {µ1 , µ2 , . . . , µ#(P) } and ΣP = {Σ1 , Σ2 , . . . , Σ#(P) },respectively.

Conversion process: 36 Mel-cepstral coefﬁcients (MCEPs),
logarithmic fundamental frequency (log F0 ), and aperiodicities
were extracted every 5 ms using the WORLD analyzer [19](D4C edition [20]). In these experiments, we applied the baseline and proposed methods only to MCEP conversion. The F0
contours were converted by logarithmic Gaussian normalization [21]. The aperiodicities were used directly without modiﬁcation. We trained the networks using the Adam optimizer [22]with a batch size of 8. The number of iterations was set to2×103 , the learning rates for G and D were set to 0.001, and themomentum term was set to 0.5. We set λcls = 1.0，λcyc = 1.0，
λid = 1.0 following the original paper [2]. We set β = 0.01
empirically.

3.2. Phoneme-dependent regularization loss
The training process of StarGAN-VC+ASR consists of two stages. The ﬁrst stage corresponds to the original StarGAN-VC
training described in Section 2. In the second stage, G is updated by using µP and ΣP obtained by Equations 10 and 11 forregularization. Speciﬁcally, we assume)(p(ytd ) = N ytd | µẑtd , Σẑtd ,(12)
as the prior distribution of yd , and deﬁne
D

LASR (G) = − ∑ ∑ log p(ytd )

(13)

d=1 t

as the regularization term for the second stage of the training.
Although very simple, this regularization is expected to bringeach element of yd corresponding to the same phoneme closer
to each other.
To summarize, the loss function of the generator in StraGANVC+ASR IG (G) is denoted as
GG
IG (G) =Ladv(G) + λcls Lcls
(G)
+ λcyc Lcyc (G) + λid Lid (G)+ β LASR (G),(14)

where β ≥ 0 is the hyperparameter that weighs the importanceof the regularization term. In addition to the above, the loss
functions for D and C are the same as those in Equations 8 and
9, respectively.

Implementation details: We used a dictation-kit (version 4.5)1
as the trained ASR system. This package contains executablesof Julius (deep neural network (DNN) version) [23], Japaneseacoustic models (AM), and Japanese language models (LM).
Julius is a high-performance, small-footprint, large-vocabularycontinuous speech recognition (LVCSR) decoder software usedby speech-related researchers and developers. The AM is aspeaker-independent triphone DNN-hidden Markov model (HMM) trained using the JNAS and CSJ corpora. It also has regression tree classes that are required for speaker adaptation
by HTK2 . The LMs are 60k-word N-gram language models
trained using the BCCWJ corpus. Moreover, we set the dictionary data, which consisted of only utterances of training data,
to improve the accuracy of speech recognition.

1 https://osdn.net/dl/julius/dictation-kit-4.5.zip
2 Hidden Markov Model Toolkit (HTK), http://htk.eng.cam.ac.uk/

1361

CER

Proposed

2nd sound

1st sound

*

0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0.0

0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0.0

Baseline

0

MOS

10

20

30

40

18.8 6.85060Preference score (%)

70

80

90

100

Figure 6: Results of the AB test for the retention of linguistic
information

Proposed

Results of Experiment 1: Figure 4 and Figure 5 show theaverage CER score for the retention of linguistic informationand MOS for naturalness, respectively. As shown by the results of the CER test, the proposed method signiﬁcantly outperformed the baseline method in terms of retaining the linguistic
information of both 1st and 2nd sounds. In addition, the MOS
test shows that the proposed method outperformed the baseline
method in terms of naturalness.
Result of Experiment 2: Figure 6 shows the score of the AB
test in terms of the clarity of linguistic information. Based on
the score, the proposed method achieved the highest score interms of the clarity of linguistic information.

*

3.0

5. Conclusion

2.0

The original StarGAN-VC can fail to preserve the linguistic
content of input speech when the number of training utterances
is very small. To address this issue, in this paper, we proposed StarGAN-VC+ASR, a method that exploits the result of
ASR applied to each training utterance during the training of
StarGAN-VC’s generator. Experimental results showed thatStarGAN-VC+ASR had a better ability to preserve linguistic
contents than the original StarGAN-VC.
In the present experiment, we used a small dataset for training and showed that our method outperformed StarGAN-VC.
However, as the size of the dataset increases, the performanceof StarGAN-VC is also expected to improve. In the future, weplan to investigate how the performance of StarGAN-VC andStarGAN-VC+ASR changes as the size of the dataset increases.
Moreover, to improve the quality of sounds converted by
the proposed method, employing a mel-spectrogram and neuralnetwork vocoder [24–26] instead of the MCEPs and WORLD
analyzer, respectively, can be a future challenge.
In this study, we applied StarGAN-VC+ASR to speakeridentity voice conversion. However, the application of nonparallel voice conversion is not limited to speaker-identity voice
conversion. For example, emotional voice conversion is an important target for voice conversion. The application of StarGANVC+ASR to emotional voice conversion will be our future work.
StarGAN-VC+ASR used phoneme index information as a regularization term. This term improved the clarity of the converted
voice. However, this may reduce the diversity of para-linguistic
expressions in utterances and negatively affect emotional voice
conversion. Validating this point will also be part of our future
work.

1.0
0.0

Baseline

74.4*

Figure 4: Character error rates (CERs) for retention of linguistic information．CERs of the 1st and 2nd sound represent theﬁrst and second sounds heard by the listeners, respectively, andthe content uttered are the same. The asterisks, *, indicate thatthe coefﬁcients are statistically different from zero at the 1 percent level.

4.0

Fair

Baseline

PropsedFigure 5: Result of the MOS test for naturalness．The asterisks,*, indicate that the coefﬁcients are statistically different fromzero at the 1 percent level.

4.2. Subjective evaluation
We conducted listening tests to analyze the performance of the
proposed method compared to StarGAN-VC [2], which is thebaseline in our proposed method. To measure the consistencyof the linguistic contents of input and converted speech, we usedthe character error rate (CER) test as an evaluation metric. CER
evaluates the error rate between the correct sentence and thesentence written by a listener after listening to the converted
speech. During the CER test, the listeners hear the same utterance of a converted speech twice because they may not beable to hear the converted speech at once. The ﬁrst trial is denoted as “1st sound,” and the second trial is denoted as “2ndsound” in Figure 4. To measure naturalness, we conducted amean opinion score (MOS) test. During the MOS test, listeners
were asked to rate the naturalness of the converted speech on a
5-point scale. To measure the clarity of linguistic information,
we conducted an AB test. “A” and “B” were the speech converted by the baseline and proposed methods, respectively. For
each sentence pair, the listeners were asked to select their preferred one, “A,” “B,” or “Fair.” We conducted CER and MOS
tests as in Experiment 1, and the AB test as in Experiment 2.
Ten sentences were randomly selected for Experiments 1 and 2
from 240 synthesized speech signals. Sentences of different utterances were selected between Experiment 1 and Experiment
2. Sixteen well-educated Japanese speakers participated in thetests.

6. Acknowledgements
This study was supported by the Japan Society for the Promotion of Science (JSPS) KAKENHI Grant-in-Aid for ScientiﬁcResearch (B), grant number 18H03308, Grant-in-Aid for Scientiﬁc Research on Innovative Areas (grant number 16H06569),
and JST CREST Grant JPMJCR19A3.

1362

7. References[17] D. P. Kingma and M. Welling, “Auto-encoding variational bayes,”
arXiv, vol. abs/1312.6114, 2013.

[1] A. van den Oord, O. Vinyals, and K. Kavukcuoglu, “Neural discrete representation learning,” in Proc. Conference on Neural Information Processing Systems (NIPS), 2017.
[2] H. Kameoka, T. Kaneko, K. Tanaka, and N. Hojo, “Stargan-vc:
Non-parallel many-to-many voice conversion using star generative adversarial networks,” in IEEE Spoken Language Technology
Workshop (SLT), 2018, pp. 266–273.
[3] K. Qian, Y. Zhang, S. Chang, X. Yang, and M. HasegawaJohnson, “Autovc: Zero-shot voice style transfer with only autoencoder loss,” in Proc. International Conference on Machine
Learning, 2019, pp. 5210–5219.
[4] Y. Stylianou, O. Cappé, and E. Moulines, “Continuous probabilistic transform for voice conversion,” IEEE Transactions on speech
and audio processing, vol. 6, no. 2, pp. 131–142, 1998.
[5] S. Desai, A. W. Black, B. Yegnanarayana, and K. Prahallad,“Spectral mapping using artiﬁcial neural networks for voice conversion,” IEEE Transactions on Audio, Speech, and Language
Processing, vol. 18, no. 5, pp. 954–964, 2010.
[6] T. Kaneko, H. Kameoka, K. Hiramatsu, and K. Kashino,“Sequence-to-sequence voice conversion with similarity metriclearned using generative adversarial networks.” in Proc. The Annual Conference of the International Speech Communication Association (Interspeech), 2017, pp. 1283–1287.
[7] C.-C. Hsu, H.-T. Hwang, Y.-C. Wu, Y. Tsao, and H.-M. Wang,“Voice conversion from non-parallel corpora using variational
auto-encoder,” in Proc. Asia-Paciﬁc Signal and Information Processing Association Annual Summit and Conference (APSIPA),
2016, pp. 1–6.
[8] Hsu, Chin-Cheng and Hwang, Hsin-Te and Wu, Yi-Chiao andTsao, Yu and Wang, Hsin-Min, “Voice conversion from unaligned
corpora using variational autoencoding wasserstein generative adversarial networks,” in Proc. The Annual Conference of the International Speech Communication Association (Interspeech), 2017,
pp. 3364–3368.
[9] T. Kaneko and H. Kameoka, “Cyclegan-vc: Non-parallel voice
conversion using cycle-consistent adversarial networks,” in Proc.
26th European Signal Processing Conference (EUSIPCO), 2018,
pp. 2100–2104.
[10] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. WardeFarley, S. Ozair, A. Courville, and Y. Bengio, “Generative adversarial nets,” in Advances in neural information processing systems, 2014, pp. 2672–2680.
[11] P. Isola, J.-Y. Zhu, T. Zhou, and A. A. Efros, “Image-to-imagetranslation with conditional adversarial networks,” in Proc. IEEE
Conference on Computer Vision and Pattern Recognition (CVPR),
2017, pp. 5967–5976.
[12] G. Rizos, A. Baird, M. Elliott, and B. Schuller, “Stargan for emotional speech conversion: Validated by data augmentation of endto-end emotion recognition,” in Proc. IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2020,
pp. 3502–3506.
[13] A. Moritani, R. Ozaki, S. Sakamoto, H. Kameoka, andT. Taniguchi, “Stargan-based emotional voice conversion forjapanese phrases,” arXiv preprint arXiv:2104.01807, 2021.
[14] L. Sun, K. Li, H. Wang, S. Kang, and H. Meng, “Phonetic posteriorgrams for many-to-one voice conversion without parallel data
training,” in Proc. IEEE International Conference on Multimedia
and Expo (ICME), 2016, pp. 1–6.
[15] T. Kinnunen, L. Juvela, P. Alku, and J. Yamagishi, “Nonparallel voice conversion using i-vector plda: Towards unifying
speaker veriﬁcation and transformation,” in Proc. IEEE International Conference on Acoustics, Speech and Signal Processing(ICASSP), 2017, pp. 5535–5539.
[16] T. Xiaohai, W. Zhichao, Y. Shan, Z. Xinyong, D. Hongqiang,
Z. Yi, Z. Mingyang, Z. Kun, S. Berrak, X. Lei, and L. Haizhou,“The NUS & NWPU system for voice conversion challenge
2020,” in Proc. Joint Workshop for the Blizzard Challenge andVoice Conversion Challenge, 2020, pp. 170–174.

[18] A. Kurematsu, K. Takeda, Y. Sagisaka, S. Katagiri, H. Kuwabara,
and K. Shikano, “Atr japanese speech database as a tool of speech
recognition and synthesis,” Speech Communication, vol. 9, no. 4,
pp. 357–363, 1990.
[19] M. Morise, F. Yokomori, and K. Ozawa, “World: a vocoder-based
high-quality speech synthesis system for real-time applications,”
IEICE TRANSACTIONS on Information and Systems, vol. 99,
no. 7, pp. 1877–1884, 2016.
[20] M. Morise, “D4c, a band-aperiodicity estimator for high-quality
speech synthesis,” Speech Communication, vol. 84, pp. 57–65,
2016.
[21] K. Liu, J. Zhang, and Y. Yan, “High quality voice conversion
through phoneme-based linear mapping functions with straightfor mandarin,” in Proc. IEEE Fourth International Conference on
Fuzzy Systems and Knowledge Discovery (FSKD), vol. 4, 2007,
pp. 410–414.
[22] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” International Conference on Learning Representations, 2014.
[23] A. Lee and T. Kawahara, “Recent development of open-source
speech recognition engine julius,” in Proc. Asia-Paciﬁc Signal andInformation Processing Association Annual Summit and Conference (APSIPA ASC), 2009, pp. 131–137.
[24] A. Tamamori, T. Hayashi, K. Kobayashi, K. Takeda, and T. Toda,“Speaker-dependent wavenet vocoder.” in Proc. The Annual Conference of the International Speech Communication Association(Interspeech), vol. 2017, 2017, pp. 1118–1122.
[25] R. Prenger, R. Valle, and B. Catanzaro, “Waveglow: A ﬂow-based
generative network for speech synthesis,” in Proc. IEEE International Conference on Acoustics, Speech and Signal Processing(ICASSP), 2019, pp. 3617–3621.
[26] K. Kumar, R. Kumar, T. de Boissiere, L. Gestin, W. Z. Teoh,
J. Sotelo, A. de Brébisson, Y. Bengio, and A. Courville, “Melgan: Generative adversarial networks for conditional waveform
synthesis,” Proc. Conference on Neural Information Processing
Systems (NIPS), pp. 14 881–14 892.

1363