Multi-Task Adversarial Training Algorithm for
Multi-Speaker Neural Text-to-Speech
Yusuke Nakai, Yuki Saito, Kenta Udagawa, and Hiroshi Saruwatari
Graduate School of Information Science and Technology, The University of Tokyo, Japan.
E-mail: nakai-yusuke@g.ecc.u-tokyo.ac.jp, yuuki saito@ipc.i.u-tokyo.ac.jp

Abstract—We propose a novel training algorithm for a multispeaker neural text-to-speech (TTS) model based on multi-task
adversarial training. A conventional generative adversarial network (GAN)-based training algorithm significantly improves thequality of synthetic speech by reducing the statistical difference
between natural and synthetic speech. However, the algorithm
does not guarantee the generalization performance of the trained
TTS model in synthesizing voices of unseen speakers who are
not included in the training data. Our algorithm alternatively
trains two deep neural networks: multi-task discriminator andmulti-speaker neural TTS model (i.e., generator of GANs).
The discriminator is trained not only to distinguish between
natural and synthetic speech but also to verify the speaker of
input speech is existent or non-existent (i.e., newly generated
by interpolating seen speakers’ embedding vectors). Meanwhile,
the generator is trained to minimize the weighted sum of the
speech reconstruction loss and adversarial loss for fooling thediscriminator, which achieves high-quality multi-speaker TTS
even if the target speaker is unseen. Experimental evaluationshows that our algorithm improves the quality of synthetic
speech better than a conventional GANSpeech algorithm.

I. I NTRODUCTION
Multi-speaker text-to-speech (TTS) [1] is a technology
for synthesizing various speakers’ voices from given text
using a computer. This technology can extend the application
range of TTS technology to personalized voice assistanceand data augmentation for speech-discriminative tasks [2],[3]. State-of-the-art multi-speaker neural TTS, i.e., a machinelearning-based framework for training deep neural networks(DNNs) [4], [5] that represent the TTS mapping, has enabled synthesizing voices of various speakers comparable tohuman speech [6]. Such quality improvement has mainly
arisen from developments of epoch-making DNN architectures (e.g., Transformer [7]), well-designed and sufficiently
large multi-speaker speech corpora [8], [9], [10], accurate
speaker-identity modeling [11], [12], and sophisticated deep
generative models [13], [14], [15]. Well-trained multi-speakerneural TTS models can be also applied to practical TTS
scenarios such as voice cloning (i.e., reproducing an unseen
speaker’s voice with few utterances) [16]. This paper focuseson the development of an algorithm for training a high-fidelity
multi-speaker neural TTS model.
Generative adversarial networks (GANs) [13] are powerfuldeep generative models that have significantly improved thequality of synthetic speech in various TTS-related tasks, suchas statistical parametric TTS [17], [18], end-to-end TTS [19],[20], and neural vocoding [21], [22]. The quality improvement derives from their frameworks to adversarially train
two DNNs: discriminator and TTS model (i.e., generator),
for learning the distribution of real data. Specifically, thediscriminator is trained to distinguish between synthetic andnatural speech and approximate the divergence (i.e., statisticaldifference) between them. Meanwhile, the generator is trained
to minimize the approximated divergence by fooling thetrained discriminator. This framework can be introduced into
the TTS training by defining the objective function as theweighted sum of speech reconstruction loss (e.g., L1 or
L2 loss between natural and generated speech parameters)
and the adversarial loss for causing the discriminator to
misclassify synthetic speech as natural. As a result, synthetic
speech generated by the trained TTS model successfully
reproduces the fine structures of natural speech parametersthat tend to be over-smoothed by learning to minimize thespeech reconstruction loss only [23].
One can extend the GAN-based training algorithm for
multi-speaker neural TTS by conditioning the discriminator
and TTS model by speaker representation. Such representation can be obtained as a one-hot speaker code [1], trainable lookup embedding [24], or intermediate vector of a
speaker encoder pretrained on speaker-discriminative tasks(e.g., d-vector [25] and x-vector [26]). Zhao et al. [27]proposed an algorithm to train a multi-speaker neural TTS
model conditioned by the speaker code and defined thetraining objective as the speech reconstruction in both melspectrogram and waveform domains and the adversarial lossof the Wasserstein GAN [28] with gradient penalty [29].
Kanagawa et al. [30] modified the training objective of adiscriminator so that it can classify the speaker identity as well
as distinguish natural/synthetic speech. Yang et al. [31] presented GANSpeech, state-of-the-art multi-speaker neural TTS
model based on FastSpeech 2 [32] trained by a GAN-basedalgorithm incorporating the joint conditional and unconditional (JCU) discriminator [33] and scaled feature matchingloss [34]. Although their algorithm achieved the quality ofsynthetic speech approaching that of natural speech, it does
not guarantee the generalization performance of the trained
TTS model in synthesizing voices of unseen speakers who
are not included in the training data. The main reason is that
the GAN-based algorithm regards the distribution of natural
speech uttered by seen speakers as the target to be trainedand never considers whether the TTS model can synthesize
realistic voices of unseen speakers.
We propose a novel GAN-based multi-task training algorithm for a multi-speaker neural TTS model that can
synthesize high-quality voices of unseen speakers. Like a
GAN-based algorithm, our algorithm alternatively trains two
DNNs: multi-task discriminator and multi-speaker neural TTS
model. The discriminator aims to not only distinguish between
natural and synthetic speech but also verify the speaker of input speech is existent or non-existent (i.e., newly generated byinterpolating seen speakers’ embedding vectors). Meanwhile,
the TTS model tries to fool the discriminator by minimizing
the weighted sum of the speech reconstruction loss andadversarial loss for fooling the discriminator. As a result, we
can expect the trained TTS model to synthesize high-quality
multi-speaker voices even if the target speaker is unseen.
Experimental evaluation shows that our algorithm improves
the quality of synthetic speech better than a conventionalalgorithm used in the training of GANSpeech [31].
II. BASELINE M ETHODS
A. FastSpeech 2
FastSpeech 2 [32] is a non-autoregressive end-to-end TTS
model widely used as the backbone of various TTS methods [31], [35]. It performs 1) phoneme duration prediction
and sequence alignment using a duration predictor and alength regulator, 2) speech feature prediction using multiple
variance adaptors, and 3) mel-spectrogram prediction using a
mel-spectrogram decoder. A trained neural vocoder takes thepredicted mel-spectrogram as input to synthesize a speech
waveform. Although the phoneme alignment information isrequired in advance for the training, this TTS model can avoid
some crucial inference errors (e.g., looping or skipping of text
in synthetic speech) that often occur in autoregressive neural
TTS methods such as Tacotron2 [36].
Let X and Y be a text set consisting of N phoneme
sequences {xn }N
n=1 and a speech set consisting of N speechparameter sequences {y n }N
n=1 , respectively. A speech parameter sequence contains phoneme durations y dur , intermediate
features y feat (e.g., F0 and energy) predicted by the varianceadaptors, and mel-spectrograms y mel . A FastSpeech 2-based
TTS model G(·) parameterized by θ (G) is trained to predictparameters of synthetic speech ŷ ∗ from x. The objectivefunction for training the model is given as:(G)

LFS2 (X , Y) = MSE (y dur , ŷ dur ) + MSE (y feat , ŷ feat )+ MAE (y mel , ŷ mel ) ,(1)

where MAE(·) and MSE(·) denote the mean absolute error and mean squared error between two given features,
respectively. The model’s parameters θ (G) are optimized
by using the mini-batch stochastic gradient descent (SGD)(G)θ (G) ← θ (G) − η∇θ(G) LFS2 with a learning rate η > 0.

B. Transfer-Learning-Based Multi-Speaker Neural TTS
We adopt Jia et al.’s method [11] for building our baseline
multi-speaker neural TTS model. In the training phase, a
DNN-based speaker encoder that extracts speaker embedding
from an input speech waveform is first trained on the objective function of speaker verification (e.g., minimizing thegeneralized end-to-end loss [37]). A multi-speaker neural TTS
model is then trained to generate speech parameters from theircorresponding phoneme sequence and speaker embedding z.
In the inference phase, the embedding of a target speaker,
who may be unseen during the training, is first extracted fromreference speech uttered by the speaker. Then, the extracted
speaker embedding and input text are fed into the trained TTS
model to generate the target speaker’s mel-spectrogram.
C. GANSpeech
GANSpeech [31] is the improved version of multi-speakerneural TTS based on FastSpeech 2, which incorporates the
GAN-based training algorithm to achieve high-quality TTS.
1) JCU Discriminator: A JCU discriminator D(·) parameterized by θ (D) is designed to capture general characteristics
of natural mel-spectrograms and speaker-specific features ofthem. Specifically, the discriminator consists of three submodules1 : 1) shared layers DS (·), 2) conditional layers DC (·),
and 3) unconditional layers DU (·), to output conditional andunconditional true/fake predictions, tC and tU , from a melspectrogram y mel and speaker embedding z. First, a hidden
vector hS is extracted by the shared layers from an input
natural mel-spectrogram y mel as hS = DS (y mel ). Then,the conditional and unconditional predictions are obtained
as tC = DC (hS , z) and tU = DU (hS ), respectively. The
same procedure is taken for a synthetic mel-spectrogramŷ mel = G(x), and predictions t̂C and t̂U are obtained.
The discriminator is trained to distinguish natural speechy mel from synthetic speech ŷ mel , with or without the conditioning by speaker embedding. The objective function isdefined as follows:

1
(D)
2
2
(tC − 1) + (tU − 1)
LGAN (X , Y) =
2
(2)

1 2
+
t̂C + t̂2U .
22) Adversarial Loss: The TTS model is trained to makethe discriminator misclassify synthetic speech as natural. The
adversarial loss to cause the misclassification is defined asfollows:
2
2 
1
(G)
t̂C − 1 + t̂U − 1
LGAN (X , Y) =.
(3)
2

1 We omit a fully-connected layer to transform speaker embedding forsimplicity.

𝒚1

𝒛1
Speaker
encoder
𝐸 ⋅

𝒛1

TTS
model
𝐺 ⋅

Speakerinterp.
𝒛2

𝒚2

Speech𝒚 reconst. loss 𝒚

𝒙

𝒛2

or
𝐷S ⋅

𝐶 ⋅

𝒛
𝐷C ⋅

𝛼1 ∼ 𝑈 0.0, 0.5
𝛼2 ∼ 𝑈 0.0, 0.5

Multi-task
discriminator 𝐷 ⋅(a) Speaker embedding & interpolation

𝐷U ⋅0: Pure𝛼: Mixed
0: Synth.
1: Nat.
0: Synth.
1: Nat.

(b) Forward propagation & loss calculation
Fig. 1. Overview of proposed algorithm. The variables x, y, and z denote text, speech, and speaker embedding,respectively. The interpolation coefficient α is sampled from the uniform distribution U (0.0, 0.5).

3) Scaled Feature Matching Loss: The feature matchingloss [29] is a well-known strategy to stabilize the GAN
training and improve the quality of generated data. The coreidea is to make the hidden vectors of the discriminator
extracted by generated data ĥl close to those of real datahl by minimizing the following loss:(G)

LFM (X , Y) =

L
X
1
l=1

Nl

to this autoencoding process, ŷ = G(E(y)), a critic C(·)
regularizes the autoencoder so that interpolation between two
embedding vectors also provides realistic data by decoding.
This regularization is formulated as the adversarial training
between the autoencoder and critic. The critic is trained tominimize the following loss function:(C)∥ĥl − hl ∥1 ,(4)

where Nl denotes the number of elements in the lth hidden
layer. In the GANSpeech training, a hyperparameter λFM tocontrol the importance of feature matching is dynamicallyadjusted in accordance with the magnitude of reconstruction(G)(G)
loss as λFM ← LFS2 /LFM .
4) GANSpeech Algorithm: The GANSpeech algorithm
consists of three phases: 1) TTS model pretraining, 2) discriminator update, and 3) TTS model update. First, the TTS model(G)
is pretrained by minimizing the reconstruction loss LFS2
only. Then, the two DNNs, D(·) and G(·), are alternatively
trained. Specifically, the discriminator is updated as θ (D) ←(D)θ (D) − η∇θ(D) LGAN , and the TTS model is trained to fool(G)the updated discriminator as θ (G) ← θ (G) − η∇θ(G) LTotal ,where the total loss for training the TTS model is given as
(G)
(G)
(G)
(G)
LTotal = LFS2 + LGAN + λFM LFM .
III. P ROPOSE A LGORITHM
As mentioned in Section I, the GAN-based training algorithm for multi-speaker neural TTS does not guarantee that thetrained model can synthesize high-quality speech of unseen
speakers. To deal with the issue, we propose a novel algorithm
for a multi-speaker neural TTS model to improve the qualityof unseen speaker’s synthetic speech.
A. Adversarially Constrained Autoencoder Interpolation(ACAI)The ACAI algorithm [38] aims to learn an autoencoderthat can generate realistic data from the convex combinationof two latent variables. The encoder E(·) extracts a latentvariable (i.e., embedding) z from input data y, and thedecoder G(·) reconstructs the data y from e. In addition

LACAI (Y) = ∥C (y)∥22 + ∥C (ỹ α ) − α∥22 ,(5)

where ỹ is decoded from the interpolated embedding withan interpolation coefficient α ∼ U (0.0, 0.5) as G(αE(y 1 ) +(1 − α)E(y 2 )). The first and second terms enforce that thecritic outputs α = 0 for pure data without interpolation2 andpredicts the mixing coefficient of interpolated data, respectively. Meanwhile, the autoencoder is trained to minimize thefollowing loss function:(E,G)

LACAI (Y) = ∥y − G (E (y))∥22 + λACAI ∥C (ỹ)∥22 ,(6)

where λACAI is a hyperparameter to control the effect of thesecond term that makes the critic recognize the interpolated
data as non-interpolated one.
B. Multi-Task Adversarial Training Algorithm
We introduce the ACAI-derived regularization term to the
GANSpeech algorithm for training a multi-speaker neural
TTS model. Specifically, we regard a pre-trained speaker
encoder and the TTS model as an encoder E(·) and decoder
G(·) used in the ACAI algorithm, respectively. Furthermore,
we extend the JCU discriminator used in GANSpeech to a
multi-task discriminator D(·) that simultaneously predicts 1)
true or fake of input mel-spectrogram with/without conditioning by speaker embedding and 2) coefficient of speaker
interpolation. Figure 1 shows the overview of our algorithm.
Our algorithm requires additional computation in training due
to the multi-task discrimination by D(·) but does not increase
the inference time.
2 To stabilize the training in initial stage, the original algorithm mixes
true data and reconstructed data with a coefficient γ and regards it as the
non-interpolated data [38]. Our algorithm omits this process because it firstpretrains the decoder (i.e., TTS model) without any adversarial constraints.

1) Multi-Task Discriminator: Our multi-task discriminatorconsists of four sub-modules: the three layers included in the
JCU discriminator of GANSpeech (i.e., DS (·), DC (·), and
DU (·)) and ACAI-derived critic layers C(·). The critic layers
predicts the speaker interpolation coefficient of input melspectrogram α̂ from the hidden vector h(S) extracted by theshared layers DS (·).
2) Forward Propagation: Let X = [x1 , x2 , . . . , xM ] ∈ X
and Y = [y 1 , y 2 , . . . , y M ] ∈ Y be mini-batches containing
M phoneme sequences and speech parameters, respectively.
In the forward propagation during the training, our algorithm
takes the following processes.
Speaker embedding extraction and interpolation: The
speaker encoder first extracts speaker embeddings Z =[z 1 , z 2 , . . . , z M ] from the speech parameters as Z = E(Y ).
Then, interpolation coefficients for each embedding α =[α1 , α2 , . . . , αM ]⊤ are sampled from Uniform(0.0, 0.5). Finally, interpolated speaker embedding Z̃ = [z̃ 1 , z̃ 2 , . . . , z̃ M ]are calculated by using the convex combinations of theoriginal embeddings Z and their reversed vectors, i.e., z̃ m =αm z m + (1 − αm )z M −m+1 for m = 1, 2, . . . , M .
Multi-speaker TTS: The TTS model generates 2M
speech parameters, Ŷ = [ŷ 1 , ŷ 2 , . . . , ŷ M ] and Ỹ =[ỹ 1 , ỹ 2 , . . . , ỹ M ], from input texts X paired with original
speaker embeddings Z and interpolated ones Z̃ as Ŷ =G(X, Z) and Ỹ = G(X, Z̃), respectively. The natural andgenerated speech parameters are used for the computation of(G)
speech reconstruction loss LFS2 shown in Eq. (1).
Multi-task discrimination: The discriminator outputs six
scalar values: true/fake predictions for { natural, generated }mel-spectrograms { with, without } conditioning by speakerembeddings and speaker interpolation coefficients for { pure,speaker-interpolated } mel-spectrograms. These values are(D)
used for the computation of the objective function LMT =(D)(D)
LGAN + LACAI , i.e., the sum of Eqs. (2) and (5).
3) Backward Propagation and Model Update: The discriminator is first updated to minimize the multi-task dis(D)(D)
crimination loss LMT as θ (D) ← θ (D) − η∇θ(D) LMT . The
TTS model is then updated to fool the updated discriminator(G)(G)
by minimizing LMT = LTotal + λACAI ∥C(DS (ỹ))∥22 as(G)θ (G) ← θ (G) − η∇θ(G) LMT . Note that the speaker encoder
is not updated because we focus on the effect of adversarial
training only, not the fine-tuning of the encoder.
IV. E XPERIMENTS
A. Experimental Conditions
For training a speaker encoder, we used the Corpus of
Spontaneous Japanese (CSJ) [39] containing 660 hours ofspeech data from 1,417 Japanese speakers (947 men and 470
women). The CSJ speech data were resampled to 16 kHz, andthe frame shift was set to 10 ms. For training, validating, andevaluating a multi-speaker neural TTS model, we used the“parallel100” subset of the Japanese Versatile Speech (JVS)
corpus [9]. The subset contains 22 hours of speech data from

100 Japanese speakers (49 men and 51 women; 100 sentences
per speaker). The ratio for each of training, validation, and test
data was 0.8, 0.1, and 0.1, respectively. Referring to Udagawa
et al.’s study [40], we regarded the following four speakers:“jvs078,” “jvs060,” ”jvs005,” and “jvs010” as unseen speakers
during the training. The JVS speech data were resampled to
22.05 kHz to match the settings of neural vocoding, and theframe shift was set to 12 ms.
We used the open source implementation of FastSpeech
2 [32] published by Wataru-Nakata3 for building our TTS
model. The TTS model predicted 80-dimensional melspectrogram from Japanese phonemes with the aid of thevariance adaptors that predicted the F0 and energy of synthetic speech. The F0 was estimated using the WORLD
vocoder [41], [42]. In the FastSpeech 2 training, we used theAdam optimizer [43] with 4,000 Warmup [44] steps, initial
learning rate of 0.0625, batch size of 8, and 30,000 training
steps.
The DNN architecture architecture of a speaker encoder
was the same as that used in Jia et al.’s method [11].
The dimensionality of speaker embeddings was 256. In the
speaker-encoder training, we used the Adam optimizer with alearning rate of 0.0001, batch size of 8, and 1,000,000 training
step.
Our multi-task discriminator had a structure of its DNN
architecture similar to that of the JCU discriminator in
GANSpeech [31]. Specifically, it consists a stack of 1D
convolutional (Conv1D) layers with the parameters of convolution operations Conv1D(c, k, s), where c, k, and s denote the number of output channels, kernel size, and stride
width, respectively. The activation functions for hidden andoutput layers were leaky ReLU [45] and sigmoid, respectively. The DS (·) had three Conv1D layers: Conv1D(64, 3, 1),
Conv1D(128, 5, 2), and Conv1D(512, 5, 2). The other submodules, DC (·), DU (·), and C(·), were all consisted of
two Conv1D layers: Conv1D(128, 5, 2) and Conv1D(1, 3, 1),
except for conditioning by speaker embedding in DC (·).
The neural vocoder for synthesizing speech waveform was
the “generator universal model” of HiFi-GAN [46] includedin the FastSpeech 2 repository published by ming0244 . We
did not finetune the HiFi-GAN on Japanese speech data.
B. Subjective Evaluations
We evaluated the performance of our algorithm on two TTS
tasks: 1) TTS for seen speakers and 2) voice cloning for unseen speakers. We compared the following three algorithms:(G)1) FS2: Minimizing LFS2 (Eq. (1)) only [32]
(G)
(G)
(G)
(G)
2) GAN: Minimizing LTotal = LFS2 + LGAN + λFM LFM(i.e., the GANSpeech algorithm [31])(G)(G)3) MT: Minimizing LMT = LTotal + λACAI ∥C(DS (ỹ))∥22(i.e., the proposed algorithm)
3 https://github.com/Wataru-Nakata/FastSpeech2-JSUT
4 https://github.com/ming024/FastSpeech2/blob/master/hifigan/generator
universal.pth.tar.zip

TABLE IS UBJECTIVE EVALUATION RESULTS WITH THEIR 95% CONFIDENCE
INTERVALS (TTS FOR SEEN SPEAKERS )Algorithm
FS2
GAN
MT

Naturalness MOS
3.18 ± 0.12
3.52 ± 0.12
3.55 ± 0.12

Speaker similarity DMOS
3.57 ± 0.12
3.79 ± 0.12
3.87 ± 0.12

TABLE IIS UBJECTIVE EVALUATION RESULTS WITH THEIR 95% CONFIDENCE
INTERVALS (TTS FOR UNSEEN SPEAKERS )Algorithm
FS2
GAN
MT

Naturalness MOS
3.13 ± 0.12
3.38 ± 0.12
3.50 ± 0.12

Speaker similarity DMOS
2.38 ± 0.12
2.40 ± 0.13
2.48 ± 0.12

We chose the hyperparameter λACAI = 1.0 empirically.
Speech samples used for the evaluation are available on
http://sython.org/demo/nakai22apsipa/demo.html.
1) TTS for Seen Speakers: We conducted the five-levelmean opinion score (MOS) and degradation MOS (DMOS)
tests to evaluate the naturalness and speaker similarity ofsynthetic speech using the test data of 96 seen speakers,
respectively. In the DMOS test, natural speech uttered bya speaker to be synthesized was used as the reference for
evaluating the similarity. For each test, we recruited 50
listeners who evaluated the quality of speech samples using
our crowdsourced subjective evaluation system. Each listener
evaluated 15 speech samples whose speakers were randomized.
Table I shows the evaluation results. From this table, we
observe that 1) “GAN” and “MT” significantly outperform“FS2” and 2) “MT” achieves similar performance to “GAN.”
These results demonstrate that our multi-task adversarial training algorithm can keep the performance of the conventional
GANSpeech algorithm for TTS of seen speakers.
2) Voice Cloning for Unseen Speakers: We also evaluatedthe performance of our algorithm for voice cloning of unseen
speakers and conducted the MOS and DMOS tests. The
conditions of the evaluation, i.e., the numbers of listeners andspeech samples, were the same as those in Section IV-B1. The
speaker embeddings of the unseen speakers were estimated byutterance-wise.
Table II shows the evaluation results. From this table, we
confirm that “MT” achieves the highest MOS and DMOS
values among the three algorithms, which suggests that ourmulti-task adversarial training algorithm can mitigate thedegradation of speech naturalness in the voice cloning task
and successfully improve the naturalness better than theconventional GANSpeech algorithm. However, we observethat there is a still large gap between the DMOS values ofsynthetic speech in seen speakers’ TTS (the third column ofTable I) and those of cloned speech. This result indicates that
the ACAI-derived regularization is insufficient to diversify thevariation of speakers and another approach such as speakergeneration [47] may be necessary for the absolute improve-

ment of speaker similarity.
V. C ONCLUSION
We proposed a GAN-based multi-task training algorithm
for a multi-speaker neural TTS model that can synthesize
high-quality voices of unseen speakers. Our algorithm alternatively trains two DNNs: multi-task discriminator and multispeaker neural TTS model. The discriminator aims to not
only distinguish between natural and synthetic speech but
also verify the speaker of input speech is existent or nonexistent (i.e., newly generated by interpolating seen speakers’
embedding vectors). Meanwhile, the TTS model tries to foolthe discriminator by minimizing the weighted sum of the
speech reconstruction loss and adversarial loss for foolingthe discriminator. Experimental evaluation showed that ouralgorithm improved the quality of synthetic speech better thana conventional algorithm used in the training of GANSpeech.
In future work, we will investigate how to improve the speakersimilarity of synthetic speech in voice cloning and conduct
TTS experiments for other languages.
ACKNOWLEDGMENT
This work was supported by JST, Moonshot R&D Grant
Number JPMJPS2011 (algorithm development) and JSPS
KAKENHI Grant Number 21K21305 (practical experiment).
R EFERENCES[1] N. Hojo, Y. Ijima, and H. Mizuno, “DNN-based speech synthesis using
speaker codes,” IEICE Transactions on Information and Systems, vol.
E101-D, no. 2, pp. 462–472, Feb. 2018.
[2] S. Ueno, M. Mimura, S. Sakai, and T. Kawahara, “Multi-speakersequence-to-sequence speech synthesis for data augmentation inacoustic-to-word speech recognition,” in Proc. ICASSP, Brighton, U.
K., May 2019, pp. 6161–6165.
[3] J. Pelecanos Y. Huang, Y. Chen and Q. Wang, “Synth2Aug: Crossdomain speaker recognition with TTS synthesized speech,” in Proc.
SLT, Shenzhen, China, Jan. 2021, pp. 316–322.
[4] H. Zen, A. Senior, and M. Schuster, “Statistical parametric speechsynthesis using deep neural networks,” in Proc. ICASSP, Vancouver,
Canada, May 2013, pp. 7962–7966.
[5] A. Oord, S. Dieleman, H. Zen, K. Simonyan, O. Vinyals, A. Graves,N. Kalchbrenner, A. Senior, and K. Kavukcuoglu, “WaveNet: A
generative model for raw audio,” arXiv, vol. abs/1609.03499, 2016.
[6] E. Casanova, J. Weber, C. D. Shulby, A. C. Junior, E. Gölge, and M. A.
Ponti, “YourTTS: Towards zero-shot multi-speaker TTS and zero-shot
voice conversion for everyone,” in Proc. ICML, Baltimore, U.S.A., Jul.
2022, pp. 2709–2720.
[7] N. Parmar J. Uszkoreit L. Jones A. N. Gomez L. Kaiser A. Vaswani,
N. Shazeer and I. Polosukhin, “Attention is all you need,” in Proc.
NIPS, Long Beach, U.S.A., Dec. 2017.
[8] H. Zen, V. Dang, R. Clark, Y. Zhang, R. J. Weiss, Y. Jia, Z. Chen,
and Y. Wu, “LibriTTS: A corpus derived from LibriSpeech for textto-speech,” in Proc. INTERSPEECH, Graz, Austria, Sep. 2019, pp.
1526–1530.
[9] S. Takamichi, R. Sonobe, K. Mitsui, Y. Saito, T. Koriyama, N. Tanji,and H. Saruwatari, “JSUT and JVS: Free Japanese voice corpora
for accelerating speech synthesis research,” Acoustical Science andTechnology, vol. 41, no. 5, pp. 761–768, Sep. 2020.
[10] Y. Shi, H. Bu, X. Xu, S. Zhang, and M. Li, “AISHELL-3: A multispeaker Mandarin TTS corpus,” in Proc. INTERSPEECH, Brno, Czech
Republic, Sep. 2021, pp. 2756–2760.
[11] Y. Jia, Y. Zhang, R. Weiss, Q. Wang, J. Shen, F. Ren, Z. Chen,
P. Nguyen, R. Pang, I. L. Moreno, and Y. Wu, “Transfer learningfrom speaker verification to multispeaker text-to-speech synthesis,” inProc. NeurIPS, Montreal, Canada, Dec. 2018, pp. 4480–4490.

[12] E. Cooper, C.-I Lai, Y. Yasuda, F. Fang, X. Wang, N. Chen, andJ. Yamagishi, “Zero-shot multi-speaker text-to-speech with state-of-theart neural speaker embeddings,” in Proc. ICASSP, Barcelona, Spain,
May 2020, pp. 6184–6188.
[13] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley,
S. Ozair, A. Courville, and Y. Bengio, “Generative adversarial nets,”in Proc. NIPS, Montreal, Canada, Dec. 2014, pp. 2672–2680.
[14] D. P. Kingma and M. Welling, “Auto-encoding variational Bayes,”
arXiv, vol. abs/1312.6114, 2013.
[15] J. Ho, A. Jain, and P. Abbeel, “Denoising diffusion probabilistic
models,” in Proc. NeurIPS, Vancouver, Canada, Dec. 2020.
[16] S. O. Arik, J. Chen, K. Peng, W. Ping, and Y. Zhou, “Neural voicecloning with a few samples,” in Proc. NeurIPS, Montreal, Canada, Dec.
2018, pp. 10019–10029.
[17] Y. Saito, S. Takamichi, and H. Saruwatari, “Statistical parametric speech synthesis incorporating generative adversarial networks,”
IEEE/ACM Transactions on Audio, Speech, and Language Processing,vol. 26, no. 1, pp. 84–96, Jan. 2018.
[18] T. Kaneko, H. Kameoka, N. Hojo, Y. Ijima, K. Hiramatsu, and
K. Kashino, “Generative adversarial network-based postfilter for
statistical parametric speech synthesis,” in Proc. ICASSP, New Orleans,
U.S.A., Mar. 2017, pp. 4910–4914.
[19] J. Donahue, S. Dieleman, M. Binkowski, E. Elsen, and K. Simonyan,“End-to-end adversarial text-to-speech,” in Proc. ICLR, Vienna, Austria,
May 2021.
[20] J. Kim, J. Kong, and J. Son, “Conditional variational autoencoder withadversarial learning for end-to-end text-to-speech,” in Proc. ICML,
Virtual Conference, Jun. 2021, pp. 5530–5540.
[21] K. Kumar, R. Kumar, T. de Boissiere, L. Gestin, W. Z. Teoh, J. Sotelo,A. de Brébisson, Y. Bengio, and A. C. Courville, “MelGAN: Generative
adversarial networks for conditional waveform synthesis,” in Proc.
NeurIPS, Vancouver, Canada, Dec. 2019, pp. 14881–14892.
[22] R. Yamamoto, E. Song, and J. Kim, “Parallel WaveGAN: a fast
waveform generation model based on generative adversarial networks
with multi-resolution spectrogram,” in Proc. ICASSP, Barcelona, Spain,
May 2020, pp. 6199–6203.
[23] T. Toda, A. W. Black, and K. Tokuda, “Voice conversion based on
maximum likelihood estimation of spectral parameter trajectory,” IEEE
Transactions on Audio, Speech, and Language Processing, vol. 15, no.
8, pp. 2222–2235, Nov. 2007.
[24] S. O. Arik, G. Diamosy, A. Gibiansky, J. Miller, K. Peng, W. Ping,
J. Raiman, and Y. Zhou, “Deep Voice 2: Multi-speaker neural text-tospeech,” in Proc. NIPS, Red Hook, New York, U.S.A., May 2017, pp.
2966–2974.
[25] E. Variani, X. Lei, E. McDermott, I. L. Moreno, and J. GonzalezDominguez, “Deep neural networks for small footprint text-dependent
speaker verification,” in Proc. ICASSP, Florence, Italy, May 2014, pp.
4080–4084.
[26] D. Snyder, D. Garcia-Romero, G. Sell, D. Povey, and S. Khudanpur,“X-vectors: robust DNN embeddings for speaker recognition,” in Proc.
ICASSP, Alberta, Canada, Apr. 2018, pp. 5329–5333.
[27] Y. Zhao, S. Takaki, H.-T. Luong, J. Yamagishi, D. Saito, and N. Minematsu, “Wasserstein GAN and waveform loss-based acoustic modeltraining for multi-speaker text-to-speech synthesis systems using a
WaveNet vocoder,” IEEE Access, vol. 6, pp. 60478–60488, Sep. 2018.
[28] M. Arjovsky, S. Chintala, and L. Bottou, “Wasserstein GAN,” arXiv,
vol. abs/1701.07875, 2017.
[29] I. Gulrajani, F. Ahmed, M. Arjovsky, V. Dumoulin, and A. Courville,“Improved training of Wasserstein GANs,” in Proc. NIPS, Long Beach,
U.S.A., Dec. 2017.
[30] H. Kanagawa and Y. Ijima, “Multi-speaker modeling for DNN-based
speech synthesis incorporating generative adversarial networks,” inProc. SSW, Vienna, Austria, Sep. 2019, pp. 40–44.
[31] J. Yang, J.-S. Bae, T. Bak, Y. Kim, and H.-Y. Cho, “GANSpeech:
Adversarial training for high-fidelity multi-speaker speech synthesis,”in Proc. INTERSPEECH, Brno, Czech Republic, Sep. 2021, pp. 2202–
2206.
[32] Y. Ren, C. Hu, X. Tan, T. Qin, S. Zhao, Z. Zhao, and T.-Y. Liu,
“FastSpeech 2: Fast and high-quality end-to-end text to speech,” inProc. ICLR, Vienna, Austria, May 2021.
[33] H. Zhang, H. Li T. Xu, S, Zhang, X. Wang, X. Huang, and D. N.
Metaxas, “StackGAN++: Realistic image synthesis with stacked gen-erative adversarial networks,” IEEE Transactions on Pattern Analysis
and Machine Intelligence, vol. 41, no. 8, pp. 1947–1962, Jul. 2018.
[34] T. Salimans, I. Goodfellow, W. Zaremba, V. Cheung, A. Radford, and
X. Chen, “Improved techniques for training GANs,” in Proc. NIPS,
Barcelona, Spain, Dec. 2016, pp. 2234–2242.
[35] Y. Nishimura, Y. Saito, S. Takamichi, K. Tachibana, and H. Saruwatari,“Acoustic modeling for end-to-end empathetic dialogue speech synthesis using linguistic and prosodic contexts of dialogue history,” inProc. INTERSPEECH, Incheon, South Korea, Sep. 2022, pp. xxxx–
xxxx, (ACCEPTED).
[36] J. Shen, R. Pang, R. J. Weiss, M. Schuster, N. Jaitly, Z. Yang, Z. Chen,
Y. Zhang, Y. Wang, RJ Skerry-Ryan, R. A. Saurous, Y. Agiomyrgiannakis, and Y. Wu, “Natural TTS synthesis by conditioning WaveNeton mel spectrogram predictions,” in Proc. ICASSP, Calgary, Canada,
Apr. 2018, pp. 4779–4783.
[37] L. Wan, Q. Wang, A. Papir, and I. L. Moreno, “Generalized end-to-end
loss for speaker verification,” in Proc. ICASSP, Alberta, Canada, Apr.
2018, pp. 4879–4883.
[38] D. Berthelot, C. Raffel, A. Roy, and I. Goodfellow, “Understanding andimproving interpolation in autoencoders via an adversarial regularizer,”in Proc. ICLR, New Orleans, U.S.A., May 2019.
[39] K. Maekawa, “Corpus of spontaneous Japanese: Its design andevaluation,” in Proc. SSPR, Tokyo, Japan, Apr. 2003, pp. 7–12.
[40] K. Udagawa, Y. Saito, and H. Saruwatari, “Human-in-the-loop speaker
adaptation for DNN-based multi-speaker TTS,” in Proc. INTERSPEECH, Incheon, South Korea, Sep. 2022, pp. xxxx–xxxx, (ACCEPTED).
[41] M. Morise, F. Yokomori, and K. Ozawa, “WORLD: a vocoder-based
high-quality speech synthesis system for real-time applications,” IEICE
Transactions on Information and Systems, vol. E99-D, no. 7, pp. 1877–
1884, 2016.
[42] M. Morise, “D4C, a band-aperiodicity estimator for high-quality speechsynthesis,” Speech Communication, vol. 84, pp. 57–65, Nov. 2016.
[43] D. Kingma and B. Jimmy, “Adam: A method for stochastic optimization,” in arXiv preprint arXiv:1412.6980, 2014.
[44] Priya Goyal, Piotr Dollár, Ross B. Girshick, P. Noordhuis, Lukasz
Wesolowski, Aapo Kyrola, Andrew Tulloch, Y. Jia, and Kaiming He,“Accurate, large minibatch SGD: Training imagenet in 1 hour,” arXiv,
vol. abs/1706.02677, 2017.
[45] A. L. Maas, A. Y. Hannun, and A. Y. Ng, “Rectifier nonlinearities
improve neural network acoustic models,” in Proc. ICML, Atlanta,
U.S.A., Jun. 2013.
[46] J. Kong, J. Kim, and J. Bae, “HiFi-GAN: Generative adversarial
networks for efficient and high fidelity speech synthesis,” in Proc.
NeurIPS, Vancouver, Canada, Dec. 2020.
[47] D. Stanton, M. Shannon, S. Mariooryad, RJ Skerry-Ryan, E. Battenberg, T. Bagby, and D. Kao, “Speaker generation,” in Proc. ICASSP,
Singapore, May 2022, pp. 7897–7901.