ドメイン適応と話者一致損失を用いた話者適応によるクロスリンガル音声合成 ∗☆辛 徳泰，齋藤 佑樹，高道 慎之介，郡山 知樹，猿渡 洋（東大院・情報理工）

1

Introduction

Cross-lingual text-to-speech (TTS) synthesis
refers to a task that requires the TTS model to synthesize a source language’s speech for a target language’s speaker. Previous work about cross-lingual
TTS usually trains a multilingual multi-speaker
TTS model on multiple monolingual datasets byconditioning the TTS model on speaker and language representations called speaker and language
embeddings, respectively [1, 2]. One of the drawbacks of such a method is that it requires a large
amount of speech data with transcriptions for speakers of the target language. However, in practice notnecessarily all speakers of the target language havesuch an amount of data. In such a case speaker
adaptation can be used, which refers to a technology that can adapt the TTS model to a newspeaker whose data is not included in the training set and enable the TTS model to synthesize thetarget speaker’s speech at a relatively lower cost.
Existing monolingual speaker adaptation methods
often use a small amount of the target speaker’sspeech data to ﬁne-tune a pretrained multi-speaker
TTS model [3]. However, since the TTS model ofthe source language never sees the target language’s
data during the training process, this method is notapplicable for cross-lingual speakers, i.e., speakers of
diﬀerent languages.
In this paper, we propose a cross-lingual speaker
adaptation method using domain adaptation andspeaker consistency loss for TTS synthesis. The basic idea of our work is inspired by Jia et al. [4],a monolingual speaker adaptation method. It only
needs multilingual data without transcriptions totrain a speaker veriﬁcation model and the source
language’s data to train a TTS model. Also, the proposed method can ﬁne-tune the source language’s
TTS model on the target language’s data by using the speaker embeddings of the target speakers,
which only needs up to 3 minutes of speech dataof each speaker. Inspired by our previous work
[1, 2], the proposed method ﬁrst trains a languageindependent speaker veriﬁcation model on multilingual data by using domain adaptation. Then thespeaker embeddings for source language’s speakers are used to train a monolingual multi-speaker
TTS model. To adapt the TTS model to crosslingual target speakers, the proposed method ﬁne∗NormalTraining

Speaker
Encoder

Proposed Domain
Adversarial Training

Speaker
EmbeddingMel-spectrogram

GE2E

Adv. Language
ClassifierFig. 1 Architecture of the speaker encoder. The
adversarial language classiﬁer is not used in the conventional method.
tunes the TTS model under a speaker consistency
loss that maximizes the cosine similarity between
speaker embeddings of the same speaker generated
from the target language’s natural speech and thesource language’s synthesized speech. Experimental
results demonstrated that (1) the proposed method
can synthesize speech of the source language ofa cross-lingual speaker with higher speech naturalness than the conventional method; (2) using
speaker consistency loss with a language-dependent
speaker veriﬁcation model will instead causing performance degradation. The audio samples of thiswork are publicated on our project page: https://aria-k-alethia.github.io/2021clsa/.

2Baseline Method

In this section we introduce a baseline method
for cross-lingual speaker adaptation based on previous work [4] used in the experiments. The baseline
method contains two components: (1) a speaker encoder based on multilingual speaker veriﬁcation, (2)a monolingual multi-speaker TTS model based on
Tacotron2.
2.1 Speaker Encoder based on Speaker Veriﬁcation
Recent work has shown that speaker veriﬁcation/recognition is an appropriate pretext task tolearn speaker embeddings for multi-speaker TTS[1, 5]. This is because learning a representativeand distinct speaker embedding for each speaker isa common requirement for speaker veriﬁcation andmulti-speaker TTS. Thus in this stage we train thespeaker encoder based on speaker veriﬁcation using
multilingual data.
The general architecture of the speaker encoder isCross-lingual Speaker Adaptation using Domain Adaptation and Speaker Consistency Loss for Text-ToSpeech Synthesis, by Detai Xin, Yuki Saito, Shinnosuke Takamichi, Tomoki Koriyama, Hiroshi Saruwatari(The University of Tokyo).

NormalTraining

Proposed Speaker
Consistency Training

Minimize theL1 DistanceNaturalMel-spectrogram

SynthesizedMel-specrogramSpeakerEncoder
Maximize theCosine SimilarityTacotron2
Text

SpeakerEmbedding

SynthesizedMel-spectrogram'sSpeaker Embedding

Fig. 2 Diagram of the proposed TTS model andthe speaker consistency loss.
illustrated in Figure 1. Basically it is similar to thearchitecture mentioned in Jia et al. [4]. The only
diﬀerence here is that we use a more powerful model
called ResCNN [6] as the speaker encoder’s implementation instead of simple long short-term memory to encode the mel-spectrograms to cope with thecomplexity of multilingual data. We use generalized
end-to-end (GE2E) loss Lge2e [7] as the speaker veriﬁcation loss.
2.2 Multi-speaker Tacotron
After we get the trained speaker encoder, we train
a monolingual multi-speaker TTS model by conditioning a Tacotron2-based TTS model [8] on thespeaker embedding generated by the speaker encoder (Figure 2). Speciﬁcally, following previous
work [4] we concatenate the speaker embedding tothe output states of the text encoder and use thisas the input of the attention module to let the decoder attend over them. In this stage we only use
the source language’s data and keep the parametersof the speaker encoder ﬁxed.
Formally, denote the mel-spectrogram of the jthutterance of the ith speaker as xij , the embedding
generated by speaker encoder fs (·) is deﬁned as:
eij = fs (xij ). Denote the Tacotron2 model and thetext sequence of the mel-spectrogram xij as ft (·)
and yij , respectively, the loss function of the TTS
model is deﬁned as the L1 distance between the natural mel-spectrogram xij and the synthesized one∑x̃ij = ft (yij , eij ): Ltts =i,j |x̃ij − xij |. After
training, to adapt to a new speaker, we ﬁrst averagespeaker embeddings generated from the speaker’s kutterances and use the averaged speaker embedding
to synthesize the source language’s speech.

3Proposed Method

In this section we describe the proposed methodby extending the baseline method described in Section 2. The proposed method ﬁrst trains a speaker
encoder based on speaker veriﬁcation using GE2E

loss and domain adaptation to construct a languageindependent speaker space. To ﬁne-tune the TTS
model on cross-lingual speech, the proposed method
then uses a speaker consistency loss to maximize thecosine similarity between the speaker embeddings ofthe natural and the same speaker’s synthesized melspectrograms, which can beneﬁt from the languageindependent speaker space learned by the domain
adaptation.
3.1 Language-independent Speaker Veriﬁcation based on Domain Adaptation
The baseline speaker veriﬁcation method introduced previously (Section 2.1) is suitable for monolingual settings. However, when training the modelon multiple datasets of diﬀerent languages, thelanguage-dependent speakers cause a domain shift
between the speakers of diﬀerent languages [9].
This impedes the knowledge transferring from thespeaker encoder to the TTS model for cross-lingualspeakers. Therefore to make it possible to ﬁne-tune
the TTS model on cross-lingual data, following ourprevious work [1, 2] we use a domain adaptation algorithm to eliminate the domain shift.
Speciﬁcally, we use domain adversarial neural network (DANN) [10]. As illustrated in Figure 1, theproposed method extends the baseline method byadding an adversarial language classiﬁer. Duringtraining the classiﬁer tries to learn the language label from the speaker embedding. Meanwhile, theadversarial training forces the model to exclude information relating to language from the speakerembedding, which ﬁnally makes the speaker encoder become language-independent. This is accomplished by a gradient reversal layer (GRL) to reverse
the gradient back-propagated from the classiﬁer tothe speaker embedding.
Formally, denote the GRL operator, the adversarial language classiﬁer and the language label of xij
as △, fl (·) and lij , respectively, the loss of DANN isdeﬁned as: ∑Lda =−lij log σ(fl (△eij ))i,j(1)− (1 − li ) log (1 − σ(fl (△eij ))),where σ(·) is the sigmoid function. The ﬁnal loss
function of the proposed speaker encoder is deﬁned
as: Ls = Lge2e + Lda . This loss makes the speakerspace generated by the speaker encoder not only reﬂect the speaker characteristics but also not dependon language.
3.2 Speaker Adaptation based on Consistency Loss
As mentioned previously cross-lingual speech data
can not be used to ﬁne-tune a monolingual TTS
model of source language directly. However, theknowledge learned by the pretrained speaker encoder can be utilized. Thus we propose a speakerconsistency loss (SCL) to maximize the cosine similarity between speaker embeddings extracted fromthe natural and the same speaker’s synthesized melspectrogram. The core idea is inspired by Nachmani
et al. [11], which has shown a possible way to apply
such loss to the cross-lingual TTS task. While theirwork only used the gradient of the loss to update thespeaker encoder’s parameters, which has little inﬂuence on the speech synthesis process, in this work
we used the gradient to update the parameters ofthe decoder in the Tacotron2 model.
As illustrated in Figure 2, to compute the SCL theproposed method ﬁrst feeds the text and the speakerembedding extracted from a target speaker’s natural mel-spectrogram to the multi-speaker Tacotron2
model to synthesize a mel-spectrogram of the source
language. The synthesized mel-spectrogram is thenfed to the speaker encoder to get the speaker embedding of the synthesized mel-spectrogram of the samespeaker. Finally the cosine similarity between thetwo embeddings of the same speaker is maximized.
In the ﬁne-tuning stage we froze the parameters of
the speaker encoder and the Tacotron2 model except for the mel-spectrogram decoder. Formally, the∑SCL is deﬁned as:Lscl = − i,j cos (fs (x̃ij ), eij )). In
addition, during experiments we found only using
cross-lingual data to ﬁne-tune would decrease thespeech quality. We then computed SCL for both
intra-lingual and cross-lingual speakers to ﬁne-tune
the TTS model and saw improvements in this case.
To stabilize the ﬁne-tuning, we still include the L1
loss in this process. Thus the loss function for ﬁnetuning is deﬁned as: Lf t = Ltts + αLscl , where α isa hyperparameter.

4Experiments

4.1

Experimental Setup

In our experiments, English is regarded as thesource language, and Japanese is regarded as thetarget language. We used English multi-speakerdataset VCTK [12] and Japanese multi-speakerdataset JVS [13] to train the speaker veriﬁcation
model. The total number of speakers was 207
in which 107 were English speakers and 100 wereJapanese speakers. We randomly picked up 8 English speakers and 8 Japanese speakers (both contained 4 female and 4 male speakers) as unseen
speakers and excluded their data from the training and ﬁne-tuning process. All audios were downsampled to 16 kHz and converted to 80-dimensionalmel-spectrograms. The frame number of the melspectrogram was segmented to [120, 150] for training and inference of the speaker veriﬁcation model.
After training the speaker encoder we trained theTTS model by the L1 loss Ltts using the VCTK
dataset. Then we ﬁne-tuned the TTS model using the ﬁne-tuning loss Lf t with both English andJapanese data. Finally we used WaveRNN [14]to convert the synthesized mel-spectrogram to thetime-domain waveform.
In all experiments, the dimension of the speakerembedding was set to 64. The number of utterances
k for computing speaker embedding and the weight
hyperparameter α of SCL were empirically set to 5and 0.1, respectively. We multiplied the gradient
back-propagated from language classiﬁer to the embedding by a factor λp = 1+exp 2(−10·p) − 1, where p
represents the training progress ranging from 0 to 1.
We trained four models for comparison. The ﬁrst
two models were trained using the baseline and theproposed methods (Section 2, 3), which are denotedby Base. and DA+SCL, respectively. For ablation experiments we additionally trained two models. The Base.+SCL model was trained by using
SCL with baseline speaker encoder but without domain adaptation. The DA model was trained by theproposed method without SCL. Note that we didn’t
ﬁne-tune the TTS model for the two models withoutusing SCL (Base. and DA).
4.2 Subjective Evaluation
We evaluated the synthesized speech from two aspects: speech naturalness and speaker similarity. Inaddition to the 16 unseen speakers, we randomly
picked 16 speakers from the training set as seen
speakers. Note that, since models without using
SCL have no ﬁne-tuning step, the 8 Japanese seen
speakers are seen by the speaker encoder but completely unseen by the TTS model of Base. and DA.
For each speaker we synthesized 20 utterances. Allof these utterances were not included in the training process. We used Amazon Mechanical Turk1 toconduct the subjective evaluation.
4.2.1 Speech Naturalness
We conducted 5-scale mean opinion score (MOS)
tests to evaluate the speech naturalness of the synthesized speech. We categorized all utterances into
four groups by their speaker (seen/unseen) and task
(intra-lingual/cross-lingual); the MOS was calculated separately for each group. Totally 720 English
listeners joined in the evaluation; each test had 90
listeners with 25 answers per listener. The result isshown in Table 2, in which ground truth (GT) represents the natural speech. It can be observed thatthe proposed DA+SCL model obtains the best performance in all tasks. For the intra-lingual task,Base.+SCL also obtains signiﬁcant improvements
compared to Base. and DA, demonstrating that theSCL can improve the intra-lingual speaker adaptation. In contrast, for the cross-lingual tasks the performance of Base.+SCL degrades signiﬁcantly while
the DA+SCL model still maintained similar performance compared to their performance on the intra1 https://www.mturk.com/

Table 1 Results of XAB tests on speaker similarity. Bold scores indicate preferred method has p value lessthan 0.05
Task

Intra-lingual
Cross-lingual

Speaker
Seen
UnseenSeen
UnseenBase. vs. DA
0.464 - 0.536
0.450 - 0.550
0.440 - 0.560
0.524 - 0.476

Base.+SCL vs. DA+SCL
0.473 - 0.527
0.448 - 0.552
0.408 - 0.592
0.531 - 0.469

Table 2 Results of MOS evaluation on naturalness.
Bold indicates better method comparing to Base.
without overlapping 95% conﬁdence interval
Task Spkr.

Base.

Seen
3.51
Unseen 3.62
Seen
3.39
Cross
Unseen 3.54
IntraBase.+
SCL
3.65
3.73
2.84
3.07

DA3.58
3.61
3.60
3.48

DA+SCL
3.67
3.77
3.61
3.55

GT
4.04
4.15
4.04
4.06

lingual tasks. This demonstrates that the domain
shift makes it diﬃcult to apply SCL directly foradapting the TTS model to cross-lingual speakers.
Finally the DA model also has comparable performance to the DA+SCL model, which shows theeﬀectiveness of removing the domain shift for thecross-lingual tasks.
4.2.2 Speaker Similarity
We then conducted preference XAB tests to evaluate the speaker similarity of the synthesized speech.
Here X is the natural speech, which is English forVCTK speakers and Japanese for JVS speakers. We
chose four pairs among all six combinations of thefour models. Totally 800 listeners joined in the evaluation; each test had 25 listeners with 10 answers
per listener. The result is shown in Table 1. The
DA and DA+SCL models both obtain relatively better performance than the Base. and Base.+SCLmodels, respectively, which shows the eﬀectivenessof the proposed method. Surprisingly, the performance of DA and DA+SCL are almost the same,
which implies the SCL has relatively little inﬂuenceon speaker similarity.

5Conclusions

This paper described a method for cross-lingual
speaker adaptation for TTS synthesis based ona language-independent speaker encoder using domain adaption and a speaker consistency loss whichmakes it possible to ﬁne-tune the TTS model ofsource language on the target language’s data. Experimental results demonstrated that the proposedmodel could signiﬁcantly improve speech naturalness compared to the baseline method.
謝 辞：本 研 究 は ，JSPS 科 研 費 19H01116，
18K18100，17H06101，JSPS-CAS 二 国 間 交 流
事業 JPJSBP120197203 による支援を受けたものです。

DA vs. DA+SCL
0.500 - 0.500
0.479 - 0.521
0.492 - 0.508
0.488 - 0.512

Base.+SCL vs. DA
0.468 - 0.532
0.504 - 0.496
0.408 - 0.592
0.496 - 0.504

参考文献[1] D. Xin et al., “Cross-lingual text-to-speech synthesis via domain adaptation and perceptual similarity
regression in speaker space,” in Proc. Interspeech,Shanghai, China, Oct. 2020.
[2] ——, “Disentangled speaker and language representations using mutual information minimizationand domain adaptation for cross-lingual TTS,” inProc. ICASSP, Toronto, Canada, Jun. 2021.
[3] H. B. Moss et al., “Boﬃn TTS: Few-shot speaker
adaptation by bayesian optimization,” in Proc.
ICASSP, Barcelona, Spain, May 2020, pp. 7639–7643.
[4] Y. Jia et al., “Transfer learning from speaker veriﬁcation to multispeaker text-to-speech synthesis,”
arXiv preprint arXiv:1806.04558, 2018.
[5] J. Cho et al., “Learning speaker embedding fromtext-to-speech,” in Proc. Interspeech, Shanghai,China, Oct. 2020, pp. 3256–3260.
[6] C. Li et al., “Deep speaker: an end-to-end neural speaker embedding system,” arXiv preprint
arXiv:1705.02304, May 2017.
[7] L. Wan et al., “Generalized end-to-end loss forspeaker veriﬁcation,” in Proc. ICASSP, Calgary,Alberta, Canada, Apr. 2018, pp. 4879–4883.
[8] J. Shen et al., “Natural TTS synthesis by conditioning WaveNet on mel spectrogram predictions,”in Proc. ICASSP, Calgary, Alberta, Canada, Apr.
2018, pp. 4779–4783.
[9] W. Xia et al., “Cross-lingual text-independentspeaker veriﬁcation using unsupervised adversarial discriminative domain adaptation,” in Proc.
ICASSP, Brighton, United Kingdom, May 2019,pp. 5816–5820.
[10] Y. Ganin et al., “Domain-adversarial training ofneural networks,” The Journal of Machine Learning Research, vol. 17, no. 1, pp. 2096–2030, 2016.
[11] E. Nachmani, L. Wolf, “Unsupervised polyglot text-to-speech,” in Proc. ICASSP, Brighton,
United Kingdom, May 2019, pp. 7055–7059.
[12] C. Veaux et al., “Superseded-CSTR VCTK corpus:
English multi-speaker corpus for cstr voice cloning
toolkit,” 2016.
[13] S. Takamichi et al., “JVS corpus: free Japanesemulti-speaker voice corpus,” arXiv preprint
arXiv:1908.06248, 2019.
[14] N. Kalchbrenner et al., “Eﬃcient neural audio synthesis,” in Proc. ICML, Stockholm, Sweden, Jul.
2018, pp. 2410–2419.