Investigating the effectiveness of various speaker embeddings for
multi-speaker end-to-end speech synthesis system using small-sized speech
data
Anonymous ACL submission

Abstract
In this paper, we investigated the effectiveness of incorporating various speaker embeddings into an end-to-end speech synthesis system, for generating a unseen speaker’s voicewith small-sized speech data. To do so, we
adopted learned speaker embeddings from various tasks, such as voice conversion and speakerverification. By combining the speaker embeddings using additive attention mechanism to an
autoregressive-based speech synthesis framework, we could evaluate the performance of
these embedding methods. To further enhance
the speaker similarity and speech quality, thepost-net for the output spectrogram sequence isreplaced by a post-filter network. Experimental
results showed that the proposed speech synthesis system with speaker embedding is capableof generating fluent arbitrary speech utterancesof a unseen speaker with only few speech utterances. Besides, the post-filter network is helpful for enhancing the speaker similarity andspeech naturalness of the output speech.

001
002
003
004
005
006
007
008
009
010
011
012
013
014
015
016
017
018
019
020
021
022
023

1

024

In recent years, end-to-end speech synthesis framework has achieved great results in the respect ofspeech naturalness and fluency, especially for synthesizing speech of a speaker whose speech corpus is available for training the voice model. If
speech data of a speaker is not large enough fortraining, adapting the available voice models is
also quite successful. Conventionally, voice models
are adapted supervisedly, which require data alignment or text transcription for the adaptation data.
Recently, speaker embedding-based method could
alleviate the problem. It encodes the speaker information directly from speech data without transcription, and the speaker embeddings are jointly trainedwith end-to-end text-to-speech (TTS) model. By
applying different speaker embeddings, the generated speech is perceived as the corresponding
speaker. This method is also popular in voice con-

025
026
027
028
029
030
031
032
033
034
035
036
037
038
039
040
041

version and vocoder model training. Besides, multispeaker speech synthesis could also be constructedby using generative adversarial network (GAN)
framework (Kameoka et al., 2018) or autoencoderbased methods (Qian et al., 2019; Chou et al., 2019)to learn speaker-related information. However, forgenerating speech of an unseen speaker, jointly
trained speaker encoder is not able to obtain the
speaker information in the training phase, therefore
it still requires model adaptation or transform learning (Jia et al., 2018; Chien et al., 2021) after modeltraining.
There are works that tried to tackle this problem
in order to generate multi-speaker TTS system for
unseen speakers (Chou et al., 2019; Cooper et al.,2020). In (Chou et al., 2019), they focused on encoding speaker and content information separately
with specially designed layers for the sequence-tosequence framework. By applying instance normalization (Huang and Belongie, 2017), which
is widely used for style transformation in computer vision, to encode the speaker informationand then the adaptive instance normalization of
the decoder takes the target speaker information
to generate speech utterances of the target speaker.
In (Cooper et al., 2020), a zero-shot multi-speaker
end-to-end text-to-speech system is implemented
based on speaker verification task. Their speakerembedding is trained from an end-to-end speakerrecognition model and inputted it at pre-net and theself-attention of an end-to-end TTS framework generated best results in both speaker verification andspeaker similarity tasks. These two methods havea common idea that only few speech data from thetarget speaker are required. Therefore, in this paper, we investigate the performance of both speakerembeddings and tried to further enhance the speech
quality and speaker similarity.
For constructing our end-to-end multi-speaker
TTS system, there are two main trends that canbe chosen. One is autogressive model which pre-Introduction

1

042
043
044
045
046
047
048
049
050
051
052
053
054
055
056
057
058
059
060
061
062
063
064
065
066
067
068
069
070
071
072
073
074
075
076
077
078
079
080
081
082

083
084
085
086
087
088
089
090
091
092
093
094
095
096

115

In this paper, we adopted the Tacotron2 as ourbaseline system due to its various applications of
TTS systems (Stanton et al., 2018). To alleviate theproblem of autoregressive model, in the decoding
process, the forward attention is used instead oforiginal local sensitive attention in Tacotron2 andthe Bahdanau attention layer is added to improve
frame repetition or skipping problem, and also accelerate the training process. For enhancing the
speech quality of output speech, we aimed to postprocessing the output spectrogram while not delaying the entire training process significantly. The
organization of the paper is listed as follows. In
Section 2, we describe our multi-speaker Tacotron
system modified by incorporating speaker embeddings and a post-filter enhancement. Section 3
shows experimental setups and the objective andsubjective evaluations. Section 4 concludes with
our findings and future work.

116

2

117

In this section, the proposed multi-speaker speechsynthesis system is introduced. The main idea ofthe proposed system is: the speaker embedding
is adopted for learning the speaker-level information such as timbre and speaking style, and thus itshould be able to separate the speaker-level information from the original speech utterance while thelanguage-level information is not altered. Then, weapplied two attention mechanisms to input speakerembeddings for Tacotron framework integration.
Finally, a post-filter network (Takamichi et al.,2014; Kaneko et al., 2017) is adopted instead of
conventional post-net for better speech enhancement.

097
098
099
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114

118
119
120
121
122
123
124
125
126
127
128
129
130

2.1

dicts output speech frames based on previous results, such as Tacotron2 (Wang et al., 2017; Stanton
et al., 2018). Tacotron2 is a sequence-to-sequencemodel that encodes the input text to a textual
embedding and then decodes it to corresponding
Mel-spectrogram using attention alignment. Another trend is non-autoregressive model like FastSpeech (Ren et al., 2020) and Transformer TTS (Liet al., 2019). This method is much faster than autoregressive method and alleviate generation errors
derived from the attention mechanism. Besides, it
could also adopt additional acoustic features, which
is helpful for generating more stable and expressive
speech output (Łańcucki, 2021).

Speaker Embedding and Encoding
process131

In the encoding process, the input character sequences are firstly encoded by 3 convolution layers
and a Bidirectional LSTM as the original Tacotron
encoder. However, to alleviate the sequential information that LSTM has to carry for long sequences,
we added a self-attention layer as another encoding
output, which is helpful for capturing the long-term
contextual information and thus should let decoding process find the mapping faster. This is inspiredby the findings of previous works (Cooper et al.,2020; Yasuda et al., 2019). The encoder process is
illustrated in fig. 1, Note that the speaker embedding is concatenated with both output sequences.
For the speaker embedding, as described in Section 1, we chose two speaker embeddings to evaluate the effectiveness for the unseen speaker problem. First is the LDE (Cooper et al., 2020), which
is trained for speaker verification task. The other isthe VAE-based method (Chou et al., 2019), which
is trained for voice conversion task. From their results, LDE is not only able to discriminate different
speakers better than conventional x-vector (Snyder
et al., 2018), but also useful for zero-shot multispeaker TTS system. For VAE-based speaker embedding, it also outperformed the x-vector in thevoice conversion task, so we are interested in comparing these two methods in the same experimental
setup.

133

Proposed Method

Figure 1: The modified framework for Tacotron encoderwith speaker embedding.

2

132

134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160

2.2

162

174

In the decoding process, as shown in fig. 2 in order to let two outputs from the encoder could be
combined together, we used forward attention forthe short-term contextual information, which canalign the sequence faster, and Bahdanau attention
for long-term one since it is flexible for selecting
suitable segments from the entire sequence. The
speaker embedding is also used in the decoding
process, which is also fed into pre-net as previous study suggested (Chien et al., 2021). Finally,a self-attention layer is added for alleviating thelong-term dependencies to sequentially propagate
information over long distances (Lin et al., 2017).

175

2.3

176

Originally in the Tacotron framework, a post-net
is consisting of 5 convolution layers to predict
residuals for improving the reconstructed Melspectrogram. However, inspired from (Kaneko
et al., 2017), we substituted a post-filter network forpost-net to see if the speech quality could be further
improved. Here, we adopted the Diffwave (Konget al., 2020), which is a non-autoregressive model
based on Markov chain to gradually convert a simple distribution (e.g., white noise) into complex
distributions (e.g., waveform), and it served as thepost-filter and for our system. The output Melspectrogram from decoder is served as the input
for training DiffWave model. Finally, a original

163
164
165
166
167
168
169
170
171
172
173

177
178
179
180
181
182
183
184
185
186
187
188
189

190

3

Experiments

191

3.1

Data and experiment setup

In our implementation, we used AISHELL-3 (Shiet al., 2020) as the training corpus, which consistsof 218 speakers. In the training process, we randomly select 173 speakers (which is about 75% of
the corpus) and 100 utterances for each speakeras the training set. We down-sampled the sampling rate to 22,050 Hz and extract 80-dims Melspectrogram as the acoustic feature. The frame sizeand step size are 1,024 and 256 samples, respectively. Pitch range is set from 20 to 8,000 Hz.
For speaker embedding training, we followed theprovided parameters in (Chou et al., 2019; Cooper
et al., 2020) and set the speaker embedding sizeas 128. The speaker embedding for each speakerhas been obtained by averaging the speaker embeddings calculated for all utterances of each speakerin the training set. For multi-speaker Tacotron TTS
model training, both encoder outputs are set to 128dims. In the decoding process, the dimension of theprevious frame is increased from 80 to 256 before
entering pre-net, therefore we also increased thedimension of the speaker embedding to 256 anduse Softsign activation function before feeding into
pre-net.
Finally, for the post-filter training, the decoded
Mel-spectrogram is inputted into DiffWave modeltraining process. We constructed a pseudo-parallel
corpus that has the same textual information as thetraining set in order to let DiffWave learns the diffusion probabilistic model for generating speechutterance similar to original speech utterances. Table 1 shows the training parameters for each modules used in this study. For the speaker embedding
training, LDE is denoted as Spkrsv and VAE isdenoted as Spkrvc , respectively.
In order to compare both models in the same situation, the batch size and total training step are set
to equal. Tacotron baseline is the original Tacotron
framework without self-attention while speaker embedding is inputted into the same place as describedin Section 2. Tacotron + Spkrvc/sv is the proposedsystems with different speaker embeddings.
These two systems are also used the same training setup to evaluate the effectiveness of selfattention. However, during model training, theTacotron baseline, without self-attention, could not
be trained properly, therefore is neglected for fur-

Figure 2: The modified framework for Tacotron decoderwith speaker embedding.

161

WaveNet vocoder is used for generating waveform.

Decoding process

Post-net and Post-filter

3

192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239

240
241
242
243
244
245
246
247
248

ther evaluations and also showed that self-attention
is necessary when speaker embedding is trained
jointly. Finally, the post-filter is trained based on
the original paper, however, since the goal of the
post-filter is to generate speech utterances as similar to the natural speech as possible. Therefore,
we set it to 320k based on our preliminary evaluations, which is slightly smaller than the setup of
the original paper.
Model
SpkrvcSpkrsv
Tacotron baseline
Tacotron + Spkrvc/sv
Post-filter

Batch size
32
32
64
64
16

Total step
1M
1M
99k
99k
320k

Table 1: Model training parameters

249

3.2

Figure 4: Original Mel-spectrogram (top figure), Melspectrogram generated from Tacotron + Spkrvc (middle
figure), and generated from Tacotron + Spkrvc + Postfilter (bottom figure).

Evaluations

evaluate the acoustic feature distance between thegenerated speech sequences and the forced aligned
natural speech sequences.
From the results from Table 2, the system using Spkrvc slightly outperformed the one using
Spkrsv . We also conduct subjective listening tests
for evaluating speaker similarity and speech naturalness. Here, a 5-point mean opinion score (MOS)
tests were held and 19 native Mandarin speakers
were participated. The same set of 50 testing utterances are used here. For speaker similarity test,
each participant is presented with several original
sentences uttered by the target speaker and asked
to score the speaker similarity of generated speechutterances. Table 3 shows that the system with
Spkrvc generated speech utterances with betterspeaker similarity. Note that for comparing speech
quality, only Spkrvc is used since it outperformed
Spkrsv in speaker similarity test. These results
imply that using the transfer learning of speakerverification task could neglect some useful information for generating natural speech while the VAEbased method, which is a generative model, is moresuitable for TTS task.
For evaluating the post-filter, we used the Resemblyzer tool to visualize the output speaker embeddings of the testing 300 utterances from 10 unseen
male and female speakers. In fig. 3, one can seethat post-filter indeed generate similar embeddings

Figure 3: Visualization of speaker embeddings (Spkrvc
is used here) that are generated w/ post-net and w/ postfilter. upper plot shows female distributions and lower
plot shows male ones.
250
251
252
253
254
255
256

First evaluation is comparing the performanceof two speaker embeddings by calculating the distortion between the generated speech and the synthesized speech, the Mel-cepstral distortion (MCD)
is used. Here, we randomly select 5 unseen speakers from the testing set and used 10 utterances foreach speaker. MCD is an objective measurement to
4

257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285

286
287
288
289
290
291
292
293
294
295
296
297
298
299

to the natural ones than the post-net. Besides, female speakers also have denser results comparedto males. This could be the result of more female
speakers in the AI-SHELL corpus (around 80%).
From these results, it could be a promising finding that incorporating speaker embedding trained
from the voice conversion task is more helpful thanthe one from the speaker verification task for generating speech utterances of unseen speakers with
only few speech data. Fig. 4 also shows an example of post-filter and post-net. One can see thatthe generated Mel-spectrogram is more detailed
using post-filter. Demo wave files are showed in
our page1 .
EmbeddingSpkrvcSpkrsv

Post-filter
Yes
No
Yes
No

MCD [db]
9.62±0.42
10.16±0.53
9.29±0.85
10.29±0.72

Table 2: Mel-cepstral distortion (with 95% confidence
interval). Note that conventional post-net is used whenpost-filter is not.

EmbeddingSpkrvcSpkrsv

Post-filter
Yes
No
Yes
No

Similarity
3.75±0.71
2.70±0.41
3.51±0.32
2.31±0.32

N/A

300

4

301

In this study, we constructed a multi-speaker endto-end Mandarin TTS system by integrating different speaker embeddings. We further enhanced
speaker similarity and speech naturalness by using
a post-filter network. This system is capable forgenerating speech utterances of unseen speakers
using only few speech utterances. In our experiments, usually 5 utterances are enough. We also
discovered that the speaker embedding trained fromgenerative method is more suitable for TTS task
comparing to the discriminative method. Future
work will focus on augmenting additional features
such as prosody or articulatory features for improving performance of the personalized TTS system.

303
304
305
306
307
308
309
310
311
312
313
314

Conclusion

1315

Chung-Ming Chien, Jheng-Hao Lin, Chien-yu Huang,
Po-chun Hsu, and Hung-yi Lee. 2021. Investigating
on incorporating pretrained and learnable speakerrepresentations for multi-speaker multi-style text-tospeech. In ICASSP 2021, pages 8588–8592. IEEE.

316
317
318
319
320

Ju-chieh Chou, Cheng-chieh Yeh, and Hung-yi Lee.
2019. One-shot voice conversion by separating
speaker and content representations with instancenormalization. arXiv preprint arXiv:1904.05742.

321
322
323
324

Erica Cooper, Cheng-I Lai, Yusuke Yasuda, Fuming
Fang, Xin Wang, Nanxin Chen, and Junichi Yamagishi. 2020. Zero-shot multi-speaker text-tospeech with state-of-the-art neural speaker embeddings. In ICASSP 2020-2020 IEEE International
Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6184–6188. IEEE.

325
326
327
328
329
330
331

Xun Huang and Serge Belongie. 2017. Arbitrary styletransfer in real-time with adaptive instance normalization. In Proceedings of the IEEE International
Conference on Computer Vision, pages 1501–1510.

332
333
334
335

Ye Jia, Yu Zhang, Ron J Weiss, Quan Wang, Jonathan
Shen, Fei Ren, Zhifeng Chen, Patrick Nguyen,
Ruoming Pang, Ignacio Lopez Moreno, et al. 2018.
Transfer learning from speaker verification to multispeaker text-to-speech synthesis. arXiv preprint
arXiv:1806.04558.

336
337
338
339
340
341

Quality Hirokazu Kameoka, Takuhiro Kaneko, Kou Tanaka, and3.70±0.5
Nobukatsu Hojo. 2018. Stargan-vc: Non-parallel
many-to-many voice conversion using star generative
2.67±0.35

Table 3: The subjective MOS results (with 95% confidence interval).

302

References

https://bit.ly/3uwCleM

5

adversarial networks. In 2018 IEEE Spoken Language Technology Workshop (SLT), pages 266–273.
IEEE.

342
343
344
345
346
347

Takuhiro Kaneko, Hirokazu Kameoka, Nobukatsu Hojo,
Yusuke Ijima, Kaoru Hiramatsu, and Kunio Kashino.
2017. Generative adversarial network-based postfilter for statistical parametric speech synthesis. In
ICASSP2017, pages 4910–4914. IEEE.

348
349
350
351
352

Zhifeng Kong, Wei Ping, Jiaji Huang, Kexin Zhao, andBryan Catanzaro. 2020. Diffwave: A versatile diffusion model for audio synthesis. arXiv preprint
arXiv:2009.09761.

353
354
355
356

Adrian Łańcucki. 2021. Fastpitch: Parallel text-tospeech with pitch prediction. In ICASSP 2021-2021
IEEE International Conference on Acoustics, Speechand Signal Processing (ICASSP), pages 6588–6592.
IEEE.

357
358
359
360
361

Naihan Li, Shujie Liu, Yanqing Liu, Sheng Zhao, andMing Liu. 2019. Neural speech synthesis with transformer network. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages
6706–6713.

362
363
364
365
366

Zhouhan Lin, Minwei Feng, Cicero Nogueira dos Santos, Mo Yu, Bing Xiang, Bowen Zhou, and Yoshua
Bengio. 2017. A structured self-attentive sentence
embedding. arXiv preprint arXiv:1703.03130.

367
368
369
370

371
372
373
374
375

Kaizhi Qian, Yang Zhang, Shiyu Chang, Xuesong Yang,
and Mark Hasegawa-Johnson. 2019. Autovc: Zeroshot voice style transfer with only autoencoder loss.
In International Conference on Machine Learning,
pages 5210–5219. PMLR.

376
377
378
379

Yi Ren, Chenxu Hu, Xu Tan, Tao Qin, Sheng Zhao,
Zhou Zhao, and Tie-Yan Liu. 2020. Fastspeech
2: Fast and high-quality end-to-end text to speech.
arXiv preprint arXiv:2006.04558.

380
381
382
383

Yao Shi, Hui Bu, Xin Xu, Shaoji Zhang, and Ming
Li. 2020. Aishell-3: A multi-speaker mandarin
tts corpus and the baselines.
arXiv preprint
arXiv:2010.11567.

384
385
386
387
388
389

David Snyder, Daniel Garcia-Romero, Gregory Sell,
Daniel Povey, and Sanjeev Khudanpur. 2018. Xvectors: Robust dnn embeddings for speaker recognition. In 2018 IEEE International Conference on
Acoustics, Speech and Signal Processing (ICASSP),
pages 5329–5333. IEEE.

390
391
392
393
394

Daisy Stanton, Yuxuan Wang, and RJ Skerry-Ryan.
2018. Predicting expressive speaking style from text
in end-to-end speech synthesis. In 2018 IEEE Spoken Language Technology Workshop (SLT), pages
595–602. IEEE.

395
396
397
398
399

Shinnosuke Takamichi, Tomoki Toda, Graham Neubig, Sakriani Sakti, and Satoshi Nakamura. 2014.
A postfilter to modify the modulation spectrum in
hmm-based speech synthesis. In ICASSP2014, pages
290–294. IEEE.

400
401
402
403
404

Yuxuan Wang, RJ Skerry-Ryan, Daisy Stanton, Yonghui
Wu, Ron J Weiss, Navdeep Jaitly, Zongheng Yang,
Ying Xiao, Zhifeng Chen, Samy Bengio, et al.
2017. Tacotron: Towards end-to-end speech synthesis. arXiv preprint arXiv:1703.10135.

405
406
407
408
409
410
411

Yusuke Yasuda, Xin Wang, Shinji Takaki, and Junichi
Yamagishi. 2019. Investigation of enhanced tacotron
text-to-speech synthesis systems with self-attention
for pitch accent language. In ICASSP 2019-2019
IEEE International Conference on Acoustics, Speechand Signal Processing (ICASSP), pages 6905–6909.
IEEE.

6