DNN-based Speaker Embedding Using Subjective Inter-speaker Similarity for Multi-speaker Speech Synthesis
Yuki Saito, Shinnosuke Takamichi, and Hiroshi Saruwatari (The University of Tokyo, Japan)
1. Research highlights

3. Proposed speaker embedding Purpose: learning speaker representation that is correlatedwith human speech perception

 Large scale scoring of subjective inter-speaker similarity
 Crowdsourcing the similarity scores involving 4,000+ listeners
 # of listeners per one speaker pair = at least 10

 Approach: using crowdsourced subjective inter-speakersimilarity scores for training speaker embedding model

To what degree do the twospeakers' voices sound similar?
(−3: dissimilar ～ +3: similar)

1. is highly correlated with the similarity scores
2. improves speech quality in multi-speaker speech synthesis

 Visualization of the obtained scores
Graph representation(speaker similarity graph) One-hot speaker code[1]Spkr. embedding
Kernel in Sim. (mat)Matrix representation(speaker similarity score matrix 𝐒 ) Correlation coef. betw. similarity scores & values of kernel

 Similarity vector embedding 𝑁d -dim. continuous vector derived from speaker recognition
 Applications: speaker verification & voice conversion[3] Training: minimizing speaker recognition loss (cross-entropy)
Spkr.
Spkr. recog. code
1loss𝒔Embedding
nets.

(vec)
𝐿SIM𝒔 Sim. scorevector(vec)
𝐿SIM 𝒔, 𝒔 Pros: high scalability
 Low-dim. representation that can deal with unseen speakers
 Cons: still low interpretability
 Speaker recognition ≠ human speech perception
Distance betw. speakers≠Subjective inter-speakersimilarity
Embedding space

Can we construct an embedding space that preserves
subjective inter-speaker similarity?

−1(mat)
𝐿SIM 𝐊 𝐃 , 𝐒00+100+1

Values of kernel

Grammatrix
𝐊𝐃Phonetic
ai posteriorgram

Speechrecognition

1=𝒔−𝒔 ⊤ 𝒔−𝒔
𝑁s

Calc.
kernel𝑘 ⋅,⋅1

 Acoustic model: variational-autoencoder[7] incorporating pretrained speech recognition & speaker embedding[3] Training DNN using the similarity score matrix 𝐒 as a constraint
on coordinates of speaker embedding

Embedding
nets.

Sim.
(mat-re) Evaluation in DNN-based multi-speaker speech synthesis

00Predictionnets.

Sim.
(mat)

Our algorithms learn speaker embedding that is
highly correlated with the similarity scores!

 Similarity matrix embedding

Embedding
nets.

Sim.
(vec)

+1−1−1𝐒Prediction
nets.

Train-Eval. Train-Train

We propose training methods for speaker embedding w/ the matrix 𝐒.

Similarity scores100 Training DNN to predict a similarity score vector 𝒔 ∈ 𝐒

Embeddingnets.

Sigmoid kernel: 𝑘 𝒙, 𝒚 = tanh 𝒙⊤ 𝒚d-vec.

𝑑-vector[2]𝑑-vecs.

Dataset(16 kHz sampling)

01001Speechparams.

JNAS[4] 153 Japanese females
Training (seen) data: 140 females except for F001—F013
Evaluation (unseen) data: 13 females (F001—F013)
STRAIGHT[5]1—39 dim. mel-cepstra (including Δ)
Feed-Forward (see our paper for details)(1) d-vec. : trained by speaker recognition(2) Sim. (vec): trained by similarity vector embedding(3) Sim. (mat): trained by similarity matrix embedding(4) Sim. (mat-re): trained by (3) w/o dissimilar spkr. pairs
8-dimensional vectorMethods

2. Conventional speaker embedding Experimental conditions

Vocoder
DNN input
DNN architecture

Instruction of the scoring Results: obtaining speaker embedding that

 𝑁s -dim. discrete vector (ID for pre-stored 𝑁s speakers)
 Pros: high simplicity when 𝑁s is small
 Cons: low interpretability & scalability Distance betw. speakers = constant Sparse representation that cannot deal with
0
unseen speakers

4. Experimental evaluation

Embedding-derived interspeaker similarity matrix(mat)
𝐿SIM𝑍s : normalization term
𝐊 𝐃 , 𝐒: matrices the same as 𝐊 𝐃 , 𝐒 w/o their diagonal components[5] H. Kawahara et al., Speech Communication, 1999. [6] J. Duchi et al., Journal of Machine Learning Research, 2011. [7] D. P. Kingma et al., arXiv, abs/1312.6114, 2013. [8] L. Sun et al., Proc. ICME, 2016.

SSW10 @ Vienna, Austria

Latentvars.

Embeddingnets.

𝒙
Decoder

Encoder

Generated
speechparams.

Spkr.
embedding Results of preference AB/XAB tests on synthetic speech quality
 A > B: A is significantly higher than B (𝑝 < 0.05)
 A ≒ B: there is no significant difference betw. A & BA vs. B
d-vec. vs. Sim. (vec)d-vec. vs. Sim. (mat)d-vec. vs. Sim. (mat-re)

12=
𝐊𝐃 − 𝐒 𝐹𝑍s[References] [1] N. Hojo et al., IEICE Trans. on Information and Systems, 2018. [2] E. Variani et al., Proc. ICASSP, 2014. [3] Y. Saito et al., Proc. ICASSP, 2018. [4] K. Itou et al., Journal of the ASJ (E), 1999.

𝐒

Naturalspeechparams.

uNaturalness(preference AB)
A＞B B>A A≒B
0121076085Spkr. similarity(preference XAB)
A＞B B>A A≒B0103445427(1) Sim. (vec) improves speech quality in almost all spkrs.
(2) Sim. (mat) degrades speech quality in some spkrs.

[Acknowledgements] This research and development was supported by the SECOM Science and Technology Foundation,
JSPS KAKENHI Grant Number 18J22090 and 17H06101, the MIC/SCOPE #182103104, and the GAP foundation program of the UTokyo.

©Yuki Saito, Sep. 20, 2019.