Perceptual-Similarity-Aware Deep Speaker Representation Learning for Multi-Speaker Generative Modeling SPE-3.5
Yuki Saito, Shinnosuke Takamichi, and Hiroshi Saruwatari (The University of Tokyo, Japan)
1. Overview

2. Methods

Introduction◼ Speaker Embedding (SE) Distributed representation of spkr. ID
 Primal usage: spkr. discriminative taskse.g., spkr. recognition / verification
 Our focus: spkr. generative taskse.g., Text-To-Speech (TTS) / Voice Conversion (VC)

◼ Deep speaker representation learning DNN-based technology for learning SEs◼ d-vector[1] & x-vector[2]→ Designed for discriminative tasks
Research question: How can we learn SEs
suitable for spkr. generative tasks?

◼

One-hot speaker code[3] 𝑁s -dim. discrete vector(i.e., ID for pre-stored 𝑁s spkrs.)
 High simplicity when 𝑁s is small 0 Low interpretability & scalability 01◼ Distance b/w spkrs. = always constant◼ Undefined for unseen spkrs.

Speechparams.

 Trained to represent perceptual similarity
among speakers → high interpretability 3 algorithms based on crowdsourced
perceptual similarity score matrix
 Human-in-the-loop Active Learning (AL) for
high cost-efficiency
Similar → closer &Dissimilar → further) → Similar) → Dissimilar) → Dissimilar Crowdsourcing similarity scores involving 4,000+ listeners◼ 1 speaker pair → scored by more than 10 listeners

100

Instruction of the scoring

Spkr. recog.
loss

d-vecs.

Embedding
DNN

Prediction
DNN,,,Matrix representation(speaker similarity score matrix 𝐒 )◼ Algorithm 1: similarity vector embedding(vec)
𝐿SIM

EmbeddingDNN) → +2) → −3) → −2

 Visualization of obtained similarity scores among 153 jp spkrs.

𝐒Prediction
DNN(vec)

𝐿SIM 𝒔, 𝒔ො =1𝑁s𝒔ො − 𝒔 ⊤ 𝒔ො − 𝒔◼ Algorithm 2: similarity matrix embedding

Graph representation(speaker similarity graph 𝐺)

SE-derived similarity b/w
𝐒(mat)𝐿SIM

Calculating
Grammatrix

Embedding
DNN&𝑍s : normalization coefficient(mat)

𝐿SIM

100𝐒, 𝐊 𝐃 =1𝑍s

𝐊 𝐃 − 𝐒 2𝐹

◼ Algorithm 3: similarity graph embeddingG◼ Learning SEs based on similarity scoresEmbedding
DNN

 Training embedding DNN to predict perceptual similarity
from SEs (i.e., human-oriented machine learning) Algorithm: similarity { vector, matrix, graph } embedding High scalability◼ Low-dim. SE that can deal with unseen spkrs.
 Still low interpretability◼ Spkr. recog. ≠ human speech perception(i.e., similar speaker-pairs → further) Uninterpretable SE leads to...
◼ Less controllability & adaptability inspeaker generative tasks

SE space(((To what degree do these two speakers'
voices sound similar?
(−3: dissimilar ～ +3: similar) 𝑁d -dim. continuous vector learned by spkr. recog.
 Training objective: spkr. recog. loss
(e.g., softmax cross-entropy or GE2E loss[4])◼ Perceptual-similarity-aware SE,,,◼ Large scale scoring of perceptual speaker similarity

010◼ d-vector[2]Summary(((Proposed method: Perceptual-similarity-aware SE learning

Traditional speaker representations

SEs
Embedding
DNN

Prediction
DNN

SE-derived
Human-derived
similarity Loss to predict similarity
similarity
Vector Matrix GraphSE space(graph)
𝐿SIM

Linkprediction

graph

𝐿SIM𝑎𝑖,𝑗

1: exist link𝑝𝑖,𝑗 = exp − 𝒅𝑖 − 𝒅𝑗𝐒

2
2: probability of link[5]𝒅𝑖 , 𝒅𝑗 = −𝑎𝑖,𝑗 log 𝑝𝑖,𝑗 − 1 − 𝑎𝑖,𝑗 log 1 − 𝑝𝑖,𝑗

◼ Human-in-the-loop AL

 Iterating SE learning & perceptual similarity scoring→ learn better SEs while reducing costs
 Query selection (priority annotation for unscored data)◼ Query strategies: { Higher, Middle, Lower }-Similarity First (SF)

3. Experimental evaluations & results

Kernel for Sim. (mat)Sigmoid kernel: 𝑘 𝒙, 𝒚Dataset(16 kHz sampling)= tanh 𝒙⊤ 𝒚

Evaluation1: interpretability of SEs◼ Scatter plots of human-/SE-derived similarity (𝑟: PCC)
Sim.
(vec)

Seen-Unseen Seen-Seen

Human-derived

d-vec

Sim.
(mat)

Sim.
(graph)1001

 d-vec never considers
perceptual similarity.
 Sim. (*) achieves higher
PCC than d-vec!
◼ Sim. (graph) works
the best among the3 algorithms.

SE-derived◼ AUC of similar speaker-pair detection using similarity b/w SEs
d-vec Sim. (vec) Sim. (mat)Sim. (graph)Seen-Seen

0.570.69

0.89

0.92

Seen-Unseen

0.64

0.69

0.77

0.82

[References]

 Sim. (graph) achieves
the highest AUC!

◼ Acoustic model: VAE[8] using pre-trained speech recog. & SE[9]Phonetic
a posteriorgram[10]iu
Latentvars.

Speech recog.
DNN
Naturalspeechparams.

Decoder

Encoder

Embedding
DNN

Generated
speechparams.

SE◼ Quality of speaker adaptation (AB/XAB tests w/ 25 listeners)
 A > B: A is significantly better than B (𝑝 < 0.05).
 A ≒ B: there is no significant difference b/w A & B.
Naturalness
A vs. Bd-vec vs. Sim. (vec)d-vec vs. Sim. (mat)d-vec vs. Sim. (graph)

A＞B

B>AA≒B

00012713160Spkr. similarity
A＞B B>A A≒B
040

10412LSF

HSF

Sim. (vec)

MSF

PS (50%)Sim. (mat)0.90

FS (100%)

Sim. (graph)

0.94
0.92
0.90

0.85

0.88

0.80

0.86
0.84

0.75

0.82

0.78

0.82

0.76

0.80

0.74

0.78

0.72

0.76

0.60
13060
90
115(50%) (62.5%) (75%) (87.5%) (100%)◼ Quality of synthetic speech (DMOS test on spkr. similarity w/ 50 listeners)

 Sim. (vec / graph) significantly improve speech quality foralmost all spkrs., but Sim. (graph) can degrade the quality.

◼ Controllability of speaker interpolation
 Horizontal axis: interpolation coefficient Vertical axis: spkr. similarity score Proposed SEs (blue lines) also improvethe controllability of d-vec (grey line)!

 Best query strategy: MSF
 AL in Sim. (vec / graph):
0.70◼ AUC comparable with 0.65
0.60
FS, while reducing
learning/scoring costs! 0.55
 AL in Sim. (mat):
0.50◼ Degraded AUC in Seen- 0.72
Unseen spkr.-pair case 0.68
→ due to high data0.64
dependency?

AL iteration (w/ % of observed scores)

3511◼ AL setting: Fully Scored (FS), Partially Scored (PS)◼ Curves of similar speaker-pair detection AUC

Seen-Seen

Vocoder
DNN input
DNN architecture
Compared SEs(8-dim.)

JNAS[6] including 153 Japanese female spkrs.
Training (seen) data: 140 females except for F001—F013
Evaluation (unseen) data: 13 females (F001—F013)
STRAIGHT[7]1—39 dim. mel-cepstral coefficients (including Δ)
Feed-forward (see our paper for details)
d-vectors[1] vs. Proposed SEs(i.e., Sim. (vec / mat / graph) embedding)

Evaluation 3: cost-efficiency of human-in-the-loop AL

Seen-Unseen

Evaluation 2: quality of multi-speaker generative modelingExperimental conditions

 BOLD scores ≒ FS (𝑝 < 0.05)
 AL in Sim. (vec / graph):◼ DMOS comparable with
FS by fewer AL iterations!
 AL in Sim. (mat):◼ Lowest DMOS among 3
SEs & no improvement

Sim. (vec)Sim. (mat)Sim. (graph)(50%)

2.85±0.14

2.90±0.13

2.86±0.13

MSF (62.5%)

2.95±0.14

2.93±0.13

3.03±0.13(75%)

3.04±0.14

3.00±0.13

3.02±0.13(87.5%)

3.05±0.14

3.03±0.13

3.06±0.13(100%)3.14±0.14

2.98±0.13

3.08±0.14

PSFS◼ Our project page: http://sython.org/demo/JSPS-DC1/index.html
0[1] Variani et al., Proc. ICASSP, 2014. [2] Snyder et al., Proc. ICASSP, 2018. [3] Hojo et al., IEICE Trans. on Info. and Systems, 2018. [4] Wan et al., Proc. ICASSP, 2018. [5] Li et al., Proc. IJCAI, 2018.

 Spkr. similarity matrix for VCTK female spkrs. is also available!
[Acknowledgements] This research and development was supported by JSPS KAKENHI Grant Number 19H0116, 18J22090 and 19K18100, the MIC/SCOPE #182103104.

[6] Itou et al., Journal of the ASJ (E), 1999. [7] Kawahara et al., Speech Communication, 1999. [8] Kingma et al., arXiv, abs/1312.6114, 2013. [9] Saito et al., Proc. ICASSP, 2018. [10] Sun et al., Proc. ICME, 2016.

ICASSP2022 @ Singapore©Yuki Saito, May, 2022.