日本音響学会 2024年春季研究発表会 1-2-8

Emotion-controllable Speech
Synthesis using Emotion Soft Labeland Word-level Prominence☆Xuan Luo, Shinnosuke Takamichi, Yuki Saito, Hiroshi Saruwatari
Graduate School of Information Science and Technology,
The University of Tokyo, Japan.

1Research Topic• Topic: Enable both emotion and word-level prominence control• Emotion and intention (expressed by prominence) consist of importantparalinguistic information• Prominence is a similar concept to emotion strength but they are
different

Emotion strength• Metrics for emotion

VSProminence• Metrics for intention• Only appears in words with emotion• Possibly appears in every word

• How strong I am feeling• How important these words are• Example: Why is it so spicy?

• Example: We should focus more on products.
2

Related research• Condition TTS model on emotion
label and strength [Y. Lei et al. 2021]

Emotion control [Y. Lei et al. 2021]
Happy
NeutralAngry
Text

Speech synthesis

Only emotion and emotion strength control• Condition TTS model on word-levelprominence [H. Li et al. 2018]

Prominence control [H. Li et al. 2018]
Prominence Go back home. Word-level
Text

Speech synthesis

E.g., Go back home.

promience
score

Only word-level prominence control

However, none of them can control emotion and prominence at the same time.
3

Proposed method and Result• Proposed method• A two-stage emotion controllable TTS model that allows emotion and word-levelprominence control using emotion soft labels and prominence.

• Result• 51% emotion-distinguishable accuracy on 3 emotions• Fair emotion discrimination ability for synthesized speech• 0.95 linear controllability on prominence• Strongly linear controllability of word-level prominence• 3.9 MOS score on naturalness• Comparable to previous research4Concept of the proposed two-stage controlling TTS model• Stage 1: Condition on emotion soft label• Emotion soft label ranges from 0 to 1
• Predict basic-conditioning prominence

Happy + Text
Angry

NeutralPredicted
basic-conditioning
prominence
0.3 0.4 0.3

Stage 1

E.g., Go back home…

Fine-conditioning
Prominence• Stage 2: Condition on fine-conditioning+0.1 +0.1 +0.3
0.4 0.5 0.6
prominence
E.g., Go back home…
Stage 2• Fine-conditioning prominence ranges from 0 to 1 E.g., Go back home…

• Basic-conditioning prominence(step1) +fine-conditioning prosodic feature bias (step 2)
are summed up for conditioning TTS control

Text+Speech synthesis

Prosodicfactors
Our model : Enable both emotion and word-level prominence control

5Architecture of proposed model (Training)
SER: Speech emotion recognizer• Estimates emotion soft labels
Text

Encoder
Multi-modal
embeddings
𝑥𝑥𝑚𝑚𝑚𝑚𝑚𝑚

LSTM• Estimates basic-conditioning prominence

Attention

SER PP

Utterance-level 𝑥𝑥𝑝𝑝𝑝𝑝𝑝𝑝
prosodic factors
𝑥𝑥𝑝𝑝𝑝𝑝𝑝𝑝Word-levelprominence
𝑥𝑥𝑤𝑤𝑤𝑤𝑤𝑤Text

PP: Prominence predictor(1)𝑝𝑝𝑒𝑒𝑒𝑒𝑒𝑒

Emotionsoft label

Decoder Mel-spectrogram
Tacotron2

𝑥𝑥𝑝𝑝𝑝𝑝𝑝𝑝

Fine-conditioning
prominence^
𝑥𝑥𝑝𝑝𝑝𝑝𝑝𝑝
Word-level
LSTM
prominence
Basic-conditioning
predictor
prominence(2)

𝑏𝑏𝑝𝑝𝑝𝑝𝑝𝑝

The proposed SER and PP models enabled both emotion and prominence control

6

Multi-modal features for SER• Word-level prominence• Weighted sum of CWT (continuous wavelettransform) amplitudes which are on the linesof LoMA (maximum amplitude) at different scales[A. Suni et al. 2017]• Text: fastText embedding [Bojanowski et al. 2017]

Signal• Mean, standard deviation, and range of pitchand energy [Sahu. 2019]0.120.25
0.5

Wavelet scale (Hz)• Utterance-level prosodic factors

f0endursum

1.0
2.0
4.0
8.0

16.0
0.00

0.25

0.50

Line of LoMA 70.75

Controlling example (Inference)• Emotion control• Don’t disappoint your public (Angry)• Don’t disappoint your public (Sad)• Emotion and prominence control

Text

Encoder

Attention𝑥𝑥𝑝𝑝𝑝𝑝𝑝𝑝

PP(1)𝑝𝑝𝑒𝑒𝑒𝑒𝑒𝑒• Sad: They forcefully (+0.3) keep them at aEmotion
black hotel.
soft label• Sad: They forcefully keep them at a blackhotel (+0.3).

Decoder

LSTM

Melspectrogram
Tacotron2(2)Fine-conditioning
prominence^𝑥𝑥𝑝𝑝𝑝𝑝𝑝𝑝
Word-levelprominence
Basic-conditioning
predictor
prominence
𝑏𝑏𝑝𝑝𝑝𝑝𝑝𝑝

8

Experiment setup• Data• IEMOCAP [Busso+08]: Used for pre-training the SER and PP models (12 hours)• Blizzard2013 [King+14]: Used for training proposed TTS model (75 hours)• Emotion labels• Emotion: Angry/Sad/Neutral• Emotion labels in Blizzard2013 are predicted by SER pre-trained on IEMOCAP• Backbone TTS model: Tacotron2 [Shen et al. 2018]• Vocoder: Parallel WaveGAN [Yamamoto et al. 2020]9• Evaluation purpose• Whether the emotion can be controlled.

• Experiment set• 10 utterances for each of angry, neutral, and sad• 10 sets of ang/neu/sad with the same sentence• Each of 50 evaluators was required to listen 10sets and select an utterance with a given emotion• Result

Emotion-distinguishable performance

Evaluation1: Controllability of emotion soft labels
0.5
0.4
0.3
0.2
0.1

0.0• Accuracy, precision, recall, and F1-score were51%, 52%, 50%, and 51% on average of 3 emotions.
• Specifically, the accuracy of angry speech was 60%-> Fair emotion discrimination ability for synthesized speech

Angry

NeutralSad

10

Evaluation2: Controllability of word-level prominence
NOUN0.3• Evaluation purpose• Whether the prominence can be linearly
controlled0• Experiment set-0.3• Experiment with content words• Set 7 fine-conditioning prominence biases ranged
from −0.3 to 0.3 by 0.1 step• 50 sentences and total 2,100 synthetic speech

Measured bias

0.3

• Result0-0.30.3• Average PCC (Pearson Correlation Coefficient)scores are 0.93 (angry), 0.97 (neutral), 0.96 (sad)• Strong linear controllability on the prominence of
the NOUN, VERB, ADJ, and ADV words.

0-> Strongly linear controllability of word-level prominence-0.3-0.3

00.311Fine-conditioning bias on prominenceEvaluation3: Subjective test of word-level prominence
controllingNOUN5• Whether the quality of synthetic speech is good

• Experiment set• 10 speech audio• 50 listeners to evaluate MOS on naturalness• Result• Shows equal performance (MOS = 3.9) synthetic
speech quality equal to the conventional method[H. Li et al. 2018]

MOS on naturalness• Experimental purpose

432

1-0.3

00.3

Fine-conditioning bias on prominence
12Summary• Purpose• Enable emotion and word-level prominence control• Model• A two-stage emotion controllable TTS model that allows emotion and word-levelprominence control using emotion soft labels and prominence.

• Result• 51% emotion-distinguishable accuracy• 0.95 linear controllability on prominence• 3.9 MOS score on naturalness• Future work

• Better emotion-distinguishable speech• Towards phoneme-level prominence control13

Appendix

14

Evaluation2: Controllability of word-level prominence• Evaluation purpose• Whether the prominence can be linearly controlled• Experiment set• Experiment on content words• Set 7 fine-conditioning prominence biases ranged
from −0.3 to 0.3 by 0.1 step• 50 sentences and totally 2,100 synthetic speech• Result• Average PCC (Pearson Correlation Coefficient)scores are 0.93 (angry), 0.97 (neutral), 0.96 (sad)• Strong linear controllability on the prominence of
the NOUN, VERB, ADJ and ADV words.

-> Strongly linear controllability of word-level prominence
15

Evaluation3: Subjective test of word-level prominence
controlling• Experimental purpose• Whether the quality of synthetic speech is good

• Experiment set• 10 speech audio• 50 listeners to evaluate MOS on naturalness• Result• Shows equal performance (MOS = 3.9) synthetic
speech quality equal to the conventional method[H. Li et al. 2018]-0.300.316Evaluation3: Prominence contours when conditioning
One example!
on word-level prominence

X, Y description• Prominence contours of fineconditioned words increased
when the conditioning bias
increased• Prominence contours of fineconditioned words decreased
when the conditioning bias
decreased
17

Prominence

18