Emotion-controllable Speech Synthesis using Emotion Soft Label andWord-level Prominence∗☆ Xuan Luo, Shinnosuke Takamichi,
Yuki Saito, Hiroshi Saruwatari (The University of Tokyo)

1

Introduction

Text-to-speech (TTS) models aim to synthesize
human-like speech including linguistic and paralinguistic information. Current TTS models [1, 2]
can synthesize understandable speech from a linguistic perspective. On the other hand, synthesizing human-like speech with diverse paralinguistic information, such as emotion and prominence, is still
not an easy task. Conventional emotion-controllable
TTS studies enable emotion controllability by conditioning on explicit emotion labels [3, 4] or implicit emotion embeddings [5, 6]. To enable more diverse controllability, subsequent studies conditioned
on emotion strength which is usually obtained byan interpolation method [7] or a ranking function atutterance- or phoneme-level [3, 8].
However, emotion strength is a limited concept
to only emotional speech (not for normal speech),and it refers to the intensity of only emotion (notfor others, like intention), which limits its broader
use. In addition, predicting emotion strength ofspeech requires speciﬁc emotional datasets [3], which
also limits its application. On the other side, wordlevel prominence is a more general concept for emotional and normal speech. It refers to the perceptual quantity of standing out from other words [9],which can be calculated by speech processing techniques [10] without training on speciﬁc emotional
datasets. Previous controllable TTS model [11]conditions their TTS model on word/phoneme-levelprominence (i.e., emphasis) to enable diverse synthesized speech. However, such models cannot control emotion.
In this paper, we propose a two-stage emotioncontrollable TTS model that we can condition on
emotion soft labels and ﬁne-condition on wordlevel prominence, which conventional models cannot. Our proposed model extends the Tacotron2
model with a speech emotion recognizer (SER) anda prominence predictor (PP) to enable this dual controllability. In the ﬁrst stage, we condition on emotion soft labels predicted by the SER. In the second
stage, we ﬁne-condition on the prominence predictedby the PP model. The experiments achieved 1) 51%
emotion-distinguishable accuracy, and 2) 0.95 linear
controllability on prominence.

∗2

Proposed method

2.1

SER model

The SER model estimates emotion soft labels
which are used for the ﬁrst stage of control. It takes
multi-modal features as input because of better performance than single features [12]. The multi-modal
features consist of prosodic factors, prominence, andtextual features.
Utterance-level prosodic factors extraction
We extract pitch and energy contours of speech at
the frame level and calculate their means, standard deviations (SD), and range as utterance-levelprosodic factors (6-dimension) because they are expected to relate to speech emotion [12]. The pitch
contour is predicted using the pYIN algorithm [13],and the energy contour is calculated by the rootmean-square value of the magnitude of each frame.
Word-level prominence extraction We extract word-level prominence by using the linesof maximum amplitude (LoMA) in the continuous wavelet transform (CWT) of a sum of signal contours of pitch, energy, and duration withweights [10]. The CWT is expected to approximate
human processing of a complex signal relevant toprominence by resembling the perceptual hierarchical structures (phoneme, syllable, word) related toprosody. This ability is more diﬃcult to achievewith traditional spectrograms. The LoMA [14]are lines that can identify and quantify word-levelprominence by connecting nearby peaks in the CWTof the signal at diﬀerent scales. The strength ofthe line for each word is the word-level prominence
which is determined by the cumulative sum of scalevalues of the line with weights, shown as follows:
xprm = Ws (a0 , ti0 ,0 ) + . . . +(1)log(j + 1)a−j/2 Ws (a0 aj , tij ,j ),where xprm is word-level prominence, a0 denotes theﬁnest scale in CWT, a deﬁnes the spacing between
chosen scales, j denotes sale, tij ,j is a time point
where the local maxima occurred in the a0 aj scale.
Ws (a0 aj , tij ,j ) denotes the CWT amplitude in tij ,jtime point at a0 aj level scale.
Word-level textual feature extraction We extract word-level textual features by applying the
fastText [15], a word-level text embedding model,
to a text embedding.
We then concatenate prosodic factors xpsd , prominence xprm , and text embedding xwrd to multimodal features xmul as the SER input, shown as

単語レベル強調と感情ソフトラベルを利用した制御可能な音声合成モデル, 羅 旋, 高道 慎之介, 齋藤 佑樹, 猿
渡 洋 (東大院・情報理工)

Text

Encoder

Attention

Decoder

Melspectrogram

Baseline TTS model

SER

PP
(&)

𝑏!"#

Duplication

Utterance-level 𝑥!"#prosodic factors
𝑥!$%
Word-levelprominence
𝑥&$#

Text

Fig. 1

👈

𝑥"'(

(%)

LSTM
Multi-modal
embeddings𝑝!"#
Emotion
soft
label

^
𝑥!"#

LSTM

Basic-conditioning
prominence

Fine-conditioning
prominence

Word-levelprominence
predictor

The proposed emotion-controllable model

follows:
xmul = Concatwrd (xpsd , xprm , xwrd ).

(2)

where Concatwrd is concatenation at word-level.
The SER model architecture The SER model
is a 2-layer LSTM model followed by a softmax out(1)put layer. It estimates emotion soft labels pemo ,where superscript 1 indicates that it is used for theﬁrst stage of control. The emotion soft labels are theposterior probabilities for predicting the emotion labels yemo , conditional on multi-modal features xmul :p(1)emo = SER(xmul ) = P (yemo |xmul ).

(3)

The SER architecture is shown on the left below
side of Fig. 1. It is trained by minimizing the crossentropy loss LSER between the groud-truth emotion
labels and estimated emotion soft labels:
N ∑C∑
LSER = −yi,c log(pemoi,c ),(4)
i=1 c=1

where yi,c is an emotion label indicator, assigned 0
or 1, indicating whether the i-th utterance belongs
to the c-th emotion (1) or not (0). N and C are thetotal numbers of utterances and emotion categories,
respectively. pemoi,c is the estimated emotion softlabel of the i-th utterance for the c-th emotion.
2.2

PP modelThe word-level prominence predictor PPprm predicts basic-conditioning prominence x̂prm from aconcatenation of text embedding xwrd and emotion(1)
soft labels pemo :x̂prm = PPprm (xwrd , p(1)emo ),

model with the concatenated SER and PP models, as shown in Fig. 1. The SER model takes
multi-modal features as input and outputs emotion soft labels, which are fed into the PP model
along with text embedding. The PP model outputs
basic-conditioning prosodic factors and prominence,
which are then fed into the TTS decoder along withphoneme embedding from the TTS encoder.
We concatenate basic-conditioning prominence
and phoneme embedding by alignment with the English grapheme-to-phoneme conversion algorithm 1 .
The proposed TTS model emoTTS is conditionedon the concatenated embeddings ĉcon to synthesize
speech yspeech :(6)

where LPP indicates the L2 loss of and predicted
and ground truth prominence.
2.3

Emotion-controllable TTS Model

The emotion-controllable TTS model enables twostage control by extending the baseline Tacotron2ĉcon = Concatphn (ĉprm , xphn ),(8)where Concatphn is a concatenation of prominenceĉprm , and phoneme embedding xphn .
The conditioning prominence ĉprm comprises twoparts: basic-conditioning prominence x̂prm and ﬁneconditioning prominence (i.e., prominence bias or(2)
ﬁne-conditioning bias) bprm , shown in Eq. (9).
(2)
ĉprm = PPprm (xwrd , p(1)emo ) + bprm

(9)
= x̂prm + b(2)prm ,where PPprm is the word-level prominence predictor.
According to Eq. (7) and Eq. (9), the proposed
TTS model enables the inter-emotion and intraemotion control by converting emotion soft labels(2)(1)
pemo and ﬁne-conditioning prominence bprm to ĉprm .
The proposed TTS model is optimized by minimizing the additive loss Lemo TTS of LTacotron2 andLPP :Lemo TTS = LTacotron2 + LPP .

(10)

In inference, the proposed two-stage control TTS
model can synthesize speech in the following ways:
1. Enabling only the ﬁrst stage of control. Givenemotion soft labels, the proposed model can
synthesize speech with a speciﬁed emotion.
2. Enabling both the ﬁrst and second stages ofcontrols. Given emotion soft labels and ﬁneconditioning prominence, the proposed model
can synthesize speciﬁed emotional speech withslightly changed prominence.

(1)
where pemo is also the SER output in training andL(SER+PP) = LSER + LPP .

(7)

where

(5)

manually assigned in inference.
The PPprm also consists of a 2-layer LSTM network followed by an FC layer and a sigmoid layer insequence, as shown in Fig. 1.
The SER and PP models are jointly trained byminimizing the sum of the SER and PP losses
L(SER+PP) on an emotion-labeled dataset. The objective function is:yspeech = emoTTS(ĉcon ),3

Experimental setup

3.1

Data

We used the IEMOCAP corpus [16] for pretraining SER and PP models and the Blizzard2013
corpus [17] for training the proposed TTS model.
The IEMOCAP corpus has 12 hours of transcriptand speech, recording from emotional dialogues ofﬁve males and ﬁve females in both acting and improvising way. We randomly split it into 80 %1 The English grapheme-to-phoneme conversion package:link

3.2

Model parameter and features

The SER model consisted of a 3-layer LSTM network with 128 hidden units and a 128 × 3 fully connected (FC) layer, followed by a softmax activation.
The PP model included a 2-layer LSTM networkwith 128 hidden units and a 128 × 1 FC layer followed by a sigmoid activation function. It took a
303-dimensional joint vector as input and output a
1-dimensional prominence.
The backbone Tacotron2 consisted of an encoder
network that converted phoneme embedding into ahidden text representation and a decoder networkthat predicted mel-spectrograms from hidden text
and prominence with attention. Speciﬁcally, theencoder network consisted of 3-layer 1-dimensional
convolutions with 512 ﬁlters and a 5 × 1 window
size. A phoneme embedding represented by a 512dimensional vector was passed through the encoder
network whose output was a hidden text representation.

4

Evaluation

4.1

Controllability of emotion soft labels(ﬁrst stage of control)

We ﬁrst evaluated the emotion controllability ofour proposed model when conditioning on emotion
soft labels during the ﬁrst stage (inter-emotion) of
control. To obtain the perceived emotion of synthesized speech, we conducted a preference test inwhich each participant was required to choose angry, neutral, and sad speech, respectively, from a
set of three synthesized speech with angry, neutral, and sad emotions. We synthesized 10 utterances for each emotion (angry, neutral, and sad)as test speech from randomly selected sentences in
the BC2013 dataset by conditioning corresponding
emotion soft labels to 1.0. We applied the emotion
soft label to 1.0 for better representativeness [18].
This test was conducted on the Amazon Mechanical Turk with 50 participants and 10 sets of speechfor each participant. The performance of emotion

0.3

Angry

NOUN

VERB

ADJ

ADV

0.0

0.3

Measured bias

and 20 % for training and testing the SER and PP
models. The Blizzard2013 corpus contains emotionunlabelled emotional speech uttered by a single English speaker. We ﬁltered out only emotional speechpart for training and testing by the following approach. First, we selected character-speaking sentences surrounded by a single or double quotation
mark. Then, we ﬁltered out weak-emotional speechwhich is estimated with more than 0.8 score by the
SER model in each category. Finally, we required 3
human annotators to randomly listen to 100 speechin each emotional category of ﬁltered data and removed perceptually non-emotional categories. As aresult we obtained 28 hours of neutral and angry
speech and followed by splitting into 80 % and 20 %for training and testing the TTS model.

0.3

Neutral

0.0

0.3
0.3

Sad

0.0

0.3

0.3

0.0

0.3

0.3

0.0

0.3

0.3

0.0

0.3

0.3

0.0

0.3

Fine-conditioning bias on prominence

Fig. 2 Correlation between ﬁne-conditioning andobserved biases when ﬁne-conditioning on theprominence of NOUN, VERB, ADJ, and ADV
words for three emotions
controllability was evaluated by the accuracy, precision, recall, and F1-score which indicates the distinguishability for each emotion category. The results
demonstrated that the accuracy, precision, recall,and F1-score were 51%, 52%, 50%, and 51% on average of three emotions. Speciﬁcally, the accuracyof angry speech was 60% which was relatively higher
than other emotions.
4.2

Linear controllability of word-levelprominence (second stage of control)

We evaluated the linear controllability of our proposed model by ﬁne-conditioning on prominence
during the second stage (intra-emotion) of control. We deﬁned a linear controllability score using the Pearson Correlation Coeﬃcient (PCC) between the ﬁne-conditioning biases and measured biases for prominence. This can be represented by
PCC(bprm , b′prm ), where bprm is ﬁne-conditioning
biases and b′prm is measured biases indicating thediﬀerence in prominence between the synthesized
speech with ﬁne-conditioning bias and without bias
(ﬁne-conditioning bias = 0).
In the experiment, we found that word-levelprominence was distributed diﬀerently depending on
the part of speech in the training dataset where
the prominence of NOUN, VERB, ADJ, and ADV
words was distributed close to a normal distribution. Therefore, we only experiment on the NOUN,VERB, ADJ, and ADV words. To synthesize theevaluation speech, we also input 50 sentences selected from the BC2013 dataset, and for each sentence, we ﬁne-conditioned on the prominence of
NOUN, VERB, ADJ, and ADV words, respectively,
with seven biases for each, ranging from −0.3 to 0.3
with a 0.1 step, for angry (angry = 1.0), neutral(neutral = 1.0), and sad emotion (sad = 1.0). In
total, we synthesized 2, 100 speech samples.
As the result, the average PCC(bprm , b′prm ) scoreof angry (0.93), neutral (0.97), sad (0.96), and overall (0.95) emotions showed strong linear controllability on the prominence of the NOUN, VERB, ADJ,

-0 .300 .3

N OU N

2 .5

VE RB

ADJ

ADV

2 .0
1 .5
1 .0

P r o m in e n c e m a g n it u d e

0 .5
0 .0

An g ry

2 .5
2 .0
1 .5
1 .0
0 .5
0 .0

N e u t ra l

2 .5

thesize adequately emotion-distinguishable speech(emotion-distinguishable score = 51%), 2) linearlyﬁne-condition on the prominence of NOUN, VERB,
ADJ, and ADV words for the angry, neutral andsad emotions, and ﬁnally 3) synthesize speech withcomparable audio quality (MOS = 3.9) to that of
the conventional methods. In future, we plan to enable more emotion controllability and their better
combination with prominence.
Acknowledgement: This work is supported byMoonshot R&D Grant Number JPMJPS2011.

2 .0

References

1 .5
1 .0
0 .5

th p
em
at
th
e
b la
ck
ho
tel

tel

ho

T
for h e yce fku
e elly

bla
ck

ce

Th
ey
ce
f
k ue l l y
e
th p
em
at
th
e
fo r

f
kue l l y
e
th p
em
at
th
e
b la
c
ho k
tel

ey
Th

fo r

fo r

Th

ho
te

l

Sad
ey
ce
f
k ue l l y
ep
th
em
at
th
e
b la
ck

0 .0

Fig. 3 Prominence contours of an utterance synthesized by conditioning on the angry, neutral, andsad emotions and ﬁne-conditioning prominence on
NOUN, VERB, ADJ, and ADV words with threebiases (−0.3, 0, and 0.3) for each emotion. The
sample sentence is “They forcefully keep them at a
black hotel”. The NOUN, VERB, ADJ, and ADV
words correspond to “hotel”, “keep”, “black”, and“forcefully”, respectively.
and ADV words with p-value < 0.05. We also visualized the correlation between bprm and b′prm on thethree emotions, as shown in Fig. 2.
We also visualized the prominence contours of utterances, synthesized by conditioning on the diﬀerent emotions and prominence, as shown in Fig. 3.
From the result, the prominence of ﬁne-conditioned
words increased (or decreased) when the conditioning bias increased (or decreased).
4.3

Subjective evaluation

We evaluated the quality of speech by a mean
opinion score (MOS) test on 360 synthesized speech
where each of the 10 sentences had 36 variations(3 emotions × 4 parts of speech × 3 biases). We
conducted the test on the Amazon Mechanical Turkwith 50 participants, each of whom was given 36
speech samples and required to choose speech quality for each speech in ﬁve stages (1: very bad, 5: verygood). The result shows the MOS = 3.9, which iscomparable to the method that can only condition
prominence.

5

Conclusion

We proposed a two-stage emotion-controllable
text-to-speech (TTS) model that can condition
on inter-emotion (e.g., angry) in the ﬁrst stage
and ﬁne-condition on intra-emotion with word-levelprominence in the second stage of control. Dueto the two-stage design, our model enables interemotion controllability and increases intra-emotion
diversity. The results show that we can 1) condition the proposed model on emotion and syn-[1] J. Shen et al., “Natural TTS synthesis by conditioning
wavenet on mel spectrogram predictions,” in 2018 IEEE
International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2018, pp. 4779–4783.
[2] Y. Ren et al., “Fastspeech 2:
Fast and highquality end-to-end text-to-speech,” arXiv preprint
arXiv:2006.04558, 2020.
[3] X. Zhu et al., “Controlling emotion strength with relative attribute for end-to-end speech synthesis,” in 2019
IEEE Automatic Speech Recognition and Understanding
Workshop (ASRU). IEEE, 2019, pp. 192–199.
[4] X. Luo et al., “Emotion-controllable speech synthesis using emotion soft labels and ﬁne-grained prosody
factors,” in 2021 Asia-Paciﬁc Signal and Information
Processing Association Annual Summit and Conference(APSIPA ASC). IEEE, 2021, pp. 794–799.
[5] T. Li et al., “Controllable emotion transfer for end-toend speech synthesis,” in 2021 12th International Symposium on Chinese Spoken Language Processing (ISCSLP). IEEE, 2021, pp. 1–5.
[6] O. Kwon et al., “An eﬀective style token weight control
technique for end-to-end emotional speech synthesis,”IEEE Signal Processing Letters, vol. 26, no. 9, pp. 1383–
1387, 2019.
[7] S.-Y. Um et al., “Emotional speech synthesis with richand granularized control,” in 2020 IEEE International
Conference on Acoustics, Speech and Signal Processing(ICASSP). IEEE, 2020, pp. 7254–7258.
[8] Y. Lei et al., “Fine-grained emotion strength transfer,control and prediction for emotional speech synthesis,”in 2021 IEEE Spoken Language Technology Workshop(SLT). IEEE, 2021, pp. 423–430.
[9] J. Terken, “Fundamental frequency and perceived
prominence of accented syllables,” The Journal of the
Acoustical Society of America, vol. 89, no. 4, pp. 1768–
1776, 1991.
[10] A. Suni et al., “Hierarchical representation and estimation of prosody using continuous wavelet transform,”
Computer Speech & Language, vol. 45, pp. 123–136,
2017.
[11] H. Li et al., “Emphasis: An emotional phoneme-based
acoustic model for speech synthesis system,” arXiv
preprint arXiv:1806.09276, 2018.
[12] G. Sahu, “Multimodal speech emotion recognition and ambiguity resolution,” arXiv preprint
arXiv:1904.06022, 2019.
[13] M. Mauch, S. Dixon, “pyin: A fundamental frequency
estimator using probabilistic threshold distributions,” in2014 ieee international conference on acoustics, speechand signal processing (icassp). IEEE, 2014, pp. 659–
663.
[14] M. Vainio et al., “Emphasis, word prominence, and continuous wavelet transform in the control of hmm-based
synthesis,” in Speech Prosody in Speech Synthesis: Modeling and generation of prosody for high quality and ﬂexible speech synthesis. Springer, 2015, pp. 173–188.
[15] P. Bojanowski et al., “Enriching word vectors with subword information,” Transactions of the association forcomputational linguistics, vol. 5, pp. 135–146, 2017.
[16] C. Busso et al., “IEMOCAP: Interactive emotional
dyadic motion capture database,” Language resourcesand evaluation, vol. 42, no. 4, pp. 335–359, 2008.
[17] S. King, V. Karaiskos, “The Blizzard Challenge 2013,”
in Blizzard challenge workshop, 2014.
[18] X. Cai et al., “Emotion controllable speech synthesis using emotion-unlabeled dataset with the assistanceof cross-domain speech emotion recognition,” arXiv
preprint arXiv:2010.13350, 2020.