APSIPA Transactions on Signal and Information Processing, 2024, 13, e2This is an Open Access article, distributed under the terms of the Creative Commons
Attribution licence (http:// creativecommons.org/ licenses/ by-nc/ 4.0/ ), which permits unrestricted re-use, distribution, and reproduction in any medium, for non-commercial use,
provided the original work is properly cited.

Original Paper

Emotion-controllable Speech Synthesis
Using Emotion Soft Label,
Utterance-level Prosodic Factors, andWord-level Prominence
Xuan Luo1 , Shinnosuke Takamichi1* , Yuki Saito1 , Tomoki Koriyama2 andHiroshi Saruwatari1
1
2

The University of Tokyo, 7-3-1 Hongo, Bunkyo-ku, Tokyo 113-8654, Japan
CyberAgent, Shibuya-ku, Tokyo, Japan

ABSTRACT
We propose a two-stage emotion-controllable text-to-speech (TTS)model that can increase the diversity of intra-emotion variation andalso preserve inter-emotion controllability in synthesized speech.
Conventional emotion-controllable TTS models increase the diversity of intra-emotion variation by controlling fine-grained emotion
strengths; however, such models cannot control various prosodic
factors (e.g., pitch). While other methods directly condition TTS
models on intuitive prosodic factors, they cannot control emotions.
Our proposed two-stage emotion-controllable TTS model extends
the Tacotron2 model with a speech emotion recognizer (SER) and aprosodic factor generator (PFG) to solve this problem. In the firststage, we condition our model on emotion soft labels predicted by
the SER model to enable inter-emotion controllability. In the second stage, we fine-condition our model on utterance-level prosodic
factors and word-level prominence generated by the PFG modelfrom emotion soft labels, which provides intra-emotion diversity.
∗ Corresponding author: Shinnosuke Takamichi, shinnosuke_takamichi@ipc.i.u-tokyo.
ac.jp. This work was supported by JST, Moonshot R&D Grant Number JPMJMS2011 (forexperiments), and JSPS KAKENHI 21H05054, 19H01116, 21H04900 (for basic technique).

Received 16 June 2023; Revised 05 October 2023
ISSN 2161-1823; DOI 10.1561/116.00000242© 2024 X. Luo, S. Takamichi, Y. Saito, T. Koriyama and H. Saruwatari2

Luo et al.

Due to this two-stage control design, we can increase intra-emotion
diversity at both the utterance and word levels, and also preserve
inter-emotion controllability. The experiments achieved 1) 51%
emotion-distinguishable accuracy on average when conditioning on
soft labels of three emotions, 2) average linear controllability scoresof 0.95 when fine-conditioning on prosodic factors and prominence,
respectively, and 3) comparable audio quality to conventional models.

Keywords: Emotion-controllable speech synthesis, expressive speech synthesis,
controllable speech synthesis, text to speech, speech emotion
recognition

1

Introduction

Text-to-speech (TTS) models synthesize human-like speech which includes
linguistic and paralinguistic information. The fast development of deep learningmodels [12, 27, 28, 31, 37, 39] has made it possible to synthesize understandable
speech from a linguistic perspective. On the other hand, synthesizing humanlike speech with diverse paralinguistic information is not an easy task. The
paralinguistic information of human speech, such as emotion, is expressedby various types and strengths of prosodic factors (e.g., pitch) [14, 29] orprominence (i.e., emphasis) [13, 43] at different levels. Even when the samewords are spoken with the same emotion, using slightly different prosodic factors
or prominence can cause listeners to have completely different perceptionsof meaning and feeling. Therefore, it is important for the TTS models tocontrol not only emotion variations but also the variations of prosodic factors
or prominence on the basis of a given emotion. However, few TTS models arecapable of synthesizing speech with such diverse emotion variations as that ofactual human speech.
The diverse emotion variations can be broadly separated into inter- andintra-emotion variations. Inter-emotion variation primarily represents significant differences between emotions, while intra-emotion variation represents
minor differences within an emotion. Conventional emotional TTS models generate an inter-emotion variation by integrating various inter-emotion
representations, such as explicit emotion labels [18, 44] or implicit emotion embeddings [17, 21], into the TTS models. Compared to inter-emotion variation,
intra-emotion variation yields richer emotion diversity. Subsequent studies
generate an intra-emotion variation by controlling a finer granularity, suchas emotion strength [19, 35, 44], which indicates the intensity of emotion.
Such models can increase the diversity of different emotion strengths but notEmotion-controllable Speech Synthesis

3various prosodic factors (e.g., pitch), which are finer granularities than emotion
strength. Considering emotion strength can be formed by various prosodic
factors depending on different conditions (such as gender), the inability tocontrol prosodic factors limits the generation of more diverse intra-emotion
variation.
To synthesize speech with more diverse prosodic factors, studies condition
TTS models on implicit prosodic factors. Subsequent studies aim to disentangle certain implicit prosodic factors in unsupervised manners by utilizing areference encoder, a variational autoencoder (VAE), and multi-head attentionbased models [32, 40, 42] from reference audio. These implicit prosodic factors
are further used to condition the TTS models, which can increase the prosodic
diversity in speech. However, these implicit prosodic factors are not always
disentangled into desired ones, such as pitch, and energy. To solve such a problem, other studies directly condition TTS models on intuitive prosodic factors(e.g., mean of pitch) at utterance-level [26, 30] or phoneme-level [38]. Thesemodels can increase intra-emotion diversity but cannot control inter-emotions.
In this paper, we propose a two-stage emotion-controllable TTS modelthat can control both inter- and intra-emotion of synthesized speech. In thefirst stage, we control the emotion of synthesized speech by conditioning on
emotion-soft labels. In the second stage, we fine-condition utterance-levelprosody factors and word-level prominence estimated on the basis of emotion
soft labels (i.e., emotion posterior probability). The overview of the proposedmodel is shown in Figure 1. The evaluation results demonstrate that ourmodel attains 51% emotion distinguishable accuracy on average which ispromising despite using only a narrative dataset. In addition, our model
achieves average linear controllability scores of 0.95 when fine-conditioning on
prosodic factors and prominence, respectively, which are comparable to theconventional method [26].
The remainder of this paper is organized as follows. In Section 2, wediscuss work related to our study. In Section 3, we elaborate on the proposedmodel by breaking it down into the speech emotion recognizer (SER) and theprosodic factor generator (PFG), and the emotion-controllable TTS models. In
Section 4, we explain the experimental setup consisting of data, preprocessing,model architecture, and training procedures. In Section 5, we first evaluate theperformance of SER and PFG models. Then we evaluate the controllabilityof emotional soft labels and the linear controllability of prosodic factors andprominence, respectively.

4

Luo et al.

Label-based[Um20]👈

Text

Speechsynthesis

Ours

Happy
Angry

Neutral👈

Neutral

Happy
Angry

Estimate
Utterance-level

Second-stage(i.e., Intra-emotion)of control
I am annoyed with him.

👈👈Condition

👈Text

I am annoyed with him.

👈Prosody factor-based[Raitio20] Pitch Energy

Word-level

Pitch Energy Prominence

Only emotion level control

I am annoyed with him.

First-stage(i.e., Inter-emotion)of control

Speechsynthesis

I am annoyed with him.

Only utterance levelprosodic factor control

Speechsynthesis

Emotion soft label, utterance- and word-levelprosody factors and prominence control

Figure 1: Overview of the proposed model2
2.1

Related Work
Emotion-controllable TTS

To control emotion, previous studies commonly conditioned their TTS models
on explicit emotion labels [18, 22, 44] or implicit emotion embeddings [17, 21].
The explicit emotion labels were usually obtained from emotion-labeled speechdatasets [18, 44], or emotion-predictive models [22]. The implicit emotion
embeddings were generally trained by learnable models, such as a speechemotion recognition model [21] or a multi-head attention model [17]. To
generate intra-emotion variation in speech, subsequent studies conditioned on
not only emotion labels or embeddings but also emotion strength, which isa finer granularity. The emotion strength was obtained by an interpolation
method between emotion categories [35], a ranking function at utterancelevel [44], or phoneme-level [19]. These models that were conditioned onemotion strengths can generate more diverse intra-emotion variation than themodels conditioned on emotion labels. However, emotion strength is still not
the smallest granularity to be controlled because it can be further represented
by various prosodic factors [14, 29] or prominence [13, 43].
2.2

Prosody-controllable TTS

To control prosodic factors, previous studies conditioned their TTS models
on prosodic factors using explicit prosodic factors or implicit prosodic factor
tokens. The former directly conditioned TTS models on intuitive prosodic
factors (e.g., pitch mean) on utterance-level [26, 30], or more fine-grained levels
such as the phoneme-level [38]. On the other side, the latter aimed to learn
disentangled prosodic factor tokens by training a reference encoder [32], aEmotion-controllable Speech Synthesis

5variational autoencoder [42], or multi-head attention-based models [40] fromreference audio. The implicit methods generated more prosodic diversity
than the explicit methods. However, there was no guarantee that they candisentangle these tokens into desired prosodic factors because the learned
tokens contained plenty of other paralinguistic information (e.g., speaker ornoise), which makes the disentanglement difficult. Meanwhile, both the explicit
and implicit methods were incapable of controlling emotion in speech.
In addition to prosodic factors, Li et al. [20] condition their TTS model on
word/phoneme-level prominence (i.e., emphasis) to make synthesized speech
more diverse. The prominence that was used to condition the TTS model was
mainly formed by specific prosodic factors, such as pitch, energy, and duration [33]. Because prominence is also related to emotion [34] and contributes
to intra-emotion diversity, we also enable prominence control in this paper.
2.3

Prediction of prosodic factors and prominence from emotion

There is a strong relationship between emotions and prosodic factors [1, 14,29] or prominence [2, 13, 43]. Akçay and Oğuz [1] reported on the correlation
between global statistics of prosodic factors including pitch, energy, andemotion states. For example, the average pitch increases in happy speech.
Arias et al. [2] found that, in addition to prosodic factors, the local prominenceis also correlated with emotion state. For example, the intonation of happy
speech usually increases at the end. Therefore, various statistics of prosodic
factors [1, 14, 29] and prominence [13, 43] are utilized to predict emotions byutilizing different discriminative models.
However, few studies utilize emotion states to predict prosodic factors andprominence. Raitio et al. [26] predicted prosodic factors on which the TTS
model is conditioned, from only text by utilizing a long short-term memory(LSTM) based module. Talman et al. [34] also predicted prominence fromthe text by utilizing BERT [9], a pre-trained language model. In controllable
TTS models, predicting prosodic factors and prominence which are used
for controlling without considering emotion will result in a limited variety
in synthesized speech. Therefore, in our proposed emotion-controllable TTS
model, we predict prosodic factors and prominence from both text and emotion,
which can increase diversity in synthesized speech.

3

Proposed Method

We propose a two-stage emotion-controllable TTS model that enables conditioning on emotion soft labels in the first stage (inter-emotion) of control
and fine-conditioning on the utterance-level prosodic factors (i.e., prosodic
factors) and word-level prominence (i.e., prominence) in the second stage

6

Luo et al.

Text

Encoder

Attention

Text
embedding

Audio

(%)

Text

^
𝑥'()

𝑝!"#

Prosodic factors
extractor(utterance level)

Training
Synthesis

Decoder🎵🔊

Speechemotionrecognizer😐
😠

Prosodicfactorgenerator🎵🎵🔊🔊

Emotionsoft label

Prominence
extractor(word level)

Mel-spectrogram

Basic-conditioning
prosodic factors^
𝑥'+"Basic-conditioning
prominence

MSE loss

Encoder
Text
embedding

Basic-conditioning
prosodic factors

(%)

𝑝!"#

👈

😐

Emotion 😠
soft label

Prosodicfeaturegenerator

First-stage of control

Mel-spectrogram

Decoder

Attention

^
𝑥'()

🎵
🎵
🔊
🔊

(,)

𝑏'()

👈

Fine-conditioning
prosodic factors
^
𝑥'+"

(,)

𝑏'+"
👈
Basic-conditioning
prominence
Fine-conditioning
prominence
Second-stage of control

Figure 2: The architecture of the proposed emotion-controllable TTS model(intra-emotion) of control. To enable this two-stage control, we extend thebaseline Tacotron2 model [31] with a speech emotion recognizer (SER) and aprosodic factor generator (PFG) model, as shown in Figure 2. The SER model
estimates the emotion soft labels, on which the TTS model is conditioned in
the first stage of control, and the PFG model generates prosodic factors andprominence, on which is fine-conditioned in the second stage of control. We
detail the SER, PFG, and the proposed emotion-controllable TTS model inthe following sections.
3.1

SER and PFG models

3.1.1

SER model

The SER model estimates emotion soft labels which are used for the firststage of control. The model takes multi-modal features consisting of prosodic
factors, prominence, and textual features, as input. We utilize the multi-modal
features as input because of their better performance on emotion estimation
than unimodal features, which has been demonstrated in previous research [29].
In addition, the emotion soft labels estimated by multi-modal features in the
SER model can be efficiently used to generate prosodic factors and prominencein the PFG model, which we will discuss later. The ground truth of prosodic
factors, prominence, and textual features can be extracted by the following
approach.
Utterance-level prosodic factors extraction We extract pitch andenergy contours of speech at the frame level and calculate their means, standard
deviations (SD), and range as utterance-level prosodic factors. The pitchEmotion-controllable Speech Synthesis

7

contour is predicted using the pYIN algorithm [23], and the energy contour
is calculated by the root-mean-square value of the magnitude of each frame.
These three statistics of pitch and energy, a total of 6-dimensional prosodic
factors, are used to condition the proposed TTS model because they are related
to speech emotion [29]. Each of the extracted prosodic factors is normalized
to the range from 0 to 1 by applying Min-Max normalization over the training
dataset.
Word-level prominence extraction We extract word-level prominenceby using the lines of maximum amplitude (LoMA) in the continuous wavelet
transform (CWT) of a sum of signal contours of pitch, energy, and duration
with weights [33]. The CWT of a composition of pitch, energy, and duration
contours can approximate human processing of a complex signal relevant toprominence by resembling the perceptual hierarchical structures (phoneme,
syllable, word) related to prosody. This ability is more difficult to achievewith traditional spectrograms. The LoMA [36] are lines that can identify andquantify word-level prominence by connecting nearby peaks in the CWT ofthe signal at different scales, as shown in Figure 3. The strength of the line
for each word (decimerical in Figure 3) is the word-level prominence which
is determined by the cumulative sum of scale values of the line with weights,
shown as follows:
xprm = Ws (a0 , ti0 ,0 ) + . . . + log(j + 1)a−j/2 Ws (a0 aj , tij ,j ),(1)where xprm is word-level prominence, a0 denotes the finest scale in CWT, tij ,j isa time point where the local maxima occurred in the a0 aj scale. Ws (a0 aj , tij ,j )denotes the CWT in tij ,j time point at a0 aj level scale. From this formula,
we can conclude that the higher levels of the hierarchy are given more weightby the logarithmic term than the lower levels one.
To extract prominence, we first align speech and its corresponding text at
the word level by using the Montreal Forced Aligner (MFA) [24], a text-speechalignment tool. Second, we extract the prominence of each word to a scalarvalue by using a wavelet prosody toolkit which is available here.1 The extracted
word prominence indicates the degree of emphasis, which is also related tospeech emotion [13]. The prominence is normalized to the range from 0 to 1
by applying the Min-Max normalization over the training dataset.
Word-level textual feature extraction We extract word-level textual
features by applying the fastText [4], a word-level text embedding model, to atext embedding. The text embedding is an L×M tensor, where the L indicatesthe number of words in a sentence and M is the embedding dimension. Same
as prosodic factors and prominence, textual features are also related to speechemotion [29].
1 The wavelet prosody toolkit: link

8

Luo et al.

Signals

f0
en
dur
sum
0.12
0.25

Wavelet scale (Hz)

0.5
1.0

0.900

1.498

1.244

2.0

0.470

4.0
8.0

0.019

16.0

y
wh

d
an
0.00

0.25

ep
we

0.50

0.75

s
thi

f or

1.00

1.25

1.50

Figure 3: The word-level prominence extraction by applying the CWT-LoMA with a sum of
signal contours of pitch, energy, and duration. The lines of maximum amplitude (LoMA) are
shown in black, while the strength of each line indicates the word-level prominence which isshown in decimal numerical. The white lines are the minimum amplitude which indicatesthe boundaries of words.

Multi-modal features We concatenate prosodic factors, prominence, andtext embedding as multi-modal features to predict emotion soft labels. To dothis, we extend prosodic factors xpsd from a 1 × 6 tensor to a word-length L × 6
tensor and concatenate it to prominence xprm and text embedding xwrd along
the L dimension. To avoid the domination of text embedding, we upsample
the prosodic factors and prominence from 6 and 1 dimensions to 16 and 8
dimensions, respectively. We denote this concatenation as Concatwrd , shownas follows:
xmul = Concatwrd (xpsd , xprm , xwrd ).

(2)

The SER model is a 2-layer LSTM model followed by a softmax output(1)
layer. It estimates emotion soft labels pemo , where superscript 1 indicatesthat it is used for the first stage of control. The emotion soft labels are theposterior probabilities for predicting the emotion labels yemo , conditional on
multi-modal features xmul :p(1)emo = SER(xmul ) = P (yemo |xmul ).
The SER architecture is shown on the left side of Figure 4.

(3)

Emotion-controllable Speech Synthesis

9

Baseline TTS model

SER

PFG

Duplication

Utterance-level 𝑥!"#prosodic factors
𝑥!$%
Word-levelprominence
𝑥&$#

Text

^
𝑥!"#

LSTM
𝑥"'(

Basic-conditioning
prosodic factors

(%)

LSTM
Multi-modal
embeddings𝑝!"#
Emotion
soft
label

^
𝑥!%&

LSTM

Basic-conditioning
prominence

Utterance-levelprosodic factors
generation
Word-levelprominence
generation

Figure 4: The SER and PFG models

Training objective The SER model is trained by minimizing the crossentropy loss (LSER ) between the groud-truth emotion labels and estimatedemotion soft labels:
LSER = −N X
CX

yi,c log(pemoi,c ),(4)

i=1 c=1

where yi,c is an emotion label indicator, assigned 0 or 1, indicating whether
the i-th utterance belongs to the c-th emotion (1) or not (0). N and C are
the total numbers of utterances and emotion categories, respectively. pemoi,cis the estimated emotion soft label of the i-th utterance for the c-th emotion.
3.1.2

PFG model

The PFG model generates basic-conditioning prosodic factors and prominenceon which the TTS model is conditioned, to synthesize speech corresponding
to a given emotion. The PFG model consists of an utterance-level prosodic
factor generator and a word-level prominence generator.
Utterance-level prosodic factor generator The utterance-level prosodic factor generator PFGpsd generates basic-conditioning prosodic factors x̂psd(1)
from a concatenation of text embedding xwrd and emotion soft labels pemo :x̂psd = PFGpsd (xwrd , p(1)emo ),(1)(5)

where pemo is the SER output in training and manually assigned in inference.
The PFGpsd consists of a 2-layer LSTM network followed by a fully connected (FC) layer and a sigmoid layer in sequence.

10

Luo et al.

Word-level prominence generator Similarly, the word-level prominence generator PFGprm generates basic-conditioning prominence x̂prm from(1)
a concatenation of text embedding xwrd and emotion soft labels pemo :x̂prm = PFGprm (xwrd , p(1)emo ),(6)(1)

where pemo is also the SER output in training and manually assigned ininference.
The PFGprm also consists of a 2-layer LSTM network followed by an FC
layer and a sigmoid layer in sequence.
The architectures of the PFGpsd and PFGprm models are shown in theupper right and lower right of Figure 4, respectively.
Training objective The PFGpsd and PFGprm models are jointly optimized
by minimizing the PFG loss LPFG which is calculated by the sum of L2 lossof prosodic factors Lpsd and prominence Lprm . The objective function is:
LPFG =N
X

Lpsdi +

i=1

N
X

Lprmi ,

(7)i=1

where i indicates an utterance index and N is the total number of utterances.
To increase the fit of the SER and PFG models, they are first trained jointly
by minimizing the sum of the SER and PFG losses on an emotion-labeleddataset. The objective function is:
L(SER+PFG) = LSER + LPFG .
3.2

(8)

Emotion-controllable TTS Model

The emotion-controllable TTS model enables two-stage control by extending
the baseline Tacotron2 model with the concatenated SER and PFG models, as
shown in Figure 2. The SER model takes multi-modal features as input andoutputs emotion soft labels, which are fed into the PFG model along with text
embedding. The PFG model outputs basic-conditioning prosodic factors andprominence, which are then fed into the TTS decoder along with phoneme
embedding from the TTS encoder.
Because the basic-conditioning prosodic factors, prominence, and phoneme
embedding have different shapes, to concatenate them together, we extend
prosodic factors and prominence to phoneme lengths by simple duplication
and alignment by the English grapheme-to-phoneme conversion algorithm,2
respectively.

2 The English grapheme-to-phoneme conversion package: link

Emotion-controllable Speech Synthesis

11

The proposed TTS model emoTTS is conditioned on the concatenatedembeddings ĉcon to synthesize speech yspeech :yspeech = emoTTS(ĉcon ),(9)ĉcon = Concatphn (ĉpsd , ĉprm , xphn ),

(10)

where

where Concatphn is a concatenation of phoneme-level conditioning of prosodic
factors ĉpsd , prominence ĉprm , and phoneme embedding xphn .
Specifically, ĉpsd include two parts: basic-conditioning prosodic factors
x̂psd and fine-conditioning prosodic factors (i.e., prosodic factors biases or(2)fine-conditioning biases) bpsd , where the superscript 2 indicates that they areused for the second stage of control, shown in Equation 11. We condition the
TTS model on x̂psd to synthesize speech with a given emotion. We can achieve
this because the x̂psd is generated from emotion soft labels by the PFG model.
(2)
We fine-condition the TTS model on bpsd to enable a slight change of basic
prosodic factors to provide diversity.
(2)ĉpsd = PFGpsd (xwrd , p(1)emo ) + bpsd
(2)

(11)

= x̂psd + bpsd ,where the PFGpsd is the utterance-level prosodic factor generator and thexwrd is the word-level text embedding.
Similarly, the conditioning prominence ĉprm also comprises two parts:
basic-conditioning prominence x̂prm and fine-conditioning prominence (i.e.,(2)
prominence bias or fine-conditioning bias) bprm and behaves in the same way
as the ĉpsd , shown in Equation 12.
(2)ĉprm = PFGprm (xwrd , p(1)emo ) + bprm

= x̂prm + b(2)
prm ,

(12)

where PFGprm is the word-level prominence generator.
According to Equations 11 and 12, the proposed TTS model that is conditioned on ĉpsd and ĉprm functionally enables the inter-emotion control by(1)
applying emotion soft labels pemo and the intra-emotion control by fine(2)(2)
conditioning prosodic factors bpsd (or prominence bprm ), respectively. It is
worth noting that although the proposed model can control both emotion
and prosodic factors (or prominence), in reality, we only need to condition
our model on the pure prosodic factors (or prominence). Such characteristics
can efficiently avoid complicated control over correlated emotion and prosodic
factors (or prominence).

12

Luo et al.

Training objective The proposed TTS model is optimized by minimizing
the additive loss Lemo_TTS of LTacotron2 and LPFG :Lemo_TTS = LTacotron2 + LPFG .

(13)

In inference, the proposed two-stage control TTS model can synthesize
speech in the following ways:
1. Enabling only the first stage of control. Given emotion soft labels, theproposed model can synthesize speech with a specified emotion.
2. Enabling both the first and second stages of controls. Given emotion softlabels and fine-conditioning prosodic factors or prominence, the proposedmodel can synthesize specified emotional speech with slightly changed
prosodic factors or prominence.

4
4.1

Experimental Setup
Data and Preprocessing

We first used the IEMOCAP dataset [5] to jointly train the SER and PFG
models. Then we trained our emotion-controllable TTS model with the SER
model frozen on the Blizzard Challenge 2013 (BC2013) dataset [15].
IEMOCAP is a multimedia English conversation dataset containing
speech, video, etc., performed by five male and five female speakers withnine different emotions. The speech part includes 10,039 utterances (about 12
hours) recorded with a sampling rate of 16,000 Hz. Each utterance is annotatedby an emotion label ranging from nine emotions, including anger, happiness,
excitement, sadness, frustration, fear, surprise, other and neutral state. To
focus only on the significant emotions, we utilized four emotion categories:
angry (i.e., anger), sad (i.e., sadness), neutral (i.e., neutral state), and happy(either happiness or excitement). Specifically, we combined happiness andexcitement into one happy emotion, following Sahu et al.’s [29] method. The
speech labeled with these four emotions was then used to train the SER topreprocess the BC2013 datasets.
BC2013 is an audiobook dataset containing 340 hours of speech recorded
by a professional female speaker in narrative and expressive styles. BC2013 isa high-quality dataset encoded at a sampling rate of 22,050 Hz.
BC2013 preprocessing To efficiently train the proposed emotion-controllable model, we preprocessed BC2013 to select a subset of the dataset that
includes a higher percentage of expressive speech than the original one. In
addition, we predicted emotion labels for each utterance. The preprocessing
consists of three steps:

Emotion-controllable Speech Synthesis

13

1. Character utterance selection. First, we selected all utterances
spoken by characters in BC2013 because they were more likely to containexpressive speech than others. The characters’ utterances were extracted
by selecting the sentences enclosed in single or double quotation marks intranscripts from the BC2013 dataset. This approach is similar to that ofprevious study [25]. To balance character and non-character utterances,
we added all utterances in Jane Eyre, Emma, A Little Princess, andTwenty Thousand Leagues Under the Seas fictions to the extracted
characters’ utterances.
2. Emotion soft labels estimation. Second, we estimated emotion
soft labels for the utterances obtained in step 1 using the SER model
pre-trained on the IEMOCAP dataset with the four emotion categories.
3. Emotion category filter. Third, we conducted a simple listening test
to filter out emotion categories that are incorrectly estimated in step2, which may result in unexpected emotion control. The details of thelistening test are provided in Appendix A.
Finally, we collected a total of 18,638 utterances (about 75 hours), including
4,416 angry, 6,762 neutral, and 7,460 sad utterances. The utterances in the
happy category were dropped out because they did not sound as happy in thelistening test during step 3.
4.2

Model Architecture and Training

4.2.1

Model Architecture

The SER model consisted of a 3-layer LSTM network with 128 hidden units
and a 128 × 3 fully connected (FC) layer, followed by a softmax activation.
The SER model took 324-dimensional multi-modal features, a combination of
300-dimensional text embedding, 16-dimensional upsampled utterance-levelprosodic factors from the original 6-dimensional, and 8-dimensional unsampled
word-level prominence from the original 1-dimensional as input and output
the 3-dimensional emotion soft labels (angry, neutral, and sad).
The PFG model consisted of an utterance-level prosodic factor generator and a word-level prominence generator. The architecture of the former
consisted of a 2-layer LSTM network with 128 hidden units and 128 × 6
FC layers followed by a sigmoid activation function. The input was a 303dimensional joint vector concatenated by a 300-dimensional text embedding
and 3-dimensional emotion soft labels. The output was 6-dimensional predictedprosodic factors. Similarly, the latter model included a 2-layer LSTM networkwith 128 hidden units and a 128 × 1 FC layer followed by a sigmoid activation
function. It also took a 303-dimensional joint vector as input and output a
1-dimensional prominence.

14

Luo et al.

The backbone Tacotron2 consisted of an encoder network that converted
phoneme embedding into a hidden text representation and a decoder networkthat predicted mel-spectrograms from hidden text and prosodic representations.
Specifically, the encoder network consisted of 3-layer 1-dimensional convolutions with 512 filters and a 5 × 1 window size. A phoneme embedding
represented by a 512-dimensional vector was passed through the encoder
network whose output was a hidden text representation.
The decoder network included an autoregressive recurrent neural network,
which consisted of a 2-layer LSTM with 1,024 hidden units, location-sensitive
attention [7], which is an extension of additive attention [3], a pre-net consistedof 2-layer FC network with 256 hidden units, and a post-net consisted of 5-layer
1-dimensional convolutional network with 512 filters. In addition, we also
introduced a psd-net which converted prosodic factors and prominence into ahidden prosodic presentation. A hidden text representation, the encoder output,
was consumed by location-sensitive attention which summarized weighted
hidden text representations into a 512-dimensional context vector. A previous
mel-spectrogram prediction was passed through the pre-net whose output wasa 256-dimensional vector, while prosodic factors and prominence were passed
through the psd-net whose output was a 32-dimensional vector. We applied
dropout with 0.5 dropout rate to the output of the pre-net for better audio
quality in both the training and inference stages, following [31]. The context
vector of attention output and the psd-net output were concatenated andpassed into the autoregressive recurrent neural network which predicted themel-spectrogram one frame at a time. The predicted mel-spectrogram wasthen fed into a post-net to improve the overall reconstruction.
The ground-truth mel-spectrograms were calculated by a short-time Fouriertransform (STFT) on 2,048 sample windows and 512 sample frame hop with aHann window function. We then transformed the STFT magnitude to the mel
scale by using an 80-channel mel filterbank spanning from 80 Hz to 7,600 Hz.
We utilized a Parallel WaveGAN model [41] as a vocoder to generate
waveform samples conditioned on the predicted mel-spectrograms. The Parallel
WaveGAN model was pre-trained on the LJSpeech dataset [11] and is accessible
online.3
4.2.2

Training

We extracted prosodic factors and prominence of the IEMOCAP and BC2013
datasets using the approach described in Section 3. In the experiment, weremoved 1, 718 samples that could not be aligned correctly by the MFA.
The training process consisted of the SER and PFG joint training and theemotion-controllable TTS training. The former trained the SER and PFG
3 The pre-trained Parallel WaveGAN model: link

Emotion-controllable Speech Synthesis

15

models on the IEMOCAP datasets by optimizing the LSER and LPFG in asupervised manner. We used the Adam optimizer [16] with a learning rateof 0.001 and 200 epochs. The latter trained the emotion-controllable TTS
model by optimizing the LPFG and LTacotron2 with the SER frozen and the
PFG fine-tuned. We also used the Adam optimizer [16], and the 0.001 learning
rate started decaying exponentially to 0.00001 after 50,000 iterations.

5

Evaluation

We conducted two principal evaluations: 1) a preliminary evaluation of the
SER and PFG models and 2) a controllability evaluation of the proposed
emotion-controllable TTS. The preliminary evaluation of the proposed SER
and PFG models was performed by comparing them with traditional SER [29]and PFG [33] models. In the latter evaluation, we first evaluated the emotion
controllability of the proposed model when conditioning on emotion soft labels,
and then we evaluated the linear controllability of utterance-level prosodic
factors and word-level prominence when fine-conditioning them with respective
biases.
5.1

SER and PFG Performance

We jointly trained the proposed SER and PFG models on the IEMOCAP
dataset and further fine-tuned the trained PFG model on the BC2013 dataset
while freezing the SER model. To evaluate the SER and PFG models, we
randomly selected 80% of the IEMOCAP and BC2013 datasets as training
datasets and evaluated them on the remaining 20% of the datasets.
5.1.1

SER performance

We evaluated the SER model on the testing part of the IEMOCAP and emotionlabeled part of BC2013 datasets (described in Appendix A) on precision, recall,and F1-score. We conducted an ablation study on the effectiveness of each of
the multi-modal features in predicting three emotions (angry, neutral, and sad)by training the SER models with only text, text, and prosodic factors [29],and multi-modal features of all three input. The results show that our SER
model with multi-modal features of text, prosodic factors, and prominence
input improves F1 scores by 7.8% and 1.3% on the IEMOCAP, 5.7%, 2.0%
and BC2013 datasets when compared with the other two benchmark models,
respectively. The details of the results are shown in Table 3 in Appendix B.
We argue that such results indicate the effectiveness of appending prosodic
factors and prominence to text input for emotion prediction.

16

5.1.2

Luo et al.

PFG performance

We also evaluated the PFG models on the testing part of the BC2013 dataset
in terms of L2 loss. We utilized the conventional prosodic factor generator [26]and prominence generator [34] as our benchmark models, both of which only
utilize text as input. The results indicated that our PFG models (both prosodic
factor and prominence generators) outperformed the corresponding benchmark
models by 0.008 (12.9%) and 0.005 (21.7%) on the absolute (relative) decreaseof L2 Loss, respectively, as shown in Table 4 in Appendix B. Thus, we canconclude that emotion soft labels, in addition to text, also contributed to
predicting both prosodic factors and prominence.
In summary, our proposed SER model with extra prominence input and
the PFG model with extra emotion soft labels input outperformed the conventional SER and PFG models in predicting emotion soft labels, and prosodic
factors/prominence, respectively.
5.2

Emotion-controllable TTS performance

5.2.1

Controllability of emotion soft labels (first stage of control)

We first evaluated the emotion controllability of our proposed model whenconditioning on emotion soft labels during the first stage (inter-emotion) of
control. To obtain the perceived emotion of synthesized speech, we conducted apreference test in which each participant was required to choose angry, neutral,and sad speech, respectively, from a set of three synthesized speech with angry,
neutral, and sad emotion. We synthesized 10 utterances for each emotion(angry, neutral, and sad) as test speech from randomly selected sentencesin the BC2013 dataset by conditioning corresponding emotion soft labels to
1.0. We applied the emotion soft label to 1.0 because the speech with thehighest posterior probability has shown better representativeness than theothers [6]. This test was conducted on the Amazon Mechanical Turk [8] with
50 participants and 10 sets of speech for each participant. The performance ofemotion controllability was evaluated by the accuracy, precision, recall, andF1-score which indicates the distinguishability for each emotion category. The
results demonstrated that the accuracy, precision, recall, and F1-score were51%, 52%, 50%, and 51% on average of three emotions, as shown in Figure 5.
Specifically, the accuracy of angry speech was 60% which was relatively higher
than other emotions. The accuracy of our model is lower than that of theconventional model (80%) [17]. We suggest the reason is two-fold. First, theconventional model was trained on an annotated private dataset recordedwith good emotion distinguishability, while our model was trained on an
unannotated narrative-style BC2013 dataset. Second, we made a trade-off
between the accuracy of emotion distinguishability and the fine-conditioning
ability of prosodic factors and prominence. Nevertheless, we still argue that

Emotion-controllable Speech Synthesis

17

our model provides good emotion controllability. Furthermore, the emotiondistinguishable accuracy of our model can be improved by training on more
emotional speech, as the conventional model did.

Emotion-distinguishable performance

Accuracy
Precision

Recall
F1-score

0.5
0.4
0.3
0.2
0.1
0.0

Angry

NeutralEmotions

Sad

Figure 5: Emotion-distinguishable performance of synthesized speech

5.2.2

Linear controllability of utterance-level prosodic factors (second stage ofcontrol)

We evaluated the linear controllability of our proposed model by fine-conditioning on prosodic factors during the second stage (intra-emotion) of control. We
expected to slightly change prosodic factors of synthesized speech to “biases”
linearly relative to the fine-conditioning biases. To measure this linear relation,
we defined a linear controllability score by utilizing the Pearson Correlation
Coefficient (PCC) between the fine-conditioning biases and measured biases(2)
for each prosodic factor. This can be expressed as PCC(bpsd , b′psd ), where(2)bpsd is the fine-conditioning biases and b′psd is the measured biases indicating
the difference in prosodic factors between the speech synthesized with fineconditioning biases and without fine-conditioning biases (fine-conditioning bias= 0), as shown in Equation 14:(2)

b′psd = PSD(emoTTS(x̂psd + bpsd ))− PSD(emoTTS(x̂psd )).

(14)

PSD indicates the prosodic factor extraction, and x̂psd denotes basic-conditioning prosodic factors, which were discussed in Section 3.1.2.

18

Luo et al.

Table 1: Pearson correlation coefficient for fine-conditioning and measured prosody factor
bias for three emotions

Angry
Prosodicfactors
Energymean
Energyrange
Energy
SD
Pitch
mean
Pitch
range
Pitch
SD

Neutral

Sad

PCC

p-valuePCC

p-valuePCC

p-value

0.99

2.87e-07

0.98

7.10e-07

0.99

1.69e-07

0.98

8.23e-05

0.99

1.21e-05

0.97

2.43e-04

0.97

1.99e-05

0.96

3.70e-04

0.98

2.16e-05

0.96

4.69e-05

0.99

6.82e-06

0.98

1.90e-05

0.83

1.83e-02

0.87

5.07e-03

0.91

1.74e-03

0.89

7.10e-03

0.91

3.82e-03

0.97

2.73e-04

To synthesize the evaluation speech, we input 50 sentences selected fromthe BC2013 dataset, and for each sentence, we fine-conditioned on six prosodic
factors with seven biases for each, ranging from −0.3 to 0.3 with a 0.1 step,for angry (angry = 1.0), neutral (neutral = 1.0), and sad emotion (sad = 1.0).
In total, we synthesized 6,300 speech samples.
(2)
We calculated PCC(bpsd , b′psd ), as shown in Table 1, and visualized the(2)correlation between bpsd and b′psd on angry, neutral, and sad evaluation speech,
as shown in Figure 6.
From the results, we can conclude that:
1. We can linearly fine-condition our model on the six prosodic factors
for angry, neutral, and sad emotions, respectively. With the p-value(statistical significance) <0.05, the average PCC score for angry (0.93),
neutral (0.95), sad (0.97), and overall (0.95) emotions showed strong
linear controllability on the proposed prosodic factors.
2. Compared with the prosodic factors of energy and pitch range/SD, thecorrelation lines (the blue lines) of the energy and pitch mean, as shownin Figure 6, exhibit higher controlling slopes and are closer to the ideal
controlling lines (the red dotted lines).

Emotion-controllable Speech Synthesis

0.3

Energy mean

Energy SD

Angry

19

Energy range

Pitch mean

Pitch SD

Pitch range

0.0

Measured biases

0.3
0.3

Neutral

0.0

0.3
0.3

Sad

0.0

0.3

0.3

0.0

0.3

0.3

0.0

0.3

0.3

0.0

0.3

0.3

0.0

0.3

0.3

0.0

0.3

0.3

0.0

0.3

Fine-conditioning biases on prosodic factors

Figure 6: Correlation between fine-conditioning and observed bias when fine-conditioning
on utterance-level prosodic factors for three emotions

3. The linear controllability of our model on the prosodic factors was fairly
comparable with the conventional method [26]. In addition, our model
can also be conditioned on emotion, while the conventional method
cannot.
We also evaluated the quality of speech which was synthesized by fineconditioning on prosodic factors biases for angry, neutral, and sad emotions.
In detail, we synthesized speech samples for evaluation by conditioning on
angry, neutral, and sad emotions and fine-conditioning on −0.3, 0, and 0.3
biases of each of six prosodic factors from 10 sentences randomly selected fromthe BC2013 test dataset. Finally, we collected 540 speech samples where eachof 10 sentences had 54 variations (3 emotions × 6 prosodic factors × 3 biases).
We conducted a mean opinion score (MOS) test on the Amazon Mechanical
Turk with 50 participants, each of whom was given 54 speech sample variationsof the same sentence and required to choose speech quality for each speech infive stages (1: very bad, 5: very good). The result is shown on the left side of
Figure 7. From the result, We can conclude that our model can condition on
both emotion and prosodic factors without degrading audio quality (MOS =3.5), which is comparable to the conventional method that can only condition
on prosodic factors.
5.2.3

Linear controllability of word-level prominence (second stage of control)

We also evaluated the linear controllability of our proposed model by fineconditioning on prominence during the second stage (intra-emotion) of control.
3 The definition of strong, moderate, and weak linear relationships are 1.0 ≥ PCC ≥ 0.6,
0.6 > PCC ≥ 0.4, 0.4 > PCC ≥ 0.0 while p-value < 0.05 [10]20

Luo et al.

Energy mean

Energy SD

Energy range

5

4

4

3

3

MOS on naturalness

MOS on naturalness

5

2
1

Pitch mean

5

Pitch SD

Pitch range

4
3

1

0.3

0.0

0.3 0.3

0.0

0.3 0.3

Fine-conditioning bias on prosodic factors

0.0

VERB

ADJ

ADV

2
1
5
4
3

Conventional
Proposed (angry)
Proposed (neutral)
Proposed (sad)

2

NOUN

2

0.3

1

0.3

0.0

0.3 0.3

0.0

Fine-conditioning bias on prominence

0.3

Figure 7: MOS scores of speech synthesized by the conventional model [26] and the proposedmodel when conditioning on angry/neutral/sad and fine-conditioning on prosodic factors(left) and prominence (right)

Similarly, we defined a linear controllability score using the PCC between
the fine-conditioning biases and measured biases for prominence. This canbe represented by PCC(bprm , b′prm ), where bprm is fine-conditioning biases
and b′prm is measured biases indicating the difference in prominence between
the synthesized speech with fine-conditioning bias and without bias (fineconditioning bias = 0), as shown in Equation 15:
b′prm = PRM(emoTTS(x̂prm + b(2)prm ))− PRM(emoTTS(x̂prm )).

(15)

PRM indicates the prominence measurement and x̂prm denotes basic-conditioning prominence, which were discussed in Section 3.1.2.
In the experiment, we found that word-level prominence was distributed
differently depending on the part of speech in the training dataset, as shown in
Figure 12 in Appendix D. The prominence of NOUN, VERB, ADJ, and ADV
words were distributed close to a normal distribution; however, the prominenceof other parts of speech was not. Because non-normally distributed parts ofspeech theoretically cannot be controlled linearly, we only experiment on the
NOUN, VERB, ADJ, and ADV words. To synthesize the evaluation speech,
we also input 50 sentences selected from the BC2013 dataset, and for eachsentence, we fine-conditioned on the prominence of NOUN, VERB, ADJ, andADV words, respectively, with seven biases for each, ranging from −0.3 to
0.3 with a 0.1 step, for angry (angry = 1.0), neutral (neutral = 1.0), and sademotion (sad = 1.0). In total, we synthesized 2, 100 speech samples.
We calculated PCC(bprm , b′prm ) based on the basis of both neutral andangry emotions, as shown in Table 2, and visualized the correlation between
bprm and b′prm on the angry, neutral, and sad emotions, as shown in Figure 8.

Emotion-controllable Speech Synthesis

21

Table 2: Pearson correlation coefficient for fine-conditioning and measured prominence bias
for three emotions

Angry

NeutralSad

Prominence

PCC

p-valuePCC

p-valuePCC

p-value

NOUN
VERB
ADJ
ADV

0.88
0.96
0.95
0.98

8.01e-3
4.81e-48.45e-4
6.89e-5

0.95
0.98
0.95
0.97

1.35e-44.49e-3
4.39e-1
4.02e-2

0.98
0.96
0.94
0.96

3.44e-4
4.53e-4
1.32e-3
3.95e-4

NOUN

VERB

0.3

Angry

ADJ

ADV

0.0

Measured bias

0.3
0.3

Neutral

0.0

0.3
0.3

Sad

0.0

0.3

0.3

0.0

0.3

0.3

0.0

0.3

0.3

0.0

0.3

0.3

0.0

0.3

Fine-conditioning bias on prominence
Figure 8: Correlation between fine-conditioning bias and observed bias when fine-conditioning
on the prominence of NOUN, VERB, ADJ, and ADV words for three emotions

From the results, we can conclude that:
1. Our model can linearly fine-condition on the prominence of certain partsof speech including NOUN, VERB, ADJ, and ADV whose prominence isdistributed close to the normal distribution in the training dataset. Withp-value < 0.05, the average PCC score of angry (0.93), neutral (0.97), sad(0.96), and overall (0.95) emotions showed strong linear controllabilityon the prominence of the NOUN, VERB, ADJ, and ADV words.
2. For the parts of speech aside from NOUN, VERB, ADJ, and ADV
words, they may also be linearly fine-controlled if the training dataset

22

Luo et al.

-0 .3

0

0 .3

N OU N

2 .5

VE RB

ADJ

ADV

2 .0
1 .5
1 .0

P r o m in e n c e m a g n it u d e

0 .5
0 .0

An g ry

2 .5
2 .0
1 .5
1 .0
0 .5
0 .0

N e u t ra l

2 .5
2 .0
1 .5
1 .0
0 .5

th p
em
at
th
e
b la
ck
ho
tel

ey

e fu
k e lly
e

for
c

tel

ho

Th

bla
ck

ce

Th
ey
ce
f
k ue l l y
ep
th
em
at
th
e
fo r

f
kue l l y
e
th p
em
at
th
e
b la
c
ho k
tel

ey

ho
te

Th

fo r

fo r

Th

l

Sad
ey
ce
f
k ue l l y
ep
th
em
at
th
e
b la
ck

0 .0

Figure 9: Prominence contours of an utterance synthesized from a given sample sentenceby conditioning on the angry, neutral, and sad emotions (first stage of control) and fineconditioning prominence on NOUN, VERB, ADJ, and ADV words with three biases (−0.3,
0, and 0.3) for each emotion (second stage of control). The sample sentence used here is“They forcefully keep them at a black hotel”. The NOUN, VERB, ADJ, and ADV words
correspond to “hotel”, “keep”, “black”, and “forcefully”, respectively.

is extended with a new one whose prominence is close to the normal
distribution.
We also visualized the prominence contours of utterances, synthesized byconditioning on the angry, neutral, and sad emotions and fine-conditioning
prominence on the NOUN, VERB, ADJ, and ADV words with three biases(−0.3, 0, and 0.3) from the same sentence. The sentence we chose should
contain NOUN, VERB, ADJ, and ADV words at the same time. As shown in
Figure 9, the prominence of fine-conditioned words increased (or decreased)
when the conditioning bias increased (or decreased). For example, by fineconditioning on the NOUN “hotel”, the prominence increased from 1.0 to 1.4(when bias is 0.3) and decreased from 1.0 to 0.6 (when bias is −0.3) for the
angry emotion. On the other side, the prominence of words that were notfine-conditioned (e.g., “They”) was also slightly changed in the experiment.
Such a phenomenon occurred because we enabled the dropout of pre-net evenin the inference stage for better audio quality which brings a slight variation
to mel-spectrogram, as described in Section 4.2. Nevertheless, such slightvariation did not affect the prominence controllability of our model.

Emotion-controllable Speech Synthesis

-0 .3

0

23

0 .3

N OU N

VE RB

ADJ

ADV

M a g n it u d e (d B)

20
25
30
35
40
45
50

En e rg y c o n t o u r

2 .4 5H e r z (d B)

2 .4 02 .3 5
2 .3 02 .2 52 .2 0

ke
th ep
em
a
t t
b la h e
ck
ho
tel

ey
Th

e fu
lly

for
c

tel

ac k

ke
th ep
em
a
t t
bl h e

ho

ey

lly
fu

Th

ce
fo r

ke

t h epem
at tb la h e
ck
ho
tel

ey

lly
fu

Th

ce
fo r

ho
te

ke
th ep
em
a
t t
b la h e
ck

lly
fu

Th

ce
fo r

l

P itc h c o n to u rey

2 .1 5Figure 10: Energy and pitch contours of a synthesized utterance when fine-conditioning
prominence on NOUN, VERB, ADJ, and ADV words with three biases (−0.3, 0, and 0.3) for
angry emotion. The interval (black dashed line) of the fine-conditioned word was computedby the average intervals of corresponding words fine-conditioned with three biases.

To further investigate how the energy and pitch contours changed whenconditioning on prominence, we also drew the energy and pitch contours whenconditioning on different parts of speech for the angry emotion, as shown in
Figure 10. We can conclude that the energy and pitch of the fine-conditioned
word increased (or decreased) simultaneously when the bias was increased to 0.3(or decreased to −0.3). In particular, the pitch was more significantly affected
than the energy. We interpret such phenomenon as follows: changing thepitch can achieve the desired prominence shift with minimal mel-spectrogramchanges compared to changing energy or duration.
We also evaluated the quality of speech which is synthesized by fineconditioning on the prominence of NOUN, VERB, ADJ, and ADV words forangry, neutral, and sad emotions. In detail, we synthesized speech samples
for evaluation by conditioning on angry, neutral, and sad emotions and fineconditioning on −0.3, 0, and 0.3 biases of the prominence of NOUN, VERB,
ADJ, and ADV words from 10 sentences randomly selected from the BC2013
test dataset. Finally, we collected 360 speech samples where each of 10
sentences had 36 variations (3 emotions × 4 parts of speech × 3 biases). We
conducted a mean opinion score (MOS) test on the Amazon Mechanical Turkwith 50 participants, each of whom was given 36 speech samples and required
to choose speech quality for each speech in five stages (1: very bad, 5: verygood). The result is shown on the right side of Figure 7. From this result,
we can conclude that our model can fine-condition the prominence of NOUN,24

Luo et al.

VERB, ADJ, and ADV words for these three emotions without degradingaudio quality (MOS = 3.9), which is comparable to the method that can only
condition prominence. The sample audio is accessible here.4

6

Conclusion and Discussion

We proposed a two-stage emotion-controllable text-to-speech (TTS) modelthat can condition on inter-emotion (e.g., angry) in the first stage and finecondition on intra-emotion including both the utterance-level prosodic factors(e.g., energy mean) and word-level prominence in the second stage of control.
Due to the two-stage design, our model enables inter-emotion controllability andincreases intra-emotion diversity. The results show that we can 1) condition the
proposed model on emotion and synthesize adequately emotion-distinguishable
speech (emotion-distinguishable score = 51%), 2) linearly fine-condition the
proposed model on the utterance-level prosodic factors for angry, neutral, andsad emotions, respectively, 3) linearly fine-condition on the prominence of
NOUN, VERB, ADJ, and ADV words for the angry, neutral and sad emotions,and finally 4) synthesize speech with audio quality (MOS = 3.5) when fineconditioning on prosodic factors and MOS = 3.9 when fine-conditioning on
prominence) comparable to that of the conventional methods. Although theemotion-distinguishable score was slightly lower (= 51%) due to the usage ofthe narrative-style BC2013 dataset, the results are still promising. The emotiondistinguishability score may be further improved by using more emotional
speech datasets.
In addition to the emotion-distinguishable score, other areas to be improved
include emotion strength controllability and the linear controllability of wordlevel prominence, especially on the parts of speech other than NOUN, VERB,
ADJ, and ADV. More generally, we can imagine that the two-stage control
approach can be utilized in other domains, such as controllable image synthesis.

AppendixA

Preference Test for Filtering Out Emotion Categories

We conducted a simple listening preference test for selecting the speech whose
annotated emotion is consistent with the estimated one. This listening preference test required three evaluators to annotate 100 randomly selected utterancesfrom each emotion category, for a total of 400 utterances (100 utterances ×
four emotions). An emotion annotated more than two times is treated as the
4 Sample audio: link

Emotion-controllable Speech Synthesis

25

Table 3: Performance of conventional and proposed SER models on the evaluation partof IEMOCAP and the preprocessed BC2013 dataset (only the annotated part) with threeemotions (angry, neutral, sad). The conventional model utilized Text, Text and PSD(prosodic factors), while the proposed SER model utilized Text, PSD, and PRM (prominence)
as input.

Dataset

Input

Precision

RecallF1

IEMOCAP

Text
Text + PSD [29]Text+PSD+PRM
Text
Text + PSD [29]Text+PSD+PRM

0.551
0.621
0.642
0.535
0.552
0.562

0.562
0.618
0.623
0.480
0.518
0.536

0.554
0.619
0.632
0.486
0.523
0.543

BC2013ground truth emotion of the speech. Given the ground truth and estimation,
we calculated the estimation accuracy and filtered out the emotion categoriesof which the accuracy is lower than 60%.

B
B.1

Performance of SER and PFG Models
SER Performance

We conducted an ablation study on the effectiveness of different features in
predicting three emotions (angry, neutral, and sad) by training the SER models
with only text, text with prosodic factors [29], and multi-modal features of text,
prosodic factors, and prominence, respectively. The SER and PFG models
were jointly trained on the training part (80%) of the IEMOCAP dataset. To
evaluate the SER performance on both the IEMOCAP and BC2013 datasets,
we evaluated it on the testing part (20%) of the IEMOCAP and the emotionlabeled part of the preprocessed BC2013 datasets (described in Appendix A)
on precision, recall, and F1-score. As a result, the F1 scores of our SER modeltrained with multi-modal features of text, prosodic factors, and prominence
input were improved by 7.8% and 1.3% on the IEMOCAP, 5.7% and 2.0% on
BC2013 datasets when compared with the other two benchmark models [29],respectively. The results are shown in Table 3.
B.2

PFG Performance

We fine-tuned the PFG model with the SER model frozen when training the
proposed emotion-controllable TTS model on the training part of the BC2013
dataset and separately evaluated the prosodic factor generator and prominence

26

Luo et al.

Table 4: Performance of conventional and proposed PFG models (prosodic factor generator
and prominence generator) on the preprocessed BC2013 dataset. The conventional modelutilized text, while the proposed PFG model utilized text and emotion soft labels as input.
The L2 loss of utterance-level prosodic factors and word-level prominence were calculated,
respectively.

Model

Input

L2 loss

Prosodic factor generator

Text [26]Text+EmoSoftLabel
Text [34]Text+EmoSoftLabel

0.062
0.054
0.023
0.018

200
0
1000

Pitch SD

750
500
250
0.5

1.0

0
0.0

Energy range
1000
750
500
250
0

400

Pitch mean
800
600
400
200
0
0.0

Energy SD
600

0.5

Pitch range
800
600
400
200
0
1.0
0.0

Energy mean

Normalized prosodic factors (angry)

Energy mean

Energy SD

1500

1000

1000

1000

500

500

500

500

500

0

0

0

0

0

Pitch mean

Pitch SD

500
1.0

Energy range
1500

1000

1000

0.5

Energy SD
1500

1000

Count

Count

Energy mean
800
600
400
200
0

0

0.0

0.5

1.0

Pitch range

1000

1000

500

500
0

0
0.0

0.5

1.0

0.0

Normalized prosodic factors (neutral)

0.5

1.0

Count

Prominence generator

Energy range
1500

1500

1000
500
0

Pitch mean

Pitch SD

Pitch range

1500

1500

1500

1000

1000

1000

500

500

0

0

0.0

0.5

1.0

500
0.0

0.5

1.0

0

0.0

Normalized prosodic factors (sad)

0.5

1.0

Figure 11: Prosodic factors distribution on six prosodic factors for three emotions (left:angry, center: neutral, right: sad)generator of our PFG model on the testing part (20%) of the BC2013 dataset
using the L2 loss. We compared our model with the PFG model without
emotion soft label as input, followed by previous methods [26, 34]. The results
are shown in Table 4.

C

Prosodic Factor Distribution of Angry, Neutral, and Sad Speech in theTraining Part of the BC2013 Dataset

We visualized the distribution of six prosodic factors, including the mean, SD,
and the range of energy and pitch contours, for angry, neutral, and sad speechin the training part of the BC2013 dataset. The results are shown in Figure 11.
The results demonstrated that the prominence of NOUN, VERB, ADJ, andADV words was distributed close to normal distribution, while the prominenceof ADP, AUX, PRON, PROPN, and INTJ was not.

Emotion-controllable Speech SynthesisD

27

Prominence Distribution of Angry, Neutral, and Sad Speech in theTraining Part of the BC2013 Dataset

We visualized the distribution of word-level prominence on NOUN, VERB,
ADJ, and ADV for angry, neutral, and sad speech in the training part of the
BC2013 dataset. The results are shown in Figure 12.

0

Count

400

500

0

PROPN

100
50

100
0.5

1.0

0
0.0

1000

0.5

1.0

0
0.0

Normalized prominence (angry)

0.5

1.0

0
0.0

1.0

0
0.0

0.5

500

1.0

50

500

0
0.0

0.5

1.0

0

200
0

ADP

AUX
1000

0

PRON
1000

ADV
400

1000

INTJ

Normalized prominence (neutral)

2000

400
200
0

100

VERB

0

ADJ

0

PROPN
200

0.5

1000

500
0

500

0

500

1000

AUX
1000

1000

PRON

NOUN

0

ADP

200

ADV

200

400
0

INTJ

400

0

ADJ

500

0
200

VERB
500

0

AUX
1000

1000

PRON

NOUN
500

0

ADP

0

ADV

200

0

ADJ

200

0
0.0

400

Count

VERB
500

Count

NOUN
500

0

PROPN

INTJ
100

200
0.0

0.5

1.0

0
0.0

50
0.5

1.0

0
0.0

Normalized prominence (sad)

0.5

1.0

Figure 12: Prominence distribution on different parts of speech for three emotions (left:angry, center: neutral, right: sad)

References[1] M. B. Akçay and K. Oğuz, “Speech emotion recognition: Emotional models, databases, features, preprocessing methods, supporting modalities,
and classifiers”, Speech Communication, 116, 2020, 56–76.
[2] J. P. Arias, C. Busso, and N. B. Yoma, “Shape-based modeling of
the fundamental frequency contour for emotion detection in speech”,
Computer Speech & Language, 28(1), 2014, 278–94.
[3] D. Bahdanau, K. Cho, and Y. Bengio, “Neural machine translation by
jointly learning to align and translate”, arXiv preprint arXiv:1409.0473,
2014.
[4] P. Bojanowski, E. Grave, A. Joulin, and T. Mikolov, “Enriching word
vectors with subword information”, Transactions of the association forcomputational linguistics, 5, 2017, 135–46.
[5] C. Busso, M. Bulut, C.-C. Lee, A. Kazemzadeh, E. Mower, S. Kim, J. N.
Chang, S. Lee, and S. S. Narayanan, “IEMOCAP: Interactive emotional
dyadic motion capture database”, Language resources and evaluation,
42(4), 2008, 335–59.
[6] X. Cai, D. Dai, Z. Wu, X. Li, J. Li, and H. Meng, “Emotion controllable speech synthesis using emotion-unlabeled dataset with theassistance of cross-domain speech emotion recognition”, arXiv preprint
arXiv:2010.13350, 2020.

28

Luo et al.

[7] J. K. Chorowski, D. Bahdanau, D. Serdyuk, K. Cho, and Y. Bengio,“Attention-based models for speech recognition”, Advances in neural
information processing systems, 28, 2015.
[8] K. Crowston, “Amazon Mechanical Turk: A research tool for organizations and information systems scholars”, in Shaping the future of ict
research. methods and approaches, Springer, 2012, 210–21.
[9] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-trainingof deep bidirectional transformers for language understanding”, arXiv
preprint arXiv:1810.04805, 2018.
[10] J. D. Evans, Straightforward statistics for the behavioral sciences. Thomson Brooks/Cole Publishing Co, 1996.
[11] K. Ito., “The lj speech dataset”, https : / / keithito . com / LJ - Speech Dataset/.
[12] M. Jeong, H. Kim, S. J. Cheon, B. J. Choi, and N. S. Kim, “Difftts: A denoising diffusion model for text-to-speech”, arXiv preprint
arXiv:2104.01409, 2021.
[13] S. Jing, X. Mao, and L. Chen, “Prominence features: Effective emotionalfeatures for speech emotion recognition”, Digital Signal Processing, 72,
2018, 216–31.
[14] R. A. Khalil, E. Jones, M. I. Babar, T. Jan, M. H. Zafar, and T. Alhussain,“Speech emotion recognition using deep learning techniques: A review”,
IEEE Access, 7, 2019, 117327–45.
[15] S. King and V. Karaiskos, “The Blizzard Challenge 2013”, in Blizzard
challenge workshop, 2014.
[16] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization”,
arXiv preprint arXiv:1412.6980, 2014.
[17] O. Kwon, I. Jang, C. Ahn, and H.-G. Kang, “An effective style token
weight control technique for end-to-end emotional speech synthesis”,
IEEE Signal Processing Letters, 26(9), 2019, 1383–7.
[18] Y. Lee, S.-Y. Lee, and A. Rabiee, “Emotional End-to-End Neural Speech
Synthesizer”, in Neural Information Processing Systems(NIPS) 2017,
Neural Information Processing Systems Foundation, 2017.
[19] Y. Lei, S. Yang, and L. Xie, “Fine-grained emotion strength transfer,control and prediction for emotional speech synthesis”, in 2021 IEEE
Spoken Language Technology Workshop (SLT), IEEE, 2021, 423–30.
[20] H. Li, Y. Kang, and Z. Wang, “EMPHASIS: An emotional phonemebased acoustic model for speech synthesis system”, arXiv preprint
arXiv:1806.09276, 2018.
[21] T. Li, S. Yang, L. Xue, and L. Xie, “Controllable emotion transfer forend-to-end speech synthesis”, in 2021 12th International Symposium on
Chinese Spoken Language Processing (ISCSLP), IEEE, 2021, 1–5.

Emotion-controllable Speech Synthesis

29[22] X. Luo, S. Takamichi, T. Koriyama, Y. Saito, and H. Saruwatari,“Emotion-Controllable Speech Synthesis Using Emotion Soft Labels
and Fine-Grained Prosody Factors”, in 2021 Asia-Pacific Signal andInformation Processing Association Annual Summit and Conference(APSIPA ASC), IEEE, 2021, 794–9.
[23] M. Mauch and S. Dixon, “pYIN: A fundamental frequency estimatorusing probabilistic threshold distributions”, in 2014 ieee international
conference on acoustics, speech and signal processing (icassp), IEEE,
2014, 659–63.
[24] M. McAuliffe, M. Socolof, S. Mihuc, M. Wagner, and M. Sonderegger,“Montreal forced aligner: Trainable text-speech alignment using kaldi.”,in Interspeech, Vol. 2017, 2017, 498–502.
[25] W. Nakata, T. Koriyama, S. Takamichi, Y. Saito, Y. Ijima, R. Masumura,and H. Saruwatari, “Predicting VQVAE-based Character Acting Style
from Quotation-Annotated Text for Audiobook Speech Synthesis”, in
Proc. Interspeech, 2022, 4551–5.
[26] T. Raitio, R. Rasipuram, and D. Castellani, “Controllable neural textto-speech synthesis using intuitive prosodic features”, arXiv preprint
arXiv:2009.06775, 2020.
[27] Y. Ren, C. Hu, T. Qin, S. Zhao, Z. Zhao, and T.-Y. Liu, “Fastspeech
2: Fast and high-quality end-to-end text-to-speech”, arXiv preprint
arXiv:2006.04558, 2020.
[28] Y. Ren, Y. Ruan, X. Tan, T. Qin, S. Zhao, Z. Zhao, and T.-Y. Liu,
“Fastspeech: Fast, robust and controllable text to speech”, arXiv preprint
arXiv:1905.09263, 2019.
[29] G. Sahu, “Multimodal speech emotion recognition and ambiguity resolution”, arXiv preprint arXiv:1904.06022, 2019.
[30] S. Shechtman and A. Sorin, “Sequence to sequence neural speech synthesis
with prosody modification capabilities”, arXiv preprint arXiv: 1909.10302,
2019.
[31] J. Shen, R. Pang, R. J. Weiss, M. Schuster, N. Jaitly, Z. Yang, Z. Chen,
Y. Zhang, Y. Wang, R. Skerrv-Ryan, et al., “Natural TTS synthesis byconditioning wavenet on mel spectrogram predictions”, in 2018 IEEE
International Conference on Acoustics, Speech and Signal Processing(ICASSP), IEEE, 2018, 4779–83.
[32] R. Skerry-Ryan, E. Battenberg, Y. Xiao, Y. Wang, D. Stanton, J. Shor,
R. Weiss, R. Clark, and R. A. Saurous, “Towards end-to-end prosodytransfer for expressive speech synthesis with tacotron”, in international
conference on machine learning, PMLR, 2018, 4693–702.
[33] A. Suni, J. Šimko, D. Aalto, and M. Vainio, “Hierarchical representationand estimation of prosody using continuous wavelet transform”, Computer
Speech & Language, 45, 2017, 123–36.

30

Luo et al.

[34] A. Talman, A. Suni, H. Celikkanat, S. Kakouros, J. Tiedemann, and M.
Vainio, “Predicting prosodic prominence from vamvouk trained contextualized word representations”, arXiv preprint arXiv:1908.02262, 2019.
[35] S.-Y. Um, S. Oh, K. Byun, I. Jang, C. Ahn, and H.-G. Kang, “Emotional
speech synthesis with rich and granularized control”, in 2020 IEEE
International Conference on Acoustics, Speech and Signal Processing(ICASSP), IEEE, 2020, 7254–8.
[36] M. Vainio, A. Suni, and D. Aalto, “Emphasis, word prominence, andcontinuous wavelet transform in the control of HMM-based synthesis”, in
Speech Prosody in Speech Synthesis: Modeling and generation of prosody
for high quality and flexible speech synthesis, Springer, 2015, 173–88.
[37] R. Valle, K. Shih, R. Prenger, and B. Catanzaro, “Flowtron: an autoregressive flow-based generative network for text-to-speech synthesis”,
arXiv preprint arXiv:2005.05957, 2020.
[38] A. Vioni, M. Christidou, N. Ellinas, G. Vamvoukakis, P. Kakoulidis,
T. Kim, J. S. Sung, H. Park, A. Chalamandaris, and P. Tsiakoulis,“Prosodic clustering for phoneme-level prosody control in end-to-end
speech synthesis”, in ICASSP 2021-2021 IEEE International Conferenceon Acoustics, Speech and Signal Processing (ICASSP), IEEE, 2021, 5719–
23.
[39] Y. Wang, R. Skerry-Ryan, D. Stanton, Y. Wu, R. J. Weiss, N. Jaitly, Z.
Yang, Y. Xiao, Z. Chen, S. Bengio, et al., “Tacotron: Towards end-to-end
speech synthesis”, arXiv preprint arXiv:1703.10135, 2017.
[40] Y. Wang, D. Stanton, Y. Zhang, R.-S. Ryan, E. Battenberg, J. Shor,
Y. Xiao, Y. Jia, F. Ren, and R. A. Saurous, “Style tokens: Unsupervised
style modeling, control and transfer in end-to-end speech synthesis”, inInternational Conference on Machine Learning, PMLR, 2018, 5180–9.
[41] R. Yamamoto, E. Song, and J.-M. Kim, “Parallel WaveGAN: A fast
waveform generation model based on generative adversarial networks withmulti-resolution spectrogram”, in ICASSP 2020-2020 IEEE International
Conference on Acoustics, Speech and Signal Processing (ICASSP), IEEE,
2020, 6199–203.
[42] Y.-J. Zhang, S. Pan, L. He, and Z.-H. Ling, “Learning latent representations for style control and transfer in end-to-end speech synthesis”, in2019 IEEE International Conference on Acoustics, Speech and Signal
Processing (ICASSP), IEEE, 2019, 6945–9.
[43] J. Zhao, X. Mao, and L. Chen, “Speech emotion recognition using deep
1D & 2D CNN LSTM networks”, Biomedical Signal Processing andControl, 47, 2019, 312–23.
[44] X. Zhu, S. Yang, G. Yang, and L. Xie, “Controlling emotion strengthwith relative attribute for end-to-end speech synthesis”, in 2019 IEEE
Automatic Speech Recognition and Understanding Workshop (ASRU),
IEEE, 2019, 192–9.