Thu-P-OS-10-3

STUDIES: Corpus of Japanese Empathetic Dialogue Speech
Towards Friendly Voice Agent
1112

Yuki Saito , Yuto Nishimura , Shinnosuke Takamichi , Kentaro Tachibana , and Hiroshi Saruwatari(1: The University of Tokyo, Japan, 2: LINE Corp.)

1

Synopsis: corpus for empathetic dialogue speech synthesis● Dialogue situation: teacher attempting to motivate students● What is “empathy”?
○ Deﬁnition: active attempt to get inside the interlocutor [1]React to his
good news

Student (human user)
Hello, teacher!
(happy) Hi! You get agood score?
(neutral)■ e.g., doctor–patient [2], teacher–student [3]■ Essentials: both text contents [4] and speech (e.g., prosody) [5]○ Different from “sympathy”■ Passive attempt to share the speaker’s feeling

Bingo!
(happy)Get it now!

Congrats!
(happy)http://sython.org/Corpus/
STUDIES

Teacher (AI voice agent)● STUDIES: Japanese empathetic dialogue speech synthesis○ Professional speakers, emotion labels, and short/long dialogues○ Speech synthesis experiment in this poster○ Opensourced for research purpose only (the above QR code!)● Empathetic dialogue speech synthesis○ For AI voice agent that empathizes with humans○ Corpus for this purpose: uncultivated so far…

Methodology for corpus construction
Dialogue scenario

Crowdsourcing chat dialogue lines● Two issues of chat-oriented
dialogue with speciﬁc persona [6]a. Inconsistent persona of the agent [7]b. Lack of long-term memory(i.e., forgetting chat history) [8]● Predetermining the agent’s persona

Voice recording● Prior to recording● Instruction to crowdworkers

a. Manual revision of dialogue lines
i. Unnatural in grammar, syntax,and/or dialogue
ii. Inconsistent persona

a. Situation: teacher-student dialogueb. Agent’s persona: see left.
c. Dialogue durations: 4 or 10~20 turns
d. etc. (see our paper for the detail)● Voice recording● Crowdworkers’ actions○ Three professional speakers○ Making teacher-student dialogue lines○ Labeling the ﬁctional characters’ emotion

a. Female in her early twenties
b. Tokyo dialectc. Cheerful and friendly personalityd. etc. (see our paper for the detail)■ Teacher and two students○ Speaker-wise recording in studio■ Due to the COVID-19 pademic● Hiring crowdworkers
a. Macrotask crowdsourcing for Long dialogues○ Recording two subsets

i. Hiring crowdworkers on the basis of their skills
ii. Diffucult task for the crowdworkers● Two types of dialogue durations○ Long (10~20 turns): challenging○ Short (4 turns): easier than “Long”■ User → Agent → User → Agentb. Microtask crowdsourcing for Short dialogues
i. Hiring crowdworkers without approvalii. Simple task for the crowdworkers■ Dialogue lines (Long and Short)■ Phoneme-balance sentences withemotion labels (ITA [9])● Neutral, Happy, Angry, and Sad● Coutermeasure for data imbalance

Corpus analysis and speech synthesis experiments
Corpus analysis

Speech synthesis experiments● Corpus speciﬁcation● Model setup○ #utterances and #hours in each subset○ #utterances for each emotion label● F0 statistics distributions
InterSpeaker’s
Hello!
Hi!
locutor’s(happy)(neutral) voice(s)voice
Aggregated byAggregated byinterlocutor’s
speaker’s
emotionemotion
Voicestatistics● 8 hrs, 6,000 utts in total○ Suﬃcient for neural TTS● Larger than existing relevant
corpora (e.g., UUDB [10])● The size is well balanced.
○ Except teacher’s “angry”● Dialogue in which the teacher
gets angry is rare.
○ “Empathy” ≠ act to shareangry emotion● Statistics of speaker’s F0○ Aggregated by two ways
○ (see our paper for otheranalysis results)● Similar distributions between
different aggregations○ Possibility to synthesize
speech using not only thespeaker’s own emotion butalso interlocutor’s one● Speech samples:Text-based chat history

Dialogue history

Text

You get a“Hi, teacher!” “Hi!
good score?”(happy)(neutral)“Bingo!”(happy)Student’semotion label
Acoustic model for TTS
Chat history encoder
Emotion label format
Training/Evaluation data

“Congrats!”(happy)Syntheticspeech

Teacher’s
emotion label

FastSpeech 2 [11]BERT-based [12]One-hot vector → lookup emb.
3,600/210 utterances (teacher)● Subjective evaluation (MOS on naturalness)○ Utterance level (i.e., teacher’s synthetic speech only)○ Dialogue level (i.e., student’s natural speech andteacher’s synthetic speech)
Emotion Text
UttDial- ● Cases better thanbaseline (ordinary TTS)
label history levellevel○ Teacher’s emotion(baseline)
3.42
3.20
○ Text history
Teacher
3.43
3.38○ Student’s emotion &Student
3.38
3.29
text history
✓
3.32
3.37
● Importance of dialogue
Teacher✓
3.38
3.34
context (history andStudent✓
3.43
3.41
situation) in empathetic
MOSs for reconst. speech: 4.08 and 4.33
dialogue speech synth.

Reference[1] M. H. Davis et al., Routedge, 1996.
[2] M. Hojat et al., Journal of Health andHuman Services Administration, 2009.
[3] C. Warren et al., The Urban Review, 2015.

[4] H. Rashkin et al., ACL, 2019.
[5] C. Regenbogen et al., Cognition andEmotion, 2012.
[6] S. Zhang et al., ACL, 2018.

Saruwatari-Koyama Lab., The University of Tokyo, Japan.
©Yuki Saito, Sep. 2022. (sythonuk@g.ecc.u-tokyo.ac.jp)[7] J. Li et al., ACL, 2016.
[8] O. Vinyals et al., ICML, 2015.
[9] J. Koguchi et al., 2021.

Interspeech 2022, Korea[10] H. Mori et al., Speech Communication,
2011.
[11] Y. Ren et al., ICLR, 2021.
[12] H. Guo et al., SLT, 2021.
This is the joint work between
The University of Tokyo & LINE Corp. Japan.