Thu-P5.21 (#1098)

CALLS: Japanese Empathetic Dialogue Speech Corpus ofComplaint Handling and Attentive Listening in Customer Center
1

1

1

2

Yuki Saito , Eiji Iimori , Shinnosuke Takamichi , Kentaro Tachibana , and Hiroshi Saruwatari(1: The University of Tokyo, Japan, 2: LINE Corp.)

1

Synopsis: corpus for empathetic dialogue speech synthesis in polite dialogue situation● New situation for EDSS: operator & customer in customer center● Task: Empathetic Dialogue Speech Synthesis (EDSS)

Customer (human user)○ Towards an AI voice agent that empathizes with humans

I bought
your product.

■ Control the prosody of synthetic speech considering theinterlocutor’s mental state (e.g., happy → high pitch) & situation● Existing corpus for EDSS: STUDIES

Thank you!

But it doesn’t
work well!

React to hiscomplaint

Get it now!

I sincerely
apologize…

https://sython.org/Corpus/
STUDIES-2

Operator (AI voice agent)[1]○ Situation: chat betw. a teacher & student in a cram school [2](informal & intensely expressive speaking style)○ Limitation: only 1 situation & 8 hours empathetic dialogues…

● CALLS: corpus of Complaint handling & Attentive Listening Lines Speech■ How can we construct a corpus for EDSS in a different situation?

○ Covering negative/positive feedbacks from customers○ The same speaker as the STUDIES teacher → multi-domain EDSS
○ Opensourced for research purpose only (scan the above QR code!)

Methodology for corpus construction
Dialogue scenario

Instructions for crowdworkers

Voice recording● Prior to the recording● For the complaint handling subset…

● Diﬃculties in recording actual
customer-center dialogues○ Privacy preservation for speakers○ Use (1) seed situation & (2) user’s proposal
w/ (3) metadata when writing dialogue lines.
1. Text data describing an anonymous user’s→ Collecting simulated dialogue lines
by crowdsourcing

complaint about a speciﬁc service or product2. An user’s opinion to deal with a complaint
3. User’s age, gender, job, and locale
Selected from the FKC corpus[3]○ Limited bandwidth of phone calls→ Recording the simulated dialogues
by a professional voice actor in studio○ Anonymize the name of a particular companyor product if the situation included it.

● Other settings○ The agent’s persona● For the attentive listening subset…

■ Female in her early twenties■ Tokyo dialect■ Gentle tone of voice, etc.

○ Write short dialogue lines where a customer &operator are talking happily on a phone call.
○ Don’t include the name of a speciﬁc service or
product in the dialogue lines.
→ Similar to “Short” dialogue in STUDIES[1]○ Two dialogue subsets■ Complaint handling (2 ~ 12 turns)■ Attentive listening (4 turns)○ Screening the obtained dialogue lines■ Corrected unnatural sentence ingrammar and/or syntax■ Removed outcomes from spamworkers (e.g., wrote only “a”) or ones
who didn’t follow our instructions● Recording the agent’s voices○ Speaker: the same as the STUDIES
teacher (a female voice actor)■ We didn’t record the customers’ voices
because the recording is unrealistic.

○ Device: a unidirectional microphone○ Period: 10 days (3 hours per day)■ Complaint handling: 6 days
■ Attentive listening: 4 days

Corpus analysis and EDSS experiments
Corpus analysis● Corpus speciﬁcation● 6,544 utterances in total○ 6,544 utterances in total○ # of utterances for each emotion label
Spkr.

NeutralHappy

SadAngryTotal

Operator

414 + 243

719 + 950

939 + 70+0

3,232 (6.5h)

Customer 760 + 389

144 + 790

263 + 21945 + 0

3,312 (N/A)● Comparison with existing corpora
Open- Dur.
Emotion# Spkr.
sourced? [h]labeled?

CorpusDialogue type

Hiraoka+[4]PersuasiveNo

5.722No

Attentive listening

No

2.38NoSTUDIES[2]Empathetic (casual)Yes

8.03YesCALLS (ours)

Empathetic (formal)

Yes

6.5

1

Yes

Kawahara+[6]○ Acoustic model: FastSpeech 2 (FS2) conditioned by:■ Emotion label of { agent, user } (A-Emo, U-Emo)■ Chat history embedded by Conversational Context Encoder[7] (CCE)■ Speaker ID (SPK: only available for multi-domain EDSS)○ Data imbalance in emotion■ 0 angry voices by theoperator→ Unfavorable in thecustomer center situation

Text

FS2

Spkr.

CCEEmo.

or○ New domain for EDSS
■ Combined with STUDIES→ 10h of multi-domainempathetic speech corpus■ Related to persuasion &counselor’s attentive listening● Prosody feature statistics & sentence embedding visualization
Interlocutor’s emotion
SadEDSSmodelor EDSSmodelFS2CCEorSingle-domain EDSS
(separately trained 2 EDSS models)Multi-domain EDSS
(1 EDSS model trained w/ 2 corpora)● Subjective evaluation: 2 MOS tests (w/ 400 listeners)

2D t-SNE plotLess expressive
in “happy” emotion● Experimental setup: single-/multi-domain EDSS

■ 6.5h of agent’s empathetic
voices w/ formal styles■ (complaint handling + attentive listening)[5]EDSS experiments

Happy○ Criteria: naturalness & speaking-style similarity

CALLSoperator
STUDIESteacher

Generally lower
stds of logF0 &Energy○ Summary
CALLS covers a different range
than STUDIES!

■ CALLS operator’s voices can be trained more easily.
■ CCE is effective, but the domain gap should be considered.

Reference[1] Y. Saito et al., INTERSPEECH, 2022.
[2] C. Warren et al., The Urban Review, 2015.
[3] K. Mitsuzawa et al., LREC NIEUW Workshop, 2016.
Saruwatari-Takamichi Lab., The University of Tokyo, Japan.
©Yuki Saito, Aug. 2023. (sythonuk@g.ecc.u-tokyo.ac.jp)[4] T. Hiraoka et al., Speech Communication, 2016.
[5] T. Kawahara et al., IWSDS, 2015.
[6] Y. Ren et al., ICLR, 2021.
INTERSPEECH2023 @ Ireland[7] H. Guo et al., SLT, 2021.

This is the joint work between
The University of Tokyo & LINE Corp. Japan.