Eliciting Cooperative Persuasive Dialogue
by Multimodal Emotional Robot
Sara Asai, Koichiro Yoshino, Seitaro Shinagawa,
Sakriani Sakti, Satoshi Nakamura
知識獲得・対話研究チーム
Knowledge Acquisition &Dialogue Research Team

奈良先端科学技術大学院大学
Nara Institute ofScience and Technology

2021/11/15 ⒸKoichiro Yoshino,
Guardian Robot Project, RIKENIWSDS 2021, Eliciting cooperative persuasive dialogue by
multimodal emotional robot

1

Emotion in persuasive dialogue◆Persuasion⚫Lead users to system’s goal

◆Emotional expression are effective for eliciting
cooperative dialogue manner⚫Positive emotion: create a cooperative atmosphere⚫Negative emotion: effectively wrest concessions
e.g.,

Persuasion to have exercise

You should have exercise!
I’m busy now.

Using angryemotion
expression forpersuasion

You said the samething yesterday!

system2021/11/15 ⒸKoichiro Yoshino,
Guardian Robot Project, RIKEN

OK, OK, I go
running

user

IWSDS 2021, Eliciting cooperative persuasive dialogue by
multimodal emotional robot2

Hypotheses and experiments◆Emotional expression by robot will increase user’s
acceptance to the persuasion, and will improve impression
from users to the system
Experiment 1: comparing systems:
w/ and w/o emotional expression◆Multimodal emotional expression will lead effective
persuasion
Experiment 2: comparing systems that have different
modalities: text, speech, speech+gesture (robot)
2021/11/15 ⒸKoichiro Yoshino,
Guardian Robot Project, RIKENIWSDS 2021, Eliciting cooperative persuasive dialogue by
multimodal emotional robot

3

System architecture◆We built persuasive system by response selection system⚫Collected multi-reference persuasion corpus with emotional
voices (acted by voice actor) [LREC2020, Asai et al.]⚫Built a BERT-based selection system⚫System can respond with pre-defined emotion
𝑟1

Let’s exercise !!
No, I don’t
like exercise.
User
Utterance

SystemUserEmbedding (BERT)𝑟𝑖

You have to takeexercise!

⊕𝑢𝑖

Calculatesimilarity

OutputI’m warried about
your health.

“Don’t you“No, I don’t”
get exercise?”

DialogContexts𝑢𝑡

𝑢2

I think you might findexercise fun.
Robot
Responses･･･_⊕

Highest similarity⊕𝑟𝑡 1

But you will getweight.

𝑢1

･･･

Previoussystemutterance𝑟2⊕AngryResponse Candidate
each Emotion
CorpusNext System Emotion

2021/11/15 ⒸKoichiro Yoshino,
Guardian Robot Project, RIKENIWSDS 2021, Eliciting cooperative persuasive dialogue by
multimodal emotional robot

4

System demo

2021/11/15 ⒸKoichiro Yoshino,
Guardian Robot Project, RIKENIWSDS 2021, Eliciting cooperative persuasive dialogue by
multimodal emotional robot

5Experiment 1: emotional persuasion◆We prepared 22 subjects (11 male, 11 female)⚫They talked with 6 systems on different conditions in random
order: neutral, angry, sad, happy, random-emo, LR-emo◆Experimental procedure⚫Dialogue robot starts persuasion and users respond to therobot’s utterance⚫Dialogue will end when 20 turns passed,
or users accept persuasion⚫Instruction:
You are living with a robot that provides daily life support.
Since you have lived with this robot for a long time, you trust it.
After learning that recently you have not been getting enoughexercise, it encourages you to start jogging.
You refuse to get any exercise.
2021/11/15 ⒸKoichiro Yoshino,
Guardian Robot Project, RIKENIWSDS 2021, Eliciting cooperative persuasive dialogue by
multimodal emotional robot

6

Result 1◆There was no significant difference on persuasion
success, between “neutral” and “single emotion” settings.
◆However, impressions from users were improved⚫Positive emotion works better in any case, Negative emotions
can improve human-likeness and considerateness
NEUTRAL

ANGRY

SAD

HAPPY

RANDOMLR

Naturalness

2.533

3.400

2.867

3.133

2.733

2.133

Persuasiveness

3.000

3.200

3.267

3.267

2.667

3.000

Expressiveness

3.000

4.200**

4.467**

4.133**

4.133**

4.000**

Human-likeness

2.667

3.800**

3.533*

3.000

3.000

2.867

Kindness

2.533

3.2003.733**

4.200**

2.933

3.067

Considerateness

3.000

4.133**

3.467

4.133**

3.933

3.333

2021/11/15 ⒸKoichiro Yoshino,
Guardian Robot Project, RIKENIWSDS 2021, Eliciting cooperative persuasive dialogue by
multimodal emotional robot

7

Experiment 2: modalities on persuasion◆We prepared 22 subjects (11 male, 11 female)⚫They talk with 3 systems on different conditions in random order⚫Text, Speech, Speech+Gesture (robot) in happy condition◆Other experimental conditions are same as Exp.1

2021/11/15 ⒸKoichiro Yoshino,
Guardian Robot Project, RIKENIWSDS 2021, Eliciting cooperative persuasive dialogue by
multimodal emotional robot

8Result 2◆Speech modality (recorded speech by actor) improved
persuasiveness a lot◆Adding gesture modality (robot) could improve
expressiveness or human-likeness
TEXT

SPEECH

SPE+GESNaturalness

3.188

3.500

3.688

Persuasiveness

3.312

3.688

3.625

Expressiveness

3.188

3.688

4.250

Human-likeness

3.438

4.000

4.625

Kindness

2.875

4.250

4.375

Considerateness

3.312

3.938

4.188

2021/11/15 ⒸKoichiro Yoshino,
Guardian Robot Project, RIKENIWSDS 2021, Eliciting cooperative persuasive dialogue by
multimodal emotional robot

9Conclusion◆We built a persuasive dialogue robot that can make emotional
expressions using multi-modality⚫ The system was based on response selection approach, and theemotional voices were pre-recorded actor’s voice◆Emotional expression could encourage users to talk with thesystem in cooperative dialogue manner⚫ Could improve impression from the users to the system◆We did not confirm significant effect on persuasion
performance⚫ Most of users agreed to the persuasion⚫ Probably some of them didn’t take exercise⚫ We should measure the real performance of persuasions
2021/11/15 ⒸKoichiro Yoshino,
Guardian Robot Project, RIKENIWSDS 2021, Eliciting cooperative persuasive dialogue bymultimodal emotional robot

10