Multimodal Database of Negative Emotion Recovery
in Dyadic Interactions: Construction and Analysis
Nurul Lubis, Michael Heck, Sakriani Sakti, Koichiro Yoshino, Satoshi Nakamura

Nara Institute of Science and Technology

18-Sep-18

Nurul Lubis

1

Affective dialogue systems

High potential of dialoguesystems to address theemotional needs of users• Increase of dialogue system works andapplications in various tasks involving
affect• Companion for the elderly[Miehle et al., 2017]• Distress clues assessment[DeVault et al., 2014]• Affect-sensitive tutoring[Forbes-Riley and Litman, 2012]18-Sep-18

Nurul Lubis

2

Works on emotional problems• Depression and suicide risk
assessment (Cummins et al., 2015)• Virtual human interviewer for distressclues assessment (DeVault et al., 2014)
Focus on clinical situation, not suitablefor common users

Lack of works on negative emotion in everyday situations
18-Sep-18

Nurul Lubis

3

Challenge and proposed solution

Lack of resources thatIn this papera multimodal database of dyadic
interactions between a professionalcounselor as an expert and 30
participants• highlights emotion problems ineveryday situation• involves an expert in theconversation

18-Sep-18

Nurul Lubis

4

Database design

18-Sep-18

Nurul Lubis

5

Processing negative emotion through social communication• Goal: Learning expert dialogue strategy in an affective conversation to recover negative emotion• Based on social sharing of emotion [Luminet+, 2000]• Interaction between an expert and a participant after a negative emotion inducer• Expert guides the conversation to allow participant’s emotion recovery
Post-recording
Questionnaire

Briefing
Recorded Session
Opening talk(neutral)
Self-assessmentof emotion (SAM)
18-Sep-18

Emotion induction(negative)

Self-assessmentof emotion (SAM)

Emotion processingand recovery(positive)

Self-assessmentof emotion (SAM)
Nurul Lubis

Self-assessmentof emotion (SAM)
6

Emotion Inducer• Video clips as emotion inducer [Gross+, 1995] [Schaefer+, 2010]• Effective and reliable• Easy to replicate• Less personally involved• Non-fictional short clips• E.g., news report, interview, documentary• Avoid unpredictable reactions to fictional clips• Two target negative emotions: anger and sadness• Each emotion target has 10 inducers

18-Sep-18

Nurul Lubis

7

Dyadic Interlocutors
Expert• An external party who helps facilitate
the emotional response and processesof the participant• 1 counselor• An accredited member of British
Association for Counseling andPsychotherapy• > 8 years of experience

18-Sep-18

Participant• Reacting to emotion inducer• Discussing the topic and their reaction
with the counselor• 30 participants• 20 males, 10 females• Fluent English speakers (L1 or L2)

Nurul Lubis

8

Result18-Sep-18

Nurul Lubis

9

Constructing the corpus• 23 hours and 41 minutes of data
were collected• Fully annotated Self-reported emotion by participant
 Perceived emotion by counselor Transcription• Emotion annotation: Circumplex
model of affect [Russel, 1980]
Emotion annotation with Gtrace [Cowie+, 2000]• Arousal: level of activity of emotion• Valence: positivity-negativity ofemotion
18-Sep-18

Nurul Lubis

10

Emotion Annotation

18-Sep-18

Nurul Lubis

11

Post-recording questionnaire
0%

20%

40%

60%

80%

100%

video_neg

chat_pos
helps_emo
understoodenjoyed• the emotion inducer videos are effective• the participants reported positive
emotional effect of the conversation• the conversation helps to process emotion• the conversation is enjoyable• the participants feel understood• emotional connection appears to be themost difficult feeling to achieve• possibly due to the limited time and lack ofcontinuity of the interaction, andemo_connect
chat_again

strongly agree

18-Sep-18

agree

neutraldisagreestrongly disagree• in general, the participants would like tointeract with the expert again in thefuture
Nurul Lubis

12

Conclusion and future works

18-Sep-18

Nurul Lubis

13

Conclusion and Future Works• We presented recordings between aprofessional counselor and 30
participants• Allowing observation of emotion
change and how an expert can guidenegative emotion processing• The corpus amounts to 23 hours and41 minutes of data

• In the future• Learn expert strategy explicitly• Use expert strategy for positive emotion
elicitation in a dialogue system• Fully annotated• Analyzed

18-Sep-18

Nurul Lubis

14

Thank you

18-Sep-18

Nurul Lubis

15