Multimodal Database of Negative Emotion Recovery in Dyadic
Interactions: Construction and Analysis ∗☆ Nurul Lubis1 , Michael Heck2 , Sakriani Sakti1,2 , Koichiro Yoshino1,2,3 ,Satoshi Nakamura1,2 (1 NAIST, 2 RIKEN AIP, 3 JST PRESTO)

1

Introduction

faction of the interaction
We opt for short video clips (a few minutes inSocial sharing of emotion is an essential part inprocessing a negative emotional experience. As thetechnology develops, the potential of dialogue systems to address this need has been growing as well.
However, there is still a lack of resources that: 1)
highlight the emotional problems commonly encountered in everyday life, and 2) involve an expert in theconversation.
To fill these gaps, in this paper we present a multimodal database containing dyadic interactions between a professional counselor as an expert and 30

length) as emotion inducers in the sessions. One
study shows that amongst a number of techniques,
the use of video clips is the most eﬀective way to induce both positive and negative emotional states [1].
We look for non-fiction clips that depict real life issues. To fit these requirements, we select short video
clips of news reports, interviews, and documentaryfilms as emotion inducers.

3Recording and Annotation

participants. Each session starts with negative emo-

We recruit a professional counselor as the expert

tion elicitation through short video clips, and in thein the recording. The expert is an accredited mem-interaction that follows, the goal of the expert is

ber of the British Association for Counseling andto aid emotion processing and elicit a positive emotional change. This database allows us to observe

Psychotherapy with more than 8 years of professional experience. As participants, we recruit 30 in-

how an external party can guide and facilitate emo-

dividuals (20 male, 10 female) who speak English

tion processing after a negative emotional response

fluently as first or second language. We record 2 ses-in a common social situation. The construction, de-sions for each participant, one induced to “anger,”velopment, and analysis of the database are reported

the other to “sadness.” In total, we record 60 ses-in this paper.

sions, amounting to 23 hours and 41 minutes of data.
We use the circumplex model of aﬀect, and anno-2Database Design
We would like to observe how an external party

can guide and facilitate emotion processing throughan interaction after a negative emotional response.
We arrange for the dyad to consist of an expert anda participant, each with a distinct role. The expert
plays the part of the external party who helps facilitate the emotional response of the participant.
The session starts with an opening talk as a neutral baseline conversation. Afterwards, we induce
negative emotion by showing an emotion inducer tothe dyad. The recording then continues with a discussion phase that targets at emotional processing
and recovery. Throughout the process, we ask theparticipants to assess their emotional states with theuse of a questionnaire. At the end of the session, we
use a questionnaire to measure participant’s satis-tate two emotion dimensions accordingly: valence
and arousal [2]. Valence measures the positivity ofemotion, and arousal measures the activity. Eachdimension is annotated separately using the FEELtrace system [3]. This results in a sequence of realnumbers ranging from -1 to 1, called a trace, thatshows how a certain emotional aspect falls and rises
within an interaction. Validation experiments haveconfirmed the reliability and indicated the precisionof the FEELtrace system [3]. We provide two typesof emotion annotation: self-reported emotion, annotated by the participants; and perceived emotion,annotated by the expert.
We transcribe the spoken language of eachrecorded session.

We employ a paid Automatic

Speech Recognition (ASR) service to obtain an automatic transcription of the data. The automatictranscription is then subject to manual revision and∗

負の感情からの回復を目的とする二者対話のマルチモーダルコーパス: 構築と分析

inspection of a professional human transcriber. Dur-ticipant is more intense.
val_participant
aro_participant

ing manual revision, we maintain non-speech information that potentially gives emotional state clues.

1.5

The following parts of speech are given special no-1.0

tations: laughter, back-channel utterances, lip, nose

0.5
value

and throat noise.

4

val_counselor
aro_counselor

Analysis0.00.5Figure 1 shows the post-recording questionnaire

1.0

reasult. Firstly, the result shows that the emotion
1.5

inducer videos are eﬀective in eliciting a negative

05

1015

20time (m)

2530

3540

emotional response (video neg). The participantsalso reported an agreement towards the positiveFig. 2

emotional eﬀect of the conversation (chat pos) as(r for valence: 0.78, arousal: 0.46).

Emotion annotations of one of the sessions

well as the feeling that the conversation helps themto process their emotion (helps emo).

Strongest

5Conclusionagreement is observed on the enjoyment of theWe present recordings of dyadic social-aﬀective in-conversation (enjoyed), followed by the feeling ofteractions between a professional counselor and 30

being understood by the counselor (understood).

participants, amounting to 23 hours and 41 min-Emotional connection appears to be the most difficult feeling to achieve through the interaction

utes of annotated data. The proposed database differs from existing ones in that it is explicitly de-(emo connect), possibly due to the limited time andsigned to allow the observation of emotion changes

lack of continuity of the interaction. In general, theat interpersonal level, involving an external party

participants express that they would like to interact

that guides the emotional process that follows nega-with the counselor again in the future (chat again).

tive emotion induction. We recruited a professional

0%

20%

40%

60%

80%

100%

counselor to fill the role of an expert in facilitating this process. The presented corpus is designed

video_neg
chat_pos

to support aﬀective computing research that focuses

helps_emo

on emotion at interpersonal or social level. Towards

understood

this direction, in the future we hope to utilize theenjoyedcorpus for developing a dialogue system with an

emo_connect
chat_again
strongly agree

emotionally intelligent dialogue strategy.
agree

neutraldisagreestrongly disagree

Acknowledgement
Fig. 1 Post-recording questionnaire result.
We investigate the correlation between the selfreport and perceived emotion annotations with
Pearson’s correlation coeﬃcient r. We found thatthe correlations for valence annotation are consistently stronger than that of activation. Strong correlation (r ≥ 0.5) for valence are observed in 68.33%
of the annotated sessions but only 8.3% for arousal.
The average correlation is 0.585 for valence and0.044 for arousal. The annotations from a session
is depicted in Figure 2. We notice that in general
the self-reported and perceived annotations are correlated more strongly when the emotion of the par-This research and development work was supportedby the MIC/SCOPE #152307004, as well as JSPS KAKENHI Grant Numbers JP17H06101 and JP17K00237.

参考文献[1] Rainer Westermann, Gunter Stahl, and F Hesse. Relative eﬀectiveness and validity of mood induction
procedures: analysis. European Journal of social psychology, 26:557–580, 1996.
[2] James A Russell. A circumplex model of aﬀect. Journal of personality and social psychology, 39(6):1161,
1980.
[3] Roddy Cowie, Ellen Douglas-Cowie, Susie Savvidou,
Edelle McMahon, Martin Sawey, and Marc Schröder.
‘FEELTRACE’: An instrument for recording perceived emotion in real time. In ISCA tutorial andresearch workshop (ITRW) on speech and emotion,
2000.