Nurul Lubis, Sakriani Sakti, Graham Neubig, Koichiro Yoshino, Satoshi Nakamura (NAIST) 1
Introduction
The study of social-affective communication is concerned with the role of emotion in human in- teraction.  Emotion plays a two way role in human conversation; we express our emotion and are af- fected by our conversational partner. This is yet to be completely replicated in human-computer inter- action (HCI). In addition to the more traditional works on recognition and simulation, there has recently been an increasing interest in emotional triggers, study- ing what causes emotion in the first place.  A recent study by Hasegawa et al. [1] addresses this issue by predicting and eliciting emotion in online conversa- tion.  We have followed up this study by perform- ing a similar task on natural spoken conversation. We recognized emotion based on a speaker's utter- ances and analyzed the conversational partner's as the emotional triggers [2, 3]. In this paper, we extend upon the previous works by trying to predict a person's emotional reaction in a social-affective conversation.  We e xamine not only the person having the reaction, but also their conversational partner. The prediction is aimed to accommodate the ability to predict the response to an emotion trigger. Two languages are examined: English and Indonesian. 2
Task Description
Figure 1 presents the workflow of this paper. We extract tri-turns from spontaneous conversation data and try to predict the emotional response oc- curred within. We collect the data from television talk shows, which contain real conversation with natural emo- tion occurrences. We utilize the English-Indonesian TV Talk show corpus [4], containing 62 minutes of English interaction and 94 minutes of Indonesian. In natural conversation, to properly analyze the fluctuation of emotion, it is necessary to ensure that the observed sequences of conversation are in re- sponse to each other.  Thus, we group consecutive sequences of conversation into a unit called a tri- turn [5]. Three consecutive sequences of speech in a conversation is considered a tri-turn when the sec- ond sequence is in response to the first, and the third is in response to the second.  On each tri-turn, we observe the first two turns and predict emotion re- sponse occurring on the third. To observe the data from the dialogue perspective, we define a set of dialogue acts adapted from [6] to describe the structure of discourse. The 17 dialogue act labels are given in Table 1.
We define the emotion scope based on the circum- plex model of affect [7].  Two dimensions of emo- tion are defined: valence and arousal. Valence mea- sures the positivity or negativity of emotion.  On the other hand, arousal measures the activity of emo- tion.  From the valence-arousal scale, we derive 5 common emotion terms: happiness, anger, sadness, contentment, and neutral.  We utilize both points of view of emotion to label the data: the value of valence and arousal; and the degree of presence of the emotion classes. The emotional responses considered in this pa- per can be categorized into three events.  As the events can happen simultaneously in different ob- servations, we examine the emotion class, valence level, and arousal level separately.  For each obser- vation, we try to predict whether it stays constant, changes from postive to negative ( drop ), or changes from negative to positive ( raise ).
3
Automatic Prediction of Emotional
Responses
As mentioned previously, on a tri-turn, we ob- serve the first two turns to predict emotion response occurring on the third.  This means, on each tri- turn, we stack the features of the first two turns to gather information of the context. The features include acoustic features and dialogue act labels. To gather information from the speech, we extract acoustic features as defined in the INTERSPEECH 2009 emotion challenge [8] using the openSMILE feature extractor. To balance the number of instances and features, we perform correlation-based feature extraction and linear discriminant analysis of our feature set.  After reducing the dimensions, we train a neural network classifier with one hidden layer using Theano and the PDNN toolkit. To summarize, for all tri-turns, based on the acoustic features and dialogue act labels of the first two turns, we attempt to predict whether emotion events defined occurred on the third.  With three classes for prediction, we have a chance rate of 33.33%. Figure 2 presents our response prediction accu- racy.  For both languages, the prediction regarding valence has the highest performance, followed by arousal and emotion class.  This could mean that valence and its fluctuations are expressed in speech more than is the case for arousal. The suboptimal performance is likely due to the limited amount of data used in this study.  Inher- ently, there are numerous factors that leads to a change of emotion in a conversation.  This means that to properly recognize patterns for such events, a large number of features are required. It is likely that if we could prepare more data in the future, the accuracy will increase significantly.
4
Conclusion
In this paper, we presented a study on social- affective communication for automatic prediction of emotional responses.  In providing users the most natural HCI, emotion is an aspect that should not be overlooked.  Our experiment on automatic predic- tion offers an approach in equipping conversational agents and dialogue systems with social-affective awareness. In future studies, we hope to include more modali- ties of interaction in observing the dynamics of emo- tion in interaction, such as textual and visual fea- tures. We also hope to incorporate this information directly into a dialogue system.
Acknowledgements
Part of this work was supported by JSPS KAK- ENHI Grant Number 24240032 and 26870371, as well as by a joint research project with Yanmar Co., Ltd.