SocialNLP EmotionX 2019 Challenge Overview: Predicting Emotions in Spoken Dialogues and Chats

Abstract: We present an overview of the EmotionX 2019 Challenge, held at the 7th International Workshop on Natural Language Processing for Social Media (SocialNLP), in conjunction with IJCAI 2019. The challenge entailed predicting emotions in spoken and chat-based dialogues using augmented EmotionLines datasets. EmotionLines contains two distinct datasets: the first includes excerpts from a US-based TV sitcom episode scripts (Friends) and the second contains online chats (EmotionPush). A total of thirty-six teams registered to participate in the challenge. Eleven of the teams successfully submitted their predictions performance evaluation. The top-scoring team achieved a micro-F1 score of 81.5% for the spoken-based dialogues (Friends) and 79.5% for the chat-based dialogues (EmotionPush).

Emotions are a central component of our existence as human beings, and are manifested by physiological and psychological changes that often affect behavior and action. Emotions involve a complicated interplay of mind, body, language, and culture BIBREF0.
Detecting and recognizing emotions is a difficult task for machines. Nevertheless, following the successful use of computational linguistics to analyze sentiment in texts, there is growing interest in the more difficult task of the automatic detection and classification of emotions in texts.
The detection of emotions in text is a complicated challenge for multiple reasons: first, emotions are complex entities, and no universally-agreed upon psychological model of emotions exists. Second, isolated texts convey less information compared to a complete human interaction in which emotions can be detected from the other person's facial expressions, listening to their tone of voice, etc. However, due to important applications in fields such as psychology, marketing, and political science, research in this topic is now expanding rapidly BIBREF1.
In particular, dialogue systems such as those available on social media or instant messaging services are rich sources of textual data and have become the focus of much attention. Emotions of utterances within dialogues can be detected more precisely due to the presence of more context. For example, a single utterance (“OK!”) might convey different emotions (happiness, anger, surprise), depending on its context. Taking all this into consideration, in 2018 the EmotionX Challenge asked participants to detect emotions in complete dialogues BIBREF2. Participants were challenged to classify utterances using Ekman's well-known theory of six basic emotions (sadness, happiness, anger, fear, disgust, and surprise) BIBREF3.
For the 2019 challenge, we built and expanded upon the 2018 challenge. We provided an additional 20% of data for training, as well as augmenting the dataset using two-way translation. The metric used was micro-F1 score, and we also report the macro-F1 score.
A total of thirty-six teams registered to participate in the challenge. Eleven of the teams successfully submitted their data for performance evaluation, and seven of them submitted technical papers for the workshop. Approaches used by the teams included deep neural networks and SVM classifiers. In the following sections we expand on the challenge and the data. We then briefly describe the various approaches used by the teams, and conclude with a summary and some notes. Detailed descriptions of the various submissions are available in the teams' technical reports.
The two datasets used for the challenge are Friends and EmotionPush, part of the EmotionLines corpus BIBREF4. The datasets contain English-language dialogues of varying lengths. For the competition, we provided 1,000 labeled dialogues from each dataset for training, and 240 unlabeled dialogues from each dataset for evaluation. The Friends dialogues are scripts taken from the American TV sitcom (1994-2004). The EmotionPush dialogues are from Facebook Messenger chats by real users which have been anonymized to ensure user privacy. For both datasets, dialogue lengths range from 5 to 24 lines each. A breakdown of the lengths of the dialogues is shown in Table .
We employed workers using Amazon Mechanical Turk (aka AMT or MTurk) to annotate the dialogues BIBREF5. Each complete dialogue was offered as a single MTurk Human Intelligence Task (HIT), within which each utterance was read and annotated for emotions by the worker. Each HIT was assigned to five workers. To ensure workers were qualified for the annotation task, we set up a number of requirements: workers had to be from an English-speaking country (Australia, Canada, Great Britain, Ireland, New Zealand, or the US), have a high HIT approval rate (at least 98%), and have already performed a minimum of 2,000 HITs.
In the datasets, each utterance is accompanied by an annotation and emotion. The annotation contains the raw count of votes for each emotion by the five annotators, with the order of the emotions being Neutral, Joy, Sadness, Fear, Anger, Surprise, Disgust. For example, an annotation of “2000030” denotes that two annotators voted for “neutral”, and three voted for “surprise”.
The labeled emotion is calculated using the absolute majority of votes. Thus, if a specific emotion received three or more votes, then that utterance is labeled with that emotion. If there is no majority vote, the utterance is labeled with “non-neutral” label. In addition to the utterance, annotation, and label, each line in each dialogue includes the speaker's name (in the case of EmotionPush, a speaker ID was used). The emotion distribution for Friends and EmotionPush, for both training and evaluation data, is shown in Table .
We used Fleiss' kappa measure to assess the reliability of agreement between the annotators BIBREF6. The value for $\kappa $-statistic is $0.326$ and $0.342$ for Friends and EmotionPush, respectively. For the combined datasets the value of the $\kappa $-statistic is $0.345$.
Sample excerpts from the two datasets, with their annotations and labels, are given in Table .
NLP tasks require plenty of data. Due to the relatively small number of samples in our datasets, we added more labeled data using a technique developed in BIBREF7 that was used by the winning team in Kaggle's Toxic Comment Classification Challenge BIBREF8. The augmented datasets are similar to the original data files, but include additional machine-computed utterances for each original utterance. We created the additional utterances using the Google Translate API. Each original utterance was first translated from English into three target languages (German, French, and Italian), and then translated back into English. The resulting utterances were included together in the same object with the original utterance. These “duplex translations” can sometimes result in the original sentence, but many times variations are generated that convey the same emotions. Table shows an example utterance (labeled with “Joy”) after augmentation.
A dedicated website for the competition was set up. The website included instructions, the registration form, schedule, and other relevant details. Following registration, participants were able to download the training datasets.
The label distribution of emotions in our data are highly unbalanced, as can be seen in Figure FIGREF6. Due to the small number of three of the labels, participants were instructed to use only four emotions for labels: joy, sadness, anger, and neutral. Evaluation of submissions was done using only utterances with these four labels. Utterances with labels other than the above four (i.e., surprise, disgust, fear or non-neutral) were discarded and not used in the evaluation.
Scripts for verifying and evaluating the submissions were made available online. We used micro-F1 as the comparison metric.
A total of eleven teams submitted their evaluations, and are presented in the online leaderboard. Seven of the teams also submitted technical reports, the highlights of which are summarized below. More details are available in the relevant reports.
BIBREF9 Two different BERT models were developed. For Friends, pre-training was done using a sliding window of two utterances to provide dialogue context. Both Next Sentence Prediction (NSP) phase on the complete unlabeled scripts from all 10 seasons of Friends, which are available for download. In addition, the model learned the emotional disposition of each of six main six main characters in Friends (Rachel, Monica, Phoebe, Joey, Chandler and Ross) by adding a special token to represent the speaker. For EmotionPush, pre-training was performed on Twitter data, as it is similar in nature to chat based dialogues. In both cases, special attention was given to the class imbalance issue by applying “weighted balanced warming” on the loss function.
BIBREF10 BERT is post-trained via Masked Language Model (MLM) and Next Sentence Prediction (NSP) on a corpus consisting of the complete and augmented dialogues of Friends, and the EmotionPush training data. The resulting token embeddings are max-pooled and fed into a dense network for classification. A $K$-fold cross-validation ensemble with majority voting was used for prediction. To deal with the class imbalance problem, weighted cross entropy was used as a training loss function.
BIBREF11 A pre-trained BERT is fine-tuned using filtered training data which only included the desired labels. Additional augmented data with joy, sadness, and anger labels are also used. BERT is then fed into a standard feed-forward-network with a softmax layer used for classification.
BIBREF12 A support vector machine (SVM) was used for classification. Words are ranked using a per-emotion TF-IDF score. Experiments were performed to verify whether the previous utterance would improve classification performance. Input to the Linear SVM was done using one-hot-encoding of top ranking words.
BIBREF13 The classifier uses a pre-trained BERT model followed by a feed-forward neural network with a softmax output. Due to the overwhelming presence of the neutral label, a classifying cascade is employed, where the majority classifier is first used to decide whether the utterance should be classified with “neutral” or not. A second classifier is used to focus on the other emotions (joy, sadness, and anger). Dealing with the imbalanced classes is done through the use of a weighted loss function.
BIBREF14 BERT is first used to generate word and sentence embeddings for all utterances. The resulting calculated word embeddings are fed into a Convolutional Neural Network (CNN), and its output is then concatenated with the BERT-generated sentence embeddings. The concatenated vectors are then used to train a bi-directional GRU with a residual connection followed by a fully-connected layer, and finally a softmax layer produces predictions. Class imbalance is tackled using focal loss BIBREF15.
BIBREF16 A word embedding layer followed by a bi-directional GRU-based RNN. Output from the RNN was fed into a single-node classifier. The augmented dataset was used for training the model, but “neutral”-labeled utterances were filtered to deal with class imbalance.
The submissions and the final results are summarized in Tables and . Two of the submissions did not follow up with technical papers and thus they do not appear in this summary. We note that the top-performing models used BERT, reflecting the recent state-of-the-art performance of this model in many NLP tasks. For Friends and EmotionPush the top micro-F1 scores were 81.5% and 88.5% respectively.
An evaluation summary of the submissions is available in Tables and . We only present the teams that submitted technical reports. A full leaderboard that includes all the teams is available on the challenge website. This section highlights some observations related to the challenge. Identical utterances can convey different emotions in different contexts. A few of the models incorporated the dialogue context into the model, such as the models proposed by teams IDEA and KU.
Most of the submissions used deep learning models. Five of the models were based on the BERT architecture, with some using pre-trained BERT. Some of the submissions enhanced the model by adding context and speaker related encoding to improve performance. We also received submissions using more traditional networks such as CNN, as well as machine learning classics such as SVM. The results demonstrate that domain knowledge, feature engineering, and careful application of existing methodologies is still paramount for building successful machine learning models.
Emotion detection in text often suffers from a data imbalance problem, our datasets included. The teams used two approaches to deal with this issue. Some used a class-balanced loss functions while others under-sampled classes with majority label “neutral”. Classification performance of underrepresented emotions, especially sadness and anger, is low compared to the others. This is still a challenge, especially as some real-world applications are dependent on detection of specific emotions such as anger and sadness.
The discrete 6-emotion model and similar models are often used in emotion detection tasks. However, such 1-out-of-n models are limited in a few ways: first, expressed emotions are often not discrete but mixed (for example, surprise and joy or surprise and anger are often manifested in the same utterance). This leads to more inter-annotator disagreement, as annotators can only select one emotion. Second, there are additional emotional states that are not covered by the basic six emotions but are often conveyed in speech and physical expressions, such as desire, embarrassment, relief, and sympathy. This is reflected in feedback we received from one of the AMT workers: “I am doing my best on your HITs. However, the emotions given (7 of them) are a lot of times not the emotion I'm reading (such as questioning, happy, excited, etc). Your emotions do not fit them all...”.
To further investigate, we calculated the per-emotion $\kappa $-statistic for our datasets in Table . We see that for some emotions, such as disgust and fear (and anger for EmotionPush), the $\kappa $-statistic is poor, indicating ambiguity in annotation and thus an opportunity for future improvement. We also note that there is an interplay between the emotion label distribution, per-emotion classification performance, and their corresponding $\kappa $ scores, which calls for further investigation.
One of the main requirements of successful training of deep learning models is the availability of high-quality labeled data. Using AMT to label data has proved to be useful. However, current data is limited in quantity. In addition, more work needs to be done in order to measure, evaluate, and guarantee annotation quality. In addition, the Friends data is based on an American TV series which emphasizes certain emotions, and it remains to be seen how to transfer learning of emotions to other domains.
This research is partially supported by Ministry of Science and Technology, Taiwan, under Grant no. MOST108-2634-F-001-004- and MOST107-2218-E-002-009-.

Figure and table captions:
Table 1: Reliability of Agreement (κ)
Table 2: Emotion Label Distribution
Table 3: Dialogue Length Distribution and Number of Utterances
Table 4: Example of Augmented Utterance
Table 5: Dialogue Excerpts from Friends (top) and EmotionPush (bottom)
Table 6: F-scores for Friends (%)
Table 7: F-scores for EmotionPush (%)
Table 8: Per-emotion Reliability of Agreement (κ)

Questions about this paper:
What is the size of the second dataset?
How large is the first dataset?
Who was the top-scoring team?

Answers:
1 000 labeled dialogues for training and 240 unlabeled dialogues for evaluation
1 000 labeled dialogues for training and 240 unlabeled dialogues for evaluation
IDEA