SlowGuess/NAACL_2022_OCR huggingface dataset:
Joosung Lee, Woon Lee
Kakao Enterprise Corp., South Korea
{rung.joo, dan.kes}@kakaoenterprise.com
As the use of interactive machines grow, the task of Emotion Recognition in Conversation (ERC) became more important. If the machine-generated sentences reflect emotion, more human-like sympathetic conversations are possible. Since emotion recognition in conversation is inaccurate if the previous utterances are not taken into account, many studies reflect the dialogue context to improve the performances. Many recent approaches show performance improvement by combining knowledge into modules learned from external structured data. However, structured data is difficult to access in non-English languages, making it difficult to extend to other languages. Therefore, we extract the pre-trained memory using the pre-trained language model as an extractor of external knowledge. We introduce CoMPM, which combines the speaker's pre-trained memory with the context model, and find that the pre-trained memory significantly improves the performance of the context model. CoMPM achieves the first or second performance on all data and is state-of-the-art among systems that do not leverage structured data. In addition, our method shows that it can be extended to other languages because structured knowledge is not required, unlike previous methods. Our code is available on github<sup>1</sup>.
As the number of applications such as interactive chatbots or social media that are used by many users has recently increased dramatically, Emotion Recognition in Conversation (ERC) plays a more important role in natural language processing, and as a proof, a lot of research (Poria et al., 2019; Zhang et al., 2019; Ghosal et al., 2020; Jiao et al., 2020) has been conducted on the task.
The ERC module increases the quality of empathetic conversations with the users and can be
Figure 1: An example of MELD dataset
utilized when sending tailored push messages to the users (Shin et al., 2019; Zandie and Mahoor, 2020; Lin et al., 2020). In addition, emotion recognition can be effectively used for opinion mining, recommender systems, and healthcare systems where it can improve the service qualities by providing personalized results. As these interactive machines increase, the ERC module plays an increasingly important role.
Figure 1 is an example of a conversation in which two speakers are angry at each other. The emotion of speaker B's utterance ("How'd you get to that?") is angry. If the system does not take into account previous utterances, it is difficult to properly recognize emotions. Like the previous studies (Ghosal et al., 2020), we show that the utterance-level emotion recognition, which does not consider the previous utterance, have limitations and experiments result in poor performances.
Therefore, recent studies are attempting to recognize emotions while taking into account the previous utterances. Representatively, DialogueRNN (Majumder et al., 2019) recognizes the present emotion by tracking context from the previous utterances and the speaker's emotion. AGHMN (Jiao et al., 2020) considers the previous utterances through memory summarizing using GRU with attention.
Many recent studies use external knowledge to improve the ERC performance. However, this exter
$^{1}$ https://github.com/rungjoo/CoMPM
Human Language Technologies, pages 5669 - 5679
nal knowledge is often only available in English. In order to utilize the previous methods in languages of other countries, it is expensive and difficult to utilize because external knowledge data must be newly constructed. In recent NLP studies, due to the effectiveness of the pre-trained language model, it has already been developed in many countries. Since pre-trained language models are trained by unsupervised learning, these models are relatively usable approaches regardless of language types. Petroni et al. (2019) introduces that these language models can be used as knowledge bases and have many advantages over the structured knowledge bases. Based on these studies, we eliminate the dependence on structured external data used in cutting-edge systems and use a pre-trained language model as a feature extractor of knowledge.
CoMPM, introduced in this paper, is composed of two modules that take into account previous utterances in dialogue. (1) The first is a context embedding module (CoM) that reflects all previous utterances as context. CoM is an auto-regressive model that predicts the current emotion through attention between the previous utterances of the conversation and the current utterance. (2) The second is a pre-trained memory module (PM) that extracts memory from utterances. We use the output of the pre-trained language model as the memory embedding where the utterances are passed into the language model. We use the PM to help predict the emotion of the speaker by taking into account the speaker's linguistic preferences and characteristics.
We experiment on 4 different English ERC datasets. Multi-party datasets are MELD (Poria et al., 2019) and EmoryNLP (Zahiri and Choi, 2018), and dyadic datasets are IEMOCAP (Busso et al., 2008) and DailyDialog (Li et al., 2017). CoMPM achieves the first or second performance according to the evaluation metric compared to all previous systems. We perform an ablation study on each module to show that the proposed approach is effective. Further experiments also show that our approach can be used in other languages and show the performance of CoMPM when the number of data is limited.
Many recent studies use external knowledge to improve the ERC performance. KET (Zhong et al., 2019) is used as external knowledge based on ConceptNet (Speer et al., 2017) and emotion lex
icon NRC_VAD (Mohammad, 2018) as the commonsense knowledge. ConceptNet is a knowledge graph that connects words and phrases in natural language using labeled edges. NRC_VAD Lexicon has human ratings of valence, arousal, and dominance for more than 20,000 English words. COSMIC (Ghosal et al., 2020) and Psychological (Li et al., 2021) improve the performance of emotion recognition by extracting commonsense knowledge of the previous utterances. Commonsense knowledge feature is extracted and leveraged with COMET (Bosselut et al., 2019) trained with ATOMIC (The Atlas of Machine Commonsense) (Sap et al., 2019). ATOMIC has 9 sentence relation types with inferential if-then commonsense knowledge expressed in text. ToDKAT (Zhu et al., 2021) improves performance by combining commonsense knowledge using COMET and topic discovery using VHRED (Serban et al., 2017) to the model.
Ekman (Ekman, 1992) constructs taxonomy of six common emotions (Joy, Sadness, Fear, Anger, Surprise, and Disgust) from human facial expressions. In addition, Ekman explains that a multimodal view is important for multiple emotions recognition. The multi-modal data such as MELD and IEMOCAP are some of the available standard datasets for emotion recognition and they are composed of text, speech and vision-based data. Datcu and Rothkrantz (2014) uses speech and visual information to recognize emotions, and (Alm et al., 2005) attempts to recognize emotions based on text information. MELD and ICON (Hazarika et al., 2018a) show that the more multi-modal information is used, the better the performance and the text information plays the most important role. Multimodal information is not always given in most social media, especially in chatbot systems where they are mainly composed of text-based systems. In this work, we design and introduce a text-based emotion recognition system using neural networks.
In the previous studies, such as Hazarika et al. (2018b); Zadeh et al. (2017); Majumder et al. (2019), most works focused on dyadic-party conversation. However, as the multi-party conversation datasets including MELD and EmoryNLP have become available, a lot of recent research is being conducted on multi-party dialogues such as Zhang et al. (2019); Jiao et al. (2020); Ghosal et al. (2020). In general, the multi-party conversations have higher speaker dependency than the
dyadic-party dialogues, therefore have more conditions to consider and result in poor performance.
Zhou et al. (2018); Zhang et al. (2018a) shows that commonsense knowledge is important for understanding conversations and generating appropriate responses. Liu et al. (2020) reports that the lack of external knowledge makes it difficult to classify implicit emotions from the conversation history. EDA (Bothe et al., 2020) expands the multi-modal emotion datasets by extracting dialog acts from MELD and IEMOCAP and finds out that there is a correlation between dialogue acts and emotion labels.
In a conversation,  $M$  sequential utterances are given as  $[(u_{1},p_{u_{1}}),(u_{2},p_{u_{2}}),\dots,(u_{M},p_{u_{M}})]$ .  $u_{i}$  is the utterance which the speaker  $p_{u_i}$  uttered, where  $p_{u_i}$  is one of the conversation participants. While  $p_{u_i}$  and  $p_{u_j}$  ( $i\neq j$ ) can be the same speaker, the minimum number of the unique conversation participants should be 2 or more. The ERC is a task of predicting the emotion  $e_t$  of  $u_t$ , the utterance of the  $t$ -th turn, given the previous utterances  $h_t = \{u_1,\dots,u_{t - 1}\}$ . Emotions are labeled as one of the predefined classes depending on the dataset, and the emotions we experimented with are either 6 or 7. We also experimented with a sentiment classification dataset which provides sentiment labels consisting of positive, negative and neutral.
Figure 2 shows an overview of our model. Our ERC neural network model is composed of two modules. The first is CoM which catches the underlying effect of all previous utterances on the current speaker's emotions. Therefore, we propose a context model to handle the relationship between the current and the previous utterances. The second one is PM that leverages only the speaker's previous utterances, through which we want to reflect the speaker's knowledge.
If the CoM and PM are based on different backbones, we consider them to be unaligned with respect to each other's output representations. Therefore, we design the PM to follow CoM so that the output representations of CoM and PM can mutually understand each other. If CoM and PM are based on different architectures, CoMPM is trained to understand each other's representations
by matching dimensions using  $\mathbf{W}_{\mathbf{p}}$  in Equation 4. The combination of CoM and PM is described in Section 4.5.
3.3 CoM: Context Embedding Module
The context embedding module predicts  $e_t$  by considering all of the utterances before the  $t$ -th turn as the dialogue context. The example in Figure 2 shows how the model predicts the emotion of  $u_6$  uttered by  $s_A$ , given a conversation of three participants  $(s_A, s_B, s_C)$ . The previous utterances are  $h_6 = \{u_1, \dots, u_5\}$  and  $e_6$  is predicted while considering the relationship between  $u_6$  and  $h_6$ .
We consider multi-party conversations where 2 or more speakers are involved. A special token  $< s_{\mathbb{P}} >$  is introduced to distinguish participants in the conversation and to handle the speaker's dependency where  $\mathbb{P}$  is the set of participants. In other words, the same special token appears before the utterances of the same speaker.
We use an Transformer encoder as a context model. In many natural language processing tasks, the effectiveness of the pre-trained language model has been proven, and we also set the initial state of the model to RoBERTa (Liu et al., 2019). RoBERTa is an unsupervised pre-trained model with large-scale open-domain corpora of unlabeled text.
We use the embedding of the special token  $<\mathrm{cls}>$  to predict emotion. The  $<\mathrm{cls}>$  token is concatenated at the beginning of the input and the output of the context model is as follows:
$$
\boldsymbol {c} _ {t} = \operatorname {C o M} \left(<   \operatorname {c l s} >, \mathbb {P}: _ {t - 1}, h _ {t}, u _ {t}\right) \tag {1}
$$
where  $\mathbb{P}_{:t - 1}$  is the set of speakers in the previous turns.  $c_{t} \in \mathbb{R}^{1 \times h_{c}}$  and  $h_{c}$  is the dimension of CoM.
3.4 PM: Pre-trained Memory Module
External knowledge is known to play an important role in understanding conversation. Pre-trained language models can be trained on numerous corpora and be used as an external knowledge base. Inspired by previous studies that the speaker's knowledge helps to judge emotions, we extract and track pre-trained memory from the speaker's previous utterances to utilize the emotions of the current utterance  $u_{t}$ . If the speaker has never appeared before the current turn, the result of the pre-trained memory is considered a zero vector.
Since  $<\mathrm{cls}>$  is mostly used for the task of classifying sentences, we use the embedding output of the  $<\mathrm{cls}>$  token as a vector representing the utterance as follows:
Figure 2: Our model consists of two modules: a context embedding module and a pre-trained memory module. The figure shows an example of predicting emotion of  $u_{6}$ , from a 6-turn dialogue context. A, B, and C refer to the participants in the conversation, where  $s_A = p_{u_1} = p_{u_3} = p_{u_6}$ ,  $s_B = p_{u_2} = p_{u_5}$ ,  $s_C = p_{u_3}$ .  $\mathbf{W}_{\mathbf{o}}$  and  $\mathbf{W}_{\mathbf{p}}$  are linear matrices.
$$
\boldsymbol {k} _ {i} = \operatorname {P M} (<   \mathrm {c l s} >, u _ {i}) \tag {2}
$$
where  $p_{u_i} = p_S$ ,  $\mathbf{S}$  is the speaker of the current utterance.  $\pmb{k}_i\in \mathbb{R}^{1\times h_k}$  and  $h_k$  is the dimension of PM.
3.5 CoMPM: Combination of CoM and PM
We combine CoM and PM to predict the speaker's emotion. In many dialogue systems (Zhang et al., 2018b; Ma et al., 2019), it is known that utterances close to the current turn are important for response. Therefore, we assume that utterances close to the current utterance will be important in emotional recognition.
3.5.1 Tracking Method
We use  $k_{i}$  tracking method using GRU. The tracking method assumes that the importance of all previous speaker utterances to the current emotion is not equal and varies with the distance of the current utterance. In other words, since the flow of conversation changes as it progresses, the effect on emotion may differ depending on the distance from the current utterance. We track and capture the sequential position information of  $k_{i}$  using a unidirectional GRU:
$$
\boldsymbol {k} \boldsymbol {t} _ {t} = \mathrm {G R U} (\boldsymbol {k} _ {i _ {1}}, \boldsymbol {k} _ {i _ {2}}, \dots , \boldsymbol {k} _ {i _ {n}}) \qquad (3)
$$
where  $t$  is the turn index of the current utterance,  $n$  is the number of previous utterances of the
speaker, and  $i_s$  ( $s = 1,2,\dots,n$ ) is each turn uttered.  $\pmb{kt}_t \in \mathbb{R}^{1 \times h_c}$  is the output of  $\pmb{k}_{i_n}$  and as a result, the knowledge of distant utterance is diluted and the effect on the current utterance is reduced.
GRU is composed of 2-layers, the dimension of the output vector is  $h_c$ , and the dropout is set to 0.3 during training. Finally, the output vector  $\boldsymbol{o}_t$  is obtained by adding  $\boldsymbol{k} \boldsymbol{t}_t$  and  $\boldsymbol{c}_t$  in Equation 4.
$$
\boldsymbol {o} _ {t} = \boldsymbol {c} _ {t} + \mathbf {W} _ {\mathbf {p}} (\boldsymbol {k} \boldsymbol {t} _ {t}) \tag {4}
$$
where,  $\mathbf{W}_{\mathbf{p}}$  is a matrix that projects the pretrained memory to the dimension of the context output, and is used only when PM and CoM are different pre-trained language models.
3.5.2 Emotion Prediction
Softmax is applied to the vector multiplied by  $\boldsymbol{o}_t$  and the linear matrix  $\mathbf{W}_{\mathbf{o}} \in \mathbb{R}^{h_e \times h_c}$  to obtain the probability distribution of emotion classes, where  $h_e$  is the number of emotion classes.  $e_t$  is the predicted emotion class that corresponds to the index of the largest probability from the emotion class distribution.
$$
P (e) = \operatorname {s o f t m a x} \left(\mathbf {W} _ {\mathbf {o}} \left(\boldsymbol {o} _ {t}\right)\right) \tag {5}
$$
The objective is to minimize the cross entropy loss so that  $e_t$  is the same as the ground truth emotional label.
Dataset	dialogues	utterance	classes	Evaluation Metrics
train	dev	test	train	dev	test
IEMOCAP	108	12	31	5163	647	1623	6	weighted avg F1
DailyDialog	11118	1000	1000	87170	8069	7740	7(6)	Macro F1 & Micro F1
MELD	1038	114	280	9989	1109	2610	3,7	weighted avg F1
EmoryNLP	713	99	85	9934	1344	1328	3,7	weighted avg F1
Table 1: Statistics and descriptions for the four datasets. DailyDialog uses 7 classes for training, but we measure Macro-F1 for only 6 classes excluding neutral. MELD and EmoryNLP are used to measure weighted avg F1 for both emotion (7) and sentiment (3) classes.
We experiment on four benchmark datasets. MELD (Poria et al., 2019) and EmoryNLP (Zahiri and Choi, 2018) are multi-party datasets, while IEMOCAP (Busso et al., 2008) and DailyDialog (Li et al., 2017) are dyadic-party datasets. The statistics of the dataset are shown in Table 1.
IEMOCAP is a dataset involving 10 speakers, and each conversation involves 2 speakers and the emotion-inventory is given as "happy, sad, angry, excited, frustrated and neutral". The train and development dataset is a conversation involving the previous eight speakers, and the train and development are divided into random splits at a ratio of 9:1. The test dataset is a conversation involving two later speakers.
DailyDialog is a dataset of daily conversations between two speakers and the emotion-inventory is given as "anger, disgust, fear, joy, surprise, sadness and neutral". Since more than  $82\%$  of the data are tagged as neutral, neutral emotions are excluded when evaluating systems with Micro-F1 as did in the previous studies.
MELD is a dataset based on Friends TV show and provides two taxonomy: emotion and sentiment. MELD's emotion-inventory is given as "anger, disgust, sadness, joy, surprise, fear and neutrality" following Ekman (Ekman, 1992) and sentiment-inventory is given as "positive, negative and neutral".
EmoryNLP, like MELD, is also a dataset based on Friends TV show, but the emotion-inventory is given as "joyful, peaceful, powerful, scared, mad, sad and neutral". Sentiment labels are not provided, but sentiment classes can be grouped as follows: positive: {joyful, peaceful, powerful}, negative: {scared, mad, sad}, neutral: {neutral}
4.2 Training Setup
We use the pre-trained model from the huggingface library  $^{2}$ . The optimizer is AdamW and the learning rate is 1e-5 as an initial value. The learning rate scheduler used for training is get_linear_schedule_with_warmup, and the maximum value of 10 is used for the gradient clipping. We select the model with the best performance on the validation set. All experiments are conducted on one V100 GPU with 32GB memory.
4.3 Previous Method
We show that the proposed approach is effective by comparing it with various baselines and the state-of-the-art methods.
KET (Zhong et al., 2019) is a Knowledge Enriched Transformer that reflects contextual utterances with a hierarchical self-attention and leverages external commonsense knowledge by using a context-aware affective graph attention mechanism.
DialogueRNN (Majumder et al., 2019) uses a GRU network to keep track of the individual party states in the conversation to predict emotions. This model assumes that there are three factors in emotion prediction: the speaker, the context from the preceding utterances and the emotion of the preceding utterances. Also, Ghosal et al. (2020) shows the performance of RoBERTa+DialogueRNN when the vectors of the tokens are extracted with a pretrained RoBERTa.
RGAT+P (Ishiwatari et al., 2020) (relational graph attention networks) proposes relational position encodings with sequential information reflecting the relational graph structure, which shows that both the speaker dependency and the sequential information can be captured.
HiTrans (Li et al., 2020) proposes a transformer-based context- and speaker-sensitive model. Hi
Models	IEMOCAP	DailyDialog	MELD	EmoryNLP
W-Avg F1	Macro F1	Micro F1	W-Avg F1 (3-cls)	W-Avg F1 (7-cls)	W-Avg F1 (3-cls)	W-Avg F1 (7-cls)
KET*	59.56	-	53.37	-	58.18	-	34.39
RoBERTa DialogueRNN	64.76	49.65	57.32	72.14	63.61	55.36	37.44
RGAT+P	65.22	-	54.31	-	60.91	-	34.42
HiTrans	64.5	-	-	-	61.94	-	36.75
COSMIC*	65.28	51.05	58.48	73.2	65.21	56.51	38.11
ERMC-DisGCN	-	-	-	-	64.22	-	36.38
Psychological*	66.96	51.95	59.75	-	65.18	-	38.8
DialogueCRN	66.05	-	-	-	58.39	-	-
ToDKAT*	61.33	52.56	58.47	-	65.47	-	43.12
CoMPM	66.33	53.15	60.34	73.08	66.52	57.14	37.37
CoM	65.05	51.17	58.63	71.67	64.9	56.27	36.34
PM	52.56	49.08	56.23	69.21	63.4	53.87	35.48
CoMPM(f)	69.46	51.67	59.02	73.04	65.77	55.44	38.93
CoMPM(s)	64.68	48.86	55.81	71.97	65.12	53.66	34.72
CoMPM(k)	64.3	52.33	59.09	72.67	66.22	56.62	36.96
Table 2: Comparison of our models with various previous models and the results on 4 datasets. Our models are trained 3 times for each experiment and the average of the scores is evaluated (same in other tables). Test performance is measured by the model with the best score in the validation dataset. Bold text indicates the best performance in each part (comparative models or ours). * indicates models that leverages structured external data.
Trans utilize BERT as the low-level transformer to generate local utterance representations, and feed them into another high-level transformer.
COSMIC (Ghosal et al., 2020) incorporates different elements of commonsense such as mental states, events and causal relations, and learns the relations between participants in the conversation. This model uses pre-trained RoBERTa as a feature extractor and leverages COMET trained with ATOMIC as the commonsense knowledge.
ERMC-DisGCN (Sun et al., 2021) proposes a discourse-aware graph neural network that utilizes self-speaker dependency of interlocutors as a relational convolution and informative cues of dependent utterances as a gated convolution.
Psychological (Li et al., 2021) uses commonsense knowledge as enrich edges and processes it with graph transformer. Psychological performs emotion recognition by utilizing intention of utterances from not only past contexts but also future context.
DialogueCRN (Hu et al., 2021) introduces an intuitive retrieving process, the reasoning module, which understands both situation-level and speaker-level contexts.
ToDKAT (Zhu et al., 2021) proposes a language model with topic detection added, and improves performance by combining it with commonsense knowledge. The performance of ToDKAT in MELD was re-released on github<sup>3</sup>.
4.4 Result and Analysis
Table 2 shows the performance of the previous methods and our models. CoM used alone does not leverage PM and predicts emotions by considering only the dialogue context. PM used alone is not used as a memory module, but the same backbone is used. PM used alone predicts emotion only with the utterance of the current turn without considering the context. CoMPM is a model in which both CoM and PM parameters are updated in the initial state of the pre-trained LM. CoMPM(f) is a model in which PM parameters are frozen in the initial state (pre-trained LM) and is not trained further, and CoMPM(s) is a model in which PM is trained from scratch. CoMPM(k) is a model in which PM is trained on ConceptNet. Following previous studies, we use the average vector for each token in  $\mathrm{PM(k)}$  as the feature of the utterance. We use the pre-trained model provided by the site  $^4$  as  $\mathrm{PM(k)}$ .
The effect of PM can be confirmed through the performance comparison between CoM and CoMPM, and the effect of CoM can be confirmed by comparing the results of CoM and PM. Since PM does not consider context, it showed worse performance than CoM, and the performance gap is larger in the IEMOCAP dataset with a higher average number of conversation turns. As a result, we show that the combination of CoM and PM is effective in achieving better performance.
We confirm the effect of PM structure in the model through the performance of CoMPM(s).
<sup>4</sup>https://huggingface.co/HungChau/distilbert-base-uncased-concept-extraction-kp20k-v1.2-concept-extraction-allwikipedia-v1.0
<sup>3</sup>https://github.com/something678/TodKat