SlowGuess/NAACL_2022_OCR huggingface dataset:
If PM parameters are not frozen and are instead randomly initialized (i.e.  $\mathrm{PM(s)}$ ), the performance deteriorates. CoMPM(s) performs worse than CoMPM, and even performs worse than CoM on the other datasets except for MELD. That is,  $\mathrm{PM(s)}$  cannot be regarded as a pre-trained memory because the parameters are randomly initialized, and simply increasing the model complexity does not help to improve the performance. CoMPM(f) shows similar performance to CoMPM and achieves better performance depending on the data.  $\mathrm{PM(f)}$  is not fine-tuned on the data, but it extracts general pre-trained memory from a pretrained language model. The comparison between PM and  $\mathrm{PM(f)}$  will be further described in Section 4.6. In addition, CoMPM(k) shows better performance than CoM, PM, and CoMPM(s) except for IEMOCAP. In IEMOCAP, CoMPM(k) has lower performance than CoM. For all datasets, CoMPM(k) performs slightly worse than CoMPM. In other words, ConceptNet improves the performance of CoMPM, but is not as effective as pretrained memory. As a result, we regard pre-trained memory as compressed knowledge, which can play a role similar to external knowledge used in cutting-edge systems.
The best performance of our approaches is CoMPM or CoMPM(f), both of which combine pre-trained memory. We achieve state-of-the-art performance among all systems that do not leverage structured external data and achieve the first or second performance even including systems that leverage external data. Therefore, our approach can be extended to other languages without structured external data as well, which is described in Section 4.7.
4.5 Combinations of CoM and PM
We experiment with the effect of pre-trained memory of different language models. To eliminate the influence of the PM structure, we freeze the parameters of PM and use it as a feature extractor. Table 3 shows the performance of the pretrained memory extracted by the different language models. If PM and CoM are based on different backbones, the pre-trained memory is projected through  $\mathbf{W}_{\mathbf{p}}$  as the dimension of the context output. RoBERTa+BERT and RoBERTa+GPT2 (combination of CoM and PM(f)) have lower performance than RoBERTa+RoBERTa, which is inferred because pre-trained memory of RoBERTa contains
Figure 3: Performance according to the size of training data of MELD and EmoryNLP
richer information than BERT and GPT2. Since there is a lot of training data in the diallydialog and  $\mathbf{W}_{\mathbf{p}}$  is fine-tuned to the data to mutually understand the pre-trained memory and context representation. Therefore, we infer that performance does not decrease even if the PM changes from the diallydialog. However, even if other PMs are used, the performance is improved compared to using only CoM, so the pre-trained memory of other language models is also effective for emotion recognition.
BERT+RoBERTa has a larger performance decrease than RoBERTa+BERT. In particular, in IEMOCAP data with a long average number of turns in the context, the performance deteriorates significantly. In addition, the performance of BERT+RoBERTa is lower than CoM (RoBERTa), which supports that the performance of CoM is a more important factor than the use of pre-trained memory. In other words, we confirm that CoM is more important than PM in our system for performance, and it is effective to focus on context modeling rather than external knowledge in the study of emotion recognition in conversation.
4.6 Training with Less Data
CoMPM is an approach that eliminates dependence on external sources and is easily extensible to any language. However, the insufficient number of emotional data available in other countries (or actual service) remains a problem. Therefore, we conduct additional experiments according to the use ratio of training data in MELD and EmoryNLP, where there is neither too much nor too little data. Figure 3 shows the performance of the model according to the ratio of the training data. In MELD and EmoryNLP, even if only  $60\%$  and  $80\%$  are used, respectively, the performance decreases by only 3 points.
CoM	PM(f)	IEMOCAP	DailyDialog	MELD	EmoryNLP
W-Avg F1	Macro F1	Micro F1	W-Avg F1 (7-cls)	W-Avg F1 (7-cls)
RoBERTa	BERT	65.93	52.74	59.97	65.41	37.25
(-3.53)	(+1.07)	(+0.95)	(-0.36)	(-1.68)
RoBERTa	GPT2	68.54	50.68	59.61	65.58	36.39
(-0.92)	(-0.99)	(+0.59)	(-0.19)	(-2.54)
BERT	RoBERTa	62.69	48.99	57.34	63.79	35.47
(-6.77)	(-2.68)	(-1.68)	(-1.98)	(-3.46)
Table 2 shows that CoMPM(f) achieves better performance than CoMPM in the emotion classification of IMEOCAP and EmoryNLP, which has fewer training data than other settings. On the other hand, if there is a lot of training data, CoMPM shows better performance. Figure 3 shows that as the number of data decreases, CoMPM(f) shows better results than CoMPM, which indicates that it is better to freeze the parameters of PM when the number of training data is insufficient. Therefore, if there is a lot of training data in the real-world application, CoMPM is expected to achieve good performance, otherwise it is CoMPM(f).
4.7 ERC in other languages
Previous studies mostly utilize external knowledge to improve performance, but these approaches require additional publicly available data, which are mainly available for English. Indeed, structured knowledge and ERC data are lacking in other languages. Our approach can be extended to other languages without building additional external knowledge and achieves better performance than simply using a pre-trained model.
4.7.1 Korean Dataset
We constructed data composed of two speakers in Korean, and emotion-inventory is given as "surprise, fear, ambiguous, sad, disgust, joy, bored, embarrassed, neutral". The total number of sessions is 1000, and the average number of utterance turns is 13.4. We use the data randomly divided into train:dev:test in a ratio of 8:1:1. This dataset is for actual service and is not released to the public.
4.7.2 Results in the Korean Dataset
In Korean, our results are shown in Table 4. The backbone of CoM and PM is Korean-BERT owned by the company, respectively. In the Korean dataset, like the English dataset, the performance is good in the order of CoMPM, CoM, and PM. Our approach
Table 3: Emotion recognition performance according to the combination of different backbones of CoM and PM. The value in parentheses is the performance difference from the original CoMPM(f) (RoBERTa + RoBERTa). We use the bert-large-uncased and GPT2-medium versions.
Models	Korean
W-Avg F1
PM	31.86
CoM	57.46
CoMPM	60.66
Table 4: Results of our approaches in Korean.
simply shows a significant performance improvement on baselines that are fine-tuned to the language model and works well for other languages as well as for English.
We propose CoMPM that leverages pre-trained memory using a pre-trained language model. CoMPM consists of a context embedding module (CoM) and a pre-trained memory module (PM), and the experimental results show that each module is effective in improving the performance. CoMPM outperforms baselines on both dyadic-party and multi-party datasets and achieves state-of-the-art among systems that do not use external knowledge. In addition, CoMPM achieves performance comparable to cutting-edge systems that leverage structured external knowledge, which is the effect of pre-trained memory of the language model.
By combining other pre-trained memories, we find that the pre-trained memory extracted with RoBERTa is richer and more effective than the pre-trained memory extracted with BERT or GPT2. Since we believe that pre-trained memory is proportional to the performance of a language model, a language model with a large training corpus and many parameters is considered to be more effective. However, we find that context modeling is more important than pre-trained memory for emotion recognition in conversation, and future research will focus on context modeling.
Additionally, our approach achieves competitive
performance and does not require externally structured data. Therefore, we show that it can be easily extended to Korean as well as English, and it is expected to be effective in other countries.
Cecilia Ovesdotter Alm, Dan Roth, and Richard Sproat. 2005. Emotions from text: Machine learning for text-based emotion prediction. In Proceedings of Human Language Technology Conference and Conference on Empirical Methods in Natural Language Processing, pages 579-586, Vancouver, British Columbia, Canada. Association for Computational Linguistics.
Antoine Bosselut, Hannah Rashkin, Maarten Sap, Chaitanya Malaviya, Asli Celikyilmaz, and Yejin Choi. 2019. COMET: Commonsense transformers for automatic knowledge graph construction. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4762-4779, Florence, Italy. Association for Computational Linguistics.
Chandrakant Bothe, Cornelius Weber, Sven Magg, and Stefan Wermter. 2020. EDA: Enriching emotional dialogue acts using an ensemble of neural annotators. In Proceedings of the 12th Language Resources and Evaluation Conference, pages 620-627, Marseille, France. European Language Resources Association.
Carlos Busso, Murtaza Bulut, Chi-Chun Lee, Abe Kazemzadeh, Emily Mower, Samuel Kim, Jeannette N. Chang, Sungbok Lee, and Shrikanth S. Narayanan. 2008. Iemocap: interactive emotional dyadic motion capture database. Lang. Resour. Evaluation, 42(4):335-359.
D Datcu and LJM Rothkrantz. 2014. Semantic audiovisual data fusion for automatic emotion recognition, pages 411-435. Blackwell, United Kingdom.
P. Ekman. 1992. An argument for basic emotions. Cognition & Emotion, 6:169-200.
Deepanway Ghosal, Navonil Majumder, Alexander Gelbukh, Rada Mihalcea, and Soujanya Poria. 2020. COSMIC: CCommonSense knowledge for eMotion identification in conversations. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 2470-2481, Online. Association for Computational Linguistics.
Devamanyu Hazarika, Soujanya Poria, Rada Mihalcea, Erik Cambria, and Roger Zimmermann. 2018a. ICON: Interactive conversational memory network for multimodal emotion detection. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2594-2604, Brussels, Belgium. Association for Computational Linguistics.
Devamanyu Hazarika, Soujanya Poria, Amir Zadeh, Erik Cambria, Louis-Philippe Morency, and Roger
Zimmermann. 2018b. Conversational memory network for emotion recognition in dyadic dialogue videos. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 2122-2132, New Orleans, Louisiana. Association for Computational Linguistics.
Dou Hu, Lingwei Wei, and Xiaoyong Huai. 2021. DialogueCRN: Contextual reasoning networks for emotion recognition in conversations. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume I: Long Papers), pages 7042-7052, Online. Association for Computational Linguistics.
Taichi Ishiwatari, Yuki Yasuda, Taro Miyazaki, and Jun Goto. 2020. Relation-aware graph attention networks with relational position encodings for emotion recognition in conversations. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7360-7370, Online. Association for Computational Linguistics.
Wenxiang Jiao, Michael R. Lyu, and Irwin King. 2020. Real-time emotion recognition via attention gated hierarchical memory network. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, pages 8002-8009. AAAI Press.
Jiangnan Li, Zheng Lin, Peng Fu, and Weiping Wang. 2021. Past, present, and future: Conversational emotion recognition through structural modeling of psychological knowledge. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 1204-1214, Punta Cana, Dominican Republic. Association for Computational Linguistics.
Jingye Li, Donghong Ji, Fei Li, Meishan Zhang, and Yijiang Liu. 2020. HiTrans: A transformer-based context- and speaker-sensitive model for emotion detection in conversations. In Proceedings of the 28th International Conference on Computational Linguistics, pages 4190–4200, Barcelona, Spain (Online). International Committee on Computational Linguistics.
Yanran Li, Hui Su, Xiaoyu Shen, Wenjie Li, Ziqiang Cao, and Shuzi Niu. 2017. DailyDialog: A manually labelled multi-turn dialogue dataset. In Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 986-995, Taipei, Taiwan. Asian Federation of Natural Language Processing.
Zhaojiang Lin, Peng Xu, Genta Indra Winata, Farhad Bin Siddique, Zihan Liu, Jamin Shin, and Pascale Fung. 2020. Caire: An end-to-end empathetic chatbot. Proceedings of the AAAI Conference on Artificial Intelligence, 34(09):13622-13623.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis,
Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692.
Zeming Liu, Haifeng Wang, Zheng-Yu Niu, Hua Wu, Wanxiang Che, and Ting Liu. 2020. Towards conversational recommendation over multi-type dialogs. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1036-1049, Online. Association for Computational Linguistics.
Wentao Ma, Yiming Cui, Nan Shao, Su He, Wei-Nan Zhang, Ting Liu, Shijin Wang, and Guoping Hu. 2019. TripleNet: Triple attention network for multi-turn response selection in retrieval-based chatbots. In Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL), pages 737-746, Hong Kong, China. Association for Computational Linguistics.
Navonil Majumder, Soujanya Poria, Devamanyu Hazarika, Rada Mihalcea, Alexander Gelbukh, and Erik Cambria. 2019. Dialoguernn: An attentive rnn for emotion detection in conversations. Proceedings of the AAAI Conference on Artificial Intelligence, 33(01):6818-6825.
Saif Mohammad. 2018. Obtaining reliable human ratings of valence, arousal, and dominance for 20,000 English words. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 174-184, Melbourne, Australia. Association for Computational Linguistics.
Fabio Petroni, Tim Rocttäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. 2019. Language models as knowledge bases? In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2463-2473, Hong Kong, China. Association for Computational Linguistics.
S. Poria, N. Majumder, R. Mihalcea, and E. Hovy. 2019. Emotion recognition in conversation: Research challenges, datasets, and recent advances. IEEE Access, 7:100943-100953.
Soujanya Poria, Devamanyu Hazarika, Navonil Majumder, Gautam Naik, Erik Cambria, and Rada Mihalcea. 2019. MELD: A multimodal multi-party dataset for emotion recognition in conversations. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 527-536, Florence, Italy. Association for Computational Linguistics.
Maarten Sap, Ronan Le Bras, Emily Allaway, Chandra Bhagavatula, Nicholas Lourie, Hannah Rashkin, Brendan Roof, Noah A. Smith, and Yejin Choi. 2019. Atomic: An atlas of machine commonsense for if-then reasoning. Proceedings of the AAAI Conference on Artificial Intelligence, 33(01):3027-3035.
Iulian Serban, Alessandro Sordoni, Ryan Lowe, Laurent Charlin, Joelle Pineau, Aaron Courville, and Yoshua Bengio. 2017. A hierarchical latent variable encoder-decoder model for generating dialogues. Proceedings of the AAAI Conference on Artificial Intelligence, 31(1).
Jamin Shin, Peng Xu, Andrea Madotto, and Pascale Fung. 2019. Happybot: Generating empathetic dialogue responses by improving user experience look-ahead. arXiv preprint arXiv:1906.08487.
Robyn Speer, Joshua Chin, and Catherine Havasi. 2017. Conceptnet 5.5: An open multilingual graph of general knowledge. In Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence, AAAI'17, page 4444-4451. AAAI Press.
Yang Sun, Nan Yu, and Guohong Fu. 2021. A discourse-aware graph neural network for emotion recognition in multi-party conversation. In *Findings of the Association for Computational Linguistics: EMNLP* 2021, pages 2949–2958, Punta Cana, Dominican Republic. Association for Computational Linguistics.
Amir Zadeh, Minghai Chen, Soujanya Poria, Erik Cambria, and Louis-Philippe Morency. 2017. Tensor fusion network for multimodal sentiment analysis. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1103-1114, Copenhagen, Denmark. Association for Computational Linguistics.
Sayyed M. Zahiri and Jinho D. Choi. 2018. Emotion detection on TV show transcripts with sequence-based convolutional neural networks. In The Workshops of the Thirty-Second AAAI Conference on Artificial Intelligence, New Orleans, Louisiana, USA, February 2-7, 2018, volume WS-18 of AAAI Workshops, pages 44-52. AAAI Press.
Rohola Zandie and Mohammad H. Mahoor. 2020. Emptransfo: A multi-head transformer architecture for creating empathetic dialog systems. In Proceedings of the Thirty-Third International Florida Artificial Intelligence Research Society Conference, Originally to be held in North Miami Beach, Florida, USA, May 17-20, 2020, pages 276-281. AAAI Press.
Dong Zhang, Liangqing Wu, Changlong Sun, Shoushan Li, Qiaoming Zhu, and Guodong Zhou. 2019. Modeling both context- and speaker-sensitive dependence for emotion detection in multi-speaker conversations. In Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, IJCAI-19, pages 5415-5421. International Joint Conferences on Artificial Intelligence Organization.
Yuxiang Zhang, Jiamei Fu, Dongyu She, Ying Zhang, Senzhang Wang, and Jufeng Yang. 2018a. Text emotion distribution learning via multi-task convolutional neural network. In Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence, IJCAI-18, pages 4595-4601. International Joint Conferences on Artificial Intelligence Organization.
Zhuosheng Zhang, Jiangtong Li, Pengfei Zhu, Hai Zhao, and Gongshen Liu. 2018b. Modeling multi-turn conversation with deep utterance aggregation. In Proceedings of the 27th International Conference on Computational Linguistics, pages 3740-3752, Santa Fe, New Mexico, USA. Association for Computational Linguistics.
Peixiang Zhong, Di Wang, and Chunyan Miao. 2019. Knowledge-enriched transformer for emotion detection in textual conversations. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 165-176, Hong Kong, China. Association for Computational Linguistics.
Hao Zhou, Tom Young, Minlie Huang, Haizhou Zhao, Jingfang Xu, and Xiaoyan Zhu. 2018. Commonsense knowledge aware conversation generation with graph attention. In Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence, IJCAI-18, pages 4623-4629. International Joint Conferences on Artificial Intelligence Organization.
Lixing Zhu, Gabriele Pergola, Lin Gui, Deyu Zhou, and Yulan He. 2021. Topic-driven and knowledge-aware transformer for dialogue emotion detection. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 1571-1582, Online. Association for Computational Linguistics.