SlowGuess/NAACL_2022_OCR huggingface dataset:
MEN performs better than all the previous baselines as measured using accuracy and F1-score. We also see an improvement in the class-wise F1 for happy, sad, neutral, and excited emotions. This improvement is possibly due to the GNN architecture (described in analysis later) that we are using in our model, and none of the previous multimodal baselines uses GNN in their architecture. Results for IEMOCAP (4-way) setting are in Table 3. In this setting, COGMEN achieves  $7.7\%$  improvement over the previous SOTA model.
MOSEI: For emotion classification across 6 emotion classes, we used two settings (as done in previous works): Binary Classification across each emotion label where a separate model is trained for every emotion class, and Multi-label Classification in which the sentence is tagged with more than 1 emotion and single model predicts multiple classes. The reason for doing this was that Multilogue-Net provides results on binary classification setting and TBJE provides results on Multi-label setting. We ran both models on these settings. For a fair comparison, we use the same utterance level textual features similar to our setting (extracted from sBERT) and train Multilogue-Net architecture on both the settings. Originally, Multilogue-Net used GloVe embeddings (Pennington et al., 2014) for textual features, and actual results in the paper are different than reported here. For TBJE, we use the features provided by the paper as it uses token-level features. COGMEN outperforms (Table 4) the baseline models in most of the cases. For 2 class sentiment classification, COGMEN outperforms the previous baselines with the highest accuracy score of  $85\%$  for  $A + T$ . For 7 class, our model shows comparable performance. All the multimodal approaches tend to perform poorly when
Sentiment Class Accuracy(%)	Emotion Class (weighted) F1-score (%)	Multi-label Emotion Class (weighted) F1-score (%)
Model	2 Class	7 Class	Happiness	Sadness	Angry	Fear	Disgust	Surprise	Happiness	Sadness	Angry	Fear	Disgust	Surprise
Multilogue-Net	T + A + V	82.88	44.83	67.84	65.34	67.03	87.79	74.91	86.05	70.6	70.7	74.4	86.0	83.4	87.8
TBJE	T	81.9	44.2	-	-	-	-	-	-	63.4	65.8	75.3	84.0	84.5	81.4
A + T	82.4	43.91	65.91	70.78	70.86	87.79	82.57	86.04	65.5	67.9	76.0	87.2	84.5	86.1
T + A + V	81.5	44.4	-	-	-	-	-	-	64.0	67.9	74.7	84.0	83.6	86.1
COGMEN	T	84.42	43.50	69.28	70.49	73.04	87.80	83.69	85.83	69.92	72.16	77.34	86.39	86.00	88.27
A + T	85.00	44.31	68.39	73.28	74.98	88.08	83.90	85.35	69.62	72.67	76.93	86.39	85.35	88.21
T + A + V	84.34	43.90	70.42	72.31	76.20	88.17	83.69	85.28	72.74	73.90	78.04	86.71	85.48	88.37
Table 4: Results on MOSEI dataset. For emotion classification, a weighted F1-score is used. For Sentiment Classification, the results are reported using accuracy. 2 class sentiment consists of only positive and negative sentiment. 7 class sentiment consists of sentiments from highly negative (-3) to highly positive (+3). For the cells showing '-', the results were not provided in the paper, and we were not able to reproduce the results since TBJE used token level features, and we are using sentence-level features.
# Utterances in Context	F1-score (%)
All Utterances in a dialogue	84.50
10 Utterances in a dialogue	77.43 (↓7.07)
3 Utterances in a dialogue	75.39 (↓9.11)
Table 5: Importance of Context in a dialogue. Experiment performed on IEMOCAP (4-way).
Modalities	T	A+T	A+T+V
(6 way)	Actual	66.00	65.42	67.63
w/o GNN	64.34 (↓1.66)	61.69 (↓3.73)	62.96 (↓4.14)
w/o Relations	60.49 (↓5.51)	65.32 (↓0.10)	62.13 (↓5.50)
(4 way)	Actual	81.55	81.59	84.50
w/o GNN	81.18 (↓0.37)	80.16 (↓1.43)	80.28 (↓4.22)
w/o Relations	76.76 (↓4.79)	80.27 (↓1.32)	79.61 (↓4.88)
Table 6: Ablation study on IEMOCAP dataset. All values are F1-score (\%). The results show the importance of GCN layer.
Model	Modality	F1-score (%)
4-way
DialogueGCN	T	71.58
DialogXL	T	73.02
DAG-ERC	T	78.08
COGMEN	T	81.55
A+T+V	84.50
6-way
EmoBERTa	T	68.57
DAG-ERC	T	68.03
CESTa	T	67.10
SumAggGIN	T	66.61
DialogueCRN	T	66.20
DialogXL	T	65.94
DialogueGCN	T	64.18
COGMEN	T	66.00
A+T+V	67.63
Table 7: Comparison with unimodal architectures on IEMOCAP dataset.
adding visual modality, possibly because of noise present in the visual modality and lack of alignment with respect to other modalities. In contrast, our model can capture rich relations across the modalities and show a performance boost while adding visual modality.
We conducted further analysis on our model. Although due to space limitations, the results below mainly describe experiments over IEMOCAP, similar trends were observed for MOSEI as well.
Effect of Local and Global Info.: We test our architecture in two information utilization settings: global and local. To test the importance of context in our architecture, we create a sub-dataset using the IEMOCAP (4-way) setting by splitting each dialogue into  $n$  utterances and training our architecture. Table 5 shows the decrease in performance
with number of utterances present in a dialogue (more details on effect of window size in Appendix G.2). This experiment helps understand the importance of context in a dialogue. Moreover, it points towards challenges in developing a real-time system (details in §6). We test the local information hypothesis by removing the GNN module and directly passing the context extracted features to the emotion classifier. Table 6 shows the drop in performance across modalities when the GNN component is removed from the architecture, making our local information hypothesis more concrete.
Effect of Relation Types: We also test the effect of inter and intra-relations in the dialogue graph by making all relations of the same type and training the architecture. We observe a drop in performance (Table 6) when the relations are kept the same in the
graph formation step. The explicit relation formation helps capture the local dependencies present in the dialogue.
Effect of Modalities: The focus of this work is multimodal emotion recognition. However, just for the purpose of comparison, we also compare with unimodal (text only) approaches. We compare (Table 7) with EmoBERTa (Kim and Vossen, 2021), DAG-ERC (Shen et al., 2021b), CESTa (Wang et al., 2020), SumAggGIN (Sheng et al., 2020), DialogueCRN (Hu et al., 2021), DialogXL (Shen et al., 2021a) and DialogueGCN (Ghosal et al., 2019). Text-based models are specifically optimized for text modalities and incorporate changes to architectures to cater to text. It is not fair to compare with our multimodal approach from that perspective. As shown in results, COGMEN, being a fairly generic architecture, still gives better (for IEMOCAP (4-way)) or comparable performance with respect to the SOTA unimodal architectures. In the case of our model, adding more information via other modalities helps to improve the performance. Results on different modality combinations are in Appendix D.
Figure 3: Confusion Matrix for IEMOCAP (6-way)
Error Analysis: After analysing the predictions made across the datasets, we find that our model falls short in distinguishing between similar emotions, such as happiness vs excited and anger vs frustration (Figure 3). This issue also exists in previous methods as reported in Shen et al. (2021b), and Ghosal et al. (2019). We also find that our model misclassifies the other emotion labels as neutral because of a more significant proportion of neutral labeled examples. Moreover, we observe the accuracy of our model when classifying examples having emotion shift is  $53.6\%$  compared to  $74.2\%$  when the emotion remains the same (more details in Appendix B).
Efficacy of the GNN Layer: For observing the
Figure 4: UMAP (Becht et al., 2019) representation of IEMOCAP (4-way) features before and after GNN.
effect of the GNN component in our architecture, we also visualize the features before and after the GNN component. Figure 4 clearly shows the better formation of emotion clusters depicting the importance of capturing local dependency in utterances for better performance in emotion recognition (more in Appendix E and Appendix Figure-9).
Importance of utterances: To verify the effect of utterances and their importance in a prediction for a dialogue, we infer the trained model on dialogues by masking one utterance at a time and calculating the F1-score for prediction. Figure 5 shows the obtained results for a dialogue (Appendix Table 10) instance taken randomly from IEMOCAP (4-way) (more in Appendix E). For the first 4 utterances, emotions state being neutral, the effect of masking the utterances is significantly less. In contrast, masking the utterances with emotion shift (9, 10, 11) completely drops the dialogue's F1-score, showing that our architecture captures the effects of emotions present in the utterances.
Figure 5: Importance of utterances in IEMOCAP (4-way). Performance drop is observed while masking  $9^{th}$ ,  $10^{th}$  and  $11^{th}$  utterances during inference.
Comparison with Baselines: Emotion recognition in a multimodal conversation setting comes with two broadly portrayed research challenges (Poria et al., 2019), first, the ability of a model to capture global and local context present in the dialogues, and second, the ability to maintain self and interpersonal dependencies among the speakers. All the popular baselines like Dialogue-GCN (Ghosal et al., 2019), DialogueRNN (Majumder et al., 2019), bc-LSTM (Poria et al., 2017) AfCAN (Wang et al., 2021a), etc., try to address these challenges by proposing various architectures. bc-LSTM (bi-directional contextual LSTM (Poria et al., 2017)) uses LSTM to capture the contextual information and maintain long relations between the utterances from the past and future. Another contemporary architecture Af-CAN (Wang et al., 2021a) utilizes recurrent neural networks based on contextual attention to model the interaction and dependence between speakers and uses bi-directional GRU units to capture the global features from past and future. We propose to address these issues using a unified architecture that captures the effect of context on utterances while maintaining the states for self and interpersonal dependencies. We make use of transformers for encoding the global context and make use of Graph Transformers to capture the self and interpersonal dependencies. Our way of forming relational graphs between the utterances comes close to DialogueGCN (unimodal architecture). We further use a shared Emotion classifier for predicting emotions from all the obtained utterance level features. Moreover, our unified architecture handles multiple modalities effectively and shows an increase in performance after adding information from other modalities.
Limitations (Offline Setting): A noteworthy limitation of all the proposed Emotion Recognition approaches (including the current one) is that they use global context from past and future utterances to predict emotions. However, baseline systems compared in this paper are also offline systems. For example, bc-LSTM (bi-directional contextual LSTM) and Af-CAN use utterances from the past and future to predict emotions. Other popular baselines like DialogueGCN and DialogueRNN (BiDialogueRNN) also peek into the future, assuming the presence of all the utterances during inference (offline setting). All such systems that depend on future information can only be used in an offline
setting to process and tag the dialogue. An Emotion Recognition system that could work in an online setting exhibits another line of future work worth exploring due to its vast use cases in live telecasting and telecommunication. A possible approach to maintain the context in an online setting would be to take a buffer of smaller context size, where the model can predict emotions taking not the complete dialogue but a smaller subset of it as input in real-time. We tried exploring this setting for our architecture with an online buffer of maintaining a smaller context window. For experimenting with it, we created a sub-dataset using the IEMOCAP (4-way) setting by splitting each dialogue into  $n$  utterances and training our architecture. Our results in Table 5 show the decrease in performance with the number of utterances present in a dialogue depicting the importance of context in a conversation. Performance improvements in these settings where the system can work in real-time are worth exploring and are an interesting direction for future research.
We present a novel approach of using GNNs for multimodal emotion recognition and propose COGMEN: COntextualized GNN based Multimodal Emotion recognitioN. We test COGMEN on two widely known multimodal emotion recognition datasets, IEMOCAP and MOSEI. COGMEN outperforms the existing state-of-the-art methods in multimodal emotion recognition by a significant margin (i.e.,  $7.7\%$  F1-score increase for IEMOCAP (4-way)). By comprehensive analysis and ablation studies over COGMEN, we show the importance of different modules. COGMEN fuses information effectively from multiple modalities to improve the performance of emotion prediction tasks. We perform a detailed error analysis and observe that the misclassifications are mainly between the similar classes and emotion shift cases. We plan to address this in future work, where the focus will be to incorporate a component for capturing the emotional shifts for fine-grained emotion prediction.
We would like to thank reviewers for their insightful comments. This research is supported by SERB India (Science and Engineering Board) Research Grant number SRG/2021/000768.
Harsh Agarwal, Keshav Bansal, Abhinav Joshi, and Ashutosh Modi. 2021. Shapes of emotions: Multimodal emotion recognition in conversations via emotion shifts. CoRR, abs/2112.01938.
AmirAli Bagher Zadeh, Paul Pu Liang, Soujanya Poria, Erik Cambria, and Louis-Philippe Morency. 2018. Multimodal language analysis in the wild: CMU-MOSEI dataset and interpretable dynamic fusion graph. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2236-2246, Melbourne, Australia. Association for Computational Linguistics.
Tadas Baltrusaitis, Amir Zadeh, Yao Chong Lim, and Louis-Philippe Morency. 2018. Openface 2.0: Facial behavior analysis toolkit. In 2018 13th IEEE International Conference on Automatic Face Gesture Recognition (FG 2018), pages 59-66.
Etienne Becht, Leland McInnes, John Healy, Charles-Antoine Dutertre, Immanuel WH Kwok, Lai Guan Ng, Florent Ginghoux, and Evan W Newell. 2019. Dimensionality reduction for visualizing single-cell data using umap. Nature biotechnology, 37(1):38-44.
Carlos Busso, Murtaza Bulut, Chi-Chun Lee, Abe Kazemzadeh, Emily Mower, Samuel Kim, Jeannette N Chang, Sungbok Lee, and Shrikanth S Narayanan. 2008. Iemocap: Interactive emotional dyadic motion capture database. Language resources and evaluation, 42(4):335-359.
Pierre Colombo, Wojciech Witon, Ashutosh Modi, James Kennedy, and Mubbasir Kapadia. 2019. Affect-driven dialog generation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 3734-3743, Minneapolis, Minnesota. Association for Computational Linguistics.
Comet.ML. 2021. Comet.ML home page.
Dragos Datcu and Leon JM Rothkrantz. 2014. Semantic audio-visual data fusion for automatic emotion recognition. Emotion recognition: a pattern analysis approach, pages 411-435.
Jean-Benoit Delbrouck, Noé Tits, Mathilde Brousmiche, and Stéphane Dupont. 2020. A transformer-based joint-encoding for emotion recognition and sentiment analysis. In Second Grand-Challenge and Workshop on Multimodal Language (Challenge-HML), pages 1-7, Seattle, USA. Association for Computational Linguistics.
Marwan Dhuheir, Abdullahatif Albaseer, Emna Baccour, Aiman Erbad, Mohamed Abdallah, and Mounir Hamdi. 2021. Emotion recognition for healthcare surveillance systems using neural networks: A survey.
Paul Ekman. 1993. Facial expression and emotion. American psychologist, 48(4):384.
Florian Eyben, Martin Wöllmer, and Björn Schuller. 2010. Opensmile: The munich versatile and fast open-source audio feature extractor. In Proceedings of the 18th ACM International Conference on Multimedia, MM '10, page 1459-1462, New York, NY, USA. Association for Computing Machinery.
Matthias Fey and Jan Eric Lenssen. 2019. Fast graph representation learning with pytorch geometric. CoRR, abs/1903.02428.
Marc Franzen, Michael Stephan Gresser, Tobias Müller, and Prof. Dr. Sebastian Mauser. 2021. Developing emotion recognition for video conference software to support people with autism.
Yahui Fu, Shogo Okada, Longbiao Wang, Lili Guo, Yaodong Song, Jiaxing Liu, and Jianwu Dang. 2021. Consk-gcn: Conversational semantic- and knowledge-oriented graph convolutional network for multimodal emotion recognition. In 2021 IEEE International Conference on Multimedia and Expo (ICME), pages 1-6.
Deepanway Ghosal, Md Shad Akhtar, Dushyant Chauhan, Soujanya Poria, Asif Ekbal, and Pushpak Bhattacharyya. 2018. Contextual inter-modal attention for multi-modal sentiment analysis. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3454-3466, Brussels, Belgium. Association for Computational Linguistics.
Deepanway Ghosal, Navonil Majumder, Soujanya Poria, Niyati Chhaya, and Alexander Gelbukh. 2019. DialogueGCN: A graph convolutional neural network for emotion recognition in conversation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 154-164, Hong Kong, China. Association for Computational Linguistics.
Tushar Goswamy, Ishika Singh, Ahsan Barkati, and Ashutosh Modi. 2020. Adapting a language model for controlled affective text generation. In Proceedings of the 28th International Conference on Computational Linguistics, pages 2787-2801, Barcelona, Spain (Online). International Committee on Computational Linguistics.
M. Hasan, Sangwu Lee, Wasifur Rahman, Amir Zadeh, Rada Mihalcea, Louis-Philippe Morency, and Ehsan Hoque. 2021. Humor knowledge enriched transformer for understanding multimodal humor. In AAAI.
Devamanyu Hazarika, Soujanya Poria, Rada Mihalcea, Erik Cambria, and Roger Zimmermann. 2018a. ICON: Interactive conversational memory network for multimodal emotion detection. In Proceedings
of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2594-2604, Brussels, Belgium. Association for Computational Linguistics.
Devamanyu Hazarika, Soujanya Poria, Amir Zadeh, Erik Cambria, Louis-Philippe Morency, and Roger Zimmermann. 2018b. Conversational memory network for emotion recognition in dyadic dialogue videos. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 2122-2132, New Orleans, Louisiana. Association for Computational Linguistics.
Dou Hu, Lingwei Wei, and Xiaoyong Huai. 2021. Dialoguecrn: Contextual reasoning networks for emotion recognition in conversations. In ACL/IJCNLP (1), pages 7042-7052. Association for Computational Linguistics.
Taichi Ishiwatari, Yuki Yasuda, Taro Miyazaki, and Jun Goto. 2020. Relation-aware graph attention networks with relational position encodings for emotion recognition in conversations. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7360-7370.
Sepehr Janghorbani, Ashutosh Modi, Jakob Buhmann, and Mubbasir Kapadia. 2019. Domain authoring assistant for intelligent virtual agent. AAMAS '19, page 104-112, Richland, SC. International Foundation for Autonomous Agents and Multiagent Systems.
Taewoon Kim and Piek Vossen. 2021. EmoBERTa: Speaker-Aware Emotion Recognition in Conversation with RoBERTa. arXiv e-prints, page arXiv:2108.12009.
Agata Kołakowska, Agnieszka Landowska, Mariusz Szwoch, Wioleta Szwoch, and Michal R Wrobel. 2014. Emotion recognition and its applications. In Human-Computer Systems Interaction: Backgrounds and Applications 3, pages 51-62. Springer.
Ayush Kumar and Jithendra Vepa. 2020. Gated mechanism for attention based multi modal sentiment analysis. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 4477-4481. IEEE.
John D. Lafferty, Andrew McCallum, and Fernando C. N. Pereira. 2001. Conditional random fields: Probabilistic models for segmenting and labeling sequence data. In Proceedings of the Eighteenth International Conference on Machine Learning, ICML '01, page 282-289, San Francisco, CA, USA. Morgan Kaufmann Publishers Inc.
Zheng Lian, Jianhua Tao, Bin Liu, Jian Huang, Zhanlei Yang, and Rongjun Li. 2020. Conversational emotion recognition using self-attention mechanisms
and graph neural networks. In INTERSPEECH, pages 2347-2351.
Navonil Majumder, Devamanyu Hazarika, Alexander Gelbukh, Erik Cambria, and Soujanya Poria. 2018. Multimodal sentiment analysis using hierarchical fusion with context modeling. Knowledge-based systems, 161:124-133.
Navonil Majumder, Soujanya Poria, Devamanyu Hazarika, Rada Mihalcea, Alexander Gelbukh, and Erik Cambria. 2019. Dialoguernn: An attentive rnN for emotion detection in conversations. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 6818-6825.
Brian McFee, Colin Raffel, Dawen Liang, Daniel P. W. Ellis, Matt McVicar, Eric Battenberg, and Oriol Ni- eto. 2015. librosa: Audio and music signal analysis in python.
Marvin Minsky. 2007. The Emotion Machine: Common Sense Thinking, Artificial Intelligence, and the Future of the Human Mind. SIMON & SCHUSTER.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. 2019. Pytorch: An imperative style, high-performance deep learning library. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems 32, pages 8024-8035. Curran Associates, Inc.
Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. GloVe: Global vectors for word representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1532-1543, Doha, Qatar. Association for Computational Linguistics.
Sally Planalp, Julie Fitness, and Beverley A. Fehr. 2018. The Roles of Emotion in Relationships, 2 edition, Cambridge Handbooks in Psychology, page 256-268. Cambridge University Press.
Soujanya Poria, Erik Cambria, Devamanyu Hazarika, Navonil Majumder, Amir Zadeh, and Louis-Philippe Morency. 2017. Context-dependent sentiment analysis in user-generated videos. In Proceedings of the 55th annual meeting of the association for computational linguistics (volume 1: Long papers), pages 873-883.
Soujanya Poria, Navonil Majumder, Rada Mihalcea, and Eduard Hovy. 2019. Emotion recognition in conversation: Research challenges, datasets, and recent advances. IEEE Access, 7:100943-100953.