SlowGuess/NAACL_2022_OCR huggingface dataset:
Nils Reimers and Iryna Gurevych. 2019. SentenceBERT: Sentence embeddings using Siamese BERTnetworks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3982-3992, Hong Kong, China. Association for Computational Linguistics.
Michael Schlichtkrull, Thomas N. Kipf, Peter Bloem, Rianne van den Berg, Ivan Titov, and Max Welling. 2018. Modeling relational data with graph convolutional networks. In *The Semantic Web*, pages 593–607, Cham. Springer International Publishing.
Nicu Sebe, Ira Cohen, Theo Gevers, and Thomas S Huang. 2005. Multimodal approaches for emotion recognition: A survey. Proceedings of SPIE - The International Society for Optical Engineering, 5670:56-67. Proceedings of SPIE-IS and T Electronic Imaging - Internet Imaging VI; Conference date: 18-01-2005 Through 20-01-2005.
Garima Sharma and Abhinav Dhall. 2021. A Survey on Automatic Multimodal Emotion Recognition in the Wild, pages 35-64.
Weizhou Shen, Junqing Chen, Xiaojun Quan, and Zhixian Xie. 2021a. Dialogxl: All-in-one xlnet for multi-party conversation emotion recognition. Proceedings of the AAAI Conference on Artificial Intelligence, 35(15):13789-13797.
Weizhou Shen, Siyue Wu, Yunyi Yang, and Xiaojun Quan. 2021b. Directed acyclic graph network for conversational emotion recognition. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 1551-1560, Online. Association for Computational Linguistics.
Dongming Sheng, Dong Wang, Ying Shen, Haitao Zheng, and Haozhuang Liu. 2020. Summarize before aggregate: A global-to-local heterogeneous graph inference network for conversational emotion recognition. In Proceedings of the 28th International Conference on Computational Linguistics, pages 4153-4163.
Aman Shenoy and Ashish Sardana. 2020. Multiloguenet: A context-aware RNN for multi-modal emotion detection and sentiment analysis in conversation. In Second Grand-Challenge and Workshop on Multimodal Language (Challenge-HML), pages 19-28, Seattle, USA. Association for Computational Linguistics.
Yunsheng Shi, Zhengjie Huang, Wenjin Wang, Hui Zhong, Shikun Feng, and Yu Sun. 2021. Masked label prediction: Unified massage passing model for semi-supervised classification. In *IJCAI*.
Aaditya Singh, Shreeshail Hingane, Saim Wani, and Ashutosh Modi. 2021a. An end-to-end network for
emotion-cause pair extraction. In Proceedings of the Eleventh Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis, pages 84–91, Online. Association for Computational Linguistics.
Gargi Singh, Dhanajit Brahma, Piyush Rai, and Ashutosh Modi. 2021b. Fine-Grained Emotion Prediction by Modeling Emotion Definitions. In 2021 9th International Conference on Affective Computing and Intelligent Interaction (ACII), pages 1-8.
Sainbayar Sukhbaatar, Arthur Szlam, Jason Weston, and Rob Fergus. 2015. End-to-end memory networks. In Proceedings of the 28th International Conference on Neural Information Processing Systems - Volume 2, NIPS'15, page 2440-2448, Cambridge, MA, USA. MIT Press.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc.
C Vinola and K Vimaladevi. 2015. A survey on human emotion recognition approaches, databases and applications. ELCVIA Electronic Letters on Computer Vision and Image Analysis, 14(2):24-44.
Tana Wang, Yaqing Hou, Dongsheng Zhou, and Qiang Zhang. 2021a. A contextual attention network for multimodal emotion recognition in conversation. In 2021 International Joint Conference on Neural Networks (IJCNN), pages 1-7.
Tana Wang, Yaqing Hou, Dongsheng Zhou, and Qiang Zhang. 2021b. A contextual attention network for multimodal emotion recognition in conversation. In 2021 International Joint Conference on Neural Networks (IJCNN), pages 1-7. IEEE.
Yan Wang, Jiayu Zhang, Jun Ma, Shaojun Wang, and Jing Xiao. 2020. Contextualized emotion recognition in conversation as sequence tagging. In Proceedings of the 21th Annual Meeting of the Special Interest Group on Discourse and Dialogue, pages 186-195.
Martin Wollmer, Angeliki Metallinou, Florian Eyben, Bjorn Schuller, and Shrikanth Narayanan. 2010. Context-sensitive multimodal emotion recognition from speech and facial expression using bidirectional LSTM modeling. In Proc. INTERSPEECH 2010, Makuhari, Japan, pages 2362-2365.
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R. Salakhutdinov, and Quoc V. Le. 2019. Xlnet: Generalized autoregressive pretraining for language understanding. In Advances in Neural Information Processing Systems, volume 32.
Zhou Yu, Jun Yu, Yuhao Cui, Dacheng Tao, and Qi Tian. 2019. Deep modular co-attention networks for visual question answering. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6281-6290.
Amir Zadeh, Minghai Chen, Soujanya Poria, Erik Cambria, and Louis-Philippe Morency. 2017. Tensor fusion network for multimodal sentiment analysis. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1103–1114, Copenhagen, Denmark. Association for Computational Linguistics.
Amir Zadeh, Paul Pu Liang, Navonil Mazumder, Soujanya Poria, Erik Cambria, and Louis-Philippe Morency. 2018a. Memory fusion network for multiview sequential learning. In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence and Thirtieth Innovative Applications of Artificial Intelligence Conference and Eighth AAAI Symposium on Educational Advances in Artificial Intelligence, AAAI'18/IAAI'18/EAAI'18. AAAI Press.
AmirAli Bagher Zadeh, Paul Pu Liang, Soujanya Poria, Erik Cambria, and Louis-Philippe Morency. 2018b. Multimodal language analysis in the wild: Cmu-mosei dataset and interpretable dynamic fusion graph. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2236-2246.
Dong Zhang, Liangqing Wu, Changlong Sun, Shoushan Li, Qiaoming Zhu, and Guodong Zhou. 2019. Modeling both context- and speaker-sensitive dependence for emotion detection in multi-speaker conversations. In Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, IJCAI 2019, Macao, China, August 10-16, 2019, pages 5415-5421. ijcai.org.
A Hyperparameter Setting
Hyperparameters used to train our model are described in Table 8 for IEMOCAP (4-way and 6-way) and Table 9 for MOSEI dataset.
Dropout	GNNHead	SeqContext	ILR
0.1	7	4	1e-4
Table 8: Hyperparameter values for our model on IEMOCAP dataset. ILR: Initial learning rate.
Modalities	Dropout	GNNHead	SeqContext	ILR
T	0.399	3	5	3.3e-3
A+T	0.103	1	2	6.9e-3
A+T+V	0.337	2	1	1.1e-3
Table 9: Hyperparameter value on MOSEI dataset. ILR: Initial learning rate.
We use PyTorch (Paszke et al., 2019) for training our architecture and PyG (PyTorch Geometric) (Fey and Lenssen, 2019) for the GNN component in our architecture. We use comet (Comet.ML, 2021) for logging all our experiments and its Bayesian optimizer for hyperparameter tuning. Our architecture trained on the IEMOCAP dataset has 55,932,052 parameters and takes around 7 minutes to train for 50 epochs on the NVIDIA Tesla K80 GPU. Comparison of the model with baselines in terms of the number of parameters is challenging, as the baselines parameters vary depending on the hyperparameter setting. Moreover, many baselines do not provide information about the number of parameters.
B Dataset Analysis
We study IEMOCAP dataset in detail for error analysis of our model. We observe the emotion transition at Utterance level (Figure 6) and Speaker level (Figure 7). We find a high percentage of transitions between similar emotions, causing the models to confuse between the similar classes of emotion. Considering the emotion transition between states that are opposite, like from happy to sad, we deduce the poor performance of emotion recognition architectures for such cases. We plan to address this issue in future work where we target a model which performs better in fine-grained emotion recognition and is robust towards the shifts in emotions.
Figure 6: Utterance-level Emotion transition for IEMOCAP. These are emotions transitions in consecutive utterances across speakers.
Figure 7: Speaker-level Emotion transition for IEMOCAP. These are emotions transitions in the consecutive utterances of the same speaker.
Weighted F1 Score: The F1 score can be interpreted as a harmonic mean of the precision and recall, where an F1 score reaches its best value at 1 and worst score at 0. The relative contribution of precision and recall to the F1 score are equal. The formula for the F1 score is:
$$
F 1 = 2 * \frac {\left(p r e c i s i o n * r e c a l l\right)}{\left(p r e c i s i o n + r e c a l l\right)}
$$
For weighted F1 score, we calculate metrics for each label, and find their average weighted by support (the number of true instances for each label).
Accuracy: It is defined as the percentage of correct predictions in the test set.
D Results on Modality Combinations
Table 11 shows results on the IEMOCAP dataset for all the modality combinations for our architectures. Figure 8 shows the confusion matrix for prediction on IEMOCAP 4-way dataset.
Speaker	Utterance Text	Emotion
M	'Why does that bother you?'	neutral
F	"She's been in New York three and a half years. Why all of the sudden?"	neutral
M	'Well maybe. Maybe she just wanted to see her again.'	neutral
F	"What did you mean? He lived next door to the girl all of his life, why wouldn't he want to see her again? Don't look at me like that, he didn't tell me any more than he told you."	neutral
M	"She's not his girl. She knows she's not."	angry
F	"I want you to pretend like he's coming back!"	angry
M	"Because if he's not coming back, then I'll kill myself."	angry
F	'Laugh. Laugh at me, but what happens the night that she goes to sleep in his bed, and his memorial breaks in pieces?"	angry
M	'Only last week, another boy turned up in Detroit, been missing longer than Larry, you read it yourself,'	angry
F	"You've got to believe. You've got to-"	sad
M	"What do you mean me above all? Look at you. You're shaking!"	angry
F	"I can't help it!"	angry
M	'What have I got to hide? What the hell is the matter with you, Kate?'	angry
Table 10: Dialogue utterances corresponding to plot shown in Figure 5.
Modalities	IEMOCAP-4way
F1 Score (%)	IEMOCAP-6way
F1 Score (%)
a	63.58	47.57
t	81.55	66.00
v	43.85	37.58
at	81.59	65.42
av	64.48	52.20
tv	81.52	62.19
atv	84.50	67.63
Table 11: Results on IEMOCAP-4way and IEMOCAP-6way datasets
Figure 8: Confusion Matrix for IEMOCAP 4-Way classification
Efficacy of the GNN Layer: We observe the efficacy of the GNN component in our architecture and visualize the features before GNN and after the GNN component (Figure 9) explained in section 5. Importance of utterances: Figure 10 shows the obtained results for a dialogue instance taken randomly from IEMOCAP 4-way. For the first 15
Figure 9: UMAP (Becht et al., 2019) representation of IEMOCAP 6-way features before and after GNN.
Figure 10: Importance of utterances in IEMOCAP classification.
utterances, emotions state being sadness, the effect of masking the utterances is more negligible for the first 5 utterances. This drop depicts the importance of utterances 5-15 that affect future utterances. Further, masking the utterances with high
Figure 11: Graph formation process in (COGMEN) architecture.
Figure	Central Node	R_intra	R_inter
(a)	ui-3(S1)	ui-3(S1)← ui-3(S1), ui-3(S1)→ ui-1, ui-3(S1)→ ui-1, ui-3(S1), ui-3(S1)→ ui-1	ui-3(S1)→ ui-2, ui-3(S1)→ ui(S2), ui-3(S1)→ ui(S2)
(b)	ui-2(S2)	ui-2(S2)← ui-2(S2), ui-2(S2)→ ui(S2), ui-2(S2)→ ui-2	ui-2(S2)← ui-3(S1), ui-2(S2)→ ui-1, ui-2(S2)→ ui(S1), ui-2(S2)→ ui-1
(c)	ui-1(S1)	ui-1(S1)← ui-1(S1), ui-1(S1)← ui-1(S1), ui-1(S1)→ ui-1(S1), ui-1(S1)→ ui-1	ui-1(S1)← ui-2(S2), ui-1(S1)→ ui(S2), ui-1(S1)→ ui-2
(d)	ui(S2)	ui(S2)← ui-2(S2), ui(S2)← ui(S2), ui(S2)→ ui-2	ui(S2)← ui-3(S1), ui(S2)← ui-1(S1), ui(S2)→ ui-1(S1), ui(S2)→ ui-1
(e)	ui+1(S1)	ui+1(S1)← ui-1(S1), ui+1(S1)← ui-1(S1), ui+1(S1)← ui-1(S1), ui+1(S1)→ ui+1	ui+1(S1)← ui-2(S2), ui+1(S1)← ui(S2), ui+1(S1)→ ui+2
(f)	ui+2(S2)	ui+2(S2)← ui-2(S2), ui+2(S2)← ui(S2), ui+2(S2)← ui+2	ui+2(S2)← ui-1(S1), ui+2(S2)← ui(S1), ui+2(S2)→ ui+1
(g)	ui+3(S1)	ui+3(S1)← ui-3(S1), ui+3(S1)← ui-1, ui+3(S1)← ui-1,S1,S1,S1,S1,S1,S1,S1,S1,S1,S1,S1,S1,S1,S1,S1,S1,S1,S1,S1,S1,S1,S1,S1,S1,S1,S1,S1,S1,S1,S1,S1,S1,S1,S1,S1,S1,S1,S1,S1,S1,S1,S1,S1,S1,S1,S1,S1,S1,S1,S1S1,S1,S1,S1,S1,S1,S1,S1,S1,S1,S1,S1,S1,S1,S1,S1,S1,S1,S1,S1,S1,S1,S1,S1,S1,S1,S1,S1,S1,S1,S1,S1,S1,S1,S1,S1,S1,S1,S1,S1,S1,S1,S1,S1,S1,S1,S1,S1,S1,S1、	ui+3(S1)← ui-2(S2), ui+3(S1)← ui(S2), ui+3(S1)→ ui+2
Table 12: Relations for each instance of Figure 11, where relations with past utterances are denoted by  $(\leftarrow)$  and relations with future utterances are denoted by  $(\rightarrow)$
Relation Type	Node A	Node B	Relation Causality	Relation
1	u(S1)	u(S1)	Past	u(S1)← u(S1)
2	u(S1)	u(S2)	Past	u(S1)← u(S2)
3	u(S2)	u(S1)	Past	u(S2)← u(S1)
4	u(S2)	u(S2)	Past	u(S2)← u(S2)
5	u(S1)	u(S1)	Future	u(S1)→ u(S1)
6	u(S1)	u(S2)	Future	u(S1)→ u(S2)
7	u(S2)	u(S1)	Future	u(S2)→ u(S1)
8	u(S2)	u(S2)	Future	u(S2)→ u(S2)
Table 13: Unique Relation types for a conversation between two speakers
emotion shift (15 to 30) drops the F1 score of the dialogue, showing the importance of fluctuations for predicting the emotion states for other utterances.
F Graph Formation
To give a clear picture of the graph formation procedure, we describe the process for utterances spoken in a dialogue. As an illustration, let's consider two speakers,  $S_{1}$  and  $S_{2}$ , present in a conversation of 7 utterances. Features corresponding to each utterance is shown as a node in Figure 11. Speaker 1 speaks utterances  $u_{i-3}, u_{i-1}, u_{i+1}, u_{i+3}$  and Speaker 2 speaks  $u_{i-2}, u_{i}, u_{i+2}$ . After creating the graphs with relations, the constructed graph would look like shown in Figure 11, and the corresponding relations for each instance would be as shown in Table 12. Since there are two speakers in the conversation ( $S_{N} = 2$ ), the total number of
unique relations would be:
$$
\begin{array}{l} \text {n u m b e r o f r e l a t i o n s} = 2 \times \left(S _ {N}\right) ^ {2} \\ = 2 \times (2) ^ {2} \\ = 8 \\ \end{array}
$$
Table 13 shows the number of possible unique relations for a conversation between two speakers.
G.1 Modality Fusing Mechanisms
While experimenting with the model architecture, we explored various mechanisms for mixing information from multiple modalities. Some of the mechanisms include pairwise attention inspired from Ghosal et al. (2018), bimodal attention present in Multilogue-Net (Shenoy and Sardana, 2020), and crossAttention layer proposed in HKT (Hasan et al., 2021). However, in our case, none of these fusing mechanisms shows significant performance improvement over simple concatenation. Moreover, all these fusing mechanisms require extra computation steps for fusing information. In contrast, a simple concatenation of modality features works well with no additional computational overhead.
G.2 Effect of window size in Graph Formation
To explore the effect of window size in the Graph Formation module of our architecture, we conduct experiments with multiple window sizes. The obtained results are present in Table 14. The window size can be treated as a hyperparameter that could be adjusted while training our architecture. Moreover, the freedom of setting the window size makes our architecture more flexible in terms of usage. A larger window size would result in better performance for cases where the inter and intra speaker dependencies are maintained for longer sequences. In contrast, setting a lower window size would be better in a use case where the topic frequently changes in dialogues and speakers are less affected by another speaker. In the future, we plan to explore a dynamic and automatic selection of window size depending on the dialogue instance.
Modalities	Window Past	Window future	F1 Score (%)
atv	1	1	81.72
atv	2	2	83.21
atv	4	4	84.08
atv	5	5	83.19
atv	6	6	82.49
atv	7	7	82.28
atv	9	9	82.77
atv	10	10	84.50
atv	11	11	83.93
atv	15	15	83.78
Table 14: Results for various window sizes for graph formation on the IEMOCAP (4-way) dataset.