SlowGuess/NAACL_2022_OCR huggingface dataset:
COGMEN: COntextualized GNN based Multimodal Emotion recognition
Abhinav Joshi Ashwani Bhat
Ayush Jain Atin Vikram Singh Ashutosh Modi
Indian Institute of Technology Kanpur (IIT-K)
{ajoshi, ashutoshm}@cse.iitk.ac.in
{ashubhat44}@gmail.com
{aayushj, atinvs}@iitk.ac.in
Emotions are an inherent part of human interactions, and consequently, it is imperative to develop AI systems that understand and recognize human emotions. During a conversation involving various people, a person's emotions are influenced by the other speaker's utterances and their own emotional state over the utterances. In this paper, we propose CoContextualized Graph Neural Network based Multimodal Emotion recognitioN (COGMEN) system that leverages local information (i.e., inter/intra dependency between speakers) and global information (context). The proposed model uses Graph Neural Network (GNN) based architecture to model the complex dependencies (local and global information) in a conversation. Our model gives state-of-the-art (SOTA) results on IEMOCAP and MOSEI datasets, and detailed ablation experiments show the importance of modeling information at both levels.
Emotions are intrinsic to humans and guide their behavior and are indicative of the underlying thought process (Minsky, 2007). Consequently, understanding and recognizing emotions is vital for developing AI technologies (e.g., personal digital assistants) that interact directly with humans. During a conversation between a number of people, there is a constant ebb and flow of emotions experienced and expressed by each person. The task of multimodal emotion recognition addresses the problem of monitoring the emotions expressed (via various modalities, e.g., video (face), audio (speech)) by individuals in different settings such as conversations.
Emotions are physiological, behavioral, and communicative reactions to cognitively processed stimuli (Planalp et al., 2018). Emotions are often a result of internal physiological changes, and
Figure 1: An example conversation between two speakers, with corresponding emotions evoked for each utterance.
these physiological reactions may not be noticeable by others and are therefore intra-personal. For example, in a conversational setting, an emotion may be a communicative reaction that has its origin in a sentence spoken by another person, acting as a stimulus. The emotional states expressed in utterances correlate with the context directly; for example, if the underlying context is about a happy topic like celebrating a festival or description of a vacation, there will be more positive emotions like joy and surprise. Consider the example shown in Figure 1, where the context depicts an exciting conversation. Speaker-1 being excited about his admission affects the flow of emotions in the entire context. The emotion states of Speaker-2 show the dependency on Speaker-1 in  $\mathcal{U}_2$ ,  $\mathcal{U}_4$  and  $\mathcal{U}_6$ , and maintains intra-personal state depicted in  $\mathcal{U}_8$  and  $\mathcal{U}_{10}$  by being curious about the responses of Speaker-1. The example conversation portrays the effect of global information as well as inter and intra dependency of speakers on the emotional states of the utterances. Moreover, emotions are a multimodal phenomenon; a person takes cues from different modalities (e.g., audio, video) to infer the emotions of others, since, very often, the in
Human Language Technologies, pages 4148 - 4164
formation in different modalities complement each other. In this paper, we leverage these intuitions and propose COGMEN: COntextualized Graph neural network based Multimodal Emotion recognition architecture that addresses both, the effects of context on the utterances and inter and intra dependency for predicting the per-utterance emotion of each speaker during the conversation. There has been a lot of work on unimodal (using text only) prediction, but our focus is on multimodal emotion prediction. As is done in literature on multimodal emotion prediction, we do not focus on comparison with unimodal models. As shown via experiments and ablation studies, our model leverages both the sources (i.e., local and global) of information to give state-of-the-art (SOTA) results on the multimodal emotion recognition datasets IEMOCAP and MOSEI. In a nutshell, we make the following contributions in this paper:
- We propose a Contextualized Graph Neural Network (GNN) based Multimodal Emotion Recognition architecture for predicting per utterance per speaker emotion in a conversation. Our model leverages both local and global information in a conversation. We use GraphTransformers (Shi et al., 2021) for modeling speaker relations in multimodal emotion recognition systems.
- Our model gives SOTA results on the multimodal Emotion recognition datasets of IEMOCAP and MOSEI.
- We perform a thorough analysis of the model and its different components to show the importance of local and global information along with the importance of the GNN component. We release the code for models and experiments: https://github.com/Exploration-Lab/COGMEN
Emotion recognition is an actively researched problem in NLP (Sharma and Dhall, 2021; Sebe et al., 2005). The broad applications ranging from emotion understanding systems, opinion mining from a corpus to emotion generation have attracted active research interest in recent years (Dhuheir et al., 2021; Franzen et al., 2021; Vinola and Vimaladevi, 2015; Kołakowska et al., 2014; Colombo et al., 2019; Janghorbani et al., 2019; Goswamy et al., 2020; Singh et al., 2021a; Agarwal et al., 2021; Singh et al., 2021b). Availability of bench
mark multimodal datasets, such as CMU-MOSEI (Zadeh et al., 2018b), and IEMOCAP (Busso et al., 2008), have accelerated the progress in the area. Broadly speaking, most of the existing work in this area can be categorized mainly into two areas: unimodal approaches and multimodal approaches. Unimodal approaches tend to consider the text as a prominent mode of communication and solve the emotion recognition task using only text modality. In contrast, multimodal approaches are more naturalistic and consider multiple modalities (audio+video+text) and fuse them to recognize emotions. In this paper, we propose a multimodal approach to emotion recognition. Nevertheless, we briefly outline some of the prominent unimodal approaches as some of the techniques are applicable to our setting.
Unimodal Approaches: COSMIC (Yu et al., 2019) performs text only emotion classification problem by leveraging commonsense knowledge. DialogXL (Shen et al., 2021a) uses XLnet (Yang et al., 2019) as architecture in dialogue feature extraction. CESTa (Wang et al., 2020) captures the emotional consistency in the utterances using Conditional Random Fields (Lafferty et al., 2001) for boosting the performance of emotion classification. Other popular approaches parallel to our work use graph-based neural networks as their baseline and solve the context propagation issues in RNN-based architectures, including DialogueGCN (Ghosal et al., 2019), RGAT (Ishiwatari et al., 2020), ConGCN (Zhang et al., 2019), and SumAgg-Gin (Sheng et al., 2020). Some of the recent approaches like DAG-ERC (Shen et al., 2021b) combine the strengths of conventional graph-based neural models and recurrence-based neural models.
Multimodal Approaches: Due to the high correlation between emotion and facial cues (Ekman, 1993), fusing modalities to improve emotion recognition has drawn considerable interest (Sebe et al., 2005). Some of the initial approaches include Datcu and Rothkrantz (2014), who fused acoustic information with visual cues for emotion recognition. Wollmer et al. (2010) use contextual information for emotion recognition in a multimodal setting. In the past decade, the growth of deep learning has motivated a wide range of approaches in multimodal settings. The Memory Fusion network (MFN) (Zadeh et al., 2018a) proposes synchronizing multimodal sequences using multi-view gated memory storing intra-view and cross-view interac
tions through time. Graph-MFN (Bagher Zadeh et al., 2018) extends the idea of MFN and introduces Dynamic Fusion Graph (DFG), which learns to model the n-modal interactions and alter its structure dynamically to choose a fusion graph based on the importance of each n-modal dynamics during inference. Conversational memory network (CMN) (Hazarika et al., 2018b) leverages contextual information from the conversation history and uses gated recurrent units to model past utterances of each speaker into memories. Tensor fusion Network (TFN) (Zadeh et al., 2017) uses an outer product of the modalities. Other popular approaches include DialogueRNN (Majumder et al., 2019) that proposes an attention mechanism over the different utterances and models emotional dynamics by its party GRU and global GRU. B2+B4 (Kumar and Vepa, 2020), use a conditional gating mechanism to learn cross-modal information. bc-LSTM (Poria et al., 2017) proposes an LSTM-based model that captures contextual information from the surrounding utterances. Multilogue-Net (Shenoy and Sardana, 2020) proposes a solution based on a context-aware RNN and uses pairwise attention as a fusion mechanism for all three modalities (audio, video, and text). Recently, Delbrouck et al. (2020) proposed TBJE, a transformer-based architecture with modular co-attention (Yu et al., 2019) to encode multiple modalities jointly. CONSK-GCN (Fu et al., 2021) uses graph convolutional network (GCN) with knowledge graphs. Lian et al. (2020) use GNN based architecture for Emotion Recognition using text and speech modalities. Af-CAN (Wang et al., 2021a) proposes RNN based on contextual attention for modeling the transaction and dependence between speakers.
In a conversation involving different speakers, there is a continuous ebb and flow in the emotions of each of the speakers, usually triggered by the context and reactions of other speakers. Inspired by this intuition, we propose a multimodal emotion prediction model that leverages contextual information, inter-speaker and intra-speaker relations in a conversation.
In our model, we leverage both the context of dialogue and the effect of nearby utterances. We model these two sources of information via two means: 1) Global Information: How to capture the impact of underlying context on the emotional
state of an utterance? 2) Local information: How to establish relations between the nearby utterances that preserve both inter-speaker and intra-speaker dependence on utterances in a dialogue?
Global Information: We want to have a unified model that can capture the underlying context and handle its effect on each utterance present in the dialogue. A transformer encoder (Vaswani et al., 2017) architecture is a suitable choice for this goal. Instead of following the conventional sequential encoding by adding positional encodings to the input, in our approach, a simple transformer encoder without any positional encodings leverages the entire context to generate distributed representations (features) efficiently corresponding to each utterance. The transformer facilitates the flow of information from all utterances when predicting emotion for a particular utterance.
Local Information: The emotion expressed in an utterance is often triggered by the information in neighboring utterances. We establish relations between the nearby utterances in a way that is capable of capturing both inter-speaker and intra-speaker effects of stimulus over the emotion state of an utterance. Our approach comes close to DialogueGCN (Ghosal et al., 2019), and we define a graph where each utterance is a node, and directed edges represent various relations. We define relations (directed edges) between nodes  $\mathcal{R}_{ij} = u_i\rightarrow u_j$  , where the direction of the arrow represents the spoken order of utterances. We categorize the directed relations into two types, for self-dependent relations between the utterances spoken by the same speaker  $\mathcal{R}_{intra}$  , and interrelations between the utterances spoken by different speakers  $\mathcal{R}_{inter}$  .We propose to use Relational GCN (Schlichtkrull et al., 2018) followed by a GraphTransformer (Shi et al., 2021) to capture dependency defined by the relations.
Figure 2 shows the detailed architecture. The input utterances go as input to the Context Extractor module, which is responsible for capturing the global context. The features extracted for each utterance by the context extractor form a graph based on interactions between the speakers. The graph goes as input to a Relational GCN, followed by GraphTransformer, which uses the formed graph to capture the inter and intra-relations between the utterances. Finally, two linear layers acting as an emotion classifier use the features obtained for all
Figure 2: The proposed model (COGMEN) architecture.
the utterances to predict the corresponding emotions.
Context Extractor: Context Extractor takes concatenated features of multiple modalities (audio, video, text) as input for each dialogue utterance  $(u_{i}; i = 1, \dots, n)$  and captures the context using a transformer encoder. The feature vector for an utterance  $u_{i}$  with the input features corresponding to available modalities, audio  $(u_{i}^{(a)} \in \mathbb{R}^{d_{a}})$ , text  $(u_{i}^{(t)} \in \mathbb{R}^{d_{t}})$  and video  $(u_{i}^{(v)} \in \mathbb{R}^{d_{v}})$  is:
$$
\mathbf {x} _ {i} ^ {(a t v)} = [ u _ {i} ^ {(a)} \oplus u _ {i} ^ {(t)} \oplus u _ {i} ^ {(v)} ] \in \mathbb {R} ^ {d}
$$
where  $d = d_{a} + d_{t} + d_{v}$ . The combined features matrix for all utterances in a dialogue is given by:
$$
\mathbf {X} = \mathbf {x} ^ {(a t v)} = [ \mathbf {x} _ {1} ^ {(a t v)}, \mathbf {x} _ {2} ^ {(a t v)} \dots , \mathbf {x} _ {n} ^ {(a t v)} ] ^ {T}
$$
We define a Query, a Key, and a Value vector for encoding the input features  $\mathbf{X} \in \mathbb{R}^{n \times d}$  as follows:
$$
Q ^ {(h)} = \mathbf {X} W _ {h, q},
$$
$$
K ^ {(h)} = \mathbf {X} W _ {h, k},
$$
$$
V ^ {(h)} = \mathbf {X} W _ {h, v},
$$
where,  $W_{h,q},W_{h,k},W_{h,v}\in \mathbb{R}^{d\times k}$
The attention mechanism captures the interaction between the Key and Query vectors to output an attention map  $\alpha^{(h)}$ , where  $\sigma_{j}$  denotes the softmax function over the row vectors indexed by  $j$ :
$$
\alpha^ {(h)} = \sigma_ {j} \left(\frac {Q ^ {(h)} (K ^ {(h)}) ^ {T}}{\sqrt {k}}\right)
$$
where  $\alpha^{(h)}\in \mathbb{R}^{n\times n}$  represents the attention weights for a single attention head  $(h)$ . The obtained attention map is used to compute a weighted sum of the values for each utterance:
$$
\operatorname {h e a d} ^ {(h)} = \alpha^ {(h)} (V ^ {(h)}) \in \mathbb {R} ^ {n \times k}
$$
$$
\mathbf {U} ^ {\prime} = \left[ \operatorname {h e a d} ^ {(1)} \oplus \operatorname {h e a d} ^ {(2)} \oplus \dots \operatorname {h e a d} ^ {(H)} \right] W ^ {o}
$$
where,  $W^{o} \in \mathbb{R}^{kH \times d}$  and  $H$  represents the total number of heads in multi-head attention. Note  $\mathbf{U}' \in \mathbb{R}^{n \times d}$ . We add residual connection  $\mathbf{X}$  and apply LayerNorm, followed by a feed forward and Add & Norm layer:
$$
\mathbf {U} = \operatorname {L a y e r N o r m} \left(\mathbf {X} + \mathbf {U} ^ {\prime}; \gamma_ {1}, \beta_ {1}\right);
$$
$$
\mathbf {Z} ^ {\prime} = \operatorname {R e L U} \left(\mathbf {U} W _ {1}\right) W _ {2};
$$
$$
\mathbf {Z} = \operatorname {L a y e r N o r m} \left(\mathbf {U} + \mathbf {Z} ^ {\prime}; \gamma_ {2}, \beta_ {2}\right);
$$
where,  $\gamma_1,\beta_1\in \mathbb{R}^d$ $W_{1}\in \mathbb{R}^{d\times m}$ $W_{2}\in \mathbb{R}^{m\times d}$  and  $\gamma_{2},\beta_{2}\in \mathbb{R}^{d}$  . The transformer encoder provides features corresponding to every utterance in a dialogue  $([\mathbf{z}_1,\mathbf{z}_2,\dots ,\mathbf{z}_n]^T = \mathbf{Z}\in \mathbb{R}^{n\times d})$
Graph Formation: A graph captures inter and intra-speaker dependency between utterances. Every utterance acts as a node of a graph that is connected using directed relations (past and future relations). We define relation types as speaker to speaker. Formally, consider a conversation between  $M$  speakers defined as a dialogue  $\mathcal{D} = \{\mathcal{U}^{S_1},\mathcal{U}^{S_2},\ldots ,\mathcal{U}^{S_M}\}$ , where  $\mathcal{U}^{S_1} = \{u_1^{(S_1)},u_2^{(S_1)},\dots,u_n^{(S_1)}\}$  represent the set of utterances spoken by speaker-1. We define intra relations between the utterances spoken by the same
speaker,  $\mathcal{R}_{intra} \in \{\mathcal{U}^{S_i} \to \mathcal{U}^{S_j}\}$ , and inter relations between the utterances spoken by different speakers,  $\mathcal{R}_{inter} \in \{\mathcal{U}^{S_i} \to \mathcal{U}^{S_j}\}_{i \neq j}$ . We further consider a window size and use  $\mathcal{P}$  and  $\mathcal{F}$  as hyperparameters to form relations between the past  $\mathcal{P}$  utterances and future  $\mathcal{F}$  utterances for every utterance in a dialogue. For instance,  $\mathcal{R}_{intra}$  and  $\mathcal{R}_{inter}$  for utterance  $u_i^{(S_1)}$  (spoken by speaker-1) are defined as:
$$
\begin{array}{l} \mathcal {R} _ {i n t r a} \left(u _ {i} ^ {(S _ {1})}\right) = \left\{u _ {i} ^ {(S _ {1})} \leftarrow u _ {i - \mathcal {P}} ^ {(S _ {1})} \dots u _ {i} ^ {(S _ {1})} \leftarrow u _ {i - 1} ^ {(S _ {1})}, \right. \\ u _ {i} ^ {(S _ {1})} \leftarrow u _ {i} ^ {(S _ {1})}, u _ {i} ^ {(S _ {1})} \rightarrow u _ {i + 1} ^ {(S _ {1})} \dots u _ {i} ^ {(S _ {1})} \rightarrow u _ {i + \mathcal {F}} ^ {(S _ {1})} \} \\ \mathcal {R} _ {i n t e r} \left(u _ {i} ^ {\left(S _ {1}\right)}\right) = \left\{u _ {i} ^ {\left(S _ {1}\right)} \leftarrow u _ {i - \mathcal {P}} ^ {\left(S _ {2}\right)}, \dots , u _ {i} ^ {\left(S _ {1}\right)} \leftarrow u _ {i - 1} ^ {\left(S _ {2}\right)}, \right. \\ u _ {i} ^ {(S _ {1})} \rightarrow u _ {i + 1} ^ {(S _ {2})}, \ldots , u _ {i} ^ {(S _ {1})} \rightarrow u _ {i + \mathcal {F}} ^ {(S _ {2})} \} \\ \end{array}
$$
where  $\leftarrow$  and  $\rightarrow$  represent the past and future relation type respectively (example in Appendix F).
Relational Graph Convolutional Network (RGCN): The vanilla RGCN (Schlichtkrull et al., 2018) helps accumulate relation-specific transformations of neighboring nodes depending on the type and direction of edges present in the graph through a normalized sum. In our case, it captures the inter-speaker and intra-speaker dependency on the connected utterances.
$$
\mathbf {x} _ {i} ^ {\prime} = \boldsymbol {\Theta} _ {\mathrm {r o o t}} \cdot \mathbf {z} _ {i} + \sum_ {r \in \mathcal {R}} \sum_ {j \in \mathcal {N} _ {r} (i)} \frac {1}{| \mathcal {N} _ {r} (i) |} \boldsymbol {\Theta} _ {r} \cdot \mathbf {z} _ {j}
$$
where  $\mathcal{N}_r(i)$  denotes the set of neighbor indices of node  $i$  under relation  $r\in \mathcal{R}$ ,  $\Theta_{root}$  and  $\Theta_r$  denote the learnable parameters of RGCN,  $|\mathcal{N}_r(i)|$  is the normalization constant and  $\mathbf{z}_j$  is the utterance level feature coming from the transformer.
GraphTransformer: For extracting rich representation from the node features, we use a GraphTransformer (Shi et al., 2021). GraphTransformer adopts the vanilla multi-head attention into graph learning by taking into account nodes connected via edges. Given node features  $H = \mathbf{x}_1', \mathbf{x}_2', \dots, \mathbf{x}_n'$  obtained from RGCN,
$$
\mathbf {h} _ {i} ^ {\prime} = \mathbf {W} _ {1} \mathbf {x} _ {i} ^ {\prime} + \sum_ {j \in \mathcal {N} (i)} \alpha_ {i, j} \mathbf {W} _ {2} \mathbf {x} _ {j} ^ {\prime}
$$
where the attention coefficients  $\alpha_{i,j}$  are computed via multi-head dot product attention:
$$
\alpha_ {i, j} = \operatorname {s o f t m a x} \left(\frac {\left(\mathbf {W} _ {3} \mathbf {x} _ {i} ^ {\prime}\right) ^ {\top} \left(\mathbf {W} _ {4} \mathbf {x} _ {j} ^ {\prime}\right)}{\sqrt {d}}\right)
$$
Dataset	Number of dialogues [utterances]
train	valid	test
IEMOCAP	120 [5810 (5146+664)]	31 [1623]
MOSEI	2249 [16327]	300 [1871]	646 [4662]
Emotion Classifier: A linear layer over the features extracted by GraphTransformer  $(\mathbf{h}_i^{\prime})$  predicts the emotion corresponding to the utterance.
$$
h _ {i} = \operatorname {R e L U} \left(W _ {1} \mathbf {h} _ {i} ^ {\prime} + b _ {1}\right)
$$
$$
\mathcal {P} _ {i} = \operatorname {s o f t m a x} \left(W _ {2} h _ {i} + b _ {2}\right)
$$
$$
\hat {y} _ {i} = \arg \max (\mathcal {P} _ {i})
$$
where  $\hat{y}_i$  is the emotion label predicted for the utterance  $u_i$ .
We experiment for the Emotion Recognition task on the two widely used datasets: IEMOCAP (Busso et al., 2008) and MOSEI (Zadeh et al., 2018b). The dataset statistics are given in Table 1. IEMOCAP is a dyadic multimodal emotion recognition dataset where each utterance in a dialogue is labeled with one of the six emotion categories: anger, excited, sadness, happiness, frustrated, and neutral. In literature, two IEMOCAP settings are used for testing, one with 4 emotions (anger, sadness, happiness, neutral) and one with 6 emotions. We experiment with both of these settings. MOSEI is a multimodal emotion recognition dataset annotated with 7 sentiments (-3 (highly negative) to +3 (highly positive)) and 6 emotion labels (happiness, sadness, disgust, fear, surprise, and anger). Note that the emotion labels differ across the datasets. We use weighted F1-score and Accuracy as evaluation metrics (details in Appendix C).
Implementation Details: For IEMOCAP, audio features (size 100) are extracted using OpenS-mile (Eyben et al., 2010), video features (size 512) are taken from Baltrusaitis et al. (2018), and text features (size 768) are extracted using sBERT (Reimers and Gurevych, 2019). Audio features for the MOSEI dataset were taken from Delbrouck et al. (2020), which are extracted using librosa (McFee et al., 2015) with 80 filter banks, making the feature vector size of 80. The video features (size 35) are taken from Zadeh et al. (2018b). The textual features (size 768) are obtained using sBERT. The textual features are
Models	IEMOCAP: Emotion Categories
Happy	Sad	Neutral	Angry	Excited	Frustrated	Avg.
F1 (%)	F1 (%)	F1 (%)	F1 (%)	F1 (%)	F1 (%)	Acc. (%)	F1 (%)
bc-LSTM	35.6	69.2	53.5	66.3	61.1	62.4	59.8	59.0
memnet	33.0	69.3	55.0	66.1	62.3	63.0	59.9	59.5
TFN	33.7	68.6	55.1	64.2	62.4	61.2	58.8	58.5
MFN	34.1	70.5	52.1	66.8	62.1	62.5	60.1	59.9
CMN	32.6	72.9	56.2	64.6	67.9	63.1	61.9	61.4
ICON	32.8	74.4	60.6	68.2	68.4	66.2	64.0	63.5
DialogueRNN	32.8	78.0	59.1	63.3	73.6	59.4	63.3	62.8
CAN	31.8	71.9	60.4	66.7	68.5	66.1	63.2	62.4
Af-CAN	37.0	72.1	60.7	67.3	66.5	66.1	64.6	63.7
COGMEN	51.9	81.7	68.6	66.0	75.3	58.2	68.2	67.6
sentence-level static features. For Audio and Visual modalities, we use sentence/utterance level features by averaging all the token level features. We fuse the features of all the available modalities (A.audio) + T(text) + V(video): ATV) via concatenation. We also explored other fusion mechanisms (Appendix G.1). However, concatenation gave the best performance. We conduct a hyper-parameter search for our proposed model using Bayesian optimization techniques (details in Appendix A).
Baselines: We do a comprehensive evaluation of COGMEN by comparing it with a number of baseline models. For IEMOCAP, we compare our model with the existing multimodal frameworks (Table 2), which includes DialogueRNN (Majumder et al., 2019), bc-LSTM (Poria et al., 2017), CHFusion (Majumder et al., 2018), memnet (Sukhbaatar et al., 2015), TFN (Zadeh et al., 2017), MFN (Zadeh et al., 2018a), CMN (Hazarika et al., 2018b), ICON (Hazarika et al., 2018a), and Af-CAN (Wang et al., 2021b). For MOSEI, COGMEN is compared (Table 4) with multimodal models, including Multilogue-Net (Shenoy and Sardana, 2020) and TBJE (Delbrouck et al., 2020) (details and analysis of baselines in §6).
Table 2: Results on IEMOCAP (6-way) multimodal  $(\mathrm{A} + \mathrm{T} + \mathrm{V})$  setting. Avg. denotes weighted average.
Model	F1-score (%)
bc-LSTM	75.13
CHFusion	76.80
COGMEN	84.50
Table 3: Results on IEMOCAP dataset for 4 emotion classes in multimodal setting (weighted F1-score).
IEMOCAP: Table 2 shows the results for IEMOCAP (6-way) multimodal setting. Overall, COG