MultiEMO: An Attention-Based Correlation-Aware Multimodal Fusion Framework for Emotion Recognition in Conversations
Year: 2023

Abstract:
Emotion Recognition in Conversations (ERC) is an increasingly popular task in the Natural Language Processing community, which seeks to achieve accurate emotion classifications of utterances expressed by speakers during a conversation. Most existing approaches focus on modeling speaker and contextual information based on the textual modality, while the complementarity of multimodal information has not been well leveraged, few current methods have sufficiently captured the complex correlations and mapping relationships across different modalities. Furthermore, existing state-ofthe-art ERC models have difficulty classifying minority and semantically similar emotion categories. To address these challenges, we propose a novel attention-based correlation-aware multimodal fusion framework named MultiEMO, which effectively integrates multimodal cues by capturing cross-modal mapping relationships across textual, audio and visual modalities based on bidirectional multi-head crossattention layers. The difficulty of recognizing minority and semantically hard-to-distinguish emotion classes is alleviated by our proposed Sample-Weighted Focal Contrastive (SWFC) loss. Extensive experiments on two benchmark ERC datasets demonstrate that our MultiEMO framework consistently outperforms existing state-of-the-art approaches in all emotion categories on both datasets, the improvements in minority and semantically similar emotions are especially significant.

heading: 1 Introduction
text: Emotion Recognition in Conversations (ERC) is an emerging task in the field of Natural Language Processing (NLP), which aims to identify the emotion of each utterance in a conversation based on textual, audio and visual cues of the speaker. ERC has attracted an enormous amount of attention from both academia and industry, due to its widespread
∗Corresponding author.
potentials in social media analysis (Chatterjee et al., 2019), health care services (Hu et al., 2021b), empathetic systems (Jiao et al., 2020), and so on.
To solve the problem of ERC, numerous approaches have been proposed. The majority of existing works concentrate on modeling speaker dependencies and conversational contexts (Poria et al., 2017; Hazarika et al., 2018a,c; Majumder et al., 2019; Ghosal et al., 2019, 2020; Shen et al., 2021; Hu et al., 2021a,b; Li et al., 2021a; Joshi et al., 2022; Lee and Lee, 2022), while there still exist several unsolved challenges: (1) The complementarity of multimodal information has not been well exploited. Apart from rich information contained in the textual modality, the tone and intonation of the speaker can indicate the intensity of the emotion, facial expressions of interlocutors are also able to explicitly reveal emotional tendencies (Li et al., 2022). Figure 1 shows an example where the complementarity of acoustic and visual signals in addition to the textual modality is essential for an accurate emotion classification. Nevertheless, most existing approaches focus on the textual modality of utterances or simply utilize feature concatenation as the multimodal fusion mechanism (Poria et al., 2017; Hazarika et al., 2018a,c; Majumder et al., 2019; Zhang and Chai, 2021; Li et al., 2022) without modeling the complicated correlations and mapping relationships across textual, audio and visual modalities, which results in an inadequate integration of multimodal cues. (2) Unsatisfactory performances in minority emotion classes. Existing benchmark datasets in ERC, such as IEMOCAP (Busso et al., 2008) and MELD (Poria et al., 2019), suffer from the problem of imbalanced classes. As illustrated in Figure 2, both MELD and IEMOCAP are class-imbalanced, especially in MELD, where the majority class neutral takes up a much larger proportion than minority classes disgust and fear. Current state-of-the-art approaches fail to solve the class imbalance problem and have poor perfor-
14752
mances in minority emotions. (3) The difficulty of distinguishing between semantically similar emotions. It remains to be a challenging task to correctly classify different emotions that are semantically related, such as disgust and anger in MELD, since they share similar underlying cognitive, affective and physiological features, and tend to be expressed by speakers in similar contexts.
To address the above problems, in this paper, we propose a novel attention-based correlationaware multimodal fusion framework named MultiEMO. Firstly, unimodal feature extraction and context modeling are performed for each modality, in which we introduce a visual feature extractor named VisExtNet based on a Multi-task Cascaded Convolutional Network (MTCNN) (Zhang et al., 2016) and a VGGFace2 (Cao et al., 2018) pretrained ResNet-101 (He et al., 2016). VisExtNet accurately captures visual cues of utterance videos by extracting emotion-rich facial expressions of interlocutors without modeling redundant scenerelated visual information. Secondly, we propose a multimodal fusion model called MultiAttn to effectively integrate multimodal information based on bidirectional multi-head cross-attention layers (Vaswani et al., 2017), which successfully captures complex cross-modal correlations and mapping relationships across contextualized textual, audio and visual features. Thirdly, in order to mitigate the difficulty of classifying minority and semantically similar emotion classes, enlightened by Focal Contrastive loss (Zhang et al., 2021), a Sample-Weighted Focal Contrastive (SWFC) loss is proposed, in which we assign more focus to hard-to-classify minority classes and make sample pairs with different emotion labels mutually exclusive with each other such that semantically similar emotions can be better distinguished. In addition, we utilize a Soft Hirschfeld-GebeleinRényi (Soft-HGR) loss (Wang et al., 2019) to maximize the correlations across multimodal-fused textual, audio and visual feature representations extracted from MultiAttn. Finally, extensive experiments are conducted on two ERC benchmark datasets, MELD and IEMOCAP. Experimental results demonstrate the effectiveness and superiority of our proposed MultiEMO framework compared with existing state-of-the-art approaches, the improvements in minority and semantically similar emotion categories are especially remarkable.
The main contributions of this work can be sum-
marized as follows:
• We propose a novel visual feature extraction network named VisExtNet, which effectively captures visual cues of interlocutors without modeling redundant scene information.
• We design a multimodal fusion model called MultiAttn based on bidirectional multi-head cross-attention layers, which successfully models the complicated correlations across textual, audio and visual modalities.
• We innovatively introduce a SWFC loss to address the difficulty of classifying minority and semantically similar emotion classes.
• We conduct extensive experiments on MELD and IEMOCAP, results show that our proposed MultiEMO framework achieves stateof-the-art performances on both datasets, the improvements in minority and semantically similar emotions are especially notable.

heading: 2 Related Work

heading: 2.1 Recurrence-based Models
text: (Poria et al., 2017) proposes a Long Short-Term Memory (LSTM) based network named BCLSTM to extract contextual information from dialogues. Interactive Conversational memory Network (ICON) is proposed by (Hazarika et al., 2018b), which models self- and inter-speaker influences based on gated recurrent units (GRUs). (Majumder et al., 2019) introduces a DialogueRNN to model speaker states and contextual information using GRUs. (Lu et al., 2020) proposes a GRUbased Iterative Emotion Interaction Network (IterativeERC), which models emotion interactions by iteratively using predicted emotion labels. (Ma et al., 2022) designs a Multi-View Network (MVN) to model emotion representations of queries from both word- and utterance-level views based on the attention mechanism and bidirectional GRUs.

heading: 2.2 Graph-based Models
text: (Ghosal et al., 2019) proposes a Dialogue Graph Convolutional Network (DialogueGCN) to model the conversational context with a directed graph. (Zhang et al., 2019) designs a graph-based model named ConGCN to capture both context- and speaker-sensitive dependencies. (Shen et al., 2021) introduces a Directed Acyclic Neural Network (DAG-ERC) to capture intrinsic structures of conversations using a directed acyclic graph (DAG). A contextualized Graph Neural Network (GNN) based model named COGMEN is proposed by (Joshi et al., 2022), which exploits both local- and global-level information in a conversation.

heading: 2.3 Transformer-based Models
text: (Li et al., 2020) introduces a transformer-based context-sensitive model named HiTrans based on two hierarchical transformers. (Li et al., 2022) designs a transformer-based model called EmoCaps to extract emotional tendencies from multimodal features. CoMPM is introduced by (Lee and Lee, 2022), which consists of a transformer-encoder based context embedding module (CoM) and a pretrained memory module (PM).

heading: 2.4 Multimodal-based Models
text: Multimodal Fused Graph Convolutional Network (MMGCN) is proposed by (Hu et al., 2021b), which leverages both multimodal information and long-distance contexts. (Li et al., 2021b) introduces a quantum-like framework named QMNN to jointly perform multimodal fusion and conversational context modeling. (Chudasama et al., 2022) designs a multimodal fusion network named M2FNet based on multi-head attention layers to capture crossmodal interactions. A unified framework named UniMSE is proposed by (Hu et al., 2022), in which multimodal representations are fused by injecting acoustic and visual signals into the T5 model.

heading: 3 Methodology

heading: 3.1 Problem Definition
text: Given a dialogue which consists of n utterances u1,u2, . . . ,un uttered by speakers Su1 , Su2 , . . . , Sun , the goal of ERC is to predict the emotion label of each utterance in the dialogue from the pre-defined k-class emotion category set Y . Each utterance has its corresponding textual (t), audio (a) and visual (v) modalities, which can be illustrated as follows:
ui = {uti,uai ,uvi }, i ∈ {1, . . . , n} (1)

heading: 3.2 Model Overview
text: The overall framework of MultiEMO is illustrated in Figure 3, which is made up of four key components: Unimodal Feature Extraction, Context Modeling, Multimodal Fusion and Emotion Classification. In the subsequent subsections, we discuss each module in detail.

heading: 3.3 Unimodal Feature Extraction and Context Modeling

heading: 3.3.1 Textual Modality
text: Existing research often adopts two different paradigms to extract contextualized textual features: (1) Two-stage paradigm (Li et al., 2020; Chudasama et al., 2022): Text sequences are first fed into a pre-trained language model to learn utterance-level local textual representations and then to another transformer to generate dialoguelevel global textual features by incorporating contextual information in the conversation. (2) Onestage paradigm (Kim and Vossen, 2021; Lee and Lee, 2022): Local utterance-level information and global dialogue-level conversational contexts are jointly captured through fine-tuning a single pretrained language model. We have explored both approaches and experimental results demonstrate
that the one-stage paradigm slightly outperforms the two-stage paradigm. For the sake of computational efficiency, we adopt the one-stage paradigm.
To be specific, following (Kim and Vossen, 2021), each textual utterance uti is prefixed with the speaker name of the utterance Sui , such that speaker information can be effectively encoded. Then, the input sequence for the i-th utterance is composed of three segments to incorporate contextual information: preceding contextual utterances {ut1, . . . ,uti−1}, current utterance uti, and succeeding contextual utterances {uti+1, . . . ,utn}. These three segments are concatenated and separated by [SEP] before being fed into a pre-trained RoBERTa model and a subsequent fully-connected layer, with the embedding of the first hidden state [CLS] utilized as the learned contextualized 256- dimensional textual representation cti for uti.

heading: 3.3.2 Audio Modality
text: Audio Feature Extraction: We follow (Majumder et al., 2019) and use a OpenSMILE (Eyben et al., 2010) to extract a 6373-dimensional feature representation for each utterance audio, then a fully-connected layer is adopted to obtain a 512- dimensional feature hai for each input audio uai . Audio Context Modeling: After unimodal audio feature extraction, we employ a DialogueRNN (Majumder et al., 2019) to capture a contextualized
audio feature cai with 256 dimensions for each audio clip. The speaker-modeling nature of DialogueRNN makes it effective in integrating audio cues from different speakers (Li et al., 2022).

heading: 3.3.3 Visual Modality
text: Visual Feature Extraction: Most existing works (Hazarika et al., 2018a,c; Majumder et al., 2019; Zhang and Chai, 2021; Li et al., 2022) utilize a 3D-CNN (Tran et al., 2015) to capture visual features from video clips. Recently, (Chudasama et al., 2022) proposes a dual network based on a Multitask Cascaded Convolutional Network (MTCNN) (Zhang et al., 2016), which demonstrates to be effective. Both approaches encode not only facial expressions of interlocutors but also scene-related information for each utterance clip. However, we
argue that these two approaches are flawed because visual surrounding information is redundant. Firstly, there are no explicit correlations between scene information and the emotion of the speaker, because dialogues that happen in the same scene do not tend to share similar emotional tendencies. To illustrate, a large proportion of conversations in MELD take place at home, but the emotions of these conversations vary significantly. In addition, the scene normally remains unchanged throughout the conversation. Therefore, capturing scenerelated visual information for each utterance is unnecessary and may lead to a wrong understanding of the speaker’s actual emotional tendency due to the influence of irrelevant scene information.
To address this problem, we propose a novel visual feature extractor named VisExtNet, which is made up of a MTCNN and a ResNet-101 (He et al., 2016) pre-trained on VGGFace2 (Cao et al., 2018). The architecture of VisExtNet is shown in Figure 4. VisExtNet aims to effectively capture visual cues by integrating facial expressions of interlocutors from multiple frames without encoding redundant scene-related information.
For an utterance video uvi , visual feature extraction is performed on 20 frames of the utterance clip, with each frame selected using a step of
number of frames 20 . Specifically, each frame is first sent into a MTCNN to accurately detect the faces of all interlocutors present in the scene at that frame, each detected face is then passed through a VGGFace2 pretrained ResNet-101 to extract a emotion-rich visual feature vector. The concatenation of facial expression features from all participants is regarded as the visual representation of that frame. The same process is repeated for each of the 20 frames, after which the output features of all frames are average pooled over the frame axis to obtain a 1000- dimensional visual feature vector hvi . Visual Context Modeling: Similar to audio context modeling, after visual feature extraction, we utilize another DialogueRNN to learn a 256- dimensional contextualized visual representation cvi for each video clip.

heading: 3.4 Multimodal Fusion
text: Existing literature fails to effectively integrate multimodal information, the complex correlations and mapping relationships across multiple modalities have not been well captured. To tackle this issue, inspired by (Chudasama et al., 2022), we propose a novel multimodal fusion network named MultiAttn based on the bidirectional multi-head cross-attention mechanism (Vaswani et al., 2017),
in which querys are generated from one modality while keys and values come from a different modality, and both preceding and succeeding contexts are leveraged when calculating attention distributions. The architecture of MultiAttn is shown in Figure 5.
MultiAttn is made up of three components: MultiAttntext, MultiAttnaudio and MultiAttnvisual, each of which aims to integrate one modality with complementary information from the other two modalities. As illustrated in Figure 5, MultiAttntext, MultiAttnaudio and MultiAttnvisual share the same building blocks and only differ in terms of input Query, Key and Value. Thus, for the sake of brevity, we use MultiAttntext to illustrate how multimodal fusion works. MultiAttntext effectively incorporates the textual modality with audio and visual cues through a three-stage approach: (1) MultiAttntext first learns cross-modal correlations and mapping relationships between textual and audio modalities by treating the textual modality as Query and the audio modality as Key and Value for the bidirectional multi-head cross-attention operation; (2) The learned output from the first stage is then utilized as the new Query while the visual modality is regarded as Key and Value for another bidirectional multi-head cross-attention layer to fuse the textual modality with visual cues; (3) Finally, a feed-forward network consisting of two fully-connected layers with a Rectified Linear Unit (ReLU) is adopted, which operates as a key-value memory (Geva et al., 2021). In addition, we employ residual connection and layer normalization over the output of each stage to facilitate the training process. To construct a deeper and more powerful network, we utilize T layers of MultiAttntext, MultiAttnaudio and MultiAttnvisual as the full model architecture of MultiAttn, where the output of each layer is fed into the next layer as the new Query.
Given the Queries of all utterances Ft (j−1) =
[ft (j−1) 1 , . . . , ft (j−1) n ]
T learned from layer j − 1 , audio and visual features Ca = [ca1, . . . , can]T and Cv = [cv1, . . . , cvn]T, the calculation of MultiAttntext at layer j is illustrated as follows:
[Qta (j) h ,K ta(j) h ,V ta(j) h ] = [F t(j−1)WQ ta(j) h , CaWK ta(j) h ,CaWV ta(j) h ], h ∈ {1, . . . ,H} (2)
Ata (j) h = Softmax( Qta
(j) h K ta(j) h
T
√ d
Kta(j)h
)Vta (j)
h ,
h ∈ {1, . . . ,H} (3)
MHta (j) = Cat(Ata (j) 1 , . . . ,A ta(j) H )W Ota
(j)
(4)
Fta (j) = LayerNorm(Ft (j−1) + MHta (j) ) (5)
[Qtav (j) h ,K tav(j) h ,V tav(j) h ] = [F ta(j)WQ
tav(j) h ,
CvWK tav(j) h ,CvWV tav(j) h ], h ∈ {1, . . . ,H} (6)
Atav (j) h = Softmax( Qtav
(j) h K tav(j) h
T
√ d
Ktav(j)h
)Vtav (j)
h ,
h ∈ {1, . . . ,H} (7)
MHtav (j) = Cat(Atav (j) 1 , . . . ,A tav(j) H )W Otav
(j)
(8)
Ftav (j) = LayerNorm(Fta (j) + MHtav (j) ) (9)
FFNt (j) 1 = max(0,F tav(j)WFFN
t(j) 1 + b FFNt(j)1 )
(10)
FFNt (j) 2 = FFN t(j) 1 W FFNt
(j)
2 + b FFNt(j)2
(11)
Ft (j) = LayerNorm(Ftav (j) + FFNt (j)
2 ) (12)
Where WQ ta(j) h , WK ta(j) h , WV ta(j) h , WO ta(j) , WQ tav(j) h , WK tav(j) h , WV tav(j) h , WO tav(j)
(h ∈ {1, . . . ,H}), WFFNt (j) 1 , WFFN t(j)
2 are projection matrices, b
FFNt(j)1 and b FFNt(j)2 are bias parame-
ters, H is the number of attention heads, Cat stands for concatenation.

heading: 3.5 Emotion Classification
text: After multimodal fusion, the learned multimodalfused textual, audio and visual feature representations fti, fai and fvi are concatenated and then sent into a fully-connected layer and a subsequent 2- layer Multilayer Perceptron (MLP) with a ReLU. Finally, a Softmax layer is utilized to compute a probability distribution over the emotion category set, where the emotion label with the highest probability is chosen as the prediction ŷi for the i-th utterance. The calculation is illustrated as follows:
fi = fti ⊕ fai ⊕ fvi (13) zi = Wzfi + bz (14)
li = max(0,Wlzi + bl) (15) pi = Softmax(W
smaxli + bsmax) (16) ŷi = argmax
t (pi[t]) (17)
Where ⊕ denotes concatenation, Wz , Wl and Wsmax are weight matrices, bz , bl and bsmax are bias parameters.

heading: 3.6 Training Objectives
text: Given a batch of N samples consisting of M dialogues, where the i-th dialogue contains C(i) utterances, training objectives are defined as follows: SWFC Loss: To alleviate the difficulty of classifying minority and semantically similar emotions, we propose a novel loss function named SampleWeighted Focal Contrastive (SWFC) loss based on the Focal Contrastive loss (Zhang et al., 2021) loss by introducing a sample-weight term and a focusing parameter, in which we assign more importance to hard-to-classify minority classes during the training phase, and make sample pairs with different emotion labels mutually exclusive with each other to maximize inter-class distances, such that semantically similar emotions can be better distinguished. The SWFC loss is defined as follows:
s (i) j,g = exp (zTi,jzi,g/τ)∑ zi,s∈Ai,j exp (z T i,jzi,s/τ)
(18)
LSWFC = − M∑
i=1
C(i)∑
j=1
( N
nyi,j )α
1 |Ri,j | ∑
zi,g∈Ri,j
(1− s(i)j,g)γ log s (i) j,g (19)
Where zi,j is the output of the fully-connected layer (Equation 14) for utterance j in dialogue i, Ai,j is the set of features in the batch other than zi,j , yi,j is the label of utterance j in dialogue i, Ri,j = {zi,g ∈ Ai,j |yi,g = yi,j} is the set of positive features that share the same label as zi,j , nyi,j is the count of label yi,j in the batch, α is a sampleweight parameter that controls the degree of focus on minority classes, τ is a temperature parameter that controls the strength of penalties on negative
samples, γ is a focusing parameter which forces the model to focus on hard-to-classify examples. Soft-HGR Loss: We utilize a Soft HirschfeldGebelein-Rényi (Soft-HGR) loss (Wang et al., 2019) to maximize the correlations across multimodal-fused textual, audio and visual features extracted from MultiAttn. The Soft-HGR loss is defined as follows:
LSoft-HGR = − ∑
Q ̸=V,Q,V∈F (E[QTV]
− 1 2 tr(cov(Q)cov(V)))
s.t. E[Q] = 0, ∀Q ∈ F.
(20)
Where F = {Ft,Fa,Fv}, Ft = [ft1, . . . , ftN ]T, Fa = [fa1, . . . , faN ]T, F
v = [fv1, . . . , fvN ]T. Expectations and covariances are approximated through sample means and sample covariances. Cross-Entropy Loss: In addition, we adopt a Cross-entropy loss to measure the difference between predicted probabilities and true labels:
LCE = − M∑
i=1
C(i)∑
j=1
log pi,j [yi,j ] (21)
Where pi,j is the probability distribution over the emotion classes for utterance j in dialogue i, yi,j is the ground-truth label of utterance j in dialogue i. Full Loss Function: A linear combination of SWFC loss, Soft-HGR loss and Cross-entropy loss is leveraged as the full loss function:
LTrain = 1
N (µ1LSWFC + µ2LSoft-HGR
+ (1− µ1 − µ2)LCE) + λ||θ||22, µ1, µ2 ∈ [0, 1] (22)
Where µ1 and µ2 are tunable hyperparameters, λ is the L2 regularization weight, θ is the set of all trainable parameters.

heading: 4 Experimental Settings

heading: 4.1 Datasets
text: IEMOCAP (Busso et al., 2008): IEMOCAP contains approximately 12 hours of videos of dyadic conversations, which are segmented into 7433 utterances and 151 dialogues. Each utterance is annotated with one of six emotion labels: happiness, sadness, neutral, anger, excitement and frustration. MELD (Poria et al., 2019): MELD is a multi-party dataset with 13708 utterances and 1433 dialogues from the TV series Friends. Each utterance is annotated with one of seven emotion categories: anger, disgust, fear, joy, neutral, sadness and surprise.

heading: 4.2 Baseline Methods
text: BC-LSTM (Poria et al., 2017): BC-LSTM models conversational contexts through bidirectional LSTMs without differentiating different speakers. DialogueRNN (Majumder et al., 2019): DialogueRNN models contextual information and
speaker states through three distinct GRUs. DialogueGCN (Ghosal et al., 2019): DialogueGCN captures the context by modeling conversations using a directed graph. IterativeERC (Lu et al., 2020): IterativeERC iteratively uses predicted emotion labels instead of gold emotion labels to model emotion interactions. QMNN (Li et al., 2021b): QMNN captures conversational contexts and conducts multimodal fusion from a novel quantum perspective. MMGCN (Hu et al., 2021b): MMGCN models long-distance conversational contexts by leveraging Graph Convolutional Networks (GCNs). MVN (Ma et al., 2022): MVN effectively captures emotion representations of queries from both wordand utterance-level views. UniMSE (Hu et al., 2022): UniMSE leverages the similarities and complementaries between emotions to achieve better predictions.

heading: 4.3 Implementation Details
text: Modality Setting: We utilize textual, audio and visual modalities of utterances to conduct experiments on both MELD and IEMOCAP. Hyperparameter Settings: (1) Dataset-specific settings: Since MELD is significantly more classimbalanced than IEMOCAP, the batch size is designed to be 64 on IEMOCAP and 100 on MELD. (2) Dataset-generic settings: The number of training epochs is 100, the optimizer is Adam (Kingma and Ba, 2015) with β1 = 0.9 and β2 = 0.99, the learning rate is initialized with 0.0001 and decays by 0.95 after every 10 epochs, the L2 regularization weight λ is 0.00001. To avoid overfitting, we apply Dropout (Srivastava et al., 2014) layers with
a dropout rate of 0.1. (3) Hyperparameters in MultiEMO: The number of layers T in MultiAttn is tuned to be 6, the temperature parameter τ , the sample-weight parameter α and the focusing parameter γ in the SWFC loss are designed to be 0.8, 0.8 and 2 respectively, the combining coefficients µ1 and µ2 in the full training loss function LTrain are tuned to be 0.4 and 0.3 respectively. Evaluation Metrics: We use the Weighted-average F1 score (Weighted-F1) for model evaluations.

heading: 5 Results and Analysis

heading: 5.1 Comparison with Baseline Models
text: The comparisons between MultiEMO and existing state-of-the-art approaches on IEMOCAP and MELD are shown in Table 1 and Table 2 respectively. Experimental results demonstrate that MultiEMO achieves the new state-of-the-art performances on both datasets and outperforms existing approaches across all emotion categories, with significant improvements in minority and semantically similar classes. Specifically, on IEMOCAP, MultiEMO surpasses MVN by 17.97% Weighted-F1 in the minority class Happiness and achieves relative Weighted-F1 improvements of 8.49% and 10.54% in two similar classes Sadness and Frustration respectively; On MELD, MultiEMO gains a remarkable 153.59% relative improvement in minority emotion Fear, and outperforms the previous best baselines in semantically-similar emotion pairs Anger and Disgust by 11.31% and 68.12%.

heading: 5.2 Different Modality Settings
text: The comparison of MultiEMO with different modality settings on IEMOCAP and MELD is illustrated in Table 3. From Table 3 we can see that the textual modality of utterances plays a major role in ERC, while the complementary cues from audio and visual modalities can bring considerable improvements over the text-based MultiEMO.

heading: 5.3 Ablation Study
text: To study the contributions of different components in MultiEMO to model performances, we conduct ablation studies on both IEMOCAP and MELD, the results are shown in Table 1 and Table 2. Impact of VisExtNet: To study the effect of VisExtNet, we implement MultiEMOw/o VisExtNet, in which the proposed VisExtNet is replaced by a 3D-CNN. Experimental results show that the performances of MultiEMOw/o VisExtNet decrease in all
emotion categories on both IEMOCAP and MELD, with a more notable decline on MELD, since the complicated multi-party conversations in MELD make it more challenging for a 3D-CNN to accurately capture visual cues. The inferior performances of MultiEMOw/o VisExtNet on both datasets prove the effectiveness of VisExtNet. Impact of MultiAttn: To analyze the impact of MultiAttn, we implement MultiEMOw/o MultiAttn, where we replace MultiAttn with feature concatenation to fuse contextualized multimodal features. As shown in Table 1 and Table 2, the performances of MultiEMOw/o MultiAttn fall sharply in all emotion classes of IEMOCAP and MELD, which proves the importance and superiority of capturing crossmodal correlations and dependencies across textual, audio and visual modalities using MultiAttn. Impact of SWFC Loss: To study the contribution of SWFC loss, we implement another variant MultiEMOw/o SWFC loss by removing the SWFC loss part from the training loss function. Experimental results demonstrate that the performances of MultiEMOw/o SWFC loss drop considerably on both IEMOCAP and MELD, the declines in minority and semantically similar emotion classes are remarkably striking, while the decreases in majority classes are merely marginal. In addition, the degree of decline is more noticeable on MELD since MELD is significantly more class-imbalanced than IEMOCAP. The results of MultiEMOw/o SWFC loss prove the effectiveness of SWFC loss in mitigating the difficulty of classifying minority and semantically similar emotion categories.

heading: 5.4 Case Study
text: A case study is illustrated in Appendix A.1.

heading: 6 Conclusion
text: In this paper, we propose a novel attention-based correlation-aware multimodal fusion framework named MultiEMO for the task of ERC, in which we design a visual feature extractor VisExtNet to accurately capture emotion-rich visual cues and introduce a multimodal fusion model MultiAttn to effectively model the cross-modal interactions and mapping relationships across multiple modalities. Furthermore, the difficulty of classifying minority and semantically similar emotions is mitigated by our proposed SWFC loss. Extensive experiments on IEMOCAP and MELD demonstrate the effectiveness and superiority of MultiEMO.
Limitations
Although our proposed MultiEMO framework has achieved state-of-the-art performances on both IEMOCAP and MELD, there are some limitations with this work:
• Our proposed visual feature extractor VisExtNet does not distinguish between speakers and irrelevant people in the scene, which can be problematic in some scenarios. For instance, one scene in MELD is the cafeteria, where a lot of background actors sit and drink coffee. The facial expressions of these background people have no impact on the emotion of the speaker since they do not participant in the conversation. However, VisExtNet captures visual features of everyone appeared in the cafeteria with no differentiation, which may lead to a wrong comprehension of the speaker’s emotional tendency due to the effects of facial expressions from irrelevant people. We plan to explore effective ways to distinguish between interlocutors and irrelevant people in the scene in our future work.
• The effects of hyperparameters in the SWFC loss (temperature parameter τ , sample-weight parameter α and focusing parameter γ) on model performances have not been fully studied, which will be thoroughly analyzed in our future research.
• Due to the class imbalanced issue with MELD, the SWFC loss requires a large batch size on MELD to ensure that for each training sample there exists at least one positive pair in the batch, which can be computationally expensive. We will investigate effective approaches to tackle this challenge in our future research.
• Even though MultiEMO has achieved remarkable improvements in minority emotion categories, the performances of MultiEMO in minority emotions are still worse than majority classes. How to further improve performances in low-resource emotion classes will be explored in the future.
Ethics Statement
The significant improvements in classifying minority emotion categories brought by our method can make MultiEMO a powerful tool in psychopathological fields such as depression detection, where
minority emotions sadness, fear and anger are important early indicators of depression (O’Connor et al., 2002).

heading: Acknowledgements
text: The research of Shao-Lun Huang is supported in part by National Key R&D Program of China under Grant 2021YFA0715202 and the Shenzhen Science and Technology Program under Grant KQTD20170810150821146.

heading: A Appendix
text: A.1 Case Study of MultiEMO Since the one-stage paradigm (Section 3.3.1) simultaneously performs unimodal textual feature extraction and textual context modeling, to better illustrate the role of context modeling to emotion classification, in the section of case study, the textual modality of the selected utterance is processed using a two-stage paradigm (Yang et al., 2021): unimodal feature extraction with a pretrained RoBERTa and context modeling with another transformer 1, such that the impact of context modeling on the textual modality can be analyzed in conjunction with audio and visual modalities.
Figure 6 depicts a visualization of a prone-tomisclassification utterance in MELD, in which the textual modality "Chandler is a great name!" appears to be positive while the true connotation of the utterance actually implies anger. The heatmaps of the utterance’s textual, audio and visual modalities on the left are obtained after unimodal feature extractions, from which we can see that: (1) Textual modality: the word “great” plays a major role in the text, revealing a strong positive emotion; (2) Audio modality: the higher intensity in the latter part of the audio indicates a flat-to-sharp tone; (3) Visual modality: the frown in the speaker’s face implies a negative emotion. The asynchronization of emotional tendencies from different modalities makes it challenging to identify the actual emotion of this utterance. However, by modeling contextual information and capturing complex cross-modal correlations across contextualized textual, audio and visual modalities, MultiEMO learns a highly representative feature for this utterance, as shown in the heatmap on the right of Figure 6. The learned multimodal-fused feature can be easily classified
1As mentioned in Section 3.3.1, the performance of MultiEMO with a two-stage paradigm is merely marginally outperformed by the one-stage paradigm, both approaches can learn good contextualized textual representations.
to the correct emotion class since it preserves useful emotional cues while discarding irrelevant information through selectively focusing on highlycorrelated information across contextualized textual, audio and visual modalities.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
The section of Limitations is after the Conclusion section.
7 A2. Did you discuss any potential risks of your work? We do not discuss potential risks of our work because of the page limit.
3 A3. Do the abstract and introduction summarize the paper’s main claims? The sections of Abstract and section 1.
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 7 Did you use or create scientific artifacts? Left blank.
B1. Did you cite the creators of artifacts you used? No response.
B2. Did you discuss the license or terms for use and / or distribution of any artifacts? No response.
B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? No response.
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? No response.
B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? No response.
B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. No response.
C 3 Did you run computational experiments? Section 4 and Section 5.
7 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? We do not report the information mentioned in the question because of the page limit.
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? Section 4.
7 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? We do not report the information mentioned in the question because of the page limit.
7 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? We do not report the information mentioned in the question because of the page limit.
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: We plan to explore effective ways to distinguish between interlocutors and irrelevant people in the scene in our future work. • The effects of hyperparameters in the SWFC loss (temperature parameter τ , sample-weight parameter α and focusing parameter γ) on model performances have not been fully studied, which will be thoroughly analyzed in our future research. • Due to the class imbalanced issue with MELD, the SWFC loss requires a large batch size on MELD to ensure that for each training sample there exists at least one positive pair in the batch, which can be computationally expensive. We will investigate effective approaches to tackle this challenge in our future research. • Even though MultiEMO has achieved remarkable improvements in minority emotion categories, the performances of MultiEMO in minority emotions are still worse than majority classes. How to further improve performances in low-resource emotion classes will be explored in the future