M3HG: Multimodal, Multi-scale, and Multi-type Node Heterogeneous Graph for Emotion Cause Triplet Extraction in Conversations
Authors: Qiao Liang, Ying Shen, Tiantian Chen, Lin Zhang
Accepted at: acl25

M^3HG: Multimodal, Multi-scale, and Multi-type Node Heterogeneous Graph for Emotion Cause Triplet Extraction in Conversations

Qiao Liang    Ying Shen*    Tiantian Chen    Lin Zhang
Tongji University, Shanghai, China
{2333091, yingshen, 2111287, cslinzhang}@tongji.edu.cn†

Abstract

Emotion Cause Triplet Extraction in Multimodal Conversations (MECTEC) has recently gained significant attention in social media analysis, aiming to extract emotion utterances, cause utterances, and emotion categories simultaneously. However, the scarcity of related datasets, with only one published dataset featuring highly uniform dialogue scenarios, hinders model development in this field. To address this, we introduce MECAD, the first multimodal, multi-scenario MECTEC dataset, comprising 989 conversations from 56 TV series spanning a wide range of dialogue contexts. In addition, existing MECTEC methods fail to explicitly model emotional and causal contexts and neglect the fusion of semantic information at different levels, leading to performance degradation. In this paper, we propose M^3HG, a novel model that explicitly captures emotional and causal contexts and effectively fuses contextual information at both inter- and intra-utterance levels via a multimodal heterogeneous graph. Extensive experiments demonstrate the effectiveness of M^3HG compared with existing state-of-the-art methods. The codes and dataset are available at https://github.com/redifinition/M3HG.

1 Introduction

Emotion Cause Analysis in Conversations (ECAC) aims at identifying emotions and their causes in conversations, which is a crucial research field in natural language processing (Li et al., 2022b; Wang et al., 2023). However, most of ECAC research (Li et al., 2022b; Wang et al., 2023; Zheng et al., 2023; Chen et al., 2023) only focuses on the textual contexts, overlooking other modalities (Soleymani et al., 2017).

To address this limitation, Wang et al. (2022) proposed a new task called Multimodal Emotion Cause Triplet Extraction in Conversations (MECTEC). The task aims to simultaneously identify the emotion utterance, the corresponding cause utterances, and the emotion category (i.e., the *utter-cause-emotion* triplet) from a conversation containing three modalities: text, audio, and video. Figure 1 illustrates a multimodal conversation between a mother and daughter. In this example, there are six non-neutral utterances, and consequently, six utter-cause-emotion triplets are identified. MECTEC differs from ECAC in 1) multimodal contexts (i.e., text, audio, and video) resulting in more complex emotional expression, and 2) multi-scale semantic information from overall conversation and utterance features like intonation and facial expressions, which pose significant challenges.

Another major challenge in MECTEC is the scarcity of datasets. While numerous text-based datasets exist for ECAC, only one dataset, namely the ECF dataset (Wang et al., 2022), is specifically designed for MECTEC. However, the videos in ECF are all from the Friends TV series with restricted speakers and scenarios, hindering MECTEC model development. Therefore, in this work, a new multimodal, multi-scenario MECTEC dataset, namely MECAD, is constructed. To the best of our knowledge, it is the first of its kind and will greatly facilitate research in this field.

Constrained by the limited dataset, existing MECTEC models have various deficiencies. Wang et al. (2022) proposed a two-stage architecture that predicts emotion and cause utterances separately. However, this approach is computationally intensive and prone to error accumulation. Therefore, recent studies (Hu et al., 2024; Wang et al., 2023; Li et al., 2024a) propose one-stage architectures using graph neural networks or prompt engineering to extract utter-cause-emotion triplets. However, these methods do not explicitly extract specific contexts
Figure 1: An example of the MECTEC task. Each utterance contains three different modalities - text, audio, and video. Arrows represent causal relationships that link the cause utterances to the corresponding emotion utterances. The dashed box at the bottom lists all the <utter-cause-emotion> triplets identified in this example.

related to emotions and their causes. According to emotion attribution theory (Weiner, 1985), the relationships of emotions and their causes are revealed by specific contexts, such as emotional words in texts, and intonations in audio and video conversations. For example, in Utterance 3 in Figure 1, a pleasant facial expression indicates happiness, while “haven’t had your cooking” and a happy tone reveal the cause. The example illustrates that emotions and their causes depend on contextual cues across multiple modalities, highlighting the necessity of explicitly modeling their specific contexts.

In addition, previous work (Wang et al., 2022; Hu et al., 2024; Wang et al., 2023; Li et al., 2024a; Wei et al., 2020) fail to effectively identify the cause utterances occurring after emotion utterances. For example, in Utterance 1 in Figure 1, the reason why Luo is happy cannot be obtained only from the historical context of Utterance 1. To find out the real cause of emotion in Utterance 1, the whole conversation should be scrutinized, which is overlooked by previous work.

Furthermore, existing models (Wang et al., 2022; Hu et al., 2024; Wang et al., 2023; Li et al., 2024a) fail to adequately extract semantic information at different scales. As shown in Figure 1, the semantic information that reveals the relationship of an utterance and its cause not only resides in inter-connections between utterances but also resides in the intra-content of each utterance. Therefore, it’s essential to comprehensively integrate semantic information in different scales during modality fusion.

To solve the aforementioned problems, we propose an MECTEC model based on the multimodal, multi-scale, and multi-type node heterogeneous graph, named \( \mathbf{M^3HG} \). \( \mathbf{M^3HG} \) accurately extracts emotion and cause-related contexts and fuses multimodal, multi-scale semantic information using multimodal heterogeneous graph attention network (HGAT) with multi-type nodes.

Our contributions can be summarized as follows:
• The first Chinese multi-scenario MECTEC dataset, MECA\(\mathbf{D}\), and an online sentiment data annotation toolkit are constructed. The dataset consists of 989 conversations with 10,519 utterances annotated with important information such as emotion labels, their causes, and types of emotional causes. It will greatly benefit the development of models in the MECTEC and related fields.
• An efficient MECTEC model, namely \( \mathbf{M^3HG} \), is proposed to identify utter-cause-emotion triplets from multimodal conversations. It explicitly extracts specific emotion and cause-related contexts to find connections between emotions and causes. Besides, it fully integrates semantic information from inter and intra-utterance levels to enhance the model’s predictive ability.
• Extensive experiments are performed to verify the performance of our proposed model and other state-of-the-art models on MECA\(\mathbf{D}\) and ECF datasets. Experimental results reveal that \( \mathbf{M^3HG} \) outperforms its counterparts, which demonstrates the effectiveness of our model.

2 Related Works

Emotion Cause Analysis in Conversations. Most existing studies on ECAC focus on Causal Emotion Entailment (CEE) and Emotion Cause Pair Extrac-
tion in Conversations (ECPEC). CEE aims to identify which cause utterances trigger the non-neutral emotions of the target utterances. Since CEE assumes emotion utterances are given, most related work (Poria et al., 2021; Li et al., 2022a; Zhang et al., 2022; Gu et al., 2023) viewed CEE as an utterance classification problem. However, because emotions of utterances are often unknown in real-world conversations, Li et al. (2022b) proposed the ECPEC task which additionally predicts emotions for the target utterances. Subsequent work (Wang et al., 2023; Zhao et al., 2023) has incorporated commonsense knowledge into GATs to improve the model’s semantic understanding of emotions and causes, achieving better performance. Besides, some methods (Ding et al., 2020a,b; Wei et al., 2020; Zheng et al., 2022) from models in the Emotion cause Pair Extraction (ECPE) field are also adapted for the ECPEC task.

Multimodal Emotion Cause Triplet Extraction in Conversations. In recent years, multimodal conversation scenarios on social media platforms have grown significantly, as more individuals share their lives and express emotions through live streaming and various online chats. To advance emotion cause analysis in multimodal conversation scenarios, Wang et al. (2022) introduced the MECTEC task and released the ECF dataset. However, few solutions have been proposed for this recently introduced task. Li et al. (2024a) incorporated emotion transition information into emotion-cause pair extraction using a novel labeling constraint, while Hu et al. (2024) fused semantic information across modalities via prompt engineering. These methods treat multimodal fusion and contextual information extraction for emotional causes as separate processes. Furthermore, they fail to effectively integrate semantic information across different scales, which significantly hampers the overall performance of models in the MECTEC task. To address these issues, we propose a model that fully integrates multi-scale semantic information from different modalities, preventing the loss of contextual information during fusion and improving triplet extraction accuracy.

Datasets for the ECAC Task. Table 1 summarizes popular datasets in ECAC. Poria et al. (2021) introduced the RECCON dataset for the ECAC task, and Li et al. (2022b) extended it by building the ConvECPE dataset. Given the multimodal nature of conversations, Wang et al. (2022) developed the ECF dataset for MECTEC. However, all scenes in ECF are drawn from the Friends, limiting the diversity of conversation scenarios and contents.

<table>
  <tr>
    <th>Dataset</th>
    <th>Modalities</th>
    <th>Sources</th>
    <th># Instances</th>
  </tr>
  <tr>
    <td>RECCON<br>ConvECPE</td>
    <td>T<br>T</td>
    <td>Act and Daily Act</td>
    <td>11,769<br>7,433</td>
  </tr>
  <tr>
    <td>ECF<br>MECAD</td>
    <td>T,A,V<br>T,A,V</td>
    <td>TV Friends<br>56 TV series</td>
    <td>13,509<br>10,516</td>
  </tr>
</table>

Table 1: A summary of datasets for ECAC task. T, A, V stand for text, audio and video respectively.

3 Proposed MECAD Dataset

To facilitate the research in MECTEC and other related fields, we constructed a multi-scenario MECTEC dataset called MECAD. Compared with ECF (Wang et al., 2022), MECAD has more diverse conversation scenarios. In addition to labeling emotion categories and their causes for each utterance, we also categorized the types of emotion causes (e.g., event, expression) and the modality of annotation (i.e., text, audio, or video) to support future studies in multimodal emotion cause analysis.

We selected the publicly available M^3ED (Zhao et al., 2022) dataset as our data source, which contains 990 segments from 56 Chinese TV series. However, M^3ED dataset only contains conversation scripts, audios, and screenshots, lacking corresponding videos. Therefore, we endeavored to collect the corresponding video segments based on the conversation timestamps provided by M^3ED. We concatenated sentences to form 989 multimodal conversations with 10,516 full utterances.

We invited 10 Chinese graduate students majored in Psychology to annotate the corresponding cause utterances, the types of emotion causes and the modal cues of annotations in the conversations. To obtain high-quality annotations, we designed detailed guidelines based on previous studies (Dirven, 1997; Steptoe and Brydon, 2009), trained the volunteers, and tested them with annotation cases. Only those passing the test participated in the final annotation process. Each volunteer was paid $50 for their annotations. Then, we randomly assigned three qualified annotators for each conversation. If divergence exists among annotations from different volunteers, the final annotation for the utterance is determined by majority voting. Two strategies were used to review and revise incorrect annotations: 1) Annotation consistency among the three annotators for each TV series is calculated. For series with low consistency, the annotators rechecked
and revised their labels as needed. 2) If disagreements remained, a fourth annotator was invited to relabel the utterances and make the final decision.

To enhance annotation efficiency and accuracy, we developed an online multimodal conversation emotion cause annotation tool. The interface of the annotation tool is shown in Figure 3 in Appendix B. This tool is highly reusable and user-friendly, making it ideal for related research in the future.

We use Cohen’s Kappa (Cohen, 1960) to assess pairwise agreement and Fleiss’s Kappa (McHugh, 2012) for overall consistency among annotators. The Cohen’s Kappa results are in Appendix A, and the Fleiss’s Kappa score of 0.6932 exceeds the threshold of 0.61 (Landis, 1977), confirming the statistical reliability of our annotations.

The dataset statistics and detailed analysis of MECAD are presented in Figure 4 in Appendix A. MECAD provides solid support for assessing the performance and generalization capabilities of MECTEC models in broader scenarios.

4 Framework of Proposed M^3HG

4.1 Task Definition

Given a conversation \( C = \{ (S_i, U_i) \}_{1 \leq i \leq n} \), where \( S_i \) denotes the speaker of the \( i \)-th utterance \( U_i \), \( n \) denotes the length of the conversation \( C \), \( U_i = \{ U_i^t, U_i^a, U_i^v \} \), and \( t, a, v \) are the text, audio and video modality, respectively. The goal of MECTEC is to identify all the utter-cause-emotion triplets from the conversation \( C \):

\[
\mathcal{P} = \{ (U_j^c, U_j^e, y_j^e) \},
\]

where \( U_j^e \) is the \( j \)-th utterance with emotion \( y_j^e \), \( U_j^c \) is the corresponding cause utterance, and \( y_j^e \in \{ \text{Anger, Disgust, Fear, Joy, Sadness, Surprise} \} \) (Ekman, 1992).

4.2 Model Overview

M^3HG is an end-to-end (E2E) MECTEC model, as illustrated in Figure 2. It consists of four key components: unimodal feature extraction, graph construction, multi-scale semantic fusion, and emotion-cause classification.

In unimodal feature extraction, M^3HG extracts local contextual representations for each utterance using modality-specific feature extractors and unimodal encoders. In graph construction, M^3HG constructs a conversation interaction graph using these feature representations to explicitly model the emotion and cause-related contexts. In multi-scale semantic fusion, M^3HG combines semantic information at different scales within the conversation interaction graph to produce a comprehensive feature representation of both emotion and cause contexts. In emotion-cause classification, emotion and cause contextual representations are concatenated and used to extract utter-cause-emotion triplets with with position embedding.

4.3 Unimodal Feature Extraction

First, we utilize SA-RoBERTa (Gu et al., 2020), Wav2Vec2 (Baevski et al., 2020), and DenseNet (Huang et al., 2017) to extract three feature representations \( E^t, E^a, \) and \( E^v \), from text, audio, and video, respectively, where \( E^t \in \mathbb{R}^{n \times d_t} \), \( E^a \in \mathbb{R}^{n \times d_a} \), and \( E^v \in \mathbb{R}^{n \times d_v} \), and \( d_t, d_a, d_v \) represent dimensions of the hidden layer representations of the three modalities. The extraction process is described in Appendix C.1.

Then, we encode each feature representation within an unimodal local context. For text, we apply multi-head self-attention (Vaswani, 2017) to \( E^t \) to capture local contextual information, resulting in \( H^t \). For \( E^a \) and \( E^v \), we use a GRU-based network (Li et al., 2024b) to extract local context by leveraging the RNN structure’s capability to handle temporal features, which is expressed as:

\[
E'^m = LN(E^m + GRU(E^m)),
\]
\[
H^m = LN(E^m + E'^m + FFN(E'^m)),
\]

where \( H^m \in \mathbb{R}^{n \times d_m}, m \in \{ a, v \} \), \( LN \) denotes layer normalization, and \( FFN \) denotes a feedforward neural network.

After encoding the local context for each modality, we obtain the sequence representations \( H^t, H^a, H^v \) for text, audio, and video. We then apply three linear layers to map \( H^t, H^a, H^v \) to \( H'^t, H'^a, H'^v \) with the same dimension \( d_h \).

4.4 Graph Construction

To enable M^3HG to fuse multi-scale semantic information across modalities, we construct a heterogeneous graph that represents both inter- and intra-utterance connections, as well as cross-modal interactions. The structure of this heterogeneous graph can be denoted by \( \mathcal{G} = (\mathcal{V}, \mathcal{E}, \mathcal{R}) \), where \( \mathcal{V} \) is the node set consisting of all graph nodes \( v_i \), \( \mathcal{R} \) is the relation set consisting of all relations \( r_{ij} \) between any two nodes \( v_i \) and \( v_j \), and \( \mathcal{E} \) is the edge set consisting of all edges represented as \( (v_i, r_{ij}, v_j) \).
Figure 2: The framework of proposed M^3HG. It consists of three main components: unimodal feature extraction, graph construction and multi-scale semantic fusion, and emotion-cause classification.

Nodes. To explicitly model emotion and cause-related contexts in conversations, we model them as emotional context nodes \( N^e \) and causal context nodes \( N^c \), respectively. To enable \( \mathcal{G} \) to accurately perceive the conversation information, we model the whole conversation as a conversation node. Each utterance is represented by an utterance node. Both the utterance node and conversation node are designed as Super-Nodes containing these modalities, denoted as \( SN^u \) and \( SN^d \), since they contain three modal features. Therefore, \( \mathcal{G} \) contains four types of nodes: \( N^e, N^c, SN^u \) and \( SN^d \).

\( N^e \) and \( N^c \) are first initialized with textual sequence representations \( H^n \), then updated with contextual information from the other two modalities, which is described in Section 4.5. Each utterance Super-Node \( SN^u = \{N^t, N^a, N^v\} \) is initialized using \( H^n, H^{ta}, H^{tv} \). The conversation node \( SN^d = \{N^t_d, N^a_d, N^v_d\} \) is initialized by averaging \( H^n, H^{ta}, H^{tv} \) to capture global information.

Edges and Relations. There are five types of Super-Edges connecting the aforementioned Super-Nodes: same speaker (\( r_{ss} \)), different speaker (\( r_{ds} \)), global connection (\( r_{gc} \)), emotion connection (\( r_{ec} \)) and cause connection (\( r_{cc} \)). The same speaker edge connects the utterance Super-Nodes \( SN^u \) from the same speaker. Inspired by the work of Shen et al. (2021), we define the local context as \( K \) preceding utterances from the same speaker of \( SN^u \), where \( K \) is a hyper-parameter. The different speaker edge connects the utterance Super-Nodes within the local context from different speakers to \( SN^u \). The bidirectional global connection edge connects all the utterance Super-Nodes \( SN^u \)'s with the conversation Super-Node \( SN^d \), facilitating the propagation of global contextual information. The emotion connection edge and the cause connection edge connect \( SN^u \) with its corresponding emotional context node \( N^e \) and causal context node \( N^c \), respectively. They explicitly capture the emotion and cause context specific to each utterance.

M^3HG is the first MECTEC model capable of handling situations where cause utterances appear after emotion utterances, as each utterance is linked through the global connection node. The detailed experiments in Appendix E.2 further validate this capability. The pseudo-code of graph construction and a constructed graph for the conversation in Figure 1 are provided in Appendix C.2 and Appendix C.3, respectively. The graph construction process of M^3HG can be expressed as:

\[
\begin{align*}
\mathcal{G} &= (\mathcal{V}, \mathcal{E}, \mathcal{R}), \\
\mathcal{V} &= \{SN^u_i, N^e_i, N^c_i, SN^d\}_{1 \leq i \leq n}, \\
SN^u_i &= \{N^t_i, N^a_i, N^v_i\}, \\
SN^d &= \{N^t_d, N^a_d, N^v_d\}, \\
\mathcal{R} &= \{r_{ss}, r_{ds}, r_{gc}, r_{ec}, r_{cc}\}, \\
\mathcal{E} &= \{(v_i, r_{ij}, v_j)\}, v_i, v_j \in \mathcal{V}, r \in \mathcal{R},
\end{align*}
\]

where superscripts \( u, e, c, d \) denote node types, and \( m \) denotes three modalities. Based on the constructed graph \( \mathcal{G} \), the emotion and cause contexts are effectively modeled.
4.5 Multi-scale Semantic Information Fusion
Based on graph \( \mathcal{G} \), we designed a comprehensive approach to integrate semantic information across different modalities and scales. This mechanism is implemented in two levels: intra-utterance fusion which captures emotion and cause-related contexts within utterances, and inter-utterance fusion which propagates semantic information among utterances and conversation-level contexts. Both levels leverage HGAT (Wang et al., 2019) to propagate and fuse semantic information through various meta-paths (Wang et al., 2019) within \( \mathcal{G} \). This ensures thorough updates to node features by integrating multi-scale semantic information.

The meta-paths in \( \mathcal{G} \) are defined as:

\[
\Phi = \{ \phi(v_i, r_{ij}, v_j) \}, v_i, v_j \in \mathcal{V},
\]
\[
\phi(v_i, r_{ij}, v_j) = v_i \xleftrightarrow{r_{ij}} v_j, r_{ij} \in \mathcal{R},
\]

where \( \phi(v_i, r_{ij}, v_j) \) represents all paths that connect node \( v_i \) to node \( v_j \) via edge type \( r_{ij} \).

Intra-utterance-level Fusion. As shown in Figure 2, for each utterance Super-Node \( SN^u \), we perform intra-utterance-level fusion by integrating semantic information within the utterance. We define the meta-path \( \Phi_{intra} \) for intra-utterance-level semantic fusion for \( SN^u \) as:

\[
\Phi_{intra} = \{ \phi(N^{m_1}, N^{m_2}, r_{m_1,m_2}) \}
\cup \{ \phi(N^m, N^e, r_{m,e}) \}
\cup \{ \phi(N^m, N^c, r_{m,c}) \},
\]

where \( m_1, m_2, m \in \{t, a, v\} \), \( N^m \) represents the nodes of modality \( m \) within the \( SN^u \), and \( r_{m_1,m_2} \) denotes the edges connecting \( N^{m_1} \) and \( N^{m_2} \). \( r_{m,e} \) denotes edges connecting nodes \( N^m \) to the emotional context nodes \( N^e \), facilitating the aggregation of emotional contexts conveyed by different modalities within the utterance. Similarly, \( r_{m,c} \) represents the edges that connect \( N^m \) to the causal context nodes \( N^c \), enabling the aggregation of causal contexts. \( \Phi_{intra} \) effectively models the process of semantic information fusion in a single utterance.

Next, we incorporate node-level attention into \( \Phi_{intra} \). For each meta-path in \( \Phi_{intra} \) and nodes \( v_i \in \{N^m, N^e, N^c\} \), the importance of its neighbors \( \mathcal{N}_i \) in \( \Phi_{intra} \) is computed as:

\[
a_{ij}^\phi = \frac{\exp \left( \sigma \left( a_\phi^T \cdot [H_i' \| H_j'] \right) \right)}{\sum_{k \in \mathcal{N}_i^\phi} \exp \left( \sigma \left( a_\phi^T \cdot [H_i' \| H_k'] \right) \right)}, \quad \phi \in \Phi_{intra},
\]

where \( \sigma \) denotes the activation function, and \( a_\phi \) is the node-level attention vector of meta-path \( \phi \). The node representation of \( v_i \) based on meta-path \( \phi \) is obtained by:

\[
Z_i = \sigma \left( \sum_{j \in \mathcal{N}_i^\phi} a_{ij}^\phi \cdot H_j' \right).
\]

This process yields the contextual features \( Z_i \in \mathbb{R}^{1 \times d_h} \) for nodes \( v_i \) under the intra-utterance-level meta-paths \( \Phi_{intra} \).

Inter-utterance-level Fusion. As illustrated in Figure 2, for any two utterance Super-Nodes \( SN^u_i \) and \( SN^u_j \) in \( \mathcal{G} \), along with the conversation Super-Node \( SN^d \), we perform inter-utterance-level fusion by connecting \( SN^u_i \) and \( SN^u_j \) to \( SN^d \), thereby integrating contextual information across utterances. We define meta-paths \( \Phi_{inter} \) for inter-utterance-level fusion between \( SN^u \) and \( SN^d \):

\[
\Phi_{inter} = \{ \phi(N^{m_1}_i, N^{m_2}_j, r_{m_1,m_2}) \}
\cup \{ \phi(N^m_i, N^d, r_{d,m}) \}
\cup \{ \phi(N^m_j, N^d, r_{d,m}) \},
\]

where \( m_1, m_2, m \in \{t, a, v\} \), \( N^{m_1}_i \) and \( N^{m_2}_j \) represent the nodes of modality \( m \) inside \( SN^u_i \) and \( SN^u_j \), respectively, \( r_{m_1,m_2} \) denotes the edges connecting \( N^{m_1}_i \) and \( N^{m_2}_j \), and \( r_{d,m} \) represents the edges connecting \( SN^u \)s to \( SN^d \) in modality \( m \).

The utterance information from each modality can be passed to \( SN^d \) though \( \Phi_{inter} \), which accomplishes inter-utterance-level fusion between utterances. As a result, \( SN^d \) comprehensively integrates information across all three modalities. The meta-path set \( \Phi_{inter} \) models multimodal connections between utterances, enabling conversation information aggregated in \( \mathcal{G} \).

Similar to Eq. 6 and Eq. 7, the contextual representations of \( SN^u \) and \( SN^d \) are obtained under the meta-path \( \Phi_{inter} \) by the node-level attention block.

After performing multi-scale semantic fusion with \( \Phi_{intra} \) and \( \Phi_{inter} \), we apply the semantic attention mechanism (Wang et al., 2019) to each node embedding \( Z_i \), integrating multi-scale semantic information from all three modalities. Following (Chen et al., 2023), each fusion iteration is followed by a position-wise feed-forward network (PFFN) layer, which updates node features through a nonlinear transformation. The emotional context node representation \( Z_i^e \) and the causal context node feature representation \( Z_i^c \) can be obtained at the end of iterations of the multi-scale semantic fusion and PFFN layers.
4.6 Emotion-cause classification
For each utterance \( U_i \), its \( Z_i^e \) and \( Z_i^c \) are fed into the emotion-specific Multi-Layer Perceptron (Emotion MLP) and the cause-specific Multi-Layer Perceptron (Cause MLP) to predict its emotion category \( \hat{y}_i^e \) and the cause indicator \( \hat{y}_i^c \) which indicates whether \( U_i \) can be a cause utterance. For each utterance pair \( U_i \) and \( U_j \), we compute a relative position encoding \( RPE_{ij} \) to capture the positional relationship between \( U_i \) and \( U_j \). We utilize the RBF kernel function (Wei et al., 2020) to compute \( RPE_{ij} \), which captures the relative positional relationships between utterances through a nonlinear relation. \( Z_j^e, Z_i^c \) and \( RPE_{ij} \) are then concatenated and fed into a new MLP to determine whether \( U_i \) is the cause utterance of \( U_j \):

\[
\hat{y}_{ij}^{ec} = \sigma(MLP(Z_j^e||Z_i^c||RPE_{ij})).
\]

\( \hat{y}_{ij}^{ec} \) represent the binary classification logits indicating whether \( U_i \) is the cause of \( U_j \). Based on \( \hat{y}_{ij}^{ec} \), we can determine whether \( U_j, U_i \) and \( \hat{y}_j^e \) can form a true utter-cause-emotion triplet.

4.7 Training
We use Focal loss (Ross and Dollár, 2017) to cope with category imbalance in emotion-cause classification. Specifically, the loss of both emotion prediction and cause utterance prediction and the emotion-cause pair prediction, can be expressed as:

\[
\mathcal{L}^\beta = -\frac{1}{N^\beta} \sum_{i=1}^{N^\beta} \alpha^\beta (1-\hat{y}_i^\beta)^\gamma \log(\hat{y}_i^\beta), \beta \in \{e, c, ec\}
\]

where \( \beta \) represents the task type, \( N^\beta \) denotes the corresponding sample number of \( \beta \), \( \alpha^\beta \) is the category balancing factor, and \( \gamma \) denotes the Focal loss modulation parameter. These three training losses are optimized jointly during the training process.

5 EXPERIMENTS

5.1 Experimental Settings
We conduct extensive experiments on two MECTEC benchmark datasets, i.e., ECF (Wang et al., 2022) and MECAD, which both contain data of three modalities: text, audio, and video. Similar to (Wang et al., 2022), we evaluate the model’s overall performance using the F1 score. The F1 score is computed for utter-cause-emotion triplets within each emotion category separately. Then the weighted average F1 score is calculated across all six emotion categories which is referred to 6 Avg. In addition, as in (Wang et al., 2023), considering the data imbalance among different emotion categories, we also report the weighted average F1 scores for the four main emotion categories except Disgust and Fear, which is referred to 4 Avg. The implementation details of the experiment are given in Appendix D.

5.2 Baselines
Due to the limited research on the MECTEC task, representative approaches in related fields of Emotion Cause Pair Extraction (ECPE) and Emotion Cause Pair Extraction in Conversations (ECPEC) are considered. The ECPE and ECPEC tasks aim to extract emotion-cause pairs from plain texts and conversations, respectively.

We compare our model with seven baselines: 1) MC-ECPE-2steps (Wang et al., 2022) is a two-step MECTEC architecture, which first extracts emotion utterances and cause utterances separately, and then performs pairing and filtering to identify emotion-cause pairs. 2) HiLo (Li et al., 2024a) is one of the SOTA approaches for the MECTEC task, which fully utilizes conversion information through a labeling constraint mechanism. 3) ECPE-2D (Ding et al., 2020a) is an E2E framework for ECPE that uses 2D-Transformer to model the interactions of emotion-cause pairs. 4) RankCP (Wei et al., 2020) is a GAT-based approach for ECPE to extract emotion-cause pairs by ranking. 5) UECA-Prompt (Zheng et al., 2022) is one of the SOTA methods for ECPE, which decomposes the task into multiple objectives and converts them into sub-prompts. 6) SHARK (Wang et al., 2023) is the SOTA method for ECPEC that incorporates commonsense into GATs to improve the model’s semantic understanding of emotions and causes. 7) GPT-4o is one of the most powerful large language models (LLMs) for open-domain conversations. Details of prompts are provided in Appendix F.

5.3 Experimental Results
Table 2 shows the experimental results of M^3HG and seven baseline models evaluated on the ECF dataset and the MECAD dataset. Our model demonstrates an excellent performance both on the ECF dataset and the MECAD dataset.

Results on the ECF dataset. First of all, as shown in Table 2, among all the baseline models, the E2E approaches such as SHARK and HiLo deliver the
<table>
  <tr>
    <th rowspan="2">Dataset</th>
    <th rowspan="2">Method</th>
    <th rowspan="2">Modality</th>
    <th colspan="6">6 Avg.</th>
    <th colspan="2">4 Avg.</th>
  </tr>
  <tr>
    <th>Anger</th>
    <th>Disgust</th>
    <th>Fear</th>
    <th>Joy</th>
    <th>Sadness</th>
    <th>Surprise</th>
    <th>6 Avg.</th>
    <th>4 Avg.</th>
  </tr>
  <tr>
    <td rowspan="5">ECF</td>
    <td>Pipeline</td>
    <td>MC-ECPE-2steps<sup>△</sup></td>
    <td>T, A, V</td>
    <td>24.39</td>
    <td>0.00</td>
    <td>0.71</td>
    <td>38.84</td>
    <td>21.60</td>
    <td>40.24</td>
    <td>29.32</td>
    <td>31.92</td>
  </tr>
  <tr>
    <td></td>
    <td>ECPE-2D<sup>△</sup></td>
    <td>T</td>
    <td>25.13</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>41.25</td>
    <td>21.62</td>
    <td>43.24</td>
    <td>30.80</td>
    <td>33.55</td>
  </tr>
  <tr>
    <td></td>
    <td>RankCP</td>
    <td>T</td>
    <td>28.29</td>
    <td>12.03</td>
    <td>3.52</td>
    <td>38.69</td>
    <td>22.17</td>
    <td>37.67</td>
    <td>30.58</td>
    <td>32.48</td>
  </tr>
  <tr>
    <td>E2E</td>
    <td>UECA-Prompt<sup>△</sup></td>
    <td>T</td>
    <td>27.37</td>
    <td>12.85</td>
    <td>7.91</td>
    <td>37.96</td>
    <td>22.51</td>
    <td>39.53</td>
    <td>30.75</td>
    <td>32.49</td>
  </tr>
  <tr>
    <td></td>
    <td>SHARK<sup>△</sup></td>
    <td>T</td>
    <td>28.65</td>
    <td>10.42</td>
    <td>5.33</td>
    <td>40.41</td>
    <td>25.35</td>
    <td>40.45</td>
    <td>32.24</td>
    <td>34.33</td>
  </tr>
  <tr>
    <td></td>
    <td>HiLo</td>
    <td>T, A, V</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>33.04</td>
    <td>35.81</td>
  </tr>
  <tr>
    <td rowspan="3">LLMs</td>
    <td>GPT-4o (5-shots)</td>
    <td>T</td>
    <td>28.49</td>
    <td>17.76</td>
    <td>12.35</td>
    <td>31.11</td>
    <td>27.27</td>
    <td>33.89</td>
    <td>29.13</td>
    <td>30.30</td>
  </tr>
  <tr>
    <td></td>
    <td>T</td>
    <td>34.47</td>
    <td>18.17</td>
    <td>12.72</td>
    <td>43.28</td>
    <td>32.22</td>
    <td>45.82</td>
    <td>37.46</td>
    <td>39.95</td>
  </tr>
  <tr>
    <td>M<sup>3</sup>HG (ours)</td>
    <td>T, A</td>
    <td>35.57</td>
    <td>18.71</td>
    <td>17.07</td>
    <td>47.73</td>
    <td>30.97</td>
    <td>46.72</td>
    <td>39.10</td>
    <td>40.97</td>
  </tr>
  <tr>
    <td></td>
    <td></td>
    <td>T, V</td>
    <td><b>37.38</b></td>
    <td><b>18.87</b></td>
    <td><b>17.07</b></td>
    <td><b>48.70</b></td>
    <td><b>32.10</b></td>
    <td><b>48.16</b></td>
    <td><b>39.37</b></td>
    <td><b>41.77</b></td>
  </tr>
  <tr>
    <td></td>
    <td></td>
    <td>T, A, V</td>
    <td><b>36.08</b></td>
    <td><b>23.33</b></td>
    <td><b>8.88</b></td>
    <td><b>49.03</b></td>
    <td><b>32.41</b></td>
    <td><b>47.46</b></td>
    <td><b>40.07</b></td>
    <td><b>41.96</b></td>
  </tr>
  <tr>
    <td rowspan="5">MECAD</td>
    <td>Pipeline</td>
    <td>MC-ECPE-2steps</td>
    <td>T, A, V</td>
    <td>28.43</td>
    <td>0.00</td>
    <td>0.23</td>
    <td>22.45</td>
    <td>27.67</td>
    <td>45.14</td>
    <td>22.01</td>
    <td>24.83</td>
  </tr>
  <tr>
    <td></td>
    <td>ECPE-2D</td>
    <td>T</td>
    <td>28.12</td>
    <td>0.00</td>
    <td>0.56</td>
    <td>24.30</td>
    <td>28.01</td>
    <td>35.87</td>
    <td>25.32</td>
    <td>28.54</td>
  </tr>
  <tr>
    <td></td>
    <td>RankCP</td>
    <td>T</td>
    <td>29.79</td>
    <td>12.50</td>
    <td>3.06</td>
    <td>21.79</td>
    <td>29.31</td>
    <td>32.36</td>
    <td>26.29</td>
    <td>28.32</td>
  </tr>
  <tr>
    <td>E2E</td>
    <td>UECA-Prompt</td>
    <td>T</td>
    <td>38.52</td>
    <td>12.12</td>
    <td>5.22</td>
    <td>29.67</td>
    <td>34.17</td>
    <td>45.57</td>
    <td>32.97</td>
    <td>37.47</td>
  </tr>
  <tr>
    <td></td>
    <td>SHARK</td>
    <td>T</td>
    <td>30.22</td>
    <td>10.16</td>
    <td>4.10</td>
    <td>25.84</td>
    <td>30.21</td>
    <td>34.59</td>
    <td>27.58</td>
    <td>29.99</td>
  </tr>
  <tr>
    <td></td>
    <td>HiLo</td>
    <td>T, A, V</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
  </tr>
  <tr>
    <td rowspan="3">LLMs</td>
    <td>GPT-4o (5-shots)</td>
    <td>T</td>
    <td>36.65</td>
    <td>20.08</td>
    <td>8.45</td>
    <td>24.52</td>
    <td>17.89</td>
    <td>39.77</td>
    <td>27.16</td>
    <td>28.42</td>
  </tr>
  <tr>
    <td></td>
    <td>T, A</td>
    <td>35.85</td>
    <td>18.05</td>
    <td>15.38</td>
    <td>25.95</td>
    <td>29.13</td>
    <td>42.11</td>
    <td>30.81</td>
    <td>32.55</td>
  </tr>
  <tr>
    <td>M<sup>3</sup>HG (ours)</td>
    <td>T, V</td>
    <td>37.29</td>
    <td>21.03</td>
    <td>15.89</td>
    <td>27.15</td>
    <td>30.34</td>
    <td>42.78</td>
    <td>32.16</td>
    <td>35.73</td>
  </tr>
  <tr>
    <td></td>
    <td></td>
    <td>T, A, V</td>
    <td><b>36.91</b></td>
    <td><b>20.48</b></td>
    <td><b>16.91</b></td>
    <td><b>25.47</b></td>
    <td><b>30.96</b></td>
    <td><b>43.14</b></td>
    <td><b>31.95</b></td>
    <td><b>33.52</b></td>
  </tr>
  <tr>
    <td></td>
    <td></td>
    <td>T, A, V</td>
    <td><b>38.34</b></td>
    <td><b>21.89</b></td>
    <td><b>8.79</b></td>
    <td><b>28.10</b></td>
    <td><b>31.17</b></td>
    <td><b>43.29</b></td>
    <td><b>32.82</b></td>
    <td><b>34.59</b></td>
  </tr>
</table>

Table 2: Performance comparison of different methods on the MECTEC task. △ denotes the results are from (Wang et al., 2023). * denotes the results are from the original paper (Wang et al., 2023; Li et al., 2024a). The best results and the second best results are in bold and underlined, respectively. Since HiLo (Li et al., 2024a) is not publicly available, we only report the results of HiLo on the ECF dataset.

best performance, indicating that the E2E framework is more effective compared to the two-step pipeline frameworks. In contrast, M<sup>3</sup>HG adopting three modalities outperforms the SOTA E2E model HiLo, with 21.28% and 17.17% improvement in 6 Avg and 4 Avg scores, respectively. We attribute this improvement to M<sup>3</sup>HG’s ability to effectively extract semantic information at inter-utterance and intra-utterance levels, which enables the model to accurately pair emotion utterances and cause utterances. Specifically, in two challenging emotion categories which have limited training samples, i.e. Disgust and Fear, M<sup>3</sup>HG also exhibits high performances. For example, compared to GPT-4o, which achieved the second highest F1 scores in the Disgust and Fear categories, M<sup>3</sup>HG shows improvements of 31.36% and 58.46%, respectively.

When only incorporating the text modality, the 6 Avg and the 4 Avg scores of M<sup>3</sup>HG are 37.46 and 39.95. When incorporating audio and video with the text modality separately, the performance of M<sup>3</sup>HG is improved to 39.10, 38.90 of 6 Avg scores and 40.97, 40.72 of 4 Avg scores. When incorporating all three modalities, M<sup>3</sup>HG achieves the highest performance with 40.07 of 6 Avg scores and 41.96 of 4 Avg scores. Meanwhile, it can be observed that M<sup>3</sup>HG outperforms all the baseline models even when only using the text modality, demonstrating its superiority on the ECF dataset.

Results on the MECAD dataset. As shown in Table 2, M<sup>3</sup>HG also achieves the highest results on the MECAD dataset. Compared to the second best model SHARK, M<sup>3</sup>HG adopting three modalities achieves the improvement of 19% on the 6 Avg scores and 15.34% on the 4 Avg score. Furthermore, despite GPT-4o’s superior semantic comprehension abilities, its performance on the MECAD dataset remains suboptimal, with its 6 Avg score and 4 Avg score of 27.16 and 28.42. Therefore, the few-shot-based LLM approach still struggles to effectively handle the MECTEC task. As shown in Table 2, M<sup>3</sup>HG exhibits a universal highest performance on the MECAD dataset, demonstrating the superiority and robustness of M<sup>3</sup>HG when dealing with multi-conversation scenarios.

More detailed experimental results and the ablation study on M<sup>3</sup>HG are presented in Appendix E.

6 Conclusion

In this work, we propose the first multimodal and multi-scenario Chinese emotion-cause analysis dataset, MECAD, for MECTEC and related emotion cause analysis tasks. Compared to ECF, the only existing dataset for multimodal emotion-cause analysis, MECAD offers more diverse conversation scenarios. It helps to enhance the generalizability and applicability of MECTEC models in complex social media environments. Moreover, MECAD is a valuable resource for cross-cultural emotion analysis and recognition. Furthermore, we propose
a generalized MECTEC framework named M^3HG, which deeply extracts emotional and causal contexts, while effectively integrating semantic information across multiple granular levels. Extensive experiments on the ECF dataset and the MECAAD dataset demonstrate the superiority of our method compared to the existing state-of-the-art methods.

Limitations

There are also some potential limitations in this work. First, the process of emotional and causal context extraction does not integrate external knowledge, which limits the model’s accuracy for emotion prediction and cause prediction. In the future, we plan to integrate external knowledge into our model and leverage the advanced semantic extraction capabilities of current LLM technology to facilitate deeper and more precise emotion cause analysis. Second, M^3HG cannot handle excessively long conversations, as its input length is constrained by the language model used. Furthermore, M^3HG may suffer from error propagation in the multimodal fusion process when emotion labels have uneven information across modalities. This imbalance can lead to inaccurate predictions, especially when modalities conflict. This challenge is common in current multimodal models for emotion-cause analysis and suggests an area for future improvement.

Ethical Considerations

We did not use real-world conversations in our data collection because such conversations may violate the privacy of the speaker. The effect of recruiting actors to play the roles is the same as in the TV series, but the scenes are not as diverse as in the TV series. Therefore, we use TV series as the data source. To further protect privacy, all data annotations were anonymized and de-identified, ensuring that our data collection adheres to ethical standards.

References

Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. 2020. wav2vec 2.0: A framework for self-supervised learning of speech representations. Advances in neural information processing systems, 33:12449–12460.

Tiantian Chen, Ying Shen, Xuri Chen, Lin Zhang, and Shengjie Zhao. 2023. Mpeg: A multi-perspective enhanced graph attention network for causal emotion entailment in conversations. IEEE Transactions on Affective Computing.

Jacob Cohen. 1960. A coefficient of agreement for nominal scales. Educational and psychological measurement, 20(1):37–46.

Zixiang Ding, Rui Xia, and Jianfei Yu. 2020a. Ecpe-2d: Emotion-cause pair extraction based on joint two-dimensional representation, interaction and prediction. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3161–3170.

Zixiang Ding, Rui Xia, and Jianfei Yu. 2020b. End-to-end emotion-cause pair extraction based on sliding window multi-label learning. In Proceedings of the 2020 conference on empirical methods in natural language processing (EMNLP), pages 3574–3583.

René Dirven. 1997. Emotions as cause and the cause of emotions. The language of emotions: Conceptualization, expression, and theoretical foundation, pages 55–83.

Paul Ekman. 1992. An argument for basic emotions. Cognition & emotion, 6(3-4):169–200.

Jia-Chen Gu, Tianda Li, Quan Liu, Zhen-Hua Ling, Zhiming Su, Si Wei, and Xiaodan Zhu. 2020. Speaker-aware bert for multi-turn response selection in retrieval-based chatbots. In Proceedings of the 29th ACM International Conference on Information & Knowledge Management, pages 2041–2044.

Xiaojie Gu, Renze Lou, Lin Sun, and Shangxin Li. 2023. Page: A position-aware graph-based model for emotion cause entailment in conversation. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1–5. IEEE.

Guimin Hu, Zhihong Zhu, Daniel Hershcovich, Hasti Seifi, and Jiayuan Xie. 2024. Unimeec: Towards unified multimodal emotion recognition and emotion cause. arXiv preprint arXiv:2404.00403.

Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. 2017. Densely connected convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4700–4708.

JR Landis. 1977. The measurement of observer agreement for categorical data. Biometrics.

Bobo Li, Hao Fei, Fei Li, Tat-seng Chua, and Donghong Ji. 2024a. Multimodal emotion-cause pair extraction with holistic interaction and label constraint. ACM Transactions on Multimedia Computing, Communications and Applications.

Jiang Li, Xiaoping Wang, Yingjian Liu, and Zhigang Zeng. 2024b. Cfn-esa: A cross-modal fusion network with emotion-shift awareness for dialogue emotion recognition. IEEE Transactions on Affective Computing.
Jiangnan Li, Fandong Meng, Zheng Lin, Rui Liu, Peng Fu, Yanan Cao, Weiping Wang, and Jie Zhou. 2022a. Neutral utterances are also causes: Enhancing conversational causal emotion entailment with social commonsense knowledge. arXiv preprint arXiv:2205.00759.

Wei Li, Yang Li, Vlad Pandelea, Mengshi Ge, Luyao Zhu, and Erik Cambria. 2022b. Ecpec: Emotion-cause pair extraction in conversations. IEEE Transactions on Affective Computing, 14(3):1754–1765.

I Loshchilov. 2017. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101.

Mary L McHugh. 2012. Interrater reliability: the kappa statistic. Biochemia medica, 22(3):276–282.

Soujanya Poria, Navonil Majumder, Devamanyu Hazarika, Deepanway Ghosal, Rishabh Bhardwaj, Samson Yu Bai Jian, Pengfei Hong, Romila Ghosh, Abhinaba Roy, Niyati Chhaya, et al. 2021. Recognizing emotion cause in conversations. Cognitive Computation, 13:1317–1332.

T-YLPG Ross and GKHP Dollár. 2017. Focal loss for dense object detection. In proceedings of the IEEE conference on computer vision and pattern recognition, pages 2980–2988.

Weizhou Shen, Siyue Wu, Yunyi Yang, and Xiaojun Quan. 2021. Directed acyclic graph network for conversational emotion recognition. arXiv preprint arXiv:2105.12907.

Mohammad Soleymani, David Garcia, Brendan Jou, Björn Schuller, Shih-Fu Chang, and Maja Pantic. 2017. A survey of multimodal sentiment analysis. Image and Vision Computing, 65:3–14.

Andrew Steptoe and Lena Brydon. 2009. Emotional triggering of cardiac events. Neuroscience & Biobehavioral Reviews, 33(2):63–70.

A Vaswani. 2017. Attention is all you need. Advances in Neural Information Processing Systems.

Fanfan Wang, Zixiang Ding, Rui Xia, Zhaoyu Li, and Jianfei Yu. 2022. Multimodal emotion-cause pair extraction in conversations. IEEE Transactions on Affective Computing, 14(3):1832–1844.

Fanfan Wang, Jianfei Yu, and Rui Xia. 2023. Generative emotion cause triplet extraction in conversations with commonsense knowledge. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 3952–3963.

Xiao Wang, Houye Ji, Chuan Shi, Bai Wang, Yanfang Ye, Peng Cui, and Philip S Yu. 2019. Heterogeneous graph attention network. In The world wide web conference, pages 2022–2032.

Penghui Wei, Jiahao Zhao, and Wenji Mao. 2020. Effective inter-clause modeling for end-to-end emotion-cause pair extraction. In Proceedings of the 58th annual meeting of the association for computational linguistics, pages 3171–3181.

Bernard Weiner. 1985. An attributional theory of achievement motivation and emotion. Psychological review, 92(4):548.

Duzhen Zhang, Zhen Yang, Fandong Meng, Xiuyi Chen, and Jie Zhou. 2022. Tsam: A two-stream attention model for causal emotion entailment. arXiv preprint arXiv:2203.00819.

Jinming Zhao, Tenggan Zhang, Jingwen Hu, Yuchen Liu, Qin Jin, Xinchao Wang, and Haizhou Li. 2022. M3ed: Multi-modal multi-scene multi-label emotional dialogue database. arXiv preprint arXiv:2205.10237.

Weixiang Zhao, Yanyan Zhao, Zhuojun Li, and Bing Qin. 2023. Knowledge-bridged causal interaction network for causal emotion entailment. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 14020–14028.

Li Zheng, Donghong Ji, Fei Li, Hao Fei, Shengqiong Wu, Jingye Li, Bobo Li, and Chong Teng. 2023. Ecqed: emotion-cause quadruple extraction in dialogs. arXiv preprint arXiv:2306.03969.

Xiaopeng Zheng, Zhiyue Liu, Zizhen Zhang, Zhaoyang Wang, and Jiahai Wang. 2022. Ueac-prompt: Universal prompt for emotion cause analysis. In Proceedings of the 29th International Conference on Computational Linguistics, pages 7031–7041.

A Dataset Statistics and Analysis of MECAD

To ensure the annotation quality of MECAD, we calculated Cohen’s kappa (Cohen, 1960) scores for every co-annotated data between two annotators, as shown in Figure 4. The Cohen’s kappa (Cohen, 1960) scores across all annotators are consistently around 0.6, indicating a good level of annotation consistency.

After the labeling was completed, we computed Cohen’s kappa scores separately for data that were not co-labeled between the two labelers, as shown in Figure 4. Table 3 lists some statistics of the MECAD dataset. The dataset contains a total of 989 conversations, 10,516 utterances, and 8,077 emotion cause pairs from 56 different TV series, which ensures the size and diversity of the dataset. Similar to M3ED (Zhao et al., 2022), we used TV-independent data segmentation to ensure the ability to validate model robustness as a benchmark dataset. The average number of utterances and the average length of an utterance of a conversation are similar in the training, validation, and test sets. At the same time, we can find that the average relative positions of the emotion cause pairs are all around
Table 3: MECAD statistics. *Rel pos of ec pairs* denotes the relative position between emotion utterances and cause utterances in emotion-cause pairs.

<table>
  <tr>
    <th>Statistic</th>
    <th>Train</th>
    <th>Val</th>
    <th>Test</th>
    <th>Total</th>
  </tr>
  <tr>
    <td># TV series</td>
    <td>38</td>
    <td>7</td>
    <td>11</td>
    <td>56</td>
  </tr>
  <tr>
    <td># conversations</td>
    <td>684</td>
    <td>126</td>
    <td>179</td>
    <td>989</td>
  </tr>
  <tr>
    <td># uttrs</td>
    <td>7,516</td>
    <td>1,668</td>
    <td>1,832</td>
    <td>10,516</td>
  </tr>
  <tr>
    <td># ec pairs</td>
    <td>421</td>
    <td>87</td>
    <td>110</td>
    <td>618</td>
  </tr>
  <tr>
    <td>Avg. uttrs/conversation</td>
    <td>10.99</td>
    <td>9.27</td>
    <td>10.24</td>
    <td>10.63</td>
  </tr>
  <tr>
    <td>Avg. uttr length</td>
    <td>18.30</td>
    <td>18.80</td>
    <td>18.15</td>
    <td>18.33</td>
  </tr>
  <tr>
    <td>Avg. rel pos of ec pairs</td>
    <td>0.72</td>
    <td>0.73</td>
    <td>0.55</td>
    <td>0.69</td>
  </tr>
  <tr>
    <td>Min. rel pos of ec pairs</td>
    <td>-1.5</td>
    <td>-1.5</td>
    <td>0.5</td>
    <td>1.5</td>
  </tr>
  <tr>
    <td>Min. rel pos of cause</td>
    <td>-14</td>
    <td>-5</td>
    <td>-9</td>
    <td>-14</td>
  </tr>
  <tr>
    <td>Emotion uttrs with cause</td>
    <td>4,526</td>
    <td>743</td>
    <td>1,062</td>
    <td>6,331</td>
  </tr>
  <tr>
    <td>ec pairs</td>
    <td>5,788</td>
    <td>977</td>
    <td>1,312</td>
    <td>8,077</td>
  </tr>
</table>

Figure 3: The interface of the developed online multi-modal conversation emotion cause annotation toolkit.

![Screenshot of the online multi-modal conversation emotion cause annotation toolkit interface](page_120_120_600_400.png)

Figure 4: Schematic representation of Cohen’s Kappa scores for the common labeled portion between every two annotators. A blank section indicates that there is no common annotation data between two annotators.

![Schematic representation of Cohen’s Kappa scores matrix](page_120_600_600_400.png)

Figure 5: Percentage of five cause types in the MECAD dataset and percentage of modal basis for emotion cause inferences.

![Pie charts showing cause type percentages and modal basis percentages](page_1040_120_600_400.png)

1, indicating that most of the emotions in the conversation are caused by the previous utterance.

We referred how the ECF (Wang et al., 2022) dataset categorizes the emotion causes and added a new category called *Self Reflection*, which differs from the remaining four categories by indicating that emotions may be triggered by an individual’s introspection or self-reflection, such as recollections of past events or worries about the future. As shown in Figure 5, the event type is the cause type with the largest share, indicating that most of the emotions are caused by specific events in the conversation. Notably, 36.1% of the causes of emotion in our dataset are reflected in both audio and video modalities, which exemplifies the need for multimodal scene studies.

B The Annotation Toolkit of MECAD

To enhance annotation efficiency and accuracy, we developed an online multimodal conversation emotion cause annotation tool based on web technology\textsuperscript{1}. As illustrated in Figure 3, the toolkit’s homepage presents a list of conversations assigned to the corresponding annotators, along with the progress of their annotations. The conversation an-

\textsuperscript{1}The annotation tool has been open-sourced at https://github.com/redifinition/MECAD-MECTEC
notation page displays speaker information, video segments, corresponding scripts, and configurable annotation items, enabling annotators to quickly and efficiently complete their annotations.

With flexible and modifiable web pages, researchers can utilize our annotation tools in dataset constructions for further multimodal sentiment analysis studies.

Algorithm 1 Super-Node-based Graph Construction for a Conversation

1: Input: the conversation \( \{S_1 : U_1, S_2 : U_2, ..., S_N : U_N\} \), speaker identity \( p(\cdot) \) satisfies \( p(U_i) = S_i \), the direct context window \( K \)
2: Output: Super-Node-based M^3HG: \( G = (\mathcal{V}, \mathcal{E}, \mathcal{R}) \)
3: \( \mathcal{V} \leftarrow \{(SN_u^1, N_e^1, N_c^1), ..., (SN_u^N, N_e^N, N_c^N), SN_d^1\} \)
4: \( \mathcal{E} \leftarrow \emptyset \)
5: \( \mathcal{R} \leftarrow \{r_{ss}, r_{ds}, r_{ge}, r_{ec}, r_{cc}\} \)
6: for \( i \in \{2, 3, ..., N\} \) do
7:   \( c \leftarrow 0, w \leftarrow i - 1 \)
8:   while \( w > 0 \) and \( c < K \) do
9:     if \( p(U_w) = p(U_i) \) then
10:        \( \mathcal{E} \leftarrow \mathcal{E} \cup \{(SN_u^w, SN_u^i, r_{ss})\} \)
11:        \( c \leftarrow c + 1 \)
12:     else
13:        \( \mathcal{E} \leftarrow \mathcal{E} \cup \{(SN_u^w, SN_u^i, r_{ds})\} \)
14:     end if
15:     \( w \leftarrow w - 1 \)
16:   end while
17: end for
18: for \( i \in \{1, 2, ..., N\} \) do
19:   \( \mathcal{E} \leftarrow \mathcal{E} \cup \{(SN_u^i, N_e^i, r_{ec})\} \)
20:   \( \mathcal{E} \leftarrow \mathcal{E} \cup \{(SN_u^i, N_c^i, r_{cc})\} \)
21:   \( \mathcal{E} \leftarrow \mathcal{E} \cup \{(SN_u^i, SN_d^1, r_{ge})\} \)
22: end for
23: return \( G = (\mathcal{V}, \mathcal{E}, \mathcal{R}) \)

C Design Details of M^3HG

C.1 Multimodal Feature Extracting

Text : We splice all the textual modal utterances and the corresponding speakers in the conversation and add a number of special tokens to get the textual modal input sequence: \( X^t = \{<cls_token> S_1 : U_1^t, <sep_token>, ..., <cls_token> S_n : U_n^t, <sep_token>\} \), where \( <cls_token> \) and \( <sep_token> \) denote the classification token and the separation token used in the pre-trained language model (PLM), respectively. To allow conversations that exceed the maximum input sequence length of the PLM to retain as much contextual information as possible when they are fed into the PLM, we sequentially truncate the last tokens of the maximum-length utterances of the conversation during preprocessing until the maximum sequence length requirement of the PLM is met. The input sequence \( X^t \) is then fed into the PLM to obtain a sequential representation of the entire conversation:

\[
I^t = PLM(X^t),
\]

where \( I^t \in \mathbb{R}^{L \times d_t} \), \( L \) is the length of the input sequence and \( d_t \) is the hidden dimension of the PLM. To obtain the sequence representation of each utterance, we make a weighted average of the sequence representations of the tokens of each conversation in \( I^t \) to obtain the sequence representation of each utterance \( E^t \in \mathbb{R}^{N \times d_t} \), where \( N \) denotes the number of utterances of that conversation. We selected Speaker-Aware RoBERTa (SA-RoBERTa) (Gu et al., 2020) as the PLM.

Audio : After resampling the audio to 16khz, we input it into an audio feature extraction model (AFE) to get a sequential representation of the audio modality of the conversation:

\[
E^a = AFE(X^a),
\]

where \( E^a \in \mathbb{R}^{n \times d_a} \), and \( d_a \) is the hidden layer dimension of the audio feature extraction model. We choose Wav2Vec2.0 (Baevski et al., 2020) as the audio feature extraction model.

Video: We first sample the video at equal intervals as a sequence of images over several frames to obtain the input sequence \( X^v, X^v \in \mathbb{R}^{F \times d_f \times d_f} \) of the video modality, where \( F \) is the number of sampled frames and \( d_f \) is the size of the picture. The image sequences are then fed into the video feature extraction model (VFE) to get a sequence representation of the video modalities:

\[
E^v = VFE(X^v),
\]

where \( E^v \in \mathbb{R}^{n \times d_v} \) and \( d_v \) is the hidden layer dimension of the video feature extraction model. We select the pre-trained DenseNet (Huang et al., 2017) as the audio feature extraction model.

C.2 Pseudo-code of Graph Construction

The pseudo-code of the graph construction process is shown in Algorithm C.2.

C.3 An Example of the Graph construction

If \( K = 1 \), the graph constructed for the conversation in Figure 1 is shown in Figure 6.
Table 4: Performance comparison of different methods for conversations with varying numbers of utterances. The best results and the second best results are in bold and underlined, respectively.

<table>
  <tr>
    <th rowspan="2">Method</th>
    <th colspan="4">ECF</th>
    <th colspan="4">MECAD</th>
  </tr>
  <tr>
    <th colspan="2">num_utt \leq 10</th>
    <th colspan="2">num_utt > 10</th>
    <th colspan="2">num_utt \leq 10</th>
    <th colspan="2">num_utt > 10</th>
  </tr>
  <tr>
    <th></th>
    <th>6 Avg.</th>
    <th>4 Avg.</th>
    <th>6 Avg.</th>
    <th>4 Avg.</th>
    <th>6 Avg.</th>
    <th>4 Avg.</th>
    <th>6 Avg.</th>
    <th>4 Avg.</th>
  </tr>
  <tr>
    <td>RankCP</td>
    <td>31.50</td>
    <td>33.29</td>
    <td>29.34</td>
    <td>31.88</td>
    <td>27.19</td>
    <td>29.23</td>
    <td>25.11</td>
    <td>27.13</td>
  </tr>
  <tr>
    <td>SHARK</td>
    <td>33.68</td>
    <td>35.57</td>
    <td>31.49</td>
    <td>33.17</td>
    <td>28.32</td>
    <td>30.75</td>
    <td>27.01</td>
    <td>29.41</td>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td>30.08</td>
    <td>31.56</td>
    <td>28.42</td>
    <td>29.36</td>
    <td>26.34</td>
    <td>27.82</td>
    <td>27.79</td>
    <td>28.88</td>
  </tr>
  <tr>
    <td>M^3HG (T)</td>
    <td>39.18</td>
    <td>41.25</td>
    <td>36.18</td>
    <td>38.98</td>
    <td>31.95</td>
    <td>33.40</td>
    <td>29.94</td>
    <td>31.90</td>
  </tr>
  <tr>
    <td>M^3HG (T, A, V)</td>
    <td><b>41.95</b></td>
    <td><b>40.42</b></td>
    <td><b>38.67</b></td>
    <td><b>41.09</b></td>
    <td><b>33.76</b></td>
    <td><b>35.21</b></td>
    <td><b>32.10</b></td>
    <td><b>34.12</b></td>
  </tr>
</table>

Figure 6: Super-Node-based edges and relations constructed from a conversation in MECAD with K = 1. The utterance Super-Nodes of the two speakers are shown in gray and blue, respectively. The black solid and dashed lines denote the Super-Edges between the same speaker and different speakers, respectively, and the red dotted lines denote the Super-Edges between the utterance Super-Nodes and the conversation Super-Nodes.

D Implement Details of the Experiment

For the ECF dataset, we use the pre-trained RoBERTa-large\(^2\) model to initialize the feature extraction parameters of the text modality. For audio modality, we use the wav2vec2-base-960h\(^3\) model and for video modality we use the DenseNet (Huang et al., 2017) model. For the MECAD dataset, we use the chinese-roberta-wwm-ext-large\(^4\) model for the initialization of textual modal features, the wav2vec2-large-chinese-zh-cn\(^5\) model for the extraction of audio modal features, and the DenseNet model is also applied to the video modal. In our experiments, none of the parameters of the PLM were frozen. During the construction of the graph, we set the hyperparameter \(K\) to 3.

\footnotetext{
\(^2\)https://huggingface.co/FacebookAI/roberta-large
\(^3\)https://huggingface.co/facebook/wav2vec2-base-960h
\(^4\)https://huggingface.co/hfl/chinese-roberta-wwm-ext-large
\(^5\)https://huggingface.co/wbbbbbb/wav2vec2-large-chinese-zh-cn
}

During training, we use the AdamW (Loshchilov, 2017) optimizer with batch size and learning rate set to 16 and 5e-6, respectively, and perform a parameter update after every two mini-batches. Our model is trained for 50 epochs on the training set, and the checkpoints corresponding to the highest values of the weighted average F1 scores of the six emotions on the validation set are used as the results of the test set.

E Supplementary Experimental Results of M^3HG

E.1 Ablation Study

Effect of different modules. We conduct ablation studies to verify the effectiveness of different modules in M^3HG on the two datasets using 6 Avg and 4 Avg scores. As shown in Table 6, w/o \(N^e\) & \(N^c\) indicates no use of emotional and causal context nodes in graph construction. Consequently, emotion-cause pair prediction is performed directly based on the features of each utterance node. w/o inter-fusion and w/o intra-fusion denote the absence of inter-utterance and intra-utterance multi-modal fusion, respectively, during multi-scale semantic information fusion. Our model outperforms the state-of-the-art baselines even without utilizing the previous three mechanisms. Specifically, the performance of M^3HG degrades on both ECF and MECAD datasets when removing the emotional and causal context nodes, demonstrating the necessity of explicitly modeling the emotion and cause-related contexts. Moreover, removing both intra-utterance and inter-utterance semantic fusion results in a drop in the model’s performance, the former of which causes a more significant degradation. It highlights the importance of effectively fusing semantic information at different scales within heterogeneous graphs, particularly within individual utterances.
Table 5: Performance comparison of different methods on four subtasks. The best results and the second best results are in bold and underlined, respectively.

<table>
  <tr>
    <th rowspan="2">Dataset</th>
    <th rowspan="2">Method</th>
    <th colspan="3">EP</th>
    <th colspan="3">ER</th>
    <th colspan="3">CE</th>
    <th colspan="3">EC</th>
  </tr>
  <tr>
    <th>P</th><th>R</th><th>F1</th>
    <th>P</th><th>R</th><th>F1</th>
    <th>P</th><th>R</th><th>F1</th>
    <th>P</th><th>R</th><th>F1</th>
  </tr>
  <tr>
    <td rowspan="3">ECF</td>
    <td>SHARK</td><td>59.00</td><td>61.21</td><td>60.74</td><td>40.34</td><td>45.65</td><td>42.83</td><td>69.25</td><td>66.13</td><td>67.64</td><td>50.12</td><td>46.31</td><td>48.14</td>
  </tr>
  <tr>
    <td>GPT-4o(5-shots)</td><td>45.17</td><td>78.62</td><td>57.37</td><td>36.42</td><td>42.70</td><td>36.76</td><td>57.02</td><td>84.85</td><td>68.21</td><td>72.90</td><td>61.13</td><td>42.78</td>
  </tr>
  <tr>
    <td>M^3HG (T)</td><td>71.36</td><td>75.11</td><td>73.19</td><td>52.24</td><td>45.63</td><td>46.60</td><td>72.32</td><td>68.40</td><td>70.30</td><td>58.03</td><td>52.05</td><td>54.88</td>
  </tr>
  <tr>
    <td rowspan="3">MECAD</td>
    <td>SHARK</td><td>69.30</td><td>67.02</td><td>68.14</td><td>39.38</td><td>36.93</td><td>38.12</td><td>64.18</td><td>66.36</td><td>65.24</td><td>49.02</td><td>42.87</td><td>45.74</td>
  </tr>
  <tr>
    <td>GPT-4o(5-shots)</td><td>71.41</td><td>63.69</td><td>67.33</td><td>38.69</td><td>36.59</td><td>34.22</td><td>65.03</td><td>69.26</td><td>67.08</td><td>39.68</td><td>41.77</td><td>40.70</td>
  </tr>
  <tr>
    <td>M^3HG (T)</td><td>72.34</td><td>67.84</td><td>70.02</td><td>43.35</td><td>40.98</td><td>41.66</td><td>66.12</td><td>70.24</td><td>68.12</td><td>54.82</td><td>46.42</td><td>50.27</td>
  </tr>
</table>

Table 6: Ablation results.

<table>
  <tr>
    <th rowspan="2">Dataset</th>
    <th rowspan="2">Model</th>
    <th colspan="2">6 Avg.</th>
    <th colspan="2">4 Avg.</th>
  </tr>
  <tr>
    <th>6 Avg.</th><th>4 Avg.</th>
    <th>6 Avg.</th><th>4 Avg.</th>
  </tr>
  <tr>
    <td rowspan="4">ECF</td>
    <td>M^3HG</td><td>40.07</td><td>41.96</td><td></td><td></td>
  </tr>
  <tr>
    <td>w/o all modules</td><td>36.81<sub>(± 2.0)</sub></td><td>38.57<sub>(± 3.39)</sub></td><td></td><td></td>
  </tr>
  <tr>
    <td>w/o N^e & N^c</td><td>38.13<sub>(± 1.94)</sub></td><td>40.11<sub>(± 1.85)</sub></td><td></td><td></td>
  </tr>
  <tr>
    <td>w/o inter-fusion</td><td>39.56<sub>(± 0.51)</sub></td><td>41.14<sub>(± 0.82)</sub></td><td></td><td></td>
  </tr>
  <tr>
    <td>w/o intra-fusion</td><td>39.12<sub>(± 0.95)</sub></td><td>40.80<sub>(± 1.10)</sub></td><td></td><td></td>
  </tr>
  <tr>
    <td rowspan="4">MECAD</td>
    <td>M^3HG</td><td>32.82</td><td>34.59</td><td></td><td></td>
  </tr>
  <tr>
    <td>w/o all modules</td><td>30.47<sub>(± 1.45)</sub></td><td>32.27<sub>(± 2.32)</sub></td><td></td><td></td>
  </tr>
  <tr>
    <td>w/o N^e & N^c</td><td>30.94<sub>(± 1.88)</sub></td><td>32.79<sub>(± 1.80)</sub></td><td></td><td></td>
  </tr>
  <tr>
    <td>w/o inter-fusion</td><td>32.57<sub>(± 0.25)</sub></td><td>33.91<sub>(± 0.68)</sub></td><td></td><td></td>
  </tr>
  <tr>
    <td>w/o intra-fusion</td><td>32.16<sub>(± 0.66)</sub></td><td>33.33<sub>(± 1.26)</sub></td><td></td><td></td>
  </tr>
</table>

![Line graph showing results of M^3HG with various K values](page_101_1042_370_246.png)

Figure 7: Results of M^3HG with various K values.

Effect of the hyperparameter K. The hyperparameter K is closely related to the spatio-temporal complexity of the M^3HG’s graph construction. We vary the size of K (ranging from 1 to 6) to test its effect, and the result of the M^3HG on both datasets is shown in Figure 7. The performance of M^3HG on both datasets initially improves with increasing K and then declines, with the best performance observed at \( K = 3 \).

E.2 In-Depth Analysis

The impact of conversation length. To evaluate the performance of M^3HG in handling longer conversations, we present a comparison of the performance of M^3HG and other baseline models across conversations of varying lengths, as shown in Table 4. We observe that M^3HG outperforms all baseline models in scenarios involving conversations with more than 10 utterances, which account for 42.65% and 43.38% of all conversations in the ECF and MECAD datasets, respectively. In long conversations, baseline models, including GPT-4o, fail to effectively extract global contextual information, thereby missing a number of triplets. Our model, through semantic fusion at different scales within multimodal heterogeneous graphs, effectively captures more triplets by extracting contextual information from long conversations.

Model performance when cause utterance appears after emotion utterances. A key challenge in the MECTEC task is when the cause of a speaker’s emotion is revealed later in the conversation, requiring the model to effectively capture and interpret the global context of the conversation. To further emphasize the superior performance of M^3HG in handling cases where the cause utterance appears after the emotion utterance, we identified and filtered all such conversations from the ECF and MECAD datasets. The performance of M^3HG, compared with two other representative models, is shown in Table 7. M^3G demonstrates superior performance, while SHARK suffers a greater performance drop compared to M^3HG. Although the performance drop for GPT-4o (5-shots) is less pronounced, its overall performance remains unsatisfactory.

Model performance on four subtasks. To evaluate M^3HG’s performance more comprehensively,
<table>
  <tr>
    <th>U_1</th>
    <td>Yunru Chen: 不嘛这说看着我啊？(Why are you looking at me like that?)</td>
    <th>U_1</th>
    <td>Zongming Tan: 怎么了？(What's going on?)</td>
  </tr>
  <tr>
    <th>U_2</th>
    <td>Junjie Mo: 没事啦，不用客气。(It's nothing, no need to thank me.)</td>
    <th>U_2</th>
    <td>Di An: 我也不知道，总觉得有人在跟着我。(I don't know. I always feel like someone's following me.)</td>
  </tr>
  <tr>
    <th>U_3</th>
    <td>Yunru Chen: 你是不是跟李子维一样，觉得我说的那些话，都是乱编的？(Are you like Ziwei Li, thinking that what I said was all made up?)</td>
    <th>U_3</th>
    <td>Zongming Tan: 你刚回来不久，上海本身就没几个朋友，谁会跟着你。(You just came back not long ago, and you don't have many friends in Shanghai itself, who would follow you.)</td>
  </tr>
  <tr>
    <th>U_4</th>
    <td>Junjie Mo: 我相信你说的都是真的啊，在你的梦里，真的是有那么一个人，你很喜欢他，他也很喜欢你，而且...(I believe what you said is true. In your dream, there was really someone you liked a lot, and he liked you too, and...)</td>
    <th>U_4</th>
    <td>Di An: 我觉得很奇怪，加上今天已经好几次了，总觉得有人在跟着我，其实，又什么都没有，就是，会不会是我自己的幻觉呢？(I also thought it was strange. It's been several times today. I always feel that someone is following me, and when I turn around, there's nothing.)</td>
  </tr>
  <tr>
    <th>U_5</th>
    <td>Junjie Mo: 没事啦(It's nothing.)</td>
    <th>U_5</th>
    <td>Zongming Tan: 安迪，别胡思乱想，可能就是工作太辛苦，太累了，(Andy, did you get any ideas; it's probably just a case of working too hard and being tired.)</td>
  </tr>
  <tr>
    <th>U_6</th>
    <td>Yunru Chen: 而且什么，你说了？(And what? Tell me!)</td>
    <th>U_6</th>
    <td>Di An: 可能吧，也许是我今天没有吃早餐，低血糖了，所以才有幻觉。(Maybe, maybe I'm hallucinating because I didn't eat breakfast today and I'm low on blood sugar.)</td>
  </tr>
  <tr>
    <th>U_7</th>
    <td>Junjie Mo: 也许只是李子维，我要像你喜欢的，只是你梦里的王强盛。(Maybe it's only Li Zhiwei, I want to be like you wish you'd like only the Wang Quansheng in your dream.)</td>
    <th colspan="2">Ground Truth (Fear,2.2), (Fear,4.2), (Fear,4.4)</th>
  </tr>
  <tr>
    <th colspan="2">Ground Truth (Surprise,1.1), (Sad,3.3), (Sad,5.7), (Surprise,6.4), (Sad,7.7)</th>
    <th colspan="2">SHARK (Fear,4.4)</th>
  </tr>
  <tr>
    <th colspan="2">SHARK (Surprise,1.1), (Sad,3.3), (Surprise,6.6), (Sad,7.6)</th>
    <th colspan="2">GPT-4o (5-shots) (Surprise,1.1), (Fear,2.2), (Sad,4.4), (Anger,5.5), (Sad,6.6)</th>
  </tr>
  <tr>
    <th colspan="2">GPT-4o (5-shots) (Anger,3.1), (Surprise,6.4), (Sad,7.7)</th>
    <th colspan="2">M^3HG (T) (Fear,2.2), (Fear,4.3), (Fear,4.4)</th>
  </tr>
  <tr>
    <th colspan="2">M^3HG (Surprise,1.1), (Sad,3.3), (Sad,5.7), (Anger,6.5)</th>
    <th colspan="2">M^3HG (T+A+V) (Fear,4.2), (Fear,4.4), (Sad,6.6)</th>
  </tr>
</table>

Figure 8: Comparison of utter-cause-emotion triplet on two test samples.

we define the following four subtasks:
• Emotion Extraction (EP): Predict whether an utterance expresses an emotion (binary classification), same as SHARK.
• Emotion Recognition (ER): Predict the emotion category of an utterance (multi-class classification).
• Cause Extraction (CE): Predict whether an utterance is a cause utterance (binary classification), same as SHARK.
• Emotion-Cause Pair Extraction (EC): Predict whether two utterances of a conversation form an emotion-cause pair (binary classification).

Table 5 demonstrates the performance comparison between M^3HG and other SOTA models across the four subtasks. For the EP subtask, M^3HG performs the best across both datasets. It is worth noting that GPT-4o (5-shots) achieves a high recall on the ECF dataset. This phenomenon can be attributed to the more pronounced label sparsity in the ECF dataset compared to MECAD. As a result, GPT-4o (5-shots) frequently predicts that an utterance carries emotion, leading to a higher recall. For the ER subtask, M^3HG achieves the best results across both datasets. This demonstrates M3HG’s ability to effectively extract the emotional context embedded in utterances. For the CE subtask, M^3HG performs best, demonstrating the importance of integrating the cause prediction subtask into the model during training. For the EC subtask, GPT-4o (5-shots) similarly exhibits high recall on the ECF dataset. This is due to the severe label sparsity problem in the ECF dataset, compared to MECAD, which leads GPT-4o to predict as many emotion-cause pairs as possible.

E.3 Case Study

To demonstrate the superiority and limitations of M^3HG, we present a case study that compares the prediction results of M^3HG with those of two other representative models (i.e. SHARK, GPT-4o (5-shots)), using two sample conversations from the MECAD dataset. As shown in Figure 8, the first test sample demonstrates that M^3HG outperforms the other models in prediction accuracy, while GPT-4o exhibits the poorest performance. This can be attributed to M^3HG’s use of a multimodal heterogeneous graph and a specially designed conversation super-node, which effectively captures global contextual information. These features enable M^3HG to more accurately handle scenarios where the cause utterance appears after the emotion utterance.

In the second sample, M^3HG (T+A+V) is less effective than M^3HG (T) in predicting Utterance 6 as “Sad” and Utterance 2 as “Neutral”. This is because the combination of text and context in Utterance 2 conveys the speaker’s worried and fearful mood, while the video and audio signals suggest a
Table 8: An example of prompt for ChatGPT.

<html>
<table>
  <tr>
    <th>Instruction</th>
    <td>You are an expert in sentiment analysis and identification of emotional causes. I will give you a conversation between multiple speakers. You are required to extract the utter-cause-emotion triplet for a given utterance. First, infer the emotion label for the utterance (select one from: Anger, Disgust, Fear, Joy, Sadness, Surprise or Neutral). Then, identify the index(es) of the cause utterance(s) that triggered this emotion (the index should represent the utterance(s) from the conversation that caused the emotion, and it must be non-negative. Multiple indices should be separated by commas). If the predicted emotion is Neutral, there is no corresponding cause utterance. The output should follow the format: emotion label, cause utterance indices. Examples of the expected output format: Example 1: happy,3. Example 2: sad,3,4,5. Example 3: neutral.</td>
  </tr>
  <tr>
    <th>Input</th>
    <td>
      Input Conversation :
      { 1. Fang Sijin: First, change your clothes, then head to this address. A decoration company will be coming over shortly. You’ll need to supervise their work and see how you can help. }
      { 2. Zhu Shanshan: Wait, am I really responsible for this? I don’t know anything about decoration. }
      { 3. Fang Sijin: You’ve been handing out flyers for two days now. Have you gotten any interested customers? }
      { 4. Zhu Shanshan: But you only told me to distribute the flyers; you never ask for phone numbers! }
      Candidate Utterances:
      { 1. Fang Sijin: First, change your clothes, then head to this address. A decoration company will be coming over shortly. You’ll need to supervise their work and see how you can help. }
      { 2. Zhu Shanshan: Wait, am I really responsible for this? I don’t know anything about decoration. }
    </td>
  </tr>
  <tr>
    <th>Demonstrations</th>
    <td>
      Target Utterance:
      { 2. Zhu Shanshan: Wait, am I really responsible for this? I don’t know anything about decoration. }
      Target emotion labels and cause index(es):
      [Surprise, 1]
      Input Conversation :
      ......
      Candidate Utterances:
      ......
      Target Utterance:
      ......
      Target emotion labels and cause index(es):
      ......
    </td>
  </tr>
  <tr>
    <th>Output</th>
    <td>output example [Happy, 1, 2]</td>
  </tr>
</table>
</html>

calmer demeanor. This discrepancy likely caused M^3HG (T+A+V) to mispredict the emotions in this case. Nevertheless, M^3HG still outperforms all other baseline models, demonstrating its robustness and superior predictive capability even under challenging conditions.

F Prompt Design for ChatGPT

We use the GPT-4o model of OpenAI public API (version up to May 13, 2024) and design a prompt elaborately to test the performance on the MECTEC task. The prompt (i.e., the input of ChatGPT) includes three parts:

• Instruction. We use instructions to guide the ChatGPT on what it needs to do. Our instruction is as follows:

You are an expert in sentiment analysis and identification of emotional causes. I will give you a conversation between two or more speakers. You need to extract the utter-cause-emotion triplet of the given utterance.

Meanwhile, we provide a detailed description of the output formats required for ChatGPT, as illustrated in Table 8.

• Demonstrations We achieve the few-shot in-context learning of ChatGPT by adding demonstrations. We use the 5-shot in-context learning due to the limitations of the input length. Each demonstration includes a conversation as input and a target utterance as the target for prediction.

Except for the aforementioned two parts, we also need to describe the conversations to be predicted and the corresponding target utterance. An example is shown in Table 8.