DualGATs: Dual Graph Attention Networks for Emotion Recognition in Conversations
Year: 2023

Abstract:
Capturing complex contextual dependencies plays a vital role in Emotion Recognition in Conversations (ERC). Previous studies have predominantly focused on speaker-aware context modeling, overlooking the discourse structure of the conversation. In this paper, we introduce Dual Graph ATtention networks (DualGATs) to concurrently consider the complementary aspects of discourse structure and speaker-aware context, aiming for more precise ERC. Specifically, we devise a Discourseaware GAT (DisGAT) module to incorporate discourse structural information by analyzing the discourse dependencies between utterances. Additionally, we develop a Speakeraware GAT (SpkGAT) module to incorporate speaker-aware contextual information by considering the speaker dependencies between utterances. Furthermore, we design an interaction module that facilitates the integration of the DisGAT and SpkGAT modules, enabling the effective interchange of relevant information between the two modules. We extensively evaluate our method on four datasets, and experimental results demonstrate that our proposed DualGATs surpass state-of-the-art baselines on the majority of the datasets.1

heading: 1 Introduction
text: With the increasing availability of conversational data on social media platforms (Poria et al., 2019a), Emotion Recognition in Conversations (ERC) has emerged as a popular research topic (Poria et al., 2019b). Its objective is to identify and track the emotional state of each utterance. ERC plays a crucial role in various applications, including opinion mining in social media (Chatterjee et al., 2019b) and the development of empathetic dialogue systems that can analyze user emotional states and gen-
∗Corresponding author. 1Available at https://github.com/BladeDancer957/
DualGATs.
erate emotion-aware responses (Zhou et al., 2018; Liu et al., 2021; Peng et al., 2022, 2023).
However, analyzing emotions in conversations poses significant challenges. Unlike emotion recognition in isolated sentences (Seyeditabari et al., 2018), ERC requires careful consideration of contextual dependencies. Previous ERC methods have primarily focused on capturing speaker or temporal dependencies between utterances, making the modeling of speaker-aware context central to these approaches (Majumder et al., 2019).
To incorporate speaker-aware contextual information, numerous methods have been proposed to model conversations as sequences (Poria et al., 2017; Hazarika et al., 2018a,b; Jiao et al., 2019; Hu et al., 2021; Ong et al., 2022) or graphs (Ghosal et al., 2019; Ishiwatari et al., 2020; Shen et al., 2021b; Li et al., 2022). Sequence-based methods capture sequential information by encoding utterances temporally using Recurrent Neural Networks (RNNs). Majumder et al. 2019 designed an independent Gated Recurrent Unit (GRU) (Cho et al., 2014) to track the emotional state of the speaker. However, these sequence-based methods often rely on limited information from nearby utterances to update the current utterance’s representation, making it challenging to capture distant contextual information and achieve satisfactory performance. To address this limitation, graph-based methods simultaneously aggregate information from surrounding contextual utterances to update the representation of the current utterance using Graph Neural Networks (GNNs) (Kipf and Welling, 2017). These methods typically treat the conversation as a directed graph, where nodes represent utterances, edges indicate dependency links between pairs of nodes, and edge labels denote the dependency types, such as speaker or temporal relationships.
Despite the remarkable progress made by sequence-based and graph-based methods, there is a need for greater emphasis on explicitly model-
7395
ing discourse structure within conversations. Discourse structure, which includes discourse dependency links and their types between utterances, offers a straightforward way to capture the essential information flow in a conversation. As illustrated in Figure 1, highly relevant utterances are linked based on discourse dependency types such as Background, Acknowledgement, and Question-Answer Pair. Explicitly incorporating these discourse dependencies in conversations can assist models in capturing significant contextual cues that influence emotions. For instance, let’s consider the first and fourth utterances in Figure 1, where there exists a direct discourse dependency link of Acknowledge type between utterances 1 and 4. In utterance 1, Speaker A expresses a positive opinion about wine, conveying a sense of joy emotion. Speaker B strongly acknowledges this opinion in utterance 4, stating that the wine improves and also experiences a sense of joy emotion.
In this paper, we propose a novel method called Dual Graph ATtention networks (DualGATs) that aims to improve the accuracy of ERC by simultaneously considering the complementarity of discourse structure and speaker-aware context. The DualGATs layer comprises three components: Discourse-aware GAT (DisGAT), Speaker-aware GAT (SpkGAT), and an interaction module. The DisGAT module is designed to capture structurallevel correlations among the interactive turns explicitly. It propagates the message over the discourse dependency graph obtained from a discourse parser (Shi and Huang, 2019), thereby incor-
porating discourse structural information. On the other hand, the SpkGAT module is implicitly organized to capture semantic-level correlations among the interactive turns. It conducts message propagation over the speaker dependency graph, constructed based on speaker identities and the relative positions of utterances, enabling the incorporation of speaker-aware contextual information. Furthermore, inspired by previous work (Li et al., 2021b; Zhang et al., 2022), the interaction module leverages mutual cross-attention to integrate the DisGAT and SpkGAT modules, facilitating the exchange of relevant information between the two modules. To enhance the complementarity of the learned representations from the DisGAT and SpkGAT modules and minimize overlap, the interaction module also includes a differential regularizer. This regularizer encourages the two modules to capture different contextual information.
Our contributions can be summarized as follows:
• We propose DualGATs to simultaneously consider the complementarity of discourse structure and speaker-aware context for more precise and accurate ERC.
• We introduce an interaction module to exchange the relevant information between the SpkGAT and DisGAT modules by mutual cross-attention, where a differential regularizer is proposed to induce the two modules to capture different contextual information.
• We conduct extensive experiments on four publicly available ERC datasets. The results of our experiments demonstrate that DualGATs outperform state-of-the-art baselines on most of the tested datasets. Further analyses validate the effectiveness of the critical components in DualGATs.

heading: 2 Related Work

heading: 2.1 ERC
text: Recently, due to the proliferation of publicly available conversational datasets (Chen et al., 2019; Chatterjee et al., 2019a), ERC has increasingly become a popular research topic, including the textmodality and multi-modality settings (Zhang et al., 2023; Chen et al., 2023). Here, we specifically focus on the former. Previous studies primarily concentrate on modeling speaker-aware conversational context. Early methods rely on RNNs to encode utterances temporally and track the speaker’s
state (Jiao et al., 2019; Hu et al., 2021). Notably, BC-LSTM (Poria et al., 2017) employs Long ShortTerm Memory networks (LSTMs) (Hochreiter and Schmidhuber, 1997), while ICON (Hazarika et al., 2018a) and CMN (Hazarika et al., 2018b) utilize GRUs (Cho et al., 2014) and memory networks. DialogueRNN (Majumder et al., 2019) utilizes three GRUs to capture speaker, temporal, and emotional dependencies among utterances. However, these sequence-based methods often rely on limited information from nearby utterances to update the state of the current utterance, which poses challenges in capturing long-range contextual information.
To model the global conversational context, various graph-based methods have emerged (Zhang et al., 2019; Shen et al., 2021a). DialogueGCN (Ghosal et al., 2019) treats each conversation as a fully-connected graph, where nodes represent utterances and edges denote speaker and temporal dependencies between utterances. RGAT (Ishiwatari et al., 2020) introduces relational position encoding to incorporate position information into the GNNs explicitly. DAG-ERC (Shen et al., 2021b) utilizes directed acyclic graphs to model the interaction between speakers and utterances. Additionally, there are several Transformer-based methods (Vaswani et al., 2017) for modeling the conversational context. Since the self-attention module in Transformer can be seen as a fully-connected graph, we consider some Transformer-based approaches as graph-based methods. CoG-BART (Li et al., 2022) employs BART (Lewis et al., 2020) as an utterance encoder and incorporates an auxiliary response generation task to enhance the model’s ability to handle contextual information. It also leverages contrastive learning to improve the identification of similar emotions. CoMPM (Lee and Lee, 2021) introduces a pre-trained memory module to consider the linguistic preferences of speakers.
Since humans do not always explicitly express their emotions in their words, there are many methods to incorporate additional general information into the sequence- or graph-based methods to enhance the understanding of implicit emotions. For example, KET (Zhong et al., 2019), KAITML (Zhang et al., 2020), and COSMIC (Ghosal et al., 2020) introduce commonsense knowledge, TODKAT (Zhu et al., 2021) integrates topic information, KI-Net (Xie et al., 2021) leverages sentiment lexicons, DialogueRole (Ong et al., 2022) incorporates utterance role informa-
tion, SKAIG (Li et al., 2021a) fuses psychological knowledge, and CauAIN (Zhao et al., 2022) includes emotion cause information to enhance ERC.
Despite significant progress, these methods above need to pay more attention to the importance of conversational discourse structure in capturing salient contextual cues that influence emotion. However, due to the complexity of humanhuman interaction, GNNs (Kipf and Welling, 2017; Yu et al., 2022) directly over the parsed discourse dependency graph like DisGCN (Sun et al., 2021) may not work well as expected on datasets that are not sensitive to discourse structure. Instead of relying solely on discourse structure, we integrate it into our carefully designed DualGATs framework to simultaneously consider discourse structure’s and speaker-aware context’s complementarity. This integration allows us to achieve more accurate ERC by leveraging the benefits of both aspects.

heading: 2.2 Discourse Parsing
text: Recently, deep sequential models have emerged as practical approaches for conversational discourse parsing (Shi and Huang, 2019; Liu and Chen, 2021). These models have proven their efficacy in various dialogue understanding tasks, such as multiturn response selection (Jia et al., 2020), as well as dialogue generation tasks, including conversation summarization (Chen and Yang, 2021; Feng et al., 2021). In our work, the discourse structures on which our DisGAT module relies are also parsed using deep sequential models (Shi and Huang, 2019). Leveraging discourse dependencies intuitively enables the model to encode unstructured human conversations better and focus on salient utterances, leading to more accurate predictions.

heading: 3 Methodology
text: We begin by providing a formal definition of the ERC task. A conversation is represented as a sequence of utterances (ui, si)|i = 1, ..., N , where each utterance ui is spoken by speaker si, and N denotes the total number of utterances. The objective of the ERC task is to assign an emotion label yi ∈ Y , such as joy, sadness, etc., to each utterance ui in the conversation, where Y represents the set of possible emotion labels.
The proposed DualGATs consist of three main components: feature extraction, DualGATs layer, and emotion prediction. The overall architecture of our DualGATs is illustrated in Figure 2.

heading: 3.1 Feature Extraction
text: In line with previous methods (Ghosal et al., 2020; Shen et al., 2021b), we employ the RoBERTa Large model (Liu et al., 2019) to extract utterance features. The RoBERTa Large model is first fine-tuned for emotion prediction using the transcript of utterances and subsequently utilized as a feature extractor with frozen parameters during the training of DualGATs. Specifically, for the i-th utterance ui, we prepend a special token “[CLS]” to its tokens, resulting in an input format of {[CLS], w1, ..., wni}, where ni denotes the number of tokens in ui. Subsequently, we extract the output activations from the last layer corresponding to the “[CLS]” token, which serves as the feature representation hi ∈ Rdu of ui. Here, du represents the dimension of the feature representation. Collectively, the feature representations for all utterances are represented as Hu ∈ RN×du .

heading: 3.2 DualGATs Layer
text: The DualGATs layer efficiently captures both discourse structure and speaker-aware context within a conversation, employing three essential modules: DisGAT, SpkGAT, and Interaction. In this section,
we first outline the computation process for each module in the initial layer and then extend it to multiple subsequent layers.
DisGAT The DisGAT module performs message propagation over a discourse dependency graph to integrate discourse structural information. We begin by explaining the construction of the discourse dependency graph, followed by an overview of the inference process employed by the DisGAT module on the constructed graph.
We define the discourse dependency graph of a conversation as GDis = (V Dis,EDis), where V Dis represents the set of nodes representing Elementary Discourse Units (EDUs), and EDis is the adjacency matrix that describes the discourse dependencies between EDUs. In our approach, each utterance in the conversation is treated as an EDU, and we leverage the 16 discourse dependency types outlined in (Asher et al., 2016). These dependency types encompass Comment, Clarification Question, Elaboration, Acknowledgment, Continuation, Explanation, Conditional, Question-Answer Pair, Alternation, Question-Elaboration, Result, Background, Narration, Correction, Parallel, and Contrast (We refer to this set of types as RDis).
Specifically, we first pre-train a discourse parser (Shi and Huang, 2019) on a humanannotated dialogue corpus (Asher et al., 2016), with a 0.78 F1 on link predictions and 0.56 F1 on relation classifications, comparable to the stateof-the-art results. Then, we use this pre-trained parser to predict the discourse dependencies within conversations present in ERC datasets. Consequently, for each conversation, we represent its corresponding discourse dependency graph as GDis = (V Dis,EDis). Here, V Dis[i] or vDisi represents the node corresponding to utterance ui, initialized with the corresponding feature representation hi. The edge EDis[i][j] or eDisi,j is assigned the dependency type rDis ∈ RDis if a link exists from ui to uj with that specific type. This is illustrated in the left part of Figure 2.
Once the discourse dependency graph for the conversation is constructed, we apply the DisGAT module to propagate and aggregate discourse structural information among the graph nodes. The DisGAT module is built upon GAT (Veličković et al., 2018) but includes type coding to account for the dependency types between nodes (utterances). Specifically, for a given node vDisi , the DisGAT aggregates the information of its neighboring nodes as follows:
αij = smi(LRL(aT [Whi∥Whj∥eDisij ])) hDisi = ∑
j∈NDisi
αijWhj (1)
where αij denotes the edge weight from node vDisi to its neighbor vDisj , sm denotes softmax function, LRL denotes LeakyReLU activation function, W and a denote trainable parameters, eDisij ∈ R|R Dis| denotes the one-hot coding (fixed during model training) corresponding to the discourse dependency type between nodes vDisi and v Dis j , |RDis| denotes the number of discourse dependency types, ∥ denotes a concatenation operation, NDisi denotes the neighbours of node vDisi in GDis, hDisi ∈ Rdh denotes the hidden representation associated with node vDisi after DisGAT update, and dh denotes the dimension of the hidden representation. The updated hidden representation of all nodes is denoted as HDis ∈ RN×dh .
We summarize the calculation process of the DisGAT in the initial layer as follows:
HDis = DisGAT(Hu,EDis) (2)
SpkGAT The SpkGAT module performs message propagation on a speaker dependency graph to incorporate speaker-aware contextual information. We will first explain the construction of the speaker dependency graph and then introduce the inference process of the SpkGAT on this constructed graph.
We define the speaker dependency graph of a conversation as GSpk = (V Spk,ESpk), where V Spk[i] or vSpki represents ui (the i-th utterance), and its representation is initialized with the corresponding feature representation hi. ESpk is the adjacency matrix that describes the speaker along with temporal dependencies between nodes (utterances). Following the conventions of previous graph-based ERC methods (Ghosal et al., 2019; Ishiwatari et al., 2020), we define five speaker dependency types: Self-Past, Self-Future, Inter-Past, Inter-Future, and SelfLoop (referred to as set RSpk). Specifically, Self represents the influence of the current utterance on other utterances expressed by the same speaker. Inter indicates the influence of the current utterance on those expressed by other speakers (excluding the speaker of the current utterance). Past and Future refer to the relative position of the current utterance and other utterances in the conversation, determining how past utterances influence future utterances and vice versa.2 SelfLoop signifies the self-influence of the current utterance. For any ui and uj , ESpk[i][j] or e Spk i,j = r
Spk if they satisfy the speaker dependency type rSpk ∈ RSpk (as depicted in the right part of Figure 2).
After constructing the speaker dependency graph for the conversation, we implement the SpkGAT to propagate and aggregate speaker-aware contextual information across the graph nodes. Similarly, the calculation process of the SpkGAT in the initial layer is summarized as follows:
HSpk = SpkGAT(Hu,ESpk) (3)
Interaction Module To capture distinct information from the discourse structure and speaker-aware context, we introduce a differential regularizer that encourages divergence between the updated representations of the DisGAT and SpkGAT modules. The regularizer is formulated as follows:
ℓreg = 1
||HDis −HSpk||F (4)
2Since ERC is viewed as an offline task and future dependencies may help the model fill in some missing information, like the speaker’s background, we consider future influence.
where the subscript F denotes the Frobenius norm. Then, to integrate the DisGAT and SpkGAT modules and effectively exchange relevant information between the two modules, we adopt a mutual crossattention as a bridge. The computation process is formulated as follows:
A1 = softmax(HDisW1(HSpk)T )
A2 = softmax(HSpkW2(HDis)T ) HDis ′ ,HSpk ′ = A1H Spk,A2H Dis
(5)
where W1,W2 ∈ Rdh×dh are learnable parameters and A1, A2 ∈ RN×N are temporary matrices projecting from HSpk to HDis and HDis to HSpk, respectively. Here, HDis
′ ∈ RN×dh can be regarded as a projection from HSpk to HDis , and HSpk
′ ∈ RN×dh follows identical principle. The Whole Process To iteratively refine and exchange discourse structural information and speaker-aware contextual information across multiple consecutive layers, we generalize the calculation process of the initial layer. The detailed procedures are as follows:
HDis,[l] = DisGAT(D[l],EDis)
HSpk,[l] = SpkGAT(S[l],ESpk)
HDis ′,[l],HSpk ′,[l] = Inter(HDis,[l],HSpk,[l])
D[l+1],S[l+1] = HDis ′,[l],HSpk ′,[l]
(6) where D[0] = S[0] = Hu and l ∈ [0, L− 1].

heading: 3.3 Emotion Prediction
text: We obtain the final representation for ui by concatenating the output (HDis ′,[L],HSpk ′,[L]) of the L-layer DualGATs. The final representation is classified via a Fully-Connected Network (FCN):
li = ReLU(Wh[h Dis′,[L] i ∥h Spk′,[L] i ] + bh)
pi = softmax(Wlli + bl)
ŷ = argmaxk∈Ypi[k]
(7)
where ŷi is the predicted emotion label for utterance ui, h Dis′,[L] i , h Spk′,[L] i ∈ Rdh denote the ith representation in HDis ′,[L] and HSpk
′,[L], Wh ∈ Rdh×2dh ,Wl ∈ Rde×dh , bh ∈ Rdh and bl ∈ Rde are learnable parameters of FCN, and de denotes the number of emotion labels in the dataset.

heading: 3.4 Loss Function
text: Our training goal is to minimize the following total objective function:
ℓtotal = ℓerc + λℓreg (8)
where λ is a regularization coefficient. ℓerc is a standard cross-entropy loss, formulated as:
ℓerc = − B∑
β=1
N(β)∑
i=1
logpβ,i[yβ,i] (9)
where B is the number of conversations, N(β) is the number of utterances in the β-th conversation, and yβ,i is the ground truth label in one-hot form.

heading: 4 Experimental Settings

heading: 4.1 Datasets
text: We evaluate our DualGATs on the following four ERC datasets. The statistics of these four datasets are drawn in Table 1.
IEMOCAP (Busso et al., 2008): Each conversation comes from the performance based on the script by two actors. There are 6 emotion labels including happiness, sadness, anger, frustration, excited, and neutral. Since IEMOCAP has no validation set, we follow (Shen et al., 2021b) to use the last 20 conversations in training set for validation.
MELD (Poria et al., 2019a): Scripts collected from the Friends TV series. There are 7 emotion labels including neutral, joy, surprise, sadness, anger, disgust, and fear.
EmoryNLP (Zahiri and Choi, 2018): Scripts collected from the Friends TV series as well. Unlike MELD, its emotion labels include sad, mad, scared, powerful, peaceful, joyful, and neutral.
DailyDialog (Li et al., 2017): Daily communications written by human. Its emotion labels are the same as the ones used in MELD.

heading: 4.2 Significance Test and Evaluation Metrics
text: To test the significance of the performance improvement, we conduct a paired t-test with a default level of 0.05 (Koehn, 2004). Following previous methods (Ghosal et al., 2019; Shen et al., 2021a,b), we adopt micro-averaged F1 score excluding the majority class (neutral) for DailyDialog and weightedaverage F1 score for the other datasets.

heading: 4.3 Compared Baselines
text: For a comprehensive performance evaluation, we compare our DualGATs with the following stateof-the-art baselines:
BC-LSTM (Poria et al., 2017), ICON (Hazarika et al., 2018a), DialogueRNN (Majumder et al., 2019), DialogueCRN (Hu et al., 2021), KET (Zhong et al., 2019), DialogueGCN (Ghosal et al., 2019), RGAT (Ishiwatari et al., 2020), DialogXL (Shen et al., 2021a), DAG-ERC (Shen et al., 2021b), CoG-BART (Li et al., 2022), CoMPM (Lee and Lee, 2021), COSMIC (Ghosal et al., 2020), TODKAT3 (Zhu et al., 2021), DialogueRole (Ong et al., 2022), CauAIN (Zhao et al., 2022), and DisGCN (Sun et al., 2021).
For a fair comparison, baseline+RoBERTa means to use RoBERTa Large (Liu et al., 2019) as an utterance feature extractor as we do. Note that most other baselines natively use pre-trained models as utterance feature extractors, such as DAG-ERC, CoMPM, COSMIC, DialogueRole, and CauAIN use RoBERTa Large, DialogXL uses XLNet (Yang et al., 2019), CoG-BART uses BART (Lewis et al., 2020), and DisGCN uses BERT (Kenton and Toutanova, 2019).

heading: 4.4 Implementation Details
text: Our DualGATs are trained with the Adam optimizer (Kingma and Ba, 2015). We conduct a hyper-parameter search for DualGATs on each dataset according to the F1 score of the validation set. The hyper-parameters to search include learning rate (lr), dropout rate, batch size, and the DualGATs Layer Number (layers) within the ranges of {1e-5,5e-5,1e-4,5e-4,1e-3,5e3}, {0.0,0.1,0.2,0.3,0.4,0.5}, {8,16,32,64,128}, and {1,2,3,4,5,6}. The details of hyper-parameters for DualGATs on each dataset are shown in Table 2. For other hyper-parameters, the dimension
3The sklearn was misused, causing the unusual high performance of MELD and EmoryNLP in TODKAT paper. Thus, we adopt the updated performance from the official Github repository: https://github.com/something678/TodKat.
of the feature representation du from the RoBERTa is 1024, the dimension of the hidden representation dh is 300, and the regularization coefficient λ is 0.3. Each training and testing process is run on an NVIDIA A100 GPU with 40GB of memory. Each training process contains 60 epochs, costing at most 50 seconds per epoch. The model with the highest F1 score on the validation set is used to evaluate the test set. The reported results for all our runs are based on the average performance of 5 random runs on the test set.

heading: 5 Results and Discussions

heading: 5.1 Main Results
text: The overall performance of all the compared baselines and our DualGATs on the four datasets is reported in Table 3.
Table 3 shows that when equipped with RoBERTa as a feature extractor, baselines such as DialogueRNN, DialogueCRN, DialogueGCN, and RGAT see considerable improvements. When feature extractors are all based on pre-trained models, graph-based methods such as DialogueGCN
Table 4: Experimental results of ablation study.
Models IEMOCAP MELD EmoryNLP DailyDialog DisGAT 64.56 64.23 37.65 58.96 SpkGAT 66.32 64.66 38.34 59.91 DualGATs w/o regularizer 66.70 65.73 39.53 60.93 DualGATs w/o cross attention 66.43 65.46 39.68 60.26 DualGATs (Ours) 67.68 66.90 40.69 61.84
or RGAT+RoBERTa, DialogXL, DAG-ERC, CoGBART, and CoMPM, overall outperform sequencebased methods such as DialogueRNN or DialogueCRN+RoBERTa across the four datasets. It indicates that sequence-based methods can not encode the context as effectively as graph-based methods, especially for long-distance contexts. Moreover, when incorporating additional information into the sequence-based or graph-based methods, such as commonsense knowledge in COMSIC, topic information in TODKAT, utterance role in DialogueRole, and emotion cause in CauAIN, we see further improvements in overall performance. It indicates that the additional information improves the model’s understanding of implicit emotions.
However, these methods neglect the importance of explicitly modeling discourse structure. Compared to focusing on speaker-aware context modeling only, our DualGATs explicitly incorporate discourse structural information by the DisGAT module, so it can capture salient contextual cues that straightforwardly influence emotion. Moreover, GNNs directly only over the parsed discourse dependency graph result in poor performance, such as DisGCN. In contrast, our DualGATs model discourse structure and speaker-aware context simultaneously, achieving competitive performance on the IEMOCAP dataset and reaching a new state of the art on the MELD, EmoryNLP, and DailyDialog datasets compared to all baselines. These results show that our DualGATs effectively integrate discourse structural and speaker-aware contextual information and consider their complementarity for more precise ERC.

heading: 5.2 Ablation Study
text: In this section, we perform ablation studies to analyze the effects of critical modules in our DualGATs, shown in Table 4.
DisGAT only models discourse structure for ERC, which does not work well on datasets that are not sensitive to discourse dependencies due to the complexity of human-human interaction. SpkGAT only models speaker-aware context for ERC and achieves better performance than DisGAT, indicat-
1, SA
2, SB
Unfortunately , he wasn't able to evacuate in time . He ended up dying in the tower .
I'm sorry . That must have really
been a nightmarish day for you
and your family .
sad
1, SA 2, SB
I like winter. Me too.
happy neutral 3, SA It’s snowing heavily. What about taking a walk? happy 4, SB That’s a good idea. Let’s go! happy 5, SA happy What a heavy snow! Look! The water is frozen. a)
b)
1, SA 2, SB
I like winter. Me too.
happy 3, SA It’s snowing heavily. What about taking a walk? happy 4, SB That’s a good idea. Let’s go!
5, SA happy What a heavy snow! Look! The water is frozen.
a)
sad
ing that for ERC, modeling speaker-aware context is more important than discourse structure. Our DualGATs model both discourse structure and speakeraware context for ERC and outperform DisGAT and SpkGAT, showing that our DualGATs can simultaneously consider the complementarity of both for more accurate ERC. DualGATs w/o regularizer means we remove the differential regularizer in the interaction module. The results show that the differential regularizer induces the DualGATs to learn more accurate complementary information. DualGATs w/o cross attention denote that we remove the mutual cross-attention transformation in the interaction module so that the DisGAT and SpkGAT modules can not interact. At this point, we concatenate the output representations of the two modules in the last layer to perform emotion prediction. Therefore, the performance drops significantly on four benchmark datasets. Overall, our DualGATs with all modules achieve the best performance.

heading: 5.3 Error Analysis
text: After going through the predicted labels on the four datasets, we find that the following two aspects primarily cause the errors.
Firstly, our DualGATs tend to misclassify utterances of other emotions to neutral. This is because most utterances contain neutral emotion in the ERC datasets, especially MELD, EmoryNLP, and DailyDialog datasets where the proportion of neutral utterance is 46.95%, 29.95%, and 83.10%,
respectively. These datasets’ highly imbalanced class distributions lead to confusion between a few non-neutral utterances and much more neutral ones, restraining the emotion recognition performance.
Secondly, our DualGATs fail to distinguish between emotion pairs that are closely related, such as happiness versus excited, anger versus frustration, and peaceful versus joyful. As shown in Figure 3, we present two cases of misclassification between happiness versus excited and anger versus frustration in the IEMOCAP dataset. Taking Figure 3 (a) as an example, it isn’t easy to distinguish whether Speaker A was happiness or excited when she or he said she or he got good news. This misclassification phenomenon between similar emotions has also been reported by (Ghosal et al., 2019; Shen et al., 2021b; Ong et al., 2022).

heading: 5.4 Impact of the DualGATs Layer Number
text: To study the impact of the DualGATs layer number, we evaluate our DualGATs with one to six layers on the IEMOCAP and MELD datasets. As demonstrated in Figure 4, our model with two DualGATs layers performs best. On the one hand, discourse structural information and speaker-aware contextual information might not be refined and exchanged well when the number of layers is small. On the other hand, if there are too many layers, the performance will drop significantly, due to the generation of redundant or compatible representations, canceling important information.

heading: 6 Conclusion
text: In this paper, we propose DualGATs with DisGAT, SpkGAT, and Interaction modules to simultaneously consider the discourse structure’s and speaker-aware context’s complementarity for more
accurate ERC. The DisGAT and SpkGAT incorporate discourse structural and speaker-aware contextual information in parallel. The subsequent interaction module integrates the DisGAT and SpkGAT and effectively exchanges relevant information between the two modules via mutual cross-attention. Experimental results show that our DualGATs outperform previous state-of-the-art baselines on most tested datasets, and further analysis validates the effectiveness of critical modules in DualGATs.
In the future, we will explore the following aspects: (1) Apply our method to similar tasks that need to incorporate the discourse structural and speaker-aware contextual information; (2) Enhance the ability of our method to handle class imbalance or similar emotion problems, such as the introduction of data augmentation or contrastive learning techniques; (3) Deal with domain gap problem when directly using pre-trained deep sequential models to parse conversations in ERC datasets (Dong et al., 2020, 2021).
Limitations
Although our DualGATs simultaneously consider the complementarity of discourse structure and speaker-aware context for more accurate ERC, it requires more computation and a longer training time. The performance of discourse parsing could be more satisfying in the current stage. Moreover, we directly utilize pre-trained deep sequential models to parse dialogues in ERC datasets, which does not address the domain gap problem well.
Ethics Statement
To consider ethical concerns, we describe the following: (1) We conduct all experiments on existing datasets derived from public scientific research. (2) Our work does not involve any sensitive tasks or data. (3) We describe the datasets’ statistics and our method’s hyper-parameter settings. Our analysis is consistent with the experimental results. (4) We will release our code on GitHub for reproducibility.

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: Limitations section
7 A2. Did you discuss any potential risks of your work? Our work does not involve any sensitive data or tasks, and there is no potential risk.
3 A3. Do the abstract and introduction summarize the paper’s main claims? 1
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? 4
3 B1. Did you cite the creators of artifacts you used? 4
7 B2. Did you discuss the license or terms for use and / or distribution of any artifacts? We will discuss the license at GitHub upon publication.

heading: 3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided
text: that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? 4

heading: 3 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any
text: information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? 4

heading: 3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and
text: linguistic phenomena, demographic groups represented, etc.? 4
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. 4
C 3 Did you run computational experiments? 4

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? 4
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? 4

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? 4

heading: 3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? 4
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: In the future, we will explore the following aspects: (1) Apply our method to similar tasks that need to incorporate the discourse structural and speaker-aware contextual information; (2) Enhance the ability of our method to handle class imbalance or similar emotion problems, such as the introduction of data augmentation or contrastive learning techniques; (3) Deal with domain gap problem when directly using pre-trained deep sequential models to parse conversations in ERC datasets (Dong et al. , 2020, 2021). Limitations
Although our DualGATs simultaneously consider the complementarity of discourse structure and speaker-aware context for more accurate ERC, it requires more computation and a longer training time. The performance of discourse parsing could be more satisfying in the current stage. Moreover, we directly utilize pre-trained deep sequential models to parse dialogues in ERC datasets, which does not address the domain gap problem well