Conversation Understanding using Relational Temporal Graph Neural Networks with Auxiliary Cross-Modality Interaction
Authors: Cam Van Thi Nguyen
Tuan Anh Mai
Son Le The
Dang Hai Kieu
Duc-Trong Le
Venue: EMNLP 2023 Main (2023)
Keywords: Multimodal Emotion Recognition
Relational Temporal GNNs
Conversation Understanding
Pairwise Cross Modality

Abstract: Emotion recognition is a crucial task for human conversation understanding. It becomes more challenging with the notion of multimodal data, e.g., language, voice, and facial expressions. As a typical solution, the global- and the local context information are exploited to predict the emotional label for every single sentence, i.e., utterance, in the dialogue. Specifically, the global representation could be captured via modeling of cross-modal interactions at the conversation level. The local one is often inferred using the temporal information of speakers or emotional shifts, which neglects vital factors at the utterance level. Additionally, most existing approaches take fused features of multiple modalities in an unified input without leveraging modality-specific representations. Motivating from these problems, we propose the Relational Temporal Graph Neural Network with Auxiliary Cross-Modality Interaction (CORECT), an novel neural network framework that effectively captures conversation-level cross-modality interactions and utterance-level temporal dependencies with the modality-specific manner for conversation understanding. Extensive experiments demonstrate the effectiveness of CORECT via its state-of-the-art results on the IEMOCAP and CMU-MOSEI datasets for the multimodal ERC task.

Conversation Understanding using Relational Temporal Graph Neural Networks with Auxiliary Cross-Modality Interaction
Van Thi Nguyen', Anh-Tuan Mai The-Son Le'
Hai-Dang Kieu', Duc- muc-Trong Le
VVN University of Engineering and Technology, Hanoi, Vietnam 2FPT Software AI Center
(vanntc, 20020269, 21020089, dangkh_uet _et, trongld]evnu. edu. vn
Abstract
Emotion
Utterance text
Utterance test
Emotion
sure standing beach, this wailing, fighting.
Emotion recognition is a crucial task for hu- man conversation understanding. becomes more challenging with the notion of multi- modal data, e.g., language, voice, and facial expressions. As typical solution, the global- and the local context information are exploited predict emotional label for every sin- gle sentence, i.c., utterance, in the dialogue. Specifically, the global representation could be captured via modeling of cross-modal in- teractions the conversation level. The lo- cal one is often inferred using the temporal information of speakers or emotional shifts, which neglects vital factors at the utterance level. Additionally, most existing approaches take fused features of multiple modalities in an unified input without leveraging modality- specific representations. Motivating from these problems, we propose the Relational Temporal Graph Neural Network with Auxiliary Cross- Modality Interaction (CORECT), an novel neu- ral network framework that effectively captures conversation-level cross-modality interactions and utterance-leve temporal dependencies with the modality-speciff manner for conversation understanding. Extensive experiments demon- strate the effectiveness of CORECT via its state- of-the-art results on the JEMOCAP and CMU- MOSEI datasets for the multimodal ERC task.
excttdd
Just couple days ago.
exctted
Oh my gosh.
Right
neutral
never
This isn't anything like thought anything would
excited
thought you would get married.
sad
excited
know me nelther.
std
excited
Oh my gosh.
This jusi this..
sad
Ses01M, Ses01M_scrip002
SesO5F
Speaker
Speaker
Figure 1: Examples of temporal effects on conversations
Multimodal ERC, which aims to automatically de- tect the speaker's emotional state during conver- sation using information from text content, facial expressions, and audio signals, has garnered signif- icant attention and research in recent years and has been applied to many real-world scenarios (Sharma and Dhall, 2021; Joshi et al., 2022).
Massive methods have been developed to model conversation's 's context. These approaches can be categorized into two main groups: graph-based methods (Ghosal et al., 2019; Zhang et al., 2019; Shen et al., 2021 and recurrence-based meth- ods (Hazarika et al., 2018a; Ghosal et al., 2020; Majumder et al., 2019; Hu et al., 2021). In addi- tion, there have been advancements multimodal models that leverage the dependencies and com- plementarities of multiple modalities to improve the ERC performance (Poria et al., 2017: Hazarika et al., 2018b; Zadeh et al., 2018). One limitation of these methods is their heavy reliance nearby utterances when updating the state of the query utterance, which can restrict their overall perfor- mance. Recently, Graph Neural Network (GNN)- based methods have been proposed for the mul- timodal ERC task due to their ability to capture
1 Introduction
Our social interactions and relationships are all in- fluenced by emotions. Given the transcript a conversation and speaker information for each con- stituent utterance, the task of Emotion Recognition in Conversations (ERC) aims to identify the emo- tion expressed in each utterance from a predefined set of emotions (Poria et al., 2019). The multimodal nature of human communication, which involves verbal/textual, facial expressions, vocal/acoustic. bodily/postural, and symbolic/pictorial expressions, adds complexity to the task of Emotion Recogni- tion in Conversations (ERC) (Wang et al., 2022).

---

long-distance contextual information through their relational modeling capabilities. However, those models rely on fused inputs being treated as a sin- gle node in the graph (Ghosal et al., 2019; Joshi et al., 2022), which limits their ability to capture modality-specifft representations and ultimately hampers their overall performance.
We conduct ablation studies to investigate the effect of various components and modalities on CORECT for conversation understanding.
Related Works
This section presents a literature review on Multi- modal Emotion Recognition (ERC) and the appli- cation of Graph Neural Networks for ERC.
The temporal aspect of conversations is crucial, past and future utterances can significantly influ- ence the query utterance as Figure 1. The sentence "I know me neither" appears with opposing labels on different dialogues, which could be caused by sequential effects from previous or future steps. There are only a few methods that take into account the temporal aspect of conversations. MMGCN (Wei et al., 2019) represents modality-specifie fea- tures as graph nodes but overlooks the temporal factor. DAG-ERC (Shen et al., 2021b) incorpo- rates temporal information, but focuses solely on text modality. Recently, COGMEN (Joshi et al., 2022) proposes to learn contextual, inter-speaker, and intra-speaker relations, but neglects modality- specific features and partially utilizes cross-modal information by fusing all modalities' representa- tions at the input stage.
2.1 Multimodal Emotion Recognition in Conversation
The complexity of conversations, with multiple speakers, dynamic interactions, and contextual de- pendencies, presents challenges for the ERC task. There are efforts to model the conversation context in ERC, with primary focus the textual modal- ity. Several notable approaches include CMN (Hazarika et al., 2018b), DialogueGCN (Ghosal et al., 2019), COSMIC (Ghosal et al., 2020), Dia- logue (Shen et al., 2021a), DialogueCRN (Hu et al., 2021), DAG-ERC (Shen et al., 2021b).
Multimodal machine learning has gained popu- larity due to its ability address the limitations of unimodal approaches in capturing complex real- world phenomena (Baltrušaitis al., 2018). It is recognized that human perception and understand- ing are influenced by the integration of multiple sensory inputs. There have been several notable approaches that aim to harness the power of multi- ple modalities in various applications (Poria et al.. 2017; Zadeh et al., 2018 8; Majumder al., 2019), etc. CMN (Hazarika et al., 2018b) combines fea- tures from different modalities by concatenating them directly and utilizes the Gated Recurrent Unit (GRU) to model contextual information. ICON (Hazarika et al., 2018a 8a) extracts multimodal con- versation features and employs global memories to model emotional influences hierarchically, result- ing in improved performance for utterance-video emotion recognition. ConGCN (Zhang et al., 2019) models utterances and speakers as nodes in graph, capturing context dependencies and speaker depen- dencies as edges. However, ConGCN focuses only on textual and acoustic features and does not con- sider other modalities. MMGCN (Wei et al., 2019), on the other hand, is a graph convolutional net- work (GCN)-based model that effectively captures both long- distance contextual information and mul- timodal interactive information.
The aforementioned limitations motivate us propose a COnversation understanding model using RElational Temporal Graph Neural Net- work with Auxiliary Cross-Modality Interaction (CORECT). It comprises two key components: the (i) Relational Temporal Graph Convolutional Network (RT-GCN); and the (ii) Pairwise Cross- modal Feature Interaction (P-CM). The RT-GCN module is based on RGCNs (Schlichtkrull et al., 2018) and GraphTransforne while the P-CM is built upon (Tsai et al., 2019). Overall, our main contributions are as follows:
(Yun et al., 2019)
We propose the CORECT framework for Mul- timodal ERC, which concurrently exploit the utterance-level local context feature from mul- timodal interactions with temporal dependen- cies via RT-GCN, and the cross-modal global context feature at the conversation level by P-CM. These features are aggregated to en- hance the performance of the utterance-level emotional recognition.
We conduct extensive experiments to show that CORECT consistently outperforms the previous SOTA baselines on the two publicly real-life datasets, including IEMOCAP and CMU-MOSEL, for the multimodal ERC task.
More recently, Lian et al. Lian et al. (2022) propose a novel framework that combines semi-

---

supervised learning with multimodal interactions. However, it currently addresses only two modali- ties, i.e., text and audio, with visual information reserved for future work. Shi and Huang (2023) introduces MultiEMO, an attention-based multi- modal fusion framework that effectively integrates information from textual, audio and visual modal- ities. However, neither of these models addresses the temporal aspect in conversations.
to explore the relationships between modalities and context. However, it primarily focuses on modality- level interactions and does not consider the tempo- ral aspect within the graph.
Methodology
Figure illustrates the architecture of CORECT to tackle the multimodal ERC task. It consists of main components namely Relational Temporal Graph Convolution Network (RT-GCN) and Pair- wise Cross-modal Feature Interaction. For a given utterance in dialogue, the former is to learn the local-context representation via leveraging various topological relations between utterances and modal- ities, while the latter infers the cross-modal global- context representation from the whole dialogue. Given a multi-speaker conversation C consisting of N utterances [M11 M2,.... MN).. let us denote S as the respective set of speakers. Each utterance u4 is associated with three modalities, including audio (a), visual (v), and textual (!), that can be represented as uf. uy uf respectively. Using local- and global context representations, the ERC task aims to predict the label for every utterance u4 E from a set of M predefined emotional labels Y = [/ 3yvl.
2.2 Graph Neural Networks
the past few years, there has been a growing in- terest in representing non-Euclidean data as graphs. However, the complexity of graph data has pre- sented challenges for traditional neural network models. From initial research on graph neural net- works (GNNs)(Gori et al., 2005; Scarselli et al., 2008), generalizing the operations of deep neural networks were paid attention, such convolution (Kipf and Welling, 2017), recurrence (Nicolicioiu et al., 2019), and attention (Velickovic et al., 2018), to graph structures. When faced with intricate inter- dependencies between modalities, GNN is a more efficient approach to exploit the potential of mul- timodal datasets. The strength of GNNs lies in its ability capture and model intra-modal and inter- modal interactions. This flexibility makes them an appealing choice for multimodal learning tasks.
3.1 Utterance-level Feature Extraction
There have been extensive studies using the ca- pability of GNNs to model the conversations. Di- alogueGCN (Ghosal et al., 2019) models conver- sation using a directed graph with utterances as nodes and dependencies as edges, fitting it into a GCN structure. MMGCN (Wei et al., 2019) adopts an undirected graph to effectively fuse multimodal information and capture long-distance contextual and moter-modal interactions. Lian et al. (2020) proposed a GNN-based architecture for ERC that utilizes both text and speech modalities. Dialogue- CRN (Hu et al., 2021) incorporates multiturn rea- soning modules to extract and integrate emotional clues, enabling a comprehensive understanding of the conversational context from a cognitive per- spective. MTAG (Yang et al., 2021) is capable of both fusion and alignment of asynchronously dis- tributed multimodal sequential data. COGMEN (Joshi et al., 2022) uses GNN-based architecture to model complex dependencies, including local and global information in a conversation. Chen al. (2023) presents Multivariate Multi-frequeee Mul- timodal Graph Neural Network, Net for short,
Here, perform pre- pro-procsssing procedures to tract utterance-level features to facilitate the learn- ing of CORECT in the next section.
3.1.1 Unimodal Encoder
Given an utterance u4 each data modality mani- fests a view of its nature. To capture this value, we employ dedicated unimodal encoders, which gen- erate utterance-level features, namely af e Rdd xf E Rdv _/ E R& for the acoustic, visual, and lexical modalities respectively, and da, dyy dj are the dimensions of the extracted features for each modality.
For textual modality, we utilize Transformer (Vaswani et al., 2017) as the unimodal encoder to extract the semantic feature of from uf as follows:
1 Transformer (ul, W
where W trans is the parameter of Transformer to be learned.
For acoustic and visual modalities, we employ a fully-connected network as the unimodal encoder

---

Unimodal Encoder
CORECT Architecture
G1
Craph Transfoooooo
Conversation
Rocu
Emotion Classifier
y/
x
Relational Temporal Graph Convolutional Network
Tosssiic Ceccatioo
(Myy-a)
(M
Transformer
(M-PP))
Predicted
labets
CMaa-v)
Transformer
CMyr-t)
(M
Transformer
Cross-modal Interaction
Multimodal Relation
Post Relation
Future Relation
Acoustic
Visual nodes
Textual nades
Speaker Embedding
Figure Framework illustration CORECT for the multimodal emotion recognition in conversations
to extract context features for each modality type via the following procedure:
3.2 Relational Temporal Graph Convolutional Network (RT-GCN)
RT-GCN is proposed to capture local context in- formation for each utterance in the conversation via exploiting the multimodal graph between utter- ances and their modalities.
o FC(u;; Wj.. T oa
(2)
where FC is the fully connected network, Wfe E Rd, xd are trainable parameters; d' the input
dimension of modality t
3.2.1 Multimodal Graph Construction
3.1.2 Speaker Embedding
Let us denote G(V, R) as the multimodal graph built from conversations, where E, R} refers to the set of utterance nodes with the three modality types ([V)
Inspired by MMGCN (Wei et al., 2019), we lever- age the significance of speaker information. Let define Embedding as procedure that takes the identity of speakers and produce the respective latent representations. The embedding of multi- speaker could be inferred as:
N), the set of edges and their relation types. Figure provides an illustrative example of the relations represented on the con- structed graph.
Nodes. Each utterance u; generates three nodes uh u., and u/. which w.. x². and of are the respec- tive audio, visual, and lexical feature vectors.
Semb Embedding(S..N))
(3)
where Somb R. NxNss and Ns is the total num- ber of participants in the conversation. The ex- tracted utterance-level feature could be enhanced by adding the corresponding speaker embedding:
Edges. The edge (uf,uf, riy) E E, E {a, v,I} represents the interaction between uf and uz with the relation type rig C R. In the scope of paper, we consider two groups of relations: Rmmatt and Reomp- Specifically, Rmoiti represents the intra connections between the three modalities within the same utterance, reflecting multimodal interac- tions. On the other hand, Reemp captures the inter connections between utterances of the same modal- ity within a specified time window. This temporal relationship includes past/previous utterances de-
X, = S/mb + +X E (a, 1,
(4)
where X, E Nxd- refers to the global-context representation from the whole dialogue obtained from the respective unimodal encoder; X, repre- sents the enhanced representation with the inclu- sion of the speaker embedding; E [0, indicates the contribution ratio.

---

noted as P and next/future utterances denoted as F. As a result, there are 15 edge types created with the definitions of the two groups.
P-P
Multimodal Relation. Emotions in dialogues can- not be solely conveyed through lexical, acoustic, or visual modalities in isolation. The interactions between utterances across different modalities play a crucial role. For example, given an utterance in graph, its visual node has different interactive magnitude with acoustic- and textual nodes. Ad- ditionally, each node has a self-aware connection to reinforce its own information. Therefore, we can formalize 9 edge types of Rymutti to capture the multimodal interactions within the dialogue as:
Xi-P
x
X
X
Figure example construction of a graph illustrat- ing the relationship among utterance nodes represent- ing audio (square), visual (circle), and text (triangle) modalities with window size [P. [2, for query utterance i-th. The solid blue, solid red, and dashed red arrows indicate modsl- past temporal- and future temporal connections respectively.
((uf. a(),, (uf uu), uk)) [(uj, , ) , (u,, u.). P) (mf, " {(u,, uf), (uf. u (u,, )
Rmulti =
(5)
where Nii is the set of the node i's neighbors with the relation E R. Wo. W, E Rd xd- are learnable parameters (h1 is the dimension of the hidden layer used by R-GCN), and zj E IR de- notes the feature vector of node = {a, u, l). To extract rich representations from node fea- tures, we utilize a Graph Transformer model (Yun et al., 2019), where each layer comprises self- attention mechanism followed by feed-forward neu- ral networks. The self-attention mechanism allows vertices to exploit information from neighborhoods as well as capturing local and global patterns in the graph. Given 9f is the representation of 7t ut- terance with modality (a, v. obtained from RGCNs, its representation transformed into:
Temporal Relation. is vital to have distinct treatment for interactions between nodes that occur in different temporal orders (Poria et al., 2017). To capture this temporal aspect, we set window slide [P.F] to control the number of past/previous and next/future utterances that set has connection to define
current node u.. This window enables the temporal context for each node and capture the relevant information from the dynamic surrounding utterances. Therefore, we have edge types of Reemp as follows:
past { {(uj + J P <j<{}
Reemp-
(6)
future (1(1 f u <
of 1 + Σ Σa W293] (8)
where E {a, i,j E I,N: future
and pust
SCN(()
indicate the past and future relation respectively.
where W1, W2 E 12 Xdy are learned parameters (h2 is the dimension of the hidden layer used by Graph Transformer): (tt) is the set of nodes that has connections to node i; is the concatenation for C head attention; and the attention coefficient of node j. i.e., a.z. is calculated by the sofimax activation function:
3.2.2 Graph Learning
With the objective of leveraging the nuances and variations of heterogeneous interactions berween utterances and modalities the multimodal graph, we seek to employ Relational Graph Convolutional Networks (RGCN) (Schlichtkrull et al., 2018). 8). For each relation type E R, node representation is inferred via mapping function fIH W,). where
ag softmar (Wag (WagF),
(9)
Vd
W, is the weighted matrix. Aggregating all 15 edge types, the final node representation could be computed by ΣR f(H, W,).
W3. Wa E IR xda; are learned parameters. After the aggregation throughout the whole graph, we obtain new representation vectors:
To be more specific, the representation for the i-th utterance is inferred as follows:
{oi 0z . o'y ow
(10)
of - E [.(1)
W. of + Wo 2o (7)
where T E {a, v,4) indicates the corresponding audio, visual, or textual modality.
JEN-(()

---

3.3 Pairwise Cross-modal Feature Interaction
Dialogues
Utterances
Dataset
train
valid
test
train
valid
test
The cross-modal heterogeneities often elevate the difficulty of analyzing human language. Exploit- ing mross-modality interactions may help to reveal the "unaligned" nature and long-term dependen- cies across modalities. Inspired by the idea (Tsai al., 2019), design the Pairwise Cross-modal Feature Interaction (P-CM) method into our pro- posed framework for conversation understanding. A more detailed illustration of the P-CM module is presented in Appendix A.1.2
IEMOCAP
108
12
31
5,146
664
1,623
(6-way)
IEMOCAP
108
12
31
3,200
400
943
(4-way)
MOSEI
2,249
300
646
16,327
1,871
4,662
Table I: Statistics for IEMOCAP, MOSEI datasets
to stabilize the learning process and enhance the convergence of the model. LN(Z LN(Z""")) _) P cx-
Given two modalities, e.g., audio a and tex- tual I, let us denote X E Rxd. X E RNxx the respective modality-sensitive representa- tions of the whole conversation using unimodal encoders. Based the transformer architecture (Vaswani et al., 2017), we define the Queries as Qa = X"Wqs, Keys as K = =''W Kt, and Val- ues as V = Wy.. The enriched representation of X once performing cross-modal attention on the modality by the modality 1, referred to as CM IR Nxy, is computed as:
presses the transformation by the position-wil wise feed-forward block as:
LN(Z 19 P)
>0 ))S1
+b1 + b (13)
where S2 and S2g are linear projection matrices; b and by are biases.
Likewise, we can easily compute the cross- modal representation 7, x indicating that infor- mation from the modality a is transferred to the modality I. Finally, we concatenate all representa- tions at the last layer, i.c., the D-th layer, to get the final cross-modal global-context representation z For other modality pairs, z!D and ( Ura could be obtained by the similar process.
X Q- (W KJ (
CM 2a
X WiW
dk
(11)
where a is the softmax function; Woe E IRdaxdy W gei E [dxx and Wy E Rdixdy are learned parameters. The value of dq. dk, dy is the dimen- sion of queues, keys and values respectively. vd is scaling factor and d( is the feature dimension.
3.4 Multimodal Emotion Classification
The local- and global context representation re- sulted in by the RT-GCN and P-CM modules are fused together to create the final representation of the conversation:
To model the modal interactions un- aligned multimodal sequences, e.g., audio, visual, and lexical, we utilize cross-modal transformer layers. Suppose that Z is the modality- global-context representation of the whole con- versation for the modality I at the i-th layer; Zij X The enriched representation of zz") de- noted as Z by applying cross-modal attention of the modality on the modality a is computed as the following procedure:
Fusion (G, Z))
(14)
(2 = [ot oz ............. ow 6 D...
D
where E {a, u,,u, Fusion represents the catenation method. H is then fed to a fully con- nected layer to predict the emotion label y for the utterance uy:
V ReLU Po/4 + bo)
(15)
P. softmax @ b1)
(16)
0
g' argmax(p:)
(17)
Z L-a
CM (LN(Z) (1), '))
where L o are learned parameters.
LN(Z
(-1)
4 Experiments
z ( - )
PFN LN(Z (12)
This section investigate the efficacy of CORECT for the ERC task through extensive experiments comparing with state-of-the-an (SOTA) baselines.
where Z is the intermediate representation; LN is a layer normalization (Ba et al., 2016), which helps

---

JEMOCAP (6-way)
Methods
Happy
Sad
Neutral
Angry
Excited
Frustrated
Acc. (%)
w-F1 (86)
be-LSTM (Poria et al., 2017)
32.63
70.34
51.14
63.44
67.91
61,06
59.58
59.10
CMN (Hazarika al., 20188)
30.38
62.41
52.39
59.83
60.25
60.69
56.56
56.13
ICON (Hazarika et al., 2018a)
29.91
64.57
57.38
63.04
63.42
60.81
59.09
58.54
DialogueRNN (Majumder et al,, 2019)
33.18
78.80
59.21
65.28
71.86
58,91
63.40
62.75
DialogueGCN (Ghosal et al., 2019)
47.10
80.88
58.71
66.08
70.97
61.21
65.54
65.04
MMGCN (Wei ct al., 2019)
45,45
77.53
61.99
66,70
72.04
64.12
65,56
65.71
DialogueCRN (Hu et al., 2021)
51.59
74.54
62.38
67.25
73.96
59.97
65.31
65.34
COGMEN (Joshi 2022)
55.76
80.17
63,21
61.69
74.91
63.90
67.04
67.27
CORECT (Ours)
59.30
80.53
66.94
69.59
72.69
68,50
69.93 (1 2.89) 70.02 (1 2.75)
Table 2: The results on IEMOCAP (6-way) multimodal (A+V+T) setting. The results in bold indicate the highest performance, while the underlined results represent the second highest performance. The f illustrates the improvement compared to the previous state-of-the-art model.
4.1 Experimental Setup
Modality Settings
JEMOCAP (4-way)
(%)
w-FI (8)
Dataset. We investigate two public real-life datasets for the multimodal ERC task including IEMOCAP (Busso et al., 2008) and CMU-MOSEI (Bagher Zadeh et al., 2018). The dataset statistics are given in Table 1.
be-LSIM (Poria al., 2017)
75.20
75.13
CHFusion (Majumder 2018)
76.59
76.80
COGMEN (Joshi al.. 2022)
82.29
82.15
CORECT (Ours)
84.73 2.44) 84.64 (1 2.49)
Table 3:
The results on the IEMOCAP (4-way) dataset in the multimodal (A+V+T) setting. The indicates the improvement compared to the previous SOTA model.
IEMOCAP contains 12 hours of videos of two- way conversations from 10 speakers. Each dialogue is divided into utterances. There are in total 7433 utterances and 151 dialogues. The 6-way dataset contains six emotion labels, i.e., happy, sad, neu- tral, angry, excited, and frustrated, assigned the utterances. As a simplified version, ambiguous pairs such as (happy, exited) and (sad, frustrated) are merged to form the 4-way dataset.
tigate multimodal models including Multilogue- Net (Shenoy and Sardana, 2020), TBJE (Delbrouck et al., 2020), and COGMEN (Joshi et al., 2022).
Implementation Details. Due to the space limit, the implementation details for feature extraction and interaction are described in Appendix A.I.
CMU-MOSEI provides annotations for 7 senti- ments ranging from highly negative (-3) to highly positive (+3), and 6 emotion labels including hap- piness, sadness, disgust, fear, surprise, and anger.
4.2 Comparison With Baselines
We further qualitatively analyze CORECT and the basclines the IEMOCAP (4-way), IEMOCAP (6-way) and MOSEI datasets.
Evaluation Metrics. We use weighted F1-score (w-F1) and Accuracy (Acc.) evaluation metrics. The w-Fl computed Σk= freq. x Flg, where freqe the relative frequency of class The accuracy is defined as the percentage correct predictions in the test set.
In the case of IEMOCAP (6-way) dataset (Table 2), CORECT performs better than the previous basclines in terms of Fl score for in- dividual labels, excepts the Sad and the Excited labels. The reason could be the ambiguity between similar emotions, such as Happy & Excited, as well as Sad & Frustrated (see more details in Figure 6 in Appendix A.2). Nevertheless, the accuracy and weighted F1 score of CORECT are 2.89% and 2.75% higher than all baseline models average. Likewise, we observe the similar phenomena on the IEMOCAP (4-way) dataset with 2.49% im- provement over the previous state-of-the-an mod- els as Table These results affirm the efficiency of CORECT for the multimodal ERC task.
IEMOCAP:
CORECT compared against SOTA baselines specific to each dataset. For TEMO- CAP, we consider two model groups namely: i) RNN-based models include bc-LSTM (Poria et al., 2017), CMN (Hazarika et al., 201 1b), ICON (Haz- arika et al., 2018a), DialogueRNN (Majumder al., 2019); ii) Graph-based methods are Dia- logueGCN (Ghosal al., 2019), MMGCN (Wei et al., 2019), DialougueCRN (Hu et al., 2021), CHFusion (Majumder et al., 2018), and COGMEN (Joshi et al,, 2022). For CMU-MOSEI, we inves-
Baseline Models.

---

Sentiment Classilcation
Emotion Classification (Binary, vs. all)
Methods
Accuracy (%)
weighted Fl-score (%)
Class
Class
Happiness
Sadmess
Angry
Fear
Disgusi
Surprise
Multiluge-Nel (Shenoy and Sardana, 2020)
82,88
44.83
67.84
65.34
67.03
87.79
74,91
86.05
TBJE (Delbrouck ct al., 2020)
82.40
43.91
65.91
70.78
70.86
87.79
82.57
86.04
COGMEN (Joshi et al., 2022)
82,95
45.22
70.88
70.91
74.20
87.79
81.83
86.05
CORECT (Ours)
83.66
46.31
71.35
72.86
76.77
87.90
84.26
86.48
Table 4: Results CMU-MOSEI dataset compared with previous works. The bolded results indicate the best performance, while the underlined results represent the second best performance.
IEMOCAP
IRMOCAP
Sub-
(6-way)
(4-way)
Modules
Acc. (%)
w-F1 (%)
Acc. (%)
w-F1 (%)
-w/o RT-GCN
66.61
66.55 (1 3.47)
80.69
80.54 (1 4.10)
-w/o P-CM
66.54
66.64 () 3.38)
82.18
82,16 () 2.48)
-wfo Renstt
66.54
66,82 (1 3.20)
82.61
82.53 () 2.11)
-w/O Rtemp
67.04
67.34 (1, 2.68)
82.08
82.07 () 2.57)
CORECT 69.93
70.02
84.73
84.64
Table 5: The performance of CORECT in different strategies under the fully multimodal (A+V+T) setting. Bolded results represent the best performance, while underlined results depict the second best. The & represents the decrease in performance when specific module ablated compared CORECT model.
Table presents a comparison of the CORECT model on the CMU-MOSEI dataset with current SOTA models in two settings: Senti- ment Classification (2-class and 7-class) and Emo- tion Classification. Apparently, CORECT consis- tently outperforms other models with sustainable improvements. One notable observation is the itali- cized results for the Fear and Surprise labels, where all the baselines have the same performance of 87.79 and 86.05 respectively. During the experi- mental process, when reproducing these baseline's results, we found that the binary classifiers were unable to distinguish any samples for the Fear and Surprise labels. However, with the help of technical components, i.e., RT-GCN and P-CM, our model shows significant improvement even in the pres- ence severe label imbalance in the dataset. Due to space limitations in the paper, we provide addi- tional experiments on the CMU-MOSEI dataset for all possible combinations of modalities in Table 7 (Appendix A.2).
CMU-MOSEI:
the 4-way IEMOCAP dataset. Therefore, we can deduce that the effect of RT-GCN in the CORECT model is more significant than that of P-CM. For different relation types, ablating either Ryulis Rtemp results in significant decrease in the performance. However, the number of labels may affect on the multimodal graph construction, thus it is no easy to distinguish the importance of Rmuttt and Rtemp for the multimodal ERC task.
Table (Appendix A.2) presents the ablation results for uni- and bi-modal combinations. In the unimodal settings, specifically for each indi- vidual modality (A, V, T), it's important to high- light that both P-CM module and multimodal re- lations Rmaiti are non-existent. However, in bi- modal combinations, the advantage of leveraging cross-modality information between audio and text (A+T) stands out, with a significant performance boost of over 2.75% compared to text and visual (T+V) modalities and a substantial 14.54% com- pared to visual and audio (V+A) modalities. Additionally,
experiments have shown slight drop in overall model performance (e.g.. 68.32% in IEMOCAP 6-way, drop of 1.70%) when excluding Speaker Embedding Semb from CORECT.
4.3 Ablation study
Effect of Main Components. The impact of main components in our CORECT model is pre- sented via Table 5. The model performance on the 6-way IEMOCAP dataset remarkably de- graded when the RT-GCN or P-CM module is not adopted with the decrease by 3.47% and 3.38% respectively. Similar phenomena observed on
Effect of the Past and Future Utterance Nodes. We conduct an analysis to investigate the influ- ence of past nodes (P) and future nodes (F) on the model's performance. Unlike previous studies

---

JEMOCAP
JEMOCAP
ing the visual and acoustic modalities produces the
Modality Settings
(6-way)
(4-way)
worst result. A similar trend is observed in the
(%6)
w-F1 (%)
(%6)
w-FI
A
$2.31
$1.49
67.02
65.48
CMU-MOSEI dataset (Table 4), where fusing all modalities together leads to a better result com-
T
67.22
67.26
82.82
82.65
V
38,63
37.67
49.73
47.97
A+T
68.27
68.36
83.14
83.13
pared to using individual or paired modalities.
T+Y
65.50
63.61
81.76
81.75
V+A
Conclusion
CORECT (A+T+V)
In this work, we propose CORECT, novel net- work architecture for multimodal ERC. consists of two main components including RT-GCN and P-CM. The former helps to learn local-context representations by leveraging modality-level topo- logical relations while the latter supports to infer cross-modal global-context representations from the entire dialogue. Extensive experiments two popular benchmark datasets, i.e., IEMOCAP and CMU-MOSEI, demonstrate the effectiveness of CORECT, which achieves the new state-of-the- art record for multimodal conversational emotion recognition. Furthermore, we also provide ablation studies to investigate the contribution of various components in CORECT. Interestingly, by analyz- ing the temporal aspect of conversations, we have validated that capturing the long-term dependen- cies, e.g.. past relation, improves the performance of the multimodal emotion recognition in conversa- tions task.
Table The performance CORECT under various modality settings.
70.0
69.5
69.0
Future Window
Weightas FT
68.0
67.5
67.0
66.5
66.0
Past Window
Figure 4: The effects P and FF nodes in the past and future of CORECT model on the IEMOCAP (6- (-a) The red-dash line implies our best setting for P and F.
Limitations
(Joshi et al., 2022; Li et al., 2023) that treated P and F pairs equally, we explore various combina- tions of P and F settings to determine their effects. Figure 4 indicates that the number of past future nodes have different impacts on the perfor- mance. From the empirical analysis, the setting [P.F] of [11,9] results the best performance. This finding shows that the contextual information from the past has a stronger influence on the multi- modal ERC task compared to the future context.
Hyper-parametee tuning is vital part of optimiz- ing machine learning models. Not exception, the learning of CORECT is affected by hyper- parameters such as the number of attention head in P-CM module, the size of Future and Past Win- dow. Due to time constraints and limited compu- tational resources, was not possible to tune or exploring all possible combinations of these hyper- parameters, which might lead to local-minima con- vergences. In future, one solution for this limitation is to employ automated syper-parametee optimiza- tion algorithms, systematically explore the hy- perparameter space and improve the robustness of the model. As another solution, we may upgrade CORECT with learning mechanisms to automati- cally leverage important information, c.g., attention mechanism on future and past utterances.
Effect of Modality. Table 6 presents the perfor- mance of the CORECT model in different modality combinations on both the IEMOCAP and CMU- MOSEI datasets.
For IEMOCAP (Table 2 and Table 3), the tex- tual modality performs the best among the uni- modal settings, while the visual modality yields the lowest results. This be attributed to the presence of noise caused by factors, e.g., camera position, environmental conditions. In the bi-modal settings, combining the textual and acoustic modal- ities achieves the best performance, while combin-
Acknowledgements
Cam-Van Thi Nguyen was funded by the Master, PhD Scholarship Programme of Vingroup Innova- tion Foundation (VINIF), code VINIF.2022 1143.

---

References
for emotion recognition in conversation. Proceed- ings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th Inter- national Joint Conference on Natural Language Pro- cessing (EMNLP-IJCNL"), pages 154-164, Hong Kong, China. Association for Computational Lin- guistics.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hin- ton. 2016. Layer normalization. arXiv preprint arXiv:1607006450
AmirAli Bagher Zadeh, Paul Pu Liang, Soujanya Poria, Brik Cambria, and Louis- Phiis-Philippe Morency. 2018. Multimodal language analysis in the wild: CMU- MOSEI dataset and interpretable dynamic fusion graph. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Vol- ume 1: Long Papers), pages 2236-2246, Melbourne, Australia. Association for Computational Linguistics.
Marco Gori, Gabriele Monfardini, and Franco Scarselli. 2005. A new model for learning in graph domains. In Proceedings. 2005 IEEE International Joint Con- ference on Neural Networks, 2005., volume 2. pages -34. IEEE
Devamanyu Hazarika, Soujanya Poria, Rada Mihal- cea, Erik Cambria, and Roger Zimmermann. 2018a. ICON: Interactive conversational memory network for multimodal emotion detection. In Proceedings of the 2018 Conference on Empirical Methods in Nat- ural Language Processing, pages 2594-2604, Brus- sels, Belgium. Association for Computational Lin- guistics.
Tadas Baltrušaitis, Chaitanya Ahuja, and Louis- Phiil-phi Morency. 2018. Multimodal machine learning: A survey and taxonomy. IEEE transactions on pattern analysis and machine intelligence, 41244:223444.
Tadas Baltrusaitis, Amir Zadeh, Yao Chong Lim, and Louis-Philippe Morency. 2018. Openface 2.0: Fa- cial behavior analysis toolkit. In 2018 13th IEEE international conference automatic face & ges- ture recognition (FG 2018), pages 59-66. IERE.
Devamanyu Hazarika, Soujanya Poria, Amir Zadeh, Erik Cambria, Louis Phiis-piilippe Morency, and Roger Zimmermann. Sb. Conversational memory net- work for emotion recognition in dyadic dialogue videos. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Tech- nologies, Volume I (Long Papers), pages 2122-2132, New Orleans, Louisiana. Association for Computa- tional I Linguistics.
Carlos Busso, Murtaza Bulut, Chi-Chun Lee, Abe Kazemzadeh, Emily Mower, Samuel Kim, Jean- nette N Chang, Sungbok Lee, and Shrikanth Narayanan. 2008. lemocap: Interactive emotional dyadic motion capture database. Language resources and evaluation, 42(4)3335-359.
Feiyu Chen, Jie Shao, Shuyuan Zhu, and Heng Tao Shen. 2023, Multivariate, multi-frequency and multi- modal: Rethinking graph neural networks for emo- tion recognition in conversation. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages 10761-10770.
Dou Hu, Lingwei Wei, and Xiaoyong Huai. 2021. Dia- logueCRN: Contextual reasoning networks for emo- tion recognition in conversations. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the I 11th International Joini Conference on Natural Language Processing (Volume 1: Long Papers), pages 7042-7052, Online. Association for Computational Linguisties.
Lean-Benoin Delbrouck, Noé Tits, Mathilde Brousmiche, and Stéphane Dupont. 2020. A transformer-basee joint-encoding for emotion recognition and sentiment analysis. In Second Grand- Chaalle-eeeeeee and Workshop on Multimodal Language (Challenge-IIM)) IIMM,, pages 1-7, Seattle, USA. Association for Computational Linguistics.
Abhinav Joshi, Ashwani Bhat, Ayush Jain, Atin Singh, and Ashutosh Modi. 2022. COGMEN: COntextual- ized GNN based multimodal emotion recognitioN. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, pages 4148-4164, Seattle, United States. Association for Computational Linguistics.
Florian Eyben, Martin Wöllmer, and Björn Schuller. 2010. Opensmile: the munich versatile and fast open- source audio feature extractor. In Proceedings of the 18th ACM international conference Multimedia, pages 1459-1462.
Thomas N. Kipf and Max Welling. 2017. Semi- supervised classification with graph convolutional networks. In International Conference on Learning Representations (ICLR).
Deepanway Ghosal, Navonil Majumder, Alexander Gel- bukh, Rada Mihalcea. and Soujanya Poria. 2020. COSMIC: COmmonSense knowledge for eMotion identification in conversations. In Findings of the As- sociation for Computational Linguistics: EMNLP 2020, pages 2470-2481. Online. Association for Computational Linguistics.
Jiang Li, Xiaoping Wang, Guoqing Lv, and Zhigang Zeng. 2023. Graphmft: A graph network based mul- timodal fusion technique for emotion recognition in conversation.
Neurocomputing, page 126427.
Zheng Lian, Bin Liu, and Jianhua Tao. 2022. Smin: Semi-supervised molti-modal interaction network for conversational emotion recognition. IEEE Transac- tions on Affective Computing.
Deepanway Ghosal, Navonil Majumder, Soujanya Poria, Niyati Chhaya, and Alexander Gelbukh. 2019. Di- alogueGCN: A graph convolutional neural network

---

Zheng Lian, Jianhua Tao, Bin Liu, Jian Huang, Zhanlei Yang, and Rongjun Li. 2020. Conversational emo- tion recognition using self-attention mechanisms and graph neural networks. In INTERSPEECH, pages 2347-2351.
Garima Sharma and Abhinav Dhall. 2021. A survey on automatic multimdall emotion recognition in the wild, Advances in data science: Methodologies and applications, pages 35-64.
Weizhou Shen, Junqing Chen, Xiaojun Quan, and Zhix- ian Xie. 2021a. Dialogxl: All-in-one xlnet for multi- party conversation emotion recognition. In Proceed- ings the AAAI Conference on Artificial Intelligence. volume 33, pages 13789-1328 13797.
Navonil Majumder. Devamanyu Hazarika, Alexander Gelbukh, Erik Cambria, and Soujanya Poria. 2018. Multimodal sentiment analysis using hierarchical fu- sion with context modeling. Knowledge-baad sys- lems, 161: 1::1:224111 133.
Weizhou Shen, Siyue Wu, Yunyi Yang, and Xiaojun Quan. 202 Directed acyclic graph network for conversational emotion recognition. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 1 11th International Joint Conference on Natural Language Processing ( (Volume 1: Long Papers), pages 1551-1566 Online. Association for Computational Linguisties.
Navonil Majumder, Soujanya Poria, Devamanyu Haz- arika, Rada Mihalcca, Alexander Gelbukh, and Erik Cambria. 2019. Dialoguernn: An attentive mn for emotion detection conversations. In Proceedings of the AAAT conference on artificial intelligence, vol- ume 33, pages 6818 68255.
Brian McFee, Colin Raffel, Dawen Liang, Daniel Ellis, Matt McVicar, Eric Battenberg. and Nieto. 2015. librosa: Audio and music signal analysis in python. In Proceedings of the 14th python in science conference, volume 8, pages 18-25.
Aman Shenoy and Ashish Sardana. 2020. Multilogue- net: context-award RNN for multi- emotion detection and sentiment analysis in conversation. In Second Grand-Challeny and Workshop on Multi- modal Language (Challenge-Haae HMLL) pages 19-28. Seattle, USA. Association for Computational Lin- guistics.
Andrei Nicolicioiu, Iulia Duta, and Marius Leordeanu. 2019. Recurrent space-time graph neural networks. Advances in neural information processing systems, 32.
Tao Shi and Shao-Lun Huang. 2023. MultiEMO: An attention- based correlation aware multimodal fusion framework for emotion recognition in conversations. In Proceedings of the st Annual Meeting of the Association for Computational Linguistics (Volume I: Long Pupers), pages 14752-14766, Toronto, Canada. Association for Computational Linguisties.
Soujanya Poria, Erik Cambria, Devamanyu Hazarika, Navonil Majumder, Amir Zadeh, and Phiiii-pi Morency. 2017. Context-dependent sentiment anal- ysis in user-generated videos. In Proceedings of the 55th Annual Meeting of the Association for Compu- tational Linguistics /Volume 1: Long Papers), pages 873-883, Vancouver, Canada. Association for Com- putational Linguisties.
Yao-Hung Hubert Tsai, Shaojie Bai, Paul Pu Liang, J. Zico Kolter, Louis Chiis-Phllippe Morency, and Rus- lan Salakhutdinov. 2019. Multimodal transformer for unaligned multimodal language sequences. In Proceedings of the 57th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 6558- 6569, Florence, Italy. Association for Computational Linguistics.
Soujanya Poria, Navonil Majumder, Rada Mihalcea, and Eduard Hovy. 2019, Emotion recognition in con- versation: Research challenges. datasets, and recent advances. IEEE Access, 7: 7:1009933100953.
Nils Reimers and Iryna Gurevych. 2019. Sentence- BERT: Sentence embeddings using Siamese BERT- networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natu- ral Language Processing (EMNLP-IJCNLPP pages 3982-3992, Hong Kong, China. Association for Com- putational Linguistics.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need, Advances neural information processing systems, 30.
Franco Scarselli, Marco Gori, Ah Chung Tsoi, Markus Hagenbuchner, and Gabriele Monfardini. 2008. The graph neural network model. IEEE transactions on neural networks, 20( 1--1-000
Petar Velickovic, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, and Yoshua Bengio. 2018. Graph attention networks. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30- May 3, 2018, Conference Track Proceedings. Openkeview.net.
Michael Schlichtkrull, Thomas N Kipf, Peter Bloem, Rianne Van Den Berg, Ivan Titov, and Max Welling. 2018. Modeling relational data with graph convolu- tional networks. In The Semantic Web: 15th Inter- national Conference, ESWC 2018, Heraklion, Crete, Greece, June 3-7, 2018, Proceedings 15, pages 593- 607. Springer.
Yan Wang, Wei Song, Wei Tao, Antonio Liotta. Dawei Yang, Xinlei Li, Shuyong Gao, Yixuan Sun, Weifeng Ge. Wei Zhang, et al. 2022. A systematic review on affective computing: Emotion models, databases, and recent advances. Information Fusion.

---

Yinwei Wei, Xiang Wang, Liqiang Nie. Xiangnan He, Richang Hong, and Tat-Seng Chua. 2019. Mmgen: Multi-modal graph convolution network for person- alized recommendation of micro-video. In Proceed- ings of the 27th ACM international conference on mullimedia, pages 1437-1445.
Jianing Yang. Yongxin Wang. Ruitao Yi, Yuying Zhu, Azaan Rehman, Amir Zadeh, Soujanya Poria, and Louis-Philiippe Morency. 2021. MTAG: Modal- temporal attention graph for unaligned human mul- timodal language sequences. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Hu- man Language Technologies, pages 1009-1021, On- line. Association for Computational Linguistics.
Seongjun Yun, Minbyul Jeong. Raehyun Kim, Jaewoo Kang. and Hyunwoo J. Kim. 2019. Graph trans- former networks. CoRR, abs/1911.06455.
Amir Zadeh, Paul Pu Liang, Navonil Mazumder, Soujanya Poria, Erik Cambria, and Louis-Philippe Morency. 2018. Memory fusion network for multi- view sequential learning. In Proceedings of AAAI conference on artificial intelligence, volume 32.
Dong Zhang, Liangqing Wu, Changlong Sun, Shoushan Li, Qiaoming Zhu, and Guodong Zhou. 2019. Mod- eling both context-and speaker-sensitive dependence for emotion detection in multi-speaker conversations. In IJCAI, pages 5415-5421.

---

A Appendix
combinations of the CORECT model, while Ta- ble presents an ablation study conducted on the CMU-MOSEI dataset, considering various modal- ity combination settings.
A.1
Implementation Details
A.1.1 Multimodal Raw Feature Extraction
The multimodal feature extraction process involves extracting features from the acoustie, lexical, and visual modalities for each utterance.
Figure 6 shows the confusion matrix for pre- diction on IEMOCAP (4-way) and IEMOCAP (6- way), respectively.
For IEMOCAP, the audio features, with size of 100, are obtained using the OpenSmile Toolkit (Eyben et al., 2010); visual features, with size of 512, extracted using OpenFace (Baltrusaitis et al., 2018); textual features, with size of 768, are derived using SBERT (Reimers and Gurevych, 2019).
Happy-
116
300
250
Actual
Sad
200
350
Noutral
120
For MOSEL, the audio features are extracted using librosa (McFee et al., 2015) with 80 filter banks, resulting in feature vector size of 80.The visual features, with size of 35, are obtained from (Bagher Zadeh et al., 2018). The textual fea- tures, with size of 768, obtained using sBERT (Reimers and Gurevych, 2019).
Angry-
19
21
129
Happy
Sad
Nautral
Angry
Predicted
Confusion matrix on the IEMOCAP way).
Happy-
102
0
26
250
200
A.1.2 Pairwise Cross-modal Feature Interaction
Actual
Noutral 19
247
15
10
150
Figure 5 illustrates details of Pairwise Cross-modal Feature Interaction (P-CM).
Angry-
0
139
0
35
Excited
Frustrated
26
37
261
TransFormer (f
Cross-Modality
Happy
Sad
Neutral
Angry
Exited
Pnustrated
Predicted
Addition
(b) Confusion matrix on the IEMOCAP (6-way).
Posiloowiis Feed-torward
Visualization the confusion matrices CORECT under multimodal (A+V+T) setting. Most of False predictions observed on IEMOCAP (6-way) came from the ambiguity between pair of labels: Happy and Excited, Neural and Frustrule.
Figure 6:
LayenVorm
ddition
CM (21--1 yo
Leyers
Mulli- beasl Crucs modal
A llention
Qc
A.3 Reproducibility
LayerNoom
LaystNoom
CORECT is implemented using Pytorch 1 and run experiments on Google Colab Pro. We choose Adam as the optimizer and the dropout rate
M
0.5. The numbers of multi-head attentions used in Graph Transformer and P-CM are selected as and 2, respectively. For LEMOCAP dataset, the learn- ing rate is 0,0003; Window size (P,F) is tested on various settings in the range of [1, [1,15]. For CMU- MOSEI dataset, the lcarning rate is 0.0006; Win- dow size [P,F] F is picked between [5,4] due to the property of short dialogue in CMU-MOSEL Refer-
Figure Musttraion of the P-CM module between modality 3 and a.
A.2 Additional Experiment Result
Table showcases the results on the IEMOCAP dataset (both 6-way and 4-way) for all the modality
'htps://pytorch.org/

---

Sentiment Class Accuracy (%)
Emotion Class
Modality Settings
Datasets
weighted Fi-score (%)
2 Class
Class
Happiness
Sadness
Angry
Fear
Disgust
Surprise
MultilgueeNett
A+T+V
82.88
44.83
67.84
65.34
67.03
87.79
74.91
86.05
(Shenoy and Sardana, 2020)
TBJE (Delhrouck et al., 2020)
A+T
82.4
43.91
65.91
70.78
70.86
87.79
82.57
86.04
COGMEN
A+T+V
82.95
45.22
70.88
70.91
74.20
87.79
81.83
86.05
(Joshi 2022)
84.13
45.80
67.82
72.12
75.55
87.79
84.63
86,
CORECT (Ours)
A+T
84.28
44.89
67.49
71.53
75.39
87.79
84,69
86.05
A+T+V
83.66
46.31
71.35
72.86
76.77
87.90
84.26
86.48
Table Ablation study on CMU-MOSEI dataset. The multimodal implementation (A+V+T) consistently outper- formed the baseline models in most of the modality combinations. For the 2-class sentiment and the Disgust class emotion, our approach reaches a competitive performance.
A+T
T+V
V+A
A+VaT
Apc
W-FI
Ace
W-FI
Ace
W-FI
Ace
W-FI
Acc
F1
Acc
W-FI
Acc
w/O RT GCN
35.12
30.01
64.7
64,34
30.99
26.88
67.10
66.92
65.37
65.50
52.13
51.80
66.61
66.55
65.00
53.54
66.54
IEMOCAP way)
w/o Rusii
66.30
66.27
64.76
64.78
53.67
53.48
66.54
66.82
w/o Reame
41.53
39.49
63.65
63.72
27.66
27.37
67.34
67.33
65.43
65.29
50.65
49.67
67.04
67.34
CORECT
52.31
51.49
67.22
67.26
38,63
37.67
68.27
68.36
65.50
65.61
54.16
53.82
69.93
70.02
w/o KT-GCN
55.25
52.18
80.38
80.25
34.04
31.33
81.87
81.18
80.17
80.04
58.96
58.57
80 69
80.54
80.38
69.25
IEMOCAP
Russit
81.76
(4-wyy)
w/o Reomo
56.84
54.88
80,70
80,70
41,04
39.75
82,08
81.99
81.34
81,36
57.16
56.62
82.08
82.07
CORECT
67.02
65.48
82.82
82.85
49.73
47.97
83.14
83.13
81.76
81.75
69.03
68.21
84.73
84.64
Table 8: Ablation study on IEMOCAP dataset. It shows the consistency of our proposal method since any ablation experiments (on both modal settings and modules) results in a reduction overall performance. On unimodal setting of {A, T). P-CM module and multimodal relation are exist. Therefore there ablation of P-CM and Rmuttt on these unimodal setting, denotes by" "-".
ring the training log on the IEMOCAP (6-way) dataset using Google Colab Pro, each mini-batch (size of 10 dialouges) takes approximately 0.4s. The similar ratio is observed on the MOSEI dataset.


Reviews:
[{"review_id": "00z8vlQGyo", "reviewer": "Reviewer_bUo5", "summary": "", "questions": "1. In the unimodal encoder, why is the text modality implemented with a transformer, while the other two modalities use FC (fully connected layers)? What is the underlying rationale for this inconsistency?\n\n2. Is the speaker embedding represented as a one-hot encoding? If not, how is it generated? Additionally, is it a necessary component? Is there a significant performance drop if the speaker embedding is not used?\n\n3. In Table 5, why does the combination of text and visual modalities (T+V) perform worse than using the text modality (T) alone?\n", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to express our heartfelt gratitude to the reviewer for taking the time to provide such insightful and valuable comments on our work. Your feedback has been incredibly helpful in guiding us towards improvements for our shortcomings. We hope that our responses would better clarify what we do and contribute to the research problem. We would like to address your concerns with the following responses: \n\n### I. Questions: \n\n**Q1.** In the unimodal encoder, why is the text modality implemented with a transformer, while the other two modalities use FC (fully connected layers)? What is the underlying rationale for this inconsistency?  \n\n**Answer**: Thank you for your interesting question. We adopt similar multimodal encoders as our state-of-the-art baseline, e.g., COGMEN [1], which uses the transformer-based encoder to capture the semantic context of each utterance in the textual modality and fully connected (FC) components for other modalities. The encoding strategy shows an efficacy for the multimodal emotion recognition (ERC) task.   \n\n**Q2.** Is the speaker embedding represented as a one-hot encoding? If not, how is it generated? Additionally, is it a necessary component? Is there a significant performance drop if the speaker embedding is not used?  \n\n**Answer**: Yes, the speaker embedding is a one-hot encoding of the speaker's identity within the conversation. Our experiments have shown a slight drop in overall model performance (e.g., 68.32% in IEMOCAP 6-way, drop of 1.70%) when excluding this embedding. Due to the space limitation, we have not shown this result in the current version. We will update it in the final version. \n\n**Q3.** In Table 5, why does the combination of text and visual modalities (T+V) perform worse than using the text modality (T) alone?\n\n**Answer**: As we mentioned in Section 4.3 about “Effects of Modality”, the reason could be that the visual modality contains noise, e.g., camera position or environmental conditions, which might negatively impact overall performance. Another factor could be the challenge of effectively integrating both modalities due to the complexity of aligning and fusing different types of data. \n\n\n### II. Suggestions: \n\n**S1.** Lack of complexity analysis for the model ... It is essential for the authors to assess the computational efficiency of this framework and compare it with the baseline to demonstrate the practical applicability of this method in real-world scenarios.\"\n\n**Answer**: Thank you for suggesting us to demonstrate computational efficiency of our framework CORECT. Referring to the trainining log on the IEMOCAP (6-way) dataset using Google Colab Pro, each mini-batch (size of 10 dialouges) takes approximately 0.3s and 0.4s for COGMEN [1] and CORECT respectively. The similar ratio is observed on the MOSEI dataset. We will update this comparison in the Reproducibility section of our final version. \n\n**S2**. In the introduction, it would be helpful to include a figure to provide a more visual representation of the article's motivation \n\n**Answer**: Thank you for your suggestion, we will update it in the final version. \n\n**S3**. Additionally, it is recommended to include some preliminary validation experiments to support the author's argument that modality-specific representations are superior to fused representation \n\n**Answer**: As demonstrated in Section 4.2, our model CORECT consistently outperform baseline models, including COGMEN [1] which relies on fused features as input. In our model, if fused features are used as input, P-CM module will be disable. The impact of removing the P-CM module has been explored and discussed in Section 4.3 within the \"Effect of Main Components\" subsection. The results demonstrate that the absence of the P-CM module indeed affects the overall model performance, leading to a reduction of 3.38% in IEMOCAP (6-way) and 2.48% in IEMOCAP (4-way). \n\n**S4**. Consider incorporating some case studies and visualizations of intermediate results in P-CM to aid readers in better understanding why the proposed method can enhance performance. \n\n**Answer**: Thank you for your great suggestion. We already run an ablation analysis on the P-CM module and consistently observe that P-CM outperforms its ablation counterparts. These findings will be included in the final version of our paper. \n\n### III. Typos Grammar Style And Presentation Improvements: \nTypos: Line 377, \"difficulty of of\" should be \"difficulty of\"; Line 388, \"are\" should be removed; Line 391, \"Querys\" should be \"Queries\" Presentation Improvements: \n\n**Answer**: Thank you for pointing out these typos. We will fix them in our final version.  \n\n### IV. References: \n\n[1] Joshi, A., Bhat, A., Jain, A., Singh, A., & Modi, A. (2022, July). COGMEN: COntextualized GNN-based Multimodal Emotion Recognition. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (pp. 4148-4164)", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The motivation is clear, and the proposed CORECT seems to be a reasonable solution.\n2. The experiments are extensive, comparing with various previous works. The ablation study sufficiently demonstrates the effectiveness of RT-GCN and P-CM, and the article also examines the impact under different modalities.\n3. Well-written and easy to follow. The literature review is sufficiently systematic and comprehensive.", "reasons_to_reject": "Lack of complexity analysis for the model. Overall, CORECT is a rather complex framework, incorporating various neural network components such as Transformer and GNN. It is essential for the authors to assess the computational efficiency of this framework and compare it with the baseline to demonstrate the practical applicability of this method in real-world scenarios.", "questions_to_authors": "1. In the unimodal encoder, why is the text modality implemented with a transformer, while the other two modalities use FC (fully connected layers)? What is the underlying rationale for this inconsistency?\n\n2. Is the speaker embedding represented as a one-hot encoding? If not, how is it generated? Additionally, is it a necessary component? Is there a significant performance drop if the speaker embedding is not used?\n\n3. In Table 5, why does the combination of text and visual modalities (T+V) perform worse than using the text modality (T) alone?\n", "paper_topic_and_main_contributions": "The authors design a novel framework, called CORECT, for addressing Multimodal Emotion Recognition in Conversations (multimodal ERC). CORECT consists of two core components, namely RT-GCN and P-CM. RT-GCN is devised to capture utterance-level local context features, whereas P-CM is proposed for extracting cross-modal global context features at the conversation level. These two kinds of features are then fused to boost the performance of the utterance-level emotional recognition.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "The literature review is sufficiently systematic and comprehensive."}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "Typos:\nLine 377, \"difficulty of of\" should be \"difficulty of\";\nLine 388, \"are\" should be removed;\nLine 391, \"Querys\" should be \"Queries\"\nPresentation Improvements:\n1. In the introduction, it would be helpful to include a figure to provide a more visual representation of the article's motivation. Additionally, it is recommended to include some preliminary validation experiments to support the author's argument that modality-specific representations are superior to fused representation.\n\n2. Consider incorporating some case studies and visualizations of intermediate results in P-CM to aid readers in better understanding why the proposed method can enhance performance."}}, {"review_id": "K2VDDrQnKj", "reviewer": "Reviewer_YYyo", "summary": "", "questions": "Same as the Reasons To Reject.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We extend our sincere appreciation to the reviewer for dedicating time to offer perceptive and precious feedback on our work. Your comments are beneficial in directing us to enhance the quality of our research. We would like to provide responses for your concerns as follows: \n### I. Questions:\n**Q1.** The evaluation is conducted on publicly available datasets (IEMOCAP and CMU-MOSEI), which have been extensively used in previous research. While these datasets are valuable resources, using them alone may not be sufficient to fully validate the proposed model's effectiveness and generalization to other domains or real-world scenarios.\n\n**Answer:** Thank you for highlighting this concern. As mentioned in Section 4.1, IEMOCAP and CMU-MOSEI are popular benchmark datasets for the ERC task. We intend to further investigate the effectiveness of CORECT in other contexts and domains as part of our future research.  \n\n**Q2.** The paper briefly mentions the limitations of previous works as motivation for proposing CORECT.  However, it does not adequately discuss the limitations of the proposed model itself. And the limitation section seems to be added for a short time before the submission deadline. \n\n**Answer:** Thank you for your comment. We will improve the discussion on the limitation of CORECT in our final version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper identifies and addresses the limitations of existing methods for multimodal ERC. It introduces the CORECT framework, which effectively captures both temporal dependencies and cross-modal interactions in conversations, thereby enhancing the overall performance of emotion recognition.\n\n2. The paper conducts extensive experiments on real-life datasets (IEMOCAP and CMU-MOSEI) to evaluate the performance of CORECT. The results demonstrate that the proposed model consistently outperforms state-of-the-art baselines, highlighting its effectiveness.\n\n3. The paper is well-structured and presents the proposed framework and experimental results in a clear and concise manner. The authors effectively convey the motivation, methodology, and results, making it easy for readers to understand the contributions and implications of the research.", "reasons_to_reject": "1. The evaluation is conducted on publicly available datasets (IEMOCAP and CMU-MOSEI), which have been extensively used in previous research. While these datasets are valuable resources, using them alone may not be sufficient to fully validate the proposed model's effectiveness and generalization to other domains or real-world scenarios.\n\n2. The paper briefly mentions the limitations of previous works as motivation for proposing CORECT. However, it does not adequately discuss the limitations of the proposed model itself. And the limitation section seems to be added for a short time before the submission deadline.", "questions_to_authors": "Same as the Reasons To Reject.", "paper_topic_and_main_contributions": "The paper focuses on the task of Emotion Recognition in Conversations (ERC), particularly in a multimodal context, where emotions are recognized from text content, facial expressions, and audio signals in a conversation. \n\nThe main contributions of the paper lie in proposing the CORECT framework, which effectively captures temporal dependencies and cross-modal interactions in multimodal ERC tasks, and demonstrates its superiority over existing approaches through comprehensive experiments on real-world datasets. The research addresses important challenges in understanding emotions in conversations, particularly in the context of diverse modalities, and contributes to advancements in multimodal emotion recognition.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "7CLp8eX1Cu", "reviewer": "Reviewer_Xeu3", "summary": "", "questions": "Question A: Why not compare with some advanced baselines, and what criteria were used to select these baselines?\n\n**I lean towards giving a rating of 2.5 on Excitement.** The performance of the method doesn't seem significantly superior, as observed in various other approaches listed below. Additionally, the paper's contributions appear rather limited, primarily focusing on refining the COGMEN model.\n\n*Shi, T., & Huang, S. L. (2023). MultiEMO: An Attention-Based Correlation-Aware Multimodal Fusion Framework for Emotion Recognition in Conversations. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (pp. 14752-14766).*  \n*Chen, F., Shao, J., Zhu, S., & Shen, H. T. (2023). Multivariate, Multi-Frequency and Multimodal: Rethinking Graph Neural Networks for Emotion Recognition in Conversation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 10761-10770).*  \n*Lian, Z., Liu, B., & Tao, J. (2022). Smin: Semi-supervised multi-modal interaction network for conversational emotion recognition. IEEE Transactions on Affective Computing.*", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for your time in providing insightful feedback on our work. Your remarks are valuable guidance for us to improve the paper quality. For your questions, we would like to respond as follows: \n\n### I. Questions:\n\n**Q1.**  Why not compare with some advanced baselines, and what criteria were used to select these baselines. The related work should be updated with more recent related works. \n\n**Answer:** Thank you for your comment. Via a literature review on related works until the submission time (Section 1, Section 2.1), we selected the most prominent baseline models (Section 4.1) for the multimodal Emotion Recognition (ERC) task, and COGMEN [1] is the state-of-the-art one.  However, we will double-check and update recent related works accordingly. \n\n**Q2.** So difficult to follow the contribution of this paper. And it looks like an incremental engineering paper. The proposed method has been introduced in many papers, such as [1] Joshi, A., Bhat, A., Jain, A., Singh, A., & Modi, A. (2022, July). COGMEN: COntextualized GNN-based Multimodal Emotion Recognition. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (pp. 4148-4164)\n\n**Answer:** Thank you for sharing your thoughts on the paper. We will work diligently to improve the presentation of our paper's contribution and originality to make it more comprehensible to readers.  Specifically, our approach differs from the COGMEN model in a significant way as follows: \n\n* While COGMEN utilizes fused features from all three modalities as input, we focus on modality-specific features in our approach. The use of fused features might blur the distinct roles of individual modalities within the model. Concatenating input representations is not ideal due to varying representation size of each modality; and the need to independently consider each modality's contributions, rather than pooling them together. Thus, our adoption of modality-specific features addresses the limitations inherent in the COGMEN model.  \n\n* Additionally, it's worth noting that COGMEN primarily focuses on harnessing contextual information of speaker flow emotions through inter-speaker and intra-speaker relations within a conversation. However, we believe that this approach does not fully exploit the rich information present in the conversation. Our model, on the other hand, operates at the utterance-level, where each utterance encapsulates abundant information including context, speaker identity, and emotions. We take into account the intricate interplay of these factors, including their past and future relationships, to achieve a comprehensive understanding of the conversational data.\n\n### II. Suggestions:\n\n**S1.** The experimental section needs some significance tests to further verify the effectiveness of the method put forward in the paper. \n\n**Answer:** Thank you for your comment. We follow the evaluation strategy of COGMEN [1], in which only the best performance is reported. Nevertheless, we will conduct significance tests and update the result in our final version.  \n\n**S2.** Error analysis plays a crucial role in evaluating model performance and identifying potential issues. We encourage the authors to conduct error analysis in the paper and provide detailed explanations of the model's performance under different scenarios. Error analysis will aid in guiding subsequent improvements and expansions of the ERC research.  \n\n**Answer:** Thank you for your suggestion. Due to the space limitation, we briefly discuss potential issues for each experimental result in Section 4.2 and Section 4.3. We will consider to add the error analysis subsection to our final version.  \n\n### III. Typos, Grammar, Style, and Presentation Improvements:\nThe structure and fluency of the sentences in the paper are poor and require improvement to enhance the quality of the writing ... On page 2, there are some words or punctuation that may have ambiguity\n\n**Answer:** Thank you for pointing out the issues. We will carefully go through your feedback and fix all the errors in the final version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "-\tReasonable method.", "reasons_to_reject": "-\tSo difficult to follow the contribution of this paper. And it looks like an incremental engineering paper. The proposed method has been introduced in many papers, such as [1] Joshi, A., Bhat, A., Jain, A., Singh, A., & Modi, A. (2022, July). COGMEN: COntextualized GNN-based Multimodal Emotion Recognition. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (pp. 4148-4164).\n-\tThe related work should be updated with more recent related works.\n-\tThe experimental section needs some significance tests to further verify the effectiveness of the method put forward in the paper.\n-\tFor the first time appearing in the text, the full name must be written, and abbreviations must be written in parentheses. When it appears in the abstract, it needs to be written once, and when it appears in the main text, it needs to be repeated again, that is, the full name+parentheses (abbreviations) should appear again.\n-\tError analysis plays a crucial role in evaluating model performance and identifying potential issues. We encourage the authors to conduct error analysis in the paper and provide detailed explanations of the model's performance under different scenarios. Error analysis will aid in guiding subsequent improvements and expansions of the ERC research.\n-\tWriting mistakes are common across the overall paper, which could be found in “Typos, Grammar, Style, and Presentation Improvements”.", "questions_to_authors": "Question A: Why not compare with some advanced baselines, and what criteria were used to select these baselines?\n\n**I lean towards giving a rating of 2.5 on Excitement.** The performance of the method doesn't seem significantly superior, as observed in various other approaches listed below. Additionally, the paper's contributions appear rather limited, primarily focusing on refining the COGMEN model.\n\n*Shi, T., & Huang, S. L. (2023). MultiEMO: An Attention-Based Correlation-Aware Multimodal Fusion Framework for Emotion Recognition in Conversations. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (pp. 14752-14766).*  \n*Chen, F., Shao, J., Zhu, S., & Shen, H. T. (2023). Multivariate, Multi-Frequency and Multimodal: Rethinking Graph Neural Networks for Emotion Recognition in Conversation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 10761-10770).*  \n*Lian, Z., Liu, B., & Tao, J. (2022). Smin: Semi-supervised multi-modal interaction network for conversational emotion recognition. IEEE Transactions on Affective Computing.*", "paper_topic_and_main_contributions": "This paper aims to propose the Relational Temporal Graph Neural Network with Auxiliary Cross-Modality Interaction (CORECT), a novel neural network framework that effectively captures conversation-level cross-modality interactions and utterance-level temporal dependencies in a modality-specific manner for conversation understanding.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "Typos, Grammar, Style, and Presentation Improvements:\n\n-\tThe structure and fluency of the sentences in the paper are poor and require improvement to enhance the quality of the writing. (e.g., “Our social interactions and relationships are all influenced by emotions.”, “MMGCN adopts an undirected graph to effectively fuse multimodal information and capture long-distance contextual and inter-modal interactions”, “Extensive experiments on two popular benchmark datasets, i.e., IEMOCAP and CMU-MOSEI, demonstrate the effectiveness of CORECT”, “we validate that capturing the long-term dependencies, e.g., past relation, improves the performance of multimodal ERC task.”)\n-\tOn page 2, there are some words or punctuation that may have ambiguity. (e.g., “ CORECT is a multimodal ERC model based on GNNs which captures both temporal dependencies and cross-modality interaction in conversational.”) In particular, the above problems appear in many parts of the paper."}}]

Author rebuttal:

Meta-review: This paper introduces a novel multimodal framework, CORECT for emotion recognition in conversations. The proposed architecture consists of two major component to capture the local context features in utterance-level along with the cross-modal global context features in conversation-level. The authors shows some modest gain in performance of CORECT when compared with other multimodal ERC research using two widely used datasets. 

All the reviewers agree that the current study provides sufficient support for its claim and offers meaningful advancements over existing emotion recognition methods. However, the reviewers have identified some minor issues, such as:
1. An insufficient discussion regarding the limitations of the proposed framework.
2. Lack of complexity analysis, more focusing on computational efficiency of the model for practical implementation among others.

The authors have addressed some of these concerns in the comment and in the rebuttal.  By addressing the reviewer’s comments, the paper quality will improve substantially. 
I believe this paper will be interesting to the Speech community.
Decision: Accept-Main