Enhancing Emotion Recognition in Conversation via Multi-view Feature Alignment and Memorization
Authors: Guiyang Hou
Yongliang Shen
Wenqi Zhang
Wei Xue
Weiming Lu
Venue: EMNLP 2023 Findings (2023)
Keywords: Emotion Recognition in Conversation
Multi-view Feature Alignment
Memorization

Abstract: Emotion recognition in conversation (ERC) has attracted increasing attention in natural language processing community. Previous work commonly first extract semantic-view features via fine-tuning PLMs, then models context-view features based on the obtained semantic-view features by various graph neural networks. However, it is difficult to fully model interaction between utterances simply through a graph neural network and the features at semantic-view and context-view are not well aligned. Moreover, the previous parametric learning paradigm struggle to learn the patterns of tail class given fewer instances. To this end, we treat the pre-trained conversation model as a prior knowledge base and from which we elicit correlations between utterances by a probing procedure. And we adopt supervised contrastive learning to align semantic-view and context-view features, these two views of features work together in a complementary manner, contributing to ERC from distinct perspectives. Meanwhile, we propose a new semi-parametric paradigm of inferencing through memorization to solve the recognition problem of tail class samples. We consistently achieve state-of-the-art results on four widely used benchmarks. Extensive experiments demonstrate the effectiveness of our proposed multi-view feature alignment and memorization.

Enhancing Emotion Recognition in Conversation via Multi-view Feature Alignment and Memorization
Guiyang Hou, Yongliang Shen, Wenqi Zhang, Wei Xue, Weiming Lu"
College of Computer Science and Technology Zhejjiag University (gyhou, syl, zhangwenqi, lokilanka, luwm)ezju. edu.
Abstract
differentt emotion
Speakert: Then all of sudden they gave him microphoee marry like, anataee
Emotion recognition in conversation (ERC) has attracted increasing attention in natural lan- guage processing community. Previous work commonly first extract semantic-view features via fine-tuning pre-trained language models (PLMs), then models context-view features based the obtained semantic-view features by various graph neural networks. However, difficult to fully model interaction between utterances simply through a graph neural net- work and the features at semantic-viers and context-view not well aligned. Moreover, the previous parametric learning paradigm struggle to learn the patterns tail class given fewer instances. To this end, treat the pre- trained conversation model as prior knowl- edge base and from which we elicit correlations between utterances by probing procedure. And we adopt supervised contrastive learning align semantic-view and context-view fea- tures, these two views of features work together complementary manner, contributing to ERC from distinct perspectives. Meanwhile, we propose a new semi aani-parametric paradigm inferencing through memorization to solve the recognition problem of tail class samples. We consistently achieve state-of-the-art results on four widely used benchmarks. Extensive experiments demonstrate the effectiveness of our proposed multi-view feature alignment and memorization!.
Speaker2:: He was toing
Speakert: Johers died yestarday, know that was coming
Speaklert: Uske just last week he was soing well.
eeootin change
Spoakert: Oh,
loolo just great on you.
change
Speaker Yeal? Are really
Speakert: You see, you look beausftul For gnd saken, dim the light.
Sunden dhange
Speaker2: LL "'m hnmhiee Nideous
Complex
imbolance
fine-grained class dication
Seal raagh
class cluss claid clasé cluss chess deuc
ENGCAP
11:00
251
630
200
DelvOlikg
12085
55572
2022
1150
274
3825
MGL0
5430
1206
301
2007
Enony
2002
535
717
Small
imbalance fine-grained classffiaation problems
Figure I: Major challenges in the ERC task.
Unlike the basic emotion classification (EC) task (Yin and Shang, 2022), ERC is more practical endeavor that involves predicting the emotion label of each utterance based on the surrounding context. Previous methods (Ghosal et al., 2019; Ishiwatari et al., 2020; Shen et al., 2021b) commonly follow two-step paradigm of first extracting semantic- view features via fine-tuning PLMs and then mod- eling context-view features based on the obtained semantic-view features by various graph neural networks, such as GCN (Kipf and Welling, 2016), RGCN (Ishiwatari et al., 2020), GAT (Veličković et al., 2017). Considering the complexity of emo- tions, some recent works (Song al., 2022; Hu et al., 2023) supervised contrast loss as auxil- iary loss improve the feature discriminability be- tween different classes of samples. Although these methods have achieved excellent performance on ERC task, three issues still remain: (1) As illus-
1 Introduction
Emotional intelligence is an advanced capability of conversational AI systems. A fundamental and critical task this domain emotion recognition in conversation (ERC), which aims to identify the emotions conveyed in each utterance within the dialogue context (Poria et al., 2019b).
*Corresponding author.
'https://gitthen wwwp:ttthu..cmmohmomo.eeee 2ttpmm/yyyoo/////AAaaa/ AM/MFAM.

---

Context-view
both semantic-view and context-view features will participate in the computations of SCL. Finally, we propose a new semi-parametric paradigm of infer- encing through memorization to solve the recogni- tion problem of tail class samples. We construct two knowledge stores, one from semantic-view and the other from context-view. Semantic-view knowl- edge store regarding comantic-view features and corresponding emotion labels as memorized key- value pairs, and the context-view knowledge store is constructed in the same way. During inference, our model not only infers emotion through the weights of trained model but also assists decision- making by retrieving examples that are memorized in the two knowledge stores. It's worth noting that semantic-view and context-view features have been well aligned, which will facilitate the joint retrieval of the semantic-view and context-view.
0-
20
Semantic-view
Figure 2: T-SNE (Van der Maaten and Hinton, 2008) visualization of features semantic-view and context- view.
The main contributions of this work are sum- marized as follows: (1) We propose multi-view feature alignment for ERC, which aligns semantic- view and context-view features, these two views of features work together in complementary man- ner, contributing to ERC from distinct perspectives. (2) We propose of inferencing through memorization solve the recognition problem of tail class samples. (3) We treat the PCM as a prior knowledge base and from which we elicit correlations between utterances by a probing procedure. (4) We achieve state-of-the- art results on four widely used benchmarks, and extensive experiments demonstrate the effective- of our proposed multi-view feature alignment and memorization.
trated in Figure 1, the interaction between utter- ances in a conversation very complex, and it's dif- ficult to fully model this interaction simply through a graph neural network. (2) Both remantic-view and context-view view are important for ERC. The for- mer focuses on the emotion conveyed by the in- dependent utterance, while the latter offers clues about the emotional context. These two views of in- formation work together in complementary man- ner, contributing ERC from distinct perspectives. However, as illustrated in Figure 2, semantic-view and context-view are not well aligned. (3) Due to the smaller number of samples in the tail class, is difficult for the parametric model to learn the pattern of the tail class during training. How to effectively recognize samples of the tail class re- mains an issue to be solved.
cami-parametric paradigm
Related Work
2.1 Emotion Recognition in Conversation
Most existing works (Ghosal al., 2019; Ishiwatari et al., 2020; Shen al., 2021b) commonly first ex- tract semantic-view features via fine-tuning PLMs and then model context-view features based on the obtained semantic-view features by various graph neural networks, there are also some works (Zhong et al., 2019; Shen et al., 2021a; Majumder et al., 2019) that use transformer-bass and recurrence- based methods to model context-view features. It's worth noting that self-attention (Vaswani al., 2017) in transformer-basee methods can be viewed as a fully-connected graph in some sense. Some recent works (Li et al., 2021; Hu et al., 2023)use supervised contrastive loss as an auxiliary loss to
To address these issues, we propose Multi-view Feature Alignment and Memorization (MFAM) for ERC. Firstly, we treat the pre-trained conversation model (PCM) (Gu et al., 2021) as a prior knowl- edge base and from which we elicit correlations between utterances by probing procedure. The correlation between utterances will be used to form the weights of edges in the graph neural network and participate in the interactions between utter- ances. Secondly, adopt supervised contrastive learning (SCL) to align the features semantic- view and context-view and distinguish semantically similar emotion categories. Unlike the regular SCL,

---

Correlation Probe Given conversation com- prised of N utterances [M1, U2, MNN, the goal of our probing technique is to derive function (()))) that captures the correaltion between bitraty pair of utterances. To this end, we employ enhance feature discriminability between samples of different classes. In the following paragraphs, we provide a detailed description of graph-based based methods and methods using supervised contrastive loss.
is the speaker and u, is the utterance of i-th turn. For utterance us it is comprised of n4 tokens u = (M,1,422: :
3.2 Multi-view Feature Extraction
DialogueGCN(HHu et al., 2021) treats the dia- logue directed graph, where each utterance is connected with the surrounding utterances. DAG- ERC(Shen et al., 2021b) uses a directed acyclic graph to model the dialogue, where each utter- ance only receives information from past utterances. CoG-BART(Li et al., 2021) adapts supervised con- trastive learning to make different emotions mutu- ally exclusive to identify similar emotions better. SACL(HU et al., 2023) propose supervised adver- sarial contrastive learning framework for learning generalized and robust emotional representations.
In this section, we will introduce how to extract the semantic-view and context-view features of utter- ance.
3.2.1 Semantic-view Feature Extraction
Semantic Feature Extraction We employ PLM to extract the semantic feature of utterance u, Fol- lowing the convention, the PLM is firstly fine-tuned each ERC dataset, and its parameters are then frozen while training. Following Ghosal al. (2020), we employ RoBERTa-Large (Liu et al., 2019) as our feature extractor. More specifically, for each utterance we prepend special token [CLS] to its tokens, making the input form of [[CLS), W4,1, Wi,,; ... Wi.n;) Then, we average the [CLS]'s embedding in the last 4 layers u,'s semantic feature vector 24 Rd.
2.2 Memorization
Memorization-based methods (or non/semi- parametric methods) performs well under low-resource scenarios, and have been ap- plied to various NLP tasks such as language modeling(Khanddelall et al., 2019), question Baswering(Ksssne and Schütze, 2020), knowledge graph embedding(Zhang et al.. 2022) and relation extraction(Chen et al., 2022).
Commensense Feature Extraction We extract six types (#Intent, xAttr, xNeed, #W ant, xEffect, x &React) of commensense feature vectors related to the utterance from COMET(Bosselut et al., 2019);
2.3 Probing PLMS
of -COMET (W,1, W1,2, "* Wi,, TT) (1)
Some work has shown that PLMs such as BERT(Devlin et al., 2018), RoBERTa(Liu et al., 2019), ECaakRA(Clark et al., 2020) store rich knowledge. Based this, PROTON(Wang et al., 2022) elicit relational structures for schema linking from PLMs through a unsupervised probing proce- dure. Unlike PROTON operating the word level, our probe procedure is based on PCM and operates at the utterance level.
where r2 ss is the token of j-th relation type, of E Rd represent the feature vectors of the j-th commensense type for uy Semantic-view feature is obtained by concate- nate semantic feature vectors with their correspond- ing six commensense feature vectors follows:
g W SVD o.... ........))))) +
(2)
where SVD D used extract key features from commensense, , is used to reduce feature dimen- sion. 9i E Bd is the final semantic-view feature of utterance u;
3 Methodology
We introduce the definition of ERC task in sec- tion 3.1, and from section 3.2 to section 3.5, introduce the proposed MFAM in this paper. The overall framework of MFAM shown in Figure 3.
3.2.2 Context-view Feature Extraction
3.1 Definition
Given collection of all speakers S, an emo- tion label set y' and conversation C, our goal is identify speaker's emotion label at each conversation turn. A conversation denoted as [(s, un), (82, U2), ... (SN, UN)], where 8; E S
PCM(GU et al., 2021), which is pretrained on masked utterance generation, next utterance gener- ation and distributed utterance order ranking task.

---

Somantic-viww Festure Extraction
Conteci-sew Feature Extraction
Training and
Inowledge-wietee
Correlation Probe
Macill Datribbtion
9r. Bs
RoBERTa
Graph Construction
BGON
A
Liteed
G Og:
Comeet
Semarticwlew
Contest-iew
edgn weight
Samantic-vicn Butrivaal
push
Contictt view Fattrival
pull
Somarin fic view
Protettpp Semanti-viee view
pull
facil tate
A
Recrival Disribution
Pull
Prototype Contweh-tew
Semanticsiess Knowledge
Contrastive Space
Contectiviees knowlddge
Multi Featuree Alignment
Knowledge Store and Retrieval
Feature Nignment collaborates
Knowledge Retrieval)
Figure 3: The overall framework of MFAM, which mainly consists of four parts: multi-view feature extraction, multi-view feature alignment, knowledge store and retrieval, training and inference.
As shown in Figure 4, first feed the N ut- terances into the PCM. We use hi to denote the contextualized representation of the j-th utterance Uj. where 1 S j N. Then, we replace the ii with a mask utterance [CLS, M ASK, SEP) and feed the corrupted N utterances into the PCM again. Accordingly, we use by to denote the new representation of the j-th utterance "y when ue is masked out. Formally, we measure the distance between hy and h to induce the correlation be- tween in and uj follows:
Tay & where R is the relation type of the edge between by and Vj and aaj the weight of the labeled edge Tij. with & aij < 1, where Okj E W and í, E [1, ..., N).
Each utterance in the conversation is represented as a vertex i C V in G. Each vertex vú is initialized with the corresponding semantic-view feature Ji, for all i E [1, 2, .... .N., and has an edge with the immediate p utterances of the past: 1.-1..... Ui-p- utterances of the future: Vi+1,...5 Us+f and it- self: Us. Considering speaker dependency and temporal dependency, there exists multiple relation types of edge. The edge weights Wij are obtained by combining the results of two computation ways. One use a similarity based attention module to cal- culate edge weights aij. weights keep changing during training, which can be regarded as adaptive training. The other takes the correlation 2ij be- tween utterance pair (uj) computed by Eq.(3) as edge weights, weights keep frozen during train- ing, which can be regarded as correlation-based knowledge injection:
f(u, u3) = d(hj. k
(3)
where d(-,-) is the distance metric to measure the difference between two vectors. We use Euclidean distance metric to implement d(.,) :):
d (ui, uz) l|u ugll2
(4)
where : denotes a distance function in Eu- clidean space.
By repeating the above process on each pair of utterances u4 Mij and calculating fu,, (z), we ob- tain correlation matrix X = [ ) 1 1= where Ty denotes the correlation between utterance pair (ug, uz).
Oij sof soffoommme llmaagg We p Welgi-p-.... ...... g+++))
|N|
(5)
Wi = ozj + bxij
where i p,..., + b denotes the pre- defined weight coefficient of zij. reflects the injec- tion intensity of correlation- based knowledge.
Following Ghosal et al. (2019), a conversation having N utterances repre- sented as a directed graph g = (V, E, R, W), with vertices/nodes U/ E V. labeled edges (relations)
Graph Construction
Interaction between Utterance Based on the constructed graph, we utilize RGCN to implement

---

For instance, with multiple prototypes, the updates for P(i) and A(i) are as follows:
2
Euclidean Space
P()) j= (1) (1c) {oc}
M/i M} [C]+(00 (7)
PCM
I.ASA
where e represents the emotion category corre- sponding to sample i, L - {4..... , and O = fo. -ty]) respectively represents the in- dex collection of semantic-view and context-view prototype vectors for all emotion categories.
bg
ug
d ( mz)
|85
Aallla
Figure Correlation Probe procedure.
3,4 Knowledge Store and Retrieval
interactions between utterances, and then obtain context-view feature fi utterance ug The de- tailed calculation process can be found in Appendix A.
Given training set (u. y) E (4,3),, paramet- ric model computes semantic-view feature x² and context-view feature f of input utterance u. Then we get semantic-viey features { and context- view features {.f))=1 of all training inputs {u) We use BERT-whitening (Su et al., 2021) to reduce the feature dimension, therefore, the retrieval speed is accelerated.
3.3 Multi-view Feature Alignment
We adopt SCL to align semantic-vier and context- view features. Specifically, in batch consisting of M training samples, for all m [1,2, ..., M). both Imu and Jm are involved in the computation of SCL, we can incorporate Im and fm into the SCL computation separately, forming 2M samples, or concatenate them for SCL computation. Taking the former approach as example, the supervised con- trastive loss of 2M samples in a multiview batch be expressed by the following equation:
{ BERT www.wwinngi.i::eeee
(8) We construct two knowledge stores, one from semantic-view and the other from context- view. Semantic-view knowledge store regarding semantic-view features fú and corresponding emo- tion labels y; as memorized key-value pairs. For context-view knowledge store, we adopt the same construction method as semantic-view.
(f)]= - BERT-whitening(())
G = 1m FM [i==ffi...... fml
Y = (GM, Fisl
(Koem Vsem) = U
(f),
LSCL Σ
-1
Σ SIM(P, i)
(LL))
I(())
(6)
(9)
iET
pEP(()
(Kcon, Voon)
U
(Fr, vi)
exp((Y; Yp)/T)
(u.))
SIM(p, i) = log Eacs(o) exp(Y; Ya/T)
During inference, given utterance u, paramet- ric model computes corresponding semantic-view feature x and context-view feature f. We do the same as (q.(8) for and f to get 5 and f. which used to respectively retrieve the semantic-viee and context-view knowledge store for the k nearest neighbors (kNN) U according to L2 distance:
where Y E R2Mxd i 21, 2M) indicate the index of the samples in multiview batch, E R denotes the temperature coef- ficient used to control the distance between in- stances, P(t)
نز= {i} represents samples with the same category as while excluding itself, A(1) = - (i, + M/i M) indicates samples in the multiview batch except itself.
N (ylu)x
Σ (ks.e))M
y DIS(Ks
To further enrich the samples under each cate- gory, incorporate the prototype vectors corre- sponding each category into the SCL computa- tion. Prototype vectors can correspond to features at different views, forming multiple prototypes, or they can correspond to the concatenation of fea- tures at different views, forming single prototype.
PENN con (vlu)
11 B1 DIS(Ki (10)
(ks
d(ki, F/f)
DIS(K, exp(
T
SSmanttcc feature an essential component of the semantic-view features, we use it to represent semantic-view features during knowledge store construction and retrieval.

---

where T is the temperature.
Dataset
DD
MELD
ENLP
IEMOCAP
#Dial
Truin
11118
1038
713
120
3.5 Training and Inference
Dev
1000
114
99
120
Test
1000
280
85
31
Training Since semantic-view and context-view features have been well aligned, we concatenate them for emotion classification and use cross- entropy loss function to calculate classification loss. Meanwhile, we use logit compensation (Menon et al., 2020) to eliminate the bias in the classifica- tion layer caused by class imbalance issues:
#CLS
7
Table Statistics four benchmark datasets.
4.2 Implementation Details
The initial weight of RoBERTa come from Hug- gingface's Transformers WWolf et al., 2020). We utilize Adam (Kingma and Ba, 2014) optimizer to optimize the network parameters of our model and the learning rate is set to 0.0003 and remains constant during the training process. We adopt faiss (Johnson et al., 2019) library to conduct re- trieval, and the dimensions of semantic-viev and context-view features respectively become 384 and 64 after the BERT- operation. We search the haper--aramettrs the validation set. All experiments are conducted on A6000 GPU with 48GB memory.
M
exp(py((,) + +dy)
LCE=
log Σ explyy (u:)) (ōg) cM
(11)
where w; gi + fi. represent feature conca- tention, ip represent classification layer, by is the compensation for class y and its value is related class-frequency. Finally, we have the following batch-wise loss for training.
L LCE + LSCL.
(12)
Inference During inference, we not only infer the utterance's 's emotion through the trained model but also assists decision-making by retrieving ex- amples that are memorized in the two knowledge stores:
4.3 Compared Methods
We compare our model with the following base- lines in our experiments. Transformer-based methods: BERT (Devlin et al., 2018), RoBERTa(Liu et al., 2019), CoG- BART(Li al., 2021), CoMPM(Lee and Lee, 2021), SPCL(Song et al., 2022), MPLP(Zhang et al., 2023).
p(ylu) = =XpModed(lle) + μРNNN seee sevv (3/24)
(2)
(13)
Recurrence-based methods: DialogueRNN (Ma- jumder et al., 2019), DialogueRNN-RRBBERRR, COSMIC (Ghosal al., 2020), DialogueCRN(Hu et al.. 2021), SACL(Hu al., 2023).
where A, μ4, Y represent the interpolation hyperpa- rameters between model output distribution and kNN distribution.
4 Experiments
Graph-based methods: DialogueGCN (Ghosal et al., 2019), DialogueGCN-RoBERTR.. RGAT (Ishiwatari et al., 2020), RGAT-RoBERTa, SGED+DAG-ERC(Baa et al., 2022) and DAG- ERC(Shen et al., 2021b).
4.1 Datasets and Evaluation metrics
We evaluate our method on four benchmark datasets: IEMOCAP (Busso al., 2008), MELD (Poria et al., 2019a), DailyDialog (Li et al., 2017) and EmoryNLP (Zahiri and Choi, 2018). Detailed descriptions of each dataset can be found in the appendix B. The statistics of four datasets are pre- sented in Table 1.
In addition to the above-mentioned baselines, we also take into account the performance of Chat- GPT et al., 2020) on the ERC task evaluated by Yang et al. (2023) and Zhao et al. (2023).
We utilize only the textual modality of the above datasets for the experiments. For evaluation met- rics, we follow (Shen al., 2021b) and choose micro-averaged Fl excluding the majority class (neutral) for DailyDialog and weighted- aveerge F! for the other datasets.
Results and Analysis
5.1 Overall Performance
The comparison results between the proposed MFAM and other baseline methods are reported in Table 2. We can observe from the results that

---

Model
IEMOCAP
MELD
DailyDialog
EmoryNLP
Avg
2hatGPT2s (Zhao et al., 2023)
44.97
57.30
40.66
37.47
45.10
ChatGPT (Yang et al., 2023)
53.35
61.18
43.27
32.64
47.61
ChatGPT (Zhao et al., 2023)
48.58
58.35
42.39
35.92
46.31
Transformmr-baseed Methods
BERT (Devlin et al., 2018)
60.98
62.28
54.85
34.87
53.25
RoBERTa (Liu et al., 2019)
63.38
62.51
54.33
35.90
54.03
BART (Lewis et al., 2019)
56.14
63.57
55.34
35.98
52.76
CoG-BART(Li et al., 2021)
66.18
64.81
56.29
38.04
56.33
CoMPM(Lee and Lee, 2021)
69.46
66.52
60.34
38.93
58.81
SPCL(Song et al., 2022)*
66.93
64.93
39.45
MPLP(Zhang et al., 2023)
66.65
66.51
59.92
Recurrence-based Methods
DialogueRNN (Majumder et al., 2019)
62.75
57.03
+RoBERTa"
64.76
63.61
57.32
37.44
55.78
COSMIC (Ghosal et al., 2020)
63.05
64.28
56.16
37.10
55.15
DialogueCRN (Hu et al., 2021)
66.20
58.39
SAU et al., 2023)
69.22
66.45
39.65
Graph-based Methods
DiN (Ghosal al., 2019)
64.18
58.10
+RoBERTa"
64.91
63.02
57.52
38.10
55.89
RGAT (Ishiwatari et al., 2020)
65.22
60.91
54.31
34.42
53.72
RoBERTa"
66.36
62.80
59.02
37.89
56.52
DAG-ERC (Shen et al., 2021b)
68.03
63.65
59.33
39.02
57.51
SGED + DAG-ERC (Bao et al., 2022)
68.53
65,46
40.24
MFAM(Ours)
70.16
66.65
62.19
41.06
60.02
Table 2: Overall results (%) against various baseline methods for ERC on the four benchmarks. i and $ represent different prompt templates, zs and fa respectively represent zero-shot and few-shot scenario. represent models with RoBERTa utterance features. The results reported in the table from the original paper their official repository. Best results are highlighted in bold. The improvement of our model over all basclines is statistically significant with p < 0.05 under I-test.
our proposed MFAM consistently achieves start-of- the-art results on four widely used benchmarks.
trained on the full dataset.
As graph-based method, MFAM achieves an average performance improvement of +4.36% when compared to preceding graph-based meth- ods. Moreover, when considering the four bench- marks individually, MFAM achieves performance improvement of +2.38%, +1.82%, +4.82%, and +2.04% respectively, marking significant enhance- ment over the earlier graph-based methods.
5.2 Ablation Study
To study the effectiveness of the modules in MFAM, we evaluate MFAM by removing knowl- edge module, alignment module and memorization module. The results of ablation study are shown in Table 3.
Concerning the knowledge module, which in- cludes commonsense knowledge and correlation- based knowledge. Its removal results in sharp performance drop on IEMOCAP and DailyDia- log, while slight drop MELD and EmoryNLP. Therefore, can infer that the conversations in IEMOCAP and DailyDialog involve more com- monsense knowledge and more complex utter- ance interactions. Concerning the alignment mod- ule, its removal leads to a similar and significant
MFAM also shows sigmificant advantage when compared to other types of methods. It achieves average performance gain of +2.06%, +7.60% respectively compared transformer-based and recurrence-based methods. Moreover, we notice that the performance of ChatGPT in zero-shot and few-shot scenarios still has a significant gap compared to the performance of models currently

---

Model
IEMOCAP
MELD
DailyDialog
EmoryNLP
Avg
MUus((Our)
70.16
66.65
62.19
41.06
60.02
w/o Know
68.66( 1.50)
65.97( 0.68) 61.04(} 1.15)
40.27(4 0.79)
58.99(L 1.03)
w/o SCL
68.64( 1.52)
65.69()
0.96)
60.61(1
1.58)
39.74(1
1.32)
58.67(L 1.35)
w/o Retrieval
68.46( 1.70) 64.64() 2.01) 60.10() 2.09) 39.10(1 1.96)
58.08(L 1.94)
w/o Retrieval & SCL
66.45( 3.71) 63.46( 3.19) 58.49( 3.70) 38. 38.26(4 2.80)
56.67(4 3.35)
Table 3: Results of ablation study on the four benchmrks.
drop in the model's performance on all datasets, demonstrating the importance of multi-view fea- ture alignment in the ERC task. Concerning the memorization module, its removal results in drastic decrease in model performance across all datasets, highlighting the effectiveness of infer- encing through memorization in addressing class imbalance issues. Moreover, the simultaneous re- moval of the alignment and memorization modules results in performance decline that is greater than the sum of the declines caused by the removal of each module individually, proving that aligning semantic-view and context-view features can facil- itate the joint retrieval of the semantic- view and context-view.
the best performance is achieved. Continuing increase the value of and reduce the value of y would result in performance decline. In addition. fluctuating the value μ while keeping and y at 0.7 and 0.1 respectively also leads to a decrease performance.
5.4 Visualization on Multi-view Feature
To conduct qualitative analysis of multi-view fea- ture alignment, we utilize t-sme(Van der Maaten and Hinton, 2008) to visualize the prototype vectors corresponding to semantic-view and context-view features under each emotion category, shown in Figure 5.
5.3 Analysis on X, 14, Y
It can be observed that semantic-view and context-view features under the same emotion cat- egory are well aligned. Meanwhile, positive emo- tions such as "happy" and "excited" are close each other, while negative emotions like "frus- trated", "angry", and "sad" are also close each other.
X, μa and very important parameters, which respectively represent the weights occupied by the model, semantic-view retrieval, and context-view retrieval during inference. Determining the appro- priate weight combination reinforce their inter- play is very important. Table shows the test fl- scores on the IEMOCAP dataset for different X, and 7.
We can observe that when setting l to 0.2, dynamically adjusting À and y leads to continuous changes in performance, when the value of A rises 0.7, corresponding to drop in y 0.1,
angry
frustrated
negative
A
μ
7
Performance
happy
0.4
0.2
0.4
67.67
proto2
0.5
0.2
0.3
67.48
protol
positive
0.6
0.2
0.2
68.91
sad
0.65
0.2
0.15
69.26
excited
neutral
0.7
0.15
0.15
68.76
0.7
0.2
0.1
70.16
0.7
0.25
0.05
68.98
0.75
0.2
0.05
69.02
Figure 5: Visualize the prototype vectors corresponding to semantic-view and context-view features under each emotion category. Protol and proto2 correspond semantic-view and context-view features, respectively.
Table 4: Test fl-scores on ЛEMOCAP dataset with dif-
ferent A, f. and 7.

---

6 Conclusion
Neclakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 1377197-1
In this paper, we propose Multi-view Feature Alignment and Memorization (MFAM) for ERC. Firstly, in order to obtain accurate context-view fea- tures, we treat the PCM as prior knowledge base and from which we elicit correlations between utter- ances by a probing procedure. Then we adopt SCL to align semantic-view and context-view features. Moreover, we improve the recognition performance of tail-class samples by retrieving examples that are memorized in the two knowledge stores during inference. We achieve state-of-the -ort----rtt-ttttt results on four widely used benchmarks, ablation study and visualized results demonstrates the effectiveness of multi-view feature alignment and memorization.
Carlos Busso, Murtaza Bulut, Chi-Chun Lee, Abe Kazemzadeh, Emily Mower, Samuel Kim, Jean- nette N Chang, Sungbok Lee, and Shrikanth S Narayanan. 2008. lemocap: Interactive emotional dyadic motion capture database. Language resources and evaluation, 42(4):335--3
Xiang Chen, Lei Li, Ningyu Zhang, Chuanqi Tan, Fei Huang, Luo Si, and Huajun Chen. 2022. Rela- tion extraction as open-book examination: Retrieval- enhanced prompt tuning. In Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 2443 -448.
K Luong Clark, MT Le, QV Manning, and CD ELEC- TRA. 2020. Pre-training text encoders as discrimina- tors rather than generators. Preprint at http://urxiv. org/abs/2003.1055
Limitations
There are two major limitations in this study. Firstly, semantic-view and context-view retrieval based on the training set may suffer from dataset and model bias. Secondly, during inference, we need to consider three probability distributions: semantic-view retrieval, context-view retrieval, and model output. Determining the appropriate weight combination to reinforce their interplay is very important, therefore, additional computational re- sources are required to find these parameters. The aforementioned limitations will be left for future research.
Jacob Devlin, Ming-Was Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre Pre-training of deep bidirectional transformers for language understand- ing. arXiv preprint
Deepanway Ghosal, Navonil Majumder, Alexander Gel- bukh, Rada Mihalcea, and Soujanya Poria. 2020. Cosmic: Commonsense knowledge for emotion identification in conversations. arXiv preprint arXiv:2010.02795.
Deepanway Ghosal, Navonil Majumder, Soujanya Poria, Niyati Chhaya, and Alexander Gelbukh. 2019. Dia- loguegen: A graph convolutional neural network for emotion recognition in conversation. arXiv preprint arXiv: 1908.1150..
Acknowledgement
This work is supported by the Fundamental Re- search Funds for the Central Universities (No. 226- 2022-00143), National Key Research and Develop- ment Project of China (No. 2018AAA0101900 and MOE Engineering Research Center of Digital Library.
Xiaodong Gu, Kang Min Yoo, and Jung- Woo Ha. 2021. Dialogbert: Discourse-awe aware response generation via learning to recover and rank utterances. In Proceed- ings of the AAAI Conference on Artificial Intelligence, volume 35, pages 12911-12919.
Dou Hu, Yinan Bao, Lingwei Wei, Wei Zhou, and Songlin Hu. 2023. Supervised adversarial contrastive learning for emotion recognition in conversations.
References
Dou Hu, Lingwei Wei, and Xiaoyong Huai. 2021. Dia- loguecrn: Contextual reasoning networks for emotion recognition in conversations, In Proceedings of the 59th Annual Meeting of the Association for Compu- tational Linguistics and the 11 International Joint Conference on Natural Language Processing (Vol- 1: Long Papers), pages 7042-7052.
Yinan Bao, Qianwen Ma. Lingwei Wei, Wei Zhou, and Songlin Hu. 2022. Speaker-guided encoder-decoder framework for emotion recognition in conversation. arXiv preprint 231v:22006.01173.
Antoine Bosselut, Hannah Rashkin, Maarten Sap. Chai- tanya Malaviya, Asli Celikyilmaz, and Yejin Choi. 2019. Comet: Commonsense transformers for auto- matic knowledge graph construction. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4762-4779.
Taichi Ishiwatari, Yuki Yasuda, Taro Miyazaki, and Jun Goto. 2020. Relation-awee aware graph attention networks with relational position encodings for emotion recog- nition in conversations. In Proceedings of the 2020 Conference on Empirical Methods in Natural Lan- guage Processing (EMNLP), pages 7360-7370.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind

---

Jeff Johnson, Matthijs Douze, and Hervé Jégou. 2019. Billion-scale similarity search with gpus. IEEE Transactions on Big Data, 7(3):535-547.
Soujanya Poria, Navonil Majumder, Rada Mihalcea, and Eduard Hovy. 2019b. Emotion recognition in con- versation: Research challenges, datasets, and recent advances. JEEE Access, -1000993-00095..
Nora Kassner and Hinrich Schütze. 2020. Bert- knn: Adding knn search component to pretrained language models for better qa. arXiv preprint arXiv:2005.00766.
Michael Schlichtkrull, Thomas N Kipf, Peter Bloem, Rianne Van Den Berg, Ivan Titov, and Max Welling. 2018. Modeling relational data with graph convolu- tional networks. In The Semantic Web: 15th Inter- national Conference, ESWC 2018, Heraklion, Crete, Greece, June 3-7, 2018, Proceedings 15, pages 593- 607. Springer.
Urvashi Khandelwal, Levy, Dan Jurafsky, Luke Zettlemoyer. and Mike Lewis. 2019, Generalization through memorization: Nearest neighbor language models. arXiv preprint arXiv:1911.00 191100172.
Diederik P Kingma and Jimmy Ba. 2014. Adam: method for stochastic optimization. arXiv preprint arXiv: 141..6980.
Weizhou Shen, Junqing Chen, Xiaojun Quan, and Zhix- ian Xie. 2021a. Dialogxl: All-in-one xinet for multi- party conversation emotion recognition. In Proceed- ings of the AAAI Conference on Artificial Intelligence, volume 33, pages 13789 137977.
Thomas N Kipf and Max Welling, 2016. Semi- supervised classification with graph convolutional networks. arXiv preprint 1rXiv:1609.02000
Weizhou Shen, Siyue Wu, Yunyi Yang, and Xiaojun Quan. 2021b. Directed acyelic graph network for conversational emotion recognition. arXiv preprint arXiv:2105.12907.
Joosung Lee and Wooin Lee. 2021. Compm: Context modeling with speaker's 's pre-trained memory track- ing for emotion recognition in conversation. arXiv preprint arXiv:210S. 1626.
Xiaohui Song. Longtao Huang, Hui Xue, and Songlin Hu. 2022. Supervised prototypical contrastive learn- ing for emotion recognition in conversation. arXiv preprint arXiv:2210.0871:
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad,, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. 2019. Bart: De- noising sequence-to sorecee----suunceee pre- pre-training for natural language generation, translation, and comprehension. arXiv preprint arXiv:79100 13461.
Jianlin Su, Jiarun Cao, Weijie Liu, and Yangyiwen Ou. 2021. Whitening sentence representations for bet- ter semantics and faster retrieval. arXiv preprint arXiv:2103.1331
Shimin Li, Hang Yan, and Xipeng Qiu. 2021. Contrast and generation make bart good dialogue emotion recognizer. arXiv preprint arXiv:2112.1 11202
Laurens Van der Maaten and Geoffrey Hinton, 2008 Visualizing data using t-sne. Journal of machine learning research, 9(1).
Yanran Li, Hui Su, Xiaoyu Shen, Wenjie Li, Ziqiang Cao, and Shuzi Niu. 2017. Dailydialog: manually labelled multi-turn dialogue dataset. arXiv preprint arXiv:1710.03957.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in neural information pro- cessing systems, pages 3998-6008
Yinhan Liu, Myle Oit, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis. Luke Zettlemoyer, and Veselin Stoyanov. 2019.
Roberta: A robustly optimized bert pretraining ap- proach. arXiv preprint arXiv: 190.119977..692.
Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. 2017. Graph attention networks. arXiv preprint 1rXiv 10903.
Navonil Majumder, Soujanya Poria, Devamanyu Haz- arika, Rada Mihalcea, Alexander Gelbukh, and Erik Cambria. 2019. Dialoguernn: An attentive rnn for emotion detection in conversations. In Proceedings of the AAAI Conference Artificial Intelligence, pages 6818-6825.
Lihan Wang, Bowen Qin, Binyuan Hui, Bowen Li, Min Yang. Bailin Wang, Binhua Li, Jian Sun, Fei Huang, Luo Si, et al. 2022. Proton: Probing schema linking information from pre-trained language models for text-to-sql parsing. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 1889-1899 1898.
Aditya Krishna Menon. Sadeep Jayasumana, Ankit Singh Rawat, Himanshu Jain, Andreas Veit, and Sanjiv Kumar. 2020. Long-tail learning via logit adjustment. arXiv preprint arXiv:2007. .0711.4
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. 2020. Transformers: State-of-the-aat natural language processing. In Proceedings of the 2020 con- ference on empirical methods natural language processing: system demonstrations, pages 38-45.
Soujanya Poria, Devamanyu Hazarika, Navonil Ma- jumder. Gautam Naik, Erik Cambria, and Rada Mi- halcea. 2019a. Meld: A multimodal multi-party dataset for emotion recognition in conversations. In Proceedings of the 57th Annual Meeting of the Associ- ation for Computational Linguistics, pages 527-536.

---

Kailai Yang. Shaoxiong Ji, Tianlin Zhang, Qianqian Xie. and Sophia Ananiadou. 2023. On the evaluations of chatgpt and emotion-eenanced prompting for mental health analysis. arXiv preprint
Wenbiao Yin and Lin Shang. 2022. Efficient nearest neighbor emotion classification with bert-whitening. In Proceedings of the 2022 Conference on Empiri- cal Methods in Natural Language Processing, pages 7738-4745.
Sayyed M Zahiri and Jinho D Choi. 2018. Emotion detection tv show transcripts with sequence-based convolutional neural networks. Workshops at the thirty-second aaai conference on artificial intelli- gence.
Ningyu Zhang, Xin Xie, Xiang Chen, Shumin Deng, Chuanqi Tan, Huang, Xu Cheng, and Huajun Chen. 2022. Reasoning through memorization: Near- est neighbor knowledge graph embeddings. arXiv preprint arXiv:2201.05572
Ting Zhang, Zhuang Chen, Ming Zhong, and Tieyun Qian. 2023. Mimicking the thinking process for emo- tion recognition conversation with prompts and paraphrasing.
Weixiang Zhao, Yanyan Zhao, Xin Lu, Shilong Wang, Yanpeng Tong, and Bing Qin. 2023. Is chat- gpt equipped with emotional dialogue capabilities? arXiv preprint arXhv:2304.09582.
Peixiang Zhong, Di Wang, and Chunyan Miao. 2019. Knowledge-enrichan transformer for emotion detec- tion in textual conversations. In Proceedings of the 2019 Conference on Empirical Methods Natu- ral Language Processing and the 9th International Joint Conference Natural Language Processing (EMNLP-IJCNLP) pages 165-176.

---

A Interaction between Utterance
C Experimental Results and Analysis
Following Ghosal et al. (2019), we use a two-step graph convolution process to obtain context-view features. In the first step, new feature vector h(") is computed for vertex by by aggregating local neighbourhood information using the relation spe- cific transformation inspired from (Schlichtkrull et al., 2018):
C.1 Generalization ability of the model
Based on the IEMOCAP dataset, we applied our proposed method to two graph network models, RGAT and DAG-ERC, and two non-graph network models, RoBERTa and COG-BART. The experi- mental results are shown table 5.
Model
Original
Original+OUrs
41 -
Cir W
RGAT
65.22
69.35
PERJENT
DAG-ERC
68.03
70.27
RoBERTa
63.38
65.96
for = ..,
COG-BART 66.18
69.12
(14)
where Wij and Wa are the edge weights, NT de- notes the neighbouring indices of vertex i under relation E R. Ca, = |NT is an activation func- tion such as ReLU, w and w are learnable parameters of the transformation. In the second step, another local neighbourhood based transfor- mation is applied over the output of the first step:
Table 5: Generalization ability of the proposed model. For the RoBERTa model, since does not model the context-view features, we only apply the memorization technique.
Based on the experimental results, applying our method to other graph networks and non-graph structured models mance improvements, indicating that our proposed method has good generalization capabilities.
bring significant perfor-
f=a Σ w(2) W n())
(1)
JENT
(15)
for 1,,2, .1,,...,,N N
C.2
Case Study
After eliciting correlations between utterances by a probing procedure from a pre-trained conversation model, the utterance will pay attention to the coher- ence of the context and the overall logic flow of the dialogue. Here is case in table 6, where nicely picks up the emotion of u1, and the emotions of W17 and U18 also correspond well. Meanwhile, by examining the relevance scores between W17 and other utterances,
where, (22) and 2( are parameters of these transformation and a is the activation function.
B Detailed Descriptions of ERC Datasets
IEMOCAP (Busso al., 2008): Multimodal ERC dataset. Each conversation within the IEMOCAP dataset comes from the performance based on script by two actors. Models are evaluated on the samples with 6 types of emotion, namely neutral, happiness, sadness, anger. frustraed and excited.
found that 1/7 has noticed the topic change at ug (a more sorrowful topic turning into an entertaining one).
MELD (Poria et al., 2019a): Multimodal ERC dataset gathered from the TV show Friends. 7 emotion labels are included: neutral, happiness, surprise, sadness, anger, disgust, and fear.
DailyDialog (Li et al., 2017): Dialogues penned by humans, collected from communications English learners. emotion labels are included: neutral, happiness, surprise, sadness, anger, disgust, and fear.
Emory (Zahiri and Choi, 2018): This dataset comprises TV show scripts obtained from the tele- vision series "Friends." " with variations in scene selection and emotion labeling. emotion labels are included: neutral, sad, mad, scared, powerful, peaceful, and joyful.

---

Index
Conversational Context
w/o probe
with probe
B: I'm really fed up with work at the moment need a break!
Disgust
Disgust
U2
A: are you doing anything this weekend
Neutral
Neutral
u3
B: I have to work on Saturday all day really hate my job
Anger
Disgust
14
A: are you available Sunday? ?
Neutral
Neutral
05
B: yes, that's my only day off until Thursday.
Neutral
Neutral
46
A: My friends and are planning on going to the beach. Come with
Neutral
Neutral
U17
B: ok, where and when should meet you ?
Neutral
Happiness
018
A: we'll pick you up at your place at noon... Be there or be square
Happiness
Happiness
Table 6: A case that our method gives the correct result.


Reviews:
[{"review_id": "EfwruT3usp", "reviewer": "Reviewer_NUwX", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper is well-motivated and well-written. It addresses an important problem in dialog and the proposed multi-view feature alignment and memorization is interesting and has potential to be used across other tasks. Extensive experiments and comparison with several baselines sufficiently demonstrate the effectiveness of the proposed approach.", "reasons_to_reject": "I do not find any major weaknesses with the proposed work. Although some of the sub-sections (in section 3) took me multiple readings, the figures helped me to follow the key idea. One aspect that I feel is missing in the current version is the qualitative analysis (I couldn’t find them in supplementary material as well). It would be very helpful to look into some of the detailed qualitative examples/results (I see some analysis in section 5.4).", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper addresses the problem of emotion recognition in conversation where the goal is to identify the emotions conveyed in utterances within the dialogue. The paper proposes Multi-view Feature Alignment and Memorization (MFAM) method which model weights in GNN by eliciting correlations between utterances by a probing procedure from a pre-trained conversation model. The paper also adopts a supervised contrastive learning to align the features at semantic-view and context-view. Furthermore, the paper addresses the problem of tail class recognition by proposing a new semi-parametric paradigm of inference through memorization. Experiments demonstrate the utility of the proposed approach.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "The paper has few typos (example: Visulization -> Visualization in line 500). Please correct them in final version."}, "extra_text": {}}, {"review_id": "Dtj0svm3ga", "reviewer": "Reviewer_Gm2P", "summary": "", "questions": "Question A： The abstract looks a little vague. For example, “However, it is difficult to fully model interaction between utterances …” What is 'interaction between utterances' and why is it difficult to model? This information is not evident from the previous context. Additionally, the misalignment between the two views might seem obvious since most ERC models aggregate information using methods like residual connections. So, why the need for alignment? Isn't the goal to leverage the advantages of both features? Or does alignment help in achieving a balance between both features?\nQuestion B： How well does the proposed method generalize, and can it achieve improvements in other graph networks or non-graph-structured models?\nQuestion C： After addressing the issue of tail class sample recognition, does the model show improvement on samples with few classes? Does the performance on samples with multiple classes get affected?\n\nI'm happy to update the review if the authors can answer my concerns.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable time and reviews.\n\n**Response to R1**: *about the motivation of the paper\n\nThis paper first presents several major challenges existing in ERC tasks, including complex emotion interaction between utterances, class imbalance and fine-grained classification problems.\n\nFor addressing the challenge of complex emotional interactions between utterances, we use a pre-trained conversation model to implement the injection of the dialogue system's prior knowledge to fully models the interactions between utterances. However, previous methods solely use graph neural networks for modeling.\n\nMeanwhile, We discover the phenomenon of feature separation between semantic-view and context-view. Other methods may consider using residual connections to connect the features of these two views. However, as shown in Figure 2 in the paper, the features of the two views are separated, and directly performing residual connections will lead to bias in the representation. Considering that some utterances need to categorize emotions more from the semantic-view, then the context-view needs to move closer to the semantic-view, and vice versa. In addition, both views are important for emotion recognition and represent the expression of emotions in different views. Therefore, we propose for the first time to align the features of the semantic-view and context-view to form a unified emotional space. This unified emotional space provides multiple views for recognizing the emotions of utterances. \n\nFor addressing the long-tail challenge, we apply the memorization technique to the ERC task for the first time.\n\n**It is worth noting that the concept of a unified emotional space effectively connects the methods proposed in this paper**. By fully modeling the features of context-view, a better emotional space is formed. The memorization technique is also based on the unified emotional space to achieve joint retrieval of semantic-view and context-view features.\n\n**Response to R2, Q1**: *about the interaction between utterances\n\nDuring a conversation, the emotion of utterance expressed by a speaker is influenced by the other speaker's utterances and their own emotional state over the utterances, utilizing relational graph convolution networks to implement interactions between utterances can take into account emotional stimulus from other speakers and their own emotional inertia. Previous methods often involved setting a fixed window size and conducting interactions between utterances within the window. However, in order to fully model the interactions between utterances, relying solely on graph neural networks is not sufficient, prior knowledge of the dialogue system itself is also needed. We believe that pre-trained conversational models have the ability to grasp interactive relationships that graph neural networks are unable to learn. Ablation experiments have demonstrated the efficiency of incorporating prior knowledge of the dialogue system itself.\n\n*about the alignment\n\nMost ERC models aggregate information using methods like residual connections, However, as shown in Figure 2 in the paper, the features of the two views are separated, and directly performing residual connections will lead to bias in the representation. Our motivation is that the semantic-view and context-view features should be unified in the same emotional space, rather than being separate. Some utterances need to categorize emotions more from the semantic-view, then the context-view needs to move closer to the semantic-view, and vice versa. Both views are important for emotion recognition and represent the expression of emotions in different views. As shown in Figure 5 of the paper, the features of the two views come close together to form a unified emotional space. \n\n\n**Response to R3, Q3**: In the ablation study section, we demonstrated the effectiveness of the proposed modules. The term \"semi-parametric\"  refers to the fact that the model's parameters are obtained through training, while the memorization technique is parameter-free. Therefore, the combination of the two is considered semi-parametric.\n\nFor performance of the model on multiple classes and tail classes, as shown in the table below, after introducing the memorization technique, the model has significantly improved recognition performance on tail-class samples. In addition, the model also improved the performance for almost all of the multiple-class samples.\n\n| Emotion class | Neu   | Sur   | Fea*  | Sad   | Joy   | Dis   | Ang   |\n| ------------- | ----- | ----- | ----- | ----- | ----- | ----- | ----- |\n| **MELD**          | +0.75 | +1.79 | +5.49 | +2.11 | +1.09 | +1.35 | +1.76 |\n| Emotion class | Hap*  | Sad   | Neu   | Ang   | Exc   | Fru   | -     |\n| **IEMOCAP**       | +3.12 | +0.96 | +0.89 | +1.43 | +2.31 | +0.59 |       |\n| Emotion class | Joy   | Mad   | Pea   | Neu   | Sad*  | Pow   | Sca   |\n| **EmoryNLP**      | +1.88 | +3.75 | +2.58 | -0.06 | +1.89 | +2.87 | +1.29 |\n| Emotion class | Ang   | Dis   | Fea*  | Hap   | Sad   | Sur   | Neu   |\n| **DailyDialog**   | +3.12 | +2.89 | +3.45 | +1.45 | -0.23 | +0.66 | -     |\n\nNote: * represents tail class.\n\nAnalysis: In the final recognition process, memorization and model work together for collaborative inference. For the multiple-class samples, the model's decision results have a higher level of confidence, so the decision results of memorization may not affect the final recognition result. For the tail-class samples, the confidence of the model's decision is relatively low, so the memorization's decision results may affect the final recognition results. The formula for collaborative inference is described as:\n$$p_{final} = \\lambda p_{model} + (1 - \\lambda) p_{memorization}$$\nwhere $\\lambda$ = 0.7 for the IEMOCAP dataset, the collaborative inference examples of multiple-class and tail-class are shown in the table below.\n\n\n\n| $p_{model}$                      | $p_{memorization}$                | $p_{final}$                              | recognition result |\n| ------------------------------ | ------------------------------- | ------------------------------------- | ------------------ |\n| [**0.9**,0.02,0.02,0.03,0.01,0.02] | [0.25,**0.35**,0.15,0.15,0.05,0.05] | [**0.705**,0.119,0.059,0.066,0.022,0.029] | class 0            |\n| [0.35,**0.4**,0.05,0.05,0.05,0.1]  | [**0.5**,0.1,0.1,0.1,0.1,0.1]       | [**0.395**,0.31,0.065,0.065,0.065,0.1]    | class 0            |\n\n\n**Response to R4**:  Since the method in this paper is based on supervised contrastive learning, the self-supervised part has not been described. We will supplement it accordingly. Thank you for your suggestion.\n\n**Response to R5**: In the results comparison section, we compared various methods. At the same time, the paper you mentioned is the latest research result accepted by ACL 2023.  The data in the Arxiv version is consistent with the ACL 2023 version, and in our revised version, we will change the citation to be from ACL.\n\n**Response to R6**: We run 10 times on each dataset, and based on the results of these 10 runs, we perform a one-sample t-test. The test results are shown in the table below. From the experimental results, it can be seen that the improvement of our model over all baselines is statistically signiﬁcant with p<0.05.\n\n| Dataset     | p     | t     | df   |\n| ----------- | ----- | ----- | ---- |\n| IEMOCAP     | 0.018 | 2.873 | 9    |\n| MELD        | 0.045 | 2.320 | 9    |\n| DailyDialog | 0.001 | 5.072 | 9    |\n| EmoryNLP    | 0.025 | 2.693 | 9    |\n\n**Response to R7**: Thank you for your suggestion, we will learn from your thoughts in the future. Your concern about robustness may be due to the large step size we used when traversing hyperparameters. We will consider using AutoML to automatically search for the best hyperparameters in the future.\n\n**Response to R8**: Thank you for your suggestion, we will add the figure of error analysis to the paper.\n\n**Response to R9-R10**: Thank you for your suggestion, we will supplement and modify the content of the paper according to your reviews.\n\n**Response to Q2**: Based on the IEMOCAP dataset, we applied our proposed method to two graph network models, RGAT and  DAG-ERC, and two non-graph network models,  Roberta and COG-BART. The experimental results are shown in the following table.\n\n| Model    | Original | Original + Our proposed method |\n| -------- | -------- | ------------------------------ |\n| RGAT     | 65.22    | 69.35（+4.12）                 |\n| DAG-ERC  | 68.03    | 70.27（+2.24）                 |\n| Roberta  | 63.38    | 65.96（+2.58）                 |\n| COG-BART | 66.18    | 69.12（+2.94）                 |\n\nNote: For the Roberta model, since it does not model the context-view features, we only apply the memorization technique.\n\nAnalysis: Based on the experimental results, applying our method to other graph networks and non-graph structured models can bring significant performance improvements, indicating that our proposed method has good generalization capabilities.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "-\tReasonable method.\n-\tThe proposed method outperforms the baselines.", "reasons_to_reject": "-\tVery difficult to follow the motivation of this paper. And it looks like an incremental engineering paper.\n-\tThe abstract looks a little vague. For example, “However, it is difficult to fully model interaction between utterances …” What is 'interaction between utterances' and why is it difficult to model? This information is not evident from the previous context. Additionally, the misalignment between the two views might seem obvious since most ERC models aggregate information using methods like residual connections. So, why the need for alignment? Isn't the goal to leverage the advantages of both features? Or does alignment help in achieving a balance between both features?\n-\tThe author used various techniques to enhance performance, including contrastive learning, external knowledge, and graph networks. However, these methods seem contradictory to the limited experiments conducted. For example, the author proposes a new semi-parametric inferencing paradigm involving memorization to address the recognition problem of tail class samples. However, the term \"semi-parametric\" is not clearly defined, and there is a lack of experimental evidence to support the effectiveness of the proposed method in tackling the tail class samples problem.\n-\tThe Related Work section lacks a review of self-supervised contrastive learning in ERC.\n-\tThe most recent comparative method is still the preprint version available on ArXiv, which lacks convincing evidence.\n-\tTable 3 needs some significance tests to further verify the assumptions put forward in the paper.\n-\tIn Chapter 5.3, the significant impact of subtle hyperparameter fluctuations on performance raises concerns about the method's robustness. The authors could consider designing an automated hyperparameter search mechanism or decoupling dependencies on these hyperparameters to address this.\n-\tThere is a lack of error analysis.\n-\tThe formatting of the reference list is disorganized and needs to be adjusted.\n-\tWriting errors are common across the overall paper.", "questions_to_authors": "Question A： The abstract looks a little vague. For example, “However, it is difficult to fully model interaction between utterances …” What is 'interaction between utterances' and why is it difficult to model? This information is not evident from the previous context. Additionally, the misalignment between the two views might seem obvious since most ERC models aggregate information using methods like residual connections. So, why the need for alignment? Isn't the goal to leverage the advantages of both features? Or does alignment help in achieving a balance between both features?\nQuestion B： How well does the proposed method generalize, and can it achieve improvements in other graph networks or non-graph-structured models?\nQuestion C： After addressing the issue of tail class sample recognition, does the model show improvement on samples with few classes? Does the performance on samples with multiple classes get affected?\n\nI'm happy to update the review if the authors can answer my concerns.", "paper_topic_and_main_contributions": "This paper proposes a new paradigm, treating a pre-trained conversation model as a knowledge base, eliciting correlations between utterances through probing, and using supervised contrastive learning to align semantic and context features. Additionally, they introduce semi-parametric inferencing through memorization to handle tail class recognition. Experimental results demonstrate that the proposed approach achieves state-of-the-art results on four benchmarks.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "O2MRM2lLOb", "reviewer": "Reviewer_oCgX", "summary": "", "questions": "1. What is \"PLMs\" in the abstract? Assuming all readers are familiar with certain abbreviations can lead to poor readability.\n2. In lines 039-041, the definition of ERC is to predict emotion labels of each utterance based on the surrounding context, which seems a bit confusing. Even when not in a conversation, I think this definition is still valid, then why is it called emotion recognition in conversation?\n3. Conflicting claims: In lines 056-060, the authors analyzed that it's difficult to fully model interaction simply through a graph neural network, but the proposed MFAM is based on graph neural networks. \n4. How is Figure 2 obtained? What are the exact meanings of points in different colors? How do you judge that they are context-view or semantic-view? Giving a figure of two separate parts of points cannot convince the audience.\n5. Why do the authors extract commonsense features in Section 3.2.1? \n6. Any theoretical reason for choosing Euclidean distance in equation (4)?\n7. Equation (3) uses $f$ to denote correlation, then in line 266, $f$ denotes future utterances, then in line 286, $f$ denotes the context-view feature; This leaves a bad impression and the authors should be careful when writing notations.\n8. How is the determination of b in Equation (5)?\n9. In lines 283-288, I understand the space is limited, but at least here the basic process should be explained clearly instead of referring readers to the appendix. What is the two-step graph convolution process? Assuming all readers know how GCN works is already not appropriate, leave alone a \"two-step\" GCN which may contain some new operations.\n10. Any specific reason for choosing BERT-whitening models for dimension reduction in Equation (8)?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable time and reviews.\n\n**Response to R1**: *about the motivation of the paper\n\nThis paper first presents several major challenges existing in ERC tasks, including complex emotion interaction between utterances, class imbalance and fine-grained classification problems.\n\nFor addressing the challenge of complex emotional interactions between utterances, we use a pre-trained conversation model to implement the injection of the dialogue system's prior knowledge to fully models the interactions between utterances. However, previous methods solely use graph neural networks for modeling.\n\nMeanwhile, We discover the phenomenon of feature separation between semantic-view and context-view. Other methods may consider using residual connections to connect the features of these two views. However, as shown in Figure 2 in the paper, the features of the two views are separated, and directly performing residual connections will lead to bias in the representation. Considering that some utterances need to categorize emotions more from the semantic-view, then the context-view needs to move closer to the semantic-view, and vice versa. In addition, both views are important for emotion recognition and represent the expression of emotions in different views. Therefore, we propose for the first time to align the features of the semantic-view and context-view to form a unified emotional space. This unified emotional space provides multiple views for recognizing the emotions of utterances.\n\nFor addressing the long-tail challenge, we apply the memorization technique to the ERC task for the first time.\n\n**It is worth noting that the concept of a unified emotional space effectively connects the methods proposed in this paper**. By fully modeling the features of context-view, a better emotional space is formed. The memorization technique is also based on the unified emotional space to achieve joint retrieval of semantic-view and context-view features.\n\n*about the alignment \n\nSome utterances need to categorize emotions more from the semantic-view, then the context-view needs to move closer to the semantic-view, and vice versa. Both views are important for emotion recognition and represent the expression of emotions in different views. Therefore, we consider aligning the features of the semantic-view and context-view to form a unified emotional space.\n*about the interaction between utterances\n\nIf we do not align the features of these two views and directly use context-view or semantic-view for emotion recognition, the information of one view will be lost. If we directly perform residual connections between the features of the two views, it will cause a bias in the representation since they are separate from each other.\n\n*about the interaction between utterances\n\nIntra-speaker and inter-speaker interactions between utterances mimic the emotional inertia and emotional stimulus, while global and local interactions represent the overall and specific emotional development in conversations. Previous methods often involved setting a fixed window size and conducting interactions between utterances within the window. However, in order to fully model the interactions between utterances, relying solely on graph neural networks is not sufficient, prior knowledge of the dialogue system itself is also needed. We believe that pre-trained conversational models have the ability to grasp interactive relationships that graph neural networks are unable to learn. Ablation experiments have demonstrated the efficiency of incorporating prior knowledge of the dialogue system itself.\n\n**Response to R2**: Firstly, to the best of our knowledge, we are the first to apply memorization to ERC tasks. Secondly, memorization techniques do not exist in isolation, but are built upon a unified emotional space. Based on the unified emotional space, which aligns semantic-view and context-view features, memorization performs joint semantic-view and context-view features retrieval.\n\n**Response to R3, Q1**: Thank you for your suggestion. The PLMs mentioned here are pre-trained language models. In the future, we will pay attention to the readability issues you mentioned.\n\n**Response to R3, Q2**: We highly agree with your opinion, and when defining it here, we need to incorporate the element of dialogue. For example, ERC is to assign emotion labels to all the utterances within a dialogue from a pre-deﬁned emotion category set.\n\n**Response to R3, Q3**: Apologies for causing any confusion. Let us explain to you that simply using graph neural networks for interaction between utterances is not enough, so we injected the prior knowledge of the dialogue system itself during the calculation process of the graph neural network.\n\n**Response to R3, Q4**: After the model training is close to convergence, the dataset samples are passed through the model to obtain the features of their semantic-view and context-view, and visualized with the help of t-SNE technology. When drawing the figure, we first draw the feature distribution of the semantic-view, then the feature distribution of the context-view, and finally summarize them in one figure. Therefore, we can find that there is a bias in the feature representation. \n\n**Response to R3, Q5**: Extracting utterance-related commonsense features helps to understand the speaker's intention and behavior, such as xWant representing what the speaker may want to do after the event, and xIntent representing the reason why the speaker would cause the event. Moreover, commonsense features also help in understanding colloquial expressions in dialogues.\n\n**Response to R3, Q6**: Since Euclidean space is a relatively common type of metric space and can effectively measure the distance between vectors, it was initially considered. Inspired by the question you raised, we researched other metric spaces and found that non-linear spaces, such as hyperbolic spaces, can better model the non-linear and hierarchical dependencies between utterances. We conducted preliminary experiments on MELD and found that hyperbolic space might have advantages over Euclidean space in modeling the correlations between utterances.\n| Space            | Performance |\n| ---------------- | ----------- |\n| Euclidean space  | 66.65       |\n| Hyperbolic Space | 66.89       |\n\n\n**Response to R3, Q7**: Thank you for your suggestion. We will pay special attention to the standardized use of symbols in the future.\n\n**Response to R3, Q8**: b represents the injection intensity of prior knowledge related to the dialogue system, so we consider its value should not be too large. We treat b as a hyperparameter, and traverse it within the range of 0.05-0.20 with a step size of 0.05, choosing the parameter with better performance. For example, in the IEMOCAP dataset, a good performance is achieved when b is 0.1.\n\n**Response to R3, Q9**: Thank you for your suggestion. We will incorporate the basic process into the main text of the paper. The first step of the operation is to compute a new feature for each utterance by aggregating local neighborhood information using the relation speciﬁc transformation, which takes into account different types of relationships such as intra-speaker, inter-speaker, past utterances, and future utterances. In the second step, another local neighborhood-based transformation is applied over the output of the ﬁrst step. \n\n**Response to R3, Q10**: BERT-whitening employs the SVD algorithm to achieve dimension reduction while enhancing the isotropy of sentence representations. Effectively improve retrieval results and accelerate retrieval speed.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. State-of-the-art results on four benchmarks and extensive experiments.\n2. An interesting manner of calculating correlations between utterances, while the novelty is not guaranteed.", "reasons_to_reject": "1. Relatively weak or unclear motivation: the features of semantic-view and context-view are not aligned, but why should they be aligned? What's the damage if they are not? Why is it difficult to fully model interactions through a graph neural network? More insightful motivations should be given. \n2. The contribution of the memorization technique is relatively invalid since the proposed technique is basically storing pairs of feature vectors and labels and using clustering techniques to help classification. This is a cliche technique, but the authors use new models within. \n3. Poor readability with many confusing descriptions, see Questions for Authors for details.", "questions_to_authors": "1. What is \"PLMs\" in the abstract? Assuming all readers are familiar with certain abbreviations can lead to poor readability.\n2. In lines 039-041, the definition of ERC is to predict emotion labels of each utterance based on the surrounding context, which seems a bit confusing. Even when not in a conversation, I think this definition is still valid, then why is it called emotion recognition in conversation?\n3. Conflicting claims: In lines 056-060, the authors analyzed that it's difficult to fully model interaction simply through a graph neural network, but the proposed MFAM is based on graph neural networks. \n4. How is Figure 2 obtained? What are the exact meanings of points in different colors? How do you judge that they are context-view or semantic-view? Giving a figure of two separate parts of points cannot convince the audience.\n5. Why do the authors extract commonsense features in Section 3.2.1? \n6. Any theoretical reason for choosing Euclidean distance in equation (4)?\n7. Equation (3) uses $f$ to denote correlation, then in line 266, $f$ denotes future utterances, then in line 286, $f$ denotes the context-view feature; This leaves a bad impression and the authors should be careful when writing notations.\n8. How is the determination of b in Equation (5)?\n9. In lines 283-288, I understand the space is limited, but at least here the basic process should be explained clearly instead of referring readers to the appendix. What is the two-step graph convolution process? Assuming all readers know how GCN works is already not appropriate, leave alone a \"two-step\" GCN which may contain some new operations.\n10. Any specific reason for choosing BERT-whitening models for dimension reduction in Equation (8)?", "paper_topic_and_main_contributions": "The manuscript proposes a multi-view feature alignment and memorization framework for emotion recognition in conversation. The main contributions are aligning semantic-view and context-view features, memorization techniques for improving tail-class classification, modeling the interactions between utterances via probing, and state-of-the-art results on four benchmarks.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "NA.", "Typos_Grammar_Style_and_Presentation_Improvements": "See Questions for Authors."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper describes a new approach to emotion recognition in context by combining a number of techniques. Reviewers were all impressed by the performance of the model, though there were multiple requests for additional comparisons/experiments to understand the performance more (e.g., through an error analysis). The approach represented a meaningful engineering contribution to ERC. However, reviewers were mixed on understanding the motivation for the approach itself and whether the individual pieces (e.g., the graph neural network) were contributing or interacting to solve the ERC task. Some of these issues were caused by the clarity in the writing, though one reviewer was able to follow completely. Overall, found the paper's methods promising but highlighted multiple ways the manuscript could be improved to strengthen the paper's clarity and subsequent impact.
Decision: Accept-Findings