A Training-Free Debiasing Framework with Counterfactual Reasoning for Conversational Emotion Detection
Authors: Geng Tu
Ran Jing
Bin Liang
Min Yang
Kam-Fai Wong
Ruifeng Xu
Venue: EMNLP 2023 Main (2023)
Keywords: Conversational Emotion Detection
Counterfactual Reasoning
Debiasing

Abstract: Unintended dataset biases typically exist in existing Emotion Recognition in Conversations (ERC) datasets, including label bias, where models favor the majority class due to imbalanced training data, as well as the speaker and neutral word bias, where models make unfair predictions because of excessive correlations between specific neutral words or speakers and classes. However, previous studies in ERC generally focus on capturing context-sensitive and speaker-sensitive dependencies, ignoring the unintended dataset biases of data, which hampers the generalization and fairness in ERC. To address this issue, we propose a Training-Free Debiasing framework (TFD) that operates during prediction without additional training. To ensure compatibility with various ERC models, it does not balance data or modify the model structure. Instead, TFD extracts biases from the model by generating counterfactual utterances and contexts and mitigates them using simple yet empirically robust element-wise subtraction operations. Extensive experiments on three public datasets demonstrate that TFD effectively improves generalization ability and fairness across different ERC models.

A Training-Free Debiasing Framework with Counterfactual Reasoning for Conversational Emotion Detection
Geng Tu Ran Jingh.; Bin Liang,, Min Yang", Kam-Fai Wong', Ruifeng Xu
Harbin Institute of Technology, Shenzhen, China
"Guangdong Provincial Key Laboratory Novel Security Intelligence Technologies
The Chinese University of Hong Kong, Hong Kong, China SIAT, Chinese Academy of Sciences, Shenzhen, China
Peng Cheng Laboratory, Shenzhen, China tugeng031 3ēgmail. com, xurui www.gehhiieeuucccmmm. cn
Abstract
Jerkd I2220
Mad esg
Pauerd
Unintended dataset biases typically exist in ex- isting Emotion Recognition in Conversations (ERC) datasets, including label bias, where models favor the majority class due to imbal- anced training data, as well the speaker and neutral word bias, where models make unfair predictions because of excessive correlations between specific neutral words or speakers and classes. However, previous studies in ERC gen- erally focus on capturing sentext-sensitive and speaker-sensitive dependencies, ignoring the unintended dataset biases data, which ham- pers the generalization and fairness in ERC. To address this issue, we propose a Training-Free Debiasing framework (TFD) that operates dur- ing prediction without additional training. To ensure compatibility with various ERC models, does not balance data or modify the model structure. Instead, TFD extracts biases from the model by generating counterfactual utter- ances and contexts and mitigates them using simple yet empirically robust element-wise sub- traction operations. Extensive experiments on three public datasets demonstrate that TFD ef- fectively improves generalization ability and faimess across different ERC models!.
Narall
Sadg
6.
Chavdier Wed
Rom Gobn witt Bode Bouck
Currt
Figure I: The probability distribution of different speak- and neutral words, confirms various biases in ERC.
et al., 2018; Majumder et al., 2019; Ghosal et al., 2020; Jiao et al., 2020; Li et al., 2022), transformer- based network (Zhong et al., 2019; Lian et al., 2021; Shen al., 2021a; Ong al., 2022), and graph-based network (Ghosal al., 2019; Shen et al., 2021b; Saxena al., 2022).
Although these endeavors have achieved satis- factory results, they all overlook unintended biases present in the datasets, thereby impeding their gen- cralization capabilities and fairness (Goyal et al.. 2017; Niu et al., 2021). Different from other tasks, in ERC, these biases may manifest at both the ut- terance level, such label and speaker biases, well the word level, like the neutral word bias. Label bias, as depicted in Fig. 1, is evident in the EmoryNLP dataset's training set, where 32.9% (2485 utterances) are labeled as 'neutral' and 6.3% (474 utterances) are labeled as 'sad'. Numerous prior studies (Dixon et al., 2018; Zhang et al., 2020) have indicated that models trained on such imbal- anced data susceptible to the tendeney of pre- dominantly predicting the majority class. Speaker and neutral word biases arise from trained mod- els exhibiting strong associations between specific words and particular emotion categories. For in- stance, in EmoryNLP, it is highly probable for
1 Introduction
Emotion recognition in conversations (ERC) has garnered substantial research interest in recent years (Mao et al.. 2021; Xic et al., 2021; Jiang et al., 2022: Tu et al., 2023b). This attention stems from its promising applications in various domains, in- cluding recommendation systems and dialogue gen- eration (Tu al., 2022a). ERC aims identify the emotion of each utterance in conversations. Current efforts primarily focus on modeling context- and speaker-sensitive dependencies (Lian et al., 2021). This includes recurrent- based network (Hazarika
"The first authors contribute equally this work. Corresponding author.
'The code is available at httts:: htttps://github com/ TuGengs/TFD.

---

Richard Burke to be associated with emotions such as 'joyful', 'neutral', and 'peaceful', rather than 'mad' as depicted in Figure I. Additionally, neutral words also display such biases in emotion distribu- tion. Consequently, models tend to unfairly assign utterances containing these emotionless keywords to specific categories based on biased statistical in- formation, rather than relying intrinsic textual semantics (Qian et al., 2021; Liu and Avei, 2019; Waseem and Hovy, 2016).
textual information for identifying emotions. RNN- based models have been commonly used to cap- ture context dependencies (Poria et al., 2017), but they struggle to distinguish between historical ut- terances (Lian et al., 2021). In order to overcome this drawback, researchers have directed their at- tention toward memory networks (Jiao et al., 2020; Hazarika et al., 2018). Additionally, the role of participants in emotional response classification (ERC) is crucial (Wen et al., 2023), leading to the development of speaker-specifer (Kim and Vossen, 2021; Majumder ct al., 2019), graph-based mod- els (Ghosal et al., 2019; Tu et al., 2022b; Shen et al., 2021b) and so on. However, these approaches still lack commonsense knowledge, which is impor- tant for human-like performance (Tu et al., 2023a). To tackle this, researchers have integrated exter- nal knowledge sources such as COMET (Bosselut et al., 2019). SenticNet (Cambria al., 2022) and ConceptNet (Speer et al., 2017) into their mod- els (Zhong et al., 2019; Ghosal et al., 2020; Zhao et al., 2022). Despite these advancements, unin- tended dataset biases have been neglected, which hinders the generalization of ERC models. Dataset debiasing: To address dataset bias, approach is manipulate the data during model training to prevent unintended biases from being captured. These manipulations include techniques like data balance or resampling methods mentioned in studies such as (Geng et al., 2007; Kang et al., 2016; Sun et al., 2018; Wang and Culotta, 202 1), as well as data augmentation methods described by Qian et al. (2020). Another common strategy is to incorporate model-level balancing mechanisms. These mechanisms aim mitigate bias by using un- biased embeddings (Kaneko and Bollegala, 2019), adjusting thresholds (Kang et al., 2019), and ap- plying instance weighting techniques (Tian et al.. 2022; Zhao et al., 2017; Zhang et al., 2020). How- ever, the data-level strategy results in additional manual costs for data manipulations and longer training time. On the other hand, the model-level strategy necessitates a meticulous selection of bal- ancing techniques and retraining whenever there are changes in the balancing mechanisms.
Existing debiasing methods mainly involve two approaches: Data-level manipulations, like resam- pling (Qian et al., 2020) and generating counter- factual samples (Wang and Culotta, 2021), aim to balance the training set but increase training time. Model-level balancing mechanisms, such counterfactual reasoning (Tian et al., 2022) and reweighting (Zhang et al., 2020), adjust category in- fluence during training but require careful strategy selection and retraining from scratch.
In contrast, our proposed Training-Free FFee Debias- ing framework (TFD) focuses on counterfactual reasoning during the prediction process without extra data augmentation and any training costs. Be- cause generating counterfactual examples while maintaining conversation structure integrity is chal- lenging, inspired by Qian al. (2021), we employ a masking mechanism to obtain counterfactual ex- amples from the original input data to extract biases suffered from trained models. We then effectively mitigate these extracted biases using empirically robust wlse subtraction operations. Our main contributions are as follows:
To our knowledge, we are the first debiasing study for ERC.
We propose TFD to tackle biases in ERC without extra data augmentation and any train- ing costs, which employs counterfactual ut- terances and contexts to extract these biases suffered from the trained model.
We conduct extensive experiments on three public datasets to illustrate the effectiveness of TFD in enhancing generalization and fair- ness across various ERC models. And the TFD-based baseline demonstrates the ability to outperform saate-of-the-ae methods.
3 Methodology
3.1 Task Definition
2
Related Work
Consider a conversation C [u,, U2, ..., Un) con-
ERC: The emotion generation theory (Gross and Barrett, 201 1) emphasizes the importance of con-
sisting of n utterances. Each utterance u, is spoken by one of the speakers in S = {si, S2,.., Sm))

---

Biased Classification
Biased Extraction (Neutral word, Speaker, and Label Bias)
Unbiased Inference
u1
Trained Classifiaation Module
Fine grainedd Classification Module
01
Label Blas
05
og
U2
02
Neutral word
Element-
113
W
Og
03
Speaker Bias
114
115
04
W Ml
u5
Differeut
Probability distribution of different emotions
og
masking strategles
5
Figure 2: The proposed TFD framework, The bias extraction and unbiased inference are only conducted during the testing phase. Mathematical symhols in the illustration are consistent with formulas paper.
The objective of the ERC task is to predict the emo- tion label y; for each utterance by incorporating the preceding utterances { (u:-w,-. -..; 2;-2; u;-1). Y {y1 .n, represents predefined set
plicitly into the input text.
x, (Xc(u:)
(1)
x = (X,,(: (u.)i)))
(2)
of emotions.
where represents the size of the context, X,, and X represent the encoding strategies for histori- cal utterances and the current utterance, respec- tively. without using future information. For X., we prepend speaker says: at the start of historical and current utterances to indicate the speaker's 's in- formation. Additionally, (s) and (/s) are employed to enclose each utterance for emphasis. In essence, the current utterance can be represented as follows:
3.2 Overview
Considering the additional manual and training costs brought by data-level and model-level de- biasing approaches. we propose TFD, Training- Free Debiasing framework, which draws inspira- tion from the effective use of counterfactual rea- soming to address biases in computer vision, as demonstrated in prior works (Tang et al., 2020; Yang et al., 2021; Niu et al., 2021). TFD aims to mitigate biases in trained models through counter- be integrated
X = (8) 8 says: u; (/s)
(3)
For Xh. speaker says: serves the sole token, considering the support of contexts and speakers.
factual reasoning prediction, into various ERC models, and we employ Roberta as the classification model to exemplify its imple- mentation. TFD comprises three primary compo- nents: biased classification, bias extraction, and unbiased inference (refer to Fig. 2).
Xu(u,) sj says uj, Vj (i w, i) (4)
Classification Module: Traditional fine-tuning methods often use the class token for classifica- tion because most tasks involve plain text without specific structures. However, in our case, the put text x; has a where the query utterance takes the leading position and the contexts play a supporting role. Moreover, there is temporal structure present. Therefore. propose fine-tuning classification module that considers the characteristies of ERC. The module's specifies will be outlined below.
subordinate structure,
3.3 Biased Classification
Utterance representation x; is typ- ically generated by directly inputting query utter- ance u, into pre-trained language model (PLM). Considering that concatenating utterances alone may significantly improve results as PLMs lack dialogue structure during pretraining. Therefore, inspired by Kim and Vossen (2021), we integrate dialogue structure and contextual information ex-
Input Format:
E, mean (PLM (x;))
(5)
E, mean (PLM (x:)
(6)
o, MLP (FC(E, ! E,))
(7)

---

mour al., 2016) with do-calculus calculate the corresponding intervention distribution:
P(o)do(x. x)) - P(ojdo(x - -- (8)
where с and č can take any form long they are no longer influenced by q. effectively breaking the connection between x, and q. To extract neutral word or speaker biases, in the causal intervention operation, c / are set to c² / or c² / which aims to retain only neutral words or speaker-related words. By revealing specific words tied to the emo- tion categories, the goal is to expose any spurious connections in the trained model and emphasize their potential negative impact.
(a) Causal Graph
(b) Causal Graph with Confounder
Figure The comparison between factual ERC and counterfactual ERC.
where o; the prediction emotion. Ez/h is the features of u, and its contexts, generated by PLM. @ denotes a concatenation operation. FC is the fully connected layer with Tanh activation func- tion. MLP represents the multilayer perceptron.
ct = (s) {WI, (mask) .... Wn. } (/5)
(9)
3.4
Bias extraction
Vw; - (mask) $ S
3.4.1
Causal Graph
c) = [W
VW: < (mask) $ S (10)
Causal graph (Glymour et al., 2016) is a compre- hensive representation of the causal relationships among variables. It consists of nodes (variables) and directed edges (arrows) that indicate causality, In this directed acyclic Bayesian graphical model, denoted as G (N,E), the nodes represent vari- ables, and the arrows represent causal relationships between them. For instance, x Y indicates that x is the cause and Y is the effect, meaning that the value of Y is influenced by x.
where S denotes the set of speakers. In this case, since the model cannot see any word except for words related to speakers x and X after mask- ing, the final predictive logits of reflect the pure influence of speakers to the trained biased model. Similarly, we can also obtain the counterfactual ut- terance and the counterfactual historical utterance éj and él by using pysentiment library² to deter- mine non-neutral words for masking.
We define the causal graph of ERC in Fig. 3.(a). Nodes & and h represent the input utterance, its context, and the combined feature of and & re- spectively. The final predictive logits take inputs from the three branches: the direct effect of the input x and & on o via x + o and & -> o, as well as the indirect effect of the input x and x on o via the combined feature h, i.c. h + o.
3.4.3 Label Bias
To address the bias of models to favor the majority class in imbalanced data, we generate counterfac- tual utterances с and contexts through causal intervention to extract label biases.
cj = (s) { {W1, (mask) Wn (/s) (11) é (W1,.... Wm J. Vw; W; + (mask) (12)
3.4.2 Speaker and Neutral Word Biases
where (mask) is used to hide a single token in the input sequence The corresponding predictive logits of only reflect the impact of the label bias because the model cannot observe any words after the complete masking.
As shown in Fig. 3.(b), there exists unobserved confounders q and q, which is the cause of the spurious correlation between the x, and o. Such unobserved confounders may occur due to trained models exhibiting strong associations between spe- cific words (such as neutral word and speaker) and particular emotion categories. As result, mod- cls tend to unfairly assign outputs containing these keywords specific categories based on biased tistical information, more than releasing on intro- ductory textual semantics, which is actually not rea- sonable for the ERC task. To decouple the spurious correlation, we use the backdoor adjustments (Gly-
3.5 Unbiased Inference
Debiasing predictive logits o, from the biased trained model can be formulated using a simple element-wise subtraction operation.
o = 0; -
(13)
2 Available at http:///wpiipyhhooo../pppppppppp

---

Dialogues
Uiterances
Lee, 2022), AGHMN (Jiao et al., 2020); Graph-
Dataset
train
vul
test
train
val
Test
based models: DAG-ERC (Shen et al., 2021b), SKAIG (Li et al., 2021); Transformer-based mod- els: KET (Zhong et al., 2019), BERT BERT_BASE (Ken- ton and Toutanova, 2019), EmoBERTa (Kim and Vossen, 2021), Roberta (Liu et al., 2019); Gen- erative models: Curie (Olmo et al., 2021), Chat- GPT (Ouyang et al., 2022)².
JEMOCAP
120
31
5,810
1,623
MELD
1039
114
280
9.989
1,109
2610
EmoryNLP
659
89
79
7,551
934
984
Datuset
Classes
Metric
JEMOCAP
6
Weighted Avg. Fl Weighted Avg. over and classes
MELD
3 and 7
EmoryNI. NL.P and 7
Weighted Avg. Fl over 3 and 7 classes
4.3
Experimental Settings
Table Statistics of experimental datasets.
All models mentioned in Table 4 have released their source codes, and we have used the same settings as the original papers. The Roberta-based method (Baseline) described in this paper is based on EmoBERTa (Kim and Vossen, 2021), with the difference being that we do not consider the impact of future utterances on the model. Furthermore, we have utilized a grid search technique determine the optimal values for the parameters X, B, and y on the validation set. The grid search is performed with a step size of 0.1 and a range spanning from -2 to 2. The results reported in the tables are the average scores from 5 random runs on the test set.
where A, 3, and y are three independent parameters. It's important to note that biases have varying im- pacts on the final classification and are not equally important. Thus, we use elastic scaling to deter- mine three adjustable scaling factors that optimize the model's performance on the validation set.
4 Experiments
4.1 Datasets
We conduct experiments on three datasets; IEMO- CAP (Busso et al., 2008), EmoryNLP (Zahiri and Choi, 2018), and MELD (Poria et al., 2019a). The statistics of datasets are shown in Table 1.
5 Experimental Results
5.1 Main Results
LEMOCAP consists of dyadic sessions where actors perform improvisations or scripted scenarios. Each utterance is labeled with one of the emotions: angry, happy, sad, neutral, excited, or frustrated. Since there is no dedicated validation set in this dataset, we follow the approach from Shen et al. (2021b) by using the last 20 dialogues from the training set for validation.
As shown in Table 2, fine-tuning at the utterance level alone is unsatisfactory in ERC because it re- lies on context and the speaker's state information. EmoBERTa, a redesigned Roberta model with ad- justed input structures, exhibits significant perfor- mance improvement. CoMPM surprisingly out- performs other methods in IEMOCAP and MELD datasets, showcasing the effectiveness of complex memory mechanisms. Graph-based models gener- ally surpass RNN-based models in IEMOCAP and EmoryNLP datasets, capturing local context well in lengthy conversations. However, in MELD, where data from TV shows, the coherence between consecutive utterances may be lacking, reducing the advantage of graph-based models. Large mod- els underperform in ERC, possibly due to limita- tions in capturing intricate interactions, especially in lengthy conversations, as evidenced by their fail- the IEMOCAP dataset. Furthermore, Chat GPT's understanding of emotions is not limited predefined categories, which may affect its emo- tional comprehension. Prompt tuning the Curie model improves performance but is not sufficient.
MELD is multi-party conversation dataset col- lected from the TV show Friends. Each utterance is annotated with one of the emotions: fear, surprise, anger, disgust, sadness, neutral, or joy, and one of the sentiments: neutral, negative, or positive.
EmoryNLP consists of multi-party sessions from the TV show Friends and each utterance is la- beled with one emotion from the set: joyful, scared, peaceful, sad, powerful, mad, or neutral, along with one sentiment, as suggested in Ghosal al. (2020), from the set: neutral, negative, or positive.
4.2 Comparison Models
We compare our proposed framework with vari- ous ERC baselines, including RNN-based models: CauAIN (Zhao al., 2022), COSMIC (Ghosal et al., 2020), DialogueRNN (Majumder et al., 2019); Memory networks: CoMPM (Lee and
Pleasee refer to the appendix for the prompt templates used for ChatGPT and Curie.

---

Methods
IEMOCAP (6-cls)
MELD
(7-cl)
EmoryNLLP (3-cls)
EmoryNL.P (7-clk)
ChaGPT!
40.07
60.07
54.37
51,93
37.55
Curic!
57.33
65.01
37.40
BERT BASE
61.19
56.21
33.15
ROBERT
54.55
72.12
62.02
55.28
37.29
EmoBERTH
65.61
DialogueNNN
61.21
66.10
56.27
48.93
31.70
KET
AGHMN
58.10
DAG-ERC"
63.65
SKAIG"
COSMIC
65.28
73.20
65.21
56.51
38.11
CauAIN
67.61
65.46
COMPM'
69.46
66,52
38,93
Buseline
69.18
72.57
65.00)
56,84
38.81
Baselinet TFD
70.43 (1 1.25%)
73.72 (1 1.15%)
66.19 (1 1,79%)
$8.28 (1 1.47%)
40.51 (1 1.70%)
w/o Label dehiasing
69.66 [L 0.77%)
72.61 (E 1.17%)
65.35 [L 0.94%)
57.05 (L 1.23%)
39.19 (1 T.32%)
w/o Neutral word debiasing
69.45 () 0.98%)
72.77 0.95%)
65.31 0. 0.88%)
51.27 (1 1.01%)
39.28 (3 1.23%)
Speaker dehiasing
() 0.62%)
(.944%)
0.97%)
(). (1.55)
Table 2: Weighted Avg. F1 score (%) of different methods. Best test scores are in bold. represents our re- implementation results. Results with and are respectively retrieved from Zhong al. (2019) and Ghosal al. (2020). Results with are retrieved from the original papers.
This does imply that generative methods are ineffective for ERC; more task-specific factors like imbalanced samples and long-term context model- ing require careful consideration.
2500
Numbers
Baselino
80
2000
- wi Label debiasing
Numbers
1300
Score
40
1000
Considering that most methods rely on using Roberta for utterance representation extraction, we use Roberta the underlying model to validate the effectiveness of TFD. We can observe that the per- formance on sentiment and emotion identification our proposed bFDed method significantly boosted. And it achieves the best improvement re- sult of Fl on the EmoryNLP dataset, i.e., 1.70%. And the TTFD outperforms all compared methods on different datasets, except for slightly underperforming the CoMPM model on the MELD dataset. The failure could possibly be attributed to the elimination of the influence of future utterances on our model, as evidenced by the performance gap between our 8 and EmoBERTa.
500
0
0
Joyfu
Mad
Peaceful Neutral
Sad
Powerkul Scared
Figure 4: Performance of Baseline on the EmoryNLP validation set across different categories.
adequate recognition performance for utterances belonging to the minority classes, as depicted Fig. 4. Fortunately, with the introduction of the label debiasing strategy, this phenomenon has been effectively mitigated. From Fig. 4, we find that the debiased model significantly improves on the other minority classes, such 'peaceful', 'sad', and 'powerful', Despite a marginal decline in per- formance for the majority classes, the model ex- hibits overall improvement, with an increase from 39.19 to 40.51. The advantages outweigh the draw- backs in this case, which suggests the effectiveness of label debiasing and the reported results presented in Table 2 further validate this assertion.
5.2 Ablation Studies
To investigate the impact of each component of our proposed method, we conduct an ablation study on B + TFD, and the results are shown in Table 2. w/o represents the removing operation. The results suggest that all components of the TFD framework have worked and all the improvements are statistically significant, as evidenced by the paired I-test results with p-value < 0.05.
5.2.2 Speaker and Neutral Word Debiasing
We have observed improvements in speaker and neutral word debiasing across various datasets. Specifically, when we applied the debiasing strat- egy of removing the speaker and neutral words the EmoryNIP dataset, we noticed a significant decrease in the model's performance, resulting in a respective drop of 1.03% and 1.23%. However,
5.2.1 Label Debiasing
Due to the tendency models to prioritize the ma- jority class, such as the emotions 'joyful', 'scared', and 'neutral', imbalanced data often leads to in-

---

IEMOCAP (6-els)
(3-ck)
(7-cls)
EmoryNLPP
EmorryNLP(77cls)
Baselinel
2.11
4.42
9.16
9.87
26,66
Label debiasing
2.07 (1 0.04%)
4,16 0.26%)
7.66 (337,))))
8.34 (.53%)
3.039) (1,
w/ Neutral word debiasing
1.97 0.14%)
4,05 (L 0.37%)
7.55 (4 1.67%)
8.54 ( 1.33%)
24.32 (L 2.34%%
w/ Speaker debiasing
2.08 0.03%)
4,12 (L 0.30%
7.10 2.50%)
8.29 (1 1.58%)
23.42 (L 3.24%)
Table Imbalance divergence or unfaimess (%) of our underlying model (lower is better) for bias analysis.
the speaker debiasing approach does not yield the expected results on the IEMOCAP dataset, as illus- trated in Table 2. This disparity can be explained by the distinction between the IEMOCAP dataset and the other two datasets obtained from TV shows, where each speaker possesses distinct personality traits and emotional tendencies. In contrast, the improvisations in IEMOCAP do not prioritize the shaping of character images, naturally reducing the inclination towards specific emotional expres- sions. In neutral word debiasing, the effectiveness varies to some degree, but overall, the debiasing process demonstrates relatively consistent improve- ment across various datasets due to the prevalent presence of neutral words conversations.
Seatal Word dcbiaiizz
62
Figure 5: Performance of Baseline with various debi- asing strategies across different debiasing coefficients on the validation set of dataset EmoryNLP.
three datasets have class imbalance issues in the validation set, the severity is more pronounced in the other two datasets. In the IEMOCAP dataset, the ratio between the most frequent and least fre- quent classes in the training 2.98:1, while in MELD and EmoryNLPP, the ratios are 17.57:1 and 5.24:1, respectively. Additionally, for speaker debiasing, since IEMOCAP is derived from improvisational performances with less em- phasis on character portrayal, the bias related to speakers is not significant in the other two datasets. The reported results in Table 2 also reflect this observation. However, neutral word debias- ing shows considerable decrease in performance across all three datasets due to the generality of neutral word debiasing.
approximately
5.3 Hyper-parameter Analysis
We also explore the effects of different coefficients these debiasing strategies, shown in Figure 5, Generally, they exhibit a trend of initially increas- ing and then decreasing, even falling below the Baseline in the case of the neutral word debiasing. However, label debiasing is quite unique. After an initial increase, it consistently maintains sta- ble performance, consistently outperforming the Baseline This reflects the severe impact of label bias model performance, while the bias associ- ated with neutral words is relatively mild.
5.5 Generalization Analysis
To evaluate the generalizability of our TFD frame- work, we conduct the experiment on various ERC models as shown in Table 4 and With the excep- tion of the imbalance divergence for AGHMN on the IEMOCAP dataset and DialogueRNN on the 3- class classification task on the MELD dataset, TFD has been effeetive in all other cases. Morcover, the Fl score for all methods has shown improvement, which demonstrates the effectiveness and general- ization ability of our TFD framework.
5.4 Bias Analysis
Based on prior research (Xiang et al., 2020;
Sweeney and Najafian, 2019; Qian al., 2021), we use imbalance divergence as metric to mea- sure the degree of unfairness in prediction results. This metric allows us to assess how much a trained model's predictions favor specific predefined cate- gories, quantifying the discriminatory nature of the model's behavior. A higher imbalance divergence in the model's 's predictions indicates more unfair- ness. As shown in Table 3, our TFD has all resulted in decrease in the imbalance divergence of pre- dicting outcomes. However, label debiasing does not achieve a significant decrease in performance on the IEMOCAP dataset compared to the other two datasets. This could be because although all
5.6 Case Study
As shown in Table in the first utterance, the speaker most likely expresses a neutral emotion, leading the model associate her with a neutral emotion despite the phrase "Nice work." " Removvig the speaker bias results the correct prediction.

---

Methods
JEMOCAP (6-cls)
MELD (3-cls)
MELD (7-cls)
EmeryNLP (3-els)
EmoryNLP (7-cls)
DialogueRNN
66.79
72.46
64,65
$5,80
38.34
w/ THD
66.94 (f 0.75%)
73.17 0.77%)
64.81 (f 0.16%)
56.69 (1 0.89%)
39.39 ( 1.05%)
DAG-ERCS
66,42
72.56
65,39
56.26
37.84
w/ TFD
66.57 (f 0.75%)
73.37 ( 0,81%)
65 53 0.14%)
56.53 0.27%)
38.59 ( 0.75%)
AGHMN
65.21
72.45
64,19
54.61
37,86
w/ THD
65.46 (f 0.25%)
72.92 0.47%)
64.76 (f 0,57%)
56.09 (f 1.48%)
38.64 (" 0.78%)
KETS
63.16
71.42
62.67
54.62
37.73
w/ TFD
64.57 (1 1.41%)
72.31 0.89%)
63 46 (1 0.79%)
56 89 (1 2.27%)
38.39 ( 0.66%)
COSMIC!
60.86
72.71
64,76
56.23
39.16
w/ TFD
61.50 (f 0.645)
73.21 ( 0.50%)
64.99 ( 0.23%)
57.02 (f 0.79%)
39.85 ( 0.69%)
Table 4: Weighted Avg. Fl score (%) of different ERC methods for generalizability analysis.
Mathods
IEMOCAP (6-cls)
MELD (3-cls)
MELD (7-cls)
EmoryNLPP (3-cls)
EmoryNLP (7-cls)
Baseline!
2.11
4.42
9.16
9.87
26.66
w/ TFD
1.96 0.15%)
4.02()
6.98 (L 2.78%)
7.84 (J. 2.03%)
18.18 (J 8.48%)
DialogueRNN
10.82
5,96
13.29
11.80
22.22
w/ IFD
8.09 (L 2.73%)
6f90 0.94%)
10.26 (L 3.03%)
7.69 () 4.17%)
8.29 (4 13.93%)
DAG-ERC
17.46
7.44
17,80
11.78
21.41
w/TFD
17.32 (1 0.14%)
7.29 (1 0.15%)
() 0.15%)
() 2.10%)
(1 3.99%
AGHMN
10.60
6.87
18,80
22.11
TFD
10.60 (, 0.00%)
6.35 0.52%)
16.99 (L 1.81%)
11.70 1.53%)
16.71 5.40%)
KET
13.01
7,43
12.52
11.44
21.83
w/ TFD
9.58 (L 3.43%)
6.78 (L 0.65%)
10.06 (1 2.46%)
5.77 5.67%)
20.55 (1 1.28%)
COSMIC!
11.60
13.72
25.38
13.21
24.20
w/ TFD
11.32 0.28%)
11.19 (L 2.53%)
10. (L 75,27%)
10.66 2.55%)
17.46 (1 6. 74%)
Table 5: Imbalance divergence or unfaimesss (%) of different ERC methods for generalizability analysis.
ID
Utterances for Prediction
w/o TFD
w/TFD joyful peaceful
Golden Label
1
RACHEL: Okay get your coat! Oh! When did you unhook this? Nice work!
neural joyful neural
joyful
2
CHANDLER: Act like processor. people will think you're processor. You're right. RACHEL: Okay wait!
peaceful
3
anger
anger
Table 6: Examples of utterances from the EmoryNLP dataset for the case study.
Methods
IEMOCAP
MELD
EmoryNLP
be because the coefficients used for the debiasing strategy are less detailed, they are applied to the entire sample set instead of individual utterances or conversations. The latter approach would require significant tuning efforts. Furthermore, there are distribution differences between the validation and test sets, which can result in prediction errors for certain samples. Another concern is the issue of emotion shift (Poria et al., 2019b), where consec- utive utterances express different emotions. This has been challenging for previous approaches as well. Table 7 shows that our TFD still struggles to effectively handle samples with emotional shifts compared to those without such shifts.
Ours
70.43
66.19
40.51
w/o Emotion Shift
74.76
75.40
52.99
w/ Emotion Shift
65.02
59.52
36.24
Table 7: Analysis of TFD on Emotional Shift
the second utterance, because other neutral words are infrequent, except for pronouns, the neutral word "think" dominates the joyful emotion, which is rectified when the neutral word bias is eliminated. In the third utterance, the majority of samples in- dicate a neutral emotion. Although the context provides some clues, the model simply predicts the majority class. However, after removing the label bias, the model correctly predicts the emotion of the utterance based on its context.
Conclusion
In this paper, we introduce TFD with counterfac- tual reasoning solution to address unintended dataset biases in ERC, which performs the debi- asing operation only during the prediction phase without additional data augmentation or training expenses. Through extensive experiments on threc public datasets, we demonstrate that TFD effec-
5.7 Error Analysis
Most errors can be attributed to class imbalance, such the low Fl score of 17.14 for the 'power- ful' emotion in the ELP dataset. Our TFD may also make mistakes, as shown in Fig. 4, where certain samples of majority classes. This could

---

tively promotes the generalization capability and fairness of various ERC methods.
ings the 2018 AAAI/ACM Conference on AI, Ethics, and Sociery, pages 67-73.
Guang Guang-Gang Geng. -hun--engg Wang. Qiu-Dan Li, Lei Xu, and Xiao-Bo Jin. 2007. Boosting the perfor- mance of web spam detection with ensemble under- sampling classification. In Fourth International Con- ference on F Fuzzy Systems and Knowledge Discovery (FSKD 2007), volume pages 583-587. IEEE.
Acknowledgements
We thank the anonymous reviewers for their valu- able suggestions improve the overall qual- ity of this manuscript. This work was par- tially supported by the National Natural Sci- ence Foundation of China (62006062, 62176076), Natural Science Foundation of GuangDong 2023A1515012922, Shenzhen Foundational Re- search Funding JCY12021032411500
Deepanway Ghosal, Navonil Majumder, Alexander Gel- bukh, Rada Mihalcea, and Soujanya Poria. 2020. Cosmic: Commonsense knowledge for emotion iden- tification in conversations. In Findings of the Associ- ation for Computational Linguistics: EMNLP 2020, pages 2470-2440
and JCYI202208 8 (02415032, Guangdong Provincial Key Laboratory of Novel Security Intelligence Technologies 2022B 121201005.
Deepanway Ghosal, Navonil Majumder, Soujanya Poria, Niyati Chhaya, and Alexander Gelbukh. 2019. Dia- loguegen: A graph convolutional neural network for emotion recognition in conversation. In In Proceed- ings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th Inter- national Joint Conference on Natural Language Pro- cessing, pages 154-164.
Limitations
When facing significant distribution differences be- tween the validation and test sets, it becomes chal- lenging to ensure the effectiveness of a method. In such cases, finding appropriate coefficients to address the biased prediction results becomes diffi- cult. As shown Fig. 5, while Label debiasing can always improve the model's performance, Neutral Word and Speaker debiasing are often unable to consistently guarantee effectiveness. Furthermore, label debiasing may not have an impact when deal- ing with samples with highly balanced classes, as label bias becomes negligible.
Madelyn Glymour, Judea Pearl, and Nicholas P Jewell. 2016. Causal inference in statistics: primer. John Wiley & Sons.
Yash Goyal, Tejas Khot, Douglas Summers-Stay Dhruv Batra, and Devi Parikh. 2017. Making the in vqa matter: Elevating the role of image understanding in visual question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6904-6913.
James Gross and Lisa Feldman Barrett, 2011. Emo- tion generation and emotion regulation: One or two depends on your point of view. Emotion Review, 3(1).
References
Antoine Bosselut, Hannah Rashkin, Maarten Sap, Chai- tanya Malaviya, Asli Celikyilmaz, and Yejin Choi. 2019. Comet: Commonsense transformers for auto- matic knowledge graph construction. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguisties, pages 4762-4779,
Devamanyu Hazarika, Soujanya Poria, Rada Mihalcea, Erik Cambria, and Roger Zimmermann. 2018. Icon: Interactive conversational memory network for multi- modal emotion detection. In Proceedings of the 2018 conference on empirical methods in natural language processing, pages 2594-2604.
Carlos Busso, Murtaza Bulut, Chi-Chun Lee, Abe Kazemzadeh, Emily Mower, Samuel Kim, Jean- nette N Chang, Sungbok Lee, and Shrikanth Narayanan. 2008 Temocap: Interactive emotional dyadic motion capture database. Language resources and evaluation, 42(4):335-355
Dazhi Jiang, Runguo Wei, Jintao Wen, Geng Tu, and Erik Cambria. 2022. Automl-emo: Automatic knowl- edge selection using congruent effect for emotion identification in conversations. IEEE Transactions Affective Computing.
Wenxiang Jiao, Michael Lyu, and Irwin King. 2020. Real-time emotion recognition via attention gated hierarchical memory network. In Proceedings of the AAAI Conference Artificial Intelligence, vol- ume 34, pages 8002-8009.
Brik Cambria, Qian Liu, Sergio Decherchi, Frank Xing, and Kenneth Kwok. 2022. Senticnet 7: A commonsense-basee neurosymbolic ai framework for explainable sentiment analysis. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages
Masahiro Kaneko and Danushka Bollegala. 2019. Gender-preserving debiasing for pre-trained word embeddings. In Proceedings of the 57th Annual Meeting of the Association for Computational Lin- guistics, pages 1641-1650.
Lucas Dixon, John Li, Jeffrey Sorensen, Nithum Thain, and Lucy Vasserman. 2018 Measuring and mitigat- ing unintended bias in text classification. In Proceed-

---

Bingyi Kang, Saining Xie, Marcus Rohrbach, Zhicheng Yan, Albert Gordo, Jiashi Feng, and Yannis Kalan- tidis. 2019. Decoupling representation and clas- sifier for long-tailed recognition. arXiv preprint 1rX:2:9910.9221..
Yuzhao Mao, Guang Liu, Xiaojie Wang. Weiguo Gao, and Xuan Li. 2021. Dialoguetrm: Exploring multi- modal emotional dynamics in conversation. In Findings of the Association for Computational Lin- guistics: EMNLP 2021, pages 2694-2704.
Qi Kang, XiaoShuang Chen, SiSi Li, and MengChu Zhou, 2016. A noise-filtered under-sampling scheme for imbalanced classification. JEEE transactions on cybernetics, 471244446334477
Yulei Niu, Kaihua Tang, Hanwang Zhang. Zhiwu Lu, Xian-Sheng Hua, and Ji- Ji-Rong Wen. 2021. Counter- factual vqa: A cause-effect look at language bias. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, pages 12700- 12710.
Jacob Devlin Ming-Wet Chang Kenton and Lee Kristina Toutanova. 2019. Bert: Pre-training of deep bidirec- tional transformers for language understanding. In Proceedings of NALC pages 4166-4186.
Alberto Olmo, Sarath Sreedharan, and Subbarao Kamb- hampati. 2021. 5pt3-to-plam, Extracting plans from text using gpt-3. arXiv preprint arXiv:2106.
Taewoon Kim and Piek Vossen. 2021.
Emoberta: Speaker-aware emotion recognition in conversation with roberta. arXiv preprint arXiv:2108.1 12000
Donovan Ong, Jian Su, Bin Chen, Anh Tuan Luu, Ashok Narendranath, Yue Li, Shuqi Sun, Yingzhan Lin, and Haifeng Wang. 2022. Is discourse role important for emotion recognition in conversation? In Proceedings of the AAAI Conference on Artificial Intelligence. volume 36, pages 11121-11129.
Joosung Lee and Wooin Lee. 2022. Compm: Context modeling with speaker's pre-trained memory track- ing for emotion recognition in conversation. In Pro- ceedings of the 2022 Conference of the North Amer- ican Chapter of the Association for Computational Linguistics: Human Language Technologies. pages 5669-5679.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroli Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instruc- tions with human feedback. Advances in Neural Information Processing Systems, 35:27730-27144.
Jiangnan Li. Zheng Lin, Peng Fu, and Weiping Wang.
2021. Past, present, and future: Conversational emo- tion recognition through structural modeling of psy- chological knowledge. In In Findings of the Associ- ation for Computational Linguistics: EMNLP 2021, pages 1204-1214.
Soujanya Poria, Erik Cambria, Devamanyu Hazarika. Navonil Majumder. Amir Zadeh, and Louis- Phiis-p Morency. 2017. Context-dependen sentiment anal- ysis user-generated videos. In Proceedings of the 55th annual meeting of the association for computa- tional linguistics, pages 873-883.
Wei Li, Wei Shao, Shaoxiong Ji, and Erik Cambria. 2022. Bieru: Bidirectional emotional recurrent unit for conversational sentiment analysis. Neurocomput- ing, 467:73-82. 82.
Soujanya Poria, Devamanyu Hazarika, Navonil Ma- jumder, Gautam Naik, Erik Cambria, and Rada Mi- halcea. 2019a. Meld: A multimodal multi-party dataset for emotion recognition in conversations. In Proceedings of the 57th Annual Meeting of the Associ- ation for Computational Linguistics, pages 527-536.
Zheng Lian, Bin Liu, and Jianhua Tao. 2021. Ct- net: Conversational transformer network for emo- tion recognition. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 29:985-1000,
Soujanya Poria, Navonil Majumder, Rada Mihalcea, and Eduard Hovy. 2019b. Emotion recognition in versation: Research challenges, datasets, and recent advances. JEEE Access, 7:100943-10099
Frederick Liu and Besim Avei. 2019. Incorporating priors with feature attribution on text classification. In Proceedings of the 57th Annual Meeting of the As- sociation for Computational Linguistics, pages 6274- 6283.
Chen Qian, Fuli Feng, Lijie Wen, Li Lin, and Tat-Seng Chua. 2020. Enhancing text classification via dis- covering additional semantic clues from logograms. In Proceedings of the 43rd International ACM SI- GIR Conference on Research and Development in Information Retrieval, pages 1201-1210
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining ap- proach. arXiv preprint arXiv: 1907.1169..
Chen Qian, Fuli Feng, Lijie Wen, Chunping Ma, and Pengjun Xie. 2021. Counterfactual inference for text classification debiasing. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Confer- ence on Natural Language Processing (Volume 1: Long Papers), pages 5434-5445.
Navonil Majumder, Soujanya Poria, Devamanyu Haz- arika, Rada Mihalcea, Alexander Gelbukh, and Erik Cambria. 2019. Dialoguernn: An attentive mnn for emotion detection in conversations. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 1885-.....

---

Prakhar Saxena, Yin Jou Huang, and Sadao Kurohashi. 2022. Static and dynamic speaker modeling based on graph neural network for emotion recognition in conversation. Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies: Student Research Workshop, pages 247-253.
Geng Tu, Bin Liang, Ruibin Mao, Min Yang, and Ruifeng Xu. 2023b. Context or knowledge is not always necessary: A contrastive learning framework for emotion recognition in conversations. In Find- ings of the Association for Computational Linguistics: ACI. 2023, pages 14054-14067.
Geng Tu, Jintao Wen, Hao Liu, Sentao Chen. Lin Zheng, and Dazhi Jiang. 2022b. Exploration meets exploitation: Multitask learning for emotion recog- nition based on discrete and dimensional models. Knowledge-Based Systems, 235:
Weizhou Shen, Junqing Chen, Xiaojun Quan, and Zhix- ian Xie. 2021a. Dialogxl: All-in-one xlnet for multi- party conversation emotion recognition. In Proceed- ings the AAAI Conference on Artificial Intellgence, volume 35, pages 13789-13797.
Zhao Wang and Aron Culotta. 2021. Robustness spurious correlations in text classification via auto- matically generated counterfactuals. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 14024-14031.
Weizhou Shen, Siyue Wu, Yunyi Yang, and Xiaojun Quan. 202 Directed acyclic graph network for conversational emotion recognition. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joins Conference on Natural Language Processing, pages 1551-1560.
Zeerak Waseem and Dirk Hovy. 2016. Hateful symbols or hateful people? predictive features for hate speech detection twitter. In Proceedings of the NAACL student research workshop, pages 88-93.
Robyn Speer, Joshua Chin, and Catherine Havasi. 2017. Conceptnet 5.5: An open multilingual graph of gen- eral knowledge. In Proceedings the 31st AAAI con- ference on artificial intelligence, pages 4444 4451.
Jintao Wen, Dazhi Jiang. Geng Tu, Cheng Liu, and Erik Cambria. 2023. Dynamic interactive multiview mem- ory network for emotion recognition in conversation. Information Fusion, 91:123-133.
Liuyu Xiang, Guiguang Ding. and Jungong Han. 2020. Learning from multiple experts: Self-paced knowl- edge distillation for long-tailed classification. In Computer MisionEECVV 2020: 16th European Con- ference, Glasgow, UK, August 23-28, 2020, Proceed- ings, Part V 16, pages 247-263, Springer.
Bo Sun, Haiyan Chen, Jiandong Wang, and Hua Xie. 2018. Evolutionary under-sampling based hagging ensemble method for imbalanced data classification. Frontiers of Computer Science, 12:331-350.
Chris Sweeney and Maryam Najafian. 2019. A trans- parent framework for evaluating unintended demo- graphic bias in word emheddings. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1667.
Yunhe Xie, Kailai Yang. Cheng-Jie Sun, Bingquan Liu, and Zhenzhou Ji. 2021. chowledge-interactvv net- work with sentiment polarity intensity-awan aware multi- task learning for emotion recognition in conversa- tions. In Findings of the Association for Computa- tional Linguistics: EMNLP 2021, pages 2879-2888 2889-2889.
Kaihua Tang. Yulei Niu, Jianqiang Huang, Jiaxin Shi, and Hanwang Zhang. 2020. Unbiased scene graph generation from biased training. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 37 373655.
Xu Yang, Hanwang Zhang, and Jianfei Cai. 2021. De- confounded image captioning: A causal retrospect. IEEE Transactions Pattern Analysis & Machine Intelligence, (01):1-1.
Bing Tian, Yixin Cao, Yong Zhang. and Chunxiao Xing. 2022. Debiasing nlu models via causal interven- tion and counterfactual reasoning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 11376-11384 11384.
Sayyed M Zahiri and Jinho D Choi. 2018. Emotion detection on tv show transcripts with sequence- based convolutional neural networks. In Workshops the thirty-second AAAI conference on artificial intelli- gence, pages 44-52.
Geng Tu, Bin Liang, Dazhi Jiang, and Ruifeng Xu. 2022a. centiment-emotion and context-guided knowledge selection framework for emotion recogni- tion in conversations. IEEE Transactions on Affective Computing, (01):1-14.
Guanhua Zhang, Bing Bai, Junqi Zhang, Kun Bai, Con- ghui Zhu, and Tiejun Zhao. 2020. Demographics should not be the reason of toxicity: Mitigating discrimination in text classifications with instance weighting. In Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguistics. pages 41 4145.
Geng Tu, Bin Liang, Xiucheng Lyu. Lin Gui, and Ruifeng Xu. 2023a. Do topic and causal consistency affect emotion cognition? a graph interactive net- work for conversational emotion detection. In In The 26th European Conference on Artificial Intelligence (ECAI'23), pages 2362-2369. IOS Press.
Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Or- donez, and Kai-Wei Chang. 2017. Men also like shopping: Reducing gender bias amplification us- ing corpus-level constraints. In Proceedings of the

---

2017 Conference on Empirical Methods Natural Language Processing, pages 2979-29999999 29899.
Fig. perform prompt-tuning, which enhances the model's performance in generating emotionally appropriate responses.
Weixiang Zhao, Yanyan Zhao, and Xin Lu. 2022. Cauain: Causal aware interaction network for emo- tion recognition in conversations. In Proceedings of the st International Joint Conference Artifiicial Intelligence, pages 4524-4530.
Peixiang Zhong, Di Wang, and Chunyan Miao. 2019. Knowledge- entrched transformer for emotion detec- tion in textual conversations. 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 165-176.
Proceedings of
Do Cedesterra to Yey, daile
1v lone teneer beregghuu oicke Well
Me sasi bavr eagy rigatt
Touama
co doca dec vastille bud dove ii, yoa and volo hald
wan you
leng oy anturm ep
aecconed you liow of tu 1i pin
momane hepp Ingh
Figure 6: Prompt template for ChatGPT.
you Scstant
capoaao do cutegiie modk
you thup
De mall coloe trom nad tustts ep comseen on happ
('scca" soader deaal curiiii
pray ca calp.
femgle happ??
(yneathin Spakent This aecilii Comprade compleo bope
Figure 7: Prompt template for the Curie model.
A Prompt Templates
this section, we primarily focus on explaining the methodology used to obtain the results for the ChatGPT and prompt-tuned Curie model in this paper. For ChatGPT, we employ a prompt tem- plate, as depicted in Fig. 6, to extract the emotion of each utterance in conversation. This approach ensures the utilization of contextual information and facilitates the generation of well-formatted out- put results. On the other hand, for the Curie model, we directly access OpenAl's API for implementa- tion. We utilize the prompt example presented in


Reviews:
[{"review_id": "8tkjEYZFVK", "reviewer": "Reviewer_X5kt", "summary": "", "questions": "A. How would/does the framework extend to models which do not explicitly encode the context of the utterance (x^hat)?\n\nB. Can you argue why the imbalance divergence is a good metric for fairness? Why should the class labels be uniformly distributed to achieve fairness? (I am aware other works have used it, but I might be missing something, so please clarify, because I understand it as a metric for measuring label distribution skewness, not fairness)", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to express our great appreciation for your valuable comments to improve the quality of this manuscript.  \n\n**Answer of Reject#1:**  \n&ensp;&ensp;Thanks for your insightful comment. According to (Sweeney and Najafian, 2019), the greater the prediction skewness of a trained model, the more it provides unfair opportunities across predefined categories. Consequently, the trained model becomes increasingly characterized by unfair discrimination. We thus follow previous work to use the metric – imbalance divergence – to evaluate whether a prediction is unfair. Based on this, to evaluate the speaker and neutral word biases of a trained model, we average the relative word (i.e. speaker and neutral word) imbalance over all the testing utterances. Similarly, to evaluate the label bias of a trained model, we average its relative label imbalance over all the testing utterances.  \n&ensp;&ensp;Regarding whether the performance of the models should be the only metric used, we acknowledge model performance is a crucial metric. Nonetheless, we recognize that the significance of model fairness cannot be disregarded in real-world applications. Perhaps, we can emphasize model performance as the primary metric and then discuss the model imbalance divergence or unfairness as a secondary metric. We sincerely appreciate your suggestions and will carefully consider them in the revision.\n\n*Sweeney, C., & Najafian, M. (2019, July). A transparent framework for evaluating unintended demographic bias in word embeddings. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (pp. 1662-1667).*\n\n**Answer of Reject#2:**  \n&ensp;&ensp;Thanks for your valuable suggestion. In the given utterance, *\"CHANDLER says Act like a processor, people will think you're a processor. You're right.\"* with the emotion \"peaceful\". We want to prevent the model from being influenced by neutral words (such as \"think\", which often appears in utterances with the emotion \"joyful\"). Similarly, we also aim to avoid the influence of the speakers (such as CHANDLER, who frequently expresses neutral and joyful emotions). Therefore, we extract neutral word biases and speaker biases, in equation (9). Specifically, in causal intervention operation, we mask tokens that do not belong to the set of speakers (CHANDLER is the only speaker in this example) to predict outcomes with speaker bias. We can also obtain prediction outcomes with neutral word biases by masking tokens that do not belong to the set of neutral words, such as \"right\". We will detail it in the revision.\n\n**Answer of Question A:**  \n&ensp;&ensp;Thanks for your insightful question. Bias extraction in our TFD operates at the feature level, ensuring its independence from the model structure. This is supported by outcomes in Tables 4 and 5, showing TFD applied to various ERC models that explicitly (e.g., KET, etc.) or implicitly (e.g., DAG-ERC, etc.) encode the context of the utterance.\n\n**Answer of Question B:**  \n&ensp;&ensp;Thanks for your valuable question. According to (Sweeney and Najafian, 2019), the greater the prediction skewness of a trained model, the more it provides unfair opportunities across predefined categories. Consequently, the trained model becomes increasingly characterized by unfair discrimination. We thus follow previous work to use the metric – imbalance divergence – to evaluate whether a prediction is unfair. Based on this, to evaluate the speaker and neutral word biases of a trained model, we average the relative word (i.e. speaker and neutral word) imbalance over all the testing utterances. Similarly, to evaluate the label bias of a trained model, we average its relative label imbalance over all the testing utterances. We will detail it in the revision.\n\n*Sweeney, C., & Najafian, M. (2019, July). A transparent framework for evaluating unintended demographic bias in word embeddings. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (pp. 1662-1667).*\n\n**Answer of Typos Grammar Style And Presentation Improvements:**  \n&ensp;&ensp;Thanks a lot for your valuable suggestions, we have corrected them in the revision and will avoid these mistakes in the future. Furthermore, Table 7 reports results indicating that the model fails to perform better on samples with emotional shifts compared to those without. We will detail it in the revision.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Interesting approach to reducing imbalance effects and the effects of co-occurence statistics on the classification performance of ERC models.\n\nGood evaluation work performed, comparisons with SoA methods.", "reasons_to_reject": "I am not convinced about using imbalance divergence to measure fairness. Also, I would not use the word fairness in the context of the biases discussed in this paper, because we are affected by their effect on the performance of the classification models only, perhaps that should be the only metric used (?). \n\nI found section 3.4 a little difficult to read. It was unclear which were notations introduced by the authors (e.g. the c's, are you masking all words in (9)?) and the causal intervention application. The examples/relation to the speaker/word biases helped.", "questions_to_authors": "A. How would/does the framework extend to models which do not explicitly encode the context of the utterance (x^hat)?\n\nB. Can you argue why the imbalance divergence is a good metric for fairness? Why should the class labels be uniformly distributed to achieve fairness? (I am aware other works have used it, but I might be missing something, so please clarify, because I understand it as a metric for measuring label distribution skewness, not fairness)", "paper_topic_and_main_contributions": "The paper introduces a framework for reducing various types of spurious correlations in models for Emotion Recognition in Conversational data (ERC). The framework is model-agnostic and uses counterfactual reasoning in a post-hoc manner (i.e. at prediction time), by applying masking to extract the model biases and obtain the counterfactual examples. The model predictions on these examples are then used to un-bias the original input predictions (by subtraction). The framework is \"instantiated\" and evaluated for reducing the effect of class imbalance on the classification performance (i.e. \"label bias\"), reducing the effect of the speaker bias and neutral words bias on the predictions. \nThe performance of the framework is compared to that of various specific ERC models (e.g. DialogueRNN, COSMIC, etc) as well as general-purpose PLMs (e.g. ChatGPT, Curie) - both from the classification performance point of view and that of improving fairness (the imbalance divergence metric is used for this).", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "\"Considering that except for generative approaches, most methods rely on using Roberta for utterance representation extraction.\" - needs rephrasing\n\n\"Our TFD may also make mistakes, as shown in Fig. 4, where certain samples of majority classes.\" - needs rephrasing\n\n\"the model simplistically predicts\" - \"... simply ...\"\n\nSome table captions are below the tables, others are above.\n\nTable 7 is confusing. Does the first  row show the average perf, while the second analyzez only dialogues w/o emotion shift, and the third row dialogues only with emotion shift, but using the framework also? (i.e. it is not a different method, but different test samples)."}}, {"review_id": "L37k7e05qe", "reviewer": "Reviewer_LsJC", "summary": "", "questions": "In Table 2, I found the results are not consistent with the three items (i.e., w/ label debiasing, w/ neutral word debiasing, and w/ speaker debiasing)", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks very much for taking the time to review this manuscript. We really appreciate all your generous comments and recognition of this paper! We will release the source code after the acceptance of this paper to facilitate reproducibility by readers.\n\n**Answer of Question#1:**  \n&ensp;&ensp;Thanks for your insightful comment. In Table 2, the term 'w/o' is employed for ablation analysis, aiming to investigate the impact of each component of our framework on model performance. For instance, w/o Neutral word debiasing aims to evaluate the impact of the removal of neutral word debiasing in TFD on model performance. **Please refer to the analysis in Section 5.2.2.**", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper proposed a new method TFD and achieve good results on three public datasets when compared to several SOTAs.", "reasons_to_reject": "No specific reason to reject from my side.", "questions_to_authors": "In Table 2, I found the results are not consistent with the three items (i.e., w/ label debiasing, w/ neutral word debiasing, and w/ speaker debiasing)", "paper_topic_and_main_contributions": "The paper proposed a debiasing framework with counterfactual reasoning for conversational emotion detection. The main contributions of this paper are:\n- They propose a TFD to tackle biases in REC without extra data augmentation and any training costs.\n- They conducted extensive experiments on three public datasets to illustrate the effectiveness of TFD.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "N/A", "Typos_Grammar_Style_and_Presentation_Improvements": "N/A"}, "extra_text": {}}, {"review_id": "g4vevWyxIK", "reviewer": "Reviewer_p6EJ", "summary": "", "questions": "MELD and EmoryNLP come from the same source and have similar statistics. What is the rationale for adding them both to the analysis?\n\nWhat do you mean by \"the replication model\" in Table 2's caption? Also, in the same table, what is the baseline model? Is it RoBERTa base? ", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to express our great appreciation for your valuable comments to improve the quality of this manuscript. We will release the source code after the acceptance of this paper to facilitate reproducibility by readers.\n\n**Answer of Reject#1:**  \n&ensp;&ensp;Thanks for your insightful question. Yes, the context representation is built by concatenating each token in the historical utterances without using special tokens to separate these utterances, following the suggestion from the previous work (Kim and Vossen, 2021).\n\n*Kim, T., & Vossen, P. (2021). Emoberta: Speaker-aware emotion recognition in conversation with roberta. arXiv preprint arXiv:2108.12009.*\n\n**Answer of Reject#2:**  \n&ensp;&ensp;Thanks for your valuable question. Yes, the term \"speaker\" appears as a variable in the phrase \"speaker says.\", evidenced by the indications in Equations 3 and 4. However, it is important to note that \"speaker\" may also appear as a variable in the utterance. We will detail it in the revision.\n\n**Answer of Reject#3:**  \n&ensp;&ensp;Thanks for your valuable comment. In fact, there is no \"<mask>\" token. The correct Equation (3) is as follows:   \n&ensp;&ensp;$X_{c}(u_{i})$ = <s\\> ${s}_i$ says: ${u}_i$ 
   \n&ensp;&ensp;Inspired by prompts, we attempted to introduce the <mask> token to signify the emotional context of utterances. For instance, in analyzing the emotion in the utterance \"Alice: I did well in the exam\", we might employ the prompt \"Alice felt <mask>\". This <mask> token allows the Pre-trained Language Model (PLM) to function within a familiar framework, thereby enhancing its performance in downstream tasks. If the <mask> token is utilized, the input format of contexts in Equation (4) should be “$s_j$ <emo> says: $u_j$”, where <emo> represents the emotion label. However, during experimentation, this method did not yield the expected results. This could be attributed to the inference stage's challenge in obtaining the <emo> token within the input context format. If the model's predicted output is used as the <emo> token, incorrect predictions might mislead the emotion detection of the current utterance. Consequently, we decided to remove it. We sincerely apologize for this oversight. We have corrected it in the revision and will avoid these mistakes in the future.\n\n**Answer of Reject#4:**  \n&ensp;&ensp;Thanks for your insightful question. The 
 and 
 are utilized to differentiate the current utterance from its context, i.e., the historical utterances.\n\n**Answer of Reject#5:**  \n&ensp;&ensp;Thanks for your insightful question. The variable \"h\" symbolizes the combined representation of an utterance and its context, indirectly affecting the prediction outcomes in the causal graph.\n\n**Answer of Reject#6:**  \n&ensp;&ensp;Thanks for your valuable comment. In the given utterance, *\"CHANDLER says Act like a processor, people will think you’re a processor. You’re right.\"* with the emotion \"peaceful\". Our objective is to mitigate the impact of neutral words, like \"think\" which commonly appears in utterances with the emotion \"joyful\". Similarly, we aim to counter the influence of speakers, such as CHANDLER, who frequently convey \"neutral\" and \"joyful\" emotions. Therefore, we extract neutral word and speaker biases. Specifically, we mask tokens that do not belong to the set of speakers (CHANDLER is the only speaker in this example) to predict outcomes with speaker bias. We can also obtain prediction outcomes with neutral word biases by masking tokens that do not belong to the set of neutral words, such as \"right\". We will detail it in the revision.\n\n**Answer of Question#1:**  \n&ensp;&ensp;Thanks for your insightful question. There is no particular rationale behind the selection of these two datasets, as they are widely utilized in ERC. This facilitates comparison with previous methods.\n\n**Answer of Question#2:**  \n&ensp;&ensp;Thanks for your valuable question. The replication model refers to our re-implementation results. Specifically, the results for ChatGPT and Curie in Table 2 are obtained by using our prompt templates provided in the appendix. The Roberta-based method (Baseline) is based on EmoBERTa, with the difference being that we did not consider the impact of future utterances on the model. **Please refer to lines 351-355 on page 5.** This exclusion is due to the impracticality of utilizing future utterances to predict the current one’s emotions in real-world scenarios.\n\n**Answer of Typos Grammar Style And Presentation Improvements:**  \n&ensp;&ensp;We greatly appreciate your input, and in response to your suggestions, we have made the necessary revisions to rectify the error. Your insightful comments are incredibly helpful in improving the quality of our work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper's core idea is interesting in several aspects. It is efficient, as it does require fine-tuning. Moreover, bias detection and logit correction are general methodologies that can find use beyond ERC. \n\nThe authors compare the performance of TDF against many similar approaches and prove it better.\n\nAn ablation study proves that every correction helps improve the model.", "reasons_to_reject": "The readability of Section 3 (and specifically 3.3 and 3.4) is the weakest point of the paper. The authors should expand it with some examples. The following is a list of less clear passages:\n- Equation 2: are the ... standing for \"concatenation\"? Is the representation of the context built by concatenating the previous utterances? If so, do you use special tokens to separate utterances from the context?\n- Is \"speaker\" a *variable* in the phrase \"speaker say\"? Because Eq. 3 and 4. suggest so.\n- The authors state that \"the \\<mask\\> token is utilized to highlight the emotional state of the sentence\": what do you mean by that, and specifically \"to highlight the emotional state\"? This passage requires more clarity, perhaps achieved by providing an example. \n- Why are you not using \\<s\\> and \\</s\\> in context utterances?\n- L251 reports **h** is the variable representing \"the combined features\": what do you mean by that?\n- Sections 3.4.2 and 3.4.3 are crucial to understanding the underlying methodology; however, the writing and notation take time to follow. I strongly recommend describing in detail and with an example of how neutral and speaker-related word sets are constructed.", "questions_to_authors": "MELD and EmoryNLP come from the same source and have similar statistics. What is the rationale for adding them both to the analysis?\n\nWhat do you mean by \"the replication model\" in Table 2's caption? Also, in the same table, what is the baseline model? Is it RoBERTa base? ", "paper_topic_and_main_contributions": "The paper introduces a new inference-time debiasing approach tailored to mitigate data collection and annotation biases in Emotion Recognition in Conversations (ERC). The proposed Training-free Debiasing Framework (TDF) defines causal graphs to model interactions between the input utterance, context, and emotion to discover biases and produce \"biased\" logits to subtract from the original logits. Experiments on three ERC datasets show that debiasing a pretrained RoBERTa with TDF leads to better emotion recognition and fairer prediction than a large set of baselines. \n\nThe paper has several strengths, including convincing results and ablation studies. Readability and structure, especially in section 3.4, can be improved by providing examples and extended descriptions.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "Several citations need to be in the correct style. When used as subjects, for example, you should use \\citet, e.g., \"inspired by (Kim and Vossen, 2021)\" should be \"inspired by Kim and Vossen, 2021\".\n\nYou have a mentioned dataset (L316), Dailydialog, that you later decided to remove from the paper."}}]

Author rebuttal:

Meta-review: This paper introduces an efficient debiasing method at the prediction time, evaluating it on the Emotion Recognition in Conversations (ERC) task. The reviewers are all excited about the originality and usefulness of this method, and the reported results in this paper. Some parts of the paper need further rewriting and improvement and the reviewers must work on that.
Decision: Accept-Main