hyy33 at WASSA 2024 Empathy and Personality Shared Task: Using the CombinedLoss and FGM for Enhancing BERT-based Models in Emotion and Empathy Prediction from Conversation Turns (2024)

Abstract: This paper presents our participation to the WASSA 2024 Shared Task on Empathy Detection and Emotion Classification and Personality Detection in Interactions. We focus on Track 2: Empathy and Emotion Prediction in Conversations Turns (CONV-turn), which consists of predicting the perceived empathy, emotion polarity and emotion intensity at turn level in a conversation. In the method, we conduct BERT and DeBERTa based finetuning, implement the CombinedLoss which consists of a structured contrastive loss and Pearson loss, adopt adversarial training using Fast Gradient Method (FGM). This method achieved Pearson correlation of 0.581 for Emotion, 0.644 for Emotional Polarity and 0.544 for Empathy on the test set, with the average value of 0.590 which ranked 4th among all teams. After submission to WASSA 2024 competition, we further introduced the segmented mix-up for data augmentation, boosting for ensemble and regression experiments, which yield even better results: 0.6521 for Emotion, 0.7376 for Emotional Polarity, 0.6326 for Empathy in Pearson correlation on the development set. The implementation and fine-tuned models are publicly-available at https://github. com/hyy-33/hyy33-WASSA-2024-Track-2.

text: Proceedings of the 14th Workshop on Computational Approaches to Subjectivity, Sentiment, & Social Media Analysis, pages 430–434 August 15, 2024 ©2024 Association for Computational Linguistics
This paper presents our participation to the WASSA 2024 Shared Task on Empathy Detection and Emotion Classification and Personality Detection in Interactions. We focus on Track 2: Empathy and Emotion Prediction in Conversations Turns (CONV-turn), which consists of predicting the perceived empathy, emotion polarity and emotion intensity at turn level in a conversation. In the method, we conduct BERT and DeBERTa based finetuning, implement the CombinedLoss which consists of a structured contrastive loss and Pearson loss, adopt adversarial training using Fast Gradient Method (FGM). This method achieved Pearson correlation of 0.581 for Emotion, 0.644 for Emotional Polarity and 0.544 for Empathy on the test set, with the average value of 0.590 which ranked 4th among all teams. After submission to WASSA 2024 competition, we further introduced the segmented mix-up for data augmentation, boosting for ensemble and regression experiments, which yield even better results: 0.6521 for Emotion, 0.7376 for Emotional Polarity, 0.6326 for Empathy in Pearson correlation on the development set. The implementation and fine-tuned models are publicly-available at https://github. com/hyy-33/hyy33-WASSA-2024-Track-2.

heading: 1 Introduction
text: Emotion detection and empathy analysis are important and inevitable topics in the processing of human interactions, which show great potential in various application scenarios (Nandwani and Verma, 2021; Sharma et al., 2020). To provide more insights into this topic, WASSA organizes workshop on related topics each year (Barriere et al., 2023). This year, WASSA 2024 focuses on Shared Task on Empathy Detection and Emotion Classification and Personality Detection in Interactions (Giorgi et al., 2024).
*The corresponding author.
In this paper, we propose a solution towards Track 2: Empathy and Emotion Prediction in Conversations Turns (CONV-turn). In this task, participants are given conversations between two users that read the same essay, which contains reaction to news articles where there is harm to a person or group (Omitaomu et al., 2022). Each of their conversation turn (text content) has been annotated in perceived empathy, emotion polarity, and emotion intensity. Other meta information such as article_id, conversation_id, turn_id and speaker_id are also provided. A sample from the dataset is demonstrated in Figure 1.
This task aims at developing appropriate methods to predict the perceived empathy, emotion polarity, and emotion intensity at the speech-turnlevel during human conversation. In previous works, BERT (Devlin et al., 2019) is frequently used for emotion classification (Luo and Wang, 2019; Kannan and Kothamasu, 2022), its variations such as RoBERTa (Liu et al., 2019) and DeBERTa (He et al., 2020) also play important roles in empathy prediction and sentiment analysis (Vasava et al., 2022; Lu et al., 2023). Based on fine-tuned encoders, different strategies are further introduced
430
to build more robust and reliable models, including adversarial training (Chen and Ji, 2022; Chang et al., 2023), data augmentation (Kwon and Lee, 2023) and ensemble strategy (Plaza-del Arco et al., 2022).
Our goal is to predict the interior emotion and empathy state of the user according to turn-level information from human-to-human conversations. To achieve this goal, we adopt BERT-based models including BERT (Devlin et al., 2019) and its variation of DeBERTa (He et al., 2020). Then, they are fine-tuned using task-oriented data from Track 2 with adversarial training using Fast Gradient Method (FGM). Also, we design a novel CombinedLoss, which consist of a structured contrastive loss and a Pearson loss. Then, after the submission to WASSA 2024 competition, data augmentation using the segmented mix-up strategy, ensemble with boosting method and regression experiments are further conducted.

heading: 2 Methodology
text: This section introduces the methodolgy of our proposed system for Track 2 in WASSA 2024. As in Figure 2, the proposed model includes: the fine-
tuned BERT or DeBERTa, the CombinedLoss and the downstream head for classification (or regression). Also, augmentation and ensemble are implemented in corresponding parts.

heading: 2.1 Fine-tuned BERT and DeBERTa
text: In this paper, we conduct task-oriented fine-tuning for Track2 based on BERT (Devlin et al., 2019) and its variation of DeBERTa (He et al., 2020). For the base models, bert-base-uncased and deberta-base are introduced as the pretrained language models. Then, fine-tuning is conducted on the training set of Track 2, so that the encoders could adapt from general language modelling to our specific prediction task by adjusting the parameters.

heading: 2.2 The CombinedLoss
text: Different from commonly-used loss functions, our work proposes the CombinedLoss, which could be expressed as:
Ltotal = Lloss + λ(1− CorrPear(ŷ,y)), (1)
where Lloss is the structured contrastive loss for classification, λ is the regularization coefficient, and CorrPear(ŷ,y) is the Pearson correlation coefficient (Cohen et al., 2009) between the prediction of ŷ and the ground truth label of y.
Since emotions are classified into multiple levels in the dataset (Omitaomu et al., 2022), the Pearson correlation coefficient is used as a regularization term in the loss function. By using the negative Pearson coefficient, this loss function aims to capture the subtle scale between emotion levels.

heading: 2.3 Adversarial Training with FGM
text: To improve the robustness and the generalization ability of the proposed model, adversarial training is introduced as follows:
Obj = min θ E(x, y) [maxL(fθ(x+ δ), y)] , (2)
in which x is the input sample, δ is the added perturbation for adversarial training, fθ is neural network function with θ as parameters. By maximizing L(fθ(x+ δ)), the most disturbing perturbation are introduced to the model, then the model is optimized to minimize the training error, which helps it to be robust to potential perturbations.
In this work, Fast Gradient Method (FGM) (Andriushchenko and Flammarion, 2020) is implemented as adversarial training strategy, which computes the most disturbing perturbation through scaling the gradient as below.
δ = · g||g||2 (3)
g = ∇xL(x, y, θ) (4)

heading: 2.4 Augmentation: the Segmented Mix-up
text: To improve the generalization ability of models, mix-up is often used as a method for data augmentation. In this work, a segmented mix-up is proposed, which mixes inputs and labels within specific label ranges. This segmentation is essential because simple mix-up (Gong et al., 2022) between highly negative and highly positive samples could not generate meaningful intermediate samples.
For each dimension, e.g. Emotion, samples are divided into two segments: the lower segment with labels smaller or equal to the middle label, and the upper one with labels larger than the middle label. Each sample (xi, yi) is paired with a partner sample (xj , yj) from the same label segment, with xi and xj denote the tokenized sentences, and yi and yj represent their labels. The mix-up coefficient µ is sampled from a Beta distribution: µ ∼ Beta(α, α), where α controls the mix-up strength. The generated inputs and labels are computed as:
x̃i = µxi + (1− µ)xj , (5)
ỹi = µyi + (1− µ)yj , (6)

heading: 2.5 Ensemble with Boosting
text: To build a more accurate and robust system, boosting is implemented as an ensemble strategy (Bühlmann, 2012), which combines fine-tuned BERT and DeBERTa models. In order to enhance the overall performance, weights are assigned according to the accuracy of each model on the development set. Through this, it is ensured that the model with the most reliable prediction has the greatest impact on the final output.

heading: 3 Experiments and Results
text: In this section, extensive experiments were conducted on the fine-tuned BERT and DeBERTa. Also, ablation study is performed to test the performance of different parts in the proposed model.

heading: 3.1 Datasets
text: The dataset of Track 2 includes a training set of 11,166 samples, a development set of 990 samples and a test set of 2,061 valid samples (Omitaomu et al., 2022). Each sample consists of the text content of a single dialogue turn and the corresponding label of Emotion, Emotional Polarity and Empathy, as well as some meta information of the speakers and the conversation. A data sample is shown in Figure 1.

heading: 3.2 Evaluation Metrics
text: To test the performance of the proposed solution, the official evaluation metric for Track 2 is the Pearson correlation (Cohen et al., 2009). Given sequences of prediction ŷ and ground truth y, their Pearson correlation coefficient can be calculated as:
CorrP (ŷ, y) =
∑n i=1( (ŷi−¯̂y) σŷ (yi−ȳ) σy )
n , (7)
in which E(ŷ) and E(y) stand for the expectations of ŷ and y, σŷ and σy stand for the standard deviations of ŷ and y.

heading: 3.3 Implementation Details
text: Baselines. To compare the performance of proposed models, BERT (Devlin et al., 2019) and its variation DeBERTa (He et al., 2021) are introduced. For BERT, bert-base-uncased is used, with 12 encoder layers and 110M parameters. For DeBERTa, deberta-base is adopted with 390M parameters. Hyper-parameters. For tokenization, input sentences are tokenized with BertTokenizer and DebertaTokenizer with the maximum length of 128. For optimization, AdamW optimizer is adopted with learning rate of 1 × 10−6 and exponential decay with γ = 0.99 after grid search, the batch size is 400 for fine-tuning BERT and 200 for fine-tuning DeBERTa. For the segmented mix-up, α = 0.2 is used. Other details could be found in our implementation. Labels and Categories. The experiments are conducted on different downstream tasks of classification and regression. Because the original labels not only contain integer values but also include float values, such as 0.3333, 0.6667 in the training set and 0.5, 1.5 in the development set, we manually divided 9 categories for Emotion, 5 categories for Emotional Polarity and 11 categories for Empathy in classification (details could be found in our code of implementation). In regression experiments, we directly use original labels as target values.

heading: 3.4 Results and Analysis
text: This section presents the results of Pearson correlation based on the experiments of the proposed models on the development set, and conducts analysis for the results. Fine-tuned BERT and DeBERTa. It can be observed from Table 1 that the average results of finetuned DeBERTa is better than fine-tuned BERT, which shows the stronger ability of DeBERTabased solution. And by implementing the CombinedLoss, both models demonstrate performance gain in Emotion, Emotional Polarity and Empathy prediction. Also, adding adversarial training using Fast Gradient Method (FGM) brings better overall performance, proving its contribution to the robustness and generalization ability of models. Our submission to WASSA 2024 competition is based on this fine-tuned DeBERTa with CombinedLoss and Fast Gradient Method (FGM). Ensemble and Augmentation. The results of Table 2 show the combined boosting yields the best overall result, which confirms the effectiveness of our boosting strategy by assigning weights to models according to their accuracy. An interesting finding is that ensembling fine-tuned DeBERTas not always achieves the highest score in single dimension, this may due to the reason that single DeBERTa already achieves its upper limit, combining them only decreases the possible lower bound, while on the other hand, single BERT may has unstable scoring performance, thus the ensemble of BERTs leads to high reliablity and better results. Also, augmentation brings further improvement, indicating our segmented mix-up strategy successfully generates meaningful intermediate samples, which contribute to the fine-tuning process. Classification and Regression. Table 3 presents the results of the fine-tuned DeBERTa (with Com-
binedLoss and FGM) in different downsteam tasks. The labelling details for classification and regression could be found in Section 3.3. From the results, it is shown that the fined-tuned DeBERTa achieved slightly better performance in regression task, which provides future research direction for us.

heading: 4 Conclusions
text: This paper presents our solution for Track 2 in WASSA 2024, which focused on the prediction of Emotion, Emotional Polarity and Empathy using turn-level information from user conversations. The submitted solution is built using fine-tuned DeBERTa with our proposed CombinedLoss and adversarial training startegy using Fast Gradient Method (FGM), which achieved Pearson correlation of 0.581 for Emotion, 0.644 for Emotional Polarity and 0.544 for Empathy on the test set, with the average value of 0.590 which ranked 4th among all teams. After the submission to WASSA 2024 competition, ensemble strategy using boosting method and data augmentation with the segmented mix-up are implemented, which further improve the performance of our model and yield better results: 0.6521 for Emotion, 0.7376 for Emotional Polarity, 0.6326 for Empathy in Pearson correlation on the development set. In the future, we plan to introduce larger datasets for model retraining at earlier stage (e.g. the Masked Language Model) for better domain adaptation, and consider introducing conversational context and speaker personality for better model construction. Also, the performance of such models in downstream regression tasks will be further investigated.
Limitations
The limitations of the proposed work included: 1) The training set was relatively small with less than 12000 samples. Fine-tuning the models on larger datasets might improve the performance. 2) The labels in the training set and the development set was mis-matched. For instance, the development set contained Emotion labels of 0.5, 1.5 and 2.5, which were not presented in the training set. If the test set had similar patterns, then, the inconsistent labels between training and testing could cause degradation of the fine-tuned models.

heading: Future Work
text: In the future, we plan to introduce larger datasets for model retraining at earlier stage (e. g. the Masked Language Model) for better domain adaptation, and consider introducing conversational context and speaker personality for better model construction. Also, the performance of such models in downstream regression tasks will be further investigated. Limitations
The limitations of the proposed work included: 1) The training set was relatively small with less than 12000 samples. Fine-tuning the models on larger datasets might improve the performance. 2) The labels in the training set and the development set was mis-matched. For instance, the development set contained Emotion labels of 0. 5, 1. 5 and 2. 5, which were not presented in the training set. If the test set had similar patterns, then, the inconsistent labels between training and testing could cause degradation of the fine-tuned models