Zhenmei at WASSA-2024 Empathy and Personality Shared Track: Incorporating Pearson Correlation Coefficient as a Regularization Term for Enhanced Empathy and Emotion Prediction in Conversational Turns (2024)

Abstract: In the realm of conversational empathy and emotion prediction, emotions are frequently categorized into multiple levels. This study seeks to enhance the performance of emotion prediction models by incorporating the Pearson correlation coefficient as a regularization term within the loss function. This regularization approach ensures closer alignment between predicted and actual emotion levels, mitigating extreme predictions and resulting in smoother and more consistent outputs. Such outputs are essential for capturing the subtle transitions between continuous emotion levels. Through experimental comparisons between models with and without Pearson regularization, our findings demonstrate that integrating the Pearson correlation coefficient significantly boosts model performance, yielding higher correlation scores and more accurate predictions. Our system officially ranked 9th at the Track 2: CONV-turn. The code for our model can be found at Link 1.

heading: 1 Introduction
text: Accurately predicting emotions is crucial for creating responsive and empathetic conversational systems. Emotions are typically classified into multiple levels, reflecting their nuanced and continuous nature. Many existing approaches focus on minimizing prediction errors but often overlook the linear relationship between predicted and actual emotion levels, resulting in extreme and unstable predictions (Acheampong et al., 2021; Zhou et al., 2024; Creanga and Dinu, 2024).
To address this, we propose enhancing emotion prediction models by incorporating the Pearson correlation coefficient as a regularization term in the loss function. The Pearson correlation measures the linear correlation between predicted and actual
1https://github.com/gongziruo/Empathy-and-EmotionPrediction-in-Conversations-Turns-CONV-turn
emotion levels. By including this metric, our approach aims to generate predictions that minimize absolute error while maintaining high correlation with actual emotions. Theoretical analysis confirms the differentiability and convergence of the proposed loss function, ensuring a stable and reliable optimization process.
Additionally, we introduce Consistent-Mixup data augmentation(CMDA) and boosting techniques to further improve model performance. CMDA creates new training samples by combining data from different classes, enhancing the model’s ability to generalize. Boosting leverages the strengths of multiple models, such as BERT (Devlin et al., 2018), RoBERTa (Liu et al., 2019) and DeBERTa (He et al., 2020), by combining their predictions based on individual accuracies, thereby improving overall accuracy.
We conducted experiments to validate our approach, comparing models trained with and without Pearson regularization, and those enhanced with CMDA and boosting. Results show that incorporating the Pearson correlation coefficient significantly improves performance, yielding higher correlation scores and more accurate emotion predictions. Furthermore, combining CMDA and boosting techniques leads to even greater improvements in model effectiveness.

heading: 2 Related Work
text: Recent research has explored various fine-tuning strategies for Transformer-based models like BERT, RoBERTa, and DeBERTa to enhance downstream performance. Sun et al. (2019) demonstrated significant improvements with techniques such as layerwise learning rate decay and data augmentation. Mosbach et al. (2020) provided insights into stable fine-tuning through learning rate schedules and early stopping. Additionally, Dong et al. (2019) proposed a unified pre-training framework for lan-
399
guage understanding and generation, while Gao et al. (2023) introduced progressive module training to incrementally fine-tune models, enhancing performance and stability.
In dialogue systems, Transformer models have been effectively applied to emotion prediction (Acheampong et al., 2021; Vazquez-Rodriguez et al., 2022). Tu et al. (2022) improved emotion recognition by leveraging context-aware embeddings and fine-tuning on emotion-labeled dialogue datasets. The WASSA 2023 shared task further explored empathy, emotion, and personality detection in conversations and reactions to news articles, highlighting the challenges and advancements in this domain (Barriere et al., 2023; Giorgi et al., 2024).
While Pearson correlation regularization remains underexplored, other methods like adversarial training (Liu et al., 2020) have been examined to enhance model robustness by adding input perturbations. These studies underscore the evolving fine-tuning methodologies for Transformer models, showcasing strategies such as layer-wise learning rate decay, context-aware embeddings, adversarial training, and progressive module training to enhance performance and stability in NLP tasks.

heading: 3 Methodology

heading: 3.1 Pearson Coefficient as Regularization Term
text: To incorporate the negative Pearson coefficient as a regularization term in the loss function, the total loss can be expressed as:
Ltotal = Lcom + λ(1− ρ(ŷ,y)), (1)
where λ is the regularization coefficient, and ρ(ŷ,y) represents the Pearson correlation between predictions ŷ and true labels y.
The combined loss Lcom is defined as:
Lcom = aLCE +β
 − 1
N
N∑
i=1
C∑
j=1
Pyij log(pij)
  .
(2) In this equation, LCE stands for Cross-Entropy Loss, a is the weight of LCE, Pyij indicates the penalty for misclassifying class yi as class j, pij is the predicted probability for class j, N is the number of samples, and C represents the number of classes. Each value in the penalty matrix P is non-negative, with higher penalties assigned for
misclassifications between labels that are numerically farther apart.

heading: 3.1.1 Differentiability
text: The Pearson correlation coefficient between two variables ŷ = (ŷ1, . . . , ŷn) and y = (y1, . . . , yn) is defined as:
ρ(ŷ,y) = ∑n i=1(ŷi − ¯̂y)(yi − ȳ)√∑n
i=1(ŷi − ¯̂y)2 √∑n i=1(yi − ȳ)2 ,
where ¯̂y and ȳ are the sample means of ŷ and y, respectively.
To derive the gradient of the Pearson correlation coefficient, we apply the quotient rule. Let u =
∑n i=1(ŷi − x̄)(yi − ȳ) and v =√∑n
i=1(ŷi − ¯̂y)2 √∑n
i=1(yi − ȳ)2. Then, the partial derivative of ρ(ŷ,y) with respect to ŷi is given by:
∂ρ(ŷ,y)
∂ŷi =
v ∂u∂ŷi − u ∂v ∂ŷi
v2 . (3)
The partial derivatives of u and v with respect to ŷi are:
∂u ∂ŷi = 1 n (yi − ȳ), (4) ∂v ∂ŷi = σy nσŷ (ŷi − ¯̂y), (5)
where σŷ and σy are the sample standard deviations of ŷ and y, respectively.
Substituting these partial derivatives into the quotient rule and simplifying, we obtain the final expression for the partial derivative of the Pearson correlation coefficient with respect to ŷi:
∂ρ(ŷ,y)
∂ŷi =
1
nσŷσy
( (yi − ȳ)− ρ(ŷ,y) · (ŷi − ¯̂y) ) .
Similarly, the partial derivative with respect to yi is given by:
∂ρ(ŷ,y)
∂yi =
1
nσŷσy
( (ŷi − ¯̂y)− ρ(ŷ,y) · (yi − ȳ) ) .

heading: 3.1.2 Convergence Analysis
text: Consider the total loss function Ltotal which includes the Pearson correlation coefficient as a regularization term, as shown in Equation (1).
The Pearson correlation coefficient ρ(ŷ,y) is a smooth function, composed of means, covariances,
and standard deviations. Consequently, the regularization term λ(1− ρ(ŷ,y)) is also smooth.
Since ρ(ŷ,y) is bounded within [−1, 1], the regularization term is bounded as well:
0 ≤ λ(1− ρ(ŷ,y)) ≤ 2λ. (6)
This boundedness ensures the regularization term does not overshadow the combined loss term Lcom, allowing it to guide the optimization process effectively. Both Lcom and the regularization term are smooth and differentiable, making Ltotal smooth and differentiable.
In gradient descent, a smooth and differentiable loss function typically ensures convergence to a local optimum with an appropriate learning rate.
The gradients of Ltotal with respect to ŷ and y are:
∂Ltotal ∂ŷi = ∂Lcom ∂ŷi − λ nσŷσy ((yi − ȳ)
−ρ(ŷ,y) · (ŷi − ¯̂y) ) , (7)
∂Ltotal ∂yi = ∂Lcom ∂yi − λ nσŷσy
( (ŷi − ¯̂y)
−ρ(ŷ,y) · (yi − ȳ)) . (8)
The gradient descent update rules for ŷ and y are:
ŷ (t+1) i = ŷ (t) i − η ∂Ltotal ∂ŷi , (9)
y (t+1) i = y (t) i − η ∂Ltotal ∂yi , (10)
where η is the learning rate and t denotes the iteration.
Given the smoothness and differentiability of the total loss function, and with an appropriate learning rate, the gradient descent algorithm is expected to converge to a local optimum, minimizing the total loss Ltotal.
In conclusion, the inclusion of the Pearson correlation coefficient as a regularization term maintains the smoothness and differentiability of Ltotal. This ensures the gradient descent algorithm converges to a local optimum, minimizing Ltotal.

heading: 3.2 Consistent-Mixup Data Augmentation
text: To enhance the robustness of emotion and empathy prediction models, we employed a ConsistentMixup data augmentation (CMDA) technique. Traditional Mixup methods, which interpolate between
pairs of inputs and their labels, can lead to inaccuracies in multi-label tasks like emotion and empathy prediction due to label complexity.
Our approach ensures that only samples with the same label are mixed, maintaining label consistency and preventing erroneous data. Given input sequences x with labels y, the data augmentation process is:
x̃ = γxi + (1− γ)xj , (11)
where yi = yj and γ ∼ Beta(α, α). The Beta distribution, denoted as Beta(α, α), is a continuous probability distribution defined on the interval [0, 1] and parameterized by two positive shape parameters α. Setting both parameters to α ensures a symmetric distribution, which results in a balanced interpolation between inputs. This preserves the integrity of the labels, enhancing the quality of the augmented data and improving model performance and generalization.

heading: 3.3 Experimental Setup
text: To validate the effectiveness of incorporating the Pearson correlation coefficient as a regularization term in emotion prediction models, we conducted a series of experiments using several state-of-the-art language models: BERT, RoBERTa, and DeBERTa. These models were chosen for their robust performance in various natural language understanding tasks. Additionally, we applied CMDA and boosting techniques to combine these models, aiming to enhance performance by leveraging their complementary strengths.

heading: 3.4 Datasets and Data Preprocessing
text: We used the Track 2 (CONV-turn) dataset, which contains 11,166 training items, 990 develop items, and around 2,300 test items, each with a dialogue text and three corresponding labels: emotional intensity, emotional polarity, and empathy. The length of each dialogue text ranges from 2 characters to 128 characters. The values of emotional intensity and empathy range from 0-5, and the values of emotional polarity range from 0-3. The data is preprocessed by identifying redundant columns and merging the redundant column contents into the correct columns to ensure that the final data is aligned with the corresponding features, and no data is lost in this process(Omitaomu et al., 2022).

heading: 3.5 Models and Training
text: Baseline Models: We implemented baseline versions of BERT, RoBERTa, and DeBERTa without Pearson regularization. These models were trained using the loss function (Lcom).
Enhanced Models: For the enhanced versions, we added the Pearson correlation coefficient as a regularization term to the loss function. For a given predicted sentiment level ŷ and actual sentiment level y is defined as:
Loss = Lcom(ŷ, y)− λ · (1− Pearson(ŷ, y)), where λ is a hyperparameter that controls the weight of the Pearson regularization term.
Training process: All models were trained using the Adam optimizer with a learning rate of 1e−5. To enhance the models, we incorporated data augmentation. Specifically, we employed CMDA ensuring the mixed labels remained consistent. Additionally, we adopted a boosting strategy by training three different models(BERT, RoBERTa, DeBERTa) and aggregating their outputs using a weighted average method to form the final prediction. This ensemble approach aimed to leverage the strengths of each individual model and improve overall performance.

heading: 3.6 Experimental Results
text: The evaluation metric used in this study is the Pearson Correlation Coefficient, which evaluates the linear correlation between the predicted and actual sentiment levels, reflecting the consistency of the predictions.
The study evaluates sentiment prediction models using the Pearson Correlation Coefficient to measure the linear correlation between predicted and
actual sentiment levels. Table 1 shows that using Pearson correlation as a regularizer significantly enhances performance across all tested configurations. Enhanced models (BERT, RoBERTa, and DeBERTa with Pearson regularization) consistently outperform their baselines in Emotion, Emotional Polarity, and Empathy tasks.
Furthermore, Boosting models demonstrate additional improvements. The Boosting (Standard) model, which combines the results of the individual standard models using weighted averages, shows better performance than the individual models. The Boosting (Pearson) model, which similarly combines the Pearson-regularized models, achieves even higher scores. The best performance is from the Boosting (Pearson, CMDA) model, with top scores in Emotional Polarity (0.765) and Empathy (0.658). The Boosting (Pearson) model excels in Emotion (0.667), underscoring the benefits of Pearson correlation regularization.
The test set results also highlight the robustness of the models. The Boosting (Pearson, CMDA) model achieved scores of 0.581, 0.644, and 0.561 in Emotion, Emotional Polarity, and Empathy respectively. It is important to note that these scores are significantly higher than the official results of -0.027, -0.020, and -0.043 respectively. The discrepancy arose because an early version of the model was submitted by mistake, leading to the lower scores. The updated results presented here reflect the true performance of the final, optimized models.

heading: 4 Conclusion
text: We proposed an enhanced approach for emotion prediction by incorporating the Pearson correlation coefficient as a regularization term in the loss function, ensuring closer alignment between predicted and actual emotion levels. This method, along with CMDA and boosting techniques, significantly improved model performance, yielding higher correlation scores and more accurate predictions. Our findings underscore the potential of correlation-based regularization and advanced training techniques in enhancing Transformer-based models for emotion prediction tasks.
Limitations
Due to time constraints, we submitted an earlier version of our results, leading to a lower score of -0.03 on TRACK CONV-turn. Here, we present
the best results to accurately represent our system’s performance, as shown in table 1. Relying solely on the Pearson correlation coefficient may not fully demonstrate our approach’s effectiveness. A 1-3% increase in the Pearson coefficient, though modest, shows consistent improvement. For a more comprehensive evaluation, we will include other metrics, such as the F1 score, in future work. These additional metrics will further validate our approach.

heading: Future Work
text: For a more comprehensive evaluation, we will include other metrics, such as the F1 score, in future work. These additional metrics will further validate our approach