EmpatheticFIG at WASSA 2024 Empathy and Personality Shared Task: Predicting Empathy and Emotion in Conversations with Figurative Language (2024)

Abstract: Recent research highlights the importance of figurative language as a tool for amplifying emotional impact. In this paper, we dive deeper into this phenomenon and outline our methods for Track 1, Empathy Prediction in Conversations (CONV-dialog) and Track 2, Empathy and Emotion Prediction in Conversation Turns (CONV-turn) of the WASSA 2024 shared task. We leveraged transformer-based large language models augmented with figurative language prompts, specifically idioms, metaphors and hyperbole, that were selected and trained for each track to optimize system performance. For Track 1, we observed that a fine-tuned BERT with metaphor and hyperbole features outperformed other models on the development set. For Track 2, DeBERTa, with different combinations of figurative language prompts, performed well for different prediction tasks. Our method provides a novel framework for understanding how figurative language influences emotional perception in conversational contexts. Our system officially ranked 4th in the 1st track and 3rd in the 2nd track.

heading: 1 Introduction
text: The computational study of empathy1 is crucial to enabling and advancing the development of innovative and resourceful tools for social good in various settings, ranging from online conversations to clinical therapy (Eysenbach et al., 2004; Elliott et al., 2018). At a broader level, recognizing emotional needs and appropriately responding to them are essential for successful interactions, making this an important step in chatbot development. However, despite the recent surge in interest in automated empathy detection (Barriere et al., 2023; Lee and Parde, 2024; Giorgi et al., 2024; Lee et al.,
*Equal contribution 1We follow Davis et al. (1980)’s definition of empathy as the ability to understand and respond to the experiences and feelings of others.
2024), research focusing on empathetic dialogue involving back-and-forth conversations (such as that by Rashkin et al. (2018)) still remains scarce. The Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis (WASSA) 2023 (Barriere et al., 2023) and 2024 (Giorgi et al., 2024) provides the opportunity to explore this domain further with the Empathic Conversations dataset (Omitaomu et al., 2022).
Analyzing linguistic features is crucial in understanding how language is used to convey empathy and emotion in dialogue. Figurative language (non-literal language; see § 2.3.2 for more information), and particularly metaphor, has been shown to enhance the performance of emotion prediction models (Dankers et al., 2019), hold more emotional charge than literal language (Citron and Goldberg, 2014), and strengthen expressions (Mohammad et al., 2016; Li et al., 2023). From our own recent work, we found that figurative language prompts improved empathy detection performance when using the domain-specialized AcnEmpathize dataset (Lee et al., 2024). Therefore, we hypothesized that identifying the use of figurative language in Empathic Conversations could likewise provide deeper insight into the study of empathy and emotion. We thus propose methods to encode figurative language prompts into large language models (LLMs) for emotion and empathy prediction in conversations.
In this shared task, we participated in two of the four tracks:
• Track 1: Empathy Prediction in Conversations (CONV-dialog), focusing on predicting perceived empathy at the dialogue level.
• Track 2: Empathy and Emotion Prediction in Conversation Turns (CONV-turn), focused on predicting perceived empathy, emotion polarity, and emotion intensity at the speech-turn level in a conversation.
441
In both tracks, we implemented transformerbased models with additional figurative language prompts. We applied combinations of figurative language prompts that yielded the best performance on the development sets. In Track 1, we used metaphor and hyperbole features with BERT (Devlin et al., 2018). In Track 2, we used idiom and hyperbole features together for empathy prediction, hyperbole for emotional polarity, and metaphor for emotion intensity with DeBERTa (He et al., 2021).

heading: 2 System Description

heading: 2.1 Track 1: Empathy Prediction in Conversations (CONV-dialog)
text: The goal of this track was to predict the perceived empathy of one person towards another based on their conversation. For each conversation ID, we were given the speaker ID of Speaker 1 and their perceived empathy from the other person, Speaker 2. We extracted all texts from Speaker 2 in each conversation and combined them to predict the empathy level perceived by Speaker 1, implementing this as a multi-class classification problem.

heading: 2.2 Track 2: Empathy and Emotion Prediction in Conversation Turns (CONV-turn)
text: In this track, we used the text of each speaker in each conversation to predict turn-level annotated emotion, emotional polarity, and empathy. Since the target labels were provided for each turn, preprocessing the text was unnecessary. We encoded the conversations and figurative language prompts at the turn level, subsequently passing them through the task-specific multilayer perception (MLP) layer to obtain the output scores.

heading: 2.3 EmpatheticFIG
text: We propose EmpatheticFIG, a framework that incorporates figurative language prompts for empathy and emotion prediction in conversations. It draws inspiration from our earlier work (Lee et al., 2024) and that of others emphasizing the importance of metaphor to emotional expression (Citron and Goldberg, 2014; Mohammad et al., 2016; Dankers et al., 2019; Li et al., 2023). The main architecture is illustrated in Figure 1.
We used mT5 (Xue et al., 2021) to extract figurative language prompts (described further in §2.3.2) and appended these features to the conversation texts. These combined inputs were then processed by pre-trained LLMs, including BERT (Devlin et al., 2018), DeBerta (He et al., 2021), RoBERTa (Liu et al., 2019), T5 (Raffel et al., 2020), and Llama (Touvron et al., 2023). Finally, we applied task-specified MLP layers for Track 1 and for different tasks in Track 2 to generate outputs.

heading: 2.3.1 Models and Hyperparameters
text: We conducted extensive experiments using different LLMs and hyperparameter settings. In Track 1, the backbone models were BERT-base, DeBerta-v3-base, RoBERTa-base, and Llama-38b. In Track 2, we fine-tuned BERT, DeBerta, RoBERTa, and T5-small. We fine-tuned BERT, DeBerta, RoBERTa, and T5 on conversation texts with figurative language prompts for both tracks. When using Llama, we utilized Parameter-Efficient Fine-Tuning (Hu et al., 2021, PEFT) to adapt the model weights for perceived empathy level prediction for Track 1.
For both tracks, we searched for the optimal hyperparameters from the following sets: learning rates ranging from {3e-4, 1e-4, 5e-5, 1e-5, 5e-6}, batch sizes from 2 to 64 depending on the model, and training epochs ranging from {1, 3, 5, 10, 30,
50}. The AdamW (Loshchilov and Hutter, 2017) optimizer was used with a weight decay of 0.01.

heading: 2.3.2 Figurative Language Prompts
text: Since empathy is primarily conveyed through language, we focused our investigation on the performance benefits of incorporating automatically extracted linguistic constructs (figurative language phenomena) from the text to assist in predicting empathy and emotion labels. Figurative language is non-literal language (Paul, 1970) that serves to compare, exaggerate, and add nuanced meaning; oftentimes, it is used as a vehicle to simplify complicated or abstract ideas.
We applied the multi-figurative language detection method from Lai et al. (2023) to identify metaphor, idiom, and hyperbole (three distinct types of figurative language, explained later in this section) in Empathic Conversations. Lai et al. (2023)’s approach requires the use of a pre-trained LLM; we specifically employed mT5 (Xue et al., 2020) using predefined prompts of the format:
Which figure of speech does this text contain? (A) Literal (B) [Task] | Text: [Text]
We filled [Task] using metaphor, idiom, and hyperbole, respectively. Using this approach, we then one-hot encoded the labels for each detected type of figurative language if at least one type was present in the text. Finally, we appended a description to the original conversation text that reads, The text contains <label>., where “label” represents the type of figurative language. Again, note that each text may contain more than one type of figurative language (which were appended separated by commas).
In Table 1, we summarize the distributions of figurative language types in the combined training and development datasets for Track 1 and Track
2. We find that the datasets for both are rich in figurative language—in total, figurative language is present in approximately 91% and 80% of the samples, respectively. Below, we briefly define and provide examples for each type of figurative language identified.
Metaphor. Metaphoric expressions frame conventional ideas in more accessible terms by assigning new meanings (Lakoff and Johnson, 1980). For example, the phrase “fight these natural disasters” from Empathetic Conversations personifies natural calamities as adversaries one must battle.
Idiom. Idiomatic expressions tie abstract ideas or meanings to more concrete anchors and often involve cultural context (Nunberg et al., 1994). For example, the phrase “pull through the hard times” from Empathetic Conversations uses idiomatic language to convey the act of enduring and surviving difficult times.
Hyperbole. Hyperbolic expressions are often used to exaggerate a statement or phenomenon (Claridge, 2010), as in “thousands of years of human progress.” This phrase from the shared task dataset likely emphasizes the significance of human progress over a long period, rather than literally referring to exactly thousands of years.

heading: 3 Results and Discussions
text: To investigate the role of figurative language in empathy and emotion prediction in conversations, we implemented EmpatheticFIG for Track 1 and Track 2. We conducted experiments using different combinations of figurative language prompts and generally found that many models with idiom features performed worse, perhaps due to the high prevalence of idioms in the dataset. We present
the experimental results on the development and test sets for Track 1 and Track 2 in Tables 2 and 3, respectively. To assess the impact of figurative language prompts, we also conducted ablation experiments using various combinations of figurative language types. Moreover, we evaluated the performance of these features across different LLMs. We discuss the results for Tracks 1 and 2 in more detail in §3.1 and §3.2, respectively.

heading: 3.1 Results for Track 1 (CONV-dialog)
text: Table 2 presents the development set results for Track 1, showcasing the averaged Pearson correlation values from three runs for each setting. Models incorporating figurative language prompts consistently outperformed the baseline models, suggesting that these features positively impact perceived empathy level predictions. Metaphors generally improved the performance for perceived empathy level prediction and seemed to enhance the ability of language models to capture the nuanced emotional cues in empathetic conversations.
We applied the settings that yielded the best performance on the development set to the test set, specifically using BERT with metaphor and hyperbole features, a learning rate of 5e-5, a batch size of 8, and an epoch length of 3. However, we observed a drop in performance on the test set (see results in Appendix A.1), potentially due to class imbalance and overfitting. We leave further investigations of test set performance for future work.

heading: 3.2 Results for Track 2 (CONV-turn)
text: We present the results for each task in Track 2 in Table 3. DeBERTa was the best performing model among all the baseline models. We observed that
task performance varied depending on the specific figurative language prompts used on the development set. DeBERTa with idiom and hyperbole features notably improved empathy predictions. DeBERTa with hyperbole achieved the highest performance on emotion polarity predictions, while DeBERTa with metaphor slightly enhanced predictions for emotion intensity.
We found that different figurative language prompts provided varying levels of information and impact on different task predictions. For example, hyperbole contributed to improved emotion polarity, while idioms and hyperbole enhanced performance in empathy prediction for turn-level conversations. In contrast, metaphor had less impact on empathy and emotion predictions at the turn level (Track 2) compared to its impact at the dialogue level in Track 1. The test set performance (see Appendix A.2) also shows consistent results.

heading: 4 Conclusion
text: Our team, EmpatheticFIG, participated in Track 1 (Empathy Prediction in Conversations) and Track 2 (Empathy and Emotion Prediction in Conversation Turns) of the WASSA 2024 shared task. Our system architecture involved fine-tuning various LLMs, such as BERT and DeBERTa, with one or more combinations of figurative language types—idioms, metaphors, and hyperbole. The results showed that incorporating figurative language prompts was beneficial for predicting empathy and emotion in conversations. Our method provides unique insights into adapting figurative language prompts into LLMs.

heading: 5 Limitations and Future Work
text: We could experiment with larger, more complex models and perform extensive hyperparameter tuning. Additionally, we could verify the detection of figurative language expressions in the dataset and conduct a deeper analysis of their usage to better understand the reasons behind the performance. Furthermore, we could explore more types of figurative language beyond those we have already investigated.

heading: Acknowledgments
text: We thank the anonymous reviewers for their feedback, and the WASSA 2024 shared task organizers for running this task. This work was partially supported by the National Science Foundation under Grant No. 2125411. Any opinions, findings, and conclusions or recommendations are those of the authors and do not necessarily reflect the views of the National Science Foundation.

heading: A Appendix
text: A.1 Test results for Track 1 We illustrate our test results for Track 1, noting a significant performance drop on the test set. The performance on the dev set was unstable and sensitive to hyperparameters. We will leave the investigation of the test set settings and improving the robustness of our system, EmpatheticFIG, for future work.
A.2 Test results for Track 2 The performance of our model on the test set in Track 2 showed similar pattern to our performance on the development set. Our model performed best in predicting emotion polarity, empathy, and emotion intensity (in this order) and achieved an average Pearson r of 0.610.

heading: Future Work
text: We leave further investigations of test set performance for future work