CHEER-Ekman: Fine-grained Embodied Emotion Classification
Authors: Phan Anh Duong, Cat Luong, Divyesh Bommana, Tianyu Jiang
Accepted at: acl25

CHEER-Ekman: Fine-grained Embodied Emotion Classification

Phan Anh Duong, Cat Luong, Divyesh Bommana, Tianyu Jiang
University of Cincinnati
{duongap, luongcn, bommandh}@mail.uc.edu, tianyu.jiang@uc.edu

Abstract

Emotions manifest through physical experiences and bodily reactions, yet identifying such embodied emotions in text remains understudied. We present an embodied emotion classification dataset, CHEER-Ekman,1 extending the existing binary embodied emotion dataset with Ekman’s six basic emotion categories. Using automatic best-worst scaling with large language models, we achieve performance superior to supervised approaches on our new dataset. Our investigation reveals that simplified prompting instructions and chain-of-thought reasoning significantly improve emotion recognition accuracy, enabling smaller models to achieve competitive performance with larger ones.

1 Introduction

Emotions are not merely abstract mental states; they are deeply intertwined with somatic experiences. When we feel joy, our faces light up with smiles; when we are scared, our hearts race and our hands tremble. These physical reactions are more than just side effects—they are part of how we experience and express emotions. This concept, known as embodied emotion, suggests that our bodies play a key role in how we feel, perceive, and understand emotions (Lakoff and Johnson, 1999; Niedenthal, 2007). In natural language, these connections surface as descriptions of physiological reactions (e.g., “my stomach churned in disgust”) or unintentional physical actions (e.g., “she stomped her feet in frustration”—phenomena termed embodied emotions. Recognizing such expressions is pivotal for understanding implicit emotional cues in narratives. While recent advances in NLP have focused more on explicit emotion classification (Mohammad et al., 2018) or sentiment analysis (Rosenthal et al., 2017), the subtler task of identifying embodied emotions remains less explored, despite its psychological grounding and practical relevance.

The CHEER dataset (Zhuang et al., 2024) filled a gap in this field by providing a collection of sentences where body parts are used to express emotions. The dataset includes 7,300 human-annotated sentences containing body part references, proposing a binary classification task which we will refer to as “embodied emotion detection.” However, one limitation of this work is that it does not distinguish between different types of emotions—for instance, whether a racing heart signals fear or excitement. The framework of Ekman’s (1992) basic emotions offers a potential solution to this limitation. By linking embodied expressions to these specific emotions, we can build systems that better understand human emotional experiences.

To achieve this goal, we extend the CHEER dataset by annotating all its 1,350 positive samples with six Ekman emotion labels (Joy, Sadness, Anger, Disgust, Fear, and Surprise), creating a new dataset, CHEER-Ekman, as illustrated in Figure 1. For clarity, we refer to the novel classification task produced by this dataset as “embodied emotion classification,” as compared to the binary task discussed above. We adopt the automatic best-worst scaling (BWS) technique (Kiritchenko and Mohammad, 2017; Bagdon et al., 2024) with large

![Illustration of embodied emotions classified into six categories.](page_1012_362_388_246.png)

Figure 1: Illustration of embodied emotions classified into six categories.
language models (LLMs) to tackle the task. Our experiments show that using Llama 3.1 8B with BWS significantly outperforms zero-shot prompting. The best BWS experiment achieved 50.6 F1-score, surpassing supervised BERT (49.6) and beating zero-shot approaches by around 20 points. Building on Zhuang et al.’s (2024) investigation of LLMs’ capability for embodied emotion detection, we further explore prompting techniques that enhanced the detection task. Our experimental results reveal that LLMs can make better recognition when instructions are rephrased in plain, easily understood language, which boosts F1 by nearly 30 points compared to technical definitions. Moreover, chain-of-thought reasoning enables an 8B parameter model to nearly match a 70B model, closing the performance gap to within 7 F1 points. In summary, our contributions are three-fold:

1. We present CHEER-Ekman, an extension of the CHEER dataset that enriches embodied emotion expressions with fine-grained Ekman emotion labels, addressing a critical gap in understanding how specific emotions manifest through bodily expressions. Our dataset is available at: https://github.com/menamerai/cheer-ekman.

2. We demonstrate that the automatic best-worst scaling technique enables LLMs to perform emotion classification without any task-specific training, achieving performance that exceeds supervised approaches.

3. We reveal that counterintuitively, simplified everyday language in prompts dramatically outperforms technical definitions for embodied emotion tasks, and that structured reasoning through chain-of-thought can allow smaller language models to perform at a level closer to larger language models on our task.

2 Related Work

Emotion recognition in natural language processing has been extensively studied, with researchers focusing more on explicit emotion using datasets like SemEval (Strapparava and Mihalcea, 2007; Mohammad et al., 2018) and GoEmotions (Demszky et al., 2020). Recent research has expanded to explore nuanced aspects of emotion expression (Li et al., 2021), including emotion intensity prediction (Mohammad, 2018; Bagdon et al., 2024) and the detection of subtle emotional cues in dialogues (Poria et al., 2019; Ghosal et al., 2020; Li et al., 2022). The advent of large language models (LLMs) has catalyzed significant advances in emotion understanding capabilities (Lee et al., 2024; Sabour et al., 2024; Zhao et al., 2024; Liu et al., 2024).

While these works contribute to a deeper understanding of emotion detection in text, the embodied nature of emotions—how physical sensations and actions encode affective states—has received comparatively less attention. The concept of embodied emotion is rooted in cognitive science, particularly in the works of Lakoff and Johnson (1999) and Niedenthal (2007), which suggest that emotional experiences are closely tied to bodily states and actions. Despite the psychological grounding of embodied emotions, computational approaches to capturing them in text remain limited. Zhuang et al. (2024) introduced the CHEER dataset, which provides a collection of sentences where body parts are explicitly used to express emotions. Our work extends theirs by incorporating Ekman’s six basic emotions into embodied emotion recognition, offering a more fine-grained classification system.

Another relevant line of work is the study of emotion taxonomy. Although recent advances in psychology have offered newer granular categories of emotions such as 27 emotions by Cowen and Keltner (2017), which has been adopted in both textual emotion datasets (Demszky et al., 2020) and visual emotion dataset (Kosti et al., 2019), we follow the vast majority of existing emotion datasets (Strapparava and Mihalcea, 2007; Mohammad et al., 2018; Poria et al., 2019) by utilizing the six basic emotions (Joy, Sadness, Anger, Disgust, Fear, and Surprise) proposed by Ekman (1992), which remain foundational due to their universality and simplicity. Future research may explore integrating alternative taxonomies into embodied emotion classification to enhance both granularity and coverage.

3 Methods

Our methodological approach comprises three key components that build upon and extend the work of Zhuang et al. (2024). First, we explore prompting strategies to enhance LLMs’ capability to detect embodied emotions. Second, we introduce CHEER-Ekman, a refinement of the original CHEER dataset that adds fine-grained emotion labels. Finally, we adopt the BWS framework for
emotion classification that leverages comparative judgments to improve classification accuracy.

3.1 Prompting LLMs for Embodied Emotion Detection

To address the gap in prompt design within the Zhuang et al.’s (2024) framework, we first sought to enhance the embodied emotion detection task using state-of-the-art LLMs and explore the impact of prompt engineering on performance. Our approach centers on two key strategies: prompt simplification to mitigate linguistic complexity and chain-of-thought (CoT) prompting.

Prompt Simplification. We investigated the effects of linguistic and domain complexity by conducting experiments with the base Llama-3.1 (Grattafiori et al., 2024) and the recently released DeepSeek-R1 distilled version (DeepSeek-AI et al., 2025). Specifically, we compared two prompts: the base prompt used in Zhuang et al. (2024) and a simplified prompt, which reduces syntactic and lexical complexity to minimize potential comprehension barriers for LLMs.

Chain-of-Thought Prompting. We further explored eliciting reasoning from the model by implementing chain-of-thought (CoT) prompting. Based on Zhuang et al.’s (2024) annotation criteria for embodied emotion detection, we developed three CoT variants: a 2-step variant that evaluates emotional causation and purposeless expression, a 3-step variant that adds body part identification, and a simplified 2-step variant with reduced linguistic complexity. These variants allowed us to examine how explicit causal reasoning affects both the model’s emotion detection performance and its understanding of body-emotion relationships.

3.2 Dataset Creation

While embodied emotion detection identifies emotional expressions through bodily movements, understanding the specific emotions conveyed requires more fine-grained annotation. To address this need, we propose CHEER-Ekman, a refined dataset extending the original CHEER corpus (Zhuang et al., 2024) by annotating its 1,350 positive embodied emotion instances with Ekman’s (1992) six basic emotions (Joy, Sadness, Anger, Disgust, Fear, and Surprise). Our adoption of Ekman’s basic emotions taxonomy balances granularity with practical considerations, as recent research by Liu et al. (2024) demonstrates that finer-grained emotion taxonomies often face sparsity issues, even in bigger datasets like GoEmotions (Demszky et al., 2020).

This approach also maintains consistency with Zhuang et al.’s (2024) methodology, which utilized emotion-associated adverbs derived from these basic emotions for weak supervision. We have also elected not to include the weakly labeled positive samples from the CHEER dataset, prioritizing our final dataset quality and reliability over quantity.

We recruited two annotators to label the 1,350 embodied emotion sentences from the original CHEER dataset. For each sentence, we provide the annotators with the sentence, the relevant body part, and up to three preceding sentences for context. We then ask the annotators to select one of the six emotions that best match the physical experience described through the body part. The pairwise inter-annotator agreement by Cohen’s Kappa is 0.64, indicating good agreement. Finally, the annotators adjudicated their disagreements to produce the final gold labels. We show some example sentences with their annotated emotions in Table 1.

Figure 2 illustrates the emotion distribution of sentences in our newly constructed CHEER-Ekman dataset. Specifically, Fear is the most prevalent emotion (24.7%), followed by Joy (21.2%), Sadness (19.3%), Surprise (13.3%), and Disgust (12.5%). Anger appeared last at 9.0%.

<table>
  <tr>
    <th>Emotion</th>
    <th>Sentence</th>
  </tr>
  <tr>
    <td>Joy</td>
    <td>... watched the fireflies with a loving look on his face.</td>
  </tr>
  <tr>
    <td>Sadness</td>
    <td>... frowning and scuffing his feet along the floor.</td>
  </tr>
  <tr>
    <td>Fear</td>
    <td>Marty nervously runs his fingers through his hair...</td>
  </tr>
  <tr>
    <td>Anger</td>
    <td>... makes me want to hit my head against the wall.</td>
  </tr>
  <tr>
    <td>Disgust</td>
    <td>Dean snorted incredulously, shaking his head in disbelief.</td>
  </tr>
  <tr>
    <td>Surprise</td>
    <td>... my eyes almost fell out of my head.</td>
  </tr>
</table>

Table 1: Examples in our CHEER-Ekman datasets.

![Pie chart showing the distribution of emotions in the CHEER-Ekman dataset](page_820_120_527_312.png)

Figure 2: CHEER-Ekman dataset distribution of emotions.
3.3 BWS for Emotion Classification

To address the fine-grained emotion classification task proposed with CHEER-Ekman, we first tested zero-shot LLM prompting. However, the model often fails to adhere to the instructions and is prone to erroneous behaviors, leading to incorrect outputs. Inspired by recent work on automatic emotion intensity annotation using LLMs (Bagdon et al., 2024), we adopted the same best-worst scaling technique. The methodology involves presenting the LLM with tuples of four different sentences, instructing it to identify the body part instances that most and least represent a specific Ekman emotion. Then, the equation \( \frac{\#Best - \#Worst}{\#Total} \) is used to calculate a score per sentence per emotion. Finally, we choose the emotion that receives the highest score to be the prediction for the sentence.

To examine how the number of comparisons affects classification accuracy, we tested a broader range of tuple counts, from 2N to 72N, increasing by 50% at each step of expansion (where N is the number of instances to be classified). While Bagdon et al. (2024) found that more tuples improved accuracy in their experiments up to 12N, we expanded this investigation to 72N to further explore performance gain behaviors from scaling comparative rounds.

<table>
  <tr>
    <th rowspan="2">Model</th>
    <th colspan="3">Macro</th>
    <th colspan="3">EE</th>
    <th colspan="3">Neutral</th>
  </tr>
  <tr>
    <th>F1</th>
    <th>Pre</th>
    <th>Rec</th>
    <th>F1</th>
    <th>Pre</th>
    <th>Rec</th>
    <th>F1</th>
  </tr>
  <tr>
    <td>Llama<sub>base</sub></td>
    <td>37.2</td>
    <td>21.5</td>
    <td>99.6</td>
    <td>35.3</td>
    <td>99.6</td>
    <td>24.2</td>
    <td>39.0</td>
  </tr>
  <tr>
    <td>Llama<sub>sample</sub></td>
    <td>66.7</td>
    <td>37.5</td>
    <td>89.3</td>
    <td>52.8</td>
    <td>96.9</td>
    <td>68.9</td>
    <td>80.6</td>
  </tr>
  <tr>
    <td>DeepSeek<sub>base</sub></td>
    <td>32.6</td>
    <td>20.3</td>
    <td>99.4</td>
    <td>33.7</td>
    <td>99.3</td>
    <td>18.7</td>
    <td>31.5</td>
  </tr>
  <tr>
    <td>DeepSeek<sub>sample</sub></td>
    <td>74.2</td>
    <td>51.2</td>
    <td>69.3</td>
    <td>58.9</td>
    <td>93.1</td>
    <td>86.2</td>
    <td>89.5</td>
  </tr>
  <tr>
    <td>GPT 3.5<sub>base</sub></td>
    <td>70.2</td>
    <td>44.0</td>
    <td>68.3</td>
    <td>53.5</td>
    <td>92.5</td>
    <td>81.9</td>
    <td>86.9</td>
  </tr>
  <tr>
    <td>BERT</td>
    <td>83.5</td>
    <td>73.2</td>
    <td>72.1</td>
    <td>72.6</td>
    <td>94.2</td>
    <td>94.5</td>
    <td>94.4</td>
  </tr>
</table>

Table 2: Results comparison for Embodied Emotion Detection. Llama: Llama-3.1-70B. DeepSeek: DeepSeek-R1-Distilled-Llama-70B. GPT 3.5 and fine-tuned BERT numbers are from Zhuang et al. (2024). The base and simple subscripts indicate the type of prompts, which can be found in Table 5.

<table>
  <tr>
    <th rowspan="2">Model</th>
    <th colspan="3">Macro</th>
    <th colspan="3">EE</th>
    <th colspan="3">Neutral</th>
  </tr>
  <tr>
    <th>F1</th>
    <th>Pre</th>
    <th>Rec</th>
    <th>F1</th>
    <th>Pre</th>
    <th>Rec</th>
    <th>F1</th>
  </tr>
  <tr>
    <td>Llama<sub>2-step</sub></td>
    <td>53.4</td>
    <td>26.2</td>
    <td>80.8</td>
    <td>39.6</td>
    <td>93.0</td>
    <td>52.7</td>
    <td>67.2</td>
  </tr>
  <tr>
    <td>Llama<sub>3-step</sub></td>
    <td>54.8</td>
    <td>24.9</td>
    <td>53.4</td>
    <td>34.0</td>
    <td>87.3</td>
    <td>66.5</td>
    <td>75.5</td>
  </tr>
  <tr>
    <td>Llama<sub>step-simple</sub></td>
    <td>60.1</td>
    <td>31.5</td>
    <td>44.5</td>
    <td>36.9</td>
    <td>87.4</td>
    <td>79.8</td>
    <td>83.4</td>
  </tr>
  <tr>
    <td>DeepSeek<sub>2-step</sub></td>
    <td>52.2</td>
    <td>26.4</td>
    <td>90.8</td>
    <td>40.9</td>
    <td>96.1</td>
    <td>47.3</td>
    <td>63.4</td>
  </tr>
  <tr>
    <td>DeepSeek<sub>3-step</sub></td>
    <td>57.4</td>
    <td>27.9</td>
    <td>62.0</td>
    <td>38.5</td>
    <td>89.4</td>
    <td>66.7</td>
    <td>76.4</td>
  </tr>
  <tr>
    <td>DeepSeek<sub>2-step-simple</sub></td>
    <td>67.5</td>
    <td>40.1</td>
    <td>65.2</td>
    <td>49.7</td>
    <td>91.7</td>
    <td>79.8</td>
    <td>85.3</td>
  </tr>
</table>

Table 3: CoT results for Embodied Emotion Detection. Llama: Llama-3.1-8B. DeepSeek: DeepSeek-R1-Distilled-Llama-8B. The 2-step, 2-step-simple, and 3-step subscripts indicate the type of prompt accompanying the model in that run. Prompt details are in Table 6.

GPT 3.5 results reported in Zhuang et al. (2024).

Chain-of-Thought (CoT) Analysis. Table 3 shows that CoT prompting enhanced performance to competitive levels with larger models in the experiments of Table 2, particularly benefiting distilled reasoning models like DeepSeek-R1-8B (distilled on Llama). The DeepSeek 8B model using simple 2-step prompts (DeepSeek<sub>2-step-simple</sub>) achieved results within 6.7 F1-points of its larger 70B counterpart (DeepSeek<sub>simple</sub>) and 2.7 F1-points of GPT 3.5. Deeper reasoning processes proved more effective, with 3-step CoT consistently outperforming 2-step variants across both models. Finally, simplified prompting substantially improved CoT performance, yielding F1-score increases of 6.7 (Llama<sub>2-step-simple</sub> vs. Llama<sub>2-step</sub>) and 15.3 (DeepSeek<sub>2-step-simple</sub> vs. DeepSeek<sub>2-step</sub>).

Error Analysis. To investigate model failures in the zero-shot experiments using the simple prompt setting with Llama-3.1-70B, we analyzed incorrect predictions. We found a pronounced false-positive bias, accounting for 93.3% of all errors. A manual inspection of 100 false-positive cases revealed
<table>
  <tr>
    <th>Model</th>
    <th>F1</th>
    <th>F1-J</th>
    <th>F1-Sa</th>
    <th>F1-F</th>
    <th>F1-A</th>
    <th>F1-D</th>
    <th>F1-Su</th>
  </tr>
  <tr>
    <td>Llama</td>
    <td>31.6</td>
    <td>39.4</td>
    <td>43.6</td>
    <td>26.6</td>
    <td>32.2</td>
    <td>19.1</td>
    <td>28.5</td>
  </tr>
  <tr>
    <td>DeepSeek</td>
    <td>28.4</td>
    <td>43.3</td>
    <td>35.7</td>
    <td>33.1</td>
    <td>23.1</td>
    <td>14.8</td>
    <td>20.2</td>
  </tr>
  <tr>
    <td>BWS<sub>4N</sub></td>
    <td>41.8</td>
    <td>62.3</td>
    <td>57.7</td>
    <td>37.9</td>
    <td>28.1</td>
    <td>30.1</td>
    <td>33.9</td>
  </tr>
  <tr>
    <td>BWS<sub>12N</sub></td>
    <td>44.6</td>
    <td>67.1</td>
    <td>59.2</td>
    <td>44.3</td>
    <td>38.6</td>
    <td>19.0</td>
    <td>39.3</td>
  </tr>
  <tr>
    <td><b>BWS<sub>36N</sub></b></td>
    <td><b>50.6</b></td>
    <td><b>66.7</b></td>
    <td><b>64.7</b></td>
    <td><b>48.0</b></td>
    <td><b>53.2</b></td>
    <td><b>22.0</b></td>
    <td><b>48.9</b></td>
  </tr>
  <tr>
    <td>BWS<sub>8N</sub></td>
    <td>49.8</td>
    <td>68.0</td>
    <td>62.8</td>
    <td>48.2</td>
    <td>51.3</td>
    <td>24.8</td>
    <td>43.8</td>
  </tr>
  <tr>
    <td>BWS<sub>72N</sub></td>
    <td>49.5</td>
    <td>68.5</td>
    <td>64.5</td>
    <td>46.2</td>
    <td>51.6</td>
    <td>20.5</td>
    <td>45.6</td>
  </tr>
  <tr>
    <td>BERT</td>
    <td>49.6</td>
    <td>68.2</td>
    <td>57.5</td>
    <td>50.1</td>
    <td>30.2</td>
    <td>56.1</td>
    <td>35.7</td>
  </tr>
</table>

Table 4: Results for Emotion Classification. Llama: Llama-3.1-8B. DeepSeek: DeepSeek-R1-Distilled-Llama-8B. BWS: Automatic BWS with Llama-3.1-8B. The first column F1 is the macro-averaged score, followed by F1-score F1-x, where J - Joy, Sa - Sad, F - Fear, A - Anger, D - Disgust, and Su - Surprise.

three main patterns. First, 17% of cases involved referenced body parts that were present in the experience or expression without acting, as in “tears falling down the face.” Second, 42% of errors stemmed from body parts performing functional or physiological roles within emotional contexts, such as eyes closing when “blackness crept across his eyes,” a natural physiological reaction associated with the character passing away within the context. Finally, 41% of errors involved metaphorical or idiomatic expressions. These included cases where emotional embodiment was implied but not explicitly stated (“I couldn’t believe my eyes,” implying widened eyes in surprise but not explicitly describing this action), expressions symbolically referring to emotional states without literal physical embodiment (“a straw that broke my back”), or purely locational expressions involving body parts without any action (“thoughts racing through my head”). These nuanced distinctions highlight the model’s challenges in accurately interpreting metaphorical, symbolic, and non-embodied references.

4.2 Embodied Emotion Classification

Our experimental results demonstrate notable performance disparities across prompting strategies and model architectures. We use the Llama-3.1-8B as the LLM interpreter for best-worst scaling (BWS). In Table 4, the first section shows the performance of zero-shot large language models, including Llama-3.1-8B and DeepSeek-R1-8B. The second section shows BWS results with different numbers of tuples. And the last section shows a fine-tuned BERT model for comparison (details in Appendix C). We see that BWS exhibits superior performance even with smaller tuple configurations (4N), exceeding Llama-3.1-8B by 10.2 points. Performance improves consistently as the number of tuples increases from 2N to 36N (40.2 to 50.6), suggesting enhanced classification from expanded pairwise comparisons. We hypothesize that this expansion helps the model better weigh emotional significance in text, improving classification accuracy. Notably, the best result comes from the expansion to 36N tuples, with the F1-score beating the supervised method BERT by 1 point. Our experimental results show that as we keep increasing N, the performance will reach a plateau as evidenced by 48N and 72N (see Appendix B).

Error Analysis. To better understand model limitations, we conducted a qualitative error analysis on the misclassified cases. We identified several consistent failure modes, including the model’s difficulty in interpreting emotionally complex inputs or making reliable distinctions between closely related emotional states. In one case, the model predicted Joy, even though the embodied expression “Ryan ducks his head down to his notebook” signaled Fear. This misclassification likely resulted from the influence of nearby positive context, such as “Brendon waves and smiles”, which distracted the model from the emotion-relevant phrase. In another case, the vivid scene “the age-old rock tradition of holding up lighters spread across the 28,000 person deep crowd … lighting up the entire audience … the hair on my arms started to raise” was misclassified as Joy, despite strong physiological cues such as raised arm hair that more closely reflect Surprise. This highlights the model’s tendency to prioritize surface-level celebratory language over conflicting embodied cues.

5 Conclusion

This work advances embodied emotion recognition through three main contributions. First, we created a new dataset called CHEER-Ekman by extending the CHEER dataset with Ekman emotion labels to better understand the connection between bodily expressions and emotional states. Second, we demonstrated that best-worst scaling outperforms both prompted LLMs and fine-tuned BERT, showing the potential for emotion classification without task-specific training. Finally, we found that simplified language and chain-of-thought reasoning significantly improve LLM performance in embodied emotion detection, enabling smaller models to achieve competitive results.
Limitations

While our approach demonstrates a promising advancement in embodied emotion detection using LLMs and the best-worst scaling technique, several limitations warrant consideration.

First, a key observation in our embodied emotion detection task was that simplifying prompts significantly improved model performance. While these findings may suggest enhanced efficiency through linguistic streamlining, they simultaneously introduce concerns about potential overfitting to these simplified phrasings. Simplified prompts may inadvertently prioritize more explicit expressions of embodied emotion over subtler or more figurative language, meaning the models might learn to recognize patterns specific to the prompt structure rather than generalizing to a wide variety of natural language expressions.

Second, the CHEER-Ekman dataset is relatively small, consisting of only 1,350 sentences. This limited size stems from our decision to annotate only the sentences already identified as containing embodied emotions in the original CHEER dataset. This selective annotation was intended to efficiently focus our efforts on instances most relevant to embodied emotion, but it may introduce a bias towards positive examples.

Finally, when addressing emotion classification via best-worst scaling, the scalability and computational overhead of this methodology present challenges: while higher tuple quantities lead to higher accuracy, they also impose significant computational costs. Due to time and computational constraints, we utilize a smaller model, which may lead to suboptimal results for higher-order tuples. Additionally, the limited context window prevents us from effectively implementing a few-shot setting, further impacting performance in scenarios requiring extended context understanding.

Acknowledgments

We thank the CincyNLP group for their constructive comments. We also thank the anonymous ACL reviewers for their valuable suggestions and feedback.

References

Christopher Bagdon, Prathamesh Karmalkar, Harsha Gurulingappa, and Roman Klinger. 2024. “you are an expert annotator”: Automatic best–worst-scaling annotations for emotion intensity modeling. In Proceedings of the 2024 Annual Conference of the North American Chapter of the Association for Computational Linguistics (NAACL 2024).

Alan Cowen and Dacher Keltner. 2017. Self-report captures 27 distinct categories of emotion bridged by continuous gradients. Proceedings of the National Academy of Sciences of the United States of America, 114.

DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihaq Zhu, Shirong Ma, and Peiyi Wang. 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. Preprint, arXiv:2501.12948.

Dorottya Demszky, Dana Movshovitz-Attias, Jeongwoo Ko, Alan Cowen, Gaurav Nemade, and Sujith Ravi. 2020. GoEmotions: A dataset of fine-grained emotions. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (ACL 2020).

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (NAACL 2019).

Paul Ekman. 1992. Are there basic emotions? Psychological Review, 99(3).

Deepanway Ghosal, Navonil Majumder, Alexander Gelbukh, Rada Mihalcea, and Soujanya Poria. 2020. COSMIC: COmmonSense knowledge for eMotion identification in conversations. In Findings of the Association for Computational Linguistics: EMNLP 2020 (Findings of EMNLP 2020).

Aaron Grattafiore, Abhimanyu Dubey, Abhinav Jauhari, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, and Alex Vaughan. 2024. The llama 3 herd of models. Preprint, arXiv:2407.21783.

Svetlana Kiritchenko and Saif Mohammad. 2017. Best-worst scaling more reliable than rating scales: A case study on sentiment intensity annotation. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (ACL 2017).

Ronak Kosti, Jose M Alvarez, Adria Recasens, and Agata Lapedriza. 2019. Context based emotion recognition using emotic dataset. IEEE Transactions on Pattern Analysis and Machine Intelligence, 42(11).

George Lakoff and Mark Johnson. 1999. Philosophy in the flesh: the embodied mind and its challenge of western thought. Basic Books. OCLC: 302020239.
Gyeongeun Lee, Christina Wong, Meghan Guo, and Natalie Parde. 2024. Pouring your heart out: Investigating the role of figurative language in online expressions of empathy. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL 2024).

Qintong Li, Piji Li, Zhaochun Ren, Pengjie Ren, and Zhumin Chen. 2022. Knowledge bridging for empathetic dialogue generation. In Proceedings of the 36th AAAI conference on artificial intelligence (AAAI 2022).

Zhengyan Li, Yicheng Zou, Chong Zhang, Qi Zhang, and Zhongyu Wei. 2021. Learning implicit sentiment in aspect-based sentiment analysis with supervised contrastive pre-training. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP 2021).

Zhiwei Liu, Kailai Yang, Qianqian Xie, Tianlin Zhang, and Sophia Ananiadou. 2024. Emolims: A series of emotional large language models and annotation tools for comprehensive affective analysis. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD 2024).

Saif Mohammad. 2018. Obtaining reliable human ratings of valence, arousal, and dominance for 20,000 english words. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (ACL 2018).

Saif Mohammad, Felipe Bravo-Marquez, Mohammad Salameh, and Svetlana Kiritchenko. 2018. SemEval-2018 task 1: Affect in tweets. In Proceedings of the 12th International Workshop on Semantic Evaluation (SemEval 2018).

Paula M. Niedenthal. 2007. Embodying Emotion. Science, 316(5827):1002–1005.

Soujanya Poria, Devamanyu Hazarika, Navonil Majumder, Gautam Naik, Erik Cambria, and Rada Mihalcea. 2019. MELD: A multimodal multi-party dataset for emotion recognition in conversations. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (ACL 2019).

Sara Rosenthal, Noura Farra, and Preslav Nakov. 2017. SemEval-2017 task 4: Sentiment analysis in Twitter. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval 2017).

Sahand Sabour, Siyang Liu, Zheyuan Zhang, June Liu, Jinfeng Zhou, Alviomna Sunaryo, Tatia Lee, Rada Mihalcea, and Minlie Huang. 2024. EmoBench: Evaluating the emotional intelligence of large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL 2024).

Carlo Strapparava and Rada Mihalcea. 2007. SemEval-2007 task 14: Affective text. In Proceedings of the Fourth International Workshop on Semantic Evaluations (SemEval 2007).

Weixiang Zhao, Zhuojun Li, Shilong Wang, Yang Wang, Yulin Hu, Yanyan Zhao, Chen Wei, and Bing Qin. 2024. Both matter: Enhancing the emotional intelligence of large language models without compromising the general intelligence. In Findings of the Association for Computational Linguistics ACL 2024 (Findings of ACL 2024).

Yuan Zhuang, Tianyu Jiang, and Ellen Riloff. 2024. My heart skipped a beat! recognizing expressions of embodied emotion in natural language. In Proceedings of the 2024 Annual Conference of the North American Chapter of the Association for Computational Linguistics (NAACL 2024).
A  LLM Prompt for Embodied Emotion Detection & Embodied Emotion Classification

Table 5-7 presents the complete prompt templates employed in our experimental methodology. Table 5 details the prompts used for zero-shot embodied emotion detection experiments. The Base prompt closely replicates the methodology of Zhuang et al. (2024), with the sole modification being the use of “True” and “False” as decision tokens rather than “Yes” and “No”. The Simple prompt maintains the fundamental logic while employing more straightforward language and structure to reduce cognitive complexity.

Table 6 presents our chain-of-thought (CoT) prompt variants. The 2-step implementation incorporates the dual criteria from the Base prompt as explicit reasoning steps. The 3-step variant augments this with an initial body movement identification phase, designed to establish concrete context and facilitate more comprehensive reasoning. The 2-step simple variant examines the effectiveness of linguistic simplification within the CoT framework.

Table 7 outlines the large language model (LLM) prompt utilized for emotion classification experiments with Llama-3.1-8B and DeepSeek-R1-8B models.

Throughout our prompt templates, we employ the following placeholder semantics:

• “<sentence|>” denotes the target sentence containing the body part for evaluation.

• “<bdypart|>” indicates the specific body part instance within the sentence.

• “<preceed|>” represents up to three preceding context sentences, when available.

This placeholder convention remains consistent across all experimental tasks presented in this research.

B  Best-Worst Scaling

Experiment Setup. Best-worst scaling (BWS) is a comparative annotation method where annotators select the best and worst items from a given set, typically a 4-tuple. This approach efficiently derives pairwise comparisons, as selecting the best and worst items provides information about most item relationships within the set. A single annotation with best-worst scaling is equivalent to an annotation with 6 pairwise comparisons. Hence, using BWS allows for fewer inferences with the same result. This is particularly beneficial for identifying emotion intensity or emotion classification.

These 4-tuples are assembled from the test data and then are presented to LLMs using the prompt in Table 8. The model then picks one instance that most represents and one instance that least represents some property (in our case, this would be one of the six Ekman’s (1992) emotions).

Once multiple 4-tuples are annotated, a simple counting procedure generates numerical scores, allowing items to be ordered according to their relevance to the given property. The score is calculated using the formula \( \frac{\#Best - \#Worst}{\#Overall} \), where the #Best and #Worst represent the number of times a sentence is ranked Best or Worst, respectively; and #Overall denotes the number of occurrences of the sentence across all 4-tuples. This approach captures a continuous measure, reflecting the relative intensity of a sentence within the given category.

With this method, the LLMs can perform accurate annotations. The results from the annotations will be calculated to get a BWS intensity score across 6 Ekman emotions. The emotion with the highest intensity score will be chosen as the predicted label. This classification process can be represented by the following expression:

\[
\hat{e} = \arg\max_{e_i \in E} S(e_i)
\]

where \( e_i \) corresponds to each emotion, \( E \) is the set of all emotions, and \( S(e_i) \) is the intensity score w.r.t. to such emotion. The resulting predictions are compared with the labels from the CHEER-Ekman dataset to assess performance using several metrics, with a particular focus on the F1-score. The approach of increasing the number of tuples to enhance performance was proposed by Bagdon et al. (2024).

Along with the embodied emotion classification prompt, we also incorporate two additional placeholder semantics:

• “<textid|>” denotes the unique instance ID from the dataset. This id helps the model easily pick out its answer from the sentence tuple when inferencing.

• “<emo|>” denotes the specific emotion required for ranking.
This placeholder convention remains consistent across all experimental tasks presented in this research.

Performance Plateau. Figure 7 illustrates the relationship between tuple count and F1-score performance in our BWS experiments. The results demonstrate significant performance improvements up to \(24N\) tuples, reaching optimal performance at \(36N\). Beyond this threshold, performance degradation is observed, with decreased F1-scores at both \(48N\) and \(72N\) tuple configurations. This pattern suggests a clear upper bound for effective tuple scaling in BWS implementations.

C BERT Experiment Setup

For the embodied emotion classification task discussed in Section 3.3, in addition to BWS, we fine-tuned BERT as a reference benchmark (Devlin et al., 2019). Inputs were constructed by concatenating the preceding context, main text, and referenced body part. We set the maximum sequence length to 512 and use a batch size of 16. The model was trained with the AdamW optimizer, a learning rate of 2e-5, and evaluated over 15 training epochs. Cross-entropy loss was used as the objective, with tokenization performed using truncation and padding. All runs were conducted with 5 seed values starting from 41 to 45 for reproducibility, and final results were averaged across these five runs.

D Data and Results Analysis

Figure 3 - 6 present a more detailed analysis of the CHEER-Ekman dataset and evaluation of models’ performance. Figure 3 illustrates the frequency of the top 10 body parts associated with each emotion, where the size of the bubble reflects the co-occurrence of the body part and emotion pair. Notably, the body parts *face*, *eye*, *head*, *hand*, and *throat* appear consistently across the top 10 body parts across all emotions, with the highest frequency observed in *face*, *eye*, and *head*.

Figure 4 illustrates the classification performance of the 10 most frequent body parts, with their frequency and corresponding accuracy across the three language models: Llama-3.1-8B, DeepSeek-R1-8B, and fine-tuned BERT. As expected, the fine-tuned BERT model consistently outperforms both Llama and DeepSeek for the most frequent body parts. Generally, the fine-tuned BERT model outperforms both zero-shot Llama and Deepseek, achieving an average accuracy increase of 11.7 over Llama and 14.8 over DeepSeek across the top 10 frequent body parts.

Figure 5 and 6 present confusion matrices comparing the models’ predicted emotions against the ground truth emotion for Llama and DeepSeek, respectively. When comparing the two figures, a notable pattern emerges. In Figure 5, strong activations across the diagonal indicate Llama’s attempt to predict emotions accurately without bias towards any one emotion. In Figure 6, however, we observe a prominent concentration in *Joy* in the DeepSeek model.
<table>
  <tr>
    <th>Task</th>
    <th>Prompt Template</th>
  </tr>
  <tr>
    <td>Base</td>
    <td>Please determine if a body part is involved in any embodied emotion. Specifically, a body part is involved in some embodied emotion if both conditions below are satisfied:
1) The physical movement or physiological arousal involving the body part is evoked by emotion.
2) The physical movement, if there is any, has no other purpose than emotion expression.
Answer "True" if the body part is involved in any embodied emotion, and "False" otherwise.

Preceding Context: <preceed|>
Sentence: <sentence|>
Body part: <bdypart|>
Answer:</td>
  </tr>
  <tr>
    <td>Simple</td>
    <td>Decide if a body part is used purely to express emotion. Ask:
- Did emotion cause the body part’s movement/response?
- Was the movement ONLY for expressing emotion (no other reason)?
If both are true, say "True." Else, say "False."

Preceding Context: <preceed|>
Sentence: <sentence|>
Body part: <bdypart|>
Answer:</td>
  </tr>
</table>

Table 5: Zero-shot templates for different tasks.

<table>
  <tr>
    <th>Setting</th>
    <th>Prompt Template</th>
  </tr>
  <tr>
    <td>2-Step</td>
    <td>Please determine if a body part is involved in any embodied emotion.

First, answer Condition 1: Is the body part’s movement/arousal caused by emotion?
Then, answer Condition 2: Does the movement lack non-emotional purposes?

If both of those conditions are true, answer "True." Otherwise, answer "False." Please reason step-by-step for your answer.
Here is the question:

Preceding Context: <preceed|>
Sentence: <sentence|>
Body part: <bdypart|></td>
  </tr>
  <tr>
    <td>3-Step</td>
    <td>Please determine if a body part is involved in any embodied emotion. Specifically, a body part is involved in some embodied emotion if both conditions below are satisfied: Before answering, reasoning step-by-step

1. Identify the body part mentioned.
2. Check if emotion directly caused its movement/arousal.
3. Verify if the movement has no functional purpose.

Only if all of the above are true, answer "True." Otherwise, answer "False."
Here is the question:

Preceding Context: <preceed|>
Sentence: <sentence|>
Body part: <bdypart|></td>
  </tr>
  <tr>
    <td>2-Step Simple</td>
    <td>Decide if a body part is used purely to express emotion. Ask:
- Did emotion cause the body part’s movement/response?
- Was the movement ONLY for expressing emotion (no other reason)?
If both are true, say "True." Else, say "False." Before answering, give your reasoning step-by-step.

Preceding Context: <preceed|>
Sentence: <sentence|>
Body part: <bdypart|></td>
  </tr>
</table>

Table 6: Chain of Thought (CoT) prompt templates for different settings.
Prompt Template

Classify the emotion expressed by the body part in a sentence into one of six categories: "Joy", "Sadness", "Anger", "Fear", "Surprise", or "Disgust".

Preceding Context: <preceed>
Sentence: <sentence>
Body part: <bdypart>
Answer:

Table 7: Emotion classification prompt template.

Prompt Template

You are an expert annotator specializing in emotion recognition. Rank the following examples based on how much <emo> the specified body part exudes in the text.

Instructions:
- Use only the Preceding Text for context.
- Identify which example conveys the MOST <emo> and which conveys the LEAST <emo> based on the body part mentioned.
- Do not repeat the text. Only provide the Example numbers in the specified format.

Example: <textid>
Preceding Context: <preceed>
Sentence: <sentence>
Body part: <bdypart>

Example: <textid>
Preceding Context: <preceed>
Sentence: <sentence>
Body part: <bdypart>

Example: <textid>
Preceding Context: <preceed>
Sentence: <sentence>
Body part: <bdypart>

Example: <textid>
Preceding Context: <preceed>
Sentence: <sentence>
Body part: <bdypart>

Format your response as:
Most <emo> Example:
Least <emo> Example:

Table 8: BWS-Emotion classification prompt template.
Figure 3: Frequency of top 10 body parts for each emotion.
Figure 4: Embodied emotion classification of top 10 frequent body parts with frequency (left) and accuracy (right) comparing zeroshot with Llama, DeepSeek and finetuned BERT.

<table>
  <tr>
    <th>Actual</th>
    <th>Anger</th>
    <th>Disgust</th>
    <th>Fear</th>
    <th>Joy</th>
    <th>Sadness</th>
    <th>Surprise</th>
  </tr>
  <tr>
    <td>Anger</td>
    <td>28</td>
    <td>6</td>
    <td>7</td>
    <td>3</td>
    <td>12</td>
    <td>7</td>
  </tr>
  <tr>
    <td>Disgust</td>
    <td>19</td>
    <td>16</td>
    <td>8</td>
    <td>11</td>
    <td>13</td>
    <td>10</td>
  </tr>
  <tr>
    <td>Fear</td>
    <td>22</td>
    <td>19</td>
    <td>31</td>
    <td>20</td>
    <td>26</td>
    <td>15</td>
  </tr>
  <tr>
    <td>Joy</td>
    <td>9</td>
    <td>11</td>
    <td>7</td>
    <td>49</td>
    <td>23</td>
    <td>19</td>
  </tr>
  <tr>
    <td>Sadness</td>
    <td>11</td>
    <td>11</td>
    <td>8</td>
    <td>9</td>
    <td>54</td>
    <td>13</td>
  </tr>
  <tr>
    <td>Surprise</td>
    <td>8</td>
    <td>11</td>
    <td>13</td>
    <td>18</td>
    <td>8</td>
    <td>23</td>
  </tr>
</table>

Figure 5: Confusion matrix of Llama predictions.
<table>
  <tr>
    <th rowspan="2">Actual</th>
    <th colspan="6">Prediction</th>
  </tr>
  <tr>
    <th>Anger</th>
    <th>Disgust</th>
    <th>Fear</th>
    <th>Joy</th>
    <th>Sadness</th>
    <th>Surprise</th>
  </tr>
  <tr>
    <td>Anger</td>
    <td>20</td>
    <td>4</td>
    <td>3</td>
    <td>26</td>
    <td>9</td>
    <td>1</td>
  </tr>
  <tr>
    <td>Disgust</td>
    <td>13</td>
    <td>9</td>
    <td>2</td>
    <td>43</td>
    <td>9</td>
    <td>1</td>
  </tr>
  <tr>
    <td>Fear</td>
    <td>10</td>
    <td>4</td>
    <td>20</td>
    <td><b>78</b></td>
    <td>12</td>
    <td>9</td>
  </tr>
  <tr>
    <td>Joy</td>
    <td>1</td>
    <td>3</td>
    <td>3</td>
    <td><b>106</b></td>
    <td>2</td>
    <td>3</td>
  </tr>
  <tr>
    <td>Sadness</td>
    <td>5</td>
    <td>4</td>
    <td>4</td>
    <td>65</td>
    <td><b>27</b></td>
    <td>1</td>
  </tr>
  <tr>
    <td>Surprise</td>
    <td>1</td>
    <td>2</td>
    <td>3</td>
    <td>59</td>
    <td>8</td>
    <td>8</td>
  </tr>
</table>

Figure 6: Confusion matrix of DeepSeek predictions.

![Confusion matrix of DeepSeek predictions.](page_324_180_800_400.png)

Figure 7: F1-score trends for BWS when increasing the number of tuples from 2N to 72N (where N is the total number of instances to be classified). BERT’s performance is shown as a reference baseline.

![F1-score trends for BWS and BERT](page_324_1040_800_400.png)

Reviewer paper summaries: This paper introduces ​CHEER-Ekman, an extension of the ​CHEER dataset for embodied emotion classification. The authors employ two annotators to label 1,300 sentences with six emotion categories (joy, sadness, fear, anger, disgust, and surprise). They establish a supervised baseline and demonstrate that by using ​BWS techniques, LLMs can achieve performance close to that of supervised models.
Summary of strengths: 1. The paper proposes a novel dataset, ​CHEER-Ekman, for embodied emotion classification, contributing to a relatively underexplored area of research.

2. The authors leverage ​BWS techniques to enhance LLM performance on this task, bridging the gap between supervised and LLM-based approaches.

3. The paper is well-written.
Summary of weaknesses: 1. The paper requires additional effort to fully understand the task. Specifically, the terms “sentence,” “relevant body,” and “three preceding sentences” need to be clearly defined, as embodied emotion recognition is a relatively new and unfamiliar task for many readers.

2. The task appears to be text-only, and current LLMs might be expected to perform well on it. However, as shown in ​Table 4, LLM performance lags behind supervised models. Therefore, I suggest that the authors could experiment with larger-scale LLMs.

3. If the task can be effectively addressed by LLMs without significant challenges, the necessity of annotating this dataset may be questioned.
Comments, suggestions and typos: Please refer to my comment on weaknesses.
Overall assessment scores: 3