Emergence of Hierarchical Emotion Representations in Large Language Models

Authors: Bo Zhao, Maya Okawa, Eric J Bigelow, Rose Yu, Tomer Ullman, Hidenori Tanaka
Primary area: alignment, fairness, safety, privacy, and societal considerations
Keywords: LLM, emotion

Abstract:
As large language models (LLMs) increasingly power emotionally engaging conversational agents, understanding how they represent, predict, and potentially influence human emotions is critical for their ethical deployment in sensitive contexts. In this work, we reveal emergent hierarchical structures in LLMs' emotion representations, drawing inspiration from psychological theories of emotion. By analyzing probabilistic dependencies between emotional states in LLM outputs, we propose a method for extracting these hierarchies. Our results show that larger models, such as LLaMA 3.1 (405B parameters), develop more intricate emotion hierarchies, resembling human emotional differentiation from broad categories to finer states. Moreover, we find that stronger emotional modeling enhances persuasive abilities in synthetic negotiation tasks, with LLMs that more accurately predict counterparts' emotions achieving better outcomes. Additionally, we explore the effects of persona biases—such as gender and socioeconomic status—on emotion recognition, revealing that LLMs can misclassify emotions when processing minority personas, thus exposing underlying biases. This study contributes to both the scientific understanding of how LLMs represent emotions and the ethical challenges they pose, proposing a novel interdisciplinary perspective on the issue.

Decision: reject
Reviewer scores: 6, 5, 3, 6

Reviews and discussion:
<thread>
creation date:
1730258581423

thread_type:
official_review

<thread_object>
creator: reviewer_6r52

<content>
summary:
This study reveals three findings about emotional intelligence in large language models (LLMs) and its practical implications. First, as LLMs scale up, they develop hierarchical representations of emotions that align with psychological models. Second, the study uncovers how different personas (based on gender, socioeconomic status, etc.) can bias LLMs' emotion recognition, particularly showing systematic biases for minority attributes. Finally, through a synthetic sales negotiation task, the research demonstrates that better emotional prediction capabilities directly correlate with improved persuasion and negotiation outcomes.

strengths:
There are three main strengths of this paper:

1. **High originality**: The exploration of hierarchical emotional representations in LLMs is novel and important. While previous studies have examined emotions in LLMs from various angles, none have investigated their hierarchical nature. Additionally, few works have explored the personalization of emotions, which this paper thoroughly investigates.


2. **High quality and clarity**: The paper presents solid evidence through multiple experiments and maintains clear, fluid expression throughout.


3. **Significant impact**: The findings on emotional hierarchy and personalized emotional biases provide valuable insights for future research and have important implications for LLMs' emotional reasoning and recognition.

weaknesses:
There are three main weaknesses of this paper:

1. **Limited data for emotion tree construction**: The study utilized 5,000 prompts to test 135 emotion types, resulting in an average of only 37.03 prompts per emotion type. This relatively small sample size per emotion suggests the need for expanded data collection to construct a more detailed and robust emotion tree.

2. **Dataset limitations**: The study exclusively relies on GPT-4 generated datasets. It would benefit from incorporating data from real-world scenarios (such as EDOS[1], EmpatheticDialogues[2], and GoEmotions[3]) for experimental validation.

3. **Format error**: (3.1) Citation formats require standardization (inconsistencies noted in lines 34, 48, 249, and 250); (3.2) Possible typo: "eutral" appears on line 362 (should this be "neutral"?)

[1] A taxonomy of empathetic response intents in human social conversations

[2] Towards empathetic opendomain conversation models: A new benchmark and dataset.

[3] Goemotions: A dataset of fine-grained emotions.

questions:
1. **Methodology Consideration**: Did the research employ various prompt types when constructing the emotion tree? Given that LLMs are typically sensitive to prompting, different prompt structures might elicit varying responses. This could potentially affect the emotional relationships identified in the study - for example, the strong connection observed between fear and shock in Llama3.1_8b might be weakened or altered with different prompt formulations. Therefore, I suggest conducting an ablation study on prompt sensitivity to quantify how different prompts affect the emotional hierarchy.

2. **Data Representation Query**: Considering that all the data used in the study was generated by GPT-4o, to what extent might this deviate from authentic human emotional expressions and patterns? I recommend that the authors compare GPT-4o generated data with existing human-annotated emotion datasets to quantify any differences. Additionally, human experts could evaluate the differences between GPT-4o generated data and real-world data.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730692150797

thread_type:
official_review

<thread_object>
creator: reviewer_9uj3

<content>
summary:
This paper studies the emergence of hierarchical emotional representations in large language models and explores their abilities to predict and manipulate emotions. The focus of this study is on models such as LLaMA and GPT, analyzing their emotional hierarchies and the potential biases they may exhibit when identifying emotions of minority character roles. The study also assesses the performance of models in comprehensive negotiation tasks, revealing the correlation between emotional prediction accuracy and negotiation outcomes.

strengths:
•  The analysis and extraction of the emotional hierarchy in LLaMA validate its similarity to human emotional structures, with the complexity of emotional hierarchy positively correlated with model parameter volume.

•  It validates that different roles and scenarios significantly affect LLMs’ emotion recognition abilities, providing guidance for how to avoid such biases in the future.

•  It analyzes the connection between emotional prediction ability and persuasive ability in negotiation tasks, offering practical insights for the application of artificial intelligence in emotionally sensitive environments.

weaknesses:
•  The first two conclusions are quite obvious and lack in-depth exploration of their underlying causes. For example, what is the relationship between the breadth and depth of model emotional stratification and model parameters and pre-training corpora?

•  The discussion on ethics and biases is somewhat coarse in terms of categorization by region, ethnicity, cultural background, and other living conditions.

•  There is a lack of discussion on how to leverage LLMs’ emotional prediction capabilities to optimize downstream dialogue tasks.

questions:
•  The bias experiment could be expanded to more detailed demographic attributes or a broader set of test roles.

•  The analysis of the relationship between emotional prediction and other abilities (such as negotiation, persuasion) could be further expanded, rather than being limited to sales.

•  The wording around ethical issues in the abstract and introduction could be strengthened by providing specific examples of potential real-world impacts.

•  The presentation of Fig 6 needs to be optimized, with biases of different roles not being prominent enough.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730703832534

thread_type:
official_review

<thread_object>
creator: reviewer_qtjt

<content>
summary:
This paper explores the development of hierarchical emotion representations in large language models (LLMs), particularly focusing on models like LLaMA 3.1 with up to 405B parameters. The authors propose methods to extract emotion hierarchies from LLM outputs by analyzing probabilistic dependencies between emotional states. They claim that larger models exhibit more intricate emotional hierarchies resembling psychological theories of emotion. Additionally, the paper examines the impact of persona biases (e.g., gender, socioeconomic status) on emotion recognition and explores the relationship between emotional modeling and persuasive abilities in synthetic negotiation tasks.

strengths:
1. Innovative Approach: The paper introduces a novel and interesting methodology for extracting hierarchical structures of emotions from LLMs, bridging computational models with psychological frameworks.

2. Relevance and Timely: The topic is timely, addressing the intersection of AI, emotion modeling, and ethics.

weaknesses:
1. Emotion Extraction Technique Concern: The method for extracting hierarchical structures based on next-word probabilities lacks rigorous justification. There is no comparison with alternative methods or validation.

2. Threshold Selection: The paper sets a threshold (0 < t < 1) for determining parent-child relationships but does not explain how this threshold is chosen or its impact on the results.

3. Quantitative Metrics: Although the visual representations of emotion hierarchies are compelling, incorporating additional quantitative metrics or comparisons with human-annotated emotion hierarchies could provide stronger validation of the proposed method.

4. The font in Figure 2 is too small to see easily.

questions:
1.  Can you provide a more detailed justification for using next-word probabilities to extract hierarchical emotion structures? 

2. How did you determine the appropriate threshold value (0 < t < 1) for establishing parent-child relationships between emotions? Was this threshold empirically validated?

3. Besides visual representations, can you use some quantitative metrics to validate the integrity and accuracy of the extracted hierarchical emotion structures?

4. Besides emotion, I guess your method can visualize the structure of other entities. Can you extend this part more to enlarge the generalization of your method?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730724683808

thread_type:
official_review

<thread_object>
creator: reviewer_lh9i

<content>
summary:
This study reveals key advancements in how LLMs perceive, predict, and influence human emotions. As model size increases, LLMs develop hierarchical emotional representations consistent with psychological models. The research highlights that personas can bias emotion recognition, underscoring the risk of stereotype reinforcement. Additionally, the study demonstrates that LLMs with refined emotional understanding perform better in persuasive tasks, raising ethical concerns about potential manipulation of human behavior. These insights call for robust ethical guidelines and strategies to mitigate risks of emotional manipulation.

strengths:
1. The writing in this paper is clear, and the figures are intuitive, making the author's ideas easy to understand.

2. The paper astutely identifies that LLMs' potential to comprehend emotions could enhance their capacity to manipulate emotions, which provides critical ethical considerations for the further development of LLMs.

3. The proposed hierarchical emotion extraction method appears simple and effective, offering a powerful tool for further analysis.

weaknesses:
1. After reading the introduction, I expected Chapter 3 to discuss the model's ability and limitations in **perceiving** emotions, especially focusing on the circumstances under which the model fails. However, the paper mainly discusses how larger models outperform smaller ones in understanding emotions, which is rather obvious and does not provide sufficient novel insight.

2. Chapter 4 employs synthetic data for testing but lacks sufficient quality validation. Including human and LLM prediction accuracy in a figure, such as Figure 6, would be beneficial, even if only for a subset.

3. The contributions of the paper are somewhat scattered, covering three different aspects, but the discussions on these points are inadequate. Given that “influencing human emotions” is highlighted as a major contribution, I expected more extensive coverage on this topic. While I understand that involving human subjects may incur additional costs, drawing conclusions solely from LLM dialogues in isolated scenarios lacks persuasiveness. This section also lacks deeper analysis.

questions:
1. What is the underlying principle behind Chapter 3? Your algorithm extracts more nuanced and hierarchical emotional information, but can you elaborate on what further conclusions can be drawn from this? If I understand correctly, does the model's ability to use more emotion-related vocabulary lead to greater hierarchical richness?

2. Chapter 4 provides quantitative analysis from multiple perspectives, but could you offer specific examples of how different character background settings lead to different model emotion predictions? This would help provide more substantial insights.

3. Could you clarify what new insights your experiments provide to advance previous work in perceiving, predicting, and potentially influencing human emotions? Some aspects have been discussed individually in previous studies.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732339040969

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Global response

comment:
Dear reviewers, 

We sincerely thank all reviewers for their thoughtful and constructive feedback. We are encouraged that the reviewers found our paper has "clear, fluid expression" (R. 6r52), presents an "innovative approach" that bridges "computational models with psychological frameworks" (R. qTjt), and offers "practical insights for the application of artificial intelligence in emotionally sensitive environments" (R. 9Uj3), with "clear and intuitive figures" that make our ideas "easy to understand" (R. Lh9i).
The reviewers raised several important concerns that we have thoroughly addressed in our revision. Below we detail how our updates specifically address each concern.
### Major Concerns and Our Responses

### 1. Motivations, Validations, and Insights from the Emotion Hierarchy Construction Method
*Raised by R. Lh9i, qTjt*

Updates:
* Enhanced motivation through psychological studies of emotion tree structures and added new figure about "emotion wheel" (**new Figure 1**)
* Demonstrated the relationship between the psychiatric condition (ASD) and emotion recognition granularity (**new Figure 12**).
* Conducted methodological comparison with baseline hierarchical clustering (**new Figure 14**)
* Validated robustness across threshold selections (**Figure 15**)
* Validated the accuracy of the emotion structures derived from LLMs by demonstrating alignment with emotion wheel (**new Figure 16 and 17**)
* Added new visualization comparing confusion matrix showcasing the performance of nine personas (**new Figure 18**)
* Added a new experiment applying our algorithm to an additional domain: scent, validating its accuracy (**new Figure 19**).
* Added probability interpretation of hierarchy construction algorithm (Appendix A)
* Included discussion on method generalization (Appendix B)

### 2. More Detailed Demographic Analysis of Emotion Biases
*Raised by R. 9Uj3, qTjt*

Updates: Extended analysis to include:
* Race (White, Black, Hispanic, Asian) - (**new Figure 6b**)
* Religion (Christian, Muslim, Buddhist, Hindu) - (**new Figure 6c**)
* Granular age categories - (**new Figure 6d**)
* Education levels - (**new Figure 6f**)
* Intersectional analysis (income, race, gender) - (**new Figure 6g**)
* Presented detailed visualizations of demographic-specific biases in emotion recognition (**new Figure 7**)
* Demonstrated that LLM’s emotion recognition biases are amplified for intersectional underrepresented groups. (**Figure 8**)

### 3. Experiments with Human Subjects
*Raised by R. Lh9i*

Updates:
* **Conducted user study with 60 human participants** comparing human and model emotion predictions
* Added **human experiments with detailed demographics** (**new Figure 9**)

### 4. Real-world Dataset Integration
*Raised by R. 6r52, Lh9i*

Updates:
* Integrated **human labeled GoEmotions dataset** for real-world validation (Section 4.1, **new Figure 10**)
* **Created interactive JavaScript visualizations of to present full results**
* Provided visualization code in supplementary materials
### 5. Emotion Prediction Applications
*Raised by R. 9Uj3*

Updates:
* Extended analysis to include complaint-handling scenarios and emotion dynamics with detailed personas (Figure 12)

### Interactive data visualization access instructions (JavaScript)
1. Download the provided ZIP file of supplementary materials
2. Extract contents (including index.html and JavaScript files) to a local folder
3. Open index.html in a modern browser (Chrome or Safari recommended)

Thank you once again for your valuable feedback. We hope the substantial updates provided above have fully addressed your concerns. Please feel free to reach out with any further questions—we're eager to continue the discussion!
</content>
</thread_object>

</thread>


Metareview:
metareview:
**Summary:** 

The authors seek to measure the degree to which LLMs can perceive and manipulate human emotions, given their rapid integration into user-facing applications. They draw from cognitive science literature, namely Shaver's Hierarchy of Emotions, and introduce a tree-construction algorithm to recover hierarchical representations of emotions in LLMs. They determine that emotions are represented in LLMs in line with intuitions about human cognition, and LLMs exhibit similar behavior in perceiving emotions to humans on synthetic data. They also run experiments with synthetic interactions that indicate improved LLM emotion prediction capabilities are correlated with proxies for emotional manipulation.

**Strengths:** 

- The paper is well-written and well-grounded in cognitive science and psychology literature.

- The authors contribute a scale analysis (illustrated by Figure 3), which shows how LLM's representations of emotions become more complex with model size. This is an interesting finding, and I'd be curious how it applies to other model families. 

- Their analysis showing LLMs have weaker emotion recognition for historically underrepresented groups could help identify errors and improve equity of deployed applications. It is also noteworthy that they consider intersectional biases and they do compare against human-labeled data (GoEmotions). 

**Weaknesses:**

- Lack of detail about quality control during Prolific recruitment. 

- It seems circular to use GPT4o-generated scenarios to assess LLMs' ability to identify and cluster emotions, since if they are unable to one would imagine the scenarios would inadequately or improperly represent these emotions. It makes more sense to use human-derived data. The authors do conduct a small-scale LLM-human comparison with 60 participants in Figure 9, but I question if LLMs really do outperform humans at emotion recognition due to the nature of the data and scale of the study. Reviewer Lh9i raises an excellent point that it would make more sense to use the human-verified emotions (rather than GPT-4 prompt labels) as the ground-truth, and the authors have attempted to address this in their rebuttal. However, this is significant enough of an issue that I believe it requires a substantial editing of results and re-review before publication. 

I actually quite like this paper, but I find the lack of comprehensive comparison to human-derived data concerning. These comparisons should be included and reviewed before publication, so I am leaning towards rejection. I think with enough time to address the reviewers' concerns, the revised version will be a strong paper.