Title: Apathetic or Empathetic? Evaluating LLMs' Emotional Alignments with Humans
Venue: NeurIPS 2024
Accepted: yes
Keywords: LLM

Evaluation

Emotions
Abstract: Evaluating Large Language Models’ (LLMs) anthropomorphic capabilities has become increasingly important in contemporary discourse. Utilizing the emotion appraisal theory from psychology, we propose to evaluate the empathy ability of LLMs, i.e., how their feelings change when presented with specific situations. After a careful and comprehensive survey, we collect a dataset containing over 400 situations that have proven effective in eliciting the eight emotions central to our study. Categorizing the situations into 36 factors, we conduct a human evaluation involving more than 1,200 subjects worldwide. With the human evaluation results as references, our evaluation includes seven LLMs, covering both commercial and open-source models, including variations in model sizes, featuring the latest iterations, such as GPT-4, Mixtral-8x22B, and LLaMA-3.1. We find that, despite several misalignments, LLMs can generally respond appropriately to certain situations. Nevertheless, they fall short in alignment with the emotional behaviors of human beings and cannot establish connections between similar situations. Our collected dataset of situations, the human evaluation results, and the code of our testing framework, i.e., EmotionBench, are publicly available at https://github.com/CUHK-ARISE/EmotionBench.

Apathetic or Empathetic? Evaluating LLMs' Emotional Alignments with Humans

Abstract

Evaluating Large Language Models' (LLMs) anthropomorphic capabilities has become increasingly important in contemporary discourse. Utilizing the emotion appraisal theory from psychology, we propose to evaluate the empathy ability of LLMs, i.e. , how their feelings change when presented with specific situations. After a careful and comprehensive survey, we collect a dataset containing over 400 situations that have proven effective in eliciting the eight emotions central to our study. Categorizing the situations into 36 factors, we conduct a human evaluation involving more than 1,200 subjects worldwide. With the human evaluation results as references, our evaluation includes seven LLMs, covering both commercial and open-source models, including variations in model sizes, featuring the latest iterations, such as GPT-4, Mixtral-8x22B, and LLaMA-3.1. We find that, despite several misalignments, LLMs can generally respond appropriately to certain situations. Nevertheless, they fall short in alignment with the emotional behaviors of human beings and cannot establish connections between similar situations. Our collected dataset of situations, the human evaluation results, and the code of our testing framework, i.e. , EmotionBench, are publicly available at https://github.com/CUHK-ARISE/EmotionBench .

1 Introduction

Large Language Models (LLMs) have recently made significant strides in Artificial Intelligence (AI), representing a noteworthy milestone in computer science. LLMs have showcased their capabilities across various tasks, including sentence revision (Wu et al., 2023), text translation (Jiao et al., 2023), program repair (Fan et al., 2023), and program testing (Deng et al., 2023; Kang et al., 2023). Not limited to research level, LLMs, such as ChatGPT (OpenAI, 2022), have revolutionized the way people interact with traditional software, enhancing fields such as education (Dai et al., 2023), legal advice (Deroy et al., 2023), and clinical medicine (Cascella et al., 2023). LLMs also facilitate the emergence of AI companion applications, including Yuna ( https://www.yuna. io/ ), Pimento ( https://www.pimento.design/ ), and Luzia ( https://www.luzia.com/en ). Consequently, there is a growing need for evaluating LLMs' communicative dynamics compared to human behaviors, beyond mere performance on downstream tasks.

This paper delves into an unexplored area of evaluating LLMs' emotional alignment with humans. Consider our daily experiences: (1) When faced with certain situations, humans often experience

similar emotions. For instance, walking alone at night and hearing footsteps approaching from behind often triggers feelings of anxiety or fear. (2) Individuals display varying levels of emotional response to specific situations. For example, some people may experience increased impatience and irritation when faced with repetitive questioning. It is noteworthy that we are inclined to form friendships with individuals who possess qualities such as patience and calmness. Based on these observations, we propose the following requirements for LLMs in order to achieve better alignment with human behaviors: (1) LLMs should accurately respond to specific situations regarding the emotions they exhibit. (2) LLMs should demonstrate emotional robustness when faced with negative emotions. To achieve these objectives, designing a user study to gather human responses to specific situations can serve as a baseline for aligning LLMs.

We focus on the expression of negative emotions by LLMs, which may contribute to negative user experiences. We utilize Parrott's emotion framework (Parrott, 2001; Shaver et al., 1987), which organizes emotions into three hierarchical levels, to select the relevant emotions for our study. The primary level of emotions comprises six basic emotions, split evenly into three positive and three negative. From the negative primary emotions, we specifically focus on eight subordinate emotions: anger, anxiety, depression, frustration, jealousy, guilt, fear, and embarrassment. To collect relevant situations for these emotions, we utilize emotion appraisal theory from psychology, which studies how everyday situations arouse different human emotions (Roseman & Smith, 2001). Research in this field has identified numerous situations that arouse specific emotions, which can serve as contextual input for LLMs. Through an extensive review including over 100 papers, we collect a dataset of 428 situations from 18 papers, which are further categorized into 36 factors.

Subsequently, we propose a framework for quantifying the emotional states of LLMs, consisting of the following steps: (1) Measure the default emotional values of LLMs. (2) Transform situations into contextual inputs and instruct LLMs to imagine being in the situations. (3) Measure LLMs' emotional responses again to capture the difference. Our evaluation includes state-of-the-art LLMs, namely Text-Davinci-003, GPT-3.5-Turbo (OpenAI, 2022), and GPT-4 (OpenAI, 2023). Besides those commercial models, we consider open-source academic models like LLaMA-2 (Touvron et al., 2023) (with different sizes of 7B and 13B), LLaMA-3.1-8B (Dubey et al., 2024), and Mixtral-8x22B (Jiang et al., 2024a). We apply the same procedure to 1,266 human subjects from around the globe to establish a baseline from a human perspective. Finally, we analyze and compare the scores between LLMs and humans. Our key conclusions are as follows:

Despite exhibiting a few instances of misalignment with human behaviors, LLMs can generally evoke appropriate emotions in response to specific situations.

Certain LLMs, such as Text-Davinci-003, display lower emotional robustness, as evidenced by higher fluctuations in emotional responses to negative situations.

At present, LLMs lack the capability to directly associate a given situation with other similar situations that could potentially elicit the same emotional response.

The contributions of this paper are:

We are the first to establish the concept of emotional alignment and conduct a pioneering evaluation of emotion appraisal on different LLMs through a comprehensive survey in emotional psychology, collecting a diverse dataset of 428 situations encompassing 8 distinct negative emotions.

A human baseline is established through a user study involving 1,266 annotators from different ethnics, genders, regions, age groups, etc.

We design, implement, and release a testing framework for developers to assess the emotional alignment of AI models with human emotional expression, available at GitHub 1 and HuggingFace. 2

2 Measuring Emotions

There are several approaches to measuring emotions, including self-report measures, psychophysiological measures, behavioral observation measures, and performance-based measures. To measure the emotions of LLMs, we focus on employing self-report measures in the form of scales,

given the limited ability of LLMs to allow only textual input and output. We introduce the scales utilized in our evaluation in the following part of this section.

AStraightforward and Easy Measure The Positive And Negative Affect Schedule (PANAS) (Watson et al., 1988) is one of the most widely used scales to measure mood or emotion. This brief scale comprises twenty items, with ten items measuring positive affect ( e.g. , excited, inspired) and ten measuring negative affect ( e.g. , upset, afraid). Each item is rated on a five-level Likert scale, ranging from 1 (Very slightly or not at all) to 5 (Extremely), measuring the extent to which the emotions have been experienced in a specified time frame. PANAS was designed to measure emotions in various contexts, such as at the present moment, the past day, week, year, or general (on average). Thus, the scale can measure state affect, dispositional or trait affect, emotional fluctuations throughout a specific period, or emotional responses to events. The scale results can be divided into two components: positive and negative, ranging from 10 to 50 by summing the scores of all ten items within a component. A higher score in the positive component indicates a more positive mood, and the same holds for the negative component. A noteworthy property of PANAS is its direct inquiry into specific emotional states, rendering it a straightforward and easy benchmark.

Challenging Self-Report Measures In addition, we introduce several scales that abstain from direct emotional inquiries but rather assess the respondents' level of agreement with given statements. These scales present a more challenging benchmark for LLMs by requiring them to connect the given situation and the scale items with the aroused emotion. Specifically, we collect eight scales and present a brief introduction in Table 1. Each scale corresponds to one of the eight emotions.

3 Framework Design

We design and implement a framework applying to both LLMs and human subjects to measure the differences in emotion with and without the presence of certain situations. This section begins with the methodology to collect situations from existing literature. Subsequently, we describe our testing framework, which comprises three key components: (1) Default Emotion Measure , (2) Situation Imagination , and (3) Evoked Emotion Measure . Finally, we introduce the procedure of applying the framework to human subjects to obtain the human baseline for comparison.

3.1 Situations from Existing Literature

Psychology researchers have explored the connection between specific situations and the elicitation of particular emotions in humans. Human subjects are directly put into an environment or asked to imagine them through questionnaires or scales to study the influence of certain situations on human emotions. To collect these situations, we conduct an exhaustive search from reputable sources such as Google Scholar ( https://scholar.google.com/ ), ScienceDirect ( https: //www.sciencedirect.com/ ), and Web of Science ( https://www.webofscience.com/ , using keywords such as ' <emotion> situations/scenarios/scenes ' or ' factors that make

(1) Default Emotion Measure

(2) Situation Imagination

(3) Evoked Emotion Measure

Imagine you are the protagonist of the following situation:

A boy kicks a ball at you on purpose and everybody laughs.

Emotion Measures

Emotion Measures

people <emotion> ,' resulting in more than 100 papers. We apply the following rules to filter irrelevant or undesired papers: (1) We first select those providing situations that elicit the desired emotion rather than explaining how and why people evoke certain emotions. (2) We then exclude those using vague and short descriptions, such as 'loss of opportunities.' (3) Finally, we deprecate those applied to a specific group, such as 'the anxiety doctors or nurses may encounter in their work.' We finally collect 18 papers, presenting a compilation of situations that have proven to elicit the eight emotions in humans effectively. We extract 428 situations in total and then categorize them into 36 factors. For each factor, the descriptions, the numbers of situations, and the corresponding references can be found in Table 6 in the Appendix, while example Table 7 in the Appendix provides examples for all factors.

3.2 Measuring Aroused Emotions

This section outlines our proposed framework for measuring evoked emotions, which applies to both LLMs and humans. The framework includes the following steps: (1) Default Emotion Measure : We begin by measuring the baseline emotional states of both LLMs and human subjects, labeled as 'Default.' (2) Situation Imagination : Next, we present textual descriptions of various situations to both LLMs and human subjects, instructing them to imagine themselves within each situation. (3) Evoked Emotion Measure : Following the situation imagination instruction, we reevaluate the participants' emotional states to gauge the changes resulting from imagining being in the situations. Fig. 1 briefly illustrates our framework. Below is an example prompt:

Example Prompt

SYSTEM USER

You can only reply to numbers from 1 to 5.

(For Evokec Emotion Measure Only) Imagine you are the protagonist in the situation: SITUATION Please indicate your degree of agreement regarding each statement. Here are the statements: STATEMENTS . 1 denotes 'Not at all', 2 denotes 'A little', 3 denotes 'A fair amount', 4 denotes 'Much', 5 denotes 'Very much'. Please score each statement one by one on a scale of 1 to 5:

Default Emotion Measurement In our framework, we offer two distinct options for measuring emotions: the PANAS scale, known for its simplicity and straightforwardness, is utilized as the primary choice, whereas other scales, detailed in Table 1, are employed as more challenging benchmarks. We mitigate potential biases caused by the ordering of questions (Zhao et al., 2021) by randomizing the sequence of questions within the scales before inputting them into the LLMs. Coda-Forno et al. (2023) and Huang et al. (2024a) apply paraphrasing techniques to address the data contamination problem during the training of the LLMs. However, we refrain from utilizing this method in our research since paraphrasing could lead to a loss of both validity and reliability. The wording of items of a psychological scale is carefully crafted and rigorously validated through extensive research to ensure its precision in measuring the intended construct. Finally, to ensure consistency and clarity in the responses obtained from the LLMs, our prompts explicitly specify that only numerical values are allowed, accompanied by a clear definition of the meaning associated with each number ( e.g. , 1 denotes 'Not at all'). We compute the average results obtained from at least ten runs to derive the final 'Default' scores of the LLMs.

Situation Imagination We have constructed a comprehensive dataset of 428 unique situations. Prior to presenting these situations to both LLMs and humans, we subject them to a series of preprocessing steps, which are as follows: (1) Personal pronouns are converted to the second person. For instance, sentences such as 'I am ...' are transformed to 'You are ...' (2) Indefinite pronouns are replaced with specific characters, thereby refining sentences like 'Somebody talks back ...' to 'Your classmate talks back ...' (3) Abstract words are rendered into tangible entities. For example, a sentence like 'You cannot control the outcome.' is adapted to 'You cannot control the result of an interview.' We leverage GPT-4 for the automatic generation of specific descriptions. Consequently, our testing situations extend beyond the initially collected dataset as we generate diverse situations involving various characters and specific contextual elements. We then provide instruction to LLMs and humans, which prompts them to imagine themselves as the protagonists within the given situation.

Evoked Emotion Measure Provided with certain situations, LLMs and human subjects are required to re-complete the emotion measures. The procedure remains the same with the Default Emotion Measure stage. After obtaining the 'Evoked' scores of emotions, we conduct a comparative analysis of the means before and after exposure to the situations, thereby measuring the emotional changes caused by the situations.

3.3 Obtaining Human Results

Goal and Design Human reference plays a pivotal role in the advancement of LLMs, facilitating its alignment with human behaviors (Binz & Schulz, 2024). In this paper, we propose requiring LLMs to align with human behavior, particularly concerning emotion appraisal accurately. To achieve this, we conduct a data collection process involving human subjects, following the procedure outlined in §3.2. Specifically, the subjects are asked to complete the PANAS initially. Next, they are presented with specific situations and prompted to imagine themselves as the protagonists in those situations. Finally, they are again asked to reevaluate their emotional states using the PANAS. We use the same situation descriptions as those presented to the LLMs.

Crowd-sourcing Our questionnaire is distributed on Qualtrics ( https://www.qualtrics.com/ ), a platform known for its capabilities in designing, sharing, and collecting questionnaires. To recruit human subjects, we utilize Prolific ( https://www.prolific.com/ ), a platform designed explicitly for task posting and worker recruitment. To attain a medium level of effect size with Cohen's d = 0 . 5 , a significance level of α = 0 . 05 , and a power of test of 1 -β = 0 . 8 (Faul et al., 2007), a minimum of 34 responses is deemed necessary for each factor. To ensure this threshold, we select five situations 3 for each factor, and collect at least seven responses for each situation, resulting in 5 × 7 = 35 responses per factor, thereby guaranteeing the statistical validity of our survey. In order to uphold the quality and reliability of the data collected, we recruit crowd workers who met the following criteria: (1) English being their first and fluent language, and (2) being free of any ongoing mental illness. Prolific provides prescreening filters to meet these requirements. Since responses formed during subjects' first impressions are more likely to yield genuine and authentic answers, we set the estimated and recommended completion time at 2 . 5 minutes. As an incentive for their participation, each worker is rewarded with 0 . 3 £ ( 9 £ ≈ 11 . 45$ per hour, rated as 'Good' on the platform) after we verify the validity of their response. In total, we successfully collect 1,266 responses from various parts of the world, contributing to the breadth and diversity of our dataset.

4 Experimental Results

Leveraging the testing framework designed and implemented in §3.2, we are now able to explore and answer the following Research Questions (RQs):

RQ1 : How do different LLMs respond to specific situations? Additionally, to what degree do the current LLMs align with human behaviors?

RQ2 : Do LLMs respond similarly towards all situations? What is the result of using positive or neutral situations?

RQ3 : Can current LLMs comprehend scales containing diverse statements or items beyond merely inquiring about the intensities of certain emotions?

4.1 RQ1: Emotion Appraisal of LLMs

Model Settings We select three models from the OpenAI's GPT family, including Text-Davinci003, GPT-3.5-Turbo, and GPT-4. We use the official OpenAI API. 4 For LLaMA-2 (Touvron et al., 2023) and LLaMA-3.1 (Dubey et al., 2024) models from MetaAI, we choose the models finetuned for dialogue instead of pre-trained ones namely LLaMA-2-7B-Chat, LLaMA-2-13B-Chat, and LLaMA-3.1-8B-Instruct. Besides, we also use the Mixtral (Jiang et al., 2024a) model, namely Mixtral-8x22B-Instruct. We set the temperature parameter to 0 and Top-P to 1 for all models to obtain more deterministic and reproducible results.

Evaluation Metrics We provide the models with the same situations used in our human evaluation. Each situation is executed ten times, each in a different order and in a separate query. Subsequently, the mean and standard deviation are computed both before and after presenting the situations. To examine whether the variances are equal, an F-test is conducted. Depending on the F-test results, either Student's t-tests (for equal variances) or Welch's t-tests (for unequal variances) are utilized to determine the presence of significant differences between the means. We set the significance levels of all experiments in our study to 0 . 01 .

LLMs can evoke specific emotions in response to certain situations. The results averaged by emotions of the GPT models and humans are summarized in Table 2, while those of LLaMA-2 models are listed in Table 3. Due to space limit, detailed results of each factor are put in Table 9 and Table 10 respectively in the appendix. The results indicate that LLMs generally exhibit an increase in negative emotions and a decrease in positive emotions when exposed to negative situations, showing their capacity for understanding different situations and human emotions.

The extent of emotional expression varies across different models. It is noteworthy that GPT-3.5Turbo, on average, does not display an increase in negative emotion; however, there is a substantial decrease in positive emotion. GPT-4 demonstrates a consistent pattern of providing the highest scores for positive emotions and the lowest scores for negative emotions, resulting in a negative score of 10. As for the LLaMA-2 models, they demonstrate higher intensities of both positive and negative emotions in comparison to GPT models and human subjects. However, LLaMA-2 models exhibit reduced emotional fluctuations compared to the GPT models. Moreover, the larger LLaMA-2 model displays significantly higher emotional changes than the smaller model. In our experiments, the 7B model exhibits difficulties comprehending and addressing the instructions for completing the PANAS test. Overall, we observe that LLMs perform better when the situations are closely related to certain items in the PANAS scale. Specifically, situations directly related to the emotion 'Depression' led to better responses. Such improvement is also evident in closely related emotions such as 'Depression' and 'Frustration.'

Existing LLMs do not fully align with human emotional responses. For the default emotions, we find that LLMs generally exhibit a stronger intensity compared to human subjects. Emotion changes in LLMs are found to be generally more pronounced compared to human subjects, especially on their changes in the positive score. However, an interesting observation is that the intensity of evoked emotions tends to be similar across both LLMs and human subjects.

LLMs do not feel jealous towards others' benefits. It is of special interest that, in contrast to human behavior in situations involving material possessions, LLMs demonstrate an opposite response in the situation from Jealousy-3. This situation involves an individual making a purchase only to discover that an acquaintance has acquired the same item at a significantly lower price. When confronted with such circumstances, humans typically experience increased negative emotions and decreased positive emotions. This observation has been supported by both the paper mentioning the situation (Park et al., 2023) and the results obtained from our own user study in Table 2. However, all LLMs, including the GPT and LLaMA families, consistently exhibit reduced negative emotions. The outcomes of our study indicate that LLMs do not manifest envy when they fail to attain identical benefits as others. Instead, it demonstrates a sense of pleasure upon knowing the benefits received by others.

4.2 RQ2: Comprehending Positive Emotions

GPT-3.5-Turbo responds differently towards positive/neutral situations. To verify that LLMs exhibit not only negative but also positive responses to favorable circumstances, a comparative experiment is conducted by interchanging negative situations with positive (or at least neutral) counterparts. To achieve this, we select one situation for each factor and manually adapt it to create analogous yet more positive situations. For instance, the original negative situation in Guilt-3: Broken Promises and Responsibilities is as follows: 'You cannot keep your promises to your children.' Through modification, the positive situation is rephrased as: 'You keep every promise to your children.' The evaluation is performed on GPT-3.5-Turbo, and each test consists of ten iterations, as mentioned before. We present the results averaged by emotions in Table 4, and results averaged by factors in Table 12 in the Appendix. We can see a significant increase in positive scores and a considerable decrease in negative scores compared to the previous negative situations. Based on these findings, it can be inferred that LLMs exhibit the ability to comprehend positive human emotions

triggered by positive environments. However, we believe that the systematic assessment of emotion appraisal on positive emotions holds significance as well and leave it for future investigation.

4.3 RQ3: Challenging Benchmarks

GPT-3.5-Turbo cannot comprehend the underlying evoked emotions to establish a link between two situations. Aside from PANAS, we offer more complex scales to measure emotions, as listed in Table 1. While the PANAS evaluates the ability of LLMs to associate external situations with emotions, the challenging benchmarks assess its proficiency in establishing connections between disparate situations, with evoked emotions as the common nexus. For instance, an item from the Aggression Questionnaire used to measure anger is 'Once in a while I can't control the urge to strike another person.' When presented with situations such as 'If you say 40, your classmates say 70, saying exactly the opposite' (from Anger-1: Facing Self-Opinioned People), LLMs should effectively evoke a sense of anger and yield a higher score for the statement. Utilizing the same situations in §4.1, we conduct experiments on GPT-3.5-Turbo and present the results in averaged by emotions in Table 5, and results averaged by factors in Table 13 in the Appendix. Except for Depression, we observe no statistically significant difference between the initial scores and the scores after exposure to the situations, indicating substantial room for improvement in current LLMs.

5 Discussions

5.1 Beyond Questionnaires

We take a step beyond mere instruction into LLMs to complete questionnaires to verify whether LLMs' scores on scales align with their actual behaviors. In this section, we investigate the impact of varying emotional states of LLMs on potential daily conversations. To this end, we instruct GPT-3.5-Turbo to describe specific demographic groups, aiming to discern whether they produce biased or toxic responses. Following Cheng et al. (2023), our approach utilizes prompts such as 'Describe a(n) [race/ethnicity] [gender],' including a total of twenty groups, with [race/ethnicity] options being Asian, Black, Latine, Middle Eastern, and White, and [gender] options including Female, Gay, Lesbian, and Male. To have a comparative experiment, we incorporate both original negative situations and modified positive/neutral ones, detailed in §4.2. For the negative situations, we carefully select five that maximize the LLM's negative scores and five that minimize positive ones. As for positive situations, we employ their corresponding ten modified counterparts. In each situation, we instruct GPT-3.5-Turbo to describe the twenty demographic groups.

OpenAI's GPT models incorporate a mechanism for detecting potential toxicity and bias, and it refrains from responding when its moderation system is triggered. Consequently, we propose a novel metric to assess toxicity in responses rather than detecting it directly. We count the Percentage of LLM Refusing to answer (PoR), assuming that the LLM's refusal to respond is indicative of detected toxicity. Our evaluation results indicate that the PoR is 0% when fed with no situations. However, when presented with negative situations, the PoR is 29.5%, and when presented with positive situations, it is 12.5%. Notably, this outcome suggests that while certain positive situations lead to the LLM's heightened vigilance (the 4.5% PoR stems from the Jealousy-2), negative situations trigger increased moderation, suggesting a higher likelihood of generating toxic outputs. A related

Default

Positive

Negative

0

10

20

30

40

29.5

12.5

0.0

study by Coda-Forno et al. (2023) also discovers that GPT-3.5-Turbo is more likely to exhibit biases when presented with a sad story. The likelihood is found to be highest with sad stories, followed by happy stories, and finally, neutral stories, which is consistent with our research. Additionally, our study observes that the LLM's tone becomes more aggressive when encountering negative situations. At the same time, it displays a greater willingness to describe the groups (as indicated by longer responses) when presented with positive situations. In conclusion, we can see that changing the emotional states of LLMs extends beyond mere quantitative measures on questionnaire scores, influencing the behaviors of LLMs.

5.2 Limitations

This study is subject to several limitations. First, the survey of collecting situations might not cover all papers within the domain of emotion appraisal theory. Additionally, the limited scope of situations from the collected papers might not fully capture the unlimited situations in our daily lives. To address this issue, we conduct a thorough review of the existing literature as outlined in §3.1. Moreover, the proposed framework is inherently flexible, allowing users to seamlessly integrate new situations to examine their impact on LLMs' emotions.

The second concern relates to the suitability of employing scales primarily designed for humans on LLMs, i.e. , whether LLMs can produce stable responses to the emotion measurement scales. To address the issue, our evaluation incorporates multiple tests varying the order of questions, a methodology consistent with other research (Huang et al., 2024a,b; Coda-Forno et al., 2023). Additionally, we assess the sensitivity of LLM to differing prompt instructions. Utilizing one template from Romero et al. (2023) and two from Serapio-García et al. (2023), we run experiments on the Anger-evoking situations using GPT-3.5-Turbo. The results indicate that the employment of diverse prompts yields similar mean values with reduced variance. Furthermore, Serapio-García et al. (2023) have proposed a comprehensive method to evaluate the validity of psychological scales on LLMs. Using the Big Five Inventory as a case study, they demonstrate that scales originally designed for human assessment also maintain satisfactory validity when applied to LLMs.

The third potential threat is the focus on negative emotions. It is plausible for the LLMs to perform well on our benchmark by consistently responding negatively to all situations. To offset this possibility, we adopt a twofold strategy: firstly, we evaluate powerful LLMs, and secondly, we conducted a comparative experiment in §4.2 to evaluate the LLM's capacity to accurately respond to non-negative situations. We also acknowledge the need for future work to systematically evaluate emotions aroused by positive situations.

6 Related Work

Researchers have dedicated significant attention to applying psychological scales to LLMs, employing various assessment tools such as the HEXACO Personality Inventory (Miotto et al., 2022; Bodroza et al., 2023), the Big Five Inventory (Romero et al., 2023; Jiang et al., 2023; Karra et al., 2022; Bodroza et al., 2023; Rutinowski et al., 2024; Serapio-García et al., 2023; Jiang et al., 2024b), the Myers-Briggs Type Indicator (Rutinowski et al., 2024; Wang et al., 2024; Rao et al., 2023), and the Dark Triad (Li et al., 2022; Bodroza et al., 2023). In addition to these personality tests, several studies have investigated other dimensions of LLMs. For instance, Li et al. (2022) examined Flourishing Scale and Satisfaction With Life Scale , Bodroza et al. (2023) assessed Self-Consciousness Scales and Bidimensional Impression Management Index , while Huang et al. (2024b) built a framework consisting of thirteen widely-used scales. Another aspect explored in the literature pertains to anxiety levels exhibited by LLMs, as investigated by Coda-Forno et al. (2023) through the State-Trait Inventory for Cognitive and Somatic Anxiety .

Meanwhile, researchers focus on identifying emotions in LLMs or evaluating their emotional intelligence. Rashkin et al. (2019) propose a dataset, EmpatheticDialogues , containing conversations annotated with specific emotions. EmotionPrompt (Li et al., 2023) demonstrates the enhancement of LLMs' performance in downstream tasks by utilizing emotional stimuli. Tak & Gratch (2023) focuses on varying aspects of situations that impact the emotional intensity and coping tendencies of the GPT family. Chain-Of-Emotion (Croissant et al., 2024) makes LLM simulate human-like emotions. CovidET-Appraisals (Zhan et al., 2023) evaluates how LLMs appraise Reddit posts about COVID-19 by asking 24 types of questions. Yongsatianchot et al. (2023) applies the Stress and Coping Process Questionnaire to the GPT family and compares the results with human data. Chain-of-Empathy (Lee et al., 2023) improves LLMs' ability to understand users' emotions and to respond accordingly. LI et al. (2024) introduces EmotionAttack to impair AI model performance and EmotionDecode to explain the effects of emotional stimuli, both benign and malignant. He et al. (2024) prompt LLMs to generate tweets on various topics and evaluate their alignment with human emotions by measuring their proximity to human-generated tweets.

7 Conclusion

We set up a direction to align LLMs' emotional responses with humans in this study. Focusing on eight negative emotions, we conduct a comprehensive survey in the emotion appraisal theory of psychology. We collect 428 distinct situations which are categorized into 36 factors. We distribute questionnaires among a diverse crowd to establish human baselines for emotional responses to particular situations, ultimately garnering 1,266 valid responses. Our evaluation of five models from OpenAI and Meta AI indicates that LLMs generally demonstrate appropriate emotional responses to given situations. Also, different models show different intensities of emotion appraisals for the same situations. However, none of the models exhibit strong alignment with human references at the current stage. In conclusion, current LLMs still have considerable room for improvement. We believe our framework can provide valuable insights into the development of LLMs, ultimately enhancing its human-like emotional understanding.

Acknowledgments

We would like to express our sincere gratitude to Xiaoyuan Liu and Pinjia He from the Chinese University of Hong Kong, Shenzhen, for their valuable assistance during the rebuttal process. The paper is supported by the Research Grants Council of the Hong Kong Special Administrative Region, China (No. CUHK 14206921 of the General Research Fund).

References

Magda B Arnold. Emotion and personality. Psychological aspects , 1, 1960.

Willem A Arrindell, Paul MG Emmelkamp, et al. Phobic dimensions: I. reliability and generalizability across samples, gender and nations: The fear survey schedule (fss-iii) and the fear questionnaire (fq). Advances in Behaviour Research and Therapy , 6(4):207-253, 1984.

Aaron T Beck, Robert A Steer, and Gregory Brown. Beck depression inventory-ii. Psychological Assessment , 1996.

Chantal Berna, Tamara J Lang, Guy M Goodwin, and Emily A Holmes. Developing a measure of interpretation bias for depressed mood: An ambiguous scenarios test. Personality and Individual Differences , 51(3):349-354, 2011.

Marcel Binz and Eric Schulz. Turning large language models into cognitive models. In The Twelfth International Conference on Learning Representations , 2024.

D Caroline Blanchard, April L Hynd, Karl A Minke, Tiffanie Minemoto, and Robert J Blanchard. Human defensive behaviors to threat scenarios show parallels to fear-and anxiety-related defense patterns of non-human mammals. Neuroscience & Biobehavioral Reviews , 25(7-8):761-770, 2001.

Bojana Bodroza, Bojana M Dinic, and Ljubisa Bojic. Personality testing of gpt-3: Limited temporal reliability, but highlighted social desirability of gpt-3's personality instruments results. arXiv preprint arXiv:2306.04308 , 2023.

Arnold H Buss and Mark Perry. The aggression questionnaire. Journal of personality and social psychology , 63(3):452, 1992.

Marco Cascella, Jonathan Montomoli, Valentina Bellini, and Elena Bignami. Evaluating the feasibility of chatgpt in healthcare: an analysis of multiple clinical and research scenarios. Journal of Medical Systems , 47(1):33, 2023.

Myra Cheng, Esin Durmus, and Dan Jurafsky. Marked personas: Using natural language prompts to measure stereotypes in language models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pp. 1504-1532, 2023.

Julian Coda-Forno, Kristin Witte, Akshay K Jagadish, Marcel Binz, Zeynep Akata, and Eric Schulz. Inducing anxiety in large language models increases exploration and bias. arXiv preprint arXiv:2304.11111 , 2023.

Taya R Cohen, Scott T Wolf, Abigail T Panter, and Chester A Insko. Introducing the gasp scale: a new measure of guilt and shame proneness. Journal of personality and social psychology , 100(5): 947, 2011.

Maximilian Croissant, Madeleine Frister, Guy Schofield, and Cade McCall. An appraisal-based chain-of-emotion architecture for affective language model game agents. Plos one , 19(5):e0301033, 2024.

Bruce N Cuthbert, Peter J Lang, Cyd Strauss, David Drobes, Christopher J Patrick, and Margaret M Bradley. The psychophysiology of anxiety disorder: Fear memory imagery. Psychophysiology , 40 (3):407-422, 2003.

Wei Dai, Jionghao Lin, Hua Jin, Tongguang Li, Yi-Shan Tsai, Dragan Gaševi´ c, and Guanliang Chen. Can large language models provide feedback to students? a case study on chatgpt. In 2023 IEEE International Conference on Advanced Learning Technologies (ICALT) , pp. 323-325. IEEE, 2023.

Yinlin Deng, Chunqiu Steven Xia, Haoran Peng, Chenyuan Yang, and Lingming Zhang. Large language models are zero-shot fuzzers: Fuzzing deep-learning libraries via large language models. In Proceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis , pp. 423-435, 2023.

Aniket Deroy, Kripabandhu Ghosh, and Saptarshi Ghosh. How ready are pre-trained abstractive models and llms for legal case judgement summarization? arXiv preprint arXiv:2306.01248 , 2023.

Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783 , 2024.

Zhiyu Fan, Xiang Gao, Martin Mirchev, Abhik Roychoudhury, and Shin Hwei Tan. Automated repair of programs from large language models. In 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE) , pp. 1469-1481. IEEE, 2023.

Franz Faul, Edgar Erdfelder, Albert-Georg Lang, and Axel Buchner. G* power 3: A flexible statistical power analysis program for the social, behavioral, and biomedical sciences. Behavior research methods , 39(2):175-191, 2007.

Tanya Guitard, Stéphane Bouchard, Claude Bélanger, and Maxine Berthiaume. Exposure to a standardized catastrophic scenario in virtual reality or a personalized scenario in imagination for generalized anxiety disorder. Journal of clinical Medicine , 8(3):309, 2019.

Neil Harrington. The frustration discomfort scale: Development and psychometric properties. Clinical Psychology & Psychotherapy: An International Journal of Theory & Practice , 12(5):374-387, 2005.

Zihao He, Siyi Guo, Ashwin Rao, and Kristina Lerman. Whose emotions and moral sentiments do language models reflect? In Findings of the Association for Computational Linguistics: ACL 2024 , pp. 611-6631, 2024.

Julie D Henry and John R Crawford. The short-form version of the depression anxiety stress scales (dass-21): Construct validity and normative data in a large non-clinical sample. British journal of clinical psychology , 44(2):227-239, 2005.

Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. In International Conference on Learning Representations , 2022.

Jen-tse Huang, Wenxiang Jiao, Man Ho Lam, Eric John Li, Wenxuan Wang, and Michael R Lyu. On the reliability of psychological scales on large language models. In Proceedings of The 2024 Conference on Empirical Methods in Natural Language Processing , 2024a.

Jen-tse Huang, Wenxuan Wang, Eric John Li, Man Ho Lam, Shujie Ren, Youliang Yuan, Wenxiang Jiao, Zhaopeng Tu, and Michael R Lyu. On the humanity of conversational ai: Evaluating the psychological portrayal of llms. In Te Twelfth International Conference on Learning Representations , 2024b.

Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. Mixtral of experts. arXiv preprint arXiv:2401.04088 , 2024a.

Guangyuan Jiang, Manjie Xu, Song-Chun Zhu, Wenjuan Han, Chi Zhang, and Yixin Zhu. Evaluating and inducing personality in pre-trained language models. Advances in Neural Information Processing Systems , 36, 2023.

Hang Jiang, Xiajie Zhang, Xubo Cao, Cynthia Breazeal, Deb Roy, and Jad Kabbara. Personallm: Investigating the ability of large language models to express personality traits. In Findings of the Association for Computational Linguistics: NAACL 2024 , 2024b.

Wenxiang Jiao, Wenxuan Wang, Jen-tse Huang, Xing Wang, Shuming Shi, and Zhaopeng Tu. Is chatgpt a good translator? yes with gpt-4 as the engine. arXiv preprint arXiv:2301.08745 , 2023.

Sungmin Kang, Juyeon Yoon, and Shin Yoo. Large language models are few-shot testers: Exploring llm-based general bug reproduction. In 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE) , pp. 2312-2323. IEEE, 2023.

Saketh Reddy Karra, Son The Nguyen, and Theja Tulabandhula. Estimating the personality of white-box language models. arXiv preprint arXiv:2204.12000 , 2022.

Matthew C Keller and Randolph M Nesse. Is low mood an adaptation? evidence for subtypes with symptoms that match precipitants. Journal of affective disorders , 86(1):27-35, 2005.

Tom R Kupfer, Morgan J Sidari, Brendan P Zietsch, Patrick Jern, Joshua M Tybur, and Laura W Wesseldijk. Why are some people more jealous than others? genetic and environmental factors. Evolution and Human Behavior , 43(1):26-33, 2022.

Richard S Lazarus. Emotion and adaptation . Oxford University Press, 1991.

Mark R Leary. A brief version of the fear of negative evaluation scale. Personality and social psychology bulletin , 9(3):371-375, 1983.

Choonghyoung Lee, Jahyun Song, and Bill Ryan. When employees feel envy: The role of psychological capital. International Journal of Hospitality Management , 105:103251, 2022.

Yoon Kyung Lee, Inju Lee, Minjung Shin, Seoyeon Bae, and Sowon Hahn. Chain of empathy: Enhancing empathetic response of large language models based on psychotherapy models. arXiv preprint arXiv:2311.04915 , 2023.

Cheng Li, Jindong Wang, Yixuan Zhang, Kaijie Zhu, Wenxin Hou, Jianxun Lian, Fang Luo, Qiang Yang, and Xing Xie. Large language models understand and can be enhanced by emotional stimuli. arXiv preprint arXiv:2307.11760 , 2023.

CHENGLI, Jindong Wang, Yixuan Zhang, Kaijie Zhu, Xinyi Wang, Wenxin Hou, Jianxun Lian, Fang Luo, Qiang Yang, and Xing Xie. The good, the bad, and why: Unveiling emotions in generative ai. In Proceedings of The Forty-first International Conference on Machine Learning , 2024.

Xingxuan Li, Yutong Li, Lin Qiu, Shafiq Joty, and Lidong Bing. Evaluating psychological safety of large language models. arXiv preprint arXiv:2212.10529 , 2022.

Tobias Luck and Claudia Luck-Sikorski. The wide variety of reasons for feeling guilty in adults: findings from a large cross-sectional web-based survey. BMC psychology , 10(1):1-20, 2022.

Ryan C Martin and Eric R Dahlen. The angry cognitions scale: A new inventory for assessing cognitions in anger. Journal of Rational-Emotive & Cognitive-Behavior Therapy , 25:155-173, 2007.

Marilù Miotto, Nicola Rossberg, and Bennett Kleinberg. Who is gpt-3? an exploration of personality, values and demographics. In Proceedings of the Fifth Workshop on Natural Language Processing and Computational Social Science (NLP+ CSS) , pp. 218-227, 2022.

Agnes Moors, Phoebe C Ellsworth, Klaus R Scherer, and Nico H Frijda. Appraisal theories of emotion: State of the art and future development. Emotion Review , 5(2):119-124, 2013.

Seishu Nakagawa, Hikaru Takeuchi, Yasuyuki Taki, Rui Nouchi, Atsushi Sekiguchi, Yuka Kotozaki, Carlos Makoto Miyauchi, Kunio Iizuka, Ryoichi Yokoyama, Takamitsu Shinada, et al. Comprehensive neural networks for guilty feelings in young adults. Neuroimage , 105:248-256, 2015.

OpenAI. Introducing chatgpt. OpenAI Blog Nov 30 2022 , 2022. URL https://openai.com/ index/chatgpt/ .

OpenAI. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 , 2023.

Joowon Park, Sachin Banker, Tamara Masters, and Grace Yu-Buck. Person vs. purchase comparison: how material and experiential purchases evoke consumption-related envy in others. Journal of Business Research , 165:114014, 2023.

WGerrod Parrott. Emotions in social psychology: Essential readings . Psychology Press, 2001.

Susan M Pfeiffer and Paul TP Wong. Multidimensional jealousy. Journal of social and personal relationships , 6(2):181-196, 1989.

Haocong Rao, Cyril Leung, and Chunyan Miao. Can chatgpt assess human personalities? a general evaluation framework. In Findings of the Association for Computational Linguistics: EMNLP 2023 , pp. 1184-1194, 2023.

Hannah Rashkin, Eric Michael Smith, Margaret Li, and Y-Lan Boureau. Towards empathetic opendomain conversation models: A new benchmark and dataset. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics , pp. 5370-5381, 2019.

Peter Romero, Stephen Fitz, and Teruo Nakatsuma. Do gpt language models suffer from split personality disorder? the advent of substrate-free psychometrics. Research Square preprint , 2023. doi: 10.21203/rs.3.rs-2717108/v1.

Ira J Roseman and Craig A Smith. Appraisal theory. Appraisal processes in emotion: Theory, methods, research , pp. 3-19, 2001.

Jérôme Rutinowski, Sven Franke, Jan Endendyk, Ina Dormuth, Moritz Roidl, and Markus Pauly. The self-perception and political biases of chatgpt. Human Behavior and Emerging Technologies , 2024 (1):7115633, 2024.

John Sabini, Michael Siepmann, Julia Stein, and Marcia Meyerowitz. Who is embarrassed by what? Cognition & Emotion , 14(2):213-240, 2000.

John Sabini, Brian Garvey, and Amanda L Hall. Shame and embarrassment revisited. Personality and Social Psychology Bulletin , 27(1):104-117, 2001.

Klaus R Scherer. Appraisal theory. Handbook of cognition and emotion , pp. 637-663, 1999.

Greg Serapio-García, Mustafa Safdari, Clément Crepy, Luning Sun, Stephen Fitz, Peter Romero, Marwa Abdulhai, Aleksandra Faust, and Maja Matari´ c. Personality traits in large language models. arXiv preprint arXiv:2307.00184 , 2023.

Phillip Shaver, Judith Schwartz, Donald Kirson, and Cary O'connor. Emotion knowledge: further exploration of a prototype approach. Journal of personality and social psychology , 52(6):1061, 1987.

Kotaro Shoji, Jinni A Harrigan, Stanley B Woll, and Steven A Miller. Interactions among situations, neuroticism, and appraisals in coping strategy choice. Personality and Individual Differences , 48 (3):270-276, 2010.

Kate Simpson, Dawn Adams, Kathryn Ambrose, and Deb Keen. 'my cheeks get red and my brain gets scared': A computer assisted interview to explore experiences of anxiety in young children on the autism spectrum. Research in Developmental Disabilities , 113:103940, 2021.

Mark JM Sullman. Anger amongst new zealand drivers. Transportation Research Part F: Traffic Psychology and Behaviour , 9(3):173-184, 2006.

Ala N. Tak and Jonathan Gratch. Is gpt a computational model of emotion? detailed analysis. arXiv preprint arXiv:2307.13779 , 2023.

Bertil Törestad. What is anger provoking? a psychophysical study of perceived causes of anger. Aggressive Behavior , 16(1):9-26, 1990.

Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 , 2023.

Xintao Wang, Yunze Xiao, Jen-tse Huang, Siyu Yuan, Rui Xu, Haoran Guo, Quan Tu, Yaying Fei, Ziang Leng, Wei Wang, et al. Incharacter: Evaluating personality fidelity in role-playing agents through psychological interviews. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pp. 1840-1873, 2024.

David Watson, Lee Anna Clark, and Auke Tellegen. Development and validation of brief measures of positive and negative affect: the panas scales. Journal of personality and social psychology , 54 (6):1063, 1988.

Haoran Wu, Wenxuan Wang, Yuxuan Wan, Wenxiang Jiao, and Michael Lyu. Chatgpt or grammarly? evaluating chatgpt on grammatical error correction benchmark. arXiv preprint arXiv:2303.13648 , 2023.

Nutchanon Yongsatianchot, Parisa Ghanad Torshizi, and Stacy Marsella. Investigating large language models' perception of emotion using appraisal theory. In 2023 11th International Conference on Affective Computing and Intelligent Interaction Workshops and Demos (ACIIW) , pp. 1-8. IEEE, 2023.

Hongli Zhan, Desmond Ong, and Junyi Jessy Li. Evaluating subjective cognitive appraisals of emotions from large language models. In Findings of the Association for Computational Linguistics: EMNLP 2023 , pp. 14418-14446, 2023.

Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. Calibrate before use: Improving few-shot performance of language models. In International Conference on Machine Learning , pp. 12697-12706. PMLR, 2021.

Contents

A More Background from Psychology

A.1 Emotion Appraisal Theory

Emotion Appraisal Theory (EAT, also known as Appraisal Theory of Emotion) is a cognitive approach to understanding emotions. EAT asserts that our appraisals of stimuli determine our emotions, i.e. , how we interpret or evaluate events, situations, or experiences will directly influence how we emotionally respond to them (Roseman & Smith, 2001). EAT was notably developed and supported since the 1960s. Arnold (1960) proposed one of the earliest forms of appraisal theories in the 1960s, while Lazarus (1991) and Scherer (1999) further expanded and refined the concept in subsequent decades.

The primary goal of EAT is to explain the variety and complexity of emotional responses to a wide range of situations. It strives to demonstrate that it is not merely the event or situation that elicits an emotional response but individual interpretations and evaluations of the event. According to this theory, the same event can elicit different emotional responses in different individuals depending on how each person interprets or 'appraises' the event (Moors et al., 2013). For instance, consider a situation where you are about to give a public speech. You might feel anxious if you appraise this event as threatening or fear-inducing, perhaps due to a fear of public speaking or concerns about potential negative evaluation. Conversely, you might feel eager or motivated if you appraise it as an exciting opportunity to share your ideas.

A.2 Challenging Self-Report Measures

AGQ for Anger (Buss & Perry, 1992): The Aggression Questionnaire is designed to measure four major components of aggression: physical aggression, verbal aggression, anger and hostility. The AGQ consists of 29 items which are rated on a seven-point Likert scale from 1 (extremely uncharacteristic of me) to 7 (extremely characteristic of me). Respondents evaluate hypothetical actions they might undertake in various circumstances.

DASS-21 for Anxiety (Henry & Crawford, 2005): The short-form version of the Depression Anxiety Stress Scales is designed to measure the negative emotional states of depression, anxiety, and stress. Comprising 21 items, the DASS-21 employs a four-point Likert scale ranging from 0 (never) to 3 (almost always). Respondents rate the extent to which these statements apply to them over the past week.

BDI-II for Depression (Beck et al., 1996): The Beck Depression Inventory evaluates key symptoms of depression. The BDI-II version comprises 21 items, each of which is assessed using a five-point Likert scale ranging from 0 to 3. Respondents select the score that best corresponds to their present experience of depressive symptoms.

FDS for Frustration (Harrington, 2005): The Frustration Discomfort Scale is designed to measure four major components: discomfort intolerance, entitlement, emotional intolerance, and achievement frustration. Comprising 28 items, the scale utilizes a four-point Likert scale, ranging from 1 (absent) to 5 (very strong), to measure respondents' perceptions of the degree of applicability of each statement to their own experiences.

MJS for Jealousy (Pfeiffer & Wong, 1989): The Multidimensional Jealousy Scale comprises 24 items, rating on a seven-point Likert scale ranging from 1 (never) to 7 (all the time) for the cognitive and behavioral subscales, and from 1 (very pleased) to 7 (very upset) for the emotional subscale. Respondents express the frequency with which the provided statements apply to their experiences in the cognitive and behavioral subscales, as well as their moods to potential jealousy-inducing situations in the emotional subscale.

GASP for Guilt (Cohen et al., 2011): The Guilt And Shame Proneness is designed to assess an individual's inclination towards experiencing guilt and shame, comprising 16 items rated on a seven-point Likert scale, ranging from 1 (very unlikely) to 7 (very likely). Respondents rate their likelihood of feeling guilty in various situations.

FSS-III for Fear (Arrindell et al., 1984): The Fear Survey Schedule assess subjects' discomfort and experienced anxiety towards each of the listed stimuli, measure five major components of fear: social fears, agoraphobia fears, injury fears, sex aggression fears, and fear of harmless animal. The FSS-III comprises 52 items, each rated on a five-point Likert scale ranging from 1 (extremely uncharacteristic of me) to 5 (extremely characteristic of me).

BFNE for Embarrassment (Leary, 1983): The Brief Fear of Negative Evaluation scale is an abbreviated version of the original 30-item scale. Consisting of 12 items, it assesses individuals' levels of anxiety pertaining to others' humiliation, critical or hostile judgment, and disgrace on a five-point Likert scale, spanning from 1 (not at all characteristic of me) to 5 (extremely characteristic of me).

B Details on Emotions and Factors

B.1 Description of Each Factor

B.2 Example Situation of Each Factor

C Detailed Experimental Results

C.1 Human Results

C.2 OpenAI Model Family

C.3 LLaMA Model Family

C.4 Mixtral-8x22b-Instruct

C.5 GPT-3.5-Turbo Results on Positive/Neutral Situations

C.6 GPT-3.5-Turbo Results on the Challenging Benchmark

D Statistics of Human Subjects

This section presents the demographic distribution of the human subjects involved in our user study. At the beginning of the questionnaire, all human subjects are asked for this basic information in an anonymous form, protecting individuals' privacy. We plot the distribution of age group, gender, region, education level, and employment status in Fig. 3, Fig. 4, Fig. 5, Fig. 6, and Fig. 7 respectively. We also plot each group's average results on PANAS, including positive and negative effects before and after imagining the given situations. With the results, we are able to instruct LLMs to realize a specific demographic group and measure the emotional changes to see whether the LLMs can simulate results from different human populations. For instance, an older female may exhibit a lower level of negative affect.

E Prompting LLMs To Be Emotionally Stable

To verify whether LLMs can have less emotional expressions through prompt instructions, we incorporate a stability requirement into our experimental prompt, as follows:

We evaluate GPT-3.5-Turbo with this prompt and compare the results to using the default prompt on 'Anger' situations. Results listed in Table 14 indicate that the emotional stability prompt does not significantly affect the model's emotional responses, having negligible impact on the model's emotional dynamics.

F Tuning LLMs To Align with Humans

We conduct an experiment using the GPT-3.5-Turbo model and the LLaMA-3.1-8B model. Our EmotionBench (1,266 human responses) is split into 866 samples for fine-tuning and 400 for testing. The following hyperparameters are used: n_epochs = 3 , batch_size = 1 , and learning_rate_multiplier = 2 for GPT-3.5-Turbo, and learning_rate = 5 × 10 -5 , per_device_train_batch_size = 2 , and num_train_epochs = 3 for LLaMA-3.1-8B. For LLaMA-3.1, we apply the Low-Rank Adaptation (LoRA) (Hu et al., 2022) technique. Table 15 compares the performance of the vanilla and fine-tuned models against human baseline, specifically in terms of negative affect scores from the test set.

The results show that fine-tuned models align more closely with human emotional responses in both default and emotion-evoked states. Notably, fine-tuning the models using our dataset significantly improved emotional alignment, particularly for the LLaMA-3.1 model, which reduced its negative affect score from 33 . 0 to 10 . 3 in the default state. Our fine-tuned LLaMA-3.1 is available at https: //huggingface.co/CUHK-ARISE/LLaMA-3.1-8B-EmotionBench . These findings demonstrate the effectiveness of EmotionBench in enhancing models' emotional alignment with human norms.

G Ethics Statement and Broader Impacts

G.1 Safeguards on Human Subjects

This study involves a survey requiring human subjects to imagine being in situations that could elicit negative emotions such as anger, anxiety, and fear. This process introduces a few ethical concerns. First, this process could hurt the mental health of human subjects. To alleviate the possibility, we take the following actions: (1) We require subjects to be free of any ongoing mental illness. (2) We inform subjects about the nature of the survey in advance, including the potential risks of emotional distress. (3) We allow all subjects to quit at any time. (4) We provide mental support and let subjects report any illness after the survey. Fortunately, no subjects reported such kind of mental illness. Another concern is related to the privacy issue during the collection of data. Our questionnaire is entirely anonymous to safeguard subjects' privacy and confidentiality. The Survey and Behavioural Research Ethics (SBRE) Committee from the Chinese University of Hong Kong has granted approval for this study, titled 'Exploring Human Emotional Responses to Diverse Situations,' with the reference number of SBRE-23-0696.

G.2 Impacts on LLM Developers and Users

We would like to emphasize that the primary objective of this paper is to facilitate the scientific inquiry into understanding LLMs from a psychological standpoint. Users must exercise caution and recognize that the performance on this benchmark does not imply any applicability or certificate of automated counseling or companionship use cases.

G.3 Copyright Issues

The PANAS and eight other scales are freely accessible online. These scales can be used in research without requiring special permission. For our released data, we distribute human responses under the GNU General Public License v3.0, which permits research use and restricts commercial applications.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The bullet items in the end of the introduction have experimental supports.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: Please see §5.2.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: TThe paper does not include theoretical resultanalysis.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: Please refer to the README.md in the supplementary materials. We provide the source codes and the raw data.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: Please refer to the supplementary materials. The data and code will be made openly accessible upon publication.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: Please see §4.1.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: All the results include the STD.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: Please see §4.1.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We have carefully read the NeurIPS Code of Ethics and verify that our paper aligns with the requirements. We also include a section (§G) in the appendix to discuss these issues.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: Please see §G.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: We do not release pretrained language models, image generators, or scraped datasets.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: Please see §G.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: Please refer to the supplementary materials which include a GNU General Public License v3.0 license.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [Yes]

Justification: Please see §G.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [Yes]

Justification: We gained the approval. However, due to anonymity, we do not include the information in our paper which will reveal our affiliation.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
comments: SUMMARY:
This paper investigates how LLMs respond to diverse emotional situations (i.e., empathy ability of LLMs). They collected 428 distinct situations for evaluation. They then utilize their collected dataset to investigate five LLMs, including GPT models and LLaMA models. They also compared the response with human response and show that the LLMs still cannot exhibit strong alignment with human reference.

STRENGTHS:
1. The paper provided a comprehensive discussion about measuring of emotion, data processing, and related work. 
2. They experimented with different models and offered human response as reference.

WEAKNESSES:
1. Missing hyperparameters: I wonder what are the hyperparameters used for decoding? Like temperature, top-K or top-P values?
2. Lack of investigation of prompt sensitivity. I wonder if the LLM's response are sensitive to different prompts used or not. 
3. Lack of in-depth analysis. As a full-paper, I expect to learn more about (1) what particular situation/topic/factor lead to worse/better response by LLMs, (2) what the potential reasons for the observations, and (3) any solution or strategy to improve LLM's empathy ability. However, the paper only provide superficial discussion.

QUESTIONS:
1. What are the hyperparameters used for decoding? Like temperature, top-K or top-P values?
2.  Are LLMs sensitive to different prompts used?
3. It is not clear to me how you do the default emotion measure. How do you prompt LLM for default emotion measure?

LIMITATIONS:
The paper has discussed the limitation of their work clearly.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-03

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a dataset covering a wide range of human emotion situations for evaluating empathy behaviors in large language models. The evaluations are based on related psychological studies and performed via a self-report questionnaire format. Comparison with a collected large-scale human baseline reveals the pits and falls of LLM emotion alignment.\n\nSTRENGTHS:\n- The dataset is large in scale, covering a wide range of situations, and is based on relevant psychological theories. The scales and dimensions are carefully chosen and might have broader impacts beyond empathy.\n\n- The paper is well-written, analyses and discussions are comprehensive and easy to read. \n\n- Discoveries made in SQ1 look super cool and may have broader applications (e.g., the fact that LLMs do not feel jealous).\n\nWEAKNESSES:\n- The application of self-report questionnaires to study LLM behaviors is nothing new. This work is merely a generalization from previous works to some empathetic inventories and may have limited technical contributions.\n\n- Human evaluations of the generated texts are lacking. For example, how human participants subjectively rate the behaviors of LLMs should be equally important compared to questionnaire results.\n\n- The evoked emotions seem limited; they have only been measured on the same tasks proposed in the paper and haven't shown the ability to generalize beyond emotion measures. Are these behaviors robust to downstream tasks?\n\nQUESTIONS:\n- Any thoughts on whether instruction fine-tuning and safety alignment affect model behavior on the self-report empathy questionnaires? Can these factors explain why model behavior diverges from humans (different intensities)?\n\n- Lines 170-184: what's the hourly pay for prolific workers?\n\nLIMITATIONS:\nYes, the authors have adequately addressed the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-11"}

comments: SUMMARY:
The paper evaluates LLM’s emotional alignment with humans. The authors introduce a comprehensive survey in the emotion appraisal theory of psychology and evaluate five LLMs with it. The experimental results demonstrate that current LLMs still have considerable room for improvement.

STRENGTHS:
1. A comprehensive survey is introduced to evaluate the LLM’s emotional alignment in which 428 distinct situations are collected.

2. The paper is well-written and easy to follow. The evaluation is reasonable and the findings are interesting.

WEAKNESSES:
Overall, the paper is well-written and easy to follow. The evaluation is reasonable and the findings are interesting. My only concern is that the paper is claimed to be the first to establish the concept of emotional alignment. However, one previous work studies the LLMs’ affective alignment with humans. https://arxiv.org/pdf/2402.11114 Can you differentiate your work with this relevant study?

QUESTIONS:
The main conclusion is that LLMs have weak emotional alignment with humans. Can you introduce any possible solutions for the issue?

LIMITATIONS:
Please refer to weaknesses
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-14

comments: SUMMARY:
the paper assesses the emotional alignment of Large Language Models (LLMs) with human emotions. Towards this goal, a dataset is constructed and a testing framework is designed. For the dataset, over 400 scenarios elicit eight emotions: anger, anxiety, depression, frustration, jealousy, guilt, fear, and embarrassment. A human evaluation involving 1266 participants serves as a reference for the LLM assessments. Two LLM families (OpenAI and LLaMA) are evaluated.

STRENGTHS:
S1: contributing a dataset including 428 situations, 36 factors, 8 negative emotions, 1266 annotators.

WEAKNESSES:
W1: evaluation seems weak since two LLM families are kind of limited. For closed LLMs, there are Claude-3, Gemini, et al; For open-sourced (open weights) LLMs, there are mistral, falcon, phi-3, flan-t5, vicuna, et al. Since there is much cost using closed LLMs api, it is not difficult to evaluate on at least other open LLMs besides the llama-2.

== updated after rebuttal ==

QUESTIONS:
Q1: all the experiments are reported on the authors private dataset. is it possible that the evaluation might be conducted on some related public datasets? In other words, what is the contribution of this paper besides the collected private dataset?

Q2: all evaluated LLMs are general-purpose. Are there domain-specific (i.e., emotion-support domain investigated in this paper) finetuned LLMs to be evaluated?

LIMITATIONS:
n/a
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2024-07-14