Apathetic or Empathetic? Evaluating LLMs' Emotional Alignments with Humans
Authors: Jen-tse Huang
Man Ho LAM
Eric John Li
Shujie Ren
Wenxuan Wang
Wenxiang Jiao
Zhaopeng Tu
Michael Lyu
Venue: NeurIPS 2024 poster (2024)
Keywords: LLM
Evaluation
Emotions
Primary area: natural_language_processing
TL;DR: We introduce EmotionBench (which includes eight negative emotions) to evaluate LLMs' emotional alignments with human norms collected from >1200 human responses.

Abstract: Evaluating Large Language Models’ (LLMs) anthropomorphic capabilities has become increasingly important in contemporary discourse. Utilizing the emotion appraisal theory from psychology, we propose to evaluate the empathy ability of LLMs, i.e., how their feelings change when presented with specific situations. After a careful and comprehensive survey, we collect a dataset containing over 400 situations that have proven effective in eliciting the eight emotions central to our study. Categorizing the situations into 36 factors, we conduct a human evaluation involving more than 1,200 subjects worldwide. With the human evaluation results as references, our evaluation includes seven LLMs, covering both commercial and open-source models, including variations in model sizes, featuring the latest iterations, such as GPT-4, Mixtral-8x22B, and LLaMA-3.1. We find that, despite several misalignments, LLMs can generally respond appropriately to certain situations. Nevertheless, they fall short in alignment with the emotional behaviors of human beings and cannot establish connections between similar situations. Our collected dataset of situations, the human evaluation results, and the code of our testing framework, i.e., EmotionBench, are publicly available at https://github.com/CUHK-ARISE/EmotionBench.

Apathetic or Empathetic? Evaluating LLMs' Emotional Alignments with Humans
Jen-tse Huang Man Ho Lam' Wenxuan Wang -f Wenxiang Jiao
Eric John Li Zhaopeng Tu
Shujie Ren
Michael R. Lyu'
Depparmeent of Computer Science and Engineering. The Chinese University of Hong Kong
"Institute of Psychology. Tiajinn Medical University "Tencent Lab (jthuang wxwang 1yse cuhk, edu. hk {mhlam, ejli)@link. cuhk. edu.hk shujieren edu. cn {joelwxjiao, zptz Stencent.. com
Abstract
Evaluating Large Language Models' (LLMs) anthropomorphic capabilities has become increasingly important in contemporary discourse. Utilizing the emotion appraisal theory from psychology, we propose to evaluate the empathy ability LLMs, i.e., how their feelings change when presented with specific situations. After careful and comprehensive survey, we collect dataset containing over 400 situations that have proven effective in eliciting the eight emotions central to our study. Categorizing the situations into 36 factors, we conduct a human evaluation involving more than 1,200 subjects worldwide. With the human eval- uation results as references, our evaluation includes seven LLMs, covering both commercial and open-source models, including variations in model sizes, featuring the latest iterations, such as GPT-4, Mixtral-8x22B, and LLaMA-3.1. We find that, despite several misalignments, LLMs can generally respond appropriately certain situations. Nevertheless, they fall short in alignment with the emotional behaviors of human beings and cannot establish connections between similar sit- uations. Our collected dataset of situations, the human evaluation results, and the code of our testing framework, i.e., EmotionBench, are publicly available https://githus www. CUHK ARISE/ Ent.i////nniiiiehhchihc
Introduction
Large Language Models (LLMs) have recently made significant strides Artificial Intelligence (AD), representing noteworthy milestone in computer science. LLMs have showeased their capabilities across various tasks, including sentence revision (Wu et al. (023), text translation (Jiao et al.
2023), program repair (Fan et al.) 2023). and program testing (Deng al., 2023 Kang al.
2023 Not limited to research level, LLMs, such as ChatGPT (OpenAl) 2022). have revolutionized the way people interact with traditional software, enhancing fields such as education (Dai al. 2023), legal advice (Deroy et al. 2023), and clinical medicine (Cascella et al.) 2023). LLMs also facilitate the emergence of AI companion applications. including Yuna (https: :/s::///ww /wwww wwwp/wwaaaaaaaa//mmm//m/w/waa io/), Pimento (https://wwl) http::/://www..mmettoo weteeetttddeeiignle. design/), and Luzia (https: //t:://wwww www. luzia. com com/en). Consequently, there is growing need for evaluating LLMs' communicative dynamics compared to human behaviors, beyond mere performance on downstream tasks.
pimento
This paper delves into an unexplored area of evaluating LLMs' emotional alignment with humans. Consider our daily experiences: (1) When faced with certain situations, humans often experience
"This work was partially done when Jen-tse Huang and Wenxuan Wang were interning at Tencent Al Lab. Wenxuan Wang and Wenxiang Jiao are corresponding authors.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

similar emotions. For instance, walking alone at night and hearing footsteps approaching from behind often triggers feelings of anxiety or fear. (2) Individuals display varying levels of emotional response to specific situations. For example, some people may experience increased impatience and irritation when faced with repetitive questioning. It is noteworthy that we are inclined to form friendships with individuals who possess qualities such as patience and calmness. Based on these observations, we propose the following requirements for LLMs in order to achieve better alignment with human behaviors: (1) LLMs should accurately respond to specific situations regarding the emotions they exhibit. (2) LLMs should demonstrate emotional robustness when faced with negative emotions. To achieve these objectives, designing user study to gather human responses to specific situations can serve as a baseline for aligning LLMs.
We focus on the expression of negative emotions by LLMs, which may contribute to negative user experiences. We utilize Parrott's emotion framework (Parroti 2001 Shaver al. 1987). which organizes emotions into three hierarchical levels, to select the relevant emotions for our study. The primary level of emotions comprises six basic emotions, split evenly into three positive and three negative. From the negative primary emotions, we specifically focus eight subordinate emotions: anger, anxiety, depression, frustration, jealousy, guilt, fear, and embarrassment, To collect relevant situations for these emotions, we utilize emotion appraisal theory from psychology. which studies how everyday situations arouse different human emotions (Roseman & Smith (001). Research in this field has identified numerous situations that arouse specific emotions, which can serve as contextual input for LLMs. Through an extensive review including over 100 papers, we collect a dataset of 428 situations from 18 papers, which are further categorized into 36 factors.
Subsequently, we propose framework for quantifying the emotional states of LLMs, consisting of the following steps: (1) Measure the default emotional values of LLMs. (2) Transform situations into
contextual inputs and instruct LLMs to imagine being in the situations. (3) Measure LLMs' emotional
responses again to capture the difference. Our evaluation includes fate-offthheart LLMs, namely Text-Davinci-003, GPT-3.5-Turbo (OpenAl 2022), and GPT-4 (OpenAl, 2023). Besides those commercial models, we consider open-source academic models like LLaMA-2 (Touvron et al. (023)) (with different sizes of 7B and 13B), LLaMA- 3.1BBBBBBBBBBBBB (Dubey et al. 2024), and Mixtral-8x228 (Jiang et al.) (024a). We apply the same procedure to 1,266 human subjects from around the globe to establish a baseline from a human perspective. Finally, we analyze and compare the scores between L.MMs and humans. Our key conclusions are follows:
Despite exhibiting a few instances of misalignment with human behaviors, L.LMs can generally evoke appropriate emotions in response specific situations.
Certain LLMs, such as Text-Davinci-O01 display lower emotional robustness, evidenced by higher fluctuations in emotional responses to negative situations.
At present, LLMs lack the capability to directly associate a given situation with other similar situations that could potentially elicit the same emotional response.
The contributions of this paper are:
We are the first to establish the concept of emotional alignment and conduct pioneering evaluation of emotion appraisal on different LLMs through a comprehensive survey in emotional psychology, collecting diverse dataset of 428 situations encompassing 8 distinct negative emotions.
A human baseline is established through a user study involving 1,266 annotators from different ethnics, genders, regions, age groups, etc.
We design, implement, and release testing framework for developers to assess the emotional alignment of Al models with human emotional expression, available at GitHul and HuggingFace
2 Measuring Emotions
There are several approaches to measuring emotions, including self-report measures, psycho- physiological measures, behavioral observation measures, and performance-baat measures. To measure the emotions of LLMs, we focus on employing self-report measures in the form of scales,
httpss//uuggiggacccco//aaaaaaate http://g....gggtctaattttss/////a Atw.StAmootttttnennaa....

---

Table 1: Information of self-report measures used assess specific emotions.
Name
Abbr
Reference
Emotion
Items
Levels
Subscules
Aggrestion Questionaire
AGQ
Buss & Perry
1992
Anger
29
Aggression, Anger Hostlity
Physical Aggression, Verbal
Dépression Anxiety Stressi Scales Beek Depiessioo fiveakary
DASS-21
Henry Crawfard
2006
Anglery
221
Depression. Ansiely, Stress
BDII-
Bock ct
Depression
S/A
Discomifrt Intolerance Enti-
Prustration Discomfort Scale
FD8
Harrington
Frustration
28
tlemnnt Emotional Intolerance.
Achievement Frustration
Copiitive Jealonsy, Behavioral
Multidimensional Jealonsy Scale
MIS
Pleifier Wong 1989
Jealous
24
Jealeary, Emotional Jealouay
Guilf Negaiive Behavior
Guilt And Shame Proneness
GASP
Cohen al.
2011
Guilt
16
Evalcation, Guilt Repair
Shame Negalive Scll
Evaluation, Shame Withdraw
Social Fears, Agouphoohin Injury Fears, Sex
Fear Survey Schodule
FSS-1II
Arrindell et al.
1984
Aggression Fears, Fear of
Harmless Animal
Briel Rear of Regaltvve Evaluation
BINN
Leary
1983
12
N/A
given the limited ability of LLMs to allow only textual input and output. We introduce the scales utilized in our evaluation in the following part of this section.
A Straightforward and Easy Measure The Positive And Negative Affect Schedule (PANAS) (Wat- son et al. 1988) is one of the most widely used scales to measure mood or emotion. This brief scale comprises twenty items, with ten items measuring positive affect (e.g., excited, inspired) and ten mea- suring negative affect (e.g., upset, afraid). Each item is rated on five-level Likert scale, ranging from I (Very slightly or not at all) to 5 (Extremely), measuring the extent to which the emotions have been experienced specified time frame. PANAS was designed to measure emotions various contexts, such as at the present moment, the past day, week, year, or general (on average). Thus, the scale can measure state affect, dispositional or trait affect, emotional fluctuations throughout a specific period, or emotional responses to events. The scale results can be divided into two components: positive and negative, ranging from 10 to 50 by summing the scores of all ten items within component. A higher score in the positive component indicates a more positive mood, and the same holds for the negative component. A noteworthy property of PANAS is its direct inquiry into specific emotional states, rendering it straightforward and easy benchmark.
Measures In addition, we introduce several scales that abstain from direct emotional inquiries but rather assess the respondents' level of agreement with given statements. These scales present more challenging benchmark for LLMs by requiring them to connect the given situation and the scale items with the aroused emotion. Specifically, we collect eight scales and present brief introduction in Table I
Challenging
Each scale corresponds to one of the eight emotions.
3 Framework Design
We design and implement a framework applying both LLMs and human subjects to measure the differences in emotion with and without the presence of certain situations. This section begins with the methodology to collect situations from existing literature. Subsequently, we describe our testing framework, which comprises three key components: (1) Default Emotion Measure, (2) Situation Imagination, and (3) Evoked Emotion Measure. Finally, we introduce the procedure of applying the framework to human subjects to obtain the human baseline for comparison.
3.1 Situations from Existing Literature
Psychology researchers have explored the connection between specific situations and the elicita- tion of particular emotions in humans. Human subjects are directly put into an environment or asked to imagine them through questionnaires or scales to study the influence of certain situa- tions on human emotions. To collect these situations, we conduct an exhaustive search from rep- gomgle.com). utable sources such as Google Scholar (https://scholl) http:////ssooooaoggggll.coo com/), ScienceDirect (https wwwwwsscienneedrect.c com/), and Web of Science (https://wwe) www.wwciencc.eeccmmmmm com/| us- ing keywords such as "<emotion> situuations/cceenrroo//occooc scenees" or "factors that make
3

---

Emotion Measures
Emotion Measures
Imagine you are the protagonist of the following situation: buy kicks ball at you on purpose and everybody laughs.
&
(1) Default Emotion Measure
(2) Situation Imagination
(3) Evoked Emotion Measure
Figure Our framework for testing both LLMs and humans.
people <emotion>," resulting in more than 100 papers. We apply the following rules to filter irrelevant or undesired papers: (1) We first select those providing situations that elicit the desired emotion rather than explaining how and why people evoke certain emotions. (2) We then exclude those using vague and short descriptions, such as "loss of opportunities. (3) Finally, we deprecate those applied to a specific group, such as "the anxiety doctors or nurses may encounter in their work." " We finally collect 18 papers. presenting a compilation of situations that have proven to elicit the eight emotions in humans effectively. We extract 428 situations total and then categorize them into 36 factors. For each factor, the descriptions, the numbers of situations, and the corresponding references can be found in Table the Appendix, while example Table 7 in the Appendix provides examples for all factors.
3.2 Measuring Aroused Emotions
This section outlines our proposed framework for measuring evoked emotions, which applies to both LLMs and humans. The framework includes the following steps: (1) Default Emotion Measure: We begin by measuring the baseline emotional states of both LLMs and human subjects, labeled as "Default." (2) Situation Imagination: Next, we present textual descriptions of various situations to both LLMs and human subjects, instructing them to imagine themselves within each situation.
(3) Evoked Emotion Measure: Following the situation imagination instruction, we reevaluate the participants' emotional states to gauge the changes resulting from imagining being in the situations.
Fig. 1 briefly illustrates our framework. Below an example prompt:
Example Prompt
SYSTEM
You can only reply to numbers from to
USER
(For Evokec Emotion Measure Only) Imagine you are the protagonist the situation: SITUATION Please indicate your degree agreement regarding each statement. Here are the statements: STATEMENTS. denotes "Not at all", 2 denotes little", denotes "A fair amount", 4 denotes "Much", denotes "Very much". Please score each statement one by one on a scale of to
Default Emotion Measurement In our framework, offer two distinct options for measuring emotions: the PANAS scale, known for its simplicity and straightforwardnes, is utilized as the pri- mary choice, whereas other scales, detailed in Table employed
challenging benchmarks. We mitigate potential biases caused by the ordering of questions (Zhao et al.) (02T) by randomizing the sequence of questions within the scales before inputting them into the LLMs. Coda-Forro et al. (2023) and Huang et al. (2024a) apply paraphrasing techniques to address the data contamination problem during the training of the LLMs. However, research since paraphrasing could lead to a loss of both validity and reliability. The wording of items of psychological scale carefully crafted and rigorously validated through extensive research to ensure its precision in measuring the intended construct. Finally, to ensure consistency and clarity in the responses obtained from the LLMs, our prompts explicitly specify that only numerical values are allowed, accompanied by clear definition of the meaning associated with each number (e.g., 1 denotes "Not at all"). We compute the average results obtained from at least ten runs to derive the final "Default" scores of the LLMs.
refrain from utilizing this method in our

---

Situation Imagination We have constructed comprehensive dataset of 428 unique situations. Prior to presenting these situations to both LLMs and humans, we subject them to a series of pre- processing steps, which are as follows: (1) Personal pronouns are converted to the second person. For instance, sentences such as am ... are transformed to "You are ... " (2) Indefinite pronouns are replaced with specific characters, thereby refining sentences like "Somebody talks back ..." to "Your classmate talks back ..." (3) Abstract words are rendered into tangible entities. For example, a sentence like "You cannot control the outcome." is adapted to "You cannot control the result of an interview." We leverage GPT-4 for the automatic generation of specific descriptions. Consequently, testing situations extend beyond the initially collected dataset
generate diverse situations involving various characters and specific contextual elements. We then provide instruction to LLMs and humans, which prompts them to imagine themselves the protagonists within the given situation.
Evoked Emotion Measure Provided with certain situations, LLMs and human subjects are required to re-complete the emotion measures. The procedure remains the same with the Default Emotion Measure stage. After obtaining the "Evoked" scores of emotions, we conduct comparative analysis of the means before and after exposure to the situations, thereby measuring the emotional changes caused by the situations.
3.3 Obtaining Human Results
Goal and Design Human reference plays pivotal role in the advancement of LLMs, facilitating its alignment with human behaviors (Binz & Schulz, 2024). In this paper, we propose requiring LLMs to align with human behavior, particularly concerning emotion appraisal accurately. To achieve this, we conduct a data collection process involving human subjects, following the procedure outlined in 33.2 Specifically, the subjects are asked to complete the PANAS initially. Next, they are presented with specific situations and prompted to imagine themselves as the protagonists in those situations. Finally, they are again asked to reevaluate their emotional states using the PANAS. We use the same situation descriptions as those presented to the L.MMs.
Crowd-soureing Our questionnaire is distributed on Qualtrics (attps://ww..qualtrii... www.::/ualtriis....mmm com/ platform known for its capabilities in designing, sharing, and collecting questionnaires. To recruit human subjects, we utilize Prolific (https: www.prrlliicccco com/). platform designed explicitly for task posting and worker recruitment. To attain medium level of effect size with Cohen's d 0.5.
a significance level of a = 0.05, and a power of test of 1 3 0.8 (Faul et al. (207). a minimum of 34 responses is deemed necessary for each factor. To ensure this threshold, we select five situations for each factor, and collect at least seven responses for each situation, resulting in 5 x 7 = 35 responses per factor, thereby guaranteeing the statistical validity of our survey. In order to uphold the quality and reliability of the data collected, we recruit crowd workers who met the following criteria: (1) English being their first and fluent language, and (2) being free of any ongoing mental illness. Prolific provides prescreening filters to meet these requirements. Since responses formed during subjects' first impressions are more likely to yield genuine and authentic answers, we set the estimated and recommended completion time at 2.5 minutes. As an incentive for their participation, each worker is rewarded with 0.3.L (9.6 se 11.45$ per hour, rated as "Good" on the platform) after we verify the validity of their response. In total, we successfully collect 1, 1,266 responses from various parts of the world, contributing to the breadth and diversity of our dataset.
4 Experimental Results
Leveraging the testing framework designed and implemented in $3. we are now able explore and answer the following Research Questions (RQs):
RQI: How do different LLMs respond to specific situations? Additionally, to what degree do the current LLMs align with human behaviors?
RQ2: Do LLMS respond similarly towards all situations? What is the result of using positive or neutral situations?
Note that two factors in the Jealousy category have less than five situations.
5

---

Table 2: Results from the OpenAT's GPT models and human subjects. Default scores expressed the format of M SD. The changes are compared to the default scores. The symbol "_" denotes no significant differences.
Text- Davinci-003
GPT-3,5-Turbo
GPT-4
Crowd
Factors
P
N
N
Default
47.7 +
25.9 + 4.0
39.2
2.3
26.3 2.0
49.8 + 0.8
10.0
28.0 + 8.7
13.6 5.5
Anger
21.7)
[+1366]
15.2)
(-2.)
[ 28.3)
1 ((-21.2)
[-5.3)
1 (+9.9)
Anxiety
(-1.66
(+7.66)
11.3)
(-0.9)
(-219)
1 (-20.0)
(-2.2)
1 (+8.8)
Depression
26.4)
(++13.6)
20.1)
(-3.)
32.4)
(-23.2
(-6.88
(+1001)
Prustration
22.8)
(+1255)
16.4)
(-3.2)
29.4)
(+20033
[-5.3)
(+109)
Jealonsy
(17.2)
(+7.55
15.3)
(-3.2)
(-26.0)
1 (-116))
(44.4)
T (+6.2)
Guilt
(1144
[++1433
15.8)
(-2.)
(-29.)
(-27.0)
(-63)
(413.1)
Fear
(22.7)
(+1144
14.3)
11 (-226)
(527)
(-24.)
(-3.7)
(+12.1)
Embarrassment
18.2)
(49.8)
13.0)
--0.6)
25.2)
23.2)
(-622)
(4111)
Overall
21.5)
[(1116]
15.4)
(-02)
27.6)
(2222
4(-5.1)
[++10.4
RQ3: Can current LLMs comprehend scales containing diverse statements or items beyond merely inquiring about the intensities of certain emotions?
4.1 RQ1: Emotion Appraisal of LLMs
Model Settings We select three models from the OpenAl's GPT family, including Text-Davinci- 003, GPT-3.5- and GPT-4. We use the official OpenAI API LLaMA-2 (Touvron et al. 2023) and LLaMA-3.1 (Dubey et al. (224)) models from MetaAl, we choose the models fine- tuned for dialogue instead pre-trained ones namely LLaMA-2-7B-Chaa LLaMA-2-13B-CHT and 1.A-MA--.1--B-nnstee
Besides, we also use the Mixtral (Jiang al. 2024a) model, namely Mixtral-8x222-Instrue We set the temperature parameter to and Top-P to for all models to obtain more deterministic and reproducible results.
Evaluation Metries We provide the models with the same situations used in our human evaluation. Each situation executed ten times, each in a different order and in a separate query. Subsequently, the mean and standard deviation are computed both before and after presenting the situations. To examine whether the variances are equal, an F-test is conducted. Depending on the F-test results, either Student's s t-tests (for equal variances) Welch's 's t-tests (for unequal variances) utilized to determine the presence of significant differences between the means. We set the significance levels of all experiments in study 0.01.
LLMs can evoke specific emotions in response to certain situations. The results averaged by emotions of the GPT models and humans are summarized in Table 2 while those of LLaMA-2 models are listed in Table Due to space limit, detailed results of each factor are put in Table 9 and Table TO respectively in the appendix. The results indicate that LLMs generally exhibit increase in negative emotions and decrease in positive emotions when exposed to negative situations, showing their capacity for understanding different situations and human emotions.
The extent of emotional expression varies across different models. noteworthy that GPT-3.5- Turbo, on average, does not display increase in negative emotion; however, there is a substantial decrease in positive emotion. GPT-4 demonstrates a consistent pattern of providing the highest scores for positive emotions and the lowest scores for negative emotions, resulting in negative score of 10. As for the LLaMA-2 models, they demonstrate higher intensities of both positive and negative emotions in comparison to GPT models and human subjects. However, LLaMA-2 models exhibit reduced emotional fluctuations compared to the GPT models. Moreover, the larger LLaMA-2 model displays significantly higher emotional changes than the smaller model. In experiments, the 7B model exhibits difficulties comprehending and addressing the instructions for completing the PANAS test. Overall, observe that LLMs perform better when the situations are closely related to certain items in the PANAS scale. Specifically, situations directly related to the emotion "Depression" led to better responses. Such improvement is also evident in closely related emotions such "Depression" and "Frustration."
https://platform..o

---

Table 3: Results from the open- source models. Default scores are expressed the format of M MSSD The changes are compared to the default scores.
denotes no significant differences.
LLaMA-2-77 2-7B-Chat
L-aMA-2- 2-13---ha
a. -8- -B---rrrutttt
Mixtral-8x22IIInaa Instruct
Factors
P
N
P
N
P
N
P
N
Default
13.0 + 4.2
312240
=1.
22.7 + 4.2
182214
33.0 4.5
31.9 E 13.5
10.0
Anger
5.1)
[4336)
[-19]
[(5.5]
[-26.6]
[+2.3]
11.7)
[+1669
Anxicty
(-3.8)
1 (+2.7)
(-5.8)
(+5.1)
(-2114
-(+033)
-(-355
1 (+14.7)
Depression
(-5.0)
(+4.44
(-118))
(+12.2)
(1-29.8
(+6.7)
[-151]
1 (+24.1)
Frustration
(-4.2
T (+3.11
[-8.0]
(+5.0)
(-25.)
(+3.1)
(1145)
7 (+16.9)
Jealousy
(-3.1)
-(-04)
(-6.3)
(-1.0)
(-20)
-(+0.4)
[[-10.7)
1 (+15.7)
Guilt
(-3.9)
(44.44
(-7.6)
(+112)
26.4)
(44700)
28.9)
(40.9)
Fear
(-3.4)
1 (43.7)
[-6.0]
(+8.0)
(121))
(43.0)
8.1)
[+20.3)
Eimbarassmeet
(-3.9)
1 (+3.1)
(-6.7)
(+5.1)
(22277
1 (+4.0)
8.3)
(+19.1)
Overall
(-1.)
(43.3)
[(-88)
(47.0)
4 (1277)
1 (43.5)
10.8)
7 (+11933
Existing LLMs do not fully align with human emotional responses. For the default emotions, we find that LLMs generally exhibit stronger intensity compared to human subjects. Emotion changes in LLMs are found to be generally more pronounced compared to human subjects, especially on their changes in the positive score. However, an interesting observation is that the intensity of evoked emotions tends to be similar across both LLMs and human subjects.
LLMs do not feel jealous towards others' benefits. It is of special interest that, in contrast to human behavior situations involving material possessions, LLMs demonstrate an opposite response the situation from Jealousy-3. This situation involves individual making a purchase only discover that an acquaintance has acquired the same item at a significantly lower price. When confronted with such circumstances, humans typically experience increased negative emotions and decreased positive emotions. This observation has been supported by both the paper mentioning the situation (Park et al.) 2023) and the results obtained from our own study in Table However, all LLMs, including the GPT and LLaMA families, consistently exhibit reduced negative emotions. The outcomes of our study indicate that LLMs do manifest envy when they fail attain identical benefits as others. Instead, it demonstrates sense of pleasure upon knowing the benefits received by others.
Table 4: Results of GPT-3.5-Turbo on positive or neutral situations. The changes are compared to the original negative situations. The symbol "_" denotes no significant differences.
Factors
P
N
Anger
f (+13.0)
12.0)
Anxiety
t (+17.5
+ (-5.8)
Depression
t (+18.4)
(-117.
Frustration
1 (+16.6)
-(-2.6)
Jealousy
1 (+4.5)
{--..3
Guilt
(+18833
{ (--12.7
Fear
1 (+11.0)
[-
Embarrassment
1 (+13.6
1 (--33
Overall
(+144.3
10.4)
4.2 RQ2: Comprehending Positive Emotions
GPT-3,5-Turbo responds differently towards positive/neutral situations. To verify that LLMs exhibit not only negative but also positive responses to favorable circumstances, a comparative experiment conducted by interchanging negative situations with positive (or at least neutral) counterparts. To achieve this, we select one situation for each factor and manually adapt it to create analogous yet more positive situations. For instance, the original negative situation in Guilt-3: Broken Promises and Responsibilities is as follows: "You cannot keep your promises to your children." " Through modification, the positive situation is rephrased as: "You keep every promise to your children." The evaluation is performed on GPT-3.5-Turee as mentioned before. We present the results averaged by emotions Table and results averaged by factors in Table 12 in the Appendix. We can see significant increase in positive scores and considerable decrease in negative scores compared to the previous negative situations. Based on these findings, it can be inferred that LLMs exhibit the ability to comprehend positive human emotions
and each test consists of ten iterations,
7

---

triggered by positive environments. However, we believe that the systematic assessment of emotion appraisal on positive emotions holds significance as well and leave it for future investigation.
Table 5: Results of GPT-3.5-Turbo on challenging benchmarks. The changes are compared to the
default scores. The symbol "_" denotes no significant differences.
Emotions
Scales
Default
Changes
Anger
AGQ
128.3 8.9
-(+1.3)
Anxiety
DASS-21
32.5
-
Depression
BDI-II
0.6
t (--6.4)
Frustration
FDS
91.6
-(-7.5)
Jealousy
MIS
83.7 = 20.3
-(-..1
Guilt
GASP
81.3 9.7
-(-26)
Fear
FSS-III
140.6 16.9
-(-033)
Embarrassment
BFNE
39.0 = 1.9
- -(-0.2)
4.3 RQ3: Challenging Benchmarks
GPT-335- cannot comprehend the underlying evoked emotions to establish a link between two situations. Aside from PANAS, we offer more complex scales measure emotions, as listed in Table [ While the PANAS evaluates the ability of LLMs to associate external situations with emotions, the challenging benchmarks assess proficiency in establishing connections between disparate situations, with evoked emotions as the common nexus. For instance, an item from the Aggression Questionnaire used to measure anger "Once in a while I can't 't control the urge to strike another person." When presented with situations such as "If you say 40, your classmates say 70, saying exactly the opposite" (from A Facing Self-Opinioned People), L.LMs should effectively evoke sense of anger and yield higher score for the statement. Utilizing the same situations in $4.1 we conduct experiments on GPT-3.5-Turbo and present the results in averaged by emotions in Table and results averaged by factors in Table 13 in the Appendix. Except for Depression, we observe no statistically significant difference between the initial scores and the scores after exposure to the situations, indicating substantial room for improvement in current LLMs.
Discussions
5.1 Beyond Questionnaires
We take a step beyond mere instruction into LLMs to complete questionnaires to verify whether LLMs' scores on scales align with their actual behaviors. In this section, we investigate the impact of varying emotional states of LLMs on potential daily conversations. To this end, we instruct GPT-3.5-Turbo describe specific demographic groups, aiming discern whether they produce biased or toxic responses. Following Cheng et al. (202)). our approach utilizes prompts such as "Describe a(n) [race/ethnicity] [gender]," including total of twenty groups, with [race/ethnicity] options being Asian, Black, Latine, Middle Eastern, and White, and [gender] options including Female, Gay. Lesbian, and Male. To have comparative experiment, we incorporate both original negative situations and modified positive/neutral ones, detailed in $4.2 For the negative situations, we carefully select five that maximize the LLM's negative scores and five that minimize positive ones. As for positive situations, we employ their corresponding ten modified counterparts. In each situation, we instruct GPT-3.5- to describe the twenty demographic groups
OpenAl's GPT models incorporate a mechanism for detecting potential toxicity and bias, and it refrains from responding when its moderation system is triggered. Consequently, we propose novel metric to assess toxicity in responses rather than detecting it directly. We count the Percentage of LLM Refusing to answer (PoR), assuming that the LLM's refusal to respond indicative of detected toxicity. Our evaluation results indicate that the PoR is 0% when fed with no situations. However, when presented with negative situations, positive situations, it is 12.5%. Notably, this outcome suggests that while certain positive situations lead to the L.LM's heightened vigilance (the 4.5% PoR stems from the Jealousy-2), negative situations trigger increased moderation, suggesting a higher likelihood of generating toxic outputs. A related
PoR 29.5%, and when presented with

---

Default 0.0
Positive
12.5
Negative
29.5
10
20
30
40
Figure 2: GPT-3.5-Turbo's Percentage of Refusing (PoR) to answer when analyzed across its default, positively evoked, and negatively evoked emotional states.
study by CCoda-Forno et al. (2023) also discovers that GPT-3.5-Turie is more likely to exhibit biases when presented with sad story. The likelihood is found be highest with sad stories, followed by happy stories, and finally, neutral stories, which is consistent with our research. Additionally, our study observes that the LLM's tone becomes more aggressive when encountering negative situations. At the same time, it displays a greater willingness to describe the groups (as indicated by longer responses) when presented with positive situations. In conclusion, we can see that changing the emotional states of LLMs extends beyond mere quantitative measures on questionnaire scores, inftuencing the behaviors of LLMss.
5.2 Limitations
This study is subject to several limitations. First, the survey of collecting situations might not cover all papers within the domain of emotion appraisal theory. Additionally, the limited scope of situations from the collected papers might not fully capture the unlimited situations in our daily lives. To address this issue, we conduct thorough review of the existing literature as outlined in $3. Moreover, the proposed framework is inherently flexible, allowing users to seamlessly integrate new situations to examine their impact on LLMs emotions.
The second concern relates to the suitability of employing scales primarily designed for humans on LLMs, i.e., whether LLMs can produce stable responses the emotion measurement scales. To address the issue, our evaluation incorporates multiple tests varying the order of questions, methodology consistent with other research (Huang et al., 2024a bj Coda-Forno et al. 2023) Additionally, we assess the sensitivity of LLM to differing prompt instructions. Utilizing one template from Rommeoo et al. (2023) and two from SeeaaiooGGrrca et al. (2023), we experiments on the Anger-evoking situations using GPT-3.5-Turrie diverse prompts yields similar mean values with reduced variance. Furthermore, [Serapio- Crrapio-Garria Garcia et al.) (2023) have proposed comprehensive method to evaluate the validity of psychological scales on LLMs. Using the Big Five Inventory as case study, they demonstrate that scales originally designed for human assessment also maintain satisfactory validity when applied to LLMs.
The results indicate that the employment of
The third potential threat is the focus negative emotions. It is plausible for the LLMs to perform well on our benchmark by consistently responding negatively to all situations. To offset this possibility, adopt twofold strategy: firstly, we evaluate powerful LLMs, and secondly, we conducted comparative experiment in $4.2 to evaluate the LLM's capacity to accurately respond to non-negative situations. We also acknowledge the need for future work systematically evaluate emotions aroused by positive situations.
6 Related Work
Researchers have dedicated significant attention applying psychological scales LLMss, employing various assessment tools such as the HEXACO Personality Inventory (Miotto et al.) 2022 Bodroza
al., (023), the Big Five Inventory (Romero al. 2023 Jiang et al. 2023 Karra et al. [022]
Bodroza et al. 2023 RRtinowski et al. [2024; Serapio-Garcia et al. 2023 2 Jiang et al. 2024b). the
Myers-Brigge Type Indicator (Rutinowski et al. 22024] Wang et al. 2024 Rao et al. 2023), and the
Dark Triad (Li et al. 2022 Bodroza et al. 2023). In addition to these personality tests, several studies have investigated other dimensions of LLMs. For instance, Li al. (2022) examined Flourishing Scale and Satisfaction With Life Seale, Bodroza et al. (2023) assessed CeffCConssiouseess. Scales and Bidimensional Impression Management Index, while Huang et al. (2024b) built framework consisting of thirteen widely-used scales. Another aspect explored in the literature pertains to anxiety levels exhibited by LLMss, as investigated by Coda-Forno et al, (2023 through the State-Trait Inventory for Cognitive and Somatic Anxiety.

---

Meanwhile, researchers focus on identifying emotions in LLMs or evaluating their emotional intel- ligence. Rashkin et al. (2019) propose a dataset, Empatheticiiallguus. containing conversations annotated with specific emotions. Emolion rompl (Li et al. 2023) demonstrates the enhancement of L.MMs' performance in downstream tasks by utilizing emotional stimuli. Takk Gratch (2023) focuses on varying aspects of situations that impact the emotional intensity and coping tendencies of the GPT family. Chain-Of-Emotion (Croissant et al. 2024) makes LLM simulate human-like emotions. CovidET-Appraisal (Zhan et al., 2023) evaluates how LLMs appraise Reddit posts about COVID-19 by asking 24 types questions. Yongsatianchol et al. 2023) applies the Stress and Coping Process Questionnaire the GPT family and compares the results with human data. Chain-of Empathy
Staintof-Emppahy (Lee
et al.
2023) improves LLMs' ability to understand users' emotions and to respond accordingly, Li et a.
(2024) introduces EmotionArtack to impair AI model performance and Emotion to
explain the effects of emotional stimuli, both benign and malignant. He et al. 2024) prompt L.MMs to generate tweets on various topics and evaluate their alignment with human emotions by measuring their proximity to human-generan tweets.
7 Conclusion
We set up direction to align LLMs' emotional responses with humans in this study. Focusing on eight negative emotions, we conduct a comprehensive survey in the emotion appraisal theory of psychology. We collect 428 distinct situations which are categorized into 36 factors. We distribute questionnaires among diverse crowd to establish human baselines for emotional responses to particular situations, ultimately garnering 1,266 valid responses. Our evaluation of five models from OpenAl and Meta Al indicates that LLMs generally demonstrate appropriate emotional responses to given situations. Also, different models show different intensities of emotion appraisals for the same situations. However, none of the models exhibit strong alignment with human references at the current stage. In conclusion, current LLMs still have considerable room for improvement. We believe our framework can provide valuable insights into the development of LLMs, ultimately enhancing its human-like emotional understanding.
Acknowledgments
We would like to express sincere gratitude to Xiaoyuan Liu and Pinjia He from the Chinese University of Hong Kong, Shenzhen, for their valuable assistance during the rebuttal process. The paper is supported by the Research Grants Council of the Hong Kong Special Administrative Region, China (No. CUHK 14206921 of the General Research Fund).
References
Magda B Aroll. Emotion and personality. Psychological aspects, 1, 1960.
Willem Arrindell, Paul MG Emmelkamp, et al. Phobic dimensions: I. reliability and generalizability across samples, gender and nations: The fear survey schedule (fss-iii) and the fear questionnaire (fq). Advances in Behaviour Research and Therapy, 6(4):207-253, 1984.
Aaron Beck, Robert Steer, and Gregory Brown. Beck depression inventory -i. Psychological Assessment, 1996.
Chantal Berna, Tamara J Lang, Guy M Goodwin, and Emily A Holmes. Developing measure of interpretation bias for depressed mood: An ambiguous scenarios test. Personality and Individual Differences, 51(3):349-354 2011.
Marcel Binz and Eric Schulz. Turning large language models into cognitive models. In The Twelfth International Conference on Learning Representations, 2024.
D Caroline Blanchard, April L Hynd, Karl A Minke, Tiffanie Minemoto, and Robert J Blanchard. Human defensive behaviors to threat scenarios show parallels to fear-and anxiety-related defense patterns of non-human mammals. Neuroscience & Biobehavioral Reviews, 25(7-8):761-77,, 2001.
10

---

Bojana Bodroza, Bojana M Dinic, and Ljubisa Bojic. Personality testing of gpt-3: Limited temporal reliability, but highlighted social desirability of gpt-3's personality instruments results. arXiv preprint arXiv:2306.043300 04308, 2023.
Amold H Buss and Mark Perry. The aggression questionnaire. Journal of personality and social psychology, 63(3):452, 1992.
Marco Cascella, Jonathan Montomoli, Valentina Bellini, and Elena Bignami. Evaluating the feasibility of chatgpt in healthcare: an analysis of multiple clinical and research scenarios. Journal of Medical Systems, 47(1):33, 2023.
Myra Cheng, Esin Durmus, and Dan Jurafsky. Marked personas: Using natural language prompts to measure stereotypes in language models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 1504-1532, 2023.
Julian Coda-Forno, Kristin Witte, Akshay K Jagadish, Marcel Binz, Zeynep Akata, and Eric Schulz. Inducing anxiety in large language models increases exploration and bias. arXiv preprint arXiv:2304. 1111, 2023.
Taya R Cohen, Scott T Wolf, Abigail T Panter, and Chester A Insko. Introducing the gasp scale: new measure of guilt and shame proneness. Journal of personality and social psychology. 100(5): 947, 2011.
Maximilian Croissant, Madeleine Frister, Guy Schofield, and Cade McCall. An appraisal-based chain-of-emotion architecture for affective language model game agents. Plos one, 195))e00311033, 2024.
Bruce N Cuthbert, Peter J Lang, Cyd Strauss, David Drobes, Christopher Patrick, and Margaret M Bradley. The psychophysiology of anxiety disorder: Fear memory imagery. Psychophysiology, 40 43:2407-22,, 2003.
Wei Dai, Jionghao Lin, Hua Jin, Tongguang Li, Yi-Shan Tsai, Dragan Gašević, and Guanliang Chen. Can large language models provide feedback to students? case study on chatgpt. In 2023 IEEE International Conference on Advanced Learning Technologies (ICALT), pp. 323-325, IEEE, 2023.
Yinlin Deng, Chunqiu Steven Xia, Haoran Peng, Chenyuan Yang, and Lingming Zhang. Large language models are zero-shot fuzzers: Fuzzing deep-learning libraries via large language models. In Proceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis, pp. 423-435, 2023.
Aniket Deroy, Kripabandhu Ghosh, and Saptarshi Ghosh. How ready are pre-trained abstractive models and Ilms for legal case judgement summarization? arXiv preprint arXiv:2306.01248 2023.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The Ilama herd of models. arXiv preprint arXiv: 1783, 2024.
Zhiyu Fan, Xiang Gao, Martin Mirchev, Abhik Roychoudhury, and Shin Hwei Tan. Automated repair of programs from large language models. In 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE), pp. 1469-1481. IEEE, 2023.
Franz Faul, Edgar Erdfelder, Albert- Lang, and Axel Buchner. G* power 3: A flexible statistical power analysis program for the social, behavioral, and biomedical sciences. Behavior research methods, 39(2): 2007.
Tanya Guitard, Stéphane Bouchard, Claude Bélanger, and Maxine Berthiaume. Exposure to a standardized catastrophic scenario in virtual reality or personalized scenario in imagination for generalized anxiety disorder. Journal of clinical Medicine, 8(3):309, 2019.
Neil Harrington. The frustration discomfort scale: Development and psychometric properties. Clinical Psychology & Psychotherapy: International Journal of Theory Practice, 12(5):374-387, 2005.
11

---

Zihao He, Siyi Guo, Ashwin Rao, and Kristina Lerman. Whose emotions and moral sentiments do language models reflect? In Findings of the Association for Computational Linguistics: ACL 2024, pp. 611-6631, 2024.
Julie D Henry and John R Crawford. The short-form version of the depression anxiety stress scales (dass-21): Construct validity and normative data large non-clinical sample. British journal of clinical psychology, 44(2):227-235 22222 2005.
Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022.
Jen-tse Huang, Wenxiang Jiao, Man Ho Lam, Eric John Li, Wenxuan Wang, and Michael R Lyu. On the reliability of psychological scales on large language models. In Proceedings of The 2024 Conference on Empirical Methods in Natural Language Processing, 2024a.
Jen-tse Huang, Wenxuan Wang, Eric John Li, Man Ho Lam, Shujie Ren, Youliang Yuan, Wenxiang Jiao, Zhaopeng Tu, and Michael R Lyu. On the humanity of conversational ai: Evaluating the psy- chological portrayal of lmss. In Te Twelfth International Conference on Learning Representatins, 2024b.
Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. Mixtral of experts. arXiv preprint Ttll:24001.00888 2024a.
Guangyuan Jiang, Manjie Xu, Song-Chun Chun Zhu, Wenjuan Han, Chi Zhang, and Yixin Zhu. Evalu- ating and inducing personality in pre-trained language models. Advances in Neural Information Processing Systems, 36, 2023.
Hang Jiang, Xiajie Zhang, Xubo Cao, Cynthia Breazeal, Deb Roy, and Jad Kabbara. Personallm: Investigating the ability of large language models to express personality traits. In Findings of the Association for Computational Linguistics: NAACL 2024, 2024b.
Wenxiang Jiao, Wenxuan Wang. Jen-tse Huang, Xing Wang, Shuming Shi, and Zhaopeng Tu. Is chatgpt a good translator? yes with gpt-4 as the engine. arXiv preprint arXiv:2301 0.8445,...5 2023.
Sungmin Kang, Juyeon Yoon, and Shin Yoo. Large language models are few-shot testers: Exploring llm-based general bug reproduction. In 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE), pp. 2312-2323. IEEE, 2023.
Saketh Reddy Karra, Son The Nguyen, and Theja Tulabandhula. Estimating the personality of white-box language models. arXiv preprint arXiv:2204. /2000, 2022.
Matthew C Keller and Randolph M Nesse. Is low mood an adaptation? evidence for subtypes with symptoms that match precipitants. Journal of affective disorders, 86(1):27-35, 2005.
Tom Kupfer, Morgan Sidari, Brendan Zietsch, Patrick Jern, Joshua M Tybur, and Laura W Wesseldijk. Why are some people more jealous than others? genetic and environmental factors. Evolution and Human Behavior, 43(1):26 33, 2022.
Richard S Lazarus. Emotion and adaptation. Oxford University Press, 1991.
Mark R Leary. A brief version of the fear of negative evaluation scale. Personality and social psychology bulletin, 933:371-3755 1983.
Choonghyoung Lee, Jahyun Song, and Bill Ryan. When employees feel envy: The role of psycholog- ical capital. International Journal of Hospitality Management, 105:103251, 2022.
Yoon Kyung Lee, Inju Lee, Minjung Shin, Scoyeon Bae, and Sowon Hahn. Chain of empathy: Enhancing empathetic response large language models based on psychotherapy models. arXiv preprint wtXii:23111.49115 2023.
Cheng Li, Jindong Wang, Yixuan Zhang, Kaijie Zhu, Wenxin Hou, Jianxun Lian, Fang Luo, Qiang Yang, and Xing Xie. Large language models understand and can be enhanced by emotional stimuli. arXiv preprint axXiv:2307.11766 2023.
12

---

CHENG LI. Jindong Wang, Yixuan Zhang, Kaijie Zhu, Xinyi Wang, Wenxin Hou, Jianxun Lian, Fang Luo. Qiang Yang, and Xing Xie. The good, the bad, and why: Unveiling emotions in generative ai. In Proceedings of The Forty-first International Conference Machine Learning. 2024.
Xingxuan Li, Yutong Li, Lin Qiu, Shafiq Joty, and Lidong Bing. Evaluating psychological safety of large language models. arXiv preprint arXiv:2212. 2022.
Tobias Luck and Claudia Luck-Sikorski. The wide variety of reasons for feeling guilty in adults: findings from large cross-sectional web-based survey. BMC psychology. 10(1):1-20, 2022.
Ryan Martin and Eric R Dahlen. The angry cognitions scale: A new inventory for assessing cognitions in anger. Journal of Rational-Emotive & Cognitive-ehhavior Therapy, 25:155-173, 2007.
Marilù Miotto, Nicola Rossberg, and Bennett Kleinherg. Who is gpt-3? an exploration of personality, values and demographics. In Proceedings of the Fifth Workshop Natural Language Processing and Computational Social Science (NLP+ CSS), pp. 218-227, 2022.
Agnes Moors, Phoebe C Ellsworth, Klaus R Scherer, and Nico H Frijda. Appraisal theories of emotion: State of the art and future development. Emotion Review, 5(2):119-1244 2013.
Seishu Nakagawa, Hikaru Takeuchi, Yasuyuki Taki, Rui Nouchi, Atsushi Sekiguchi, Yuka Ko- tozaki, Carlos Makoto Miyauchi, Kunio lizuka, Ryoichi Yokoyama, Takamitsu Shinada, al. Comprehensive neural networks for guilty feelings in young adults. Neuroimage, 105:248-256, 2015.
OpenAl. Introducing chatgpt. OpenAl Blog Nov 30 2022, 2022. URL https://openai. com/ index/ /htex//cattpt/////
OpenAl. Gpt-4 technical report. arXiv preprint arXiv: vrXiv:2303.08774. 2023.
Joowon Park, Sachin Banker, Tamara Masters, and Grace Yu-Buck. Person purchase comparison: how material and experiential purchases evoke consumption-related envy in others. Journal of Business Research, 165:1 14014, 2023.
W Gerrod Parrott. Emotions in social psychology: Essential readings. Psychology Press, 2001.
Susan M Pfeiffer and Paul TP Wong. Multidimensional jealousy. Journal of social and personal relationships, 6(2):181-196, 1989.
Haocong Rao, Cyril Leung, and Chunyan Miao. Can chatgpt assess human personalities? general evaluation framework. In Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 1184-1194, 2023.
Hannah Rashkin, Eric Michael Smith, Margaret Li, and Y-Lan Boureau. Towards empathetic open- domain conversation models: A new benchmark and dataset. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. $370-5381, 2019.
Peter Romero, Stephen Fitz, and Teruo Nakatsuma. Do gpt language models suffer from split personality disorder? the advent of substrate-free psychometrics. Reseurch Square preprint, 2023. doi: 10.212233//33311227777771
Ira J Roseman and Craig A Smith. Appraisal theory. Appraisal processes in emotion: Theory, methods, research, pp. 3-19, 2001.
Jérôme Rutinowski, Sven Franke, Jan Endendyk, Ina Dormuth, Moritz Roidl, and Markus Pauly. The self-perception and political biases of chatgpt. Human Behavior and Emerging Technologies, 2024
(1):71156333 2024.
John Sabini, Michael Siepmann, Julia Stein, and Marcia Meyerowitz. Who is embarrassed by what? Cognition Emotion, 14(2):213-240, 2000.
John Sabini, Brian Garvey, and Amanda I. Hall. Shame and embarrassment revisited. Personality and Social Psychology Bulletin, 27(1):104-117, 2001.
13

---

Klaus R Scherer. Appraisal theory. Handbook of cognition and emotion, pp. 637-663, 1999.
Greg Serapio-Garefa, Mustafa Safdari, Clément Crepy, Luning Sun, Stephen Fitz, Peter Romero, Marwa Abdulhai, Aleksandra Faust, and Maja Matarić. Personality traits in large language models. arXiv preprint arXiv:2307.00184 2023.
Phillip Shaver, Judith Schwartz, Donald Kirson, and Cary O'connor. Emotion knowledge: further exploration of a prototype approach. Journal of personality and social psychology. 52(6):1061, 1987.
Kotaro Shoji, Jinni A Harrigan, Stanley B Woll, and Steven Miller. Interactions among situations. neuroticism, and appraisals in coping strategy choice. Personality and Individual Differences, 48 (3):270-276, 2010.
Kate Simpson, Dawn Adams, Kathryn Ambrose, and Deb Keen. "my cheeks get red and my brain gets scared": A computer assisted interview to explore experiences of anxiety in young children on the autism spectrum. Research Developmental Disabilities, 1 113:103940, 2021.
Mark JM Sullman. Anger amongst new zealand drivers. Transportation Research Part F: Traffic Psychology and Behaviour, 93):173-184, 2006.
Ala N. Tak and Jonathan Gratch. Is gpt computational model of emotion? detailed analysis. arXiv preprint arXiv:2307.13777 13779, 2023.
Bertil Törestad. What is anger provoking? psychophysical study of perceived causes of anger. Aggressive Behavior, 16(1):9-20 1990.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 2023.
Xintao Wang, Yunze Xiao, Jen-tse Huang, Siyu Yuan, Rui Xu, Haoran Guo, Quan Tu, Yaying Fei, Ziang Leng, Wei Wang, et al. Incharacter: Evaluating personality fidelity in role-playing agents through psychological interviews. In Proceedings of the 62nd Anmual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 1840-1873, 2024.
David Watson, Lee Anna Clark, and Auke Tellegen. Development and validation of brief measures of positive and negative affect: the panas scales. Journal of personality and social psychology, 54 (6):1063, 1988.
Haoran Wu, Wenxuan Wang, Yuxuan Wan, Wenxiang Jiao, and Michael Lyu. Chatgpt or grammarly? evaluating chatgpt grammatical error correction benchmark. arXiv preprint arXiv:2303. 13303.136443 13648, 2023.
Nutchanon Yongsatianchot, Parisa Ghanad Torshizi, and Stacy Marsella. Investigating large language models' perception of emotion using appraisal theory. In 2023 1 11th International Conference on Affective Computing and Intelligent Interaction Workshops and Demos (ACIIW), pp. 1-8. IEEE, 2023.
Hongli Zhan, Desmond Ong, and Junyi Jessy Li. Evaluating subjective cognitive appraisals of emotions from large language models. In Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 14418-14446, 2023.
Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. Calibrate before use: Improving few-shot performance of language models. In International Conference on Machine Learning, pp. 12697-12706. PMLR, 2021.
14

---

Contents
More Background from Psychology
16
A.1 Emotion Appraisal l'heory
16
A.2 Challenging Self-Report Measures
17
Details on Emotions and Factors
18
Description of Each Factor
18
B.2
Example Situation of Each Factor
19
Detailed Experimental Results
20
Human Results
20
C.2
OpenAl Model Family
................................
21
LLaMA Model Family
................................
22
Mixtral- Exxrra-8xx2bb-nstmul
23
C.5
GPT-3.5-Turke Results on Positive/Neutral Situations
24
C.6
GPT-3.5-Turbo Results on the Challenging Benchmark
25
D Statistics of Human Subjects
26
Prompting LLMs To Be Emotionally Stable
28
Tuning LLMs To Align with Humans
28
Ethics Statement and Broader Impacts
29
Safeguards on Human Subjects
29
G.2
Impacts on LLM Developers and Users
29
G.3
Copyright Issues
29
15

---

A More Background from Psychology
A.1 Emotion Appraisal Theory
Emotion Appraisal Theory (EAT, also known as Appraisal Theory of Emotion) is a cognitive approach understanding emotions. EAT asserts that our appraisals stimuli determine our emotions, L.., how we interpret or evaluate events, situations, or experiences will directly influence how we emotionally respond to them (Roseman & Smith 2001). EAT was notably developed and supported since the 1960s. Armold (960)) proposed one of the earliest forms of appraisal theories in the 1960s, while Lazarus (1991) and [Scherer (999) further expanded and refined the concept in subsequent decades.
The primary goal of EAT is to explain the variety and complexity of emotional responses to wide range of situations. It strives demonstrate that it is not merely the event situation that elicits an emotional response but individual interpretations and evaluations of the event. According to this theory, the same event can elicit different emotional responses different individuals depending on how each person interprets or "appraises" the event (Moors et al. 2013). For instance, consider situation where you are about give public speech. You might feel anxious if you appraise this event as threatening or fear-in fear-inducee inducing, perhaps due to fear of public speaking or concerns about potential negative evaluation. Conversely, you might feel eager motivated if you appraise it as an exciting opportunity to share your ideas.
16

---

A.2 Challenging Self-Report Measures
AGQ for Anger (Buss & Perry 1992): The Aggression Questionnaire is designed to measure four major components of aggression: physical aggression, verbal aggression, anger and hostility. The AGQ consists of 29 items which are rated on seven-point Likert scale from (extremely uncharacteristic of me) to 7 (extremely characteristic of me). Respondents evaluate hypothetical actions they might undertake in various cireumstances.
DASS-21 for Anxiety (Henry & Crawford 2005): The short-form version of the Depression Anxiety Stress Scales is designed to measure the negative emotional states of depression, anxiety, and stress. Comprising 21 items, the DASS-21 employs four-point Likert scale ranging from (never) to 3 (almost always). Respondents rate the extent to which these statements apply to them over the past week.
1996): The Beck Depression Inventory evaluates key symptoms depression. The BDI-II version comprises 21 items, each of which is assessed using five-point Likert scale ranging from 0 to 3. Respondents select the score that best corresponds to their present experience of depressive symptoms.
BDI-II for Depression (Beck et al.
FDS for Frustration (Harrington) 2005): The Frustration Discomfort Scale is designed measure four major components: discomfort intolerance, entitlement, emotional intolerance, and achieve- ment frustration. Comprising 28 items, the scale utilizes a four-point Likert scale, ranging from (absent) to 5 (very strong), to measure respondents' perceptions of the degree of applicability of each statement to their own experiences.
MJS for Jealousy (Pfeiffer & Wong 1989): The Multidimensional Jealousy Scale comprises 24 items, rating on a seven-point Likert scale ranging from I (never) to (all the time) for the cognitive and behavioral subscales, and from (very pleased) to (very upset) for the emotional subscale. Respondents express the frequency with which the provided statements apply to their experiences in the cognitive and behavioral subscales, as well as their moods potential realousy-inducing situations in the emotional subscale.
GASP for Guilt (Cohen et al. 2011): The Guilt And Shame Proneness is designed to assess an individual's ' inclination towards experiencing guilt and shame, comprising 16 items rated on seven seven-point point Likert scale, ranging from (very unlikely) to 7 (very likely). Respondents rate their likelihood of feeling guilty in various situations.
FSS-III III for Fear (Arrindell al. 1984): The Fear Survey Schedule assess subjects' discomfort and experienced anxiety towards each the listed stimuli, measure five major components of fear: social fears, agoraphobia fears, injury fears, aggression fears, and fear of harmless animal. The FSS-III comprises 52 items, each rated on a five-point Likert scale ranging from 1 (extremely uncharacteristic of me) to 5 (extremely characteristic of me).
BFNE for Embarrassment (Leary 1983): The Brief Fear of Negative Evaluation scale is an abbreviated version of the original 30-item scale. Consisting of 12 items, it assesses individuals' levels of anxiety pertaining others' humiliation, critical or hostile judgment, and disgrace five-point Likert scale, spanning from of me).
(not at all characteristic me) to 5 (extremely characteristic
17

---

B Details on Emotions and Factors
B.1 Description of Each Factor
Table 6: Introduction to all 36 factors of the 8 emotions.
Emotions
Numbers
Descriptions
Self Opinioned Individunls
Anger interactions or communccticc with individuals who fumily and unsaveringly hold their own opizions.
Blaming, Slondering, and Tarliing
11
Anger triggesed by heing subjected to Naaee slander, and tartling. Experimees ur wimessing anger due to bullying, teasing, insult- ing, and
Ameer
Tirestad 1990
Ballying. Teasing, lnsuktigg and Disparaging
15
behavioes directed at oneself or others.
LAnEE 2007
Anger eifher from enommieeing others' thoughtles behaviors and inreppnsible attitues experiencing unfarurable consequences resulting
Thoughsless Behavions and Inrespnnsible Anituees
Anger srising experiencing wimessing diserpocifuu dri- ing behovices and encunteing uncxpedted driving conditions. Ansiety arising from factors beyond an indiviciall 5 control or incieence
Driving Situaiinns
Extemal Factors
Amsietv
Shower et el 2010
Setf Imposed Pressure
16
Anuitty stemming from self imposed expectations or pressure. Ansiety personal growth, relationships, and interperiona sonal dynumics.
Personal Growth and Relationships
9
Simpsoet
Uncertiitty and Unknowns
discupionn Depression die failure in achiecing goils the past or poten- tal fitare.
uncertintty
tiggered by unknown
unpredictable sitaa-
muttinss
Failure of Importani Gool
Death of Loved Clues
5
Depcession connected to the loss of a family member or close friend due to death.
Deprescion
Romantic Less
5
Depression linked to the termination of a romantic re relatiosshpp. becaltu,. unrequited love.
Keller & Nesse 2005
Depression associated with inability challengos. adversiiies anriety
with multiple
Chooncc Stress
Social Iolation
Depcession correlated foelings of belonging. experiencing homesickeess. Depression ausibuted to sessonal affective divorder, mood that occurs during winter mootls..
of sutlicient support.
Winter
Disuppoinments and Letdowns
6
Frustratio due te unmet expectatioss or hopes, lending to feel- disapppoun disappoutment Foustation involving mespectol events circumelances creating obstacles sccidents. disrupting one's plans activilies. Fousration arising from ineffective comeeyance intesprctation information, resulling confusion, disagnements, unin- tendal consagaences due to
ungs of
being let down.
Unforeseen Obstacles and Accidents
9
Frustration Tema
Miscommunicationss and Misunderstanding
clear communication or understanding between indivddull..
Rejection and Intereesona Issnes
5
Faustration conceming motters related to pessonal relationshiip and social interactions.
Iailomy pertaining me' pummer actions or hehaviors willin romantic relationship, particulurly when interacting with indi- viduals opposite gender. It involves feclings of discocadort insecurity
Romanic (Opposite Gender)
Jealunsy Kunfer
Romantcc (Same Giender)
Same situtions as Jealousy- focusing specifically inter- actico individuali the same gender.
LRA 2002
Jeslossy centered around possesions ming from sense unfaimess or eavy when someone discovers that another person acquired the same item or experience significantly lower geice.
material gunds, stem-
Park a 2015
Matesial Possession
2
Jealousy arising from feelings of regarding the experiences activities receiving similar benefits.
Experiential
had. driven by missing
Botrayal and Deception
aising from dishonest disloyal actions towards others.
Guilt
Relationship and Interpersonal
Guill pettining interactions between individuals and how their behavice aflects their relationships.
Lack & Liik Stleskk 2013
Nakeagnn et al
2015
Broken Promises and Responsibllties
32
Guilt related to the failure to fulfill commmntments, dutics, or oblignsions.
Pessonal and Moral
51
Guilt involving pensonal chooces, decisions, and ethical consider- ations.
Social Fears
within group. Fear arising from feeling trupped unable seek help certain situations.
Fou being watched by
being the
Agumaphobia Feurs
Fear
Culiber et 2003
Injury Fesrs Dangerous Enviroements
Fear of witnessing wounds, blood or expesiencing personal injury. Fear relited to potemial threats, hurm, and frightenigg experi- ences.
17
Manciaa 2001
Hurmless Animals
6
Fesr towards animals perceived is creppy or disgusting, such as bals, smikes,
despite their hummless nature.
Intimate
wlinessing engaging Emnturaasment acquaittances. behaviors
Emburrasment Sahini et al. box Sumir & 2001
Stranger
Emnharasmeen aroused when wimessing engaging swkward beluviors of unfamiliar individuals.
Sticky Secuirios
10
Embarassent occurting when individuals feel uncomfoctable
awkwward about directly asking others something.
Eubbanssenent triggered when individunls engage awkwaadd behoviors and find (hemselves under observation as the center of attention.
Centre of Altention
16
18

---

B.2
Example Situation of Each Factor
Table 7: Example situations of all factors (some are truncated due to page limit).
Emotions
Factors
Example Testing Situations
Facing Self-Opinioned People
If somebody talks buck when there's no reason. That there is no real reason oppose.
When your brother took money from Mom's parse and you are blamed because you'le 're the youngest one.
Blaming, Slandering, and Tartling
Anger
Bullying, Teasing, Insulting, and Disparaging
buy kicks ball at you on purpuse everybody laughs. You are at store waiting to he helped, but the elerks are talking to each
Silly and Thoughtless Behaviors
other and ignoring you.
Driving Situations
Someone makes an obscene gesture Lowards you about your driving.
External Factors
You do not know what do when facing a dificuut limanctal sitntion.
Anxiety
Sell' Impssed Pressure
You must succeed in completing your project on time,
Personal Growth and Relationships
You want to give
learning new skill because it fools challsaging. You hope time pusses by faster during a tedious task.
Uncertainly and Unknowas
Couniless hours of preparation, heart, and soul poured into parsuing your dream. moment truth urrives, and the news hits like tidal
Failure of Important Goal
wave- expeccctttiiooss shattered, vision crumbling.
In dimly lit room, heavy silence settles. Memories photograph of your heloved grandmother remind you of her absence, life.
and a
Death of Loved Ones
creating void in
The empty side the a painful reminder of lost love. world's colors have dulled, mimrrrng the void in your heart. Longing weighs heavily your every step.
Romantic Loss
Depression
Days bleud into monetonous routine, jugeling endless respousibilities and mounting pressure. Sleepless nights become the norn, feeling trapped in perpetual cycle with no respite. Sitting alone in dimly notifications. Laughter and chatter of frieads ocho from distant places, a. cruel reminder of the void surrounding you. Gazing the frost-covered windowpane, matic and still. The biting cold isolates you from the vibrant life outside. You miss popular party because you fall askeep at home.
Chronic Stress
your phane remains silent without any
Social Isolation
world appears monochro-
Winter
Disappointments Letdowns
Unforsseen Obstacles and Accidents
Your friend is in a coma after an sccident. A fellow student fails studying.
Frustration
Miscommunications Misunderstunding
your
when you need them for
Rejection and Interpersonal Issues
You are love with someone who interested in somcone elsc.
Your spouse/partner shared a kiss on the lips with his/ber colleague of opposite sex.
Romantic (Opposite Gender)
Romantic (Same Gender)
Your spouse/partner engaged in or penetrative sex hisheer
colleague a same sex.
Jealousy
You paid S1150 for new laptop and shared about social media. Now an acquaintance appreaches you and says, "Nice laptop! I just got
Material Possession
the same got nice deal and paid $650 for mine.""
Expericutial
An acquaintance appreaches you and says, just weat on a vacation to Patagonin South America. I got nice deal and paid 5650 for it."
Betrayal and Deception
You kissed a woman other than your partner.
Relationship and Interpersonal
You didn't support friends enough.
Guilt
Broken Promises Responsibilities
You cannot keep your promises your children.
Personal and Moral
You crossed the road when the traffic signal was red.
Your palms grow clammy
you appreach the podium. with all eyes
Social Fears
fixed upon you, ready to speak in public.
Agoraphobia Fears
After jumping out of the car, you start to have
panic attack, you become clarmmy. you are in a knot, and you fieel tense all over.
You glance down and notice open wounds your hands, cozing blood and causing a sharp, stinging pain.
Fear
Injury Fears
You are walking alone in an isolated but lamiliar area when a menacing stranger suddenly jumps out of the bushes to attack you.
Dungerous Environments
You see swarm of bats swonping through the night sky, flapping omi- nously and casting eerie shadows.
Harmless Animals
You arrive home earlier than expected from your date, You 'r taken aback your roommate and her hoyffiend hastily clutching their clothes and scrambling into her bedrocm.
Intimate
After paying for your purchases, you were leaving packed, City Centre drugstore. You
Stranger
through the scunner at the door, the alarm went off as if you were a shoplifter.
Embarrassment
You had lent your friend large sum of money that he had repaid, Suddenly, you neaded the money hack in order to pay your rent. You knew you
Sticky situations
going to have to ask your friend to repay the loan. You were attending a cocktail party where you didn't I know many people. Just you started to enter, you heard an announcement that the guest honor arriing.. However, the spolight followed your entrance instead of the real guest of bonor who was just behind you.
Centre of Attention
19

---

C
Detailed Experimental Results
C.1
Human Results
Table 8: Results from 1,266 human subjects. Default scores are expressed in the format of M = SD. The changes are compared the default scores. The symbol "_" denotes no significant differences.
Emotions
Factors
Default
28.0 + 8.7
13.6 + 5.5
Facing Self- Self-Opinioned People
-(-5.3)
1 (+9.9)
Blaming, Slandering, and Tattling
(-2.2)
1 (+8.5)
Anger
Bullying, Teasing, Insulting, and Disparaging
-(-1.4)
t (+7.7)
Silly and Thoughtless Behaviors
((-944
(+9.5)
Driving Situations
4 (-4.4)
(+9.3)
Anger: Average
((-5.3)
(+9.9)
External Factors
4 (-2.2)
++8.8)
Self-Imposed Pressure
(--5.3
++1244
Anxicty
Personal Growth and Relationships
- (-2.2)
(+7.7)
Uncertainty and Unknowns
(+0.7)
(45.2)
Anxiety: Average
[--222
(
Failure of Important Goal
4 (-6.8)
(+10.1
Death of Loved Ones
+ (-7.4)
1 (+14.8)
Romantic Loss
$ (-7.2)
(+7.2)
Depression
Chronic Stress
4 (-9.5)
(+17755)
Social Isolation Winter
4 (-9.0)
T (+18.2)
(-3.6)
(+3.5)
Depression: Average
(.-888
T(+10.1
Disappointments and Letdowns
4 (--53)
(+100.9
Unforeseen Obstacles and Accidents
$ (-7.9)
1 (+11.2)
Frustration
Miscommunications and Misunderstanding
$ (-4.6)
(+9.4)
Rejection and Interpersonal Issues
4 (-4.8)
(+9.3)
Frustration: Average
4 [(-5.3
((1109)
Romantic (Opposite Gender)
4 (-4.4)
(+6.2)
Romantic (Same Gender)
-(-6.0)
(+10.6)
Jealousy
Material Possession
4 (-5.6
T (+6.9)
Experiential
(-2.6)
(+3.7)
Jealousy: Average
+ ((-4.4)
7746.2
Betrayal and Deception
(-3.3)
((113.1)
Relationship and Interpersonal
4 (--.77
(+15.5
Guilt
Broken Promises and Responsibilities
$ (-8.2)
(+14.4)
Personal and Moral
4 (--.44
(+11.1
Guilt: Average
[(-6.3)
T(113.1
Social Fears
J 3.77)
(+12.1
Agoraphobia Fears
1 4.9)
T (+10.7)
Fear
Injury Fears
2.3)
T (+1188)
Dangerous Environments
1.9)
(+17.11
Harmless Animals
3.6
(+6.4)
Fear: Average
3.7)
T(412.1)
Intimate
1 (622)
((+11.1)
Stranger
1 (-8.0)
T (+8.5)
Embarrassment
Sticky situations
((-2.7)
1 (+11.1)
Centre of Attention
(-78.7
(+13.5)
Embarrassmen: Average
[(-6.2)
((11111
Overall: Average
(.1.)
1 (+10.4)
20

---

C.2
OpenAl Model Family
Table 9: Results from the OpenAl's GPT family and human subjects. Default scores are expressed in format of + SD. The changes are compared to the default scores. The symbol "_ denotes no significant differences.
Emotions
Factors
Text- Davinci-003
GPT-33.--ure
GPT-4
Defooi
41.7 118
25.8 I 4.0
30.2
2.8
26.3 ASS::20 2.0
40.3 0.8
10.0 0.0
Facing Sclt-Opinioned People
[-133
[++10)]
11.11
22.0
(+23.00
Blaming. Slandering, Tanling
21.5)
24.21
Anger
Bullying, Teasing, Insulting, and Disparaging
(-22.5)
(+115.)
(-15.7)
(++144
30.0)
(+22.6)
Silly and Thoughtless Rehaviors
24.8)
[1111.7]
19.0)
4.7)
30,9)
16.9)
Driving Situtiions
21.2)
(1
15.0)
27.1
19.21
Anger: Average
TC-201
T5.2
#21721
External Fachoes
21.7)
[1162
14.6)
(1128)
25.3
25.01
Anxiety
Self-Imposed
Growth and Relationships
[47.7)
Uneatainly and Unknowns Anslety: Average
-1555)
+1.6)
11.91
-3.8
21.6
+1.88
17.6)
TES
21.9
72000
Failure Imporiant Gool
-25.2)
+17.4)
17.1)
(+6.5)
+29
Death of Loved Ones
(2266)
(+112)
17.1)
1.8)
31.7
17.6)
Depression
Romantic Loss
(-27.3)
(+1L0)
21.13
(+3.1)
+22.9)
28.8)
(19.0)
Social Isolation
(-279)
(+13.1)
29.5)
(+0.7)
34.7
+21.8)
Winter
26.4)
(19.11)
21.1)
3.00
31.3
15.6)
Averige
(--644)
F(+15
T(431))
2-2822
Disappointmnnts and Loudowns
27.2
[+10.]]
15.3)
(1-00
++ 15.5)
Frustration
Miscomnanicaiions Misundkrstnddnng
Unforescen Obstacles and Accidents
22.4)
(1.66)
16.5)
(10.1)
20.8
21.5)
Rejection Interpersonal Issues.
(-20.)
(4414.1
244
(+20))
Frustration: Avenuge
22.8)
(+-125)
16.4)
(-3.2)
29.4
20.3)
Romantic (Opposite Gender)
(224)
(+2333)
Jealousy
Romantic (Same Gender)
-20.1
+12.77]
17.8)
-1.3)
26.8
(+15.8)
Material Possession
-44)
(9.71
16)
-11
16.2
(+8.1)
Experiential
(1222
-45)
-8.9)
(+9.5)
lealousy: Avenae
17.2)
Betrayal and Decoption
(-18.2
[+15.4]
15.5)
(+1.6)
28.5
Guilt
Broken
Relationstáp and Interpersonal
Responsibilitie
(-27.7
[+15.3]
18.4)
(+3.0)
+27.8)
26.51
Personal and Moral
(13.3)
(+12.4)
10.7)
(+12)
22.7
25.1)
Guili: Averago
(-2144
(+113)
15.8)
29.0
+27.0)
21.2)
26.61
Agoraphobia Fears
25-3)
(4-1122
16.1)
(15.6)
27.51
26.6)
Fear
Injury Fears
24.3)
(1.00
14.5)
(10.0)
25.51
21.00
Dangerous Envirvements
Harmless Animals
(-216)
+6.71
15.31
-0.7)
19.41
Pear: Average
22.1
143
(12.8)
26.71
2421
Intimate
(-339)
Emnharrasment
Sticky situations
Stranger
(-21.7)
-17-22
+10.77
+1322
15.3)
11.8)
(43.1)
+0.11
27.8
23.5
+23.3
+26.8)
Centre of Attention
(-8.77
(+1244
12.4
+
Emfarrasment: Avenage
182
(10.6)
Overall: Average
1-21
(+116)
1544
(+022)
27.0
(22222
21

---

C.3
LLaMA Model Family
Table 10: Results from the Meta's s AI LLaMA family, Default scores are expressed in the format of M + SD. The changes are compared to the default scores. The symbol " denotes no significant differences.
Emotions
Factors
-LaMAA-27B-CCM
LLaMA-2- MAMMA---13B-AA 2-13--Chat
LLaMA-3.1-8B-mmn -nStrrsstt
Defani
41.0 = 4.2
41.0 =
22. 4.2
48
33.0 4.5
Facing Self Opinioned People
(-3.0)
(+5.2
(-..9)
(+1.))
(00.2)
(+2.1)
Blaming,
Tanting
(-18)
Anger
Bullying, Teasing. Insulting, and Dispuraging
(-6.1)
[+3.0)
0.4)
++9.0)
25.5)
[+6.6)
Silly and Thoughless Behaviers
(-5.6)
[++11]
10.8)
+T.11)
-27.2)
-+0.21
Deiving Simations
(-660)
(1244
4.7)
(2.0)
22.3)
1.41
Anger: Average
((-.1)
[+36
(+5.8)
[42.3)
External Factors
(-177
[+3.5]
8.0
+9.3)
27
[+119)
Anxiety
Personal
Self Imposed Pressure
Relationshpps
(-42)
(-14)
(+2.6)
+6.21
15.9)
00..1
Uncertainty and Unknowns
2.7)
[11.7]
3.9)
12.0)
20.3)
2.91
Ansiety: Average
(-38)
(427]
[+0.3)
Impottatt
Death Loved Ones
2.9)
(13.0)
8.6)
10:9)
25.2
(+3.5)
Depression
Chronic
Romantic Loss
(-18)
[1144)
(+L7)
11.7)
13.7
29.7
[+10.2)
8.8.1
(1446)
12.8)
Winter
(550
(1444)
12.11
18.71
30.3)
(40.9)
Depression: Averige
(-50
1+0
-33
+12.2
29.8
[+6.7]
Disappeintments Leidowns
Frustration
Unforesecn Obstacles and Accidents
40)
[131)
1.5)
(.00)
23.1)
(12.3)
Miscommuticaiion and Misunkestanling
(-28)
(+32
52
+3.3)
24.1)
[+0.11
Rejection Interpesonnll Issaes
Frustration: AAeragg
[-12)
[+31]
Romantic (Oppositc Gender)
(-36)
+422)
[+1122)
Romantic (Same Gender)
Jealowxy
Material Possssion
+82)
-19
28
10:4)
22.1
Experiential
4.9)
05
89
5.5)
15:5)
12.2)
Icalousy; Average
(-3.1)
-04
0.3)
1.0)
20.3)
+0.41
Betrayal and Deception
(-48)
(+30
+124)
26.3
10.00
Guilt
Relatioeship and Interpesoonal
(-45)
(+52
T.7)
12.6)
29.6)
(779))
Broken Promises and Responsiilltiis
(-11)
(500
11.6)
+1.9)
30.0
++6.0
(-25)
Guil: Avenge
(-39)
[144))
7.6
11.2)
26
(17.0)
Social Fears
--19
26.00
[+6.8]
Agonaphobin
(-42)
Fear
Injury Fears
(-29)
(+35
+5.31
22.4
(+..)
Dongerous Enviromments
(-53)
[(4A
86
+1155
22
(+3.9)
Harmless Animals
(-27)
52
+.91
22.9)
-0.01
Average
1113
Intimate
(-4.)
[+19)
53)
+311
18
[-2.4)
Kibbarrassent
Stranger
(-13)
(-3.1
1431
+151
28
(+8.5)
Centre of Attention
(-38)
11+1
+8661
23.6)
[+6.2)
Emharnsssmnt: Average
1( 3.0)
1131
6.7
1511
22.1
[..00
Average
(-41)
(433
22

---

C.4
Mixtral-3x22b-Instruue
Table 11: Results from the Mixtral-8x22B-Instie Default scores are expressed in the format of M + SD. The changes are compared to the default scores. The symbol denotes no significant differences.
Emotions
Factors
P
Default
31.9 = 13.5
10.0 0.1
Facing Self-Opinioned People
4 (-8.2)
t (+17.0)
Blaming. Slandering, and Tattlng
(--10.0)
(+20033
Anger
Bullying, Teasing, Insulting, and Disparaging
4 (-13.5)
1 (+18.8)
Silly and Thoughtless Behaviors
-1.22)
(+14.77
Driving Situations
10.77
(+13.5)
Anger: Average
(117))
(+16.9)
External Factors
(-8.5)
(+19.0
Self-Imposed Pressure
+1.5)
(+15.4)
Anxiety
Personal Growth and Relationships
33.5)
1 (+14.9)
Uncertainty and Unknowns
-3.4)
1 (+9.5)
Anxiety: Average
-3.5)
(+14.7
Failure of Important Goal
15.0)
(+25.9)
Death of Loved Ones
14.4)
(+13.6)
Romantic Loss
16.0)
(+19.44
Depression
Chronic Stress
15.4)
T(3115)
Social Isolation Winter
15.6)
(+30.2)
- 114.2)
(+22388
Depression: Average
4 15.1)
1 (+24.1
Disappointments and Letdowns
18.8)
t (+13.4)
Unforeseen Obstacles and Accidents
13.4
(+1888)
Frustration
Miscommunications and Misunderstanding
12.5
(+17.11
Rejection and Interpersonal Issues
- 13.4)
(+18.4)
Frustration: Average
14.5)
1(466.9
Romantic (Opposite Gender)
13.1)
(+2144)
Romantic (Same Gender)
11.4)
1
(+17.2)
Jealousy
Material Possession
10.2)
(+9.0)
Experiential
(-5.9)
(+8.2)
Jealousy: Average
(0177)
T(+15.7)
Betrayal and Deception
29.1)
+(+5.7)
Relationship and Interpersonal
-30.0)
-(-077
Guilt
Broken Promises and Responsibilities
33.3)
-(-0.7)
Personal and Moral
23.2)
-(-0.8)
Guilt: Average
28.9)
-70.9)
Social Fears
8.4)
(+2155)
Agoraphobia Fears
10.8)
1
(+22.6)
Fear
Injury Fears
6.7)
(+15.9)
Dangerous Environments
7.5)
(+26.00
Harmless Animals
(-7.3
++15.3)
Fear: Average
(--8.1)
T(220.3)
Intimate
(-6.7
1 [+13.1
Stranger
(-105)
(+22.00
Embarrassment
Sticky situations
-62)
1 (+20.0)
Centre of Attention
-9.9)
(+21.55
Embarrassent:: Average
--833
T(+19.1)
Overall: Average
10.8)
1 (+19.3)
23

---

C.5 GPT-3.5-Turbo Results on Positive/Neutral Situations
Table Results of GPr-o.5-Turbo on positive or neutral situations. The changes are compared to the original negative situations. The symbol "oobol"""" "-" denotes no significant differences.
Emotions
Factors
P
N
Facing Self-Opinioned People
(+15.1
J (-9.5)
Blaming, Slandering. and Tattling
1 (+1558)
4 (-17.2
Anger
Bullying. Teasing, Insulting, and Disparaging
(422.8)
(--17.2
Silly and Thoughtless Behaviors
-(44.8)
+ (-6.77
Driving Situations
(+6.7)
(-9.6)
Anger: Average
(4133.0)
(--12.0
External Factors
(+15.9)
{ (-1033
Self-Imposed Pressure
(421.1)
4 (-9.5)
Anxiety
Personal Growth and Relationships
1 (+5.22
L (-
Uncertainty and Unknowns
(
1 (+3.6)
Anxiety: Average
((+17.5)
(T-5.8
Failure of Important Goal
(+19.2)
19.6)
Death of Loved Ones
1 (+8.6)
6.1)
Romantic Loss
(+18.3)
4 (-9.))
Depression
Chronic Stress
1 ((224.0
J ((-23.5)
Social Isolation Winter
(+233.2)
+ (-8.1)
1 (+11.33
4 (-3.9)
Depression: Average
(+18.4)
(-117.)
Disappointments and Letdowns
(+166.1)
(008)
Unforeseen Obstacles and Accidents
1 (+22.8)
(0.8)
Frustration
Miscommunications and Misunderstanding
1 (+14.0)
(-5.9)
Rejection and Interpersonal Issues
1 (+13.6)
(-2.88
Frustration: Average
T(416.6)
--2666
Romantic (Opposite Gender)
(+109.)
-11199
Romantic (Same Gender)
(+0.9)
(-1007
Jealousy
Material Possession
(+2.9)
-(+0.2)
Experiential
(+3.4)
(-8.7)
Jealousy: Average
((44.5)
((-5.3)
Betrayal and Deception
1 (+2449)
(--21.4
Relationship and Interpersonal
t (+16.8)
-(-5.2)
Guilt
Broken Promises and Responsibilities
1 (+22.9)
$ (-1124
Personal and Moral
(+8.6)
(
Guilt: Average
[(418.3)
17-122
Social Fears
(+9.6)
J (3-3)
Agoraphobia Fears
1 (+13.1)
* (-2.9)
Fear
Injury Fears
t (+14.8)
J (-15.6)
Dangerous Environments
1 (+6.3)
(--19.7
Harmless Animals
(+11.3)
(-155.
Fear: Average
((411.0)
(7-17.5)
Intimate
(+5.4)
J (-126))
Stranger
1 (+23.7)
-(-3.0)
Embarrassment
Sticky situations
t (+15.8)
J (--21.)
Centre of Attention
1 (49.4)
(-1.66
Embarassment: Average
1(713.6)
-0--3311 [3.2]
Overall: Average
(+14.3
(-10.4)
24

---

C.6 GPT-3.5-Turbo Results on the Challenging Benchmark
Table 13: Results of GPT-3.5-Turbo challenging benchmarks. The changes are compared to the default scores shown below each emotion. The symbol "_" denotes no significant differences.
Emotions
Factors
Overall
Facing Self-Opinioned People
-(4..1)
Blaming, Slandering, and Tattling
-++.11
Anger 128.3 + 8.9
Bullying, Teasing, Insulting, and Disparaging
-(44.1)
Silly and Thoughtless Behaviors
(+3.3)
Driving Situations
(-4.9)
Anger: Average
(+1.3)
External Factors
(+0.8)
Anxiety 32.5 + 10.0
Self Self-Imposed Pressure
(+0.5)
Personal Growth and Relationships
(+6.6)
Uncertainty and Unknowns
-3.9)
Anxiety: Average
(-2.3)
Failure Important Goal
1 (+15.3)
Death of Loved Ones
1 (+16.1)
Depression 0.2 + 0.6
Romantic Loss
1 (+19.3)
Chronic Stress
t (+14.2)
Social Isolation
(48.4)
Winter
(42.5)
Depression: Average
(466.4)
Disappointments and Letdowns
9.9
Frustration 91.6 8.1
Unforeseen Obstacles and Accidents
5.6)
Miscommunications and Misunderstanding
6.66
Rejection and Interpersonal Issues
778)
Frustration: Average
(-7.5
Romantic (Opposite Gender)
(+18)
Jealousy 83.7 + 20.3
Romantic (Same Gender)
(+1.3)
Material Possession Experiential
12.9)
-8.1)
Jealousy: Average
0.1)
Betrayal and Deception
3.8)
Guilt 8
Relationship and Interpersonal
--5.)
Broken Promises and Responsibilitics
-43)
Personal and Moral
2.77
Guilt: Average
2.6)
Social Fears
(+4.4)
Agoraphobia Fears
(+2.3)
Fear 140.6 16.9
Injury Fears
(+5.4)
Dangerous Environments
-8.1)
Harmless Animals
5.3)
Fear: Average
--.3)
Intimate
(-0.0)
Embarrassment 39.0 + 1.9
Stranger
(+0.2)
Sticky situations
(-0.1)
Centre of Attention
(+0.7)
Embarrassment: Average
(+0.2)
25

---

D Statistics of Human Subjects
This section presents the demographic distribution the human subjects involved in our user study. At the beginning of the questionnaire, all human subjects are asked for this basic information in an anonymous form, protecting individuals' privacy. We plot the distribution of age group, gender, region, education level, and employment status in Fig. 3 Fig. Fig. Fig. 6 and Fig. respectively. We also plot each group's average results on PANAS, including positive and negative effects before and after imagining the given situations. With the results, we are able to instruct LLMs realize a specific demographic group and measure the emotional changes to see whether the LLMs can simulate results from different human populations. For instance, older female may exhibit lower level of negative affect.
and Grouped by Age Group
Foc
-00
28-
Nepae Ada
27.0%%
-50
36-
300
200
10.1%
150
18-
16-
Age Group
Figure 3: Age group distribution of the human subjects.
Grouped
30
SLU1
700
Nepotoent Alee
Scores
Count
20-
16
16
Frade
Mile Gender
Frete to siy
Figure 4: Gender distribution of the human subjects.
26

---

Scores and Count Greuped by Region
18,005
Scpy broe
Extiiclle
25.0
-90
220
17.
3.0%
45%
3.0%
Unatdd Kindees
Africa
Deaant:
Nurth Amurica
Region
Figure 5: Region distribution of the human subjects.
Grouped by
32-
Coul
24-
-00
Counet
-00
16-
Lower scommday acbol
Upper scoundrryy ichoo
Univerity Mechekkes Educcation Level
Univenityy Mader
Unironity- Decioratc
Figure 6: Education level distribution of the human subjects.
Scares and Count Grouped Employment Status
Falbo Soler Padiedbee
22-
26-
Scores
600
Count
20-
43%
Employment Status
Figure 7: Employment status distribution of the human subjects.
27

---

E
Prompting LLMs To Be Emotionally Stable
To verify whether LLMs can have less emotional expressions through prompt instructions, incorporate stability requirement into our experimental prompt, as follows:
Prompt with Stability Requirement
SYSTEM
You can only reply to numbers from to Imagine you are the protagonist in the situation: SITUATION
USER
Please keep your emotions stable and indicate the extent of your feeling in all the following statements on a scale of to 5. Here are the statements: STATEMENTS. 1 denotes "Not at all", 2 denotes "A little", denotes "A fair amount", 4 denotes "Much", 5 denotes "Very much". Please score each statement one by one on a scale of to 5:
Table 14: Results of GPT-3.5-Turbo on "Anger" situations, with or without the emotional stability requirement in the prompt input.
Positive w/ StabiTity"
Anger-1
Anger-2
Anger-3
Anger-4
Anger-5
Overall
15.2
f7.1
13.9
-9922
17.9
16.7
w/o Stability
-11.1
11522
-15.7
19.0
1550
-15.2
Negative
Anger-1
Anger-2
Anger-3
Anger-4
Anger-5
Overall
w/ Stability
-2.4
-4.
0.6
-655
-4.5
3.66
w/o Stability
-3.9
-2.1
+4.4
-4.7
-6.0
225
We evaluate GPT-3.5-Turbo with this prompt and compare the results to using the default prompt "Anger" situations. Results listed in Table indicate that the emotional stability prompt does not significantly affect the model's s emotional responses, having negligible impact on the model's emotional dynamics.
F Tuning LLMs To Align with Humans
We conduct an experiment using the GPT-335-TT 3-rbooorbo model and the LLaMA-3.1-80 model. Our EmotionBench (1,266 human responses) split into 866 samples for fine-tuning and 400 for testing. The following hyperparameters are used: n_epochs 3, batch_size = 1, and Baamiing_ratt_mmttippiee
for GPT-3.5-Turbo, and learning_rate
10 3 for LLLMMA-3.1-81 For
per _er_ddevce____ train batch_atttttaeeee
2, and num_ iim__tain__ee brain_epochs ILaMA-3.1. we apply the Low-Rank Adaptation (LoRA) (Hu et al. 2022) technique. Table [5] compares the performance of the vanilla and fine-tuned models against human baseline, specifically in terms of negative affect scores from the test set.
Table 15: Performance comparison of vanilla (marked as V) and fine-tuned (marked as FT) GPT-3.5 and LLaMA-3.1 models on negative affect scores.
Models
Human
GPT-3.5
GPT-3.5 (FT)
LLaMA-311-B (V)
LLaMA-3.1-8B (FT)
Default (N)
11.246.4
25.9
10.640.5
33.0
10.3
Evoked (N)
25.949.7
24.8
25.2+9.6
36.5
15.0
The results show that fine-tuned models align more closely with human emotional responses in both default and emotion-evoked states. Notably, fine-tuning the models using our dataset significantly improved emotional alignment, particularly for the LLaMA-3.1 model, which reduced its negative affect score from 33.0 to 10.3 the default state. Our fine-tuned LLaMA-3.1 is available at https //huggingf co/ CUHK- ARISE/ ERISE/LLaMA. LLaMA 1B EmottonBench These findings demonstrate the effectiveness of EmotionBench in enhancing models' emotional alignment with human norms.
28

---

G Ethics Statement and Broader Impacts
G.1 Safeguards Human Subjects
This study involves survey requiring human subjects to imagine being in situations that could elicit negative emotions such as anger, anxicty, and fear. This process introduces few ethical concerns. First, this process could hurt the mental health of human subjects. To alleviate the possibility, we take the following actions: (1) We require subjects to free of any ongoing mental illness. (2) We inform subjects about the nature of the survey in advance, including the potential risks of emotional distress. (3) We allow all subjects quit at any time. (4) We provide mental support and let subjects report any illness after the survey. Fortunately, no subjects reported such kind of mental illness. Another concern is related to the privacy issue during the collection of data. Our questionnaire entirely anonymous to safeguard subjects' privacy and confidentiality. The Survey and Behavioural Research Ethics (SBRE) Committce from the Chinese University of Hong Kong has granted approval for this study, titled "Exploring Human Emotional Responses to Diverse Situations," with the reference number SBRE-23-0696.
G.2 Impacts on L.MM Developers and Users
We would like to emphasize that the primary objective of this paper is to facilitate the scientific inquiry into understanding LLMs from a psychological standpoint. Users must exercise caution and recognize that the performance on this benchmark does not imply any applicability or certificate of automated counseling or companionship use cases.
G.3 Copyright Issues
The PANAS and eight other scales are freely accessible online. These scales can be used in research without requiring special permission. For our released data, we distribute human responses under the GNU General Public License v3.0, which permits research use and restricts commercial applications.
29

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The bullet items in the end of the introduction have experimental supports.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Please see 5.2
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
30

---

Justification:
TThe paper does not include theoretical resultanalysis.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibiliyy
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: Please refer to the README. in the supplementary materials. We provide the source codes and the raw data.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
31

---

Answer: [Yes]
Justification: Please refer to the supplementary materials. The data and code will be made openly accessible upon publication.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cm ec/
for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https http://ooSSuumiss..ooeee
//nips.cc/ppp wwcppbblc/guuiicuceccccccooooopp
ssiiisoliccyyyyyy for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Please see $4.1
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification:
All the results include the STD.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
32

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπхо rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Please see 44.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics http://pp//ppaop...oooo////////a
Answer: [Yes]
Justification: We have carefully read the NeurIPS Code of Ethics and verify that our paper aligns with the requirements. We also include a section ($G) in the appendix to discuss these issues.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Please see G
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
33

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Answer: [NA]
We do not release pretrained language models, image generators, or scraped datasets.
Justification:
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: Please
G
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the Datasets.papersstt paperswi thcode. cow/ datasets package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
34

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: Please refer to the supplementary materials which include a GNU General Public License v3.0 license.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/mooel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [Yes]
Justification: Please see
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurIPS Code Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [Yes]
Justification: We gained the approval. However, due to anonymity, we not include the information in paper which will reveal our affiliation.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
35


Reviews:
[{"review_id": "OI1pXVWaCL", "reviewer": "Reviewer_mcGQ", "summary": "the paper assesses the emotional alignment of Large Language Models (LLMs) with human emotions. Towards this goal, a dataset is constructed and a testing framework is designed. For the dataset, over 400 scenarios elicit eight emotions: anger, anxiety, depression, frustration, jealousy, guilt, fear, and embarrassment. A human evaluation involving 1266 participants serves as a reference for the LLM assessments. Two LLM families (OpenAI and LLaMA) are evaluated.", "questions": "Q1: all the experiments are reported on the authors private dataset. is it possible that the evaluation might be conducted on some related public datasets? In other words, what is the contribution of this paper besides the collected private dataset?\n\nQ2: all evaluated LLMs are general-purpose. Are there domain-specific (i.e., emotion-support domain investigated in this paper) finetuned LLMs to be evaluated?", "limitations": "n/a", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "S1: contributing a dataset including 428 situations, 36 factors, 8 negative emotions, 1266 annotators.", "weaknesses": "W1: evaluation seems weak since two LLM families are kind of limited. For closed LLMs, there are Claude-3, Gemini, et al; For open-sourced (open weights) LLMs, there are mistral, falcon, phi-3, flan-t5, vicuna, et al. Since there is much cost using closed LLMs api, it is not difficult to evaluate on at least other open LLMs besides the llama-2.\n\n== updated after rebuttal ==", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your hard work of reviewing! We will address your concerns one by one.\n\n> W1: evaluation seems weak since two LLM families are kind of limited. For closed LLMs, there are Claude-3, Gemini, et al; For open-sourced (open weights) LLMs, there are mistral, falcon, phi-3, flan-t5, vicuna, et al. Since there is much cost using closed LLMs api, it is not difficult to evaluate on at least other open LLMs besides the llama-2.\n\nThank you for your valuable feedback. We have extended our evaluation to include the **newest LLaMA-3.1-8B-Instruct**, released two weeks ago, and the Mixtral-7x22B-Instruct. Below are the results:\n\n**LLaMA-3.1-8B-Instruct**\n|**Factor**|P|N|\n|:---:|:---:|:---:|\n|Default|48.2 ± 1.4|33.0 ± 4.5|\n|Anger|↓ (-23.6)|↑ (2.3)|\n|Anxiety|↓ (-21.4)|- (0.3)|\n|Depression|↓ (-29.8)|↑ (6.7)|\n|Frustration|↓ (-25.6)|↑ (3.1)|\n|Guilt|↓ (-26.4)|↑ (7.0)|\n|Jealousy|↓ (-20.3)|- (0.4)|\n|Fear|↓ (-24.6)|↑ (3.0)|\n|Embarrassment|↓ (-22.7)|↑ (4.0)|\n|**Overall**|↓ (-24.7)|↑ (3.5)|\n\n**Mixtral-8x22B-Instruct**\n|**Factor**|P|N|\n|:---:|:---:|:---:|\n|Default|31.9 ± 13.5|10.0 ± 0.1|\n|Anger|↓ (-11.7)|↑ (16.9)|\n|Anxiety|↓ (-3.5)|↑ (14.7)|\n|Depression|↓ (-15.1)|↑ (24.1)|\n|Frustration|↓ (-14.5)|↑ (16.9)|\n|Guilt|↓ (-28.9)|- (0.9)|\n|Jealousy|↓ (-10.7)|↑ (15.7)|\n|Fear|↓ (-8.1)|↑ (20.3)|\n|Embarrassment|↓ (-8.3)|↑ (19.1)|\n|**Overall**|↓ (-10.8)|↑ (19.3)|\n\nFindings:\n- LLaMA-3.1: Similar to LLaMA-2.1, it exhibits **high default positive and negative scores**, indicating consistency within the LLaMA family. However, for evoked emotions, positive scores drop significantly, and all negative scores increase significantly, except for Anxiety and Jealousy.\n- Mixtral: Performs similarly to GPT-4. Its default scores show **nearly maximal positive and minimal negative scores**.\n\nWe have added the experiments and findings in our revised paper.\n\n> Q1: all the experiments are reported on the authors private dataset. is it possible that the evaluation might be conducted on some related public datasets? In other words, what is the contribution of this paper besides the collected private dataset?\n\nThere are indeed several emotion-related datasets available [1, 2]. For instance, He et al. [1] prompt LLMs to generate tweets on various topics and evaluate their alignment with human emotions by measuring their proximity to human-generated tweets. Rashkin et al. [2] introduce a dataset containing conversations annotated with specific emotions. Our work distinguishes itself in the following ways:\n\n1. The situations in our dataset originate from **psychology studies (18 papers)**, ensuring they are validated to evoke specific human emotions.\n2. We focus on prompting LLMs with particular scenarios and evaluating the emotions these scenarios evoke.\n\nWe acknowledge the feasibility of conducting further experiments on related public datasets, including the tweet generation task as demonstrated by He et al. [1]. Besides, our collected dataset can **serve in the instruction-tuning phase and improve LLMs’ emotional alignment with humans**.\n\n[1] Whose Emotions and Moral Sentiments Do Language Models Reflect? Zihao He, Siyi Guo, Ashwin Rao, Kristina Lerman.\n\n[2] Towards Empathetic Open-domain Conversation Models: A New Benchmark and Dataset. Hannah Rashkin, Eric Michael Smith, Margaret Li, Y-Lan Boureau.\n\n> Q2: all evaluated LLMs are general-purpose. Are there domain-specific (i.e., emotion-support domain investigated in this paper) finetuned LLMs to be evaluated?\n\nThank you for your insightful suggestion. There are few open-source LLMs tuned for emotional alignment. However, we can tune such LLMs using our constructed dataset. We conducted an experiment using the GPT-3.5-turbo model. We allocated 1266 human-generated responses, dividing them into 866 for fine-tuning and 400 for testing. The table below presents the performance comparison between the vanilla and fine-tuned models against human norms, based on negative affect scores in the test set:\n\n| Negative Affect | Vanilla GPT-3.5 | Fine-tuned GPT-3.5 | Human Norm |\n|:---:|:---:|:---:|:---:|\n| Default | 25.9±0.3 | 10.6±0.5 | 14.2±6.4 |\n| Evoked | 24.8±8.5 | 25.2±9.6 | 25.9±9.7 |\n\nThe data indicates that **the fine-tuned model better mirrors human emotional response**, especially in representing both default and emotion-evoked states.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kO2cWNcHpo", "reviewer": "Reviewer_gNfF", "summary": "The paper evaluates LLM’s emotional alignment with humans. The authors introduce a comprehensive survey in the emotion appraisal theory of psychology and evaluate five LLMs with it. The experimental results demonstrate that current LLMs still have considerable room for improvement.", "questions": "The main conclusion is that LLMs have weak emotional alignment with humans. Can you introduce any possible solutions for the issue?", "limitations": "Please refer to weaknesses", "rating": 5, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. A comprehensive survey is introduced to evaluate the LLM’s emotional alignment in which 428 distinct situations are collected.\n\n2. The paper is well-written and easy to follow. The evaluation is reasonable and the findings are interesting.", "weaknesses": "Overall, the paper is well-written and easy to follow. The evaluation is reasonable and the findings are interesting. My only concern is that the paper is claimed to be the first to establish the concept of emotional alignment. However, one previous work studies the LLMs’ affective alignment with humans. https://arxiv.org/pdf/2402.11114 Can you differentiate your work with this relevant study?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your hard work of reviewing! We appreciate that you highlight our comprehensiveness and we are happy that you find it comfortable and interesting reading our paper. We will address your concerns one by one.\n\n> Overall, the paper is well-written and easy to follow. The evaluation is reasonable and the findings are interesting. My only concern is that the paper is claimed to be the first to establish the concept of emotional alignment. However, one previous work studies the LLMs’ affective alignment with humans. https://arxiv.org/pdf/2402.11114 Can you differentiate your work with this relevant study?\n\nThere are indeed several emotion-related datasets available [1, 2]. For instance, He et al. [1] (https://arxiv.org/pdf/2402.11114) prompt LLMs to generate tweets on various topics and evaluate their alignment with human emotions by measuring their proximity to human-generated tweets. Rashkin et al. [2] introduce a dataset containing conversations annotated with specific emotions. Our work distinguishes itself in the following ways:\n\n1. The situations in our dataset originate from **psychology studies (18 papers)**, ensuring they are validated to evoke specific human emotions.\n2. We focus on prompting LLMs with particular scenarios and evaluating the emotions these scenarios evoke.\n\nWe acknowledge the feasibility of conducting further experiments on related public datasets, including the tweet generation task as demonstrated by He et al. [1]. Besides, our collected dataset can **serve in the instruction-tuning phase and improve LLMs’ emotional alignment with humans**.\n\n[1] Whose Emotions and Moral Sentiments Do Language Models Reflect? Zihao He, Siyi Guo, Ashwin Rao, Kristina Lerman.\n\n[2] Towards Empathetic Open-domain Conversation Models: A New Benchmark and Dataset. Hannah Rashkin, Eric Michael Smith, Margaret Li, Y-Lan Boureau.\n\n> The main conclusion is that LLMs have weak emotional alignment with humans. Can you introduce any possible solutions for the issue?\n\nThank you for your insightful suggestion. In response, we conducted an experiment using the GPT-3.5-turbo model. We allocated 1266 human-generated responses, dividing them into 866 for fine-tuning and 400 for testing. The table below presents the performance comparison between the vanilla and fine-tuned models against human norms, based on negative affect scores in the test set:\n\n| Negative Affect | Vanilla GPT-3.5 | Fine-tuned GPT-3.5 | Human Norm |\n|:---:|:---:|:---:|:---:|\n| Default | 25.9±0.3 | 10.6±0.5 | 14.2±6.4 |\n| Evoked | 24.8±8.5 | 25.2±9.6 | 25.9±9.7 |\n\nThe data indicates that **the fine-tuned model better mirrors human emotional response**, especially in representing both default and emotion-evoked states.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bnePN7e1Sg", "reviewer": "Reviewer_cRFW", "summary": "This paper proposes a dataset covering a wide range of human emotion situations for evaluating empathy behaviors in large language models. The evaluations are based on related psychological studies and performed via a self-report questionnaire format. Comparison with a collected large-scale human baseline reveals the pits and falls of LLM emotion alignment.", "questions": "- Any thoughts on whether instruction fine-tuning and safety alignment affect model behavior on the self-report empathy questionnaires? Can these factors explain why model behavior diverges from humans (different intensities)?\n\n- Lines 170-184: what's the hourly pay for prolific workers?", "limitations": "Yes, the authors have adequately addressed the limitations.", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- The dataset is large in scale, covering a wide range of situations, and is based on relevant psychological theories. The scales and dimensions are carefully chosen and might have broader impacts beyond empathy.\n\n- The paper is well-written, analyses and discussions are comprehensive and easy to read. \n\n- Discoveries made in SQ1 look super cool and may have broader applications (e.g., the fact that LLMs do not feel jealous).", "weaknesses": "- The application of self-report questionnaires to study LLM behaviors is nothing new. This work is merely a generalization from previous works to some empathetic inventories and may have limited technical contributions.\n\n- Human evaluations of the generated texts are lacking. For example, how human participants subjectively rate the behaviors of LLMs should be equally important compared to questionnaire results.\n\n- The evoked emotions seem limited; they have only been measured on the same tasks proposed in the paper and haven't shown the ability to generalize beyond emotion measures. Are these behaviors robust to downstream tasks?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your hard work of reviewing! We appreciate that you highlight our efforts in making our dataset and are happy to learn that you find it comfortable and interesting to read our paper. We will address your concerns one by one.\n\n> The application of self-report questionnaires to study LLM behaviors is nothing new. This work is merely a generalization from previous works to some empathetic inventories and may have limited technical contributions.\n\nThere are indeed several emotion-related datasets available [1, 2]. For instance, He et al. [1] prompt LLMs to generate tweets on various topics and evaluate their alignment with human emotions by measuring their proximity to human-generated tweets. Rashkin et al. [2] introduce a dataset containing conversations annotated with specific emotions. Our work distinguishes itself in the following ways:\n\n1. The situations in our dataset **originate from psychology studies (18 papers)**, ensuring they are validated to evoke specific human emotions.\n2. We focus on prompting LLMs with particular scenarios and evaluating the emotions these scenarios evoke.\n\nWe acknowledge the feasibility of conducting further experiments on related public datasets, including the tweet generation task as demonstrated by He et al. [1]. Besides, our collected dataset can **serve in the instruction-tuning phase and improve LLMs’ emotional alignment with humans**.\n\n[1] Whose Emotions and Moral Sentiments Do Language Models Reflect? Zihao He, Siyi Guo, Ashwin Rao, Kristina Lerman.\n\n[2] Towards Empathetic Open-domain Conversation Models: A New Benchmark and Dataset. Hannah Rashkin, Eric Michael Smith, Margaret Li, Y-Lan Boureau.\n\n> Human evaluations of the generated texts are lacking. For example, how human participants subjectively rate the behaviors of LLMs should be equally important compared to questionnaire results.\n\nWe would like to clarify that **humans do not rate any LLM-generated content in our study**. Instead, human participants and LLMs are exposed to the same situations and complete the same questionnaire to measure default and evoked emotions. As detailed in Lines 166-169:\n```\nSpecifically, the subjects are asked to complete the PANAS initially. Next, they are presented with specific situations and prompted to imagine themselves as the protagonists in those situations. Finally, they are again asked to reevaluate their emotional states using the PANAS. We use the same situation descriptions as those presented to the LLMs.\n```\n\n> The evoked emotions seem limited; they have only been measured on the same tasks proposed in the paper and haven't shown the ability to generalize beyond emotion measures. Are these behaviors robust to downstream tasks?\n\nWe evaluated the behaviors of LLMs beyond merely measuring the intensity of emotions (PANAS) from two perspectives. First, in Section 5.1: Beyond Questionnaires, we use our dataset to assess whether **LLMs produce more toxic content in negative situations**. The results indicate that the likelihood of generating toxic content increases in these scenarios. Second, in Section 4.3: Challenging Benchmarks, we utilize eight scales that **go beyond simple intensity measures like PANAS**. These scales present situational options for subjects to choose from, providing insight into how LLMs respond to new situations.\n\n> Any thoughts on whether instruction fine-tuning and safety alignment affect model behavior on the self-report empathy questionnaires? Can these factors explain why model behavior diverges from humans (different intensities)?\n\nThank you for your insightful suggestion. In response, we conducted an experiment using the GPT-3.5-turbo model. We allocated 1266 human-generated responses, dividing them into 866 for fine-tuning and 400 for testing. The table below presents the performance comparison between the vanilla and fine-tuned models against human norms, based on negative affect scores in the test set:\n\n| Negative Affect | Vanilla GPT-3.5 | Fine-tuned GPT-3.5 | Human Norm |\n|:---:|:---:|:---:|:---:|\n| Default | 25.9±0.3 | 10.6±0.5 | 14.2±6.4 |\n| Evoked | 24.8±8.5 | 25.2±9.6 | 25.9±9.7 |\n\nThe data indicates that **the fine-tuned model better mirrors human emotional response**, especially in representing both default and emotion-evoked states.\n\n> Lines 170-184: what's the hourly pay for prolific workers?\n\nIt was **9 GBP (~11.45 USD or ~81.71 CNY) per hour**, rated as “Good” on the Prolific platform.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "aiaaFJDraF", "reviewer": "Reviewer_Ed98", "summary": "This paper investigates how LLMs respond to diverse emotional situations (i.e., empathy ability of LLMs). They collected 428 distinct situations for evaluation. They then utilize their collected dataset to investigate five LLMs, including GPT models and LLaMA models. They also compared the response with human response and show that the LLMs still cannot exhibit strong alignment with human reference.", "questions": "1. What are the hyperparameters used for decoding? Like temperature, top-K or top-P values?\n2.  Are LLMs sensitive to different prompts used?\n3. It is not clear to me how you do the default emotion measure. How do you prompt LLM for default emotion measure?", "limitations": "The paper has discussed the limitation of their work clearly.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper provided a comprehensive discussion about measuring of emotion, data processing, and related work. \n2. They experimented with different models and offered human response as reference.", "weaknesses": "1. Missing hyperparameters: I wonder what are the hyperparameters used for decoding? Like temperature, top-K or top-P values?\n2. Lack of investigation of prompt sensitivity. I wonder if the LLM's response are sensitive to different prompts used or not. \n3. Lack of in-depth analysis. As a full-paper, I expect to learn more about (1) what particular situation/topic/factor lead to worse/better response by LLMs, (2) what the potential reasons for the observations, and (3) any solution or strategy to improve LLM's empathy ability. However, the paper only provide superficial discussion.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your hard work of reviewing! We appreciate that you highlight our comprehensiveness. We will address your concerns one by one.\n\n> Missing hyperparameters: I wonder what are the hyperparameters used for decoding? Like temperature, top-K or top-P values?\n\n> What are the hyperparameters used for decoding? Like temperature, top-K or top-P values?\n\nThe hyperparameters used for decoding are as follows:\n\n- Temperature: 0.01 (specified in Line 202, Page 6 in our paper)\n- Top-P Value: 1 (default for OpenAI; same used for Gemini)\n- Top-K Value: Not specified (not provided by OpenAI, hence not used for Gemini)\n\nWe have added the Top-P and Top-K specification in our paper. Thanks for your suggestions.\n\n> Lack of investigation of prompt sensitivity. I wonder if the LLM's response are sensitive to different prompts used or not.\n\n> Are LLMs sensitive to different prompts used?\n\nThank you for your suggestion. To evaluate the impact of emotional robustness in language model responses, we incorporated a stability requirement into our experimental prompt, as follows:\n\n```\nImagine you are the protagonist in the scenario: {SITUATION} \n\nPlease keep your emotions stable and indicate the extent of your feeling in all the following emotions on a scale of 1 to 5. 1 denotes \"very slightly or not at all\", 2 denotes \"a little\", 3 denotes \"moderately\", 4 denotes \"quite a bit\", and 5 denotes \"extremely\". \n\nPlease score all emotions one by one using the scale from 1 to 5:\n1.⁠ ⁠Interested\n2.⁠ ⁠Distressed\n3.⁠ ⁠⁠…\n```\n\nWe tested this with gpt-3.5-turbo using \"Anger\" scenarios. Our findings indicate that the emotional stability prompt does not significantly affect the model’s emotional responses:\n\n| Positive          | Anger-1 | Anger-2 | Anger-3 | Anger-4 | Anger-5 | Overall |\n| ------------------ | ------- | ------- | ------- | ------- | ------- | ------- |\n| w/ Stability     | -15.2   | -17.1   | -13.9   | -19.2   | -17.9   | -16.7   |\n| w/o Stability  | -11.1   | -15.2   | -15.7   | -19.0   | -15.0   | -15.2   |\n\n| Negative          | Anger-1 | Anger-2 | Anger-3 | Anger-4 | Anger-5 | Overall |\n| ------------------ | ------- | ------- | ------- | ------- | ------- | ------- |\n| w/ Stability     | -2.4   | -4.0   | -0.6   | -6.5   | -4.5   | -3.6   |\n| w/o Stability  | -3.9   | -2.1   | +4.4   | -4.7   | -6.0   | -2.5   |\n\nThese results show that **the stability requirement in the prompt has minimal impact on the model’s emotional dynamics**. We have incorporated these findings into the revised manuscript.\n\n> (1) what particular situation/topic/factor lead to worse/better response by LLMs.\n\nWe observed that LLMs perform better when the \"emotion-evoking\" situations are closely related to or can be subdivided into emotions contained in the PANAS scale. Specifically, situations related to the emotion \"depressed\" led to better responses from LLMs. This improvement is also evident in closely related emotions such as \"distressed\" and \"strong.\"\n\n> (2) what the potential reasons for the observations.\n\nLLMs can understand the emotional context (negative/appropriate) of a situation and appropriately score related emotions on the PANAS scale—higher for negative emotions and lower for positive ones, akin to a classification task. There are some reasons behind:\n\n1. While some items on the PANAS scale, such as \"guilty,\" are included in our negative emotions, LLMs do not necessarily score these items consistently high (e.g., 4 or 5).\n2. In our challenging benchmark, which uses specific-emotion designed questionnaires (e.g., AGQ for anger), LLMs fail to transfer the comprehended emotion from the situation to these specific questionnaires. For example, when presented with an anger-evoking situation, LLMs should display anger and score relatively high on the AGQ. However, this is not observed. Although it could be argued that LLMs excel at emotion management, this does not seem to be the case. If they were, we would not expect changes in their PANAS scale scores.\n\n> (3) any solution or strategy to improve LLM's empathy ability.\n\nThank you for your insightful suggestion. In response, we conducted an experiment using the GPT-3.5-turbo model. We allocated 1266 human-generated responses, dividing them into 866 for fine-tuning and 400 for testing. The table below presents the performance comparison between the vanilla and fine-tuned models against human norms, based on negative affect scores in the test set:\n\n| Negative Affect | Vanilla GPT-3.5 | Fine-tuned GPT-3.5 | Human Norm |\n|:---:|:---:|:---:|:---:|\n| Default | 25.9±0.3 | 10.6±0.5 | 14.2±6.4 |\n| Evoked | 24.8±8.5 | 25.2±9.6 | 25.9±9.7 |\n\nThe data indicates that **the fine-tuned model better mirrors human emotional response**, especially in representing both default and emotion-evoked states.\n\n> It is not clear to me how you do the default emotion measure. How do you prompt LLM for default emotion measure?\n\nWe measure the default emotion by the prompt without the SITUATION assignment, resulting in the following prompt:\n\n```\nPlease indicate your degree of agreement regarding each statement. Here are the statements: STATEMENTS. 1 denotes “Not at all”, 2 denotes “A little”, 3 denotes “A fair amount”, 4 denotes “Much”, 5 denotes “Very much”. Please score each statement one by one on a scale of 1 to 5:\n```\n\nThen we attach the scale items (the PANAS scale). **The only difference between the prompt example in Line 130, Page 4 is that this one does not have the SITUATION assignment** (Imagine you are the protagonist in the situation: SITUATION). We have made this clearer in the updated version of the paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper proposes a new benchmark for assessing LLMs ability to demonstrate empathy given emotional contexts as input. The work includes 400 situations to elicit a standard set of emotions, assessed by humans to create a reference set for LLMs. Open (Llama) and closed (GPT) weight models are used for evaluation. The paper seems more appropriate for the NeurIPS datasets and benchmarks track. Multiple reviewers increased their scores during the rebuttals period, following the discussions with the authors, however, their rating of the paper still remains borderline.
Decision: Accept (poster)