Evaluating Subjective Cognitive Appraisals of Emotions from Large Language Models
Authors: Hongli Zhan
Desmond Ong
Junyi Jessy Li
Venue: EMNLP 2023 Findings (2023)
Keywords: Emotion
Cognitive Appraisal
Large Language Models

Abstract: The emotions we experience involve complex processes; besides physiological aspects, research in psychology has studied cognitive appraisals where people assess their situations subjectively, according to their own values (Scherer, 2005). Thus, the same situation can often result in different emotional experiences. While the detection of emotion is a well-established task, there is very limited work so far on the automatic prediction of cognitive appraisals. This work fills the gap by presenting CovidET-Appraisals, the most comprehensive dataset to-date that assesses 24 appraisal dimensions, each with a natural language rationale, across 241 Reddit posts. CovidET-Appraisals presents an ideal testbed to evaluate the ability of large language models — excelling at a wide range of NLP tasks — to automatically assess and explain cognitive appraisals. We found that while the best models are performant, open-sourced LLMs fall short at this task, presenting a new challenge in the future development of emotionally intelligent models. We release our dataset at https://github.com/honglizhan/CovidET-Appraisals-Public.

Evaluating Subjective Cognitive Appraisals of Emotions from Large Language Models
Hongli Zhan' Desmond C. Ong Junyi Jessy Li
l Departmeet of Linguisties, The University of Texas at Austin 2 Department of Psychology. The University of Texas at Austin mhonglizhan, desmond. ong, jessy jessya@u edu
Abstract
I've recently heard of stories that people who are getting mildy sick after being vaccinated are still coming down with long covid. People long huge problem an our hands because no vaccine protects against long covid... This Isn't geing end, Ian't ht? Right think I've tasted freedom, Iind out information that makes me want to stay inside forever like hermitt...
subreddit are saying that
going
The emotions we experience involve complex processes; besides physiological aspects, re- search in psychology has studied cognitive appraisals where people assess their situa- tions subjectively, according their own val- ues (Scherer, 2005). Thus, the same situa- tion can often result in different emotional experiences. While the detection of emotion is a well-established task, there is very lim- ited work so far on the automatic prediction of cognitive appraisals. This work fills the the
1. what extent did the nerretor think that THEY responsible causing situation?
[Saale]:1 1 - [Rationale]: The nerrator does not talk about themsolivss relation
COVID situation way that makes it seem like they belleve they are responsible.
To what extent did the narrator think that OTHER PEOPLE were responaible for causing the situstion?
[Scala]: [Rationalo]: The narrator mentions how many people who are vaccinated and get mildly slck still contract long COVID, but they
them way that seems like they are blerning them for the situstion.
gap by presenting most comprehensive dataset to-date that as- sesses 24 appraisal dimensions, each with natural language rationale, across 241 Reddit presents an ideal testbed to evaluate the ability of large language models- - excelling at a wide range of NLP tasks to automatically assess and explain cognitive appraisals. We found that while the best models are performant, open- sourced LLMs fall short at this task, present- ing a new challenge in the future develop- ment of emotionally intelligent models. We release our dataset al / thub. com/ hanpgliihhann/oiidd--ppraaaaaaa -uuuul
3. To what extent did the narrttr think that CIRCUMSTANG ES BEYOND ANYONE'S CONTROL were responsible for causing the situation?
[Scale]: [Rationale]: The narrittor mentions long COVID and Inability to currently protect against COVID, This a factor that outside anyone's control. including doctors and nurses, the situation continues to evolve in uncontrolisble ways.
posts. COVIDET-APPRAISAAN
?
24. To what extent did the narrator EXPECT the situation to occur?
[Scalo]: 3 - [Rationalo]:
narrator really expoct
this situation since they mention being able to taste freedam, beleving the pandemicc andinng they heard news that vaccinated people are still getting long covid and now they think the pendemic will never end.
suddenly
An example from COVIDET-APPRAISAN The fact that the narrator is blaming nobody but cir- cumstances beyond anyone's control for causing long- COVID contributes to their feeling of sadness. We showcase an annotation together with LLMs' responses in Appendix SA.
Figure 1:
1 Introduction
Emotions constitute crucial aspect of people's s lives, and understanding them has profound im- pact on improving public mental health problems as
Ellsworth, 1985; Weiner, 1985; Clore and Ortony,
well as policy-making (Choudhury and De, 2014;
2000; Roseman and Smith, 2001; Scherer et al.. 2001; Ellsworth and Scherer, 2003; Sander et al., 2005; Ong et al., 2015, 2019; Ortony et al., 2022: Yeo and Ong, 2023). For instance, being fired from job, if judged to be due to one's own controllable mistakes, could result in regret; if evaluated to be unfair and due to someone else's intentional actions, would make one feel angry; and if appraised be leaving a toxic work environment, could instead re- sult in relief and even happiness. The different di- mensions along which people subjectively inter-
Gjurković and Snajder, 2018; Arora et al., 2021;
Uban et al., 2021). The emotions we experience
involve complex processes: the same situation can often result in different emotional experiences, based on individual's subjective evaluations. These are called cognitive appraisals, and have been extensively studied in psychology through theoretical, behavioral, and hand-coded studies (Amold, 1960; Lazarus, 1966; Lazarus et al., 1980; Roseman, 1984; Scherer et al., 1984; Smith and

---

pret or appraise the situation characterizes the specific emotions they feel (Moors et al., 2013).
cover implicit information for emotion understand- ing. Benchmarking on COVIDET-APPRAISALL we evaluate the performance of LLMs to (1) pro- vide Likert-scale ratings for the appraisal dimen- sions; and (2) generate natural language rationales for their ratings. The elicitation of the rationales can be seen way of probing (Le Scao and Rush, 2021; Gu et al., 2022), where we prefix a question with elaborated situation. We evaluate a range of LLMs, including ChatGPT, Flan-T5 (Chung et al., 2022), Alpaca (Taori et al., 2023), Dolly (Conover et al., 2023). With an extensive human evalua- tion of the natural language rationales from LLMs as well as our annotators, we find that ChatGPT performs on par with (and in some cases better than) human-annotated data; this opens new av- enue of investigation to improve its performance on emotion-related tasks (Kocoń et al., 2023). In comparison, other open-sourced LLMs fall short on this task, presenting a new challenge the fu- ture development of emotionally intelligent open models.
Although emotion detection a well-established NLP task (Strapparava and Mihalcea, 2007: Mi-
halcea and Strapparava, 2012: Wang et al., 2012;
Lei et al., 2014; Abdul-Mageed and Ungar, 2017;
Khanpour and Caragea, 2018; Liu et al., 2019;
Sosea and Caragea, 2020; Demszky et al., 2020;
Desai et al., 2020; Sosea et al., 2022), it mostly involves classification from text to emotion labels directly, skipping the appraisal step that is neces- sary to interpret why the emotion is experienced by an individual in a particular event. Hence, we do not yet have a data-driven understanding of these cognitive appraisals in textual data. Yet recent work has started to show its necessity: Hofmann et al. (2020) showed that appraisals are informative for emotion detection model; Zhan et al. (2022) further recognized appraisals be an integral part of emotion triggers, though appraisals were not explicit in their work.
This work aims at construing an empirical, ex- plicit understanding of perceived cognitive ap- praisals in human readers and large language mod- els (LLMs) alike, via a comprehensive 24 di- mensions, along with their corresponding natural language rationales. A language model's capa- bility of assessing cognitive appraisals reflects more nuanced understanding of emotions, where it could contextualize individual subjectivity in re- sponses to the same situation, while offering ex- planations ('they are feeling [emotion] because of lappraisal]"). This could be groundwork for emo- tional support agents, e.g., one capable of positive reframing (Ziems et al., 2022) producing empa- thetic responses.
We publicly release our annotated dataset COVIDET-APPRAISALS. model oulputs, and our human evaluation data https: //gi tigb..hu...... comb Soaphhan/CoviiE--Appppaee
Background and Related Work
Cognitive Appraisal Theories. The cognitive ap- praisal theories of emotion state that emotions arise from individual's subjective understanding and interpretation of situations that hold personal im- portance for their overall well-being (Amold, 1960; Lazarus, 1966; Lazarus et al., 1980; Roseman, 1984; Scherer et al., 1984; Smith and Ellsworth, 1985; Weiner, 1985; Clore and Ortony, 2000; Rose- man and Smith, 2001; Scherer et al., 2001; Sander et al., 2005; Ortony et al., 2022). In practical terms, people interpret and appraise situations along a range of different dimensions, and it is the specific manner in which they appraise their situations that give rise to the distinct emotions they experience. The primary focus of cognitive appraisal theories of emotions revolves around the identification of these appraisal dimensions that are associated with specific emotional experiences and how these di- mensions contribute to distinguishing between dif- ferent emotional states (Lazarus, 1993; Roseman, 1996; Scherer al., 2001; Moors, 2010; Scherer and Moors, 2019).
We first introduce dataset of 24 appraisal dimensions annotated across 241 Reddit posts sourced from Zhan al. (2022) about COVID-19. Each post was manually anno- tated with 24 appraisal dimensions from recent meta-analysis covering all appraisal dimensions proposed and studied in the literature (Yeo and Ong, 2023). For each appraisal dimension, annotators not only rated the extent to which they perceived the narrator is experiencing the said dimension, but also provided rationale in their own language to justify their rating selection. An example from CovIDET-APPRR APPRAISLLS is shown Figure
APPRAAALPPA serves as an idcal testbed to evaluate the capability of a model to un-
While appraisal theorists agree on the impor-

---

tance of motivatioonllly-relevnn appraisals in trig- gering emotions. they have reached a consensus on the specific appraisal dimensions that play sig-
2023), emotion recognition (Kocoń et al., 2023), disclosing the representation of human emotions encapsulated in LLMs (Li et al., 2023), and inter- preting mental health analysis (Yang et al., 2023). However, few have tapped into the understand- ing of cogmitive appraisals of emotions innate in LLMs. In this work, we dive into the extent to which LLMs comprehend the profound cognitive appraisals underlying emotions in situations, and further elicit natural language rationales from the language models disclose the reason behind such predictions from the otherwise baffling black-box LLMs (Gilpin et al., 2018). 1).. Aligning with Maraso- vić et al. (2020) who performed human evaluation on rationales generated by GPT, we additionally perform an in-depth human evaluation of the ratio- nales from human annotators and LLMs alike the novel task of providing natural language expla- nations for cognitive appraisals of situations that underlie narrators' emotional experiences.
nificant role in this process (Yeo and Ong, 2023). Various theories have put forth distinct sets of ap- praisal dimensions that are considered crucial in triggering and distinguishing emotions. From prior literature, Yeo and Ong (2023) identified and as- sembled taxonomy of all appraisal dimensions that have been studied, and produced a condensed list of 24 cognitive appraisal dimensions which we focus on in this paper.
Cognitive Appraisals in NLP. Appraisals pro- vide the necessary computational structure allow- ing for the distillation of real-life situations that depend on a multitude of factors into a (large but) finite set of appraisal dimensions (Ong et al.. 2015). Despite its importance, however, few works have explored the implications of cognitive appraisals emotions in NLP. Hofmann al. (2020) ex- perimented with small set of cognitive appraisal dimensions (including attention, certainty, effort, pleasantness, responsibility, control, and circum- stance) to assist the automatic detection of emo- tions text, and found that accurate predictions of appraisal dimensions boost emotion classification performance. They introduced a dataset of 1, 001 sentences following the template "1 feel Lemotion], when ..." (average sentence length: 27 tokens). In comparison, our work covers much wider range of 24 appraisal dimensions found in prior literature, over lengthy (176 tokens on average) Reddit posts that were natural and emotionally charged. We also collect natural language rationales as a key contri- bution to reveal human's 's in-depth understanding of such cognitive appraisals in context.
3 The COVIDET-APPRAISALS Dataset
COVIDET-APPRATSAA contains 241 Reddit posts sampled from the COVIDET dataset (Zhan et al., 2022), where the Reddit posts are sourced from /CCVID1 9 9_sppport. Each post is manually anno- tated with one or more of the emotions: anger, anticipation, joy, trust, frust.fear, sadness, and disgust. The 241 posts in DOVIDET-APPRAISS average of 175.82 tokens and 2.67 emotions per post. From Yeo and Ong (2023)'s work, we iden- tify 24 cognitive emotion appraisal dimensions (Ta- ble 1). We provide the instructions given to the annotators (including the full questions for each of these 24 dimensions) in Appendix SB.
have an
Annotators.
We recruited 2 linguistics students at university to work on our annotation task; both of them are native speakers English. Both an- notators underwent training using a set of posts already annotated by our group. Throughout the annotation, we monitored the inter-annotator agrec- ment and provided feedback on their work.
Recent studies (Zhan et al., 2022: Sosea et al., 2023) acknowledged both what happened and how one appraised the situation as inherent compo- nents of emotion triggers, although the appraisal of events was not explicit in their work. Instead we provide datasets and perform evaluation ap- praisals explicitly, such that language models can build on this work to achieve comprehensive and explicit understanding of cognitive appraisals from written text.
Instructions. Given Reddit post from COVIDET, annotators are asked to judge 24 emotion appraisal dimensions pertaining to how the narrator feels about and views the situation that they are going through (e.g., whether the narrator feels the situation they are in something they could control). For each appraisal dimension, annotators need to select a Likert rating on the
LLMs on Emotion-Related Tasks. Autoregres- sive LLMs have been explored extensively in emotion-related tasks such as sentiment analysis (Zhong et al., 2023; Qin et al., 2023; Susnjak,

---

Abbrv.
Reader-Friendly Labels
ply (Yeo and Ong, 2023). This can be seen in Figure 2 which plots the percentage of NA labels: dimensions such as civ (consistency with internal values). fair (faimess)) and csn (consistency with social norms) contain mostly NA labels (around 80%). Therefore, we remove these dimensions from subsequent analyses and evaluations of the dataset. This results in a total of 21 applicable appraisal dimensions in COVIDET-APPRAIS..
srap
Self relfiresppsiiiiity
orsp
repoosiisyyyiiiillly
3
crsp
Diramattnces-repponsiiiiin
4
Problem-focuser coping
5
grlv
Goal Relevance
6
atin
Attentional activity
7
Emotiion Fonuton-focused coping
8
scn!
Self- Controllllle
9
ocrl
Other- ConerrContllllable
10
ccrl
Circumstances Controllleeeeo
pri
Predicuabliiy
Threal
We collected 241 posts in total. For the subset of 40 posts that are doubly annotated, we aggregate the Likert-scale ratings by taking the mean of cach post's ratings for each appraisal dimension (if an annotator labels a dimension as NA, we then ex- clude the particular dimension of that post that they annotate). In terms of the rationales, we consider both rationales ground truth references and use multi-reference metries in our experiments.
Pleasantness
14
crt
Certainry
15
gend
Goal Conduciveness
fuir
Fairness
Future expectancy
Consistency with social
19
loss
Loss
20
finl
Familiarity
eff
Effort
Challenge
Consistency with internal values
24
exp
Expectedness
Table I: The 24 appraisal dimensions and their abbrevi- ations we used throughout this paper. See Appendix $B for full questions for each dimension, and Figure for example of how the items for I: relfsreesponsiiiillty 2: responiiiyp, 3: pirumstonce---espposiiiii
Agreement. We report inter- annotator agreement on the Likert-scale ratings, Since there is no reliable, automatic way to evaluate natural language rationales (as discussed in $4), we evaluate them with human validation in $7.2.
and 24: expectedness framed.
To measure the agreement for selecting the NA label, we average the Fleiss' Kappa values (Fleiss, 1971; Randolph, 2005) across all 24 appraisal di- mensions, yielding a value 0.769 indicating sub- stantial agreement (Artstein and Poesio, 2008).
Percentage Nf values
NA Values Percentage in Each Dimension
ao
60
40
For the 1-9 Likert-scale ratings, we report on the 21 applicable dimensions: (1) Spearman's 's be- tween two annotators, calculated per dimension then averaged across all dimensions; (2) Krippen- dorff's alpha (using interval distance) (Krippen- dorff, 1980); and (3) mean absolute difference (abs. delta). Here the agreement calculated if neither annotator gave a NA judgment. Krippendorff's alpha yields a value of 0.647 indicating substan- tial agreement (Artstein and Poesio, 2008). The average Spearman's 's correlation is 0.497 with sig- nificance, and the absolute delta values also have small mean of 1.734. These measures indicate that while the task is subjective, annotators do align with each other with only small difference com- pared to the scale of ratings (1-9). Agreement val- ues differ by dimension, which we showcase in Appendix C.
20
fair
csn
prd
fex
off
exp
eit
chi
Uhr
scri
crsp
pis cull
or srsp ptc
Dimensions
Figure 2: Percentage of "not mentioned" labels in each
dimension in COVIDET-PPPRISALS.
scales of 1 to 9. A "not mentioned" (NA) option is provided in case the dimension being asked is absent in the given post. In addition, we also ask the annotators to provide rationales for their ratings in the form of natural language explanations.
On average, trained annotators spent around 30 minutes complete the annotation of one post, Owing to the immense effort involved, we dou- bly annotate 40 posts to measure inter-annotator agreement while leaving the rest annotated by one annotator.
4 Dataset Analysis
Tost-Processing and Aggregation. Given fixed topic (COVID-19 in our case), highly likely that certain dimensions frequently don't ap-
How do the scales distribute across dimensions and emotions? The distribution of the Likert- scale ratings is shown in Figure 3. The rat-

---

RATIONALE ROUGE-L
BLEU-4
BERTSC
ANNOTATORS 0.042
0.253
0.357
BASELINE-P 0.060
0.261
0.336
BASELINE-D 0.059
0.247
0.332
Table 2: Automatic measures of similarity on the nat- ural language rationales of BASELINE-P denotes "baseline (same dimension, differ- ent posts)", and BASELINE- denotes "baseline (same post, different dimensions)".
grsp
orsp
crsp
plc griv atti
efo
soht
cott
conl
thr
pls
art
gend
fox loss
fml
eff chi exp
Dimensions
Figure 3: Distribution of the ratings for each dimension.
FER
structions (i.e., evaluating specific appraisal di- mension) that the annotators given. As shown in Table 4, human annotators attain % of novel bigrams of 36,7%, indicating a high abstrac- tiveness. We showcase the most prominent topics extracted from the annotated rationales using La- tent Dirichlet Allocation (LDA) (Blei et al., 2003) in Appendix $D.2.
D96
SDN
9999
018p
ptc
gtv
att
efc
s2r
oct
thr
pla
gond
Fex loss
fmil
off
chi O0g
Dimemsions
Figure 4: Mean Likert-scale ratings for each dimension
in each emotion.
Are rationales repetitive? We also look into automatic measures of similarity to assess how much rationales from different annotators, or from different dimensions/posts, differ from one an- other. Specifically, we calculate BLEU-4 (Papineni et al., 2002), ROUGE-L (Lin, 2004), and re-scaled BERTSccoe (Zhang et al., 2019) between our two annotators' rationales. We establish 2 random base- lines for comparison: (1) rationales of the same dimension from different posts; (2) rationales from different dimensions within the same post. In each case we report similarity between 3 randomly sam- pled rationales and the annotated ones.
ings for some dimensions are consistent (e.g.. di- mensions crsp (ciruustaanes-responibbiiii corl and chl (challenge)), whereas for some other dimensions, the ratings have higher variance (e.g., dimensions ocrl (other- controllable) and loss).
feiruusttnnces-contrlllllll
We analyze the connections between our Likert- scale annotations and COvIDET's emotion annota- tions. Figure shows the mean Likert-scale rating for each dimension within each post with respect to the perceived emotion. While it evident that most dimensions show consistency (the posts are all related to COVID-19), some emotions stand out distinctly in particular dimensions. For example, trust and joy have higher Likert-scale ratings on dimensions pfc (problem-focused coping) and gend (goal conduciveness) compared to other emotions, suggesting the inter-correlation between these ap- praisal dimensions with positive emotions. We further explore whether appraisal dimensions alone are indicative of perceived emotions already anno- tated in COVIDET in Appendix $D.1.
Table 2 shows that the textual similarity in all conditions are somewhat low; the BLEU and ROUGE scores show that there is very little lex- ical overlap, although BERTScore shows higher semantic similarity between annotators for the same dimension within the same post. Upon closer inspection, we observe that these commonly used automatic measures do not adequately capture se- mantic similarity in our dataset (see Appendix $D.3 for an example). This adds to the challenge of eval- uating rationales; as result, resort the human evaluation in $7.2.
What are the characteristics of the natural lan- guage rationales? On average, each rationale is 1.2 sentences (std.dev 0.4) and 28.9 tokens (std.dev 10.0) long. Following Marfurt and Hen- derson (2021), we also measure the abstractiveness of the rationales from our human annotators by calculating the percentage of novel bigrams in the rationales with respect to the Reddit posts and in-
5
Can LLMs understand emotional appraisals?
provides an ideal testbed that evaluates models' performance on predict- ing both the Likert ratings, as well as their nat-

---

ural language explanations. Using COVIDET- APPRAISALS, we evaluate the zero-shot perfor- mance of LLMs in an attempt to evaluate their innate ability to comprehend emotional appraisals from social media text without in-context learning.
SCALE SPEARMAN'S p
NA
MAE
F1
CHATGPT
1.694
0.388
0.918
FLAN-TS
3.266
0.225
0.852
ALPACA-78
2.353
0.081
0.918
ALPACA-13B
3.872
-0.035
0.602
DOLLY-7B
2.812
-0.013
0.645
Models. We evaluate the following instruction- tuned LLMs': 1) ChatGPT, ie.. GPT-3.5- 2) FLAN-T5-XXL (IIB) (Chung et al., 2022), which is the instruction fine-tuned version of T5 (Raffel et al., 2020); 3) Alpaca (7B, 13B) (Taori et al., 2023) is fine-tuned from LLaMA (7B and 13B) (Touvron et al., 2023) 52K instruction- following examples created with GPT text-davinci- 003 in the manner of self-instruct (Wang et al.. 2022); 4) Dolly-V2 (7B, 12B) (Conover et al., 2023) an instruction-tume tund LLM trained on -15k demonstrations consisting of both instructions and responses.
DOLLY-12B
2.747
0.022
0.711
Table Experiment results from LLMs. < 0.1 for Spearman correlation, and indicates p < 0.05. addition, we also provide results the F1 score measuring the agreement between the models' ratings and the gold ratings for selecting the "not mentioned" label across all 24 dimensions.
indicates
standard regression metries: Mean Absolute Error (MAE) and Spearman's ''s correlation. We treat the selection of the NA labels as a binary classification task and report F1 measures across all 24 dimen- sions. For the 10 gold examples that were doubly annotated by human annotators, we consider a di- mension NA when both annotators select the label.
Prompts and Setup. The templates for prompt- ing the LLMs are shown in Appendix Figure 17. After extensive experimentation, we found that only ChatGPT is able to generate both rating and rationale with single prompt; this type of "I-step" prompting leads to ill-formed responses for other models. Thus, for models other than Chat- GPT, we instead use pipeline "2-step" prompt- ing similar to the strategy used in Press al. (2022): we first elicit the rating for the appraisal dimension, then conditioned on the response for the rating we further elicit the rationale for the selection.
To evaluate the performance, we clean the responses elicited from the LLMs. Specifi- cally, we use regular expressions to extract the first numeric value ranging from 1-9 from the scale re- sponses The results of the models' performance are shown in Table 3. We showcase examples of the models' responses in Appendix $A. Additional analyses of the LLMs' responses are shown in Ap- pendix $G.
Results.
We carry out all our experiments on 4 Nvidia A40 GPUs. We use the HuggingFace Transformers (Wolf al., 2020) library for model inference. We set the temperature value of all models 0. To enable a fair comparison of models, we sample from the LLMs five times with different model initializations and report average values for both scales and rationales.
For the NA labels (Table 3, right), ChatGPT and Alpaca-7B score the highest with an Fl of 0.918. In general, the average performance across the lan- guage models we evaluate is 0.774 for F1, indi- cating these models performant at predicting whether a dimension applies.
For the Likert-rating predictions, results show that ChatGPT-335 consistently yields the highest performance compared to the other language mod- els, with significant Spearman's 's correlation of 0.388 and an MAE of 1.694. We note that FLAN- T5-XXL is the second best-performing model. Al- paca and Dolly perform poorly on our task, with negative correlations with the gold labels Inter-
6 Evaluation: Likert-Scale Ratings
We report model performance for Likert-scale rat- ings on the 21 applicable dimensions using two
"Wiie we have also experimented with non-instruction- tuned LLMss (including GPT-3 davinci and LLaMA (7B and 13B), they largely fail to generate sensible outputs for this task. We showease examples of responses from non-instruction- tuned models Appendix $A. For these reasons, we do not include their results this paper.
FFor example, one of Alpaca- scale responses "The narrator thought that Circumsiances Beyond Anyone's Control were responsible for causing the situation a moderate extent on a scale of 1-9).
' After cleaning, the response is formatted to "4".
"W experimented with higher temperatures on a validation set consisting of 10 Reddit posts annotated by our group which are not included in COVIDET-A APRR-AAPAAAAAA and the models yielded worse and more unstable performance.
As shown in Appendix Figure 9, the ratings generated hy

---

LENGTH TOKENS
ABSTRACTIVENESS NNOVEL BIGRAMS
HUMAN
EVAL
BLEU-4
ROUGE-I
BERTSC
FAC
REL
JUS
USE
ANNOTATORS
28.9
86.7%
0.73
0.88
0.95
0.72
CHATGPT 58.0
81.8%
0,044
0.224
0.347
0.84
0.88
0.93
0.85
FLAN-TS
45.3
16. 0%
0.008
0.066
0.053
0.40
0.29
0.24
0.13
ALPACA-7B 48.6
71.9%
0.040
0.230
0.297
0.55
0.82
0.82
0.51
Table 4: Experiment results from LLMs. Additional evaluations of all language models (including Alpaca- 13B, Dolly-7B, and D2I are provided Table 11. more comprehensive report the automatic metries BLEU-4, ROUGE-L, and BERTSCORE provided in Table Appendix $F.
estingly, we notice a drop in performance when the size the model parameters increases for Alpaca. The results highlight the challenging nature of our task, and the gap between open- sourcs LLMs vs. ChatGPT (Gudibande et al., 2023).
despite its relatively poor capability in terms of selecting Likert-scale ratings. We note that FLAN- T5 lags behind considcrably compared to Chat- GPT and Alpaca-7B. We provide the additional auto-evaluation statistics for other LLMs includ- ing Dolly-7B, Dolly-12B, and Alpaca- in Ap- pendix Table 11.
Additionally, we also measure the systems' per- formance on all 24 appraisal dimensions, including the 3 appraisal dimensions where the NA rates are around 80%. Results revealed marginal change in performance across all LLMs. For most LLMs the performance dropped as expected: measured with Spearman's 's ChatGPT-3.5 (J 0.018), Alpaca-7B (L 0.008), and DoB (1 0.007). On the other hand, the performance of FLAN-T5 (f 0.005), Alpaca-13B (t 0.027), and Dolly-7B (t 0.020) increased.
How long and how abstractive are the rationales generated by LLMs? In addition, we also mea- sure the length and abstractiveness of the rationales generated by LLMs. Following the setup in $4, we evaluate abstractiveness using % of novel bigrams, comparing LLMs' generated rationales against the Reddit posts as well as the prompts (i.e., evaluating a specific appraisal dimension) they were given. As shown in Table 4, rationales generated by LLMs are at least 1.5x longer than those provided by our annotators, with ChatGPT being the most verbose. The LLMs also provide rationales that are more extractive compared to annotators, with FLAN- T5 being the most extractive.
7 Evaluation: Rationales
As rationalizing emotional uppraisals with natural language is a novel task, we perform both auto- matic ($7.1) and human evaluation ($7.2).
7.1 Automatic Evaluation
7.2 Human Evaluation
We use commonly used automatic reference- based metrics including BLEU (Papineni et al., 2002), ROUGE (Lin, 2004) and BERTScore (Zhang et al., 2019), comparing generated rationales vs. anno- tated ones (in multi-reference fashion).
Data. Because the natural language rationales are explanations for a particular rating, we only eval- uate and analyze LLM LLM-generated rationales when the model made near-correct prediction of the Likert-scale rating for that particular dimension compared against the gold human ratings. Specifi- cally, we sample the intersection of (post, dimen- sion) tuples where the 3 best-performing L.MMs' (i.c., ChatGPT, FLAN-TS, and Alpaca-7B) ratings fall in the range of an absolute difference of to one of the annotated scale-ratings. In cases where there are 2 gold annotations for particular dimen- sion, both are evaluated. In Appendix $F we also show the human evaluation of rationales for such intersection of all LLMs. We additionally evaluate human-written rationales as well, and we mix those (in random order) with LLMs' responses.
Results. Similar to the performance in selecting Likert-scale ratings, ChatGPT remains the best- performing language model in providing natural language rationales (Table 4). The values Chat- GPT achieves are lower than, though compara- ble to, those between different rationales from our two annotators. Alpaca-7B also achieves compa- rable performance in these automatic measures,
the language models (specifically, Alpaca-7B and D2Il for some of the dimensions lack variance (i.c., they gave a constant rating for certain appraisal dimensions). Therefore, the Spearman correlation is set to zero in these dimensions, indicating no correlation.
The above desiderata results in an evaluation

---

of 108 rationales annotated by human annotators and 65 natural language rationales from each LLM. The evaluation covers 19 out of the 21 applicable dimensions (no such overlap is found for dimen- sions crsp ccicuumstanees-eesoossiit
involved in a pre-annotation qualification as well as training process before commencing the evalua- tion of the natural language rationales. We assign 2 crowd workers per natural language rationale eval- uation. We ensure that the crowd workers earn a minimum salary of $10 per hour.
and pls (pleasantess)). Moreover, we make sure that there are no ground truth labels annotated by the human annotators in which the rating is NA.
We report the inter-evaluator agreement using Krippendorff's s Alpha with interval distance in Ta- ble showing substantial agreement (Artstein and Poesio, 2008) across all criteria.
Instructions. Given a Reddit post and the scale provided by the human annotators or the LLM (blinded to the annotators), annotators are asked to judge the rationales pertaining to the emotion appraisal dimension regarding the post as well as the stated scale. The rationales are distributed to annotators at random. We evaluate the natural lan- guage rationales based on the following criteria. In Appendix $H, We provide the detailed instructions and examples given the annotators, together with the layout the human evaluation task.
Label Transformation. For the convenience of measuring mnter-annotator agreement as well in- terpreting the results, we convert the labels of each criterion to numeric values within the range of 0 to 1. Specifically, for criteria Factuality, Justification, and Usefulness, "Yes" is converted to 1, "Minor Error/Maybe" to 0.5, and "No" to O. As for the criterion Relevance which is judged on 5-scale Likert rating, map the Likert scale of into 0, 2 into 0.25, 3 into 0.5, 1 into 0.75, and 5 into 1.
1) Factuality: For the rationale, the model may not generate something that is factual: sometimes it generates rationales for the sole purpose of jus- tifying its answer (Ye and Durrett, 2022). There- fore, we include the aspect of hallucination and factuality as one of our evaluation criteria, and ask evaluators whether the rationale faithfully reflects what's 's stated in the post. Options of "Yes", "Minor Error", and "No" are provided.
Results. The result of the mean ratings for each criterion from the human evaluation task is pro- vided in Table 4. We provide box plots of the ratings as well as the human evaluation results for the rationales from all LLMs in Appendix $I. From Table 4 we observe that our human anno- tators and ChatGPT provide natural language ratio- nales of the highest quality among all models, ac- cording to human evaluators. Surprisingly, we find ChatGPT performs on par with our human anno- tators, with (slightly) better performance in terms of factuality and usefulness. This can be attributed to the verbosity and extractiveness of ChatGPT (as shown in Table 4), especially in dimensions where the scale rating is low. We showcase an example in Appendix ST.
2) Relevance: We evaluate whether the rationale directly addresses the specific appraisal dimension question that is being asked about the post. We ask evaluators on Likert-scale of 1 to 5, with 1 being "least relevant" and 5 being "most relevant"" whether the rationale focuses on the specific aspect of the post that is being appraised, and whether it strays off-topic provides irrelevant information.
3) Justification: We ask human evaluators whether the rationale justifies the selected scale by adequately explaining why the selected rating scale is the most appropriate or relevant one to use for the aspect being evaluated. Annotators need to select either "Yes" or "No".
Alpaca-7B attains lower results compared to the other LLMs, especially in terms of the criteria fac- tuality and usefulness. FLAN-TS, on the other hand, ranks the worst on all criteria among the LLMs. Further analysis reveals that FLAN-TS oc- casionally generates responses for natural language rationales that are the same as its scale answers, resulting in irrelevant and useless rationales.
4) Usefulness: Finally, we evaluate whether the rationale provides useful or informative insights or explanations of useful information pertaining to the appraisal dimension being judged. Options of "Yes". "Maybe", and "No" can be selected.
Conclusion
To achieve a more accurate and holistic understand- ing of emotions from written text, NLP models need to work towards understanding the subjec- tive cognitive appraisals of emotions underlying
Annotators. We recruit annotators from the Amazon Mechanical Turk (MTurk) to work on our human evaluation task. The crowd workers were

---

FAC
REL
JUS
USE
dimensions analyze per post, and the annotation for one post for one trained annotator takes half an hour. Future work may establish the validity of training data obtained from LLMs, and explore approaches such as distillation.
EVALUATORS
0.590
0.718
0.576
0.668
Table 5: Inter-annotator agreement statistics for the human evaluation task, measured using Krippendorff's Alpha with interval distance.
In addition, we experiment with LLMs under zero-shot setup only, while we highlight that this is the first work towards the assessment of cognitive appraisals of emotions in language models, and it lays the foundation for future research on decipher- ing the intrinsic emotional dynamics that remain unexplored in current state-of- -ftt---rt- models. We believe that this warrants careful construction of the dataset with thorough analysis; and we leave these interesting engineering questions to future work.
situations. In this work, we construe empirical and explicit understanding of perceived cognitive appraisals in human readers and LLMs alike. We dataset of 241 Reddit posts annotated with a comprehensive range of 24 subjective cogmitive appraisals that follow a situation, along with their corresponding natu- ral language rationales. Experiments reveal that COVIDET--APPRAISAAS.. is a vital resource eval- uate the capability of language model to uncover implicit information for emotional understanding. Our thorough evaluation of LLMs' performance on assessing emotion appraisal dimensions empha- is a challenging
present
Acknowledgements
This research was partially supported by National Science Foundation (NSF) grant IIS-2107524. We thank Kathryn Kazanas and Keziah Kaylyn Reina for their dedication and hard work the anno- tation of COVIDET-APEE APPRAISALS. We also thank our reviewers for their insightful feedback and com- ments.
sizes that benchmark, and our in-depth human evaluation of the natural language rationales indicates potential areas of improvement (e.g., improving the factual- ity and usefulness of the rationales) for open-source LLMs.
Limitations
References
This work presents a new dataset entitled COVIEE-APPRAISAL. to evaluate LLMs' capa- bility in cognitive emotion appraisals. Due to the highly demanding nature of our task (e.g.. the same situation can result in different subjective evalua- tions), SEVIDE--PPRAASALLS is annotated by annotators. Future work can explore a larger pool annotators. Furthermore, it should be acknowl- restricted
Muhammad Abdul-Magecd and Lyle Ungar. 2017. EmoNet: fine-grained emotion detection with gated recurrent neural networks. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume Long Papers), pages 718-728, Vancouver, Canada. Association for Computational Linguistics.
Magda B Arnold. 1960. Emotion and personality. Columbia University Press.
edged that COVIDET--APPRAISAA to social media posts during the COVID-19 pan- demic, and they are written in English solely. This makes it challenging to evaluate LLMs ability in other domains as well as languages. Also, we note the appraisals we collect are from the perceived end, which are not subjective appraisals from the narrators and authors themselves.
Anshika Arora, Pinaki Chakraborty, M. P. S. Bhatia, and Prabhat Mittal. 2021. Role of Emotion in Excessive Use of Twitter During COVID-19 Imposed Lock- down in India. Journal of Technology in Behavioral Science. 6(2)37003377.
Ron Artstein and Massimo Poesio. 2008. Survey article: Inter-coder agreement for computational linguistics. Computational Linguistics, 34(4):555-596.
We note that the size of is relatively small. We have not intended this dataset to be one for supervised model training but rather a very high-quality dataset for evaluation (since this is the first dataset of its kind). key rea- son that the collection of appraisal annotations is both challenging and time-consuming: we have 24
David M Blei, Andrew Y Ng, and Michael I Jordan. 2003. Latent dirichlet allocation. Journal of Machine Learning Research, 393312:93-11022.
Asli Celikyilmaz, Elizabeth Clark, and Jianfeng Gao. 2020. Evaluation text generation: A survey, arXiv preprint arXiv:2006. 14799.

---

Mummun De Choudhury and Sushovan De. 2014. Men- tal health discourse on reddit: Self- disclosure, social support, and anonymity. In ICWSM.
Matej Gjurković and Jan Snajder. 2018. Reddit: A gold mine for personality prediction. In Proceedings of the Second Workshop on Computational Modeling of People's 's Opinions, Personality, and Emotions in Social Media, pages 87-97, New Orleans, Louisiana, USA. Association for Computational Linguistics.
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang. Mostafa Dehghani, Siddhartha Brahma, Albert Web- son, Shixiang Shane Gu, Zhuyun Dai, Mirac Suz- gun, Xinyun Chen. Aakanksha Chowdhery, Sharan Narang. Gaurav Mishra, Adams Yu, Vincent Zhao, Yanping Huang. Andrew Dai, Hongkun Yu, Slav Petrov, Ed H. Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoc V. Le, and Jason Wei. 2022. Scaling cnrructionfinettnee language models. arXiv preprint arXiv:2210. 11416.
Yuling Gu, Bhavana Dalvi, and Peter Clark. 2022. DREAM: Improving situational QA by first elab- orating the situation. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1115-1127, Seattle, United States. Association for Computational Lin- guistics.
Gerald I. Clore and Andrew Ortony. 2000. Cogmition in emotion: Always, sometimes, never. Cognitive neuroscience of emotion, pages 61.
Arnav Gudibande, Eric Wallace, Charlie Snell, Xinyang Geng, Hao Liu, Pieter Abbeel, Sergey Levine, and Dawn Song. 2023. The false promise of imitating proprietary 1lms. arXiv preprint arXiv:2305.15711
Mike Conover, Matt Hayes, Ankit Mathur, Jianwei Xie, Jun Wan, Sam Shah, Ali Ghodsi, Patrick Wendell, Matei Zaharia, and Reynold Xin. 2023. Free dolly: Introducing the world's first truly open instruction- tuned Ilm.
Jan Hofmann, Enrica Troiano, Kai Sassenberg, and Ro- man Klinger. 2020. Appraisal theories for emotion classification in text. In Proceedings the 28th Inter- national Conference Computational Linguistics. pages 125-138, Barcelona, Spain (Online). Interna- tional Committee on Computational Linguistics.
Dorottya Demszky, Dana Movshovitz-Attias, Jeongwao Ko, Alan Cowen, Gaurav Nemade, and Sujith Ravi. 2020. GoEmotions: A dataset of fine aee-grained emo- tions. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4040-405 Online. Association for Computational Linguisties.
Hamed Khanpour and Cornelia Caragea. 2018. Fine- grained emotion detection in health-related online posts. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1160-1166, Brussels, Belgium. Association for Computational Linguisties.
Shrey Desai, Comeela Caragea, and Junyi Jessy Li. 2020. Detecting perceived emotions in hurricane disasters.
Jan Kocoñ, Igor Cichecki, Oliwier Kaszyca, Mateusz Kochanek, Dominika Szydio, Joanna Baran, Julita Bielaniewicz, Marcin Gruza, Arkadiusz. Janz, Kamil Kanclerz, Anna Kocoñ, Bartlomiej Koptyra, Wik- toria Brazzzzznnk-KKoszzewiee
In Proceedings of the 58th Annual Meeting of the As- sociation for Computational Linguistics, pages 5290- 5305, Online. Association for Computational Lin- guistics.
Piotr Milkowski, Marcin Oleksy, Maciej Piasecki, Eukasz Radlíński, Konrad Wojtasik, Stanistaw Wožniak, and Prze- myslaw Kazienko. 2023. ChatGPT: Jack of all trades, master of none. Information Fusion, 99: 101861.
Phoebe C Ellsworth and Klaus R Scherer. 2003. Ap- praisal processes in emotion. Oxford University Press.
Jan Kocoñ, Igor Cichecki, Oliwier Kaszyca, Mateusz Kochanek, Dominika Szydto. Joanna Baran, Julita Bielaniewicz, Marcin Gruza, Arkadiusz Janz, Kamil Kanclerz, et al. 2023. Chatgpt: Jack of all trades. master of none. Information Fusion, page 101861.
JL Fleiss. 1971. ment among many raters. Psychological bulletin, 76(5):378 382.
Measuring nominal scale agree-
Sebastian Gehrmann, Tosin Adewumi, Karmanya Aggarwal, Pawan Sasanka Ammanamanchi, Anuoluwapo Aremu, Antoine Bosselut, Khy- athi Raghavi Chandu, Miruna Clinciu, Dipanjan Das, Kaustubh Dhole, et al. 2021. The gem benchmark: Natural language generation. its evaluation and metrics. In Proceedings of the 1st Workshop on Natural Language Generation, Evaluation, and Metrics (GEM 2021), pages 96-120.
Klaus Krippendorff. 1980. Content analysis intro- duction to its methodology. Sage commtext series. Sage Publications. Beverly Hills.
Richard S Lazarus. 1966. Psychological stress and the coping process. McGraw-Hii
Richard S Lazarus. 1993. From psychological stress to the emotions: history of changing outlooks. Annual review of psychology. 44(1):1-22.
Leilani H. Gilpin, David Bau, Ben Z. Yuan, Ayesha Ba- jwa, Michael A. Specter, and Lalana Kagal. 2018. Ex- plaining explanations: An overview of interpretabil- ity of machine learning, 2018 IEEE 5th International Conference on Data Science and Advanced Analytics (DSAA), pages 80-89.
Richard S Lazarus, Allen D Kanner, and Susan Folkman, 1980. Emotions: A analysis. In Theories of emotion, pages 189-217. Elsevier.

---

Teven Le Scao and Alexander Rush. 2021. How many data points is a prompt worth? In Proceedings of the 2021 Conference of the North American Chap- ter the Association for Computational Linguistics: Human Language Technologies, pages 2627-2636, Online. Association for Computational Linguistics.
Desmond C. Ong. Zhengxuan Wu. Zhi-Xuan Tan, Mar- ianne C. Reddan, Isabella Kahhalé, Alison Mattek, and Jamil Zaki. 2019. Modeling emotion in complex stories: The stanford emotional narratives dataset. IEEE Transactions on Affective Computing, 12:579- 594.
Desmond C. Ong. Jamil Zaki, and Noah D. Goodman. 2015. Affective cognition: Exploring lay theories of emotion. Cogmition, 143:141-162.
Jingsheng Lei, Yanghui Rao, Qing Li, Xiaojun Quan, and Liu Wenyin. 2014. Towards building social emotion detection system for online news. Future Generation Computer Systems, 37:438-444
Andrew Ortony, Gerald L Clore, and Allan Collins. 2022. The cognitive structure of emotions. Cam- bridge university press.
Ming Li, Yusheng Su, UisiuYYuan Huang. Jiali Cheng, Xin Hu, Xinmiao Zhang, Huadong Wang. Yujia Qin, Xiaozhi Wang. Zhiyuan Liu, and Dan Zhang. 2023. Human emotion knowledge representation emerges in large language models and supports discrete emo- tion inference. arXiv preprint 2rXiv:2302.09582
Kishore Papineni, Salim Roukos, Todd Ward, and Wei- Jing Zhu. 2002. Bleu: a method for automatic evalu- ation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Compu- national Linguistics, pages 311-318, Philadelphia. Pennsylvania, USA. Association for Computational Linguistics.
Chin- Yew Lin. 2004. ROUGE: A package for auto- matic evaluation of summaries. In Text Summarizu- fion Branches Out, pages 74-81, Barcelona, Spain. Association for Computational Linguistics.
Ofir Press, Muru Zhang, Sewon Min, Ludwig Schmidt, Noah A Smith, and Mike Lewis. 2022. Measuring and narrowing the compositionality gap in language models. arXiv preprini arXiv:2210.0333
Chen Liu, Muhammad Osama, and Anderson De An- drade. 2019. DENS: A dataset for multi-class emo- tion analysis. In Proceedings of the 2019 Confer- ence on Empirical Methods in Natural Language Pro- cessing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)), pages 6293-6298, Hong Kong, China, Association for Computational Linguistics.
Chengwei Qin, Aston Zhang, Zhuosheng Zhang. Jiaao Chen, Michihiro Yasunaga, and Diyi Yang. 2023. Is chatgpt general-purpose natural language process- ing task solver? arXiv preprint arXiv:2302.06477
Colin Raffel, Noam Shazeer, Adam Roberts, Kather- ine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1-67.
Ana Marasović, Chandra Bhagavatula, Jae sung Park, Ronan Le Bras, Noah A. Smith, and Yejin Choi. 2020. Natural language rationales with full-stack visual reasoning: From pixels to semantic frames to commonsense graphs. In Findings of the Association for Computational Linguistics: EMNLP 2020. pages 2810-2822 Online. Association for Computational Linguistics.
Justus J Randolph. 2005.
Free- tree-marginal multirater kappa (multirater k [free]]: An alternative to fleiss' Exed-marginal multirater kappa. Online submission.
Ira Roseman, 1984. Cognitive determinants of emo- tion: A structural theory. Review of personality & social psychology.
Andreas Marfurt and James Henderson. 2021. Sentence- level planning for especially abstractive summariza- tion. In Proceedings of the Third Workshop on New Frontiers in Summarization, pages 1-14, Online and Dominican Republic. Association for Computa- tional Linguistics.
Ira J Roseman. 1996. Appraisal determinants of emo- tions: Constructing more accurate and comprehen- sive theory. Cognition & Emotion, 10(3):241225)
Rada Mihalcea and Carlo Strapparava. 2012. Lyrics, music, and emotions. In Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, pages 590-599, Jeju Island, Ko- rea. Association for Computational Linguistics.
Ira J Roseman and Craig A Smith. 2001. Appraisal the- ory. Appraisal processes in emotion: Theory, meth- ods, research, pages 3-19.
David Sander, Didier Grandjean, and Klaus R Scherer. 2005. A systems approach to appraisal mechanisms in emotion. Neural networks, 18(4)3317-32.
Agnes Moors. 2010. Theories of emotion causation: review. Psychology Press.
Klaus R. Scherer. 2005. What are emotions? and how can they be measured? Social Science Information, 44:695-729
Agnes Moors, Phoche C. Ellsworth, Klaus R. Scherer, and Nico H. Frijda. 2013. Appraisal theories of emo- tion: State the art and future development. Emo- tion Review, ((2):119-122
Klaus R Scherer and Agnes Moors. 2019. The emotion process: Event appraisal and component differentia- tion. Annual review of psychology, 70:719-745.

---

Klaus R Scherer 1984, On the nature and function of emotion: A component process approach. Ap- proaches to emotion, 2293(317:31..
Yizhong Wang. Yeganch Kordi, Swaroop Mishra, Al- isa Liu, Noah A Smith, Daniel Khashabi, and Han- naneh Hajishirzi. 2022. Self-instruct: Aligning lan- guage model with self generated instructions. arXiv preprimt arXiv:2212. 10560.
KR Scherer, A Schorr, and T Johnstone. 2001. Ap- praisal theory: Overview, assumptions, varieties.
Bernard Weiner. 1985. An attributional theory of achievement motivation and emotion. Psychologi- cal review, 92(4):548.
Craig A Smith and Phoebe C Ellsworth. 1985. Pat-
temss of cognitive appraisal in emotion. Journal of personality and social psychology, 48(4))813.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- rie Cistac, Tim Rault, Remi Louf, Morgan Funtow- icz, Joe Davison, Sam Shleifer. Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. Trans- formers: State-of-the-an thr-trrrrtttt natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38-45, Online. Association for Computational Linguistics.
Tiberiu Sosea and Cornelia Caragea. 2020. Cancer- Emo: A dataset for fine-grained emotion detection. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 8892-8904, Online. Association for Computa- tional Linguistics.
Tiberiu Sosea, Chau Pham, Alexander Tekle, Cornelia Caragea, and Junyi Jessy Li. 2022. Emotion analysis and detection during COVID-19. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 6938-6947, Marseille, France. Eu- ropean Language Resources Association.
Kailai Yang. Shaoxiong Ji, Tianlin Zhang. Qianqian Xie, and Sophia Ananiadou. 2023. Towards interpretable mental health analysis with large language models. arXiv preprint 2:X.h:230400344.
Tiberiu Sosea, Hongli Zhan, Junyi Jessy Li, and Cor- nelia Caragea. 2023. Unsupervised extractive sum- marization of emotion triggers. In Proceedings of the 61st Annual Meeting of the Association for Compu- lational Linguistics (Volume 1: Long Papers), pages 9550-9569, Toronto, Canada. Association for Com- putational Linguistics.
Xi Ye and Greg Durrett. 2022. The unreliability of ex- planations few-shot prompting for textual reason- ing. In Advances in Neural Information Processing Systems.
Carlo Strapparava and Rada Mihalcea. 2007. SemEval- 2007 task 14; Affective text. In Proceedings of the Fourth International Workshop on Semantic Evalua- tions (SemEval-2007), pages 70-74, Prague, Czech Republic. Association for Computational Linguistics.
Gerard Yeo and Desmond C. Ong. 2023. A meta- analytic review of the associations between cognitive appraisals and emotions in cognitive appraisal theory. PsyArXiv.
Teo Susujak. 2023. Applying bert and chatgpt for senti- ment analysis of lyme disease in scientific literature. arXiv preprint arXiv:2302.0644
Hongli Zhan, Tiberiu Sosea, Cornelia Caragea, and Junyi Jessy Li. 2022. Why do you feel this way? summarizing triggers of emotions in social media posts. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 9436-9453, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. Stanford alpaca: An instruction- followiig Ilama model. https:// gi git.. cowm/atsu-lleec Saattsuullbb/se stw/sstdd_aaaaaaaa
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Wein- berger, and Yoav Artzi. 2019. Bertscore: Evaluating text generation with BERT. In International Confer- ence on Learning Representations.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Maric-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. Ilama: Open and effi- cient foundation language models. arXiv preprint arXiv:2302.1397
Qihuang Zhong, Liang Ding. Juhua Liu, Bo Du, and Dacheng Tao. 2023. Can chatgpt understand too? comparative study on chatgpt and fine-tuned bert. arXiv preprint a30iv:2302.1 10198.
Ana-Sabina Uban, Berta Chulvi, and Paolo Rosso. 2021. An emotion and cognitive based analysis of men- tal health disorders from social media data. Future Generation Computer Systems, 124:480 494.
Caleb Ziems, Minzhi Li, Anthony Zhang, and Diyi Yang. 2022, Inducing positive perspectives with text reframing. In Proceedings of the 60th Annual Meet- ing of the Association for Computational Linguistics (Volume I: Long Papers), pages 3682 -37000 Dublin, Ireland. Association for Computational Linguistics.
Wenbo Wang, Lu Chen, Krishnaprasad Thirumarayan. and A. Sheth. 2012. Harnessing twitter "big data" for automatic emotion identification. 2012 International Conference Privacy, Security, Risk and Trust and 2012 International Confernece on Social Computing, pages 587-592.

---

A Dataset Example and LLM Responses
Expert Lingustt' inter-Aunotator Agreement
0.8
Spaamnn's
0.6
Figure 10, Figure 11, and Figure 12, we show- case an annotation from GOVIDET-APPRAISAA together with LLMs' responses regarding dimen- sion 3 crsp (crreumstancee-responssbi dition to LLMs evaluated in this paper (including ChatGPT, FLAN-T5-XXL, Alpaca (7B, 13B), and Dolly-V2 (7B, 12B)), we also present responses models
-0
0.0
efc
ptc
Srsp
cor
pla
oR
the
gend
crse
Cn fex
prd
8ttn
fim
loss
exp
griv
In ad-
Dimensione
Figure Inter-annotator agreement of the Likert-scale ratings within each dimension. The dimensions are ranked by the order of Spearman's 's p. and the colors in- dicate the inter-annolator agreement measured by Krip- pendorff's 's alpha using interval distance.
elicited from other non-instruction-tun such as GPT-3-davinci (a vanilla base model of GPT-3) and LLaMA (7B, 13B) (Touvron et al., 2023) using the "2-step" prompting template given in Figure 17. As the example shows, these non- cnstruction-tuness LLMs perform poorly of cognitive emotion appraisal, generating nonsen- sical responses for both selecting Likert-scall ings as well as providing natural language ratio- nales.
AGR
DSG
FER
JOY
SDN
TRS
ANC
AVG
F1
0.18
0.13
0.40
0.26
0.29
0.06
0.23
0.22
task
Table 6: Fl scores of each emotion using the trained logistic regression model the test set.
D Additional Dataset Analyses
D.1 Are the Dimensions Informative for Emotions?
B Dataset Annotation Framework
The cognitive appraisal theories provide insights into the nature of the appraisal dimensions in distin- guishing various emotions (Hofmann et al., 2020; Yeo and Ong, 2023): while different individuals may appraise the same situation distinctively, they are more likely to experience the same emotion when consistent appraisal pattern emerges. For example, the cognitive dimension pls (pleasant- ness) is often linked to joy, but unlikely to be sociated with disgust (Smith and Ellsworth, 1985). Therefore, specific emotions are hypothesized stem from corresponding appraisal patterns (Yeo and Ong, 2023). By understanding how individ- uals appraise the situations they experience, we can subsequently make predictions regarding their emotional state. As a result, appraisal dimensions valuable in differentiating emotional states, pecially in cases where the emotions are highly interchangeable (e.g., disgust and anger).
We provide the instructions given to the annota- tors in Figure 13. In addition, we also provide the layout for the annotation task (which includes the full questions for each of the 24 cognitive emotion appraisal dimensions abbreviated in Table 1) in Figures 14, 15, 16.
C Inter-Annotator Agreement by Dimension in CovIDET-APPRAIANT
To better understand the inter-annotator agreement pertaining to each emotion appraisal dimension in COVIDET-APPRAISAN
measure Spearman's P and Krippendorff's alpha on each of the 21 appli- cable dimensions. We provide the inter-annotator agreement statistics per dimension in Figure As the plot shows, the human annotators have strong agreement on dimensions such as efc (emotion- focused coping) and pfc pproblem-focuset coping), whilst disagreeing with each other most often on dimensions griv (goal relevance), exp (expected- ness), and loss. This can be attributed the nature of our domain: in these Reddit posts, the narrator is mainly sharing their experiences life around COVID-19, while preserving doubts about the fu- ture.
Here, using the cognitive appraisal dimensions annotated in COVIDET-APPRAISALS we further explore and validate whether these appraisal di- mensions alone are indicative of perceived emo- tions already annotated in COVIDET. While in the ideal scenario, both the appraisal and the objec- tive event need be present for emotion predic- tion, this small experiment will allow us to gauge which dimensions are more likely discriminative for a particular emotion. For each of the emo-

---

ID
Abbrv.
Reader-Friendly Labels
Anger
Fear
Joy
Sadness
Disgust
1
srsp
Self-responsibility
2
orsp
Olher--esponsiiiiity
3
crsp
Ciruusstunces-resoonssit
4
Pfc
frobsem-focused coping
5
grlv
Goal Relevance
6
ain
Attentional activity
7
efc
Emotion- focused coping
8
scrl
Self-Comtrollable
9
ocrl
Cottr-Conte
10
cerl Aueuustances-Contrllaan
11
prd Predictability
thr Threat
13
ply
Pleasantness
14
crt
Certainty
15
gend
Goal Conduciveness
17
fex
Future expectancy
19
loss
Loss
20
fill
Familiarity
eff
Effort
22
chl
Challenge
24
exp
Expectedness
Table 7: Cognitive emotion appraisal dimensions that are predictive of emotions (including anger. fnaer.fear, joy, sadness, and disgust), identified by recent meta-analysis conducted by Yco and Ong (2023). + indicates appraisal dimensions that are significantly positively predictive of emotions, and indicates appraisal dimensions that are significantly negatively predictive of emotions. We highlight in red the indicative appraisal dimensions captured by our logistic regression models that in line with Yeo and Ong (2023)'s 's findings. signifies weights our logistic regression models with 0, and if signifies significant weights with < 0.05.
tion classes labeled in COVIDET we train a lo- gistic regression model using the scales the an- notated 21 applicable appraisal dimensions as fea- tures. We split COVIDET- APTT-APARAIALL using a random 80:20 train-test partitioning, and aggregate the Likert-scale ratings for the 40 posts that are dou- bly annotated by human annotators following the aggregation setup discussed in $3. We down- sample the training data for each logistic regression model to handle class imbalance issues. addi- tion, we encode the "not mentioned" (NA) labels as an independent real-valued feature, and substitute their values with 0. To prevent features of different scales or magnitudes from having disproportion- ate influence on the models, Z-normalize the scale ratings within each dimension for each anno- tator.
test set across all emotions.
To reveal the appraisal dimensions that are in- dicative of each emotion, we examine the weights from the trained logistic regression models. Specif- ically, we aim to validate the emotion appraisal dimensions that Yeo and Ong (2023) identified be predictive of emotions (including anger, fear, joy, sadness, and disgust) from prior studies in psy- chology. In Table 7, we show the appraisal dimen- sions found to either positively predictive (+) or negatively predictive (-) of emotions. Please note that these indications are extracted from re- cent meta-analysis from Yeo and Ong (2023) with significance (p < 0.05). In Table we highlight the indicative appraisal dimensions captured by our logistic regression models that are in line with Yeo and Ong (2023)'s findings. We observe a certain degree of overlap between Yeo and Ong (2023)* ss identified emotion appraisal dimensions that are predictive of emotions and those captured by our logistic regression models. It should be noted that some uppraisal dimensions may not be useful for all emotions included in Table since in COVIDET there are no Reddit posts annotated with neutral emotions: for example, as shown in Table 7, crsp (crrumstaacc--ssposiiiillll responsibility) is found to be posi- tively indicative for fear and sadness, while neutral
The Fl scores for each emotion using the trained logistic regression models on the test set are re- ported in Table 6. We observe that the models are most capable at predicting emotions such as fear and sadness, whilst performing poorly on emotions disgust and frust. This is possibly due to the do- main of our dataset: in COVIDET, fear and sadness are the most commonly found emotions whereas disgust and trust are scarcely present. On average, the classifiers achieve an average F1 0.22 on the
(circumstances

---

srsp
orsp
crsp
pfc
grly
alln
efc
believe
responsible
control
cope
finds
attend
cope
responsible
people believes
believes
believe
concerns
believes
emotionally
does
cireumstances
doesn highly
need
somewhat
docsn
does
covid
coping
relevant
want
feeling
causing
covid
responsible
having covid
believe
struggling
focused
vaceinated
blame
vaccine
infected
covid
helieve
reaction
believe
delta
believes stuck
advice
covid
believes
somewhat
outside
covid
dose
asking
believes
somewhat
blame
pandemic
difficult
ending
pandemic
doesn
vaccinated
causing
worried
time pandemic
trying coping
scrl
ocrl
ccrl
prd thr
pls
crt
control
people
control
happen threatened finds
uncertain
believe
control covid
believe covid
unpleasant
unsure
does believes
believes
predict feels feeling
certain
believes
wait
cireumstunces
doesn does
covid
consequences
doesn vaccine
outside
covid don
express
pandemic
vaccine
covid covid
delta
feeling
worried
covid
somewhat vaccine does
feel
understands
unable prediction makes
health
pleasant
understand
understand
threat
confused
somewhat
vaccinated
believe
believe
somewhat
feel
delta
pandemic september
pandemic
information
sense
vaccine
fully
gend fex
loss
fml
elf
chl
want
worse
sense
subject
effort deal
finds
occur
finds better
does
information
challenging
did
inconsistent
believe
express
meaning
mental covid
expect
covid
does
loss
advice believes
vaccinated
mentions
highly
believes
lost
asking lot mentions
highly
somewhat
wants getting
believes
exert pandemic
expected
vaccinated
covid
covid
unfamiliar
try vaccine
covid
don
delta
pandemic
familiar
believe worried covid need
expecting
feel
worried variant
vaccinated
covid
delta
mention
trying
opportunity
somewhat
variant
vaccinated
Table 8: I.DA results the annotated rationales for each appraisal dimension.
for all other emotions. However, when compared to neutral emotions (i.e., in texts where no emotions are present), crsp pntruumstancee-rrspossihiii may be negative indicator for disgust. Therefore, experimenting with WWWIIDE--APRRALEE not reveal the extensive range of appraisal dimen- sions indicative of each emotion. Further investi- gations are needed to explore the predictability of these appraisal dimensions for emotions compared against neutral emotions.
praisal dimension being assessed. For example, dimension crsp (ciruunstance--resoosiiiii observe narrators of Reddit posts worrying about and blaming Delta, COVID- 19 variant, for caus- ing the status quo, whereas in dimension fil (fa- miliarity) we note people are generally unfamiliar with the situation, they are prone seek advice and probe for information on the forum.
we
may
D.3 An Example of Semantic Similarity
As discussed in $4, commonly used automatic measures such as BLEU (Papineni et al., 2002). ROUGE (Lin, 2004), and BERTScore (Zhang et al., 2019) do not adequately capture semantic similar- ity in COVIDET-APPRAISAIS Taking the post in Figure/ for example. Both rationales for di- mension 24, namely "The narrator mentions how people who vaccinated and mildly sick are still experiencing long COVID symptoms. They seem surprised by the continued COVID symptoms peo- ple are experiencing and how the situation seems to evolve." " and "The narrator really didn't 't expect
D.2 Topic Variations in Rationales
We use Latent Dirichlet Allocation (LDA) (Blei et al., 2003) to extract topics from the natu- ral language rationales annotated in COVIDET- APPRAISAL.S. Stop-words such common En- glish function words and words that occur fre- quently in our instructions (e.g., narrator, silu- ation) are removed prior the topic modeling. The most prominent topic extracted by the LDA model for each dimension is shown Table 8. We notice clear patterns of topics related to the ap-

---

BLEU BLEU-3
ROUGE ROUGE-2
BERTSCORE RE-SCALED
BLEU-2
BLEU-4
ROUGE-1
ROUGE-L
BERTSCORE
CHATGPT
0.147
0.078
0.044
0.317
0.111
0.224
0.890
0.347
ALPACA-78
0.136
0.069
0.040
0.292
0.101
0.230
0.881
0.297
A1PACA-13B
0.007
0.004
0.003
0.019
0.005
0.017
0.842
0.066
DOLLY-7B
0.067
0.034
0.020
0.185
0.047
0.142
0.858
0.157
DOLLY-12B
0
0.043
0.024
0.223
0.066
0.165
0,865
0,199
FLAN-T5-XXL
0.026
0.014
0.008
0.091
0.018
0.066
0.840
0.053
Table The full rationale statistics measured for LLMs' responses against the gold annotations, measured across independent runs.
FAC
REL
JUS
USE
gold annotations in Table 9.
EVALUATORS
0.721
0.711
0.632
0.672
As discussed $7.1, ChatGPT is the most per- formant language model in providing natural lan- guage rationales, with values from these metrics comparable those between different rationales from our annotators. Alpaca-7B also achieves comparable performance in these automatic mea- sures, despite its relatively poor capability terms of selecting aikert-scale ratings.
Table 10: Inter-annotator agreement statistics for the human evaluation task, measured using Krippendorff's Alpha with interval distance.
this situation since they mention being able to taste freedom, believing the pandemic is ending, when suddenly they heard news that vaccinated people still getting long covid and now they think the pandemic will never end." convey the reasons for why the narrator fails to expect the situation to oc- cur. However, the automatic metries reveal low agreement between these two rationales, with a BLEU-4 score of 0.018, ROUGE-L of 0. 0.231, and a re-scaled BERTSCORE of 0.237. This finding is in line with work showing the challenges of eval- uating generation (Gehrmann et al., 2021; Celiky- ilmaz al., 2020); similarly conclude that au- tomatic evaluation metries may poorly reflect the correctness of rationale for subjective emotion appraisal dimension.
In addition, we observe that other language mod- els such as FLAN-T5 and Dolly lag behind con- siderably compared to ChatGPT and Alpaca-7B. Enchantingly, the automatic metrics suggest that Alpaca- 13B is the worst language model among our LLMs under assessment, with markable degrada- tion from Alpaca-7B. Further investigation reveals that Alpaca-13B tends to respond with "Tell us why. 
" when prompted to generate the natural language rationale for the Likert-scale rating it se- lects, which takes up more than 84% of its rationale responses. The debasement of the Alpaca model in spite of the increase in the model's scale raises questions regarding the scaling law in our current task of appraising cognitive emotion dimensions in context. The templates for prompting the LLMs are shown in Figure 17. We use "1-step" prompting to clicit both rating and rationale with single prompt from ChatGPT. For all other language models, we apply "2-step" prompting, which first elicits the rating for the appraisal dimension, then conditioned on the response for the rating we further elicit the rationale for the selection.
E Prompt Templates
Rationale Human Evaluation. We provide the box plots of the results from the human evaluation for the most morf-performant 3 language models (i.c., ChatGPT, Alpaca- 7B, and FLAN-T5) in Figure 6. Furthermore, we also provide the results for the human evaluation regarding all 6 LLMs assessed in this paper. Following the setup in $7.2, we evaluate and analyze LLM-generated rationales when the model made a near-correct prediction of the Likert- scale rating for that particular dimension compared against the gold human ratings. Specifically, we sample the intersection of dimensions (post, di- mension) tuples where all 6 L.MM' (i.e., ChatGPT, FLAN-T5, Alpaca-7B. 7B, Alpaca-1333 Dolly-7B, and Dolly-12B) ratings fall in the range of an abso-
F Full LLM Rationale Measures
Rationale Automatic Evaluation. We provide the full statistics of the automatic rationale agree- ment measured using BLEU (Papineni et al., 2002), ROUGE (Lin, 2004), and BERTScore (Zhang et al., 2019) for the all 6 LLMs' responses against the

---

LENGTH TOKENS
ABSTRACTIVENESS NoVEEL BIGRAMS
HUMAN
EVAL
BLEU-4
ROUGE-I
BERTSC
FAC
REL
JUS
USE
ANNOTATORS
28.9
86.7%
0.68
4.43
0.92
0.77
CHATGPT
58.0
81.8%
0.044
0.224
0.347
0.88
4.42
0.85
0.88
FLAN-T5
45.3
16.0%
0.008
0.066
0.053
0.44
2.27
0.25
0.19
ALPACA-7B
48.6
71.9%
0,040
0.230
0.297
0.57
4.23
0.79
0.64
ALPACA-13R
19.7
10.9%
0.003
0.017
0.066
0.03
1.13
0.02
0.02
DOLLY-7B
79.7
0.157
0.32
2.
0.21
DOLLY-12B
73.3
55.1%
0.021
0.165
0.199
0.38
2.79
0.56
0.38
Table 11: Experiment results from LLMs. We report the average performance across five independent runs, A more comprehensive report of the automatic metrics BLEEU-4, ROUGE-I, and BERTSCORE is provided in Table 9, Appendix $F.
festuaity
16
ea
usddumm mssdumess
04
R
22
80
Bod
ehagee
eipasa-7 7
Bede
chatgst
fan-S-ad
appree 7B
mid
chatignt
Bon-B-dd
appaca-1
alpaste (3B
deljui2-7
dell 12B
gold
eteepol
appesae
alpaca- BB
dely--2-2 47
dady 128
Figure 6: Box plots of the results from the human evalu- ation task for the most-performant LLMs (i.c., Chat- GPT, Alpaca-7B, and FLAN-T5).
Figure 7: Box plots of the results from the human evalu- ation task for all 6 LLMs.
the box plots for each LLM's Likert-scale ratings are shown in Figure 9.
lute difference of 1 to one of the annotated scale- ratings. This results in 30 rationales annotated by human annotators and 26 natural language ratio- nales from each LLM. We report the inter-evaluator agreement using Krippendorff's Alpha with inter- val distance in Table 10, which shows substantial agreement (Artstein and Poesio, 2008) across all criteria.
H Human Evaluation Framework
We provide the instructions given to the human evaluators of the rationales (described in $7.2) in Figure 18 and Figure 19. Additionally, we show- case the human evaluation task layout in Figure 20.
Results from the human evaluation for all 6 LLMs reported in Table 11. We observe that apart from ChatGPT and Alpaca-7B, all other LLMs including FLAN-T5, Alpaca-13B. Dolly-7B, and Dolly-12B achieve similarly low performance on providing natural language rationales for cogni- tive emotion appraisals. We provide the box plots of the results from the human evaluation for all 6 language models in Figure 7.
Why Does ChatGPT Perform (Slightly) Better Than Human Annotators in Providing Rationales?
As discussed $7.2, ChatGPT was scored slightly higher in terms of factuality and usefulness on pro- viding natural language rationales than our human annotators, according to human evaluators. This can be attributed to ChatGPT's wordiness and ex- tractiveness (as shown in Table 4), especially in cognitive emotion appraisal dimensions where the scale rating is low. As an example, we showcase in Table 12 where both ChatGPT and our human annotator give the same rating for a dimension, but ChatGPT scores higher than our human experts on
G Model Responses Analyses
The LLMs' performance in terms of Likert-scale rating selections measured using Spearman corre- lation and Krippendorff's alpha against the gold annotations are shown in Figure 8. Additionally,

---

chelgpt: Per Dimension Performance
Ian-tbood Per Ulmension Performance
0.50
0.8
0.8
Speereerrpp
-0.4
0.00
0..
-0.25
-0.00
0.0
gond
P
gM
001
crsp
Eml
ath
pro
0..0
gond
phc
pr
mimansions
Dimerwiore
aipaca-D:: Per Dimension Performance
alparne 13B Cimentiion Performenco
Speements g
0.2
0.2
0.0
-02
-02
-04
pb
1h
021
gM
Tni
pid
aond
46
chi
1050
e59
corl
ail
Dimenstons
Dimensoone
doly-2277:: Per Dimmmiionn Parfommenee
dolly22 1222 Per Dimension Performance
00
0.1
0.0
0.10
Sperrmnts
0.0
Speurments
0.00
0.00
-048
-0.06
-06
36
pd
pa
kmi
spp
prd
fm
po
chi
ptc
ath
51
et
ON tel.
gand orsp
Dimensions
Figure 8: L.MSs' performance terms of Spearman correlation and Krippendorff's alpha (using interval distance) against the gold annotations within each group of dimensions (averaged performance across 5 independent runs).
metrics factuality and usefulness.
REDDIT POST
As shown in the example, given the same Red- dit post as well as the instruction to evaluate the cognitive emotion appraisal dimension orcl (other- controllable), both our human annotator and Chat- GPT give Likert rating of 1 indicating small extent which the narrator believes other people were controlling what was happening in the situa- tion. Similarly, in their natural language rationales, both our annotator and ChatGPT express that there is no mention of other people controlling the sit- uation in the Reddit post. However, in the post, although to a small degree, the narrator is hinting that other people are in control of the situation since their school is deciding the mask policy for the school year. This lowers the human annotation to a mean factuality rating of 0.25 and a mean useful- ness rating of 0.50. On the other hand, ChatGPT iterates and summarizes the information provided in the post, mentioning that the narrator is "seek- ing advice on where to find masks that are more suitable for their needs". The provision of such information thus leads to higher score than our annotator, resulting in a mean rating of 0.75 for both factualit and usefulness.
want to elarify that am in no way shape an anti masker, believe we do need them during the pandemic and any future outhreaks. However, the masks a nightmare for me person- ally. They trigger off my migraines, could hurdly concentrute at school because of it. also have sensory disability. which makes wearing masks 10 times harder. Everything is hrighter, louder, and everything feels more crowded when I wear them. Its horrible. And it looks like we have to wear them for the next school year,
my final year. cant it. So know if any you know where get masks that migraine/sensory friendly? While still doing the job normal medical mask does. Thunks.
APPRAISAL DIMENSION ocrl To what extent did the narrator think that OTHER PEOPLE were controlling what was happening in the situation?
HUMAN ANNOTATIONS
SCALE:
RATIONALE: The narrator does mention anyone else
controlling what's 's happening the situation.
FACTUALITY: 0.25
(Minor Error + No)
USEFULNESS: 0.50
(Maybe + MMybb)
CHATGPT'S RESPONSES
SCALE: 1
RATIONALE: The text does not suggest that other people are controlling the situation. The varndor sumply seeking advice on where to find masks that are more suitable for their needs.
FACTUaLITY: 0.75
(Yes
Minor Error)
USEFULNESS: 0.75
(Yes + Maybe)
Table 12: An example found in our human evaluation where ChatGPT scores higher than our human annota- tors on metrics factuality and usefulness.

---

chatgpt: Scale Distribution of Each Dimension
flan-t5-oll: Scale Distrbution of Each Dimension
500p
0190
pM on
gond
fmd
eif
chi
eep
509
0000
coo
m
Tox gndd
l0s0
elf
Dimensions
Dimonsions
alpaca-78: Scale Distribution of Each Dimension
alpaca-13 Scale Distribution of Each Dimension
Scale
1000
crsp
0790
plc
gttu
dfc
val
oct
pPd
the
pls
on
gand
chi
oxp
oreg
pfc
gfv
atn
elc
good
tax
tmi
Eimensiona
Dimensions
dolly-v2-7B: Scale Distribution of Each Dimension
dolly-v2-12B: Scale Distribution of Each Dimension
Scale
50p
orsp
gri
oct
201
on
gond
en
cHi
exg
stop
onp
gth
atn
elc
gond
Tox
loss
eff dN esp
Dimensions
Dimensions
Figure 9: Box-plots for the LLMSs Likert-scale rating responses, measured independent

---

Reddit Post
I've recently heard of stories that people who are getting mildly sick after being vaccinated are still coming down with long covid. People on the covid long hauler subreddit are saying that we're going to have huge problem on our hands because no vaccine protects against long covid...This isn't going end, isn't Right as think tasted freedom, out information that makes me want stay inside forever like hermit...
Cognitive Emotion Appraisal Dimension
Dimension
extent did the narrator think CIRCUMSTANCES BEYOND ANYONE'S CONTROL were responsible causing the situation? (For example, the thought that the situation caused luck, weather, by other things outside anyone's control.)
Example Annotations and Models' Responses
[An example from CovidET-Appprisals)
Human Annotators
[Scale]:
[Rationate]: The narrator mentions long COVID and the inability to currently protect against long COVID. This is factor that is still outside of anyone's control, including doctors and nurses, and the situation continues to evolve in uncontrollable ways.
[Responses from LLMs evaluated in the paperl
ChatGPT
[Scale]: 7
[Rationale]: The narrator seerss acknowledging that situation is beyond anyone's control, they mention
of helplessness and resignation, stating that "This isn't going end, isn't it?" and feeling like they want "stay inside forever like hermit." However, they do not completely responsible, they may still believe that there some actions
"stories" and information that they have heard. They also express
explicitly state that circumstances that can be taken to mitigate the situation.
FLAN-TS-XXXL (11B)
[Scale]:
[Rationale]: 1
Alpaca (7B)
[Scale]:
|Rationale]: The narrator believes that circumstances beyond anyone's control were responsible for causing the situation a moderate extent. They believe that luck, weather, and other external factors played role in creatinng the current situation, but that it was not completely due to luck or external factors.
Alpaca (13B)
[Scale]:
|Rationale]: Tell us why.
ca Dol (7B)
> [Scale]: 0
[Rationale]: don't what you're talking about, going go outside play my new remote-controlled helicopter.
Dolly-V2 (12B)
[Scale]: 7
[Rationale]: I think circumstances were completely responsible for causing the situation. The narrator believes that getting vaccinated gives you the same immunity to the long covid as natural infection would. However, there is research to suggest that this is not the case. The narrator also assuming that everyone who has gotten vaccinated has gotten the same ineffective vaccination as him, which is not the case. Additionally, the text suggests that the long
Figure 10: A comprehensive example from COVIDET-APPRISAAA for Dimension 3 (circumsttanceessssppoiiibblllll
(Page 1/3).

---

covid is not going to end, which is also not the case. The long covid is pandemic that is going to last for many years, not a few months. The pandemic going to cause huge problem on the US population, because there no vaccine that protects against long covid, and the long covid can cause serious health issues.
|Responses from other LLMs]
LLMM (7B)
[Scale]: think the narrator thinks that circumstances beyond anyone's control are responsible for causing the situation. The narrator says that people who are getting mildly sick after being vaccinated are still coming down with long covid. The narrator also says that people on the covid long hauler subreddit are saying that we're going to have a huge problem going to end, isn't it? Right as think I've tasted freedom, I find out information that makes me want to stay inside forever like a hermit. An Instruction: \nTo what extent did the narrator think that PEOPLE'S BEHAVIOR was responsible for causing the situation? (For example, if the narrator thought that the situation was caused by people's bad decisions, or by people's lack of knowledge, or by people's lack of effort, or by people's lack of concern, or by people's lack of cooperation, or by people's lack of self-control, or by people's lack of self-dis |Rationale]: I think the narrator thinks that people's behavior is respansible for causing the situation. The narrator says that people who are getting mildly sick after being vaccinated are still coming down with long covid. The narrator also says that people on the covid long hauler subreddit are saying that we're going to have a huge problem on our hands because no vaccine protects against long covid. The narrator also says that this isn't going to end, Isn't it? Right as think I've tasted freedom, I find out information that makes me want to stay inside forever like hermit. Vn'n#ss Instruction: AnTo what extent did the narrator think that PEOPLE'S BEHAVIOR was responsible for causing the situation? (For example, if the narrator thought that the situation was caused by people's bad decisions, or by people's lack of knowledge, or by people's lack of effort, or by peaple's lack of concem, or by people's lack of cooperation, or by people's lack of self-control, or by people's lack of self-dis. Tell
our hands because vaccine protects against long covid. The narrator also says that this isn't
caLLaMA (13B)
[Scale]: think that the narrator thinks that circumstances were not at all responsible causing the situation. The narrator says that people are getting mildly sick after being vaccinated, and that people are still getting long covid. The narrator also says that people are saying that we are going to have huge problem on our hands because vaccine protects against long covid. The narrator also says that this isn't going end, and that people are saying that we are going to have huge problem on our hands because no vaccine protects against long covid. The narrator also says that this isn't going to end, isn't it? The narrator also says that right as they think they have tasted freedom, they find out information that makes them want to stay inside forever like hermit. The narrator also says that this isn't going to end, isn't it? The narrator also says that right they think they have tasted freedom, they find out information that makes them want to stay inside forever like hermit. The narrator also says that this isn't going end, isn't it? The narrator also says that right
|Rationale|: think that the narrator thinks that the circumstances were not responsible for causing the situation. The narrator says that people are getting mildly sick after being vaccinated, and that people still getting long covid. The narrator also says that people are saying that
going have a huge problem on our hands because no vaccine protects against long covid. The narrator also says that this isn't going to end, and that people are saying that are going to have huge problem says that this isn't going end, isn't it? The narrator also says that right as they think they have tasted freedom, they find out information that makes them want to stay inside forever like hermit. The narrator also says that this isn't going to end, isn't it? The narrator also says that right as they think they have tasted freedom, they find out information that makes them want stay inside forever like a hermit. The narrator also says that this isn't going to end, isn't it? The narrator also says that right
hands because vaccine protects against long covid. The narrator also
GGT-----aviiig (vanilla model)
[Scale]: \nThe narrator thinks that CIRCUMSTANCES BEYOND ANYONE'S CONTROL are responsible for causing the situation. The narrator thinks that the situation is caused by luck. The narrator thinks that the situation is caused by weather. The narrator thinks that the situation is caused by some other things outside anyone's control. The narrator doesn't think that the situation is caused by CIRCUMSTANCES BEYOND ANYONE'S CONTROL. thinks that CIRCUMSTANCES BEYOND ANYONE'S CONTROL are responsible for causing the situation. The narrator thinks that the situation is caused by luck. The narrator thinks that the situation is caused by weather. The narrator thinks that the situation is caused by some other things outside anyone's control. The narrator doesn't think that the situation caused by CIRCUMSTANCES BEYOND ANVONE'S CONTROLAInNTTh narrator thinks that CIRCUMSTANCES BEYOND ANYONE'S CONTROL are responsible for causing the situation. The narrator thinks that
narrator
2
Figure 1I: A comprehensive example from COVIDET-APPRAM for Dimension 3 (cicummstancces-esppnniiiiii (Page 2/3).

---

the situation caused by luck. The narrator thinks that the situation caused by weather. The narrator thinks that the situation caused by some other things outside anyone's control. The narrator
[Rationale]: \nThe narrator thinks that CIRCUMSTANCES BEYOND ANYONE'S CONTROL are responsible for causing the situation. The narrator thinks that the situation is caused by luck. The narrator thinks that the situation is caused by weather. The narrator thinks that the situation is caused by some other things outside anyone's control. The narrator doesn't think that the situation is caused by CIRCUMSTANCES BEYOND ANYONE'S CONTROL 'nhnTTh narrator thinks that CIRCUMSTANCES BEYOND ANYONE'S CONTROL are responsible for causing the situation. The narrator thinks that the situation is caused by luck. The narrator thinks that the situation is caused by weather. The narrator thinks that situation is caused by some other things outside anyone's control. The narrator doesn't think that the situation caused by CIRCUMSTANCES BEYOND ANYONE'S CONTROL.MNnTTH narrator thinks PRCUMSTANCES BEYOND ANYONE'S CONTROL are responsible causing the situation. The narrator thinks that the situation caused by luck. narrator thinks that situation is caused by weather. narrator thinks that situation caused by some other things outside anyone's control. The narrator
Figure 12: A comprehensive example from COVIDET-APPRAISALS for Dimension 3 (cirammstances-rspoonsiiiil (Page 3/3).

---

Help Us Understand Emotion Appraisals!
You'll read Reddit post about someone's experiences during COVID-19.
We ask to judge 24 aspects about how they feel about and view the situation. (For example, whether they felt the situation they were in was something they could control.)
For each of these aspects, please select rating using the scales we provide (from 1 to 9). You can also select the (NOT MENTIONED] option case the aspect is mentioned in given post.
In addition, please tell why you picked that rating! Examples are provided below.
Please read the following examples:
Reddit Post: "Just recently went grocery shopping. also forgot to bring receipt to the tailor, so I had to back home after grocery shopping just to pick it and deliver it to her to pick my trimmed down pants. And now feel like I'm spreading COVID-19 to my parents, despite myself getting vaccinated against it. Like, know the best I'm going to experience is mild symptoms anytime "do" get it, plus my parents are vacoinated as well, so either they'll experience mild symptoms or no symptoms at all any time sspread to them. But COVID-19 has the potential to mutate thus evade our immune systems much more easily. And with that many unvaccinated people spreading among each other for the nonexistent "freedoms", it's going mutate and infect all of us vaccinated people and kill all. So there anyone who will help me with this? Thanks!"
their otherwise
Question: To what extent did the narrator think that
THEY were responsible for causing the situation?
..> Rating: 6 (out of 9)
Reason: The narrator expresses concern about
potentially spreading COVID-19 to their parents, even though they have been vaccinated and their parents have also been vaccinated. They seem to recognize that there is a potential for the virus to mutate and evade immunity, but also seem to feel some level of personal responsibility outcome. The text suggests some level guilt or responsibility for causing the situation.
narrator feels
Close
Figure 13: Instructions to annotators for COVIDET-APPRAISAL

---

Annotate the Appraisal Bimersanns
evample posts cavelaly.
"Be the Nacome Juan la laving nar shy for noxeh that untands over the time uni war sapaos lo geiing secoms Aote Zenona ahst Trny affoed get sooce buttthe hoard will docrass it's off etveere ess alut Brossl get the shat wosks ofter my shocor should wot undil tay cone book and gott lood wooks after my anst. Ioslard has slopped cltor
Prow de messer
A, whet oxlent cbdtthe camettr thin COu veruuces DEYOND ANYoves CONTrOL were respents e ceantng the staatee??
por coomply, the nesruttr Frouuhh shadkie ased wnether smags outiee cestee)
ixtert clithre namitrr
conet she event?
For etarçii, Ihte ramaar toought but they kad researses the for sellngee make siuudios bitter beattraragghha
1901
end
ixtent nammtor rall DWATT poabr?
ecarçáo, tilorraree rarraar Bhouuht the smaatic was persuriity inportattio whurithy
Prem
dhegerchrrhhy they
Prem
happaning
Wan de
intert sarmtto contrelingg whatway huppeeing stheles?
Figure 14: Annotation task layout for COVIDET-APPRAISAIS (Page 1/3)

---

hopperng
244 yeureesers
To what cateet ded the rerrotor thi buuthhig tyy were oble PREDECT what veru goiing hoppee wost ahbrdor??
Korneee pee pedtt
to wrttl caturt au sumattrr to that seing indcuaiinnsss
For coumpti resmater thought that they helay Prevatmend se seeerhingg phy
dangerous wwweeeun-eeeeennneeeeee anmelneetoy crnceofoyrnd filligg count
Zier Jown crecoeet
whateeeen thise shaatiee PUCAGANITY
Prom
entent
they WoNTESS
To what-ettnn dil the numaterr the that sttulicee was 7A/12
Harr Mirreoo
whaneetent di WORSGEETTER
your
entent did tht sometiiing Insortenble has boen sllatiie?
Figure 15: Annotation task layout for COVIDET-APPRAISALE (Page 2/3).

---

chate sorster th whfudime Poli sorr
coarpe, of nomooar Bhought Bust they And reptrreeet the adtuttie barker kithe
situalic
ansunn revrta ded sitatioe)
modery
Par rasse
O
from de messen hest
cnatet the tituiiie with
To romsster scONect sibariiee souar??
Complealy anceettd
Figure Annotation task layout for COVIDET-APPRAISALE (Page 3/3).
ChatGPT: 1-Step Prompting
Other 2-Step Prompting
[Reddit Post)
1st-Step: Below is instruction that describes a task, paired with an input that provides further context. Write response that appropriately completes the request.
Given the above what extent did the narrator think
were responsible causing the situation? Rate on scale from 9, with being "Narrator thought they were: Not at all responsible" and 9 being "Narrator thought they were: Completely responsible". If the text doesn't 't address this question, please mark "NA". Additionally, tell why. The format of the should follows: |a/raiionale
input: (Reddit Post)
### Instruction:
To what extent did the narrator think that THEY were responsible for causing the situation? Rate a scale from 1 to 9, with 1 being "Narrator thought they were: Not all responsible" and being "Narrator thought they were: Completely responsible". If the text doesn't address this question, please mark it as "NA".
Response:
2nd-Step: Below is an instruction that describes a task, paired with an input that provides further context. Write a response that appropriately completes the request.
### input: (Reddit Post)
Instruction:
narrator think that THEY were responsible causing the situation? Rate on a scale from to with being "Narrator thought they were: Not at all responsible" and being "Narrator thought they were: Completely responsible". If the text doesn't address this question, please mark at "NA". The selected scale (scale answer from the 1st-step). Tell us why.
what
##lfi Response:
Figure 17: Prompt templates (taking dimension as example).

---

Instructions and Examples for Evaluating Rationales for Emotion Dimensions
[Instructions]
This an annotation task for evaluating the rationales for selected ratings. During the evaluation, you will read Reddit post. Then, you will see pair of (question, answer) relating to the emotional state of the author of the post. The answer will first give rating (that verified to be correct)on scale of 1 (including "Not Mentioned" label provided in case the aspect is not mentioned in the post), followed by a rationale explaining why the rating is selected. The rationale is the portion ask you evaluate.
conducted based on criteria, namely "factual consistency", "relevance" "justifiability", and "usefulness". The detailed instructions for each question are shown below.
The evaluation
1) Is the rationale factually consistent with the post?
* Whether the rationale faithfully reflects what's stated in the post. In other words, does the rationale accurately describe what post saying, or does misrepresent or hallucinate content some way?
"Yes": " if the rationale is accurate with no errors
"Minor Error": the rationale contains some minor errors or omissions
"No": if the rationale contains significant errors, misrepresentations, or significant hallucinations to the
question
2) Is the rationale relevant to the aspect question being asked?
Whether the rationale directly addresses the specific question that is being asked about the post. This that the rationale should be focused on the specific aspect of the post that is being evaluated, and should not stray off-topic provide irrelevant information.
(Most Relevant)
(Least Relevant)
5
3
2
3) Does the rationale justify the selected scale?
Does the rationale adequately explain why selected rating scale is most appropriate or relevant one use for the aspect being evaluated?
Yes
No
4) Is rationale useful (informative)?
* Whether the rationale provides useful or informative insights or explanations related to the aspect being evaluated. other words, does the rationale provide helpful information insights that aid evaluating the aspect question being asked?
"Yes"" if the rationale provides clear and informative explanations or insights for the reasons why the
Likert-scale rating is selected
0 "Maybe": if the rationale provides some information that may be useful, but is not entirely clear or may not
fully address the reasons why the Likert-scale rating is selected
0 "No": if the rationale does not provide any useful or informative insights or explanations for the reasons why the Likert-scale rating is selected
Figure 18: Instructions for the human evaluation described in $7.2 (Page 1/2).

---

[Examples]
Reddit Post:
I'm being vague as to not give away my employer but I'm frustrated and wondering how you're 're coping with being one of the only fields CA that will be stuck in the past with end date in sight. I have lot of various traumas from this, I'm sure many do, and being left as the rest of my (all vaccinated, myself included) family gets to finally unmask at work is definitely not helping mental state.
Example Rationales Evaluate:
Dimension 20: To what extent did the narrator think that the situation FAMILIAR? (For example, if the narrator thought that they had experienced this situation before the past.) [Scale]:
Situation was
(Not at all familiar)
(Completely familiar)
1 2
4
5
5
7
Not mentioned
[Rationale]:
The narrator seems to be familiar with the situation of being frustrated and traumatized due to their personal health conditions, as indicated by rating of 6. This indicates that the narrator believes that the situation cannot be reversed
something of value been permanently lost.
Human Evaluation Example:
1) Is the rationale factually consistent with the post?
Yes
Minor Error
No
2) Is the rationale relevant to the question being asked? Most Relevant
Least Relevant
3
2
3) Does the rationale justify the selected scale?
Yes
No
4) Is the rationale useful (informative)?
Yes
Maybe
No
Figure 19; Instructions for the human evaluation described in $7.2 (Page 2/2).

---

Instructions
Shortoute
Evaluate the Appraisal Rationales
See here instructions and examples. Please read them carefully prior evaluating!
"My mental health was never great before the pandemic but felt more grounded. Even though pandemic-wise things are objectively much better than they were, I'm atill feeling the after-effects of year of isolation and high atress at work. can feel noticeable notch down my mood just compared to last summer. I feel dead inside and doing anything requires huge amount of energy. I feel sad all the time, cry more. feel sad when see people having fun with friends and family; don't have that. My family isn't great and have no contact with them. I had no close friends but lost all my distant friends during the pandemic. have therapist see infrequently since can't afford more, but I'm starting to feel like he can't help me. I've known him for quite some time that's sad thought, but feel like everything he says repeat of something he's said befors rather than tallored advice to actual situstion in the present. We all repeat ourselves lold can call him get very bad way, least have his support on that level, don'l expect advice solution this. the end of the we're alone struggles, can't expect anybody else but get myself of But
extent. probably need fresh perspective, don't know. He's
sharing somebody."
example, did thought situation required expending large physical effort deal situation.) thought that: effort was needed" being "Narrator thought that: Very much effort was needed".
they needed EFFORT
(For mental 9, being "Narrator
doesn't address this question, please mark as
- Selected Scale:
Rationale for the Scale Selected: "The narrator mentions that doing anything requires a huge amount of energy and that they feel dead inside. They also mention feeling sad all the time and crying more. These are all indicators that the narrator feels that significant amount of effort is needed to deal with their situation."
the rationale factually consistent with the post?
Emor
2) the rationate relevant the question being asked?
(Mess
Relevart)
Does the rationale justify the selected scale?
Yes
rationale (informativee?
Yes
Maybe
No
Figure 20: Task layout the human evaluation.


Reviews:
[{"review_id": "2Ng7LSAWXr", "reviewer": "Reviewer_jg4L", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "&nbsp;&nbsp;&nbsp;&nbsp;Thank you very much for your valuable comments! We address your questions and comments below.\n\n> Small dataset size:\n\n&nbsp;&nbsp;&nbsp;&nbsp;We have not intended this dataset to be one for supervised model training but rather a very high-quality dataset for evaluation (since this is the first dataset of its kind). To annotate a single post in our dataset, our human annotators were asked to rate each of the 24 appraisal dimensions on a scale of 1 to 9, accompanied by the provision of rationales in their own words to justify their ratings. The result is 6,744 ratings and their corresponding natural language rationales. Given the length of our source posts (175.82 tokens on average) as well as the multitude of appraisal dimensions (24 in total), the annotation was difficult and time-consuming as it required annotators to comprehend the document in depth. Even an experienced annotator who has gone through rounds of annotations for our dataset still needs to spend around 30 minutes annotating a single example.\n\n> Why should LLMs understand the cognitive appraisals of emotions? How would they benefit from such a subjective cognitive assessment?\n\n&nbsp;&nbsp;&nbsp;&nbsp;The importance of a deeper understanding of cognitive appraisals of emotions in LLMs is two-fold. First, going beyond the simple task of “emotion detection”, it is important to also consider contextual factors that lead to emotion (see recent NLP work in extracting causes/triggers of emotions as another example). Cognitive appraisals are the subjective interpretations that people make that determine their emotional reactions: different interpretations of the same situation result in different emotions. For example, if a person experiencing a breakup perceives it to be their responsibility, they might feel guilty and sad; if they perceive it to be the other party’s responsibility, they might feel angry instead. (There is extensive literature in psychology on appraisals, which we briefly cover in Appendix Table 7.) A language model that can also extract appraisals of emotion will result in a more accurate and interpretable understanding of emotions, e.g. understanding individual subjectivity in responses to the same situation, and at the same time offering explanations/rationale (“they are feeling guilty because of X appraisal”). This current work focuses on the validation of appraisal identification in LLMs, and future work can expand upon it in these areas.\n\n&nbsp;&nbsp;&nbsp;&nbsp;Second, from an application perspective, the understanding of emotion appraisals in LLMs lays the groundwork for useful AI that can improve users’ mental health using positive reframing (“reappraisals”) or empathic responses, such as emotional support systems. In practice, we observe that while current LLMs are capable of responding to human’s emotional distresses, the responses are generally templated and generic. An understanding of appraisals can result in more targeted and psychologically beneficial responses (e.g., helping someone feeling guilty feel better by helping them reappraise their perception that they caused the situation: “It was not your fault”).\n\n&nbsp;&nbsp;&nbsp;&nbsp;We appreciate your valuable comments, and will include some more discussion of this motivation in our final draft of the paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Another novel dataset was introduced and manually annotated, acknowledging the contributions made to the community.\n2. Novel insights were derived from cognitive appraisals, leading to new perspectives.\n3. A comprehensive comparison was conducted among various large-scale models, accompanied by an in-depth analysis of their performance.", "reasons_to_reject": "1. Dataset size is too small(only 241 reddit posts)\n2. Did not provide why to achieve a more accurate and holistic understanding of emotions from written text, NLP models need to work towards understanding the subjective cognitive appraisals of emotions underlying situations(line 599) or why an LL.M. would benefit from understanding subjective cognitive assessment..", "questions_to_authors": "", "paper_topic_and_main_contributions": "This study introduces a new dataset, CovidET-Appraisals, which evaluates 24 appraisal dimensions present in 241 Reddit posts. Each dimension is accompanied by a natural language explanation. The dataset was assessed using a large language model. The paper's findings reveal that while the top-performing models demonstrate competence, open-sourced large language models do had shortages on this task.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "PGqnJUgdzV", "reviewer": "Reviewer_TGaY", "summary": "", "questions": "What are the agreement for each individual appraisal dimension? Surely some dimensions will be more difficult to annotate whereas other are easier.", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "&nbsp;&nbsp;&nbsp;&nbsp;Thank you very much for your valuable review!\n\n> Comparison with other similar datasets:\n\n&nbsp;&nbsp;&nbsp;&nbsp;Thank you for pointing this out! In the field of NLP, the appraisal of emotions is rarely studied. The only prior dataset was discussed in lines 163-170, Hofmann et al. (2020) utilized cognitive appraisals to assist the task of emotion classification. They introduced a dataset including 1,001 *sentences* of the format “I feel [emotion name], when . . . ”, on top of which 7 cognitive appraisal dimensions were examined. The average length of the instances in their dataset is only 27.41 tokens. In comparison, our work studies the cognitive appraisal of emotions on Reddit posts, which are naturally occurring, emotionally charged lengthy texts where the average span is 175.82 tokens. Besides, we examine 24 appraisal dimensions, covering a more comprehensive set of cognitive appraisal dimensions found in prior psychology literature.\n\n> The LLMs were experimented under a zero-shot set-up, and other commonly used techniques such as in-context learning and few-shot fine-tuning were not included:\n\n&nbsp;&nbsp;&nbsp;&nbsp;We highlight that this is the first work towards the assessment of cognitive appraisals of emotions in language models, and it lays the foundation for future research on deciphering the intrinsic emotional dynamics that remains unexplored in current state-of-the-art models. We believe that this warrants a careful construction of the dataset with thorough analysis; and we leave these interesting engineering questions, such as low-rank adaptation, to future work.\n\n> Compare the LLMs’ performance with supervised models:\n\n&nbsp;&nbsp;&nbsp;&nbsp;Given the poor performance of Flan-T5, we believe that a supervised model will perform poorly without a large number of training examples. However this is challenging: we have 24 dimensions to analyze per post, and the annotation for one post for one trained annotator takes half an hour. Approaches such as distillation may work, but only if we first establish the validity of training data obtained from LLMs. This work aims to lay this type of ground work.\n\n> What is the agreement for each individual appraisal dimension? Surely some dimensions will be more difficult to annotate whereas others are easier.\n\n&nbsp;&nbsp;&nbsp;&nbsp;The inter-annotator agreement for each dimension is in Appendix Figure 5 (Page 13). Results show that our human annotators have strong agreement on dimensions such as “emotion-focused coping” and “problem-focused coping”, whilst disagreeing with each other most often on dimensions “goal relevance”, “expectedness”, and “loss”. This can be attributed to the nature of our domain: in the Reddit posts we annotated, the narrator is mainly sharing their experiences in life around COVID-19 and seeking support on the forum, while preserving doubts about the future. This leads to a stronger consensus in a low perception of the narrators’ coping abilities, and more disagreement on the other appraisal dimensions.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper is generally well-written. The analysis of dataset and evaluation is comprehensive.\n\nCOVIDET-APPRAISALS, a dataset containing expert-annotated emotional appraisal dimensions for Reddit posts, has been constructed. The inter-annotator agreement looks good.", "reasons_to_reject": "The comparison with other similar dataset is missing. The claim \"... the most comprehensive dataset to-date ...\" (line 13) needs to be supported. Is it the most comprehensive because it contains the largest number of samples, appraisal dimensions, labels, or something else? No other established dataset have been mentioned, evaluated or compared with.\n\nThe LLMs were experimented under a zero-shot set-up. Other commonly used techniques such as in-context learning and few-shot fine-tuning (for open-sourced models, it shouldn't be expensive with the low-rank adaptation configuration) were not included. It would also bring about more insight if LLM performance could be compared with smaller but supervised models.", "questions_to_authors": "What are the agreement for each individual appraisal dimension? Surely some dimensions will be more difficult to annotate whereas other are easier.", "paper_topic_and_main_contributions": "This work presents COVIDET-APPRAISALS, a dataset containing 241 reddit posts about COVID. Each post is annotated for 24 emotional appraisal dimension by at least one expert.\n\nAuthors evaluated their dataset with a range of large language models (LLMs) for likert ratings and rationales. ", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "8MpwnzGWFR", "reviewer": "Reviewer_8u8A", "summary": "", "questions": "- Line 503: how come some human-annotated rationales do not pass the filtering?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "&nbsp;&nbsp;&nbsp;&nbsp;We sincerely appreciate your attention to the details in our paper. Here we would like to address each of your points and provide clarifications for them one by one:\n\n> Clarifications regarding the experimental setup:\n\n&nbsp;&nbsp;&nbsp;&nbsp;(1) Annotators’ label aggregation: With annotation tasks that are highly costly like ours, it is standard to trust well-trained annotators to singly annotate and perform inter-annotator agreement analysis on a smaller subset annotated by multiple annotators. As shown in Section 3 (lines 270-297), our inter-annotator agreement measures indicate that the human annotators tend to agree strongly with each other on the Likert-scale ratings with a Krippendorff’s alpha of 0.647 and a small mean absolute delta of 1.734 on a scale of 1-9 between the two annotators on the subset of doubly annotated data. This we believe is a strong indication that taking the average for aggregation is a reasonable approach.\n\n&nbsp;&nbsp;&nbsp;&nbsp;(2) 1-step and 2-step prompting setup: We consider the 1-step prompting strategy for ChatGPT-3.5 as it is a more congruent and efficient way to elicit the understanding of emotion appraisals from the language model. We have also experimented with the 2-step prompting strategy for ChatGPT-3.5, and it yielded similar responses as the 1-step prompts. On the other hand, while we have also experimented with other LLMs using the 1-step prompts, they generally fail to generate sensible responses.\n\n> Human evaluation setup results could be misleading and difficult to compare to in future works:\n\n&nbsp;&nbsp;&nbsp;&nbsp;We filter the rationales based on the correctness of the appraisal dimension. This is intentional, because rationales serve to justify the ratings. We recognize that a joint evaluation of both rationale and ratings would be appealing, however we have already quantitatively evaluated the Likert scores, and we believe that evaluating the rationales only when the scores are correct would give a much clearer picture of where the errors come from and leave a more informative picture for future work. We will release the judgments of these dimensions, which are fixed and can be referred to by future work. We will also release the protocol for evaluation.\n\n> The final corpus doesn’t cover all 24 dimensions, and the 3 appraisal dimensions where the NA rates are ~80% should be included as part of the task:\n\n&nbsp;&nbsp;&nbsp;&nbsp;We agree that the inclusion of all 24 appraisal dimensions will be useful for a more thorough examination of the systems’ performance. Since the data we plan to release includes the annotations on all 24 dimensions, we will add the prediction of the 3 additional dimensions in our final draft of the paper. After including the 3 dimensions where the NA rates are around 80%, the performance across all LLMs showed marginal change. For most LLMs dropped as expected: measured with Spearman’s ρ, ChatGPT-3.5 (↓0.018), Alpaca-7B (↓0.008), and Dolly-12B (↓0.007). On the other hand, the performance of FLAN-T5 (↑0.005), Alpaca-13B (↑0.027), and Dolly-7B (↑0.020) increased.\n\n> Line 503: How come some human-annotated rationales do not pass the filtering?\n\n&nbsp;&nbsp;&nbsp;&nbsp;We do not perform filtering on the human-annotated rationales. Rather, we carry out human evaluation on the subset of (post, dimension) tuples where all LLMs’ ratings are near-correct with the human ratings.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Brings light to an important and challenging task in NLP, i.e. emotion cognitive appraisal\n- contributes a corpus which could be use as benchmark for the task\n- strong correlation on the subset which are annotated by 2 labels\n- Extensive evaluation and analysis on the experiments", "reasons_to_reject": "- The set up is not as clean as it could be. For example, in the dataset a subset is labeled by 2 annotators and an average score is taken, and the rest is labeled only by 1 annotator. Another example is in the experiment, one LLM uses 1-step prompting while the rest use 2-step. Although the authors provide the reason, to me it is not convincing enough as the set up can be easily made consistent, e.g. still take rating from just 1 annotator (use the 2nd only for sanity check through correlation) or annotate the entire corpus with 2 annotators, and use 2-step prompting for all systems.\n- Human evaluation set up result could be a) misleading, as the outputs are already filtered prior to evaluation (the numbers then are an upper bound or over-estimation of the average performance), b) difficult to compare to in future works as it doesn't reflect system performance on a static test set (filtering result will differ across works).\n- the final corpus doesn't cover all 24 dimensions. I'm not convinced that exclusion of the 3 categories is warranted given that the NA rates is only around 80%, and prediction of NA is actually considered part of the task. If anything, these dimensions would be suitable to check system performance on rare labels which is arguably more challenging.", "questions_to_authors": "- Line 503: how come some human-annotated rationales do not pass the filtering?", "paper_topic_and_main_contributions": "Emotion arises from a person's cognitive appraisal of the situation. Although emotion recognition is an established task in NLP, emotion appraisal is not yet deeply explored. This work fills the gap by presenting COVIDET-APPRAISALS, which assesses 24 appraisal dimensions, each with a natural language rationale, across 241 Reddit posts. The annotation procedure is carefully elaborated. The corpus construction is followed up with an experiment to gauge LLMs performance on this task. Result shows that ChatGPT performs remarkably and there are room for improvements on the other tested LLMs.\n\n\nMain Contributions\n- Corpus consisting of reddits posts with detailed annotation and rationale on 24 cognitive appraisal dimensions. \n- Experiments comparing the cognitive appraisal abilities of state-of-the-art LLMs, the result highlight that this task is challenging.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper introduces the COVIDET-APPRAISALS dataset, focusing on the appraisal dimensions of emotions in Reddit posts. It also evaluates large language models (LLMs) on this dataset. While the paper makes significant contributions to the understanding of cognitive appraisal in emotions, there are areas that need attention.

Pros in Reviews:
- The paper presents a novel and valuable dataset, COVIDET-APPRAISALS, with detailed annotations and rationales for 24 cognitive appraisal dimensions.
- It addresses an important and underexplored task in NLP, emotion cognitive appraisal, filling a gap in the field.
- The paper conducts extensive evaluations and analyses of state-of-the-art LLMs on the task, shedding light on the challenges involved.
- There is a strong correlation on the subset of data annotated by two labels, demonstrating the reliability of the annotations.

Cons in Reviews:
- The dataset size is relatively small, comprising only 241 Reddit posts. The paper lacks a clear rationale for why understanding subjective cognitive appraisals is essential, limiting the broader context of the research's significance.
- The claim that COVIDET-APPRAISALS is "the most comprehensive dataset to date" lacks support and a comparison with other similar datasets.
- The paper experiments with LLMs under a zero-shot setup, missing the opportunity to explore other techniques like in-context learning or few-shot fine-tuning. Comparing LLM performance with smaller supervised models could provide additional insights.
- The experimental setup could be improved for consistency. The use of one or two annotators for labeling and variations in prompting for LLMs should be better justified.
- The human evaluation setup may be misleading, as it filters outputs prior to evaluation, potentially overestimating system performance. It also makes comparisons with future works challenging.
- The exclusion of three appraisal dimensions from the final corpus is questionable, given the relatively low NA rates, and NA prediction is considered part of the task.

In summary, the paper contributes significantly to the understanding of emotion cognitive appraisal and presents a valuable dataset, but it should address concerns about dataset size, support claims regarding dataset comprehensiveness, explore different experimental setups, and reconsider the exclusion of certain appraisal dimensions. Despite all these problems, the authors have written good rebuttals regarding the main criticism. Looking at the work that would need to go into the paper to address all those comments, I would argue for a similar rating in-line with those of the reviewers.
Decision: Accept-Findings