Introduction
According to appraisal theories, emotions emerge from the individual’s subjective appraisals of significant events (Scherer, 2009 ) . Appraisals are the person’s evaluations of what situations mean for their needs, goals, and other concerns (Moors et al., 2013 ) . Appraisals consist of values on abstract dimensions representing key aspects of a situation, such as how important, desirable, self-caused, certain, and controllable it is. Appraisals orchestrate changes in other components of an emotional episode, including tendencies to act in some way (motivational component), visceral preparations for these actions (somatic component), facial and bodily expressions (motor component), and a conscious feeling (experiential component) (Moors et al., 2013 ) . Even as all of these components can influence each other, appraisals are often considered as pivotal for initiating and shaping the dynamic interactions that underlie a person’s experience of emotion.

Despite their centrality in emotion theory, there have been only few attempts to apply NLP methods to automatically extracting appraisals from text (Hofmann et al., 2020 , 2021 ; Troiano et al., 2023 ) .
In those previous works, appraisals have been of interest as a component of emotions, with the goal of eventually predicting the emotion themselves (Hofmann et al., 2020 ; Troiano et al., 2023 ) . These early results are promising, but more research is needed to further improve the precision and robustness of appraisal prediction models.

Advancing the NLP research on appraisal requires suitable annotated datasets. The most accurate way to annotate appraisals is the so-called experiencer-annotator paradigm, where a person
provides both a textual description of some emotional event and ratings on appraisal dimensions. However, the experiencer-annotator method can only be used on texts collected specifically for this purpose. In an alternative reader-annotator paradigm, the appraisal ratings are provided by another person reading the textual descriptions of emotional events. The reader-annotator method is more flexible, as it enables using existing datasets of emotional event descriptions, even those that have not been specifically collected for that purpose but rather have been generated spontaneously by people, for instance in social media.

Generating appraisal ratings with the reader-annotator procedure requires human labor that can become prohibitively expensive for large real-world datasets such as social media and blog posts. Therefore, we are interested whether this process can be automated. Previously, Hofmann et al. ( 2021 ) adopted a rule-based approach to assign appraisal labels to texts, given that the emotion label of the text was known. However, this approach requires the emotion labels which are often unavailable in real-world datasets. Also, Hofmann et al. ( 2021 ) considered only six appraisals and represented them as binary variables. Thus, the deterministic rule-based approach is clearly not feasible with a larger number of appraisal dimensions assessed on Likert scales.

The goal of this work is to assess the suitability of Large Language Models (LLMs) to act as an annotator in the reader-annotator paradigm for emotional appraisals. Several previous works have shown the utility of using LLMs as a viable alternative for human annotators for labeling data for NLP tasks using both the proprietary GPT models (Ding et al., 2023 ) as well as open-source LLMs (Alizadeh et al., 2023 ) . Other studies have found that using GPT-4 is generally a more reliable annotator than crowdworkers for social science related text annotations (Gilardi et al., 2023 ) , and it is better than vocabulary-based methods for annotating several psychological constructs (Rathje et al., 2024 ) . Although several studies have assessed the ability of GPT-4 to detect discrete emotion labels (Niu et al., 2024 ; Lian et al., 2024 ) , Kocoń et al. ( 2023 ) found that GPT-4 was performing worse than fine-tuned classification models, especially on emotion prediction tasks.

Our study focuses on adopting GPT-4 for annotating emotion appraisals in the reader-annotator paradigm to assess the viability of the generative LLM to act as an alternative to human reader-annotators. Few studies have explored the capabilities of GPT-4 and similar LLMs in this context. Tak and Gratch ( 2023 ) analyzed the emotional reasoning abilities of GPT models, finding that while they align well with human appraisals, they struggled with predicting emotion intensity and coping response. Yongsatianchot et al. ( 2023 ) studied how LLMs perceive emotions using appraisal theory, demonstrating that these models can effectively generate context-specific emotional appraisals. However, these studies focused more on the rationales of LLMs than appraisal rating predictions and used a limited list of appraisal dimensions. Zhan et al. ( 2023 ) evaluated annotating emotion appraisals by multiple LLMs, showing that they can produce human-like emotional appraisals.
However, the dataset used in this research was relatively small and, more importantly, did not contain original experience-annotator ratings, which prevents comparison between the event experiencer and reader annotations.

For our study, we use the crowd-enVent dataset collected by Troiano et al. ( 2023 ) , which consists of crowd-sourced emotional event descriptions, supplied by the appraisal rating of both the experiencers and readers. We formulate research questions about the reliability of GPT-4 in generating appraisal ratings (Q1) and the accuracy of GPT-4 ratings compared to both experiencer-annotators and human reader-annotators (Q2). In addition, we test whether the application of a majority voting algorithm and a model confidence tiebreaker improves accuracy (Q3). Also, we examine the impact of adding the emotion prediction on the accuracy of the appraisal ratings (Q4). Finally, we study the impact of the event description length on the accuracy of the ratings by both the GPT-4 and human reader-annotators (Q5).