The PEACE-Reviews dataset: Modeling Cognitive Appraisals in Emotion Text Analysis
Authors: Gerard Christopher Yeo
Kokil Jaidka
Venue: EMNLP 2023 Findings (2023)
Keywords: cognitive appraisals
emotions
language modeling
computational social science
TL;DR: PEACE-Reviews dataset: Annotated autobiographical writing on significant purchases, revealing how cognitive appraisal and emotions interact with consumer behavior and demographic traits, with baseline models for feature prediction.

Abstract: Cognitive appraisal plays a pivotal role in deciphering emotions. Recent studies have delved into its significance, yet the interplay between various forms of cognitive appraisal and specific emotions, such as joy and anger, remains an area of exploration in consumption contexts. Our research introduces the PEACE-Reviews dataset, a unique compilation of annotated autobiographical accounts where individuals detail their emotional and appraisal experiences during interactions with personally significant products or services. Focusing on the inherent variability in consumer experiences, this dataset offers an in-depth analysis of participants' psychological traits, their evaluative feedback on purchases, and the resultant emotions. Notably, the PEACE-Reviews dataset encompasses emotion, cognition, individual traits, and demographic data. We also introduce preliminary models that predict certain features based on the autobiographical narratives.

The PEACE-Reviews dataset: Modeling Cognitive Appraisals in Emotion Text Analysis
Gerard Christopher Yeo Institute of Data Science National University of Singapore e05451 59@u. nus. nus.ede. edu
Kokil Jaidka
NUS Centre for Trusted Internet and Community National University of Singapore
jaidka@nus edu.sg
Abstract
Ellsworth, 1985), and this study focuses on con- structing a dataset to model such evaluations that people make about their situations.
Cogmitive appraisal plays a pivotal role in deci- phering emotions. Recent studies have delved into its significance, yet the interplay between various forms of cognitive appraisal and spe- cific emotions, such joy and anger, remains an area of exploration in consumption contexts. Our research introduces the PEACE-Reviese dataset, a unique compilation of annotated au- tobiographical accounts where individuals de- tail their emotional and appraisal experiences during interactions with personally significant products or services. Focusing on inher- ent variability in consumer experiences, this dataset offers an in-depth analysis of partic- ipants' psychological traits, their evaluative feedback purchases, and the resultant emo- tions. Notably, the PEACE-Reviews dataset en- compasses emotion, cognition, individual traits, and demographic data. We also introduce pre- liminary models that predict certain features based the autobiographical narratives.
Cognitive appraisal theories proposed that emo- tions are elicited from the evaluation, or ap- praisal, of situations along a set of motivationally- relevant dimensions (Scherer et al., 1984; Smith and Ellsworth, 1985). Consider the following sce- nario. Suppose Andy encountered rude server in a restaurant that only attends politely other customers except him. Andy might interpret this personal situation in the following way- a) the rude behavior of the server is unexpected as he ex- pects that servers should always be polite to their customers, b) the rude server is responsible for him having such an unpleasant experience in the restau- rant, and c) he finds that the rude behavior directed only to him and not other customers is unfair. By interpreting the situation in this manner, Andy might experience an emotion like anger.
Specific emotions are characterized by profiles of cognitive appraisals that represent the evalu- ation of the person-situation relationship that is important to an individual (Lazarus et al., 1980). Emotions with a certain appraisal profile have been shown to influence decision-making and behav- ioral processes (Lerner et al., 2015). For example, individuals experiencing fear (an emotion that is characterized by the appraisals of high threat and uncertainty) made more risk-averse decisions about an event (Lerner and Keltner, 2001). The effect of emotions on decision-making processes and subse- quent behaviors is also found in the domain of con- sumption (buying or using product). For instance, emotions affect various processes such as purchase decisions (Achar et al., 2016), recommendations (Richins, 2008), and brand-switch intentions (Bui et al., 2011). Moreover, recent studies have also shown that cognitive appraisals directly and indi- rectly (via emotions) influence pre- pre-purchase inten- tions and post-consumption behaviors (Sari, 2022:
1 Introduction
Language modeling provides insights into the lin- guistic features associated with how people think, feel, and behave. Some studies have modeled vari- ous psychological constructs in online text such as world beliefs (Clifton, 2020), personality (Christian et al., 2021), stress (Guntuku et al., 2019), locus of control (Jaidka et al., 2018), and even psycholog- ical disorders such as depression (De Choudhury et al., 2013). There is also work modeling emo- tions using language models (Sookarah and Ram- wodin, 2022). However, to understand emotional experiences of individuals comprehensively, what needs to be added in the current literature under- standing the cognitive antecedents of emotions how people evaluate their situations that results in specific emotional experiences. These evalua- tions are known as cognitive appraisals (Smith and
'The dataset is accessible at https://github... com/ GerardYeo/PEACE-RRee Bew.cE-Reeiiwsse... git

---

Wang al., 2019). Considering emotions and their antecedents (i.e., cognitive appraisals) can offer more holistic models of consumers' thoughts, emo- tions, and behaviors.
and cognitive appraisals from linguistic fea- tures using state-of-the-ari language models.
Theoretical Framework
Online product reviews provide implicit evi- dence of consumer behavior. They include descrip- tions of subjective feelings and non-emotional fac- tual recounts (Lee and Koo, 2012; Moore, 2015). Analyzing the emotional content of reviews can infer reviewer behaviors. However, these reviews may not reflect the necessary emotions and cogni- tive appraisals to study the psychological link be- tween the two. Public datasets of product reviews do not consist of rich emotional labels (emotions and cognitive appraisals) to model the associations between consumer language and emotional con- structs. Furthermore, given that most datasets com- prise annotations inferred by independent annota- tors, is worthwhile to pause and consider whether they offer an accurate first-hand representation of consumer appraisal. In other words, are public datasets of product reviews suitable to study emo- tions, cognitive appraisals, and subsequent con- sumer behavior in the context of consumption?
One of the primary theoretical perspectives of un- derstanding and predicting emotions in people is Cognitive Appraisal Theory, as studied in Psychol- ogy. This theory posits that emotions are elicited from one's evaluation of situations that are of per- sonal significance to one's well-being (Arnold, 1960; Frijda, 1987: Lazarus et al., 1980; Ortony et al., 2022; Roseman, 1984; Scherer et al., 1984; Smith and Ellsworth, 1985; Weiner, 1985). These evaluations, appraisals, are done along a set of motivationally relevant variables. For example, if person appraises situation as 1) not consis- tent with their goals (goal conduciveness), 2) some other person is responsible for causing the situa- tion (other-accoumtaablity)) and 3) unfair (fairness), they will feel an emotion like anger. Therefore, instead of just knowing what emotion a person is feeling, these appraisal dimensions are essential in understanding why such an emotion is felt by the person. Thus, according to cognitive appraisal theories, how people appraise their situations re- sults in specific emotions experienced. Moreover, appraisals are proposed to central to the emotion process and are considered to be antecedents of emotions (Moors et al., 2013). Therefore, analyz- ing appraisals is essential in understanding what specific emotions get elicited in situation and can also answer why person is feeling the way he she is feeling (Siemer and Reisenzein, 2007).
To address these research gaps, we conducted a pilot study of which kinds of writing prompts elicit high-quality text of emotions and cognitive ap- praisals and subsequently used the writing prompts derived from the pilot to collect the full dataset. Our dataset, called Perception and Experience of Appraisals and Consumption Emotions in Reviews (PEACE- PEACE-Reviewy offers review texts annotated with rich first-person emotional variables such as cognitive appraisal dimensions and emotional intensity. This work contributes to the extant literature that overlaps between the disciplines of Emotion Science and Natural Lan- guage Processing by enhancing the understanding of linguistic features of appraisals reflected in the emotional expression of autobiographical review text.
Autobiographical writing, such as reviews, re- flects and communicates psychological aspects about the writer such as one's 's thoughts and feelings (Boyd and Pennebaker, 2015; Tausczik and Pen- nebaker, 2010). Words or phrases used in such writ- ings can suggest cognitive processes (c.g. how peo- ple think about themselves or objects, their goals, and intentions), affective processes (e.g. how peo- ple are feeling), and even personality and traits (Pennebaker et al., 2003). Moreover, previous re- search has found that stronger emotional responses experienced in such consumption situations of- ten lead to the posting of product/service reviews (Derbaix and Vanhamme, 2003: Hennig-Thure et al., 2004). Thus, reviews are typically informa- tive enough to reflect features of emotions and ap- praisals that underlie the text and therefore provide a suitable context for modeling consumer behavior.
The key contributions of our work are as follows:
A framework for evaluating appraisal and emotion elicitation through writing prompts. A novel dataset, the first to contain review text labeled with a large set of first-person emotional, cognitive, and trait variables, with cross-domain applications in computational linguisties, consumer psychology, computa- tional psychology, and interaction design. Baseline experiments on identifying emotions

---

In summary, motivated by how emotions and cog- nitive appraisals are critical in understanding con- sumer behavior, our broad aim is to analyze these emotional constructs in review text. The following research questions guided our dataset curation:
et al., 2007: He and McAuley, 2016), and the Yelp restaurant review dataset (Asghar, 2016) are widely utilized in studies that analyzed sentiments prod- uct/services. Other review datasets include restau- rant (Gojali and Khodra, 2016; Li et al., 2021; McAuley and Leskovec, 2013; Pontiki et al., 2015, 2016; Zahoor al., 2020), and hotel reviews (Cal- heiros et al., 2017; Yordanova and Kabakchieva, 2017). Therefore, the lack of product/service review dataset containing emotionally rich vari- ables such as cognitive appraisals and emotional intensity is a primary motivation for the proposed dataset. Such a dataset can be useful not only for researchers from computational linguistics and con- sumer psychology to model linguistic correlates of emotional constructs and behaviors in the do-
How can we elicit high-quality text consisting of emotions and cognitive appraisals of con- sumption experiences using writing prompts? How do these compare to the product reviews posted to online marketplaces, such as Ama- zon?
3 Related Work
Numerous publicly available text datasets are an- notated with emotion labels that follow various theoretical emotion taxonomies (Murthy and Ku- mar, 2021). These datasets span a wide range of domains such as tweets (Mohammad and Bravo- Marquez, 2017: 77 Mohammad et al., 2018; Oberlän- der and Klinger, 2018 8; Wang et al., 2016), sto- ries (Alm, 2008; Alm et al., 2005), news headlines (Buechel and Hahn, 2017; Rosenthal et al., 2019; Strapparava and Mihalcea, 2007), and personal de- scription of emotional events (Scherer and Wallbott, 1994).
main of consumption but also for practical appli- cations in the fields of marketing, advertising, and business analyties to understand how specific prod- uct/service designs can be improved to result targeted emotions.
4 Research design
This study aimed to curate a dataset of text re- sponses labeled with rich emotional variables such as appraisal and emotional intensity. Therefore, we conducted pilot study to evaluate which kinds of writing prompts elicit high-quality text of emotions and cognitive appraisals. In an online panel, quali- fied participants were assigned to one of four con- ditions eliciting different kinds of detail about their cognitive appraisal of product they purchased. After evaluating the data collected for its emotional and cognitive richness through automatic content analysis methods, we subsequently used the most effective writing prompts to collect the full dataset.
Although there are some annotated emotional textual datasets with third-person appraisal dimen- sions (Hofmann al., 2020; Skerry and Saxe, 2015; Troiano et al., 2019, 2022), these are perceived ap- praisal ratings by third-person readers rather than first-person appraisals that are personally evaluated by the writer who experienced the situation. Third- person ratings are conceptually different from first- person ratings as they measure emotional percep- tion rather than experience. The former ratings are indications of how a third-person (i.e. reader) in- terprets situation that someone else (i.e. writer) is experiencing, and then from these interpretations, reason how the first-person writer is feeling (Ong al., 2015; Skerry and Saxe, 2015). These third- person appraisal ratings might differ from how the writers personally appraise the situation in terms of the kinds of appraisal dimensions used and the intensity of the appraisal ratings. Thus, the exist- ing datasets cannot address questions relating to first-person emotional experiences.
Pilot Study
The purpose of the pilot study was to evaluate the quality of emotional text responses elicited by dif- ferent writing prompts. Its protocol design is il- lustrated in Figure 1. Therefore, we employed a 2 x 2 factorial between-subjects design to examine whether a questionnaire format might yield more detailed emotional responses or whether review format is as emotionally detailed the question- naire format. The two factors that are manipu- lated are the presence of emotion prompts (emotion prompts or no emotion prompts), and 2) response format (review or questionnaire). This design was motivated by the concern that the review format
Datasets of products/service reviews are primar- ily investigated in sentiment analysis where the outcome labels are either 1) positive, negative, or neutral, or 2) an integer rating scale of to 5. The Amazon product reviews dataset (Blitzer

---

responses might emphasize product-specific details instead of emotional reactions even when we ask participants to write about the emotional aspect of it. Another primary concern is that the review format responses might not contain text pertain- ing to appraisal dimensions. Therefore, the four conditions are:
In the no-Emotion treatment, participants were tasked to think about a time that they have used or experienced an expensive product/service that they purchased without any emotional prompts.
Next, we manipulate how participants are re- quired to express themselves. In the Question- naire treatment, participants will answer series of open-ended questions about their experience with the product that they have thought about. We only included four specific open-ended appraisal questions (goal relevance, goal conduciveness, ex- pectedness, and pleasantness) due to the feasibility of data collection (i.c. eliciting responses for a large number of appraisals might result in poor response due to fatigue) although praisal dimensions might be relevant in the context of consumption (see Table of Appendix A for the appraisal questions). The choice of these appraisal dimensions to elicit text responses is supported by previous studies that show certain appraisals that are particularly relevant to the context of consump- tion (Demir et al., 2009; Desmet, 2008). The design of the questions was adapted from methodologi- cal frameworks of previous studies that investigate appraisals and emotions in Psychology (Scherer. 1997; Smith and Ellsworth, 1985; Tong, 2015). We tried our best to facilitate recall by including ad- ditional prompts such as 'How did you feel when you used the product/service? and 'What about the product that caused you to feel the emotion stated?" which might also elicit other appraisals other than the responses the four appraisal prompts.
Emotion Review: Presence of emotion
prompts, and review format
Emotion Questionnaire: Presence of emo- tion prompts, and questionnaire format Review: Absence of emotion prompts, and review format
Questionnaire: Absence of emotion prompts, and question format
think that other ap-
Groption::
2 wer--
Figure 1: Schematic diagram of procedure used in the Pilot study.
The writing prompts for the different conditions are found in Table 5 of Appendix A. The context examined in the pilot and the main study is spe- cific to using expensive product/service. This class of products has the highest likelihood of ex- periencing an emotion and also results in higher emotional intensity while using it as its usage rep- resents a situation that the participants think deeply about before purchasing and is personally impor- tant to them, compared to the experience of using a mundane economical product such pen toothbrush (Richins, 2008). Therefore, we pect that the responses to using these products are emotionally rich.
Finally, in the Review treatment, participants were tasked to write a review about their experience of using the product that they have thought about as if they were going to publish online.
After completing one of the four conditions. participants will also complete a validated cogni- tive appraisal and emotion questionnaire that was adapted from previous studies (Scherer and Wall- bott, 1994; Smith and Ellsworth, 1985; Tong and Jia, 2017; Yeo and Ong, 2023), where they will be asked to rate how they appraised and felt when describing the experiences on set of 20 appraisal (see Table 6 of Appendix A), and 8 emotion dimen- sions (see Table 7 of Appendix A), Likert scale. The emotions are chosen based on the literature on consumption emotions commonly experienced during the usage of products (Richins, 2008). This is to account for the fact that partici- pants might feel multiple emotions during a con-
In the Emotion treatment, participants were tasked to recall when they felt either the emotion of Joy or Anger when they used or experienced an expensive product/service they purchased. These emotions are chosen based the literature con- sumption emotions commonly experienced during the usage of products (Richins, 2008). Therefore. the 'presence of emotion prompts' condition asks the participants to think about an experience that involves a particular emotion.
7-point

---

sumption episode.
example, the social and causation dimensions in LIWC-22 might reflect accountability- and control- related appraisals. general, the responses from the questionnaire format have a greater number of appraisal-related words than the review format with the exception of fuilfil, need, social, and dis- crepancy. Despite this, the correlations between these LIWC22 dimensions and related appraisal dimensions (Figure of Appendix A) are greater for the responses in the questionnaire format. For example, the correlation between goal relevance and the needs dimension of LIWC22 for the ques- tionnaire format is -.55 which is greater than -.12 for the review format. Therefore, appraisal ratings are generally consistent with the expressed text for the questionnaire format compared to the review format.
We created a Qualtries survey that reflects these conditions with the respective questions and prompts. Once participants have consented to par- ticipate, they will be randomly assigned to one of the conditions.
5.1 Data Collection
We recruited participants using the crowdsoureing called Prolific. This platform was specifically de- signed for academic research and has pool of participants from the Organization for Economic Co-operation and Development (OECD) countries. Although Prolific has a pool of participants from different countries, we only recruited participants from the United States. This is to reduce the het- erogeneity of the data for subsequent analyses. A total of 18 participants were recruited in De- cember 2022. We also set quota to obtain equal proportion of gender in our sample (50% male, 50% female). The pilot dataset published on Zenodo at http:///oooo..o//rrrood//////8
On the other hand, the responses received in the review format are mixed (see Table of Ap- pendix A for sample responses). Although some were very detailed and included emotional state- ments and appraisals, some responses were rela- tively short. On top of that, responses in this format typically talk about the characteristics of the prod- uets instead of the emotional experience of using the product.
5.2 Expressed text in different conditions
One of the primary motivations for this study is to collect review text responses that express appraisal dimensions. In terms of the response format (re- view vs. questionnaire format), the questionnaire format generally resulted in more detailed ex- planation of the participants' emotional situation (see Table 8 of Appendix A for sample responses). The average word count for texts in the question- naire and review format is 150 and 62 words, re- spectively. Morcover, the participant answering in the questionnaire format provides detailed ex- planations on specific appraisal questions asked (e.g. why is the product consistent with what you wanted?). Thus, the text responses from the ques- tionnaire format conditions clearly express the ap- praisal dimensions.
Figure 2: Means of selected LIWC-22 dimensions com- paring questionnaire and review formats, respec- tively. "socref" refers to "social referents." "discrep" refers to "discrepancy." The full description of the cat- egories, their definitions, and contents are available in the LIWC 2022 manual.2
Moreover, we coded for several appraisal di- mensions using the Linguistic Inquiry and Word Count 2022 dictionaries (Boyd et al., 2022) (Fig- ure 2). LIWC22 dictionaries measure the evidence of emotional, cognitive, and social processes ex- emplified in writing by looking at simple word fre- quency counts. Their association with psycholog- ical processes, emotions, and cognitive appraisal has been well-established in prior work (Jaidka et al., 2020). Although LIWC22 does not code for cognitive appraisals per se, certain dimensions of LIWC22 might reflect cognitive appraisals. For
5.3 Linguistic feature comparison
Figure 3 compares the presence of words denoting cognitive processes in the pilot study to topically similar reviews in the Kaggle Amazon dataset (N 18078), matched through a keyword search of the products mentioned in the pilot study. The first barplot compares the distribution of Affect scores, which comprises words used for both positive and

---

negative tones and emotional expressions, swear words (good, well, love, happy, bad, wrong, hate), and Cognition scores, which denote styles of think- ing and processing information (know, how, think, feel), discrepancies (would, can, want, could), com- parison (but, not, if. or), and so on. The second barplot compares the differences in the psychologi- cal states reflected in the text. Psychological states indicate signals of acquisition (get, take, getting), fulfillment (enough, full, complete, extra) and want- ing (want, hope).
(Richins, 2008).
During the main data collection, participants fol- lowed the framework of the presence of emotion prompts and questionnaire format. That is, partic- ipants were tasked to state the dominant emotion experienced when they were using the expensive product/service. Unlike the pilot study where the emotion prompts involved only the emotion of joy or anger, participants were instead asked to state the dominant emotion (amongst the eight emotions listed and an 'other' option). Following this, they answer the same set of questions as in the pilot study in the text regarding their experiences and appraisals with respect to the dominant emotion that they have stated.
The barplots show how the text elicited through the questionnaire format overshadows Amazon re- views in terms of the simple average proportion of words that reflect emotional and cognitive ap- praisal. A two-tailed Wilcoxon signed-rank test of independent samples found that the level of cognition words reported in the two questionnaire conditions was significantly higher than the corre- sponding number for Amazon reviews (p- < 0.001). Moreover, across all dimensions of psychological states, the questionnaire format has greater av- erage proportion of words compared to Amazon reviews except for the dimension of acquire.
Additionally, we also added a trait measure called Preference for Intuition and Deliberation Scale (Betseh, 2004) measure individual differ- ences in the tendency utilize a particular deci- sion mode. This trait measure is highly relevant in the context of consumer behavior to understand the decision makiing process underlying the con- sumers' experiences. Finally, they answered the same validated 8-emotions and 20-appraisals ques- tionnaires in the pilot study. We also collected other responses such the product/service that the participants have recalled and the associated cost, intention to buy the product again, and intention recommend the product to others.
6 Main Study
Based on the results of the pilot study, we observed that indeed the structured nature of the question- naire format elicits richer emotional responses that contain appraisal dimensions that could be used to analyze such text and so we chose the ques- tionnaire format as the writing prompts to elicit appraisal and emotional text responses in the main study. In terms of the presence of emotion prompts, although there were generally no significant dif- ferences between questionnaire conditions with or without emotion prompts (with the exception of the cognition dimension of LIWC22), we decided to go along with the presence of emotion prompts as it allows to determine the ground truth emotion labels for each participant's s experiences.
This process was then repeated again and partic- ipants were tasked to recall another experience of using an expensive product/service that is opposite in terms of valence (positive or negative) to the ex- perience that they have previously stated. That is, if the participants recalled a positive experience pre- viously, they were now tasked recall and write about negative experience. This is typically done in appraisal studies and allows us to obtain another set of responses (Tong, 2015; Smith and Ellsworth, 1985).
Prior to the main data collection, we con- ducted a screening study that asked one question- whether participants have purchased expensive products/services recently. Participants who an- swered yes are then eligible to participate in our main study, More broadly, this question serves as a proxy for products/services that are personally important to the participants, which we are inter- ested in as this class of products has the highest likelihood of experiencing an emotion while using
6.1 Data Collection
A total of 700 participants were recruited in March 2023 (See Table 10 of Appendix B for demographic statistics). Since each participant provided text responses and ratings for two experiences (one positive and one negative), our dataset consists of 1,400 instances. The data can be found in the following repository- https: //gi ttg.thuu.comm GerArdVeo/PEACE- Reviewwwweewww........ gittt

---

EXPRESSIONS COGNITION
EXPRESSIONS OF PSYCHOLOGICAL STATES
Figure 3: Distribution of linguistic features compared Amazon Reviews sample (N 18,078) matched to the topics of the
Pilot dataset through keyword search.
6.2
Data Characteristics
Prompts
Mean
Min
Max
What about the product/service made you feel the emotion that you have stated? Why were you feeling that emo- tion when you were using or ex- periencing the product/service? Was the product/service im- portant to you? Please also state the reason(s) of why was the product/service impor- tant/unimportant you. Was the product/service consis- with what you wanted? Please also state the reason(s) of why was product/service con- sistent with what you wanted. Was a pleasant experience when you were using the prod- uct/service? Please also state the reason(s) of why was it a pleasant experience when you were using the product/service.
53.5
780
6.2.1
Statistics
The statistics of the text responses for each prompt are presented in Table 1. When combining the text responses from all the prompts, the average number of tokens significantly longer than previous au- tobiographical emotion dataset (ISEAR; (Scherer and Wallbott, 1994), and other reviews datasets (Maas et al., 2011). 1). Datasets with small number of tokens for each instance hinder the identification of linguistic features of large number of appraisal and emotion constructs. On the other hand, the relatively large number of tokens for each instance in our dataset has the potential to study and seg- ment linguistic features of uppraisals and emotions in first-person emotionally rich autobiographical writings.
29.7
7
360
27.5
256
25.7
243
26.3
260
While using the product, did ev- erything turn out to be what had expected? Please also state what turned out to be pected/unexpected.
25.8
7
285
6.2.2 Analysis of appraisals and emotions ratings
In terms of ratings, focus on appraisals and emotions, although we collected other data such as decision-making and person-level traits. For the cognitive appraisal questionnaire, there is an option for participants to indicate whether an ap- praisal is relevant during their experience of using the product/service. It is important to study these NA responses as not all appraisal dimensions might be relevant to the context of consumption. Table 6 shows the percentage of NA responses for each appraisal dimension. The mean percentage of NA responses across all the appraisals 9.49%. The appraisals of goal conduciveness and pleasantness have the lowest number of NA responses which sug- gests that they are highly relevant in emotional ex- periences. On the contrary, appraisals might not be that relevant in the context of consumption experiences.
Total
190.2
1821
Table I: The values refer to the number of tokens across
all participants' responses to the various prompts.
called expensive product/service. The distribution of emotions helps understand what kind of emotions felt during the usage of expensive product/service. On top of providing the list of eight emotions for participants to select, we also included a free text component for participants to write down the dominant emotion felt. This is for cases when participants did not endorse any of the eight emotions provided. Joy and disappointment appear to be the largest dominant positive and neg- ative emotions felt, respectively.
ceeuusttances-elattee
Next, since each emotion is proposed to have a unique appraisal profile according to cognitive appraisal theory, we conducted a MANOVA on the 20 appraisal ratings to observe whether there are
Table 2 indicates the distribution of the dominant emotion felt when participants were using the re-

---

Emotion
%
to predict each of the appraisal dimensions into 3 classes- high, medium, and low, typical of how psy- chologists study appraisals (Ellsworth and Smith, 1988; Scherer and Wallbott, 1994).
Anger
4.8
Disappointment
32.2
Disgust
2.0
Gratitude
11.3
Joy
19.2
Pride
4.6
7.2 Baseline Models
Regret
14.4
Surprise
1.7
We used the following baseline models to run the experiments-
Others
9.7
Table 2: Distribution of emotion classes in the PEACE- Reviews dataset. These are the dominant emotions en- dorsed by the participants when recalling specific ex- perience.
DistillRoBERTa (Hartmann, 2022), transformer-based model, was fined tuned set of six emotion datasets such GoEmotions (Demszky et al., 2020) and SemEval-2018 (Mohammad et al., 2018). This model contains six layers, 768 dimensions, and 12 attention heads.
differences between the eight emotions. We group all observations based on the dominant emotion se- lected by the participants. The omnibus MANOVA shows that there are significant differences amongst the eight emotion groups based the 20 appraisal ratings, Wilk's A is F(140, 5358.68) 8.86, p .05. We also conduct linear discriminant analyses that extend MANOVA to visualize the differences amongst emotion groups using appraisals. Figure shows the boxplots of the first linear discriminant of appraisals for different emotions. We can ob-
BERT (Devlin et al., 2018), a transformer model that applies bidirectional training to learn informative contextual features from quences. We used the base model that con- tains 12 layers and attention heads, and 768 dimensions.
XLNet (Yang et al., 2019) builds upon the BERT's limitations of failing to account for the dependencies between masked positions in sequences and is based on autoregressive language modeling that can learn bidirectional context. We used the base model that contains the same number of layers, attention heads, and dimensions as BERT.
serve that this discriminant is function of the valence of emotions where emotions within the positive (gratitude, joy, pride) and negative (anger, disappointment, disgust, regret) emotion groups are similar. Surprise is considered have both positive and negative valences.
OpenAl's GPT3-ada (OpenAI, 2023) a GPT- 3-based model (Brown et al., 2020). We ran the model using the OpenAl API.
The dataset was split into training (80%). valida- tion (10%), and test sets (10%), where the training and validation sets used in finetuning to the respective tasks. The last layer of the models was modified accordingly suit the number of classifi cation outputs for the respective tasks. All models are fine-tuned on 10 epochs and implemented Python.
Figure 4: Boxplots of the first linear discriminant func- tion of emotions.
7.3 Emotion prediction
For the emotion and appraisal prediction tasks, we use the classification accuracy and average weighted F1 scores as the evaluation metrics. Ta- ble 3 presents the results of the eight-way emotion classification. OpenAl's ada model appears to have the best results, and it even performs better than Distil which was pre-trained on multiple emotion corpus. Nevertheless, the performances of all of the models are comparable given that our
7 Baseline Experiments
7.1 Tasks
First, we conducted eight-way emotion classifi- cation task to predict the emotions of the combined text responses to the prompts. The second task also uses the combined text responses to the prompts

---

Model
Accuracy
F1
Appraisal/Model Pleasantness
Accuracy
F1
BERT
67.9
0.62
XLNet DistillRoBERTa
0.67
BERT
0.74
68.6
0.67
XLNet
0.73
OpenAl ada
72.1
0.71
DistilRoBERTa
74,8
0.75
OpenAl
69.8
0.69
Table 3: Results of waght-way emotion classification. The values the AAcurrcy'' column are in percentages.
Goal conduciveness
BERT
70.5
0.71
XLNet
71.2
0.70
DistilRRoBERTa
70.5
0.70
OpenAl ada
73.0
0.73
dataset has imbalanced emotion classes, with joy and disappointment dominating. Table 12 of Ap- pendix B presents the accuracies and Fl scores for the individual emotions. As anticipated, since the emotions of joy and disappointment are the most dominant emotions in our dataset, the models ex- cel in predicting these emotions but occasionally misclassify them into other positive and negative emotion classes. This suggests models primarily capture valence and pleasantness but struggle to differentiate specific emotions within each positive and negative emotion class.
Expectedness
BERT
50.8
0.49
XLNNt
57.8
0.55
DistllRoBERTa
55.5
0.53
OpenAl ada
56.1
53.7
Goal relevance
BERT
68.3
0.68
XLNet
73.5
0.73
DistilRoBERTa
69.1
0.68
OpenAl
66.1
0.65
Table 4: Results of three-way (high, medium, low) praisal classification for goal conduciveness, goal rele- vance, pleasantesss, and expectedness.
Given the unique emotion classes in our dataset, direct comparisons with other datasets are challeng- ing. However, close comparison is the ISEAR dataset (Scherer and Wallbott, 1994), which has 3 emotions that overlapped with our of emo- tions, where BERT and XLNet achieved 70.1% and 73.0% accuracy in seven-way emotion classifica- tion, respectively (Adoma et al., 2020).
Overall, the models are comparably similar the prediction of appraisals except for BERT's predic- tion for expectedness.
Conclusion
We curated a text dataset that described autobio- graphical expressions of participants' experiences when using an expensive product/service, anno- tated with large number of self-reported psycho- logical variables such as cognitive appraisals, emo- tions, traits, and future consumer behavior. We experimented with various writing prompts the pilot study so as to determine which method clic- its the best expression of emotional and cognitive appraisals of consumption experiences. Further- more, we conducted experiments with state-of-the- art language models to provide baseline evalua- tions of appraisal and emotion classifications. Al- though we primarily examined appraisals and emo- tions in this paper, future research could utilize the other variables in this dataset such as decision- making traits, and post-purchase consumer behav- ior (e.g. intention to purchase and product again, and recommending the product to someone else) to model how appraisals and emotions interact with these variables. In summary, the PEACE- PEACE-Reviews Reviews dataset enables researchers to study the correlates between expressed linguistic features and cognitive- emotional processes and affords a comprehensive understanding of emotional experiences in the do- main consumption and product/service reviews.
7.4 Appraisal prediction
Table shows the results of the three-way appraisal classification for the appraisals of goal conducive- ness, goal relevance, pleasantness, and expected- ness. Since these are the appraisals that have text responses, we prioritize them and present them in the main text. The predictions of the other ap- praisals and the distribution of appraisal classes can be found in Table 14 and Table 13 of Appendix B. Across the three appraisals, the prediction accura- cies of models on expectedness are the lowest. Un- like the appraisals of pleasantness and goal congru- ence, where positive/negative valence words could be easily identified and thus enhance the predictive performance, expectedness is a more complicated appraisal where participants described whether the experience was expected or unexpected. Even though BEsti
pre-trained emo- tion multiple emotion corpora, it did not have an edge in predicting the various appraisal dimensions except for pleasantness, which expected since pre-training on multiple emotion corpora might learn words that are related to emotional valence.

---

Limitations
Acknowledgments: We thank Mike Cheung. Harold Soh, and Desmond Ong for early feedback on the research design. This work was supported by a grant from the NUS Centre for Trusted Internet and Community.
Although this curated dataset contains autobio- graphical text describing participants' appraisals and its associated ratings, we only included four appraisal questions (goal relevance, goal conducive- ness, expectedness, and pleasantness) to elicit text responses about appraisals. Nevertheless, we se- lected these four appraisals because they are most relevant to the context of consumption (Desmet, 2008) and we feel that introducing more appraisal questions for participants' complete could results in poor quality responses due fatigue. Despite this, on top of the four appraisal questions, partic- ipants also provided text responses on questions such as 'what about the product/service that made you feel this way?' and 'why did you feel this way when you use the product/service? alongside an appraisal questionnaire that contains 20 different appraisals. From the text responses elicited from these two questions and its associated appraisal ratings, linguistic features of appraisal dimensions could still be extracted.
References
Chethana Achar, Jane So, Nidhi Agrawal, and Adam Duhachek. 2016. What we feel and why we buy: the influence of emotions on consumer decision-makenn making. Current Opinion in Psychology, 10: 166-170.
Acheampong Francisca Adoma, Menoo-Mensah Henry, and Wenyu Chen. 2020. Comparative analyses of bert, roberta, distilhert, and xlnet for text-based emo- tion recognition. 2020 17th International Com- puter Conference on Wavelet Active Media Tech- nology and Information Processing (ICCWAMTIP). pages 117-121. IEEE.
Cecilia Ovesdotter Alm, Dan Roth, and Richard Sproat. 2005. Emotions from text: machine learning for text- based emotion prediction. In Proceedings of human language technology conference and conference empirical methods in natural language processing. pages 579-586.
Ethies Statement
Ebba Cecilia Ovesdotter Alm. 2008. Affect in text and speech. University of Illinois at Urbana-Champaii
8.1 Composition
This dataset consists of participants' review text responses and the self-reported first-person ratings of emotion variables. These are the primary fields that are of interest to us. We also included an op- tional demographic questionnaire at the end of our survey. This demographic questionnaire was taken from Qualtrics's databases of demographic ques- tionnaire templates that apply to the context of the US population. These questions are typically asked in research to determine the sample's composition. There was no personal data collected during the process of constructing this dataset.
Magda B Arnold. 1960. Emotion and personality.
Nabiha Asghar. 2016. Yelp dataset challenge: Review rating prediction. arXiv preprint 1rXiv:1605.08 05362..
Cornelia Betsch. 2004. Präferenz für intuition und de- liberation (pid). Zeitschrift für Differentielle und Diagnostische Psychologie, 25(4):179-197)
John Blitzer, Mark Dredze, and Fernando Pereira. 2007. Biographies, bollywood, hoom-boxes and blenders: Domain adaptation for sentiment classification. In Proceedings of the 45th annual meeting of the asso- ciation of computational linguistics, pages 440 447.
Ryan L Boyd, Ashwini Ashokkumar, Sarah Seraj. and James W Pennebaker. 2022. The development and psychometric properties of liwe-22, Austin, TX: Uni- versity of Texas at Austin.
8.2 Data Collection
The Departmental Ethics Review Committee of the Department of Communications and New Media at the National University of Singapore reviewed this study and determined it to be exempt. We respected the autonomy and protected the participants' pri- vacy during every stage of the data collection pro- cess. Informed consent was obtained before the start of the study, and participants could withdraw at any point time, including after providing their data, which we have discarded and not used in any subsequent analyses.
Ryan L Boyd and James W Pennebaker. 2015. A way with words: Using language for psychological sci- ence in the modern era. In Consumer psychology in social media world, pages 222-236. Routledge.
Tom Brown, Benjamin Mann, Nick Ryder, Melanic Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, al. 2020. Language models are few-shot learners. Advances neural information processing systems, 33: 197-1

---

Sven Buechel and Udo Hahn. 2017. Readers vs, writers Sharath Chandra Guntuku, Anneke Buffone, Kokil Jaidka, Johannes C Eichstaedt, and Lyle H Ungar. 2019. Understanding and measuring psychological stress using social media. In Proceedings of the inter- national AAAI conference on web and social media. volume 13, pages 214-225.
vs. texts: Coping with different perspectives of text understanding in emotion annotation. In Proceedings of the 1 11th Linguistic Annotation Workshop, pages 1-12.
My Bui, Anjala Krishen, and Kenneth Bates. 2011. I. Modeling regret effects on consumer post pusthaee decisions. European Journal of Marketing.
Jochen Hartmann. 2022. Emotion english distilroberta- base. /tttps///uggiii ingggce jo emotion-english--ee -istiiiiddddddsrrbberrrr tar base/.
Ana Catarina Calheiros, Sérgio Moro, and Paulo Rita. 2017. Sentiment classification of consumer- generated online reviews using topic modeling. Journal of Hospitality Marketing Management, 26(7):675-692
Ruining He and Julian McAuley. 2016. Ups and downs:
Modeling the visual evolution of fashion trends with one-class collaborative filtering. proceedings of the 25th international conference on world wide web. pages 507-517.
Hans Christian, Derwin Suhartono, Andry Chowanda, and Kamal 7. Zamli. 2021. Text based personality prediction from multiple social media data sources using pre-trained language model and model averag- ing. Journal of Big Data, 8-1):1-20.
Thorsten Hennig- TTuriuu Kevin P Gwinner, Gian- franco Walsh, and Dwayne D Gremler. 2004. Elec- tronic word-of-mouth via consumer-opinion plat- forms: what motivates consumers to articulate them selves on the internet? Journal of interactive market- ing, 18(1):88-52
Jeremy DW Clifton. 2020. Testing if primal world beliefs reflect experiences- or at least some expe- riences identified ad hoc. Frontiers in Psychology, 11:1145.
Jan Hofmann, Enrica Troiano, Kai Sassenberg, and Roman Klinger. 2020. Appraisal theories for emotion classification in text. arXiv preprint a:Xiv:2003.14155
Mummun De Choudhury, Scott Counts, and Eric Horvitz. 2013. Social media as measurement tool of depres- sion in populations. In Proceedings of the Sth annual ACM web science conference, pages 47-56.
Kokil Jaidka, Anneke Buffone, Johannes Eichstaedt, Masoud Rouhizadeh, and Lyle H Ungar. 2018. Mod- cling and visualizing locus of control with facebook language. In Twelfih International AAAI Conference Weh and Social Media.
Erdem Demir, Pieter MA Desmet, and Paul Hekkert. 2009. Appraisal patterns of emotions in human- product interaction. International journal of design, 3(2).
Kokil Jaidka, Salvatore Giorgi, H Andrew Schwartz, Margaret L Kern, Lyle H Ungar, and Johannes C Eichstaedt. 2020. Estimating geographic subjective and from twitter: A comparison of dictionary language methods. Proceedings of the National Academy of Sciences, 117(19):1016- 10171.
Dorottya Demszky, Dana Movshovitz-Atiias, Jeongwao Ko, Alan Cowen, Gaurav Nemade, and Sujith Ravi. 2020. Goemotions: A dataset of fine- mrained emo- tions. arXiv preprint arXivv2005.008 00547.
Christian Derbaix and Joelle Vanhamme. 2003. Induc- ing of oord-of-mouth by eliciting surprise-a pilot inves- tigation. Journal of economic psychology. 24(1):99- 116.
Richard S Lazarus, Allen D Kanner, and Susan Folkman. 1980. Emotions: A pognitve-phenomenollgiio analysis. In Theories of emotion, pages 189-217. Elsevier.
Pieter MA Desmet. 2008. Product emotion. In Product experience, pages 379-397. Elsevier.
Kyung-Tag Tag Lee and Dong-Mo Koo. 2012. Effects of attribute and valence of e-wom on message adop- tion: Moderating roles of subjective knowledge and regulatory focus. Computers in human behavior, 28(5):1974-199 1984.
Jacob Devlin, Ming-We Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understand- ing. Xiv preprint arXiv: 1.1.:0880004005
Phoebe C Ellsworth and Craig A Smith. 1988. From appraisal to emotion: Differences among unpleasant feelings. Motivation and emotion, 12(3):271-300
Jennifer Lemne and Dacher Keltner. 2001. Fear, anger. and risk. Journal of personality and social psychol- ogy, 81(1):146.
Nico H Frijda. 1987. Emotion, cognitive structure, and action tendency. Cognition and emotion, 1(2):1 155- 143.
Jennifer S Lerner, Ye Li, Piercarlo Valdesolo, and Karim S Kassam. 2015. Emotion and decision mak- ing. Annual review of psychology, 66(1).
Susanti Gojali and Masayu Leylia Khodra. 2016. As- pect based sentiment analysis for review rating pre- diction. In 2016 International Conference On Ad- vanced Informatics: Concepts, Theory And Applica- tion (ICAICTA), pages 1-6. IEEE.
I Liangqiang Li, Liang Yang, and Yuyang Zeng. 2021. Improving sentiment classification of restaurant re- views with attention-based bi-gru neural network. Symmetry, 1 1368)

---

Andrew Maas, Raymond E Daly, Peter T Pham, Dan Huang, Andrew Y Ng, and Christopher Potts. 2011. I Learning word vectors for sentiment analysis. In Proceedings of the 49th annual meeting of the associ- ation for computational linguistics: Human language technologies, pages 142-150.
Bing Qin, Orphée De Clereq, et al. 2016. Semeval- 2016 task 5: Aspect based sentiment analysis. In In- ternational workshop on semantic evaluation, pages 19-30.
Maria Pontiki, Dimitrios Galanis, Harris Papageorgiou. Suresh Manandhar, and Ion Androutsopoulos. 2015. Semeval- task 12: Aspect based sentiment analy- sis. In Proceedings of the 9th international workshop on semantic evaluation (SemEval 2015), pages 486- 495.
Julian John McAuley and Jure Leskovec. 2013. From
amateurs to connoisseurs: modeling the evolution of user expertise through online reviews. In Proceed- ings of the 22nd international conference on World Wide Web, pages 897-908.
Marsha I. Richins. 2008. Consumption emotions. In Product experience, pages 399-422. Elsevier.
Saif Mohammad, Felipe Bravo-Marquez. Mohammad Salameh, and Svetlana Kiritchenko. 2018, Semeval- 2018 task I: Affect in tweets. In Proceedings of the 12th international workshop on semantic evaluation, pages 1-17.
Ira Roseman, 1984. Cognitive determinants of emo- tion; A structural theory. Review of personality & social psychology.
Saif M Mohammad and Felipe Bravo- Pravoeaaquue.. 2017. Wassa- 2017 shared task on emotion intensity. arXiv preprint arXiv: 1w1.177780000000
Sara Rosenthal, Noura Farra, and Preslav Nakov. 2019. 2001 task 4; Sentiment analysis in twitter. arXiv preprint 1912v=191..00744
Sarah G Moore. 2015, Attitude predictability and help- fulness in online reviews: The role of explained ac- tions and reactions. Journal of Consumer Research, 42(1 +204)430-4
Ovi Hamidah Sari. 2022. Theory of planned behaviour in marketing: Cognitive consideration purchase decision. Golden Ratio of Mapping Idea and Litera- ture Format, 2( 107701-077
Agnes Moors. Phoebe C Ellsworth, Klaus R Scherer, and Nico H Frijda. 2013. Appraisal theories of emo- tion: State of the art and future development. Emo- tion Review, 5(2):119-11)) 124.
Klaus R Scherer and Harald G Wallbott. 1994. Evidence for universality and cultural variation of differential emotion response patterning. Journal of personality and social psychology. 66(2):110.
Ashritha R Murthy and KM Anil Kumar. 2021. A re- view of different approaches for detecting emotion from text. In TOP Conference Series: Materials Sci- ence and Engineering, volume 1110, page 212009. IOP Publishing.
Klaus R Scherer et al. 1984, On the nature and function of emotion: component process approach. Ap- proaches to emotion, 2293(317):31.
KlausR Scherer. 1997. Profiles of appraisal: Testing theoretical predictions across cul- tures. Cognition & Emotion, (1):: 13-150.
Laura Ana Maria Oberländer and Roman Klinger. 2018. An analysis of annotated corpora for emotion clas- sification in text. In Proceedings of the 27th Inter- national Conference on Computational Linguistics, pages 2104-2119.
Matthias Siemer and Rainer Reisenzein. 2007. The process of emotion inference. Emotion, 7(1):1.
Amy Skerry and Rebecca Saxe. 2015. Neural repre- sentations of emotion are organized around abstract event features. Current biology, 25(15):1945-1954
Desmond C Ong, Jamil Zaki, and Noah D Goodman. 2015. Affective cognition: Exploring lay theories of emotion. Cognition, 143:141-162.
Craig A Smith and Phoebe C Ellsworth. 1985. Pat- terns of cognitive appraisal in emotion. Journal of personality and social psychology, 48(4):813.
OpenAI. 2023. docs/models.
com/
Andrew Ortony, Gerald L Clore, and Allan Collins. 2022. The cognitive structure of emotions. Cam- bridge university press.
Doorgesh Sookarah and Loovesh S Ramwodin. 2022. Combatting online harassment by using transformer language models for the detection of emotions. hate speech and offensive language social media. In 2022 4th International Conference on Emerging Trends in Electrical, Electronic and Communications Engineering (ELECOM), pages 1-6. IEEE
James W Pennebaker, Matthias Mehl, and Kate Niederhoffer. 2003. Psychological aspects of natural language use: Our words, our selves, Annual review of psychology, 54(1):547-577.
Carlo Strapparava and Rada Mihalcea. 2007. Semeval- 2007 task 14: Affective text. In Proceedings of the Fourth International Workshop on Semantic Evalua- tions (SemEval- 2007), pages 70-74.
Maria Pontiki, Dimitrios Galanis, Haris Papageorgiou, Androutsopoulos, Suresh Manandhar, Moham- mad Al-Smadi, Mahmoud Al-Ayyoub, Yanyan Zhao,

---

Yla Tausczik and James W Pennebaker. 2010. The psychological meaning of words: Liwe and comput- erized text analysis methods. Journal of language and social psychology, 29(1):24-54,
"Recollection phase" is when participants are re- calling an experience of using an expensive prod- uct/service. During the "Writing phase", partici- pants then write down their experiences according to the questions asked in the different conditions.
Eddie MW Tong. 2015. Differentiation of 13 posi- tive emotions by appraisals. Cognition and Emotion,
29(3):484-503
Condition Elaoodionn Review)
Rocallcction phose
Writing phase
Eddie MW Tong and Lile Jia. 2017. Positive emotion, appraisal, and the role of appraisal overlap in positive emotion co-occurrence. Emotion, 17(1):40.
Think about the time when
After you have thought about your experience, describe this moment the form review the re- called productiervice as if you are going publish this reviow culine. Please write in complese sentences and detailed as possible.
you folt happylangry
you were using the expen- sive product/ervice that you thought shout. Picture this situation in your mind. Try your best remember this situation as vivilly ss you com
Enrica Troiano, Laura Oberländer, Maximilian Wegge, and Roman Klinger. 2022. x-envent: corpus of event descriptions with experiencer-specific emo- tion and appraisal annotations. arXiv preprint 11Xi:22033.10009
2 (Famotion Questionaaie)
-same above-
After you have thought abour your experience, describe this in the following questions,
1. Wix the praluctoervice impurtant you?? Why??
Enrica Troiano, Sebastian padó, and Roman Klinger. 2019. Crowdsourcing and validating event-focused emotion corpora for german and english. arXiv prepriat
2. What about the product that made you feel happylangry?
3. Why were you hapgy/angry when you used or experienced the product?
13618.
4. Wis the produstserviis consistant with what you wanted? Why?
Bo Wang. Maria Liakata, Arkaitz Zubiaga, Rob Procter, and Eric Jensen. 2016. Smiles: Twitter emotion classification using domain. In SAAIP@ IJCAL.
5. Was a pleasent experience when you were using the producthevvcs? Why?
6, While using the product. did erything turn out what you expucta?? Why?
Xiaopan Wang. Junpeng Guo, Yi Wu, and Na Liu. 2019. Emotion as signal of product quality: Its effect on purchase decision based on online customer reviews. Internet Research.
(Review)
Please think about the time when you were ing the expensive peod- acthersicu that you thought about. Picture this siturtion your mind. Try your best remeebbrr this sitution is vividly as
thought about your experience, describe this momentt in the form of review of the re- calked productisrvics as if you going to publish this review online. Please write complete sentenees. and dettilld as possible.
After
Bernard Weiner. 1985. An attributional theory of achievement motivation and emotion. Psychologi- cal review, 92(4):548.
(Quodionmire)
-oatn above-
Aßter you luve thought about your experiener, describe this moment the following questions,
Zhilin Yang. Zihang Dai, Yiming Yang, Jaime Car- bonell, Russ R Salakhutdinov, and Quoc V Le. 2019. Xlnet: Generalized autoregressive pretraining for lan- guage understanding. Advances in neural informa- tion processing systems, 32.
1. Wis the productivers impurtent to you! Way?
you usod the prodeeiserice? Please also state what aboutt the producttarrvice that you teel this way?
How did you
3. Was the produstvervie ice consis- tent with what you wanted? Why? 5. Was it plensnt experience when you ere using the producthservice? Why?
Gerard Christopher Zheng Jie Yeo and Desmond C Ong- 2023. A analytin review of the associations be- tween cognitive appraisals and emotions cognitive appraisal theory.
6. While using the product. did ev- erything turn out be what you had expected? Why?
Stanimira Yordanova and Dorina Kabakchieva. 2017. Sentiment classification of hotel reviews in social me- dia with decision tree learning. International Journal of Computer Applications, 158(7).
Table 5: Prompts and questions for the different condi- tion.
Table presents the appraisal questionnaire used in both the pilot and main studies (Yeo and Ong, 2023). The third column contains values for the percentage of NA responses for each appraisal di- mensions in the main study. As not all appraisal dimensions are relevant in the context of consump- tion, the percentage of NA responses allows us discern which appraisals are important in consump- tion experiences. Table 7 presents the emotion questionnaire used in both the pilot and the main studies. Both the appraisal and emotion question-
Kanwal Zahoor, Narmeen 7akaria Bawany, and Soomaiya Hamid. 2020. Sentiment analysis and classification of restaurant reviews using machine learning. In 2020 21 International Arab Confer- ence on Information Technology (ACIT), pages pages 1-6.
IEEE.
A Supplementary Information for Pilot Study
Table provides information about the prompts and questions used across different conditions. The

---

naires and rated on a 7-point Likert scale.
Emotion
Measure
Anger
To what extent did you Jeel angry?
Disappointment
To what extent did you feel disappointed?
Appraisal
Measure
% of NA 16.43
Disgust
To what extent did you feel disgusted?
Accountability-
To what extent did you think that circumstances beyond any- one's s control were responsible for what was happening in the situation?
Gratitude
To what extent did you Ieel gruteful?
circumstances
Joy Pride
To what extent did you feel happy?
To what extent did you feel proud?
Regret
To what extent did you feel regretful?
Accountahility-ote
To what extent you think that someone else other than you was responsible for what happening the situation? To what extent did you think that you were responsible for what was happening in the situ- ation?
11.79
Surprised
To what extent did you feel surprised??
Table 7:
The emotion dimensions measured in this dataset on a 7-point Likert scale. The questionnaire is sourced from prior work (Scherer et al., 1984; Smith and Ellsworth, 1985; Tong and Jia, 2017)
Accounaaility-self
4.93
Attentional activity
To what extent did you think that you needed to attend to the situation further?
14.64
resulted in higher quality responses in terms of the expression of cognitive appraisals and emotions in the text.
Certainty
To what extent did you under- stand what was happening in the situation?
5.14
cintumiiiics
To what extent you think that circumstances beyond any- one's 's control were controlling what was happening the situ- ation?
14.14
Question
Condition
Condition 4
Was the prod-
It war importont us part of self cure: I'd never spead nuch on amount on just my own
Yes becouse this prod-
uctiservice impor- tant you? Please also state the rea- sunfs) of why the product/service impor- tant/onimportant you.
uct was important to since was expensive meal with my friends, have good memories of this experience, and got to try things that have not tried before.
Contml-other
To what extent did you think that other people were control- ling what was happening in the situation?
14.21
enjoyment before. and
was tiberatigg There
Controlself
To what extent you think you had control over the situa- tion?
2.07
were comery cut all reganding the experi. ence, which was three day stay at
Coping potential
To what extent were you able to cope with any negative con- soquenees the situation?
16.36
exclie sive resort. (Juy)
How did feel when you used the prod- uctherrice? Pleae also state what abont the product/servioe mude you feel this wuy?
was best desoribed a constunt stote of be- wilderment at being pan- posed as was. It was elaborately stclusive and husurious, with amazing views and amenities. If not for the lousy weather, it would have been per- foet, but even preciated the scenery in falling rain. (Joy)
folt happy and jebt that was eating one of best meals my life. The product was excel- lent sushi chareen sushi chef, and it had just the right amount of wasolii and any sauce on each piece.
Difficulty
To what extent did you think the sitution was difficult?
6.79
Effort
To what extent did you think that you nceded to exert effort deal with the situation? ?
11.57
Expectedness
To what extent did you expect the situation occur? To what extent did you think that the situation was consistent with external and social norms? To what extent did you think the situation was fair?
9.00
External normative sig- nificance
15.50
ap-
Faireess
5.71
Was the prod.
It very consistent with what I wanted; I wanted absolute pin- nacle of resort experi. ence terms of exclutis- and comfort and and detail, and the Inter- contiontal provided that from landing to depar- ture. I never lacked for any service and every- thing was consistently ex- cellent. (Joy)
Yes because expected great mush for the high price, and subhi that had in my life. Therefore, the prod- consistent with what I expereddwoneed.
Future expectancy
To what extent did you think that the situation would ger worsehetter?
13.21
uct/service consistent what you wanted? Please also state the reason(s) of why was the productiseviice consistent with what you wanted.
best
Goul conduciveness
To what extent was the situ-- tion consistent with what you wanted?
Goal relevance
To what extent did you think that the situation was relevant what you wanted?
3.14
Novelty
To what extent did you think the situation was familiar? To what extent did you think there were problems that had to be solved before you could get what you wanted?
12.07
Perceived obstacle
11.57
Was a pleasant experience when you were using the prod- uct/service? Please also state the reason(s) of why was it a pleas- ant experience when you were using the product/ervice.
Yes bevzuse is was very pleusant: every detail the experience was expently curated and could note meaning fault in any of it. The food exquiste, the views were incredible.
Yes because I had a good time with friends, and it vas pleasant din- ing experience as well as just good time with friends.
Pleasantness
To what extent did you think that the situation was pleasant?
0.71
Table 6: The cognitive appraisal dimensions measured in this dataset. The questionnaire is sourced from prior work (Scherer et al., 1984; Smith and Ellsworth, 1985; Tong and Jia, 2017). The third column contains values for the percentage of NA responses for each appraisal obtained in the main study.
the comfortable the ameraties lasurious. (Joy)
Table 8: Examples of review text responses for the ques- tion format conditions (condition 2-presence of emotion prompts, condition 4- absence of emotion prompts). The general emotion that the participants felt when using the product is in parentheses for condition 1.
We present some sample responses of the ques- tionnaire and review formats in Table 8 and Table 9, respectively. In general, the questionnaire format

---

AD
12
Figure 5: Correlation between selected LIWC-22 dimensions and the appraisal dimensions in the current study. The left and right panel represent the correlations for the responses of the questionnaire and review formats, respectively. The vertical axis labels are the LIWC-22 dimensions, while the horizontal axis labels are the appraisal dimensions our study. "discrep" represents "discrepancy." and "socref" represents "social referents." " The full description of the categories, their definitions, and contents available in the LIWC 2022 manual.".
Emotion
Condition
Condition 3
relevance. The full description of the categories, their definitions, and contents are available in the LIWC 2022 manual.
Juy
I Loyota gaz mileage is top notch. wometmess travel to and from work for two weeks before I have Bl When I do fill up doeun't brook the bank. bought this product hoping to better my self tape procedhures, since I am an actress. Howerver the product
The
The previded excep- tional. We were treated and premises were pristine. Would do again!
Anger
were lucky enough seated right away because of
B Supplementary Information for Main Study
eaacelation. The service was ex-
defective from diffoalt
collent and the food was lughy ong food breshestbest in owhide. The food had an up- state foel without the uprcale
very heginning wok with. The entire product bioke a feo weeks oftex.
We collected an optional demographics question- naire at the end of the main study. Table 10 presents the distribution of participants across different de- mographic variables. Do note that we only sampled participants from the United States to reduce the heterogeneity of the responses collected.
Table 9: Examples of review text responses for the review format conditions (condition 1-presence of emo- tion prompts, condition 3- absence of emotion prompts). The self- self-reported general emotion that the participants felt when using the product is in column I.
Figure 6 shows the correlation between a subset of all the measured variables in our study. From these correlations, we observe that different spe- cific emotions have different relationships to dif- ferent appraisals and positive emotions positively correlate with intention to recommend to others and purchase again.
Figure 5 presents the correlations between se- lected LIWC22 dimensions and the 20 appraisal dimensions for the questionnaire and review exam- ined in the current study. Although there are no validated appraisal lexicon in the literature, certain LIWC-22 dimensions might be useful in indicating some of the cognitive appraisals. For instance, the dimensions of certitude and insight are indicative the appraisal dimension certainty and the corre- lation of these LIWC22 dimensions and certainty are greater for the questionnaire compared to the review format. The dimensions of causation, so- cial, and socrefs might be indicative of cognitive appraisals of accountability and control which the questionnaire format has in general a high corre- lation between these dimensions compared to the review format. Also, the questionnaire format gen- erally has high correlations between dimensions of need, and want with the cognitive appraisal of goal
We present the means of the appraisals with spect to the eight emotions in Table 11. Most of the relationships are generally consistent with previous theories (Yeo and Ong. 2023). For example, anger is associated with high scores on accountability- other, perceived obstacle, pleasantness, and low scores on external normative significance, faimmes, goal conduciveness. Pride is associated with high scores on accountability-seff, certainty, control- self. goal conduciveness, pleasaniness. However there are some appraisal-emotion relationships that are not consistent. For example, surprise hy- pothesized to be associated with high attentional activity but in our study we found that it has a low
Availaabl at http://tinarr..oom///c2222222.a

---

700
Gender
female: 42.3%
male: 54.9%
Non-binary/third gender: 2.3%
no respouse: 0.6%
Age group
18-24 years: 9.1%
25-34 years: 29.6%
35-44 years: 22.3%
45-54 years: 15.1%
55-64 years: 14.9%
64 years: 9.1%
Race
White or Caucasian: 63.3%
Black or African American: 22.6%
American Indian/Native American: 2.6%
Asian: 13.3%
Others: 2.0%
no response: 1.0% Yes: 12.0
Hispanic/Latinn origin
No: 86.6%
no response: 1.4%
Highest education level
High school diploma or less: 12.7%
Technical degree: 10.3%
Some college but degree: 16.9%
Bachelor Degree: 37.9%
Graduate degree: 22.0%
response: 0.3%
Annual household income
less than $25,000: 13.7%
155.000-$499 999: 22.4%
$50,000 (0000-$79,9999 18.0%
$75,000-$994 99909999999999999999999999999999 14.1%
$100,000-$1 18.0%
$150,000 %11.0
response: 2.7%
Table 10: Demographic statistics of participants in the main study.
rating. Gratitude is hypothesized to be associated with high accountabilit-ooher but found here that it has low rating.
balanced except for accumaabilttyyirrunstteee control-circumstancee difficulty, expectedness, and novelty, which have very little instances in the "high" class (less than 10%).
Figure shows example of how specific emo- tions (e.g. joy and disappointment) have different appraisal profiles. The emotion of joy is associated with high pleasantness, goal conduciveness, and moderate expectedness, while disappointment is a negative emotion that is low in pleasantness, goal conduciveness, and expectedness.
Table 14 shows the three-way prediction results of the other 16 appraisal dimensions. We can ob- serve that certain appraisals such as accountability- self, control-circumstancess. control-other, control- self, difficulty, external normative significance, and fairness have prediction results that have least one model predicting above an accuracy of 60%. This suggests that these mentioned appraisals ex- hibit certain systematic linguistic features that mod- els readily pick up and use it for predictions, com- pared to other appraisals. The appraisal with the lowest accuracy is accountability-otee which is surprising because expected that the models are able to pick up linguistic features of attribution and responsibility of the situation, nouns and pronouns (e.g. waiter, service staff, they) which might be indicative of this appraisal.
Table 12 presents the performance accuracies and Fl FI-scores for individual emotions concerning the different models. Across all the models, the ac- curacies for the emotions of disappointment, grat- itude, and joy were the highest. This is expected these emotions are the most dominant emotions endorsed by the participants in our dataset. The models perform poorly in predicting the least en- dorsed emotions such as disgust and surprise as there are not many instances of such emotions.
Table 13 shows the distribution of the cogni- tive appraisals terms of low, medium, and high used in the appraisal prediction task. general, most the distributions across the three classes are

---

Appraisal
Anger 3,28
Disappointment
Disgust 2.92
Gratitude
Joy
Pride
Regret 3,07
Surprise
Accountability-
3.28
2.66
2.74
3.02
2.80
circumstances
Accountability-onm
5,43
4.53
4.89
3.95
3.64
3.07
3.79
4,48
Accountability-cee
2.83
2.72
3.36
5.24
5.27
5.73
4.31
3.83
Attentional actvitiy
5.14
4.46
4.65
2.73
2.72
3.23
4.12
3.60
Certainty
4.52
4.48
4.43
6.51
6.44
6.08
4,96
5.87
Control-circumstance
2.89
3.19
3,46
2.74
2.74
3.11
3.08
3.24
Control-other
4.95
3.89
4.72
3.02
2.93
2.85
3,45
4.10
Control-self
3.15
3.03
3.64
5.82
5.65
6.09
4.19
4.38
Coping potential
4.18
4.39
4.22
5.94
5,70
5.38
4.50
4.67
Difficult
5.12
4.45
5.04
1.81
1.92
2.39
4.27
2.55
Effort
5.45
4.54
4.88
2.92
2.81
3.45
4.53
3.32
Expectedness
3.09
3.06
3.26
3.87
3.61
4.21
3.23
3.18
External normative signifi-
3.43
3.54
3.36
5.42
5.71
5.50
4.06
4.43
Fairness
2,49
2.66
3.36
6.27
6.22
Future expectancy
3.81
3.58
3.62
5.39
5.33
5.50
3.57
4,95
Goal conduciveness
2,43
3.07
6.46
6.34
3.37
Goal relevance
2.86
3,64
3.89
Novelty
3.06
3,22
Perceived obstacle Pleasantness
5.35
4.41
2.76
4.43
3.18
2.46
3.18
6.46
6.46
3.49
5.13
Table I1: Means of the appraisal dimensions with respect to different emotions. These appraisals are rated on
7 7-point Likert scale.
BERT
XLNet
DistillRoBERTa
Emotion
Acc
F1
Acc
F1
Acc
Fi
Anger
20.2
0.33
33.3
0.33
33.3
0.38
Disappointment
92.0
0.78
90.0
0.83
84.0
0.79
Disgust
0.00
0.00
0.00
0.00
14.3
0.25
Gratitude
80.0
0.67
80.0
0.69
80.0
0.71
Joy
91.3
0.76
87.0
0.74
78.3
0.75
Pride
25.0
0.40
50.0
0.67
62.5
0.56
Regret
54.5
0.67
50.0
0.63
54.5
0.59
Surprise
0.00
0.00
33.3
0.40
50.0
0.67
Table 12: Accuracy and Fl scores for each emotion class of each model.

---

anansoon actiity
Rium. eipectoncy
palcaiee obstacte
QO cons
contro
control-
goal
cinducleeness
0.8
fermess
coplng potersal
difccityy
0.4
attentional activity
future expeciancy
effort
perceived obetacd
external
accountab liy-
research
-04
again
disappaintmmnt
-0.6
diogust
pri50
-08
Accountability
Appraisal
59.4
(%)
Medium (%)
34.4
High (%)
6.2
circumstances
Accomiahility-ohee
39.8
40.4
19.8
Figure 6: Correlation matrix of subset of vari- ables in dataset. "control-s" refers to "control- self", "accountability-o" refers to "accountability- other", "accoumtability-s" refers to "accountabillty-sel""" "control-c" refers to "controlcirruusstanees"" "exter- nal normative" refers to "external normative signif- icance", "accountability-e" refers to "accountability- circumstances", "research" refers whether the partici- pant put in effort in researching for the product/service before purchasing. "again" refers to whether the partici- pant will purchase the product/service again.
Accountability-sei
40.4
Attentional activity
43.4
42,6
14,0
Certainly
16.3
42.0
41.6
contmol-cincumstannees
59.8
Control-other
50.7
38.2
11.1
Control-self
34.4
45.1
20.5
Coping potential
22.8
Difficulty
52.0
39.4
Effort
40.8
45.9
13.3
Expectedness
50.6
External normative sig-
26.3
53.8
19.9
nificance
Faireess
37.0
35.6
27.4
Future expectancy
26.3
Goal conduciveness
40.4
29.9
29.7
Goal relevance
33.3
32.6
34.3
Novelty
48.4
Perceived obstacle Pleasantness
42.7
41.6
15.7
37.2
32.8
30.0
Table 13: Distribution of cognitive appraisals in terms of low, medium, and high used the appraisal prediction task.
Figure 7: Comparing different appraisal ratings the emotion of jjy"" and 'disappointment.'

---

BERT
XLNet
DistilRooERTT
OpenAl ada
Appraisal
Acc
F1
Acc
F1
Acc
F1
Acc
F1
Accountability-ciruussstt
cireumstances
50.4
0.45
47.0
44.5
53.0
0.48
57.7
0.55
Accountability-otte
45.2
0.43
46.8
0.46
35.5
0.35
44.9
0.44
Accountability-se
53.7
0.54
50.0
0.50
56.0
0.56
61.3
0.61
Attentional activity
48.3
0.44
45.0
0.42
51.7
0.51
56.5
0.54
Certainty
55.6
0.55
56.4
0.56
52.6
0.52
50.4
0.50
Control-circumstances
64.5
0.58
65.3
0.52
63.6
0.61
55.0
0.54
Control-other
62.0
54.5
0.55
52.9
0.54
55.0
0.53
Control-self
61.6
0.60
59.4
0.58
60.1
0.59
65.0
0.65
Coping potential
$1.7
45.8
44.1
0.43
57.7
0.55
Difficulty
67.1
64.9
70.2
0.69
64.8
0.62
Eiffort
54.8
56.4
58.9
0.57
55.6
0.53
External normative significance
64.7
59.7
63.0
0.62
53.6
0.54
Faireess
66.7
68.2
64.4
0.64
Future expectancy
58.2
59.0
0.55
Novelty
57.3
56.5
56.5
0.53
Perceived obstacle
58.9
0.54
54.0
0.52
48.4
0.57
52.8
0.53
Table 14: Results of three-way (high, medium, low) appraisal classification for the other 16 appraisal dimensions.


Reviews:
[{"review_id": "SaBob5IuFh", "reviewer": "Reviewer_5opM", "summary": "", "questions": "1. Lines 67-81 and lines 490-492: Why do you include the decision-making and person-level traits data if it’s not discussed in this paper? Can you talk about the decision-making and person-level traits data that you collected in more detail?\n\n2. On lines 314-316, you mentioned that participants might feel multiple emotions during a consumption episode. However, in both the main study and the baseline modeling, you focus on the dominant emotion only. Why don’t you take into account multiple emotions in your dataset?\n\n3. The discussion of the analyses from the pilot study is unclear and ambiguous. Can you discuss the results using LIWC22 in Sec 5.3 in more detail? The results from Figure 2 are barely discussed. For the main study, the authors mentioned that they chose the [questionnaire format] as the writing prompts based on results from the pilot study. But in Figure 2, the [questionnaire format] only surpasses the [review format] in terms of the expressions of “cognition”. For the expressions of “affect”, I don’t see significant differences in distribution between the two formats. In fact, for expressions of psychological states such as “fulfilled”, “acquire”, and “want”, the [review format] actually often expresses much more psychological states than the [questionnaire format], as manifested in Figure 2. Can the authors provide more explanations why the [questionnaire format] is better than the [review format] based on the results in Figure 2?\n\n4. Lines 407-413: Can you explain why there were no significant differences between conditions with or without emotion prompts? Because based on Figure 2, it looks like there is a large difference between the [Review] and [Emotion + Review] formats, especially in terms of the expressions of psychological states.\n\n5. Section 7.3: Why don’t you include a simpler baseline such as the logistic regression model, so that the performance of the transformer models can be more clearly compared to?\n\n6. Section 7.3: In addition, can you provide more explanations and details of the results of the emotion prediction experiments? For example, why is the performance of the models decent just because they are base models? What results should we expect to observe on more advanced models? Also, what is the accuracy and F1 score for each emotion? I am curious whether the models are more capable in predicting “joy” and “disappointment”, since they are the largest dominant positive and negative emotions felt on your domain (lines 518-520). Can you provide more details on the discussion of these?\n\n7. Section 7: Can you explain the appraisal prediction task more clearly? It seems like you’re only trying to predict the appraisals “pleasantness”, “goal conduciveness”, and “expectedness” only, as shown in Table 4? Why do you pick these 3 dimensions and not include the other 17?\n\n8. Section 7.3: Similar to the discussion of results for emotion prediction, a more detailed discussion of results for appraisal prediction is needed.\n\n9. In conclusion you mentioned that you only included 4 appraisal questions (goal relevance, goal conducive-ness, expectedness, and pleasantness) to elicit text responses about appraisals. Did you mention that anywhere in the experiments? The design of the modelling is very unclear.", "limitations": "", "rating": 4, "confidence": 5, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful for your insightful feedback and would be happy to address your suggestions in the camera-ready version of the paper. As we say in our \"Summary of Rebuttal\" comment, the research design and the dataset collection are the main contributions of the paper. We plan to explore directions in predictive language and behavioral modeling in subsequent projects, while also opening this dataset up for other researchers to use. Many of your concerns have already been addressed in the paper and the appendix. \n\n1. As mentioned in Table 4, lines 140-143, 166-169, and 604-606, we have clarified our focus on four appraisal dimensions for most of the analysis: goal relevance, goal conduciveness, pleasantness, and expectedness. In lines 655-657, we motivate our choice of appraisal dimensions based on the context of consumption; we say that “we only included four appraisal questions (goal relevance, goal conduciveness, expectedness, and pleasantness) to elicit text responses about appraisals. Nevertheless, we selected these four appraisals because they are most relevant to the context of consumption (Desmet, 2008) and we feel that introducing more appraisal questions for participants to complete could result in poor quality responses due to fatigue.” These appraisal dimensions are not new; instead, they are based on decades of literature on cognitive appraisal and consumption. In the camera-ready version, we will add more details and offer a lengthier motivation for our choices. Other papers that motivate and describe our choice of appraisal dimensions are Demir et al., 2009; Desmet, 2008; Desmet & Hekkert, 2007. We apologize for leaving out the meta-analysis the reviewer has provided and can include it in the camera-ready version. We regret the reviewer’s perceived brevity in our rationale.\n\n2. Answers to the questions: \n\n-1. Given that EMNLP focuses on computational linguistics, we’ve not included the person-level variables in this version of the dataset. For the sake of transparency, we still mention that this was collected. We’re exploring these variables for a subsequent project on the cognitive antecedents and linguistic predictors of decision-making, targeting a psychological journal. Thank you for your interest!\n\n-2. As mentioned in lines 431-433 and line 514, we collected information about eight emotions from the respondent, while we did ask what the “dominant” emotion was. This information is used to collect information regarding cognitive appraisal, thereby allowing us to relate the cognitive antecedents of the dominant emotion to their consequence. We could not have asked these questions for each of the emotions the respondent experienced to be mindful of respondent fatigue.\n\n-3. Thank you for your keen observation and great question! These would be important directions to investigate in a follow-up study, as they reflect findings from preliminary data, based on 18 responses. But in general, data inspection revealed that in the review format, the emotion scores may reflect authors’ subjective assessment of the product, rather than their own emotional experience. For instance “bad food” is a different, and less useful text for the study of appraisal, as compared to “horribly upset.” We decided to focus our paper on the dataset collection and save these interesting explorations for future work.\n\n-4. For lines 407-413, we were referring to the comparison between [Questionnaire] and [Emotion + Questionnaire] format. We have conducted Wilcoxon signed-rank tests and found that there were no significant differences across LIWC categories except for ‘cognition’ in which the [Emotion + Questionnaire] format yielded greater cognition words. The reason would be because of the high variance in the measurements (based on 18 reviews). We will state more clearly in the camera-ready version of the paper.\n\n-5. Thank you for raising this suggestion. We’d like to highlight that while our main contribution is the research design and dataset collection, we went the extra mile to add more sophisticated deep-learning models. We hope the reviewers are satisfied with our efforts.\n\n-6. Thank you for your suggestion. As we noted on line 599, model performances on our dataset are consistent. A comprehensive set of emotion prediction experiments was beyond the scope of our research project because our main contribution is the research design and the dataset collection. Our emotion prediction experiments aim to establish baseline evaluations with state-of-the-art models on this novel dataset. Given its unique emotion classes, direct comparisons with other datasets are challenging. However, a close comparison is the ISEAR dataset (Scherer, & Wallbott, 1994), where XLNet achieved a 73.0% accuracy in seven-way emotion classification (Adoma et al., 2020). Notably, our dataset has imbalanced emotion classes, with joy and disappointment dominating. Using XLNet, we recorded accuracies of 95.7% and 92.0% for these emotions, respectively. As anticipated, the models excel in predicting these emotions but occasionally misclassify them into other positive or negative categories. This suggests models capture valence and pleasantness but struggle to differentiate specific emotions within a valence. We would be happy to clarify this in the camera-ready version.\n\n-7. Answers 7 and 9: We appreciate the feedback. We focused on four appraisal questions: 1. goal relevance, 2. goal conduciveness, 3. expectedness, and 4. pleasantness, backed by prior research (Demir et al., 2009; Desmet, 2008; Desmet & Hekkert, 2007). We also gathered responses to two other questions (see Table 8) and ratings for all 20 appraisal dimensions. We apologize for leaving out the meta-analysis the reviewer has provided and can include it in the camera-ready version. For classification, we prioritized dimensions with text responses. In the update, we'll expand the classification tasks and provide a distribution of class labels.\n\n-8. Answer 8: Our appraisal prediction experiments aim to establish baselines using top-tier models. Our choice of appraisal dimensions was based on prior work as cited above. We sincerely apologize for missing the reference of the meta-analysis the reviewer has provided and will include it in the camera-ready version. Furthermore, pleasantness and goal congruence predictions are the most accurate, likely due to their close alignment with positive/negative valence words.\n\nFinally, we sincerely apologize for the typos that remain in the paper and would be glad to address these in the camera-ready version. Thank you for your help in helping us to clarify the impact of our paper.\n\nReferences\n\nAdoma, A. F., Henry, N. M., & Chen, W. (2020, December). Comparative analyses of bert, roberta, distilbert, and xlnet for text-based emotion recognition. In 2020 17th International Computer Conference on Wavelet Active Media Technology and Information Processing (ICCWAMTIP) (pp. 117-121). IEEE.\n\nDesmet, P., & Hekkert, P. (2007). Framework of product experience. International journal of design, 1(1), 57-66.\n\nDesmet, P. M. (2008). Product emotion. In Product experience(pp. 379-397). Elsevier.\n\nDemir, E., Desmet, P. M., & Hekkert, P. (2009). Appraisal patterns of emotions in human-product interaction. International journal of design, 3(2).\n\nScherer, K. R., & Wallbott, H. G. (1994). Evidence for universality and cultural variation of differential emotion response patterning. Journal of personality and social psychology, 66(2), 310.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. This work presents PEACE-Reviews, which offers review texts annotated with rich first-person emotional variables such as cognitive appraisal dimensions and emotional intensity. The first-person perspective autobiographical writing in the dataset fills the gap of prior work which focuses on emotions and appraisals from a 3rd-person perceived perspective.\n\n2. Prior to collecting the PEACE-Reviews dataset, a pilot study was conducted to scrutinize which kinds of writing prompts elicit high-quality text of emotions and cognitive appraisals. After inspecting the results from the pilot study, the authors subsequently adopted the framework of  [Emotion Prompts] + [Questionnaire Format] to elicit first-person reviews from participants.\n\n3. The data collection process is well-designed: 1) The authors included a screening question before collecting the main dataset that serves as a proxy for product/service that are personally important to the participants. This increases the likelihood of participants having emotion experiences when using the product/service. 2) Lines 451-461: both positive and negative experiences from the same participant are recorded. This better accounts for individual differences in the dataset. 3) There are 700 participants in total that contribute to the PEACE-Reviews dataset, which is a good indicator of the diversity involved in this dataset.", "reasons_to_reject": "1. The authors failed to explain many critical aspects of the study. For example, where do the 20 appraisal dimensions come from? Are they extracted from prior studies or are they newly proposed in this specific work? The authors did not successfully convince me of the theoretical grounding for the appraisal dimensions they chose as they didn’t even discuss the origins of the 20 appraisal dimensions they included in this study.\n\n2. The discussion of the analyses and experimental results from the study remains unclear and ambiguous. Please refer to the “Questions for the Authors” section for my questions.\n\n3. The writing of this paper could be improved to a great extent, both in terms of the wording as well as the formulation of paragraphs. See the “Typos Grammar Style and Presentation Improvements” section for suggestions for improvements.", "questions_to_authors": "1. Lines 67-81 and lines 490-492: Why do you include the decision-making and person-level traits data if it’s not discussed in this paper? Can you talk about the decision-making and person-level traits data that you collected in more detail?\n\n2. On lines 314-316, you mentioned that participants might feel multiple emotions during a consumption episode. However, in both the main study and the baseline modeling, you focus on the dominant emotion only. Why don’t you take into account multiple emotions in your dataset?\n\n3. The discussion of the analyses from the pilot study is unclear and ambiguous. Can you discuss the results using LIWC22 in Sec 5.3 in more detail? The results from Figure 2 are barely discussed. For the main study, the authors mentioned that they chose the [questionnaire format] as the writing prompts based on results from the pilot study. But in Figure 2, the [questionnaire format] only surpasses the [review format] in terms of the expressions of “cognition”. For the expressions of “affect”, I don’t see significant differences in distribution between the two formats. In fact, for expressions of psychological states such as “fulfilled”, “acquire”, and “want”, the [review format] actually often expresses much more psychological states than the [questionnaire format], as manifested in Figure 2. Can the authors provide more explanations why the [questionnaire format] is better than the [review format] based on the results in Figure 2?\n\n4. Lines 407-413: Can you explain why there were no significant differences between conditions with or without emotion prompts? Because based on Figure 2, it looks like there is a large difference between the [Review] and [Emotion + Review] formats, especially in terms of the expressions of psychological states.\n\n5. Section 7.3: Why don’t you include a simpler baseline such as the logistic regression model, so that the performance of the transformer models can be more clearly compared to?\n\n6. Section 7.3: In addition, can you provide more explanations and details of the results of the emotion prediction experiments? For example, why is the performance of the models decent just because they are base models? What results should we expect to observe on more advanced models? Also, what is the accuracy and F1 score for each emotion? I am curious whether the models are more capable in predicting “joy” and “disappointment”, since they are the largest dominant positive and negative emotions felt on your domain (lines 518-520). Can you provide more details on the discussion of these?\n\n7. Section 7: Can you explain the appraisal prediction task more clearly? It seems like you’re only trying to predict the appraisals “pleasantness”, “goal conduciveness”, and “expectedness” only, as shown in Table 4? Why do you pick these 3 dimensions and not include the other 17?\n\n8. Section 7.3: Similar to the discussion of results for emotion prediction, a more detailed discussion of results for appraisal prediction is needed.\n\n9. In conclusion you mentioned that you only included 4 appraisal questions (goal relevance, goal conducive-ness, expectedness, and pleasantness) to elicit text responses about appraisals. Did you mention that anywhere in the experiments? The design of the modelling is very unclear.", "paper_topic_and_main_contributions": "This paper revolves around emotion appraisals, i.e., how people evaluate their situations that results in specific emotional experiences. A set of 20 cognitive appraisal dimensions were examined. Using a pilot study, the authors were able to settle on a framework in the format of both the presence of emotion prompts and questionnaires to elicit high-quality first-person narratives that are rich in both emotional and cognitive appraisals. Using this framework, the authors then created the PEACE-Reviews dataset, which consists of annotated autobiographical writing where participants describe their appraisals and emotions during the usage of products or services of personal significance. Subsequent analyses and experiments were performed on the dataset.\n\nThe paper has the following main contributions: 1) This work identifies and proposes a useful framework for eliciting emotions together with the appraisals behind the emotions through writing prompts. 2) The authors propose the PEACE-Reviews dataset, which is the first to present emotion and cognition along multiple dimensions along with person-level trait and demographic information. 3) The authors present baseline models on identifying emotions and cognitive appraisals from autobiographical writing using language models.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "“A meta-analytic review of the associations between cognitive appraisals and emotions in cognitive appraisal theory” (Yeo and Ong, 2023). The appraisal dimensions used in this study are *all* found in Yeo and Ong’s work, but not mentioned at all here.", "Typos_Grammar_Style_and_Presentation_Improvements": "1. The writing of this paper could be improved to a great extent, both in terms of the wording as well as the formulation of paragraphs.\n\n2. Line 029: delete “also”.\n\n3. Lines 043 and 058: a dash is needed instead of a hyphen.\n\n4. Lines 67-81: The connection between the 2nd and 3rd paragraph of Introduction is loose: the authors could do a better job at linking [the explanation of what emotion appraisal is] to [how emotions influence subsequent decisions]. In addition, the mention of the decision-making processes is not relevant to the paper as it’s not discussed in the data collection and the experiments.\n\n5. Line 121: emotion.\n\n6. Line 384: comprises *of*.\n\n7. Lines 544-547 is misleading and not appropriate as a paragraph starting sentence. What does “Although we curated the dataset with 1,400 autobiographical writings annotated numerous variables, we will focus on modeling emotions and appraisals here” mean?\n\n8. Line 602: the authors forgot to delete the manuscript, leaving a note “(need to explain more?)."}}, {"review_id": "8kpKFwPbzs", "reviewer": "Reviewer_rMKV", "summary": "", "questions": "Did you run experiments with linear models for the classification of emotions and appraisal dimensions? This could be interesting to analyze feature importances of the model, which would provide potential insights into which words drive classification performance.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable and constructive feedback. As we say in our \"Summary of Rebuttal\" comment, the research design and the dataset collection are the main contributions of the paper. The choice to report only simple deep-learning baselines was deliberate so that we could focus on our unique contribution. We plan to explore directions in predictive language and behavioral modeling in subsequent projects, while also opening this dataset up for other researchers to use.\n\n1.\tThe NA option was included in the response options provided to the participants so that we did not accidentally encode the absence/irrelevance of an appraisal to be the same as a low-rated appraisal. \n\n2.\tSelection of appraisal dimensions: The choice of these appraisal dimensions is supported by previous studies that show certain appraisals that are particularly relevant to the context of consumption (Demir et al., 2009; Desmet, 2008; Desmet & Hekkert, 2007). Nevertheless, we also added two questions that we feel might elicit other appraisals (see Table 8- a) How did you feel when you used the product/service?, and b) What about the product/service that made you feel this way?).\n\n3.\tA comprehensive set of emotion prediction experiments was beyond the scope of our research project because our main contribution was the research design and the dataset collection. Our choice of appraisal dimensions was based on prior work. Furthermore, we expect that pleasantness and goal congruence predictions are the most accurate likely due to their close alignment with positive/negative valence words.\n\n4.\tRegarding label distribution: Sure, we would be happy to add the distribution of class labels. They are already in the online repo at https://anonymous.4open.science/r/PEACE-Reviews-1DCF\n\n5.\tRegarding further analysis: as mentioned above, we wanted to focus our paper on our major contribution - the research design and data collection of reviews annotated by participants. We are working on subsequent projects for further analyses such as topic modeling and using validated lexicons (e.g. LIWC) to identify what words and tokens are indicative of emotions and appraisals so as to better understand the emotional experience of reviews.\n\nReferences\n\nDesmet, P., & Hekkert, P. (2007). Framework of product experience. International journal of design, 1(1), 57-66.\n\nDesmet, P. M. (2008). Product emotion. In Product experience(pp. 379-397). Elsevier.\n\nDemir, E., Desmet, P. M., & Hekkert, P. (2009). Appraisal patterns of emotions in human-product interaction. International journal of design, 3(2).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* The paper presents a novel dataset, with product/service reviews annotated according to first-person emotional and appraisal states, which represents the first of its kind in this context.\n* The paper provides an extensive review of the literature to give context to the theory of cognitive appraisal.\n* The dataset creation process is well-documented. The authors furthermore conducted pilot studies to identify to optimize their collection process, which speaks to the quality of the dataset.\n", "reasons_to_reject": "* The selection of appraisal dimensions used for classification seems unclear. In lines 501-503, it is mentioned that goal conduciveness and pleasantness have the lowest number of NA responses. In Section 7.4, results are shown for a 3-class classification task for goal conduciveness, pleasantness, and expectedness. In the Limitations Section, it is mentioned that only four appraisal questions are used to elicit responses. However, Table 6 shows many more dimensions and the caption states that all these dimensions were measured in the dataset. Apologies if I am confusing this, but can you clarify and elaborate on how you selected those 3 (4?) categories and why you included them for the classification task?\n* Related to the above, the distribution of class labels for the appraisal task (low-medium-high) does not seem to be shown in the paper. I would recommend adding this since it has an impact on how to interpret performance scores in Table 4.\n* The analysis of linguistic features regarding the dataset could be more extensive. While the collection process is described in great detail, the data analysis is rather thin and could be addressed further (e.g., by providing results on word and n-gram occurrences across emotions and appraisal dimensions, as well as topic modeling). This would serve to provide the reader with a better understanding of how individual reviews are composed, and what words are indicative of individual emotional states and appraisal dimensions.\n", "questions_to_authors": "Did you run experiments with linear models for the classification of emotions and appraisal dimensions? This could be interesting to analyze feature importances of the model, which would provide potential insights into which words drive classification performance.", "paper_topic_and_main_contributions": "This paper presents the Perception and Experience of Appraisals and Consumption Emotions in Reviews (PEACE) dataset, a collection of reviews annotated with first-person emotional variables, intended to be used to measure emotional states and cognitive appraisal dimensions in product and service reviews. This is the first dataset of such kind since existing datasets are only annotated according to third-person appraisal (and therefore model perception rather than experience). \nTo construct the dataset, the authors first ran a pilot study to identify the most effective writing prompt to use in the main study. For the main data collection, 700 participants were then recruited via Prolific and asked to rate their emotions (among an option pool of eight emotions) and appraisal (from a set of appraisal dimensions) experienced when using an expensive product or service that they recently purchased. The resulting dataset contains a total of 1400 examples (each participant provided two responses, one positive and one negative). The results show that joy and disappointment are the most dominant emotions selected by participants. \nThe paper then presents baseline experiments for predicting emotions and appraisal dimensions from text. To do so, they cast the problem as an 8-class classification task. Additionally, the authors aim to predict each of the appraisal dimensions as a 3-class classification task (for the appraisal dimensions goal conduciveness, pleasantness, and expectedness). Baseline models include DistillRoBERTa, BERT, XLNet, and GPT-3. Performance metrics are reported in terms of accuracy and F1, showing that GPT-3 provides the best results in terms of F1 for emotion classification. For appraisal classification, expectedness is hardest to predict, and performances are higher for the other two categories. The different models perform similarly across comparisons.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "None"}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "* Line 218: “third-person” appearing twice\n* Line 333: Lower case start of sentence\n* Line 524: typo “is are”\n* Line 534: Add whitespace after “Figure 3”\n* Line 602: There seems to be a comment that hasn’t been removed prior to submission\n"}}, {"review_id": "Id0fVDlO0w", "reviewer": "Reviewer_Miqh", "summary": "", "questions": "A. The term “expensive” can be somewhat ambiguous. One might assume that people tend to be more critical of pricier products or services and more easily pleased with economical ones. \n\nB. Many individuals, if not the majority, tend to be more introspective and less emotionally charged when reflecting on past experiences, having had time to process. Have you considered this factor? Furthermore, the recency of an experience can influence recall and emotion. Did you set guidelines regarding how recent these experiences should be?\n\nC. Is there a quantitative analysis supporting the conclusions/observations made in section 5.2?\n\nD. While it seems intuitive that a questionnaire format might yield more detailed results due to its structured nature, what is the core discovery here? Is it that structured guidance elicits richer responses? If the concern is that review format responses tend to emphasize product specifics (as noted in lines 361-362), you could potentially direct participants to concentrate on their emotional reactions and thought processes rather than product details.\n\nE. Did you measure any time discrepancies between completing a questionnaire-based survey versus a review-based one?\n\nF. How do you differentiate between functional words (e.g., the word “get” in sentences like \"This gets better\") and genuine psychological state lexicons (e.g., \"I got this product\")?\n\nG. How do you ensure authenticity in the responses? Is there a mechanism to detect fabricated stories?\n\nH. Are your dataset's statistics consistent with the proposed emotion-appraisal profiles? Specifically, do you identify which appraisal dimensions rank high or low for particular emotions, and how these compare to existing theories?\n\nI. Is there a theoretical or quantitative basis for selecting the four appraisal questions to elicit text responses about appraisals? At a glance, I perceive that factors like customer service quality, associated with attributes such as “other-accountability” and “difficulty”, might also influence emotions. \n\nJ. In appraisal prediction experiments, why were only the appraisal dimensions of “pleasantness”, “goal conduciveness”, and “expectedness” considered? In particular, why was “goal relevance” excluded, especially since it was one of the prompt queries?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are extremely grateful for your valuable feedback. We'll be sure to increase the size of our tables and figures in the camera-ready version of the paper. As we mention in the \"Summary of Rebuttal\" comment, the research design and the dataset collection are the main contributions of the paper. The choice to report only simple predictive deep-learning baselines on the final dataset was deliberate so that we could focus on our unique contribution. We plan to explore directions in predictive language and behavioral modeling in subsequent projects, while also opening this dataset up for other researchers to use.\n\nAnswer A: Thank you for the great observation! Anticipating these differences, we also asked the respondents how much they spent on the particular product so that we could control for these factors in subsequent analyses. This information is included in the dataset. Nevertheless, despite these differences, we were motivated by our goal of curating a dataset that is rich in emotional expressions. According to Richins (2008), this class of products has the highest likelihood of experiencing an emotion and also results in higher emotional intensity while using it as its usage represents a situation that the participants think deeply about before purchasing and is personally important to them. Compared to someone using a mundane or economical product such as a pen or a toothbrush, expensive products (regardless of whether the participant is critical of it or not, i.e. think it is not worth it or regret buying it) are able to increase the likelihood of experiencing an emotion and the intensity of felt emotions. \n\nAnswer B: The design of the data collection closely followed the methodological frameworks of previous studies that investigate appraisals and emotions in Psychology (Scherer, 1997; Smith & Ellsworth, 1985; Tong, 2015). We tried our best to facilitate recall by including prompts such as ‘How did you feel when you used the product/service?’ and ‘What about the product that caused you to feel the emotion stated?’. Moreover, prior to the main study, we also conducted screening studies to ensure that participants were able to recall an emotional episode of using an expensive product/service. Similar to Answer A, we feel that the context of using an expensive product/service helps to facilitate recall and ensure that rich emotional experiences are collected. \n\nAnswer C: The conclusion and observations were based on a few quantitative analyses. Firstly, in terms of how detailed the responses were, we used a word count approach to analyze how detailed the text responses were. For example, the average word count for a text of the review format is 61.8 words whereas the average word count for a text of the questionnaire format is 150 words. Since we are interested in understanding appraisals from such text, a detailed account would be preferable as it has the potential to extract various appraisal dimensions from the text. Secondly, in terms of whether the appraisal ratings are consistent with the expressed responses, we coded for several appraisal dimensions using LIWC and correlated them with the corresponding appraisal ratings. We will include these findings in the revised paper. \n\nAnswer D: One of the main concerns when designing the data collection procedure is that the review format responses might emphasize product-specific details instead of emotional reactions even when we ask participants to write about the emotional aspect of it. Another primary concern is that the review format responses might not contain text pertaining to appraisal dimensions. Therefore, we carried out the pilot study to determine whether a questionnaire format might yield more detailed emotional responses or whether a review format is as emotionally detailed as the questionnaire format. From the results, we observed that indeed the structured nature of the questionnaire format elicits richer emotional responses that contain appraisal dimensions that could be used to analyze such text. \n\nAnswer E: Yes since we have data on how long each participant completed the tasks, we found that the review-based task took an average of 11.15 minutes while the questionnaire-based task took an average of 13.70 minutes to complete. \n\nAnswer F: We did not differentiate between functional and genuine psychological state lexicons. Since there are no appraisal lexicons in the literature, we used some of the psychological state lexicons from LIWC as a proxy for certain appraisal dimensions. For example, the appraisal of ‘goal conduciveness’ could be measured by a combination of various psychological states such as ‘get’, ‘obtain’, and ‘fulfill’. Therefore, the aggregation of various psychological states might mitigate the presence of functional words in the dataset. \n\nAnswer G: We could also ensure that the responses are written by real humans instead of bots due to the validation and vetting of participants in the Prolific platform. The Prolific platform is also recently working on various ways to identify participants who used artificial intelligence such as ChatGPT to respond to tasks. However, we did not implement any mechanism to detect fabricated stories by human participants. \n\nAnswer H: Due to space constraints and the scope of the paper, we only presented preliminary results on how the ratings of appraisals are related to the eight included emotions (see line 521-541). From the MANOVA results, we found that there are significant differences amongst the eight emotion groups based on the 20 appraisal ratings. However, post-hoc analyses to observe which appraisals rank high or low for particular emotions are not presented. We are planning to analyze the appraisal-emotion profiles in another separate paper. Nevertheless, we could present these results in the Appendix of the revised paper. \n\nAnswer I: We agree that other appraisal dimensions might be relevant in the context of consumption but due to the resources constraints and feasibility of data collection, we only collected text responses for four specific appraisal questions - 1. goal relevance, 2. goal conduciveness, 3. expectedness, and 4. pleasantness, instead of eliciting responses for all 20 appraisal dimensions. The choice of these appraisal dimensions to elicit text responses is supported by previous studies that show certain appraisals that are particularly relevant to the context of consumption (Demir et al., 2009; Desmet, 2008; Desmet & Hekkert, 2007). Nevertheless, we also added two questions that might elicit other appraisals (see Table 8- a) How did you feel when you used the product/service?, and b) What about the product/service that made you feel this way?). \n\nAnswer J: For the selection of appraisal dimensions used for the classification task, we only included the appraisal dimensions that have text responses that are elicited from the appraisal questions (i.e. goal relevance, goal conduciveness, expectedness, and pleasantness). Due to space constraints, we only presented goal conduciveness, expectedness, and pleasantness for the appraisal classification tasks. The primary purpose of the appraisal classification task is to demonstrate that the text and ratings are useful in measuring and understanding appraisals in consumption. In the revised version, we will include the classification of goal relevance and other appraisal dimensions for completeness.\n\nReferences\n\nDesmet, P., & Hekkert, P. (2007). Framework of product experience. International journal of design, 1(1), 57-66.\n\nDesmet, P. M. (2008). Product emotion. In Product experience (pp. 379-397). Elsevier. \n\nDemir, E., Desmet, P. M., & Hekkert, P. (2009). Appraisal patterns of emotions in human-product interaction. International journal of design, 3(2). Richins, M. L. (2008). Consumption emotions. In Product experience (pp. 399-422). Elsevier.\n\nScherer, K. (1997). Profiles of emotion-antecedent appraisal: Testing theoretical predictions across cultures. Cognition & Emotion, 11(2), 113-150. \n\nSmith, C. A., & Ellsworth, P. C. (1985). Patterns of cognitive appraisal in emotion. Journal of personality and social psychology, 48(4), 813. \n\nTong, E. M. (2015). Differentiation of 13 positive emotions by appraisals. Cognition and Emotion, 29(3), 484-503.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This study explores the nuanced connection between self-cognitive appraisal and its consequent effects on emotions and behaviors. By probing into the feasibility of extracting self-cognitive appraisal from autobiographical writing reviews, the author(s) contribute a unique and insightful perspective to the field. The proposed novel dataset could serve as a valuable asset for further research in this domain.", "reasons_to_reject": "The rationale for selecting the specific topic (expensive product/service purchase experience) requires further justification (refer to question A below). Additionally, the dataset seems to lack a quantitative analysis (see questions C and H). The baseline experiments appear preliminary. It would be beneficial to see experiments exploring if cognitive appraisal inference/modeling enhances emotion detection, and potentially, future customer behaviors, especially since data regarding intentions of future purchases and recommendations has been gathered.", "questions_to_authors": "A. The term “expensive” can be somewhat ambiguous. One might assume that people tend to be more critical of pricier products or services and more easily pleased with economical ones. \n\nB. Many individuals, if not the majority, tend to be more introspective and less emotionally charged when reflecting on past experiences, having had time to process. Have you considered this factor? Furthermore, the recency of an experience can influence recall and emotion. Did you set guidelines regarding how recent these experiences should be?\n\nC. Is there a quantitative analysis supporting the conclusions/observations made in section 5.2?\n\nD. While it seems intuitive that a questionnaire format might yield more detailed results due to its structured nature, what is the core discovery here? Is it that structured guidance elicits richer responses? If the concern is that review format responses tend to emphasize product specifics (as noted in lines 361-362), you could potentially direct participants to concentrate on their emotional reactions and thought processes rather than product details.\n\nE. Did you measure any time discrepancies between completing a questionnaire-based survey versus a review-based one?\n\nF. How do you differentiate between functional words (e.g., the word “get” in sentences like \"This gets better\") and genuine psychological state lexicons (e.g., \"I got this product\")?\n\nG. How do you ensure authenticity in the responses? Is there a mechanism to detect fabricated stories?\n\nH. Are your dataset's statistics consistent with the proposed emotion-appraisal profiles? Specifically, do you identify which appraisal dimensions rank high or low for particular emotions, and how these compare to existing theories?\n\nI. Is there a theoretical or quantitative basis for selecting the four appraisal questions to elicit text responses about appraisals? At a glance, I perceive that factors like customer service quality, associated with attributes such as “other-accountability” and “difficulty”, might also influence emotions. \n\nJ. In appraisal prediction experiments, why were only the appraisal dimensions of “pleasantness”, “goal conduciveness”, and “expectedness” considered? In particular, why was “goal relevance” excluded, especially since it was one of the prompt queries?", "paper_topic_and_main_contributions": "This study introduced a novel dataset, PEACE-Reviews, an autobiographical writing review collection, designed to tackle the limitations observed in currently available public datasets:\n\n1. Many existing datasets lack annotations for both emotions and cognitive appraisals, hindering the study of the psychological connection between them.\n2. Existing datasets often fall short in capturing a rich variety of emotional language and labels.\n3. Present datasets predominantly reflect a third-person's perception of the written text, rather than the original author's (first-person) appraisals.\n\nAdditionally, baseline experiments were conducted on this dataset to identify emotions and cognitive appraisals utilizing linguistic features.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "1. Line 397-398: misuse of period and comma\n2. Line 407, 408: extra comma\n3. Fonts in figures are too small and hard to read"}, "extra_text": {}}]

Author rebuttal:

Meta-review: Thank you to authors for their work and reviewers for their reviews. This paper builds a new data set in which participants describe their appraisals and emotions during the usage of products or services of personal significance.

Pros:
- Reviewers agree that the data set, which aims to capture self-reported cognitive appraisals and emotions is novel in design and captures dimensions not present in current data sets
- Reviewers agree that the data set collection is well-thought-out, with a pilot study to determine the best format and contextualization within prior work

Cons:
- After reviewing the paper myself as well as the reviews and the author responses, I am missing some of the main motivation of the work. The authors state: "Therefore, the lack of a product/service review dataset containing emotionally-rich variables such as cognitive appraisals and emotional intensity is a primary motivation for the proposed dataset." While reviewers (and I) agree that a dataset like this does not currently exist, that does not address why such a data set should exist. The only direct use case I can think of for studying emotions in product reviewers is to improve targeted advertising. If the authors have other use cases in mind, I would suggest adding them to the paper if it is accepted.
- Emotion classification in general is an ethically complex topic: [1] outlines some of the debates regarding NLP specifically. I think the paper can more thoroughly address the ethical implications of their work, not only in the ethical implications section, but also in clarifying the limitations of their dataset throughout the work, for example, how post-hoc descriptions of emotions may differ from actual emotions. Given that the authors are planning to release the data, it would be useful to add more guidance on how future researchers should and should not use it to help prevent future researchers from making misinterpretations or inaccurate claims based on this data.
- Reviewers note areas for clarification in the details of the data set. These look like minor clarifications to me, some of which are addressed in the appendix and could be moved to the main paper for a final version.

Overall, I think this is a very interesting and well-collected data set and my concerns with the work are primarily about potential misuse.

[1] Mohammad, Saif M. "Ethics sheet for automatic emotion recognition and sentiment analysis." Computational Linguistics 48.2 (2022): 239-278
Decision: Accept-Findings