Modeling Empathic Similarity in Personal Narratives
Authors: Jocelyn J Shen
Maarten Sap
Pedro Colon-Hernandez
Hae Won Park
Cynthia Breazeal
Venue: EMNLP 2023 Main (2023)
Keywords: empathy
semantic similarity
personal narratives
TL;DR: We introduce a new task, dataset and model for identifying similarity in personal stories based on empathic resonance as opposed to raw semantic similarity, and show that this task improves feelings of empathy in a user study.

Abstract: The most meaningful connections between people are often fostered through expression of shared vulnerability and emotional experiences in personal narratives. We introduce a new task of identifying similarity in personal stories based on empathic resonance, i.e., the extent to which two people empathize with each others' experiences, as opposed to raw semantic or lexical similarity, as has predominantly been studied in NLP. Using insights from social psychology, we craft a framework that operationalizes empathic similarity in terms of three key features of stories: main events, emotional trajectories, and overall morals or takeaways. We create EmpathicStories, a dataset of 1,500 personal stories annotated with our empathic similarity features, and 2,000 pairs of stories annotated with empathic similarity scores. Using our dataset, we fine-tune a model to compute empathic similarity of story pairs, and show that this outperforms semantic similarity models on automated correlation and retrieval metrics. Through a user study with 150 participants, we also assess the effect our model has on retrieving stories that users empathize with, compared to naive semantic similarity-based retrieval, and find that participants empathized significantly more with stories retrieved by our model. Our work has strong implications for the use of empathy-aware models to foster human connection and empathy between people.

Modeling Empathic Similarity in Personal Narratives
Jocelyn Shen'
Maarten Sap2. Pedro Colon-Hernandez!
Hae Won Park' Cynthia Breazeal
Massaacuseetss Institute of Technology, Cambridge, MA, USA
Carnegie Mellon University, Pittsburgh, PA, USA
3 Allen Institute for Artificial Intelligence, Seattle, WA, USA
joceshen@mit, t. edu, maartensapécmu.erm edu, {pe25171, haewon, cynthiabJ@mit..
Abstract
fell prady bust paluranc my compony lonelness of nat haviug a roat covnectian
Snepathicall cirb
The most meaningful connections between people are often fostered through expression shared vulnerability and emotional experi- ences in personal narratives. We introduce a new task of identifying similarity in per- sonal stories based on empathic resonance, ie.. the extent which two people empathize with each others' experiences, as opposed to raw semantic or lexical similarity, has pre- dominantly been studied in NLP. Using in- sights from social psychology, we craft framework that operationalizes empathic sim- ilarity in terms of three key features of sto- ries: main events, emotional trajectories, and overall morals takeaways. We create EM- PATHICSTORIES, a dataset of 1,500 personal stories annotated with our empathic similar- ity features, and 2,000 pairs of stories anno- tated with empathic similarity scores. Using our dataset, finetune model compute empathic similarity of story pairs, and show that this outperforms semantic similarity mod- els automated correlation and retrieval met- rics. Through a user study with 150 partici- pants, we also assess the effect our model has retrieving stories that users empathize with, compared to naive semantic similarity-basee retrieval, and find that participants empathized significantly more with stories retrieved by our model. Our work has strong implications for the use of empathy-aware models to foster hu- man connection and empathy between people.
boving people who you pannect
Bew in small lown)
overwtemee with foan
Homooo
imparteoreng
Narverr
having pecale my Me...
"I went to a city this past surtmer for vacution mikking seeving peopie busting ... tale nime for yourse]
Somandicall emrt
Farmator
Figure I: Examples of empathically similar and dis- similar stories. Highlighted are the features of our em- pathic similarity framework (main event, emotion, and moral/takeaway). Narrator A and B are more likely to empathize with one another over their shared feelings of isolation.
and apathy widespread (Buecker al., 2021; Konrath, 2013; Konrath al., 2011). While these challenges cannot be solved with technology alone, AI systems can be developed to bolster emotional support, empathy, and truly meaningful connec- tions through fostering personal experience sharing (Sagayaraj et al., 2022; Chaturvedi et al., 2023; Berridge et al., 2023). In order to do so, these sys- tems must be able to reason about complex social and emotional phenomena between people.
In this work, we introduce the task of modeling empathic similarity, which we define as people's s perceived similarity and resonance to others' expe- riences. For example, in Figure 1, empathic sim- ilarity aims to capture that Narrator A, who feels lonely in their small town, likely to empathize with Narrator B, who is feeling isolated their new job. Crucially, empathic similarity differs from tra- ditional notions of textual similarity that have been the main focus of NLP work (e.g., semantic simi- larity; Reimers and Gurevych, 2019); Narrator A will likely not empathize with Narrator C, despite both stories having higher semantic similarity.
1 Introduction
Through personal experience sharing, humans are able to feel the sting of another person's 's pain and the warmth of another person's joy. This process of empathy is foundational in the ability to connect with others, develop emotional resilience, and take prosocial actions in the world (Coke et al., 1978; Morelli et al., 2015; Vinayak and Judge, 2018; Cho and Jeon, 2019). Today, there is more visibility into the lives others than ever before, yet loneliness
We operationalize empathic similarity around alignment in three features of a personal story

---

(highlighted in Figure 1): its main event, its emo- tional reaction, and its overall moral or story take- away (Hodges et al., 2010; Morelli et al., 2017; Krebs, 1976; Wondra and Ellsworth, 2015; Bal and Veltkamp, 2013; Walker and Lombrozo, 2017; Labov and Waletzky, 1997), as motivated by social psychology and narratology literature. From our definition, empathic similarity arises from the in- terplay of the main events, emotions, and morals in story, where some components or all compo- nents must be similar in order for two narrators to resonate with onc another. For example, Narrator A and B both experience loneliness, even though their actual situations are different (living in a small town versus working at company).
narratives based on shared emotional experiences (Chaturvedi et al., 2018; Lin et al., 2014). One study used Latent Dirichlet Allocation (LDA) to cluster eyberbullying stories and match these sto- ries based on similarity in theme (Dinakar et al., 2012), but discovered that only 58.3% found the matched story to be helpful if provided to the nar- rator of the original story.
Other work has explored ways to bridge the fea- tures of story and human-perceived similarity of stories (Nguyen et al., 2014). Saldias and Roy (2020) found that people use Labov's action (se- ries of events) and evaluation (narrator's needs and desires) clauses to identify similarity in personal narratives (Labov and Waletzky, 1997). Their find- ings support our decision focus modeling events, emotions, and morals within stories.
To enable machines to model empathic similar- ity, we introduce EMPATHICSTORIES, ! corpus of 1,500 personal stories, with crowdsourced an- nontations of the free-text summarics of the main event, emotion, and moral of the stories, as well as an empathic similarity score between 2,000 pairs of stories. We find that finetuning on our paired sto- ries dataset to predict empathic similarity improves performance on automatic metrics as compared to off-the-shelff semantic similarity methods.
Most relevant to our work are recent advances in social and emotional commonsense reasoning using using language models. Specifically, prior methods have used finetuning of language models such as BERT (Devlin et al., 2019; Reimers and Gurevych, 2019) and GPT-2 (Radford al.) model events and the emotional reactions caused by everyday events (Rashkin et al., 2019, 2018; Sap et al., 2019b; Bosselut et al., 2019; Wang et al., 2022; West et al., 2022; Mostafazadeh et al., 2020) as well as predicting empathy, condolence, or prosocial outcomes (Lahnala et al., 2022a; Kumano et al.; Boukricha et al., 2013; Zhou and Jurgens, 2020; Bao et al., 2021). Understanding the emo- tional reactions elicited by events is challenging task for many NLP systems, as requires com- monsense knowledge and extrapolation of mean- ings beyond the text alone. Prior works use com- monsense knowledge graphs to infer and automat- ically generate commonsense knowledge of emo- tional reactions and reasoning about social inter- actions (Sap et al., 2019c,b; Bosselut al., 2019; Hwang et al., 2021). However, there are still many under-explored challenges in developing systems that have social intelligence and the ability to infer states between people (Sap et al., 2022).
While automatic evaluation a valuable signal of model quality, it is crucial to showcase the real- world impact of our task on improving empathy towards people's stories. As such, we conducted full user study with 1.50 participants who wrote their own personal journal entries and were pre- sented stories retrieved by our model (and by a semantic similarity baseline). Our results show that users empathize significantly more with sto- ries retrieved by our finetuned empathic similarity model compared to those from a semantic simi- larity baseline (SBERT; Reimers and Gurevych, 2019). Our findings highlight the applicability of framework, dataset, and model towards fos- tering meaningful human-human connections by enabling NIP systems to reason about complex interpersonal social-emotional phenomena.
2 Related Work
In contrast to previous works, we present a task for reasoning between pairs of stories, beyond pre- dicting social commonsense features of texts alone. Our work builds on top of prior work by devel- oping a framework around empathic resonance in personal narratives in addition to assessing the hu- man effect of Al-retrieved stories on empathic re- sponse beyond automatic metrics. Unlike previous
Document similarity is a well-defined task in NLP (Salton et al., 1997; Damashek, 1995; Deerwester et al., 1990; Landauer and Dumais, 1997), but few have applied this work to matching personal
'ee publicly release our dataset, annotation proce- dure, model, and user study at https://githuu com/ mi ammdiaalabemmaahh--ssttre

---

works, our human evaluation is full user study to see how the model performs given story that the users told themselves, which is much more aligned with weal-world impact.
We operationalize an empathic similarity frame- work grounded in research from social and narra- tive psychology discussed in $3.1. Our framework differs from prior work (Sharma et al., 2020) in that is expanded to the relationship between two peo- plc' 's experiences, rather than how empathetically someone responds, and focuses on learning a con- tinuous similarity signal opposed to detecting the presence of empathy. This distinction is important, as someone may be able express condolences a personal experience, but not necessarily relate the experience itself. The core features of empathic similarity we identify are explained below, and we show how these features contribute to empathic similarity in Appendix A. (1) Main event. Prior work demonstrates that peo- ple empathize more with experiences that are simi- lar to their own (Hodges et al., 2010; Morelli et al., 2017: Krebs, 1976). We formalize this the main event of the story expressed in short phrase (e.g. "living in a small town").
3 Empathic Aspects of Personal Stories
Modeling empathic similarity of stories requires reasoning beyond their simple lexical similarities (see Figure 1). In this section, we briefly discuss how social science scholars have conceptualized empathy ($3.1) and draw on empathy definitions relevant for the NLP domain (Lahnala al., 2022b). Then, we introduce our framework for modeling empathic similarity of stories and its three defining features ($3.2).
3.1 Background on Empathy and Stories
Empathy, broadly defined as the ability to feel or understand what a person is feeling, plays a cru- cial role in numan-human connections. Many prior works in social psychology and narrative psychol- ogy find that the perceived similarity of personal experience has effects empathy (Roshanaei et al., 2019; Hodges et al., 2010; Wright, 2002; Morelli et al., 2017; Krebs, 1976; Wondra and Ellsworth. 2015). For example, Hodges et al. (2010) found that women who shared similar life events to speak- ers expressed greater empathic concern and re- ported greater understanding of the speaker.
(2) Emotional Reaction. Although two people may relate over an experience, they may differ how they emotionally respond to the experience (e.g. "overwhelmed with fear of being all alone" vs "loneliness of not having real connection"). Prior work shows that people have harder time em- pathizing with others if they felt that the emotional response to an event was inappropriate (Wondra and Ellsworth, 2015).
As with these prior works, our work uses shar- ing of personal stories as a means to expressing similarity in shared experiences. Personal story- telling as a medium itself has the ability to reduce stress, shift attitudes, elicit empathy, and connect others (Green and Brock, 2000; Andrews et al., 2022; Brockington et al., 2021). In fact, some re- search has shown that when telling a story a second listener, speakers and listeners couple their brain activity, indicating the neurological under- pinnings of these interpersonal communications (Honey et al., 2012; Vodrahalli et al., 2018).
(3) Moral. Readers are able abstract a higher- level meaning from the story, often referred as the moral of the story (Walker and Lombrozo, 2017) (e.g. "the importance of having people around").. In studying fictional narratives, prior work has found that people can empathize with the takeaway of story, despite its fictional nature (Bal and Veltkamp, 2013).
4 EMPATHICSTORIES Dataset
We introduce EMPATHICSTORIES. a corpus of per- sonal stories containing 3,568 total annotations. Specifically, the corpus includes empathic similar- ity annotations of 2,000 story pairs, and the main events, emotions, morals, and empathy reason an- notations for 1,568 individual stories. An overview of our data annotation pipeline is shown in Fig- ure and data preprocessing steps are included in Appendix D. In Appendix H, we show that using LLMs for human annotation is not viable for our task.
3.2 Empathic Similarity in Personal Stories
We define empathic similarity as measure of how much the narrators of a pair of stories would em- pathize with one another. While there are many ways to express empathy, situational empathy, which is empathy that occurs in response to a social context, conveyed through text-based personal narratives (Fabi et al., 2019).
focus specifically on

---

(a) Individual Annotation
Sampling Strategy
(c) Paired Annotation
Story A
8
events Emotional reactions
Aal
(Story summary, Story B summary)
Annotator
Morals takeswayy
SBERT
(2:4)
8
Story B
Empathic
8 Annotator B
Annotator
Main events
Aa
Similarity Distribution
C
Emotional reactions
Morals takeawayys
Figure 2: Overview of annotation pipeline starting with (a) individual story event, emotion, and moral to (b) using these annotations to sample balanced story pairs and (c) rating empathic similarity scores
# sents
# words
Topic
Keywords
% Stories
Story Main Event
13.17
235.14
romantic relationships posisive life events depression
relationships, divorced, passion opportunities, wedding. depression, therapy, psych families, parents, relatives recovery, drugs. addiction encouragement. caring, distress
15.63%
1.48
32.51
13.20%
Emotional Reaction
2.39
46.08
12.95%
family
10.33%
Moral
1.38
31.35
substancee use encouragement college and school loneliners
9.38%
8.42%
Table I: Story and annotation statistics
stadents, classes, cullege loucliness, relationships, baireut
7,08%
3,87%
youth
teenage, childbood, twenties goodbyes, retired. gradusting
4.97%
changes
4.40%
4.1
Data Sources
work
mundane, coworkers, volunteering abused, traumas, therapist
4.34%
trauma
3.44%
We collect diverse set of stories from sources in- cluding social media sites, spoken narratives, and crowdsourced stories. We take approximately 500 stories from each of the following sources (for a full breakdown see Appendix F). These sources contain English-written stories revolving around deep emotional experiences and open-ended con- versation starters.
Table 2: Themes across main events of the stories.
main event of the story, (2) the main emotional state induced by the main event, and (3) moral(s) of the story. The story and annotated summary statistics are shown in Table 1. The themes from stories are shown Table 2, and themes for annotated sum- maries as well as our topic modeling approach are presented in Appendix E.
(1) Online Personal Stories. We scrape stories from subreddits about personal ex- periences (r/offnychest, r/todayiamhappy, and ricasualcomversation). We also include a small set stories from a public college confessions forum.
4.3 Paired Story Annotation
Sampling Empathic Story Pairs. We devise a sampling method create a sample of balanced empathically similar and dissimilar story pairs, since random sampling across all possible pairs would likely result in an unbalanced dataset with more dissimilar stories than similar stories. First, split the 1,568 stories into train, dev, and test set using 75/5/20 split. Using SBERT (Reimers and Gurevych. 2019), we compute a composite similarity score using average cosine similarity of the embeddings for the story and our empathy features for every possible story pair within the dataset. We randomly sample stories from each bin such that bins with higher composite similarity scores are more likely to be chosen.
(2) Crowdsourced Personal Stories. We use subset of autobiographical stories from the exist- ing Hippocorpus dataset (Sap et al., 2020), which contains recalled and imagined diary-like personal stories obtained from crowdworkers.
(3) Spoken Personal Narratives. We use stories from the Roadtrip Nation corpus (Saldias and Roy, 2020), which contains transcribed personal stories about people's 's career trajectories and life stories,
4.2 Individual Story Annotation
Using these stories, we designed an annotation framework on Amazon Mechanical Turk (MTurk) that asks workers to label individual story features. Then, we asked for short free responses on (1) the
Annotation Procedure With the sampled story pairs, we released an annotation task on Amazon
Phttps: //api, www.siipssssifi...mm io/

---

Annotation
PPA
KA
pathy, event, emotion, and moral similarity across the entire dataset are shown in Table 3. While these agreement scores are seemingly on the lower side, using a softer constraint, we see that most common disagreements are at most likert point away (73% of points are at most distance away). We are aiming for more descriptive annotation paradigm and thus do not expect annotators to perfectly agree (Rottger al., 2022). Furthermore, our agreement rates are in line with other inherently personal and affect-driven annotation tasks (Sap al., 2017; Rashkin et al., 2018). Given the difficulty of our task (reading longer stories and projecting the men- tal state of characters), our agreement is in line with prior work, which achieve around 0.51 0.91 PPA and 0.29 0.34 KA.
Overall
.80
.14
Train
.79
Empathic similarity
Dev
81
Test
.33
.17
Overall
86
.27
Train
866
.26
Event similarity
Dev
.84
.25
Test
.87
-30
Overall
.83
.23
Train
.33
.23
Emotion similarity
Dev
-79
.15
Test
84
.25
Overall
.80
.19
Train
.80
.18
Moral similarity
Dev
.80
.14
Test
.82
.20
Table 3:
Similarity agreement scores (PPA = pairwise percent agreement, KA Krippendorff's Alpha)
MTurk, asking workers to read pairs of stories and rate various aspects of empathic similarity between the stories. Two annotators rated each story pair. From early testing, we found that the task was dif- ficult because of the large amount of text in the stories and the cognitive load of projecting into two narrator's ss mental states. To simplify the task, we used ChatGPT (gpt-3.5-turbal 5-turbo) to summarize all the stories before presenting the pairs annotators. While summarization may remove specific details the stories, we find that the main event, emotion, and moral takeaway are still present.*
Modeling Empathic Similarity
To enable the retrieval and analysis of empathically similar stories, we design task detailed below. Appendix B, we also propose an auxiliary reason- ing task to automatically extract event, emotion, and moral features from stories, which could be used in future work to quickly generate story anno- tations.
5.1 Task Formulation
Our ultimate retrieval task is given query story Q and selects a story S; from a set of N stories S2, Sy} such that i argmaz; sim((a(S.)) fo(Q)). Here, sim(-.-) is similarity metric (e.g. cosine similarity) between two story representations fo(S.) and fo(Q) that are learned from human ratings of empathic similarity.
At the beginning of the task, we first provide the annotator with 6 examples of empathically simi- lar stories: one positive and one negative example for stories that are empathically similar/dissimilar based on each feature: main event, emotion, and moral the story. After reading the two stories, we ask workers to provide explanations of whether and why the narrators would empathize with one another, to prime annotators to think about the em- pathic relationship between the stories. We then ask workers to provide four similarity ratings 4-point Likert scale (1 strongly disagree, strongly agree): (1) overall empathic similarity (how likely the two narrators would empathize with each other), (2) similarity in the main events, (3) emotions, and (4) morals of the stories.
Empathic Similarity Prediction. The overall task is, given story pair (S1, S2), return a similar- ity score fo)) such that sim(:,) :) is large for empathically similar stories and small for empathically dissimilar stories.
5.2 Models
We propose finetuning LLMs to learn embeddings that capture empathic similarity using cosine dis- tance, for efficient retrieval at test time. In contrast, popular approach is to use few-shot prompting of very large language models (c.g., GPT-3 and Chat- GPT), which have shown impressive performance across variety of tasks (Brown et al., 2020). How- ever, in real deployment setting, retrieval through prompting every possible pair of stories is expen- sive and inefficient.
Agreement We aggregate annotations by averag- ing between the raters. Agreement scores for em-
"By comparing the cosine similarity of human annotated event, emotion, and moral to the ChatGPT summarized sto- ries, we find that there is high semantic overlap of the human ground-truths to the automatically generated summaries (0.66 for event, 0.64 for emotion, and ().49 for moral).

---

Model
0
P
R
P
Trank
Prank
SBERT
30.93
29.86
62.75
57.81
90.24
70.48
57.92
17.46
18.74
+ finetuning
35.93
35.21
64.75
58.68
90.73
71.26
57.43
17.59
18.98
BART
10.24
11.54
57.00
52.19
99.02
68.35
49.51
7.56
9.28
+ finetuning
34.20
34.43
64.75
58.2
88.29
70.16
65.84
24.68
26.55
GPT-3
3.24
2.79
51.25
51.25
100
67.77
90.59
0.33
0.79
+ examples
4.94
6.71
51.25
51.27
98.54
67.45
72.77
-4.8
-5.33
ChatGPT
19.56
20.16
56.25
55.24
77.07
64.36
80.69
13.48
14.10
+ 5 examples
27.75
28.07
63.25
60.43
81.95
69.57
85.15
21.27
22.10
Table 4: Model performance for empathic similarity prediction task across correlation, accuracy, and retrieval metrics, r = Pearson's 's correlation, p= Spearman's correlation, Acc = accuracy, P precision, R recall, Pa-1 = precision at & where & I, Trank Kendall Tau of ranking and Prank Spearman of ranking. Note that all scores multiplied by 100 for easier comparison, and the maximum for each metric is 100. In bold is the best performing and underlined is the second-bes best performing condition for the metric.
Baseline Models. We compare performance to finetuning with SBERT (multi-qa-mmpeet-aas--dd v1) (Reimers and Gurevych, 2019; Brown al., 2020) and BART model (bart-base) (Lewis al., 2019). As a few-shot baseline, we evaluate GPT-3 ((ext-davinci-003) and ChatGPT's (gpt-3.5-turbo) ability to distinguish empathically similar stories by using a k-shot prompting setup as done in Sap et al. (2022); Brown et al. (2020). For the query story pair, we ask for empathic similarity score from 1-4, We compare across k- 0 examples and k examples from the training set. We also evaluate these models' ability to generate human- like main event, emotion description, and moral summaries for each story. Again, we use a k-shot prompting setup, comparing across k - 0 and k = 10 examples. See Appendix G and Appendix C for prompts used and finetuning details.
A 1 (what proportion of the top-ranked stories by our model the top-ranked story as rated by human annotators), Kendall's Tau (Abdi, 2007), and Spearman's 's correlation (Schober al., 2018) for the ranking of the stories (how close the overall rankings are).
Shown in Table 4, our results indicate that fine- tuning SBERT and BART with EMPATHICSTO- RIES results in performance gains across all metries. SBERT has relatively high off-the-shelf perfor- mance, as is trained with 215M examples specif- ically for semantic similarity tasks. However, we see that finetuning with our dataset, which contains far fewer training examples relative to SBERT's pretraining corpus, improves performance. (+ 5.35 p. +2 accuracy). BART, which is not specifically pre-trained for semantic similarity tasks, shows even greater gains across retrieval metrics when finetuned on our dataset. (22.89 p. +7.75 accuracy). We find that for BART models, fine tuning im- provements (p = 0.02, p = 0,0006 respectively). as measured with McNemar's 's test on the accuracy scores and Fisher's transformation on correlations, are significantly higher than baselines.
Empathy Similarity Prediction. We propose a bi- encoder architecture finetuned with mean-squared error (MSE) loss of the cosine-similarity between story pairs, as compared to the empathic similar- ity gold labels. For each of the encoders, we use a shared pretrained transformer-based model and further finetune on the 1,500 annotated story pairs in our training set. We obtain the final embedding using mean pooling of the encoder last hidden state,
While GPT-3 and ChatGPT have high perfor- mance on the precision k retrieval metric, in practice, not feasible prompt the models with every pair of stories the retrieval corpus.
6 Automatic Evaluation
7 User Study
To evaluate the quality of empathic similarity pre- dictions, we first compare the Spearman's 's and Pear- son's 's correlations between the cosine similarity of the sentence embeddings and the gold empathic similarity labels. Next, bin scores into binary similar/dissimilar categories (> 2.5 and < 2.5 re- spectively) compute the accuracy, precision, re- call, and Fl scores. Finally, we compute a series of retrieval-based metrics including precision at
Prior work's versions of human evaluations (Zhou and Jurgens, 2020; Bao et al., 2021; Sharma et al., 2020) are humans verifying ranking model out- puts based inputs from test data. This provides valuable signal of model quality, but isn' t represen- tative of how model could be used in world applications due to input distribution mismatch and lack of personal investment in the task. Our hu-

---

Total Empathy Across Conditions
Empathy Dimensions Across Conditions
Empathy Value
Empathy
SOERT
Affective Empathy
Cognitive Empathy
Associative Empathy
Our Model
SBERT
Figure 4: Breakdown of empathy dimensions for the
story retrieved by our model vs. SBERT
Figure 3: Total empathy for the story retrieved by our
model vs. SBERT. Error bars show standard error.
(4x Nvidia A40s, 256GB of RAM, and 64 cores), which retrieves story responses in real time.
man evaluation a full user study to see how the model performs in retrieving a story that is empath- ically similar to story that the users told them- selves. Through our user study, we demonstrate the applicability of the task to improve empathy towards retrieval of human stories, well how our dataset was used develop the empathic sim- ilarity retrieval task and why the task matters in the rorl-world. Our hypothesis is: Users will em- pathize more with stories retrieved by our model (BART finetuned on EMPATHICSTORIES) than sto- ries retrieved by SBERT.
Writing Prompts and Stories Retrieved. We carefully designed writing prompts to present the participants to elicit highly personal stories, in- spired by questions from the Life Story Interview (McAdams, 2007), an approach from social science to gather key moments from person's life.
Conditions. We used within-subject study de- sign, where each participant was exposed to 2 con- ditions presented in random order. In Condition 1, participants read story retrieved by our best performing model the empathic similarity task (BART + finetuning). Condition 2, participants read a story retrieved by SBERT. For both models, we select the best response that minimizes cosine distance.
7.1 Participants and Recruitment
We recruited a pool of 150 participants from Pro- lific. Participants were primarily women (58%, 38% men, 3% non-binary, 1% undisclosed) and white (73%, 8% Black, 9% other or undisclosed, 4% Indian, 3% Asian, 2 % Hispanic, 1% Native American). The mean age for participants was average said they would consider themselves empathetic people (mean 4.3, s.d. 0.81 for Likert scale from 1-5).
Measures. To measure empathy towards each story, we used a shortened version of the State Empathy Survey (Shen, 2010). which contains 7 questions covering affective (sharing of others' feel- ings), cognitive (adopting another's point of view), and associative (identification with others) aspects of situational empathy. We also ask users to provide a free-text explanation of whether and why they found the retrieved story empathically resonant, to gain qualitative insights into their experience.
37 (s.d. 11.6), and participants
7.2 Study Protocol
Participants rated their mood, wrote personal story, then rated their empathy towards the sto- ries retrieved by the baseline and proposed models, They additionally answered questions about the story they wrote (main event, emotion, and moral the story) and their demographic information (age, ethnicity, and gender).
7.3 Effects on Empathy
With our results shown in Figure 3, we found through a paired t-test (N = 150) that users signif- icantly empathized more with stories retrieved by our model finetuned EMPATIICSTORIES than off-the-shelf SBERT (I(149) 2.43, p < 0.01, Cohen's d 0.26), validating our hypothesis. In addition, this effect was present across all three dimensions of empathy: affective (t(149) = 1.87, 0.03, Cohen's 's = 0.21), cognitive (t(149) =
User Interface. We designed web interface simi- lar to a guided journaling app and distributed the link to the interface during the study. The inter- face connects to a server run on GPU machine

---

Reasans why participants empathized
Reasons why participants didn't empathize
SBEAT
SSERT
Consistant
Our Model
Not consistent
Our Model
Genuine
Not ganuine
Read well
Not well
Original
original
Emotions
Emotions
Moralitakzeewwy
Moraltakeeway
100
Frequency mentioned
Frequency mentioned
Figure 5: Reasons why participants did did not empathize with the retrieved story.
2.05, = 0.02, Cohen's = 0.21), and associa- tive empathy (1(149) 2.61, = 0.005, Cohen's d = 0.27), shown in Figure (empathy values are the summed scores from the empathy survey). Interestingly, the difference in empathic response across conditions strongest for associative empa- thy, which measures how much the user can iden- tify with the narrator of the story.
narrator felt, and the takeaway of the story. For example, one participant shared that "I foound no moment where didn't 't fully understand the author, and I share very similar story about my father...its absolutely amazing...I enjoyed this study very much." Other participants wrote. "I empathize heavily with this story because it has many similarities my own. Kind of a 'started from the bottom, now we' here' vibe, which love to see" and "I can relate to the feelings of abandonment and regret expressed."
We examine reasons why with retrieved stories across conditions (Figure 5). Across both conditions, empathy towards story was often related how well-read, genuine, and consistent the story was, and if the user could empathize with the narrator's emotional reactions. When participants did not empathize with re- trieved story, this was more often than not duc to stark differences in the main events of their own story and the model's selected story. This effect was strongest for our finetuned model, as it was trained on data with a more open definition of em- pathy than just sharing the same situation. cer- tain cases, this could result in the events being too different for the user to empathize with.
empathized
8 Future Directions for Empathic Similarity
In summary, few prior works on text-based empa- thy have looked at modeling empathy in woowway interpersonal settings for tuman-to-humme connec- tion, most focus on detecting empathy generat- ing empathetic utterances, and even fewer of these works have shown langible outcomes in human studies. With increasing polarization, loneliness, and apathy (Buecker et al., 2021), personal experi- ences are a fundamental way people connect, yet existing social recommendation is not targeted for numan-humaa connectivity and empathy. Empath- ically encoded story embeddings could be useful for a variety of NL.P tasks, including retrieval, text generation, dialogue, and translation, for example in the following settings:
Interestingly, we see that our model chose stories that aligned better on events and emotions with respect to the story they wrote, and participants thought the stories were more original compared to SBERT-retrieved stories. In cases where the participant did not empathize with the retrieved story, SBERT-retrieved stories were considered less consistent, less genuine, less, original, did not read as well, and did not match on emotions as well compared to our model.
Using empathic reasoning to incorporate story retrieval in dialogue generation.
Generating stories that users resonate with more in conversational AI
From qualitative responses, we see that our model retrieved stories that user empathized with based on the situation described, the emotions the
Extending this work to multilingual settings and better understand translating experiences

---

in ways that preserve empathic meaning
Limitations
With regards to our data collection and annotation framework, our annotations for empathic similarity are not first-person, which are sub-optimal given that it may be difficult for annotator's to project the emotional states of two narrators. In addition, because of the complexity of our annotation task, we opted use ChatGPT summaries of the stories during our paired story annotation, which could introduce biases depending on the quality of the generated summaries. However, given the inherent difficulty of the task, we found this reduction neces- sary to achieve agreement and reduce noise in our dataset, and we found that important features will still present in the summaries. Future work could use our human experimental setup collect first person labels over the entire stories, rather than the automatic summaries.
Better understand cognitive insights, such as linguistic patterns of emotion-driven commu- nication
Applications and building interactions that foster story sharing across geographic, eth- nic, and cultural bridges, such as developing better social media recommendation or per- sonalization.
We encourage future works explore these di- rections in developing more human-centered ap- proaches for interactions with NLP systems.
9
Conclusion
This work explores how we can model empathic resonance between people's personal experiences. We focused specifically on unpacking empathy in text-based narratives through our framework of the events, emotions, and moral takeaways from per- sonal narratives. We collected EMPATHICSTORITE diverse dataset of high-qualig, personal narratives with rich annotations on individual story features and empathic resonance between pairs of stories, We presented novel task for retrieval of empathi- cally similar stories and showed that large-language models finetuned on our dataset can achieve con- siderable performance gains in our task. Finally, we validated the wal-worldd efficacy of our BART- finetuned retrieval model in user study, demon- strating significant improvements in feelings of em- pathy towards stories retrieved by our model com- pared off-the-shelf semantic similarity retrieval.
Another limitation of our modeling approach is that our finetuned model takes in data that cap- tures empathic relations across our framework of events, emotions, and morals. However, the learned story representations general purpose and are not personalized to user's empathic preferences. Personalization could improve model performance across automatic and human evaluation metries, there may exist finer-grained user preferences how users empathize with certain stories, and what aspects users focus on. Furthermore, future work could explore training using contrastive setup to learn contextualized story embeddings. Lastly, future work should explore longitudinal effects of recieving stories retrieved by our system. Our survey measures (State Empathy Scale) are used for short, quick assessments of immediate em- pathy rather than "fixed" or "trait" empathy. While our model might perform well in this one-shot in- teraction settings, it is also important to study the last empathic effects of reading stories retrieved by the model and measure changes in user's longer term empathy, mood, and feelings of connection.
Empathy is a complex and multi-dimensional phenomenon, intertwined with affective and cogni- tive states, and it is foundational in our ability to form social relationships and develop meaningful connections. In a world where loneliness and apa- thy are increasingly present despite the numerous ways we are now able to interact with technology- based media, understanding empathy, developing empathic reasoning in AI agents, and building new interactions to foster empathy imperative chal- lenges. Our work lays the groundwork towards this broader vision and demonstrates that Al systems that can reason about complex interpersonal dy- namics have the potential to improve empathy and connection between people in the rorldworld.
Ethics Statement
While such system might foster empathy and con- nectedness, it is important to consider the potential harms brought about by this work. As with many recommenders, our model is susceptible algorith- mie biases in the types of stories it retrieves, as well as creating an echo chamber for homogeneous per- spectives (Kirk et al., 2023). Embedding diversity in the recommended stories is important both

---

broadening the perspective of users and preventing biases.
of the acl workshop on intrinsic and extrinsic evalu- ation measures for machine translation and/or sum- marization, pages 65-72.
Many social platforms struggle with the issue of content moderation and content safety. In its proposed state, our model does not do anything to guarantee the safety of content that is shared with users. Hateful speech and triggering experiences should not be propagated by our model regardless of the extent to which users relate to these stories (Goel ct al., 2023; Lima et al., 2018).
Jiajun Bao, Junjie Wu, Yiming Zhang, Eshwar Chan- drasekharan, and David Jurgens. 2021. Conver- sations Gone Airight: Quantifying and Predicting Prosocial Outcomes in Online Conversations. Proceedings of the Web Conference 2021, pages 1134 1145, Ljubljana Slovenia. ACM.
Clara Berridge, Yuanjin Zhou, Julie M. Robillard, and Jeffrey Kaye. 2023. Companion robots to mitigate loneliness among older adults: Perceptions of hene- fit and possible deception. Frontiers in Psychology. 14:1 106633.
Finally, the goal of our work is to connect peo- ple to other human experiences. Story generation and NLG that aims mimic appropriate hu- man experiences is not something we endorse, and we encourage the use of machine-text detectors in systems that retrieve empathic stories. In line with Oren Etzioni (2018)'s three rules of AI, we also discourage presenting humans with machine- generated stories without disclosing that the story is written by an AT author.
Antoine Bosselut, Hannah Rashkin, Maarten Sap, Chai- tanya Malaviya, Asli Celikyilmaz, and Yejin Choi. 2019. COMET: Commonsense Transformers for Automatic Knowledge Graph Construction.
Hana Boukricha, Ipke Wachsmuth, Maria Nella Carmi- nati, and Pia Knoeferle. 2013. Computational
Model of Empathy: Empirical Evaluation. In 2013 Humaine Association Conference on Affective Computing and Intelligent Interaction, pages 1-6, Geneva, Switzerland. IEEE
Acknowledgements
Guilherme Brockington, Ana Paula Gomes Moreira. Maria Stephani Buso, Sérgio Gomes da Silva, Edgar Altszyler, Ronald Fischer, and Jorge Moll. 2021. Storytelling increases oxytocin and positive emo- tions and decreases cortisol and pain in hospitalized children. Proceedings of the National Academy of Sciences, 118(22):e201 1800918811.11.
We would like to thank all of our participants, an- notators, and teammates for their invaluable con- tributions to this project. Special thanks to Sharifa Algohwinem and Wonjune Kang for their technical feedback throughout the project and thanks to Ji Min Mun, Akhila Yerukola, and Ishaan Grover for paper feedback. This work was supported by an NSF GRFP under Grant No. 2141064 and the ITTP grant funded by the Korean Ministry of Science and ICT No.2020-0-00842.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neclakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger. Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam Me- Candlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language Models are Few-Shot Learners. arXiv:2005. 14165 [cs].
References
Hervé Abdi. 2007. The kendall rank correlation coeffi- cient. Encyclopediaa of Measurement and Statistics. Suge, Thousand Ouks, CA, pages 508-510.
Susanne Buecker, Marcus Mund, Sandy Chwastek, Melina Sostmann, and Maike Luhmann. 2021. Is loneliness in emerging adults increasing over time? A preregistered cross-temporal meta-analysis and systematic review. Psychological Bulletin, 147:787- 805.
Mary E. Andrews, Bradley D. Mattan, Keana Richards, Samantha L. Moore-Berg, and Emily B. Falk. 2022. Using finst-perso narratives about healthcare work- ers and people who are incarcerated to motivate help- ing behaviors during the COVID-19 pandemic. So- cial Science & Medicine, 299:114870.
Rijul Chaturvedi, Sanjeev Verma, Ronnie Das, and Yogesh K. Dwivedí. 2023. Social companionship with artificial intelligence: Recent trends and fu- ture avenues. Technological Forecasting and Social Change, 123:442634.
P. Matthijs Bal and Martijn Veltkamp. 2013. How Does Fiction Reading Influence Empathy? An Experimen- tal Investigation on the Role of Emotional Trans- portation. PLOS ONE, 8(1)x55344.
Snigdha Chaturvedi, Shashank Srivastava, and Dan Roth. 2018. Where Have Heard This Story Be- fore? Identifying Narrative Similarity in Movie Re- makes. In Proceedings of the 2018 Conference of
Satanjeev Banerjee and Alon Lavie. 2005. Meteor: An automatic metric for mt evaluation with improved correlation with human judgments. In Proceedings

---

the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies. Volume 2 (Short Papers), pages 673-678, New Orleans, Louisiana, Association for Computa- tional Linguistics.
Sara D. Hodges, Kristi J. Kiel, Adam I. Kramer, Darya Veach, and B. Renee Villanueva. 2010. Giv- ing Birth Empathy: The Effects of Similar Experi- ence on Empathic Accuracy, Empathic Concern, and Perceived Empathy. Personality and Social Psychol- ogy Bulletin, 36(3):398 -09..
Eun Cho and Soohyun Jeon. 2019, The role of empa- thy and psychological need satisfaction in pharmacy students' burouu and well-being. BMC Medical Ed- ucation, 19(1):43.
Christopher J. Honey, Christopher R. Thompson, Yulia Lerner, and Uri Hasson. 2012. Not Lost in Trans- lation: Neural Responses Shared Across Languages. The Journal of Neuroscience, 32(44))15277-155583
Jay S. Coke, C. Daniel Batson, and Katherine McDavis. helping. A two-stage model. Journal of Personality and Social Psychol- ogy; 36(7))552-766
Jena D. Hwang, Chandra Bhagavatula, Ronan Bras, Jeff Da Keisuke Sakaguchi, Antoine Bosselut, and Yejin Choi. 2021. COMET-ATOMIC 2020: On Symbolic and Neural Commonsense Knowledge Graphs.
1978. Empathic mediation of
Mare Damashek. 1995. Gauging Similarity with n- Grams: Language-Independee Categorization of Text. Science, (67($199) Publisher: American Association for the Advancement of Sci- ence.
Hyunwoo Kim, Youngjae Yu, Liwei Jiang. Ximing Lu, Daniel Khashabi, Gunhee Kim, Yejin Choi, and Maarten Sap. 2022. ProsocialDialog: A Prosocial Backbone for Conversational Agents.
Scott Deerwester, Susan T. Dumais, George W. Fur- nas, Thomas K. Landauer, and Richard Harshman. 1990. Indexing by latent semantic analysis. Jour- nal of the American Society for Information Science, 41(6):391-44
Hannah Rose Kirk, Bertie Vidgen, Paul Rötteer, and Scott A. Hale. 2023. Personalisation within bounds: A risk taxonomy and policy framework for the align- ment of large language models with personalised feedback. ArXiv:2303.00 3.330330053333. [es].
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. arXiv: 1870,04805 [cs]. ArXiv: 1810.04805.
Sara Konrath. 2013.
The Empathy Para- dox: Increasing Disconnection the Age of Increasing Connection. https://www.igi- habbaa..onn/caaaere/ooeeetwwwwwo gabbal.coo//caapperrconeee///0
Karthik Dinakar, Birago Jones, Henry Lieberman, Ros- alind Picard, Carolyn Rose, Matthew Thoman, and Roi Reichart. 2012. You Too?! Mixed-Initiation LDA Story Matching to Help Teens in Distress. Pro- ceedings of the International AAAI Conference on Web and Social Media, 6(1):74-81.
Sara H. Konrath, Edward H. O' O'Brien, and Courtney Hsing. 2011. Changes in Dispositional Empathy in American College Students Over Time: A Mcta- Analysis. Personality and Social Psychology Re- view, (12):880-198.
Dennis Krebs. 1976. of Persomality and Social Psychology. 32(6); 133..
Empathy and altruism. Journal
Sarah Fabi, Lydia Anna Weber, and Hartmut Leuthold. 2019, Empathic concern and personal distress de- pend on situational but not dispositional factors. PLoS ONE, (211))xe2225022-022255
Shiro Kumano, Ryo Ishii, and Kazuhiro Otsuka. Com- paring Empathy Perceived by Interlocutors in Multi- party Conversation and External Observers.
Fabrizio Gilardi, Meysam Alizadeh, and Maël Kubli. 2023. ChatGPT Outperforms Crowd-Worken for Text-Annotation Tasks, ArXiv:2303. 15555 [cs].
William Labov and Joshua Waletzky. 1997. Narrative analysis: Oral versions of personal experience. Jour- nol of Narrative & Life History, 7(1-4)3--38
Vasu Goel, Dhruv Sahnan, Subhabrata Dutta, Anil Bandhakavi, and Tanmoy Chakraborty. 2023. Hate- mongers ride on echo chambers to escalate hate speech diffusion. PNAS Nexus, pgad04.,
Allison Lahnala, Charles Welch, and Lucie Flek. 2022a. CAISA at WASSA 2022: Adapter-Tume for Em- pathy Prediction. In Proceedings of the 12th Work- shop on Computational Approaches Subjectiviry, Sentiment & Social Media Analysis, pages 280-285, Dublin, Ireland. Association for Computational Lin- guistics.
Melanie C. Green and Timothy C. Brock. 2000. The role of transportation in the persuasiveness of public narratives. Journal of Personality and Social Psy- chology, -915)
Allison Lahnala, Charles Welch, David Jurgens, and Lucie Flek. 2022b. A Critical Reflection and For- ward Perspective on Empathy and Natural Language Processing. In Findings of the Association for Com- putational Linguistics: EMNLP 2022, pages 2139- 2158. Abu Dhabi, United Arab Emirates. Associa- tion for Computational Linguistics.
Maarten Grootendorst. 2020. Keybert: Minimal key- word extraction with bert.
Laura Hanu and Unitary team. 2020. Detoxify. Github. http:://gitubb..ommmuiiaaaaaaaet

---

Thomas K. Landauer and Susan T. Dumais. 1997. so- lution to Plato's problem: The latent semantic anal- ysis theory of acquisition, induction, and representa- tion of knowledge. Psychological Review, 104:211 1- 240. Place: US Publisher: American Psychological Association.
Oren Etzioni. 2018, Three rules of Artificial Intelli- genee.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei- Jing Zhu. 2002. Bleu: a Method for Automatic Eval- uation of Machine Translation. In Proceedings of the 40th Anmual Meering of the Association for Com- putational Linguistics, pages 311-318, Philadelphia. Pennsylvania, USA. Association for Computational Linguistics.
Mike Lewis. Yinhan Liu, Naman Goyal, Mar- jan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. 2019. BART: Denoising seuueence-t-Seqqeene Pre- training for Natural Language Generation, Transla- tion, and Comprehension. ArXiv:1910.1344 [es, stat].
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language Mod- cls are Unsupervised Multitask Learners.
Lucas Lima, Julio C.S. Reis, Philipe Melo, Fabricio Murai, Leandro Araujo, Pantelis Vikatos, and Fabri- cio Benevenuto. 2018. Inside the Right-Leaning Echo Chambers: Characterizing Gab, an Unmoder- ated Social System. In 2018 IEEE/ACM Interna- tional Conference on Advances in Social Networks Analysis and Mining (ASONAM), pages 515-522.
Hannah Rashkin, Antoine Bosselut, Maarten Sap. Kevin Knight, and Yejin Choi. 2018. Modeling Naive Psychology of Characters in Simple Com- monsense Stories. In Proceedings of the 56th An- nual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2289- 2299, Melbourne, Australia. Association for Com- putational Linguistics.
ISSN: 2473-991X.
Chin- Lin. 2004. ROUGE: A Package for Auto- matic Evaluation of Summaries. Text Summariza- tion Branches Out, pages 74-81, Barcelona, Spain. Association for Computational Linguistics.
Hannah Rashkin, Maarten Sap. Emily Allaway, Noah A. Smith, and Yejin Choi. 2019. Event2Mind: Commonsense Inference on Events, Intents, and Re- actions.
Yung-Shen Lin, Jung-Yi Yi Jiang, and Shie-Jue Lee. 2014. A Similarity Measure for Text Classification and Clustering. IEEE Transactions on Knowledge and Data Engineering, 26(7):1575-1590.
Nils Reimers and Iryna Gurevych. 2019. Sentence- BERT: Sentence Embeddings using Siamese BERT- Networks.
Mahnaz Roshanaei, Christopher Tran, Sylvia Morelli, Cornelia Caragea, and Elena Zheleva. 2019. Paths to Empathy: Heterogeneous Effects of Reading Per- sonal Stories Online. In 2019 IEEF International Conference on Data Science and Advanced Analyt- ics (DSAA), pages 570-579, Washington, DC, USA. IEEE.
Dan P McAdams. 2007, The Life Story Interview page 5.
Sylvia A. Morelli, Matthew D. Lieberman, and Jamil Zaki. 2015. The Emerging Study of Positive Em- pathy. Social and Personality Psychology Compass, 9(2):77-68.
Paul Rottger, Bertie Vidgen, Dirk Hovy, and Janet Pierrehumbert. 2022. Two Contrasting Data An- notation Paradigms for Subjective NLP Tasks. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, pages 175-190, Seattle, United States. Association for Computational Linguistics.
Sylvia A. Morelli, Desmond C. Ong, Rucha Makati, Matthew O. Jackson, and Jamil Zaki. 2017. Empa- thy and well-being correlate with centrality in dif- ferent social networks. Proceedings of the National Academy of Sciences, 1 114(37):9843 9847.
Nasrin Mostafazadeh, Aditya Kalyanpur, Lori Moon. David Buchanan, Lauren Berkowitz, Or Biran, and Jennifer Chu-Carroll, 2020. GLUCOSE: General.- ized and COntextualized Story Explanations. In Proceedings of the 2020 Conference on Empirical Methods Natural Language Processing (EMNLP), pages 4569-4586 Online. Association for Computa- tional Linguistics.
Mary Fabiola Sagayaraj, Ignisha Rajathi George, R. Vedhapriyavadhana, and L. R. Priya. 2022. Ar- tificial Intelligence to Combat the Sting of the Pan- demic on the Psychological Realms Human Brain. SN Computer Science, 3(3):182.
Belen Saldias and Deb Roy. 2020. Exploring aspects of similarity between spoken personal narratives by disentangling them into narrative clause types.
Dong Nguyen, Dolf Trieschnigg, and Mariët Theune. 2014. Using Crowdsourcing to Investigate Percep- tion of Narrative Similarity. In Proceedings of the 23rd ACM International Conference on Conference on Information and Knowledge Management, CIKM 14, pages 321-330, New York, NY, USA. Associa- tion for Computing Machinery.
Gerard Salton, Amit Singhal, Mandar Mitra, and Chris Buckley. 1997. Automatic text structuring and sum- marization. Information Processing & Management. 1932)

---

Maarten Sap, Dallas Card, Saadia Gabriel, Yejin Choi, and Noah A. Smith. 2019a. The Risk Racial Bias in Hate Speech Detection. In Proceedings of the S7th Annual Meeting of the Association for Com- putational Linguistics, pages 1668-1678, Florence, Italy. Association for Computational Linguisties.
Zhilin Wang, Anna Jafarpour, and Maarten Sap. 2022. Uncovering Surprising Event Boundaries in Narra- tives. page 12.
Peter West, Chandra Bhagavatula, Jack Hessel, Jena D. Hwang. Liwei Jiang, Ronan Le Bras, Xim- ing Lu, Sean Welleck, and Yejin Choi. 2022.
Maarten Sap, Eric Horvitz, Yejin Choi, Noah Smith, and James Pennebaker. 2020. Recollection versus Imagination: Exploring Human Memory and Cogni- tion via Neural Language Models. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1970-1978, On-
Symbolic Knowledge Distillation: from Gen- eral Language Models Commonsense Models. ArXiv:2110.07178 [es].
Joshua D. Wondra and Phoebe C. Ellsworth. 2015. An appraisal theory of empathy and other vicari- ous emotional experiences. Psychological Review, 122(3):411-422
line. Association for Computational Linguistics.
Maarten Sap, Ronan Le Bras, Emily Allaway, Chan dra Bhagavatula, Nicholas Lourie, Hannah Rashkin, Brendan Roof, Noah A. Smith, and Yejin Choi. 2019b. ATOMIC: An Atlas of Machine Common- sense for If-Then Reasoning. Proceedings of the AAAI Conference Artificial Intelligence, 33:3027- 3035.
Kevin Wright. 2002. Motives for communication within on-line support groups and antecedents for in- terpersonal use. Communication Research Reports. 19(1):89-98.
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2020. BERT'Score: Evaluating Text Generation with BERT.
Maarten Sap. Ronan LeBras, Daniel Fried, and Yejin Choi. 2022. Neural Theory-of-Minde On the Limits of Social Intelligence in Large LMs.
Naitian Zhou and David Jurgens. 2020. Condolence and Empathy in Online Communities. In Proceed- ings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 609-626. Online. Association for Computational Linguistics.
Maarten Sap. Marcella Cindy Prasettio, Ari Holtzman, Hannah Rashkin, and Yejin Choi, 2017. Connota- tion Frames of Power and Agency in Modern Films. page 6.
Maarten Sap, Hannah Rashkin, Derek Chen, Ronan Le- Bras, and Yejin Choi. 2019c. SocialIQA: Common- sense Reasoning about Social Interactions.
Patrick Schober, Christa Boer, and Lothar A Schwarte. 2018. Correlation coefficients: appropriate use and interpretation. Anesthesia & analgesia,
126(5): 1763-17766677
Ashish Sharma, Adam Miner, David C. Atkins, and Tim Althoff. 2020. A Computational Approach to Understanding Empathy Expressed in Text-B Based Mental Health Support.
Lijiang Shen. 2010. On Scale of State Empathy Dur- ing Message Processing. Western Journal of Com- munication, 74(5):504-52 524.
Seema Vinayak and Jotika Judge. 2018. Resilience and Empathy Predictors of Psychological Well- being among Adolescents. International Journal of Health Sciences, (4):10.
Kiran Vodrahalli, Po-Hsuan Chen, Yingyu Liang, Christopher Baldassano, Janice Chen, Esther Yong. Christopher Honey, Uri Hasson, Peter Ramadge, Kenneth A. Norman, and Sanjeev Arora. 2018. Mapping between fMRI responses movies and their natural language annotations. Neurolmage, 180:223-231.
Caren M. Walker and Tania Lombrozo. 2017. Explain- ing the moral of the story. Cognition, 167:266-281.

---

A
Understanding Aspects of Empathic Similarity
Feature
r
Similarity in Main Event
0.69
0.69
Similarity in Emotion Description Similarity in Moral
0.65
0.65
0.76
0.76
Empathic Similarity vs semantic Similarity Distribution
Table S: Correlation between similarity scores for indi- vidual features compared to overall empathic similarity score. Pearson's correlation coefficient, / Spear- man's 's correlation coefficient.
Empathic similarity aimilntly
600
500
ROUGE
BART
1.16
16.87
21.26
13.30
+ finetuning
9.56
32.72
29.14
39.79
200
Event
GPT3
1.40
24.77
26.31
33.39
examples
7.72
ChatGPT
1.85
25.38
25,36
34.93
+ 10 examples
7.23
30.02
32.81
37.59
BART
0.40
15.73
16.95
0.0
finctoning
2.08
23.54
26.24
Normalized score
GPT-3
1.56
22.37
27.90
21.28
Emetion
Figure 6: Comparing the empathic similarity and se- mantic similarity core distributions
examples
0.08
21.09
12.08
19.97
ChatGPT
examples
25.43
27.67
26.46
BART
0.02
11,78
15.52
0.40
* finetaning
13.77
33.52
29.66
32.26
Before training any models to learn empathic similarity ratings, is important to understand the mechanisms behind empathic similarity in text- based personal narratives. In particular, we are interested in how structural elements of stories (events, emotional trajectories. and morals) relate to empathy. The question we aim to answer through our analysis of the text is what qualities of personal experiences people resonate with most and how does this relate to the personal experience they self disclose.
GPT-3
27.87
3164
Moral
10 examples
4.38
28.63
18.97
28.15
ChatGPT
4.45
25.03
26.16
30.99
+ examples
27.91
27.51
33.97
Table 6: Quality of event, emotion, and moral sum- maries across models. Scores are multiplied by 100
for readability, and the max. for each metric is 100.
weakly positively correlated with empathic sim- ilarity (p 0.17), with event-based features corre- lating the most (p 0.067), followed by emotion- based features (p = 0.0069) and lastly moral fea- tures (p 0.048). These results indicate that semantic similarity naturally related to empathic similarity, but might not capture relationships be- tween emotions and takeaways in pairs stories.
First, we look the correlation between human- rated similarity in event, emotion, and moral of the stories to the empathic similarity rating. We show in Table that the correlation of the similarity be- tween events, emotions, and morals to the empathic similarity rating is high for all three features. This indicates that similarity in these components is re- lated to similarity in empathic resonance between stories. Using paired t-test between high and low empathically similar story pairs, find that empathically similar story pairs have statistically significantly higher similarities events, emotions, and morals, with the largest increase in moral simi- larity and roughly equivalent increases in event and emotion similarities.
B Empathy Reasoning Task
Empathy Reasoning Task Definition. Given a story context c, finetune ssequence-t--sequeen (seq2seq) model to generate event (v), emo- tion (e), and moral (m), concatenating annotated summaries to construct the gold label and mod- eling p(v, e, m|c) (Kim et al., 2022). The model is trained to minimize negative log likelihood predicting each word in the constructed gold label.
Next, we look at the differences between se- mantic similarity and human-rated empathic sim- ilarity. As shown in Figure 6, can see that the distributions of similarity scores are differ- ent for human-rated empathic similarity scores as compared to semantic similarity scores obtained from SBERT. Semantic similarity of stories is
Empathy Reasoning Results. We evaluate pathy reasoning performance using BLEU (Pa- pineni al., 2002), ROUGE (Lin, 2004), MEB TEOR (Banerjee and Lavie, 2005), and BertScore (Zhang et al., 2020), taking the human-writter free-text annotations as gold references. From Ta-

---

ble we see that finetuning BART with human- written story summaries improves performance across all metrics. The BART model finetuned on EMPATHICSTORIES demonstrates improved per- formance across 3/4 metrics event and moral rea- sons. For emotion reasons, ChatGPT demonstrates better performance in 2/4 metrics, with the fine- tuned BART model close behind. We note that the BART-base model has 140M parameters, whereas ChatGPT has upwards of 175B parameters.
Note that we did not include topics for the events since these were similar Table 2. To identify these topics, we use Latent Dirichlet Allocation (LDA) and KeyBERT on the clusters (Grootendorst, 2020).
Topic
Keywards
% Stocies
depressioo
meluncholy. depression, unhappy
happinezz and sariffction ansiety
happiness, satisfaction, overwhelmed
20.92%
ansicty, Inustraedd, upsct
11.03%
motivaaioo
motivoted, sucrews, achioving
10.40%
compassion
compassioatl. happiness, gradchildren
9,38%
gratinde
grittude, gencrosity, happincss
9.31%
desiie
desire, possion, youth grief, weber, litsstyle
6.12%
Finetumed Model Training Details
gridf
3,80%
Table 7: Themes across emotion descriptions of the sto- ries.
We use a 75:5:20 train:dev.tesl split on both indi- vidual stories and pairs of stories. For the empathic similarity prediction task, use learning rates of le-6 and 5e-6 for SBERT and BART respectively, and linear scheduler with warmup. For the em- pathic reasoning task, we use a learning rate of 1e-5. For both tasks, we use batch size of 8 and finetune for 30-50 epochs, monitoring correlation and validation loss to select the best-perforning models. We trained all models on 4x Nvidia A40s with 256GB of RAM and 64 cores, and all model training times were under 12 hours.
Topic motivattion and encounnnement overooming and revillence happiners and fuifimen
Keywords
motivation, success, schieving resillent, rehab opporanities, happincss, meaningful companinnetip, gratitude, stress
40.31%
25.07%
17.60%
social support and gratimude
16.52%
Table 8: Themes across morals of the stories.
F Collected Stories Breakdown
A breakdown of the amount of stories per source can be found in Table 9.
D Data Pre-Processing
Data Source
Number of Stories
For all of the data sources, we remove stories that are shorter than sentences long, longer than 500 words, and which have a severe toxicity score of less than 0.005 using Detoxify (Hanu and Unitary team, 2020). While the latter step may filter meaningful stories and introduce bias in the story selections (Sap et al., 2019a), we eтr on the side removing any stories that could potentially harmful, even if not severely so.
Hippocorpus
483
Road Trip Narratives
476
Reddit Today I Am Happy
198
Reddit-Cassen Conversations
195
Reddit Off My Chest
Facebook [Redacted] Confessions
54
Table 9: Breakdown retrieved stories per data source.
G GPT-3 and ChatGPT Prompts
Our research team then selected stories that were appropriate to share (did not contain excessive pro- fanity or explicit sexual content), and which had first-person narrator and concrete resolution to the story. We chose stories with concrete resolution in order to avoid rant posts, which were common on social media pages. In addition, we manually corrected overt grammatical errors as well as ref- erences to the platform the story was shared on (c.g. addressing Redditors). Our final set of sto- ries contains 1,568 curated, high-quality personal narratives.
Below are prompts we fed to GPT-3 and ChatGPT for our few-shot baselines. Note that in addition to the prompts, we provided sampled examples from our training corpus.
Event summary: What is the main event be- ing described in the story? Response must be least sentence and 50-1000 characters including spaces.
Emotion summary: Describe the emotions the narrator feels before and after the main event and why they feel this way. Answer as though you were explaining how the narra- tor felt to someone who knew nothing about the situation. Response must be at least 2
E Story and Annotation Themes
Below we show the top themes across each story's emotion (Table 7) and moral (Table 8) annotations.

---

sentences and 150--000 characters including spaces.
Summary
BLEU
ROUGE
METEOR
BertScore
Main Event
2.86
26.37
28.20
36.53
Emotion Description
1,43
23.01
28.87
23.36
Moral
7.67
27.64
27.33
33.24
Moral summary: What is the high-level lesson takeaway (ie. moral) of the story? Response must be at least sentence and 100- 1000 characters including spaces.
Table 10: Quality of ChatGPT story empathy reasoning annotations (scores are multiplied by 100 for readabil- ity, and the maximum for each metric is 100)
Empathic similarity: Rate the extent to which you agree with the statement "the nar- rators of the two stories would empathize with each other." " We define empathy as feeling, un- derstanding, and relating to what another per- son is experiencing. Note that it is possible to have empathy even without sharing the ex- act same experience or circumstance. Impor- tantly, for two stories to be empathetically similar, both narrators should be able to pathize with each other (if narrator A' story was shared in response to narrator B's story, narrator B would empathize with narrator and vice versa). Give your answer on a scale from 1-4 (1 not at all, 2 not so much, 3 very much, 4 extremely)
Empathic Similaniyy Score Distribution vs Human)
Human
800
600
Frequency
400
200
2.0
3.5
Empathic Similarity Score (1-4)
Figure Comparing the empathic similarity score dis- tributions between ChatGPT and human labels
1-4 for the empathic similarity between two sto- rics. The Spearman's 's correlation between human and ChatGPT generated labels is 0.22 (p < 0.001), indicating weakly positive correlation between hu- man annotations and ChatGPT annotations. In ad- dition, we perform one-sample t-test on the mean- squared error between automatically generated la- bels and human annotations across all story pairs in the training data, obtaining p-value 0.001, indicating that the mean of all the errors nonzero with statistical significance.
H
Using LLMs as a Proxy for Human Annotations
Recent works raise the question of whether LLMSs can be used to proxy human annotations (Gilardi al., 2023). The motivation behind this method is that obtaining human labels across many pairs of stories is costly, and this cost only compounds as the number of stories in the corpus increases. As such, we provide additional analyses as to whether or not these models can truly perform at the same level as human annotators for task, which in- volves heavy empathy and emotion reasoning.
Finally, we bin the ChatGPT annotations into agree/disagree categories, and compute the classi- fication precision (0.59), recall (0.40), Fl score (0.48), and accuracy (0.59) compared to hu- gold labels. These scores offer insight how well ChatGPT predicts the direction of the empathic similarity annotation, but we see that ac- curacy is low when comparing to human labels. Figure 7, we see that ChatGPT similarity scores are skewed to the left, indicating that humans are more likely to find empathic similarities between expe- riences. These results are also supported by the higher number of false negatives when comparing ChatGPT classification human gold labels.
H.1 Individual Story Annotation
We prompt ChatGPT (gpt-3.5-turbo) to generate summaries of each story's main event, emotion, and moral, in addition to a list of reasons why nar- rator might empathize with the story. We compare these summaries against human-written summaries using BLEU, ROUGE, METEOR, and BertScore (Table 10), showing that ChatGPT has relatively low performance across all four metries.
H.2 Paired Story Annotation
We feed the same prompt given to human annota- tors into ChatGPT, asking for a Likert score from


Reviews:
[{"review_id": "qTOgV9vA0R", "reviewer": "Reviewer_g6sS", "summary": "", "questions": "The impact of a personal story and its similarity is useful to get some valid conclusion ?? though it is not in the scope, the impact needs to be calculated in order to find the success of the method.  Please write a paragraph about this point in a paper.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your  insightful comments, and for acknowledging the efforts of our work in modeling social-emotional relevance of stories with respect to empathy.\n\n1. _Clarification of impact of our method on empathic similarity prediction and empathy summaries_:\n* We see that the impact of our method in retrieving emphatically similar stories might not be clear.  For metric-wise impact, we show through automatic evaluation that *models fine tuned on our dataset significantly outperform baselines* in empathetically similar story retrieval. We also find that for BART models, fine tuning improvements, as measured with McNemar’s test on the accuracy scores and Fisher’s transformation on correlations, are significantly higher than baselines (***p*= 0.02, *p*= 0.0006** respectively; results seen in Table 4).  In lines 374-386, we further elaborate on the impact fine-tuning with our dataset has across a variety of automated metrics for semantic similarity and retrieval. We will clarify this in the final version.\n* A core contribution of our work is showing the _real-world impact of our model on user’s empathy_ towards stories selected by our model. In our human evaluation, we report Cohen’s effect size (*d*=0.26) and statistical significance (*p*<0.01) showing that *our methods retrieve stories that real world users empathize significantly better with  (Section 6.3).*\n\n* We thank the reviewer for mentioning evaluating the impact of the empathy summaries. We provide automated evaluation metrics for the quality of these summaries using BLEU, ROUGE, METEOR, and BertScore in Table 5. For the empathy reasons summarization task, the input is a single personal story, and the outputs are summaries of the (1) main event of the story, (2) emotional trajectory of the story, and (3) the overall takeaway/moral of the story. These stories also may contain conversational elements, which are also included in the output summaries depending on their relevance.\n\n2.  _Exploration of more models and datasets_: \n\n* Currently, no existing datasets containing empathy relationships between personal stories exist. As such, we searched for the most relevant literature (Sharma et al. 2020; Saldias et al. 2020) [1] and used stories from these datasets, as well as our own collected data, for our final dataset. \n\n* We used state-of-the-art models (gpt-3.5, gpt-3, BART, and SBERT). In preliminary experiments, we also tested other transformer-based models (ie. GPT-2, Flan-T5), but found their performance to be worse than the BART model, so we did not include these in paper. We will make a note in the paper.\n\n3. _Personal story conversational elements_:\n* Personal stories contain conversational elements (e.g., quotes), and are typically included in the summary when the conversation is important enough to affect the overall trajectory of the story.  We will make a note of this in the camera ready. \n\n4. _Participant selection_:\n* We selected participants over the age of 18 who are from predominantly English-speaking countries (US and UK), as approved by our institution’s IRB. This will be clarified in the  camera ready.\n\n5. _Clarifying methodology_:\n* “Why is cosine similarity used?” As shown in (Reimers et al. 2017), cosine similarity is an effective distance function for semantic similarity tasks, which is most analogous to our new task of empathic similarity prediction. It is also used generally for text generation metrics (Zhang et al. 2020). There is more literature on classical word embeddings that supports this [2].  As such, we incorporated this metric as our primary means of comparing empathic similarity.\n* “Line No 301,302 says the score is less for dissimilar. How?” - For empathically similar stories, we aim to have the model return a high similarity score, pulling empathically similar stories (as rated by human labelers) closer in the embedding space. Conversely, for dissimilar stories, one would have a smaller score. We will clarify this in the camera ready.\n* “explain fig 2 working with an example.”\nAn example for each element would be:\n    - Story A example: “Once upon a time, there was a curious dog named Biscuit who loved exploring. While playing fetch, he accidentally stumbled on a hidden treasure chest filled with delicious bones. From that moment on, Biscuit was always eager to embark on new adventures.”\n    - Annotator A summarizes event (“Biscuit stumbled on a treasure chest”), emotion (“Biscuit is excited and curious when he finds the chest”), and moral (“curiosity can lead to unexpected positive discoveries”).\n    - Story B example: “There was an imaginative child named Lily who loved to explore. She was curious about an abandoned treehouse in her neighborhood and went to explore. There, she found a kitten and they became good friends.”\n    - Annotator B summarizes event (“While exploring a treehouse, Lily finds a kitten.”), emotion (“Lily feels curious about the treehouse and happy when she makes a new friend.”), and moral (“crossing uncharted territories can lead to new joys”)\n    - Concatenating encode(event_A + emotion_A + moral_A) compared with encode(event_B + emotion_B + moral_B). Based on the sampling strategy described in Lines 234-248, we sample the pair (Story A, Story B) to provide to raters.\n    - Annotator C decides how empathically similar  story A and story B are to one another (on scale of 1-4).\n\n[1] Giorgi, S., Zhao, K., Feng, A. H., & Martin, L. J. (2023). Author as Character and Narrator: Deconstructing Personal Narratives from the r/AmITheAsshole Reddit Community. Proceedings of the International AAAI Conference on Web and Social Media, 17(1), 233-244. \n\n[2] Lahitani, A. R., Permanasari, A. E., and Setiawan N. A. Cosine similarity to determine similarity measure: Study case in online essay assessment. 2016 4th International Conference on Cyber and IT Service Management, Bandung, Indonesia, 2016, pp. 1-6.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper is a good attempt to enable social-emotional correlation between people based on their stories.  The dataset of 2000 stories is constructed. 150 participants feedback on the story is also created and compared with the model. The stories are classified into main, empathy and moral events.", "reasons_to_reject": "The concerns are addressed in the revision.", "questions_to_authors": "The impact of a personal story and its similarity is useful to get some valid conclusion ?? though it is not in the scope, the impact needs to be calculated in order to find the success of the method.  Please write a paragraph about this point in a paper.", "paper_topic_and_main_contributions": "The paper is a good attempt to enable social-emotional correlation between people based on their stories.  The dataset of 2000 stories is constructed. 150 participants feedback on the story is also created and compared with the model. The stories are classified into main, empathy and moral events.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "s632myiXtI", "reviewer": "Reviewer_rDq9", "summary": "", "questions": "Will you make the code available online?\nWill the dataset be available online?\nAlthough you list many different similarity scores and model performance metrics (tables 3, 4, 5) I'm unconvinced these are very significant. Of course, fine-tuning will make the model more domain-specific and this is where I think you should bring the results from section 6 to show that these are not just generic domain-adaptation improvements but actually useful from a real-world perspective where your model is the best at detecting empathic similarity (which I do believe it probably is). Could you present this disconnect better somehow to improve the narrative of your paper?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful for the reviewer’s positive comments in valuing our contributions in interdisciplinary approaches as well as the usefulness of the dataset for future work in empathy. Thank you for your comments on how “some of the concepts introduced in this paper are fairly novel as well as useful and interesting.” We also appreciate the additional references provided — we will include these in the camera ready!\n\n1.  _Clarifying core contributions and flow of the paper and role of user study_: \n* We apologize that the structure and flow of the paper muddled the contributions of the paper, and specifically the role of our human study. In summary, the core contributions of the work are (1) a new framework defining empathic similarity based on social and narrative psychology (2) a new dataset of personal stories annotated with story events, emotions, morals, and empathic similarity between pairs of stories, (3) automatic evaluation on our new task of empathic similarity retrieval and (4) a real-world user study demonstrating that our model significantly improves empathy towards people’s stories compared to off-the-shelf retrieval systems (see rebuttal 4 for full detailed list of contributions). \n\n    The user study in this work is crucial to showcase the promising real-world impact of our new task. Prior work's versions of human evaluations (Sharma et al. 2020) [1, 2] are humans verifying or ranking model outputs based on inputs from dev/test data. This provides a valuable signal of model quality, but isn't representative of how a model could be used in real-world applications (due to input distribution mismatch and lack of personal investment in the task). Our human evaluation is a full user study to see how the model performs given a story that the users told themselves, which is much more aligned with real-world applications. We demonstrate the applicability of the task to improve empathy towards retrieval of human stories, and because of this extensive human evaluation, we included separate hypotheses. The user study fits the overall narrative of why we collected this dataset, how it was used to develop this task, and why this task matters in the real world. We will be sure to improve the structure and narrative of the paper to make this clearer.\n* As you also mentioned, showing actual real-world usefulness of the models is important for the narrative of the paper, which is why we show in our human evaluation with 150 participants that stories retrieved using our model significantly improve empathy towards selected stories over current baselines such as SBERT story retrieval. This is particularly important given the highly personalized aspects of the study — participants wrote a story from their own life, and our approach, motivated by social psychological theory, was able to capture more empathically relevant features of their personalized stories and result in meaningful impact on their empathy levels. We will make sure to tie this discussion into the automatic evaluation results in the camera ready.\n\n2. _Significance of results_:\n* Good point on domain-specificity of the fine tuning task! The improvements we reported are statistically significant (***p* = 0.02** for BART finetuning improvements based on McNemar’s test on the accuracy scores, similarly we obtain ***p* = 0.0006** comparing Spearman’s correlation with/without fine-tuning on BART using Fisher’s z-transformation). We will make this clearer in the camera ready by indicating statistically significant elements in Table 4.\n\n3.  _Related work_: \n* Thank you for your detailed feedback, we will correct all of the structural elements of the paper and tighten up the related work earlier in the paper. In the camera ready, we will rearrange the related work and motivate our work in relation to prior work better, as suggested. In summary, few prior works on text-based empathy have looked at modeling empathy in two-way interpersonal settings for human-to-human connection (most focus on detecting empathy or generating empathetic utterances), and even fewer of these works have shown tangible outcomes in human studies. The authors believe this task is crucial given the increased loneliness and apathy between people over time [3]. With increasing polarization, personal experiences are a fundamental way people connect, yet existing social recommendation isn’t targeted for human-human connectivity and empathy. Empathically encoded story embeddings could be useful for a variety of NLP tasks, including retrieval, text generation, dialogue, and translation, for example in the following settings:\n    *  Using empathic reasoning to incorporate story retrieval in dialogue generation.\n    * Generating stories that users resonate with more in conversational AI\n    * Extending this work to multilingual settings and better understand translating experiences in ways that preserve empathic meaning\n    * Better understand cognitive insights, such as linguistic patterns of emotional communication\n    * Applications/building interactions that foster story sharing across geographic, ethnic, and cultural bridges, such as developing better social media recommendation or personalization.\n\n* We will make sure to ground these motivations in the related work and help support the significance of our results.\n\n4. _Reproducibility of work_:\n* The dataset we created is unique and adds a new contribution to the field (as mentioned in footnote 1 on page 2). The dataset will be made publicly available with all annotations, and we will release all code including annotation templates, pre-processing of data, model fine tuning, and evaluation code (anything used to replicate the entire study). We will also make all user study interfaces publicly available to allow replication of the user study.\n\n[1] Zhou, N. and Jurgens, D. 2020. Condolence and Empathy in Online Communities. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 609–626, Online. Association for Computational Linguistics.\n\n[2] Bao, J., Wu, J., Zhang, Y., Chandrasekharan, E., and Jurgens, D. 2021. Conversations Gone Alright: Quantifying and Predicting Prosocial Outcomes in Online Conversations. In Proceedings of the Web Conference 2021 (WWW '21). Association for Computing Machinery, New York, NY, USA, 1134–1145. \n\n[3] Buecker S., Mund M., Chwastek S., Sostmann M., and Luhmann M. Is loneliness in emerging adults increasing over time? A preregistered cross-temporal meta-analysis and systematic review. Psychol Bull. 2021 Aug;147(8):787-805.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The methodology is overall sound and some of the concepts introduced in this paper are fairly novel as well useful and interesting. The authors also manage to discuss the matter from an interdisciplinary perspective bringing in psychology and social studies work.\n\nThe dataset is also likely to be useful to other researchers specifically in empathy and intimacy-related projects, but also sentiment analysis and summarization projects.\n\nAdditionally, to the delight of CS people everywhere, the authors evaluate their model using BLEU, ROUGE, and METEOR.", "reasons_to_reject": "This paper is a little confused about what it wants to be and is in some ways two papers merged into one as exemplified by section 6 which is a standalone study with its own hypothesis, previous work, data, and method as well as results section and is followed by section 7 \"Further related work\" just before the \"Conclusions\" section. Although these issues are structural they confound the core message of the paper which should be the pairwise empathic dataset.\n\nI am also not entirely convinced of the results and their significance. At the very least, the improvements of the model presented here should be discussed more in-depth in the text. As it currently stands the link between the results in table 3-5 and figures 3 and 4 are confusing at best.", "questions_to_authors": "Will you make the code available online?\nWill the dataset be available online?\nAlthough you list many different similarity scores and model performance metrics (tables 3, 4, 5) I'm unconvinced these are very significant. Of course, fine-tuning will make the model more domain-specific and this is where I think you should bring the results from section 6 to show that these are not just generic domain-adaptation improvements but actually useful from a real-world perspective where your model is the best at detecting empathic similarity (which I do believe it probably is). Could you present this disconnect better somehow to improve the narrative of your paper?", "paper_topic_and_main_contributions": "This paper presents an empathic similarity dataset with pairwise annotations for 2000 pairs of stories from a pool of 1500 stories. The authors show that their finetuned model is better aligned with human interpretations of empathic similarity than previous models.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "One glaring omission in the reference list is work done by David Jurgens et al. on sympathy and empathy in online fora.\n\nZhou, N. and Jurgens, D., 2020, November. Condolence and empathy in online communities. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) (pp. 609-626).\n\nLahnala, A., Welch, C., Jurgens, D. and Flek, L., 2022. A critical reflection and forward perspective on empathy and natural language processing. arXiv preprint arXiv:2210.16604.\n\nBao, J., Wu, J., Zhang, Y., Chandrasekharan, E. and Jurgens, D., 2021, April. Conversations gone alright: Quantifying and predicting prosocial outcomes in online conversations. In Proceedings of the Web Conference 2021 (pp. 1134-1145).\n\nOther more NLP-focused work could also be added to strengthen the arguments made by the authors:\n\nLahnala, A., Welch, C. and Flek, L., 2022, May. CAISA at WASSA 2022: Adapter-tuning for empathy prediction. In Proceedings of the 12th Workshop on Computational Approaches to Subjectivity, Sentiment & Social Media Analysis (pp. 280-285).\n\nKumano, S., Ishii, R. and Otsuka, K., 2017, October. Comparing empathy perceived by interlocutors in multiparty conversation and external observers. In 2017 Seventh International Conference on Affective Computing and Intelligent Interaction (ACII) (pp. 50-57). IEEE.\n\nBoukricha, H., Wachsmuth, I., Carminati, M.N. and Knoeferle, P., 2013, September. A computational model of empathy: Empirical evaluation. In 2013 Humaine Association Conference on Affective Computing and Intelligent Interaction (pp. 1-6). IEEE.", "Typos_Grammar_Style_and_Presentation_Improvements": "I'm not entirely certain that the flow of the paper is the best. I understand the choice to place section 6 where it is, but it could very well be merged into previous sections as another hypothesis. Currently, section 6 reads like its own short paper so perhaps the authors could consider redistributing the information of section 6 into the other sections (intro, previous work, data & method, results etc).\n\nSimilarly, it is strange to see section 7 \"Further related work\" as the next to last chapter just before \"conclusions\". I suggest that baking section 6 into the whole paper includes moving section 7 as a subsection to section 2. Otherwise, the paper is just too disjointed and reads as if trying to cram two papers into one. If your paper is not accepted, you could consider resubmitting it elsewhere as two short papers rather than one long paper."}}, {"review_id": "3PrODBjAKV", "reviewer": "Reviewer_RNMN", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are thankful for the reviewer’s appreciation of our contributions in “a new and interesting task in conjunction of NLP and Social Psychology,” as well as our “new dataset for this task, which is carefully analyzed” and “first benchmark on this dataset.”\n\n1.  _Effectiveness of work relative to other related works_: \n* We apologize that the structure of the paper and the related work muddled the contributions of our work relative to prior literature.\nPrior works either focus on predicting empathy [1,2], improving empathy towards AI agents through improving empathetic dialogue generation [3], or understanding patterns of empathy online (Roshanei et al. 2019; Sharma et al. 2020) [4]. These existing tasks are typically one-way, looking at expression of empathy in utterances or generating more empathetic responses given a context. Our task is novel relative to these works because it focuses on pairwise empathy and modeling interpersonal dynamics to improve human-human connection. Furthermore, we validate our dataset, framework, and task in a user study with positive results and also briefly look at patterns of empathy (Roshanei et al. 2019; Sharma et al. 2020) [4] in section 6.3 of the paper. We will be sure to make this clearer in the camera ready.\n* Prior works (Sharma et al. 2020) [4] developed frameworks for empathy or social support that are psychologically inspired, but did not conduct real-world studies to scientifically validate frameworks or measure changes in cognitive and affective states of people.  As discussed in lines 523-526 of our paper, works that conducted studies to assess effect of story matching (Dinakar et al. 2012) report that only 58.3% found the matched story to be helpful if provided to the narrator of the original story. In contrast, we found that our model retrieved stories, users empathized with _significantly_ more than standard retrieval. \n* Since this is a new task, as a first evaluation benchmark for this task, making direct comparison of metrics is challenging — however we do show significance of improvements in both automatic evaluation and our user study. Relative to prior works, our methodology in data collection, annotation, and model training is similar and uses current, state-of-the-art trends in quantitative NLP (including comparisons of fine-tuning LLMs and few-shot prompting of very large language models). In addition, as part of our development, we also evaluated other models like GPT-2 and Flan-T5 and conducted extensive ablations and selected and presented only the best performing models (BART, GPT-3/3.5). We will make sure to include this in the camera ready.\n    * __Dataset__: While prior works (Sharma et al. 2020; Saldias et al. 2020) [5] have personal story datasets, none of them have empathy annotations suitable for our task. As such, we collect additional personal stories and combine these datasets into our evaluation.\n\n    * __Model training and automatic evaluation__: For fine-tuning, we used similar approaches to Sharma et al. 2020, and we used the same few shot prompting setup as Sap et al. 2022. Our automatic evaluation metrics are standard for semantic similarity and text generation tasks.\n* We will restructure the related work earlier in the paper, as you mentioned, and all of the above points about how we position our contributions relative to related work.\n\n2.  _Typos and presentation improvements_: Thank you for your detailed feedback! We will correct the typos you pointed out.\n\n[1] Lahnala, A., Welch, C., and Flek, L. 2022. CAISA at WASSA 2022: Adapter-Tuning for Empathy Prediction. In Proceedings of the 12th Workshop on Computational Approaches to Subjectivity, Sentiment & Social Media Analysis, pages 280–285, Dublin, Ireland. Association for Computational Linguistics.\n\n[2] Buechel, S., Buffone, A., Slaff, B., Ungar, L., and Sedoc, J. 2018. Modeling Empathy and Distress in Reaction to News Stories. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4758–4765, Brussels, Belgium. Association for Computational Linguistics.\n\n[3] Rashkin, H., Smith, E. M., Li, M., Boureau, Y. 2019. Towards Empathetic Open-domain Conversation Models: A New Benchmark and Dataset. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5370–5381, Florence, Italy. Association for Computational Linguistics.\n\n[4] Bao, J., Wu, J., Zhang, Y., Chandrasekharan, E., and Jurgens, D. 2021. Conversations Gone Alright: Quantifying and Predicting Prosocial Outcomes in Online Conversations. In Proceedings of the Web Conference 2021 (WWW '21). Association for Computing Machinery, New York, NY, USA, 1134–1145. \n\n[5] Giorgi, S., Zhao, K., Feng, A. H., & Martin, L. J. (2023). Author as Character and Narrator: Deconstructing Personal Narratives from the r/AmITheAsshole Reddit Community. Proceedings of the International AAAI Conference on Web and Social Media, 17(1), 233-244.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper introduces a new and interesting task in conjunction of NLP and Social Psychology to identify the similarity of two texts, specifically two personal stories, based on the extend of their narrator empathy to the other story. \n\nThe paper introduces a new dataset for this task., which is carefully analyzed and they also provide the first benchmark on this dataset.\n\nThe author(s) formulated the task with mathematical notations which helps in understanding the task and further development of the task by new research teams.\n\nThey did human evaluations to asses their model performance.", "reasons_to_reject": "My concerns have been addressed in the rebuttal.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper introduces a new task of similarity detection in personal stories based on the extend of empathy people have to each other stories instead of using raw semantic similarity.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "In section 7, further relation work, in paragraph 2, line 3, you used two usings."}, "extra_text": {}}, {"review_id": "groniG3mHd", "reviewer": "Reviewer_Pwhy", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the insightful comments and for your appreciation of our dataset as a resource “valuable for future work on empathic analysis” as well as your interest in our user study’s efforts to “conduct personalized evaluation of the model’s ability to retrieve similar stories the user can empathize with.”\n\n1.  _Summarization methodology_: \n* This is a great point, as summarization definitely distills the meaning of a story and could filter out valuable main event, emotion, and moral information, as you mention. We initially ran annotation on pairs of full stories, but found that each task took on average > 10 minutes and had very low agreement between annotators. Longer, more demanding tasks (e.g., reading two full stories and comparing them) can be challenging to run on MTurk due to worker attentiveness issues. Summarization drastically reduced the complexity of the task and made it more doable for annotators (average of 8.5 minutes per HIT). \n* We do not believe that summarizing filtered critical final features in our results: We ran experiments showing that there is great presence of the human rated main event, emotion, and moral in automated summaries. We found that cosine similarity of the _human annotated_ event, emotion, and moral to the ChatGPT summarized stories (using SBERT embeddings) are **0.66 for event, 0.64 for emotion, and 0.49 for moral**.  Concatenating the human-written summaries of event, emotion and moral, and comparing to the ChatGPT summaries gives us a **semantic similarity of 0.67**,  again suggesting there is high semantic overlap in human annotated features of the story and the automatically generated summaries\n* Nevertheless, we will run and include a more in-depth analysis and human evaluation of the quality of the ChatGPT summaries for the final version of the work. Ultimately, the empathic similarity labels after summarization still provided sufficient learning signal, which resulted in improved performance not only across automatic metrics, but also in people’s actual empathic responses during the user study.\n\n2. _Interrater Agreement_:\n* We aggregate annotations by averaging between the 2 raters. Averaging provides a good learning signal for our ultimate task of predicting empathic similarity, as most common disagreements are at most 1 likert point away (73% of points are at most 1 distance away). We will add agreement measures with this softer constraint in the paper to make this clearer.\n\n* Inter-annotator agreement is on average 0.82 for PPA and 0.21 for similarity ratings of empathic similarity, event, emotion, and moral similarity. We mention in lines 277-286 that this is in-line with other subjective, affect based tasks (Sap et al. 2017; Rashkin et al. 2018). For example, tasks on annotating power and agency (Sap et al. 2017) achieved .56 and .51 PPA and 0.34 KA and other work on annotating basic emotions and Maslow's needs achieved 0.7 - 0.91 PPA and 0.29 - 0.32 KA (Rashkin et al. 2018). Given the difficulty of our task (reading longer stories and projecting the mental state of 2 characters), our agreement is in line with prior work. \n\n3. _Human evaluation of empathy reasoning task_:\n* Thank you for your point on providing additional human evaluation of the empathy reasoning task. We could indeed conduct a human evaluation on the summaries, but the primary aim of our human study was to show how empathic similarity makes people feel more connected, indicating the larger downstream impact of the main task, so we resorted to automatic metrics for the auxiliary task. For the final work, we will consider running human evaluations to supplement the automatic evaluations in Table 5.\n\n4.  _Clarifying contributions_: \n* Thank you so much for your detailed feedback! We will make sure to include a list summarizing contributions at the end of the introduction. To summarize, the core contributions of this work are:\n    * A new framework/design approach for defining empathic similarity backed by social psychology and narrative psychology theory on empathic similarity that is validated by the results of a user study.\n    * The collection and annotation of a new dataset of personal stories annotated with story events, emotions, and morals, as well as empathic similarity between pairs of stories.\n    * A novel task and first evaluation benchmark for quantifying empathic similarity between personal stories, comparing current approaches, including fine-tuning and prompting LLMs.\n    * Automatic evaluation showing significant improvements of finetuning on our dataset for the new task of empathic similarity prediction.\n* A user study assessing the effect our model has on retrieving stories users empathize with __significantly more__ than off the shelf retrieval, improving empathy towards people’s stories in the real world. \nWe will make sure to include this in the camera ready to make the contributions clearer.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The collected dataset, is a resource that might prove valuable for future work on empathic analysis by utilizing personalized narratives of the users\n2. A user study that conducts a personalized evaluation of the model's ability to retrieve similar stories that the user can empathize. I found this user study particularly interesting.", "reasons_to_reject": "1. The paper mentions that it summarizes stories before presenting the pairs to the annotators. However, valuable information might be lost in the process. For example, the individual annotations of the first part of the annotation, like main events, emotional reaction, or moral, which are an important part of the framework defined for empathy similarity\n2. In addition, there is a very low agreement between the annotators, and it is not stated how the annotations are aggregated, or how the disagreements are solved\n3. Analysis for the empathy reasoning task is vague. It does not include any human evaluation and no further analysis except reporting automated metrics.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper addresses empathic resonance between people, by utilizing their personalized narratives.  They compare empathy similarity through people's stories by using three components of the stories: main event, emotional reaction, and moral. The authors introduce a new dataset, which contains empathic similarity annotations between pairs of stories. Moreover, they introduce two tasks: empathic similarity prediction and empathy reasoning. The main models that they utilize are SBERT, BART (with fine-tuning), and GPT3, ChatGPT (with few-shot learning). In addition, to the automatic evaluation, the authors also conduct a user study where the main goal is to evaluate the models for story retrieval.\n\nThe main contributions of the paper are as follows:\n- They introduce a framework to model empathic similarity, which focuses on the relationship between the narratives of a pair of people. \n- They release “EmpathaticStories”  a corpus of 1500 stories with data from three different sources.\n- A user study that involves personalized human evaluation of the models predictions", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "I would suggest that in the introduction, you include a paragraph where you clearly state your contributions."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a framework operationalizing empathic similarity (i.e., similarity between personal stories based on empathic resonance). It contributes 1) a dataset of personal stories annotated with features from the framework and pairs of stories annotated with empathic similarity scores, 2) a model fine-tuned to compute empathic similarity of story pairs, and 3) a user study showing that participants empathize more with model-retrieved stories.

There is strong consensus from the reviewers that this paper presents a valuable contribution. They appreciate that the paper draws on work outside NLP (e.g., social psychology) to engage meaningfully with empathy and its expression in narratives, and find the concepts and task introduced to be new and interesting. They find the study to be well-scoped and the methodology sound, and appreciate the concepts’ and dataset’s likely utility not only for researchers working directly on empathy-related questions, but also for researchers working on sentiment, summarization, and other tasks that would benefit from engagement with the expression of personal experiences.
Decision: Accept-Main