Unsupervised Extractive Summarization of Emotion Triggers
Year: 2023

Abstract:
Understanding what leads to emotions during large-scale crises is important as it can provide groundings for expressed emotions and subsequently improve the understanding of ongoing disasters. Recent approaches (Zhan et al., 2022) trained supervised models to both detect emotions and explain emotion triggers (events and appraisals) via abstractive summarization. However, obtaining timely and qualitative abstractive summaries is expensive and extremely time-consuming, requiring highlytrained expert annotators. In time-sensitive, high-stake contexts, this can block necessary responses. We instead pursue unsupervised systems that extract triggers from text. First, we introduce COVIDET-EXT, augmenting (Zhan et al., 2022)’s abstractive dataset (in the context of the COVID-19 crisis) with extractive triggers. Second, we develop new unsupervised learning models that can jointly detect emotions and summarize their triggers. Our best approach, entitled Emotion-Aware Pagerank, incorporates emotion information from external sources combined with a language understanding module, and outperforms strong baselines. We release our data and code at https://github.com/tsosea2/CovidET-EXT.

heading: 1 Introduction
text: Language plays a central role in social, clinical, and cognitive psychology (Pennebaker et al., 2003), and social media presents a gold mine for such analysis: people turn to social media to share experiences around challenges in their personal lives and seek diagnosis, treatment, and emotional support for their conditions (Choudhury and De, 2014; Gjurković and Šnajder, 2018). During crises, such as natural disasters or global pandemics, large-scale analysis of language on social media — both how people feel and what’s going on in their lives to lead to these feelings — can have a profound impact on improving mental health solutions as well
*Tiberiu Sosea and Hongli Zhan contributed equally.
as helping policymakers take better-informed decisions during a crisis.
Recent work (Zhan et al., 2022) taps into this broad challenge by jointly detecting emotions and generating a natural language description about what triggers them (triggers include both objective events and subjective appraisals of those events (Ellsworth and Scherer, 2003; Moors et al., 2013)). Trigger explanation is formulated as a supervised, abstractive summarization task that is emotion-specific. Unlike generic summarization however, due to the high cognitive load to provide judgments for each emotion, obtaining humanwritten summaries for this task is time-consuming and requires significant annotator training. This results in small, domain-specific datasets that are difficult to scale — especially in the face of new crisis events where the timing of such analysis is often pivotal.
This work instead takes a fully unsupervised approach such that we do not rely on any labeled data, thus becoming agnostic to distributional shifts in domain or types of crisis, and robust for time-
9550
critical events. We posit that emotion triggers can be summarized effectively in an extractive manner where unsupervised methods are well-suited; we thus tackle the challenge of simultaneous emotion prediction and trigger extraction.
For this new task, we first introduce COVIDETEXT, augmenting Zhan et al. (2022)’s COVIDET with manually annotated extractive summaries corresponding to each of their abstractive summaries. The result is a dataset of 1, 883 Reddit posts about the COVID-19 pandemic, manually annotated with 7 fine-grained emotions (from COVIDET) and their corresponding extractive triggers (Figure 1). For every emotion present in a post, our annotators highlight sentences that summarize the emotion triggers, resulting in 6, 741 extractive summaries in total. Qualitative analyses of the dataset indicate good agreement among the annotators, and followup human validations of the annotations also reveal high correctness. COVIDET-EXT provides an ideal test bed to facilitate the development of extractive (supervised or unsupervised) techniques for the tasks of emotion detection and trigger summarization in crisis contexts.
We propose Emotion-Aware PageRank (EAP), a novel, fully unsupervised, graph-based approach for extractive emotion trigger summarization from text. The core of our method is to decompose the traditional PageRank (Page et al., 1999) ranking algorithm into multiple biased PageRanks (Haveliwala, 2003), one for each emotion. To bias our model towards various emotions, our approach harnesses lexical information from emotion lexicons (Mohammad and Turney, 2013; Mohammad, 2018). Critically, unlike previous graph-based unsupervised approaches (Mihalcea and Tarau, 2004; Liu et al., 2010; Gollapalli and Caragea, 2014; Florescu and Caragea, 2017; Patel and Caragea, 2021; Singh et al., 2019), which represent the text as a bag-of-words or word embeddings, EAP incorporates a language understanding module leveraging large language models to ensure that the summaries for an emotion are coherent in the context of that emotion. Results on our COVIDET-EXT indicate the effectiveness of our EAP, which significantly pushes the Rouge-L score of our summaries by an average of 2.7% over strong baselines.
Our contributions are as follows: 1) We introduce COVIDET-EXT, a manually annotated benchmark dataset for the task of emotion detection and trigger summarization. 2) We propose
Emotion-Aware PageRank, a variation of PageRank that combines a language understanding module and external emotion knowledge to generate emotion-specific extractive summaries. 3) We carry out a comprehensive set of experiments using numerous baselines to evaluate the performance on COVIDET-EXT and show that our proposed EAP significantly outperforms strong baselines.

heading: 2 Background and Related Work
text: Emotion Tasks. Most of the prior work on emotions on social media focuses solely on detecting emotions or emotional support from text (Wang et al., 2012; Biyani et al., 2014; Abdul-Mageed and Ungar, 2017; Khanpour et al., 2018; Khanpour and Caragea, 2018; Demszky et al., 2020; Desai et al., 2020; Sosea and Caragea, 2020; Adikari et al., 2021; Calbi et al., 2021; Kabir and Madria, 2021; Beck et al., 2021; Mohammed Abdulla et al., 2019; Sosea and Caragea, 2021; Hosseini and Caragea, 2021a,b; Saakyan et al., 2021; Ils et al., 2021; Sosea et al., 2022; Sosea and Caragea, 2022a,b). Our task is directly related to emotion cause extraction (Gao et al., 2015; Gui et al., 2016; Gao et al., 2017) which focused on identifying phrase-level causes from Chinese news or micro-blogs, which are distinct from the spontaneous writing on social media. In our context, similar to the work of Zhan et al. (2022), what triggers an emotion includes both what happened and how the writer appraised the situation. A major difference of our work from Zhan et al. (2022) is that we consider extractive summaries instead of abstractive and take a fully unsupervised perspective, eliminating the reliance on labeled data. For a comprehensive overview of COVIDET introduced by Zhan et al. (2022), refer to Appendix §A.
Unsupervised Extractive Summarization. Extractive summarization aims to condense a piece of text by identifying and extracting a small number of important sentences (Allahyari et al., 2017; Liu and Lapata, 2019; El-Kassas et al., 2021) that preserve the text’s original meaning. The most popular approaches in unsupervised extractive summarization leverage graph-based approaches to compute a sentence’s salience for inclusion in a summary (Mihalcea and Tarau, 2004; Zheng and Lapata, 2019). These methods represent sentences in a document as nodes in an undirected graph whose edges are weighted using sentence similarity. The sentences in the graph are scored and ranked using node cen-
trality, computed recursively using PageRank (Page et al., 1999). In contrast, our EAP considers words instead of sentences as nodes in the graph and employs multiple separate biased PageRanks (Haveliwala, 2003) to compute an emotion-specific score for each word, which is combined with a sentencesimilarity module to produce one sentence score per emotion, indicating the salience of the sentences under each emotion.

heading: 3 Dataset Construction
text: Since there is no annotated data for extractive emotion triggers summarization in crisis contexts, we first bridge this gap by extending COVIDET, Zhan et al. (2022)’s abstractive-only dataset with extractive trigger summaries. Doing so (a) creates benchmark data for extractive systems; (b) allows in-depth analyses to understand how and when emotion triggers are expressed on social media. This will also create a parallel abstractiveextractive dataset for future research. We name our new dataset COVIDET-EXT (COVIDET {extractive, extension}).
Annotating Emotion Triggers. Given a post from COVIDET annotated with an emotion e, we ask annotators to highlight sentences in the post that best describe the trigger for e. An overview of our annotation scheme can be viewed in Appendix §B. We recruit both undergraduate students (in a Linguistics department) as well as prequalified crowd workers (from the Amazon Mechanical Turk) for this task.1 Each post is annotated by two annotators. We monitor the annotation quality and work with the annotators during the full process. Similar to COVIDET, the test set is annotated by undergraduate students.
Benchmark Dataset. We follow the benchmark setup in Zhan et al. (2022) with 1, 200 examples for training, 285 examples for validation, and 398 examples for testing. If two annotators highlight different sentences as triggers for the same emotion, we consider both sets of sentences as the gold summaries and evaluate them using multi-reference ROUGE. We anonymize COVIDET-EXT. Note that since we explore unsupervised methods, the training set is not used in our summarization models. Nevertheless, we emphasize that while the fo-
1These crowd workers have an ongoing working relationship with our group and have prior experience in related complex tasks, and we make sure they are paid at least $10/hr.
cus of this work is the unsupervised setup, we hope that COVIDET-EXT can spur further research into both supervised and unsupervised methods, hence we maintain the splits in Zhan et al. (2022). For completeness, we carry out experiments in a fully supervised setup in Appendix §F.
Human Validation. We validate the annotated extractive summaries of emotion triggers in COVIDET-EXT through inspections from thirdparty validators on the Amazon Mechanical Turk crowdsourcing platform. A subset of our training data including 300 randomly selected examples which contain annotations of extractive summaries of emotion triggers are validated. Given an annotated extractive trigger summary, we first ask the validators whether the summary leans towards the annotated emotion. It yes, we ask the validator to further point out if the trigger — rather than the emotion itself — is present in the summary. The percentage of examples that validators confirm for the two steps is shown in Table 1. Overall, the human validation results showcase moderately high correctness in the annotations of COVIDET-EXT, considering the subjective nature of our task.2
Inter-Annotator Agreement. We measure the inter-annotator agreement between two extractive trigger summaries for the same emotion in a post, as shown in Table 2. Results show that, within the examples where we find emotion overlaps, 29.9% of the extractive summaries of triggers for the same emotion share completely identical annotations from both annotators, and 25.6% have partial sentence-level overlaps. In total, we find overlaps
2The same sentence can be interpreted to be triggers for different emotions. For example, the sentence “I miss my room and I dont have many clothes or my meds here, but hes hitting these mics every fucking night and Im scared of contracting it” expresses anger, sadness, and fear simultaneously under the same context.
in 55.5% of the summaries, and the experts who were responsible for the test set (65.8%) have more overlapping summaries than the crowd workers who were responsible for the training and validation sets (52.3%). Furthermore, the average Fleiss’ kappa (Fleiss, 1971; Randolph, 2005) is 0.89 across all the emotions in COVIDET-EXT. This suggests substantial agreement among our annotators.
In addition, we also employ automatic metrics including self-BLEU (with smoothing methods 1) and self-ROUGE to capture the overlap between annotators’ summaries. To establish a baseline, we report these metrics between the annotators’ work and a randomly selected sentence from the original post. We repeat this process five times. Results reveal that both the self-BLEU and self-ROUGE of our annotations significantly outperform that of the random baseline (as shown in Table 2). We also observed higher values of these measures for student annotators compared with crowd workers. (c.f. Appendix §D). These results indicate strong accordance among our annotators.
Dataset Statistics. Here we elaborate on the overview of COVIDET-EXT. On average, there are 1.35 sentences (std.dev = 0.79) consisting of 32.54 tokens (std.dev = 20.68) per extractive summary of emotion trigger in COVIDET-EXT. As shown in Figure 2, when broken down into unique trigger sentences, fear has the most trigger sentences in the dataset, closely followed by anticipation. On the other hand, trust has the lowest number of trigger sentences. This can be attributed to the calamitous nature of the domain of our dataset. Besides, unlike generic news summarization (Fabbri et al., 2021), the emotion-trigger extractive summarization task is not lead-based. This is manifested through our scrutiny of the position of emotion trigger sentences in the original posts (Figure 6 and Figure 7, Appendix §E), where a large number of triggers cluster in the later parts of the post.
Additional analyses of COVIDET-EXT can be found in Appendix §E.
Emotion Explicitness. To examine the explicitness of emotions in the extractive summaries of emotion triggers, we apply EmoLex (Mohammad and Turney, 2013), an English lexicon for the Plutchik-8 primary emotions. Specifically, for the extractive summaries of triggers to a certain emotion e, we measure the average ratio of e’s words in EmoLex being present in the sentence-level lem-
matized summaries. The results are presented in Figure 2. Interestingly, we notice that sadness is the most explicit emotion in the annotated extractive summaries of triggers in our dataset, while anger is the most implicit one.

heading: 4 Unsupervised Extractive Summarization
text: In this section we introduce Emotion-Aware Pagerank (EAP), our fully unsupervised, graph-based, emotion trigger extractive summarization method that incorporates information from emotion lexicons to calculate a biased PageRank score of each sentence in a post. EAP then fuses this score with an additional similarity-based sentence-level score that ensures the summary for a specific emotion e does not diverge in meaning from other summaries of the same emotion e. We show an overview of our model architecture in Figure 3.
Task Formulation. Let P be a Reddit post. P is composed of an ordered sequence of n sentences: P = {s1, s2, ..., sn}. Generic extractive summarization aims to output an ordered set of sentences S with S ⊂ P that captures the essence of post P . In our emotion trigger summarization, however, we aim to generate multiple extractive summaries conditioned on the expressed emotions. To this end, we are interested in a set of summaries Semo = {Se1 , Se2 , ..., Sem} where m is the total number of emotions present in P and Sei is the summary of the triggers that lead to the expression of emotion ei with Sei ⊂ P . Note that P usually conveys a subset of emotions, in which case the summaries for the emotions that are not present in text are empty.
Graph Construction. We build an undirected graph G = (V,E), where V is vocabulary set of words. To build V we employ various processing and filtering techniques. First, we only select nouns, adjectives, verbs, adverbs and pronouns and remove any punctuation. Next, we stem all the selected words to collapse them in a common base form. Finally, we remove infrequent words which appear less than 20 times in the entire training set. The remaining words form the vocabulary V . A pair of words (wi, wj) ∈ E defines an edge between wi and wj and the operator β(wi, wj) denotes the weight of edge (wi, wj). We compute the weight of an edge in our graph using word co-occurences in windows of text. Given a window size of ws, we say that two words wi and wj co-occur together if the number of words between them in text is less than ws. We build a co-occurence matrix C of size |V | × |V | from the documents in our training set where Cij is the number of times words wi and wj co-occur together. Using C we simply define the weight of an edge as:
β(wi, wj) = 2× Cij
∑|V | k=0(Cik + Cjk)
(1)
Intuitively, the more frequently two words co-occur together, the higher the weight of the edge between them becomes.
Emotion Decomposition. In PageRank, the importance or relevance R(wi) of an arbitrary word wi is computed in an iterative fashion using the following formula:
R(wi) = λ |V |∑
k=1
β(wk, wi)R(wk) + (1− λ) 1
|V | (2)
where |.| is the set size operator and λ is the damping factor, a fixed value from 0 to 1 which measures the probability of performing a random jump to any other vertex in the graph. The idea of PageRank is that a vertex or word is important if other important vertices point to it. The constant term 1|V | is called a random jump probability and can be viewed as a node preference value, which in this case assigns equal weights to all the words in the graph, indicating no preference.
In this current formulation, the PageRank model calculates the weights of words irrespective of the expressed emotion. We claim that for our purpose words should bear different importance scores in different emotion contexts. For example, the word agony should have a higher importance in the context of sadness or fear than in the context of joy.
To this end, we propose to decompose the text into multiple components, one for each emotion, where the relevance of a word differs from component to component. Biased PageRank (Haveliwala, 2003) is a variation of PageRank where the second term in Equation 2 is set to be non-uniform, which can influence the algorithm to prefer particular words over others. We propose to run a separate biased PageRank for each emotion and leverage a custom importance function ie(wi) that yields high values for words that are correlated with an emotion e and low values otherwise. For-
mally, the relevance computation for the PageRank corresponding to emotion e becomes:
Re(wi) = λ |V |∑
k=1
β(wk, wi)Re(wk)+(1−λ) ie(wi)
N
(3)
where N is a normalization factor such that∑ w∈V ie(w) N = 1. Since the model prefers those vertices with higher random jump probabilies, using an accurate importance function ie(wi) for emotion e can lead to accurate relevance scores in the context of e. We define this function using the NRC emotion intensity (Mohammad, 2018) lexicon. EmoIntensity associates words with their expressed emotions and also indicates the degree of correlation between a word and a particular emotion using real values from 0 to 1. For example, outraged has an intensity for anger of 0.964 while irritation has an intensity of 0.438. In our context, assigning importance values using intensity is appropriate since a sentence containing high intensity words for an emotion e is more likely to be relevant in the context of e compared to a sentence containing lower intensity words. Denoting the set of words in EmoIntensity correlated with emotion e by Ie, all words w ∈ Ie also come with intensity value annotations denoted by inte(w). Therefore, we define the importance function as:
ie(w) = { inte(w) if w ∈ Ie c if w ∈ V \ Ie
(4)
where c is a constant that we find using the validation set. Since our summaries are at the sentence level, we simply score a sentence si as the average relevance of its words:
Re(si) =
∑ wj∈si Re(wj)
|si| (5)
Encoding the meaning. A major drawback of prior graph-based approaches is that they exclusively represent the input as a bag-of-words, ignoring the structure of text. We propose to solve this drawback by introducing a language model-based component to encode the meaning of a sentence. Our component is based on the assumption that a sentence s that is highly relevant for an emotion e should be similar in meaning to other sentences si relevant to e. We capture this property by scoring each sentence based on its similarity with other important (i.e., in the context of e) sentences. We
leverage the popular Sentence-BERT (Reimers and Gurevych, 2019) model, which produces meaningful sentence embeddings that can be used in operations such as cosine similarity. Given a sentence si, let si be its embedding and sim(si, sj) be the cosine similarity between the embeddings of sentences si and sj . Denoting by T the set of sentences in the entire dataset, we score si in the context of emotion e as follows:
Me(si) =
∑ s∈T sim(si, s) ∗ Re(s)
|T | (6)
Intuitively, Me(si) yields high values if si is similar in meaning to sentences relevant in the context of emotion e.
Constructing the Summaries. Given a post P = {s1, s2, ..., sn}, we first combine the meaning and the relevance scores into a final, sentence level, per-emotion score, which we use to score every sentence si in P along all the emotions:
Fe(si) = Re(si) ∗Me(si) (7) We use this per-emotion score to rank the sentences in the post P . For an emotion e, we only select the sentences si where Fe(si) > t to be part of the final summary for e. t is a threshold value that we infer using our validation set. Note that given P , we compute the score Fe for every emotion e. In the case that none of the sentences in P exceed the threshold for a particular emotion, we consider that the emotion is not present in the post (i.e., we do not generate a summary).

heading: 5 Experiments and Results
text: In this section, we first introduce our emotionagnostic and emotion-specific baselines. Next, we present our experimental setup and discuss the results obtained by EAP against the baselines.
Emotion-agnostic baselines. We explore two standard heuristic baselines, namely 1) Extracting the first sentence in the post (1 sent) and 2) Extracting the first three sentences in the post (3 sent). Next, we design three graph centrality measurebased methods: 3) PacSum (Zheng and Lapata, 2019), 4) PreSum (Liu and Lapata, 2019) and wordlevel 5) TextRank (Mihalcea and Tarau, 2004). Note that these methods are emotion-oblivious and the generated summary will be identical for different emotions.
Emotion-specific baselines. We first employ two lexical-based methods: 6) EmoLex - we use the EmoLex (Mohammad and Turney, 2013) lexicon to identify lexical cues that indicate the expression of emotions. If a sentence contains a word that is associated with an emotion e, we consider the sentence to express e. The final summary for e contains all sentences expressing e. 7) EmoIntensity - we leverage the NRC Affect Intensity Lexicon (Mohammad, 2018) to build a more fine-grained approach of identifying if a sentence expresses an emotion or not. For each sentence and emotion, we calculate the average emotion word intensity and compare it to a pre-defined threshold t. If the average intensity for e is higher than t we label the sentence with e. t is a tunable parameter that we select based on our validation set performance.
Finally, we leverage models trained on emotion detection datasets to build our emotion-specific summaries. For a post P , we use our model to make predictions on each sentence in P and build summaries by concatenating sentences that express the same emotions. We mainly experiment with a model trained on the 8) GoEmotions (Demszky et al., 2020) dataset.
Experimental Setup. We carry out our experiments on an Nvidia A5000 GPU. We use the HuggingFace Transformers (Wolf et al., 2019) library for our Sentence-BERT implementation and we
will make the code for our methods and data available for reasearch purposes. We report the performance in terms of Rouge-2 and Rouge-L (Lin, 2004) to evaluate the summarization performance. Additionally, we also calculate the performance in terms of F1 and show the results in Appendix I. We provide extensive details about the hyperparameters used in EAP and the baselines, such as our various thresholds and constants in Appendix §G.
Results. We show the results obtained in Table 3. First, we note that emotion-specific approaches outperform the emotion-oblivious methods considerably. Notably, EmoIntensity outperforms PacSum by an average of 1.1% in Rouge-2. Among the emotion-specific baselines, EmoIntensity, which uses the intensity of emotion words to extract relevant sentences for a particular emotion obtains good performance, outperforming the EmoLex method by 5.1% Rouge-2 on disgust and 3.3% on fear. This result emphasizes that having a degree of association between a word and an emotion (i.e., the intensity) is a stronger signal than the plain word-emotion association in our emotion-based extractive summarization context.
EAP consistently yields the highest results both in terms of Rouge-2 and Rouge-L compared to the other approaches. Concretely, we obtain an average improvement of 2.7% in Rouge-L and 2.5% in Rouge-2 score over our strongest EmoIntensity baseline. For example, on anger and joy we see improvements in Rouge-2 of 1.7% and 6.3% respectively. Moreover, our emotion-aware PageRank considerably outperforms TextRank (Mihalcea and Tarau, 2004) by as much as 5.5% Rouge-L and 4.5% Rouge-2 on average.
Emotion Detection. While EAP shows strong results in our emotion trigger summarization experiments, we want to evaluate our approach in a traditional emotion detection task. To this end,
highlighted.
we ask how well EAP can detect emotions at the post level. Given a post P, we label the post with emotion e if we identify any sentence s ∈ P as a summary for e. If no sentence is selected to be included in the summary, we consider that EAP does not predict e.
We show the results obtained in Table 4, where we compare EAP to lexical methods (EmoLex and EmoIntensity) and a domain adaptation method, which trains a BERT (Devlin et al., 2019) model on the GoEmotions dataset (Demszky et al., 2020). We observe that EAP consistently outperforms prior work on all the emotions by an average of 0.9% in F1 score. Notably, we see 1.5% improvements in F1 on fear and 1.9% on anticipation.
Ablation Study. We perform a thorough ablation study to tease apart and analyze the components lead to the success of EAP. First, we analyze the influence of emotion intensity on the performance of the model. Here, we slightly modify the importance function from Equation 4 to a constant value. Instead of using the variable inte(w) we use a constant value ce where ce > c. Intuitively, we still bias the model towards a particular emotion e, however, every word associated with e weighs equal in this ablated version of EAP. We denote this modifi-
cation of the algorithm by -int. Second, we remove the meaning score Me from our algorithm and use only the word-based relevance Re. This approach is denoted by -sim. We also analyze the behaviour of EAP when removing both components.
We show the results obtained in Table 5. Removing emotion intensity leads to a performance degradation of 1% in Rouge-L while the lack of our similarity module decreases the performance by 1.2% in Rouge-L. Removing both further decreases the performance by 2.9% in Rouge-2. These results emphasize that both similarity and intensity are core components of EAP and both consistently contribute to its success.
Anecdotal Evidence. To offer additional insights into our EAP, we provide anecdotal evidence in Figure 4, where we show a post expressing both joy and fear. We indicate for each word both its relevance for joy and for fear. Additionally, we show the meaning score for each sentence and emotion. Interestingly, we observe that the scores produced by our model are very relevant. For instance, protection has a very large value for joy of 0.531 and a very small value of 0.076 for fear. Along the same lines, worried has a relevance of 0.523 for fear and 0.074 for joy. The similarity scores are also accu-
rate. For example, glad I am fully vaccinated has a score for joy of 0.463, 9 times as large of the score of the same sentence for fear. We show additional analysis on the effect of the most relevant terms on EAP performance in Appendix §H.

heading: 6 Conclusion
text: We introduce COVIDET-EXT, a new benchmark dataset composed of 1, 883 Reddit posts annotated for the task emotion detection and extractive trigger summarization in the context of the COVID-19 pandemic. Our proposed Emotion-Aware Pagerank approach yields strong results on our datasets, consistently outperforming prior work in an unsupervised learning context. In the future, we plan to study abstractive trigger summarization from an unsupervised point of view to bridge the gap between the extractive and abstractive summarization performance.
Limitations
Since our EAP builds its graph representation from social media data, our method may carry inductive biases rooted in this type of data. Moreover, note that the scope of our study is limited to English social media posts and our approach does not consider inputs larger than 512 tokens. Therefore using our approach in long document summarization may be challenging. Finally, the general applicability of EAP in a different domain is highly dependent on the existence of high-quality lexicons for the domain in question, which may not be available.

heading: Acknowledgements
text: This research was partially supported by National Science Foundation (NSF) grants IIS-1912887, IIS-2107487, ITE-2137846, IIS-2145479, IIS2107524, IIS-2107487. We thank Jamie Pennebaker for useful discussions and comments. We also thank our reviewers for their insightful feedback and comments.

heading: C Crowd Workers
text: Both groups of annotators for COVIDET-EXT come from the United States. The crowd workers are recruited from the Amazon Mechanical Turk crowdsourcing platform, with restrictions that their locale is the US and that they have completed 500+ HITs with an acceptance rate of at least 95%. The undergraduate students are hired from a university in the United States.
D Inter-annotator Agreement Among Undergraduate Students and Crowd Workers
As shown in Table 6, the inter-annotator performance of the undergraduate students consistently exceeds the crowd workers.
E Additional Analyses of COVIDET-EXT
Trigger Positions. We examine the position of the emotion trigger sentences in the original posts. The sentence-level distribution of the annotated triggers is reported in Figure 6. Results reveal that the trigger sentences spread evenly across the posts, with a large number of triggers clustering in the later parts of the post. This means that the emotion-trigger extractive summarization task is not lead-based, unlike generic news summarization (Fabbri et al., 2021; Sebastian et al., 2019). This is especially true for anticipation, as demonstrated in Figure 7.
Trigger Components. In addition to the explicitness of emotion triggers, we also examine the syntactic components of the extractive summaries of emotion triggers. Results are shown in Figure 8. We observe that nouns and verbs take up the majority of triggers, closely followed by the use of pronouns.
Pronoun Distributions. Psycho-linguistic studies reveal that the analysis of function words such as pronouns can disclose psychological effects of life experiences and social processes (Campbell and Pennebaker, 2003; Tausczik and Pennebaker, 2010; Pennebaker et al., 2014; Seraj et al., 2021; Singh et al., 2018). Specifically, overusing the firstperson singular pronouns may imply a high level of self-involvement, whereas the increased use of other pronouns may signify improvement of social engagement (Cohn et al., 2004; Simmons et al.,
2008; Kumari and Singh, 2017). We evaluate the percentage of personal pronoun usage per annotated emotion trigger sentence. In particular, we discover an inverse correlation between first-person singular pronouns (e.g., I, me, my, mine, myself ) and second-person pronouns (e.g., you, your, yours, yourself, yourselves). We provide the average percentage of the personal pronouns per emotion trigger in Figure 9. Further statistical tests reveal negative Pearson correlations between the percentage distribution of first-person singular pronouns and second-person pronouns in each emotion (with substantial significance in all 7 emotions; shown in Table 7). We note that when expressing negative emotions such as sadness and fear, authors used more first-person singular pronouns in triggers. On the other hand, authors used more second-person pronouns in expressing the triggers for positive emotions like joy and trust. The inverse correlation between first-person singular pronouns and second-person pronouns suggests more self-involvement in negative emotions and more social engagement in positive emotions in COVIDET-EXT.
Topical Variations. To better interpret the annotated emotion triggers, we train a multi-class bag-of-words logistic regression model to predict the emotion label of each annotated extractive emotion trigger sentence. The trained model’s weights pertaining to each class of emotions are then extracted to locate the tokens that are most indicative of each emotion. The multi-class logistic regression model achieved a micro F1 score of 0.33 after
training and evaluating on our benchmark dataset. The most indicative tokens associated with each emotion are reported in Table 8.
Connections to COVIDET. To understand the ties between COVIDET-EXT and COVIDET, we measure the self-BERTScore between the extractive summaries of triggers from COVIDETEXT and the abstraction summaries of triggers from COVIDET. Results reveal that the average BERTScore F1 is 0.872 between the extractive and abstractive summaries, indicating strong correlations between the two datasets.
Same Triggers for Different Emotions. The status of overlapping trigger sentences for different emotions is shown in Figure 10. Specifically, we measure the percentage of sentences that are triggers for an emotion i that are also triggers for emotion j in COVIDET-EXT.

heading: F Supervised Extractive Summarization
text: Although our focus is exclusively on unsupervised approaches to eliminate the reliance on labeled data, we note that Covid-EXT can be a suitable benchmark for developing supervised methods as well. In this section, we compare two supervised methods against our unsupervised EAP. We experiment with two methods for emotion trigger extraction. 1) First, we experiment with the BART-FTJOINT (Zhan et al., 2022) model which is trained to jointly predict emotions and their summary. We train this model on the training set of Covid-EXT in a supervised manner. Second we employ a simple 2) BERT (Devlin et al., 2019) classifier that is trained in a supervised manner to detect emotions at sentence level. We consider as positive examples the sentences that are included in the summary,
and negative examples the rest of the sentences. Note that we train 7 different models, one for each emotion.
We show the results obtained in Table 9. We observe that BART-FT-JOINT outperforms our EAP considerably by 1.5% in Rouge-L score. However, we see that the BERT-based approach is much closer to the performance of the unuspervised EAP, outperforming it by less than 1% in Rouge-L and F1.

heading: G Hyperparameters
text: In this section we detail the values of the hyperparameters used and the search space considered in the development of our EAP. First in terms of the constant c in Equation 4, we experiment with values in the range 0.1 → 0.5 but observed that 0.1 works well. We mentioned that the minimum frequency of a word necessary for selection in our vocabulary V is 20. We also experimented with other values ranging from 5 to 50. The threshold t from Equation 7 is emotion-specific and inferred using the validation set. We experiment with values between 0.2 and 0.7 and observed that 0.35 works well in general.

heading: H Model Analysis
text: To offer additional insights into our approach, we show in Figure 11 an analysis on the effect of the top relevant terms on the performance of EAP. For
each emotion, we experiment with completely dropping the top k most relevant terms (i.e., words) in the graph, with k ranging from 1 to 40 and report the average performance obtained. This analysis can be seen as a way to measure the reliance of EAP and the top relevant words. We observe that the performance drops considerably while dropping the first 28 terms and the starts to plateau.

heading: I Extractive Summarization Results in terms of F1
text: In Table 10 we present the performance on extractive summarization in terms of F1. While Rouge captures the overlap between extracted summaries and human references at word level, F1 measures the number of extracted sentences from the post that are correctly part of the gold summary (human
references). Specifically, we compute F1 as if we dealt with a traditional classification problem. For every emotion, the sentences belonging to the trigger summaries are positive examples, and all the other sentences are negative examples. If our EAP model selects a sentence that does not appear in the trigger summary, we view it as a false positive. On the other hand, if our EAP model does not extract a sentence which belongs to the trigger summary, we count it as a false negative. We calculate F1 as the harmonic mean between precision and recall.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: Limitations Section

heading: 3 A2. Did you discuss any potential risks of your work?
text: Limitations Section

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: Left blank.
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? Section 3

heading: 3 B1. Did you cite the creators of artifacts you used?
text: Section 3

heading: 3 B2. Did you discuss the license or terms for use and / or distribution of any artifacts?
text: Section 5
B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Not applicable. Left blank.

heading: 3 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any
text: information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Section 3

heading: 3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and
text: linguistic phenomena, demographic groups represented, etc.? Section 3
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Section 3
C 3 Did you run computational experiments? Section 5

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? Section 5 + Appendix G
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? Section 5 + Appendix

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Section 5

heading: 3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Section 5 experimental setup
D 3 Did you use human annotators (e.g., crowdworkers) or research with human participants? Section 3

heading: 3 D1. Did you report the full text of instructions given to participants, including e.g., screenshots,
text: disclaimers of any risks to participants or annotators, etc.? Section 3
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.

heading: 3 D4. Was the data collection protocol approved (or determined exempt) by an ethics review board?
text: This data collection is reviewed and exempted by the IRB board of our institution

heading: 3 D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data?
text: Appendix C

heading: Future Work
text: In the future, we plan to study abstractive trigger summarization from an unsupervised point of view to bridge the gap between the extractive and abstractive summarization performance. Limitations
Since our EAP builds its graph representation from social media data, our method may carry inductive biases rooted in this type of data. Moreover, note that the scope of our study is limited to English social media posts and our approach does not consider inputs larger than 512 tokens. Therefore using our approach in long document summarization may be challenging. Finally, the general applicability of EAP in a different domain is highly dependent on the existence of high-quality lexicons for the domain in question, which may not be available