Data Collection & Annotation ::: Adjudication of Annotations ::: Longest Common Subsequence Rule.	This rule is only applied two different intersections are the most common (previous rule), and these two intersect. We then accept the longest common subsequence. Revisiting the example for deciding on the cause role with the annotations “by landing their helicopter in the nature reserve”, “by landing their helicopter”, “landing their helicopter in the nature reserve”, “a couple infuriated officials”, “infuriated” the adjudicated gold is “landing their helicopter in the nature reserve”. Table TABREF27 shows through examples of how each rule works and how many instances are “solved” by each adjudication rule.

Data Collection & Annotation ::: Annotation Procedure ::: Quality Control and Results	To control the quality, we ensured that a single annotator annotates maximum 120 headlines (this protects the annotators from reading too many news headlines and from dominating the annotations). Secondly, we let only annotators who geographically reside in the U.S. contribute to the task. We test the annotators on a set of $1,100$ test questions for the first phase (about 10% of the data) and 500 for the second phase. Annotators were required to pass 95%. The questions were generated based on hand-picked non-ambiguous real headlines through swapping out relevant words from the headline in order to obtain a different annotation, for instance, for “Djokovic happy to carry on cruising”, we would swap “Djokovic” with a different entity, the cue “happy” to a different emotion expression. Further, we exclude Phase 1 annotations that were done in less than 10 seconds and Phase 2 annotations that were done in less than 70 seconds.

Related Work	Our annotation is built upon different tasks and inspired by different existing resources, therefore it combines approaches from each of those. In what follows, we look at related work on each task and specify how it relates to our new corpus.

Analysis ::: Inter-Annotator Agreement ::: Roles	Table TABREF41 presents the mean of pair-wise inter-annotator agreement for each role. We report average pair-wise Fleiss' $\kappa $, span-based exact $\textrm {F}_1$ over the annotated spans, accuracy, proportional token overlap, and the measure of agreement on set-valued items, MASI . We observe a fair agreement on the open annotation tasks. The highest agreement is for the role of the Experiencer, followed by Cue, Cause, and Target. This seems to correlate with the length of the annotated spans (see Table TABREF42). This finding is consistent with Kim2018. Presumably, Experiencers are easier to annotate as they often are noun phrases whereas causes can be convoluted relative clauses.

Data Collection & Annotation ::: Sampling Headlines	We stratify the remaining headlines by source (150 headlines from each source) and subsample equally according to the following strategies: 1) randomly select headlines, 2) select headlines with high count of emotion terms, 3) select headlines that contain named entities, and 4) select the headlines with high impact on social media. Table TABREF16 shows how many headlines are selected by each sampling method in relation to the most dominant emotion (see Section SECREF25).

Related Work ::: Reader vs. Writer vs. Text Perspective	Studying the impact of different annotation perspectives is another little explored area. There are few exceptions in sentiment analysis which investigate the relation between sentiment of a blog post and the sentiment of their comments  or model the emotion of a news reader jointly with the emotion of a comment writer . Fewer works exist in the context of emotion analysis. 5286061 deal with writer's and reader's emotions on online blogs and find that positive reader emotions tend to be linked to positive writer emotions. Buechel2017b and buechel-hahn-2017-emobank look into the effects of different perspectives on annotation quality and find that the reader perspective yields better inter-annotator agreement values.

Analysis ::: Inter-Annotator Agreement ::: Emotion	We use Fleiss' Kappa ($\kappa $) to measure the inter-annotator agreement for closed questions , . In addition, we report the average percentage of overlaps between all pairs of annotators (%) and the mean entropy of annotations in bits. Higher agreement correlates with lower entropy. As Table TABREF38 shows, the agreement on the question whether a headline is emotional or not obtains the highest agreement ($0.34$), followed by the question on intensity ($0.22$). The lowest agreement is on the question to find the most dominant emotion ($0.09$). All metrics show comparably low agreement on the closed questions, especially on the question of the most dominant emotion. This is reasonable, given that emotion annotation is an ambiguous, subjective, and difficult task. This aspect lead to the decision of not purely calculating a majority vote label but to consider the diversity in human interpretation of emotion categories and publish the annotations by all annotators. Table TABREF40 shows the counts of annotators agreeing on a particular emotion. We observe that Love, Pride, and Sadness show highest intersubjectivity followed closely by Fear and Joy.

GoodNewsEveryone: A Corpus of News Headlines Annotated with Emotions, Semantic Roles, and Reader Perception	Most research on emotion analysis from text focuses on the task of emotion classification or emotion intensity regression. Fewer works address emotions as structured phenomena, which can be explained by the lack of relevant datasets and methods. We fill this gap by releasing a dataset of 5000 English news headlines annotated via crowdsourcing with their dominant emotions, emotion experiencers and textual cues, emotion causes and targets, as well as the reader's perception and emotion of the headline. We propose a multiphase annotation procedure which leads to high quality annotations on such a task via crowdsourcing. Finally, we develop a baseline for the task of automatic prediction of structures and discuss results. The corpus we release enables further research on emotion classification, emotion intensity prediction, emotion cause detection, and supports further qualitative studies.

The rest of $1,688$ headlines (up to $5,000$) ended in ties for the most dominant emotion category and were adjudicated manually. The emotion category Negative Surprise has the highest number of annotations, while Love has the lowest number of annotations. In most cases, Cues are single tokens (e. g., “infuriates”, “slams”), Cause has the largest proportion of annotations that span more than seven tokens on average (65% out of all annotations in this category), For the role of Experiencer, we see the lowest number of annotations (19%), which is a very different result to the one presented by Kim2018, where the role Experiencer was the most annotated. We hypothesize that this is the effect of the domain we annotated; it is more likely to encounter explicit experiencers in literature (as literary characters) than in news headlines. As we can see, the cue and the cause relations dominate the dataset (27% each), followed by Target (25%) relations. Table TABREF42 also shows how many times each emotion triggered a certain relation.

Data Collection & Annotation ::: Adjudication of Annotations ::: Relative Majority Rule.	This filter is applied to all questions regardless of their type. Effectively, whenever an entire annotation is agreed upon by at least two annotators, we use all parts of this annotation as the gold annotation. Given the headline depicted in Figure FIGREF1 with the following target role annotations by different annotators: “A couple”, “None”, “A couple”, “officials”, “their helicopter”. The resulting gold annotation is “A couple” and the adjudication process for the target ends.

Data Collection & Annotation ::: Annotation Procedure ::: Phase 2: Emotion and Role Annotation	The annotations collected during the first phase are automatically ranked and the ranking is used to decide which headlines are further annotated in the second phase. Ranking consists of sorting by agreement on P1Q1, considering P1Q2 in the case of ties. The top $5,000$ ranked headlines are annotated by five annotators for emotion class, intensity, reader emotion, and other emotions in case there is not only a dominant emotion. Along with these closed annotation tasks, the annotators are asked to answer several open questions, namely (1) who is the experiencer of the emotion (if mentioned), (2) what event triggered the annotated emotion (if mentioned), (3) if the emotion had a target, and (4) who or what is the target. The annotators are free to select multiple instances related to the dominant emotion by copy-paste into the answer field. For more details on the exact questions and example of answers, see Table TABREF17. Figure FIGREF1 shows a depiction of the procedure.

After we collected all annotations, we found unreliable annotators for both phases in the following way: for each annotator and for each question, we compute the probability with which the annotator agrees with the response chosen by the majority. If the computed probability is more than two standard deviations away from the mean we discard all annotations done by that annotator. On average, 310 distinct annotators needed 15 seconds in the first phase. We followed the guidelines of the platform regarding payment and decided to pay for each judgment $$0.02$ (USD) for Phase 1 (total of $$816.00$ USD). For the second phase, 331 distinct annotators needed on average $\approx $1:17 minutes to perform one judgment. Each judgment was paid with $0.08$$ USD (total $$2,720.00$ USD).

Data Collection & Annotation ::: Adjudication of Annotations	In this section, we describe the adjudication process we undertook to create the gold dataset and the difficulties we faced in creating a gold set out of the collected annotations. The first step was to discard obviously wrong annotations for open questions, such as annotations in other languages than English, or annotations of spans that were not part of the headline. In the next step, we incrementally apply a set of rules to the annotated instances in a one-or-nothing fashion. Specifically, we incrementally test each instance for a number of criteria in such a way that if at least one criteria is satisfied the instance is accepted and its adjudication is finalized. Instances that do not satisfy at least one criterium are adjudicated manually.

Data Collection & Annotation ::: Adjudication of Annotations ::: Most Common Subsequence Rule.	This rule is only applied to open text questions. It takes the most common smallest string intersection of all annotations. In the headline above, the experiencer annotations “A couple”, “infuriated officials”, “officials”, “officials”, “infuriated officials” would lead to “officials”.

Data Collection & Annotation	We gather the data in three steps: (1) collecting the news and the reactions they elicit in social media, (2) filtering the resulting set to retain relevant items, and (3) sampling the final selection using various metrics. The headlines are then annotated via crowdsourcing in two phases by three annotators in the first phase and by five annotators in the second phase. As a last step, the annotations are adjudicated to form the gold standard. We describe each step in detail below.

Data Collection & Annotation ::: Annotation Procedure	Using these sampling and filtering methods, we select $9,932$ headlines. Next, we set up two questionnaires (see Table TABREF17) for the two annotation phases that we describe below. We use Figure Eight.

Data Collection & Annotation ::: Adjudication of Annotations ::: Noun Chunks	For the role of experiencer, we accept only the most-common noun-chunk(s). The annotations that are left after being processed by all the rules described above are being adjudicated manually by the authors of the paper. We show examples for all roles in Table TABREF29.

Related Work ::: Cue or Trigger Words	The task of finding a function that segments a textual input and finds the span indicating an emotion category is less researched. Cue or trigger words detection could also be formulated as an emotion classification task for which the set of classes to be predicted is extended to cover other emotion categories with cues. First work that annotated cues was done manually by one expert and three annotators on the domain of blog posts . Mohammad2014 annotates the cues of emotions in a corpus of $4,058$ electoral tweets from US via crowdsourcing. Similar in annotation procedure, Yan2016emocues curate a corpus of 15,553 tweets and annotate it with 28 emotion categories, valence, arousal, and cues. To the best of our knowledge, there is only one work  that leverages the annotations for cues and considers the task of emotion detection where the exact spans that represent the cues need to be predicted.

Conclusion & Future Work	We introduce GoodNewsEveryone, a corpus of $5,000$ headlines annotated for emotion categories, semantic roles, and reader perspective. Such a dataset enables answering instance-based questions, such as, “who is experiencing what emotion and why?” or more general questions, like “what are typical causes of joy in media?”. To annotate the headlines, we employ a two-phase procedure and use crowdsourcing. To obtain a gold dataset, we aggregate the annotations through automatic heuristics. As the evaluation of the inter-annotator agreement and the baseline model results show, the task of annotating structures encompassing emotions with the corresponding roles is a very difficult one. However, we also note that developing such a resource via crowdsourcing has its limitations, due to the subjective nature of emotions, it is very challenging to come up with an annotation methodology that would ensure less dissenting annotations for the domain of headlines. We release the raw dataset, the aggregated gold dataset, the carefully designed questionnaires, and baseline models as a freely available repository (partially only after acceptance of the paper).

Data Collection & Annotation ::: Annotation Procedure ::: Phase 1: Selecting Emotional Headlines	The first questionnaire is meant to determine the dominant emotion of a headline, if that exists, and whether the headline triggers an emotion in a reader. We hypothesize that these two questions help us to retain only relevant headlines for the next, more expensive, annotation phase. During this phase, $9,932$ headlines were annotated by three annotators. The first question of the first phase (P1Q1) is: “Which emotion is most dominant in the given headline?” and annotators are provided a closed list of 15 emotion categories to which the category No emotion was added. The second question (P1Q2) aims to answer whether a given headline would stir up an emotion in most readers and the annotators are provided with only two possible answers (yes or no, see Table TABREF17 and Figure FIGREF1 for details). Our set of 15 emotion categories is an extended set over Plutchik's emotion classes and comprises anger, annoyance, disgust, fear, guilt, joy, love, pessimism, negative surprise, optimism, positive surprise, pride, sadness, shame, and trust.

Our contributions are: (1) we design a two phase annotation procedure for emotion structures via crowdsourcing, (2) present the first resource of news headlines annotated for emotions, cues, intensity, experiencers, causes, targets, and reader emotion, and, (3), provide results of a baseline model to predict such roles in a sequence labeling setting. We provide our annotations at http://www.romanklinger.de/data-sets/GoodNewsEveryone.zip.

Related Work ::: Semantic Role Labeling of Emotions	Semantic role labeling in the context of emotion analysis deals with extracting who feels (experiencer) which emotion (cue, class), towards whom the emotion is expressed (target), and what is the event that caused the emotion (stimulus). The relations are defined akin to FrameNet's Emotion frame . There are two works that work on annotation of semantic roles in the context of emotion. Firstly, Mohammad2014 annotate a dataset of $4,058$ tweets via crowdsourcing. The tweets were published before the U.S. presidential elections in 2012. The semantic roles considered are the experiencer, the stimulus, and the target. However, in the case of tweets, the experiencer is mostly the author of the tweet. Secondly, Kim2018 annotate and release REMAN (Relational EMotion ANnotation), a corpus of $1,720$ paragraphs based on Project Gutenberg. REMAN was manually annotated for spans which correspond to emotion cues and entities/events in the roles of experiencers, targets, and causes of the emotion.

Related Work ::: Emotion Intensity	In emotion intensity prediction, the term intensity refers to the degree an emotion is experienced. For this task, there are only a few datasets available. To our knowledge, the first dataset annotated for emotion intensity is by Aman2007, who ask experts for ratings, followed by the datasets released for the EmoInt shared tasks , , both annotated via crowdsourcing through the best-worst scaling. The annotation task can also be formalized as a classification task, similarly to the emotion classification task, where the goal would be to map some textual input to a class from a set of predefined classes of emotion intensity categories. This approach is used by Aman2007, where they annotate high, moderate, and low.

Analysis ::: Inter-Annotator Agreement	We calculate the agreement on the full set of annotations from each phase for the two question types, namely open vs. closed, where the first deal with emotion classification and second with the roles cue, experiencer, cause, and target.

Analysis ::: General Corpus Statistics	In the following, we report numbers of the adjudicated data set for simplicity of discussion. Please note that we publish all annotations by all annotators and suggest that computational models should consider the distribution of annotations instead of one adjudicated gold. The latter for be a simplification which we consider to not be appropriate. GoodNewsEveryone contains $5,000$ headlines from various news sources described in the Media Bias Chart . Overall, the corpus is composed of $56,612$ words ($354,173$ characters) out of which $17,513$ are unique. The headline length is short with 11 words on average. The shortest headline contains 6 words while the longest headline contains 32 words. The length of a headline in characters ranges from 24 the shortest to 199 the longest. Table TABREF42 presents the total number of adjudicated annotations for each role in relation to the dominant emotion. GoodNewsEveryone consists of $5,000$ headlines, $3,312$ of which have annotated dominant emotion via majority vote.

Data Collection & Annotation ::: Sampling Headlines ::: Sampling via NRC.	For the second sampling strategy we hypothesize that headlines containing emotionally charged words are also likely to contain the structures we aim to annotate. This strategy selects headlines whose words are in the NRC dictionary .

Such a diverse set of emotion labels is meant to provide a more fine-grained analysis and equip the annotators with a wider range of answer choices.

Anger and Annoyance show, given their similarity, lower scores. Note that the micro average of the basic emotions (+ love) is $0.21$ for when more than five annotators agree.

Data Collection & Annotation ::: Sampling Headlines ::: Random Sampling.	The goal of the first sampling method is to collect a random sample of headlines that is representative and not biased towards any source or content type. Note that the sample produced using this strategy might not be as rich with emotional content as the other samples.

In this sense, Negative Surprise and Positive Surprise has triggered the most Experiencer, and Cause and Target relations, which due to the prevalence of the annotations for this emotion in the dataset. Further, Figure FIGREF44, shows the distances of the different roles from the cue. The causes and targets are predominantly realized right of the cue, while the experiencer occurs more often left of the cue.

Related Work ::: Emotion Classification	Emotion classification deals with mapping words, sentences, or documents to a set of emotions following psychological models such as those proposed by Ekman1992 (anger, disgust, fear, joy, sadness and surprise) or Plutchik2001; or continuous values of valence, arousal and dominance . One way to create annotated datasets is via expert annotation , , , , . The creators of the ISEAR dataset make use of self-reporting instead, where subjects are asked to describe situations associated with a specific emotion . Crowdsourcing is another popular way to acquire human judgments , , , , . Another recent dataset for emotion recognition reproduces the ISEAR dataset in a crowdsourcing setting for both English and German . Lastly, social network platforms play a central role in data acquisition with distant supervision, because they provide a cheap way to obtain large amounts of noisy data , , , . Table TABREF3 shows an overview of resources. More details could be found in Bostan2018.

In this study, we focus on an annotation task with the goal of developing a dataset that would enable addressing the issues raised above. Specifically, we introduce the corpus GoodNewsEveryone, a novel dataset of news English headlines collected from 82 different sources analyzed in the Media Bias Chart  annotated for emotion class, emotion intensity, semantic roles (experiencer, cause, target, cue), and reader perspective. We use semantic roles, since identifying who feels what and why is essentially a semantic role labeling task . The roles we consider are a subset of those defined for the semantic frame for “Emotion” in FrameNet . We focus on news headlines due to their brevity and density of contained information. Headlines often appeal to a reader's emotions, and hence are a potential good source for emotion analysis. In addition, news headlines are easy-to-obtain data across many languages, void of data privacy issues associated with social media and microblogging.

The released dataset will be useful for social science scholars, since it contains valuable information about the interactions of emotions in news headlines, and gives interesting insights into the language of emotion expression in media. Note that this dataset is also useful since it introduces a new dataset to test on structured prediction models. We are currently investigating the dataset for understanding the interaction between media bias and annotated emotions and roles.

Related Work ::: Emotion Cause Detection	Detecting the cause of an expressed emotion in text received relatively little attention, compared to emotion detection. There are only few works on English that focus on creating resources to tackle this task , , , . The task can be formulated in different ways. One is to define a closed set of potential causes after annotation. Then, cause detection is a classification task . Another setting is to find the cause in the text. This is formulated as segmentation or clause classification , . Finding the cause of an emotion is widely researched on Mandarin in both resource creation and methods. Early works build on rule-based systems , ,  which examine correlations between emotions and cause events in terms of linguistic cues. The works that follow up focus on both methods and corpus construction, showing large improvements over the early works , , , , , , , , . The most recent work on cause extraction is being done on Mandarin and formulates the task jointly with emotion detection , , . With the exception of Mohammad2014 who is annotating via crowdsourcing, all other datasets are manually labeled, usually by using the W3C Emotion Markup Language.

Data Collection & Annotation ::: Sampling Headlines ::: Sampling based on Reddit & Twitter.	The last sampling strategy involves our Twitter and Reddit metadata. This enables us to select and sample headlines based on their impact on social media (under the assumption that this correlates with emotion connotation of the headline). This strategy chooses them equally from the most favorited tweets, most retweeted headlines on Twitter, most replied to tweets on Twitter, as well as most upvoted and most commented on posts on Reddit.

Data Collection & Annotation ::: Sampling Headlines ::: Sampling Entities.	We further hypothesize that headlines that mention named entities may also contain experiencers or targets of emotions, and therefore, they are likely to present a complete emotion structure. This sampling method yields headlines that contain at least one entity name, according to the recognition from spaCy that is trained on OntoNotes 5 and on Wikipedia corpus. We consider organization names, persons, nationalities, religious, political groups, buildings, countries, and other locations.

Data Collection & Annotation ::: Filtering & Postprocessing	We remove any headlines that have less than 6 tokens (e. g., “Small or nothing”, “But Her Emails”, “Red for Higher Ed”), as well as those starting with certain phrases, such as “Ep.”,“Watch Live:”, “Playlist:”, “Guide to”, and “Ten Things”. We also filter-out headlines that contain a date (e. g., “Headlines for March 15, 2019”) and words from the headlines which refer to visual content, like “video”, “photo”, “image”, “graphic”, “watch”, etc.

Data Collection & Annotation ::: Collecting Headlines	The first step consists of retrieving news headlines from the news publishers. We further retrieve content related to a news item from social media: tweets mentioning the headlines together with replies and Reddit posts that link to the headlines. We use this additional information for subsampling described later. We manually select all news sources available as RSS feeds (82 out of 124) from the Media Bias Chart , a project that analyzes reliability (from original fact reporting to containing inaccurate/fabricated information) and political bias (from most extreme left to most extreme right) of U.S. news sources. Our news crawler retrieved daily headlines from the feeds, together with the attached metadata (title, link, and summary of the news article) from March 2019 until October 2019. Every day, after the news collection finished, Twitter was queried for 50 valid tweets for each headline. In addition to that, for each collected tweet, we collect all valid replies and counts of being favorited, retweeted and replied to in the first 24 hours after its publication.

Questions like: “Who feels a particular emotion?” or “What causes that emotion?” still remain unaddressed. There has been almost no work in this direction, with only few exceptions in English ,  and Mandarin , . With this work, we argue that emotion analysis would benefit from a more fine-grained analysis that considers the full structure of an emotion, similar to the research in aspect-based sentiment analysis , , , . Consider the headline: “A couple infuriated officials by landing their helicopter in the middle of a nature reserve”  depicted on Figure FIGREF1. One could mark “officials” as the experiencer, “a couple” as the target, and “landing their helicopter in the middle of a nature reserve” as the cause of anger. Now let us imagine that the headline starts with “A cheerful couple” instead of “A couple”. A simple approach to emotion detection based on cue words will capture that this sentence contains descriptions of anger (“infuriated”) and joy (“cheerful”). It would, however, fail in attributing correct roles to the couple and the officials, thus, the distinction between their emotion experiences would remain hidden from us.

Acknowledgements	This research has been conducted within the CRETA project (http://www.creta.uni-stuttgart.de/) which is funded by the German Ministry for Education and Research (BMBF) and partially funded by the German Research Council (DFG), projects SEAT (Structured Multi-Domain Emotion Analysis from Text, KL 2869/1-1). We thank Enrica Troiano and Jeremy Barnes for fruitful discussions. same

They also provide baseline results for the automatic prediction of these structures and show that their models benefit from joint modeling of emotions with its roles in all subtasks. Our work follows in motivation Kim2018 and in procedure Mohammad2014.

The last step in the pipeline is aquiring the top (“hot”) submissions in the /r/news, /r/worldnews subreddits, and their metadata, including the number of up and downvotes, upvote ratio, number of comments, and comments themselves.

Baseline	As an estimate for the difficulty of the task, we provide baseline results. We formulate the task as sequence labeling of emotion cues, mentions of experiencers, targets, and causes with a bidirectional long short-term memory networks with a CRF layer (biLSTM-CRF) that uses Elmo embeddings as input and an IOB alphabet as output. The results are shown in Table TABREF45.

Introduction	Research in emotion analysis from text focuses on mapping words, sentences, or documents to emotion categories based on the models of Ekman1992 or Plutchik2001, which propose the emotion classes of joy, sadness, anger, fear, trust, disgust, anticipation and surprise. Emotion analysis has been applied to a variety of tasks including large scale social media mining , literature analysis , , lyrics and music analysis , , and the analysis of the development of emotions over time . There are at least two types of questions which cannot yet be answered by these emotion analysis systems. Firstly, such systems do not often explicitly model the perspective of understanding the written discourse (reader, writer, or the text's point of view). For example, the headline “Djokovic happy to carry on cruising”  contains an explicit mention of joy carried by the word “happy”. However, it may evoke different emotions in a reader (e. g., the reader is a supporter of Roger Federer), and the same applies to the author of the headline. To the best of our knowledge, only one work takes this point into consideration . Secondly, the structure that can be associated with the emotion description in text is not uncovered.

Question: How is quality of annotation measured?
Answer: Annotators went through various phases to make sure their annotations did not deviate from the mean.