Thus, we conceptualize a set of aesthetic emotions that are predictive of aesthetic appreciation in the reader, and allow the annotation of multiple labels per line to capture mixed emotions within context. We evaluate this novel setting in an annotation experiment both with carefully trained experts and via crowdsourcing. Our annotation with experts leads to an acceptable agreement of $\kappa =.70$, resulting in a consistent dataset for future large scale analysis. Finally, we conduct first emotion classification experiments based on BERT, showing that identifying aesthetic emotions is challenging in our data, with up to .52 F1-micro on the German subset. Data and resources are available at https://github.com/tnhaider/poetry-emotion. Emotion, Aesthetic Emotions, Literature, Poetry, Annotation, Corpora, Emotion Recognition, Multi-Label

The dataset presented in this paper can be of use for different application scenarios, including multi-label emotion classification, style-conditioned poetry generation, investigating the influence of rhythm/prosodic features on emotion, or analysis of authors, genres and diachronic variation (e.g., how emotions are represented differently in certain periods). Further, though our modeling experiments are still rudimentary, we propose that this data set can be used to investigate the intra-poem relations either through multi-task learning  and/or with the help of hierarchical sequence classification approaches.

PO-EMO: Conceptualization, Annotation, and Modeling of Aesthetic Emotions in German and English Poetry	Most approaches to emotion analysis regarding social media, literature, news, and other domains focus exclusively on basic emotion categories as defined by Ekman or Plutchik. However, art (such as literature) enables engagement in a broader range of more complex and subtle emotions that have been shown to also include mixed emotional responses. We consider emotions as they are elicited in the reader, rather than what is expressed in the text or intended by the author. Thus, we conceptualize a set of aesthetic emotions that are predictive of aesthetic appreciation in the reader, and allow the annotation of multiple labels per line to capture mixed emotions within context. We evaluate this novel setting in an annotation experiment both with carefully trained experts and via crowdsourcing. Our annotation with experts leads to an acceptable agreement of kappa=.70, resulting in a consistent dataset for future large scale analysis. Finally, we conduct first emotion classification experiments based on BERT, showing that identifying aesthetic emotions is challenging in our data, with up to .52 F1-micro on the German subset. Data and resources are available at https://github.com/tnhaider/poetry-emotion

We primarily adopt and adapt emotion terms that schindler2017measuring have identified as aesthetic emotions in their study on how to measure and categorize such particular affective states. Further, we consider the aspect that, when selecting specific emotion labels, the perspective of annotators plays a major role. Whether emotions are elicited in the reader, expressed in the text, or intended by the author largely changes the permissible labels. For example, feelings of Disgust or Love might be intended or expressed in the text, but the text might still fail to elicit corresponding feelings as these concepts presume a strong reaction in the reader. Our focus here was on the actual emotional experience of the readers rather than on hypothetical intentions of authors. We opted for this reader perspective based on previous research in NLP ,  and work in empirical aesthetics , that specifically measured the reception of poetry. Our final set of emotion labels consists of Beauty/Joy, Sadness, Uneasiness, Vitality, Suspense, Awe/Sublime, Humor, Annoyance, and Nostalgia.

In cases like these, the emotional response actually implies an aesthetic evaluation: narratives that have the capacity to move readers are evaluated as good and powerful texts for this very reason. Similarly, feelings of suspense experienced in narratives not only respond to the trajectory of the plot's content, but are also directly predictive of aesthetic liking (or disliking). Emotions that exhibit this dual capacity have been defined as “aesthetic emotions” . Contrary to the negativity bias of classical emotion catalogues, emotion terms used for aesthetic evaluation purposes include far more positive than negative emotions. At the same time, many overall positive aesthetic emotions encompass negative or mixed emotional ingredients , e.g., feelings of suspense include both hopeful and fearful anticipations. For these reasons, we argue that the analysis of literature (with a focus on poetry) should rely on specifically selected emotion items rather than on the narrow range of basic emotions only. Our selection is based on previous research on this issue in psychological studies on art reception and, specifically, on poetry. For instance, knoop2016mapping found that Beauty is a major factor in poetry reception.

Expert Annotation ::: Emotion Labels	We opt for measuring the reader perspective rather than the text surface or author's intent. To closer define and support conceptualizing our labels, we use particular `items', as they are used in psychological self-evaluations. These items consist of adjectives, verbs or short phrases. We build on top of schindler2017measuring who proposed 43 items that were then grouped by a factor analysis based on self-evaluations of participants. The resulting factors are shown in Table TABREF17. We attempt to cover all identified factors and supplement with basic emotions , , where possible. We started with a larger set of labels to then delete and substitute (tone down) labels during the initial annotation process to avoid infrequent classes and inconsistencies. Further, we conflate labels if they show considerable confusion with each other. These iterative improvements particularly affected Confusion, Boredom and Other that were very infrequently annotated and had little agreement among annotators ($\kappa <.2$). For German, we also removed Nostalgia ($\kappa =.218$) after gold standard creation, but after consideration, added it back for English, then achieving agreement.

Expert Annotation	In the following, we will explain how we compiled and annotated three data subsets, namely, (1) 48 German poems with gold annotation. These were originally annotated by three annotators. The labels were then aggregated with majority voting and based on discussions among the annotators. Finally, they were curated to only include one gold annotation. (2) The remaining 110 German poems that are used to compute the agreement in table TABREF20 and (3) 64 English poems contain the raw annotation from two annotators. We report the genesis of our annotation guidelines including the emotion classes. With the intention to provide a language resource for the computational analysis of emotion in poetry, we aimed at maximizing the consistency of our annotation, while doing justice to the diversity of poetry. We iteratively improved the guidelines and the annotation workflow by annotating in batches, cleaning the class set, and the compilation of a gold standard. The final overall cost of producing this expert annotated dataset amounts to approximately 3,500.

Crowdsourcing Annotation	After concluding the expert annotation, we performed a focused crowdsourcing experiment, based on the final label set and items as they are listed in Table TABREF27 and Section SECREF19. With this experiment, we aim to understand whether it is possible to collect reliable judgements for aesthetic perception of poetry from a crowdsourcing platform. A second goal is to see whether we can replicate the expensive expert annotations with less costly crowd annotations. We opted for a maximally simple annotation environment, where we asked participants to annotate English 4-line stanzas with self-perceived reader emotions. We choose English due to the higher availability of English language annotators on crowdsourcing platforms. Each annotator rates each stanza independently of surrounding context.

Crowdsourcing Annotation ::: Results	In the following, we determine the best aggregation strategy regarding the 10 annotators with bootstrap resampling. For instance, one could assign the label of a specific emotion to an instance if just one annotators picks it, or one could assign the label only if all annotators agree on this emotion. To evaluate this, we repeatedly pick two sets of 5 annotators each out of the 10 annotators for each of the 59 stanzas, 1000 times overall (i.e., 1000$\times $59 times, bootstrap resampling). For each of these repetitions, we compare the agreement of these two groups of 5 annotators. Each group gets assigned with an adjudicated emotion which is accepted if at least one annotator picks it, at least two annotators pick it, etc. up to all five pick it. We show the results in Table TABREF27. The $\kappa $ scores show the average agreement between the two groups of five annotators, when the adjudicated class is picked based on the particular threshold of annotators with the same label choice.

To reflect on the emotional complexity of poetry, we allow a maximum of two labels per line while avoiding heavy label fluctuations by encouraging annotators to reflect on their feelings to avoid `empty' annotations. Rather, they were advised to use fewer labels and more consistent annotation. This additional constraint is necessary to avoid “wild”, non-reconstructable or non-justified annotations. All subsequent batches (all except the first three) were only annotated by two out of the three initial annotators, coincidentally those two who had the lowest initial agreement with each other. We asked these two experts to use the generated gold standard (48 poems; majority votes of 3 annotators plus manual curation) as a reference (“if in doubt, annotate according to the gold standard”). This eliminated some systematic differences between them and markedly improved the agreement levels, roughly from 0.3–0.5 Cohen's $\kappa $ in the first three batches to around 0.6–0.8 $\kappa $ for all subsequent batches. This annotation procedure relaxes the reader perspective, as we encourage annotators (if in doubt) to annotate how they think the other annotators would annotate.

We see that some emotions tend to have higher agreement scores than others, namely Annoyance (.66), Sadness (up to .52), and Awe/Sublime, Beauty/Joy, Humor (all .46). The maximum agreement is reached mostly with a threshold of 2 (4 times) or 3 (3 times). We further show in the same table the average numbers of labels from each strategy. Obviously, a lower threshold leads to higher numbers (corresponding to a disjunction of annotations for each emotion). The drop in label counts is comparably drastic, with on average 18 labels per class. Overall, the best average $\kappa $ agreement (.32) is less than half of what we saw for the expert annotators (roughly .70). Crowds especially disagree on many more intricate emotion labels (Uneasiness, Vitality, Nostalgia, Suspense). We visualize how often two emotions are used to label an instance in a confusion table in Figure FIGREF18. Sadness is used most often to annotate a stanza, and it is often confused with Suspense, Uneasiness, and Nostalgia.

The results of the crowdsourcing experiment, on the other hand, are a mixed bag as evidenced by a much sparser distribution of emotion labels. However, we note that these differences can be caused by 1) the disparate training procedure for the experts and crowds, and 2) the lack of opportunities for close supervision and on-going training of the crowds, as opposed to the in-house expert annotators. In general, however, we find that substituting experts with crowds is possible to a certain degree. Even though the crowds' labels look inconsistent at first view, there appears to be a good signal in their aggregated annotations, helping to approximate expert annotations to a certain degree. The average $\kappa $ agreement (with the experts) we get from $N=10$ crowd workers (0.24) is still considerably below the agreement among the experts (0.70).

Crowdsourcing Annotation ::: Data and Setup	For consistency and to simplify the task for the annotators, we opt for a trade-off between completeness and granularity of the annotation. Specifically, we subselect stanzas composed of four verses from the corpus of 64 hand selected English poems. The resulting selection of 59 stanzas is uploaded to Figure Eight for annotation. The annotators are asked to answer the following questions for each instance. Question 1 (single-choice): Read the following stanza and decide for yourself which emotions it evokes. Question 2 (multiple-choice): Which additional emotions does the stanza evoke? The answers to both questions correspond to the emotion labels we defined to use in our annotation, as described in Section SECREF19. We add an additional answer choice “None” to Question 2 to allow annotators to say that a stanza does not evoke any additional emotions. Each instance is annotated by ten people. We restrict the task geographically to the United Kingdom and Ireland and set the internal parameters on Figure Eight to only include the highest quality annotators to join the task. We pay 0.09 per instance. The final cost of the crowdsourcing experiment is 74.

Concluding Remarks	In this paper, we presented a dataset of German and English poetry annotated with reader response to reading poetry. We argued that basic emotions as proposed by psychologists (such as Ekman and Plutchik) that are often used in emotion analysis from text are of little use for the annotation of poetry reception. We instead conceptualized aesthetic emotion labels and showed that a closely supervised annotation task results in substantial agreement—in terms of $\kappa $ score—on the final dataset. The task of collecting reader-perceived emotion response to poetry in a crowdsourcing setting is not straightforward. In contrast to expert annotators, who were closely supervised and reflected upon the task, the annotators on crowdsourcing platforms are difficult to control and may lack necessary background knowledge to perform the task at hand. However, using a larger number of crowd annotators may lead to finding an aggregation strategy with a better trade-off between quality and quantity of adjudicated labels. For future work, we thus propose to repeat the experiment with larger number of crowdworkers, and develop an improved training strategy that would suit the crowdsourcing environment.

Modeling	To estimate the difficulty of automatic classification of our data set, we perform multi-label document classification (of stanzas) with BERT . For this experiment we aggregate all labels for a stanza and sort them by frequency, both for the gold standard and the raw expert annotations. As can be seen in Figure FIGREF23, a stanza bears a minimum of one and a maximum of four emotions. Unfortunately, the label Nostalgia is only available 16 times in the German data (the gold standard) as a second label (as discussed in Section SECREF19). None of our models was able to learn this label for German. Therefore we omit it, leaving us with eight proper labels. We use the code and the pre-trained BERT models of Farm, provided by deepset.ai.

Crowdsourcing Annotation ::: Comparing Experts with Crowds	To gauge the quality of the crowd annotations in comparison with our experts, we calculate agreement on the emotions between experts and an increasing group size from the crowd. For each stanza instance $s$, we pick $N$ crowd workers, where $N\in \lbrace 4,6,8,10\rbrace $, then pick their majority emotion for $s$, and additionally pick their second ranked majority emotion if at least $\frac{N}{2}-1$ workers have chosen it. For the experts, we aggregate their emotion labels on stanza level, then perform the same strategy for selection of emotion labels. Thus, for $s$, both crowds and experts have 1 or 2 emotions. For each emotion, we then compute Cohen's $\kappa $ as before. Note that, compared to our previous experiments in Section SECREF26 with a threshold, each stanza now receives an emotion annotation (exactly one or two emotion labels), both by the experts and the crowd-workers. In Figure FIGREF30, we plot agreement between experts and crowds on stanza level as we vary the number $N$ of crowd workers involved.

Introduction	Emotions are central to human experience, creativity and behavior. Models of affect and emotion, both in psychology and natural language processing, commonly operate on predefined categories, designated either by continuous scales of, e.g., Valence, Arousal and Dominance  or discrete emotion labels (which can also vary in intensity). Discrete sets of emotions often have been motivated by theories of basic emotions, as proposed by Ekman1992—Anger, Fear, Joy, Disgust, Surprise, Sadness—and Plutchik1991, who added Trust and Anticipation. These categories are likely to have evolved as they motivate behavior that is directly relevant for survival. However, art reception typically presupposes a situation of safety and therefore offers special opportunities to engage in a broader range of more complex and subtle emotions. These differences between real-life and art contexts have not been considered in natural language processing work so far. To emotionally move readers is considered a prime goal of literature since Latin antiquity , , . Deeply moved readers shed tears or get chills and goosebumps even in lab settings .

Expert Annotation ::: Workflow	The annotation process was initially conducted by three female university students majoring in linguistics and/or literary studies, which we refer to as our “expert annotators”. We used the INCePTION platform for annotation . Starting with the German poems, we annotated in batches of about 16 (and later in some cases 32) poems. After each batch, we computed agreement statistics including heatmaps, and provided this feedback to the annotators. For the first three batches, the three annotators produced a gold standard using a majority vote for each line. Where this was inconclusive, they developed an adjudicated annotation based on discussion. Where necessary, we encouraged the annotators to aim for more consistency, as most of the frequent switching of emotions within a stanza could not be reconstructed or justified. In poems, emotions are regularly mixed (already on line level) and are more interpretable within the whole poem. We therefore annotate lines hierarchically within the larger context of stanzas and the whole poem. Hence, we instruct the annotators to read a complete stanza or full poem, and then annotate each line in the context of its stanza.

In a first study, the annotators work on the annotations in a closely supervised fashion, carefully reading each verse, stanza, and poem. In a second study, the annotations are performed via crowdsourcing within relatively short time periods with annotators not seeing the entire poem while reading the stanza. Using these two settings, we aim at obtaining a better understanding of the advantages and disadvantages of an expert vs. crowdsourcing setting in this novel annotation task. Particularly, we are interested in estimating the potential of a crowdsourcing environment for the task of self-perceived emotion annotation in poetry, given time and cost overhead associated with in-house annotation process (that usually involve training and close supervision of the annotators). We provide the final datasets of German and English language poems annotated with reader emotions on verse level at https://github.com/tnhaider/poetry-emotion.

In addition to selecting an adapted set of emotions, the annotation of poetry brings further challenges, one of which is the choice of the appropriate unit of annotation. Previous work considers words , , sentences , , utterances , sentence triples , or paragraphs  as the units of annotation. For poetry, reasonable units follow the logical document structure of poems, i.e., verse (line), stanza, and, owing to its relative shortness, the complete text. The more coarse-grained the unit, the more difficult the annotation is likely to be, but the more it may also enable the annotation of emotions in context. We find that annotating fine-grained units (lines) that are hierarchically ordered within a larger context (stanza, poem) caters to the specific structure of poems, where emotions are regularly mixed and are more interpretable within the whole poem. Consequently, we allow the mixing of emotions already at line level through multi-label annotation. The remainder of this paper includes (1) a report of the annotation process that takes these challenges into consideration, (2) a description of our annotated corpora, and (3) an implementation of baseline models for the novel task of aesthetic emotion annotation in poetry.

Expert Annotation ::: Agreement	Table TABREF20 shows the Cohen's $\kappa $ agreement scores among our two expert annotators for each emotion category $e$ as follows. We assign each instance (a line in a poem) a binary label indicating whether or not the annotator has annotated the emotion category $e$ in question. From this, we obtain vectors $v_i^e$, for annotators $i=0,1$, where each entry of $v_i^e$ holds the binary value for the corresponding line. We then apply the $\kappa $ statistics to the two binary vectors $v_i^e$. Additionally to averaged $\kappa $, we report micro-F1 values in Table TABREF21 between the multi-label annotations of both expert annotators as well as the micro-F1 score of a random baseline as well as of the majority emotion baseline (which labels each line as Beauty/Joy). We find that Cohen $\kappa $ agreement ranges from .84 for Uneasiness in the English data, .81 for Humor and Nostalgia, down to German Suspense (.65), Awe/Sublime (.61) and Vitality for both languages (.50 English, .63 German).

Related Work ::: Emotion Annotation	Emotion corpora have been created for different tasks and with different annotation strategies, with different units of analysis and different foci of emotion perspective (reader, writer, text). Examples include the ISEAR dataset  (document-level); emotion annotation in children stories  and news headlines  (sentence-level); and fine-grained emotion annotation in literature by Kim2018 (phrase- and word-level). We refer the interested reader to an overview paper on existing corpora . We are only aware of a limited number of publications which look in more depth into the emotion perspective. buechel-hahn-2017-emobank report on an annotation study that focuses both on writer's and reader's emotions associated with English sentences. The results show that the reader perspective yields better inter-annotator agreement. Yang2009 also study the difference between writer and reader emotions, but not with a modeling perspective. The authors find that positive reader emotions tend to be linked to positive writer emotions in online blogs.

Both annotators have a similar emotion frequency profile, where the ranking is almost identical, especially for German. However, for English, Annotator 2 annotates more Vitality than Uneasiness. Figure FIGREF18 shows the confusion matrices of labels between annotators as heatmaps. Notably, Beauty/Joy and Sadness are confused across annotators more often than other labels. This is topical for poetry, and therefore not surprising: One might argue that the beauty of beings and situations is only beautiful because it is not enduring and therefore not to divorce from the sadness of the vanishing of beauty . We also find considerable confusion of Sadness with Awe/Sublime and Vitality, while the latter is also regularly confused with Beauty/Joy. Furthermore, as shown in Figure FIGREF23, we find that no single poem aggregates to more than six emotion labels, while no stanza aggregates to more than four emotion labels. However, most lines and stanzas prefer one or two labels. German poems seem more emotionally diverse where more poems have three labels than two labels, while the majority of English poems have only two labels. This is however attributable to the generally shorter English texts.

Appendix	We illustrate two examples of our German gold standard annotation, a poem each by Friedrich Hölderlin and Georg Trakl, and an English poem by Walt Whitman. Hölderlin's text stands out, because the mood changes starkly from the first stanza to the second, from Beauty/Joy to Sadness. Trakl's text is a bit more complex with bits of Nostalgia and, most importantly, a mixture of Uneasiness with Awe/Sublime. Whitman's poem is an example of Vitality and its mixing with Sadness. The English annotation was unified by us for space constraints. For the full annotation please see https://github.com/tnhaider/poetry-emotion/

The term Sublime originated with kant2000critique as one of the first aesthetic emotion terms. Awe is a more common English term. Beauty/Joy (found it beautiful/pleasing/makes me happy/joyful): kant2000critique already spoke of a “feeling of beauty”, and it should be noted that it is not a `merely pleasing emotion'. Therefore, in our pilot annotations, Beauty and Joy were separate labels. However, schindler2017measuring found that items for Beauty and Joy load into the same factors. Furthermore, our pilot annotations revealed, while Beauty is the more dominant and frequent feeling, both labels regularly accompany each other, and they often get confused across annotators. Therefore, we add Joy to form an inclusive label Beauty/Joy that increases annotation consistency. Humor (found it funny/amusing): Implies feeling amused by the line/stanza or if it makes one laugh. Nostalgia (makes me nostalgic): Nostalgia is defined as a sentimental longing for things, persons or situations in the past. It often carries both positive and negative feelings.

Further, Beauty/Joy partially overlaps with Awe/Sublime, Nostalgia, and Sadness. On average, each crowd annotator uses two emotion labels per stanza (56% of cases); only in 36% of the cases the annotators use one label, and in 6% and 1% of the cases three and four labels, respectively. This contrasts with the expert annotators, who use one label in about 70% of the cases and two labels in 30% of the cases for the same 59 four-liners. Concerning frequency distribution for emotion labels, both experts and crowds name Sadness and Beauty/Joy as the most frequent emotions (for the `best' threshold of 3) and Nostalgia as one of the least frequent emotions. The Spearman rank correlation between experts and crowds is about 0.55 with respect to the label frequency distribution, indicating that crowds could replace experts to a moderate degree when it comes to extracting, e.g., emotion distributions for an author or time period. Now, we further compare crowds and experts in terms of whether crowds could replicate expert annotations also on a finer stanza level (rather than only on a distributional level).

With these word lists, they distinguish the categories `Love', `Songs to Lisi', `Satire' and `Philosophical-Moral-Religious' in Quevedo's poetry. Similarly, alsharif2013emotion classify unique Arabic `emotional text forms' based on word unigrams. Mohanty2018 create a corpus of 788 poems in the Indian Odia language, annotate it on text (poem) level with binary negative and positive sentiment, and are able to distinguish these with moderate success. Sreeja2019 construct a corpus of 736 Indian language poems and annotate the texts on Ekman's six categories + Love + Courage. They achieve a Fleiss Kappa of .48. In contrast to our work, these studies focus on basic emotions and binary sentiment polarity only, rather than addressing aesthetic emotions. Moreover, they annotate on the level of complete poems (instead of fine-grained verse and stanza-level).

However, we found that this formulation improves the usability of the data and leads to a more consistent annotation.

Acknowledgements	A special thanks goes to Gesine Fuhrmann, who created the guidelines and tirelessly documented the annotation progress. Also thanks to Annika Palm and Debby Trzeciak who annotated and gave lively feedback. For help with the conceptualization of labels we thank Ines Schindler. This research has been partially conducted within the CRETA center (http://www.creta.uni-stuttgart.de/) which is funded by the German Ministry for Education and Research (BMBF) and partially funded by the German Research Council (DFG), projects SEAT (Structured Multi-Domain Emotion Analysis from Text, KL 2869/1-1). This work has also been supported by the German Research Foundation as part of the Research Training Group Adaptive Preparation of Information from Heterogeneous Sources (AIPHES) at the Technische Universität Darmstadt under grant No. GRK 1994/1.

However, since this label is quite infrequent, and not available in all subsets of the data, we annotated it with an additional Beauty/Joy or Sadness label to ensure annotation consistency. Sadness (makes me sad/touches me): If the line/stanza makes one feel sad. It also includes a more general `being touched / moved'. Suspense (found it gripping/sparked my interest): Choose Suspense if the line/stanza keeps one in suspense (if the line/stanza excites one or triggers one's curiosity). We further removed Anticipation from Suspense/Anticipation, as Anticipation appeared to us as being a more cognitive prediction whereas Suspense is a far more straightforward emotion item. Uneasiness (found it ugly/unsettling/disturbing / frightening/distasteful): This label covers situations when one feels discomfort about the line/stanza (if the line/stanza feels distasteful/ugly, unsettling/disturbing or frightens one). The labels Ugliness and Disgust were conflated into Uneasiness, as both are seldom felt in poetry (being inadequate/too strong/high in arousal), and typically lead to Uneasiness.

Data Collection	For our annotation and modeling studies, we build on top of two poetry corpora (in English and German), which we refer to as PO-EMO. This collection represents important contributions to the literary canon over the last 400 years. We make this resource available in TEI P5 XML and an easy-to-use tab separated format. Table TABREF9 shows a size overview of these data sets. Figure FIGREF8 shows the distribution of our data over time via density plots. Note that both corpora show a relative underrepresentation before the onset of the romantic period (around 1750).

Data Collection ::: German	The German corpus contains poems available from the website lyrik.antikoerperchen.de (ANTI-K), which provides a platform for students to upload essays about poems. The data is available in the Hypertext Markup Language, with clean line and stanza segmentation. ANTI-K also has extensive metadata, including author names, years of publication, numbers of sentences, poetic genres, and literary periods, that enable us to gauge the distribution of poems according to periods. The 158 poems we consider (731 stanzas) are dispersed over 51 authors and the New High German timeline (1575–1936 A.D.). This data has been annotated, besides emotions, for meter, rhythm, and rhyme in other studies , .

On average, there is roughly a steady linear increase in agreement as $N$ grows, which may indicate that $N=20$ or $N=30$ would still lead to better agreement. Concerning individual emotions, Nostalgia is the emotion with the least agreement, as opposed to Sadness (in our sample of 59 four-liners): the agreement for this emotion grows from $.47$ $\kappa $ with $N=4$ to $.65$ $\kappa $ with $N=10$. Sadness is also the most frequent emotion, both according to experts and crowds. Other emotions for which a reasonable agreement is achieved are Annoyance, Awe/Sublime, Beauty/Joy, Humor ($\kappa $ > 0.2). Emotions with little agreement are Vitality, Uneasiness, Suspense, Nostalgia ($\kappa $ < 0.2). By and large, we note from Figure FIGREF18 that expert annotation is more restrictive, with experts agreeing more often on particular emotion labels (seen in the darker diagonal).

Related Work ::: Poetry in Natural Language Processing	Natural language understanding research on poetry has investigated stylistic variation , , , with a focus on broadly accepted formal features such as meter , ,  and rhyme , , as well as enjambement ,  and metaphor , . Recent work has also explored the relationship of poetry and prose, mainly on a syntactic level , . Furthermore, poetry also lends itself well to semantic (change) analysis , , as linguistic invention ,  and succinctness  are at the core of poetic production. Corpus-based analysis of emotions in poetry has been considered, but there is no work on German, and little on English. kao2015computational analyze English poems with word associations from the Harvard Inquirer and LIWC, within the categories positive/negative outlook, positive/negative emotion and phys./psych. well-being. hou-frank-2015-analyzing examine the binary sentiment polarity of Chinese poems with a weighted personalized PageRank algorithm. barros2013automatic followed a tagging approach with a thesaurus to annotate words that are similar to the words `Joy', `Anger', `Fear' and `Sadness' (moreover translating these from English to Spanish).

We test the multilingual-uncased model (Multiling), the german-base-cased model (Base), the german-dbmdz-uncased model (Dbmdz), and we tune the Base model on 80k stanzas of the German Poetry Corpus DLK  for 2 epochs, both on token (masked words) and sequence (next line) prediction (Base$_{\textsc {Tuned}}$). We split the randomized German dataset so that each label is at least 10 times in the validation set (63 instances, 113 labels), and at least 10 times in the test set (56 instances, 108 labels) and leave the rest for training (617 instances, 946 labels). We train BERT for 10 epochs (with a batch size of 8), optimize with entropy loss, and report F1-micro on the test set. See Table TABREF36 for the results. We find that the multilingual model cannot handle infrequent categories, i.e., Awe/Sublime, Suspense and Humor. However, increasing the dataset with English data improves the results, suggesting that the classification would largely benefit from more annotated data.

Related Work ::: Emotion Classification	The task of emotion classification has been tackled before using rule-based and machine learning approaches. Rule-based emotion classification typically relies on lexical resources of emotionally charged words , ,  and offers a straightforward and transparent way to detect emotions in text. In contrast to rule-based approaches, current models for emotion classification are often based on neural networks and commonly use word embeddings as features. Schuff2017 applied models from the classes of CNN, BiLSTM, and LSTM and compare them to linear classifiers (SVM and MaxEnt), where the BiLSTM shows best results with the most balanced precision and recall. AbdulMageed2017 claim the highest F$_1$ with gated recurrent unit networks  for Plutchik's emotion model. More recently, shared tasks on emotion analysis ,  triggered a set of more advanced deep learning approaches, including BERT  and other transfer learning methods .

Nostalgia is still available in the gold standard (then with a second label Beauty/Joy or Sadness to keep consistency). However, Confusion, Boredom and Other are not available in any sub-corpus. Our final set consists of nine classes, i.e., (in order of frequency) Beauty/Joy, Sadness, Uneasiness, Vitality, Suspense, Awe/Sublime, Humor, Annoyance, and Nostalgia. In the following, we describe the labels and give further details on the aggregation process. Annoyance (annoys me/angers me/felt frustrated): Annoyance implies feeling annoyed, frustrated or even angry while reading the line/stanza. We include the class Anger here, as this was found to be too strong in intensity. Awe/Sublime (found it overwhelming/sense of greatness): Awe/Sublime implies being overwhelmed by the line/stanza, i.e., if one gets the impression of facing something sublime or if the line/stanza inspires one with awe (or that the expression itself is sublime). Such emotions are often associated with subjects like god, death, life, truth, etc.

Vitality (found it invigorating/spurs me on/inspires me): This label is meant for a line/stanza that has an inciting, encouraging effect (if the line/stanza conveys a feeling of movement, energy and vitality which animates to action). Similar terms are Activation and Stimulation.

Data Collection ::: English	The English corpus contains 64 poems of popular English writers. It was partly collected from Project Gutenberg with the GutenTag tool, and, in addition, includes a number of hand selected poems from the modern period and represents a cross section of popular English poets. We took care to include a number of female authors, who would have been underrepresented in a uniform sample. Time stamps in the corpus are organized by the birth year of the author, as assigned in Project Gutenberg.

:::  ::: 	1.1.1.1em Thomas Haider$^{1,3}$, Steffen Eger$^2$, Evgeny Kim$^3$, Roman Klinger$^3$, Winfried Menninghaus$^1$ $^{1}$Department of Language and Literature, Max Planck Institute for Empirical Aesthetics $^{2}$NLLG, Department of Computer Science, Technische Universitat Darmstadt $^{3}$Institut für Maschinelle Sprachverarbeitung, University of Stuttgart {thomas.haider, w.m}@ae.mpg.de, eger@aiphes.tu-darmstadt.de {roman.klinger, evgeny.kim}@ims.uni-stuttgart.de Most approaches to emotion analysis regarding social media, literature, news, and other domains focus exclusively on basic emotion categories as defined by Ekman or Plutchik. However, art (such as literature) enables engagement in a broader range of more complex and subtle emotions that have been shown to also include mixed emotional responses. We consider emotions as they are elicited in the reader, rather than what is expressed in the text or intended by the author.

::: 	1.1.1em

The best model overall is DBMDZ (.520), showing a balanced response on both validation and test set. See Table TABREF37 for a breakdown of all emotions as predicted by the this model. Precision is mostly higher than recall. The labels Awe/Sublime, Suspense and Humor are harder to predict than the other labels. The BASE and BASE$_{\textsc {TUNED}}$ models perform slightly worse than DBMDZ. The effect of tuning of the BASE model is questionable, probably because of the restricted vocabulary (30k). We found that tuning on poetry does not show obvious improvements. Lastly, we find that models that were trained on lines (instead of stanzas) do not achieve the same F1 (~.42 for the German models).

1.1em

Question: How is the annotation experiment evaluated?
Answer: confusion matrices of labels between annotators