Modeling Emotional Trajectories in Written Stories Utilizing Transformers and Weakly-Supervised Learning (2024)

Abstract: Telling stories is an integral part of human communication which can evoke emotions and influence the affective states of the audience. Automatically modeling emotional trajectories in stories has thus attracted considerable scholarly interest. However, as most existing works have been limited to unsupervised dictionary-based approaches, there is no benchmark for this task. We address this gap by introducing continuous valence and arousal labels for an existing dataset of children’s stories originally annotated with discrete emotion categories. We collect additional annotations for this data and map the categorical labels to the continuous valence and arousal space. For predicting the thus obtained emotionality signals, we fine-tune a DeBERTa model and improve upon this baseline via a weakly supervised learning approach. The best configuration achieves a Concordance Correlation Coefficient (CCC) of .8221 for valence and .7125 for arousal on the test set, demonstrating the efficacy of our proposed approach. A detailed analysis shows the extent to which the results vary depending on factors such as the author, the individual story, or the section within the story. In addition, we uncover the weaknesses of our approach by investigating examples that prove to be difficult to predict.

heading: 1 Introduction
text: Stories are central to literature, movies, and music, but also human dreams and memories (Gottschall, 2012). Storytelling has received widespread attention from various disciplines for many decades (Polletta et al., 2011), e. g., in the fields of psychology (Sunderland, 2017), cognitive sciences (Burke, 2015), and history (Palombini, 2017). A crucial aspect of stories is their emotionality, as stories typically evoke a range of different emotions in the listeners or readers, which also serves the purpose of keeping the audience interested (Hogan, 2011).
Several efforts have been made to model emotionality in written stories computationally. However, these studies have often been constrained to dictionary-based methods (Reagan et al., 2016; Somasundaran et al., 2020). In addition, existing work often models emotions in stories on the sentence level only (Agrawal and An, 2012; Batbaatar et al., 2019) without taking into account surrounding sentences, missing out on important contextual information. In this study, we address the aforementioned issues by employing a pretrained Large Language Model (LLM) to predict emotionality in stories automatically. In combination with an emotional Text-to-Speech (TTS) system (Triantafyllopoulos et al., 2023; Amiriparian et al., 2023), our system could serve naturalistic human-machine interaction, educational, and entertainment purposes (Lugrin et al., 2010). For example, stories could be automatically read to children (Eisenreich et al., 2014) by voice assistants. Furthermore, the prediction of emotions in literary texts is of interest in the field of Digital Humanities (Kim and Klinger, 2018a), especially in Computational Narratology (Mani, 2014; Piper et al., 2021).
We conduct our experiments on the children’s story dataset created by Alm (2008). Specifically, our contributions are the following. First, we extend the annotations provided by Alm (2008) and, subsequently, map the originally discrete emotion labels to the continuous valence and arousal (Russell, 1980) space (cf. Section 3). We then employ DeBERTaV3 in combination with a weakly-supervised learning step to predict valence and arousal in the stories provided in the dataset (cf. Section 4). To the best of our knowledge, our work is the first to model emotional trajectories in stories over the course of complete stories, also referred to as emotional arcs, using supervised machine learning, and, in particular, LLMs. While predicting such valence and arousal signals is common in the field of multimodal affect analysis (Ringeval
7144
et al., 2019; Stappen et al., 2021; Christ et al., 2022a), it has not been applied to textual stories, yet.

heading: 2 Related Work
text: Various unsupervised, lexicon-based approaches to model emotional trajectories in narrative and literary texts have been proposed. With a lexicon-based method, Reagan et al. (2016) identified six elementary sentiment-based emotional arcs such as ragsto-riches in a corpus of about 1,300 books. Moreira et al. (2023) generate lexicon-based emotional arcs and demonstrate their usability in predicting the perceived literary quality of novels. Further Examples include the works of Strapparava et al. (2004), Wilson et al. (2005), Kim et al. (2017) and (Somasundaran et al., 2020). While these previous works use dictionaries to directly predict emotionality, we only utilize them to map existing annotations into the valence/arousal space.
Moreover, a range of datasets of narratives annotated for emotionality exists. In a corpus of 100 crowdsourced short stories, (Mori et al., 2019) provided annotations both for character emotions as well as for emotions evoked in readers. The Dataset for Emotions of Narrative Sequences (DENS) (Liu et al., 2019) contains about 10,000 passages from modern as well as classic stories, labeled with 10 discrete emotions. In the authors’ experiments, fine-tuning BERT (Devlin et al., 2019) proved to be superior to more classic approaches such as Recurrent Neural Networks (RNNs). The Relational EMotion ANnotation (REMAN) dataset (Kim and Klinger, 2018b) comprises 1,720 text segments from about 200 books. These passages are labeled on a phrase level regarding, among others, emotion, the emotion experiencer, the emotion’s cause, and its target. Kim and Klinger (2018b) conducted experiments with biLSTMs and Conditional Random Fields (CRFs) on REMAN. The Stanford Emotional Narratives Dataset (SEND) (Ong et al., 2019) is a multimodal dataset containing 193 video clips of subjects narrating personal emotional events, annotated with valence values in a time-continuous manner.
The corpus of children’s stories (Alm, 2008) we are using for our experiments is originally labeled for eight discrete emotions (cf. Section 3). Alm and Sproat (2005) modeled emotional trajectories in a subset of this corpus, while in (Alm et al., 2005), the authors conducted machine learning ex-
periments with several handcrafted features such as sentence length and POS-Tags as well as Bag of Words. While the corpus has frequently served as a benchmark for textual emotion recognition, scholars have so far limited their experiments to subsets of this dataset, selected based on high agreement among the annotators or certain emotion labels. Examples of such studies include an algorithm combining vector representations and syntactic dependencies by Agrawal and An (2012), the rule-based approach proposed by Udochukwu and He (2015), and a combination of Convolutional Neural Network (CNN) and Long Short-Term Memory (LSTM) introduced by Batbaatar et al. (2019). No existing work, however, aims at modeling the complete stories provided in the dataset.

heading: 3 Data
text: We choose the children’s story dataset by Alm (2008), henceforth referred to as ALM, for our experiments. From the mentioned datasets, the ALM dataset is the only suitable one as it is reasonably large, comprising about 15,000 sentences, and contains complete, yet brief stories, with the longest story consisting of 530 sentences. Moreover, the data is labeled per sentence, allowing us to model emotional trajectories for stories. We extend the dataset by a third annotation, as described in Section 3.1, and modify the originally discrete annotation scheme by mapping it into the continuous valence/arousal space (cf. Section 3.2).
Originally, the dataset comprises 176 stories from 3 authors. More precisely, 80 stories from the German Brothers Grimm, 77 stories by Danish author Hans-Christian Andersen, and 19 stories written by Beatrix Potter are contained. Every sentence is annotated with the emotion experienced by the primary character (feeler) in the respective sentence, and the overall mood of the sentence. For both label types, two annotators had to select one out of eight discrete emotion labels, namely anger, disgust, fear, happiness, negative surprise, neutral, positive surprise, and sadness. For a detailed description of the original data, the reader is referred to (Alm and Sproat, 2005; Alm, 2008). We limit our experiments to predicting the mood per sentence, as it refers to the sentence as a whole instead of a particular subject.

heading: 3.1 Additional Annotations
text: In addition to the existing annotations, we collect a third mood label for every sentence. This allows us to create a continuous-valued gold standard (cf. Section 3.2) via the agreement-based EvaluatorWeighted Estimator (EWE) (Grimm and Kroschel, 2005) fusion method, for which at least three different ratings are required. Compared to the original dataset, however, we utilize a reduced labeling scheme, eliminating both positive surprise and negative surprise from the set of emotions. We follow the reasoning of Susanto et al. (2020) and Ortony (2022), who argue that surprise can not be considered a basic emotion, as it is not valenced, i. e., of negative or positive polarity, in itself but can only be polarised in combination with other emotions.
Krippendorff’s alpha (α) for all three annotators is .385, when calculated based on single sentences. Details on agreements are provided in Appendix B. Removal of 7 low-agreement stories (cf. Appendix B) leaves us with a final data set of 169 stories. Key statistics of the data are summarized in Table 1.
The label distribution statistics listed in Table 1 indicate stylistic differences between the different authors. To give an example, sadness is rare in Potter’s stories (3.97% of all annotations) compared to the other two authors. Overall, neutral is the most frequent label, while other classes, especially positive surprise and disgust, are underrepresented.
Figure 1 shows confusion matrices comparing the annotations of annotator 1 with the annotations of annotators 2 and 3. The decision of whether a sentence is emotional or neutral is the most im-
portant source of disagreement in both annotator pairs. Furthermore, Figure 1 demonstrates that disagreement about the valence, i. e., pleasantness, of a sentence’s mood is rare. To give an example, in both depicted confusion matrices, sentences labeled with happiness by annotator 1 are rarely labeled with a negative emotion (anger, disgust, fear) by annotator 2 and 3, respectively.

heading: 3.2 Label Mapping
text: Motivated by low to moderate Krippendorff agreements (cf. Appendix B) and underrepresented classes in the discrete annotations (cf. Table 1), we project all emotion labels into the more generic, continuous valence/arousal space. Proposed by (Russell, 1980), the valence/arousal model characterizes affective states among two continuous dimensions where valence corresponds to pleasantness, while arousal is the intensity or degree of agitation. As depicted in Figure 1, the annotators often agree on the polarity of the emotion, i. e., whether it is to be understood as positive or negative in terms of valence. Hence, it can be argued that disagreement between annotators is not always as grave as suggested by low α values, which do not take proximity between different emotions into account. To give an example, disagreement on whether a sentence’s mood is happiness or neutral is certainly less severe than one annotator labeling the sentence sad, while the other opts for happy. Moreover, a projection into continuous space unifies the two different label spaces defined by the original and our additional annotations, respectively. To implement the desired mapping, we take up an idea proposed by Park et al. (2021), who map discrete emotion categories to valence and arousal values by looking up the label (e. g., anger) in the NRC-VAD dictionary (Mohammad, 2018) that assigns crowd-sourced valence and arousal val-
ues in the range [0...1] to words. For instance, the label anger is mapped to a valence value of .167 and an arousal value of .865. The full mapping and further explanations can be found in Appendix C.
After label mapping, we create a gold standard for every story by fusing the thus obtained signals over the course of a story for valence and arousal, respectively. We apply the EWE (Grimm and Kroschel, 2005) method which is well-established for the problem of computing valence and arousal gold standards from continuous signals (e. g., (Ringeval et al., 2019; Stappen et al., 2021; Christ et al., 2022b)). Figure 2 presents an example for this process, presenting both the discrete labels and the valence and arousal signals constructed from them for a specific story.

heading: 3.3 Split
text: We split the data on the level of stories. Three partitions for training, development, and test are created, with 118, 25, and 26 stories, respectively. In doing so, we make sure to include comparable portions of stories and sentences by each author in all three partitions. A detailed breakdown is provided in Appendix D.

heading: 4 Experimental Setup
text: We fine-tune (cf. Section 4.1) the 304M parameter large version of DeBERTaV3 (He et al., 2023), additionally utilizing a weakly supervised learning approach (Section 4.2). Further details regarding the computational resources can be found in Appendix E.

heading: 4.1 Finetuning
text: Since the context of a sentence in a story is relevant to the mood it conveys, we seek to leverage multiple sentences at once in the fine-tuning process. Specifically, we create training examples as follows. We denote a story as a sequence of sentences s1 . . . sn. For a sentence si and a context window size C, we also consider the up to C sentences preceding (si−C . . . si−1) and the up to C sentences following (si+1 . . . si+C) si. We construct an input string from the sentences si−C . . . si+C by concatenating them, separated via the special [SEP] token. The i-th [SEP] token in this sequence is intended to represent the ith sentence. We add a token-wise feed-forward layer on top of the last layer’s token representations. It projects each 1024-dimensional embedding to 2 dimensions and is followed by
Sigmoid activation for both of them, corresponding to a prediction for valence and arousal, respectively. As the loss function, we sum up the Mean Squared Errors (MSEs) of valence and arousal predictions for each [SEP] token. We optimize C, for C ∈ {1, 2, 4, 8}. If the length of an input exceeds the model’s capacity, we decrease C for this specific input. Figure 3 provides an example of an input sequence.
We train the models for at most 10 epochs but abort the training process early if no improvement on the development set is achieved for 2 consecutive epochs. The evaluation metric is the mean of the Concordance Correlation Coefficient (CCC) (Lawrence and Lin, 1989) values achieved for arousal and valence, computed over the whole dataset, respectively. Equation (1) gives the formula for CCC between two signals Y and Ŷ of equal length.
CCC(Y, Ŷ ) = 2 Cov(Y, Ŷ )
Var(Y ) + Var(Ŷ ) + (Y − Ŷ )2 (1)
CCC is a well-established correlation measure to assess agreement between (pseudo-)time continuous annotations and predictions, particularly common in Affective Computing Tasks, e. g., (Ringeval et al., 2018; Schoneveld et al., 2021; Christ et al., 2023). It can be thought of as a bias-corrected modification of Pearson’s correlation. Different from Pearson’s correlation, it is sensitive to location and scale shifts, i. e., it measures not only correlation but also takes into account absolute errors. Same as for the Pearson correlation, the chance level is 0, and two identical signals would have a CCC value of 1. AdamW (Loshchilov and Hutter, 2019) is chosen as the optimization method. Following a preliminary hyperparameter search, the learning rate is set to 5 × 10−6. We do not optimise any hyperparameter besides the learning rate. Every experiment is repeated with five fixed seeds. In every experiment, we initialize the model with the checkpoint provided by the DeBERTaV3 authors1.

heading: 4.2 Weakly Supervised Learning
text: The Alm dataset comprises 169 stories by only three different authors, making our models prone to overfitting. Thus, we seek to augment our data set with in-domain texts written by other authors,
1https://https://huggingface.co/microsoft/deberta-v3large
thereby covering more topics and also spanning more cultures. We collect 45 books containing in total 801 different stories from Project Gutenberg, more specifically the Children’s Myths, Fairy Tales, etc. category2. These stories comprise fairytales, myths, and other tales from different geographic regions, including Japan, Ireland, and India. This newly collected unlabeled data set, henceforth referred to as Gutenberg Corpus or GB, amounts to 101529 sentences. A more detailed description of GB is given in Appendix F.
2https://www.gutenberg.org/ebooks/bookshelf/216
Figure 4 illustrates our overall finetuning approach. Given 1) a DeBERTa model finetuned on the labeled dataset (cf. 4.1), we 2) utilize its predictions on GB as pseudo-labels, yielding a labeled dataset GBALM. Subsequently, 3) another pretrained DeBERTa model is finetuned on GBALM only. This training process is limited to 1 epoch and employs a learning rate of 5× 10−1. Lastly, 4) this model is further trained on ALM. Here, we utilize the same hyperparameters as for training M , but we find a smaller learning rate of 10−7 to be beneficial.
Train
Alm DeBERTa FT (Alm)
1) Finetune pretrained DeBERTa with labelled Alm data

heading: 5 Results
text: Table 2 presents the results of our experiments with different C values. We report the mean CCC results when tuning a) only on ALM (FT ALM, step 1 in Figure 4), b) only on GBALM (FT GBALM, step 3
in Figure 4) and c) additionally on ALM (FT GBAlm + ALM, step 4 in Figure 4). There is a clear trend for both arousal and valence to increase with larger Cs. The models trained with a context size of 8 account for the best valence and arousal results in every set of experiments, e. g., CCC values of .8168 and .6809 for arousal and valence, respectively, on the development set when trained on both corpora. These are also the best results encountered overall. In contrast, the models with C = 0 always perform worst, yielding e. g., only CCC values of .6798 (valence) and .5576 (arousal) on the development set in the ALM-only configuration. This supports the assumption that the context of a sentence is oftentimes key to correctly assessing its mood. The gap between valence and arousal CCC values is in line with previous studies showing that text-based classifiers are typically better suited for valence prediction than for arousal prediction (Kossaifi et al., 2019; Wagner et al., 2023). Further, our results demonstrate the benefits of the weakly supervised approach. Training on GBALM always improves upon training on ALM only, especially for smaller context sizes C. To give an example, both the valence and arousal CCC values on the development set increase by more than 1.5 for C = 0 on the development set. Further tuning on ALM afterward leads to additional performance gains for both prediction targets and all context sizes. However, the increase never exceeds 1 percentage point in comparison to training on GBALM.

heading: 5.1 Author-Wise Results
text: Since ALM comprises stories of three different authors, we investigate the relevance of an author’s individual style for learning to predict their stories. For each author (AUTH), we create a dataset ALM\{AUTH} by removing AUTH’s stories from the training and development partitions and keep-
ing only AUTH’s stories as test data. We then repeat steps 1-3 in Figure 4, using ALM\{AUTH} instead of the full ALM dataset. Only the best configuration, i. e., C = 8 is considered here. The results of these experiments, alongside the corresponding author-wise results, when employing the full ALM dataset, are given in Table 3. Performance, in general, differs by author, e. g., both valence and arousal CCC for Potter are lower than for the other two authors when training on the full ALM dataset. Furthermore, test set performance for every author drops when removing the author from the training and development data. The clearest example is Potter’s arousal CCC value of .5886 when training on ALM\{Potter} compared to .6442 when training on the full ALM data. The weakly supervised learning step, implying exposure to a wider range of styles, proves to be beneficial for every author, regardless of the dataset. Nevertheless, for every author AUTH the performance of the weakly supervised approach on ALM\{AUTH} never reaches the performance for fine-tuning on ALM alone. In conclusion, it is crucial to include targeted authors in training data in order to capture their individual styles.

heading: 5.2 Further Statistics
text: In the remainder of the paper, we limit our analysis to the best-performing seed for C = 8 and the full training pipeline (cf. Figure 4).
The CCC values given in Table 2 are calculated over the entire dataset, i. e., a concatenation of all stories per partition. Table 4, in contrast, lists story-wise CCC results for the predictions of our best model.
It shows that results are highly story-dependent. To give an example, the arousal CCC values for the test partition display a standard deviation of .1679 over the 25 stories in this partition.
We find that the model’s performance for arousal and valence per story correlates: we obtain a Pearson’s correlation of .3499 (statistically significant with p < .02) between our best model’s valence CCC values per story and the respective CCC arousal values. Hence, there exist stories whose emotional trajectories are difficult (or easy) to predict for the model in general, regardless of the two different emotional dimensions.
This can partly be explained by the correlation between model performance and human agreement per story. There is a Pearson’s correlation of .3659 between all story-wise human CCC agreements and the CCC values achieved by the best model on the corresponding stories. Analogously, for arousal, this correlation is .4095. Both correlations are statistically significant with p < .02. It can be concluded that the model particularly struggles to learn stories that also pose a challenge to humans.
Another analysis reveals that the model’s performance also tends to vary for different parts of the same story. We divide every story into 5 parts of equal size. This way, we evaluate the performance of our best models on 5 different subsets of the data corresponding to positions in the story. Roughly, the first part can be expected to correspond to the beginning of the story, while the last part comprises its end. Table 6 displays the results of this evaluation. Both valence and arousal results are, on average, better at the very beginning (.8260 valence, .7304 arousal CCC) and the very end (.8576
valence, .7306 arousal CCC) of the stories than during their middle parts. We hypothesize that this is due to many stories’ beginnings and endings being drawn from a limited set of archetypical situations. Hence, the model may easily learn the emotional connotations of such common events from the large corpus. To give a few examples, fairytales in particular often start, e. g., with the death or absence of a parent, the hero leaving home, an act of villainy against the hero, or a combination thereof. Endings often involve reunion, marriage, and the villain receiving punishment (Propp, 1968).
As a measure of model performance on the sentence level, we compute the best model’s absolute prediction errors on the development and test set. Table 5 presents the results.
It is evident from the median values that more than half of arousal and valence predictions miss the gold standard by less than .1. From the percentiles, it can be concluded that errors larger than .3 occur in less than 5% of sentences for valence and in less than 10% of sentences for arousal.

heading: 5.3 Qualitative Analysis
text: To gain qualitative insights into the model’s limitations, we manually analyze around 200 text spans for which high absolute errors in terms of valence or arousal prediction are observed. First, we find that the model seems to learn emotional connotations of events, but is prone to ignore the roles of the protagonists involved in them. Table 7 provides an example of this phenomenon. In this text passage, a typically positive event, namely being granted a wish, is salient. The model assigns rel-
atively high valence values. However, the actual mood in these sentences is rather negative, as they describe the implicitly jealous reaction of a negative character to this situation.
Probably closely related to these observations, we figure that our model sometimes struggles to accurately assess situations, because it disregards the general sentiment of the respective story. To give an example, Table 8 lists a passage from Andersen’s story grandmot. This story displays a rather positive sentiment overall, as it is presented as a loving memory of a deceased grandmother. For the passages describing her peaceful death, our model underestimates the valence gold standard by a large margin, probably due to the typically sad topic of death.
Stories within stories pose another facet the model faces difficulties with. Frequently, protagonists tell stories or recall memories. Narrated stories or memories typically contain emotionally significant events, but they are not directly experienced and thus are not always heavily influencing the mood of the actual story. Table 9 presents an example, where a cat tells another one about a fearful incident. The corresponding gold standard arousal values are moderate, arguably as the inci-
dent is over and has not harmed the protagonist. The model nevertheless predicts high arousal.
To summarise, the model tends to miss out on a holistic understanding of stories such as the roles of different protagonists, nested stories, and a story’s overall tone. This can partially be attributed to inputs not consisting of complete stories, cf. Section 4.1. Further examples for all aspects discussed above can be found in Appendix G.

heading: 6 Discussion
text: We demonstrate the efficacy of our approach to model emotional trajectories via LLMs, achieving CCC values of .8221 and .6809 for valence and arousal on the test set, respectively. We find that considering a sentence’s context is crucial for predicting its emotionality. Furthermore, our analysis reveals the author-dependence of these results, which, in addition, vary from story to story. Even within a story, certain parts (namely, beginning and ending) are often easier to predict than others. Further analysis of our models’ predictions uncovers additional challenges, such as assigning the correct role to protagonists and understanding the overall tone of a story. All these aspects combined shed light on the complexity of the task at hand. Keeping this in mind, our methodology can be understood as a first benchmark for predicting emotional trajectories in a supervised manner.

heading: 7 Conclusion
text: We proposed a valence/arousal-based gold standard for the Alm dataset (Alm, 2008). Moreover, we provide first results for the prediction of these signals via finetuning DeBERTa combined with a weakly supervised learning step. We obtain promising results, but, at the same time, demonstrate the limits of this methodology in our analysis. Future work may include attempts at a more holistic story understanding, involving e. g., the roles
of protagonists. Besides, our analysis of the results by author suggests that personalization methods (e. g., (Kathan et al., 2022)) may improve the results. Further, the potential of even larger LLMs such as LLaMA (Touvron et al., 2023) or (Chat-)GPT (Achiam et al., 2023) remains to be explored for this task. Such models may even assist in refining the rather simplistic mapping method we utilized for the creation of the gold standard, as they have been shown to come with inherent emotional understanding capabilities (Broekens et al., 2023; Tak and Gratch, 2023). Code, data, and model weights are released to the public3.

heading: 8 Limitations
text: Our work comes with several constraints. The simple mapping from discrete emotions into the dimensional valence/arousal space (cf. Table 12) may be too coarse to capture some texts’ emotional connotations. When analyzing the best model’s predictions, we encounter texts where such shortcomings of our label mapping approach (cf. Table 12) surface. For instance, all instances of disgust are mapped to high arousal, leaving no room for less frequent low-arousal variants of disgust as can be found in passages like the one given in Table 10. Here, the sentences’ mood was predominantly labeled as disgust and thus set to high gold standard arousal values. The considerably lower arousal predictions by our model, however, are arguably more appropriate than the gold standard here, as the described situation is rather characterized by distanced arrogance than by actual disgust.
Besides, our approach to weakly supervised learning is obviously limited to high-resource languages. Story is a broad term applicable to all texts in both ALM and the crawled GB data. The included stories could be distinguished in a more fine-grained manner, e. g., the data contains fairytales, myths, fables, and other types of stories. Such
3https://github.com/lc0197/emotional_trajectories_stories
distinctions may have methodologically relevant implications we do not consider in our experiments. We also show that emotional arcs are highly authordependent (cf. Section 5.2), implying that future datasets should seek to comprise a wider range of authors and writing styles. In particular, our results may not generalize well to authors of backgrounds that are not represented in the data used. Lastly, we analyse our method’s limitations in Section 5.3, without claim of completeness.

heading: Acknowledgements
text: Shahin Amiriparian, Manuel Milling, and Björn W. Schuller are also with the Munich Center for Machine Learning (MCML). Additionally, Björn W. Schuller is with the Munich Data Science Institute (MDSI) and the Konrad Zuse School of Excellence in Reliable AI (relAI), all in Munich, Germany and acknowledges their support.

heading: A Annotation Details
text: Our additional annotations (cf. Section 3.1) are carried out by a 24-year-old male PhD student with a solid background in Affective Computing concepts, in particular, different emotion models. Hence, A3 is the same person for all stories, while this is not the case for A1 and A2 (cf. (Alm and Sproat, 2005; Alm, 2008)). Figure 5 shows a screenshot of the annotation tool.

heading: B Agreement Statistics
text: Krippendorff’s alpha (α) for all three annotators is .385, when calculated based on single sentences and ignoring the different label schemes. This is possible, as annotator 3’s label scheme is a subset of the labels available to annotators 1 and 2. The mean α per story is µα = .341, with a standard deviation of σα = .126, indicating that the level of agreement is highly dependent on the story. We remove stories whose α is smaller than µα − 2σα. A detailed listing of α values for the remaining data on both the sentence and the story level is provided in Table 11.
Table 11 illustrates that agreement is also authordependent, e. g., for all combinations of annotators, the sentence-wise agreement for the Grimm brothers is lower than for both other authors.

heading: C Label Mapping Details
text: Table 12 lists the mapping for all discrete emotion labels as obtained from the NRC-VAD dictionary (Mohammad, 2018). However, the dictionary does not contain entries for positive surprise and negative surprise. For positive surprise, we take the valence and arousal values of surprise (both .875). The valence value for negative surprise is set to the
mean valence value of the negative emotions anger, disgust, and fear (.097), while the arousal value is the same as for positive surprise (.875).
There are a few similar attempts to mapping discrete to continuous emotion models, but no agreedupon gold standard method to do so. Our decision for this particular method is motivated by three criteria: 1) the method should yield a numeric value (in contrast to approaches like (Amiriparian et al., 2024; Gerczuk et al., 2021) that utilize categories such as “low valence” etc.) 2) the values should, of course, match our expectations based on Russel’s circumplex model (Russell, 1980) regarding the position of the discrete emotions in the V/A space, and 3) the method must be able to account for all labels in the dataset. We could, e.g., not utilize the V/A mappings for discrete emotions collected
in (Hoffmann et al., 2012), as they do not obtain values for surprise, disgust, and neutral. Admittedly, the method we selected has this problem for negative surprise as well, but we found a relatively straightforward way to make up for this shortcoming.
We validate the mapping approach by obtaining additional valence/arousal labels from the same annotator for 3 randomly selected stories. Table 13 reports the Pearson correlation between these direct valence/arousal annotations and those obtained by the proposed mapping.
The correlations illustrate again that the difficulty of the problem varies for different stories. Moreover, the correlations for valence are higher than those for arousal, indicating that the method may capture valence better than arousal. This observation may also contribute to explaining why the automatic prediction of arousal proves to be
more difficult than the prediction of valence.

heading: D Split Statistics
text: Table 14 displays detailed statistics for the split into training, development, and test sets.
Figure 6 shows that the continuous label distributions are fairly similar in the different partitions.

heading: E Further Experiment Details
text: All experiments were carried out on an NVIDIA RTX3090 GPU and took about 200 GPU hours in total. For illustration, we calculate the rough number of training/prediction steps for one experiment, i. e., one configuration (e. g., C = 4) and one seed. The Alm dataset comprises about 15k data points (about 10k of which are used for training), the Gutenberg dataset contains about 100k sentences. Assuming that steps 1) and 4) in Figure 4 run for 5 epochs each and step 3) takes one epoch, all of them using a batch size of 4, we end up with (5 ∗ (100k+10k) + 110k)/4 = 165k training steps. Multiplying this with 5 seeds and 5 Cconfigurations, results in about 4M training steps overall, notwithstanding some preliminary hyperparameter optimization and the additional authorindependent experiments. These rather large resource requirements also motivate our choice for the relatively small 304M parameter DeBERTa model.

heading: F Gutenberg Corpus
text: In Table 21, all books used for creating the GB are listed. We make sure not to include tales written by the three authors in the labeled dataset. We do not carry out any further filtering or manual screening steps. Basic preprocessing steps such as the removal of footnotes and images are conducted before we split the stories into sentences utilizing the PySBD (Sadvilkar and Neumann, 2020) library.

heading: G Further Qualitative Analysis
text: In this section, we provide further examples of passages for which the model’s predictions result in large errors, thus extending Section 5.3.
Table 15 displays a text passage revolving around the theme of marriage. The model predicts high valence values, arguably due to this oftentimes positive topic. However, in this particular context, the planned marriage is viewed as negative by the protagonist.
In Table 16, an example is provided in which the model assesses a situation as positive, in which a person has gained a great amount of power. The gold standard, in contrast, assigns low valence values to this passage, as the protagonist exhibits greed and megalomania, aspects seemingly ignored by the model.
The phenomenon of our model missing out on the overall tone of stories is further exemplified by the text in Table 17. Here, the protagonists are pigs behaving like and interacting with humans, which gives the entire story a funny mood. In this context, a typically rather exciting situation (being interrogated by the police) is not assigned a high arousal value by the gold standard – different from the model.
Another example from an overall funny story is given in Table 18. The protagonists encounter several robbers, but the situation is labeled with a neutral valence value in the gold standard. The model assigns low valence values, missing out on the funny tone of the entire story.
Moreover, we provide further examples of the model struggling with stories within stories. In the passage given in Table 19, the model overestimates the arousal value, as the story told about a great fire is arguably very exciting.
The example presented in Table 20 demonstrates a passage where valence is overestimated by the model. Here, happy memories are recalled in a
sad context, giving the text a sad mood that is not properly assessed by the model.

heading: Future Work
text: 2), implying that future datasets should seek to comprise a wider range of authors and writing styles. In particular, our results may not generalize well to authors of backgrounds that are not represented in the data used. Lastly, we analyse our method’s limitations in Section 5. 3, without claim of completeness