Dataset ::: Passage Selection	We selected both classic and modern narratives in English for this dataset. The modern narratives were sampled based on popularity from Wattpad. We parsed selected narratives into passages, where a passage is considered to be eligible for annotation if it contained between 40 and 200 tokens. In long-form narratives, many non-conversational passages are intended for transition or scene introduction, and may not carry any emotion. We divided the eligible passages into two parts, and one part was pruned using selected emotion-rich but ambiguous lexicons such as cry, punch, kiss, etc.. Then we mixed this pruned part with the unpruned part for annotation in order to reduce the number of neutral passages. See Appendix SECREF25 for the lexicons used.

Introduction	Humans experience a variety of complex emotions in daily life. These emotions are heavily reflected in our language, in both spoken and written forms. Many recent advances in natural language processing on emotions have focused on product reviews  and tweets , . These datasets are often limited in length (e.g. by the number of words in tweets), purpose (e.g. product reviews), or emotional spectrum (e.g. binary classification). Character dialogues and narratives in storytelling usually carry strong emotions. A memorable story is often one in which the emotional journey of the characters resonates with the reader. Indeed, emotion is one of the most important aspects of narratives. In order to characterize narrative emotions properly, we must move beyond binary constraints (e.g. good or bad, happy or sad). In this paper, we introduce the Dataset for Emotions of Narrative Sequences (DENS) for emotion analysis, consisting of passages from long-form fictional narratives from both classic literature and modern stories in English. The data samples consist of self-contained passages that span several sentences and a variety of subjects. Each sample is annotated by using one of 9 classes and an indicator for annotator agreement.

Though passages may lose some emotional context when read independently of the complete narrative, we believe annotator agreement on our dataset supports the assertion that small excerpts can still convey coherent emotions. During the annotation process, several annotators had suggested for us to include additional emotions such as confused, pain, and jealousy, which are common to narratives. As they were not part of the original Plutchik’s wheel, we decided to not include them. An interesting future direction is to study the relationship between emotions such as ‘pain versus sadness’ or ‘confused versus surprise’ and improve the emotion model for narratives.

Dataset ::: Mechanical Turk (MTurk)	MTurk was set up using the standard sentiment template and instructed the crowd annotators to `pick the best/major emotion embodied in the passage'. We further provided instructions to clarify the intensity of an emotion, such as: “Rage/Annoyance is a form of Anger”, “Serenity/Ecstasy is a form of Joy”, and “Love includes Romantic/Family/Friendship”, along with sample passages. We required all annotators have a `master' MTurk qualification. Each passage was labelled by 3 unique annotators. Only passages with a majority agreement between annotators were accepted as valid. This is equivalent to a Fleiss's $\kappa $ score of greater than $0.4$. For passages without majority agreement between annotators, we consolidated their labels using in-house data annotators who are experts in narrative content. A passage is accepted as valid if the in-house annotator's label matched any one of the MTurk annotators' labels. The remaining passages are discarded. We provide the fraction of annotator agreement for each label in the dataset.

Benchmarks ::: Hierarchical RNN	For this benchmark, we considered a Hierarchical RNN, following . We used two BiLSTMs  with 256 units each to model sentences and documents. The tokens of a sentence were processed independently of other sentence tokens. For each direction in the token-level BiLSTM, the last outputs were concatenated and fed into the sentence-level BiLSTM as inputs. The outputs of the BiLSTM were connected to 2 dense layers with 256 ReLU units and a Softmax layer. We initialized tokens with publicly available embeddings trained with GloVe . Sentence boundaries were provided by SpaCy. Dropout was applied to the dense hidden layers during training.

Dataset ::: Dataset Statistics	The dataset contains a total of 9710 passages, with an average of 6.24 sentences per passage, 16.16 words per sentence, and an average length of 86 words. The vocabulary size is 28K (when lowercased). It contains over 1600 unique titles across multiple categories, including 88 titles (1520 passages) from Project Gutenberg. All of the modern narratives were written after the year 2000, with notable amount of themes in coming-of-age, strong-female-lead, and LGBTQ+. The genre distribution is listed in Table TABREF8. In the final dataset, 21.0% of the data has consensus between all annotators, 73.5% has majority agreement, and 5.48% has labels assigned after consultation with in-house annotators. The distribution of data points over labels with top lexicons (lower-cased, normalized) is shown in Table TABREF9. Note that the Disgust category is very small and should be discarded. Furthermore, we suspect that the data labelled as Surprise may be noisier than other categories and should be discarded as well.

Dataset	In this section, we describe the process used to collect and annotate the dataset.

The final annotation categories for the dataset are: Joy, Sadness, Anger, Fear, Anticipation, Surprise, Love, Disgust, Neutral.

Benchmarks ::: Bi-directional RNN and Self-Attention (BiRNN + Self-Attention)	One challenge with RNN-based solutions for text classification is finding the best way to combine word-level representations into higher-level representations. Self-attention , ,  has been adapted to text classification, providing improved interpretability and performance. We used  as the basis of this benchmark. The benchmark used a layered Bi-directional RNN (60 units) with GRU cells and a dense layer. Both self-attention layers were 60 units in size and cross-entropy was used as the cost function. Note that we have omitted the orthogonal regularizer term, since this dataset is relatively small compared to the traditional datasets used for training such a model. We did not observe any significant performance gain while using the regularizer term in our experiments.

Background	Using the categorical basic emotion model , ,  studied creating lexicons from tweets for use in emotion analysis. Recently, ,  and  proposed shared-tasks for multi-class emotion analysis based on tweets. Fewer works have been reported on understanding emotions in narratives. Emotional Arc  is one recent advance in this direction. The work used lexicons and unsupervised learning methods based on unlabelled passages from titles in Project Gutenberg. For labelled datasets on narratives,  provided a sentence-level annotated corpus of childrens' stories and  provided phrase-level annotations on selected Project Gutenberg titles. To the best of our knowledge, the dataset in this work is the first to provide multi-class emotion labels on passages, selected from both Project Gutenberg and modern narratives. The dataset is available upon request for non-commercial, research only purposes.

Dataset ::: Plutchik’s Wheel of Emotions	The dataset is annotated based on a modified Plutchik’s wheel of emotions. The original Plutchik’s wheel consists of 8 primary emotions: Joy, Sadness, Anger, Fear, Anticipation, Surprise, Trust, Disgust. In addition, more complex emotions can be formed by combing two basic emotions. For example, Love is defined as a combination of Joy and Trust (Fig. 1). The intensity of an emotion is also captured in Plutchik's wheel. For example, the primary emotion of Anger can vary between Annoyance (mild) and Rage (intense). We conducted an initial survey based on 100 stories with a significant fraction sampled from the romance genre. We asked readers to identify the major emotion exhibited in each story from a choice of the original 8 primary emotions. We found that readers have significant difficulty in identifying Trust as an emotion associated with romantic stories. Hence, we modified our annotation scheme by removing Trust and adding Love. We also added the Neutral category to denote passages that do not exhibit any emotional content.

DENS: A Dataset for Multi-class Emotion Analysis	We introduce a new dataset for multi-class emotion analysis from long-form narratives in English. The Dataset for Emotions of Narrative Sequences (DENS) was collected from both classic literature available on Project Gutenberg and modern online narratives available on Wattpad, annotated using Amazon Mechanical Turk. A number of statistics and baseline benchmarks are provided for the dataset. Of the tested techniques, we find that the fine-tuning of a pre-trained BERT model achieves the best results, with an average micro-F1 score of 60.4%. Our results show that the dataset provides a novel opportunity in emotion analysis that requires moving beyond existing sentence-level techniques.

Appendices ::: Sample Data	Table TABREF26 shows sample passages from classic titles with corresponding labels.

Conclusion	We introduce DENS, a dataset for multi-class emotion analysis from long-form narratives in English. We provide a number of benchmark results based on models ranging from bag-of-word models to methods based on pre-trained language models (ELMo and BERT). Our benchmark results demonstrate that this dataset provides a novel challenge in emotion analysis. The results also demonstrate that attention-based models could significantly improve performance on classification tasks such as emotion analysis. Interesting future directions for this work include: 1. incorporating common-sense knowledge into emotion analysis to capture semantic context and 2. using few-shot learning to bootstrap and improve performance of underrepresented emotions. Finally, as narrative passages often involve interactions between multiple emotions, one avenue for future datasets could be to focus on the multi-emotion complexities of human language and their contextual interactions.

Benchmarks	We performed benchmark experiments on the dataset using several different algorithms. In all experiments, we have discarded the data labelled with Surprise and Disgust. We pre-processed the data by using the SpaCy pipeline. We masked out named entities with entity-type specific placeholders to reduce the chance of benchmark models utilizing named entities as a basis for classification. Benchmark results are shown in Table TABREF17. The dataset is approximately balanced after discarding the Surprise and Disgust classes. We report the average micro-F1 scores, with 5-fold cross validation for each technique. We provide a brief overview of each benchmark experiment below. Among all of the benchmarks, Bidirectional Encoder Representations from Transformers (BERT)  achieved the best performance with a 0.604 micro-F1 score. Overall, we observed that deep-learning based techniques performed better than lexical based methods. This suggests that a method which attends to context and themes could do well on the dataset.

Benchmarks ::: Bag-of-Words-based Benchmarks	We computed bag-of-words-based benchmarks using the following methods: Classification with TF-IDF + Linear SVM (TF-IDF + SVM) Classification with Depeche++ Emotion lexicons  + Linear SVM (Depeche + SVM) Classification with NRC Emotion lexicons ,  + Linear SVM (NRC + SVM) Combination of TF-IDF and NRC Emotion lexicons (TF-NRC + SVM)

Benchmarks ::: Doc2Vec + SVM	We also used simple classification models with learned embeddings. We trained a Doc2Vec model  using the dataset and used the embedding document vectors as features for a linear SVM classifier.

Benchmarks ::: ELMo embedding and Bi-directional RNN (ELMo + BiRNN)	Deep Contextualized Word Representations (ELMo)  have shown recent success in a number of NLP tasks. The unsupervised nature of the language model allows it to utilize a large amount of available unlabelled data in order to learn better representations of words. We used the pre-trained ELMo model (v2) available on Tensorhub for this benchmark. We fed the word embeddings of ELMo as input into a one layer Bi-directional RNN (16 units) with GRU cells (with dropout) and a dense layer. Cross-entropy was used as the cost function.

Benchmarks ::: Fine-tuned BERT	Bidirectional Encoder Representations from Transformers (BERT)  has achieved state-of-the-art results on several NLP tasks, including sentence classification. We used the fine-tuning procedure outlined in the original work to adapt the pre-trained uncased BERT$_\textrm {{\scriptsize LARGE}}$ to a multi-class passage classification task. This technique achieved the best result among our benchmarks, with an average micro-F1 score of 60.4%.

Table TABREF10 shows a few examples labelled data from classic titles. More examples can be found in Table TABREF26 in the Appendix SECREF27.

Question: How many annotators were there?
Answer: 3