SlowGuess/NAACL_2022_OCR huggingface dataset:
Effi Levi<sup>1</sup>, Guy Mor<sup>2</sup>, Tamir Sheafer<sup>2,3</sup>, Shaul R. Shenhav<sup>2</sup>
$^{1}$ Institute of Computer Science, The Hebrew University of Jerusalem
$^{2}$ Department of Political Science, The Hebrew University of Jerusalem
$^{3}$ Department of Communication and Journalism, The Hebrew University of Jerusalem
efle@cs.huji.ac.il
{guy.mor|tamir.sheafer|shaul.shenhav}@mail.huji.ac.il
Automatic extraction of narrative elements from text, combining narrative theories with computational models, has been receiving increasing attention over the last few years. Previous works have utilized the oral narrative theory by Labov and Waletzky to identify various narrative elements in personal stories texts. Instead, we direct our focus to informational texts, specifically news stories.
We introduce NEAT (Narrative Elements AnnoTation) - a novel NLP task for detecting narrative elements in raw text. For this purpose, we designed a new multi-label narrative annotation scheme, better suited for informational text (e.g. news media), by adapting elements from the narrative theory of Labov and Waletzky (Complication and Resolution) and adding a new narrative element of our own (Success). We then used this scheme to annotate a new dataset of 2,209 sentences, compiled from 46 news articles from various category domains<sup>1</sup>. We trained a number of supervised models in several different setups over the annotated dataset to identify the different narrative elements, achieving an average  $F_{1}$  score of up to 0.77. The results demonstrate the holistic nature of our annotation scheme as well as its robustness to domain category.
Automatic extraction of narrative elements from texts is a multidisciplinary field of research, combining narrative theories with computational models, which has been receiving increasing attention over the last few years. Examples include modeling narrative structures for story generation (Gervás et al., 2006), using unsupervised methods to detect narrative event chains (Chambers and Jurafsky, 2008) and detecting content zones (Baiamonte
et al., 2016) in news articles, using semantic features to detect narreme boundaries in fictitious prose (Delmonte and Marchesini, 2017), identifying turning points in movie plots (Papalampidi et al., 2019) and using temporal word embeddings to analyze the evolution of characters in the context of a narrative plot (Volpetti et al., 2020).
A recent and more specific line of work focuses on using the theory laid out by Labov and Waletzky (1967) and later refined by Labov (2013) to characterize narrative elements in personal experience texts. Swanson et al. (2014) relied on Labov and Waletzky (1967) to annotate a corpus of 50 personal stories from weblogs posts, and tested several models over hand-crafted features to classify clauses into three narrative clause types: orientation, evaluation and action. Ouyang and McKeown (2014) constructed a corpus from 20 oral narratives of personal experience collected by Labov (2013), and utilized logistic regression over hand-crafted features to detect instances of complicating actions. More recently, Li et al. (2017) utilized a combination of ideas from Labov and Waletzky (1967) and Freytag (1894) to annotate a collection of short stories, and Saldias and Roy (2020) used convolutional neural networks (CNNs) to classify clauses from spoken personal texts into the same three narrative clause types as Swanson et al. (2014).
While these works concentrated their effort on narrative analysis of personal experience texts, we direct our focus to detecting narrative patterns in informational texts, such as news stories. The social impact of news stories distributed by the media and their role in creating and shaping of public opinion incentivized our efforts to adapt narrative analysis approaches to this domain. To the best of our knowledge, this is the first attempt to automatically detect narrative elements based on Labov and Waletzky (1967) and later works by Labov (1972, 2013) in news articles.
In this work, we introduce NEAT (Narrative Ele
'https://github.com/efle/NEAT
Findings of the Association for Computational Linguistics: NAACL 2022, pages 1755 - 1765 July 10-15, 2022 ©2022 Association for Computational Linguistics
ments AnnoTation) – a novel NLP task for detecting narrative elements in raw text. For this purpose, we adapted two elements from the narrative theory presented in Labov and Waletzky (1967); Labov (1972, 2013), namely Complication and Resolution, while adding a new narrative element, Success, to create a new multi-label narrative annotation scheme. This scheme was designed with two main objectives in mind. First, capturing elements oriented towards discourse structure, rather than semantic content. Second, possessing the flexibility required to capture narrative characteristics within a wide variety of text types, specifically informational text (as opposed to personal experience), and not only literary and well-structured stories. We used this scheme to annotate a newly-constructed dataset of 2,209 sentences, compiled from 46 English news articles; each sentence was tagged with a subset of the three narrative elements (or, in some cases, none of them), thus defining a novel multi-label classification task.
We explored two different approaches towards solving our new task: splitting into three unrelated binary classification tasks (Complication, Resolution and Success), and jointly learning the three narrative categories as a multi-label classification task. We experimented with three supervised models, each based on fine-tuning a different pre-trained language model: BERT (Devlin et al., 2018), RoBERTa (Liu et al., 2019) and DistilBERT (Sanh et al., 2020), achieving an average  $F_{1}$  score of up to 0.77. An analysis of the results indicates that our narrative categories are strongly connected and form a coherent narrative scheme which is more than just the sum of its parts. Additional experimentation with cross-domain classification demonstrates the task's robustness to domain category, suggesting that our annotation scheme is more grounded in discourse characteristics rather than semantic context.
The remainder of this paper is organized as follows: Section 2 gives a theoretical background and describes the adjustments we have made to the scheme in Labov (2013) in order to adapt it to informational text. Section 3 provides a complete description of the dataset and of the processes and methodologies which were used to construct and annotate it, along with a short analysis and some examples for annotated sentences. Section 4 describes the experiments conducted on the dataset, and Section 5 provides an analysis and a discus
sion of the results. Finally, Section 6 contains a summary of our contributions as well as several potential directions for future work.
2 Narrative Analysis
Ever since the emergence of formalism and structuralistic literary criticism (Propp, 1968) and throughout the development of narratology (Genette, 1980; Fludernik, 2009; Chatman, 1978; Rimmon-Kenan, 2003), narrative structure has been the focus of extensive theoretical and empirical research. While most of these studies were conducted in the context of literary analysis, the interest in narrative structures has made inroads into social sciences (Shenhav, 2015). The classical work by Labov and Waletzky (1967) on oral narratives, as well as later works (Labov, 1972, 2013), signify this stream of research by providing a schema for an overall structure of narratives, according to which a narrative construction encompasses the following building blocks (Labov, 1972, 2013): abstract (what is the narrative about), orientation (information on the time, the place, the persons and the behavior involved), complicating action (or simply complication; the forward progression of narrative clauses), evaluation (establishing the narrative's "point"), resolution (what finally happened), and coda (bringing the time of reference back to the present time of narration). These building blocks provide useful and influential guidelines for oral narratives analysis.
2.2 Adaptation
Despite the substantial influence of Labov and Waletzky (1967) and Labov (2013), scholars in the field of communication have noticed that this overall structure does not necessarily comply with the form of informational text, such as news stories (Thornborrow and Fitzgerald, 2004; Van Dijk, 1988), and consequently proposed modified narrative structures (Thornborrow and Fitzgerald, 2004). Unlike well-tailored narrative texts, such as personal experience texts, narrativity in informational text is somewhat more challenging as it does not necessarily follow conventional or predefined genre-related structures. This requires a flexible coding scheme, unconstrained by a specific type of text. Instead, it should be open to a wide range of text types (such as informational text), and allow the presence of micro stories, encompassing
Complication	Resolution	Success
# Sentences	1,092	541	312
Proportion in Dataset	49%	24%	14%
Table 1: Overview of the NEAT dataset. Note that the categories are not mutually exclusive, due to the multi-labeled nature of the annotation scheme.
any combination of all narrative categories even at the sentence level. We set to accomplish that via two objectives: first, formalizing narrative categories which are oriented towards discourse structure, rather than semantic context. Second, defining our task as a multi-labeled one, to allow the flexibility required to capture sentence-level narrative characteristics. A special consideration was given to the variety of contents, forms and writing styles typical for media texts. For example, we required a coding scheme that would fit laconic or problem-driven short reports (too short for full-fledged "Labovian" narrative style), as well as complicated texts with multiple story-lines moving from one story to another. We addressed this challenge by focusing on two of Labov's six elements - complicating action and resolution, considered to be the most fundamental and relevant for informational text analysis (Labov, 2013). There are several reasons for our focus on these particular elements: first, it goes in line with the understanding that worth-telling stories usually consist of protagonists facing and resolving problematic experiences (Eggins and Slade, 2005). Moreover, these elements resonate with what is considered by Entman (2004) to be the most important Framing Functions - problem definition and remedy.
In order to adapt the original complicating action and resolution categories to informational content, we designed our annotation scheme as follows. Complicating action – hence, Complication – was defined in our narrative scheme as an event, series of events or situation, that point at problems or tensions. Resolution refers to the way the story is resolved or to the release of the tension. An improvement from – or a manner of coping with – an existing or a hypothetical situation was also considered to be a Resolution. This choice was made in order to follow the often tentative or speculative notion of future resolutions in news stories (Thornborrow and Fitzgerald, 2004; Bell, 1991). We have therefore included in this category any temporary or partial resolutions. The transitional characteristic of the Resolution motivated us
to add a new category defined as Success. Unlike Resolution, which refers, implicitly or explicitly, to a prior situation, this category was designed to capture any description or indication of an achievement or a desirable outcome.
3.1 Pilot Study
We started by conducting a pilot study, for the purpose of formalizing an annotation scheme and training our annotators. For this study, sample sentences were gathered from print news articles, published between 1995 and 2017 and collected via LexisNexus. These were used to refine the annotation scheme described in Section 2.2, as well as perform extensive training for our annotators.
Following the conclusion of the pilot study, we used the sentences which were collected and manually annotated during the pilot to train a multi-label classifier, later used to provide labeled candidates for the annotators during the annotation stage of the NEAT dataset, in order to optimize annotation rate and accuracy. The pilot samples were then discarded.
3.2 News Articles
The news articles for the dataset were sampled from leading news websites in the English language, all published between 2017 and 2020. The result is a corpus of 2,209 sentences taken from 46 news articles, with an average of 48 sentences per article  $(\sigma^2 = 39.44)$ , and an average of 20.2 tokens per sentence  $(\sigma^2 = 11.2)$ . The articles are semantically diverse, as they were sampled from a wide array of domain categories.
3.3 Preprocessing
The news articles' content was extracted using diffbot. The texts were scraped and split into sentences using the Punkt unsupervised sentence segmenter (Kiss and Strunk, 2006). Remaining segmentation errors were manually corrected.
3.4.1 Guidelines
Following the pilot study (Section 3.1), a code book containing annotation guidelines was produced. For each of the three categories in the annotation scheme - Complication, Resolution and Success - the guidelines provide:
A general explanation of the category
- A list of well-defined criteria for identifying the category
- Select examples of sentences labeled with the category
3.4.2 Process
We employed a three-annotator setup for annotating the collected sentences. First, the pilot stage model (Section 3.1) was used to produce annotation suggestions for each of the sentences in the corpus. Each sentence was then separately annotated by two trained annotators according to the guidelines described in Section 3.4.1. Each annotator had the choice to either accept the suggested annotation or to change it by adding or removing any of the suggested labels. Disagreements were later decided by a third expert annotator. Table 2 reports inter-coder reliability scores for each of the three categories, averaged across pairs of annotators: pairwise percent agreement (PPA), and Cohen's Kappa coefficient, accounting for chance agreement (Artstein and Poesio, 2008). Article-level domain categories (Table 3) were initially assigned according to the news section from which the articles were taken, and later verified by two annotators.
PPA (avg.)	κ (avg.)
Complication	89.6%	0.79
Resolution	87.5%	0.65
Success	92.8%	0.7
Table 2: Inter-coder reliability: average pairwise percent agreement (PPA) & average Cohen's Kappa  $(\kappa)$
Narrative categories vary significantly in their prevalence in the corpus; their respective proportions in the dataset are given in Table 1. The categories are unevenly distributed: Complication is significantly more frequent than Resolution and Success. This was to be expected, considering the known biases of "newsworthiness" towards
problems, crises and scandals (Esser et al., 2016), and due to the fact that in news media, resolutions often follow reported complications.
Interestingly, the distribution over narrative categories varies significantly between the different category domains (see Table 3). Most domains contain many more Complications than Resolutions or Successes, which is consistent with the distribution in the complete dataset (Table 1); the "Crime" domain is an extreme example, with a very small number of Resolutions and no Successes at all. However, some domains exhibit a completely different distribution. For example, the "Travel" and the "Science & Technology" domains possess a relatively uniform distribution over the three narrative categories. The "Sports" domain contains a similar number of Complications and Successes, with a smaller number of Resolutions.
Table 4 reports pairwise Pearson correlations between the categories. The Complication and Resolution categories are completely uncorrelated  $(r = 0.016)$ . A minor negative correlation was found between Complication and Success  $(r = -0.234)$ , and a minor positive one was found between Resolution and Success  $(r = 0.228)$ . These minor correlations – in our opinion – indicate that the Success category does indeed bring added value to our narrative scheme.
All the possible combinations of narrative categories appear in the dataset; Table 5 summarizes the occurrences of each of the possible category combinations. Examples of sentences annotated with various category combinations are given in Appendix A. There is, however, a significant variability to the frequency in which different combinations occur in the dataset. For example, the Complication-Resolution combination, designating a typical narrative tension-relief pattern (Shenhav, 2015), is by far the most frequent one with 226 sentences. Complication-Success, on the other hand, is a very rare combination with only 15 sentences, embodying a far less trivial or common logic, where a success is accompanied by an unresolved problem.
The fact that the dataset is assembled from full coherent news articles allows the analysis of a range of micro and macro stories in narrative texts. For example, an article in the dataset concerning
Domain Category	# Sentences	Complication	Resolution	Success
Economy & Job Market	221	99	71	25
Politics	860	514	220	89
Health	66	38	13	10
Travel	200	47	46	42
Arts & Culture	345	166	63	49
Crime	28	25	3	0
Accidents & Disasters	12	9	4	2
Welfare	47	30	11	1
Sports	115	51	21	47
Science & Technology	223	53	60	40
Immigration	92	60	29	7
Table 3: Domain category distribution in the NEAT dataset
Comp.	Res.	Suc.
Comp.	1
Res.	0.016	1
Suc.	-0.234	0.228	1
Table 4: Inter-category Pearson correlations
Combination	# Sentences
Complication & Resolution	226
Complication & Success	15
Resolution & Success	103
All Three Categories	49
Table 5: Narrative category co-occurrences in the NEAT dataset
the coronavirus outbreak in South Korea<sup>2</sup> opens with a one-sentence summary, tagged with both Complication and Resolution:
"South Korea's top public health official hopes that the country has already gone through the worst of the novel coronavirus outbreak that has infected thousands inside the country."
(Complication, Resolution)
This problem-solution (in this case, hopeful solution) plot structure reappears in the article, this time detailed over a series of sentences:
"More than 7,300 coronavirus infections have been confirmed throughout South Korea, killing more than 50." (Complication)
"It is one of the largest outbreaks outside mainland China, where the deadly virus was first identified." (Complication)
However, the number of new daily infections in South Korea has declined in recent days. (Complication, Resolution)
"... while he believes the aggregate number of infections is high, he is confident in the job South Korea did to combat the virus' spread and would advise other governments..." (Complication, Resolution)
"The South Korean government has been among the most ambitious when it comes to providing the public with free and easy testing options." (Success)
The sequence starts with two sentences tagged with Complication, followed by two sentences tagged with Complication and Resolution, and concludes with a sentence tagged with Success, demonstrating a more gradual transition from problem through solution to success.
4.1 Dataset Partition
We randomly divided the dataset into article-wise mutually-exclusive train, validation and test sets (details in Table 6), while keeping the distribution over the three narrative categories in each of the sets as similar as possible to the one in the complete dataset. The train set was used to train a supervised model for the task; the validation set was used to select the best model configuration during the training phase by tuning the model's hyper-parameters (see Section 4.3 for details), and the test set was used to evaluate the chosen model and produce the results reported in Section 5.
$^{2}$ https://edition.cnn.com/2020/03/09/asia/south-korea-coronavirus-int1-hnk/index.html
# Sentences (Articles)	Ratio
Train	1,767 (37)	80%
Validation	222 (5)	10%
Test	220 (4)	10%
Table 6: Train, validation & test set statistics
4.2 Task Definition
We explored two different approaches for solving the task: (1) addressing each of the three narrative categories as a separate classification task, and (2) treating the task as multi-label classification with three labels (one for each narrative category).
4.2.1 Separate Classification Tasks
In this approach, we defined a separate binary classification task for each of the narrative categories: Complication, Resolution and Success. For each such task, we trained a dedicated supervised model (further details given in Section 4.3), specifically optimized for the learned category. However, any potential information stemming from inter-correlations between the different categories was ignored, effectively treating them as three unrelated tagging schemes.
4.2.2 Multi-Label Classification
Here, the task was treated as a three-way multi-label classification problem (each sentence may contain any combination of the three narrative categories), thus taking advantage of inter-correlations between the three narrative categories to better learn them as part of a coherent narrative scheme. We trained and optimized a single multi-label model to jointly predict the three categories (further details given in Section 4.3).
4.3 System Architecture
We employed the method of fine-tuning a pretrained language model for our task. In each experimental setup, we chose a pre-trained language model as a backbone, applied a multilayer perceptron (MLP) classifier on top of it, and fine-tuned the entire model over the train set.
4.3.1 Backbone
We experimented with three state-of-the-art transformer-based language models as the backbone for our inference model, using pre-trained weights from the transformers python package (Wolf et al., 2019).
BERT. Following common practice, we first utilized the base-sized BERT (Devlin et al., 2018) as the backbone model.
RoBERTa. This BERT variant was developed by training the original BERT model with altered design choices and training techniques, and has been recently shown to produce better results on various NLP tasks (Liu et al., 2019). We used the base-sized RoBERTa as the backbone model.
DistilBERT. A recent body of work has focused on developing "lighter" transformer-based language models which allow for faster fine-tuning for downstream NLP tasks (Sanh et al., 2020; Lan et al., 2020). In order to assess robustness to a decrease in the model's size, we also experimented with DistilBERT (Sanh et al., 2020), which follows the same basic architecture as BERT but consists of 66M parameters (as opposed to 110M in BERT), as the backbone model.
4.3.2 Classifier
In order to fine-tune the backbone language model, we appended a multilayer perceptron (MLP) over the output of the language model. The MLP consisted of one hidden layer (increasing the number of hidden layers produced no improvement in performance), the size of which was optimized as a hyper-parameter. In the case of a single binary classification task (Section 4.2.1), the output layer consisted of a single sigmoid output, while in the case of a multi-labeled task, it consisted of three sigmoid outputs, one for each narrative category.
4.3.3 Training Procedure
All models were optimized using the AdamW algorithm (Loshchilov and Hutter, 2017) and the binary cross entropy loss function. Positive weighting was used in order to compensate for class imbalance (evident in Table 1). Hyper-parameters – batch size, learning rate and MLP hidden layer size – were chosen via a standard grid search (see Appendix B for more details). For each configuration of task definition, backbone model and hyper-parameters, the model was evaluated over the validation set after every epoch of training, and the best-performing checkpoint was tested on the test set to produce the results reported in Section 5.
4.4 Cross-Domain Classification
Given the semantic diversity in the dataset, as well as the variability in distribution over the narrative categories between the various domains (Table 3),