SlowGuess/NAACL_2022_OCR huggingface dataset:
Task	Backbone	Complication	Resolution	Success	Average
P	R	\(F_1\)	P	R	\(F_1\)	P	R	\(F_1\)	P	R	\(F_1\)
Separate	BERT	0.89	0.87	0.88	0.67	0.40	0.50	0.33	0.72	0.45	0.63	0.66	0.61
Separate	RoBERTa	0.92	0.85	0.88	0.69	0.56	0.62	0.83	0.56	0.67	0.81	0.66	0.72
Separate	DistilBERT	0.87	0.84	0.85	0.69	0.40	0.51	0.53	0.44	0.48	0.70	0.56	0.61
M.Label	BERT	0.88	0.89	0.88	0.72	0.60	0.65	0.53	0.50	0.51	0.71	0.66	0.68
M.Label	RoBERTa	0.90	0.90	0.90	0.76	0.67	0.71	0.64	0.78	0.70	0.77	0.78	0.77
M.Label	DistilBERT	0.86	0.87	0.86	0.73	0.53	0.61	0.75	0.50	0.60	0.77	0.63	0.69
Table 7: Test set precision (P), recall (R) and  $F_{1}$  scores, for every combination of task definition (a separate task for each narrative category / a multi-labeled task) and backbone model. See Sections 4.2 and 4.3.1 for details.
we wished to assess the domain category's effect on learning our narrative scheme. For this purpose, we experimented with a cross-domain classification setup. For each of the eleven category domains, we concatenated the sentences from all other domains into a train set, which was then used to train a classification model. The training process was done using the configuration of the best-performing model from the previous stage (described in Sections 4.1-4.3), including task definition, backbone model and hyper-parameters (i.e. no hyper-parameter tuning was performed in this setup). The trained model was then evaluated on the test set.
Results are reported in Table 7. For each task definition and choice of backbone model, we report the precision, recall and  $F_{1}$  score for each of the three narrative categories, as well as their average, over the test set.
First, we observe that addressing the task as a multi-labeled one proved to be a better strategy than learning each narrative category separately. This is evident across backbone models as well as across narrative categories; for each backbone model, the multi-label model produced a higher  $F_{1}$  score for each and every one of the narrative categories. This is a clear indication that these categories are substantially connected as they constitute intertwining elements in an underlying story. Therefore, the three categories form a coherent narrative scheme that is more than just the sum of its parts.
Interestingly, while this effect is relatively small for Complication ( $F_{1}$  increased by 0.00–0.02), it is much more prominent for Resolution ( $F_{1}$  increased by 0.09–0.15) and Success ( $F_{1}$  increased by 0.03–0.12), meaning that incorporating all three categories into one coherent scheme contributes mostly to learning the Resolution and Success categories. This
suggests that perhaps the narrative properties of the Complication category make it more independent and self-contained than the other two categories. Resolution and Success, on the other hand, are more relative in nature, and seem to be anchored, implicitly or explicitly, by a prior situation or condition.
Among the three narrative categories, Complication gained the highest  $F_{1}$  scores by all the models, ranging between 0.85 and 0.90. The models were less successful predicting Resolution, with  $F_{1}$  scores ranging between 0.50 and 0.71, and Success, with  $F_{1}$  scores ranging between 0.45 and 0.70. This is consistent with the proportion of instances belonging to each category in the dataset (see Table 1), which may provide a possible explanation for this observation. However, the fact that positive weighting was used in the training process (Section 4.3.3) to counter class imbalance, motivates us to search for another explanation. Defined by Labov and Waletzky (1967)'s overall structure of narratives as "the main body of narrative clause", Complication may just be an easier narrative category to learn.
Comparing different backbone models, the DistilBERT-based model performed similarly to the BERT-based one - an average  $F_{1}$  score of 0.61 for both in the separate-task setting, and 0.69 compared to 0.68 in the multi-label setting - suggesting that the task is fairly robust to a decrease in the backbone model's size. However, RoBERTa significantly outperformed the other two language models as the backbone in both settings - an average  $F_{1}$  score of 0.71 compared to 0.61 in the separate-task setting, and an average  $F_{1}$  score of 0.77 compared to 0.68 and 0.69 in the multi-label setting. We also note that the difference in performance between Complication and the other two categories is less extreme in the RoBERTa-based models compared to the other backbone models.
Domain Category	Complication	Resolution	Success	Average
P	R	F1	P	R	F1	P	R	F1	P	R	F1
Accidents & Disasters	1.00	0.89	0.94	1.00	0.50	0.67	1.00	0.50	0.67	1.00	0.63	0.76
Arts & Culture	0.83	0.84	0.84	0.67	0.75	0.71	0.51	0.78	0.62	0.67	0.79	0.72
Crime	1.00	0.92	0.96	0.67	0.67	0.67	-	-	-	0.83	0.79	0.81
Economy & Job Market	0.86	0.78	0.81	0.77	0.68	0.72	0.50	0.84	0.63	0.71	0.76	0.72
Health	0.97	0.82	0.89	0.63	0.77	0.69	0.67	0.80	0.73	0.75	0.80	0.77
Immigration	0.86	0.93	0.90	0.80	0.83	0.81	0.71	0.71	0.71	0.79	0.83	0.81
Politics	0.83	0.95	0.89	0.58	0.86	0.69	0.60	0.65	0.63	0.67	0.82	0.74
Science & Technology	0.75	0.83	0.79	0.66	0.77	0.71	0.64	0.88	0.74	0.68	0.82	0.74
Sports	0.87	0.65	0.74	0.53	0.76	0.63	0.84	0.81	0.83	0.75	0.74	0.73
Travel	0.89	0.87	0.88	0.66	0.89	0.76	0.69	0.83	0.75	0.75	0.87	0.80
Welfare	1.00	0.97	0.98	0.80	0.73	0.76	0.33	1.00	0.50	0.71	0.90	0.75
Average	0.90	0.86	0.87	0.71	0.74	0.71	0.65	0.78	0.68	0.76	0.79	0.76
Table 8: Precision (P), recall (R) and  ${F}_{1}$  scores for cross-domain classification. See Section 4.4 for details.
5.1 Cross-Domain Classification
The best-performing configuration (a multi-label classifier based on the RoBERTa language model) was used to perform the cross-domain classification experiment. As stated in Section 4.4, hyperparameters were fixed to the values obtained in the train-validation-test setup. Results are presented in Table 8. Averaged over domain categories, they are virtually identical to the results obtained on the test set (reported in Table 7), with a precision, recall and  $F_{1}$  score of 0.76, 0.79 and 0.76 compared to 0.77, 0.78 and 0.77 (respectively). In our opinion, this demonstrated invariance to domain category is a strong indication that our narrative elements are more grounded in discourse characteristics rather than in the semantic field.
We introduced NEAT (Narrative Elements Annotation) - a novel NLP task for detecting narrative elements in raw text. For this purpose, we designed a new flexible multi-label narrative annotation scheme, specifically suited for informational text, by adapting two elements from the theory introduced in Labov and Waletzky (1967); Labov (1972, 2013) - Complication and Resolution - and adding a new element - Success. The scheme was used to annotate a new dataset of 2,209 sentences, compiled from 46 articles, which were collected from news websites.
We explored two alternate settings for solving this task - one in which each narrative category was treated as a separate classification task, and another in which the entire task was addressed as multi-label classification. In each of these setups,
we experimented with fine-tuning three different language models, achieving an average  $F_{1}$  score of up to 0.77 on the test set, and showcasing the potential of supervised-learning methods in detecting the narrative information encoded into our scheme. The multi-label setting consistently provided significantly better results across all models and narrative categories, demonstrating that our narrative categories are strongly connected and form a coherent narrative scheme which is more than just the sum of its parts. Additional cross-domain classification results demonstrate the task's invariance to domain category, suggesting that our annotation scheme is more grounded in discourse characteristics rather than semantic context.
We are currently engaged in an ongoing effort for improving the annotation quality of the dataset and increasing its size. In addition, we have several interesting directions for future work. The first one, which we are currently pursuing, includes enriching the scheme with token-level annotation of the narrative elements, effectively converting the task from multi-label classification to a sequence prediction one. Alternatively, we could introduce additional layers of information to encode more global narrative structures in the text, such as inter-sentence – or even interarticle – references between narratively-related elements (e.g., a Resolution referencing its inducing Complication). Another potential direction is incorporating additional narrative elements into our annotation scheme. For example, the evaluation element from (Labov, 2013) may be beneficial in encoding additional information in the context of news media, such as the sever
ity of a Complication or the 'finality' of a Resolution. We could also add completely new narrative elements, tailored to capture specific informational aspects, such as actor-based elements identifying entities which are related to one or more of the currently defined narrative categories.
This research was supported by the Israel Science Foundation (Grants No. 1400/14; 2315/18). We thank the three annotators - Gal Ron, Vered Porzycki and Avishai Green - for their dedicated work, as well as their insightful comments and suggestions during the annotation processes.
Ron Artstein and Massimo Poesio. 2008. Survey article: Inter-coder agreement for computational linguistics. Computational Linguistics, 34(4):555-596.
Daniela Baiamonte, Tommaso Caselli, and Irina Prodanof. 2016. Annotating content zones in news articles. CLiC it, page 40.
Allan Bell. 1991. The language of news media. Blackwell Oxford.
Nathanael Chambers and Dan Jurafsky. 2008. Unsupervised learning of narrative event chains. In Proceedings of ACL-08: HLT, pages 789-797, Columbus, Ohio. Association for Computational Linguistics.
Seymour B Chatman. 1978. Existents. Story and Discourse: Narrative Structure in Fiction and Film, pages 96-145.
Rodolfo Delmonte and Giulia Marchesini. 2017. A semantically-based computational approach to narrative structure. In IWCS 2017 — 12th International Conference on Computational Semantics — Short papers.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805.
Suzanne Egginss and Diana Slade. 2005. Analysing casual conversation. Equinox Publishing Ltd.
Robert M Entman. 2004. Projections of power: Framing news, public opinion, and US foreign policy. University of Chicago Press.
Frank Esser, Sven Engesser, and Jörg Matthes. 2016. Negativity. In Comparing political journalism, pages 89-109. Routledge.
Monika Fludernik. 2009. An introduction to narratology. Routledge.
Gustav Freytag. 1894. Die technik des dramas. S. Hirzel.
Gérard Genette. 1980. Narrative discourse, tr. J. Lewin. Ithaca: Cornell  $[= 1972$ , Figures III.
Pablo Gervás, Birte Lönneker-Rodman, Jan Christoph Meister, and Federico Peinado. 2006. Narrative models: Narratology meets artificial intelligence. In International Conference on Language Resources and Evaluation. Satellite Workshop: Toward Computational Models of Literary Analysis, pages 44-51.
Tibor Kiss and Jan Strunk. 2006. Unsupervised multilingual sentence boundary detection. Computational Linguistics, 32(4):485-525.
William Labov. 1972. Language in the inner city: Studies in the Black English vernacular, volume 3. University of Pennsylvania Press.
William Labov. 2013. The language of life and death: The transformation of experience in oral narrative. Cambridge University Press.
William Labov and Joshua Waletzky. 1967. Narrative analysis. oral versions of personal experience. Essays on the Verbal and Visual Arts: Proceedings of the American Ethnological Society, edited by June Helm, 12-44. Seattle: American Ethnological Society.
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2020. Albert: A lite bert for self-supervised learning of language representations.
Boyang Li, Beth Cardier, Tong Wang, and Florian Metze. 2017. Annotating high-level structures of short stories and personal anecdotes. arXiv preprint arXiv:1710.06917.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach.
Ilya Loshchilov and Frank Hutter. 2017. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101.
Jessica Ouyang and Kathy McKeown. 2014. Towards automatic detection of narrative structure. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC-2014), pages 4624-4631, Reykjavik, Iceland. European Languages Resources Association (ELRA).
Pinelopi Papalampidi, Frank Keller, and Mirella Lapata. 2019. Movie plot analysis via turning point identification. arXiv preprint arXiv:1908.10328.
Vladimir Yakovlevich Propp. 1968. Morphology of the Folktale, volume 9. University of Texas Press.
Shlomith Rimmon-Kenan. 2003. Narrative fiction: Contemporary poetics. Routledge.
Belen Saldias and Deb Roy. 2020. Exploring aspects of similarity between spoken personal narratives by disentangling them into narrative clause types.
Victor Sanh, Lysandre Debut, Julien Chaumont, and Thomas Wolf. 2020. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter.
Shaul R Shenhav. 2015. Analyzing social narratives. Routledge.
Reid Swanson, Elahe Rahimtoroghi, Thomas Corcoran, and Marilyn Walker. 2014. Identifying narrative clause types in personal stories. In Proceedings of the 15th Annual Meeting of the Special Interest Group on Discourse and Dialogue (SIGDIAL), pages 171-180, Philadelphia, PA, U.S.A. Association for Computational Linguistics.
Joanna Thornborrow and Richard Fitzgerald. 2004. Storying the news through category, action, and reason. The Communication Review, 7(4):345-352.
Teun A Van Dijk. 1988. News as discourse. hillsdale, nj: L.
Claudia Volpetti, K Vani, and Alessandro Antonucci. 2020. Temporal word embeddings for narrative understanding. In Proceedings of the 2020 12th International Conference on Machine Learning and Computing, pages 68-72.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumont, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, and Jamie Brew. 2019. Huggingface's transformers: State-of-the-art natural language processing.
A Sample Annotated Sentences
1. How did some of the biggest brands in care delivery lose this much money? (Complication)
2. Bleeding from the eyes and ears is also possible after use, IDPH said. (Complication)
3. His proposal to separate himself from his business would have him continue to own his company, with his sons in charge. (Resolution)
4. Instead, hospitals are pursuing strategies of market concentration. (Resolution)
5. With its centuries-old canals, vibrant historic center and flourishing art scene, Amsterdam takes pride in its cultural riches. (Success)
6. Mr. Trump chose to run for president, he won and is about to assume office as the most powerful man in the world. (Success)
7. Soon after, her administration announced a set of measures intended to curb misconduct. (Complication, Resolution)
8. Avant-gardists, the couple opened an art gallery in 1875 within the department store, offering artists who had been turned away by the Paris Salon – the official art exhibition of the Academy of Fine Arts in Paris – a home for their works and a large public audience, Burckhardt wrote. (Complication, Resolution)
9. Unlike Macy's, the well-known US department store which has been closing its doors around the country, Le Bon Marché isn't fighting to stay alive. (Complication, Success)
10. He defeated Bolt in a close semifinal heat, ending Bolt's 28-race winning streak in the 100 meters. (Complication, Success)
11. The Utah man's mother, Laurie Holt, thanked Mr. Trump and the lawmakers for her son's safe return, adding: "I also want to say thank you to President Maduro for releasing Josh and letting him to come home." (Resolution, Success)
12. And if Tony Hughes can keep his job without the weeks away, "in that aspect," he said, "it's going to make my life better." (Resolution, Success)
13. They were fortunate to escape to America and to make good lives here, but we lost family in Kristallnacht. (Complication, Resolution, Success)
14. He was vulnerable, no doubt, but in the past, with the spotlight at its brightest, he still found the speed and the will to remain the fastest man in the world. (Complication, Resolution, Success)
B Hyper-Parameters
Table 9 lists the ranges of values which were used for hyper-parameter tuning during the experiments described in Section 4.
Hyper-Parameter	Values
Batch Size	16, 32, 64
Learning Rate	1e-5, 2e-5, 3e-5, 4e-5, 5e-5
MLP Hidden Size	50, 100, 200, 300
Table 9: Hyper-parameters value ranges
The best-performing model - a multi-label classifier based on the RoBERTa language model - was trained for 4 epochs with a batch size of 32, a learning rate of 4e-5 and an MLP classifier with a 100-node hidden layer.