Automatic Annotation of Direct Speech in Written French Narratives
Year: 2023

Abstract:
The automatic annotation of direct speech (AADS) in written text has been often used in computational narrative understanding. Methods based on either rules or deep neural networks have been explored, in particular for English or German languages. Yet, for French, our target language, not many works exist. Our goal is to create a unified framework to design and evaluate AADS models in French. For this, we consolidated the largest-to-date French narrative dataset annotated with DS per word; we adapted various baselines for sequence labelling or from AADS in other languages; and we designed and conducted an extensive evaluation focused on generalisation. Results show that the task still requires substantial efforts and emphasise characteristics of each baseline. Although this framework could be improved, it is a step further to encourage more research on the topic.

heading: 1 Introduction
text: Prose fiction makes whole worlds emerge. Authors make use of different strategies to create narratives and convey the storyworld. Novels intertwine narrators’ words to build the atmosphere and tell the story, with words stemming from characters inhabiting the fictive world that disclose their personality and depict them directly via dialogues or direct speech (DS) (James, 2011; Hühn et al., 2014).
The development of algorithms to perform the automatic annotation of direct speech (AADS) in written text has been of high interest for literary studies. This task consists in retrieving lines uttered by the characters of a narrative in contrast to words delivered by the narrator of the story. One goal of AADS has been to compare fiction works by different authors or stemming from different genres or time periods. DS was then studied as a literary device carrying specific purposes and disclosing compelling cultural information (Muzny et al., 2017; Egbert and Mahlberg, 2020). AADS is also central
in narrative understanding endeavors. DS has been then considered as the main realisation of characters, their means to gain volume and depth, and come alive to the readers. In this context, AADS is often regarded as a pre-processing step that enables downstream analysis such as DS speaker attribution (Cuesta-Lazaro et al., 2022), that can in turn serve to assemble characters networks (Labatut and Bost, 2019), or model personas (Sang et al., 2022).
AADS has been widely performed for English literature, leveraging strict formatting conventions (e.g. quotes or long dashes) to extract DS through simple regular expression—regex (Bamman et al., 2014; O’Keefe et al., 2012; Elson and McKeown, 2010). Yet, in other languages, dialogues may be less strictly segregated from narration and typographic conventions can be more flexible. Hence, more complex solutions based on lexical features have been developed, mainly for German (Brunner, 2013; Jannidis et al., 2018; Brunner et al., 2020). These lexical features were either manually defined and exploited with classical machine learning algorithms such as Random Forest (Brunner, 2013), or were inferred indirectly from text in deep learning frameworks (Jannidis et al., 2018; Brunner et al., 2020) using Recurrent Neural Networks or lan-
7129
guage models such as BERT (Devlin et al., 2019). For other languages, including French, there are very few AADS efforts. Schöch et al. (2016) propose Straight Talk!, a corpus of 40 chapters from 19th century French novels annotated per sentence if containing DS or not, and performed binary classification using 81 engineered features. The corpus was quite large, but sentences were poorly segmented with a high impact on results; annotations did not consider incises (i.e. narrative breaks within the same DS turn as in Figure 1); despite a high overall F1-score (93%), some writing styles were very challenging (for instance in homodiegetic narratives, where the narrator is a fully fledged character in the storyworld and may relate the story at the first person). In another work, Sini et al. (2018a) adopted a feature engineering approach as well. They combined it with rules to segment and identify paragraphs containing DS, and then to extract incises from mixed paragraphs. Still, the method was tested on a small corpus, a subset of SynPaFlex (Sini et al., 2018b) with excerpts from only two novels. Finally, Byszuk et al. (2020) considered AADS in multilingual settings using BERT, but on an even smaller French corpus.
The goal of the current work is to create an unified framework for designing and evaluating AADS models in French, which in return we hope to encourage more research on the topic1. Specifically, we address existing limitations on multiple fronts:
1. We catalogued and consolidated the largestto-date dataset of French narratives manually annotated with DS tags at the word level based on 4 existing corpora. First, we re-annotated Straight Talk! (Schöch et al., 2016) to reach a finer granularity: from sentence to word level. Second, we extended the SynPaFlex (Sini et al., 2018a) sparse annotations, initially done on chapter excerpts, to cover the whole chapters. We also incorporated two new corpora as they were: fr-LitBank, the French variant of the Multilingual BookNLP project (Lattice, 2022) and an extension of SynPaFlex (Sini et al., 2018a) provided by the authors. Our dataset is made of 86 whole chapters (680K annotated tokens), extracted from French novels published during the 19th and 20th centuries.
1Code and data are publicly available at https:// github.com/deezer/aads_french
2. We modelled AADS as a token classification task, which we argue as more suitable for incises identification. This approach allowed us to benchmark state-of-the-art sequence labelling models such as French finetuned transformers (Martin et al., 2020) for the first time for AADS. We also re-implemented the most popular AADS baselines from other languages to fit French language peculiarities and trained them on our dataset. In our selection, we included baselines that did not require extensive manual feature engineering to encourage generalisation over various writing styles.
3. We devised an extensive evaluation covering text with varied formatting quality. Apart from traditional token- and span-level strict precision, recall and F1-score metrics (Yadav and Bethard, 2018), we adapted ZoneMap (Galibert et al., 2014), a metric stemming from page segmentation literature, to our task. This allowed us to quantify the effect of various error types made by the models and deepen our understanding of their limitations.
Results show that rule-based baselines using regular expressions remain a good choice when texts are well-formatted. Deep learning solutions are however more effective and achieve satisfactory results even on narratives with poor formatting quality. Their most common issue is that they still miss to catch whole DS sequences. We also conducted a qualitative analysis to bring insights on the strengths and weaknesses of various models, and defined the directions for future endeavors.

heading: 2 Literature Review
text: We further review AADS solutions for any language.

heading: 2.1 Rule-based AADS
text: While conventions may vary across languages and novels, DS tends to be enclosed within quotation marks (e.g. «...»; “...”), or introduced with long dashes (e.g. —...; –...). Regarded as pre-processing, simple AADS methods relying on regex with low computational costs are favored (Thomas, 2012; Cunha and Arabyan, 2004). The AADS module of BookNLP (Bamman et al., 2014), the reference pipeline developed for computational narrative understanding in English, first determines
the most used quotation mark type from a predefined set; then it tags every passage in between the selected quotation mark pair as DS. This yields performances around an F1-score of 90% when evaluated as a token-level binary classification task on the LitBank 19th century book corpus (Sims and Bamman, 2020). Variations of this approach considering more quotation mark types than the most used one are also common (Cuesta-Lazaro et al., 2022; Yoder et al., 2021; Byszuk et al., 2020; O’Keefe et al., 2012). Almost perfect F1-scores (96 − 99%) are then reported on various English corpora.
However, when working with heterogeneous corpora, texts with poorer encoding quality (because of Optical Character Recognition errors or changing editing standards over time), or other languages, typographic AADS appears to be limited (Byszuk et al., 2020; Muzny et al., 2017). For instance, performances on French and German decrease to a F1-score of 92% and down to 65% for Norwegian (Byszuk et al., 2020). Similarly, we observe the F1-score decreasing to 77% on a more challenging English corpus (Muzny et al., 2017).
To overcome these issues, more complex rulebased systems that leverage semantic and syntactic cues besides typographic markers have been proposed for English (Muzny et al., 2017) and German (Tu et al., 2019). Empirical studies revealing writing style differences between DS and narration (Egbert and Mahlberg, 2020) have supported this direction. The lack of DS markers and the prevalence of incises in French literature has also led Sini et al. (2018a) to devise more sophisticated regex based on dependency parsing and Part-of-Speech (POS) tags, yielding an F1-score of 89.1%.

heading: 2.2 Machine Learning-based AADS
text: With an increasing availability of annotated corpora, AADS based on machine learning has been explored more and more, in particular on German literature (Brunner, 2013; Tu et al., 2019; Brunner et al., 2020). Works on other languages, such as French (Schöch et al., 2016) or Swedish (Ek and Wirén, 2019), have also emerged, while remaining sparse and isolated. ELTeC multilingual initiative (Odebrecht et al., 2021) has encouraged the investigation of multilingual approaches too (Byszuk et al., 2020; Kurfalı and Wirén, 2020).
All these endeavors exploit syntactic and semantic features of DS segments beyond typographic
cues, either through feature engineering or by learning features from text with end-to-end deep learning. Brunner (2013) trained a Random Forest on 80 syntactic and semantic features extracted at the sentence level from a corpus of 13 short German narratives. Her method showed a 3 point improvement compared to the rule-based AADS baseline, though with a large standard deviation (19%). This approach was later adapted to French by Schöch et al. (2016) on a corpus of 40 book chapters.
In the recent years, the successful application of deep learning to a wide-range of NLP tasks has led to the adoption of these models for AADS too. Brunner et al. (2020) proposed to use a BiLSTMCRF (Huang et al., 2015) on text encoded with Flair (Akbik et al., 2018), FastText (Mikolov et al., 2018) and a multilingual BERT (Devlin et al., 2019), as well as to fine-tune the German-language BERT (Chan et al., 2020) for AADS on German narratives. Byszuk et al. (2020) fine-tuned a multilingual BERT and reported an overall F1-score of 87.3% at the token level. However, the score per language is missing, making it challenging to assess the benefits of the approach for individual cases. Kurfalı and Wirén (2020) adopt a zero-shot framework and remove DS typographic markers from the test corpora. They trained a multilingual BERT on silver-labelled data obtained with regex AADS and report token-level F1-score of 85% on English, 73% on Swedish and 64% on German.
In summary, research dedicated to French remains very sparse and suffers from a lack of comparability because of differences among the studied corpora, task modeling focuses (token vs. sentence classification), or imposed research scenario (without typographic markers, multilingual, zero-shot).

heading: 3 French Narrative Corpora for AADS
text: We consolidate a large dataset of French novel excerpts, manually annotated with DS labels at the word level. Built upon existing endeavors, the final dataset is a compilation of four sub-corpora, individually referred to as Straight Talk! (ST!) (Schöch et al., 2016), SynPaFlex (SPF) (Sini et al., 2018a), an extension of SynPaFlex provided to us by the authors (SB), and fr-LitBank (fr-LB) (Lattice, 2022). While fr-LB, SPF, and SB have overall good encoding and segmentation quality, ST! is poorly formatted with some files lacking line breaks, for instance.
Each sub-corpus contains French novels from
public-domain published between 1830 and 19372. It results in an aggregated corpus gathering 86 chapters extracted from 44 novels. The full dataset comprises more than 680K words, 8826 DS spans which represent 37% of the total tokens. However, we can observe large variations of DS presence across files (see Appendix A), from no DS in the excerpt named madame_bovary_première_9 to 92% of the words being labelled as DS in mystères_de_paris_2_troisième_16. Appendix A shows the excerpts and more dataset details.
The sub-corpora, fr-LB, and SB, were kept in the form provided by the original works. In contrast, we modified the ground-truth annotations of ST! and SPF in order to align them with the other two sub-corpora and the adopted approach to model the problem—binary classification at the token level– and to exhaustively cover chapters, not only excerpts. In particular, ST! annotations lacked granularity as text segments were labelled as Narration or Mixed (comprising both Narration and DS), so we corrected those. As for SPF, the annotations were very sparse among the 27 chapters; hence we extended them to whole chapters.
The re-annotation process was mainly led by one author using Doccano (Nakayama et al., 2018). A selection of 5 files were doubly annotated by a coauthor to check labeling reliability. The obtained pairwise Cohen’s κ score (Cohen, 1960) was 97%, which is considered almost perfect agreement. The re-annotated dataset is shared with the code.
The dataset is then split into train, validation and test sets. The files from the three well-formatted sub-corpora (fr-LB, SPF, SB) are randomly divided in order to ensure a proportion of 0.8/0.1/0.1 for train, validation, and test, respectively, and that at least one file from each sub-corpus can be found in each split. Each file can be found in only one split, but we sometimes have files from the same novel present in all splits, especially those origi-
2This period is chosen because it is copyright-free.
nating from the SPF sub-corpus (Les Mystères de Paris by Eugène Sue and Madame Bovary by Gustave Flaubert). Finally, ST! is kept for test only as a challenge dataset. Indeed, contrary to the other subcorpora mentioned above, this latter sub-corpus suffers from largely unequal formatting quality across files. Some chapters are completely devoid of line break which makes them, wrongly, appear as one unique paragraph, while others exhibit misplaced line breaks, sometimes in the middle of sentences. ST!’s formatting peculiarities make it a good test for generalisation, especially on noisier text. This challenging set is also referred to as a noisy test set (TestN ) in contrast to the clean test set (TestC) stemming from the split of the three well-formatted sub-corpora —that are also used for training and validation.
Dataset statistics are shown in Table 1. More details on split composition in terms of files can be found in Appendix A.

heading: 4 Methods
text: Popular baselines from the two AADS approaches (rule-based and machine learning-based), including those designed for other languages, were modified to fit the characteristics of French. AADS was then formalized either as a text matching and extraction task, when using regex, or as a sequence labelling task, when using deep learning models. For the latter, the AADS models returned a binary label per token, (O / DS) as in other related works (Brunner et al., 2020; Ek and Wirén, 2019; Jannidis et al., 2018). While regex has been more common, to our knowledge, this is the most extensive attempt to explore deep learning for AADS in French narratives.

heading: 4.1 Rule-based AADS Baselines
text: We adapted two rule-based systems (Byszuk et al., 2020; Bamman et al., 2014) for our framework.
Byszuk et al. (2020) compiled a list of various quotation marks and dashes used to introduce characters’ DS, which we kept the same. However, we modified the definition of paragraphs, the input to the regex system, to be spans of text until a break line. Regular expressions were after applied, as they were, to extract the text enclosed by quotation marks or introduced by a dialogue dash.
In contrast, Bamman et al. (2014)’s method was driven by the hypothesis that each text used a single typographic convention for DS. Thus, they identified the most used quotation mark in the analyzed
document from a predefined list. Then, regex was applied considering only the selected symbols. To make it applicable to French narratives, we added other types of dialogue cues to the original DS markers list, which we release with the code.
Although Sini et al. (2018a) propose a rule-based algorithm focused on the French language, they relied heavily on crafted syntactic and semantic rules. Our aim was to avoid extensive manual feature engineering in order to encourage generalisation over various writing styles. Also, this method was strongly dependent on other external tools for syntactic analysis that introduced further errors too. Hence, we did not include it in the benchmark.

heading: 4.2 Deep Learning-based AADS Baselines
text: Deep learning-based AADS was modelled as a token classification task, which we considered more suitable for identifying incises. We further discuss how we preprocessed the text in order to maintain a certain degree of contextual coherence for our objective. Then, we present the two models we included in our benchmark: 1) we adapted the state-of-the-art AADS deep learning model for German (Brunner et al., 2020) to fit French language peculiarities and re-trained it from scratch , and 2) we fine-tuned CamemBERT (Martin et al., 2020) to perform sequence labelling on our dataset.
Input Preprocessing. We used spaCy (Honnibal and Johnson, 2015) to segment text in sentences and each sentence into words and punctuation.
The input length supported by contemporary language or text embedding models is limited. For instance, BERT (Devlin et al., 2019) accepts a maximum of 512 sub-word tokens, while Flair embeddings (Akbik et al., 2019) initially could handle 512 characters. This makes them unfitted to represent or produce inferences over whole books, chapters, or even larger paragraphs, which is an important limitation in computational narrative understanding. However, to preserve a certain degree of coherence within each individual text segment with regard to the DS task, we implemented an informed split as follows. Given text in reading direction, a new sentence was added to the existing segment only if the maximum input size LC was not reached. Otherwise, the current segment was stored and a new one initialized starting with this last sentence. We discuss the choice of LC in Section 5.
Fine-tuned CamemBERT. To specialize the general linguistic knowledge of the pre-trained
language models for a precise purpose—here, to recognize DS, we use fine-tuning. We work with CamemBERT (Martin et al., 2020), one of the reference BERT-like model for French, available in the HuggingFace library (Wolf et al., 2020). However, as another tokenization of our preprocessed input is performed by CamemBERT, some adjustments were necessary to address out-of-vocabulary limitations and to handle larger sub-word sequences.
First, CamemBERT tokenizer was not be able to project all of the encoded symbols into the model’s vocabulary. This was the case for breaklines as we worked with paragraphs as input, or special space encodings such as "\xa0". We spotted these unknown symbols during a first model tokenization round over the whole set of tokens, initially obtained with spacy, and replaced them with a special token [UK]. Another strategy could have been to remove them but we found these tokens potentially informative for AADS, as text structure cues.
Second, after the CamemBERT tokenization, a sequence of LC tokens created during preprocessing might result in more sub-word tokens allowed as input. Similar to BERT, CamemBERT has the input limited to 512 sub-words. Here, in order to avoid the model automatically truncating the long sequence, the sequence is split in half if it overflows the input limit. Thus, it is less likely to have very short sub-sequences and context is evenly shared amongst resulting chunks. This chunking choice is closely linked to the tested LC values (up to 512, see section Section 5). However, splits are unlikely most of the time, as SpaCy tokens are common French words—most likely represented by one or two sub-words in the model’s vocabulary.
BiLTSM-CRF. We adopt the same architecture as in the state-of-the-art AADS model for German proposed by Brunner et al. (2020). Typical for sequence labelling tasks (Huang et al., 2015), it consists of two bi-directional Long-Short Term Memory (BiLSTM) layers and one Conditional Random Field (CRF) layer. The model is implemented using the SequenceTagger class of the Flair framework (Akbik et al., 2019). To embed the input, we test multiple options: Flair (Akbik et al., 2019), FastText (Athiwaratkun et al., 2018), or Flair and FastText stacked. Regarding input representation, Flair comes with a native way to handle long sequences, if these are encountered. They are chunked and each chunk is pushed to the model while
keeping the last hidden state as a new hidden state3.

heading: 5 Experiments

heading: 5.1 Evaluation Metrics
text: We assess the performance of models both at the token- and sequence-levels. Results are reported overall and per file. Token-level metrics measure the quality of the binary classification per word / token. Precision, recall and F1 scores are then computed with the scikit-learn library (Pedregosa et al., 2011). Strict sequence match (SSM) scores, such as precision, recall and F1 scores, measure the extent to which the predicted DS sequences strictly match the ground-truth ones. These are computed with the seqeval library (Nakayama, 2018)
We also employ another sequence-level score: Zone Map Error (ZME). This is our custom adaptation of the error computation method originally developed for page segmentation (Galibert et al., 2014). We include ZME because: 1) we wanted to have complementary scores that alleviate SSM’s strictness; 2) we aimed to leverage it to get more insights into the quality of the output by studying the impact of various types of errors a model makes.
ZME relies on a classification of error types that depends on the overlap between ground-truth and predicted spans. The overlap can be perfect, overlapping, including or included. The error types we could obtain are then: Match Error (1-to-1 non-perfect overlapping between ground-truth and predicted spans), Miss (non-detected ground-truth DS span), False Alarm (falsely detected DS span), Merge (several ground-truth DS spans are covered by only one predicted span), or Split (several predicted spans within a unique ground-truth one). The score is also dependent on the span length and the number of correctly classified tokens within a span (Galibert et al., 2014). Note that this is an error score, thus it should be minimized. We present ZME in more detail in Appendix B.
A final remark is that sequences are not necessarily utterances or turns. A single turn can be split into several sequences if it contains incises by the narrator. Reversely, several utterances can be merged in the same sequence if they are not separated by any token labeled as non-DS (O).

heading: 5.2 Experiment Details
text: The deep-learning based models were trained using the train split and the best configuration was
3https://github.com/flairNLP/flair/pull/444
identified using the validation split. Only a part of the hyper-parameters were tuned as explained further in this section. The rule-based baselines do not need training. However, for space limitation, we report in Section 6 only the results of the best performing regex baseline on the validation split. In accordance with the task formalization and most of the existing literature, token-level F1-score was the metric used for model selection, averaged over files to mitigate the influence of longer chapters.
The two rule-based baselines exhibited similar token-level F1-scores on the validation data (over all files): 89% for BookNLP-inspired method (Bamman et al., 2014) and 87% for Byszuk et al. (2020)’s baseline. However, the BookNLP-inspired regex system showed large variance across files and scored 8 points less than its counterpart baseline adapted from (Byszuk et al., 2020) on the averaged token-level F1-score. Thus, we retained only this latter in further analyses, which we denote Regex.
We trained BiLSTM-CRF model for 10 epochs with a batch size of 8 and learning rate set to 0.1. After each epoch, performance was assessed on the validation set and the best configuration over epochs was retained. Regarding the input embeddings, we obtained the largest results for the stacked Flair and FastText, similar to the original work on German AADS (Brunner et al., 2020). We also benchmarked different values (from 64 to 448) for the input size LC . Both token-level and SSM F1-scores peaked for LC = 192 on the validation split, which is the value we keep for test4.
We fine-tuned CamemBERT for 3 epochs with a batch size of 8. Similar to the experimental setup of BiLSTM-CRF, we retained the model that yielded the best results on the validation set after any epoch. We also investigated multiple input size values, LC , from 128 to 512. For each value, training was repeated with 6 different initialisation seeds. LC = 320 led to the best results. By manually analysing the sub-word sequences, we noticed that this value corresponded to the maximal input sequence length accepted by the transformer model after the inner preprocessing for length adjustment. Indeed, smaller word sequences are likely to result in sub-optimal context use while longer word sequences would more often overflow the input size accepted by the model and be automatically split.
4The training for each LC was done with one seed.

heading: 6 Results
text: Table 2 shows the obtained results, overall (top) and averaged over files (bottom). The scores are computed separately on clean (TestC) and noisy (TestN ) data to assess generalization.

heading: 6.1 Performance on well-formatted files
text: The scores on TestC show that Regex is a strong baseline on well-formatted texts, reaching a tokenlevel F1-score of 90% and a SSM F1-score of 45% despite its design limitations (e.g. inability to spot incises). The fine-tuned CamemBERT (F.CamemBERT) substantially outperforms Regex on all computed metrics, especially on span-level metrics. Though BiLSTM-CRF has a poorer tokenlevel performance compared to F.CamemBERT, it yields a competitive SSM F1-score when averaged over files but with a larger variance. In contrast, BiLSTM’s ZME scores are much worse than the F.CamemBERT’s ones and are even worse than those of the simple Regex.
ZME depends on the span length when computing the contribution of each error type (see Appendix B) and BiLSTM appears to make errors concerning longer spans. Also, as further shown by the performances per file in Figure 2, BiLSTM-CRF struggles on La_morte_amoureuse. This can be, at least partly, explained by the nature of this text. The chapter from Théophile Gautier’s work is homodiegetic: it is written at the first person ("je") and the character / narrator directly addresses the reader (frequent use of the second person pronoun "vous"). Thus, it could be particularly hard to distinguish DS from narration on this type of text, especially if the model indirectly relies on such cues. The F.CamemBERT seems more robust even in these challenging settings, although it struggles with identifying full spans in this case.

heading: 6.2 Performance on noisy files
text: The results on TestN allows us to get insights on the generalization capabilities of the baselines, in particular when handling low formatting quality. Regex displays poor generalization which was expected given its design and reliance on typographic and formatting cues. Its token-level F1-score is 53 points less compared to the clean setup in Table 2. In fact, Regex cannot even detect any DS token on some files as shown in Appendix C.
In contrast, deep learning-based models are less impacted by changes in formatting quality in terms of token-level F1 scores. In this regard, the F.CamemBERT remains the best model overall and
averaged over files. BiLSTM-CRF shows a better overall token-level F1 score on TestN than on TestC (88% vs. 83%). As shown in Appendices A and C, it is linked to the model obtaining very good scores on chapters with many DS tokens.
Moreover, the deep learning models are much better than Regex on the span-level metrics. BiLSTM-CRF is slightly more competitive than F.CamemBERT, but the average over files SSM F1-scores are not significantly different. Indeed, as emphasized by the results per file in Appendix C, the performance is chapter-dependent. While F.CamemBERT consistently outperforms the other baselines on token-level F1-score on all files, BiLSTM-CRF is better at recognizing DS spans in about 22 out of 37 files (i.e. 60% of the time). However, we could notice again that the BiLSTMCRF’s ZME scores are quite large but more stable than F.CamemBERT when the test set moves from clean (C) to noisy (N ) (0.02 vs. 0.19 between the two setups). In spite of that, F.CamemBERT clearly appears as the best-performing model in both cases.

heading: 6.3 Qualitative analysis and Discussion
text: We conducted a qualitative analysis by checking the detailed contribution of each ZME error type for all models and by manually comparing a selection of files5 with their corresponding predictions. Table 3 reveals interesting differences (despite a lack of statistical significance) between BiLSTM-CRF and F.CamemBERT on TestN . While BiLSTM-CRF exhibits more Miss, False Alarm and Merge error type contributions to ZME, F.CamemBERT’s ZME score is more impacted by Split errors. The manual investigation of the selected files showed that both deep learning-based models identified much better incises than Regex. This is also consistent with the much lower Merge (21.8 and 12.3 vs. 633.4). Nonetheless, other semantic, syntactic or lexical cues seemed to mislead these models.
On the one hand, BiLSTM-CRF seemed to systematically classify parts of text written at the first person ("je") as DS, which makes it especially unfitted for homodiegetic novels (hence the low performance on La_ morte_ amoureuse). The punctuation seemed to be a strong cue for the model as it tended to classify sentences with exclamation or interrogation marks as DS. Then, BiLSTM-CRF could not handle long and uninterrupted DS para-
5Files from TestC : La_morte_amoureuse and mystères_de_paris_2_troisième_7. Files from TestN : rd0002_1, rd0724_0, and rd0367_1.
graphs. These long DS spans often share registries or production strategies similar to narration (Egbert and Mahlberg, 2020), such as the use of past tense or descriptions, which likely misled the model.
On the other hand, the manual analysis showed that F.CamemBERT appeared better at identifying long DS spans or at classifying homodiegetic narration. However, this model bears other weaknesses. For instance, proper noun phrases seemed to be systematically classified as non-DS. Another common error was the miss-classification as non-DS of [UK] tokens in files using unrecognized non-breaking spaces (e.g. "\xa0") after quotation marks. Plus, the model regularly produced chains of alternating labels on very short groups of words as in Figure 3. These aspects correlated with the high contribution to ZME from False Alarm and Split error types.
Finally, these observations also motivated a final post-processing AADS experiment. A simple heuristic is used a posteriori to hinder incoherent predictions that mix different narrative levels within the same segment of a sentence. The correction of the predicted labels post-model using a majority vote per clause lead to significant improvements on sequence-level metrics for both of the deep learning-based models. Indeed, in all settings –overall and averaged on both clean and noisy files– F.CamemBERT’s SSM F1 scores gained from 5 to 8 points. The performances of the BiLSTM-CRF model are only slightly impacted on TestC but its SSM F1 scores gained in average 5 points on TestN . After this post-processing step, F.CamemBERT shows weaker performances than BiLSTM-CRF only on SSM F1 scores averaged over files on TestN . Details of this clauseconsistent post-processing step, as well as ensuing results, are reported in Appendix E. Altogether the different experiments tend to show that F.CamemBERT is the most promising model for AADS, when computational resources and groundtruth annotation are available for training.

heading: 7 Limitations
text: The current framework bears several limitations. First, although a common strategy in the related literature (Brunner et al., 2020; Ek and Wirén, 2019; Jannidis et al., 2018) which we also adopted, the binary annotation at the token-level is limiting. With this schema, the focus is not on speakers’ utterances or turns, but on DS sequences. A subsequent issue is that consecutive turns by different characters are considered as one DS sequence if there is no "O" labeled tokens between them. One solution could have been to mark the start and end of a DS turn while paying attention to handle imbricated narration (ie. incises). However, this would have required significant more re-annotation efforts, which we left for a future research cycle within the proposed framework.
Second, because of copyright issues the corpus contains excerpts exclusively from a specific period, 1830-1937. Thus, the models were trained and tested on a specific type of literature and may not generalize well to other forms of narratives, in particular modern and contemporary. In this direction, the curation of the test corpus could benefit from more literary insights considering that the evaluation showed high variance of the performance over chapters. This could help to better determine the application scope of the models, and which kind of narratives require further work.
With regard to the deep neural network baselines, we did not perform an extensive parameter search and model optimisation. This could have further improved the results. However, performances on recognizing full DS spans were clearly lower than token-level metrics, which had most likely other causes. Regarding the evaluation, although we adopted ZME scores from page segmentation to have more qualitative insights, there are still other aspects we have not quantified and could be particularly relevant. For instance, does the model tend to miss the beginning, the end or some other specific parts of a DS sequence? We tried to capture some of these phenomena through our manual analysis, but it is challenging to apply it at scale
without introducing methods to automatically compute metrics.

heading: 8 Conclusion
text: We have presented an unified framework to design and evaluate AADS in written French narratives. To our knowledge, this is the largest AADS study to date in French. We consolidated a large dataset annotated per word. Then, we benchmarked two families of baselines, rule and deep learning-based, using as inspiration AADS advances in other languages (German and English). We designed an evaluation which focuses on generalization and on learning about the advantages and weaknesses of each baseline. Results show that rule-based systems work well on bounded DS conventions (quotation marks) in clean text. Other DS formats, incises, and poorly formatted files pose many problems. Deep learning baselines prove to be far more robust, reaching token-level F1-scores up to 95%, but with large variance across files. Yet, recognizing full spans of DS is still challenging, even when texts have good formatting quality.
While for macro analyses in literary studies, imperfect AADS may be sufficient, some use-cases require almost perfect performance when recognizing DS spans (e.g. audiobook generation from text). If a more thorough parameter optimization might help, our qualitative analysis conveys that performance gain should be instead sought by integrating domain knowledge into the models—without feature over-engineering though. Studying the models’ performances after the removal of typographic cues could lead to other insights on how to increase robustness. Multilingual language models and existing AADS corpora could be also exploited for French. Another needed step would be to transition to identifying full DS turns and their corresponding speakers, with the implied manual re-annotation efforts.

heading: A Corpus Details
text: Corpus details (file names, authors, publication years and DS percentages per excerpt) are given in Table 5 for the clean (C) corpus and in Table 4 for the noisy (N ) corpus.

heading: B ZoneMap
text: The ZoneMap Error metric (Galibert et al., 2014) was originally developed for page segmentation. ZoneMap offers a configurable way to compute area segmentation errors based on a typology of possible errors.
Let NT , NP be respectively the number of positive (here, DS) spans from the ground truth, and from the model’s predictions. The corresponding sets can respectively be written as {si}NTi=1 and {s̃j}NPj=1. The length of a span sk (given in terms of tokens) is written as |sk|. Ground truth and predicted spans are grouped according to rules described further into N groups Gk, k = 1, ..., N . Then, the error score attributed to the model is
given by:
EZM = ∑N k=1E(Gk)∑NT
i=1 |si| (1)
where E(Gk) = (1− αC)ES(Gk) + αCEC(Gk) (2)
with αC ∈ [0, 1]. E(Gk) is a linear interpolation of the segmentation error rate ES and the classification error rate EC within group k. Both error types can be defined purposely to penalize the model differently depending on the group type of Gk. Groups’ constructions, types and compositions are defined below.
Groups are constructed based on a link force between true and predicted spans computed as:
fi,j := f(si, s̃j) = ( |si ∩ s̃j | |si| )2 + ( |si ∩ s̃j | |s̃j | )2
i ∈ {1, · · · , NT }, j ∈ {1, · · · , NP } (3)
Non-zero links are then sorted in descending order and areas are combined into groups incrementally according to one rule: if adding a new area to a group leads to the situation where a group contains multiple ground truth or predicted areas, then such an area is not added to the group in question. This process ultimately results in five types of groups:
1. Match: one ground truth area overlaps with one predicted area and none of them overlap with other predicted or ground truth areas (even if the covered areas are not aligned).
2. Miss: one ground truth area is not covered at all by any predicted area.
3. False Alarm: one predicted area is not covered at all by any ground truth area.
4. Split: one ground truth area is covered by at least two predicted areas.
5. Merge: one predicted area is covered by at least two ground truth areas.
Considering the nature of the AADS task as a binary classification, spans will be used instead of areas and classification error rates will be omitted further (set αC = 0).
For both Miss and False Alarm groups, the segmentation error rate is strictly proportional to their length: if Gk = {si}, respectively Gk = {s̃j}; the group contribution to the Zone Map error is
E(Gk) = |si|, respectively E(Gk) = |s̃j |. For Match groups, the group error is proportional to the number of non-overlapping tokens: if Gk = {si, s̃j}, then E(Gk) = |si∪ s̃j |− |si∩ s̃j |, so that E(Gk) = 0 for strict span matches.
Finally, Split and Merge groups are divided into sub-zones that are in turn classified as strict Match and Miss or False Alarm. Miss and False Alarm sub-zones contribute to the error like Miss or False Alarm groups (strictly proportionally to the length of the sub-zones). In contrast, the largest Match sub-zone is not counted as an error and does not contribute to EZM, while the remaining Match subzones are partially counted as errors. Their contribution to the segmentation error rate is proportional to their length, an introduced Merge/Split mitigating parameter αMS ∈ [0, 1] and the relative number of split, respectively merged sub-zones. Given a Split group Gk = {si, {sj,l}nkl=1}, the group is sub-divided into nk strict match sub-zones {zm}nkm=1 and nk ∈ {nk − 1, nk, nk + 1} nonoverlapping spans {zm}nkm=1. The segmentation error rate of such group would then be the sum of non-detected tokens EF (Gk) = ∑nk m=1 |zm| and
split penalization ET = αMS ( ∑nk
m=2 |zm|) nk−1nk . Those formula can then be rewritten in terms of original spans as:
EF (Gk) = ∣∣si ∪ ( ∪nkl=1s̃j,l )∣∣− ∣∣si ∩ ( ∪nkl=1s̃j,l )∣∣
(4)
ET (Gk) = αMSVi,j nk − 1 nk
(5)
Vi,j = |si ∩ ( ∪nkl=1s̃j,l ) | − max
l∈{1,··· ,nk} |si ∩ s̃j,l|
(6)
and E(Gk) = EF (Gk) + ET (Gk). Merge groups error contribution is computed comparably.

heading: C Out-of-distribution results per file
text: Figure 5 shows the results obtained by the considered baselines on each file from TestN , the out-ofdistribution corpus.

heading: D Computing information
text: We trained the models on a 32-core Intel Xeon Gold 1051 6134 CPU @ 3.20GHz CPU with 128GB RAM 1052 equipped with 4 GTX 1080 GPUs with 11GB RAM 1053 each. The required time for train, where applicable, validation and test both on TestC and TestN was a bit less than 2 hours: 15 minutes for Regex, 45 minutes for BiLSTM-CRF, and 40 minutes for CamemBERT.

heading: E Clause-consistent predictions
text: We lead a final post-processing experiment on top of the predictions made by the different models.
This step is meant to ensure the consistency of the automatic annotations at the clause6 level. It relies on a simple heuristic drawn from the knowledge of the task: all words between two consecutive punctuation marks (full stop, question mark, hyphen, quotation mark, etc.) lie at the same narrative level, ie. the sequence of words is either uttered by a character or part of the narrator’s discourse. Thus, all words stemming from a common clause must be associated with the same label.
In practice, this is implemented as a postprocessing step. Based on a model’s predictions, the clause-consistency is ensured by imposing all words from the same clause to be associated with the same label. For each clause, a majority vote is carried out from the predicted labels to determine a consistent unique label for all the words of the clause.
Results of the clause-consistent (CC) postprocessing experiment are disclosed in Table 6. As expected, this post-processing step has only few to no impact on the Regex model’s output. Indeed, this method directly labels sequences of words caught with regular expressions that are redundant with the definition of clauses. However, imposing clause-consistent predictions allows to significantly improve the performances of the BiLSTM-CRF and F.CamemBERT based models. Overall, this heuristic never deteriorates the performances of the models on all performance scores,
6Here, the term clause does not strictly follow its grammatical definition, but it is used to designate any sequence of words between two punctuation marks.
and the improvements are particularly significant for sequence-level metrics.
Enhancements are striking for the F.CamemBERT in all evaluation configurations. This post-processing step allows to alleviate one of the observed weaknesses of the model (see subsection 6.3) by hindering sequences of alternated labels within the same clause. This results in major performance boosts of up to 8 points for the overall SSM F1 on TestN . On the other hand, sequence level performances of BiLSTM-CRF also benefit from the CC-predictions but mainly on noisy files with a gain of 4 points on the overall SSM F1.
Clause consistent predictions allow to reach fairly high scores even on the most challenging task of strict sequence match on well-formated documents, F.CamemBERT reaching on average a SSM F1 score of 78 on TestC . Yet, performances on noisy files remain curtailed F.CamemBERT and BiLSTM-CRF SSM F1 scores on TestN are on average, respectively, 31 and 33 with large variances among files.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: A1. Did you describe the limitations of your work?
Left blank.
A2. Did you discuss any potential risks of your work? Left blank.
A3. Do the abstract and introduction summarize the paper’s main claims? Left blank.
A4. Have you used AI writing assistants when working on this paper? Left blank.

heading: B Did you use or create scientific artifacts?
text: Left blank.
B1. Did you cite the creators of artifacts you used? Left blank.
B2. Did you discuss the license or terms for use and / or distribution of any artifacts? Left blank.
B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Left blank.
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Left blank.
B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? Left blank.
B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Left blank.

heading: C Did you run computational experiments?
text: Left blank.
C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? Left blank.
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? Left blank.
C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Left blank.
C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Left blank.

heading: D Did you use human annotators (e.g., crowdworkers) or research with human participants?
text: Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? Left blank.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? Left blank.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? Left blank.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? Left blank.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? Left blank.

heading: Future Work
text: However, this would have required significant more re-annotation efforts, which we left for a future research cycle within the proposed framework. Second, because of copyright issues the corpus contains excerpts exclusively from a specific period, 1830-1937. Thus, the models were trained and tested on a specific type of literature and may not generalize well to other forms of narratives, in particular modern and contemporary. In this direction, the curation of the test corpus could benefit from more literary insights considering that the evaluation showed high variance of the performance over chapters. This could help to better determine the application scope of the models, and which kind of narratives require further work. With regard to the deep neural network baselines, we did not perform an extensive parameter search and model optimisation. This could have further improved the results. However, performances on recognizing full DS spans were clearly lower than token-level metrics, which had most likely other causes. Regarding the evaluation, although we adopted ZME scores from page segmentation to have more qualitative insights, there are still other aspects we have not quantified and could be particularly relevant. For instance, does the model tend to miss the beginning, the end or some other specific parts of a DS sequence? We tried to capture some of these phenomena through our manual analysis, but it is challenging to apply it at scale
without introducing methods to automatically compute metrics