SlowGuess/NAACL_2022_OCR huggingface dataset:
Dian Yu<sup>1</sup> Ben Zhou<sup>2</sup> Dong Yu<sup>1</sup>
$^{2}$ University of Pennsylvania, Philadelphia, PA
{yudian, dyu} @tencent.com, xyzhou@seas.upenn.edu
Speaker identification (SI) in texts aims to identify the speaker(s) for each utterance in texts. Previous studies divide SI into several sub-tasks (e.g., quote extraction, named entity recognition, gender identification, and coreference resolution). However, we are still far from solving these sub-tasks, making SI systems that rely on them seriously suffer from error propagation. End-to-end SI systems, on the other hand, are not limited by individual modules, but suffer from insufficient training data from the existing small-scale datasets. To make large end-to-end models possible, we design a new annotation guideline that regards SI as span extraction from the local context, and we annotate by far the largest SI dataset for Chinese named CSI based on eighteen novels. Viewing SI as a span extraction task also introduces the possibility of applying existing strong extractive machine reading comprehension (MRC) baselines. Surprisingly, simply using such a baseline without human-annotated character names and carefully designed rules, we can already achieve performance comparable or better than those of previous state-of-the-art SI methods on all public SI datasets for Chinese. Furthermore, we show that our dataset can serve as additional training data for existing benchmarks, which leads to further gains (up to  $6.5\%$  in accuracy). Finally, using CSI as a clean source, we design an effective self-training paradigm to continuously leverage hundreds of unlabeled novels.
Speaker identification (SI) aims to identify the corresponding speakers for utterances in texts (Zhang et al., 2003; Glass and Bangay, 2007). Most existing SI datasets (He et al., 2013; Chen et al., 2021) provide ground-truth character aliases and utterance spans as inputs. However, such annotations are unavailable in realistic settings, under which SI is usually divided into interrelated sub-tasks (Pan et al., 2021; Yoder et al., 2021) (e.g., utterance
identification, named entity recognition, coreference resolution, and candidate speaker generation).
However, this pipeline faces several challenges. First, these modules are imperfect, and they inevitably introduce errors that propagate and seriously affect the final performance (C-I). For example, the performance of the state-of-the-art coreference resolution model is about  $80.3\%$  in F1 (Kirstain et al., 2021). Second, classical SI datasets and approaches assume that the speaker to be linkable to one of the named entities, which cannot handle the more realistic settings where the speakers are not humans or only exist as nominals (e.g., "a young girl" or "smartwatch") (C-II). Third, features (e.g., speech verb list and position information) and rules are usually carefully created and selected by experts for a certain language, which may make these resources difficult to be used for other languages (C-III). Finally, one of the main reasons that people heavily rely on pipeline methods is that book-level exhaustive annotations of SI datasets are too expensive. As a result, existing small-scale annotations are insufficient to train large models, especially the advanced pre-trained language models (Devlin et al., 2019) (C-VI).
This work focuses on the abovementioned four challenges. We first design a new annotation guideline that simplifies the task to span extraction from the local context and thus viewing SI as an end-to-end task (C-I): given a snippet that contains several contiguous paragraphs, for each paragraph that may contain utterances, we annotate the most informative reference to a speaker (i.e., speaker mention) if one exists, otherwise the content within quotation marks. As a result, speaker mentions are not limited to entities only (C-II). This simplification is built on two assumptions: utterances in a single paragraph usually correspond to a single speaker (He et al., 2013) (e.g., all utterances in paragraph U1 in Table 1 are said by "Brandi") and an explicit speaker mention is very likely to appear
Human Language Technologies, pages 2274 - 2285
in the local context (Glass and Bangay, 2007), especially when the context starts and ends with a paragraph that does not contain any utterances (Section 3.3). Under this guideline, annotation efforts are greatly reduced because annotators only need to select a span based on a text snippet and avoid steps such as creating and maintaining a book-specific list of characters and their aliases, in-depth chapter-level or book-level understanding. In total, we annotate 66K Speaker Identification instances based on eighteen Chinese novels named CSI.
Considering the similarity between simplified SI and extractive machine reading comprehension (MRC), which aims to extract an answer span from a given document for a question (Hermann et al., 2015), we can easily adapt an MRC baseline to SI, which does not consider any language-specific features (C-III). Surprisingly, simply using such an extractive method (Xu et al., 2020) already yields comparable or better performance than that of state-of-the-art systems on two public SI datasets for Chinese (WP (Chen et al., 2021) and JY (Jia et al., 2021)). Furthermore, our experimental results demonstrate that CSI can serve as additional high-quality training data for existing SI datasets, though it follows an extractive annotation guideline based on local context, which is quite different from that of traditional book-level annotation. Finally, using CSI as a clean source, we develop a simple yet effective self-training paradigm to continue leveraging hundreds of unlabeled novels (C-IV), which further reduces the gap between supervised and zero-shot performance on WP and JY.
- We design a new annotation guideline that simplifies book-level SI to span extraction based on the local context, which alleviates the annotation burden and covers diverse types of speakers instead of entities alone.
- We offer a large-scale dataset for Chinese to support end-to-end extractive SI, which can also serve as high-quality training data for existing SI datasets.
- We propose the first end-to-end SI method, which achieves comparable or better performance than that of state-of-the-art methods on all SI datasets for Chinese without requiring any manually designed rules and features.
- We are the first to leverage large-scale unlabeled novels to improve SI via self-training, and our recipes to make self-training work for these tasks may shed light on future studies.
N1 Layla didn't give Brandy anything dangerous, so she put away things like a silver knife. She only asked her to use a mill, and then Brandi was able to sit there and grind all the peppers in the house into powder.
U1 Brandi was very careful. When she showed Layla, she said, "The particles of this bottle are a little bit thicker." She put down a crystal bottle and said "The particles of this bottle are a little finer". After she put another crystal bottle and then star, Brandy asked Layla, "Mom, will you give me anything that needs to be ground?".
U2 "Okay." The mother was dizzy and dizzy at the moment when her daughter raised her face and hold Brandi's small hands to the kitchen and told her to grind whatever she wanted.
N2 Coarse sugar is all ground into fine sugar, cooked sesame seeds are all ground into sesame powder, and there are other things such as cinnamon.
U1 Brandi
U2 Layla
Table 1: An translated example containing two utterance paragraphs (U) in CSI (N: narrative paragraph).
We will compare in detail existing SI tasks/datasets and CSI in Section 3.1 and Section 3.2.
This work is the first attempt to apply self-training (Yarowsky, 1995; Riloff and Wiebe, 2003) to SI. Previous studies on other natural language understanding tasks using self-training mostly generate pseudo-labeled data based on in-domain unannotated data (Du et al., 2021) or data in the same domain (Wang et al., 2021). In addition, those studies usually fix the unannotated data pool in each iteration. We propose continual self-training to feed a model with pseudo-labeled data based on different unlabeled out-of-domain data in each iteration, removing the burden of widely adopted strategies such as selecting ample in-domain unlabeled data (which may not exist) and filtering some of the pseudo-labeled data after each iteration (Chen et al., 2011; Ye et al., 2020; Cascante-Bonilla et al., 2021) to either improve the quality or control the difficulty of noisy pseudo-labeled data.
Different from continual learning (Ring et al., 1994), we stick to the SI task. As the clean data of the target task is always used during training, the proposed paradigm tends not to suffer from catastrophic forgetting (McCloskey and Cohen, 1989).
<sup>1</sup>We will release the code, pre-trained model, and resources without distributing copies of any copyrighted work at https://github.com/yudiandoris/csi.
3 Guideline and Dataset Annotation
3.1 Existing Task Formulations and Datasets
Most of the existing SI datasets are in rich-resource languages such as English (e.g., CQSAC (Elson and McKeown, 2010), P&P (He et al., 2013), QuoteLi3 (Muzny et al., 2017), and RiQuA (Papay and Padó, 2020)) and Chinese (WP (Chen et al., 2019, 2021) and JY (Jia et al., 2021)), and the annotated texts are mostly classical novels or nonfiction texts (e.g., RWG (Brunner, 2013)). See data statistics in Table 2.
Some of the datasets such as P&P and WP also provide a human-labeled list of main characters in a novel, which contains different mentions (if any exists) of each main character, or a small number of candidate speakers for each utterance instance (e.g., JY). When a character list is unavailable, person names that appear in the surrounding context of the utterance is regarded as candidate speakers (Pan et al., 2021). Thus SI tasks are usually formulated as ranking (He et al., 2013) or classification (Muzny et al., 2017) problems, and golden-standard gender information of speakers can be used as features to facilitate speaker identification.
3.2 Assumptions for Annotation
Main characters who are important to the story are usually named, and they play essential roles in many downstream tasks such as character personality prediction (Flekova and Gurevych, 2015) and character network construction (Labatut and Bost, 2019). And this might explain why previous SI resource studies put more emphasis on person entities and their anaphoric mentions during annotation, leading to entity-centric designs for most SI methods. However, unnamed speakers (e.g., "pedestrian" and "cat"), who are usually created as minor characters, and non-living things (e.g., "robot") are seldom annotated, limiting the usage of SI in real-world applications such as audiobook reading (Hinterleitner et al., 2011) that require exhaustive identification of all kinds of speakers.
Another challenge is that existing SI tasks mostly regard ground truth utterances as inputs, which, unfortunately, are not readily available in real-world book-based applications. Worse still, quote identification itself is a research challenge (e.g., overall F1 around  $50\% - 60\%$  (Lee et al., 2020)).
To address the two issues and support end-to-end training, we first propose a new annotation guideline for SI that considers different types of speakers
(e.g., multiple speakers, entities, person names in other languages, and phrases) and at the same time addresses non-utterance quotation identification. Given a snippet that contains several contiguous paragraphs, for each paragraph that may contain utterances, we annotate the most informative mention of the corresponding speaker if one exists, otherwise the earliest mentioned content punctuated with quotation marks (quotation marks included). This simplification is built on two widely held assumptions: (I) utterances in a single paragraph usually correspond to a single speaker (He et al., 2013) and (II) an explicit speaker mention is very likely to appear in the surrounding context of the target utterance (Glass and Bangay, 2007). See more discussions about the two assumptions and exceptions based on our annotated corpus in Section 3.4.
3.3 Candidate Utterance Paragraph Identification and Context Selection
Based on Assumption (I), we aggressively regard that all utterances in a paragraph are said by the same speaker. Thus, we do not conduct quote identification as previous studies, which saves annotation cost. We use (context, paragraph) pairs in which the paragraph may contain utterances as annotation instances. We aim to annotate the most informative speaker mention within the surrounding context of the paragraph or from the paragraph.
To save annotation efforts, we simply regard all paragraphs that contain at least one double quotation mark as candidate utterance paragraph (i.e., a paragraph that contains one or multiple utterances) and regard others as narrative paragraphs. Paragraphs are split by line breaks. To select local context, we argue that we can regard the nearest narrative paragraphs before and after the candidate as context boundaries. We refer to this method of context selection as narrative window. Considering the role of supporting dialogue understanding, this kind of context can somehow be regarded as a mini-scene, similar to those in movie/TV show scripts that provide structured information such as a narrative description of the location and time, the events of a scene, and non-verbal behaviors (e.g., actions or attitudes) of speakers beyond dialogues. We only keep instances whose context contains fewer than ten candidate utterance paragraphs, as long-text understanding is also quite challenging.
For speaker mentions that are enclosed by quotation marks, we only annotate the speaker mention.
dataset	language	types of speakers	# of books	# of utterances*	avg. context length (tokens)
CQSAC	English	entity/mention†	11	3,176	-
P&P	English	entity	3	1,901	-
QuoteLi3	English	entity/mention†	3	2,296	-
RiQuA	English	entity/mention†	11	5,963	-
RWG	German	entity/mention†	13	9,451	-
JINYONG	Chinese	entity	3	28,597	191
WP	Chinese	entity	1	2,596	353
CSI (this work)	Chinese	entity/phrase/pronoun/multi-speaker	18	65,540	180
Table 2: Existing publicly available speaker identification and profiling datasets for novels in English and Chinese (  $\star$  assuming one speaker (if any) per paragraph; †: some mentions could be mapped onto a named person).
In contrast, using a fixed-length window is less flexible to cover sufficient context across diverse types of books by different authors, though it can be used to augment pseudo-labeled training data: for example, using a six-paragraph window based on the same book corpora as that of CSI, we can generate about  $33.7\%$  more unlabeled instances and can easily obtain more by changing the sliding window (e.g., length and center). We will have more discussions about the impact of window selection on performance in Section A.2.
To simplify annotation, we do not further distinguish utterances whose speakers are unclear and expressions enclosed by quotation marks that can be expressions such as idioms, proverbs, or poetics. For convenience, we call them non-utterance paragraphs (e.g., NU1 in Table 3). To ensure the quality of the annotation, the data is independently annotated by the first author of this paper (EA) and a group of annotators (GA) who are native speakers of Chinese from a commercial data annotation company (\(0.071 per instance). The inter-annotator agreement between EA and GA is measured using Cohen's kappa, which yields a value of 0.76 (substantial agreement (Viera et al., 2005)). The disagreements are reviewed and re-annotated by the author to obtain the final annotation.
Entity-Level vs. Mention-Level: Speaker mentions that are entities are more informative and therefore support better disambiguation of different speakers than other types of mentions. However, mention-level annotation can support diverse types of speakers (see Figure 1) and is relatively easy for annotators as no long-text comprehension is needed. Though there exist differences between entity-level and mention-level SI datasets, we argue that the latter one can help entity-level SI tasks, which is supported by our results in Section 5.3.
N1 Si Teng's Hongmen banquet was set at a high-end clubhouse near Qingcheng Mountain. At that time, he would dine on a glass terrace extending out of the lake. It was next to the water by the railing, and the opposite side were silent green mountains.
NU1 It is said that one or two girls in blue calico clothes will be arranged at that time. The girl hold a paper umbrella on one or two flat boats floating across the lake in the distance. If it rains that day, it means "staying in breeze and drizzle meets his will", and if the sun is out, it means "the brimming waves delight the eyes on sunny days."
U2 The proprietoress strongly recommended to Qin Fang: "It is very comfortable. When you eat here, what you eat is not food, but spiritual enjoyment."
N2 Those Taoist masters will be probably mentally nervous, so it's okay to let them have some spiritual enjoyment and adjustment.
NU1 "staying in breeze and drizzle meets his will"  
U2 proprietress
Table 3: An translated example with utterance paragraph (U) and non-utterance paragraph (NU) in CSI.
Figure 1: Distribution of general phenomena in CSI.
Assumption I: There has been some disagreement among different literature (e.g., (Muzny et al., 2017)) about this assumption that all utterances within a paragraph could be attributed to the same speaker. We randomly select 400 instances from CSI and only find one exception  $(0.25\%)$ . This is perhaps because of linguistic changes over years (Nettle, 1999): the books we use for annotation are mostly web novels published after the year 2010 (full list in Table 8 (Appendix A.1)), compared with classical novels published hundred years ago (Papay and Padó, 2020). Also, we restrict the scope of this observation to Chinese as there may exist significant differences among different
languages. Furthermore, our experimental results show that such kind of data annotated following this assumption can be used as useful additional training data for other SI datasets in which the ground truth utterances are provided (Section 5.3). Assumption II: Most previous SI datasets provide human-summarized book-level characters and their different name mentions (e.g., WP provides 125 characters, each with 1-5 different mentions (Chen et al., 2021)) or a small number of candidate characters for each instance. In CSI, we skip the step of clustering different mentions of one speaker. Instead, we assume that we can find an informative mention within a narrative window centered around an utterance paragraph. To examine this assumption and its potential negative impacts, we first dig into CSI. Based on all instances in it that involve eighteen novels, the annotated speaker mentions for merely  $2.7\%$  of utterance paragraphs (63,165 in total) are personal pronouns, indicating that in most cases we can find the corresponding speaker mentions more informative than personal pronouns. This also indirectly supports that context constructing using narrative window contains relatively sufficient information for SI. Furthermore, without using any forms of character lists, applying a baseline that will be introduced in Section 4 to extract a speaker mention from the given context can achieve promising results on existing SI datasets (Section 5.3). Though we admit that additional mappings can be particularly important for first-person narratives, where “ $T$ ” will be frequently annotated as the most informative mention.
4.1 Extractive Machine Reading Comprehension
As the speaker mention must be a span in the context based on the guideline, we consider an extractive MRC model built upon a pre-trained language model (e.g., (Devlin et al., 2019)) that aims to extract an answer of a give question from a document.
Given a paragraph  $q$  that may contain an utterance and its context  $d$ , we follow previous work (e.g., (Devlin et al., 2019)) to concatenate a special token [CLS], tokens in  $q$ , a special token [SEP], and tokens  $d$  that covers the piece of text in  $q$ . Two vectors  $p_{\mathrm{start}}$  and  $p_{\mathrm{end}}$  are introduced to represent the estimated probabilities of each token in  $d$  to be the start or end token of the correct answer span  $a$  that appears in  $d$ , respectively. Let  $a_{\mathrm{start}}$  and  $a_{\mathrm{end}}$
denote the start offset and end offset of  $a$ , respectively.
We optimize the extractive SI model with parameters  $\theta$  by minimizing  $\sum_{t\in V}L(t,\theta)$ , where  $V$  represents the set of speaker identification instances, and  $L$  is defined as:
$$
L (t, \theta) = - \log_ {p _ {\text {s t a r t}, \theta} (a _ {\text {s t a r t}} | t)} - \log_ {p _ {\text {e n d}, \theta} (a _ {\text {e n d}} | t)}.
$$
4.2 Self-Training
We first generate a fixed set  $W$  of unlabeled extractive SI instances from the unlabeled books and conduct the following self-training paradigm. First, we use the labeled data  $V$  to train a teacher model. Then the resulting teacher model generates pseudo-labels for the unlabeled instances. Finally, we train a student model with the combination of pseudo-labeled and labeled data. We simply regard different types of data equally and thus optimize the model by minimizing  $\sum_{t \in V \cup W} L(t, \theta)$ . As this loss is already reasonably stable in our experiments (Section 5), we leave the exploration of other choices such as weighted normalized loss (e.g., (Zoph et al., 2020)) for future studies.
The resulting model can be used as a new teacher to generate new pseudo-labeled data  $W$ , and we can iterate the self-training procedure until no gains are observed or a pre-defined number of iterations.
4.3 Continual Self-Training
In our experiments (Section 5.5), we observe that iterative training over the same unlabeled corpus does not lead to gains for SI as considerable as those obtained on vision tasks such as image classification (Xie et al., 2020). Furthermore, we aim to let our SI model keep learning from different books in diverse domains written by different authors to improve its generalization ability.
Thus, different from previous studies, we collect  $n$  pairwise disjoint sets of unlabeled instances  $\{W_1, W_2, \ldots, W_n\}$ , each based on an set of books. In each iteration, we use an unvisited set of unlabeled instances (e.g.,  $W_1$ ) as  $W$  in Section 4.2. See Figure 2 for the overview of the paradigm.
5.1 Data Statics and Evaluation Metrics
There is no book overlap between the training and dev sets in CSI as in previous datasets JY written by Jin Yong (Jia et al., 2021) and WP authored
Figure 2: Continual self-training for speaker identification.
by Lu Yao (Chen et al., 2021) (Table 4). None of the books in CSI or the unlabeled corpora are written by the two authors to avoid data leakage. Previous methods use accuracy as the evaluation metric since a candidate list is provided. We report macro-averaged F1 and exact match (EM) following previous extractive MRC work (Rajpurkar et al., 2016). As each instance in WP and JY must have a ground truth speaker, EM equals accuracy.
name	set	candidate provided	# of books	# of instances
JY	train	✓	17,159
dev	✓	3	5,719
test	✓	5,719
WP	train	✓	2,000
dev	✓	1	298
test	✓	298
CSI	train	×	10	48,037
dev	×	8	17,503
CORPUS_1	train	×	65	232,239
CORPUS_2	train	×	26	232,240
CORPUS_3	train	×	60	141,251
CORPUS_4	train	×	99	187,524
Table 4: Statistics of the speaker identification datasets for Chinese: JY, WP, CSI (this work), and pseudolabeled training data from unlabeled books.
We adopt the MRC model (Xu et al., 2020) as the baseline model that is built upon a pre-trained language model for Chinese. We experiment with BERT-wwm-ext-base and RoBERTa-wwm-ext-large (Cui et al., 2020), and our method can be easily used upon other recently release pre-trained language models. We use the same model architecture for teacher and student models. We train a model for five epochs when only clean data is used during training (i.e., CSI, JW, and PW) and train it for only one epoch when relatively large-scale