SlowGuess/ACL_2020_OCR huggingface dataset:
Belinda Z. Li†* Gabriel Stanovsky\* Luke Zettlemoyer\*  
\* University of Washington Allen Institute for AI Facebook belindali@fb.com {gabis,1sz}@cs.washington.edu
We improve upon pairwise annotation for active learning in coreference resolution, by asking annotators to identify mention antecedents if a presented mention pair is deemed not coreferent. This simple modification, when combined with a novel mention clustering algorithm for selecting which examples to label, is much more efficient in terms of the performance obtained per annotation budget. In experiments with existing benchmark coreference datasets, we show that the signal from this additional question leads to significant performance gains per human-annotation hour. Future work can use our annotation protocol to effectively develop coreference models for new domains. Our code is publicly available. $^1$
Coreference resolution is the task of resolving anaphoric expressions to their antecedents (see Figure 1). It is often required in downstream applications such as question answering (Dasigi et al., 2019) or machine translation (Stanovsky et al., 2019). Exhaustively annotating coreference is an expensive process as it requires tracking coreference chains across long passages of text. In news stories, for example, important entities may be referenced many paragraphs after their introduction.
Active learning is a technique which aims to reduce costs by annotating samples which will be most beneficial for the learning process, rather than fully labeling a large fixed training set. Active learning consists of two components: (1) a task-specific learning algorithm, and (2) an iterative sample selection algorithm, which examines the performance of the model trained at the previous iteration and selects samples to add to the annotated
A volcano in Mexico, known to locals as Po-po, just started spewing molten rock. Are the two mentions coreferent? No What is the first appearance of the entity that the yellowhighlighted text refers to? A volcano in Mexico
Figure 1: Discrete annotation. The annotator is shown the document, a span (yellow), and the span's predicted antecedent (blue). In case the answer to the coreference question is negative (i.e., the spans are not coreferring), we present a follow-up question ("what is the first appearance of the entity?"), providing additional cost-effective signal. Our annotation interface can be seen in Figure 5 in the Appendix.
training set. This method has proven successful for various tasks in low-resource domains (Garrette and Baldridge, 2013; Kholghi et al., 2015; Syed et al., 2016, 2017).
Sachan et al. (2015) showed that active learning can be employed for the coreference resolution task. They used gold data to simulate pairwise human-annotations, where two entity mentions are annotated as either coreferring or not (see first question in Figure 1).
In this paper, we propose two improvements to active learning for coreference resolution. First, we introduce the notion of discrete annotation (Section 3), which augments pairwise annotation by introducing a simple additional question: if the user deems the two mentions non-coreferring, they are asked to mark the first occurrence of one of the mentions (see second question in Figure 1). We show that this simple addition has several positive implications. The feedback is relatively easy for annotators to give, and provides meaningful signal which dramatically reduces the number of annotations needed to fully label a document.
Second, we introduce mention clustering (Section 4). When selecting the next mention to label, we take into account aggregate model predictions
*Work done while at the University of Washington.  
<sup>1</sup>https://github.com/belindal/discrete-active-learning-coref
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8320-8331 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
for all antecedents which belong to the same cluster. This avoids repeated labeling that would come with separately verifying every mention pair within the same cluster, as done in previous methods.
We conduct experiments across several sample selection algorithms using existing gold data for user labels and show that both of our contributions significantly improve performance on the CoNLL-2012 dataset (Pradhan et al., 2012). Overall, our active learning method presents a superior alternative to pairwise annotation for coreference resolution, achieving better performing models for a given annotation budget.
Our work relies on two main components: a coreference resolution model and a sample selection algorithm.
Coreference resolution model We use the span ranking model introduced by Lee et al. (2017), and later implemented in AllenNLP framework (Gardner et al., 2018). This model computes span embeddings for all possible spans  $i$  in a document, and uses them to compute a probability distribution  $P(y = \mathrm{ant}(i))$  over the set of all candidate antecedents  $\mathcal{Y}(i) = \{K \text{ previous mentions in the document}\} \cup \{\epsilon\}$ , where  $\epsilon$  is a dummy antecedent signifying that span  $i$  has no antecedent. This model does not require additional resources, such as syntactic dependencies or named entity recognition, and is thus well-suited for active learning scenarios for low-resource domains.
Sample selection algorithm Previous approaches for the annotation of coreference resolution have used mostly pairwise selection, where pairs of mentions are shown to a human annotator who marks whether they are co-referring (Gasperin, 2009; Laws et al., 2012; Zhao and Ng, 2014; Sachan et al., 2015). To incorporate these binary annotations into their clustering coreference model, Sachan et al. (2015) introduced the notion of must-link and cannot-link penalties, which we describe and extend in Section 4.
3 Discrete Annotation
In discrete annotation, as exemplified in Figure 1, we present the annotator with a document where the least certain span  $i$  ("Po-po", in the example) and  $i$ 's model-predicted antecedent,  $A(i)$  ("locals"), are
highlighted. Similarly to pairwise annotation, annotators are first asked whether  $i$  and  $A(i)$  are coreferent. If they answer positively, we move on to the next sample. Otherwise, we deviate from pairwise sampling and ask the annotator to mark the antecedent for  $i$  ("A volcano in Mexico") as the follow-up question. The annotator can abstain from answering the follow-up question in case  $i$  is not a valid mention or if it does not have an antecedent in the document. See Figure 5 in the Appendix for more example annotations.
In Section 5, we show that discrete annotation is superior to the classic pairwise annotation in several aspects. First, it makes better use of human annotation time, as often an annotator needs to resolve the antecedent of the presented mention to answer the first question. For example, identifying that "Po-po" refers to the volcano, and not the locals. Second, we find that discrete annotation is a better fit for mention ranking models (Lee et al., 2017), which assign the most-likely antecedent to each mention, just as an annotator does in discrete annotation.
4 Mention Clustering
We experiment with three selection techniques by applying popular active learning selectors like entropy or query-by-committee (Settles, 2010) to clusters of spans. Because our model outputs antecedent probabilities and predictions, we would like to aggregate these outputs, such that we have only one probability per mention cluster rather than one per antecedent. We motivate this with an example: suppose span  $i$ 's top two most likely antecedents are  $y_{1}$  and  $y_{2}$ . In scenario 1,  $y_{1}$  and  $y_{2}$  are predicted to be clustered together, and in scenario 2, they are predicted to be clustered apart. Span  $i$  should have a "higher certainty" in scenario 1 (and thus be less likely to be picked by active learning), because its two most likely antecedents both imply the same clustering, whereas in scenario 2, picking  $y_{1}$  vs.  $y_{2}$  results in a different downstream clustering. Thus, rather than simply using the raw probability  $i$  refers to a particular antecedents, we use the probability  $i$  belongs to a certain cluster. This implies modelling  $y_{1}$  and  $y_{2}$  "jointly" in scenario 1, and separately in scenario 2.
Formally, we compute the probability that a span  $i$  belongs in a cluster  $C$  by summing  $P(\operatorname{ant}(i) = y)$
For consistency, we ask annotators to select the first antecedent of  $i$  in the document.
8321
for all  $y$  that belong in some cluster  $C$ , since  $i$  having an antecedent in a cluster necessarily also implies  $i$  is also in that cluster. This allows us to convert the predicted antecedent probabilities to in-cluster probabilities:
$$
P (i \in C) = \sum_ {y \in C \cap \mathcal {Y} (i)} P (\operatorname {a n t} (i) = y) \tag {1}
$$
Similarly, for query-by-committee, we aggregate predictions such that we have one vote per cluster rather than one vote per antecedent:
$$
V (i \in C) = \sum_ {y \in C \cap \mathcal {Y} (i)} V (A (i) = y) \tag {2}
$$
where  $V(A(i) = y)\in \{0,1,\dots ,\mathcal{M}\}$  refers to the number of models that voted  $y$  to be the antecedent of  $i$ .
The cluster information  $(y\in C\cap \mathcal{V}(i))$  we use in Equations 1 and 2 is computed from a combination of model-predicted labels and labels queried through active learning. Antecedents which were not predicted to be in clusters are treated as singleton clusters.
Additionally, to respect user annotations during the selection process, we must keep track of all prior annotations. To do this, we use the concept of must-link (ML; if two mentions are judged coreferent) and cannot-link (CL; if two mentions are judged non-coreferent) relations between mentions introduced by Sachan et al. (2015), and adapt it for our purposes. Specifically, in our discrete setting, we build the links as follows: if the user deems the pair coreferent, it is added to ML. Otherwise, it is added to CL, while the user-corrected pair (from the second question) is always added to ML.
In addition, we use these links to guide how we select for the next mention to query. For example, if a CL relation exists between spans  $m_{1}$  and  $m_{2}$ , we will be less likely to query for  $m_{1}$ , since we are slightly more certain on what  $m_{1}$ 's antecedent should be (not  $m_{2}$ ). Formally, we revise probabilities and votes  $P(i \in C)$  and  $V(i \in C)$  in accordance to our link relations, which affects the selector uncertainty scores. $^{3}$
Finally, following (Sachan et al., 2015), we impose transitivity constraints, which allow us to model links beyond what has been explicitly
pointed out during annotation:
$$
M L \left(m _ {i}, m _ {j}\right) \wedge M L \left(m _ {j}, m _ {k}\right)\rightarrow M L \left(m _ {i}, m _ {k}\right) \tag {3}
$$
$$
C L \left(m _ {i}, m _ {j}\right) \wedge M L \left(m _ {i}, m _ {k}\right)\rightarrow C L \left(m _ {j}, m _ {k}\right) \tag {4}
$$
However, recomputing these closures after each active learning iteration can be extremely inefficient. Instead, we build up the closure incrementally by adding only the minimum number of necessary links to maintain the closure every time a new link is added.
We experiment with the following clustered selection techniques:
Clustered entropy We compute entropy over cluster probabilities and select the mention with the highest clustered entropy:
$$
E (i) = - \sum_ {C \in \text {a l l c l u s t e r s}} P (i \in C) \cdot \log P (i \in C) \tag {5}
$$
Where  $P(i \in C)$  is defined as in Equation 1.
Clustered query-by-committee We train  $\mathcal{M}$  models (with different random seeds) and select the mention with the highest cluster vote entropy:
$$
\mathrm {V E} (i) = - \sum_ {C \in \text {a l l c l u s t e r s})} \frac {V (i \in C)}{\mathcal {M}} \cdot \log \frac {V (i \in C)}{\mathcal {M}} \tag {6}
$$
Using votes counted over clusters, as defined in Equation 2.
Least coreferent clustered mentions / Most coreferent unclustered mentions (LCC/MCU) We aim to select a subset of spans for which the model was least confident in its prediction. For each span  $i$  which was assigned a cluster  $C_i$ , we compute a score  $s_C(i) = P(i \in C_i)$ , and choose  $n$  spans with the smallest  $s_C(i)$ . For each singleton  $j$ , we give an "unclustered" score  $s_U(i) = \max_{C \in \text{all clusters}} P(j \in C)$  and choose  $m$  spans with the largest  $s_U(i)$ .  $P(i \in C_i)$  and  $P(j \in C)$  are computed with Equation 1.
We compare discrete versus pairwise annotation using the English CoNLL-2012 coreference dataset (Pradhan et al., 2012). Following Sachan et al. (2015), we conduct experiments where user judgments are simulated from gold labels.
See Section A.2 in the appendix for more details.
8322
Figure 2: Comparing various selectors for discrete versus pairwise annotation (dashed orange line).
Set	# labels/doc	Active learning
iteration	# docs	# ?s
A	20	1st (retrained 0x)	5	15
20	7th (retrained 6x)	5	15
200	2nd (retrained 1x)	5	15
200	8th (retrained 7x)	5	15
B	20	2nd (retrained 1x)	5	15
20	8th (retrained 7x)	5	15
200	1st (retrained 0x)	5	15
200	7th (retrained 6x)	5	15
Annotation time estimation To compare annotation times between pairwise and discrete questions, we collected eight 30-minute sessions from 7 in-house annotators with background in NLP. Annotators were asked to answer as many instances as they could during those 30 minutes. We additionally asked 1 annotator to annotate only discrete questions for 30 minutes. To be as representative as possible, the active learning queries for these experiments were sampled from various stages of active learning (see Table 1). On average, an annotator completed about 67 questions in a single session, half of which were answered negatively, requiring the additional discrete question. Overall, these estimates rely on 826 annotated answers. Our annotation interface is publicly available, $^{4}$  see examples in Figure 5 in the Appendix.
Timing results are shown in Table 2. Answering
Table 1: Timing experiments sampling. For each of the 2 datasets, we collected 60 total active learning questions from 20 documents. We collected 5 documents and 15 questions for each of the 4 categories: trained with many/few labels per document, and early/late in active learning process. The 15 questions were sampled randomly from within an iteration.
Avg. Time per ?
Initial question	15.96s
Follow-up question	15.57s
ONLY Follow-up questions	28.01s
Table 2: Average annotation time for the initial pairwise question, the discrete followup question, and the discrete question on its own.
the discrete question after the initial pairwise question takes about the same time as answering the first question (about  $16s$ ). Furthermore, answering only discrete questions took  $28.01s$  per question, which confirmed that having an initial pairwise question indeed saves annotator time if answered positively.
In the following experiments, we use these measurements to calibrate pairwise and discrete followup questions when computing total annotation times.
Baselines We implement a baseline for pairwise annotation with entropy selector. We also implement two discrete annotation baselines with random selection. The partially-labelled baseline follows the standard active learning training loop, but selects the next mention to label at random. The fully-labelled baseline creates a subset of the training data by taking as input an annotation time  $t$  and selecting at random a set of documents that the user can fully label in  $t$  hours using ONLY discrete annotation. By comparing the fully-labelled baseline against our active learning results, we can determine whether active learning is effective over labelling documents exhaustively.
Hyperparameters We use the model hyperparameters from the AllenNLP implementation of Lee et al. (2017). We train up to 20 epochs with a patience of 2 before adding labels. After all documents have been added, we retrain from scratch. We use a query-by-committee of  $\mathcal{M} = 3$  models, due to memory constraints. For LCC/MCU, given  $L$  annotations per document, we split the annotations equally between clusters and singletons.
Results Figure 2 plots the performance of discrete annotation with the various selectors from Section 4, against the performance of pairwise annotation, calibrated according to our timing experiments. In all figures, we report MUC, B3, and CEAFe as an averaged F1 score.
The three non-random active learning frameworks outperform the fully-labelled baseline, show
4https://belindal.github.io/timing_experiments
8323
Figure 3: Mention detection accuracy (in document-micro F1) for pairwise versus discrete selection per human annotation time.
ing that active learning is more effective for coreference resolution when annotation budget is limited.
Most notably, Figure 2 shows that every nonrandom discrete selection protocol outperforms pairwise annotation. Where the gap in performance is the largest ( $>15$  minutes per document), we consistently improve by  $\sim 4\%$  absolute  $F1$  over pairwise selection.
A major reason for discrete annotations outperforms the pairwise baseline. The number of pairwise annotations needed to fully label a document is much larger than the number of discrete annotations. In an average development document with 201 candidates per mention, the number of pairwise queries needed to fully label a document is 15,050, while the maximum number of discrete queries is only 201 (i.e., asking for the antecedent of every mention). Thus, the average document can be fully annotated via discrete annotation in only  $2.6\%$  of the time it takes to fully label it with pairwise annotation, suggesting that our framework is also a viable exhaustive annotation scheme.
Further analysis shows that the improvement in discrete selection stems in part from better use of annotation time for mention detection accuracy (Figure 3) and pronoun resolution (Figure 4), in which we measure performance only on clusters with pronouns, as identified automatically by the spaCy tagger (Honnibal and Montani, 2017).
Finally, Table 3 shows ablations on our discrete annotation framework, showing the contribution of each component of our paradigm.
Figure 4: Pronoun resolution accuracy (average F1) for pairwise versus discrete selection per human annotation time.
F1 score
Discrete annotation	57.08
—clustered probabilities	56.49
—incremental link closures	56.98
Pairwise annotation	54.27
Table 3: Ablations over the different model elements, at a single point ( $\sim$ 315 annotation hours). Entropy selector was used for all experiments.
We presented discrete annotation, an attractive alternative to pairwise annotation in active learning of coreference resolution in low-resource domains. By adding a simple question to the annotation interface, we obtained significantly better models per human-annotation hour. In addition, we introduced a clustering technique which further optimizes sample selection during the annotation process. More broadly, our work suggests that improvements in annotation interfaces can elicit responses which are more efficient in terms of the obtained performance versus the invested annotation time.
We would like to thank Christopher Clark, Terra Blevins, and the anonymous reviewers for their helpful feedback, and Aaron Jaech, Mason Kamb, Madian Khabsa, Kaushal Mangipudi, Nayeon Lee, and Anisha Uppugonduri for their participation in our timing experiments.
8324
Pradeep Dasigi, Nelson F. Liu, Ana Marasovic, Noah A. Smith, and Matt Gardner. 2019. Quoref: A reading comprehension dataset with questions requiring coreferential reasoning. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP).
Matt Gardner, Joel Grus, Mark Neumann, Oyvind Tafjord, Pradeep Dasigi, Nelson F. Liu, Matthew E. Peters, Michael Schmitz, and Luke S. Zettlemoyer. 2018. Allennlp: A deep semantic natural language processing platform. CoRR, abs/1803.07640.
Dan Garrette and Jason Baldridge. 2013. Learning a part-of-speech tagger from two hours of annotation. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 138-147, Atlanta, Georgia. Association for Computational Linguistics.
Caroline Gasperin. 2009. Active learning for anaphora resolution. In Proceedings of the NAACL HLT 2009 Workshop on Active Learning for Natural Language Processing, HLT '09, pages 1-8, Stroudsburg, PA, USA. Association for Computational Linguistics.
Matthew Honnibal and Ines Montani. 2017. spacy 2: Natural language understanding with bloom embeddings, convolutional neural networks and incremental parsing. To appear.
Mahnoosh Kholghi, Laurianne Sitbon, Guido Zuccon, and Anthony Nguyen. 2015. Active learning: a step towards automating medical concept extraction. Journal of the American Medical Informatics Association, 23(2):289-296.
Florian Laws, Florian Heimerl, and Hinrich Schütze. 2012. Active learning for coreference resolution. In Proceedings of the 2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 508-512, Montréal, Canada. Association for Computational Linguistics.
Kenton Lee, Luheng He, Mike Lewis, and Luke S. Zettlemoyer. 2017. End-to-end neural coreference resolution. ArXiv, abs/1707.07045.
Sameer Pradhan, Alessandro Moschitti, Nianwen Xue, Olga Uryupina, and Yuchen Zhang. 2012. Conll-2012 shared task: Modeling multilingual unrestricted coreference in onthonotes. In Joint Conference on EMNLP and CoNLL-Shared Task, pages 1-40. Association for Computational Linguistics.
Mrinmaya Sachan, Eduard Hovy, and Eric P. Xing. 2015. An active learning approach to coreference resolution. In Proceedings of the 24th International Conference on Artificial Intelligence, IJ-CAI'15, pages 1312-1318. AAAI Press.
Burr Settles. 2010. Active learning literature survey. University of Wisconsin, Madison, 52(55-66):11.
Gabriel Stanovsky, Noah A. Smith, and Luke Zettlemoyer. 2019. Evaluating gender bias in machine translation. In ACL, page (to appear), Florence, Italy. Association for Computational Linguistics.
A. R. Syed, A. Rosenberg, and E. Kislal. 2016. Supervised and unsupervised active learning for automatic speech recognition of low-resource languages. In 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 5320-5324.
A. R. Syed, A. Rosenberg, and M. Mandel. 2017. Active learning for low-resource speech recognition: Impact of selection size and language modeling data. In 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 5315-5319.
Shanheng Zhao and Hwee Tou Ng. 2014. Domain adaptation with active learning for coreference resolution. In Proceedings of the 5th International Workshop on Health Text Mining and Information Analysis (Louhi), pages 21-29, Gothenburg, Sweden. Association for Computational Linguistics.
8325
A.1 Timing Experiment Details and Computations.
In order to properly calibrate the results from discrete and pairwise querying, we conducted experiments (eight 30-minute sessions) to time how long annotators take to answer discrete and pairwise questions. See Figure 5 for the interface we designed for our experiments.