SlowGuess/ACL_2020_OCR huggingface dataset:
The questions we ask for the experiment are all sampled from real queries from full runs of our active learning simulations. To obtain representative times, we sampled a diverse selection of active learning questions—at various stages of active learning (first iteration before retraining vs. after retraining  $n$  times) and various numbers of annotation per document (20 vs. 200). For each document, we randomly selected between 1-5 questions (of the total 20 or 200) to ask the annotator. Full details on how we sampled our queries can be found in Table 1. Note that we divided our samples into two datasets. We ran four 30-minute sessions with Dataset A before Dataset B and four 30-minute sessions with Dataset B before Dataset A—for a total of eight 30-minute sessions across 7 annotators (1 annotator completed a 1-hour session).
Since pairwise annotation is the same as answering only the initial question under the discrete setting, we run a single discrete experiment for each annotation session and use the time taken to answer an initial question as a proxy for pairwise annotation time. Our results show that answering the initial question took an average of  $15.96s$  whereas answering the follow-up question took  $15.57s$ . Thus, we derive the following formulas to compute the time it takes for pairwise and discrete annotation:
$$
t = 1 5. 9 6 p \tag {7}
$$
$$
t = 1 5. 9 6 d _ {c} + 1 5. 5 7 d _ {n c} \tag {8}
$$
where  $p = \#$  of pairwise instances.  $d_{c}, d_{nc} = \#$  of discrete instances for which the initial pair was "coreferent"  $(d_{c})$  and "not coreferent"  $(d_{nc})$ , respectively. We also compute the number of pairwise examples  $p$  we can query in the same time it takes to query  $d_{c} + d_{nc}$  discrete examples:
$$
1 5. 9 6 p = 1 5. 9 6 d _ {c} + 1 5. 5 7 d _ {n c}
$$
$$
p = d _ {c} + 0. 9 7 6 d _ {n c} \tag {9}
$$
Moreover, we additionally conduct a single 30-minute experiment to determine how long it takes
to answer only discrete questions (without the initial pairwise step). We find that it takes  $28.01s$  per question under the only-discrete setting. This is longer than the time it takes to answer a pairwise question, thus confirming that having an initial pairwise question indeed saves time if the pair is coreferent. Moreover, this also shows that answering the initial pairwise question significantly helps with answering the follow-up discrete question.
A.2 Additional Model Adaptations
Adapting Link Relations for our Model We use must-link and cannot-link relations between mentions to guide our active learning selector. We revise probabilities and model outputs (from which the model computes uncertainty scores for entropy, QBC, and LCC/MCU) in accordance to the following rules:
1. Clustered entropy. For every  $CL(a, b)$  relationship, we set  $P(\mathrm{ant}(a) = b) = 0$  and re-normalize probabilities of all other candidate antecedents. This decreases the probability that the active learning selector chooses  $a$ . Moreover, for every  $ML(a, b)$  relationship, we set  $P(\mathrm{ant}(a) = b) = 1$  and  $P(\mathrm{ant}(a) = c) = 0$  for all  $c \neq b$ . If there are multiple  $ML$  relationships involving  $a$ , we choose only one of  $a$ 's antecedent to set to 1 (to maintain the integrity of the probability distribution). This guarantees that the active learning selector will never select  $a$ , as any ML link out of  $a$  means we have already queried for  $a$ .
2. Clustered query-by-committee. To ensure we do not choose a mention we have already queried for, after each user judgment, for every  $ML(a,b)$  relation, we set  $V(A(a) = b) = \mathcal{M}$ , and  $V(A(a) = c) = 0$  for all other  $c \neq b$ . Moreover, for every  $CL(a,b)$  relation, we set  $V(A(a) = b) = 0$ , which decreases the vote entropy of  $a$ , making it less likely for the selector to choose  $a$ .
3. LCC/MCU. We revise the probabilities in the same way as in clustered entropy and add the constraint that, when choosing MCU spans  $j$ , we disregard those that already have probability 1 (signifying that we have already queried for them).
Incremental Closures Algorithm We introduce an algorithm to compute link closures incrementally. Instead of re-computing and re-adding the
8326
Figure 5: Timing experiments interface. Top: The initial pairwise question. Bottom: The user is presented with the discrete question when they click "No". They are asked to select the appropriate tokens in the text representing the first occurrence of the yellow entity in the text.
entire set of closures (based on a set of all prior human annotations that we keep track of) each time we query for a new mention, we add the minimum set of necessary links. See Algorithm 1.
To determine how much time our incremental closure algorithm saves over recomputing closures from scratch, we simulated annotations on a single document with 1600 mentions, and recorded how long it took to re-compute the closure after each annotation. Our experiments show that recomputing from scratch takes progressively longer as more labels get added: at 1600 labels, our incremental algorithm is 556 times faster than recomputing from scratch (1630ms vs. 2.93ms).
Figure 6 plots the runtime of our incremental closure algorithm ("incremental closure") against the run-time of recomputing closures from scratch ("closure") using Equations 3 and 4. In the latter case, we keep track of the set of user-added edges which we update after each annotation, and re-compute the closures from that set.
A.3 Additional Analysis
Computing the time to fully-label a document under discrete and pairwise annotation. First, we compute the maximum number of pairwise questions we can ask. We consider the setup of Lee et al. (2017)'s model. This model considers only spans with highest mention scores (the "top spans"), and only considers at most  $K$  antecedents per top span. Thus, for a document with  $m$  top spans, we can ask up to
$$
\frac {K (K - 1)}{2} + (m - K) K \tag {10}
$$
pairwise questions. The first factor  $\frac{K(K - 1)}{2}$  comes from considering the first  $K$  spans in the document. For each of these spans  $i = 1\cdots K$ , we can ask about the first  $i - 1$  spans. The second factor  $(m - K)K$  comes from considering the spans after the  $K$ -th span. For each of these  $m - K$  spans in the document, we can only consider up to  $K$  antecedents. Using statistics for the average document  $(m = 201)$  and the standard hyper-parameter settings  $(K = 100)$ , we plug into Equation 10 to
Figure 6: Under each closure algorithm, the time to compute the closure after the next annotation is added, as # of existing annotations increases.
get 15,050 overall pairwise questions needed to fully label a document (in worst-case). Meanwhile, the maximum number of discrete questions we can ask is only 201 (i.e., asking for the antecedent of every mention). Using timing Equations 7 and 8, we compute that it takes at most  $6337.53s$  to answer 201 discrete questions in the worst-case scenario, and  $240198s$  to answer 15050 pairwise questions. Thus, in the worst-case scenario for both discrete and pairwise selection, discrete selection will take only  $2.64\%$  of the time it takes pairwise selection to fully label a document.
Quantifying "Information Gain" from Discrete and Pairwise Annotation. Let  $\overline{D_U}$  be the set of training documents we are annotating for in a given round of active learning. To better quantify how much information discrete and pairwise annotation can supply in same amount of time, we define  $\Delta F1$  as the change in the  $F1$  score on  $\overline{D_U}$ , before and after model predictions are supplemented with user annotation.
Figure 7 shows average  $\Delta F1$  as annotation time increases for discrete and pairwise annotation. Across the 10 annotation times we recorded, discrete annotation results in an average  $\Delta F1$  that more than twice that of pairwise, in the same annotation time.
Model. We preserve the hyperparameters from the AllenNLP implementation of Lee et al. (2017)'s model. The AllenNLP implementation mostly maintains the original hyperparameters, except it sets the maximum number of antecedents considered to  $K = 100$ , and excludes speaker features
Figure 7: Comparing F1 score improvement on  $\overline{D_U}$  for discrete vs. pairwise annotation.
and variational dropout, due to machine memory limitations.
Training. We use a 700/2102 fully-labelled/unlabelled initial split of the training data, and actively label 280 documents at a time. We train to convergence each round. Before all documents have been added, we train up to 20 epochs with a patience of 2 before we add more training documents. After all documents have been added, we retrain from scratch and use the original training hyperparameters from Lee et al. (2017).
**Operators.** For query-by-committee, we use a committee of  $\mathcal{M} = 3$  models. We were not able to experiment with more due to memory constraints.
For LCC/MCU, given  $L$  annotations per document, we allocate  $n$  annotations to least-coreferent clustered mentions and the remaining  $m$  to most-coreferent unclustered mentions. We use  $n = \min(L/2, \text{number of clustered spans})$ , and  $m = \min(L - n, \text{number of un-clustered spans})$ .
A.5 Active Learning Training Setup Full Details
In our active learning setup, we begin by training our model on a 700-document subset of the full training set. We discard the labels of the remaining 2102 documents. In each round of active learning, we choose 280 unlabelled documents, and query up to  $Q$  annotations per document. We then add these documents to the labelled set and continue training our model on this set (now with new documents). After all documents have been labelled, we retrain our model on the full document set from scratch, resetting all model and trainer parameters.
8328
In Algorithm 2, we show our main training loop for active learning using discrete selection. This is the training loop we use for our clustered entropy and LCC/MCU selectors, and our partially-labelled random baseline. In Algorithm 3, we modify that loop for the clustered query-by-committee selector.
In Algorithm 1, we show our incremental closures algorithm, which builds up the transitive closure incrementally by adding only the minimum number of necessary links to maintain the closure each time a new link is added.
8329
Algorithm 1: Incremental Link Closures Algorithm
Let  $(a,b) =$  link pair being added,  $A = a$  's old cluster before the pair is added,  $B = b$  's old cluster before the pair is added,  $\overline{A} =$  set of element  $a$  has a CL relationship to before the pair is added,  $\overline{B} =$  set of elements  $b$  has a CL relationship to before the pair is added.
1. If pair  $(a, b)$  was added to must-link, both must-link and cannot-link needs to be updated.
First, resolve the MLs by adding a ML relationship between every element in  $A$  and every element in  $B$ :
$$
\forall a ^ {\prime}, b ^ {\prime} \quad (M L (a, a ^ {\prime}) \wedge M L (b, b ^ {\prime})) \rightarrow (M L (a, b ^ {\prime}) \wedge M L (a ^ {\prime}, b) \wedge M L (a ^ {\prime}, b ^ {\prime}))
$$
Next, resolve the CLs by adding a CL relationship between every element of  $A$  and  $\overline{B}$ , and every element of  $B$  and  $\overline{A}$ :
$$
\forall a ^ {\prime}, \hat {b} \quad (M L (a, a ^ {\prime}) \wedge C L (\hat {b}, \hat {b})) \rightarrow (C L (a, \hat {b}) \wedge C L (a ^ {\prime}, \hat {b}))
$$
$$
\forall b ^ {\prime}, \hat {a} \quad (M L (b, b ^ {\prime}) \wedge C L (a, \hat {a})) \rightarrow (C L (b, \hat {a}) \wedge C L (b ^ {\prime}, \hat {a}))
$$
2. If pair  $(a, b)$  was added to cannot-link, only cannot-link needs to be updated. Add a CL relationship between every element of  $A$  and every element of  $B$ :
$$
\forall a ^ {\prime}, b ^ {\prime} \quad (M L (a, a ^ {\prime}) \wedge M L (b, b ^ {\prime})) \rightarrow (C L (a, b ^ {\prime}) \wedge C L (a ^ {\prime}, b) \wedge C L (a ^ {\prime}, b ^ {\prime}))
$$
Algorithm 2: Training loop for active learning
$D_F = \{\text{fully-labelled docs}\}$ ,  $D_U = \{\text{unlabelled docs}\}$ ,  $D_A = \{\text{docs labelled through active learning}\}$ ,  $M = \text{model}$ ,  $ML = \text{must-link pairs}$ ,  $CL = \text{cannot-link pairs}$ ;
Init:  $D_F = \{\text{first 700 docs}\}$ ,  $D_U = \{\text{remaining docs}\}$ ,  $D_A = \emptyset$ ,  $ML = CL = \emptyset$ ;
while  $D_U$  is not empty do
Algorithm 3: Training loop for active learning with QBC selector (Differences from Algorithm 2 are highlighted)
8331