SlowGuess/ACL_2020_OCR huggingface dataset:
Kianté Brantley
kdrbrant@cs.umd.edu
Amr Sharaf
amr@cs.umd.edu
Hal Daumé III
me@hal3.name
Imitation learning algorithms provide state-of-the-art results on many structured prediction tasks by learning near-optimal search policies. Such algorithms assume training-time access to an expert that can provide the optimal action at any queried state; unfortunately, the number of such queries is often prohibitive, frequently rendering these approaches impractical. To combat this query complexity, we consider an active learning setting in which the learning algorithm has additional access to a much cheaper noisy heuristic that provides noisy guidance. Our algorithm, LEAQI, learns a difference classifier that predicts when the expert is likely to disagree with the heuristic, and queries the expert only when necessary. We apply LEAQI to three sequence labeling tasks, demonstrating significantly fewer queries to the expert and comparable (or better) accuracies over a passive approach.
Structured prediction methods learn models to map inputs to complex outputs with internal dependencies, typically requiring a substantial amount of expert-labeled data. To minimize annotation cost, we focus on a setting in which an expert provides labels for pieces of the input, rather than the complete input (e.g., labeling at the level of words, not sentences). A natural starting point for this is imitation learning-based "learning to search" approaches to structured prediction (Daumé et al., 2009; Ross et al., 2011; Bengio et al., 2015; Leblond et al., 2018). In imitation learning, training proceeds by incrementally producing structured outputs on piece at a time and, at every step, asking the expert "what would you do here?" and learning to mimic that choice. This interactive model comes at a substantial cost: the expert demonstrator must be continuously available and must be able to answer a potentially large number of queries.
We reduce this annotation cost by only asking an expert for labels that are truly needed; our algorithm, Learning to Query for Imitation (LEAQI, /'li:tfi:/) achieves this by capitalizing on two factors. First, as is typical in active learning (see §2), LEAQI only asks the expert for a label when it is uncertain. Second, LEAQI assumes access to a noisy heuristic labeling function (for instance, a rule-based model, dictionary, or inexpert annotator) that can provide low-quality labels. LEAQI operates by always asking this heuristic for a label, and only querying the expert when it thinks the expert is likely to disagree with this label. It trains, simultaneously, a difference classifier (Zhang and Chaudhuri, 2015) that predicts disagreements between the expert and the heuristic (see Figure 1).
The challenge in learning the difference classifier is that it must learn based on one-sided feedback: if it predicts that the expert is likely to agree with the heuristic, the expert is not queried and the classifier cannot learn that it was wrong. We address this one-sided feedback problem using the Apple Tasting framework (Helmbold et al., 2000), in which errors (in predicting which apples are tasty) are only observed when a query is made (an apple is tasted). Learning in this way particularly important in the general case where the heuristic is likely not just to have high variance with respect to the expert, but is also statistically biased.
Experimentally (§4.5), we consider three structured prediction settings, each using a different type of heuristic feedback. We apply LEAQI to: English named entity recognition where the heuristic is a rule-based recognizer using gazetteers (Khashabi et al., 2018); English scientific keyphrase extraction, where the heuristic is an unsupervised method (Florescu and Caragea, 2017); and Greek part-of-speech tagging, where the heuristic is a small dictio
$^{1}$ Code is available at: https://github.com/xkianteb/leaqi
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2093-2105 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
Figure 1: A named entity recognition example (from the Wikipedia page for Clarence Ellis).  $\pmb{x}$  is the input sentence and  $\pmb{y}$  is the (unobserved) ground truth. The predictor  $\pi$  operates left-to-right and, in this example, is currently at state  $s_{10}$  to tag the 10th word; the state  $s_{10}$  (highlighted in purple) combines  $\pmb{x}$  with  $\hat{\pmb{y}}_{1:9}$ . The heuristic makes two errors at  $t = 4$  and  $t = 6$ . The heuristic label at  $t = 10$  is  $y_{10}^{h} = \mathsf{ORG}$ . Under Hamming loss, the cost at  $t = 10$  is minimized for  $a = \mathsf{ORG}$ , which is therefore the expert action (if it were queried). The label that would be provided for  $s_{10}$  to the difference classifier is 0 because the two policies agree.
nary compiled from the training data (Zesch et al., 2008; Haghighi and Klein, 2006). In all three settings, the expert is a simulated human annotator. We train LEAQI on all three tasks using fixed BERT (Devlin et al., 2019) features, training only the final layer (because we are in the regime of small labeled data). The goal in all three settings is to minimize the number of words the expert annotator must label. In all settings, we're able to establish the efficacy of LEAQI, showing that it can indeed provide significant label savings over using the expert alone and over several baselines and ablations that establish the importance of both the difference classifier and the Apple Tasting paradigm.
We review first the use of imitation learning for structured prediction, then online active learning, and finally applications of active learning to structured prediction and imitation learning problems.
2.1 Learning to Search
The learning to search approach to structured prediction casts the joint prediction problem of producing a complex output as a sequence of smaller classification problems (Ratnaparkhi, 1996; Collins and Roark, 2004; Daumé et al., 2009). For instance, in the named entity recognition example from Figure 1, an input sentence  $x$  is labeled one word at a time, left-to-right. At the depicted state  $(s_{10})$ , the model has labeled the first nine words and must next label the tenth word. Learning to search approaches assume access to an oracle policy  $\pi^{\star}$ , which provides the optimal label at every position.
In (interactive) imitation learning, we aim to imitate the behavior of the expert policy,  $\pi^{\star}$ , which provides the true labels. The learning to search view allows us to cast structured prediction as a (degenerate) imitation learning task, where states
are (input, prefix) pairs, actions are operations on the output, and the horizon  $T$  is the length of the sequence. States are denoted  $s \in S$ , actions are denoted  $a \in [K]$ , where  $[K] = \{1, \ldots, K\}$ , and the policy class is denoted  $\Pi \subseteq [K]^S$ . The goal in learning is to find a policy  $\pi \in \Pi$  with small loss on the distribution of states that it, itself, visits.
A popular imitation learning algorithm, DAgger (Ross et al., 2011), is summarized in Alg 1. In each iteration, DAgger executes a mixture policy and, at each visited state, queries the expert's action. This produces a classification example, where the input is the state and the label is the expert's action. At the end of each iteration, the learned policy is updated by training it on the accumulation of all generated data so far. DAgger is effective in practice and enjoys appealing theoretical properties; for instance, if the number of iterations  $N$  is  $\tilde{O}(T^2\log(1/\delta))$  then with probability at least  $1 - \delta$ , the generalization error of the learned policy is  $O(1/T)$  (Ross et al., 2011, Theorem 4.2).
Active learning has been considered since at least the 1980s often under the name "selective sam
pling" (Rendell, 1986; Atlas et al., 1990). In ag- nostic online active learning for classification, a learner operates in rounds (e.g. Balcan et al., 2006; Beygelzimer et al., 2009, 2010). At each round, the learning algorithm is presented an example  $x$  and must predict a label; the learner must decide whether to query the true label. An effective margin-based approach for online active learning is provided by Cesa-Bianchi et al. (2006) for linear models. Their algorithm defines a sampling probability  $\rho = b / (b + z)$ , where  $z$  is the margin on the current example, and  $b > 0$  is a hyperparameter that controls the aggressiveness of sampling. With probability  $\rho$ , the algorithm requests the label and performs a perceptron-style update.
Our approach is inspired by Zhang and Chaudhuri's (2015) setting, where two labelers are available: a free weak labeler and an expensive strong labeler. Their algorithm minimizes queries to the strong labeler, by learning a difference classifier that predicts, for each example, whether the weak and strong labelers are likely to disagree. Their algorithm trains this difference classifier using an example-weighting strategy to ensure that its Type II error is kept small, establishing statistical consistency, and bounding its sample complexity.
This type of learning from one-sided feedback falls in the general framework of partial-monitoring games, a framework for sequential decision making with imperfect feedback. Apple Tasting is a type of partial-monitoring game (Littlestone and Warmuth, 1989), where, at each round, a learner is presented with an example  $\pmb{x}$  and must predict a label  $\hat{y} \in \{-1, +1\}$ . After this prediction, the true label is revealed only if the learner predicts  $+1$ . This framework has been applied in several settings, such as spam filtering and document classification with minority class distributions (Sculley, 2007). Sculley (2007) also conducts a through comparison of two methods that can be used to address the one-side feedback problem: label-efficient online learning (Cesa-Bianchi et al., 2006) and margin-based learning (Vapnik, 1982).
2.3 Active Imitation & Structured Prediction
In the context of structured prediction for natural language processing, active learning has been considered both for requesting full structured outputs (e.g. Thompson et al., 1999; Culotta and McCallum, 2005; Hachey et al., 2005) and for requesting only pieces of outputs (e.g. Ringger et al.,
2007; Bloodgood and Callison-Burch, 2010). For sequence labeling tasks, Haertel et al. (2008) found that labeling effort depends both on the number of words labeled (which we model), plus a fixed cost for reading (which we do not).
In the context of imitation learning, active approaches have also been considered for at least three decades, often called "learning with an external critic" and "learning by watching" (Whitehead, 1991). More recently, Judah et al. (2012) describe RAIL, an active learning-for-imitation-learning algorithm akin to our ACTIVATEDAGGER baseline, but which in principle would operate with any underlying i.i.d. active learning algorithm (not just our specific choice of uncertainty sampling).
3 Our Approach: LEAQI
Our goal is to learn a structured prediction model with minimal human expert supervision, effectively by combining human annotation with a noisy heuristic. We present LEAQI to achieve this. As a concrete example, return to Figure 1: at  $s_{10}$ ,  $\pi$  must predict the label of the tenth word. If  $\pi$  is confident in its own prediction, LEAQI can avoid any query, similar to traditional active learning. If  $\pi$  is not confident, then LEAQI considers the label suggested by a noisy heuristic (here: ORG). LEAQI predicts whether the true expert label is likely to disagree with the noisy heuristic. Here, it predicts no disagreement and avoids querying the expert.
3.1 Learning to Query for Imitation
Our algorithm, LEAQI, is specified in Alg 2. As input, LEAQI takes a policy class  $\Pi$ , a hypothesis class  $\mathcal{H}$  for the difference classifier (assumed to be symmetric and to contain the "constant one" function), a number of episodes  $N$ , an expert policy  $\pi^{\star}$ , a heuristic policy  $\pi^{\mathrm{h}}$ , and a confidence parameter  $b > 0$ . The general structure of LEAQI follows that of DAGger, but with three key differences:
(a) roll-in (line 7) is according to the learned policy (not mixed with the expert, as that would require additional expert queries),
(b) actions are queried only if the current policy is uncertain at  $s$  (line 12), and
(c) the expert  $\pi^{\star}$  is only queried if it is predicted to disagree with the heuristic  $\pi^{\mathrm{h}}$  at  $s$  by the difference classifier, or if apple tasting method switches the difference classifier label (line 15; see §3.2).
In particular, at each state visited by  $\pi_{i}$ , LEAQI estimates  $z$ , the certainty of  $\pi_{i}$ 's prediction at that state (see §3.3). A sampling probability  $\rho$  is set to  $b / (b + z)$  where  $z$  is the certainty, and so if the model is very uncertain then  $\rho$  tends to zero, following (Cesa-Bianchi et al., 2006). With probability  $\rho$ , LEAQI will collect some label.
When a label is collected (line 12), the difference classifier  $h_i$  is queried on state  $s$  to predict if  $\pi^{\star}$  and  $\pi^{\mathrm{h}}$  are likely to disagree on the correct action. (Recall that  $h_1$  always predicts disagreement per line 4.) The difference classifier's prediction,  $\hat{d}_i$ , is passed to an apple tasting method in line 15. Intuitively, most apple tasting procedures (including the one we use, STAP; see §3.2) return  $\hat{d}_i$ , unless the difference classifier is making many Type II errors, in which case it may return  $\neg \hat{d}_i$ .
A target action is set to  $\pi^{\mathrm{h}}(s)$  if the apple taste
ing algorithm returns "agree" (line 17), and the expert  $\pi^{\star}$  is only queried if disagreement is predicted (line 20). The state and target action (either heuristic or expert) are then added to the training data. Finally, if the expert was queried, then a new item is added to the difference dataset, consisting of the state, the heuristic action on that state, the difference classifier's prediction, and the ground truth for the difference classifier whose input is  $s$  and whose label is whether the expert and heuristic actually disagree. Finally,  $\pi_{i + 1}$  is trained on the accumulated action data, and  $h_{i + 1}$  is trained on the difference dataset (details in §3.3).
There are several things to note about LEAQI:
If the current policy is already very certain, a expert annotator is never queried.
If a label is queried, the expert is queried only if the difference classifier predicts disagreement with the heuristic, or the apple tasting procedure flips the difference classifier prediction.
Due to apple tasting, most errors the difference classifier makes will cause it to query the expert unnecessarily; this is the "safe" type of error (increasing sample complexity but not harming accuracy), versus a Type II error (which leads to biased labels).
$\diamond$  The difference classifier is only trained on states where the policy is uncertain, which is exactly the distribution on which it is run.
3.2 Apple Tasting for One-Sided Learning
The difference classifier  $h\in \mathcal{H}$  must be trained (line 27) based on one-sided feedback (it only ob
serves errors when it predicts "disagree") to minimize Type II errors (it should only very rarely predict "agree" when the truth is "disagree"). This helps keep the labeled data for the learned policies unbiased. The main challenge here is that the feedback to the difference classifier is one-sided: that is, if it predicts "disagree" then it gets to see the truth, but if it predicts "agree" it never finds out if it was wrong. We use one of (Helmbold et al., 2000)'s algorithms, STAP (see Alg 3), which works by random sampling from apples that are predicted to not be tasted and tasting them anyway (line 12). Formally, STAP tastes apples that are predicted to be bad with probability  $\sqrt{(m + 1) / t}$ , where  $m$  is the number of mistakes, and  $t$  is the number of apples tasted so far.
We adapt Apple Tasting algorithm STAP to our setting for controlling the number of Type II errors made by the difference classifier as follows. First, because some heuristic actions are much more common than others, we run a separate apple tasting scheme per heuristic action (in the sense that we count the number of error on this heuristic action rather than globally). Second, when there is significant action imbalance<sup>2</sup> we find it necessary to skew the distribution from STAP more in favor of querying. We achieve this by sampling from a Beta distribution (generalizing the uniform), whose mean is shifted toward zero for more frequent heuristic actions. This increases the chance that Apple Tasting will have on finding bad apples error for each action (thereby keeping the false positive rate low for predicting disagreement).
3.3 Measuring Policy Certainty
In step 11, LEAQI must estimate the certainty of  $\pi_{i}$  on  $s$ . Following Cesa-Bianchi et al. (2006), we implement this using a margin-based criteria. To achieve this, we consider  $\pi$  as a function that maps actions to scores and then chooses the action with largest score. The certainty measure is then the difference in scores between the highest and second highest scoring actions:
$$
\operatorname {c e r t a i n t y} (\pi , s) = \max  _ {a} \pi (s, a) - \max  _ {a ^ {\prime} \neq a} \pi (s, a ^ {\prime})
$$
3.4 Analysis
Theoretically, the main result for LEAQI is an interpretation of the main DAgger result(s). Formally, let  $d_{\pi}$  denote the distribution of states visited by  $\pi$ ,  $C(s,a) \in [0,1]$  be the immediate cost of performing action  $a$  in state  $s$ ,  $C_{\pi}(s) = \mathbb{E}_{a \sim \pi(s)} C(s,a)$ , and the total expected cost of  $\pi$  to be  $J(\pi) = T\mathbb{E}_{s \sim d_{\pi}} C_{\pi}(s)$ , where  $T$  is the length of trajectories.  $C$  is not available to a learner in an imitation setting; instead the algorithm observes an expert and minimizes a surrogate loss  $\ell(s,\pi)$  (e.g.,  $\ell$  may be zero/one loss between  $\pi$  and  $\pi^{\star}$ ). We assume  $\ell$  is strongly convex and bounded in  $[0,1]$  over  $\Pi$ .
Given this setup assumptions, let  $\epsilon_{\mathrm{pol - approx}} = \min_{\pi \in \Pi}\frac{1}{N}\sum_{i = 1}^{N}\mathbb{E}_{s\sim d_{\pi_i}}\ell (s,\pi)$  be the true loss of the best policy in hindsight, let  $\epsilon_{\mathrm{dc - approx}} = \min_{h\in \mathcal{H}}\frac{1}{N}\sum_{i = 1}^{N}\mathbb{E}_{s\sim d_{\pi_i}}err(s,h,\pi^{\star}(s)\neq \pi^{\mathrm{h}}(s))$  be the true error of the best difference classifier in hindsight, and assuming that the regret of the policy learner is bounded by  $reg_{\mathrm{pol}}(N)$  after  $N$  steps, Ross et al. (2011) shows the following:
Theorem 1 (Thm 4.3 of Ross et al. (2011)). After  $N$  episodes each of length  $T$ , under the assumptions above, with probability at least  $1 - \delta$  there exists a policy  $\pi \in \pi_{1:N}$  such that:
$$
\begin{array}{l} \mathbb {E} _ {s \sim d _ {\pi}} \ell (s, \pi) \leq \\ \epsilon_ {\mathrm {p o l - a p p r o x}} + r e g _ {\mathrm {p o l}} (N) + \sqrt {(2 / N) \log (1 / \delta)} \\ \end{array}
$$
This holds regardless of how  $\pi_{1:N}$  are trained (line 26). The question of how well LEAQI performs becomes a question of how well the combination of uncertainty-based sampling and the difference classifier learn. So long as those do a good job on their individual classification tasks, DAgger guarantees that the policy will do a good job. This is formalized below, where  $Q^{\star}(s,a)$  is the best possible cumulative cost (measured by  $C$ ) starting in state  $s$  and taking action  $a$ :
Theorem 2 (Theorem 2.2 of Ross et al. (2011)). Let  $u$  be such that  $Q^{\star}(s,a) - Q^{\star}(s,\pi^{\star}(s))\leq u$  for all  $a$  and all  $s$  with  $d_{\pi}(s) > 0$ ; then for some  $\pi \in \pi_{1:N}$ , as  $N\to \infty$ :
$$
J (\pi) \leq J (\pi^ {\star}) + u T \epsilon_ {\mathrm {p o l - a p p r o x}}
$$
Here,  $u$  captures the most long-term impact a single decision can have; for example, for average Hamming loss, it is straightforward to see that  $u = \frac{1}{T}$
2For instance, in named entity recognition, both the heuristic and expert policies label the majority of words as  $\bigcirc$  (not an entity). As a result, when the heuristic says  $\bigcirc$ , it is very likely that the expert will agree. However, if we aim to optimize for something other than accuracy—like F1—it is precisely these disagreements that we need to find.
3Proving a stronger result is challenging: analyzing the sample complexity of an active learning algorithm that uses a difference classifier—even in the non-sequential setting—is quite involved (Zhang and Chaudhuri, 2015).
Task	Named Entity Recognition	Keyphrase Extraction	Part of Speech Tagging
Language	English (en)	English (en)	Modern Greek (el)
Dataset	CoNLL'03 (Tjong
Kim Sang and De Meulder,
2003)	SemEval 2017 Task 10
(Augenstein et al., 2017)	Universal Dependencies
(Nivre, 2018)
# Ex	14,987	2,809	1,662
Avg. Len	14.5	26.3	25.5
# Actions	5	2	17
Metric	Entity F-score	Keyphrase F-score	Per-tag accuracy
Features	English BERT (Devlin et al.,
2019)	SciBERT (Beltagy et al.,
2019)	M-BERT (Devlin et al.,
2019)
Heuristic	String matching against an offline gazeteer of entities from Khashabi et al. (2018)	Output from an unsupervised keyphrase extraction model
Florescu and Caragea
(2017)	Dictionary from Wiktionary, similar to Zesch et al. (2008) and Haghighi and Klein (2006)
Heur Quality	P 88%, R 27%, F 41%	P 20%, R 44%, F 27%	10% coverage, 67% acc
Table 1: An overview of the three tasks considered in experiments.
because any single mistake can increase the number of mistakes by at most 1. For precision, recall and F-score,  $u$  can be as large as one in the (rare) case that a single decision switches from one true positive to no true positives.
The primary research questions we aim to answer experimentally are:
Q1 Does uncertainty-based active learning achieve lower query complexity than passive learning in the learning to search settings?
Q2 Does learning a difference classifier improve query efficiency over active learning alone?
Q3 Does Apple Tasting successfully handle the problem of learning from one-sided feedback?
Q4 Is the approach robust to cases where the noisy heuristic is uncorrelated with the expert?
Q5 Is casting the heuristic as a policy more effective than using its output as features?
To answer these questions, we conduct experiments on three tasks (see Table 1): English named entity recognition, English scientific keyphrase extraction, and low-resource part of speech tagging on Modern Greek (el), selected as a low-resource setting.
4.1 Algorithms and Baselines
In order to address the research questions above, we compare LEAQI to several baselines. The baselines below compare our approach to previous methods:
DAGGER. Passive DAGger (Alg 1)
ACTIVEDAGGER. An active variant of DAgger that asks for labels only when uncertain. (This is equivalent to LEAQI, but with neither the difference classifier nor apple tasting.)
DAGGER+FEAT. DAGGER with the heuristic policy's output appended as an input feature.
ACTIVEDAGGER+FEAT. ACTIVEDAGGER with the heuristic policy as a feature.
The next set of comparisons are explicit ablations:
LEAQI+NOAT LEAQI with no apple tasting.
LEAQI+NOISYHEUR. LEAQI, but where the heuristic returns a label uniformly at random.
The baselines and LEAQI share a linear relationship. DAGGER is the baseline algorithm used by all algorithms described above but it is very query inefficient with respect to an expert annotator. ACTIVEDAGGER introduces active learning to make DAGGER more query efficient; the delta to the previous addresses Q1. LEAQI+NOAT introduces the difference classifier; the delta addresses
Q2. LEAQI adds apple tasting to deal with one-sided learning; the delta addresses Q3. Finally, LEAQI+NOISYHEUR. (vs LEAQI) addresses Q4 and the +FEAT variants address Q5.
4.2 Data and Representation