SlowGuess/NAACL_2021_OCR huggingface dataset:
Tianze Shi*
tianze@cs.cornell.edu
Adrian Benton
abenton10@bloomberg.net
Igor Malioutov
imalioutov@bloomberg.net
Ozan Irsoy
Oirsoy@bloomberg.net
While the predictive performance of modern statistical dependency parsers relies heavily on the availability of expensive expert-annotated treebank data, not all annotations contribute equally to the training of the parsers. In this paper, we attempt to reduce the number of labeled examples needed to train a strong dependency parser using batch active learning (AL). In particular, we investigate whether enforcing diversity in the sampled batches, using determinantal point processes (DPPs), can improve over their diversity-agnostic counterparts. Simulation experiments on an English newswire corpus show that selecting diverse batches with DPPs is superior to strong selection strategies that do not enforce batch diversity, especially during the initial stages of the learning process. Additionally, our diversity-aware strategy is robust under a corpus duplication setting, where diversity-agnostic sampling strategies exhibit significant degradation.
Though critical to parser training, data annotations for dependency parsing are both expensive and time-consuming to obtain. Syntactic analysis requires linguistic expertise and even after extensive training, data annotation can still be burdensome. The Penn Treebank project (Marcus et al., 1993) reports that after two months of training, the annotators average 750 tokens per hour on the bracketing task; the Prague Dependency Treebank (Böhmová et al., 2003) cost over $600,000 and required 5 years to annotate roughly 90,000 sentences (over $5 per sentence). These high annotation costs present a significant challenge to developing accurate dependency parsers for under-resourced languages and domains.
Active learning (AL; Settles, 2009) is a promising technique to reduce the annotation effort required to train a strong dependency parser by intel
ligently selecting samples to annotate such that the return of each annotator hour is as high as possible. Popular selection strategies, such as uncertainty sampling, associate each instance with a quality measure based on the uncertainty or confidence level of the current parser, and higher-quality instances are selected for annotation.
We focus on batch mode AL, since it is generally more efficient for annotators to label in bulk. While early work in AL for parsing (Tang et al., 2002; Hwa, 2000, 2004) cautions against using individually-computed quality measures in the batch setting, more recent work demonstrates empirical success (e.g., Li et al., 2016) without explicitly handling intra-batch diversity. In this paper, we explore whether a diversity-aware approach can improve the state of the art in AL for dependency parsing. Specifically, we consider samples drawn from determinantal point processes (DPPs) as a query strategy to select batches of high-quality, yet dissimilar instances (Kulesza and Taskar, 2012).
In this paper, we (1) propose a diversity-aware batch AL query strategy for dependency parsing compatible with existing selection strategies, (2) empirically study three AL strategies with and without diversity factors, and (3) find that diversity-aware selection strategies are superior to their diversity-agnostic counterparts, especially during the early stages of the learning process, in simulation experiments on an English newswire corpus. This is critical in low-budget AL settings, which we further confirm in a corpus duplication setting. $^{1}$
2 Active Learning for Dependency Parsing
2.1 Dependency Parsing
Dependency parsing (Kübler et al., 2008) aims to find the syntactic dependency structure,  $y$ , given a length- $n$  input sentence  $x = x_{1}, x_{2}, \ldots, x_{n}$ , where
1Our code is publicly available at https://github.com/tzshi/dpp-al-parsing-naacl21.
*Work done during an internship at Bloomberg L.P.
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2616-2626 June 6-11, 2021. ©2021 Association for Computational Linguistics
$y$  is a set of  $n$  arcs over the tokens and the dummy root symbol  $x_0$ , and each arc  $(h,m)\in y$  specifies the head,  $h$ , and modifier word,  $m$ . In this work, we adopt the conceptually-simple edge-factored deep biaffine dependency parser (Dozat and Manning, 2017), which is competitive with the state of the art in terms of accuracy. The parser assigns a locally-normalized attachment probability  $P_{\mathrm{att}}(\mathrm{head}(m) = h\mid x)$  to each attachment candidate pair  $(h,m)$  based on a biaffine scoring function. Refer to Appendix A for architecture details.
We define the score of the candidate parse tree  $s(y\mid x)$  as  $\sum_{(h,m)\in y}\log P_{\mathrm{att}}(\mathrm{head}(m) = h\mid x)$ . The decoder finds the best scoring  $\hat{y}$  among all valid trees  $\mathcal{V}(x)$ :  $\hat{y} = \arg \max_{y\in \mathcal{V}(x)}s(y\mid x)$ .
2.2 Active Learning (AL)
We consider the pool-based batch AL scenario where we assume a large collection of unlabeled instances  $\mathcal{U}$  from which we sample a small subset at a time to annotate after each round to form an expanding labeled training set  $\mathcal{L}$  (Lewis and Gale, 1994). We use the superscript  $i$  to denote the pool of instances  $\mathcal{U}^i$  and  $\mathcal{L}^i$  after the  $i$ -th round.  $\mathcal{L}^0$  is a small set of seed labeled instances to initiate the process. Each iteration starts with training a model  $\mathcal{M}^i$  based on  $\mathcal{L}^i$ . Next, all unlabeled data instances in  $\mathcal{U}^i$  are parsed by  $\mathcal{M}^i$  and we select a batch  $\mathcal{U}'$  to annotate based on some criterion  $\mathcal{U}' = \mathcal{C}(\mathcal{M}^i, \mathcal{U}^i)$ . The resulting labeled subset  $\mathcal{L}'$  is added to  $\mathcal{L}^{i+1} = \mathcal{L}^i \bigcup \mathcal{L}'$  and  $\mathcal{U}^{i+1} = \mathcal{U}^i - \mathcal{U}'$ .
The definition of the selection criterion  $\mathcal{C}$  is critical. A typical strategy associates each unlabeled instance  $\mathcal{U}_i$  with a quality measure  $q_{i}$  based on, for example, the model uncertainty level when parsing  $\mathcal{U}_i$ . A diversity-agnostic criterion sorts all unlabeled instances by their quality measures and takes the top- $k$  as  $\mathcal{U}'$  for a budget  $k$ .
2.3 Quality Measures
We consider three commonly-used quality measures adapted to the task of dependency parsing, including uncertainty sampling, Bayesian active learning, and a representativeness-based strategy.
Average Marginal Probability (AMP) measures parser uncertainty (Li et al., 2016):
$$
\mathsf {A M P} = 1 - \frac {1}{n} \sum_ {(\hat {h}, m) \in \hat {y}} P _ {\operatorname {m a r}} (\operatorname {h e a d} (m) = \hat {h} \mid x),
$$
where  $P_{\mathrm{mar}}$  is the marginal attachment probability
$$
P _ {\operatorname {m a r}} (\operatorname {h e a d} (m) = h \mid x) = \sum_ {(h, m) \in y} P (y \mid x),
$$
and  $P(y\mid x) = \frac{\exp(s(y|x))}{\sum_{y^{\prime}\in\mathcal{Y}(x)}\exp(s(y^{\prime}|x))}$ . The marginal probabilities can be derived efficiently using Kirchhoff's theorem (Tutte, 1984; Koo et al., 2007).
Bayesian Active Learning by Disagreement (BALD) measures the mutual information between the model parameters and the predictions. We adopt the Monte Carlo dropout-based variant (Gal et al., 2017; Siddhant and Lipton, 2018) and measure the disagreement among predictions from a neural model with  $K$  different dropout masks, which has been applied to active learning in NLP. We adapt BALD to dependency parsing by aggregating disagreement at a token level:
$$
\mathsf {B A L D} = 1 - \frac {1}{n} \sum_ {m} \frac {\operatorname {c o u n t} \left(\operatorname {m o d e} \left(h _ {m} ^ {1} , . . . , h _ {m} ^ {K}\right)\right)}{K},
$$
where  $h_m^k$  denotes that  $(h_m^k, m)$  appears in the prediction given by the  $k$ -th model.
Information Density (ID) mitigates the tendency of uncertainty sampling to favor outliers by weighing examples by how representative they are of the entire dataset (Settles and Craven, 2008):
$$
\mathsf {I D} = \mathsf {A M P} \times \left(\frac {1}{| \mathcal {U} |} \sum_ {x ^ {\prime} \in \mathcal {U}} \operatorname {s i m} _ {\cos} (x, x ^ {\prime})\right),
$$
where cosine similarity is computed from the averaged contextualized features (§3.2).
2.4 Learning from Partial Annotations
We follow Li et al. (2016) and select tokens to annotate their heads instead of annotating full sentences. We first pick the most informative sentences and then choose  $p\%$  tokens from them based on the token-level versions of the quality measures (e.g., marginal probability instead of AMP).
3 Selecting Diverse Samples
Near-duplicate examples are common in real-world data (Broder et al., 1997; Manku et al., 2007), but they provide overlapping utility to model training. In the extreme case, with a diversity-agnostic strategy for active learning, identical examples will be selected/excluded at the same time (Hwa, 2004). To address this issue and to best utilize the annotation budget, it is important to consider diversity. We adapt Biryik et al. (2019) to explicitly model diversity using determinantal point processes (DPPs).
3.1 Determinantal Point Processes
A DPP defines a probability distribution over subsets of some ground set of elements (Kulesza, 2012). In AL, the ground set is the unlabeled
2For clarity, here we describe unlabeled parsing. In our experiments, we train labeled dependency parsers, which additionally predict a dependency relation label  $l$  for each arc.
pool  $\mathcal{U}$  and a subset corresponds to a batch of instances  $\mathcal{U}'$  drawn from  $\mathcal{U}$ . DPPs provide an explicit mechanism to ensure high-quality yet diverse sample selection by modeling both the quality measures and the similarities among examples. We adopt the  $L$ -ensemble representation of DPPs using the quality-diversity decomposition (Kulesza and Taskar, 2012) and parameterize the matrix  $L$  as  $L_{ij} = q_i\phi_i\phi_j^T q_j$ , where each  $q_i \in \mathbb{R}$  is the quality measure for  $\mathcal{U}_i$  and each  $\phi_i \in \mathbb{R}^{1 \times d}$  is a  $d$ -dimensional vector representation of  $\mathcal{U}_i$ , which we refer to as  $\mathcal{U}_i$ 's diversity features. The probability of selecting a batch  $B$  is given by  $P(B \subseteq \mathcal{U}) \propto \operatorname*{det}(L_B)$ , where  $\operatorname*{det}(\cdot)$  calculates the determinant and  $L_B$  is the submatrix of  $L$  indexed by elements in  $B$ .
DPPs place high probability on diverse subsets of high-quality items. Intuitively, the determinant of  $L_{B}$  corresponds to the volume spanned by the set of vectors  $\{q_i\phi_i \mid i \in B\}$ , and subsets with larger  $q$  values and orthogonal  $\phi$  vectors span larger volumes than those with smaller  $q$  values or similar  $\phi$  vectors. We follow Kulesza (2012) and adapt their greedy algorithm for finding the approximate mode  $\arg \max_{B} P(B \subseteq \mathcal{U})$ . This algorithm is reproduced in Algorithm E1 in the appendix.
3.2 Diversity Features
We consider two possibilities for the diversity features  $\phi$ . Each feature vector is unit-normalized.
Averaged Contextualized Features are defined as  $\frac{1}{n}\sum_{i}\mathbf{x}_{i}$ , where  $\mathbf{x}_i$  is a contextualized vector of  $x_{i}$  from the feature extractor used by the parser. By this definition, we consider the instances to be similar to each other when the neural feature extractor returns similar features such that the parser is likely to predict similar structures for these instances.
Predicted Subgraph Counts explicitly represent the predicted tree structure. To balance richness and sparsity, we count the labeled but unlexicalized subgraph formed by the grandparent, the parent and the token itself. Specifically, for each token  $m$ , we can extract a subgraph denoted by  $(r_1, r_2)$ , assuming the predicted dependency relation between its grandparent  $g$  and its parent  $h$  is  $r_1$ , and the relation between  $h$  and  $m$  is  $r_2$ . The parse tree for a length-  $n$  sentence contains  $n$  such subgraphs. We apply tfidf weighting to discount
Batch	5	10
Strategy	w/o DPP	w/ DPP	w/o DPP	w/ DPP
Random	85.68±.26	86.61±.28	87.84±.26	88.55±.23
AMP	85.98±.22	86.77±.43	88.80±.18	89.23±.29
BALD	86.24±.40	86.86±.31	88.66±.36	89.03±.10
ID	86.68±.26	86.56±.24	88.96±.20	89.06±.16
Table 1: LAS after 5 and 10 rounds of annotation for strategies with and without modeling diversity through DPP.
the influence from frequent subgraphs.
Dataset We use the Revised English News Text Treebank $^{4}$  (Bies et al., 2015) converted to Universal Dependencies 2.0 using the conversion tool included in Stanford Parser (Manning et al., 2014) version 4.0.0. We use sections 02-21 for training, 22 for development and 23 for test.
Setting We perform experiments by simulating the annotation process using treebank data. We sample 128 sentences uniformly for the initial labeled pool and each following round selects 500 tokens for partial annotation. We run each setting five times using different random initializations and report the means and standard deviations of the labeled attachment scores (LAS). Appendix B has unlabeled attachment score (UAS) results.
Baselines While we construct our own baselines for self-contained comparisons, the diversity-agnostic AMP (w/o DPP) largely replicates the state-of-the-art selection strategy of Li et al. (2016).
Implementation We finetune a pretrained multilingual XLM-RoBERTa base model (Conneau et al., 2020) as our feature extractor. See Appendix E for implementation details.
Main Results Table 1 compares LAS after 5 and 10 rounds of annotation. Our dependency parser reaches 95.64 UAS and 94.06 LAS, when trained with the full dataset (more than one million tokens). Training data collected from 30 annotation rounds  $(\approx 17,500$  tokens) correspond to roughly  $2\%$  of the full dataset, but already support an LAS of up to 92 through AL. We find that diversity-aware strategies generally improve over their diversity-agnostic counterparts. Even for a random selection strategy, ensuring diversity with a DPP is superior
4https://catalog.ldc.upenn.edu/LDC2015T13
5To construct the averaged contextualized features, we also use the fine-tuned feature extractor. In our preliminary experiments, we have tried freezing the feature extractors, but this variant did not perform as well.
3Although certain applications of DPPs may learn  $q$  and  $\phi$  representations from supervision, we define  $q$  and  $\phi$  a priori, since acquiring supervision in AL is, by definition, expensive.
Figure 1: Learning curves for our DPP-based diversity-aware selection strategies, comparing predicted subgraph counts versus averaged contextualized features as diversity features. Both use AMP as their quality measures.
to simple random selection. With AMP and BALD, our diversity-aware strategy sees a larger improvement earlier in the learning process. ID models representativeness of instances, and our diversity-aware strategy adds less utility compared with other quality measures, although we do notice a large improvement after the first annotation round for ID:  $82.40_{\pm .48}$  vs.  $83.36_{\pm .54}$  (w/DPP) - a similar trend to AMP and BALD, but at an earlier stage of AL.
Experiments with Different Diversity Features
Figure 1 compares our two definitions of diversity features, and we find that predicted subgraph counts provide stronger performance than that of averaged contextualized features. We hypothesize this is due to the fact that the subgraph counts represent structures more explicitly, thus they are more useful in maintaining structural diversity in AL.
Intra-Batch Diversity To quantify intra-batch diversity among the set of sentences  $B$  picked by the selection strategies, we adapt the measures used by Chen et al. (2018) and define intra-batch average distance (IBAD) and intra-batch minimal distance (IBMD) as follows:
$$
\begin{array}{l} \operatorname {IBAD} = \max  _ {i, j \in B, i \neq j} (1 - \operatorname {s i m} _ {\cos} (i, j)), \\ \operatorname {IBMD} = \max  _ {i \in B} \min  _ {j \in B, i \neq j} (1 - \operatorname {s i m} _ {\cos} (i, j)). \\ \end{array}
$$
A higher value on these measures indicates better intra-batch diversity. Figure 2 compares diversity-agnostic and diversity-aware sampling strategies using the two different diversity features. We confirm that DPPs indeed promote diverse samples in the selected batches, while intra-batch diversity naturally increases even for the diversity-agnostic strategies. Additionally, we observe that the benefits of DPPs are more prominent when using pre
Figure 2: Intra-batch average distance (IBAD) and intrabatch minimal distance (IBMD) measures comparing diversity-agnostic and diversity-aware AMP-based sample selection strategies. The distances are derived from averaged contextualized features (top) and predicted subgraph counts (bottom). A higher value indicates better intra-batch diversity.
dicted subgraph counts compared with averaged contextualized features. This can help explain the relative success of the former diversity features.
Corpus Duplication Setting In our qualitative analysis (Appendix C), we find that diversity-agnostic selection strategies tend to select near-duplicate sentences. To examine this phenomenon in isolation, we repeat the training corpus twice and observe the effect of diversity-aware strategies. The corpus duplication technique has been previously used to probe semantic models (Schofield et al., 2017). Figure 3 shows learning curves for strategies under the original and corpus duplication settings. As expected, diversity-aware strategies consistently outperform their diversity-agnostic counterparts across both settings, while some diversity-agnostic strategies (e.g., AMP) even underperform uniform random selection in the duplicated setting.
Interpreting the Effectiveness of Diversity-Agnostic Models Figure 4 visualizes the density distributions of the top 200 data instances by AMP over the diversity feature space reduced to two dimensions through t-SNE (van der Maaten and Hinton, 2008). During the initial stage of active learning, data with the highest quality measures are con
Figure 3: Learning curves of different sampling strategies based on AMP (left), BALD (middle) and ID (right), comparing diversity-aware (w/ DPP) and diversity-agnostic variants using the original and duplicated corpus (dup). The  $x$ -axis shows the number of rounds for annotation. Random (dup) curves overlap with those of Random and are omitted for readability.
Figure 4: t-SNE visualization of the distributions of the 200 highest-quality unlabeled sentences over the diversity feature space after the  $1^{\mathrm{st}}$  (left) and the  $10^{\mathrm{th}}$  (right) annotation rounds using AMP without DPPs. Darker region indicates more data points residing in that diversity feature neighborhood. The left figure contains a dense region, while the data in the right figure are spread out in the feature space.
centrated within a small neighborhood. A diversity-agnostic strategy will sample similar examples for annotation. After a few rounds of annotation and model training, the distribution of high quality examples spreads out, and an AMP selection strategy is likely to sample a diverse set of examples without explicitly modeling diversity. Our analysis corroborates previous findings (Thompson et al., 1999) that small annotation batches are effective early in uncertainty sampling, avoiding selecting many near-duplicate examples when intra-batch diversity is low, but a larger batch size is more efficient later in training once intra-batch diversity increases.
Modeling diversity in batch-mode AL (Brinker, 2003) has recently attracted attention in the machine learning community. Kirsch et al. (2019) introduce a Bayesian batch-mode selection strategy by estimating the mutual information between a set of samples and the model parameters. Ash et al. (2020) present a diversity-inducing sampling method using gradient embeddings. Most related to our work, Biryik et al. (2019) first apply DPPs
to batch-mode AL. Building on their approach, we flesh out a DPP treatment for AL for a structured prediction task, dependency parsing. Previously, Shen et al. (2018) consider named entity recognition but they report negative results for a diversity-inducing variant of their sampling method.
Due to the high annotation cost, AL is a popular technique for parsing and parse selection (Osborne and Baldridge, 2004). Recent advances focus on reducing full-sentence annotations to a subset of tokens within a sentence (Sassano and Kurohashi, 2010; Mirroshandel and Nasr, 2011; Majidi and Crane, 2013; Flannery and Mori, 2015; Li et al., 2016). We show that AL for parsing can further benefit from diversity-aware sampling strategies.
DPPs have previously been successfully applied to the tasks of extractive text summarization (Cho et al., 2019a,b) and modeling phoneme inventories (Cotterell and Eisner, 2017). In this work, we show that DPPs also provide a useful framework for understanding and modeling quality and diversity in active learning for NLP tasks.
We show that compared with their diversity-agnostic counterparts, diversity-aware sampling strategies not only lead to higher data efficiency, but are also more robust under corpus duplication settings. Our work invites future research into methods, utility and success conditions for modeling diversity in active learning for NLP tasks.
We thank the anonymous reviewers for their insightful reviews, and Prabhanjan Kambadur, ChenTse Tsai, and Minjie Xu for discussion and comments. Tianze Shi acknowledges support from Bloomberg's Data Science Ph.D. Fellowship.
Jordan T. Ash, Chicheng Zhang, Akshay Krishna-murthy, John Langford, and Alekh Agarwal. 2020. Deep batch active learning by diverse, uncertain gradient lower bounds. In International Conference on Learning Representations, Online. OpenReview.net.
Ann Bies, Justin Mott, and Colin Warner. 2015. English news text treebank: Penn Treebank revised (LDC2015T13).
Erdem Bıyık, Kenneth Wang, Nima Anari, and Dorsa Sadigh. 2019. Batch active learning using determinantal point processes. arXiv preprint arXiv:1906.07975.
Alena Böhmová, Jan Hajic, Eva Hajicova, and Barbora Hladká. 2003. The Prague dependency treebank. In Anne Abeille, editor, Treebanks: Building and Us ing Parsing Corpora, Text, Speech and Language Technology, pages 103-127. Springer Netherlands, Dordrecht.
Klaus Brinker. 2003. Incorporating diversity in active learning with support vector machines. In Proceedings of the 20th International Conference on Machine Learning, pages 59-66, Washington, DC, USA. AAAI Press.
Andrei Z. Broder, Steven C. Glassman, Mark S. Manasse, and Geoffrey Zweig. 1997. Syntactic clustering of the Web. Computer Networks and ISDN Systems, 29(8):1157-1166.
Laming Chen, Guoxin Zhang, and Hanning Zhou. 2018. Fast greedy MAP inference for determinantal point process to improve recommendation diversity. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, pages 5627-5638, Montreal, Canada.
Sangwoo Cho, Logan Lebanoff, Hassan Foroosh, and Fei Liu. 2019a. Improving the similarity measure of determinantal point processes for extractive multi-document summarization. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1027-1038, Florence, Italy. Association for Computational Linguistics.
Sangwoo Cho, Chen Li, Dong Yu, Hassan Foroosh, and Fei Liu. 2019b. Multi-document summarization with determinantal point processes and contextualized representations. In Proceedings of the 2nd Workshop on New Frontiers in Summarization, pages 98-103, Hong Kong, China. Association for Computational Linguistics.
Yoeng-Jin Chu and Tseng-Hong Liu. 1965. On the shortest arborescence of a directed graph. Science Sinica, 14:1396-1400.
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised