SlowGuess/NAACL_2021_OCR huggingface dataset:
cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8440-8451, Online. Association for Computational Linguistics.
Ryan Cotterell and Jason Eisner. 2017. Probabilistic typology: Deep generative models of vowel inventories. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1182-1192, Vancouver, Canada. Association for Computational Linguistics.
Timothy Dozat and Christopher D. Manning. 2017. Deep bioaffine attention for neural dependency parsing. In International Conference on Learning Representations, Toulon, France. OpenReview.net.
Jack Edmonds. 1967. Optimum branchings. Journal of Research of the National Bureau of Standards, 71B(4):233-240.
Daniel Flannery and Shinsuke Mori. 2015. Combining active learning and partial annotation for domain adaptation of a Japanese dependency parser. In Proceedings of the 14th International Conference on Parsing Technologies, pages 11-19, Bilbao, Spain. Association for Computational Linguistics.
Yarin Gal, Riashat Islam, and Zoubin Ghahramani. 2017. Deep Bayesian active learning with image data. In Proceedings of the 34th International Conference on Machine Learning, pages 1183-1192, Sydney, Australia. PMLR.
Rebecca Hwa. 2000. Sample selection for statistical grammar induction. In Proceedings of the 2000 Joint SIGDAT Conference on Empirical Methods in Natural Language Processing and Very Large Corpora, pages 45-52, Hong Kong, China. Association for Computational Linguistics.
Rebecca Hwa. 2004. Sample selection for statistical parsing. Computational Linguistics, 30(3):253-276.
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In International Conference on Learning Representations, San Diego, California, USA.
Andreas Kirsch, Joost van Amersfoort, and Yarin Gal. 2019. BatchBALD: Efficient and diverse batch acquisition for deep Bayesian active learning. In Advances in Neural Information Processing Systems 32, pages 7026-7037, Vancouver, Canada. Curran Associates, Inc.
Nikita Kitaev, Steven Cao, and Dan Klein. 2019. Multilingual constituency parsing with self-attention and pre-training. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3499–3505, Florence, Italy. Association for Computational Linguistics.
Terry Koo, Amir Globerson, Xavier Carreras, and Michael Collins. 2007. Structured prediction models via the matrix-tree theorem. In Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP-CoNLL), pages 141-150, Prague, Czech Republic. Association for Computational Linguistics.
Sandra Kübler, Ryan McDonald, and Joakim Nivre. 2008. Dependency Parsing, volume 2 of Synthesis Lectures on Human Language Technologies. Morgan and Claypool.
Taku Kudo and John Richardson. 2018. SentencePiece: A simple and language independent subword tokenizer and tokenizer for neural text processing. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 66-71, Brussels, Belgium. Association for Computational Linguistics.
Alex Kulesza and Ben Taskar. 2012. Determinantal point processes for machine learning. Foundations and Trends in Machine Learning, 5(2-3):123-286.
John A. Kulesza. 2012. Learning with Determinantal Point Processes. Ph.D. thesis, University of Pennsylvania.
David D. Lewis and William A. Gale. 1994. A sequential algorithm for training text classifiers. In Proceedings of the 17th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 3-12, Dublin, Ireland. ACM/Springer.
Zhenghua Li, Min Zhang, Yue Zhang, Zhanyi Liu, Wenliang Chen, Hua Wu, and Haifeng Wang. 2016. Active learning for dependency parsing with partial annotation. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 344-354, Berlin, Germany.
Saeed Majidi and Gregory Crane. 2013. Active learning for dependency parsing by a committee of parsers. In Proceedings of the 13th International Conference on Parsing Technologies (IWPT 2013), pages 98-105, Nara, Japan. Association for Computational Linguistics.
Gurmeet Singh Manku, Arvind Jain, and Anish Das Sarma. 2007. Detecting near-duplicates for web crawling. In Proceedings of the 16th International Conference on World Wide Web, pages 141-150, Banff, Alberta, Canada. Association for Computing Machinery.
Christopher Manning, Mihai Surdeanu, John Bauer, Jenny Finkel, Steven Bethard, and David McClosky. 2014. The Stanford CoreNLP natural language processing toolkit. In Proceedings of 52nd Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 55-60, Baltimore, Maryland, USA. Association for Computational Linguistics.
Mitchell P. Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz. 1993. Building a large annotated corpus of English: The Penn Treebank. Computational Linguistics, 19(2):313-330.
Seyed Abolghasem Mirroshandel and Alexis Nasr. 2011. Active learning for dependency parsing using partially annotated sentences. In Proceedings of the 12th International Conference on Parsing Technologies, pages 140-149, Dublin, Ireland. Association for Computational Linguistics.
Miles Osborne and Jason Baldridge. 2004. Ensemble-based active learning for parse selection. In Proceedings of the Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics: HLT-NAACL 2004, pages 89–96, Boston, Massachusetts, USA. Association for Computational Linguistics.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. 2019. PyTorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems 32, pages 8026-8037, Vancouver, Canada. Curran Associates, Inc.
Manabu Sassano and Sadao Kurohashi. 2010. Using smaller constituents rather than sentences in active learning for Japanese dependency parsing. In Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, pages 356-365, Uppsala, Sweden. Association for Computational Linguistics.
Alexandra Schofield, Laure Thompson, and David Mimno. 2017. Quantifying the effects of text duplication on semantic models. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2737-2747, Copenhagen, Denmark. Association for Computational Linguistics.
Burr Settles. 2009. Active learning literature survey. Computer Sciences Technical Report 1648, University of Wisconsin-Madison.
Burr Settles and Mark Craven. 2008. An analysis of active learning strategies for sequence labeling tasks. In Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing, pages 1070-1079, Honolulu, Hawaii, USA. Association for Computational Linguistics.
Yanyao Shen, Hyokun Yun, Zachary C. Lipton, Yakov Kronrod, and Animashree Anandkumar. 2018. Deep active learning for named entity recognition. In International Conference on Learning Representations, Vancouver, Canada. OpenReview.net.
Aditya Siddhant and Zachary C. Lipton. 2018. Deep Bayesian active learning for natural language processing: Results of a large-scale empirical study. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2904-2909, Brussels, Belgium. Association for Computational Linguistics.
Min Tang, Xiaoqiang Luo, and Salim Roukos. 2002. Active learning for statistical natural language parsing. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 120-127, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.
Cynthia A. Thompson, Mary Elaine Califf, and Raymond J. Mooney. 1999. Active learning for natural language parsing and information extraction. In Proceedings of the Sixteenth International Conference on Machine Learning, pages 406-414, Bled, Slovenia. Morgan Kaufmann Publishers Inc.
William T. Tutte. 1984. Graph Theory. Addison-Wesley Publishing Company.
Laurens van der Maaten and Geoffrey Hinton. 2008. Visualizing data using t-SNE. Journal of machine learning research, 9(Nov):2579-2605.
Appendix A Dependency Parser
We adopt the deep biaffine dependency parser proposed by Dozat and Manning (2017). The parser is conceptually simple and yet competitive with state-of-the-art dependency parsers. The parser has three components: feature extraction, unlabeled parsing and relation labeler.
Feature Extraction For a length- $n$  sentence  $x = x_0, x_1, x_2, \ldots, x_n$ , where  $x_0$  is the dummy root symbol, we extract contextualized features at each word position. In our experiments, we use a pretrained multilingual XLM-RoBERTa base model (Conneau et al., 2020), and fine-tune the feature extractor along with the rest of our parser:
$$
\left[ \mathbf {x} _ {0}, \mathbf {x} _ {1}, \dots , \mathbf {x} _ {n} \right] = \operatorname {X L M - R} \left(x _ {0}, x _ {1}, \dots , x _ {n}\right).
$$
Each word input to the XLM-RoBERTa model is processed with the SentencePiece tokenizer (Kudo and Richardson, 2018), and we follow Kitaev et al. (2019) and retain the vectors corresponding to the last sub-word units as their representations. For  $\mathbf{x}_0$ , we use the vector of the [CLS] token, which is appended by XLM-RoBERTa to the beginning of each sentence.
Unlabeled Parser The parser uses a deep bi-affine attention mechanism to derive locally-normalized attachment probabilities for all potential head-dependent pairs:
$$
\begin{array}{l} \mathbf {h} _ {i} ^ {\text {a r c - h e a d}} = \mathrm {M L P} ^ {\text {a r c - h e a d}} \left(\mathbf {x} _ {i}\right) \\ \mathbf {h} _ {j} ^ {\operatorname {a r c - d e p}} = \operatorname {M L P} ^ {\operatorname {a r c - d e p}} (\mathbf {x} _ {j}) \\ s _ {i, j} = \left[ \mathbf {h} _ {i} ^ {\text {a r c - h e a d}}; 1 \right] ^ {\top} U ^ {\text {a r c}} \left[ \mathbf {h} _ {j} ^ {\text {a r c - d e p}}; 1 \right] \\ \end{array}
$$
$$
P _ {\text {a t t}} (\text {h e a d} (j) = i \mid x) = \operatorname {s o f t m a x} _ {i} \left(s _ {:, j}\right),
$$
where  $\mathbf{MLP}^{\mathrm{arc - head}}$  and  $\mathbf{MLP}^{\mathrm{arc - dep}}$  are two multilayer perceptrons (MLPs) projecting  $\mathbf{x}$  vectors into  $d^{\mathrm{arc}}$ -dimensional  $\mathbf{h}$  vectors, [;1] appends an element of 1 at the end of the vectors, and  $U^{\mathrm{arc}} \in \mathbb{R}^{(d^{\mathrm{arc}} + 1) \times (d^{\mathrm{arc}} + 1)}$  is a bilinear scoring matrix. This component is trained with cross-entropy loss of the gold-standard attachments. During inference, we use the Chu-Liu-Edmonds algorithm (Chu and Liu, 1965; Edmonds, 1967) to find the spanning tree with the highest product of locally-normalized attachment probabilities.
Relation Labeler The relation labeling component employs a similar deep biaffine scoring func
Round #	5	10
Strategy	w/o DPP	w/ DPP	w/o DPP	w/ DPP
Random	89.01±.28	89.67±.30	90.78±.27	91.22±.22
AMP	89.67±.29	90.24±.39	92.03±.10	92.17±.22
BALD	89.82±.36	90.29±.20	91.87±.36	92.00±.08
ID	90.24±.20	90.03±.18	92.16±.17	92.06±.16
Table B1: UAS after 5 and 10 rounds of annotation (roughly 5,000 and 7,000 training tokens respectively), comparing strategies with and without modeling diversity through DPP.
tion as the unlabeled parsing component:
$$
\begin{array}{l} \mathbf {h} _ {i} ^ {\text {r e l - h e a d}} = \mathrm {M L P} ^ {\text {r e l - h e a d}} (\mathbf {x} _ {i}) \\ \mathbf {h} _ {j} ^ {\text {r e l - d e p}} = \operatorname {M L P} ^ {\text {r e l - d e p}} \left(\mathbf {x} _ {j}\right) \\ t _ {i, j, r} = [ \mathbf {h} _ {i} ^ {\mathrm {r e l - h e a d}}; 1 ] ^ {\top} U _ {r} ^ {\mathrm {r e l}} [ \mathbf {h} _ {j} ^ {\mathrm {r e l - d e p}}; 1 ] \\ \end{array}
$$
$$
P (\operatorname {r e l} (i, j) = r) = \operatorname {s o f t m a x} _ {r} (t _ {i, j,:}),
$$
where each  $U_r^{\mathrm{rel}} \in \mathbb{R}^{(d^{\mathrm{rel}} + 1) \times (d^{\mathrm{rel}} + 1)}$ , and there are as many such matrices as the size of the dependency relation label set  $|R|$ . The relation labeler is trained using cross entropy loss on the gold-standard head-dependent pairs. During inference, the labeling decision for each arc is made independently given the predicted unlabeled parse tree.
Appendix B Results with UAS Evaluation
We also evaluate different learning strategies based on unlabeled attachment scores (UAS), and the results are shown in Table B1. In line with LAS-based experiments, we find that modeling diversity is more helpful during initial stages of learning. For ID, we observe this effect even earlier than the fifth round of annotation:  $86.57_{\pm .44}$  vs.  $87.40_{\pm .51}$  after the first annotation round.
Appendix C Sentence Selection Examples
In Table C2 we compare batches sampled by a diversity-aware selection strategy with a diversity-agnostic one. We observe that by modeling diversity in the sample selection process, DPPs avoid selecting duplicate or near-duplicate sentences and thus the annotation budget can be maximally utilized.
Appendix D BALD under High Duplication Setting
Figure D1 shows the learning curves for BALD-based selection strategies under a high corpus duplication setting where the corpus is repeated
Sentences selected by AMP (highest-quality ones first):
Sentences selected by AMP with diversity-inducing DPP:
Downgraded by Moody 's were Houston Lighting ' s first - mortgage bonds and secured pollution - control bonds to single - A - 3 from single - A - 2 ; unsecured pollution - control bonds to Baa - 1 from single - A - 3 ; preferred stock to single - A - 3 from single - A - 2 ; a shelf registration for preferred stock to a preliminary rating of single - A - 3 from a preliminary rating of single - A - 2 ; two shelf registrations for collateralized debt securities to a preliminary rating of single - A - 3 from a preliminary rating of single - A - 2 , and the unit ' s rating for commercial paper to Prime - 2 from Prime - 1 .
4 . When a RICO TRO is being sought , the prosecutor is required , at the earliest appropriate time , to state publicly that the government ' s request for a TRO , and eventual forfeiture , is made in full recognition of the rights of third parties - that is , in requesting the TRO , the government will not seek to disrupt the normal , legitimate business activities of the defendant ; will not seek through use of the relation - back doctrine to take from third parties assets legitimately transferred to them ; will not seek to vitiate legitimate business transactions occurring between the defendant and third parties ; and will , in all other respects , assist the court in ensuring that the rights of third parties are protected , through proceeding under RICO and otherwise .
COMMERCIAL PAPER placed directly by General Motors Acceptance Corp. : 8.40 % 30 to 44 days ; 8.325 % 45 to 59 days ; 8.10 % 60 to 89 days ; 8 % 90 to 119 days ; 7.85 % 120 to 149 days ; 7.70 % 150 to 179 days ; 7.375 % 180 to 270 days .
Table C2: Sentences picked by a diversity-agnostic (top) and a diversity-aware (bottom) selection strategy from the same unlabeled pool after the initial round of model training on the seed sentences. The diversity-agnostic strategy selects many near-duplicate sentences (the two near-duplicate clusters are marked by red  $\rightarrow$  and blue  $\triangleright$ ), effectively wasting the annotation budget, where DPPs largely alleviate this issue by enforcing diversity.
Figure D1: Learning curves for BALD-based selection strategies under a five-fold corpus duplication setting.
five times. In this extreme setting, the diversity-agnostic strategy significantly underperforms the diversity-aware one. We posit that the relative success of BALD compared to AMP in the twice-duplicated setting is due to the fact that BALD randomly draws dropout masks to estimate model uncertainty, so that identical examples could still have different quality measures.
Appendix E Implementation Details and Hyperparameters
We do not tune our hyperparameters since in practice, active learning systems only have a single shot at success, without tuning. Instead, we follow recommendations from relevant prior work in setting our learning details and hyperparameters.
Active Learning Following Li et al. (2016), our active learning set-up proceeds in two stages for each annotation round. In the first stage, we select sentences filling in a budget of 2500 tokens; in the second stage, we pick 500 tokens out of the subset of sentences. For a diversity-agnostic strategy, we choose the top- $k$  highest-quality candidates within the token budget, while our diversity-aware selec
tion strategy uses a separate DPP for each stage. The active learning process is bootstraped with a seed set of 128 labeled sentences. For the BALD quality measure, we set  $K = 5$ .
Greedy MAP Inference for DPPs Algorithm E1 illustrates the procedure for selecting items from DPPs under a budget constraint. This greedy MAP inference algorithm is adapted from Kulesza (2012). During sentence selection, the size of a sentence is its number of tokens, and each token has a size of 1 in the token selection stage.
Dependency Parser We set the hyperparameters according to Dozat and Manning (2017). All the MLPs in the deep biaffine attention architecture have single hidden layers with ReLU activation functions and a dropout probability of 0.33, and we set  $d^{\mathrm{arc}}$  and  $d^{\mathrm{rel}}$  to be 500 and 100 respectively.
Training and Optimization Each training batch contains 16 sentences and gradient norms are clipped to 5.0. We use the Adam optimizer (Kingma and Ba, 2015) with a learning rate of  $10^{-5}$  with 640 warmup steps with a linearly-increasing learning rate starting from 0.
Implementation Our implementation is in PyTorch (Paszke et al., 2019), and we use the transformers package to interface with the pretrained XLM-RoBERTa model.