SlowGuess/NAACL_2022_OCR huggingface dataset:
We keep a single pre-selected set of hyperparameters for all AL iterations. Tables 4, 6, 7 in Appendix A describe the hyperparameter setup. Hyperparameter tuning on each AL iteration is very time-consuming. This is an important research problem but out of the scope of the current work.
5.2.1 Acquisition-Successor Mismatch
First of all, we illustrate the ASM problem on the selected datasets with various acquisition-successor pairs (Figure 1a and Figures 5a, 6a, 7a, 8a, 9a in
Figure 1: AL experiments on CoNLL-2003, in which a successor model does not match an acquisition model (DistilBERT).
Figure 2: The performance of PLASM (BERT is a pseudo-labeling model) on CoNLL-2003 compared with the standard approach to AL.
Appendix B). The presented results correspond to the findings of Lowell et al. (2019) and Shelmanov et al. (2021). In each experiment, we see a significant reduction in the performance of successor models when for acquisition, a distilled model from a different family is used. The performance drop is especially notable when we compare results of ELECTRA(acq.)-ELECTRA(succ.) to results of DistilBERT(acq.)-ELECTRA(succ.) on the CoNLL-2003 dataset in Figure 1a and to results of DistilRoBERTa(acq.)-ELECTRA(succ.) on AG News in Figure 8a in Appendix B. Moreover, Figure 10 in Appendix B shows that even if use full-fledged BERT for acquisition and ELECTRA as a successor (and vice versa), a similar performance drop is also present.
The ASM problem appears to be even more severe with the modern uncertainty estimation technique based on MD. Figure 11 in Appendix B shows the results of experiments with MD and DistilBERT(acq.)-ELECTRA(succ.) on the AG
Figure 3: Uncertainty correlation matrix of various Transformers on the AG News dataset. The correlations were obtained by training a model on the  $1\%$  of the training data and calculating the LC score for the rest  $99\%$  of instances.
News dataset. On most iterations, the performance drop for MD is even bigger than the drop for LC shown in Figure 6a.
In the next series of experiments, we demonstrate on both text classification and tagging tasks that when an acquisition model is a distilled version of the full-fledged successor model, the ASM problem is substantially alleviated (Figure 1b, and Figures 5b, 6b, 7b, 8b, 9b in Appendix B). Previously, this effect was also revealed by Shelmanov et al. (2021) for tagging. As we can see in Figure 1b, when DistilBERT is used as an acquisition model, the successor model based on BERT does not experience a performance drop. A similar effect with other model pairs can be noted for sequence tagging on OntoNotes and for text classification on AG News and IMDb.
Figure 3 shows the correlation between the output probabilities of various Transformer-based models fine-tuned on  $1\%$  of the AG News dataset. As we can see, for each model, the other most similar model is its student / teacher (except for DistilRoBERTa, which has a slightly larger correlation with DistilBERT). This explains the absence of the ASM problem for such model pairs. Since, after fine-tuning, the distilled version of a model produces similar uncertainty estimates, it will strive to query similar instances during AL.
Although we can mitigate the ASM problem for such model pairs as DistilBERT-BERT, it is still a serious constraint for applying AL. Obviously, such an approach is not feasible if for the final applica
tion, one would like to train a completely different model (e.g. XLNet). In the next section, we show that the proposed method based on pseudo-labeling helps to overcome this limitation and resolve the ASM problem in a more general case.
5.2.2 Pseudo-labeling for Acquisition-Successor Mismatch
Figure 2, and Figures 12, 13, 14 in Appendix C present the performance of PLASM on considered datasets for various combinations of acquisition, successor, and pseudo-labeling Transformer models in comparison with the case when acquisition and successor models are the same and with the case of ASM. On the AG News dataset (Figure 12), we investigate the effect of PLASM for three different successors: ELECTRA, RoBERTa, XLNet and for three different distilled acquisition models: DistilBERT, DistilRoBERTa, and our custom DistilELECTRA model. In all experiments, PLASM substantially alleviates the ASM problem yielding higher results compared to directly fine-tuning on data acquired with a different acquisition model.
Usually, PLASM yields a similar or slightly better results than the case when the same model is used both for acquisition and as a successor. PLASM might be superior than this case when a pseudo-labeling model is better suited to the dataset than a successor model. For example, in Figure 12d, PLASM shows better results on early AL iterations than ELECTRA(acq.)-ELECTRA(succ.) due to the fact that RoBERTa used for pseudo-labeling has generally higher performance on AG News than ELECTRA when fine-tuned on the same amount of labeled data. However, we argue that PLASM also effectively helps to deal with the ASM problem when the successor model is more expressive than the pseudo-labeling model. This is the case of the experiment on CoNLL-2003 (Figure 2), where PLASM completely mitigates the ASM problem, while ELECTRA successor shows generally better results than BERT used for pseudo-labeling.
Figures 15 and 16 in Appendix C show that PLASM also mitigates the performance drop due to ASM between a DistilBERT acquisition model and classical CNN-BiLSTM-CRF or CNN successor models. In this case, PLASM gives a very big boost to performance compared to the case when the same classical model is used both for acquisition and as a successor. This happens because the BERT-based pseudo-labeling model is better suited for fine-tuning on small data than the
classical models and produces good automatically labeled instances that are reused by successors.
Figure 17a presents the results of the first ablation study, in which, for pseudo-labeling, we leverage the same distilled model used for acquisition instead of a more expressive teacher. In particular, DistilBERT performs acquisition and pseudolabeling instead of BERT, while ELECTRA is a successor. The performance drop in this case compared to PLASM demonstrates that using an expressive model (e.g. BERT) for pseudo-labeling is necessary for achieving high scores at the beginning of annotation. Figure 17b presents the results of the second ablation study, in which we use DistilBERT for acquisition and ELECTRA for pseudo-labeling and as a successor. This study demonstrates that pseudo-labeling on its own cannot alleviate the ASM completely. It is better to use an expressive pseudo-labeling model that also matches the lightweight acquisition model (e.g. distilled model for acquisition, its teacher – for labeling), as it is proposed in PLASM.
The ablation study of the methods for filtering erroneous instances in the pseudo-labeling step is conducted on the AG News dataset in Figures 18a,b in Appendix C. Applying each of the methods gives substantial improvements over the PLASM without the filtering step, while TracIn is slightly better than thresholding uncertainty of pseudo-labeling model predictions. We note that for XLNet as a successor, PLASM without filtering alleviates the ASM problem, but does not approach the performance of the case when XLNet is used as an acquisition model.
Table 2 and Table 8 in Appendix D summarize the time required for conducting AL iterations with different acquisition functions on the AG News and CoNLL-2003 datasets. As we can see, since PLASM uses DistilBERT for acquisition, our method reduces the iteration time by more than  $30\%$  compared to the standard approach, in which ELECTRA is used for acquisition. Thereby, empirical results show that PLASM offers two benefits: (1) it helps to alleviate the ASM problem in AL; (2) it reduces the time of an AL iteration and required computational resources for training and running acquisition models. These benefits substantially increase the practicality of using AL in interactive annotation tools.
5.2.3 Unlabeled Pool Subsampling
Table 2 compares the duration of AL iterations on the AG News dataset, including the duration of
Top-k% / Curr. AL iter.	1	2	6
10%	0.503	0.649	0.924
20%	0.789	0.883	0.992
30%	0.915	0.947	0.995
40%	0.958	0.976	1.000
50%	0.980	0.991	1.000
Table 1: A fraction of instances that would be standardly selected on the current AL iteration, contained in top- $k\%$  uncertain instances according to the acquisition model on the previous iteration (AG News corpus).
Figure 4: The performance of UPS with PLASM (BERT is a pseudo-labeling model) on AG News compared with baselines  $(\gamma = 0.1, T = 0.01)$ .
the acquisition model training step and the duration of inference on instances from the unlabeled pool. We can see that the inference step is very time-consuming, especially on early iterations, and takes more than half of the time required for performing an AL iteration. Therefore, we claim that in such cases, it is more important to accelerate the inference step rather than the training step as it was done in previous work (Shen et al., 2017).
To justify our approach to accelerating the inference step, we show that many unlabeled instances have similar uncertainty estimates across different AL iterations. Table 1 presents the fraction of instances, which would be standardly queried on the current iteration if we selected them from the whole unlabeled pool that are contained in  $\mathrm{k} - \%$  of most uncertain instances, according to the acquisition model built on the previous AL iteration. For example, we observe that  $50\%$  of the most uncertain instances according to the model trained on the first iteration contain more than  $99\%$  of instances from the "standard query" on the second iteration, and  $30\%$  contains almost  $95\%$  of instances from the "standard query". Later iterations have even a better trade-off. Thereby, it is reasonable to avoid spending computational resources on instances that were most certain in previous iterations.
ELECTRA	BERT	DistilBERT	ELECTRA with UPS (ours)	DistilBERT with UPS (ours)
Iter. 2	Train	176.3 ± 1.4	174.8 ± 1.4	87.4 ± 0.8	178.0 ± 1.4	87.9 ± 0.5
Inference	622.2 ± 9.4	623.8 ± 7.5	481.8 ± 17.2	630.9 ± 12.3	483.2 ± 23.0
Overall	798.6 ± 9.6	798.6 ± 8.4	569.2 ± 17.5	808.8 ± 12.8	571.1 ± 22.6
Iter. 6	Train	342.8 ± 5.7	339.9 ± 4.2	174.1 ± 2.9	342.2 ± 5.3	173.0 ± 1.4
Inference	600.5 ± 10.4	596.4 ± 6.6	455.1 ± 8.9	58.9±3.3	50.0±6.4
Overall	943.4 ± 15.9	936.3 ± 8.8	629.1 ± 9.7	401.1±3.4	222.9±5.9
Iter. 10	Train	504.6 ± 6.3	498.8 ± 3.9	257.5 ± 3.9	502.7 ± 6.0	255.1 ± 3.4
Inference	573.0 ± 6.9	577.5 ± 7.7	434.6 ± 4.6	55.5±2.9	42.6±7.1
Overall	1077.6 ± 13.1	1076.4 ± 10.9	692.1 ± 5.5	558.2±4.4	297.7±10.3
Iter. 15	Train	701.9 ± 7.2	714.9 ± 20.5	358.3 ± 3.0	704.8 ± 11.7	359.3 ± 5.4
Inference	548.6 ± 9.2	541.0 ± 5.0	415.9 ± 10.2	59.4±3.1	39.3±2.6
Overall	1250.5 ± 16.0	1255.9 ± 18.4	774.2 ± 10.8	764.2±10.6	398.6±6.8
Overall train	6323.7 ± 72.1	6294.8 ± 73.7	3215.1 ± 38.5	6333.3 ± 92.8	3204.5 ± 32.5
Overall inference	8799.2 ± 150.7	8787.5 ± 102.7	6682.1 ± 96.2	3110.9±85.3	2332.2±86.2
Overall	15122.9 ± 213.4	15082.2 ± 141.1	9897.1 ± 112.8	9444.2±113.6	5536.7±100.8
Table 2: Duration of training and inference steps of AL iterations in seconds on AG News. Hardware configuration: 2 Intel Xeon Platinum 8168, 2.7 GHz, 24 cores CPU; NVIDIA Tesla v100 GPU, 32 Gb of VRAM.
If we exclude a big part of the unlabeled pool from consideration during acquisition, the benefits of AL can potentially deteriorate. Results of experiments presented in Figure 4 and Figures 19, 20 in Appendix D show that the proposed UPS algorithm does not lead to the performance drop compared to the standard approach, in which we consider the whole unlabeled pool for instance selection. Meanwhile, the results of the ablation study in Figure 21 in Appendix D demonstrate that the baseline, which randomly subsamples the unlabeled dataset, has a performance drop compared to UPS. In another ablation study, we set  $T = 0$ , which means that UPS just takes a fraction of the most uncertain instances (Figure 22). On some iterations, this results in a slight reduction of performance.
From Table 2, we can see that UPS accelerates the query process up to 10 times. The corresponding results for CoNLL-2003 are presented in Table 8 in Appendix D. Overall, applying both PLASM and UPS algorithms on AG News reduces the duration of AL iterations by more than  $60\%$  compared with the standard approach. We can also tune the hyperparameters  $\gamma$  and  $T$  to reduce duration further in exchange for slightly worse scores.
We investigated several obstacles to deploying AL in practice and proposed two algorithms that help to overcome them. In particular, we considered the acquisition-successor mismatch problem revealed by Lowell et al. (2019), as well as the problem related to the excessive duration of AL iterations with uncertainty-based query strategies and deep learning models. We demonstrate that the proposed
PLASM algorithm helps to deal with both of these issues: it removes the constraint on the type of the successor model trained on the data labeled with AL and allows the use of lightweight acquisition models that have good training and inference performance, as well as a small memory footprint. The unlabeled pool subsampling algorithm helps to substantially decrease the inference time during AL without a loss in the quality of successor models. Together the PLASM and UPS algorithms help reduce the duration of an AL iteration by more than  $60\%$ . We consider that the conducted empirical investigations and the proposed methods will help to increase the practicality of using deep AL in interactive annotation tools.
We note that applying PLASM requires some conditions to be met. Particularly, when a pseudolabeling model is of considerably lower performance than a successor model, and filtering is not strict enough, training a successor directly on labeled instances acquired during AL with a different acquisition model may result in higher performance. Consequently, despite the pseudo-labeling model may be less expressive compared to the successor model, it should not be too "weak". In practice, we suggest comparing results obtained by the models trained with pseudo-labeling and without on a hold-out set and selecting the best model.
There are still many issues that hinder the application of AL techniques. We consider that one of the most important obstacles is the necessity of hyperparameter optimization of deep learning models that can take a prohibitively long time to keep the annotation process interactive. We are looking forward to addressing this problem in future work.
We thank anonymous reviewers for their insightful suggestions to improve this paper. The work was supported by a grant for research centers in the field of artificial intelligence (agreement identifier 000000D730321P5Q0002 dated November 2, 2021 No. 70-2021-00142 with ISP RAS).
Jason Baldridge and Miles Osborne. 2004. Active learning and the total cost of annotation. In Proceedings of the 2004 Conference on Empirical Methods in Natural Language Processing, pages 9-16, Barcelona, Spain. Association for Computational Linguistics.
Charles Blundell, Julien Cornebise, Koray Kavukcuoglu, and Daan Wierstra. 2015. Weight uncertainty in neural network. In Proceedings of the 32nd International Conference on Machine Learning, ICML 2015, Lille, France, 6-11 July 2015, volume 37 of JMLR Workshop and Conference Proceedings, pages 1613-1622. JMLR.org.
Kianté Brantley, Hal Daumé III, and Amr Sharaf. 2020. Active imitation learning with noisy guidance. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2093-2105, Online. Association for Computational Linguistics.
Kevin Clark, Minh-Thang Luong, Quoc V. Le, and Christopher D. Manning. 2020. ELECTRA: pretraining text encoders as discriminators rather than generators. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020.
David A Cohn, Zoubin Ghahramani, and Michael I Jordan. 1996. Active learning with statistical models. Journal of artificial intelligence research, 4:129-145.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Liat Ein-Dor, Alon Halfon, Ariel Gera, Eyal Shnarch, Lena Dankin, Leshem Choshen, Marina Danilevsky, Ranit Aharonov, Yoav Katz, and Noam Slonim. 2020. Active Learning for BERT: An Empirical Study. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7949-7962, Online. Association for Computational Linguistics.
Alexander Erdmann, David Joseph Wrisley, Benjamin Allen, Christopher Brown, Sophie Cohen-Bodénès, Micha Elsner, Yukun Feng, Brian Joseph, Beatrice Joyeux-Prunel, and Marie-Catherine de Marneffe. 2019. Practical, efficient, and customizable active learning for named entity recognition in the digital humanities. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 2223-2234, Minneapolis, Minnesota. Association for Computational Linguistics.
Meng Fang, Yuan Li, and Trevor Cohn. 2017. Learning how to active learn: A deep reinforcement learning approach. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 595-605, Copenhagen, Denmark. Association for Computational Linguistics.
Yarin Gal and Zoubin Ghahramani. 2016. Dropout as a Bayesian approximation: Representing model uncertainty in deep learning. In Proceedings of the 33nd International Conference on Machine Learning, ICML 2016, New York City, NY, USA, June 19-24, 2016, volume 48 of JMLR Workshop and Conference Proceedings, pages 1050-1059.
Geoffrey E. Hinton, Oriol Vinyals, and Jeffrey Dean. 2015. Distilling the knowledge in a neural network. CoRR, abs/1503.02531.
Jeremy Howard and Sebastian Ruder. 2018. Universal language model fine-tuning for text classification. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 328-339.
Rong Hu, Brian Mac Namee, and Sarah Jane Delany. 2016. Active learning for text classification with reusability. Expert Systems with Applications: An International Journal, 45(C):438-449.
Bulat Ibragimov and Gleb Gusev. 2019. Minimal variance sampling in stochastic gradient boosting. In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc.
Guolin Ke, Qi Meng, Thomas Finley, Taifeng Wang, Wei Chen, Weidong Ma, Qiwei Ye, and Tie-Yan Liu. 2017. Lightgbm: A highly efficient gradient boosting decision tree. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc.
Hoa T Le, Christophe Cerisara, and Alexandre Denis. 2018. Do convolutional networks need to be deep for text classification? In Workshops at the Thirty-Second AAAI Conference on Artificial Intelligence.
Kimin Lee, Kibok Lee, Honglak Lee, and Jinwoo Shin. 2018. A simple unified framework for detecting out-of-distribution samples and adversarial attacks. In Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc.
David D. Lewis and William A. Gale. 1994. A sequential algorithm for training text classifiers. In Proceedings of the 17th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval. Dublin, Ireland, 3-6 July 1994 (Special Issue of the SIGIR Forum), pages 3-12. ACM/Springer.
Ming Liu, Wray Buntine, and Gholamreza Haffari. 2018. Learning how to actively learn: A deep imitation learning approach. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1874-1883, Melbourne, Australia. Association for Computational Linguistics.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692.
David Lowell, Zachary C. Lipton, and Byron C. Wallace. 2019. Practical obstacles to deploying active learning. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 21-30, Hong Kong, China. Association for Computational Linguistics.
Xuezhe Ma and Eduard Hovy. 2016. End-to-end sequence labeling via bi-directional LSTM-CNNs-CRF. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1064-1074, Berlin, Germany. Association for Computational Linguistics.
Andrew Maas, Raymond E Daly, Peter T Pham, Dan Huang, Andrew Y Ng, and Christopher Potts. 2011. Learning word vectors for sentiment analysis. In Proceedings of the 49th annual meeting of the association for computational linguistics: Human language technologies, pages 142-150.
Katerina Margatina, Loic Barrault, and Nikolaos Aletras. 2021. Bayesian active learning with pretrained language models. arXiv preprint arXiv:2104.08320.
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Distributed representations of words and phrases and their compositionality. In Proceedings of the 26th International Conference on Neural Information Processing Systems - Volume 2, NIPS'13, page 3111-3119, Red Hook, NY, USA. Curran Associates Inc.
Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. GloVe: Global vectors for word representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1532-1543, Doha, Qatar. Association for Computational Linguistics.