SlowGuess/NAACL_2022_OCR huggingface dataset:
Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 2227-2237, New Orleans, Louisiana. Association for Computational Linguistics.
Alexander Podolskiy, Dmitry Lipin, Andrey Bout, Ekaterina Artemova, and Irina Pionkovskaya. 2021. Revisiting mahalanobis distance for transformer-based out-of-domain detection. In Thirty-Fifth AAAI Conference on Artificial Intelligence, AAAI 2021, Thirty-Third Conference on Innovative Applications of Artificial Intelligence, IAAI 2021, The Eleventh Symposium on Educational Advances in Artificial Intelligence, EAAI 2021, Virtual Event, February 2-9, 2021, pages 13675-13682. AAAI Press.
Ameya Prabhu, Charles Dognin, and Maneesh Singh. 2019. Sampling bias in deep active classification: An empirical study. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 4049-4059.
Sameer Pradhan, Alessandro Moschitti, Nianwen Xue, Hwee Tou Ng, Anders Björkelund, Olga Uryupina, Yuchen Zhang, and Zhi Zhong. 2013. Towards robust linguistic analysis using OntoNotes. In Proceedings of the Seventeenth Conference on Computational Natural Language Learning, pages 143-152, Sofia, Bulgaria. Association for Computational Linguistics.
Garima Pruthi, Frederick Liu, Satyen Kale, and Mukund Sundararajan. 2020. Estimating training data influence by tracing gradient descent. In Advances in Neural Information Processing Systems, volume 33, pages 19920-19930. Curran Associates, Inc.
Erik F. Tjong Kim Sang and Fien De Meulder. 2003. Introduction to the CoNLL-2003 shared task: Language-independent named entity recognition. Proceedings of the Seventh Conference on Natural Language Learning, CoNLL 2003, Held in cooperation with HLT-NAACL 2003, Edmonton, Canada, May 31 - June 1, 2003, pages 142-147.
Victor Sanh, Lysandre Debut, Julien Chaumont, and Thomas Wolf. 2019. DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01108.
Burr Settles. 2009. Active learning literature survey. Computer Sciences Technical Report 1648, University of Wisconsin-Madison.
Burr Settles and Mark Craven. 2008. An analysis of active learning strategies for sequence labeling tasks.
In 2008 Conference on Empirical Methods in Natural Language Processing, EMNLP 2008, Proceedings of the Conference, 25-27 October 2008, Honolulu, Hawaii, USA, A meeting of SIGDAT, a Special Interest Group of the ACL, pages 1070-1079. Association for Natural Language Processing.
Artem Shelmanov, Dmitri Puzyrev, Lyubov Kupriyanova, Denis Belyakov, Daniil Larionov, Nikita Khromov, Olga Kozlova, Ekaterina Artemova, Dmitry V. Dylov, and Alexander Panchenko. 2021. Active learning for sequence tagging with deep pre-trained models and Bayesian uncertainty estimates. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 1698-1712, Online. Association for Computational Linguistics.
Yanyao Shen, Hyokun Yun, Zachary Lipton, Yakov Kronrod, and Animashree Anandkumar. 2017. Deep active learning for named entity recognition. In Proceedings of the 2nd Workshop on Representation Learning for NLP, pages 252-256, Vancouver, Canada. Association for Computational Linguistics.
Aditya Siddhant and Zachary C. Lipton. 2018. Deep Bayesian active learning for natural language processing: Results of a large-scale empirical study. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2904-2909, Brussels, Belgium. Association for Computational Linguistics.
Katrin Tomanek and Katherina Morik. 2011. Inspecting sample reusability for active learning. In Active Learning and Experimental Design workshop In conjunction with AISTATS 2010, pages 169-181. JMLR Workshop and Conference Proceedings.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, 4-9 December 2017, Long Beach, CA, USA, pages 5998-6008.
Thuy-Trang Vu, Ming Liu, Dinh Phung, and Gholamreza Haffari. 2019. Learning how to active learn by dreaming. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4091-4101, Florence, Italy. Association for Computational Linguistics.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumont, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, and Jamie Brew. 2019. Huggingface's Transformers: State-of-the-art natural language processing. CoRR, abs/1910.03771.
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. 2019. Xlnet: Generalized autoregressive pretraining for
language understanding. In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc.
Michelle Yuan, Hsuan-Tien Lin, and Jordan Boyd-Graber. 2020. Cold-start active learning through self-supervised language modeling. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7935-7948, Online. Association for Computational Linguistics.
Xiang Zhang, Junbo Zhao, and Yann LeCun. 2015. Character-level convolutional networks for text classification. In Proceedings of the 28th International Conference on Neural Information Processing Systems - Volume 1, Neural Information Processing Systems'15, page 649-657, Cambridge, MA, USA. MIT Press.
Wenxuan Zhou, Fangyu Liu, and Muhao Chen. 2021. Contrastive out-of-distribution detection for pretrained transformers. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 1100-1111, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
A Dataset Statistics and Model Hyperparameters
Table 3: Dataset statistics. We provide a number of sentences/tokens for the training and test sets. k stands for a size of seeding datasets (\% of the training dataset) and a size of sets of instances selected for "annotation" on each iteration. C is a number of classes/entity types.
Dataset	Train	Test	k	C
CoNLL-2003	15K/203.6K	3.7K/46.4K	2%	4(5)
OntoNotes 5.0	59.9K/1088.5K	8.3K/152.7K	2%	18
AG News	120K/4541.7K	7.6K/286.7K	1%	4
IMDb	25K/5844.7K	25K/5713.2K	1%	2
Table 4: Hyperparameter values of Transformers. "Sequence tagging" incorporates CoNLL-2003 & OntoNotes datasets, while "Classification" combines AG-News & IMDB. The hyperparameters are chosen according to evaluation scores on the validation datasets when models are trained using the whole available training data on CoNLL-2003 for sequence tagging & AG-News for classification.
Hparam	Sequence tagging	Classification
Number of epochs	15	5
Batch size	16	16
Min. number of training steps	1000	1000
Max. sequence length	-	256
Optimizer	AdamW	AdamW
Learning rate	5e-5	2e-5
Weight decay	0.01	0.01
Gradient clipping	1.	1.
Scheduler	STLR	STLR
% warm-up steps	10	10
A.1 Distillation Details for the Custom DistilELECTRA Model
The DistilELECTRA model is distilled from the ELECTRA-base model. It has the same architecture, but half as many layers initialized by taking from the teacher one layer out of two. Distillation is performed on the AG News dataset using a linear combination  $L = L_{ce} + L_{mlm} + L_{cos} + L_{mse}$  of the following loss functions as a training objective:  $L_{ce} = \sum_{i} t_{i} \cdot \log(s_{i})$  is a distillation loss of the student's probabilities  $s_{i}$  over the soft target probabilities of the teacher  $t_{i}$ ;  $L_{mlm}$  is the student's self-supervised masked language modeling loss;  $L_{cos}$  is the cosine embedding loss that aligns the directions of the student's and teacher's hidden state
Table 5: Transformers model checkpoints from HuggingFace repository (Wolf et al., 2019).
Dataset	Model	Checkpoint	#Param.
AG-News /IMDb	BERT	bert-base-uncased	110M
DistilBERT	distilbert-base-uncased	67M
ELECTRA	google/electra-base-discriminator	110M
DistilELECTRA	lsanochkin/distilectra-base	67M
XLNet	xlnet-base-cased	117M
RoBERTa	roberta-base	125M
DistilRoBERTa	distilroberta-base	82M
CoNLL-2003 /OntoNotes 5.0	ELECTRA	google/electra-base-discriminator	110M
BERT	bert-base-cased	110M
DistilBERT	distilbert-base-cased	67M
Table 6: Hyperparameter values of the CNN-BiLSTM-CRF model.
Hparam	CoNLL-2003
Word embeddings	GloVe (Pennington et al., 2014)
pre-trained model
Word embedding dim.	100
Char embedding dim.	30
CNN dim.	30
CNN filters	[2, 3]
RNN num. layers	1
RNN hidden size	200
RNN word dropout prob.	0.3
RNN locked dropout prob.	0.2
Encoder dropout prob	0.0
Feed forward num. layers	1
Feed forward hidden size	200
Feed forward activation	Tanh
Feed forward dropout prob.	0.0
Batch size	32
Learning rate	0.015
Momentum	0.9
Number of epochs	50
Optimizer	SGD
Gradients clipping	5
Table 7: Hyperparameter values of the CNN model for text classification on AG News.
Hparam	AG News
Word embeddings	Word2Vec (Mikolov et al., 2013)
pre-trained model
Word embedding dim.	300
CNN dim.	100
CNN filters	[3, 4, 5]
Dropout prob.	0.5
Batch size	128
Learning rate	0.001
Momentum	0.9
Number of epochs	20
Optimizer	SGD
Gradients clipping	1
vectors;  $L_{mse}$  is a mean squared error between student's and corresponding teacher's hidden states vectors. DistilELECTRA is trained with the following hyperparameters: 50 epochs, batch size 5, 50 gradient accumulation steps, AdamW optimizer with a learning rate  $5e - 4$ , epsilon  $1e - 6$ .
https://huggingface.co/models  
https://flair.informatik.hu-berlin.de/resources/embeddings-token/glove.  
gensim  
https://drive.google.com/file/d/0B7XkCwpI5KDYNlNUTT1SS21pQmM/edit?usp=sharing
B Additional Experimental Results with Acquisition-successor Mismatch
Figure 5: AL experiments on OntoNotes, in which a successor model does not match an acquisition model (DistilBERT).
a) ELECTRA is a successor model.
Figure 6: AL experiments on AG News, in which a successor model does not match an acquisition model (Distil-BERT).
a) RoBERTa is a successor model.
Figure 7: AL experiments on AG News, in which a successor model does not match an acquisition model (Distilled ELECTRA).
b) ELECTRA is a successor model.
b) RoBERTa is a successor model.
Figure 8: AL experiments on AG News, in which a successor model does not match an acquisition model (Distil-RoBERTa).
b) RoBERTa is a successor model.
Figure 9: AL experiments on IMDb, in which a successor model does not match an acquisition model (Distil-RoBERTa).
Figure 10: AL experiments on CoNLL-2003, in which a successor model does not match an acquisition model. This experiment demonstrates that models with similar expressiveness and size (BERT and ELECTRA) cannot be used interchangeably for acquisition in AL.
Figure 11: AL experiments on AG News with Mahalanobis distance used as an uncertainty measure in a query strategy. This experiment demonstrates that the acquisition-successor mismatch problem also persists for this modern uncertainty estimation technique.
a) DistilBERT is an acquisition model, BERT is a pseudo-labeling model, ELECTRA is a successor model.
b) DistilBERT is an acquisition model, BERT is a pseudo-labeling model, XLNet is a successor model.
c) DistilELECTRA is an acquisition model, ELECTRA is a pseudo-labeling model, RoBERTa is a successor model.
d) DistilRoBERTa as an acquisition model, RoBERTa as a pseudo-labeling model, ELECTRA is a successor model.
Figure 12: The performance of PLASM compared with the standard approach to AL on AG News with various acquisition - pseudo-labeling model pairs and successor models.
Figure 13: The performance of PLASM (BERT is a pseudo-labeling model) compared with the standard approach to AL on OntoNotes.
C Additional Experimental Results with PLASM
Figure 14: The performance of PLASM (RoBERTa is a pseudo-labeling model) compared with the standard approach to AL on IMDb.
Figure 15: Experiments with PLASM and standard approaches on CoNLL-2003, in which CNN-BiLSTM-CRF is used as a successor model. We can see that due to using PLASM and the expressiveness of the pseudo-labeling model (BERT), the successor achieves substantial improvements over the baseline.
Figure 16: Experiments with PLASM and standard approaches on AGNews, in which simple CNN is used as a successor model. We can see that due to using PLASM and the expressiveness of the pseudo-labeling model (BERT), the successor achieves substantial improvements over the baseline. We also note that using AL with DistilBERT as an acquisition model results in worse performance than using the baseline random sampling; this corresponds to findings of (Lowell et al., 2019).
a) DistilBERT for pseudo-labeling.
b) ELECTRA for pseudo-labeling.
Figure 17: Ablation studies of PLASM on the CoNLL-2003 dataset, in which an inappropriate model is used for pseudo-labeling.
a) ELECTRA as a successor.
Figure 18: Ablation studies of filtering methods in PLASM on the AG News dataset.
b) XLNet as a successor.
D Additional Experimental Results with UPS
a) AG News dataset.
b) CoNLL-2003 dataset.
Figure 19: The performance of UPS compared with the standard approach to AL on AG News and CoNLL-2003 datasets with ELECTRA as a successor model ( $\gamma = 0.1$ ,  $T = 0.01$ ).
Figure 20: The performance of UPS in conjunction with PLASM (BERT is a pseudo-labeling model) on CoNLL-2003 compared with baselines  $(\gamma = 0.1, T = 0.01)$ .
Figure 21: The comparison of UPS with a random-subsampling baseline on the AG News dataset ( $\gamma = 0.1$ ,  $T = 0.01$ ). A pseudo-labeling model in PLASM is BERT.
Figure 22: Ablation study for the parameter  $T$  in the UPS algorithm. We see that when sampling a total of  $5\%$  of the dataset to select for the query, using a non-zero value for the parameter  $T$  gives an increase in performance compared to a case when only  $5\%$  most uncertain samples are considered for query (i.e.  $T = 0$ ).
ELECTRA	BERT	DistilBERT	ELECTRA with UPS (ours)	DistilBERT with UPS (ours)
Iter. 2	Train	44.8 ± 0.3	50.9 ± 1.6	29.1 ± 0.3	43.3 ± 0.8	26.4 ± 2.5
Inference	25.9 ± 0.3	25.9 ± 0.3	19.6 ± 0.3	25.7 ± 0.4	19.9 ± 0.9
Overall	70.6 ± 0.6	76.8 ± 1.7	48.7 ± 0.5	69.0 ± 1.0	46.3 ± 3.1
Iter. 6	Train	74.9 ± 1.6	81.4 ± 1.4	49.7 ± 1.3	66.9 ± 1.6	44.2 ± 4.0
Inference	23.8 ± 0.0	23.4 ± 0.3	17.9 ± 0.0	3.2±0.2	2.3±0.2
Overall	98.6 ± 1.5	104.8 ± 1.1	67.5 ± 1.4	70.1±1.6	46.5±4.2
Iter. 10	Train	95.6 ± 1.1	105.7 ± 1.5	63.6 ± 2.0	88.4 ± 1.2	57.1 ± 5.5
Inference	21.3 ± 0.1	21.4 ± 0.2	15.9 ± 0.5	2.6±0.2	2.3±0.1
Overall	116.9 ± 1.2	127.1 ± 1.5	79.5 ± 2.4	91.0±1.3	59.4±5.6
Iter. 15	Train	122.2 ± 1.2	133.4 ± 3.1	79.0 ± 1.3	129.9 ± 3.2	74.6 ± 6.4
Inference	18.9 ± 0.2	18.6 ± 0.1	14.0 ± 0.2	2.0±0.1	1.4±0.1
Overall	141.1 ± 1.0	151.9 ± 3.2	92.9 ± 1.2	131.9±3.1	76.0±6.5
Overall train	1266.6 ± 16.9	1387.1 ± 26.3	838.6 ± 19.2	1195.0 ± 25.0	748.3 ± 70.4
Overall inference	339.1 ± 3.5	335.5 ± 4.7	252.9 ± 3.9	128.9±5.6	97.5±5.1
Overall	1605.7 ± 18.8	1722.6 ± 24.1	1091.4 ± 18.4	1323.9±28.5	845.8±75.1
Table 8: Duration of training and inference steps of AL iterations in seconds on CoNLL-2003. We highlight with the bold font the values affected by UPS. Hardware configuration: 2 Intel Xeon Platinum 8168, 2.7 GHz, 24 cores CPU; NVIDIA Tesla v100 GPU with 32 Gb of VRAM.