SlowGuess/NAACL_2022_OCR huggingface dataset:
Akim Tsvigun $^{1,3\diamond}$ , Artem Shelmanov $^{1,6\diamond}$ , Gleb Kuzmin $^{1,5}$ , Leonid Sanochkin $^{1}$ , Daniil Larionov $^{2,5}$ , Gleb Gusev $^{1,2,4}$ , Manvel Avetisian $^{1,4}$ , and Leonid Zhukov $^{1,3}$
$^{1}$ AIRI,  $^{2}$ MIPT,  $^{3}$ HSE,  $^{4}$ Sber AI Lab,  $^{5}$ FRC CSC RAS,
$^{6}$ ISP RAS Research Center for Trusted Artificial Intelligence
{tsvigun, shelmanov, kuzmin, sanochkin, gusev, manvel, zhukov} @airi.net dslarionov@isa.ru
Active learning (AL) is a prominent technique for reducing the annotation effort required for training machine learning models. Deep learning offers a solution for several essential obstacles to deploying AL in practice but introduces many others. One of such problems is the excessive computational resources required to train an acquisition model and estimate its uncertainty on instances in the unlabeled pool. We propose two techniques that tackle this issue for text classification and tagging tasks, offering a substantial reduction of AL iteration duration and the computational overhead introduced by deep acquisition models in AL. We also demonstrate that our algorithm that leverages pseudo-labeling and distilled models overcomes one of the essential obstacles revealed previously in the literature. Namely, it was shown that due to differences between an acquisition model used to select instances during AL and a successor model trained on the labeled data, the benefits of AL can diminish. We show that our algorithm, despite using a smaller and faster acquisition model, is capable of training a more expressive successor model with higher performance. $^{1}$
Active learning (AL) (Cohn et al., 1996) is an approach for reducing the amount of dataset annotation required for achieving the desired level of machine learning model performance. This is especially important in domains where obtaining labeled instances is expensive or wide crowdsourcing is unavailable. For example, annotation of clinical and biomedical texts usually requires the help of physicians or biomedical researchers. The time of such highly qualified experts is extremely valuable
and should be spent wisely. Straightforward annotation of datasets can be very redundant, wasting the time of annotators on unimportant instances. AL alleviates this problem by asking human experts to label only the most informative instances selected according to the information acquired from a machine learning model. The algorithm for selection of such instances is called a query strategy, and a model used to estimate the informativeness of yet unlabeled instances is called an acquisition model.
AL starts from a small seeding set of labeled instances, which are used to train an initial acquisition model. A query strategy ranks unlabeled instances in a large pool according to a criterion that measures their informativeness based on the acquisition model output. One of the most widely adopted criteria is the uncertainty of the acquisition model on instances in question (Lewis and Gale, 1994). Eventually, top selected instances are presented to annotators, and this active annotation process iteratively continues.
After labels are collected, we would like to train a model for a final application. In the same vein as (Lowell et al., 2019), we call it a successor model. AL can help reduce the amount of annotation required to achieve a reasonable quality of the successor text processing model by multiple times (Settles and Craven, 2008; Settles, 2009).
Recently, deep learning has given us a tool for solving one of the essential problems of AL. When we start annotating, we have to build an acquisition model almost without insights from the data that could help us to do feature engineering or to introduce inductive bias. Deep learning does not require feature engineering and transfer learning with deep pre-trained models like ELMo (Peters et al., 2018), BERT (Devlin et al., 2019), and followers such as ELECTRA (Clark et al., 2020) provide near state-of-the-art performance on a variety of tasks without any modifications to their architectures. However, deep learning introduces another problem related
1The code for reproducing the experiments is available at https://github.com/AIRI-Institute/al_nlpfeasible
$\diamond$  Equal contribution, corresponding authors
Findings of the Association for Computational Linguistics: NAACL 2022, pages 1198 - 1218 July 10-15, 2022 ©2022 Association for Computational Linguistics
to computational performance. Since AL annotation typically is an interactive process, we have to train acquisition models and perform inference on a huge unlabeled pool of instances very quickly. This imposes constraints on the acquisition model size and entails another issue.
Ideally, the architectures of acquisition and successor models should be the same. Lowell et al. (2019) demonstrate that when the acquisition model is different from the successor model, the performance of the latter one can degrade compared to the performance of the model trained on the same amount of annotation obtained without AL. The performance drop in the case of acquisition-successor mismatch (ASM) raises the question of whether AL is a practical technique at all since the usage of different models on the annotated dataset is a common practice. The problem is complicated by a contradiction between the fact that the acquisition model is required to be as lightweight as possible to mitigate computational overhead and the successor model should be as expressive as possible because we apparently care about the quality of our final application.
In this work, we propose a simple algorithm based on pseudo-labeling and demonstrate that it is able to alleviate the ASM problem. Moreover, we show that it is possible to substitute a resource-intensive acquisition model with a smaller one (e.g., take DistilBERT instead of BERT) but train a more powerful successor model of an arbitrary type (e.g., ELECTRA) without loss of quality. This helps to accelerate the execution of AL iterations and reduce computational overhead.
We also find that the most time-consuming part of an AL iteration with uncertainty-based query strategies can be the inference on the unlabeled pool of instances, while a set of the most certain instances usually does not change substantially from iteration to iteration. Therefore, the straightforward approach to instance acquisition wastes much time on instances shown to be unimportant in previous iterations. We leverage this finding and propose an algorithm that subsamples instances in the unlabeled pool depending on their uncertainty scores obtained on previous AL iterations. This helps to speed up the AL iterations further, especially when the unlabeled pool is large. A series of experiments on text classification and tagging benchmarks widely used in recent works on AL demonstrate the efficiency of the proposed algorithms.
The contributions of the paper are the following:
- We propose a novel algorithm denoted as Pseudo-Labeling for Acquisition Successor Mismatch (PLASM) that allows the use of computationally cheap models during the acquisition of instances in AL, while it does not introduce constraints on the type of the successor model and effectively alleviates the ASM problem. It helps to reduce the hardware requirements and the duration of AL iterations.
- We propose a novel algorithm denoted as Unlabeled Pool Subsampling (UPS) that helps to reduce the time required for calculating informativeness of instances in AL based on the fact that the set of instances that model is certain about does not change substantially. This helps to further speed up the AL iteration.
Deep learning, to a large extent, has freed data scientists from doing feature engineering, which has been one of the essential obstacles to annotation with AL. This advantage has sparked a series of works on deep active learning (DAL) in natural language processing (NLP).
Shen et al. (2017) conduct one of the first investigations on DAL in sequence tagging tasks. They propose an efficient way of quantifying the uncertainty of sentences, namely maximal normalized log probability (MNLP), by averaging log probabilities of their tokens. They also address the problem of excessive duration of a neural network training step during an AL iteration by interleaving online learning with training from scratch. In our work, we take MNLP as a query strategy for experiments on sequence tagging tasks since it has demonstrated a good trade-off between quality and computational performance. We consider that online learning can potentially be used as a complement to our algorithms. Since the most time-consuming part of an AL iteration can be model inference instead of training, in this work, we also pay attention to the acceleration of the inference step.
Several recent publications investigate deep pretrained models based on the Transformer architecture (Vaswani et al., 2017), ELMo (Peters et al., 2018), and ULMFiT (Howard and Ruder, 2018) in AL on NLP tasks (Prabhu et al., 2019; Ein-Dor et al., 2020; Yuan et al., 2020; Shelmanov et al., 2021). We continue this line of works by relying on pre-trained Transformers since this architecture
has been shown promising for AL in NLP due to its good qualitative and computational performance.
A few works have experimented with Bayesian query strategies for AL. Shen et al. (2017), Siddhant and Lipton (2018), Ein-Dor et al. (2020), and Shelmanov et al. (2021) leverage Monte Carlo dropout (Gal and Ghahramani, 2016) for quantifying uncertainty of models. Siddhant and Lipton (2018) also apply the Bayes by backprop algorithm (Blundell et al., 2015) for performing variational inference of a Bayesian neural network. This approach demonstrates the best improvements upon the baseline but introduces large computational overhead both for training and uncertainty estimation of a model, as well as the memory overhead for storing parameters of a Bayesian neural network. The query strategies based on Monte Carlo dropout do not affect the model training procedure and do not change the memory footprint. However, they also suffer from slow uncertainty estimation due to the necessity of making multiple stochastic predictions, while their empirical evaluations with Transformers in recent works (Ein-Dor et al., 2020; Shelmanov et al., 2021) do not demonstrate big advantages. Therefore, we do not use Bayesian query strategies in our experiments and adhere to the classical uncertainty-based query strategies.
Recently proposed alternatives to uncertainty-based query strategies leverage reinforcement learning and imitation learning (Fang et al., 2017; Liu et al., 2018; Vu et al., 2019; Brantley et al., 2020). This series of works aims at constructing trainable policy-based query strategies. However, this requires an excessive amount of computation while the transferability of learned policies across domains and tasks is underresearched.
Finally, Lowell et al. (2019) question the usefulness of AL techniques in general. They demonstrate that due to the ASM problem, AL can be even detrimental to the performance of the successor. This finding is also revealed for classical machine learning models by Baldridge and Osborne (2004), Tomanek and Morik (2011), Hu et al. (2016) and supported by experiments with Transformers in (Shelmanov et al., 2021). Our work directly addresses the question raised by Lowell et al. (2019) and suggests a simple solution to the ASM problem. Moreover, we combine it with the method proposed by Shelmanov et al. (2021), who suggest using distilled models for instance acquisition and their teacher models as successors.
This section describes models and AL query strategies used in this work.
3.1 Query Strategies
We conduct experiments with four basic AL query strategies. We note that despite their simplicity, these strategies are usually on par with more elaborated counterparts (Ein-Dor et al., 2020; Shelmanov et al., 2021; Margatina et al., 2021).
Random sampling is used for both text classification and sequence tagging experiments. Applying this strategy means that we do not use AL at all and just emulate that an annotator labels a randomly sampled piece of a dataset.
Least Confident (LC) is used for text classification experiments. This strategy sorts texts in the ascending order of their maximum class probabilities given by a machine learning model. Let  $y$  be a predicted class of an instance  $x$ , then  $LC_{cls}$  is:
$$
\operatorname {L C} _ {\mathrm {c l s}} = 1 - \max  _ {y} \mathbb {P} \left(y | x\right).
$$
Maximum Normalized Log-Probability (MNLP) is proposed by Shen et al. (2017) to mitigate the drawback of the standard LC when it is applied to sequence tagging tasks. Let  $y_{i}$  be a tag of a token  $i$ , let  $x_{j}$  be a token  $j$  in an input sequence of length  $n$ . The MNLP score can be formulated as follows:
$$
\mathrm{MNLP}_{\text{ner}} = -\max_{y_{1},\dots ,y_{n}}\frac{1}{n}\sum_{i = 1}^{n}\log \mathbb{P}\left[y_{i}|\left\{y_{j}\right\} \backslash y_{i},\left\{\mathbf{x}_{j}\right\} \right].
$$
This modified version of LC works slightly better for sequence tagging tasks (Shen et al., 2017), and is adopted in many other works on DAL (Siddhant and Lipton, 2018; Erdmann et al., 2019; Shelmanov et al., 2021).
Mahalanobis Distance (MD) between a test instance and the closest class-conditional Gaussian distribution is suggested by Lee et al. (2018) for detection of out-of-distribution instances and adversarial attacks. MD is a strong baseline for uncertainty estimation of NLP model predictions (Podolskiy et al., 2021) and is also a backbone for other subsequent techniques (Zhou et al., 2021). We use it as an informativeness score in AL since previous work shows that MD captures epistemic uncertainty well (Podolskiy et al., 2021).
$$
\mathrm {M D} _ {\mathrm {c l s}} = \min  _ {c \in C} \left(h _ {i} - \mu_ {c}\right) ^ {T} \Sigma^ {- 1} \left(h _ {i} - \mu_ {c}\right), \tag {1}
$$
where  $h_i$  is a hidden representation of a  $i$ -th instance,  $\mu_c$  is a centroid of class  $c$ , and  $\Sigma$  is a covariance matrix for hidden representations of training instances.
We use the standard models based on the Transformer architecture (Vaswani et al., 2017): BERT, RoBERTa (Liu et al., 2019), ELECTRA, and XLNet (Yang et al., 2019). For supplementary experiments, we also employ two classical neural models: a CNN-BiLSTM-CRF sequence tagging model (Ma and Hovy, 2016) and a CNN-based text classification model (Le et al., 2018).
Besides full-fledged Transformers, we leverage their three smaller distilled versions: DistilBERT (Sanh et al., 2019), DistilRoBERTa, and a custom DistilELECTRA trained by ourselves. The distillation procedure aims at creating a smaller-size model (student) while keeping the behavior of the original model (teacher) by minimizing the distillation loss over the student predictions and soft target probabilities of the teacher (Hinton et al., 2015):  $L_{distil} = -\sum_{i,c} t_{ic} \cdot \log(s_{ic})$ , where  $t_{ic}$  and  $s_{ic}$  are probabilities estimated by the teacher and the student correspondingly for each instance  $i$  and class  $c$ . Typically, distillation loss is supplemented with additional techniques that help to align a student with a teacher (Sanh et al., 2019).
Distilled models are usually much more compact than their teachers. For example, DistilBERT reduces the memory footprint by  $40\%$  compared to the original BERT-base. It achieves the  $60\%$  speedup, sacrificing only  $3\%$  of its qualitative performance (Sanh et al., 2019). Since the qualitative performance during acquisition is not essential, we would like to use such lightweight models for instance acquisition to reduce AL iteration duration and the requirements for the computational power of the hardware.
This section outlines two proposed algorithms that help to reduce the computational cost of AL.
4.1 Pseudo-labeling for Acquisition-Successor Mismatch
We propose a simple algorithm for constructing a successor model of an arbitrary type using AL: Pseudo-Labeling for Acquisition-Successor Mismatch (PLASM). The algorithm is designed for
reducing the amount of computation required for instance acquisition during AL with uncertainty-based query strategies.
PLASM leverages the finding of Shelmanov et al. (2021) that the successor model can be trained on instances labeled during AL without a penalty to the quality if its distilled version was used for instance acquisition. However, this idea alone does not resolve the question, how we can train new models of arbitrary type on datasets collected via AL (Lowell et al., 2019).
The algorithm consists of the following steps:
1. Consider we have a resource-intensive pretrained teacher model (e.g. BERT). We construct a lightweight distilled version of this model (e.g. DistilBERT) using unlabeled data.
2. We apply a distilled model to perform acquisition during AL for collecting the gold labels.
3. The collected labels are used for fine-tuning a resource-intensive teacher model of a higher quality than the distilled acquisition model.
4. The teacher model is used for pseudo-labeling of the whole unlabeled pool of instances.
5. The automatically acquired annotations are filtered to reduce noise introduced by mistakes of the pseudo-labeling model. In the main experiments, we use TracIn – a strong and practical method for mislabelled data identification (Pruthi et al., 2020). In the ablation study, we also test a simpler solution: filtering instances with high uncertainty of the pseudo-labeling model predictions. The fraction of the filtered out instances in both cases is determined from the evaluation score of the pseudo-labeling model on a held-out subset of the training corpus (100%-score).
6. Finally, we train a successor model of an arbitrary type on the dataset that contains automatically labeled instances and instances with gold labels obtained from human experts.
If the teacher model is expressive enough, it will generate reasonable pseudo labels, which can be filtered and reused by another model of a different type and architecture. This additional annotation helps to mitigate the performance drop due to ASM and to keep the benefits of AL even when the successor model is more expressive than the model used for pseudo-labeling. Meanwhile, PLASM helps to reduce the duration of AL iterations similarly to the approach of Shelmanov et al. (2021),
and it does not introduce any additional computational overhead during the annotation process since training the teacher model and pseudo-labeling are performed after the AL annotation is completed.
4.2 Unlabeled Pool Subsampling
If the unlabeled pool of instances is large, which is a common situation, and a deep neural network is used as an acquisition model, the most time-consuming step of the AL cycle is the generation of predictions for unlabeled instances, which is necessary for uncertainty-based query strategies (refer to Table 2). We note that uncertainty estimates of the most certain instances in the unlabeled pool do not alter substantially across multiple AL iterations (Table 1). This means that AL wastes much time and resources on these unimportant instances. We claim that it is possible to recalculate uncertainty scores on the current iteration only for the top instances of the unlabeled pool, which were the most uncertain on previous iterations, while not sacrificing the benefits of AL.
We propose an unlabeled pool subsampling (UPS) algorithm, in which uncertainty estimates only for a fraction of instances are updated. On the current iteration, we suggest always selecting a fraction of the most uncertain instances on previous iterations equal to  $\gamma \in [0,1]$  and sample a small portion of instances with a probability that depends on their rank in a list sorted by their uncertainty. Formally, this can be written as follows. Let  $u$  be the last recalculated uncertainty score of an instance on one of the previous iterations. We order the instances according to this value:  $u_0 \leq u_1 \leq \dots \leq u_i \leq \dots \leq u_M$  and denote a normalized rank of an instance as  $r_i = \frac{i}{M}$ . Let  $T > 0$  be a "temperature" hyperparameter. Then the probability of keeping an instance  $i$  for recalculation of uncertainty on the current iteration is:
$$
\mathbb {P} (i) \propto \exp \left(- \frac {\operatorname* {m a x} (0 , r _ {i} - \gamma)}{T}\right).
$$
Sampling certain instances with a non-negative probability instead of just ignoring them gives a chance of overcoming a situation when an informative instance is occasionally assigned a high certainty score and is never selected ever since. This method is inspired by subsampling techniques used in gradient boosting algorithms for selecting a training subset for decision trees (Ke et al., 2017; Ibragimov and Gusev, 2019).
On initial AL iterations, an acquisition model is trained on an extremely small amount of data, which leads to unreliable uncertainty estimates. To mitigate this problem, we suggest keeping the standard approach to performing instance acquisition on several first iterations and switching to the optimized process later during AL. We also note that interleaving the optimized selection with the standard approach, in which we recalculate the uncertainty for the whole unlabeled pool of instances, can help to keep the high performance of AL.
We follow the common schema of AL experiments adopted in many previous works (Settles and Craven, 2008; Shen et al., 2017; Siddhant and Lipton, 2018; Shelmanov et al., 2021). We emulate the AL annotation cycle starting with a small random sample of the dataset used as a seed for the construction of the initial acquisition model. On each iteration, we pick a fraction of top instances from the unlabeled pool sorted using the query strategy and, instead of demonstrating them to annotators, automatically label them according to the gold standard. These instances are removed from the unlabeled pool and added to the training dataset for the next iterations. On each iteration, we train the successor model on the data acquired so far and evaluate it on the whole available test set. Acquisition and successor models are always trained from scratch. We run several iterations of emulation to build a chart, which demonstrates the performance of the successor depending on the amount of "labor" invested into the annotation process. To report standard deviations of scores, we repeat the whole experiment five times with different random seeds. In most experiments, we use LC or MNLP query strategies for classification and sequence tagging correspondingly. Results with MD are presented only in Figure 11 in Appendix B.
For classification, accuracy is used as the evaluation metric. For sequence tagging, we use the strict span-based F1-score (Sang and Meulder, 2003).
We experiment with widely-used datasets for the evaluation of AL methods on text classification and sequence tagging tasks.
For text classification, we use the English AG News topic classification dataset (Zhang et al.,
a) ELECTRA is a successor model.
b) BERT is a successor model.
2015) and the binary sentiment classification IMDb dataset (Maas et al., 2011). We randomly select  $1\%$  of instances of the training set as a seed to train the initial acquisition model and select  $1\%$  of instances for "annotation" on each AL iteration.
For sequence tagging, we use English CoNLL2003 (Sang and Meulder, 2003) and English OntoNotes 5.0 (Pradhan et al., 2013). We randomly sample instances with a total number of tokens equal to  $2\%$  of all tokens from the training set as a seed. On each AL iteration, we select instances from the unlabeled pool until a total number of tokens equals  $2\%$  of all training tokens.
The corpora statistics are presented in Table 3 in Appendix A.
5.1.2 Model Choice, Training Details, and Hyperparameter Selection
We conduct experiments with pre-trained Transformers used in several previous works on AL. The exact checkpoints and parameter numbers are presented in Table 5 in Appendix A. Section A.1 contains the distillation details of the custom Distile ELECTRA model.