SlowGuess/NAACL_2022_OCR huggingface dataset:
Shujian Zhang $^{1,2}$ , Chengyue Gong $^{1}$ , Xingchao Liu $^{1}$ , Pengcheng He $^{2}$ , Weizhu Chen $^{2}$ , Mingyuan Zhou $^{1}$
<sup>1</sup>The University of Texas at Austin 2Microsoft Azure AI
{szhang42, cygong, xcliu}@utexas.edu
Active learning, which effectively collects informative unlabeled data for annotation, reduces the demand for labeled data. In this work, we propose to retrieve unlabeled samples with a local sensitivity and hardness-aware acquisition function. The proposed method generates data copies through local perturbations and selects data points whose predictive likelihoods diverge the most from their copies. We further empower our acquisition function by injecting the select-worst case perturbation. Our method achieves consistent gains over the commonly used active learning strategies in various classification tasks. Furthermore, we observe consistent improvements over the baselines on the study of prompt selection in prompt-based few-shot learning. These experiments demonstrate that our acquisition guided by local sensitivity and hardness can be effective and beneficial for many NLP tasks.
Crowdsourcing annotations (Rajpurkar et al., 2016; Bowman et al., 2015) has become a common practice for developing NLP benchmark datasets. Rich prior works (Pavlick and Kwiatkowski, 2019; Nie et al., 2020; Ferracane et al., 2021) show that the time-consuming and expensive manual labeling in crowdsourcing annotations are not an annotation artifact but rather core linguistic phenomena. Active Learning (AL) is introduced to efficiently acquire data for annotation from a (typically large) pool of unlabeled data. Its goal is to concentrate the human labeling effort on the most informative data in hopes of maximizing the model performance while minimizing the data annotation cost.
Popular approaches to acquiring data for AL are uncertainty sampling and diversity sampling. Uncertainty sampling selects data that the model predicts with low-confidence (Lewis and Gale, 1994;
Culotta and McCallum, 2005; Settles, 2009). Diversity sampling selects batches of unlabeled examples that are prototypical of the unlabeled pool to exploit heterogeneity in the feature space (Xu et al., 2003; Bodó et al., 2011). Different from these two perspectives, recent works focus on the informativeness of the selected data. For example, Zhang and Plank (2021) acquire informative unlabeled data using the training dynamics based on the model predictive log likelihood. Margatina et al. (2021) construct contrastive examples in the input feature space. However, these methods either ignore the local sensitivity of the input features or take no consideration of the difficulty of the learning data. Consequently, they may ignore examples around the decision boundary, or select hard-to-train or even noisy examples. Their performance may further suffer under some practical settings, such as those with imbalanced labels and when there is a very limited annotation budget.
In this work, we determine the informativeness by considering both the local sensitivity and learning difficulty. For local sensitivity, we take the classical definition from Chapelle et al. (2009), which is widely used in both classic machine learning problems (e.g. Blum and Chawla, 2001; Chapelle et al., 2002; Seeger, 2000; Zhu et al., 2003; Zhou et al., 2004) and recent deep learning settings (e.g. Wang et al., 2018b; Sohn et al., 2020; Xu et al., 2021). Specifying a local region  $\mathcal{R}_{region}(\pmb{x})$  around an example  $\pmb{x}$ , we assume in our prior that all examples in  $\mathcal{R}_{region}(\pmb{x})$  have the same labels. If the examples in  $\mathcal{R}_{region}(\pmb{x})$  give us different labels, we say the local region of  $\pmb{x}$  is sensitive. Data augmentation has been chosen as the way to create label-equivalent local regions in many recent works (e.g., Berthelot et al., 2019b; Xie et al., 2020). We utilize data augmentation as a tool to capture the local sensitivity and hardness of inputs and present
See the paragraph 'unlabeled bias as regions' and the section 'Regions and Smoothness' for details.
Code is available at https://github.com/szhang42/allsh
Findings of the Association for Computational Linguistics: NAACL 2022, pages 1328 - 1342 July 10-15, 2022 ©2022 Association for Computational Linguistics
ALLSH: Active Learning guided by Local Sensitivity and Hardness. Through various designs on local perturbations, ALLSH selects unlabeled data points from the pool whose predictive likelihoods diverge the most from their augmented copies. This way, ALLSH can effectively ensure the informative and local-sensitive data to have correct human-annotated labels. Figure 1 illustrates the scheme of the proposed acquisition strategy.
We conduct a comprehensive evaluation of our approach on datasets ranging from sentiment analysis, topic classification, natural language inference, to paraphrase detection. To measure the proposed acquisition function in more realistic settings where the samples stem from a dissimilar input distribution, we (1) set up an out-of-domain test dataset and (2) leak out-of-domain data (e.g., adversarial perturbations) into the selection pool.
We further expand the proposed acquisition to a more challenging setting: prompt-based few-shot learning (Zhao et al., 2021), where we query a fixed pre-trained language model via a natural language prompt containing a few training examples. We focus on selecting the most valuable prompts for a given test task (e.g., selecting 4 prompts for one given dataset). We adapt our acquisition function to retrieve prompts for the GPT-2 model.
Furthermore, we provide extensive ablation studies on different design choices for the acquisition function, including the designs of augmentations and divergences. Our method shows consistent gains in all settings with multiple datasets. With little modification, our data acquisition can be easily applied to other NLP tasks for a better sample selection strategy.
Our contributions are summarized as follows: (1) Present a new acquisition strategy, embracing local sensitivity and learning difficulty, such as paraphrasing the inputs through data augmentation and adversarial perturbations, into the selection procedure. (2) Verify the effectiveness and general applicability of the proposed method in more practical settings with imbalanced datasets and extremely few labeled data. (3) Provide comprehensive study and experiments of the proposed selection criteria in classification tasks (both in-domain and out-of-domain evaluations) and prompt-based few-shot learning. (4) The proposed data sampling strategy can be easily incorporated or extended to many other NLP tasks.
In this section we present in detail our proposed method, ALLSH (Algorithm 1).
2.1 Active Learning Loop
The active learning setup consists of an unlabeled dataset  $\mathcal{D}_{pool}$ , the current training set  $\mathcal{D}_{label}$ , and a model  $\mathcal{M}$  whose output probability is  $p_{\theta}(\cdot \mid x)$  for input  $\mathbf{x}$ . The model  $\mathcal{M}$  is generally a pre-trained model for NLP tasks (Lowell et al., 2018). At each iteration, we train a model on  $\mathcal{D}_{label}$  and then use the acquisition function to acquire  $s_{acq}$  sentences in a batch  $\mathcal{T}$  from  $\mathcal{D}_{pool}$ . The acquired examples from this iteration are labeled, added to  $\mathcal{D}_{label}$ , and removed from  $\mathcal{D}_{pool}$ . Then the updated  $\mathcal{D}_{label}$  serves as the training set in the next AL iteration until we exhaust the budget. Overall, the system is given a budget of  $S$  queries to build a labeled training dataset of size  $S$ .
2.2 Acquisition Function Design
To fully capture the data informativeness and train a model with a limited amount of data, we consider two data-selection principals: local sensitivity and learning hardness.
Local Sensitivity Based on theoretical works on the margin theory for active learning, the examples lying close to the decision boundary are informative and worth labeling (Ducoffe and Precioso, 2018; Margatina et al., 2021). Uncertainty sampling suffers from the sampling bias problem as the model is only trained with few examples in the early phase of training. In addition, high uncertainty samples given the current model state may not be that representative to the whole unlabeled data (Ru et al., 2020). For example, if an input has high confidence while its local perturbation generates low-confidence output, then it is likely that this input lies close to the model decision boundary. This information can be captured by measuring the difference between an input example and its augmentation in the output feature space. We utilize the back-translation (Sennrich et al., 2016; Edunov et al., 2018; Zhang et al., 2021b) and TF-IDF (Xie et al., 2020) as effective augmentation methods which can generate diverse paraphrases while preserving the semantics of the original inputs (Yu et al., 2018b).
Instead of simply using augmentation, adversarial perturbation can measure the local Lipschitz and sensitivity more effectively. We therefore further
exploit adversarial perturbation to more accurately measure local sensitivity. For NLP problems, generating exact adversarial perturbations in a discrete space usually requires combinatorial optimization, which often suffers from the curse of dimensionality (Madry et al., 2017; Lei et al., 2018). Hence, we choose the hardest augmentation over  $K$  random augmentations as a "lightweight" variant of adversarial input augmentation which optimizes the worst case loss over the augmented data.
Learning Hardness: From Easy to Hard Learning from easy examples or propagating labels from high-confidence examples is the key principle for curriculum learning (Bengio et al., 2009) and label propagation based semi-supervised learning algorithms (Chapelle et al., 2009). For example, FixMatch (Sohn et al., 2020), a SOTA semi-supervised method, applies an indicator function to select high confident examples at each iteration. This will facilitate the label information from high confidence examples to low-confidence ones (Chapelle et al., 2009). In our selection criterion, as the model is trained with limited data, we also want to avoid the hard-to-learn examples, which in some cases frequently correspond to unlabeled or erroneous instances (Swayamdipta et al., 2020; Zhang and Plank, 2021). These examples may stuck the model performance at the beginning of the selection.
2.3 Acquisition with Local Sensitivity and Hardness
We come to the definition of our acquisition function. Given a model  $p_{\theta}$  and an input  $\mathbf{x}$ , we compute the output distribution  $p_{\theta}(\cdot \mid \mathbf{x})$  and a noised version  $p_{\theta}(\cdot \mid \mathbf{x}')$  by injecting a random transformation  $\mathbf{x}' = g(\mathbf{x})$  to the inputs. Here,  $g(\cdot)$  is sampled from a family of transformations and these random transformations stand for data augmentations. This procedure can select examples that are insensitive to transformation  $g(\cdot)$  and hence smoother with respect to the changes in the input space (Berthelot et al., 2019b,a; Sohn et al., 2020). We calculate
$$
\ell (\boldsymbol {x}, \boldsymbol {x} ^ {\prime}) = \mathbb {D} \left(p _ {\theta} (\cdot \mid \boldsymbol {x}), p _ {\theta} (\cdot \mid \boldsymbol {x} ^ {\prime})\right), \tag {1}
$$
where  $\mathbb{D}$  denotes a statistical distance such as the Kullback-Leibler (KL) divergence (Kullback and Leibler, 1951). Model  $p_{\theta}$  here can be a pretrained language model such as BERT (Devlin et al., 2018).
Data Paraphrasing via Augmentation Paraphrase generation can improve language models (Yu et al., 2018a) by handling language varia
Figure 1: Overview of active learning framework guided by local sensitivity and hardness. Some notations are labeled along with corresponding components. 'Select' refers to the select worst-case augmentation.
tion. TF-IDF and backtranslation can generate diverse inputs while preserving the semantic meaning (Singh et al., 2019; Xie et al., 2020). For TF-IDF, we replace uninformative words with low TF-IDF scores while keeping those with high. Specifically, Suppose  $\mathrm{IDF}(w)$  is the IDF score for word  $w$  computed on the whole corpus, and  $\mathrm{TF}(w)$  is the TF score for word  $w$  in a sentence. We compute the TF-IDF score as  $\mathrm{TFIDF}(w) = \mathrm{TF}(w)\mathrm{IDF}(w)$ . For backtranslation, we use a pre-trained EN-DE and DE-EN translation models (Ng et al., 2019) to perform backtranslation on each sentence. We denote  $\pmb{x}$  as  $(x_0,\dots ,x_n)$ . Here,  $n$  denotes the original length of the input. For  $\pmb{x}$ , we pass them through two translation models to get  $\pmb{x}^{\prime} = (x_{0}^{\prime},\dots ,x_{m}^{\prime})$ , where  $m$  denotes the length after backtranslating. More details can be found in Appendix A.
Select Worst-Case Augmentation (WCA) In order to construct effective local sensitivity, the most direct approach is calculating the local Lipschitz constant or finding the worst case adversarial perturbation. However, estimating the Lipschitz constant for a neural network is either model dependent or computationally hard (Scaman and Virmaux, 2018; Fazlyab et al., 2019). Instead, we select worst-case augmentation over  $K$  copies, which can still roughly measure the norm of the first-order gradient without a huge computation cost and is easy to implement. Given input examples  $\pmb{x}$ , and  $K$  augmentation of  $\pmb{x}$  as  $\{x_{i}^{\prime}\}_{i = 1}^{K}$ , we propose the following acquisition function to select data:
$$
\ell^ {\max } (\boldsymbol {x}) = \max  _ {i \in [ K ]} \ell \left(\boldsymbol {x}, \boldsymbol {x} _ {i} ^ {\prime}\right). \tag {2}
$$
Inspired by some simple and informal analysis in continuous space, we draw the connection between calculating  $\ell^{max}(\pmb{x})$  and local sensitivity by
Figure 2: The solid line is model decision boundary. Orange circles refer to the unlabeled data and green circles refer to the corresponding augmentation of the orange unlabeled data.
$$
\begin{array}{l} \ell^ {\max } (\boldsymbol {x}) = \ell \left(\boldsymbol {x}, \boldsymbol {x} ^ {\prime}\right) + \left[ \ell^ {\max } (\boldsymbol {x}) - \ell \left(\boldsymbol {x}, \boldsymbol {x} ^ {\prime}\right) \right] \tag {3} \\ = \boldsymbol {\ell} (\pmb {x}, \pmb {x} ^ {\prime}) + \left[ \max _ {i \in [ K ]} \langle \nabla_ {\pmb {x}} \boldsymbol {\ell} (\pmb {x}, \pmb {x} ^ {\prime}), \pmb {x} - \pmb {x} ^ {\prime} \rangle \right] + \mathbf {O} (\sigma^ {2}). \\ \end{array}
$$
Recent works in computer vision (Gong et al., 2020; Wang et al., 2021) have provided more formal connections between local gradient norm estimation and  $K$ -worst perturbations.
The text sentences in NLP are in the discrete space, which lacks the definition of local Lipschitz, but finding the worst perturbation in a local discrete set can still be a better measurement of local sensitivity in the semantic space.
Choice of Divergence We use the KL divergence as the primary measure of the statistical distance between the distribution of the original examples and that over augmented examples. We also empirically provide detailed analysis of the Jensen-Shannon Distance (JSD) (Endres and Schindelin, 2003) and  $\alpha$ -divergence (Minka et al., 2005) as a complementary measure in Section 5. The  $\alpha$ -divergence (Pillutla et al., 2021) is a general divergence family, which includes the most popular KL divergence and reverse KL divergence. Different value of  $\alpha$  makes the divergence trade-off between overestimation and underestimation. JSD is a metric function based on a mathematical definition which is symmetric and bounded within the range [0, 1]. These divergences are calculated as:
$$
\operatorname {K L} \left(\mathfrak {p} \| q\right) = \sum_ {i} p _ {i} (\boldsymbol {x}) \log \frac {p _ {i} (\boldsymbol {x})}{q _ {i} (\boldsymbol {x})},
$$
$$
\operatorname {J S D} (p \| q) = \sqrt {\frac {1}{2} (\operatorname {K L} (p \| m) + \operatorname {K L} (q \| m))}, \tag {4}
$$
$$
\mathbf {D} _ {\alpha} (p \| q) = \frac {1}{\alpha (\alpha - 1)} \sum_ {i} [ (\frac {p _ {i} (\boldsymbol {x})}{q _ {i} (\boldsymbol {x})}) ^ {\alpha} - 1 ],
$$
where  $p$  is the output probability distribution of an example,  $q$  is the output probability distribution of an augmented example, and  $m = \frac{1}{2} (p + q)$ .
Local Sensitivity and Informativeness The divergence objective exploits unlabeled data by measuring predictions across slightly-distorted versions of each unlabeled sample. The diverse and adversarial augmentations capture the local sensitivity and informativeness of inputs and project examples to the
decision boundary (Ducoffe and Precioso, 2018). Thus, the examples and their copies with highly inconsistent model predictions lie close to the decision boundary of the model (Gao et al., 2020). These examples are valuable to have human annotations because they 1) contain high-confidence region in a local perturbation and are therefore easy to train 2) are highly likely to promote the model with large-margin improvements (see example in Figure 2). Under our local sensitivity and hardness guided acquisition, we argue the selected examples would not be necessarily the examples with the highest uncertainty, which do not always benefit the training. For instance, an example may have low-confidence prediction of both original inputs and augmented inputs thus making the samples most hard to train.
2.4 More Details
Compute Distance We compute the divergence in the model predictive probabilities for the pairs of the input and its augmentations in Eqn (1). Specifically, we use a pretrained BERT in classification tasks and GPT-2 in prompt-based few-shot learning as the base model  $p_{\theta}$  to obtain the output probabilities for all unlabeled data points in  $D_{pool}$ . We then compute the divergence value with Eqn (1). Rank and Select Candidates We apply these steps to all candidate examples from  $D_{pool}$  and obtain the divergence value for each. Our acquisition function selects the top  $s_{acq}$  examples that have the highest divergence value from the acquired batch  $\mathcal{T}$ .
Table 1 shows the experimental data configuration. In classification tasks, we use five datasets, including Stanford Sentiment Treebank (SST-2; (Socher et al., 2013)), Internet Movie Database (IMDB; (Maas et al., 2011)), AG's News Corpus (AG News; (Zhang et al., 2015)), Quora Question Pairs (QQP; (Wang et al., 2018a)), and Question NLI (QNLI; (Wang et al., 2018a)). The validation and test splits are provided in Margatina et al. (2021). Following Desai and Durrett (2020), we test domain generalization and robustness on three challenging out-of-domain (OD) datasets. For sentiment analysis, SST-2 and IMDB are the source and target domains, respectively, and vice versa; for paraphrase detection, TwitterPPDB (Lan et al., 2019) serves as the out-of-domain test dataset for QQP.
In the prompt-based few-shot learning, we fol
Algorithm 1: Acquisition with Local Sensitivity and Hardness
1: Input: labeled data  $\mathcal{D}_{\text{label}}$ , unlabeled data  $\mathcal{D}_{\text{pool}}$ , acquisition size  $s_{\text{acq}}$ , model  $\mathcal{M}$  with output probability  $p_{\theta}(\cdot \mid \boldsymbol{x})$ .
2: while Select examples before reaching the budget do
3: for  $\pmb{x}$  in  $\mathcal{D}_{pool}$  do
4: Generate  $K$  augmentations,  $\{\pmb{x}_i^{\prime}\}_{i = 1,\dots ,K}\gets g(\pmb {x})$  //data paraphrasing via augmentation
5: Compute  $p_{\theta}(\cdot \mid \boldsymbol {x})$  and  $p_{\theta}(\cdot \mid \boldsymbol{x}_i^{\prime})$  for  $i = 1,\dots ,K$  //compute probabilities
6: Select the worst case augmentation  $\pmb{x}^{\prime}$  for each input  $\pmb{x}$  as  $\ell^{max}(\pmb{x}) = \max_{i=1,\dots,K} \ell(\pmb{x},\pmb{x}_i^{\prime})$ .
7: end for
8: Select top  $s_{acq}$  largest examples in  $\mathcal{D}_{pool}$ , according to the value of  $\mathbb{D}(p_{\theta}(\cdot \mid x), p_{\theta}(\cdot \mid x'))$ .
9: Label these  $s_{acq}$  examples.
0: end while
1: Curriculum learning the model parameters with Eqn (5).
Dataset	Train	Val	Test	OD Dataset
SST-2	60.6K	6.7K	871	IMDB
IMDB	22.5K	2.5K	25K	SST-2
AG News	11.4K	6K	7.6K	-
QNLI	99.5K	5.2K	5.5K	-
QQP	327K	36.4K	80.8K	TwitterPPDB
SST-2	60.6K	6.7K	871	-
TREC	4.5K	500	500	-
RTE	2.5K	277	3K	-
Table 1: Dataset Configuration. The top block is for the classification tasks and the bottom block is for the prompt-based few-shot learning. OD represents out-of-domain datasets.
low Zhao et al. (2021) to use SST-2 (Socher et al., 2013) for sentiment analysis, TREC (Voorhees and Tice, 2000) for question classification, and RTE (Dagan et al., 2005) for recognizing textual entailment. See Appendix A for more details of the data.
3.1 Classification Task
We compare the proposed ALLSH against four baseline methods. We choose these baselines as they cover a spectrum of acquisition functions (uncertainty, batch-mode, and diversity-based).
Random samples data from the pool of unlabeled data  $\mathcal{D}_{\text {pool }}$  following a uniform distribution.
Entropy selects  $s_{acq}$  sentences with the highest predictive entropy (Lewis and Gale, 1994) measured by  $-\sum_{\boldsymbol{x}} p_{\theta}(\boldsymbol{x}) \ln p_{\theta}(\boldsymbol{x})$ .
BADGE (Ash et al., 2020) acquires  $s_{acq}$  sentences based on diversity in loss gradient. The goal of BADGE is to sample a diverse and uncertain batch of points for training neural networks. It acquires data from  $\mathcal{D}_{pool}$  by first passing the input through the trained model and computing the gradient embedding with respect to the parameters of the model's last layer.
CAL (Margatina et al., 2021) The acquisition function samples contrastive examples. It uses information from the feature space to create neighborhoods for unlabeled examples, and uses predictive
likelihood for ranking the candidates.
3.2 Prompt-based Few-Shot Learning
Following Zhao et al. (2021), we adapt our acquisition function for state-of-the-art generation based model GPT-2 and propose to retrieve examples that are semantics and sensitivity aware to formulate its corresponding prompts. We compare ALLSH's acquisition function with random, contextual calibrated, and uncertainty prompt. For random prompt, we randomly select in-context examples from the training set for each test sentence. For Calibrated, Zhao et al. (2021) inject calibration parameters that cause the prediction for each test input to be uniform across answers. See Zhao et al. (2021) for more details. For Uncertain, we sample the highest uncertain prompt for the test sentences. For ALLSH, we augment the in-context examples and select the prompts with the highest divergence of the predicted likelihood between the original examples and their augmentations.
For classification, we use BERT-base (Devlin et al., 2018) from the HuggingFace library (Wolf et al., 2020). We train all models with batch size 16, learning rate  $2 \times 10^{-5}$ , and AdamW optimizer with epsilon  $1 \times 10^{-8}$ . For all datasets, we set the default annotation budget as  $1\%$ , the maximum annotation budget as  $15\%$ , initial accumulated labeled data set  $\mathcal{D}_{label}$  as  $0.1\%$  of the whole unlabeled data, and acquisition size as 50 instances for each active learning iterations, following prior work (e.g., Gissin and Shalev-Shwartz, 2019; Dor et al., 2020; Ru et al., 2020). Curriculum Learning (CL) We further combine our acquisition function with advances in semi-supervised learning (SSL) (Berthelot et al., 2019a; Sohn et al., 2020), which also integrates abundant unlabeled data into learning. A recent line of work in SSL utilizes data aug
mentations, such as TF-IDF and back-translation, to enforce local consistency of the model (Sajjadi et al., 2016; Miyato et al., 2018). Here SSL can further distill information from unlabeled data and gradually propagate label information from labeled examples to unlabeled one during the training stage (Xie et al., 2020; Zhang et al., 2021c). We construct the overall loss function as