SlowGuess/NAACL_2021_OCR huggingface dataset:
Active $^2$  Learning: Actively reducing redundancies in Active Learning methods for Sequence Tagging and Machine Translation
Rishi Hazra, Parag Dutta, Shubham Gupta, Mohammed Abdul Qaathir, Ambedkar Dukkipati
{rishihazra, parag Dutta, shubhamg,
mohammedq, ambedkar}@iisc.ac.in
While deep learning is a powerful tool for natural language processing (NLP) problems, successful solutions to these problems rely heavily on large amounts of annotated samples. However, manually annotating data is expensive and time-consuming. Active Learning (AL) strategies reduce the need for huge volumes of labeled data by iteratively selecting a small number of examples for manual annotation based on their estimated utility in training the given model. In this paper, we argue that since AL strategies choose examples independently, they may potentially select similar examples, all of which may not contribute significantly to the learning process. Our proposed approach, Active $^2$  Learning  $(\mathrm{A}^2\mathrm{L})$ , actively adapts to the deep learning model being trained to eliminate such redundant examples chosen by an AL strategy. We show that  $\mathrm{A}^2\mathrm{L}$  is widely applicable by using it in conjunction with several different AL strategies and NLP tasks. We empirically demonstrate that the proposed approach is further able to reduce the data requirements of state-of-the-art AL strategies by  $\approx 3 - 25\%$  on an absolute scale on multiple NLP tasks while achieving the same performance with virtually no additional computation overhead.
Active Learning (AL) (Freund et al., 1997; McCallum and Nigam, 1998) reduces the need for large quantities of labeled data by intelligently selecting unlabeled examples for expert annotation in an iterative process. Many Natural Language Processing (NLP) tasks like sequence tagging (NER, POS) and Neural Machine Translation (NMT) are very data-intensive and require a meticulous, time-consuming, and costly annotation process. On the other hand, unlabeled data is practically unlimited. Due to this, many researchers have explored applications of active learning for NLP (Thompson et al., 1999; Figueroa et al., 2012). A general AL method
proceeds as follows: (i) The partially trained model for a given task is used to (possibly incorrectly) annotate the unlabeled examples. (ii) An active learning strategy selects a subset of the newly labeled examples via a criterion that quantifies the perceived utility of examples in training the model. (iii) The experts verify/improve the annotations for the selected examples. (iv) These examples are added to the training set, and the process repeats. AL strategies differ in the criterion used in step (ii).
We claim that all AL strategies select redundant examples in step (ii). If one example satisfies the selection criterion, then many other similar examples will also satisfy it (see the next paragraph for details). As the examples are selected independently, AL strategies redundantly choose all of these examples even though, in practice, it is enough to label only a few of them (ideally just one) for training the model. This leads to higher annotation costs, wastage of resources, and reduces the effectiveness of AL strategies. This paper addresses this problem by proposing a new approach called  $\mathrm{A}^2\mathrm{L}$  (read as active-squared learning) that further reduces the redundancies of existing AL strategies.
Any approach for eliminating redundant examples must have the following qualities: (i) The redundancy should be evaluated in the context of the trained model. (ii) The approach should apply to a wide variety of commonly used models in NLP. (iii) It should be compatible with several existing AL strategies. The first point merits more explanation. As a model is trained, depending on the downstream task, it learns to focus on certain properties of the input. Examples that share these properties (for instance, the sentence structure) are similar from the model's perspective. If the model is confused about one such example, it will likely be confused about all of them. We refer to a similarity measure that is computed in the context of a model as a model-aware similarity (Section 3.1).
Contributions: (i) We propose a Siamese twin
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1982-1995 June 6-11, 2021. ©2021 Association for Computational Linguistics
(Bromley et al., 1994; Mueller and Thyagarajan, 2016) based method for computing model-aware similarity to eliminate redundant examples chosen by an AL strategy. This Siamese network actively adapts itself to the underlying model as the training progresses. We then use clustering based on similarity scores to eliminate redundant examples. (ii) We develop a second, computationally more efficient approach that approximates the first one with a minimal drop in performance by avoiding the clustering step. Both of these approaches have the desirable properties mentioned above. (iii) We experiment with several AL strategies and NLP tasks to empirically demonstrate that our approaches are widely applicable and significantly reduce the data requirements of existing AL strategies while achieving the same performance. To the best of our knowledge, we are the first to identify the importance of model-aware similarity and exploit it to address the problem of redundancy in AL.
Active learning has a long and successful history in the field of machine learning (Dasgupta et al., 2009; Awasthi et al., 2017). However, as the learning models have become more complex, especially with the advent of deep learning, the known theoretical results for active learning are no longer applicable (Shen et al., 2018). This has prompted a diverse range of heuristics to adapt the active learning framework to deep learning models (Shen et al., 2018). Many AL strategies have been proposed (Sha and Saul, 2007; Haffari et al., 2009; Bloodgood and Callison-Burch, 2010; Blundell et al., 2015; Gal and Ghahramani, 2016a), however, since they choose the examples independently, the problem of redundancy (Section 1) applies to all.
We experiment with various NLP tasks like named entity recognition (NER) (Nadeau and Sekine, 2007), part-of-speech tagging (POS) (Marcus et al., 1993), neural machine translation (NMT) (Hutchins, 2004; Nepveu et al., 2004; Bahdanau et al., 2014; Cho et al., 2014; Sutskever et al., 2014; Ortiz-Martínez, 2016) and so on (Landes et al., 1998; Tjong Kim Sang and Buchholz, 2000). The tasks chosen by us form the backbone of many practical problems and are known to be computationally expensive during both training and inference. Many deep learning models have recently advanced the state-of-art for these tasks (Bahdanau et al., 2014; Lample et al., 2016; Siddhant and Lip
ton, 2018). Our proposed approach is compatible with any NLP model, provided it supports the usage of an AL strategy.
Existing approaches have used model-independent similarity scores to promote diversity in the chosen examples. For instance, in Chen et al. (2015), the authors use cosine similarity to pre-calculate pairwise similarity between examples. We instead argue in favor of model-aware similarity scores and learn an expressive notion of similarity using neural networks. We compare our approach with a modified version of this baseline using cosine similarity on Inferent embeddings (Conneau et al., 2017).
We use  $\mathcal{M}$  to denote the model being trained for a given task.  $\mathcal{M}$  has a module called encoder for encoding the input sentences. For instance, the encoder in  $\mathcal{M}$  may be modeled by an LSTM (Hochreiter and Schmidhuber, 1997).
3.1 Model-Aware Similarity Computation
A measure of similarity between examples is required to discover redundancy. The simplest solution is to compute the cosine similarity between input sentences (Chen et al., 2015; Shen et al., 2018) using, for instance, the InferSent encodings (Connieu et al., 2017). However, sentences that have a low cosine similarity may still be similar in the context of the downstream task. Model  $\mathcal{M}$  has no incentive to distinguish among such examples. A good strategy is to label a diverse set of sentences from the perspective of the model. For example, it is unnecessary to label sentences that use different verb forms but are otherwise similar if the task is agnostic to the tense of the sentence. A straightforward extension of cosine similarity to the encodings generated by model  $\mathcal{M}$  achieves this. However, a simplistic approach like this would likely be incapable of discovering complex similarity patterns in the data. Next, we describe two approaches that use more expressive model-aware similarity measures.
3.2 Model-Aware Siamese
In this approach, we use a Siamese twin's network (Bromley et al., 1994) to compute the pairwise similarity between encodings obtained from model  $\mathcal{M}$ . A Siamese twin's network consists of an encoder (called the Siamese encoder) that feeds on the output of model  $\mathcal{M}$ 's encoder. The outputs of the
Siamese encoder are used for computing the similarity between each pair of examples  $a$  and  $b$  as:
$$
\sin (a, b) = \exp \left(- \| \mathbf {o} ^ {a} - \mathbf {o} ^ {b} \| _ {2}\right), \tag {1}
$$
where  $\mathbf{o}^a$  and  $\mathbf{o}^b$  are the outputs of the Siamese encoder for sentences  $a$  and  $b$  respectively. Let  $N$  denote the number of examples chosen by an AL strategy. We use the Siamese network to compute the entries of an  $N\times N$  similarity matrix  $\mathbf{S}$  where the entry  $S_{ab} = \mathrm{sim}(a,b)$ . We then use the spectral clustering algorithm (Ng et al., 2002) on the similarity matrix  $\mathbf{S}$  to group similar examples. A fixed number of examples from each cluster are added to the training dataset after annotation by experts.
We train the Siamese encoder to predict the similarity between sentences from the SICK (Sentences Involving Compositional Knowledge) dataset (Marelli et al., 2014) using mean squared error. This dataset contains pairs of sentences with manually annotated similarity scores. The sentences are encoded using the encoder in  $\mathcal{M}$  and then passed on to the Siamese encoder for computing similarities. The encoder in  $\mathcal{M}$  is kept fixed while training the Siamese encoder. The trained Siamese encoder is then used for computing simi
larity between sentences selected by an AL strategy for the given NLP task as described above. As  $\mathcal{M}$  is trained over time, the distribution of its encoder output changes, and hence we periodically retrain the Siamese network to sustain its model-awareness.
The number of clusters and the number of examples drawn from each cluster are user-specified hyper-parameters. The similarity computation can be done efficiently by computing the output of the Siamese encoder for all  $N$  examples before evaluating equation 1, instead of running the Siamese encoder  $O(N^2)$  times. The clustering algorithm runs in  $O(N^3)$  time. For an AL strategy to be useful, it should select a small number of examples to benefit from interactive and intelligent labeling. We expect  $N$  to be small for most practical problems, in which case the computational complexity added by our approach would only be a small fraction of the overall computational complexity of training the model with active learning (see Figure 1).
3.3 Integrated Clustering Model
While the approach described in Section 3.2 works well for small to moderate values of  $N$ , it suffers from a computational bottleneck when  $N$  is large. We integrate the clustering step into the similarity computation step to remedy this (see Figure 1) and call the resultant approach as Integrated Clustering Model (Int Model). Here, the output of model  $\mathcal{M}$ 's encoder is fed to a clustering neural network  $\mathcal{C}$  that has  $K$  output units with the softmax activation function. These units correspond to the  $K$  clusters, and each example is directly assigned to one of the clusters based on the softmax output.
To train the network  $\mathcal{C}$ , we choose a pair of similar examples (say  $a$  and  $b$ ) and randomly select a negative example (say  $c$ ). We experimented with both SICK and Quora Pairs dataset<sup>3</sup>. All examples are encoded via the encoder of model  $\mathcal{M}$  and then passed to network  $\mathcal{C}$ . The unit with the highest probability value for  $a$  is treated as the ground-truth class for  $b$ . Minimizing the objective given below maximizes the probability of  $b$  belonging to its ground truth class while minimizing the probability of  $c$  belonging to the same class:
$$
\begin{array}{l} \mathcal {L} (a, b, c) = - \lambda_ {1} \log p _ {i _ {a}} ^ {b} - \lambda_ {2} \log \left(1 - p _ {i _ {a}} ^ {c}\right) \\ + \lambda_ {3} \sum_ {k = 1} ^ {K} p _ {k} ^ {b} \log p _ {k} ^ {b}. \tag {2} \\ \end{array}
$$
Here  $\lambda_1, \lambda_2$ , and  $\lambda_3$  are user-specified hyperparameters,  $p_j^x$  is the softmax output of the  $j^{th}$  unit
for example  $x, j = 1,2,\ldots ,K$ ,  $x = a,b,c$ , and  $i_{a} = \arg \max_{j\in \{1,2,\dots K\}}p_{j}^{a}$ . The third term encourages the utilization of all the  $\mathbf{K}$  units across examples in the dataset. As before, a trained network  $\mathcal{C}$  is used for clustering examples chosen by an AL strategy, and we select a fixed number of examples from each cluster for manual annotation.
It is important to note that: (i) These methods are not AL strategies. Rather, they can be used in conjunction with any existing AL strategy. Moreover, given a suitable Siamese encoder or clustering network  $\mathcal{C}$ , they apply to any model  $\mathcal{M}$ . (ii) Our methods compute model-aware similarity since the input to the Siamese or the clustering network is encoded using the model  $\mathcal{M}$ . The proposed networks also adapt to the underlying model as the training progresses. Algorithm 1 describes our general approach called Active Learning.
We establish the effectiveness of our approaches by demonstrating that they: (i) work well across a variety of NLP tasks and models, (ii) are compatible with several popular AL strategies, and (iii) further reduce the data requirements of existing AL strategies, while achieving the same performance. In particular, we experiment<sup>1</sup> with two broad categories of NLP tasks: (a) Sequence Tagging (b) Neural Machine Translation. Table 1 lists these tasks and information about the corresponding datasets (including the two auxiliary datasets for training the Siamese network (Section 3.2)) used in our experiments. We begin by describing the AL strategies for the two kinds of NLP tasks.
4.1 Active Learning Strategies for Sequence Tagging
Margin-based strategy: Let  $s(\mathbf{y}) = \mathrm{P}_{\theta}(\mathbf{Y} = \mathbf{y}|\mathbf{X} = \mathbf{x})$  be the score assigned by a model  $\mathcal{M}$  with parameters  $\theta$  to output  $\mathbf{y}$  for a given example  $\mathbf{x}$ . Margin is defined as the difference in scores obtained by the best scoring output  $\mathbf{y}$  and the second best scoring output  $\mathbf{y}'$ , i.e.:
$$
\mathrm {M} _ {\text {m a r g i n}} = \max  _ {\mathbf {y}} s (\mathbf {y}) - \max  _ {\mathbf {y} ^ {\prime} \neq \mathbf {y} _ {\max }} s \left(\mathbf {y} ^ {\prime}\right), \tag {3}
$$
where,  $\mathbf{y}_{\mathrm{max}} = \arg \max_{\mathbf{y}}s(\mathbf{y})$  . The strategy selects examples for which  $\mathrm{M}_{\mathrm{margin}}\leq \tau_1$  , where  $\tau_{1}$  is a hyper-parameter. We use Viterbi's algorithm (Ryan and Nudd, 1993) to compute the scores  $s(\mathbf{y})$
Entropy-based strategy: All the NLP tasks that we consider require the model  $\mathcal{M}$  to produce an output for each token in the sentence. Let  $\mathbf{x}$  be an input sentence that contains  $n(\mathbf{x})$  tokens and define  $\bar{s}_j = \max_{o\in \mathcal{O}}\mathrm{P}_{\theta}(y_j = o|\mathbf{X} = \mathbf{x})$  to be the probability of the most likely output for the  $j^{th}$  token in  $\mathbf{x}$ . Here  $\mathcal{O}$  is set of all possible outputs and  $y_{j}$  is the output corresponding to the  $j^{th}$  token in  $\mathbf{x}$ . We define the normalized entropy score as:
$$
\mathrm {M} _ {\text {e n t r o p y}} = - \frac {1}{n (\mathbf {x})} \sum_ {j = 1} ^ {n (\mathbf {x})} \bar {s} _ {j} (\mathbf {y}) \log \bar {s} _ {j} (\mathbf {y}). \tag {4}
$$
A length normalization  $n(\mathbf{x})$  is added to avoid bias due to the example length as it may be undesirable to annotate longer length examples (Claveau and Kijak, 2017). The strategy selects examples with  $\mathrm{M}_{\text{entropy}} \geq \tau_2$ , where  $\tau_2$  is a hyper-parameter.
Bayesian Active Learning by Disagreement (BALD): Due to stochasticity, models that use dropout (Srivastava et al., 2014) produce a different output each time they are executed. BALD (Houlsby et al., 2011) exploits this variability in the predicted output to compute model uncertainty. Let  $\mathbf{y}^{(t)}$  denote the best scoring output for  $\mathbf{x}$  in the  $t^{th}$  forward pass, and let  $N$  be the number of forward passes with a fixed dropout rate, then:
$$
\mathrm {M} _ {\text {b a l d}} = 1 - \frac {\operatorname {c o u n t} \left(\operatorname {m o d e} \left(\mathbf {y} ^ {(1)} , \dots , \mathbf {y} ^ {(N)}\right)\right)}{N}. \tag {5}
$$
Here the mode(.) operation finds the output which is repeated most often among  $\mathbf{y}^{(1)},\ldots ,\mathbf{y}^{(N)}$ , and the count(. ) operation counts the number of times this output was encountered. This strategy selects examples with  $\mathrm{M}_{\mathrm{bald}}\geq \tau_3$  (hyper-parameter).
4.2 Active Learning Strategies for Neural Machine Translation $^{2}$
Least Confidence (LC) This strategy estimates the uncertainty of a trained model on a source sentence  $\mathbf{x}$  by calculating the conditional probability of the prediction  $\hat{\mathbf{y}}$  conditioned on the source sentence(Lewis and Catlett, 1994).
$$
\mathrm {M} _ {\mathrm {L C}} = \frac {1}{\mathrm {n} (\hat {\mathbf {y}})} \log \mathbf {P} (\hat {\mathbf {y}} | \mathbf {x}) \tag {6}
$$
A length normalization of  $\mathrm{n}(\hat{\mathbf{y}})$  (length of the predicted translation  $\hat{\mathbf{y}}$ ) is added.
2The AL strategies for NMT are ranking-based techniques, so we select the top  $50\%$  of the candidates after sorting them in ascending (6,7) or descending (9) order.
<sup>1</sup>Codes for the experiments are available at the following github link: https://github.com/parag1604/A2L.
Task	Dataset	#Train/#Test	Example (Input/Output)
NER	CoNLL 2003	14987 / 3584	Fischler proposed EU measures after reports from Britain B-PER 0 B-MISC 0 0 0 0 B-LOC
POS	CoNLL 2003	14987 / 3584	He ended the World Cup on the wrong note PRP VBD DT NNP NNP IN DT JJ NN
CHUNK	CoNLL 2000	8936 / 2012	The dollar posted gains in quiet trading B-NP I-NP B-VP B-NP B-PP B-NP I-NP
SEMTR	SEMCOR2	13851 / 4696	This section prevents the military departments 0 Mental Agentive 0 0 Object
NMT	Europarl (en → es)	100000 / 29155	(1) that is almost a personal record for me this autumn! (2) es la mejormarca que he alcanzado este otono .
AUX	SICK	9000/1000	(1) Two dogs are fighting. (2) Two dogs are wrestling and hugging. Similarity Score: 4 (out of 5)
AUX	Quora Pairs3	16000 / 1000 (sets)5	(1) How do I make friends? (2) How to make friends? Label: 1
Table 1: Task and dataset descriptions. AUX is the task of training the Siamese network (Section 3.2) or Integrated network  $\mathcal{C}$  (Section 3.3). Citations: CoNLL 2003 (Sang and De Meulder, 2003), CoNLL 2000 (Tjong Kim Sang and Buchholz, 2000), SEMCOR $^3$ , Europarl (Koehn, 2005), SICK (Marelli et al., 2014), Quora Pairs $^4$ .
Coverage Sampling (CS) A translation model is said to cover the source sentence if it translates all of its tokens. Coverage is estimated by mapping a particular source token to its appropriate target token, without which the model may suffer from under-translation or over-translation issues (Tu et al., 2016). Peris and Casacuberta (2018) proposed to use translation coverage as a measure of uncertainty by:
$$
\mathrm {M} _ {\mathrm {C S}} = \frac {\sum_ {\mathrm {j} = 1} ^ {n (\mathbf {x})} \log \left(\min \left(\sum_ {\mathrm {i} = 1} ^ {n (\hat {\mathbf {y}})} \alpha_ {\mathrm {i} , \mathrm {j}} , 1\right)\right)}{n (\mathbf {x})} \tag {7}
$$
Here  $\alpha_{\mathrm{i,j}}$  denotes the attention probability calculated by the model for the  $j^{th}$  source word in predicting the  $i^{th}$  target word. It can be noted that the coverage score will be 0 for samples for which the model almost fully covers the source sentences.
Attention Distraction Sampling (ADS) Peris and Casacuberta (2018) claimed that in translating an uncertain sample, the model's attention mechanism would be distracted (dispersed throughout the sentence). Such samples yield attention probability distribution with light tails (e.g., uniform distribution), which can be obtained by taking the Kurtosis of the attention weights for each target token  $\mathbf{y}_i$ .
$$
\operatorname {K u r t} (\mathbf {y} _ {\mathrm {i}}) = \frac {\frac {1}{n (\mathbf {x})} \sum_ {\mathrm {j} = 1} ^ {n (\mathbf {x})} \left(\alpha_ {\mathrm {i} , \mathrm {j}} - \frac {1}{n (\mathbf {x})}\right) ^ {4}}{\left(\frac {1}{n (\mathbf {x})} \left(\sum_ {\mathrm {j} = 1} ^ {n (\mathbf {x})} \alpha_ {\mathrm {i} , \mathrm {j}} - \frac {1}{n (\mathbf {x})}\right)\right) ^ {2}} \tag {8}
$$
where  $\frac{1}{n(\mathbf{x})}$  is the mean of the distribution of the attention weights (for a target word) over the source words. The kurtosis value will be lower for distributions with light tails, so the average of the negative
Figure 1: Comparison of time taken for one data selection step in NMT task by the Model Aware (MA) Siamese and Integrated Clustering (Int) Model across different ALS. It can be observed that  $\mathrm{A}^2\mathrm{L}$  adds a negligible overhead ( $\approx \frac{1}{12}$  of the time taken for ALS) to the overall process.
kurtosis values for all words in the target sentence is used as the distraction score.
$$
\mathrm {M} _ {\mathrm {A D S}} = \frac {\sum_ {\mathrm {i} = 1} ^ {n (\mathbf {y})} - \operatorname {K u r t} \left(\mathbf {y} _ {\mathrm {i}}\right)}{n (\mathbf {y})} \tag {9}
$$
4.3 Details about Training
For sequence tagging, we use two kinds of architectures: CNN-BiLSTM-CRF model (CNN for character-level encoding and BiLSTM for word-level encoding) and a BiLSTM-BiLSTM-CRF model (BiLSTM for both character-level and word-level encoding) (Lample et al. (2016); Siddhant
<sup>3</sup>From a subset of the Brown Corpus (Burchfield, 1985), using splits from Martínez Alonso and Plank (2017)
4https://www.quora.com/q/quoradata/First-QuoraDataset-Release-Question-Pairs
Task	Dataset	% of train data used to reach 99% of full-data F-Score	% less data required to reach 99% of full-data F-score
POS	CoNLL 2003	25%	16%
NER	CoNLL 2003	37%	3%
SEMTR	SEMCOR	35%	25%
CHUNK	CoNLL 2000	23%	11%
Table 2: Fraction of data used for reaching full dataset performance and the corresponding absolute percentage reduction in the data required over the None baseline that uses active learning strategy without the  $A^2 L$  step for the best AL strategy (BALD in all cases). Refer Fig 8 in Appendix for CHUNK plots.
and Lipton (2018)). For the translation task, we use LSTM based encoder-decoder architecture with Bahdanau attention (Bahdanau et al., 2014). These models were chosen for their performance and ease of implementation.
The Siamese network used for model-aware similarity computation (Section 3.2) consists of two bidirectional LSTM (BiLSTM) encoders. We pass each sentence in the pair from the SICK dataset to model  $\mathcal{M}$  and feed the resulting encodings to the Siamese BiLSTM encoder. The output is a concatenation of terminal hidden states of the forward and backward LSTMs, which is used to compute the similarity score using (1). As noted before, we keep model  $\mathcal{M}$  fixed while training the Siamese encoders and use the trained Siamese encoders for computing similarity between examples chosen by an AL strategy. We maintain the model-awareness by retraining the Siamese after every 10 iterations.